[RFC] Introduce LightningFetcher and BatchProcessor

🚀 Feature

Provide a set of building blocks and APIs for PyTorch Lightning users to manipulate fetching and data based to the the LightningModule training_step.
Motivation

Currently, several features being developed within Lightning requires a formalisation on how data are being fetched such as Fault Tolerant Training to properly handle iterators state checkpointing, iter-batch-parellism and refactors within the Loops.
Furthermore, there is a growing need to have a higher control on the data being pass to the training_step.
Concrete goals for this proposal:

Provide a composable blocks for users to control the flow of their fetching and data passed to the training_step
Support fault tolerant training with progress tracking in the case iterator is directly passed the training_step.

API Design proposal.
The LightningFetcher would provide inter-batch-parallism with pre-fetching=1 for its default behaviour.
Further in the future, the LightningFetcher could provide an auto mode by default which adapts to the user model / data and auto-tune the fetching using inter-batch-parallism to maximise through-put.
The code below is a pseudo-code and likely to be modified.
class BachProcessor:
    def __init__(self, trainer):
        self.trainer = trainer

    @property
    def should_fetch(self):
        return True

    def __call__(self, fetcher, batch):
        return batch

class TBPTTBachProcessor:
    def __init__(self, trainer):
        self.trainer = trainer

    @property
    def should_fetch(self):
        return True

    def __call__(self, fetcher, batch):
        return self.trainer.lightning_module.tbptt_split_batch(batch)

class DataLoaderIterBachProcessor:
    def __init__(self, trainer):
        self.trainer = trainer

    @property
    def should_fetch(self):
        return False

    def __call__(self, fetcher, batch):
        return fetcher.iterator

class LightningFetcher

    def __init__(self, prefetch_batches, batch_processor=...):
        ...

    def setup(self, dataloader, batch_to_device: Callable, profiler: ...):
        ...

def select_batch_processor(trainer, user_batch_processor):

    if user_batch_processor:
        return user_batch_processor

    if "dataloader_iter" in signature(trainer.lightning_module.training_step):
        return DataLoaderIterBachProcessor(...)

    elif trainer.lightning_module.truncated_bptt_steps > 1:
        return TBPTTBachProcessor(...)

    return BachProcessor()

fetcher = LightningFetcher(prefetch_batches=1, batch_processor=BachProcessor())
iterator = iter(fetcher)
batch, is_last, batch_idx = next(fetcher)

# user facing API.
Trainer(prefetch_batches=OPTIONS[0, 1, ..., "auto", LightningFetcher])

Pitch

Alternatives

Additional context

If you enjoy Lightning, check out our other projects! ⚡

Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

Dear @ananthsub @yifuwang, mind sharing your thoughts ?

Hi @carmocca , @tchaton , I'm an eng. in @ananthsub 's team. Trying to understand this proposal better. I have been looking at the stack of diffs, and I'm having a hard time to understand what is the value proposal of AbstractDataFetcher beyond the regular Iterable (or even Iterator) interface in python.
I understand the value of providing pre-fetching iterators and state saving iterators, but those are implementations of iterators, not requiring any additional interfaces on top of iterator itself. Also, iterators are composable so I believe it would be possible to implement these functionalities in a composable way.
Am I missing anything here?

Hey @aazzolini,
Thanks for reaching out !
The AbstractDataFetcher  is implementing all the Fault Tolerant Logic related to the data fetching which you can find in setup / _apply_patch.
The DataFetcher is the interface already implementing the preventing logic with some hooks to be overriden if required.
Right now, they are tailored to InterBatchParallelDataFetcher.
Finally, under the request from @yifuwang, we added support for a fault tolerant DataLoaderIterDataFetcher subclassing directly AbstractDataFetcher with allowed us to remove the InterBatchProcessor introduced in https://github.com/PyTorchLightning/pytorch-lightning/issues/8316.
The API is in its V0 form and we plan to improve it with more use-case before exposing this to the users.
If you are interested in improving it, feel free to come with a proposal and a PoC.
Best,
T.C


ORA 28040 No matching Authentication Protocol

docker run -d -p 1521:1521 -e ORACLE_PASSWORD=123456 -v oracle-volume:/opt/oracle/oradata -v /Users/jpsl/DosPlayas-backup:/opt/backup gvenzl/oracle-xe \n
Already tried setting env variables to lower version but the error persists:
-e SQLNET.ALLOWED_LOGON_VERSION_CLIENT=8
-e SQLNET.ALLOWED_LOGON_VERSION_SERVER=8
how do I fix this?

Hi @admin-relu,
Thanks for using these images!
Both of these parameters are, as per Oracle documentation, to be set in the sqlnet.ora file, and not as environment variables.
You can find the sqlnet.ora file under <your volume>/dbconfig/XE/sqlnet.ora


Atom Feed

Impelement atom feed, for now i think to use https://github.com/hexojs/hexo-generator-feed
<link rel="alternate" type="application/atom+xml" title="m0k1.org blog Atom feed" href="/feed.atom">

Becouse of moving to Angular #15 i need to find out a new way to generate Atom feed.


Minor CMake issue?

I'm getting an error in CMake while building. It says it requires "language dialect C89" but that CMake doesn't know the compile flags for it.
As far as I can tell CMake only supports C90, C99 and C11 for CMAKE_C_STANDARD.Is it ok if I change it to C99? C89 and 90 apparently don't support single line '//' comments and I saw a whole bunch of those in the project.
While I'm at it, I wanted to add an option to build shared libs for CMake-gui and a pkg-config file. Any interest?

I changed it to 99, but I think the intention was to set it to 90 instead of 99.
See the latest change to the main CMake script.


can 't run at win 10,could you help me?

Windows 10 isn't linux. You may wish to look into https://store.steampowered.com/app/431960/Wallpaper_Engine/ for windows

Since the core functionality is handled by SDL2 which is cross-platform, I made a Windows port of this project. If you want to, have a look: https://github.com/nwxnk/animwal

Give this one a shot too:
https://github.com/TrAyZeN/sdl-wallpaper

https://github.com/TrAyZeN/sdl-wallpaper ya try this it is good

after cmake . , how could I use it?


[1pt] Switch badges for Missions - Conditional components

Missions that the user has joined already should show a badge "Active Member" instead of the default "NOT A MEMBER" and a button "Leave Mission" instead of the "Join Mission" button (as per design).
Use the React conditional rendering syntax:
{rocket.reserved && (
    // render Cancel Rocket button
)}

Hi @fobadara this task is done you can review it here https://github.com/fobadara/rocket-space-missions/pull/36


Add an ability to replay and loop FIFO data

This update adds 2 write bits to VERA registers: AUDIO_CTRL bit 6 and AUDIO_RATE bit 7.
Writing 1 to AUDIO_CTRL bit 6 will only reset FIFO's read position to 0. This allows a sample data in the buffer to be replayed any time without refilling it again. AUDIO_RATE bit 7 set will enable looped playback where the read position is automatically reset to 0 if the buffer becomes empty in the next sample. Although an AUDIO_RATE value of 128 will still play at the maximum rate without looping.
Note that a position of 0 here means an internal buffer's address. Which means a FIFO reset (write 1 to AUDIO_CTRL bit 7) is required in order to reset the write position to 0 too before uploading a sample data and make these methods above play correctly.

I have tried to include these changes into the fx-branch. After doing so I found out a few things:

The change raised the LUT count by 46, which is more than you would expect.
Attempts to reduce the LUT-count were partially successful (down to 16 LUTs) but resulted in a different way of interfacing with the looping/reset settings, which may be undesired.
The change resets rdidx_r (in audio_fifo.v) when a restart of the read position is triggered. However, the state_r (in pcm.v) is not reset. This means that the fifo-reading might be in an intermediate state (say: 8-bit stereo, and only the left byte has been read, but not the right one) when the fifo read position is reset. This may cause strange/unwanted effects.
It is probably required to wait for the state to be DONE or IDLE before executing a read-reset. This means some logic needs to be added and/or adjusted.

It is probably prudent to re-think how the logic should work and how the interface should work for this feature. Then implement that. Then LUT-optimization (and review) can be performed.
JeffreyH

I updated the loop method to be a combination of AUDIO_CTRL bits 6-7 now. Also, I made it reset only in IDLE state.

I accidentally merged this one.  If we need to revert, we can merge #19

This merge has been reverted after discussions on Discord.  JeffreyH has some concerns about potential timing problems in this implementation that might unfortunately be costly in LUTs to fix, and it blocks the potential merge of fx into main.  Please rework this feature against the fx branch and open a new PR.


SW-5047 Implement top bar button for 'remove global roles' into Accelerator->People table

Make the TableWithSearchFiltersProps extend the OrderPreservedTablePropsFull so we can pass more props through the underlying table without identifying them individually
Implement the "remove access" top bar button that will remove global roles for selected users in the Accelerator->People table
Need to build BE API for this, so the calls will come later.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#2341  👈
#2331
#2330
#2329
#2328
#2327
#2326
#2333
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @nickgraz and the rest of your teammates on  Graphite

Merge activity

Mar 19, 11:32 AM EDT: @nickgraz started a stack merge that includes this pull request via Graphite.


Prefix all env vars with TT_UMD

Prefix all env vars that can be used in the cde with TT_UMD, similar to how tt-metal env vars are prefixed with TT_METAL. This issue requires investigating which clients use these env vars in order to not break anything. For example, look at the comment https://github.com/tenstorrent/tt-umd/pull/130#discussion_r1793119760

I would suggest also to document them all in a single place, so that it is easy to understand how to change the runtime behavior of a UMD-based application without grepping the code for getenv.


Peek error: Use of deprecated "Deno.run()" API

Describe the bug
Showing error in nvim-notify after closed the browser. i have try close peek with PeekClose and exit the browser when is running, both of them give me notify error.
To Reproduce

Open neovim and markdown file
Run command PeekOpen
After the browser open and run, close with PeekClose or exit the browser

Expected behavior
Not showing notify error after closed
Screenshots

Desktop (please complete the following information):

OS: Endeavour
nvim --version:

NVIM v0.10.0-dev-2110+g2fce95ec4
Build type: RelWithDebInfo
LuaJIT 2.1.1703358377

deno --version:

deno 1.40.1 (release, x86_64-unknown-linux-gnu)
v8 12.1.285.6
typescript 5.3.3

plugin setup:

return
{
    "toppair/peek.nvim",
    event = { "VeryLazy" },
    build = "deno task --quiet build:fast",
    config = function()
        require("peek").setup({
            app = {
                'chromium',
                '--new-window'
            }
        })
        -- refer to `configuration to change defaults`
        vim.api.nvim_create_user_command("PeekOpen", require("peek").open, {})
        vim.api.nvim_create_user_command("PeekClose", require("peek").close, {})
    end,
}

Additional context
Sorry i don't know how to debug and get full log. so i just copied from command :Notifications based nvim-notify.
message = { 'Peek error: warning: Use of deprecated "Deno.run()" API. This API will be removed in Deno 2.', "", "Stack trace:", "  at open (file:///home/aquila/.l
ocal/share/nvim/lazy/peek.nvim/public/main.bundle.js:1537:29)", "  at file:///home/aquila/.local/share/nvim/lazy/peek.nvim/public/main.bundle.js:107268:5", "  at file
:///home/aquila/.local/share/nvim/lazy/peek.nvim/public/main.bundle.js:107276:3", "", 'hint: Use "Deno.Command()" API instead.', "", 'warning: Use of deprecated "Deno
.serveHttp()" API. This API will be removed in Deno 2.', "", "Stack trace:", "  at file:///home/aquila/.local/share/nvim/lazy/peek.nvim/public/main.bundle.js:107237:3
5", "", "hint: Use `Deno.serve()` instead.", "", "" },

@rizkyilhampra did you figure something out to surpress that warning?


Use msg.topic to set the variable name

The node would be significantly enhanced if the variable name could be set from the TOPIC passed in via msg.
See: https://groups.google.com/forum/#!topic/node-red/YOs4F_Jcajo

Now updated and bumped to v2
Much simpler to use and more consistent.


Search clears input element value with ngModel

I have implemented the highlight functionality. I have a div with huge content with a form as well containing fields where ngModel is wrapped.
When I search, all the form inputs gets cleared though ngModel value remains intact and all I see is blank form.

Hi, I encountered the same issue with my toggle components which use @Input for content, did you find any fix for this ? Thanks in advance

@code-vagabond
Unfortunately, it didn't work for input fields. because it tries to add  tag around search term so it breaks the Input fields.
I have solved this by not adding search on inputs, you can search on other fields by adding multiple []highlight]="searchTerms".


Feature Request: Export Keras Model

Issue Description
Currently we can import a model from a HDF5 Keras model file. It would be helpful if we can also do the other way around - exporting a DL4J MultiLayerNetwork or ComputationGraph to a HDF5 Keras-compatible file.
Version Information
1.0.0-M2

@bnnthang we have the apis in place for that for onnx and tf but not h5...I'd have to think on that one a bit. Most people train models in python and just want to deploy them in java (at least for most of our use cases) - what's the main use case here? People who like java wanting to hand off models to a data science team that mainly uses python? I don't see that a lot in the industry.

So I have a trained Keras model, deploy it to Android devices, and do on-device training using DL4J. Later on I want to take those trained on device models and train in Python with another dataset different than the dataset I first trained the model with.


Fix issue 7

Adds the "p-note" class name to satisfy the indiewebify validate-h-card test.

Yes, of course. My bad for not switching back branches before continuing.

I want to add tests for h-card and h-feed parsing before taking this one, as there is some trickiness between post and Index pages at the moment that I want to be more comfortable about.


Replace raster image resources with vector drawables

Actual Behaviour
Currently, the application uses png icons which are both heavy in size and blurry in vision.
Expected Behaviour
Since now, through AppCompat, Android supports Vector Drawables back to API 9, we should update the project with them
Would you like to work on the issue?
Yes, I would like to work on the issue.
@the-dagger

Sounds good, please go ahead and send a pull request.

Please take a look @the-dagger


Secure cookies

Gitblit already set the "secure" flag for cookies, i.e. send them only over secured transports, as well as 'httpOnly' flag for the Jetty session cookies already.
With this PR it will also do so for the user authentication cookies.
It will also set the 'secure' flag when HTTP is redirected to HTTPS by Gitblit.

:+1:

It fails to compile:
compile:
  [mx:keys] =========================================================
  [mx:keys] mx:Keys  (com.gitblit.Keys)
  [mx:keys] ---------------------------------------------------------
  [mx:keys]    input: /data/src/gitblit/src/main/distrib/data/defaults.properties
  [mx:keys]    output: /data/src/gitblit/src/main/java/com/gitblit/Keys.java
 [mx:javac] =========================================================
 [mx:javac] mx:Javac  (com.gitblit:gitblit:1.8.1-SNAPSHOT, compile)
 [mx:javac] ---------------------------------------------------------
 [mx:javac] Cleaning apt source directory src/main/gen
 [mx:javac] Cleaning output directory /data/src/gitblit/build/classes
 [mx:javac] Compiling 457 source files to /data/src/gitblit/build/classes
 [mx:javac] /data/src/gitblit/src/main/java/com/gitblit/models/UserModel.java:40: error: a type with the same simple name is already defined by the single-type-import of SecureRandom
 [mx:javac] import com.gitblit.utils.SecureRandom;
 [mx:javac] ^
 [mx:javac] /data/src/gitblit/src/main/java/com/gitblit/models/UserModel.java:669: error: cannot find symbol
 [mx:javac] 		return StringUtils.getSHA1(RANDOM.randomBytes(32));
 [mx:javac] 		                                 ^
 [mx:javac]   symbol:   method randomBytes(int)
 [mx:javac]   location: variable RANDOM of type SecureRandom
 [mx:javac] Note: Some input files use or override a deprecated API.
 [mx:javac] Note: Recompile with -Xlint:deprecation for details.
 [mx:javac] Note: Some input files use unchecked or unsafe operations.
 [mx:javac] Note: Recompile with -Xlint:unchecked for details.
 [mx:javac] 2 errors

BUILD FAILED
/data/src/gitblit/build.xml:110: Compile failed; see the compiler error output for details.

Hey Tamas, the compile problem is not related to this PR. It does look like we'll need to review the class name and import collision on the SecureRandom class recently introduced.

Do you want me to submit a separate issue?

@tomposmiko Good idea.

@tomposmiko, don't bother, it is a quick fix. Already pushed to master.

Thanks, I didn't have time until now:)


Disable DTLSUDP & TLSTCP

These require linking against OpenSSL - and on Ubuntu 16.04 onwards, OpenSSL has been upgraded to 1.1.0 which Net-SNMP 5.7.3 does not play nice with.
Will review and patch in the future to re-enable.
Also fixed a print format string and updated documentation.

Just on my way to merge it too.


Nested texts don't seem to inherit parent styles

I stumbled upon an issue where creating nexted Texts don't work like they do in React-Native. Eg. this snippet:
import {Text} from 'dripsy';

...

<Text sx={{color: '$neutral/10'}}>
  Demo 1
  <Text>Demo 2</Text>
</Text>

Results in this:

Whereas if I were to use the Text component coming from react-native, like this:
import {Text} from 'react-native';

...

<Text style={{color: 'white'}}>
  Demo 1
  <Text>Demo 2</Text>
</Text>

It would work as I'd expect:

Hey, so they should inherit, however, they might be getting overridden by theme.text.body, is that possible?

Or if you have text in your theme.colors, I believe this would override it

I only have fontSize, lineHeight and fontWeight set in my theme.text.body; and I also don't have any key called text in the colors, only very color-y names like $yellow/90 and so. :D
But even then, I think the color coming from the parent component should take precedence instead of the ones in the theme, as the default theme values have been overwritten by it 🤔

But even then, I think the color coming from the parent component should take precedence instead of the ones in the theme, as the default theme values have been overwritten by it 🤔

I agree, it would just require adding an additional react context which would add some decent performance overhead…unfortunately RN doesn’t have easy APIs for controlling inheritance the way CSS does.
That said, if the color is unset everywhere, it should inherit. There’s nowhere you’re setting color then, right?

In that case, I'm not totally sure what's going on. Can you try:

Use a normal Text from react-native for the child
Do the same but for the parent instead
And lastly, do normal RN text for both parent and child

And compare what happens?


Allow multiple instances, and/or multiple sprite projects open at once

It would be nice to have something like a "create new piskel" button on the desktop app that either opens up a new instance of Piskel desktop or opens the new Piskel in a new tab.  The web version basically already does this, but I see no similar functionality in the desktop app.

That's actually a duplicate of #271
And that should be fixed now.


New release: Version 0.3.1

Release Checklist

[x] ci:jest-coverage passes
[x] npm outdated red dependencies are updated
[x] npm audit passes
[x] npm run build completes successfully
[x] manually tested in all supported browsers
[x] CHANGELOG.md is updated to include details of new release
[x] version bumped to 0.3.1 in package.json using npm version patch
[x] above changes reviewed and merged to master
[x] push to CodeCommit succeeds
[x] deployment to AWS succeeds
[x] #prj_move notified: MOVE back up after new version deployment

Release process complete.


[🚀esl-utils]: (POC) create isVisible predicate

As an ESL consumer, I want to be able to check element visibility using ESL utils.
Suggested implementation
elem.offsetHeight || elem.offsetWidth || elem.getClientRects().length
*Additional notes
Consider the option to include elements with visibility: hidden or, in general, the option to choose visibility check strategy
** Nice To Have (follow-ups)**

[ ] TraversingQuery: support of ::visible syntax

Split into 2 story:

[ ] POC: investigate options to check visibility, create isVisible predicate under esl-utils
[ ] TaaversinfQuery: support ::visible syntax

@ala-n to write spec


error running in docker  - npm ERR! cb() never called!

windows 10
npm 15
i get this error
docker build -t health-wallet-demo .
[+] Building 1200.9s (8/13)
=> [internal] load build definition from Dockerfile                                                                                0.1s
=> => transferring dockerfile: 329B                                                                                0.1s
=> [internal] load .dockerignore                                                                                0.1s
=> => transferring context: 54B                                                                                0.0s
=> [internal] load metadata for docker.io/library/node:13                                                                                3.6s
=> [1/9] FROM docker.io/library/node:13@sha256:70d4fffcab39a1f9f7161d58e674ddcc56c7f0724196b68d52a87bab15cb4a04                                561.7s
=> => resolve docker.io/library/node:13@sha256:70d4fffcab39a1f9f7161d58e674ddcc56c7f0724196b68d52a87bab15cb4a04                                  0.0s
=> => sha256:70d4fffcab39a1f9f7161d58e674ddcc56c7f0724196b68d52a87bab15cb4a04 1.21kB / 1.21kB                                                    0.0s
=> => sha256:2b9604a36e4911d15d2916dac4f1d853e2da612e9bb77df1016f8a51b3e333a1 7.88kB / 7.88kB                                                    0.0s
=> => sha256:1c6172af85ee14a8db5a3a51d406b768dfa94d196c06d0d06d591507cf8199f0 45.38MB / 45.38MB                                                199.6s
=> => sha256:b194b0e3c928807cfabf081055a117585ba5bf6697f65b2fede02225a5d73ad2 10.80MB / 10.80MB                                                 56.4s
=> => sha256:1e8d7127072cdbaae1935656444c3ec2bef8882c8c14d459e3a92ca1dd313c28 2.21kB / 2.21kB                                                    0.0s
=> => sha256:1f5ec00f35d5b2d1db6b8e925a3005c1a285365775028db0339903ddaeec4763 4.34MB / 4.34MB                                                   33.1s
=> => sha256:93b1353672b6861da5f1b58b0eca02ec10373a25d2898bddafa1b4bae2271c55 50.08MB / 50.08MB                                                286.1s
=> => sha256:3d7f38db3cca2c74df9a146d8419f5bf79d79b18de9eaee6351dccde16ab1f4a 214.91MB / 214.91MB                                              552.7s
=> => sha256:21e102f9fe89a18627c0ce50945bd1e0a11d0fecd4800bbbd999944d3940efc6 4.16kB / 4.16kB                                                  200.1s
=> => extracting sha256:1c6172af85ee14a8db5a3a51d406b768dfa94d196c06d0d06d591507cf8199f0                                                         1.6s
=> => sha256:d5431b24825a3297da35afe3d32786e01ec3fe7a8d1685adf59f82138e916e10 34.44MB / 34.44MB                                                358.6s
=> => extracting sha256:b194b0e3c928807cfabf081055a117585ba5bf6697f65b2fede02225a5d73ad2                                                         0.3s
=> => extracting sha256:1f5ec00f35d5b2d1db6b8e925a3005c1a285365775028db0339903ddaeec4763                                                         0.1s
=> => sha256:f780e3352c1809c08a5e6e4168206425ce703018baae8d6efd8d18efb101405b 2.38MB / 2.38MB                                                  301.5s
=> => extracting sha256:93b1353672b6861da5f1b58b0eca02ec10373a25d2898bddafa1b4bae2271c55                                                         1.9s
=> => sha256:4d28937582d0e76cbe8ed78ed921823a349a8a0755f91e13648e7636c974b0b6 295B / 295B                                                      303.3s
=> => extracting sha256:3d7f38db3cca2c74df9a146d8419f5bf79d79b18de9eaee6351dccde16ab1f4a                                                         6.7s
=> => extracting sha256:21e102f9fe89a18627c0ce50945bd1e0a11d0fecd4800bbbd999944d3940efc6                                                         0.0s
=> => extracting sha256:d5431b24825a3297da35afe3d32786e01ec3fe7a8d1685adf59f82138e916e10                                                         1.5s
=> => extracting sha256:f780e3352c1809c08a5e6e4168206425ce703018baae8d6efd8d18efb101405b                                                         0.1s
=> => extracting sha256:4d28937582d0e76cbe8ed78ed921823a349a8a0755f91e13648e7636c974b0b6                                                         0.0s
=> [internal] load build context                                                                                0.4s
=> => transferring context: 1.14MB                                                                                0.3s
=> [2/9] WORKDIR /usr/src/app                                                                                0.8s
=> [3/9] COPY package.json package-lock.json ./                                                                                0.0s
=> ERROR [4/9] RUN npm ci                                                                                634.7s

[4/9] RUN npm ci:
#8 634.6 npm ERR! cb() never called!
#8 634.6
#8 634.6 npm ERR! This is an error with npm itself. Please report this error at:
#8 634.6 npm ERR!     https://npm.community
#8 634.7
#8 634.7 npm ERR! A complete log of this run can be found in:
#8 634.7 npm ERR!     /root/.npm/_logs/2021-01-16T20_27_55_657Z-debug.log

executor failed running [/bin/sh -c npm ci]: exit code: 1

Is this happening consistently, or have you only seen it once?
You could try and get the logs from npm following the strategy here.
Also, what branch are you pulling from?

Is this happening consistently, or have you only seen it once?
You could try and get the logs from npm following the strategy here.
Also, what branch are you pulling from?

Closing this as it hasn't been reproduced, but please reopen if this is continuing.

Closing this as it hasn't been reproduced, but please reopen if this is continuing.


Remove default event

We need to remove "First App Open" event on dashboard  using code on mixpanel . how it will be possible ??
I have not got any way to block or predefined event  .

Hi @abhayjaiswal - "First App Open" is part of our Mixpanel default events. If you do not wish to collect those events,

Hi @abhayjaiswal ! If you want to disable automatic common mobile events tracking you should follow this steps:

Log in into mixpanel.com
Go to your project settings (click on gear icon on the right upper corner)
On the left panel select Autotrack
Disable "Automatically collect common mobile events"

Hope that helps!


[Discussion] - Goals, hurdles & opened questions

Hey peeps!
After a brief discussion on the OpenUI Discord I'm opening this issue to help point out some of the hurdles we might face implementing this polyfill and highlight some of the goals.

Goals

Feature Detection
Pop-up element attributes

HTML Content Attribute (popup | defaultopen)
IDL Attribute (popUp | defaultOpen)

Pop-up's triggering element attributes

HTML Content Attribute (popuptoggletarget | popupshowtarget | popuphidetarget)
IDL Attribute (popUpToggleTarget | popUpShowTarget | popUpHideTarget)

Showing and Hiding

Via Declarative Triggers (HTML attributes)
Via JavaScript Triggers (JS APIs, either via the methods: (show|hide)PopUp(), or via the IDL Attributes: el.popUp = '...'?)
Via Page Load Trigger (defaultopen)
Via "Forced/Mandated" Triggers (<dialog>.showModal(), <div>.requestFullscreen(), otherPopUp.show(), ...)

Events (show | hide)
Focus Management
Anchoring (partially out of scope as it would be subject to a dedicated polyfill?)
Backdrop

Hurdles to overcome

Stacking context /!\
Keeping track of the last focused element when necessary
Keeping track of ancestral pop-ups when necessary
Detecting element removal
Carving to higher priority actions (modal dialog shown or entered fullscreen)
Carving to other triggered pop-ups when appropriate

Open questions

Need a concept of stacks/layers? (as in const stack = new StackOfPopUps; stack.add(el); if (condition) stack.pop(el))
Needs a "#layers" element as last child of the body? Or should we play with z-index (like the dialog's polyfill seem to do?)?
A ".top-layer" class emulating the ":top-layer" pseudo class?
A ".backdrop" class emulating the "::backdrop" pseudo element (like the dialog's polyfill)?
To emulate UA CSS rule specificity the supplied default styles should make use of :where()?
How to handle the defaultopen attribute? querySelector on page load & MutationObserver for if the attribute get's added later on?
Events (show | hide)

Does the event need to be of a specific type or CustomEvent will do? (Dialog's polyfill uses CustomEvent)
Do we need on(show|hide) handlers? (Dialog's polyfill triggers them)

@VicGUTT Thanks so much! I'm going to spend a lot of today spinning this into more issues and documents, @mirisuzanne alerted me people are wanting to start helping!
Thanks so much for writing this all out, is it okay to come back with any questions?

Also cc: @keithamus, since he might have some more time to work on this!

Thanks so much!

Thank you for setting things up!

I'm going to spend a lot of today spinning this into more issues and documents

Nice! I'll keep an eye out, provide feedback when necessary and even chip in :). Don't hesitate to ping me 👌

Thanks so much for writing this all out, is it okay to come back with any questions?

Oh please do :)

First question @VicGUTT,  is there a more recent spec you are referring to re: anchoring? We started this polyfill without https://github.com/oddbird/css-anchor-positioning being ready due to not thinking they were that related.

Oh nice I somehow didn't see that repo ^^.
I'm not referring to any specific spec and from what I understand the two features: pop-up & anchoring are loosely related.
I made my bullet-points above mostly reading through the pop-up explainer:

A new attribute, anchor, can be used on a pop-up element to refer to the pop-up's "anchor". The value of the attribute is a string idref corresponding to the id of another element (the "anchor element").

Based on that sentence I figured the two polyfills would need to play nice with each other.

Open questions

Need a concept of stacks/layers? (as in const stack = new StackOfPopUps; stack.add(el); if (condition) stack.pop(el))

We have early work in that area: https://github.com/oddbird/popup-polyfill/blob/337c69c46ff3554d86580ec5edf11b2efbcd4126/src/popup.ts#L11

Needs a "#layers" element as last child of the body? Or should we play with z-index (like the dialog's polyfill seem to do?)?

zIndex might be the easiest path forward, but it does come with significant concerns as the top layer is one of the major benefits to popup (also the hardest to polyfill :smile:. Moving elements in the DOM to a last-child of body avoids some issues but present others (css selectors may be invalidated, events may propagate in the wrong places).

How to handle the defaultopen attribute? querySelector on page load & MutationObserver for if the attribute get's added later on?

I believe this is a working implementation: https://github.com/oddbird/popup-polyfill/pull/2. The explainer document speaks only to defaultopen having an effect on pageload. It does not speak about the effects of this attribute after page load. Chrome's implementation currently only observes defaultopen on load, and removing/adding it after load has no effect.

Events (show | hide)

Does the event need to be of a specific type or CustomEvent will do? (Dialog's polyfill uses CustomEvent)

Generally I'd say we should use Event. I don't know why the dialog polyfill uses CustomEvent, especially as it doesn't make use of detail.

Do we need on(show|hide) handlers? (Dialog's polyfill triggers them)

Yes we will do. Generally speaking in the HTML spec any event triggered should call on<event>?.(event) before calling dispatchEvent(event), and they should use the same event object.

Hey thanks for the feedback Keith!
Yes I have seen some early answers and implementations in the code to some of my open questions. I believed it'd still be useful to write them down here :).

zIndex might be the [...]

Yeah I totally agree with your two points. It's not easy to figure out ^^.
A third option, expressed in the OpenUI Discord server, would be to try and make use of/hijack the <dialog> element for it's capacity to be in the actual top-layer. I don't have more details about it but it could be an interesting venue to experiment. I'm thinking about opening a different issue about it.

I believe this is a working implementation [...]

Yeah that is true, the explainer only outlines the initial page load scenario. Find by me!
And I 100% agree with your last two points.
I'll make sure to update my initial comment with your provided insights.

Using dialog would give us the backdrop, the top-layer, and autofocus support. But it presents two new problems:

it makes the rest of the document inert, which popup shouldn't do. Unsure if we can work around that.
it means this polyfill must also depend on the dialog polyfill for browsers which don't have dialog, and that means we're kind of back to square one.

it makes the rest of the document inert, which popup shouldn't do. Unsure if we can work around that.

The dialog element only renders the page inert when invoked with .showModal() (in "dialog mode"). This is not the case for when it is invoked with .show().
The only issue I see is the accessibility / semantics that comes backed in with the dialog element (role=dialog) and according to the pop-up explainer: a "popped-up" element shouldn't see it's semantics changed. But unless I'm missing something, this shouldn't be to much of a problem to fix.

it means this polyfill must also depend on the dialog polyfill for browsers which don't have dialog, and that means we're kind of back to square one.

According to canIUse, the dialog element has 90+% support except for the ::backdrop pseudo-element missing in Firefox, bringing it down to 87.75%. I believe, if there's a way to work around the cons of using the dialog element, it would be worth it.

The dialog element only renders the page inert when invoked with .showModal() (in "dialog mode"). This is not the case for when it is invoked with .show().

This is true, however a dialog shown without showModal() won't have a ::backdrop. Perhaps the ::backdrop is the easier piece to poly-fill though (maybe with another dialog).

According to canIUse, the dialog element has 90+% support

While current versions of browsers all support this, the fall-off on older versions is quite steep. For example the current Firefox ESR (91) does not support dialog, and only the current major version of Safari supports it. The dialog polyfill claims to work back to IE9, so it's a viable strategy, but it does mean this polyfill would inherit all problems of the existing dialog polyfill(s).

Started a bunch of issues I will add notes to!

Right. And we can always go the adjacent .backdrop route.
True, and I wasn't aware of ESR versions of Firefox, although for the examples provided, older versions of Firefox & Safari combined seem to account for 1.30% total. I guess we'll need to decide what threshold of "not supported" browsers would be acceptable.
To be fair, I'm usually not best suited to judge "acceptable browser support" as I'm usually forward facing and use anything with more than 90+% support without thinking much about it, which in the case of working on a polyfill might to be the best strategy 😅 .
Yay thanks @oluoluoxenfree :). I'll see what I can pick up ✌🏾

Another bummer about making use of the dialog element is that, as stated in the OpenUI Discord channel, .show() does not promote the dialog to be in the top-layer. Which defeats the biggest benefit of using the dialog element.
So, a summary on making use of the <dialog> element:
Pros

top-layer
::backdrop
autofocus

Cons

in certain situations, heavy accessibility & semantics backed-in that may not be easy to overwrite
this polyfill would depend on the dialog polyfill and would inherit it's quirks
juggle between features (or lack of) provided by either .show() or .showModal()

.show():
❌ top-layer
❌ ::backdrop
❌ autofocus (?)
✔ light accessibility & semantics backed-in, not much unwanted behavior
.showModal():
✔ top-layer
✔ ::backdrop
✔ autofocus
❌ heavy accessibility & semantics backed-in, quite some unwanted behavior

I'm a bit weary about dialog as an approach, simply because of all the a11y articles that warn it shouldn't be used to mimic other patterns. I don't know if there are ways around those issues or not.


[FLINK-23489][docs] Change flink-config.yaml to flink-conf.yaml

The relevant pages are as follows：
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/cli/
https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/deployment/cli/
The related documents are as follows：
docs/content/docs/deployment/cli.md
docs/content.zh/docs/deployment/cli.md
In these pages, conf/flink-config.yaml should be changed to conf/flink-conf.yaml

Could you also update the English doc docs/content/docs/deployment/cli.md as well?

Hi，I also update the English doc docs/content/docs/deployment/cli.md in this pr.
Would you like to review it for me again? [抱拳][呲牙]


Relax GRPC requirements

Since 1.51.1 grpc does not fail client memory growth test, relax grpc requirements.

fixes issues raised in https://github.com/triton-inference-server/client/pull/92


How do I change a Cosmos container to Autoscale throughput via Azure CLI

Azure documentation issue guidance
Thanks for opening an issue in the Azure technical documentation repository.
We use GitHub issues as the primary channel for customer and community feedback about the Azure documentation.
Creating an issue
We prefer that you create documentation feedback issues using the Feedback link on the published article - the feedback control on the doc page creates an issue that contains all the article details so you can focus on the feedback part.
You can also create a feedback issue here in the repo. If you do this, please make sure your issue lists:

[ ] The relevant Azure service or technology.
[ ] A link to the published documentation article that you have feedback about.
[ ] Clear, specific feedback that the author can act on.

Pull requests and article contributions
If you know the change that is needed in an article, we encourage you to submit the changes directly using a pull request. If the change is large, or if you want to contribute an entire article, follow these guidelines:

[ ] Don't surprise us with a big pull request or a pull request with a new article! Submit an issue that describes the details of the proposed large change or new article.
[ ] Include the service or technology area.

We'll route the issue to the appropriate content team for review and discussion.
Tech support and product feedback
If you would like to contact Microsoft about other things, such as product feedback or tech support, please review these guidelines:

If you need technical support using Azure, the paid and free support options are described here: https://azure.microsoft.com/support/options/.

Each article in the Azure technical documentation contains a product feedback button - it's best to submit product feedback directly from a relevant article. Otherwise, you can submit product feedback for most Azure products in the following product feedback forum: https://feedback.azure.com/forums/34192--general-feedback.

Thanks for the feedback @sdg002! We are currently investigating and will update you shortly.

@sdg002 Thanks for reaching out to us. You can refer to the below articles to update the throughput of the container using Azure CLI:

https://docs.microsoft.com/en-us/azure/cosmos-db/scripts/cli/sql/autoscale?toc=/cli/azure/toc.json
https://docs.microsoft.com/en-us/cli/azure/cosmosdb/sql/container/throughput?view=azure-cli-latest

There are many ways to do it, you can set it at the time of creating the container. Or else you can update it once container is created as mentioned in above article.
Please let us know if this is what you were looking for.

@sdg002 We will now proceed to close this thread. If there are further questions regarding this matter, please comment and we will gladly continue the discussion.


Blockchain Module imports not working?

We've found our smart contract stopped compiling on the latest neo-boa (both master and development branches). It looks like this commit is the culprit:
https://github.com/CityOfZion/neo-boa/commit/2c6e01ad89b398092610cff89a23f11484b3bfb1
The compile works fine on the prior commit:
https://github.com/CityOfZion/neo-boa/commit/aad18d8c47a9b3f99ec15bfdb7653c673f4691c3
(venv) neo-boa $ git checkout 2c6e01ad89b398092610cff89a23f11484b3bfb1
Previous HEAD position was aad18d8... Merge branch 'development' of github.com:CityOfZion/neo-boa into development
HEAD is now at 2c6e01a... Blockchain no longer imports other modules
(venv) neo-boa $ python compile.py
Traceback (most recent call last):
  File "compile.py", line 3, in <module>
    Compiler.load_and_save('../tokensale-neo-smartcontract/ico_template.py')
  File ".../neo-boa/boa/compiler.py", line 105, in load_and_save
    data = compiler.write()
  File ".../neo-boa/boa/compiler.py", line 81, in write
    out_bytes = bytes(module.write())
  File ".../neo-boa/boa/code/module.py", line 403, in write
    self.link_methods()
  File ".../neo-boa/boa/code/module.py", line 462, in link_methods
    raise Exception("Target method %s not found" % vmtoken.target_method)
Exception: Target method GetTimestamp not found
(venv) neo-boa $ git checkout aad18d8c47a9b3f99ec15bfdb7653c673f4691c3
Previous HEAD position was 2c6e01a... Blockchain no longer imports other modules
HEAD is now at aad18d8... Merge branch 'development' of github.com:CityOfZion/neo-boa into development
(venv) neo-boa $ python compile.py
(venv) neo-boa $

Note that "GetTimestamp" compile error is the same on the latest master, as well.
Our project is https://github.com/NarrativeNetwork/tokensale-neo-smartcontract if you want to test against the same source.
If we remove the Header.Timestamp property reference, the GetTimestamp error goes away and turns into:
(venv) neo-boa brian$ python compile.py
Traceback (most recent call last):
  File "compile.py", line 3, in <module>
    Compiler.load_and_save('../tokensale-neo-smartcontract/ico_template.py')
  File ".../neo-boa/boa/compiler.py", line 105, in load_and_save
    data = compiler.write()
  File ".../neo-boa/boa/compiler.py", line 81, in write
    out_bytes = bytes(module.write())
  File ".../neo-boa/boa/code/module.py", line 403, in write
    self.link_methods()
  File ".../neo-boa/boa/code/module.py", line 462, in link_methods
    raise Exception("Target method %s not found" % vmtoken.target_method)
Exception: Target method GetUnspentCoins not found
(venv) neo-boa brian$

I tried to isolate a more limited use case, but it seems to be challenging, as there appear to be non-deterministic properties of this issue. Here's one example, but this doesn't seem to be affected by the specific changeset I mentioned above:
from boa.blockchain.vm.Neo.Transaction import Transaction

class Test():
    def Main(self):
        return 0

That results in what appears to be the same GetUnspentCoins error as above:
(venv) neo-boa brian$ python compile.test.py
Traceback (most recent call last):
  File "compile.test.py", line 3, in <module>
    Compiler.load_and_save('../tokensale-neo-smartcontract/test.py')
  File ".../neo-boa/boa/compiler.py", line 105, in load_and_save
    data = compiler.write()
  File ".../neo-boa/boa/compiler.py", line 81, in write
    out_bytes = bytes(module.write())
  File ".../neo-boa/boa/code/module.py", line 403, in write
    self.link_methods()
  File ".../neo-boa/boa/code/module.py", line 462, in link_methods
    raise Exception("Target method %s not found" % vmtoken.target_method)
Exception: Target method GetUnspentCoins not found
(venv) neo-boa brian$

For now, to work around the issue, we are using neo-boa 0.2.1, as the issue is present in 0.2.2.

This should be fixed in the new version of the compiler, v0.3.3

when i try to compile neo-ico-template , using command :-
sc build_run /home/adarsh.jha/Documents/neoproject/neo-ico-template/compile
.py True False False '' 02 ''
i get this error : -
from boa.compiler import Compiler
^
IndentationError: unexpected indent


Search for post field doesn't yield any results

Describe the bug
Currently I am unable to search for any posts using the search field. This makes it impossible for me to find posts.
To Reproduce
Steps to reproduce the behavior:

Go to DevHub's homepage
Type any key word in the "Search Posts" field
See hovering circle inside the search posts (e.g. loading button) without any search outputs

Expected behavior
I should see the posts that contain the search term/
Screenshots

Desktop (please complete the following information):

OS: iOS
Browser: Safari
Version: Latest

The error is because of calling the contract method get_posts fails like this:
NEAR_ENV=mainnet near view devgovgigs.near get_posts '{}'
View call: devgovgigs.near.get_posts({})
An error occured
Error: Querying [object Object] failed: wasm execution failed with error: HostError(GuestPanic { panic_msg: "Cannot deserialize element" }).
{
  "block_hash": "8iGTZLLcTcWB6BhrVhdMZGNXScnJMdYraRdHN3ZuXcy3",
  "block_height": 103304522,
  "error": "wasm execution failed with error: HostError(GuestPanic { panic_msg: \"Cannot deserialize element\" })",
  "logs": []
}
    at JsonRpcProvider.query (/usr/local/lib/node_modules/near-cli/node_modules/near-api-js/lib/providers/json-rpc-provider.js:123:19)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Account.viewFunction (/usr/local/lib/node_modules/near-cli/node_modules/near-api-js/lib/account.js:366:24)
    at async exports.callViewFunction (/usr/local/lib/node_modules/near-cli/index.js:103:48)
    at async Object.handler (/usr/local/lib/node_modules/near-cli/utils/exit-on-error.js:52:9)
TypedError: Querying [object Object] failed: wasm execution failed with error: HostError(GuestPanic { panic_msg: "Cannot deserialize element" }).
{
  "block_hash": "8iGTZLLcTcWB6BhrVhdMZGNXScnJMdYraRdHN3ZuXcy3",
  "block_height": 103304522,
  "error": "wasm execution failed with error: HostError(GuestPanic { panic_msg: \"Cannot deserialize element\" })",
  "logs": []
}
    at JsonRpcProvider.query (/usr/local/lib/node_modules/near-cli/node_modules/near-api-js/lib/providers/json-rpc-provider.js:123:19)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Account.viewFunction (/usr/local/lib/node_modules/near-cli/node_modules/near-api-js/lib/account.js:366:24)
    at async exports.callViewFunction (/usr/local/lib/node_modules/near-cli/index.js:103:48)
    at async Object.handler (/usr/local/lib/node_modules/near-cli/utils/exit-on-error.js:52:9) {
  type: 'UntypedError',
  context: undefined
}

And it is because of particular posts like no 146:
NEAR_ENV=mainnet near view devgovgigs.near get_post '{"post_id": 146}'
View call: devgovgigs.near.get_post({"post_id": 146})
An error occured
Error: Querying [object Object] failed: wasm execution failed with error: HostError(GuestPanic { panic_msg: "Cannot deserialize element" }).
{
  "block_hash": "2dkpWC71CwFbw8uZiDES2SVDVprMYBgRTsBkAYEzA8Qv",
  "block_height": 103305310,
  "error": "wasm execution failed with error: HostError(GuestPanic { panic_msg: \"Cannot deserialize element\" })",
  "logs": []
}
    at JsonRpcProvider.query (/usr/local/lib/node_modules/near-cli/node_modules/near-api-js/lib/providers/json-rpc-provider.js:123:19)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Account.viewFunction (/usr/local/lib/node_modules/near-cli/node_modules/near-api-js/lib/account.js:366:24)
    at async exports.callViewFunction (/usr/local/lib/node_modules/near-cli/index.js:103:48)
    at async Object.handler (/usr/local/lib/node_modules/near-cli/utils/exit-on-error.js:52:9)
TypedError: Querying [object Object] failed: wasm execution failed with error: HostError(GuestPanic { panic_msg: "Cannot deserialize element" }).
{
  "block_hash": "2dkpWC71CwFbw8uZiDES2SVDVprMYBgRTsBkAYEzA8Qv",
  "block_height": 103305310,
  "error": "wasm execution failed with error: HostError(GuestPanic { panic_msg: \"Cannot deserialize element\" })",
  "logs": []
}
    at JsonRpcProvider.query (/usr/local/lib/node_modules/near-cli/node_modules/near-api-js/lib/providers/json-rpc-provider.js:123:19)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Account.viewFunction (/usr/local/lib/node_modules/near-cli/node_modules/near-api-js/lib/account.js:366:24)
    at async exports.callViewFunction (/usr/local/lib/node_modules/near-cli/index.js:103:48)
    at async Object.handler (/usr/local/lib/node_modules/near-cli/utils/exit-on-error.js:52:9) {
  type: 'UntypedError',
  context: undefined
}

While post no 145 works fine.
Need to find a way to reproduce the state of post 146, or a similar state in a test environment.


Option to change bodyAttr

Hi @tulios
Prior to version 2.0.0 we had something like this:
Mappersmith.forge(Manifest, Gateway, "data")
Is there a way to achieve the same (pass a custom bodyAttr) on version >= 2.0.0?
PS: If possible, can you put together a minimal migration guide?

Yes :)
You can take a look here (https://github.com/tulios/mappersmith#body), but in a nutshell you can use the bodyAttr when defining your methods, example:
// ...
{
  create: { method: 'post', path: '/blogs', bodyAttr: 'payload' }
}
// ...

client.Blog.create({
  payload: {
    title: 'Title',
    tags: ['party', 'launch']
  }
})

Is it not possible to make it a global config like before?

Currently it's not possible, is it something you would like to have?

Yeah. It would be nice to have some of the resource attributes i.e bodyAttr, headersAttr and authAttr configurable at a higher level than resource definition, like host.
My thinking behind this is that they are less likely to change than path and method. So that something like the following is possible:
const client = forge({
host: 'https://status.github.com',
bodyAttr: 'data',
headersAttr: 'h',
resources: {
User: {
all: { path: '/users' }
}
}
})


Something wrong about training

When running: sh ./table_recognition/expr/table_recognition_dist_train.sh
AssertionError: OCRFinTabDataset: TableHardDiskLoader: /datasets/mmocr_fintabnet_recognition_0726_train/StructureLabelAddEmptyBbox_train/ is not exist
How can I deal with this problem?

Is there a trained model？I want to run a demo.

Yes, we have released the pre-trained models of both FinTabNet and PubtabNet.
Please check the pre-trained models and how to run the demo in README.md.

Did you prepare the training data for FinTabNet?
You need to run "data_preprocess_fintabnet.py" to prepare the training data for FinTabNet.

When running: sh ./table_recognition/expr/table_recognition_dist_train.sh AssertionError: OCRFinTabDataset: TableHardDiskLoader: /datasets/mmocr_fintabnet_recognition_0726_train/StructureLabelAddEmptyBbox_train/ is not exist How can I deal with this problem?

Did you prepare the training data for FinTabNet?
You need to run "data_preprocess_fintabnet.py" to prepare the training data for FinTabNet.

Is there a trained model？I want to run a demo.

Yes, we have released the pre-trained models of both FinTabNet and PubtabNet.
Please check the pre-trained models and how to run the demo in README.md.

When running: sh ./table_recognition/expr/table_recognition_dist_train.sh AssertionError: OCRFinTabDataset: TableHardDiskLoader: /datasets/mmocr_fintabnet_recognition_0726_train/StructureLabelAddEmptyBbox_train/ is not exist How can I deal with this problem?

Did you prepare the training data for FinTabNet? You need to run "data_preprocess_fintabnet.py" to prepare the training data for FinTabNet.

Yes,I did.But the mistake still appeared.I'll try it again.

Is there a trained model？I want to run a demo.

Yes, we have released the pre-trained models of both FinTabNet and PubtabNet. Please check the pre-trained models and how to run the demo in README.md.

Thank you very much for replying!

When running: sh ./table_recognition/expr/table_recognition_dist_train.sh AssertionError: OCRFinTabDataset: TableHardDiskLoader: /datasets/mmocr_fintabnet_recognition_0726_train/StructureLabelAddEmptyBbox_train/ is not exist How can I deal with this problem?

Did you prepare the training data for FinTabNet? You need to run "data_preprocess_fintabnet.py" to prepare the training data for FinTabNet.

Yes,I did.But the mistake still appeared.I'll try it again.

Can you show the whole error message, then I will try to check the reason.

When running: sh ./table_recognition/expr/table_recognition_dist_train.sh AssertionError: OCRFinTabDataset: TableHardDiskLoader: /datasets/mmocr_fintabnet_recognition_0726_train/StructureLabelAddEmptyBbox_train/ is not exist How can I deal with this problem?

Did you prepare the training data for FinTabNet? You need to run "data_preprocess_fintabnet.py" to prepare the training data for FinTabNet.

Yes,I did.But the mistake still appeared.I'll try it again.

Can you show the whole error message, then I will try to check the reason.

After running "data_preprocess_fintabnet.py",I find the several folders were empty.Is there anything wrong with fintabnet datasets?

When running: sh ./table_recognition/expr/table_recognition_dist_train.sh AssertionError: OCRFinTabDataset: TableHardDiskLoader: /datasets/mmocr_fintabnet_recognition_0726_train/StructureLabelAddEmptyBbox_train/ is not exist How can I deal with this problem?

Did you prepare the training data for FinTabNet? You need to run "data_preprocess_fintabnet.py" to prepare the training data for FinTabNet.

Yes,I did.But the mistake still appeared.I'll try it again.

Can you show the whole error message, then I will try to check the reason.

After running "data_preprocess_fintabnet.py",I find the several folders were empty.Is there anything wrong with fintabnet datasets?

I think you have some problems when preparing the jsonl file of FinTabNet.
This jsonl file is the same as the one of PubTabNet.
I will publicize the jsonl as well as the table images of FinTabNet and let you know soon.

When running: sh ./table_recognition/expr/table_recognition_dist_train.sh AssertionError: OCRFinTabDataset: TableHardDiskLoader: /datasets/mmocr_fintabnet_recognition_0726_train/StructureLabelAddEmptyBbox_train/ is not exist How can I deal with this problem?

Did you prepare the training data for FinTabNet? You need to run "data_preprocess_fintabnet.py" to prepare the training data for FinTabNet.

Yes,I did.But the mistake still appeared.I'll try it again.

Can you show the whole error message, then I will try to check the reason.

After running "data_preprocess_fintabnet.py",I find the several folders were empty.Is there anything wrong with fintabnet datasets?

I think you have some problems when preparing the jsonl file of FinTabNet. This jsonl file is the same as the one of PubTabNet. I will publicize the jsonl as well as the table images of FinTabNet and let you know soon.

Thank you so much!

When running: sh ./table_recognition/expr/table_recognition_dist_train.sh AssertionError: OCRFinTabDataset: TableHardDiskLoader: /datasets/mmocr_fintabnet_recognition_0726_train/StructureLabelAddEmptyBbox_train/ is not exist How can I deal with this problem?

Did you prepare the training data for FinTabNet? You need to run "data_preprocess_fintabnet.py" to prepare the training data for FinTabNet.

Yes,I did.But the mistake still appeared.I'll try it again.

Can you show the whole error message, then I will try to check the reason.

After running "data_preprocess_fintabnet.py",I find the several folders were empty.Is there anything wrong with fintabnet datasets?

I think you have some problems when preparing the jsonl file of FinTabNet. This jsonl file is the same as the one of PubTabNet. I will publicize the jsonl as well as the table images of FinTabNet and let you know soon.

I upload the Python script extract_table_images_FinTabNet.py for extracting table images and making the annotation file (like PubTabNet) from FInTabNet dataset. After that, you can run "data_preprocess_fintabnet.py" to prepare the training dataset for FinTabNet.

OK，I'll give another try.It is appreciated for answering my question in your busy time.

When running: sh ./table_recognition/expr/table_recognition_dist_train.sh AssertionError: OCRFinTabDataset: TableHardDiskLoader: /datasets/mmocr_fintabnet_recognition_0726_train/StructureLabelAddEmptyBbox_train/ is not exist How can I deal with this problem?

Did you prepare the training data for FinTabNet? You need to run "data_preprocess_fintabnet.py" to prepare the training data for FinTabNet.

Yes,I did.But the mistake still appeared.I'll try it again.

Can you show the whole error message, then I will try to check the reason.

After running "data_preprocess_fintabnet.py",I find the several folders were empty.Is there anything wrong with fintabnet datasets?

I think you have some problems when preparing the jsonl file of FinTabNet. This jsonl file is the same as the one of PubTabNet. I will publicize the jsonl as well as the table images of FinTabNet and let you know soon.

I upload the Python script extract_table_images_FinTabNet.py for extracting table images and making the annotation file (like PubTabNet) from FInTabNet dataset. After that, you can run "data_preprocess_fintabnet.py" to prepare the training dataset for FinTabNet.

OK，I'll give another try.It is appreciated for answering my question in your busy time.

Is there a trained model？I want to run a demo.

Yes, we have released the pre-trained models of both FinTabNet and PubtabNet. Please check the pre-trained models and how to run the demo in README.md.

Thank you very much for replying!

While running on CPU Machine - python demo.py
Facing ERROR -
Traceback (most recent call last):
File "demo.py", line 6, in
from mmdet.apis import init_detector
File "/home/n14688/mmdetection/mmdet/apis/init.py", line 2, in
from .det_inferencer import DetInferencer
File "/home/n14688/mmdetection/mmdet/apis/det_inferencer.py", line 22, in
from mmdet.evaluation import INSTANCE_OFFSET
File "/home/n14688/mmdetection/mmdet/evaluation/init.py", line 4, in
from .metrics import *  # noqa: F401,F403
File "/home/n14688/mmdetection/mmdet/evaluation/metrics/init.py", line 5, in
from .coco_metric import CocoMetric
File "/home/n14688/mmdetection/mmdet/evaluation/metrics/coco_metric.py", line 16, in
from mmdet.datasets.api_wrappers import COCO, COCOeval, COCOevalMP
File "/home/n14688/mmdetection/mmdet/datasets/init.py", line 31, in
from .utils import get_loading_pipeline
File "/home/n14688/mmdetection/mmdet/datasets/utils.py", line 5, in
from mmdet.datasets.transforms import LoadAnnotations, LoadPanopticAnnotations
File "/home/n14688/mmdetection/mmdet/datasets/transforms/init.py", line 6, in
from .formatting import (ImageToTensor, PackDetInputs, PackReIDInputs,
File "/home/n14688/mmdetection/mmdet/datasets/transforms/formatting.py", line 11, in
from mmdet.structures.bbox import BaseBoxes
File "/home/n14688/mmdetection/mmdet/structures/bbox/init.py", line 2, in
from .base_boxes import BaseBoxes
File "/home/n14688/mmdetection/mmdet/structures/bbox/base_boxes.py", line 9, in
from mmdet.structures.mask.structures import BitmapMasks, PolygonMasks
File "/home/n14688/mmdetection/mmdet/structures/mask/init.py", line 3, in
from .structures import (BaseInstanceMasks, BitmapMasks, PolygonMasks,
File "/home/n14688/mmdetection/mmdet/structures/mask/structures.py", line 12, in
from mmcv.ops.roi_align import roi_align
File "/home/n14688/anaconda3/envs/MTL/lib/python3.8/site-packages/mmcv/ops/init.py", line 2, in
from .active_rotated_filter import active_rotated_filter
File "/home/n14688/anaconda3/envs/MTL/lib/python3.8/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in
ext_module = ext_loader.load_ext(
File "/home/n14688/anaconda3/envs/MTL/lib/python3.8/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "/home/n14688/anaconda3/envs/MTL/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /home/n14688/anaconda3/envs/MTL/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor6narrowElll
(MTL) n14688@n14688:~/MTL-TabNet/table_r

Can we demo.py on CPU  machine ?


man: update missingcrypto.txt file

Remove some entries which have been documented meanwhile.
Note
The entries were checked using the following python script after installing the documentation to /opt/openssl-dev
#!/usr/bin/env python3

import shutil
import subprocess

manpath = '/opt/openssl-dev/share/man'

with open("util/missingcrypto.txt", "r") as f:
    lines = f.readlines()

with open("util/missingcrypto.txt", "w") as f:
    for entry in lines:
        if subprocess.run(["man", entry.strip()], capture_output=True, env={'MANPATH': manpath}).returncode:
            f.write(entry)

CI failure is relevant @mspncp

find-doc-nits and man seem to disagree on whether those two functions are documented:
msp@office:~/src/openssl/master$ make doc-nits
/usr/bin/perl ./util/find-doc-nits -c -n -l -e
crypto: function ERR_load_CRYPTO_strings(3) undocumented
crypto: function OCSP_crlID_new(3) undocumented
# 2 libcrypto names are not documented
make: *** [Makefile:4144: doc-nits] Error 1
msp@office:~/src/openssl/master$ find /opt/openssl-dev/share/man/ -iname 'ERR_load_CRYPTO_strings*' -o -iname 'OCSP_crlID_new*'
/opt/openssl-dev/share/man/man3/OCSP_CRLID_new.3ossl
/opt/openssl-dev/share/man/man3/ERR_load_crypto_strings.3ossl

Oh I see: both capitalizations exist. Since man doesn't care about capitalization, it found the other variant. I'll add those two entries back in.

See force-push diff

However were these functions with different capitalization just aliases? You could have added them to the manual pages fairly trivially then.

However were these functions with different capitalization just aliases?

No, they are not.
ERR_load_crypto_strings and ERR_load_CRYPTO_strings are different and both deprecated anyway.
https://github.com/openssl/openssl/blob/0f9caad5b95e901b87fe45cf85c9582071ca0b23/include/openssl/err.h.in#L468-L470
https://github.com/openssl/openssl/blob/0f9caad5b95e901b87fe45cf85c9582071ca0b23/include/openssl/cryptoerr_legacy.h#L41
OCSP_CRLID_new creates the corresponding ASN1 object, while OCSP_crlID_new wraps that ASN1 object into an X509_EXTENSION:
https://github.com/openssl/openssl/blob/0f9caad5b95e901b87fe45cf85c9582071ca0b23/crypto/ocsp/ocsp_ext.c#L360-L366

Merged to master. Thank you.


Add Link Directories

Add the link directories for Vulkan, GLFW, and GLM for CMake.

git fetch origin
git checkout 17-add-link-directories


Updated key name in dictionary to match device dict key name

With this change, we're now able to see the changes on the UI from using the physical switches plugged to the Dual R3.
I haven't changed the tests, as I am not completely sure if it would be impacting other places in the code.

As I can see logs here - currLocation for read position, location for write position.
Maybe something have changed in new device firmwares. Have you collect debug logs?

When I use the manual wall switches on my Dual R3, I get one line once the motion stops (consistent with the behaviour on the eWeLink app). But only with the change proposed here, I get to see the Home Assistant UI updated with the new final position.
2022-07-31 17:33:00 [D] 1001436ec2 <= Cloud3 | {'motorTurn': 0, 'currLocation': 82, 'location': 82} | None
I cannot use them locally in this 3.x version. When in this mode, I know they update live (as seen in 2.x version and eWeLink app).
Thoughts?
I can try and debug further into the code, if you'd like.

Just to add that with the latest updates from HA, I am now unable to reach the debug page for the tool. Any other way to reach there @AlexxIT ? Thanks!

Yes. Link is broken. Haven't tried to fix it yet.

Further debugging, appears that if using "cloud" mode, everything works. Need to debug further when using "local". Closing this. Thanks for the support @AlexxIT .

Ok. Thanks for debugging :)


feat(tests): add integration tests

PR to origin: https://github.com/google-github-actions/ssh-compute/pull/7


OSX - macCredits and macPlist options not being seen in manifest

OSX 10.11.6 El Capitan  using nwjs 0.23.5 (node 8.1.2).
Just tried out nw-builder 3.4.1 and found that macIcns worked fine but macCredits and macPlist options were not being parsed in the package.json and therefore not appearing in the final app build.
Had a look at nwbuild file and it appeared to only cater for macIcns.
Adding 2 lines for the other two options allowed the build to work perfectly.
62  macCredits: argv.macCredits || false,
63  macPlist: argv.macPlist || false,
Without these 2 lines the conditional test in NwBuilder.prototype.handleMacApp was always false as per the option default settings at top of index.js
EDIT:
Also tried versioning the release folder with buildType versioned and always got the default with appName. Same issue with options in nwbuild - not specified.
Added line  67  buildType: argv.buildType || 'default'  and the folder type naming works well.

Sorry yeah the CLI isn't quite as powerful as the programmatic API. Nice find. I'd really appreciate it if you created a pull-request 👍


Add toggle for menu

On tablet and desktop screens there is a sidebar instead of a tab bar. It would be ideal to be able to add a toggle to collapse the menu down to just icons, especially when using the horizontal songsheet view.

Hey, I love your app! Can I take a look at implementing this?

@jahtomini Thanks, that would be great! Let me know if you have any questions.

I opened a discussion explaining an issue I had. Was able to get around it, at least partially, but would love any assistance in figuring out what really went wrong.


feat: add wasm support for blas/base/ccopy-wasm

Progresses #2039.
Description

What is the purpose of this pull request?

This pull request adds wasm support for blas/base/ccopy.
Related Issues

Does this pull request have any related issues?

This pull request:

progresses #2039

Questions

Any questions for reviewers of this pull request?

No.
Other

Any other information relevant to this pull request? This may include screenshots, references, and/or implementation notes.

No.
Checklist

Please ensure the following tasks are completed before submitting this pull request.

[x] Read, understood, and followed the contributing guidelines.

@stdlib-js/reviewers

/stdlib merge


KOGITO-1787 NullPointerException X-KOGITO-StartFromNode header

Start date is now fixed in startFrom method, which means
internalSetStartDate is not longer internal.
https://issues.redhat.com/browse/KOGITO-1787
Many thanks for submitting your Pull Request :heart:!
Please make sure that your PR meets the following requirements:

[x] You have read the contributors guide
[x] Pull Request title is properly formatted: KOGITO-XYZ Subject
[x] Pull Request title contains the target branch if not targeting master: [0.9.x] KOGITO-XYZ Subject
[x] Pull Request contains link to the JIRA issue
[x] Pull Request contains link to any dependent or related Pull Request
[x] Pull Request contains description of the issue
[x] Pull Request does not include fixes for issues other than the main ticket

@fjtirado this PR needs reviewers, ask @cristianonicolai for this.
Also, remember to update the PR description with the JIRA link.

@tiagodolphine why not extending this test to make sure that start date is set?
https://github.com/kiegroup/kogito-runtimes/blob/master/kogito-codegen/src/test/java/org/kie/kogito/codegen/tests/ServiceTaskTest.java#L94
plus you might start using file system based persistence for tests to at least cover the marshalling part

Hi, @mswiderski thanks! yeah, this is an option to cover this with a test since there is no easy way now to add a unit test for the AbstractProcessInstance, @fjtirado can you take a look at the ServiceTaskTest.java, it would be great to cover with some test this change.
Thanks!

Hi, @mswiderski thanks! yeah, this is an option to cover this with a test since there is no easy way now to add a unit test for the AbstractProcessInstance, @fjtirado can you take a look at the ServiceTaskTest.java, it would be great to cover with some test this change.
Thanks!

I was looking at ActivityGenerationModelTest, there we are testing start method, but not startFrom method.


question on where is spark in the full version?

according to this list of kubeflow bundle , it should contain spark.. but I fail to find it in either my deployed Applications or in the bundle file
https://charmed-kubeflow.io/docs/operators-and-bundles
Full Kubeflow bundle - [Store](https://charmhub.io/kubeflow) | [YAML](https://github.com/canonical/bundle-kubeflow/blob/master/bundle-kubeflow.yaml)

    [admission-webhook](https://charmhub.io/admission-webhook)
    [argo-controller](https://charmhub.io/argo-controller)
    [dex-auth](https://charmhub.io/dex-auth)
    [istio-ingressgateway](https://jaas.ai/u/kubeflow-charmers/istio-ingressgateway)
    [istio-pilot](https://charmhub.io/istio-pilot)
    [jupyter-controller](https://charmhub.io/jupyter-controller)
    [jupyter-ui](https://charmhub.io/jupyter-ui)
    [katib-controller](https://charmhub.io/katib-controller)
    [katib-db](https://charmhub.io/charmed-osm-mariadb-k8s)
    [katib-db-manager](https://charmhub.io/katib-db-manager)
    [katib-ui](https://charmhub.io/katib-ui)
    [kfp-api](https://charmhub.io/kfp-api)
    [kfp-db](https://charmhub.io/charmed-osm-mariadb-k8s)
    [kfp-persistence](https://charmhub.io/kfp-persistence)
    [kfp-schedwf](https://charmhub.io/kfp-schedwf)
    [kfp-ui](https://charmhub.io/kfp-ui)
    [kfp-viewer](https://charmhub.io/kfp-viewer)
    [kfp-viz](https://charmhub.io/kfp-viz)
    [kubeflow-dashboard](https://charmhub.io/kubeflow-dashboard)
    [kubeflow-profiles](https://charmhub.io/kubeflow-profiles)
    [mlmd](https://charmhub.io/mlmd)
    [minio](https://charmhub.io/minio)
    [oidc-gatekeeper](https://charmhub.io/oidc-gatekeeper)
    [pytorch-operator](https://charmhub.io/pytorch-operator)
    [spark](https://charmhub.io/spark-charmers-spark)
    [seldon-controller-manager](https://charmhub.io/seldon-core)
    [tfjob-operator](https://charmhub.io/tfjob-operator)

https://github.com/canonical/bundle-kubeflow/blob/master/bundle-kubeflow.yaml

perhaps I just need a tutorial how to get started and it's already installed.. but when I looked for spark in a new notebook I hadn't a clue where to start

just tried the spark notebook image but.. seemed to fail

Hello @EMCP, you can install spark separately by executing the following command in the kubeflow juju model:
juju deploy spark-k8s
We removed spark from the default bundle as it wasn't entirely related to Kubeflow, sorry for the inconvenience.

thank you @DomFleischmann , I need to start over from scratch.. as my k8s controller is now acting up.. the bug is already reported here https://bugs.launchpad.net/juju/+bug/1968105
but I will retry deploying this in a moment, thanks for the insights.. now its clear why it is not included


Link-style custom file control

It would be really nice to have a custom file control which would style like a link, as the one GitHub uses for attaching files to issues.

What do you think?

To clarify, github's own implementation is a combination of an actual <input type="file"...> that is visually hidden, and a separate <button...> with custom javascript that then triggers the actual file input itself

also, github's file chooser does not provide a built-in indication of which file was selected (as it integrates it in the big comment/text field itself), so some additional way of indicating the chosen file for upload would need to be devised

Foundation uses a fairly simple solution: http://foundation.zurb.com/sites/docs/forms.html#file-upload-button. However, it does not display the name of the file that was selected (it's pure CSS, no JS).
Osvaldas Valutis wrote an article about a similar method that has a few more features: http://tympanus.net/codrops/2015/09/15/styling-customizing-file-inputs-smart-way/
I tend to favor an adaptation of the 2nd method.

Thank you @patrickhlauke for your response. I understand GitHub's implementation uses JavaScript, but in my opinion Bootstrap's implementation could be done with an hidden <input type="file"> inside a customized —as on .custom-file in v4.

@RyanZim I tried the Foundation's method and it works, now it's just a matter of styling a  as a link.
<input id="files" type="file" multiple class="sr-only">
Attach files by dragging & dropping, selecting them, or pasting from the clipboard.

@heuan sure i wasn't being dismissive, just adding some info/context for when we look at this more closely :)

A side note:
Maybe I'm wrong, but wrapping the input in a  seems a bit un-semanatic. Are there any drawbacks of the for attribute that I'm unaware of?
Osvaldas Valutis has an example that pretty well duplicates what .custom-file does: http://tympanus.net/Tutorials/CustomFileInputs/#file-7. It also doesn't rely on ::before and ::after nearly as much. Would this implementation be something that could be considered for .custom-file?

wrapping an input inside its own  is perfectly legit semantically, and something that's been used for a long time. there's no particular drawback to for, it's just a different way of marking up the same thing (commonly referred to as implicit labelling, see for instance https://www.w3.org/WAI/tutorials/forms/labels/#associating-labels-implicitly)

OK, just wondered.

I've achieved this with for and id.
Attach files by dragging and dropping,
selecting them,
or pasting from the clipboard.
<input id="attachmentsInputElement" type="file" multiple class="sr-only">

Thank you all for the suggestions!


I I wish you continued further development of the following features

new SMS message alert
read PDU and text SMS
GSM 7-bit alphabet and UCS2 support
concatenated (long) SMS support
delete SMS

Hi,
this library is meant for creating SMS messages, not to communicate with GSM modem (reading, alerts, deleting)
I 7-bit encoding and UCS2 is supported for creating new messages, same goes for concatenated messages.


Application Gateway v2 Pricing

Please clarify the pricing model for the v2 Application Gateway. In the Azure Portal I can set Capacity type, Minimum instances, and Maximum instances.
In this article there are references to Fixed price and Capacity Unit price. Can a better definition of these be provided? Which of these corresponds to the minimum and maximum instances set in the Portal? I'm thinking Capacity Unit? But then the pricing example here is confusing me.
"Fixed price = 744(hours) * $0.20 = $148.8
Capacity units = 744 (hours) * 10 capacity unit per instance * 5 instances * $0.008 per capacity unit hour = $297.6"
Is Fixed price per Application Gateway resource in Azure?
What is the pricing per instance?
Where do you set capacity units per instance, or is this automatic? If it's automatic, what is the need for having multiple instances?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: faa5b8a2-11a1-6a2d-07cb-ffcf39643ed4
Version Independent ID: 298a134e-fb27-f16d-bff2-d43eb0f766fd
Content: Autoscaling and Zone-redundant Application Gateway v2
Content Source: articles/application-gateway/application-gateway-autoscaling-zone-redundant.md
Service: application-gateway
GitHub Login: @vhorne
Microsoft Alias: victorh

@JoeRistine Thanks for the feedback!

Is Fixed price per Application Gateway resource in Azure?

The fixed price is independent of scale count, and is a flat cost associated to an App Gateway. If you have 2 App Gateway V2s running for a month with different instance counts, the fixed price will be the same.

What is the pricing per instance?

The Capacity Unit price is where the variable pricing comes in, and where the instance price is calculated. Capacity units per instance is usually at 10 per instance, but can change depending upon how you are utilizing the app gateway. Because the Capacity Units are partially dependent on the load, it can be difficult to estimate an exact price for an app gateway until you know how App Gateway will handle your applications load.

Where do you set capacity units per instance, or is this automatic? If it's automatic, what is the need for having multiple instances?

It is possible to both Set the number of instances, as well as have App Gateway automatically scale the number of instances. This doc is designed to cover both scenarios.
To set this number, you can use the "Configuration" area of  the portal:

Please let me know if you have any other questions!

@JoeRistine We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

@TravisCragg-MSFT Now that billing has started for Application Gateway v2, I find that this article does not explain the pricing model well at all. The article gives the impression that capacity unit billing is consumption based, which is partly true. It fails to mention that 10 capacity units is the minimum amount that is billed per instance. Also, the minimum instance count allowed via the Azure portal is 2 (while this can be set to 1 via PowerShell - this might be helpful to include in this article for non-production environments).
The minimum price for an Application Gateway WAF_v2 instance created in the Portal would be as follows (East US pricing).
Fixed price - $0.36 per gateway-hour * 730 hours = $262.80
Consumption price - Minimum of 2 instances allowed by the Portal * minimum of 10 capacity units per instance - $0.0144 per capacity unit-hour * 20 capacity units * 730 hours = $210.24
Total minimum monthly cost = $473.04
One non-production environment I've implemented averages 0.42 Current Capacity Units since billing has started. It is still billed at 20 capacity units. Consumption can be slightly reduced by $105.12/month by setting the instance count to 1 via PowerShell.
Both Example 2 and Example 3 seem incorrect to me. How is it possible to get 4 capacity units in Example 2 or 8 capacity units in Example 3? Even at the bare minimum of 1 instance, minimum capacity units would be 10.

Example 2
An Application Gateway standard_v2 is provisioned for a month and during this time it receives 25 new SSL connections/sec, average of 8.88-Mbps data transfer. Assuming connections are short lived, your price would be:
Fixed price = 744(hours) * $0.20 = $148.8
Capacity unit price = 744(hours) * Max (25/50 compute unit for connections/sec, 8.88/2.22 capacity unit for throughput) * $0.008 = 744 * 4 * 0.008 = $23.81
Total price = $148.8+23.81 = $172.61

Example 3
An Application Gateway WAF_v2 is provisioned for a month. During this time, it receives 25 new SSL connections/sec, average of 8.88-Mbps data transfer and does 80 request per second. Assuming connections are short lived, and that compute unit calculation for the application supports 10 RPS per compute unit, your price would be:
Fixed price = 744(hours) * $0.36 = $267.84
Capacity unit price = 744(hours) * Max (compute unit Max(25/50 for connections/sec, 80/10 WAF RPS), 8.88/2.22 capacity unit for throughput) * $0.0144 = 744 * 8 * 0.0144 = $85.71
Total price = $267.84 + $85.71 = $353.55

Either the examples provided are incorrect or Azure billing is incorrect. The consumption details for the non-production instance I mentioned are below. Capacity units used for this instance averaged 0.42 over the time period below, but is billed at 20 capacity units.
$              0.86 | Application Gateway WAF v2 - Capacity Units - US East | 60.00 | 7/3/2019 0:00 | 7/3/2019 23:59
$              6.91 | Application Gateway WAF v2 - Capacity Units - US East | 480.00 | 7/4/2019 0:00 | 7/4/2019 23:59
$              6.91 | Application Gateway WAF v2 - Capacity Units - US East | 480.00 | 7/5/2019 0:00 | 7/5/2019 23:59
$              6.91 | Application Gateway WAF v2 - Capacity Units - US East | 480.00 | 7/6/2019 0:00 | 7/6/2019 23:59
$              6.91 | Application Gateway WAF v2 - Capacity Units - US East | 480.00 | 7/7/2019 0:00 | 7/7/2019 23:59
$              1.08 | Application Gateway WAF v2 - Fixed Cost - US East | 3.00 | 7/3/2019 0:00 | 7/3/2019 23:59
$              8.64 | Application Gateway WAF v2 - Fixed Cost - US East | 24.00 | 7/4/2019 0:00 | 7/4/2019 23:59
$              8.64 | Application Gateway WAF v2 - Fixed Cost - US East | 24.00 | 7/5/2019 0:00 | 7/5/2019 23:59
$              8.64 | Application Gateway WAF v2 - Fixed Cost - US East | 24.00 | 7/6/2019 0:00 | 7/6/2019 23:59
$              8.64 | Application Gateway WAF v2 - Fixed Cost - US East | 24.00 | 7/7/2019 0:00 | 7/7/2019 23:59
$              0.00 | Bandwidth - Data Transfer Out - Zone 1 | 0.01 | 7/3/2019 0:00 | 7/3/2019 23:59
$              0.00 | Bandwidth - Data Transfer Out - Zone 1 | 0.07 | 7/4/2019 0:00 | 7/4/2019 23:59
$              0.01 | Bandwidth - Data Transfer Out - Zone 1 | 0.09 | 7/5/2019 0:00 | 7/5/2019 23:59
$              0.00 | Bandwidth - Data Transfer Out - Zone 1 | 0.05 | 7/6/2019 0:00 | 7/6/2019 23:59
$              0.00 | Bandwidth - Data Transfer Out - Zone 1 | 0.05 | 7/7/2019 0:00 | 7/7/2019 23:59

@JoeRistine If you are being charged different from how this doc describes, your best bet is to create a support request for them to validate the charges and rates you received. If you do not have a support plan, please let me know!

@joeRistine @lenCardinal can you post your Support Request # (SR#) here as well? I would love to see what they say.
I am also going to re-open this issue and assign it to the doc author to re-evaluate the pricing descriptions & examples and make sure they are correct.

Sure, it's<PHONE_NUMBER>04940 @TravisCragg-MSFT

Mine was:<PHONE_NUMBER>04021
I did not receive any response from Support on this case. Eventually, I just figured out that V2 AG’s are billed at a minimum of 20 CU’s per hour (explained in my earlier posts). Support did not refute it.
As for the documentation (assuming I’m understanding this correctly, based on Joe’s, others’, and my experiences), the Pricing section of the doc should be updated to state: (a) a full 10 CU’s are billed for each of the “minimum instances” – regardless of usage – and (b) The minimum # instances per AG is 2. So a minimum of 20 CU’s (10x2) are going to be billed for every AG. The documentation examples would be updated accordingly.
Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Joe Ristine<EMAIL_ADDRESS>Sent: Wednesday, September 4, 2019 4:00:49 PM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Len Cardinal<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Application Gateway v2 Pricing (#33601)
Sure, it's<PHONE_NUMBER>04940 @TravisCragg-MSFThttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FTravisCragg-MSFT&data=02|01||57292da86eae4ac6120908d731729596|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637032240505948862&sdata=QqfOw6XK8ZKzGsWB8kuvL%2FW%2FjNUF4ZohwPpRa%2F4ONr4%3D&reserved=0
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F33601%3Femail_source%3Dnotifications%26email_token%3DADRDDTQKOVPH56C5RU2FF3DQIAHXDA5CNFSM4HZLGVQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD54ZHNY%23issuecomment-528061367&data=02|01||57292da86eae4ac6120908d731729596|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637032240505958867&sdata=I3%2Bf2WmhDi9GtT%2BEWmoDZSU2iwTur8g%2FRe0t8xThfbc%3D&reserved=0, or mute the threadhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADRDDTV6NSFAZNF5DH5V73DQIAHXDANCNFSM4HZLGVQQ&data=02|01||57292da86eae4ac6120908d731729596|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637032240505968872&sdata=LBFVuC8bp0VlvKsMHZPP8PrmOgCvt3sfa5ONd9475eI%3D&reserved=0.

Response from Microsoft:

Hello Joe,
Thank you for your patience.
I have investigated this further with the Product Group for Application Gateway and have a confirmation that Capacity Units will be billed as
10 * No of Instances. Hence there is a minimum charge of 10 CUs per instance.
We have currently documented the above here:
https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-autoscaling-zone-redundant#pricing
Relevant fragments:
Capacity Unit price - This is consumption-based cost that is charged
in addition to the fixed cost. Capacity unit charge is also computed hourly or partial hourly.
Note
Each instance can currently support approximately 10 capacity units
But, we have also acknowledged the fact that there is little ambiguity in the documentation and
we could improve the communication of this on our public facing documentation.
My PG  is discussing this internally and will move the change into production in next 2-3 weeks (Expected ETA).
Do let me know if you have any further questions or concerns on this case.

Thanks, Joe,
It looks to me like they are still missing the point. I guess let's wait and see.

Based on their docs here for V2_standards
https://azure.microsoft.com/en-us/pricing/details/application-gateway/

Fixed
$0.20 per gateway-hour

Capacity Unit1
$0.008 per capacity unit-hour

I spun up a new V2_standard AG and left it inactive for 2 days.  Each day I had a $8.20 cost.  Based on the info about the 20 CUs default per instance, this still doesn't make sense to me.
24h * $0.2 = $4.80
20CU * 24h * $0.008 = $3.84
That brings the total to $8.64, which is $0.44 higher than actual.  Now I'm guessing because I'm using a corporate account, there might be discounts, but it's hard to actually plan what the cost would be w/o some detailed documentation on how AG pricing is calculated.  It would really be nice to get some clarity around this.
I spun up a V2_Standard for two days with no activity on it.

(dat: I'm guessing that the price difference is a discount.)
It looks like there has been some progress here. The Azure Portal now lets you set minimum instances=0 for AutoScale deployments. And the detailed pricing explanation at https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-autoscaling-zone-redundant has been updated to mention the '0' option for minimum instances. Such a configuration should result in CU pricing based purely on usage.
Details: Back when I and others were carping about CU charges for no usage, the detailed pricing information at https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-autoscaling-zone-redundant#pricing did not mention a 'minimum CU's.'. Lower in the page, however, the section https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-autoscaling-zone-redundant#scaling-application-gateway-and-waf-v2 mentioned that 'reserved instances' were charged at 'full capacity' (10 CU's). 'Reserved' being the 'minimum instances' in an AutoScale deployment, or the fixed number of instances in a Manual deployment. On top of that, the Azure Portal forced a minimum of two instances* (at least with AutoScale), which resulted in 20 CU's per month minimum.
Now, the Azure Portal allows configuration of zero minimum instances for AutoScale. And the section https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-autoscaling-zone-redundant#scaling-application-gateway-and-waf-v2 mentions the 0 minimum instance option.**
So....now, you should be able to create an Auto-Scaling AG with Minimuminstances='0', and have CU's be calculated solely based on usage. That is great!

A few months ago, @JoeRistine was able to use PowerShell to set min instances to '1' (though the Portal marked that as invalid). POSH or some other automation might have allowed AutoScale deployments with zero minimum instances...avoiding the minimum CU charges.

** Would be interesting & important to know if a deployment with '0' minimum instances potentially results in a noticeable 'warm-up' problem, when traffic starts up after a period of inactivity.

Shorter answer: the following should result in CU charges based purely on usage:

Use Auto-Scaling configurations
Set "Minimum autoscale instances" to '0':

@LenCardinal That's very helpful information, thank you!  On an inactive AG that is a ~42% savings over a 2 instance manual, so that's great.  Would still be nice to have all this clearly added to the price calculators @TravisCragg-MSFT.  I'll run some tests with this setup today.

@JoeRistine @LenCardinal @datfinesoul  Thank you all for your additional follow ups on this issue!
@vhorne currently has the action to update the pricing in the doc.

Am I understanding the pricing correctly, in that the v2 standard sku fixed price is about 10 times more expensive than the price for a v1 small standard sku? That's crazy!
We are currently setting up our test and stage environment, and the plan was to use an application gateway in front of each of our four VM scale sets (each scale set is a completely different service, so we can't join them). That would mean a total of 8 application gateways in non-production. That's $1400 per month in fixed costs! That is a massive increase in costs compared to the v1 sku.
Is there really no v1 version suitable for development/testing environments, with virtually no traffic (3 person team, only traffic is when we actively test something)? We are willing to reduce the "power" of each application gateway, not the number of gateways used. And we don't want a vastly different setup compared to production (meaning: we want v1 in both prod and non-prod, or v2 in both).

Am I understanding the pricing correctly, in that the v2 standard sku fixed price is about 10 times more expensive than the price for a v1 small standard sku? That's crazy!
We are currently setting up our test and stage environment, and the plan was to use an application gateway in front of each of our four VM scale sets (each scale set is a completely different service, so we can't join them). That would mean a total of 8 application gateways in non-production. That's $1400 per month in fixed costs! That is a massive increase in costs compared to the v1 sku.
Is there really no v1 version suitable for development/testing environments, with virtually no traffic (3 person team, only traffic is when we actively test something)? We are willing to reduce the "power" of each application gateway, not the number of gateways used. And we don't want a vastly different setup compared to production (meaning: we want v1 in both prod and non-prod, or v2 in both).

Yes - I just found this out the hard way.  I selected V2 due to the Azure documentation stating that V1 was being phased out not realising it is more than 10X the cost. Not sure if that changes, but I am now trying to get V1 APPGW to work for my solution. There is clearly a difference between V2 and V1 because having a lot of issues with the exact same configuration.

#reassign:winthrop28

Hi all,
I'm confused about the v2 pricing model. I understand there is a fixed price for using an APGW v2 and a price related to CUs usage. It does not mention how APGW distributed on Availability Zones are priced. So if I deploy an APGW on 3 AZ, will I be charge for 3x the fixed price? Or if I have 10 CUs configured, will I be charge 3x 10CUs price? Or anything else?
Thnaks.

Hi @TravisCragg-MSFT I'm looking at AppGW and find it very confusing as well. Here's an example:

Pricing page https://azure.microsoft.com/en-us/pricing/details/application-gateway/
It mentions Capacity Unit, with a per-capacity unit hour charge.

Pricing on MSDN docs https://docs.microsoft.com/en-au/azure/application-gateway/application-gateway-autoscaling-zone-redundant mentions Capacity Unit price

When in the azure portal, it talks about scaling using Scale units which is not defined anywhere

Confirming a "scale unit" is 1x VM and has 10x Capacity Units?

Scaling from 2 to 3 Scale units shows an increase from 20 Capacity Units to 23.33 Capacity Units, hows that work? It would make sense if this was 20 -> 30 (1 scale unit, which is really just a VM, is 10 CUs) so its not clear to me the relation between Scale unit and Capacity units (which supports a certain workload + cost).

The sentence Please note that even if you configure autoscaling with zero minimum instances the service will still be highly available, which is always included with the fixed price contradicts However, creating a new instance can take some time (around six or seven minutes). Hence, if you do not want to cope with this downtime you can configure a minimum instance count of 2, ideally with Availability Zone support --- I'm assuming the "fixed cost" is really 1 VM instance, so if it goes down AppGW is offline until a new instance is made, hence the suggestion for 2 scale units/20 CUs? https://docs.microsoft.com/en-au/azure/application-gateway/application-gateway-autoscaling-zone-redundant

This may not be fully on topic, but I feel it needs to be addressed here. The pricing makes no sense what-so-ever. How is it that the smallest manual deployment of Application Gateway v2 ($0.28/hour), costs nearly as much as a single Large Application Gateway v1 ($0.32/hour)?
I know this isn't the location for recommending Microsoft make the pricing better. Instead, the documentation needs to show how the price is justified. There is no cost effective way to deploy a small scale v2 app gateway, unless the v2 app gateway will NEVER be meant for smaller deployments, and v1 will NEVER go away.
Right now, I just don't see how it compares...

Hey Team! We are approaching a year since I last commented on how the price explanation is confusing. I am not asking anyone here to change the prices. But, it is still nearly impossible to decipher how much the Application Gateway v2 is going to cost.
Can you please add more examples to the documentation?

Hi @adagioajanes Is the information at https://learn.microsoft.com/en-us/azure/application-gateway/overview-v2#pricing and https://learn.microsoft.com/en-us/azure/application-gateway/understanding-pricing sufficient to close this issue?
I took ownership of the articles for this product last year and I'm currently going through all the github issues. This one is quite old, but I see you've commented several times so I want to pursue this.  Please let me know if the current docs are satisfactory.

#reassign:greg-lindsay

I am closing issues that are older than a year in order to research and cross reference them individually. I am recording each issue and compiling a list of articles to review. If you feel there is an urgent need to address something related to this GitHub issue, please open a new issue and I will reply in the new thread. Please also reference this GitHub issue number in the new issue. Thanks. #please-close


Add encoder samples and update CHANGELOG.md

For issue https://github.com/Azure/azure-sdk-for-python/issues/26318

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-data-tables


Replace all 9999 availability in non-stdlib tests.

Follow-on to https://github.com/apple/swift/pull/26108

@jckarter Mainly tagging you to check that you folks haven't yet added anything to these tests that actually shouldn't be available in 5.1-aligned SDKs, and that there aren't other reasons to maintain 9999 for these.

@swift-ci please test

@swift-ci please test

Build failed
Swift Test Linux Platform
Git Sha - a62137ece0734e2f39a39af0cb7d964f539f9867

Build failed
Swift Test OS X Platform
Git Sha - a62137ece0734e2f39a39af0cb7d964f539f9867

Linux failure is in LLDB, looks unrelated to this change: 09:44:25 error: missing required modules: 'Swift', 'SwiftOnoneSupport'
Modules again?

@swift-ci smoke test linux

Checked in with @dcci about the LLDB failures, we think this is not a PR blocker, so merging with the linux smoke-test only.