[Performance] Onyx.get batching

In the benchmarks performed here: https://github.com/Expensify/Expensify.cash/issues/2667#issuecomment-874160913 I've noticed that sequential reads happen very fast 8-9ms, much faster than what it takes in E.cash
I think the reason is that Onyx will blast AsyncStorage with as many calls as it need.
Browsing the AsyncStorage code I've found this optimization
AsyncStorage#multiGet
    if (!this._immediate) {
      this._immediate = setImmediate(() => {
        this._immediate = null;
        this.flushGetRequests();
      });
    }
/.../

    this._getRequests.push(getRequest);
    // avoid fetching duplicates
    keys.forEach((key) => {
      if (this._getKeys.indexOf(key) === -1) {
        this._getKeys.push(key);
      }
    });

When multiGet is called multiple times in a very short span of time (1 tick) it would batch all the calls that happen and invoke flushGetRequests on the next tick

1 trip across the bridge
1 sql query

From the docs of flushGetRequests

The following batched functions are useful for executing a lot of operations at once, allowing for native optimization

I think that's exactly what happens when E.cash starts -  a lot of operations at once

Proposal
Test the following theory:
Update Onyx.get to use multiGet internally when it fetches data from AsyncStorage this way the operations that happen at the same time would be batched
We can try implementing our own get batching that waits a bit longer 2-3ms to capture more get calls and then make a multiGet request
Sample benchmarks
Reading a random key and ~1kb text value
Based on 50+ tests of each

Getting 10 items using AsyncStorage.getItem in parallel:

time to complete: min: 53.7ms, max: 72.1ms, avg: 62ms

Getting 10 items using a single AsyncStorage.multiGet:

time to complete: min: 21.3ms, max: 95.7ms, avg: 32.3ms

Getting 100 items using AsyncStorage.getItem in parallel:

time to complete: min: 546.1ms, max: 644.8ms, avg: 602.7ms

Getting 100 items using a single AsyncStorage.multiGet:

time to complete: min: 46.4ms, max: 76.6ms, avg: 59.6ms

I've observed that there are gains even for 10 parallel calls made to AsyncStorage.getItem, while as we get higher the performance degrades exponentially, multiGet is not affected as much by the 10x increase of throughput
Additional notes
Write operations take about twice as much and can be optimized the same way
For my sample chat there are around 80-100 calls to Onyx.set during init with the following timings:

max: 2.08sec
min: 22.200ms
avg: 275.237ms

I like the idea of batching everything in one tick, but I'm not a fan of batching arbitrarily based upon time frames. "Magic number"  solutions where we pick some arbitrary number (why 2ms vs 20m or 200ms) out of the hat require constant maintenance find the right numbers, and those numbers can change that are effectiveness unpredictably.
I like the idea of batching until the JS engine is idle (which if I understand correctly, is what you mean by one tick), but I'm not sure the argument why we would wait longer than that.

why 2ms vs 20m or 200ms

Let's say an average read from disk takes anywhere between 50ms to 500ms, by delaying execution by up to 10% (2-5ms) we can save the roundtrip for potential requests that might happen in this window of time
This is very likely to happen during init where more data can be captured this way

a tick might be enough to batch 2 requests where a "forgivable" delay would capture 20
it's a tradeoff hard reads would happen 2-5ms slower but every time a 2nd request is batched you save 50-500ms

The idea here is the first get triggers the batch of 2-5ms then the batch flushes along with any other gets arrived by that time

I like the idea of batching until the JS engine is idle (which if I understand correctly, is what you mean by one tick)
We can't reliably detect that, and it might be too long to wait to get idle, usually reads are triggered by components that need data "now" and can't afford to wait to idle. Going this direction would make complicated logic "to feed the most hungry first"

"next tick" is the next iteration of the event loop, it's very similar to a setTimeout() with a 0ms delay

Just to confirm, reads are triggered by user or network action right?  So
the only difference between "batching one tick" or "batching one tick +
20ms" is the possibility that additional user/network activity happened in
that tiny 20ms window, right?  It feels like the odds of that helping are
extremely low.
On Tue, Jul 6, 2021 at 9:28 AM Peter Velkov @.***>
wrote:

why 2ms vs 20m or 200ms
Let's say an average read from disk takes anywhere between 50ms to 500ms,
by delaying execution by up to 10% (2-5ms) we can save the roundtrip for
potential requests that might happen in this window of time
This is very likely to happen during init where more data can be captured
this way

a tick might be enough to batch 2 requests where a "forgivable"
delay would capture 20
it's a tradeoff hard reads would happen 2-5ms slower but every time
a 2nd request is batched you save 50-500ms

The idea here is the first get triggers the batch of 2-5ms then the batch
flushes along with any other gets arrived by that time
I like the idea of batching until the JS engine is idle (which if I
understand correctly, is what you mean by one tick)
We can't reliably detect that, and it might be too long to wait to get
idle, usually reads are triggered by components that need data "now" and
can't afford to wait to idle. Going this direction would make complicated
logic "to feed the most hungry first"
"next tick" is the next iteration of the event loop, it's very similar to
a setTimeout() with a 0ms delay
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Expensify/react-native-onyx/issues/84#issuecomment-874906632,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEMNUQSJ4MEGY6A7KLIMBLTWMVJFANCNFSM474CEQZQ
.

Just to confirm, reads are triggered by user or network action right? So the only difference between "batching one tick" or "batching one tick + 20ms" is the possibility that additional user/network activity happened in that tiny 20ms window, right? It feels like the odds of that helping are extremely low.

Rendering a screen does not happen all at once, it also does not trigger the needed Onyx connections at once - it's more like an avalanche effect
A tree is rendered from the children to the parent
First the child components mount and their lifecycle methods triggered (componentDidMount)
Some of these children will initiate Onyx connections
Then the component branch above this tree renders, it again triggers Onyx connections
Then the next and so on
This process does not happen in a single tick, but happens very fast
Implementing batching with a delay/debounce can capture more get calls as the avalanche is building up
We did something similar with hooking into existing reads - when 2 components want the same thing, but it's not in cache. When the 2nd component mounts instead of starting a new call to get the "thing" it hooks into the pending read result
It was proven that this happens and is beneficial
This happens mostly during init where many components are added to the tree as the app initializes
This is the theory

App init
static Onyx.connect calls in .js files are evaluated first. These are the calls most likely to get picked up in a batch within a single tick
Auth state and data is evaluated and the app continues to render the relevant component Tree
Tick(s) pass
Navigation is mounting
Tick(s) pass
Top level navigator stacks are mounting
Tick(s) pass
The active screen begins mounting
As we get deeper different Onyx data needs to be resolve

We won't be able to capture all of this in a single batch as some items are codependent - you need to resolve Auth to continue with the rest of the app, but if we manage to group larger batches we should see a big improvement in time to resolve data from storage

Great, thank you for that!  However, there still isn't actually any "delay" between the avalanching cycles -- if one cycle of Onyx.gets() triggers a second cycle of Onyx.gets(), it's not like it waits 2-20ms between "ticks".  Rather, immediately after one tick completes, the next one starts.
So aren't we really trying to wait for all the cycles to stop avalanching, before doing any Onyx.gets()?  In other words, we aren't actively looking to wait any amount of time at all.  Rather, we just want to wait until the avalanche is "done" before sending all the get over in a large batch.
If that's correct, are you saying that it's in practice very difficult to detect when the avalanche is done, and thus there's no more need to wait?  Why wouldn't setTimeout(0) work -- why does the delay need to be nonzero?

So aren't we really trying to wait for all the cycles to stop avalanching, before doing any Onyx.gets()?  In other words, we aren't actively looking to wait any amount of time at all.  Rather, we just want to wait until the avalanche is "done" before sending all the get over in a large batch.

Yes ideally we would hook to some event instead of waiting an arbitrary time, but I'm not sure if that's possible at all.

Why wouldn't setTimeout(0) work -- why does the delay need to be nonzero?

setTimeout(0) would be about the same as with the next tick, and is just not enough
Hey, remember the xlsx that I did originally they contain start time information of each Onyx.get call, check the start times of the get calls here: https://github.com/Expensify/react-native-onyx/pull/70#issuecomment-840220867
2ms window

1350859.00ms
1350861.00ms

5ms window

1350935.00ms
1350937.00ms
1350939.00ms

Here's a full xls: https://docs.google.com/spreadsheets/d/15GkUEUOyVLk02y8_-PFAJFF4Ej7OsyXfdbyJgFoC_4Q/edit?usp=sharing
The only time I see where get calls are on the same millisecond is for calls that happen in a foreach cycle, but this is handled in https://github.com/Expensify/react-native-onyx/issues/78 where instead of for-eaching we can directly call multiGet with all the keys
So in reality there are calls that happen in very close proximity, it's just how React rendering works and Onyx relies on that.
A batching with delay can serve as a good POC whether we'll gain something out of this and then we can try to improve by hooking to something exposed by React to sync with it's rendering process

Got it, if we can't accurately "hook" when the thread is otherwise idle,
then a short timeout might be the best we can do.  Thanks for walking me
through it!
On Tue, Jul 6, 2021 at 12:45 PM Peter Velkov @.***>
wrote:

So aren't we really trying to wait for all the cycles to stop avalanching,
before doing any Onyx.gets()? In other words, we aren't actively looking to
wait any amount of time at all. Rather, we just want to wait until the
avalanche is "done" before sending all the get over in a large batch.
Yes ideally we would hook to some event instead of waiting an arbitrary
time, but I'm not sure if that's possible at all.
Why wouldn't setTimeout(0) work -- why does the delay need to be nonzero?
setTimeout(0) would be about the same as with the next tick, and is just
not enough
Hey, remember the xlsx that I did originally they contain start time
information of each Onyx.get call, check the start times of the get calls
here: #70 (comment)
https://github.com/Expensify/react-native-onyx/pull/70#issuecomment-840220867
2ms window

1350859.00ms
1350861.00ms

5ms window

1350935.00ms
1350937.00ms
1350939.00ms

Here's a full xls:
https://docs.google.com/spreadsheets/d/15GkUEUOyVLk02y8_-PFAJFF4Ej7OsyXfdbyJgFoC_4Q/edit?usp=sharing
The only time I see where get calls are on the same millisecond is for
calls that happen in a foreach cycle, but this is handled in #78
https://github.com/Expensify/react-native-onyx/issues/78 where instead
of for-eaching we can directly call multiGet with all the keys
So in reality there are calls that happen in very close proximity, it's
just how React rendering works and Onyx relies on that.
A batching with delay can serve as a good POC whether we'll gain
something out of this and then we can try to improve by hooking to
something exposed by React to sync with it's rendering process
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Expensify/react-native-onyx/issues/84#issuecomment-875034133,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEMNUXWRT5TZXVRUEMIA5DTWNMMTANCNFSM474CEQZQ
.

Some more timing information, this time on iOS
iOS Onyx.get timings durring init.xlsx
Calls are ordered by startTime
A lot of keys are retrieved between 1.563 to 1.567
The ones taking less than 10ms are coming from cache - you can ignore them
But there are a a lot of other keys that can be batched together for a hard read

I think that's exactly what happens when E.cash starts - a lot of operations at once

I'm having trouble following some of this as it's a bit over my head. But it sounds like we are proposing that batching calls to Onyx.get() will improve the time it takes for the app to start (+ other things more generally) since we "get a lot of things from storage" when the app first boots up.
Does that sound accurate?
Can we easily test this by:

Time out how long it takes to cold boot the app before and after these changes
Time how long it takes to render chat A while chat B is open before and after these changes

I've tested this while working on #88
just using multiGet instead of get is enough to batch ~30 keys right at the start of the app
Then as the app continues to load 2-3 keys are fetched together from storage occasionally

We can try implementing our own get batching that waits a bit longer 2-3ms to capture more get calls and then make a multiGet request

Looks like this wont be needed, or we can use setImmediate
Even though retrieval became about twice as fast, the app seem to became slower as many withOnyx component would receive data at once , which caused hangs
Having some delays between notifying Onyx subscribers seem to make the app to run better...

just using multiGet instead of get is enough to batch ~30 keys right at the start of the app
Then as the app continues to load 2-3 keys are fetched together from storage occasionally
Even though retrieval became about twice as fast, the app seem to became slower as many withOnyx component would >receive data at once , which caused hangs

Cool thanks for the summary! Lemme know if I've got the general idea correct...

We could batch calls to Onyx.get() when the app inits and this will make data retrieval twice as fast vs. not doing this

Even though retrieval became about twice as fast, the app seem to became slower as many withOnyx component would receive data at once , which caused hangs

However, this creates a new problem to deal with which is...

the app seem to became slower as many withOnyx component would receive data at once

And this is not normally observable because withOnyx connected components are usually updated in a more staggered way and not all at once? Any more thoughts on the React rendering side of this? Why is it that when we put space between updating subscribers there are no issues, but when we update them all at once there is a hanging effect?

The "hang" effect happens when switching chats and only on Android.
It seems to be related to:

general CPU intensive rendering of that view on a "weaker" device cause it to hang. This might be improved if the chat items become cheaper to render

staggering seems to help a lot in Android, otherwise it's overwhelmed with data it has to render, maybe this is a part where the Virtualization Config can play a role and not allow to pre-render too much. Other FlatList / ActionItem improvements can help as well

iOS native implementation of the FlatList might be better than Android

Since I already have the change in git history, I can put it in a separate branch to test again after improvements in E.cash

Since I already have the change in git history, I can put it in a separate branch to test again after improvements in E.cash

Sounds good. Let's put this on HOLD for now.

Gonna take this back off hold now so we can keep discussing this...

I think that's exactly what happens when E.cash starts - a lot of operations at once

@kidroca Seems like this is happening for sure. For good measure, did some logging from the native layer and can see about 400 separate multiGet calls in the native layer when the app inits. Saved some of the data here (though not sure it's helpful as we already know there are many native calls happening).
https://docs.google.com/spreadsheets/d/e/2PACX-1vSsR3pqvLkNGOSWVx72OdTngYXjKnPPj1jPYxA2W8gtiNDoNREtZTi-SkyHiEAQqsjryT6I-BNSKrFs/pubhtml
Feels like we have a few options to explore, but I'm not sure which ones are worth pursuing next...

Switch over to multiGet instead of getItem
Do the second proposal in this issue

For both of these, I'm curious how the new Onyx version of a multiGet would utilize the cache.getPendingTask() logic we are using here. Just looking at what kinds of keys are getting retrieved on init and it seems likely that we might try to multiGet keys we are already in the process of getting.

AsyncStorage.getItem would internally call the native module multiGet method - this is the reason for the big number of separate multiGet call that fetch a single key
But AsyncStorage.multiGet JS implementation is actually better suited for single gets, because it uses setImmediate to schedule a single native multiGet request for all the individual calls that happened up until that moment
https://github.com/react-native-async-storage/async-storage/blob/58f5f0e066c97c1932fe29f23dd120ae67f6f71f/src/AsyncStorage.native.js#L302-L311
I'm surprised why that's not used internally for getItem as well
I think we can implement both proposals - they both need a multiGet implementation anyway

Implement mutliGet and use it instead of getItem (https://github.com/kidroca/react-native-onyx/commit/f29cfc9ce689ca83312ee36720926ced6af1e223)
Just use mutliGet when we're dealing with collection keys  (https://github.com/kidroca/react-native-onyx/commit/137012b91664d95a74ccb956ba518f30b6345a96)

I'm curious how the new Onyx version of a multiGet would utilize the cache.getPendingTask()

Here's how that is handled:

We're still registering a separate pending task for each key we don't have in cache
Then make a single multi get request for those keys
At the end all separate pending tasks are resolved at once when the multiGet finishes

https://github.com/kidroca/react-native-onyx/blob/f29cfc9ce689ca83312ee36720926ced6af1e223/lib/Onyx.js#L38-L98
One thing to note is there won't be much use for the capture/pending task logic if we fetch all finite items with one multiGet at the start
There just won't be overlapping key requests

I was thinking of opening a separate ticket but here seems a good place to discuss as well
Proposal
Onyx keeps track of keys with infinite size - the "Safe Eviction Keys". The data stored under such keys can grow as long as there is storage for it.
ATM the list is solely the Report Action history
Using the "safe eviction keys" and "all keys" we can derive the "finite keys"
These are small size keys that are no larger than a kB each
They contain frequently used information like session, network, currentUrl, and general report information
The typical size these keys take is around 20-30 kB of data
The only thing that makes the size vary is the general report information, each report would add 1kB of information
Problem
Currently the entire "finite" information is requested and read during init, it just happens in chunks as each components subscribes to different data
Solution
Identify all "finite" keys and load them into cache with one multiGet request
Then initialize the app
Additional Consideration
Since general report information (typically needed to display reports in the LHN) is stored in separate keys and each one is about 1kB it might not be desired to fetch these in one multiGet
Perhaps these keys should be flagged as safe eviction as well e.g. what if you have 1000 such keys
Correct me if I'm wrong but I think currently we read all such keys to fill the LHN list of conversations
IMO these keys should be flagged as safe eviction as well and at some point a paging solution introduced so that we don't have to read 1000 keys, but perhaps read them incrementally as the users scrolls the LHN

I think we should create a new issue if it's OK. I have some thoughts and would like to reply, but feels like a different problem to the problem of making many unoptimized data retrieval across the bridge.

I'm surprised why that's not used internally for getItem as well

This is a good question. I'd be curious to hear what the maintainers think.

We're still registering a separate pending task for each key we don't have in cache

Oh I see, so we will still use get(key) it just won't call AsyncStorage.getItem() and AsyncStorage.multiGet() handles resolving the promises to the correct places in this code?
Btw, it also looks like this code avoids fetching duplicates so perhaps the pending task is not needed
    // avoid fetching duplicates
    keys.forEach((key) => {
      if (this._getKeys.indexOf(key) === -1) {
        this._getKeys.push(key);
      }

FWIW, just swapping out getItem for multiGet with no other changes will batch a large number of keys during init.
I logged this from flushGetRequests while starting up the Android app:

DEBUG  [AsyncStorage] Getting keys  ["currentURL", "preferredLocale", "credentials", "myPersonalDetails", "currentlyViewedReportID", "user", "countryCode", "policy_010931807F672475", "policy_08D0600B8AAE868D", "policy_137DA25D273F2423", "policy_43BC6076D3D1418B", "policy_456619EAC8A1A4BF", "policy_474ABBB288DAC2E0", "policy_6B40F9444C3F2EFF", "policy_7A35FD885AE104E1", "policy_A55A7962A1F001A4", "policy_C25E8D06D27010EB", "policy_CD36E1BEF9329100", "reportIOUs_73847679", "reportIOUs_74142904", "personalDetails", "reportActions_60432478", "reportActions_61607959", "reportActions_64217852", "reportActions_64218347", "reportActions_64218390", "reportActions_64218473", "reportActions_64220748", "reportActions_64220825", "reportActions_64224572", "reportActions_64224777", "reportActions_64224791", "reportActions_64224801", "reportActions_64226902", "reportActions_64533058", "reportActions_64867940", "reportActions_64889352", "reportActions_65092250", "reportActions_65094882", "reportActions_65219439", "reportActions_65279574", "reportActions_65298677", "reportActions_65341306", "reportActions_65357746", "reportActions_65501330", "reportActions_65504418", "reportActions_65608585", "reportActions_65637646", "reportActions_65639802", "reportActions_65728247", "reportActions_65759622", "reportActions_65764596", "reportActions_65969070", "reportActions_66048205", "reportActions_66135402", "reportActions_66282008", "reportActions_66387047", "reportActions_66600097", "reportActions_66701368", "reportActions_66903522", "reportActions_67776049", "reportActions_68940003", "reportActions_69104135", "reportActions_69104809", "reportActions_69353005", "reportActions_69379828", "reportActions_70008016", "reportActions_70270997", "reportActions_70768603", "reportActions_71054527", "reportActions_71245420", "reportActions_71401209", "reportActions_71787592", "reportActions_72162014", "reportActions_72643259", "reportActions_72645341", "reportActions_74050763", "reportActions_74200583", "reportActions_74538596", "reportActions_75168433", "reportActions_75320708", "reportActions_75326778", "reportActions_75362078", "reportActions_75380790", "reportActions_75380798", "reportActions_75380810", "reportActions_75382121", "reportActions_75691589", "reportActions_75950515", "reportActions_76185608", "reportActions_76408728", "reportActions_76578590", "reportActions_77036495", "reportActions_77154182", "reportActions_77291418", "reportActions_77334248", "reportActions_77543235", "reportActions_77583619", "reportActions_77883250", "reportActions_77996837", "reportActions_78058869", "reportActions_78395461", "report_60432478", "report_61607959", "report_61717071", "report_64217661", "report_64217852", "report_64218347", "report_64218390", "report_64218473", "report_64220748", "report_64220825", "report_64224572", "report_64224777", "report_64224791", "report_64224801", "report_64226902", "report_64533058", "report_64867940", "report_64889352", "report_64981914", "report_64993910", "report_65030406", "report_65083262", "report_65092250", "report_65094882", "report_65125296", "report_65170196", "report_65171050", "report_65219439", "report_65279574", "report_65285640", "report_65298677", "report_65341306", "report_65357746", "report_65376110", "report_65501330", "report_65504418", "report_65608585", "report_65637646", "report_65639802", "report_65693396", "report_65728247", "report_65759622", "report_65764596", "report_65851328", "report_65969070", "report_66048205", "report_66112934", "report_66120017", "report_66120977", "report_66135402", "report_66282008", "report_66378905", "report_66379571", "report_66382218", "report_66382245", "report_66384585", "report_66387047", "report_66431860", "report_66546050", "report_66553044", "report_66600097", "report_66603474", "report_66674648", "report_66701368", "report_66706140", "report_66838520", "report_66867504", "report_66903522", "report_66927665", "report_67381113", "report_67776049", "report_67858265", "report_67865722", "report_67968295", "report_67977534", "report_67999703", "report_68074734", "report_68104913", "report_68231412", "report_68248275", "report_68374837", "report_68374863", "report_68374866", "report_68411449", "report_68444993", "report_68597795", "report_68600061", "report_68621407", "report_68673342", "report_68694171", "report_68696210", "report_68696216", "report_68867940", "report_68918628", "report_68940003", "report_68992666", "report_69104135", "report_69104809", "report_69171208", "report_69180191", "report_69213673", "report_69289882", "report_69335433", "report_69353005", "report_69367134", "report_69379828", "report_69396275", "report_69396438", "report_69423271", "report_69530167", "report_69564775", "report_69590301", "report_69590603", "report_69624153", "report_69865968", "report_69905551", "report_69962302", "report_69993653", "report_70008016", "report_70270997", "report_70437049", "report_70491072", "report_70515871", "report_70768603", "report_70808211", "report_71054527", "report_71088813", "report_71163286", "report_71245420", "report_71304196", "report_71304204", "report_71304364", "report_71334762", "report_71338647", "report_71357584", "report_71401209", "report_71578546", "report_71615930", "report_71764953", "report_71787592", "report_71799824", "report_71867514", "report_72059927", "report_72162014", "report_72188098", "report_72206989", "report_72426645", "report_72626744", "report_72643259", "report_72645341", "report_73026878", "report_73115712", "report_73120478", "report_73734882", "report_73734893", "report_73734922", "report_73848337", "report_74050763", "report_74200583", "report_74486476", "report_74525032", "report_74538596", "report_75168433", "report_75320708", "report_75326778", "report_75362078", "report_75380790", "report_75380798", "report_75380800", "report_75380810", "report_75382121", "report_75573325", "report_75573326", "report_75573424", "report_75573426", "report_75573427", "report_75573429", "report_75573430", "report_75573431", "report_75573432", "report_75573433", "report_75573435", "report_75573436", "report_75691589", "report_75950515", "report_76185608", "report_76408728", "report_76578590", "report_77036495", "report_77154182", "report_77291418", "report_77334248", "report_77543235", "report_77583619", "report_77869117", "report_77883250", "report_77996837", "report_78058869", "report_78395461"]

That is about 300 or so keys normally are fetched one at a time over the bridge.
I kind of expected to maybe see some big reduction in start up time since the bridge flooding issue isn't happening anymore. But things are more or less the same. Which makes me wonder how much of a problem we have with the "bridge" or if the problem is somewhere else. Here's a quick side by side comparison of release builds (w/ multiGet is a bit faster - but not much - and also take with a grain of salt as I am only focusing on start up time which varies a bit).
https://user-images.githubusercontent.com/32969087/128400320-e801a1aa-9b25-4a6d-b9c1-15a797e18cda.mp4
I think these changes are maybe still worth doing, but the lack of a difference just makes me think that we are just "pushing the limits of AsyncStorage".

Btw, it also looks like this code avoids fetching duplicates so perhaps the pending task is not needed

It avoids duplicates but only for the current batch, while the pending tasks consider retrievals that are half way through

FWIW, just swapping out getItem for multiGet with no other changes will batch a large number of keys during init.

Yep tried that as well some time ago, the effect was noticeable but the app was slower back then

Which makes me wonder how much of a problem we have with the "bridge" or if the problem is somewhere else.

This way of making get calls still has the overhead of creating 300+ promises since calls are still happening one by one even though they are batched in the end. This will also capture 300 more promises for the capture/pending task optimisation.
So creating 600 callbacks in such a short time might be the same as flooding the bridge.
We know we have a problem with how withOnyx works, but I don't think that alone would cause this slowdown. Looking at the huge storage operation times I thought it was the bridge, perhaps something else is keeping it busy as well?
I've suggested to strip everything from the app try put it back together to identify what might be causing this, could it be the navigation setup with all the different Navigators that mount after auth? I colleague Android developer told me once that having many view stacks hurts performance

@marcaaron
I've created this ticket: https://github.com/Expensify/App/issues/4433 regarding https://github.com/Expensify/react-native-onyx/issues/84#issuecomment-893487588

I think this can be closed since we don't use async storage anymore?

Plus in addition we are now working on this PR:

https://github.com/Expensify/react-native-onyx/pull/315


Add Trivia minigame

Please do a thorough review.
Intended exposed class is trivia.cs, which owns a questions.cs, which owns many AnsweredQuestions. AnsweredQuestions own AskableQuestions, which are restricted versions that don't own the knowledge of which answer is correct. Wrote three tests, including one that simulates an entire round of trivia. A fourth test that I am not willing to put in main simulated a successful trivia round, and passed.
Trivia.cs implements Minigame.cs, though it implements only one required method.

git says your code failed

The failed build is happening because the testing packages from nuget aren't being installed correctly. If I remember correctly we fixed this at some point, but this branch is old enough that it might not have those fixes. We might have to do a rebase?

Ok the issue is fixed now, github-actions needed no changes, it was the project that was broken :(
Will make a PR for this


Create a reference document describing the code framework

If an R savvy person wishes to dive into the code for the project, they should have a road map. This document should plainly describe the roles of each function, and the general order in which they are called. It should contain a complete example that shows how to go from raw data into summarized bootstrap output.
There are a wide range of functions, but they all fall into these main groups:

Preparation of data
Core harvest/effort estimation (including sensitivity analyses)
Summary/reporting (tables, figures)
Interfaces
Utility

Somehow, the Rmd source framework should be described as well.

Closed in #129


doc: Add module docstrings and info about ~q()

I'm sure much more could be written here, but this makes common patterns in mathlib a little more understandable from the vscode hovers alone.

Are you planning to put general explanations somewhere else?

No, but I figured docstrings somewhere now was better than a plan much later.

Then this looks good to me, but I don't have merge rights

@dwrensha, could you take a quick look too? I seem to remember you writing some Qq code in the past.
If it looks good to you, I'll go ahead and merge.

Looks good to me. Thanks for doing this!


Shouldn't shuffle = True be set to shuffle = False in create_data_loaders_dictionairy?

Question about the following:
File: PyTorchModelTrainer.py
Class: PyTorchLSTMTrainer
Function: create_data_loaders_dictionary
In the parameters in the call to the DataLoader function in the above method, the parameter shuffle is set to True in your code. Shouldn't this be set to False since we are training a LSTM model (where the order of the data matters since we are training timeseries data)?

You're right.
It should be and I think it is set to false in the example config file. I'm not 100% sure if it handles the shuffle:false from the config file but it's worth checking!


Deps get upgraded without the upgrade command

As reported by @dpiddy, rebar3, on some source dependencies, will end up updating the lock file when tasks such as rebar3 ct get run.
In this case the source dependency was locked on {branch, master} and this one ended up fast-forwarded without ever calling upgrade.
There is currently no public repo to reproduce this, this is in private code.

The cause is two fold. First, the rebar.config had the source dep meck in both the default deps list and in test deps list. Second, when rebar3 ct was run without previously running rebar3 compile it installs all deps to _build/test/lib (as opposed to linking to the default deps in _build/default/lib and overriding with new ones if need be in test).
These two factors caused the issue.
The second one is the only one we need to consider fixing here. An obvious one is running a compile as default if it hasn't already. A trickier one, which I don't want to do since I tried it before, is tracking which profile a dep should originally belong to and making sure it is fetched to that dir and only linked to from any other.

Would we see the same problem running rebar3 as prod,test compile if a compile as prod never ran before? My understanding is that we only lock default deps, therefore the problem really only exists in running default first, right?

That shouldn't matter. Correct, only default deps are locked. Right now we do track if a dep came from the default profile to know to lock it, but don't use that info to know where to install it to (adds a surprising amount of complexity). We actually could just not do that... Meaning if you have no lock file and run rebar3 ct one will not be created.

That actually sounds reasonable-ish, though it also sounds like a quirky jagged corner.
 Do we want to keep this consistent here?

Yea, I suppose it is reasonable in that sense.


buyAmountMarket Question

Hi,
In the index.js for the strategies, is the buyAmountMarket in the targetMarket currency?  For example with targetMarket: "USDT" then is the buyAmountMarket: 200 in USD?
Thanks
Awesome effort btw!

Yes, that's it. As my strategies are only for the BTC market, I didn't test BitProphet with other markets, but I know people that do.
To avoid having losses, I advise to use any strategy first with paper trading active. Let me know if you run into problems.

Awesome effort btw!

Thank you! :blush:


Add regex matching on path.

Added regex property on mocks to enable Regex matching on path property.
To support JSON files and javascript objects, I've used the approach.
With this, all regex patterns are interpreted as strings, and proper escapings of path slashes etc. are necessary. See the examples for details on this.
Further more cleaned some minor jshint cleanup.
Implements #89

This looks great @Crevil . I was just about ready to work on this when I saw your PR.
Thanks!

👍

Glad it fits. It was an easy implementation and I'd love to help on this.


Decorate per-architecture images with base image annotations.

Currently we only decorate the topmost index/image with base image data for the base index/image (for everything except docker manifest lists).
This change makes multi-arch builds decorate the manifest of the per-architecture images with the reference of the base index, and the digest of the specific base image used for that particular architecture's image.
This results in the per-architecture SBOMs starting to encode DESCENDENT_OF links as well.

Codecov Report

Merging #759 (907b8d5) into main (59c4264) will increase coverage by 0.02%.
The diff coverage is 63.63%.

@@            Coverage Diff             @@
##             main     #759      +/-   ##
==========================================
+ Coverage   51.22%   51.24%   +0.02%
==========================================
  Files          44       44
  Lines        3315     3323       +8
==========================================
+ Hits         1698     1703       +5
- Misses       1404     1407       +3
  Partials      213      213

Impacted Files
Coverage Δ

pkg/build/gobuild.go
60.22% <63.63%> (+0.02%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59c4264...907b8d5. Read the comment docs.

Gonna rebase this on main now that my other change landed, so I can get it all in a local install.


No Units behind the values

Describe the bug
It is possible to write Units behind the values

I have added unites to the values in version 0.1.4.


feat: vLLM integration for PagedAttention

Signed-off-by: aarnphm-ec2-dev<EMAIL_ADDRESS>

What does this do? Add Llama 2 support?

no, this adds paged attention with vllm

will include a build PR afterward


Use esprima to parse source files

Use JavaScript parser to parse function signatures, build tests, etc.
Compared to previous solution, these are the limitations and assumptions

each source file is checked for module.exports. If there is more than one in a file, the process fails with a custom error message
some data pieces in Util.js are not included anymore in params. They didn't seem to be used anywhere
comments are looked up only between function signature (module.exports) and first function body element

Please help me make sure that I didn't break anything in the build process. I'm not yet very familiar with all details.

Wondering if this PR could have caused that we're seeing undefined now on all functions, e.g. http://locutus.io/php/array/array_diff_assoc/ ?

I suspect so. I will debug as soon as possible.


Guest mode for apps

Add in base url to the real site (for redirect purposes)
Add in Login to use button everywhere that was the action buttons
fix a glitch with mobile action buttons not seen
details page gets a login to use button

:+1:

:+1:

:+1:


[ONBOARDING] Basic Navigation

Adds basic navigation between onboarding screens

"Login" button in the sign in screen navigates to the profile creation screen (placeholder to demonstrate behavior of successful login)
"Skip" button in the sign in screen navigates directly to the home screen (bypasses remainder of onboarding)
Get started button in profile creation screen navigates to the home screen (completes onboarding)
Bottom nav bar doesn't display during onboarding flow

For the profile creation screen with the checkmarks, I think the current background is pinkish, so we might need to change background color. Also, it might be because of my emulator, but I think part of the "Complete your profile" title gets covered by the camera; not too sure how to fix though.

For the Elevated Card items in UpliftUsesCardList(), I think you can extract them into a composable that takes in text and a painterResource for more reusability, and I think for the shadow effect you can make use of the elevation property of ElevatedCard instead of creating the shadows manually in Modifier.

For the profile creation screen with the checkmarks, I think the current background is pinkish, so we might need to change background color. Also, it might be because of my emulator, but I think part of the "Complete your profile" title gets covered by the camera; not too sure how to fix though.

You should be able to fix this by using something like Modifier.statusBarsPadding(), but I'm not actually sure if this takes into account stuff like the camera... also, it doesn't even look like the status bars are there? It's worth a shot, but if that modifier doesn't work, there may be a similar one that'll give you the proper insets padding for the camera.


Revert "Fire Spree::PromotionHandler::Page on static page show"

Reverts spree-contrib/spree_static_content#213

Hi,
is there any issues with this?

Hi @nilcolor some builds are failing with this change -> https://travis-ci.org/spree-contrib/spree_static_content/builds/184236169


Disable about popup

Stop the about popup from displaying automatically

This is as not been addressed and the pop up still appears when opening the page.
@kamicut @olafveerman  could you please disable when you have a chance?
Thanks and kind regards,
-Yann


Android Gradle Plugin 4.1.2 in buildSrc module

Follow-up on #16 : one more place needs to be updated.

@mariusc83 interesting point, I'm going to take a look


Minor patch

Hello @devblackops. I just submitted a pull request for a minor change (added a parameter alias) to the Get-NSLBServiceGroupMemberBinding function. Unfortunately I failed to read the CONTRIBUTING.md until after fact and did not run the pester tests prior to submitting the pull request. I'll try to do better in the future.
Love your module! I intend to be a heavy user and will try to contribute if I have any suggestions or run into any problems - in the unlikely event that your code isn't already perfect :)

No worries and thanks for the PR! An AppVeyor build will kick off on any PR submissions but it's good to run .\build.ps1 -task test before hand. I only have Pester testing the module manifest, file encoding, and spaces/vs tabs at the moment. A bunch of work needs to be done on creation proper tests for the functions.
Any suggestions for improvements are welcome.


Is there any way to sniff on a unix socket?

Short answer: No.
Long answer:

You could mount a "Man in the Middle"-Attack: Rename the Unix socket, create a new one in it's place and then copy all the data from the new one to the original one. But for this you have to restart the communicating program
Use something like strace on the program you want to observe and filter for everything that gets written or read to the fd belonging to the Unix socket in question.


[Chore] Fix flaky e2e test

Purpose of the pull request

Brief change log
Fix flaky e2e test
Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
Pull Request Notice
Pull Request Notice
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 41.16%. Comparing base (565a917) to head (682503f).

Additional details and impacted files
@@             Coverage Diff              @@
##                dev   #16246      +/-   ##
============================================
- Coverage     41.17%   41.16%   -0.02%
+ Complexity     5093     5090       -3
============================================
  Files          1391     1391
  Lines         43803    43803
  Branches       4645     4645
============================================
- Hits          18038    18033       -5
- Misses        23997    24003       +6
+ Partials       1768     1767       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Improvements to build system

Description
Added two scripts to help with the CMake build. The workflow now looks like:
./configure
./9_sycl_of_hell -install

Both of the scripts support various options:
thilina@uan-0002:~/sycltrain/9_sycl_of_hell> ./configure -help
Usage: ./configure [options]
Options:
  -help Print this help message and exit.
  -cxxflags <compiler flags> Set compiler flags to use for the build.
  -build-type <Release|RelWithDebInfo|Debug> Build type.
  -build-dir <build directory> Build directory.
  -install-prefix <install prefix> Install prefix.
  -enable-asan Enable address sanitizer.

thilina@uan-0002:~/sycltrain/9_sycl_of_hell> ./9_sycl_of_hell -help
Usage: ./9_sycl_of_hell [options]
Options:
  -help Print this help message and exit.
  -install Install sycl examples.
  -format Format the source code with clang-format.
  -format-check Check if source formatting is compliant with clang-format.
  -tidy Run clang-tidy.

Todo

[ ] Test the build system changes in Polaris
[ ] Update 9_sycl_of_hell/README.md

Look good!  A few comment

Do we really need a large clang format, or we can just put google or whatever inside?

Please do not change argparse. It's an external lib, so would like to keep it "raw", so diff are easier when updating.

Also if you don't mind renaming configure, it's the traditional name of the autotool configure script. Mayve configure-cmake?

Thanks!


MSID: 95898 Version: 1 DOI: 10.1101/2<IP_ADDRESS>4840

MSID: 95898
Version: 1
Preprint DOI: https://doi.org/10.1101/2<IP_ADDRESS>4840
Step 1. Awaiting reviews
Editorial to post reviews via hypothesis
Useful links:

DocMap: https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=95898
New model tracking: https://docs.google.com/spreadsheets/d/1_fHaoOy7hjyocptKtVJRijeNpUY4hBS7Ck_aVmx6ZJk/
Reviews on sciety: https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>4840

For trouble shooting (e.g. no Docmaps available):

DocMap issue addressing: https://miro.com/app/board/uXjVNCwK6EI=/
Explore DataHub DocMaps API: https://lookerstudio.google.com/reporting/4c2f0368-babb-4beb-b5b3-497e7e7b0f08/page/ejphD
Unmatched submissions and preprints: https://lookerstudio.google.com/u/0/reporting/9f86204f-3bf7-477c-9b18-5c5ef141bf69/page/p_gxi57ha93c
Unmatched manuscripts spreadsheet: https://docs.google.com/spreadsheets/d/15QcK8w-ssB7109RQEDtFpJPZ0J5HTGxoHa_2TtpMBbg/edit#gid=1336081641

Step 2. Preview reviewed preprint
Production QC content ahead of publication
Instructions:

QC preview: https://prod--epp.elifesciences.org/previews/95898v1
Update ticket with any problems (add blocked label)
When QC OK, add QC OK label to ticket and add publication date and time to https://docs.google.com/spreadsheets/d/1amAlKvdLcaDp5W8Z8g77NmkwbMF5n_u89ArSqPMO8jg
Move card to next column
(At end of the day post link in #enhanced-preprint and ask for PDF to be generated)

Useful links:

Preprint DOI :  https://doi.org/10.1101/2<IP_ADDRESS>4840
Confirm reviews returned by EPP: https://prod--epp.elifesciences.org/api/reviewed-preprints/95898/v1/reviews
To update the MECA path in the docmap: https://docs.google.com/spreadsheets/d/1mctCQuNFBjSn97Lihy7_vBO6z7-N-oqyLv4clyi6zHg

Step 3: Awaiting search reindex
This step adds the reviewed preprint to the homepage: https://elifesciences.org
The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.
Useful links:

Jenkins pipeline to reindex search can be triggered sooner or monitored here: https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 4: Published! PDF requested
Waiting for PDF to be generated
Useful links:

PDF tracking: https://docs.google.com/spreadsheets/d/106_XeDjmuBae7gexOTNzg60lapeqjl2aRn9DzupGyS8/

Step 5: Introduce PDF to data folder and git repo
Upload PDF to relevent folder in git repo https://github.com/elifesciences/enhanced-preprints-data/
Step 6: Done!

[ ] Kettle is on!

Thanks Rebecca. Should be sorted in the next few hours

Thanks @fred-atherden, the references appear the same to me?

Thanks @rkcook312. Not sure why that was, I re-ran the temporal workflow, and see now that they've updated. Could be related to do with the weirdness we were seeing in EPP on Friday (missing Icons).
Should be good to go now I think.

Great, thanks!


chore(main): release 0.7.1

:robot: I have created a release beep boop
0.7.1 (2024-06-24)
Bug Fixes

comfyui: remove hardcoded url (81281ea)
readme: update readme and docstrings with installation instructions (caec523)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/briancaffey/agents-of-inference/releases/tag/v0.7.1 :sunflower:


impl Parse for UseTree

Closes #618 by exposing impl Parse for UseTree.
Diff is best viewed without whitespace.

Fixed. (Thought I could get away without local testing. I was wrong.)

Published in 0.15.31.
Still eager to have #568 when you get back to it.


Ref docs issue

Error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:copy (default-cli) on project standalone-pom: Unable to resolve artifact.: Could not transfer artifact com.microsoft.azure:azure-mgmt-redis:pom:1.41.4 from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/com/microsoft/azure/azure-mgmt-redis/1.41.4/azure-mgmt-redis-1.41.4.pom
[ERROR]   com.microsoft.azure:azure-mgmt-redis:pom:RELEASE
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR]   azure-sdk-for-java (https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-java/maven/v1, releases=true, snapshots=true),
[ERROR]   maven-cognitiveservices-speech (https://csspeechstorage.blob.core.windows.net/maven/, releases=true, snapshots=true),
[ERROR]   central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false): Connect to repo.maven.apache.org:443 [repo.maven.apache.org/<IP_ADDRESS>] failed: Connection timed out: connect
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

For more details check here:

https://apidrop.visualstudio.com/Content CI/_build/results?buildId=329220&view=results

@danieljurek for notification.

This error was transient and was fixed in a re-run.


Blog app - Add forms

Description
In this project, I added forms to my Blog app.
Project requirements

Create a method called current_user in your ApplicationController to make current user data available to all controllers.

It will return the first user from the database.
Don't worry about proper authentication at this time. You will add it in the upcoming projects.

Create forms to perform the following functions:

Creates a Post on behalf of the current_user (use the method that you created in your ApplicationController).
Create a comment on behalf of the current_user (use the method that you created in your ApplicationController).

Allow Users to add likes to Posts.

Hi @Ibizugbe
Could you tell me if you get any error, because in my computer it works fine.

@MarcoDiaz2000 I actually didn't get any error. as you can see on the image I have uploaded.
do you mind if we get on a call together, so we can go through it?

Hi @Ibizugbe
I am checking and apparently I have a problem with the creation of new users.
I will fix it.
Thanks

Alright @MarcoDiaz2000 👋🏼.
when you solve the issue, and you have resubmitted, I will suggest you stay online both on slack and zoom just incase a reviewer wants to reach out to you😉.

Thanks @Ibizugbe  👋🏼

Thanks @elisha2kyakpo1


Outreachy/MDF task

This is the branch I created for my MDF task. In this branch, I created a repository called  Outreachy_Modeci_Projects where I created a separate folder for each of my task. In each folder is a Readme.md file that described the outline of each of this task. I humbly await your feedback on all this tasks. Thanks

Thanks for this @TaofeeqatDev.

For the pytorch to mdf example, please make sure you have run it with the latetst version of mdf, which gives colors to the generated graph of the mdf model. Aslo, use the new solid_color=True option in mdf_model.to_graph_image
Can you run the model multiple times with different input digits to assess the % accuracy of the MDF model?
The keras_mdf.py file looks like it comes from a previous commit, which is not in our main branch. Can you be clear where this came from?
Can you add a new layer to your keras model and retrain it, and demonstrate that the newly generated mdf model also has the new structure? Note you should use these helper methods: https://github.com/ModECI/MDF/blob/main/examples/TensorFlow/Keras/keras2mdf.py

@pgleeson thanks for the feedbacks

I have tried including solid_color for the pytorch to MDF example but I kept on getting error, that was the reason I removed it.  But I will definitely work on it to fix the error
The second point is not very clear to me, do you mean I should supply a different input image and try to convert it to MDF again  to see if the pytorch and MDF will still produce the same result.
As for the third point, I wrote the keras_mdf script myself base on the tutorial I saw in one of the docs, as a matter of fact, I wrote 2 python script (one with 3 layers and another with four) if you want me to send a screenshot of the the second python script for clarification I will gladly do that.
As for the last point, I will do that right away and make another commit. Thanks

@pgleeson As regards the last feedback on my project, I have added a new layer to my keras model as instructed and have also retrained it and generated the graph. Then i also use the helper function to convert it to MDF. However, everything works perfectly except for the graph. i got the error parameter object has no attribute shape, while trying to plot the graph. The parameter object is a list and list objects do not have the attribute shape. Looking forward to your feedback
Thanks

Thanks @TaofeeqatDev.

It seems though some of your notebooks have disappeared in this version, particularly where the pytorch and keras generated models are converted to mdf.
Can you remove the files used to generate the images with graphviz from this commit (e.g. AB, example - text files which begin with digraph)?
Can you tell me where this file came from originally?

Thanks @pgleeson

I removed the 2 notebooks in order to work on the changes you talked about base on your last feedback.
I will remove the files from the folder
I saw the script in one  of the repository I came across while going through some example codes. But since you referred me to another script am already working on that.
Thanks.

@pgleeson  as regards your comment on too many files in PR. My question is, do you want to upload just the two notebook, I mean the notebook for building the model and the one for converting to MDF. Just want to confirm this before I reupload my projects.

@pgleeson as regards the last feedback on my task.

I have added a new layer to my keras model and retrained it.
I also use the helper functions you referred to for my keras to MDF conversion.
I'm currently working on the pytorch to MDF conversion and I will reupload it soon.

@TaofeeqatDev there shouldn't be any reason to remove files and reupload them, e.g. your notebooks. You should be managing files by adding/committing changes using git, rather adding/deleting them on the website.
Please do add the pytorch->mdf version and update that by committing changes to the same file.
Re the keras example, it seems to be just a toy example with dummy data in/out, correct? It would be better if it was a "real world" example, e.g. like the pytorch one
The file referred to in your keras notebook (from Keras_MDF import *) does not exist.

@TaofeeqatDev there shouldn't be any reason to remove files and reupload them, e.g. your notebooks. You should be managing files by adding/committing changes using git, rather adding/deleting them on the website. Please do add the pytorch->mdf version and update that by committing changes to the same file. Re the keras example, it seems to be just a toy example with dummy data in/out, correct? It would be better if it was a "real world" example, e.g. like the pytorch one The file referred to in your keras notebook (from Keras_MDF import *) does not exist.

Thanks for the feedback... I will use the MNIST dataset that I use for my pytorch example

I have opened a new pull request for my project. Thanks


inconsistent terminology for spy_return and result=

Since we cannot use "return" as a parameter key in the agent, we use "result". We should then change the terminology to be consistency: spy_result in the spy_point and .result as the spy point name,

Fixed for pybond, leaving open for rbond

Fixed in rbond as well


Spice86: The C# stubs generator produces code that does not build

The C# stubs generator is to be deleted. The code generator will be used instead.


ClangFormat not used in some files

When I was writing #6, I noticed that you offered a ClangFormat specification. When I tried to use it, I noticed that imgui_plot.cpp does not apply the specified style. ClangFormat reformatted the entire file (and even made it unusable).
It's great to offer a .clang-format in your repository. I like working with auto-formatting a lot. It allows you to focus on what you want to write instead of how to indent it. It would be great if you could reformat all the files in the repository with ClangFormat, so the tools can actually be used.

Yes, I recall trying to format imgui_plot.cpp and it breaking the build as well. The issue is:
SortIncludes: 'true'
which messes up the imgui include order. I just changed it, and applied the formatting to all files. Thanks!


docs: add azure private cluster page

slight bump to @alexsomesan. this is for https://jira.prod.coreos.systems/browse/DOCS-120

@alexsomesan if you're good with this, I will pass to @joshix

from Alex:
So there are two valid options regarding DNS there: 1) use Azure MANGED DNS (not the default one against which we already warn) or 2) Use your own externally managed DNS server about which the installer doesn’t care but which needs to resolve requests coming from the cluster nodes
number 2) is an advanced use case where users already need to be familiar with managing DNS
so we don’t need to elaborate very much
but we need to point out that the specific DNS server needs to be configured as the default resolver for the nodes

@alexsomesan I am going to interpret Beth's comment above, and the eventual content in this document, in the following way:
There are three DNS options for Azure:

Azure "managed" DNS -- your hostnames, with Azure-region domain names
Azure DNS with your own domain name
External DNS, not Azure DNS service

Of those, only 1 and 3 can be used with a private cluster on Azure.
Please let me know if that's accurate.

@joshix Number 1) is not supported: the default hostnames that you can resolve without a managed zone are not enough. We need a way to publish multiple-entry A records, which we can only get via 2) and 3).
So to summarize, the two options that will work are:
2) Azure DNS with your own domain name
and
3) External DNS, not Azure DNS service

@alexsomesan Aha. Thank you!

@zbwright Fixed a typo and ironed out one sentence for (hopefully) clarity.


Pop Overs prevent from selecting Code

Issue:
The pop ups that appear when hovering the mouse over variables/code appears too quickly, and covers the code that I was about to click on, preventing me from selecting to write or edit.
Step to reproduce:

enable pop-ups
position variable to click on towards middle of screen
hover over it with mouse. Pop up should cover variable
attempt clicking - you will not be able to select the variable

If this box could be moved upwards, or take longer to appear, that would be helpful.
ScreenShot example:
I go to hover over goalLookup on line 43

goalLookup is covered by the pop up, and I cannot click on goalLookup

See https://github.com/microsoft/vscode/issues/112736#issuecomment-747923034 for a setting controlling the delay.

See https://github.com/microsoft/vscode/issues/112736#issuecomment-747923034 for a setting controlling the delay.

The hover is positioned a bit too low, appearing on top of the text that is being hovered. Duplicate of https://github.com/microsoft/vscode/issues/112391 . A fix is available on Insiders.

The hover is positioned a bit too low, appearing on top of the text that is being hovered. Duplicate of https://github.com/microsoft/vscode/issues/112391 . A fix is available on Insiders.


[security] Upstream vulnerability in decompress module

Describe the bug
There is a NPM high severity warning for kevva/decompress which means yarn audit fails on the JS cli - due its being used in the imagemin family of algorithms.
See: https://www.npmjs.com/advisories/1217
Issue here: kevva/decompress#71
Potential mitigation strategy: https://github.com/kevva/decompress/issues/71#issuecomment-594242824
The discussion is somewhat long-winded - with a variety of opinions on the security implications of the matter. For our usage of this library I am not concerned since the library is only used for downloading known binaries.
Nevertheless, the "high" severity warning is disconcerting and I wanted to publish this opinion so that you are aware that we are aware of it. Updates will be found here on this issue as well as at the wg-security channel on our Discord.

This was fixed in #545 by @not-matthias


Security question.

-when choosing a security question and keying in the the answer: it does not accept a cap letter for first lettter of answer.
-Just skips over the entry. For example, if I type in “Montana” it will register as “ontana”. Without warning, people might not realize.

Pls use this to change the checks we do on the answer.
We don't need special character or capital letter restriction. Length can also be longer.


"component.astro has no default export when using interpolation with astro/i18next

What version of astro are you using?
1.0.6
Are you using an SSR adapter? If so, which one?
None
What package manager are you using?
npm
What operating system are you using?
Mac
Describe the Bug
When using string interpolation in any html tag in an .astro file while using astro/i18next, for instance
<title>{t(`meta:${page}:title`)}</title>

it throws a *component.astro* has no default export warning.
And, it does it with any interpolation in the t function, so if I do this  it happens as well.
<title>{t(`${page}`)}</title>

If I remove the interpolation the warning goes away.
The code works perfectly fine. It just throws a warning so I thought I'd report it.
I can report it on their repo as well but my hunch was that this was a framework issue, but if I'm wrong, happy to move it over there.
Link to Minimal Reproducible Example
https://stackblitz.com/edit/github-pjrsds?file=src/layouts/Layout.astro
Participation

[ ] I am willing to submit a pull request for this issue.

Hello, thank you for creating this issue! This is caused by a parsing error in our language tooling. We're currently in the process of migrating to a new parser which should fix this issue


Correction: Purple-70

Discovered a discrepancy with purple-70.
HEX code should = #5D4DAB
This color has a contrast ratio of 6.74, which is more in-line with the other grade-70 swatches.

@Apatil15, this should be straightforward and I think this can be bundled with #104. Would you mind opening a PR for these changes?

@sh0ji This issue has been fixed and push to branch "issues"


Populates the cohort model with users

When querying a Cohort, the participants array will now present the _id, name, email and role of each user listed in that Cohort

Works great! Just one more request: Could you also populate the request with the isActivated field? Thanks


Implement invalid input handling for Find command

Fixes #157
The find command does not handle invalid inputs, including empty criteria n/ e/, invalid age criteria a/invalid, and invalid gender criteria g/invalid.
The application throws exceptions, but they are not handled properly, resulting in no error message on the GUI.
Let's make the find command handle invalid inputs, and return error messages to the user.
Example:
find n/

find a/invalid

find g/invalud

LGTM! Just some merge conflicts to resolve~


Set form name for collection entries + email content

Added an optional formName key that users can enable to add a name to each submission collection item. Also adds this to the email content when enabled.
Docs will need to be updated to reflect this.

@Zaya1101 Can you add something to the readme outlining how to implement it.


Could not start the htpp server: None

help
yeah i cant start the app

You need to give me a bit more information. What are you trying to do? What is the error message? Are you on the latest version of everything? Any other relevant information?

Okay I think I've found the error and created an issue with the developers of the library I'm using.
https://github.com/sanic-org/sanic/issues/2568


Configure default folder for opening

When using the File -> Open in VSCode on Mac the default folder to display is the home folder.
It would be convenient if there was a configuration open to change the default folder.
I keep all my code in a sub-directory of home, and it would be convenient if VSCode could be configured to default to that subdirectory.
This problem has been discussed prior in a few places, so it seems others are also interested in this capability:

Stack Overflow: https://stackoverflow.com/questions/46419613/visual-studio-code-default-folder
Reddit: https://www.reddit.com/r/learnprogramming/comments/a1sign/how_to_change_default_working_directory_in_vscode/

Related https://github.com/microsoft/vscode/issues/115730

I think this might be a duplicate of #115730. I'll post details of this on that issue.


Refer url protocol and npm config to use appropriate proxy agent

This PR fixes #4
Problem
I'm using proxy setting like the below
$ cat ~/.npmrc
proxy=http://xx.xx.xxx.xx:8080/
https-proxy=http://xx.xx.xxx.xx:8080/

But vscode-test is not using HttpsProxyAgent to request list of stable release version, when proxy address is not started with https.
I think it needs to use HttpsProxyAgent because it needs to access https protocol url.

stable release version url: https://update.code.visualstudio.com/api/releases/stable

Changes
To solve this problem, i refer given url to select proper ProxyAgent.
If you got any issue, feel free to contact me :smile:

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: amazingguni sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

The indentation is wrong, but I'll merge and fix it myself. Thanks for your contribution!

Thanks 😄
I'll check coding rule after this time!


Update Dependencies

Changes Introduced
Updates dependencies
Applicable linked issues
N/A
Checklist (check all that apply)

[X] This change maintains backwards compatibility
[ ] I have introduced tests for all new features and changes
[ ] I have added documentation covering all new features and changes
[X] This pull-request is ready for review

Codecov Report

Merging #219 (67e5b9b) into main (e40cad9) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #219   +/-   ##
=======================================
  Coverage   88.85%   88.85%
=======================================
  Files          91       91
  Lines        2772     2772
  Branches       70       70
=======================================
  Hits         2463     2463
  Misses        309      309

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e40cad9...67e5b9b. Read the comment docs.


Migrate from legacy to new React lifecycle

React will soon deprecate componentWillMount, componentWillReceiveProps and componentWillUpdate methods due to their potential misuse, especially once async rendering is launched. In fact, enabling strict mode already warns about these unsafe lifecycle methods usage.
This PR moves module loading logic from componentWillMount to componentDidMount, which is the recommended upgrade path in this case.

Did you test this with regards of SSR (cWM is called but cDM not)?

I assumed tests were already covering this. Are there any use cases missing in this regard?

isn't this should be migrate to constructor as new lifecycle suggested?

Any timeframe as to when to expect this to me merged?

@tajo @jamiebuilds any feedback on that?

@bjminhuang I don't think so. If you do that, you'll potentially add a side effect (this.setState) in the constructor that is not recommended. The React way for doing this is componentDidMount

Would really like to see this get merged. Currently trying to integrate this with a project on React 16.4 and the use of componentWillMount is causing the component not to render properly for SSR. I went in to the react-loadable module and swapped componentWillMount for componentDidMount and no more warnings or SSR issues.

Actually, looks like this may have partially been addressed by this merged PR https://github.com/jamiebuilds/react-loadable/pull/123, although componentWillMount still remains along with the added componentDidMount.

Any traction on this? The PR appears ready to merge.

Any news about this PR?

LGTM! Anything I can do to help move this along?

@abenchi @mshwery
This includes your PR + migration to babel .7 and webpack 5
https://www.npmjs.com/package/jaybe-react-loadable
Cheers

@jamiebuilds can we please get this merged?

Would love to see this merged.

go and use lazy load which comes now natively in React

@abenchi yes you can lazy load with latest version of react, though suspense is not supported in SSR mode... So for people like me who server side render their app, there's no option beside react universal or react loadable

Another alternative, beyond React.lazy, is https://github.com/smooth-code/loadable-components

will this ever be merged?!!

Hey @jamiebuilds @tajo
any news to fix the concerning issue of this PR?

For anyone interested, here's my solution to get rid of the warning (React 16 + 17) and still get this working (client+SSR).
https://github.com/jamiebuilds/react-loadable/pull/213#issuecomment-778246548

React will soon deprecate componentWillMount, componentWillReceiveProps and componentWillUpdate methods due to their potential misuse, especially once async rendering is launched. In fact, enabling strict mode already warns about these unsafe lifecycle methods usage.
This PR moves module loading logic from componentWillMount to componentDidMount, which is the recommended upgrade path in this case.

This author doesn't seem to be updating this component, but you can look at similar components that implement SSR PROMISE asynchronous loading，github address ：https://github.com/qq281113270/react-lazy-router-dom


Does element-ui have a type declaration file（for typescript）?

Does element-ui have a type declaration file（dts） for Typescript ? I recently developed a project based on typescript, but using the latest npm install @ types / element-ui , But the server returns 404 .

According to this issue #2651, not yet. 😕

I have solved this problem, and there is no use of type declaration file,  the configuration of the project template on the respository  vue-typescript-template


Update Consultation "Referred For"

I think let's rename this to "Referral for"
without the within the clinic
and then in the drip down, we include

Diagnostics
Acute
Chronic
Advanced Vision [Within clinic]
Glasses fitting [Within clinic]

@davidgohzk resolved by #274 . Should I close the issue haha or do you wanna check


added request and json_request helpers - fixes #50

[X] Add synchronous helpers
[x] Make sure rpapa likes the API
[x] Make sure  pdehaan likes the API
[x] Add an example in the documentation
[x] profit

Coverage increased (+0.2%) to 93.115% when pulling 469495ade88f7657de227c5bbe9d01e182f4027a on synchronous into 2221c76fe8b325c249721338199072612d73edc7 on master.

Coverage increased (+0.2%) to 93.115% when pulling f5a61bb0b8b43fa5395cda4f2e0c6847bd68dcd3 on synchronous into 2221c76fe8b325c249721338199072612d73edc7 on master.

looks like request and json_request provide a clean way to create and manage individual loop(s) to essentially allow for activity to take place synchronously.  looks good!

Thanks @rpappalax ! I will wait for @pdehaan feedback and then add the doc

Coverage increased (+0.2%) to 93.115% when pulling daf11e492310ddd0a8cc5c184e1e1bb1e208dee1 on synchronous into 2221c76fe8b325c249721338199072612d73edc7 on master.


With validator API, debug tokenscripts are also invalidated.

This is what I see. Do you observe or expect something different?
(Also posted to https://community.tokenscript.org/t/if-a-tokenscript-in-debug-mode-has-a-bad-signature-or-is-not-signed-should-the-token-still-be-rendered-with-it/261 to clarify)

Invalid Sig
Valid sig


Update HTTP/TLS/TCPRoute status

What does this PR do?
This PR implements updating status resource for HTTP/TLS/TCPRoute objects, so that they appear attached to the gateway from the API perspective.
Motivation
For background see #9884
More

[x] Added/updated tests

Additional Notes
n/a

Sorry for radio silence. I became a father recently, so haven't found time to get back to this 🐣

Hello @romantomjak,
Thanks for the recent work on this.
I discussed the PR with other maintainers, and we have some feedback.
Regarding the ParentRef sectionName discussion: we agree with your stand, we should specify the sectionName in the status only if the route requires it.
Concerning the route status conditions update, by updating the route status for each listener independently it will still remain as creating a sort of "state".
For instance, if a listener is dropped from a gateway definition, it won't be removed from the status.
There is also an induced complexity due to the update operation per listener, versus a complete rebuild of the status related to the controller with one single update operation that would replace any previous status for that controller.
We have also some concerns about the bump of gateway-api, because gateway.networking.k8s.io/v1beta1 is released a year now (v0.5.0), and we have in mind to provide the support of it.
Because it may have impacts on the current provider design, we think that it would be better to address the status route update on top of the bump to the latest version (v0.7.1) of the gateway API spec and the support of gateway.networking.k8s.io/v1beta1.
This would give the opportunity to take into account any clarification in the spec related to status updates, if there are, or to build status updates on top of any rework of the provider if the bump requires it.
As a side concern, it would have been better for the review process to address the listener status update independently from the route status update, or even to address each route type status update.
To sum up, we think that it would be better to bump gateway-api before addressing the route status update, and hence wait for it before moving on this PR.
Is there any chance that you'll be willing to address that bump?
ps: For both the bump and the future of this PR, the changes should target the v3 branch.

Ah, perfect! Thanks for the feedback - all suggestions seem reasonable and I'll gladly pick up the version bump. I couldn't find an issue for the version bump, shall I open one to chat through any details before starting the work?
What do we do with this PR? I reckon I'll close it for now and open a new one after bumping the API version work is finished?

Hello @romantomjak,

Ah, perfect! Thanks for the feedback - all suggestions seem reasonable and I'll gladly pick up the version bump. I couldn't find an issue for the version bump, shall I open one to chat through any details before starting the work?

Good news!
I have found this one: https://github.com/traefik/traefik/issues/9476, which is a proposal for the bump to APIVerison v1beta1.
I think there is no need to discuss it further, as long as the scope is limited. I would say that bumping the dependency version and moving from APIVersion v1alpha1 to v1beta1, without widening the current feature support is enough for one PR.

What do we do with this PR? I reckon I'll close it for now and open a new one after bumping the API version work is finished?

Yes, closing it is probably better, and it would also give it a "fresh start" on its reopening.
Thanks!


Incorporate latest storehaus release

Corresponding changes to changes in FutureCollector type signature.

@pankajroark  Looks like some tests are failing on travis
There has been no public 0.10.0 release (only RCs) so we may not strictly need to change to 0.11.0-SNAPSHOT, but it doesn't hurt I guess.

Fixed tests.

👍

looks like this was added here:
https://github.com/twitter/summingbird/commit/4a80201c9b30d2dd9d034e898df8f0fb6b4f8ebc
@egonina do you remember why it was added?


i2c Address?

Not an issue - just a question.
It seems like there can only ever be 1 of these on a given i2c bus, or am I reading the datasheet incorrectly?
Datasheeet:
VCNL3040 applies single slave address 0x60 (HEX) of 7-bit addressing following I2C protocol
If true, would the only way to have more than one be something like I/O expander, with enable for each channel (only 1 enabled at a time/polling cycle for each channel)?

Yes, that's right!
There is only one I2c adress for this sensor.
In order to connect multiple device you should use io expenses or some other method.
Another way would use some MCU that has multiple I2c interfaces.
I am still working on this board, but after that I have application where I have to connect multiple sensors. Will post here what method I decide to use.

Thanks.
I am total noob - so my understanding of all this is pretty basic.
Look forward to seeing what you come up with.


I want to know how KinectDK works in bodyTracking

I noticed that the Joint 3d coordinate (especially the depth value) is not same with the pixel depth (get from 2d coord)
So, what algorithms are used?

The "slides" link on this page links to a powerpoint that describes how the body tracking algorithms work.


Installing ucp on a server with autolock enabled aks for key but fails

Related to 17.06.2-ee-13
Hi there,
I am trying to install ucp on an autolock enabled swarm and I am asked for setting the UNLOCK_KEY variable. What does this mean anyway? No documentation on this variable exists. Setting a Linux CLI variable does not help either.
Running the installation command with
--unlock-key
Results in
INFO[0000] Your engine version 17.06.2-ee-13, build ac44d73 (3.10.0-862.3.2.el7.x86_64) is compatible with UCP 3.0.1 (2f7c7e1)
Unlock Key:

However, providing the key fails with the message
`FATA[0054] this swarm-mode cluster has an unlock key, which is required for UCP installation. Please set the UNLOCK_KEY environment variable``
Providing the key directly after --unlock-key does not change anything.
The key I provide is the current one, as checked with
docker swarm unlock
Please look also on issue https://github.com/docker/swarm/issues/2871 as I am not able do deactivate autolock anymore.
Best regards,
Martin

Please try with  "docker swarm update --autolock=false" and try with ucp installation.


Ambiguity in Training Data - Prioritization guideance

Cell 1.5 - Ambiguity in Training Data - is a very useful report.
Can we quantify or prioritize what overlap should be addressed - perhaps an "overlap score" - showing the worst overlaps or the ones most impactful to improving the training.
Ideally the tool suggests what I do next, this would be informed by:
Work on ambiguity between my most confused intents first (leveraging earlier chi-square or a k-folds analysis)
Work on the most significant overlaps, ie which ones give me the most bang for the buck.

@andrewrfreed Hi, Andrew. In our current design, the output is sorted by the similarity score at example-pairs level. We aim to tackle this problem here at a more microscopic level and if the user is seeing a pattern between intents, we direct them to the Watson  Assistant Intent Conflict Resolution.
I agree with your point for the time-to-reward for customers. Therefore, We ask the user to go through the sort list starting with the examples with the highest similarity scores. However, the user will still need to go through the list because higher similarity score does not guarantee ambiguity at the semantic level.

issue closed and we will revisit if necessary.

issue closed and we will revisit if necessary.


add error handler middleware in the plugin router

This may be mildly controversial.
On the upside, few plugins were using the error handler middleware, so their errors will now be more precisely logged and reported, tagged with the plugin ID. And we could now remove ALL MiddlewareFactory usage in the entire plugins directory which surely is a nice simplification.
On the downside, this is now a "forced" behaviour, and the httpRouter does not yet have a configure like the rootHttpRouter does. Those who relied on errors bubbling all the way down to the root http router for custom error handling, will now have to hoist their error handling up to the plugin level. This is therefore marked as a breaking change.
This also does mean that in the response body you'll have the "local" route, i.e. without the /api/{pluginId} prefix. This is a matter of taste but I kinda would not have minded if it were the full path including the prefix.

When we lean on the root error handler, there are two outward visible differences:

The reported URL is the full one including the prefix
The log uses the root logger, so isn't tagged with the plugin id (or other customer specific plugin level adaptations)

I think maybe we should either do what's in this PR, or just stop ever putting these middlewares inside plugin code.


React on Transaction Failures

Fixed by preventing stuck transactions (hopefully, ha) in https://github.com/codex-storage/nim-ethers/pull/55.


Sending authorization model twice duplicates it

Hi guys,
recently I made a mistake (fortunately, in development environement) and sent a request to create an authorization model twice, which created the same model without asking any question. I believe it would be nice to avoid such situations by checking the structure of the model and comparing it to others with hashes before inserting it.
Please let me know if you need further information, cheers!

Thanks for the suggestion @geoffroy! Do you expect this to work across all previous models or only the latest one?

We'd have to make it in the WriteAuthorizationModel command: https://github.com/openfga/openfga/blob/087ce392595f3c319ab3028b5089118ea4063452/pkg/server/commands/write_authzmodel.go#L31
The downside of this change is, let's say you want to be able to quickly stress test this API, you'd have to send a different model every time.

Hi, I'm also interested by this enhancement.

The downside of this change is, let's say you want to be able to quickly stress test this API, you'd have to send modelA, modelB, modelA, modelB, etc..

maybe this behavior could be disabled with an option like idempotent=false?

IMO if we add this it should not be at the API layer, it would be at the SDK/CLI layers - they can accept an optional --reject-if-matching-latest (or something similar, we can iterate on a better name), in which case the SDKs should:
1- get latest model
2- remove unnecessary fields (e.g. ID)
3- normalize both (e.g. default all fields to empty state)
4- convert to json
5- do a hash that sorts the fields before hashing
6- compare the hashes
That said we would still have issues in eventually consistent environments. If you post, you might get an error that it matches latest even if latest has been updated in the meantime

On this aspect, I'd like to understand the best practices to deploy models in production. The proposed change here would break our model, which is as follows. We might be doing something wrong or overcomplicated, so please share any best practices available:
Our goal is for teams to be able to contribute to the policy models, allowing some control on what they can change. And to automate as much as possible deployment of such models to different environments.
To achieve that we:

Have a git repo with parts of the policy. Each team owns a directory on that repo.
Each team writes part of the policy and stores them as .pol files in the repo
Every time there's a commit on that repo, a service consolidate all .pol files in that particular commit point and create a new policy model
The git commit SHA and the policy model ID are returned by this service and available to the developer/user.

Then a developer, or the person in charge of the deployment can choose which model ID should be used on a particular environment, by deciding on which git commit/branch we want. For example, we can have a tag in the git repo that identifies a particular version.
For that to work, we need either each commit ID to have a model ID or at least, we would need if the model creation fails because it's the same as before, to have the previous model ID available as part of the response.
Thanks,

We're also encountering challenges with deployment due to unexpected behavior, similar to the issue discussed here.

IMO if we add this it should not be at the API layer, it would be at the SDK/CLI layers - they can accept an optional --reject-if-matching-latest (or something similar, we can iterate on a better name).
In which case the SDKs should: 1- get latest model 2- remove unnecessary fields (e.g. ID) 3- normalize both (e.g. default all fields to empty state) 4- convert to json 5- do a hash that sorts the fields before hashing 6- compare the hashes
That said we would still have issues in eventually consistent environments. If you post, you might get an error that it matches latest even if latest has been updated in the meantime
For JS, this might be helpful (though it has not been maintained for a while): https://github.com/puleos/object-hash
The need to add an external dependency to do this reliably doesn't really increase my enthusiasm for doing this though, right now our SDKs have between 0 and 2 prod dependencies.

Considering that model creation is an API endpoint, which might be used by tools not dependent on the CLI, it seems more logical to include this feature at the API level.


tools have no installed version (IDFGH-4979)

Hi,
I'm currently trying to build and pack the esp-idf toolchain inside an (alpine 3.13 linux-base) docker image.
git clone --recursive https://github.com/espressif/esp-idf.git
./install.sh
. ./export.sh
Things work up to install, but export.sh fails for several reasons:
. ./export.sh
Setting IDF_PATH to '/root/esp-idf'
Detecting the Python interpreter
Checking "python" ...
bash: python: command not found
Checking "python3" ...
Python 3.8.8
"python3" has been detected
Adding ESP-IDF tools to PATH...
WARNING: directory for tool xtensa-esp32-elf version esp-2020r3-8.4.0 is present, but tool was not found
ERROR: tool xtensa-esp32-elf has no installed versions. Please run '/root/esp-idf/install.sh' to install it.
WARNING: directory for tool xtensa-esp32s2-elf version esp-2020r3-8.4.0 is present, but tool was not found
ERROR: tool xtensa-esp32s2-elf has no installed versions. Please run '/root/esp-idf/install.sh' to install it.
WARNING: directory for tool xtensa-esp32s3-elf version esp-2020r3-8.4.0 is present, but tool was not found
ERROR: tool xtensa-esp32s3-elf has no installed versions. Please run '/root/esp-idf/install.sh' to install it.
WARNING: directory for tool riscv32-esp-elf version <IP_ADDRESS>_64eb9ff-8.4.0 is present, but tool was not found
ERROR: tool riscv32-esp-elf has no installed versions. Please run '/root/esp-idf/install.sh' to install it.
WARNING: directory for tool esp32ulp-elf version 2.28.51-esp-20191205 is present, but tool was not found
ERROR: tool esp32ulp-elf has no installed versions. Please run '/root/esp-idf/install.sh' to install it.
WARNING: directory for tool esp32s2ulp-elf version 2.28.51-esp-20191205 is present, but tool was not found
ERROR: tool esp32s2ulp-elf has no installed versions. Please run '/root/esp-idf/install.sh' to install it.
WARNING: directory for tool openocd-esp32 version v0.10.0-esp32-20200709 is present, but tool was not found
ERROR: tool openocd-esp32 has no installed versions. Please run '/root/esp-idf/install.sh' to install it.
env: can't execute 'idf.py': No such file or directory
although it detects that there is python3 only and not python (deprecated old python2) idf.py still starts with
#!/usr/bin/env python
so it is hardcoded to use python2 which is absent.

https://github.com/espressif/esp-idf/issues/5908

I think the python 2/3 problem near the end is simply the result of install script not completing successfully, it fails before it can set up a Python virtual environment. The main issue here is failure to verify that the tools installed by install.sh can run.
Errors such as ERROR: tool openocd-esp32 has no installed versions. Please run '/root/esp-idf/install.sh' to install it. typically indicate that the program couldn't be executed, typically due to missing shared libraries or due to architecture mismatch.
On Alpine, it is most likely the former, and most likely related to glibc. Please run ldd on one of the executables to check if all the shared libraries are found. E.g. find $HOME/.espressif -name xtensa-esp32-elf-gcc | xargs ldd.

Yes, "python" will be set up after the export script finishes successfully and the hardcoding in the sheebang won't matter.
We are currently working on improving these error messages so the installer will print a more useful error message.
Please run the command suggested by igrr and then we will advise you further.

I'm closing this for lack of activity. In the meantime, we have improved the error message: https://github.com/espressif/esp-idf/commit/ae6f602b6bb53c47cb938f510c1dae57eb0f58a9

find $HOME/.espressif -name xtensa-esp32-elf-gcc | xargs ldd
gives me the output not a dynamic executable

If seeing this on an M1 mac it's because Rosetta is not yet installed. Open an Intel app to get the prompt to install Rosetta 2.

the prebuilt binaries are linked against glibc and so won't work on a system with a different libc

@nekopsykose is there any way to build the tool chain ourselves? With musl linc.
I tried building the esp32ulp-binutil and it errored out while compiling. On a relevant issues someone said to downgrade gcc.
For now I've been using a mix of container and chroot

is there any way to build the tool chain ourselves?

no idea, i assume this literal repository and the .zip in releases with all the code in it is what you want, and the instructions would be somewhere. but the issue is just that the prebuilt binaries won't work on a musl system, and that's all

prebuilt binaries won't work on a musl system, and that's all

:(

i guess https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/linux-setup-scratch.html tells you how? but you'd have to figure it out yourself, and any intermediate issues of needing to patch things to work with musl you'd have to do yourself.
also all of this has nothing to do with this issue :)


Global CLI installation should not be required for unpack

Since unpack requires an empty directory, the oz binary cannot be installed locally and run via npx oz unpack - it must be installed globally. Even worse, the CLI is then immediately installed locally as it's part of the starter kit dev dependencies.
Reevaluate the unpack flow to ensure the CLI can be installed locally. Some options that come to mind are:

Run unpack from outside the CLI, potentially in a package that follows the create- convention to use npm init (see #44)
Allow the local directory to have a package.json, and just add the kit's dependencies to it

Using npm init would be the best IMO.


Translate: Search doc

Sent to translation #450
Done when:

[x] Documentation has been translated
[ ] Conducted a bilingual review
[ ] EN & FR documentation has been passed to devs for publishing

See issue #450 for progress tracking


Improve error message when lazy collection is not initialized

Currently, the error is:
org.hibernate.LazyInitializationException: HR000056: Collection cannot be initialized: com.example.Parent.lazyChildren

We should tell the user that they need to fetch the connection:

Using a fetch join in the query
Or, using the Mutiny.fetch or the Stage.fetch

We should also add this on the WebSite in the Faq section.

Hi, I have the same issue with panache reactive on Quarkus 2.8.3.Final
com.fasterxml.jackson.databind.JsonMappingException: HR000056: Collection cannot be initialized
The non-reactive version of this code works fine.
Thanks

@liconalicona You are seeing this error message because you are trying to access a lazy association. You can do that with Hibernate ORM - the non-reactive version - but you cannot do it with Hibernate Reactive.
This paragraph in the documentaion should explain to you what's happening: https://hibernate.org/reactive/documentation/1.1/reference/html_single/#_fetching_lazy_associations
This issue is not about the adding a new feature to Hibernate Reactive, it's about updating the error message and make it easier for people to find an explanation of the behavior.


Adyen Hosted CSE is not gzipped

Google PageSpeed is not happy with Adyen Hosted CSE, because content is not gzipped

Hi Phil,
Sorry for not responding. I'll put in a request on if we can enable gzip for the Hosted Payment Pages in general or the hosted CSE in specific. I can't promise on a ETA though.

Arnoud

@ArnoudAdyen Thanks!


FUELHH and other outturn reports do not include solar data

Describe the issue
Using the BMRS, we were able to get a proper breakdown of generation outturn using the B1620 report. The report gave us data including wind onshore, wind offshore, and solar. Using either the new generation summary report or the FUELHH report, wind is aggregated (not an issue), but solar is a fuel type.
This is confirmed in the fuel type references endpoint, that solar is not its own category.
Steps to Reproduce
Steps to reproduce the behavior:

Go to https://bmrs.elexon.co.uk/api-documentation/endpoint/datasets/FUELHH/stream
Make request hoping for solar data
Be sad

Expected behaviour
@zghanty raising this now to feed into the battery as a fuel type discussion for Elexon, but it would be great if solar could be a fuel type as well. Feels like its about time to try to push for some better change on this one as well.
Checks
Please answer the following questions for yourself before submitting an issue.

[x] I checked the documentation and found no answer
[x] I checked to make sure that this issue has not already been filed
[x] I'm reporting the issue to the correct repository
[x] I'm reporting a problem and not a feature request
[x] I have added screenshots or other relevant evidence (if applicable)

@Samreay : Thanks for raising this. The inclusion of new Fuel Type categories will need to be made the upstream source data governed by  BSC. We will liaise with NGESO to get the ball rolling on the BSC change proposal.

@Samreay - We've started discussion to get the change proposal for additional Fuel-Type category


Error from stream worker: Stream ended; no additional packets

Home Assistant Version

Home Assistant 2021.12.10

Error from stream worker: Stream ended; no additional packets
21:39:28 – (ERROR) Stream - el mensaje se produjo por primera vez a las 17:22:25 y aparece 56 veces
Device info (please complete the following information, which can be found in log):
Logger: homeassistant.components.stream
Source: components/stream/init.py:305
Integration: Stream (documentation, issues)
First occurred: 17:22:25 (56 occurrences)
Last logged: 21:39:28
Error from stream worker: Stream ended; no additional packets
Error from stream worker: Error opening stream rtsps://:@aws-wework8.tuyaeu.com:443/v1/bfc5d7b007044c021e1k0p/c857uqecmgr80djs5gb04zjWm0fETBpI?signInfo=8W9MuRXm6hwMGGjHuYnnqPIttq%2FLMhdZxbYo0ApsnRTp8tMQiGLToVuZBERX7RW0YnOrASPXp5j9ug49ddwMFAZ43DpoqFwaikO7PF513xcNChz7pAGHguT%2FhkfvfTqw%2BvyxN43IElRyUDDXo3lhKbKIaz7ua9Sv5jB5%2ByF2%2FYQ%3D

Hi,
Probably the same issue. Pity, good camera but not currently usable at home. Works for few minutes after integration has started, and after a while I get this error:
2022-03-21 22:03:37 ERROR (stream_worker) [homeassistant.components.stream.stream.camera.security_camera] Error from stream worker: Error opening stream (HTTP_NOT_FOUND, Server returned 404 Not Found) rtsps://:@aws-wework8.tuyaeu.com:443/v1/andmanyadditionalcharacters
then restarting the integration fixes the issue for few minutes again.

Hi, I have the same Problem.
Would be nice if anyone can fix it :-)

Still not working - 2022.5.0
Logger: homeassistant.components.stream.stream.camera.security_camera
Source: components/stream/init.py:347
Integration: Stream (documentation, issues)
First occurred: 10:00:28 (9 occurrences)
Last logged: 10:06:31
Error from stream worker: Stream ended; no additional packets
Error from stream worker: Error opening stream (HTTP_NOT_FOUND, Server returned 404 Not Found) rtsps://:@aws-wework7.tuyaeu.com:443/v1/bf4de924bc8c0b2e6fidik/c9pljj3eu3kp38se8m00pH6j5MevjuHn?signInfo=CGGjyczYipSjrRiFbSXGW26HKkT7AysYwh%2FKrplJMYIFG1A%2FFAijrgvz9ijMGje3HWwm9v6AOiM1u3Sb0rVosQOVDF7a2X7j9athAL3J35s2wI19R%2BPWN%2B98coNKHnVDbgxbJtDEMJjwpNqgeMv%2FWZewAYP59R7c6Gev3%2BjK0aw%3D

Still not working - 2022.6.2

Same problem here, I can confirm if you open the camera in Tuya smart life app then it will work for some minutes in HA

Still not working - 2022.7.5

Same problem here too

Same. 2022.8.5

so, any news ?

I also have the same issue. Would be great if someone where to look into it.

Still not? Home Assistant 2023.2.4

Still not working in 2023.5.3 .

Still not working in 2023.11.3

Still not working in 2024.2.1 with new method tuya authentication.

Still not working in 2024.2.1