flush handleBeginFrame/handleDrawFrame microtasks in the same event

Description
Flush handleBeginFrame/handleDrawFrame microtasks in the same event to prevent an animation frame event from being injected in between.
Related Issues
Fixes https://github.com/flutter/flutter/issues/69313
Tests
I added the following tests:
TBD

I would very strongly prefer that we not introduce more code that uses Zones. We have regretted every time we have used them. They are super subtle and extremely bug prone with very hard-to-understand bugs.

Closing in favor of https://github.com/flutter/flutter/pull/72115


Make minimap links only point to object hits

Is your feature request related to a problem? Please describe.
With minimap feature, usability testing showed that users did not expect/were confused when map links pointed to a collection hit.
Describe the solution you'd like
Add logic to make sure that minimap links always point to an object hit, rather than a collection hit. Consider how to deal with cases where there are no object hits.

fixed in f54e977285c8833f32e19fe839e4ec15e978b3fb


Mrksr fix mlp acos numerics

In house branch of #411.

Codecov Report

Merging #412 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #412      +/-   ##
==========================================
+ Coverage   99.74%   99.74%   +<.01%
==========================================
  Files          28       28
  Lines        2334     2336       +2
==========================================
+ Hits         2328     2330       +2
  Misses          6        6

Impacted Files
Coverage Δ

GPflow/kernels.py
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5190ada...a152d15. Read the comment docs.

Done! Thanks @mrksr!


denoised audio doesn't sound right

Not sure why but the denoised audio sounds like cut off sounds or little pulses of sound that have nothing to do with the original audio. The original audio is a speaker with some background noise, but the output sounds as if someone is plugging in and out their electric guitar really quickly (like the speaker pop sound). I am new to this so I do not understand the underlying tech, but why does the generateNoiseProfile function overwrite the input audio file. Should i be passing the input audio file to the generateNoiseProfile function? without calling generateNoiseProfile, my input and output audio sounds the exact same. I am also getting the warnings below, but I believe I read somewhere that the divide by 0 error is okay. Any explanation of how to use this properly and what might be causing my problems would be appreciated.
Thank you for the help!
UserWarning: Level value of 2 is too high: all coefficients will experience boundary effects.
  warnings.warn(s]
                      /home/ubuntu/.local/lib/python3.9/site-packages/pywt/_thresholding.py:22: RuntimeWarning: invalid value encountered in divide
  thresholded = (1 - value/magnitude)

you don't need to do generateNoiseProfile

Thank you for your response. When I don't do generateNoiseProfile, the output audio is the same as the input audio with no noise reduction.
This is the script i'm using:
import os
from tqdm import tqdm

# input_dir = '/home/ubuntu/denoising/DNS_data/datasets_fullband/noisy_audios'
input_dir = '/home/ubuntu/denoising/audios/noisy_audios'

output_dir = '/home/ubuntu/kevin/denoising/Audio-Denoising/'

os.makedirs(output_dir, exist_ok=True)

wav_files = [f for f in os.listdir(input_dir) if f.endswith('.wav')]

for file in tqdm(wav_files):
    in_audio_file = os.path.join(input_dir, file)
    out_audio_file = os.path.join(output_dir, f"results2/denoised_{file}")
    print(in_audio_file)
    print(out_audio_file)
    audioDenoiser = AudioDeNoise(inputFile=in_audio_file)
    audioDenoiser.deNoise(outputFile=f"{out_audio_file}")
    # audioDenoiser.generateNoiseProfile(noiseFile=in_audio_file)```


update: readme cleanup and fxes

PTAL @thaJeztah @Jamstah

Just squashed, waiting for the CI and then I'll merge.


Caddy with QUIC = no login possible (POST)

1. What version of Caddy are you running (caddy -version)?
UPDATE: Caddy 0.9.5 (+dc6c986 Mi Feb 01 17:27:47 UTC 2017)
2. What are you trying to do?
Trying to log into my ownCloud, PhpMyAdmin and rainLoop Webmail Account
3. What is your entire Caddyfile?
https://henrock.net {

	tls /etc/caddy/ssl/1_ecc1_henrock.net_bundle.crt /etc/caddy/ssl/2_ecc1.henrock.net.key {

		protocols	tls1.0 tls1.2
		curves		p521 p384 p256

	}

	gzip {

		ext *
		level 4

	}

	fastcgi / unix:/run/php/php7.0-fpm.sock php {

		root		/etc/caddy/html/henrock.net

	}

	header / {

		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "SAMEORIGIN"
		Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;"
		Server "henrock-caddy"

	}

	errors {

		log /etc/caddy/log/error.log
		404 /etc/caddy/errdoc/404.html # Not Found
		500 /etc/caddy/errdoc/500.html # Internal Server Error

	}

		root /etc/caddy/html/henrock.net
		ext .html .htm .php
		log /etc/caddy/log/access.log

}

https://www.henrock.net {

	tls /etc/caddy/ssl/1_ecc1_henrock.net_bundle.crt /etc/caddy/ssl/2_ecc1.henrock.net.key {

		protocols	tls1.0 tls1.2
		curves		p521 p384 p256

	}

	gzip {

		ext *
		level 4

	}

	fastcgi / unix:/run/php/php7.0-fpm.sock php {

		root		/etc/caddy/html/henrock.net

	}

	header / {

		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "SAMEORIGIN"
		Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;"
		Server "henrock-caddy"

	}

	errors {

		log /etc/caddy/log/error.log
		404 /etc/caddy/errdoc/404.html # Not Found
		500 /etc/caddy/errdoc/500.html # Internal Server Error

	}

		root /etc/caddy/html/henrock.net
		ext .html .htm .php
		log /etc/caddy/log/access.log

}

https://blog.henrock.net {

	tls /etc/caddy/ssl/1_ecc1_henrock.net_bundle.crt /etc/caddy/ssl/2_ecc1.henrock.net.key {

		protocols	tls1.0 tls1.2
		curves		p521 p384 p256

	}

	gzip {

		ext *
		level 4

	}

	fastcgi / unix:/run/php/php7.0-fpm.sock php {

		root		/etc/caddy/html/blog.henrock.net

	}

	header / {

		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "SAMEORIGIN"
		Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;"
		Server "henrock-caddy"

	}

	errors {

		log /etc/caddy/log/error.log
		404 /etc/caddy/errdoc/404.html # Not Found
		500 /etc/caddy/errdoc/500.html # Internal Server Error

	}

		root /etc/caddy/html/blog.henrock.net
		ext .html .htm .php
		log /etc/caddy/log/access.log

}

https://sql.henrock.net {

	tls /etc/caddy/ssl/1_ecc1_henrock.net_bundle.crt /etc/caddy/ssl/2_ecc1.henrock.net.key {

		protocols	tls1.0 tls1.2
		curves		p521 p384 p256

	}

	gzip {

		ext *
		level 4

	}

	fastcgi / unix:/run/php/php7.0-fpm.sock php {

		root		/etc/caddy/html/sql.henrock.net

	}

	header / {

		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "SAMEORIGIN"
		Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;"
		Server "henrock-caddy"

	}

	errors {

		log /etc/caddy/log/error.log
		404 /etc/caddy/errdoc/404.html # Not Found
		500 /etc/caddy/errdoc/500.html # Internal Server Error

	}

		root /etc/caddy/html/sql.henrock.net
		ext .html .htm .php
		log /etc/caddy/log/access.log

}

https://cloud.henrock.net {

	tls /etc/caddy/ssl/1_ecc1_henrock.net_bundle.crt /etc/caddy/ssl/2_ecc1.henrock.net.key {

		protocols	tls1.0 tls1.2
		curves		p521 p384 p256

	}

	gzip {

		ext *
		level 4

	}

	fastcgi / unix:/run/php/php7.0-fpm.sock php {

		root		/etc/caddy/html/cloud.henrock.net

	}

	rewrite {
        r ^/index.php/.*$
        to /index.php?{query}
    }

    redir /.well-known/carddav /remote.php/carddav 301
    redir /.well-known/caldav /remote.php/caldav 301

    rewrite {

        r ^/remote.php/(webdav|caldav|carddav|dav)(\/?)$
        to /remote.php/{1}

    }

    rewrite {

        r ^/remote.php/(webdav|caldav|carddav|dav)/(.+?)(\/?)$
        to /remote.php/{1}/{2}

    }

    rewrite {

        r  ^/(?:\.htaccess|data|config|db_structure\.xml|README)
        to status 403

    }

	header / {

		Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;"
		Server "henrock-caddy"

	}

	errors {

		log /etc/caddy/log/error.log
		404 /etc/caddy/errdoc/404.html # Not Found
		500 /etc/caddy/errdoc/500.html # Internal Server Error

	}

		root /etc/caddy/html/cloud.henrock.net
		ext .html .htm .php
		log /etc/caddy/log/access.log

}

https://postfix.henrock.net {

	tls /etc/caddy/ssl/1_ecc1_henrock.net_bundle.crt /etc/caddy/ssl/2_ecc1.henrock.net.key {

		protocols	tls1.0 tls1.2
		curves		p521 p384 p256

	}

	gzip {

		ext *
		level 4

	}

	fastcgi / unix:/run/php/php7.0-fpm.sock php {

		root		/etc/caddy/html/postfix.henrock.net

	}

	header / {

		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "SAMEORIGIN"
		Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;"
		Server "henrock-caddy"

	}

	errors {

		log /etc/caddy/log/error.log
		404 /etc/caddy/errdoc/404.html # Not Found
		500 /etc/caddy/errdoc/500.html # Internal Server Error

	}

		root /etc/caddy/html/postfix.henrock.net
		ext .html .htm .php
		log /etc/caddy/log/access.log

}

https://mail.henrock.net {

	tls /etc/caddy/ssl/1_ecc1_henrock.net_bundle.crt /etc/caddy/ssl/2_ecc1.henrock.net.key {

		protocols	tls1.0 tls1.2
		curves		p521 p384 p256

	}

	gzip {

		ext *
		level 4

	}

	fastcgi / unix:/run/php/php7.0-fpm.sock php {

		root		/etc/caddy/html/mail.henrock.net

	}

	header / {

		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "SAMEORIGIN"
		Strict-Transport-Security "max-age=15768000; includeSubdomains; preload;"
		Server "henrock-caddy"

	}

	errors {

		log /etc/caddy/log/error.log
		404 /etc/caddy/errdoc/404.html # Not Found
		500 /etc/caddy/errdoc/500.html # Internal Server Error

	}

		root /etc/caddy/html/mail.henrock.net
		ext .html .htm .php
		log /etc/caddy/log/access.log

}

4. How did you run Caddy (give the full command and describe the execution environment)?
screen ./caddy -conf /etc/caddy/caddy.cf -cpu 2 -log /etc/caddy/log/access.log -quic
5. What did you expect to see?
I can log into my accounts on the above sites.
6. What did you see instead (give full error messages and/or log)?

At owncloud: Zugriff verboten, CSRF check failed (in english: Access prohibited, CSRF check failed),

At phpmyadmin: Site simply reloads when trying to login without any error.

At Rainloop webinterface: Unbekannter Fehler (in english: unknown error)

When I turn off QUIC by removing the -quic Parameter, every login works!!
7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?
Creating a login page with php and enabling QUIC when starting the caddy web server.

I can confirm I had the same issue trying to log in to an Elixir application earlier. The error was:
panic: interface conversion: io.ReadCloser is struct { io.Reader; io.Closer }, not *h2quic.requestBody
goroutine 36 [running]:
panic(0x9ca1c0, 0xc4200fae80)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/lucas-clemente/quic-go/h2quic.(*Server).handleRequest.func1(0xc4201da160, 0xc420564190, 0xc4200b9590, 0x0, 0xd1e3a0, 0xc420100820)
	/home/caddy/go/src/github.com/lucas-clemente/quic-go/h2quic/server.go:193 +0x17c
created by github.com/lucas-clemente/quic-go/h2quic.(*Server).handleRequest
	/home/caddy/go/src/github.com/lucas-clemente/quic-go/h2quic/server.go:202 +0x628

This issue should be fix by this. Please build caddy with the latest quic-go to confirm. Thanks.

For @Gazler,  the patch from @marten-seemann should work, as his server crashes with POST requests over QUIC, but for me, as you can see above, i recompiled it yesterday, where the fix already existed! So my server does not crash when handling POST Requests.
But I still experience these symptoms I explained above.

I can confirm that the latest dev build doesn't have the issue I reported anymore. Sorry for piggybacking on your issue @Henrocker - I thought it was the same thing.

@Henrocker: I can't reproduce this. Please provide us a minimal working example.

Hi @marten-seemann,
You could reproduce this by installing an owncloud behind a caddy webserver, with php7. 0-fpm and use QUIC parameter for Caddy, then you wont be able to login at the owncloud login.

@Henrocker, I used php-fpm and it works. I have no idea how owncloud works.

@marten-seemann, Can you please somehow let me see how you reproduced the problem?

I set up a fastcgi with this line in my Caddyfile
fastcgi / <IP_ADDRESS>:9000 php
And then I use an HTML form
<form action="test.php" method="post">
  <input name="val" type="text" value="Lorem ipsum">
  <input type="submit">
</form>

that gets posted to test.php, where I do a
var_dump($_POST);

Can it be that no login is possible at my server when QUIC enabled, because I use a unix socket for php unix:/run/php/php7.0-fpm.sock?

Your latest assumption would be a configuration error on your site though, not an issue with Caddy.
I've got a very similar setup running and i can't reproduce any of the symptoms you're experiencing either.

Guess, I will leave this issue closed for now, as I am working on a better solution and it seems like I messed up with some configuration... Sorry for the noise.


Avoid pushing gh-pages when creating a PR

As per title, the gh-pages push will be done only once merged

It skips the push:


Adapter eats up UDP ports

** Adapter eats UPD ports **
Having a problem on a Raspberry Pi4 that a simple ping command gave the error message "ping: connect: Resource temporarily unavailable"
root@shsrv01:~# ping <IP_ADDRESS>
ping: connect: Resource temporarily unavailable
Tracing the error I used the command
# lsof -i -n | grep snmp
which gave me an endless list of entries like
io.snmp.0  17911 iobroker *092u  IPv4 1366517      0t0  UDP *:37626
So I used to the command
# lsof -i -n | grep snmp | wc -l
28237
to count the lines. The result was 28237 open UDP sockets by the io.smp process.
To reproduce I restarted because a simple iob stop snmp.0 did not make it. After a few days I got the same result. After a few days again I tried iob stop and iob start and that worked. But that can not be a solution.
Versions:

Adapter version: 2.4.7
JS-Controller version: 5.0.5
Node version: v18.16.0
Operating system:
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

Here the configuration of the SNMP Adapter

At the first look I cannot reproduce this. I've 2 instances running at my production system. They use exactly one udb port per configured device. The are running since weeks...
Do you have any relevangt logs (exceptions, errors) which might be related?
Do you use writeable states?

Code Review noticed for internal purpose only:
Session could get lost here:
https://github.com/iobroker-community-adapters/ioBroker.snmp/blob/a8abe79000731c12ad7285d85cb4e5e4839482be/main.js#L1401

I have two different iobroker installations and both show up the same behavior. I just checked the use of the USP port after the restart in the afternoon when I wrote the error report.
Since now there is only one UDP port per configured device as you mentioned.
I have enabled the "debug logging" on both installations now and will come back when I have any news.

I've asked at forum test thread. Until now I did not get any feedback.
Currently only one code part could cause lost udb port. In this case you should have a warning logged:
````device ${pCTX.name} (${pCTX.ipAddr}) reported error ${pErr.toString}```
Could you please check you logs (including older logs if possible) if you have such warnings logegd from snmp adapter. Each such warning could possibly result in one lost udp port.

Unfortunately I could not reproduce that problem again. I had it before a few times with having traced it down to the SNMP adapter.
As mentioned before I have 2 Raspberry Pi installations and both had that problem in the past. I have configured the instance as to set it to "debug log" so that I will get the error message in case it will occur.
Looks very strange to me and I feel a bit embarrassed that it could not be reproduced. One the other hand it is good that this problem does not  occur anymore.
Lets stay calm for a few days and if it does not occur anymore I will close the issue with the option to re-open it in case it will happen again.
I have also mis-configured one of the installations with an ip that does not exist and an ip without a running snmp daemon just in case it a bad line is the reason for that problem. Backgound: I do have a wireless connection between two buildings and this might have bad connections sometimes.
Regards

Yes i would keep the ossue open any2ay.
Could you please check if older logs exist and scsn for ghe warning i mentioned befote. Maybe an instable line could cause udp errors.
On he other side i do not gave a real idea how to force errors at udp port as its connectionless.

After I wrote my comment this morning without any new issues I have just ran into the problem now again on both machines.
But the missing UDP ports prevent the generating if a log file print  and even to open the iobroker admin page

pi@shsrv02:/opt/iobroker $ iob logs snmp
Uncaught Rejection: Error: Cannot create a string longer than 0xffffff0 characters
at Object.slice (node:buffer:652:37)
at Buffer.toString (node:buffer:871:14)
at CLILogs.execute (/opt/iobroker/node_modules/@iobroker/js-controller-cli/build/lib/cli/cliLogs.js:39:67)
at processCommand (/opt/iobroker/node_modules/@iobroker/js-controller-cli/build/lib/setup.js:1056:25)
at Object.execute (/opt/iobroker/node_modules/@iobroker/js-controller-cli/build/lib/setup.js:3138:5)
at Object. (/opt/iobroker/node_modules/iobroker.js-controller/iobroker.js:1:40)
at Module._compile (node:internal/modules/cjs/loader:1256:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
at Module.load (node:internal/modules/cjs/loader:1119:32)
at Module._load (node:internal/modules/cjs/loader:960:12)
pi@shsrv02:/opt/iobroker $

Now I created a log file with all SNMP adapter message which I will attach here.  Be aware that the compressed file is only 13MB but the uncompressed file is about 500MB. Maybe that help to track down the problem.
This log file is NOT from the system with the wireless connection. So that seem not to be the problem.
iobroker.current.log.snmp.12.txt.gz

Thanks i ll check the log.
But has to wait for pc, my mobile cannot handle this

Just take your time.

2023-06-29 15:40:10.252  - warn: snmp.0 (17911) device ShAp01 (<IP_ADDRESS>) reported error function toString() { [native code] }
frank@lnx03:~/tmp/iobroker/shsrv01$ cat iobroker.*.log | grep -i snmp | grep -i "reported error"|wc -l
72077
frank@lnx03:~/tmp/iobroker/shsrv01$
Just a short snippet from the log. No other messages between.

2023-06-29 12:33:16.217  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.218  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.219  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.219  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.220  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.221  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.222  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.222  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.223  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.223  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 12:33:16.224  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp00 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 15:31:41.626  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp01 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 15:40:10.252  - ESC[33mwarnESC[39m: snmp.0 (17911) device ShAp01 (<IP_ADDRESS>) reported error function toString() { [native code] }
2023-06-29 19:21:02.341  - ESC[32minfoESC[39m: snmp.0 (17911) [ShAp01] device disconnected - request timeout

Similar messages on both installations.

Thanks for investigation.
The meassage is from exact the suspect code line. Each of this warning indicates one lost udp port.
And there seems to be a second error there - normally there should be an error text output...
So at the meoment I have no glue why the udp port raises an error at your side.
I'll correct the code and give feedback as soon as it is testable.

The SNMP instance asks my network components (access points) regularly. These AP do have the mini_snmpd running. Would you need the config files of them to eventually run a virtual machine?
Some other note: after a while the adapter seems to release all the ports and the same repeats.

should be fixed with 2.4.8
@TheKilroy
There were two indipendent error in  code:
a) normalle the errortext should be output with the warning filtered by you. Du to atyp this was not the case. Starting with 2.4.8 the error text delivered by node-net-snmp should be visible now. In addition the complete stringified arror objet is output when enebling debug logging. This should enable you (or me?) to solve the original problem causing the errors
b) whenever an error was signalled the udp port was not close but a new one was opended. Thus for every error one udp port got lost. This should be fixed now to - although I hope the errors occure at your system still to verify this.
So please test 2.4.8 id possible at one of your systems. The release is available at github and will be available at latest repository tomorrow. Please be aware that this is a beta/latest release so do not use it at productive systems or take at least special care and ensure that you have a working (and restoreable) backup.
Thanks for any feedback.
McM1957

Thank you for your effort. I have installed the new version directly from github and will observe it the next time.
I have installed it on one of the two machines, so there should be a difference to see somehow.
I will report what I will observe.
Thanks
Kilroy

Some new info and and a snipped from the log file. Checked the log file today and the number of used UDP ports. Here a snippet from the system with the 2.4.8 version. Result: no missing ports, but an error message.
On the system with the 2.4.7 version no error yet.

2023-07-05 17:47:25.092  - debug: snmp.0 (14920) session for device "ShBridge" (<IP_ADDRESS>) created
2023-07-05 17:47:36.863  - debug: snmp.0 (14920) handleConnectionInfo
2023-07-05 17:47:44.641  - debug: snmp.0 (14920) readOIDs - device "wrt_eg" (<IP_ADDRESS>)
2023-07-05 17:47:44.641  - debug: snmp.0 (14920) [wrt_eg] processing oid chunk index 0
2023-07-05 17:47:44.642  - debug: snmp.0 (14920) readChunkOIDs - device "wrt_eg" (<IP_ADDRESS>), chunk idx 0
2023-07-05 17:47:44.651  - debug: snmp.0 (14920) [wrt_eg] session.get completed for chunk index 0
2023-07-05 17:47:44.709  - debug: snmp.0 (14920) processVarbind - [wrt_eg] wrt_eg.sysName
2023-07-05 17:47:44.710  - debug: snmp.0 (14920) varbindDeode - decode varbind
2023-07-05 17:47:44.710  - debug: snmp.0 (14920) oidObjType2Text - stringify oid object type
2023-07-05 17:47:44.711  - debug: snmp.0 (14920) [wrt_eg] wrt_eg.sysName (OctetString){"oid":"<IP_ADDRESS>.<IP_ADDRESS>.0","type":4,"value":{"type":"Buffer","data":[119,114,116,95,101,103,46,107,105,108,114,111,121,46,108,111,99,97,108]}}
2023-07-05 17:47:44.711  - debug: snmp.0 (14920) [wrt_eg] update wrt_eg.sysName: wrt_eg.kilroy.local
2023-07-05 17:47:44.711  - debug: snmp.0 (14920) initobject wrt_eg.sysName
2023-07-05 17:47:44.712  - debug: snmp.0 (14920) processVarbind - [wrt_eg] wrt_eg.sysUptime
2023-07-05 17:47:44.712  - debug: snmp.0 (14920) varbindDeode - decode varbind
2023-07-05 17:47:44.712  - debug: snmp.0 (14920) oidObjType2Text - stringify oid object type
2023-07-05 17:47:44.713  - debug: snmp.0 (14920) [wrt_eg] wrt_eg.sysUptime (TimeTicks){"oid":"<IP_ADDRESS>.<IP_ADDRESS>.0","type":67,"value":32295186}
2023-07-05 17:47:44.713  - debug: snmp.0 (14920) [wrt_eg] update wrt_eg.sysUptime: 32295186
2023-07-05 17:47:44.713  - debug: snmp.0 (14920) initobject wrt_eg.sysUptime
2023-07-05 17:47:44.713  - debug: snmp.0 (14920) processVarbind - [wrt_eg] wrt_eg.hrSystemUptime
2023-07-05 17:47:44.714  - debug: snmp.0 (14920) varbindDeode - decode varbind
2023-07-05 17:47:44.714  - debug: snmp.0 (14920) oidObjType2Text - stringify oid object type
2023-07-05 17:47:44.714  - debug: snmp.0 (14920) [wrt_eg] wrt_eg.hrSystemUptime (TimeTicks){"oid":"<IP_ADDRESS>.<IP_ADDRESS>.1.0","type":67,"value":34101217}
2023-07-05 17:47:44.715  - debug: snmp.0 (14920) [wrt_eg] update wrt_eg.hrSystemUptime: 34101217
2023-07-05 17:47:44.715  - debug: snmp.0 (14920) initobject wrt_eg.hrSystemUptime
2023-07-05 17:47:44.715  - debug: snmp.0 (14920) setStates - wrt_eg.sysName
2023-07-05 17:47:44.716  - debug: snmp.0 (14920) setStates - wrt_eg.sysUptime
2023-07-05 17:47:44.716  - debug: snmp.0 (14920) setStates - wrt_eg.hrSystemUptime
2023-07-05 17:47:44.716  - debug: snmp.0 (14920) [wrt_eg] processing oid chunk index 0 completed
2023-07-05 17:47:44.912  - debug: snmp.0 (14920) createReaderSession - device ShBridge (<IP_ADDRESS>)
2023-07-05 17:47:44.924  - debug: snmp.0 (14920) snmpCreateSession - device ShBridge (<IP_ADDRESS>)
2023-07-05 17:47:44.925  - debug: snmp.0 (14920) session for device "ShBridge" (<IP_ADDRESS>) created
2023-07-05 17:47:44.925  - debug: snmp.0 (14920) readOIDs - device "ShBridge" (<IP_ADDRESS>)
2023-07-05 17:47:44.925  - debug: snmp.0 (14920) [ShBridge] processing oid chunk index 0
2023-07-05 17:47:44.925  - debug: snmp.0 (14920) readChunkOIDs - device "ShBridge" (<IP_ADDRESS>), chunk idx 0
2023-07-05 17:47:44.930  - debug: snmp.0 (14920) onReaderSessionError - device ShBridge (<IP_ADDRESS>) - RangeError: Read integer -5108 is outside the unsigned 32-bit range
2023-07-05 17:47:44.931  - warn: snmp.0 (14920) device ShBridge (<IP_ADDRESS>) reported error RangeError: Read integer -5108 is outside the unsigned 32-bit range
2023-07-05 17:47:44.931  - debug: snmp.0 (14920) error dump
2023-07-05 17:47:44.932  - debug: snmp.0 (14920) snmpCloseSession - device ShBridge (<IP_ADDRESS>
2023-07-05 17:47:44.932  - debug: snmp.0 (14920) onReaderSessionClose - device ShBridge (<IP_ADDRESS>)
2023-07-05 17:47:44.933  - debug: snmp.0 (14920) [ShBridge] session.get completed for chunk index 0
2023-07-05 17:47:44.933  - debug: snmp.0 (14920) [ShBridge] session.get: Error: Socket forcibly closed
2023-07-05 17:47:44.934  - debug: snmp.0 (14920) setStates - ShBridge.sysName
2023-07-05 17:47:45.003  - debug: snmp.0 (14920) setStates - ShBridge.sysUptime
2023-07-05 17:47:45.052  - debug: snmp.0 (14920) setStates - ShBridge.hrSystemUptime
2023-07-05 17:47:45.112  - debug: snmp.0 (14920) [ShBridge] processing oid chunk index 0 completed

It seems to be solved but I will observe it for a few days.
The error probably comes from the use of the mini_smpd on my OpenWrt router. As far as I have seen are all values limited to 32bit numbers, but I will double check it.

Ok,
The primary erro is now logged. Theres a problem that node-net-snmp seem to receive data which do not fit into the data type specified. The dump of the error bock is missing - I'll have to check this again. The port / session is now closed as expected after an error.
What I'm missing is a create session afterwards, This should occure after the retry timeout (I think 5 sec per default). Without this createSession no future data would be recorded for this device. Please verify that the device is not dead after an error forever.
By checking use raw states and use type states you should be able to see which data type and which (binary) data is transmitted  - but only if the data can be read and passed to the adapter. Mayby this can help diagnosing the oid with problems.
The -5108 could either be an overflow or the type of the oid should be signed int32.
There could be an error at node-net-snmp too, but befor opening a issue there more data would be required.
Thanks for testing.

I've created a new release 2.4.9
-) the extended json stringified dump of the error will now be displyed (at least I hope so :-) )
-) I've added a ne option to disable closing the session in case of an error.

I'm not sure if closing the session is really required or node-net-snmp (the layer provising the snmp encoding/decoding) could continue without a reset of the session. With this option both possibilities are available now. So please try disabling closing the session. The main difference will be that an active read block will not be aborted and that no retry wait time is set up.

Ypu are right, that a new session is created. I just did not copy the lines from the log file.

2023-07-05 17:47:45.112  - ESC[34mdebugESC[39m: snmp.0 (14920) [ShBridge] processing oid chunk index 0 completed
2023-07-05 17:47:45.113  - ESC[34mdebugESC[39m: snmp.0 (14920) session for device "ShBridge" (<IP_ADDRESS>) created
2023-07-05 17:47:51.863  - ESC[34mdebugESC[39m: snmp.0 (14920) handleConnectionInfo
2023-07-05 17:48:04.933  - ESC[34mdebugESC[39m: snmp.0 (14920) createReaderSession - device ShBridge (<IP_ADDRESS>)
2023-07-05 17:48:04.951  - ESC[34mdebugESC[39m: snmp.0 (14920) snmpCreateSession - device ShBridge (<IP_ADDRESS>)
2023-07-05 17:48:04.953  - ESC[34mdebugESC[39m: snmp.0 (14920) session for device "ShBridge" (<IP_ADDRESS>) created

I have checked both options and will observe the behavior and tomorrow I will install 2.4.9.
Great job done.

Thanks for the update

Here finally some info about the logging output of version 2.4.9
2023-07-08 14:48:13.902  - debug: snmp.0 (28213) session for device "ShBridge" (<IP_ADDRESS>) created
2023-07-08 14:48:13.902  - debug: snmp.0 (28213) readOIDs - device "ShBridge" (<IP_ADDRESS>)
2023-07-08 14:48:13.902  - debug: snmp.0 (28213) [ShBridge] processing oid chunk index 0
2023-07-08 14:48:13.903  - debug: snmp.0 (28213) readChunkOIDs - device "ShBridge" (<IP_ADDRESS>), chunk idx 0
2023-07-08 14:48:13.912  - debug: snmp.0 (28213) onReaderSessionError - device ShBridge (<IP_ADDRESS>) - RangeError: Read integer -6870200 is outside the unsigned 32-bit range
2023-07-08 14:48:13.913  - warn: snmp.0 (28213) device ShBridge (<IP_ADDRESS>) reported error RangeError: Read integer -6870200 is outside the unsigned 32-bit range
2023-07-08 14:48:13.913  - debug: snmp.0 (28213) error dump: {JSON.stringify(pErr)}
2023-07-08 14:48:13.913  - debug: snmp.0 (28213) snmpCloseSession - device ShBridge (<IP_ADDRESS>
2023-07-08 14:48:13.914  - debug: snmp.0 (28213) onReaderSessionClose - device ShBridge (<IP_ADDRESS>)
2023-07-08 14:48:13.915  - debug: snmp.0 (28213) [ShBridge] session.get completed for chunk index 0
2023-07-08 14:48:13.915  - debug: snmp.0 (28213) [ShBridge] session.get: Error: Socket forcibly closed
2023-07-08 14:48:13.915  - debug: snmp.0 (28213) setStates - ShBridge.sysName
2023-07-08 14:48:13.991  - debug: snmp.0 (28213) setStates - ShBridge.sysUptime
2023-07-08 14:48:14.052  - debug: snmp.0 (28213) setStates - ShBridge.hrSystemUptime
2023-07-08 14:48:14.120  - debug: snmp.0 (28213) [ShBridge] processing oid chunk index 0 completed
2023-07-08 14:48:14.120  - debug: snmp.0 (28213) session for device "ShBridge" (<IP_ADDRESS>) created

That is the part of the log file showing the error output. Still the range check error about the "unsigned 32bit integer" which is fairly easy to understand that a negative number does not work with it.
I will also attach the a file holding some more lines. These are all lines logged by the SNMP adapter from 14:00 until I copied the file - so about 45 minutes continues log.
For your explanation:
rt01 - is an OpenWrt router with OpenWrt 22.03.5 r20134 and mini_smpd
shbridge - is an OpenWrt router with OpenWrt Backfire 10.03.1
dummy1 - is an ip address with no connected device
dummy2 - is an ip address with a connected device but there is no snmp service running
Thank your for all your effort. I really appreciate your work.
snmp.snip.3.log

Thanks for the feedback.
I've created a 2.4.10 after trying to correct the error output again. I hope the json stringified text is now output. (Sorry fpr the multipel attempts, as I have no glue how to cause an error during oid reading I cannot test the code here.).
So 2.4.10 should no longer display
'2023-07-08 14:48:13.913  - debug: snmp.0 (28213) error dump: {JSON.stringify(pErr)}'
but display the content of the error block instead of {JSOIN.stringify(pErr)}

I will test it and will report

Here is the latest output to the log file showing the error message
2023-07-09 09:52:30.814  - debug: snmp.0 (19952) createReaderSession - device ShBridge (<IP_ADDRESS>)
2023-07-09 09:52:30.830  - debug: snmp.0 (19952) snmpCreateSession - device ShBridge (<IP_ADDRESS>)
2023-07-09 09:52:30.831  - debug: snmp.0 (19952) session for device "ShBridge" (<IP_ADDRESS>) created
2023-07-09 09:52:30.831  - debug: snmp.0 (19952) readOIDs - device "ShBridge" (<IP_ADDRESS>)
2023-07-09 09:52:30.832  - debug: snmp.0 (19952) [ShBridge] processing oid chunk index 0
2023-07-09 09:52:30.832  - debug: snmp.0 (19952) readChunkOIDs - device "ShBridge" (<IP_ADDRESS>), chunk idx 0
2023-07-09 09:52:30.837  - debug: snmp.0 (19952) onReaderSessionError - device ShBridge (<IP_ADDRESS>) - RangeError: Read integer -4304 is outside the unsigned 32-bit range
2023-07-09 09:52:30.837  - warn: snmp.0 (19952) device ShBridge (<IP_ADDRESS>) reported error RangeError: Read integer -4304 is outside the unsigned 32-bit range
2023-07-09 09:52:30.838  - debug: snmp.0 (19952) error dump: {}
2023-07-09 09:52:30.838  - debug: snmp.0 (19952) snmpCloseSession - device ShBridge (<IP_ADDRESS>
2023-07-09 09:52:30.839  - debug: snmp.0 (19952) onReaderSessionClose - device ShBridge (<IP_ADDRESS>)
2023-07-09 09:52:30.839  - debug: snmp.0 (19952) [ShBridge] session.get completed for chunk index 0
2023-07-09 09:52:30.840  - debug: snmp.0 (19952) [ShBridge] session.get: Error: Socket forcibly closed
2023-07-09 09:52:30.840  - debug: snmp.0 (19952) setStates - ShBridge.sysName
2023-07-09 09:52:30.875  - debug: snmp.0 (19952) handleConnectionInfo
2023-07-09 09:52:30.935  - debug: snmp.0 (19952) setStates - ShBridge.sysUptime
2023-07-09 09:52:30.991  - debug: snmp.0 (19952) setStates - ShBridge.hrSystemUptime
2023-07-09 09:52:31.065  - debug: snmp.0 (19952) [ShBridge] processing oid chunk index 0 completed
2023-07-09 09:52:31.066  - debug: snmp.0 (19952) session for device "ShBridge" (<IP_ADDRESS>) created

I will attach also a bigger part of the log file where the error message disappears after the value turns positive again. That behavior makes me curious to investigate and I will try to figure out what might be the reason for it.

Thanks and greetings to Austria

[snmp.snip.5.log](https://github.com/iobroker-community-adapters/ioBroker.snmp/files/11994859/snmp.snip.5.log)

Thanks for Info.
Unfortunatly the error data seesm to be a simple string only and does not contain additional data.
BUT
I think I found a related issue and PR at the snmp base module:
https://github.com/markabrahams/node-net-snmp/pull/235
As a PR exists I would expect an improved release within a few months.
The reason could be (and this would be fixed by the PR) that you snmp client sends incorrect data. A big unit32 number seems to need a leading zero byte (if the MSB of the real data is set) otherwiese ist is assumed to be negative.
So I see the following real solutions:
-) you cantact the development of your snmp cleint and ask them to check / fix the issue
-) you wait for a ne release of node-net-snmp
-) you manage to keep the affected oids at a lower numreic level, i.e. by restting some counters etc.
I do not see any change to work around the problem at ioBroker.snmp adapter.

Well, the first issue loosing the UDP ports is solved.
I will wait for the mentioned pull request to be merged and close that issue now.
Thank you.

I've just released v2.4.11 which included a new version of net-node-snmp. According to releasenotes this new version has a change adding more tolarance for incorrect uint32 data.
Feel free to test the new release and let me know if the problem is fixed completly if you have time for a test. Adapter version 2.4.11 will be available at latest repo tomorrow.

Thank you. Will install it right away now and let you know.

@TheKilroy
Sorry to distzurb
Did you have time to update the adapter in the meantime?
Is the original error fixed now (or better didi it no longer occure)?

Sorry for the late reply.I have updated both installations and all seems to be fixed. No error anymore - not the original one and not the resulting. So you may close it from my point of view.
Big thank you.


[Event Hubs] Fix for Buffered Producer Flush

Summary
The focus of these changes is to fix an error where flushing the buffered producer leaves a disposed cancellation source reference in place, causing an error the next time an event is enqueued and publishing restarts.
Also included:

Some Event Hubs test quality and stability fixes
Normalization of paths for the Azure.ResourceManager.HealthBot source

Details
The HealthBot paths had mixed casing, causing some local git commands to fail.  These have now been unified.

References and Resources

[BUG] EventHubBufferedProducerClient: Flushing the client results in ObjectDisposedException on subsequent enqueueing of events (#32871)

API change check
API changes are not detected in this pull request.

All tests passing, outside of Ben's test pipeline.   I'm going to ignore failures there and override check enforcer to avoid having to artificially push changes.

/check-enforcer override


dnm: test fc ci

Because the tests repo doesn't test fc, I have to use the runtime
repo ci to validate a tests PR for fc CI, for now.
Fixes: #999999
Depends-on: github.com/kata-containers/tests#1847

/test

@bergwolf still using this test PR or we can close it now?


Adds story label background colours.

Description

Adds background style to labels + epics for stories.
In fact, the css is a modified version of Bootstrap badge style.
The code assumes the stories have only one epic and that pic is the current epic.

Pivotal Tracker labels are all in lower case even if the epics or labels contain upper case.
The first badge will always be the current epic being drawn.

For Reviewers

It has been a very long time since I have done straight css.  I could not really get nested css to work.  Is there a better way?

Screenshots
Story that contains multiple labels

Story that contains only the epic

I could not really get nested css to work. Is there a better way?

Not really? I think this is fine.


GH-35166: [C++] Increase precision of decimals in aggregate functions

Rationale for this change
As documented in #35166, when Arrow performs a sum, product, or mean of an array of type decimalX(P, S), it returns a scalar of type decimalX(P, S). This is true even if the aggregate does not fit in the specified precision. For instance, a sum of two decimal128(1, 0)'s such as 1 + 9 is a decimal128(2, 0). But (in Python):
import pyarrow as pa
from decimal import Decimal

arr = pa.array([Decimal("1"), Decimal("9")], type=pa.decimal128(1, 0))
assert arr.sum().type == pa.decimal128(1, 0)

This is recognized in the rules for binary addition and multiplication of decimals (see footnote 1 in this section), but this does not apply to array aggregates.
In #35166 I did a bit of research following a question from @westonpace , and it seems that there's no standard approach to this across DBMS's, but a common solution is to set the precision of the result of a sum to the maximum possible precision of the underlying type. That is, a sum of decimal128(1, 0)'s becomes a decimal128(38, 0).
However, products and means differ further. For instance, in both instances, duckdb converts a decimal to a double, which makes sense as the precision of the product of an array of decimals would likely be huge, e.g., an array of size N with precision 2 decimals would have precision at least 2^N.
This PR implements the minimum possible change: replace all return types of a product, sum, or mean aggregate of decimal128(P, S) to decimal128(38, S) and decimal256(P, S) to decimal256(76, S).
Please note, this PR is not done (see the checklist below), as it would be good to get feedback on the following before going through the whole checklist:

Is this the correct change to make (especially for products and means); and
The implementation relies on overriding "out types," which is how the current mean implementation works, but perhaps there's a better way to approach this.

What changes are included in this PR?

[x] Update C++ kernels to support the change
[x] Update docs to reflect the change
[x] Fix tests in the languages that depend on the C++ engine
[x] Determine if there are other languages which do not depend on the C++ engine which should also be updated

Are these changes tested?
They are tested in the following implementations:

[x] Python
[x] C++
[ ] Java Java does not currently implement any dependent tests

Are there any user-facing changes?
Yes. This changes the return type of a scalar aggregate of decimals.
This PR includes breaking changes to public APIs.
Specifically, the return type of a scalar aggregate of decimals changes. This is unlikely to break downstream applications as the underlying data has not changed, but if an application relies on the (incorrect!) type information for some reason, it would break.

GitHub Issue: #35166

@zeroshade I noticed in #43957 that you were adding in Decimal32/64 types, which I think will have the same problem that this PR addresses. I was curious if you might have interest in reviewing this PR?

@khwilson Sure thing, i'll try to take a look at this in the next day or so

Hi @zeroshade just checking in! Thanks again for taking a look

Thanks for the review!
By a user-side cast, do you mean that users should essentially do:
select avg(cast(blah as decimal(big-precision)))

instead of
select avg(blah)

or do you mean that this code should "inject" a cast on the "user" side?
If you mean putting the cast onto the user, then I would think you'd want to add an error if the answer can't fit into the default precision, but that seems like it would be more disruptive (and out of step with how other systems handle decimal aggregates).
If you mean "injecting" the cast on the user side, would that end up creating a copy of the array?

@khwilson Hey sorry for the delay from me here, I've been traveling a lot lately for work and have been at ASF Community Over Code this week. I promise i'll get to this soon. In the meantime, you're in the very capable hands of @mapleFU

No problem! Hope your travels were fun!

Generally this method is ok for me, but I'm not so familiar with the "common solutions" here. I'll dive into Presto/ClickHouse

Would you mind making this Ready for review?

Sure!

@mapleFU I believe this is done now. Some notes on the diff:

The hash aggregates had to be updated as well (missed them in the first pass)
I've also added in Decimal32/64 support the basic aggregates (sum, product, mean, min/max, index). However, there's quite a lot of missing support for these types still in compute (most notably in casts)
Docs are updated to reflect the change

And a note that quite a few tests are failing for what appears to be the same reason as #41390. Happy to address them if you'd like.

I'm lukewarm about the approach here. Silently casting to the max precision discards metadata about the input; it also risks producing errors further down the line (if e.g. the max precision is deemed too large for other operations). It also doesn't automatically eliminate any potential overflow, for example:
>>> a = pa.array([789.3] * 20).cast(pa.decimal128(38, 35))
>>> a
<pyarrow.lib.Decimal128Array object at 0x7f0f103ca7a0>
[
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440,
<PHONE_NUMBER>9999995452526491135358810440
]
>>> pc.sum(a)
<pyarrow.Decimal128Scalar: Decimal('-1228.11834604692408266343214451664848480')>

We should instead check that the result of an aggregate fits into the resulting Decimal type, while overflows currently pass silently:
>>> a = pa.array([123., 456., 789.]).cast(pa.decimal128(4, 1))
>>> a
<pyarrow.lib.Decimal128Array object at 0x7f0ed06261a0>
[
  123.0,
  456.0,
  789.0
]
>>> pc.sum(a)
<pyarrow.Decimal128Scalar: Decimal('1368.0')>
>>> pc.sum(a).validate(full=True)
Traceback (most recent call last):
  ...
ArrowInvalid: Decimal value 13680 does not fit in precision of decimal128(4, 1)

Two problems with just validating afterward: First, I'd expect in reasonable cases for the validation to fail. A sum of 1m decimals of approximately the same size you'd expect to have 6 more digits of precision. I assume this is why all the DBMSs I looked at increase the precision by default.
Second, just checking for overflow doesn't solve the underlying problem. Consider:
a = pa.array([789.3] * 18).cast(pa.decimal128(38, 35))
print(pc.sum(a))
pc.sum(a).validate(full=True)  # passes

In duckdb, they implement an intermediate check to make sure that there's not an internal overflow:
tab = pa.Table.from_pydict({"a": a})
duckdb.query("select sum(a) from tab")
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# duckdb.duckdb.OutOfRangeException: Out of Range Error: Overflow in HUGEINT addition:
#<PHONE_NUMBER>99999990905052982270717620880 +<PHONE_NUMBER>9999995452526491135358810440

Notably, this lack of overflow checking also applies to integer sums in arrow:
>>> pa.array([9223372036854775800] * 2, type=pa.int64())
<pyarrow.lib.Int64Array object at 0x10c1d8b80>
[
<PHONE_NUMBER>854775800,
<PHONE_NUMBER>854775800
]
>>> pc.sum(pa.array([9223372036854775800] * 2, type=pa.int64()))
<pyarrow.Int64Scalar: -16>
>>> pc.sum(pa.array([9223372036854775800] * 2, type=pa.int64())).validate(full=True)

Two problems with just validating afterward: First, I'd expect in reasonable cases for the validation to fail. A sum of 1m decimals of approximately the same size you'd expect to have 6 more digits of precision.

It depends obviously if all decimals are of the same sign, and what their actual magnitude is.

Second, just checking for overflow doesn't solve the underlying problem.

In the example above, I used a validate call simply to show that the result was indeed erroneous. I didn't mean we should actually call validation afterwards. We should instead check for overflow at each individual aggregation step (for each add or multiply, for example). This is required even if we were to bump the result's precision to the max.

Notably, this lack of overflow checking also applies to integer sums in arrow:

Yes, and there's already a bug open for it: https://github.com/apache/arrow/issues/37090

Nice! I'm excited for the checked variants of sum and product!
With the integer overflow example, I only meant to point out that the compute module currently allows overflows, so I think it would be unexpected for sum to complain about an overflow only if the underlying type was a decimal. But if the goal with #37090 is to replace sum with a checked version, then the solution of erroring makes a lot of sense, and I'd be happy to implement it when #37536 gets merged. :-)
Still, I do think that users would find it unexpected to get an error if the sum fit in the underlying storage since this is how all the databases I've used (and the four I surveyed in #35166) have operated.

Hi @pitrou sorry for dropping the ball on this a bit earlier. Are you still interested in merging this at some point? I'm happy to explore some alternative options for how to do the decimal expansion, or alternatively am happy to push #37536 forward


Fixes Release Notes 0.15 canonical form yaml

Summary
Fixes Reference to dagdependency_to_dag() in add-canoical-form-*.yaml wrt Issue #4930
Details and comments
Change ~qiskit.converters.dagdependency_to_ciruit to ~qiskit.converters.dagdependency_to_dag
Fixes #4930

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@SauravMaheshkar please don't manually update the branch, this isn't necessary after the automerge label is applied. The automerge label is used as a trigger for mergify to queue this PR for ci to try and avoid CI resource starvation (although typically to limited effect) and will automatically update the branch when it's ready. Updating the branch manually will force a queue flush and rebuild which just delays everything approved pending CI results (including this PR) from merging.

Ah okay thanks @mtreinish for letting me know. Sorry, didn't know about this.


Twist Gate Refactor

Removed remote control functionality.
Removed main loop so that commands are passed through immediately.
Other small cleanups and refactors.

See original MR for more details: https://gitlab.com/astuff/autoware.ai/core_planning/-/merge_requests/84

@icolwell-as I'm not a big fan of removing existing APIs without a lot of discussion with the community, especially in the last feature release of the software. Could AStuff reformat this MR to leave that functionality but provide the other improvements?

Unfortunately, removing the remote command capabilities is what made it much easier to streamline/simplify the operation of this node.
I'll leave this PR open and take a closer look later to see how much additional work would be required.

@icolwell-as Any updates?

Hi @JWhitleyWork, I think I'll close this PR. In retrospect it was mostly tailored to our specific use case and may not be appropriate for the wider community. Also, we don't really have a clear understanding of the remote control use case in order to make sure its re-implemented and fully tested along with the other control input types.
Thanks!

Thanks, @icolwell-as.


Duplicated disk_consistent_lsn in IndexPart

We maintain disk_consistent_lsn at IndexPart::disk_consistent_lsn and IndexPart::metadata::disk_consistent_lsn. Do we actually need to maintain both? Currently scrubber is performing a check to see if the two are the same. We can get rid of this check if one of them is no longer needed.

As a precursor to fully removing the weird binary metadata blob, we could make a change to remove the disk_consistent_lsn rust struct member, and just skip that field when decoding it.


Implementation of finishOnCallbackQueue

Just posting an issue for the implementation of finishOnCallbackQueue. You call finish(error:error or nil) from the web request completion blocks within performCKOperation, that then causes finishOnCallbackQueue where you call then call the operation specific completion block and then call super to end the operation. The reason for calling the completion block from a dispatch queue is to hide the stack trace from whatever real URL request operation was actually getting the data. The progress blocks should probably also be called on this queue too but I left them as is just now.
Also within the web blocks you gather any partial errors in to a property. And any data required like savedRecords or savedTokens. Then within override finishOnCallbackQueue(error) if the error was null, i.e. it wasn't cancelled, couldn't be started because of invalid params, or any other major error, then you can build an error that has a userInfo dict to hold the partial errors.
The advantage to having the op's completion block only called from finishOnCallbackQueue is that CKOperation can then call finish (e.g. if cancelled) and since it feeds through to the operation subclass, it can then call the correct completion handler.

Continued in Issue #11


Design tile layout

As a project developer, I want to predefine the set of tiles that should be displayed visually, so that project developers can build the necessary supporting queries

Mentioned in #193


Unable to parse input JSON

I get the next error when benchmark finish:
Unable to parse input JSON!

Phoronix Test Suite v10.0.1
An outdated version of the Phoronix Test Suite is installed.
The version in use is 10.0.1 (10010), but the latest is pts-core 10810.
Visit https://www.phoronix-test-suite.com/ to update this software.

{
    "title": "resultdata",
    "results": [
        {
            "test": "pts\/c-ray-1.2.0",
            "arguments": "Total Time - 4K, 16 Rays Per Pixel",
            "units": "Seconds",
            "results": {
                "test_result": {
                    "value": "546.696",
                    "all_results": "532.65:548.734:558.704"
                }
            }
        }
    ]
}

Traceback (most recent call last):
  File "/extract_and_add_meta.py", line 43, in <module>
    main()
  File "/extract_and_add_meta.py", line 26, in main
    data = in_data["results"][0]

These are the steps I did after creating the machine "EOSC-Synergy benchmark virtual machine" from IM-Dashboard:
#> cd $HOME
#> mkdir out
#> docker run -v $(pwd)/out:/out_dir thechristophe/openbench-c-ray

It seems the issue is that the phoronix test suite displays the update banner when requesting just the json (why???), so the script that adds CPU metadata fails to parse it

Luckily a new version of phoronix test suite provides the ability to write it to a file directly without having to redirect outputs, hopefully avoiding other such issues in the future. Unfortunately, the json output structure changed somewhat, too.
Please try with the new image: perfbench/openbench-c-ray:v1
I likely will have to re-run the tests on all other sites...

Hi,
The new image now generate the JSON file, but when I try to sumbit the restult to the EOSC-Perf website (https://performance.services.fedcloud.eu/result-submission#) I get an error:
Error: Server could not process our submission:

{
    "code": 422,
    "errors": {
        "error": "'arguments' is a required property",
        "path": "deque([])"
    },
    "status": "Unprocessable Entity"
}

Your input may be malformed or wrong, please verify if everything is correct. If you believe this to be a bug, please submit an issue about it on [GitHub](https://github.com/EOSC-synergy/eosc-perf/issues)!

You may have submitted it under thechristophe/openbench-c-ray, which has a different schema so results from perfbench/openbench-c-ray are not compatible. I have not yet added perfbench/openbench-c-ray, because so far most of the machines I've been able to deploy on IM did not work correctly so I could gather data.

Ok, I understand it.
I await you upload perfbench/openbench-c-ray or change the thechristophe/openbench-c-ray with the changes that correct the json output file.
regards

It seems I've forgotten about this issue, oops.
The image is published at https://hub.docker.com/repository/docker/perfbench/openbench-c-ray.
Instructions on running it are available at https://github.com/EOSC-synergy/eosc-perf-benchmarks/blob/main/phoronix-c-ray-docker/README.md.


Process fault messages

The error handling looks great. Seems to work fine too.

About the refactor Hub: I kept that separated to make it possible to change the readout for the inverter static data to a longer interval. Once per hour or so. As it is data that never changes.

About the refactor Hub: I kept that separated to make it possible to change the readout for the inverter static data to a longer interval. Once per hour or so. As it is data that never changes.

It's still separated only tried to prevent some unnecessary indents.

The error handling looks great. Seems to work fine too.

Waiting for an error to a cure to fully test is. That's also why I added the debug message.
Once I processed a real error msg I will remove the draft flag from the PR.
Did you already have a error message?

The error handling looks great. Seems to work fine too.

Waiting for an error to a cure to fully test is. That's also why I added the debug message.
Once I processed a real error msg I will remove the draft flag from the PR.
Did you already have a error message?

Yes, I forced an error and tested with a separate script.
(https://github.com/wimb0/saj_modbus/blob/main/read_invertor_errors.py)
Did not check it with the actual integration in HA.
Forcing an error is easy, just turn off the Current Breaker.
What I got is:
faultMsg 0x00000000 0x00800000 0x00800000
Fault message: Code 24: Master Phase1 No Grid Error, Code 73: Slave Phase1 No Grid Error

All looks good, ready to go.
Tnx for the test hint ;)

Thanks for the work on this. It's available in release 1.4.0.
I was struggling with this for a while. Didn't know you could do hex lookup/conversion this easy in Python.

Thanks for the work on this. It's available in release 1.4.0.

Your welcome :)

I was struggling with this for a while. Didn't know you could do hex lookup/conversion this easy in Python.

Had some experience with hex and binary comparison due to my work on a Bluetooth HA integration ...
But in the end HEX values are just integers in python. The 0x... is mostly a visual representation. I kept the values in the const.py in HEX as that makes it easier to check them against the modbus documentation


Reverse K nodes in a linked list.

We will be given a linked list :
1->2->3->4->5->6->NULL
if the value of k=2, after reversing it will look like:
2->1->4->3->6->5->NULL
Will implement the concept of pointers and recursion to implement the problem, with comments at every step to make it clean and understandable
Want to do it in C++

hello , can i contribute to this issue in python
thank you!

I would like to implement this using c++.kindly assign me this issue.

@raksha009 please assign


pickle renders the student and glovebox setups tightly coupled

we currently use the pickle format, which is language specific and package version sensitive, in order to communicate challenge results between the student repo and glovebox
transmitting a serialised memory object over the network contraints our ability to process it : students need to match precisely the glovebox setup (python + package versions), and we have yet to find a way of preventing them from upgrading or downgrading their packages through pip install during the bootcamp
a solution to loosen the coupling could be to follow the path of the web and add support for standard data types such as json for the exchange between our apps

Hello @gmanchon 👋
Do you have a specific example / a repo reproducing a problem at hand? Looking at the code, we are dumping an instance of ChallengeResult which should store simple attributes. If attributes are only Python primitive types, then how could package versions interfere?
Maybe my reasoning is missing something, that's why a specific challenges where this problem actually arises will help me understand this a bit better. One resolution could be to update the test challenge result not to use classes from packages but indirection with Python primitive types (integers, floats, lists, dict, etc.)
Thanks!

Hello @ssaunier 👋
I was able to reproduce the issue with pandas 1.3.5 on student setup and pandas 1.2 on glovebox setup.
The challenge on which the issue was encountered stores a dataframe.
This is not a practical issue as long as we stick to simple data types because pickles is unlikely to change its serialisation format.
The setup is reproduced in this repo (no LW data inside).
Which raises this error :
➜  nbresult_test git:(master) pytest tests/test_data.py                                     [🐍 nbr_glovebox 🍓 3.0.3]
================================================= test session starts =================================================
platform darwin -- Python 3.8.12, pytest-7.0.0, pluggy-1.0.0
rootdir: /Users/gmanchon/code/sandbox/nbresult_test
collected 1 item

tests/test_data.py F                                                                                [100%]

====================================================== FAILURES =======================================================
___________________________________________ TestData.test_data_is_above_82 ____________________________________________

self = <tests.test_data.TestData testMethod=test_data_is_above_82>

    def setUp(self):
        """Load the pickle file"""
        klass = self.__class__.__name__
        name = re.sub(r'(?<!^)(?=[A-Z])', '_', klass).lower()[len('test_'):]
        result_file = os.path.join(os.getcwd(), "tests", f"{name}.pickle")
>       self.result = pickle.load(open(result_file, 'rb'))
E       AttributeError: Can't get attribute 'new_block' on <module 'pandas.core.internals.blocks' from '/Users/gmanchon/.pyenv/versions/3.8.12/envs/nbr_glovebox/lib/python3.8/site-packages/pandas/core/internals/blocks.py'>

../../../.pyenv/versions/3.8.12/envs/nbr_glovebox/lib/python3.8/site-packages/nbresult/__init__.py:52: AttributeError
=============================================== short test summary info ===============================================
FAILED tests/test_data.py::TestData::test_data_is_above_82 - AttributeError: Can't get attribute 'new_block' on <mod...
================================================== 1 failed in 0.28s ==================================================

Thanks for investigating 🙏 - What do you think, should we update the challenge's test to try and stick to simpler data?

Good idea 💡 ! Before the GHA maybe have a script to list challenges which are at risk, to allow time to fix them. Once they are all fixed => implement that script as a GHA to keep it green in the future!

script is pending, would not have thought to see that diversity in the content
➜  data-solutions git:(master) nbr run .                                                                                [🐍 lewagon 🍓 3.0.3]
run check

Pickles containing:
- str: 155
- ndarray: 27
- dtype: 2
- tuple: 39
- int64: 18
- set: 1
- int: 31
- float64: 51
- Index: 4
- dict: 7
- list: 13
- Series: 2
- float: 23
- LinearRegression: 1
- DataFrame: 10
- bool_: 1
- RandomizedSearchCV: 1

Pickles containing str:
- 02-Data-Toolkit/01-Data-Analysis/01-Notebook/tests/import_hello.pickle: (name, sentence)
- 02-Data-Toolkit/01-Data-Analysis/02-Numpy/tests/numpy.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/date.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/filtered_gas.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/full_gas.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/merged_dataframes.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/oil.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/all_df.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/inner_merge.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/left_merge.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games_combined.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games_event.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games_season.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/outer_merge.pickle: (name)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/right_merge.pickle: (name)
- 02-Data-Toolkit/02-Data-Sourcing/00-Warmup/tests/warmup.pickle: (name)
- 02-Data-Toolkit/02-Data-Sourcing/01-Stock-Market-API/tests/apple.pickle: (name, index_name)
- 02-Data-Toolkit/02-Data-Sourcing/02-Scraping/tests/books.pickle: (name, title)
- 02-Data-Toolkit/02-Data-Sourcing/03-Text-Extraction-with-Regex/tests/patterns.pickle: (name, zipcode_re, date_re, quantity_re, amount_re, quantity_grp_re, amount_grp_re)
- 02-Data-Toolkit/02-Data-Sourcing/03-Text-Extraction-with-Regex/tests/receipts.pickle: (name)
- 02-Data-Toolkit/03-Data-Visualization/01-Matplotlib-Intro/tests/quiz.pickle: (name, answer1, answer2)
- 03-Maths/01-Algebra-Calculus/01-real-estate-estimator/tests/features.pickle: (name)
- 03-Maths/01-Algebra-Calculus/01-real-estate-estimator/tests/solution.pickle: (name)
- 03-Maths/01-Algebra-Calculus/01-real-estate-estimator/tests/target.pickle: (name)
- 03-Maths/01-Algebra-Calculus/02-real-estate-advanced-estimator/tests/flats.pickle: (name)
- 03-Maths/01-Algebra-Calculus/02-real-estate-advanced-estimator/tests/univariate.pickle: (name)
- 03-Maths/01-Algebra-Calculus/03-Scipy/tests/global_optimization.pickle: (name)
- 03-Maths/01-Algebra-Calculus/03-Scipy/tests/minimize2d.pickle: (name)
- 03-Maths/01-Algebra-Calculus/03-Scipy/tests/minimize_constraints.pickle: (name)
- 03-Maths/02-Statistics-Probabilities/01-Law-of-Large-Numbers/tests/expected_value_coins.pickle: (name)
- 03-Maths/02-Statistics-Probabilities/01-Law-of-Large-Numbers/tests/expected_value_dice.pickle: (name)
- 03-Maths/02-Statistics-Probabilities/02-Random-Variables/tests/factorial.pickle: (name)
- 03-Maths/02-Statistics-Probabilities/02-Toss-a-Coin/tests/factorial.pickle: (name)
- 03-Maths/02-Statistics-Probabilities/03-Central-Limit-Theorem/tests/distribution.pickle: (name, skewness)
- 03-Maths/02-Statistics-Probabilities/03-Central-Limit-Theorem/tests/probability.pickle: (name)
- 03-Maths/02-Statistics-Probabilities/03-Central-Limit-Theorem/tests/zscore.pickle: (name)
- 04-Decision-Science/01-Project-Setup/02-Data-Preparation/tests/get_data.pickle: (name)
- 04-Decision-Science/01-Project-Setup/03-Exploratory-Analysis/tests/exploratory.pickle: (name)
- 04-Decision-Science/01-Project-Setup/Optional-Metric-Design/tests/orders.pickle: (name)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/distance.pickle: (name)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/get_wait_time.pickle: (name)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/number_products.pickle: (name)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/number_sellers.pickle: (name)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/price.pickle: (name)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/review_score.pickle: (name)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/training.pickle: (name)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/wait_time.pickle: (name)
- 04-Decision-Science/03-Linear-Regression/02-Sellers/tests/seller.pickle: (name)
- 04-Decision-Science/03-Linear-Regression/Optional-Products/tests/products.pickle: (name)
- 04-Decision-Science/04-Logistic-Regression/01-Logit/tests/logit.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/cv_results.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/prediction.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/trained_model.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/variables.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/02-Learning-Curves/tests/added_features.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/02-Learning-Curves/tests/cv_score.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/02-Learning-Curves/tests/prediction.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/02-Learning-Curves/tests/reduced_dataset.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/03-Problems/tests/problem_1.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/03-Problems/tests/problem_2.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/04-Logistic-Regression/tests/accuracy.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/04-Logistic-Regression/tests/predictions.pickle: (name)
- 05-ML/01-Fundamentals-of-Machine-Learning/04-Logistic-Regression/tests/reduced_accuracy.pickle: (name)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/duplicates.pickle: (name)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/encoding.pickle: (name)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/missing_values.pickle: (name)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/scaling.pickle: (name)
- 05-ML/02-Prepare-the-dataset/02-Feature-Selection/tests/base_model.pickle: (name)
- 05-ML/02-Prepare-the-dataset/02-Feature-Selection/tests/correlation.pickle: (name)
- 05-ML/02-Prepare-the-dataset/02-Feature-Selection/tests/feature_permutation.pickle: (name, feature)
- 05-ML/02-Prepare-the-dataset/02-Feature-Selection/tests/reduced_complexity_model.pickle: (name)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/base_model.pickle: (name)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/collinearity.pickle: (name)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/encoding.pickle: (name)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/missing_values.pickle: (name)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/scaling.pickle: (name)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/strong_model.pickle: (name)
- 05-ML/03-Performance-metrics/01-KNN/tests/best_model.pickle: (name, model)
- 05-ML/03-Performance-metrics/01-KNN/tests/default_score.pickle: (name)
- 05-ML/03-Performance-metrics/01-KNN/tests/optimal_k.pickle: (name)
- 05-ML/03-Performance-metrics/01-KNN/tests/price_error.pickle: (name)
- 05-ML/03-Performance-metrics/01-KNN/tests/scale_sensitivity.pickle: (name)
- 05-ML/03-Performance-metrics/02-Electrocardiograms/tests/best_model.pickle: (name, model)
- 05-ML/03-Performance-metrics/02-Electrocardiograms/tests/class_balance.pickle: (name)
- 05-ML/03-Performance-metrics/02-Electrocardiograms/tests/logistic_regression_evaluation.pickle: (name)
- 05-ML/03-Performance-metrics/02-Electrocardiograms/tests/precision.pickle: (name)
- 05-ML/03-Performance-metrics/02-Electrocardiograms/tests/prediction.pickle: (name, prediction)
- 05-ML/03-Performance-metrics/03-Threshold-Adjustments/tests/base_precision.pickle: (name)
- 05-ML/03-Performance-metrics/03-Threshold-Adjustments/tests/decision_threshold.pickle: (name)
- 05-ML/03-Performance-metrics/03-Threshold-Adjustments/tests/recommendation.pickle: (name, recommendation)
- 05-ML/03-Performance-metrics/03-Threshold-Adjustments/tests/scaled_features.pickle: (name)
- 05-ML/04-Under-the-hood/01-Loss-Functions/tests/loss_functions.pickle: (name)
- 05-ML/04-Under-the-hood/02-Solvers/tests/new_data_prediction.pickle: (name)
- 05-ML/04-Under-the-hood/02-Solvers/tests/solvers.pickle: (name, fastest_solver)
- 05-ML/04-Under-the-hood/03-Batch-Gradient-Descent/tests/descent.pickle: (name)
- 05-ML/05-Model-Tuning/01-Workflow/tests/knn.pickle: (name)
- 05-ML/05-Model-Tuning/01-Workflow/tests/r2.pickle: (name)
- 05-ML/05-Model-Tuning/02-Regularization/tests/lasso.pickle: (name)
- 05-ML/05-Model-Tuning/02-Regularization/tests/ridge.pickle: (name)
- 05-ML/05-Model-Tuning/02-Regularization/tests/unregularized.pickle: (name)
- 05-ML/05-Model-Tuning/03-SVM/tests/generalization.pickle: (name)
- 05-ML/05-Model-Tuning/03-SVM/tests/linear_svm.pickle: (name)
- 05-ML/05-Model-Tuning/03-SVM/tests/poly_svm.pickle: (name)
- 05-ML/05-Model-Tuning/03-SVM/tests/rbf_svm.pickle: (name)
- 05-ML/05-Model-Tuning/03-SVM/tests/sigmoid_svm.pickle: (name)
- 05-ML/05-Model-Tuning/03-SVM/tests/svm_rbf.pickle: (name)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/blobs.pickle: (name)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/color_count.pickle: (name)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/image_shape.pickle: (name)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/imageshape.pickle: (name)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/two_means.pickle: (name)
- 05-ML/06-Unsupervised-Learning/02-Face-Recognition/tests/classification.pickle: (name)
- 05-ML/06-Unsupervised-Learning/02-Face-Recognition/tests/components.pickle: (name)
- 05-ML/06-Unsupervised-Learning/02-Face-Recognition/tests/projection.pickle: (name)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/datasets.pickle: (name)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/feature_engineering.pickle: (name)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/metadata.pickle: (name)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/ratings.pickle: (name)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/recommender.pickle: (name, best_similarity)
- 05-ML/07-Ensemble-Methods/01-Houses-Kaggle-Competition/tests/features_overview.pickle: (name)
- 05-ML/07-Ensemble-Methods/01-Houses-Kaggle-Competition/tests/preproc_baseline.pickle: (name)
- 05-ML/07-Ensemble-Methods/01-Houses-Kaggle-Competition/tests/submission_baseline.pickle: (name, submission_dtypes)
- 05-ML/08-Workflow/01-Preprocessor-Tuning/tests/solution.pickle: (name)
- 05-ML/08-Workflow/02-Custom-Transformer/tests/pipe.pickle: (name)
- 05-ML/08-Workflow/02-Custom-Transformer/tests/prediction.pickle: (name)
- 05-ML/08-Workflow/03-Tuning-Pipeline/tests/solution.pickle: (name)
- 05-ML/08-Workflow/04-Pickle-Pipe/tests/solution.pickle: (name)
- 05-ML/08-Workflow/05-Hand-Made-Standardizer/tests/standardizer.pickle: (name)
- 06-Deep-Learning/01-Fundamentals-of-Deep-Learning/02-Your-first-Neural-Network/tests/deeper_model.pickle: (name)
- 06-Deep-Learning/01-Fundamentals-of-Deep-Learning/02-Your-first-Neural-Network/tests/first model.pickle: (name)
- 06-Deep-Learning/01-Fundamentals-of-Deep-Learning/02-Your-first-Neural-Network/tests/first_model.pickle: (name)
- 06-Deep-Learning/01-Fundamentals-of-Deep-Learning/03-Multiclass-classification/tests/baseline.pickle: (name)
- 06-Deep-Learning/02-Optimizer-loss-and-fitting/03-Finetune-your-Neural-Network/tests/solution.pickle: (name)
- 06-Deep-Learning/02-Optimizer-loss-and-fitting/04-Credit-Card-Challenge/tests/solution.pickle: (name)

Pickles containing ndarray:
- 02-Data-Toolkit/01-Data-Analysis/02-Numpy/tests/numpy.pickle: (ten, from_five, A, B, lin_twenty, C, E, F, reshaped_G, hi_sum)
- 03-Maths/01-Algebra-Calculus/01-real-estate-estimator/tests/features.pickle: (features)
- 03-Maths/01-Algebra-Calculus/01-real-estate-estimator/tests/solution.pickle: (theta)
- 03-Maths/01-Algebra-Calculus/01-real-estate-estimator/tests/target.pickle: (target)
- 03-Maths/01-Algebra-Calculus/02-real-estate-advanced-estimator/tests/univariate.pickle: (squared_errors)
- 03-Maths/01-Algebra-Calculus/03-Scipy/tests/global_optimization.pickle: (Xmin_shgo, Xmin_dual)
- 03-Maths/01-Algebra-Calculus/03-Scipy/tests/minimize_constraints.pickle: (X0, Xmin)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/trained_model.pickle: (slope)
- 05-ML/03-Performance-metrics/01-KNN/tests/scale_sensitivity.pickle: (rescaled_features)
- 05-ML/03-Performance-metrics/03-Threshold-Adjustments/tests/scaled_features.pickle: (scaled_features)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/blobs.pickle: (lower_centroid)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/two_means.pickle: (clusters)
- 05-ML/08-Workflow/02-Custom-Transformer/tests/prediction.pickle: (prediction)
- 05-ML/08-Workflow/04-Pickle-Pipe/tests/solution.pickle: (predicted_class)
- 05-ML/08-Workflow/05-Hand-Made-Standardizer/tests/standardizer.pickle: (X_train_transformed, X_test_transformed)

Pickles containing dtype:
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/date.pickle: (month_type)
- 02-Data-Toolkit/02-Data-Sourcing/01-Stock-Market-API/tests/apple.pickle: (index_type)

Pickles containing tuple:
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/filtered_gas.pickle: (yearly_gas)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/full_gas.pickle: (yearly_gas_shape)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/merged_dataframes.pickle: (merged_df_shape)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/oil.pickle: (filtered_oil_shape)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/all_df.pickle: (all_df_shape)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/inner_merge.pickle: (inner_merged_shape)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/left_merge.pickle: (left_merged_shape)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games_combined.pickle: (top_combined_shape)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games_event.pickle: (top_country_event_shape)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games_season.pickle: (top_country_season_shape)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/outer_merge.pickle: (outer_merged_shape)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/right_merge.pickle: (right_merged_shape)
- 02-Data-Toolkit/02-Data-Sourcing/03-Text-Extraction-with-Regex/tests/receipts.pickle: (df_size)
- 03-Maths/01-Algebra-Calculus/02-real-estate-advanced-estimator/tests/flats.pickle: (shape)
- 03-Maths/01-Algebra-Calculus/03-Scipy/tests/minimize2d.pickle: (X0_shape, minimum_shape)
- 03-Maths/01-Algebra-Calculus/03-Scipy/tests/minimize_constraints.pickle: (bounds)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/get_wait_time.pickle: (shape)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/number_products.pickle: (shape)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/number_sellers.pickle: (shape)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/price.pickle: (shape)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/review_score.pickle: (shape)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/training.pickle: (shape)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/wait_time.pickle: (shape)
- 04-Decision-Science/03-Linear-Regression/02-Sellers/tests/seller.pickle: (shape)
- 04-Decision-Science/03-Linear-Regression/Optional-Products/tests/products.pickle: (shape)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/blobs.pickle: (shape)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/image_shape.pickle: (img_shape)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/imageshape.pickle: (img_shape)
- 05-ML/06-Unsupervised-Learning/02-Face-Recognition/tests/projection.pickle: (shape)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/datasets.pickle: (movies_shape, tags_shape, ratings_shape)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/metadata.pickle: (counter_shape, latent_shape)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/ratings.pickle: (latent_shape)
- 05-ML/07-Ensemble-Methods/01-Houses-Kaggle-Competition/tests/preproc_baseline.pickle: (shape)
- 05-ML/07-Ensemble-Methods/01-Houses-Kaggle-Competition/tests/submission_baseline.pickle: (submission_shape)
- 05-ML/08-Workflow/02-Custom-Transformer/tests/pipe.pickle: (shape)

Pickles containing int64:
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/full_gas.pickle: (index_year, us_total)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/merged_dataframes.pickle: (yearly_oil_2009)
- 02-Data-Toolkit/01-Data-Analysis/03-US-Oil-and-Gas-Production/tests/oil.pickle: (filtered_oil_index_year, us_total)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games.pickle: (top_country_1, top_country_10)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games_event.pickle: (top_country_1_event, top_country_10_event)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games_season.pickle: (top_country_1_summer, top_country_10_winter)
- 02-Data-Toolkit/02-Data-Sourcing/02-Scraping/tests/books.pickle: (rating)
- 04-Decision-Science/01-Project-Setup/03-Exploratory-Analysis/tests/exploratory.pickle: (n)
- 05-ML/01-Fundamentals-of-Machine-Learning/04-Logistic-Regression/tests/predictions.pickle: (prediction)
- 05-ML/03-Performance-metrics/02-Electrocardiograms/tests/class_balance.pickle: (healthy, at_risk)
- 05-ML/04-Under-the-hood/02-Solvers/tests/new_data_prediction.pickle: (predicted_class)
- 05-ML/05-Model-Tuning/01-Workflow/tests/knn.pickle: (best_k)

Pickles containing set:
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/all_df.pickle: (all_df_columns)

Pickles containing int:
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/inner_merge.pickle: (inner_merged_nulls)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/left_merge.pickle: (left_merged_nulls)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/outer_merge.pickle: (outer_merged_nulls)
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/right_merge.pickle: (right_merged_nulls)
- 03-Maths/02-Statistics-Probabilities/01-Law-of-Large-Numbers/tests/expected_value_coins.pickle: (expected_value_coins)
- 03-Maths/02-Statistics-Probabilities/02-Random-Variables/tests/factorial.pickle: (count_total_possibilities_10)
- 03-Maths/02-Statistics-Probabilities/02-Toss-a-Coin/tests/factorial.pickle: (count_total_possibilities_10)
- 03-Maths/02-Statistics-Probabilities/03-Central-Limit-Theorem/tests/probability.pickle: (n)
- 04-Decision-Science/01-Project-Setup/02-Data-Preparation/tests/get_data.pickle: (keys_len)
- 04-Decision-Science/01-Project-Setup/Optional-Metric-Design/tests/orders.pickle: (keys_len, reviews_number)
- 04-Decision-Science/03-Linear-Regression/Optional-Products/tests/products.pickle: (avg_review_score, avg_price, avg_quantity)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/variables.pickle: (variable_y)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/duplicates.pickle: (duplicates)
- 05-ML/02-Prepare-the-dataset/02-Feature-Selection/tests/correlation.pickle: (correlated_features)
- 05-ML/03-Performance-metrics/01-KNN/tests/optimal_k.pickle: (optimal_k)
- 05-ML/05-Model-Tuning/03-SVM/tests/generalization.pickle: (number_misclassified_test)
- 05-ML/05-Model-Tuning/03-SVM/tests/rbf_svm.pickle: (best_c, best_gamma)
- 05-ML/05-Model-Tuning/03-SVM/tests/svm_rbf.pickle: (best_c, best_gamma)
- 05-ML/06-Unsupervised-Learning/01-Image-Compression/tests/color_count.pickle: (color_count)
- 05-ML/06-Unsupervised-Learning/02-Face-Recognition/tests/classification.pickle: (best_pc)
- 05-ML/06-Unsupervised-Learning/02-Face-Recognition/tests/components.pickle: (min_pc)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/feature_engineering.pickle: (merged_df_rows)
- 05-ML/07-Ensemble-Methods/01-Houses-Kaggle-Competition/tests/features_overview.pickle: (n)
- 05-ML/08-Workflow/01-Preprocessor-Tuning/tests/solution.pickle: (n_best)
- 06-Deep-Learning/02-Optimizer-loss-and-fitting/04-Credit-Card-Challenge/tests/solution.pickle: (fraud_number, non_fraud_number)

Pickles containing float64:
- 02-Data-Toolkit/01-Data-Analysis/04-Multiple-Files-With-Pandas/tests/olympic_games_combined.pickle: (top_combined_1_event, top_combined_1_medal, top_combined_10_event, top_combined_10_medal)
- 02-Data-Toolkit/02-Data-Sourcing/02-Scraping/tests/books.pickle: (price)
- 03-Maths/01-Algebra-Calculus/02-real-estate-advanced-estimator/tests/univariate.pickle: (mse, theta1, theta0)
- 03-Maths/02-Statistics-Probabilities/03-Central-Limit-Theorem/tests/probability.pickle: (sigma_expected, proba)
- 03-Maths/02-Statistics-Probabilities/03-Central-Limit-Theorem/tests/zscore.pickle: (z, proba)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/cv_results.pickle: (min_score, max_score, mean_score)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/prediction.pickle: (prediction)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/trained_model.pickle: (intercept)
- 05-ML/01-Fundamentals-of-Machine-Learning/02-Learning-Curves/tests/added_features.pickle: (score)
- 05-ML/01-Fundamentals-of-Machine-Learning/02-Learning-Curves/tests/cv_score.pickle: (score)
- 05-ML/01-Fundamentals-of-Machine-Learning/02-Learning-Curves/tests/prediction.pickle: (prediction)
- 05-ML/01-Fundamentals-of-Machine-Learning/02-Learning-Curves/tests/reduced_dataset.pickle: (score)
- 05-ML/01-Fundamentals-of-Machine-Learning/03-Problems/tests/problem_1.pickle: (answer)
- 05-ML/01-Fundamentals-of-Machine-Learning/03-Problems/tests/problem_2.pickle: (answer)
- 05-ML/01-Fundamentals-of-Machine-Learning/04-Logistic-Regression/tests/accuracy.pickle: (accuracy)
- 05-ML/01-Fundamentals-of-Machine-Learning/04-Logistic-Regression/tests/predictions.pickle: (probability)
- 05-ML/01-Fundamentals-of-Machine-Learning/04-Logistic-Regression/tests/reduced_accuracy.pickle: (accuracy)
- 05-ML/02-Prepare-the-dataset/02-Feature-Selection/tests/base_model.pickle: (score)
- 05-ML/02-Prepare-the-dataset/02-Feature-Selection/tests/reduced_complexity_model.pickle: (model_score)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/base_model.pickle: (score)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/strong_model.pickle: (score)
- 05-ML/03-Performance-metrics/01-KNN/tests/default_score.pickle: (score)
- 05-ML/03-Performance-metrics/01-KNN/tests/price_error.pickle: (error)
- 05-ML/03-Performance-metrics/01-KNN/tests/scale_sensitivity.pickle: (base_score, rescaled_score)
- 05-ML/03-Performance-metrics/02-Electrocardiograms/tests/logistic_regression_evaluation.pickle: (accuracy, recall, precision, f1)
- 05-ML/03-Performance-metrics/03-Threshold-Adjustments/tests/base_precision.pickle: (score)
- 05-ML/04-Under-the-hood/01-Loss-Functions/tests/loss_functions.pickle: (r2, r2_mae, max_error, max_error_mae)
- 05-ML/04-Under-the-hood/02-Solvers/tests/new_data_prediction.pickle: (predicted_proba_of_class)
- 05-ML/04-Under-the-hood/03-Batch-Gradient-Descent/tests/descent.pickle: (a_100, b_100)
- 05-ML/05-Model-Tuning/01-Workflow/tests/knn.pickle: (best_score)
- 05-ML/05-Model-Tuning/01-Workflow/tests/r2.pickle: (r2_test)
- 05-ML/05-Model-Tuning/03-SVM/tests/sigmoid_svm.pickle: (sigmoid_svm_cv_accuracy)
- 05-ML/07-Ensemble-Methods/01-Houses-Kaggle-Competition/tests/submission_baseline.pickle: (score_baseline)
- 05-ML/08-Workflow/01-Preprocessor-Tuning/tests/solution.pickle: (cv_score)

Pickles containing Index:
- 02-Data-Toolkit/02-Data-Sourcing/00-Warmup/tests/warmup.pickle: (df_columns)
- 02-Data-Toolkit/02-Data-Sourcing/01-Stock-Market-API/tests/apple.pickle: (columns)
- 02-Data-Toolkit/02-Data-Sourcing/02-Scraping/tests/books.pickle: (columns)
- 03-Maths/01-Algebra-Calculus/02-real-estate-advanced-estimator/tests/flats.pickle: (columns)

Pickles containing dict:
- 02-Data-Toolkit/02-Data-Sourcing/02-Scraping/tests/books.pickle: (books_dict)
- 02-Data-Toolkit/02-Data-Sourcing/03-Text-Extraction-with-Regex/tests/receipts.pickle: (receipts)
- 03-Maths/02-Statistics-Probabilities/02-Random-Variables/tests/factorial.pickle: (probability_1, probability_100)
- 03-Maths/02-Statistics-Probabilities/02-Toss-a-Coin/tests/factorial.pickle: (probability_1, probability_100)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/cv_results.pickle: (cv_result)

Pickles containing list:
- 02-Data-Toolkit/02-Data-Sourcing/03-Text-Extraction-with-Regex/tests/receipts.pickle: (raw)
- 03-Maths/01-Algebra-Calculus/03-Scipy/tests/global_optimization.pickle: (bounds)
- 04-Decision-Science/01-Project-Setup/02-Data-Preparation/tests/get_data.pickle: (keys, columns)
- 04-Decision-Science/01-Project-Setup/Optional-Metric-Design/tests/orders.pickle: (key_names)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/training.pickle: (columns)
- 04-Decision-Science/04-Logistic-Regression/01-Logit/tests/logit.pickle: (answers)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/encoding.pickle: (new_features)
- 05-ML/05-Model-Tuning/02-Regularization/tests/lasso.pickle: (zero_impact_features)
- 05-ML/05-Model-Tuning/02-Regularization/tests/ridge.pickle: (top_2)
- 05-ML/05-Model-Tuning/02-Regularization/tests/unregularized.pickle: (top_1_feature)
- 05-ML/05-Model-Tuning/03-SVM/tests/poly_svm.pickle: (poly_svm_performance)
- 05-ML/07-Ensemble-Methods/01-Houses-Kaggle-Competition/tests/submission_baseline.pickle: (submission_columns)

Pickles containing Series:
- 02-Data-Toolkit/02-Data-Sourcing/03-Text-Extraction-with-Regex/tests/receipts.pickle: (receipt)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/datasets.pickle: (genres_cleaned)

Pickles containing float:
- 03-Maths/02-Statistics-Probabilities/01-Law-of-Large-Numbers/tests/expected_value_dice.pickle: (expected_value)
- 03-Maths/02-Statistics-Probabilities/02-Random-Variables/tests/factorial.pickle: (count_possibilities_11, count_possibilities_43)
- 03-Maths/02-Statistics-Probabilities/02-Toss-a-Coin/tests/factorial.pickle: (count_possibilities_11, count_possibilities_43)
- 03-Maths/02-Statistics-Probabilities/03-Central-Limit-Theorem/tests/distribution.pickle: (mu, sigma)
- 03-Maths/02-Statistics-Probabilities/03-Central-Limit-Theorem/tests/probability.pickle: (mu_expected, mu, sigma)
- 04-Decision-Science/02-Statistical-Inference/01-Orders/tests/distance.pickle: (mean)
- 04-Decision-Science/03-Linear-Regression/02-Sellers/tests/seller.pickle: (median)
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/variables.pickle: (variable_X)
- 05-ML/03-Performance-metrics/02-Electrocardiograms/tests/precision.pickle: (precision)
- 05-ML/03-Performance-metrics/03-Threshold-Adjustments/tests/decision_threshold.pickle: (threshold)
- 05-ML/05-Model-Tuning/03-SVM/tests/linear_svm.pickle: (linear_svm_score)
- 06-Deep-Learning/01-Fundamentals-of-Deep-Learning/02-Your-first-Neural-Network/tests/deeper_model.pickle: (accuracy)
- 06-Deep-Learning/01-Fundamentals-of-Deep-Learning/02-Your-first-Neural-Network/tests/first model.pickle: (accuracy)
- 06-Deep-Learning/01-Fundamentals-of-Deep-Learning/02-Your-first-Neural-Network/tests/first_model.pickle: (accuracy)
- 06-Deep-Learning/01-Fundamentals-of-Deep-Learning/03-Multiclass-classification/tests/baseline.pickle: (accuracy)
- 06-Deep-Learning/02-Optimizer-loss-and-fitting/03-Finetune-your-Neural-Network/tests/solution.pickle: (mae_test)
- 06-Deep-Learning/02-Optimizer-loss-and-fitting/04-Credit-Card-Challenge/tests/solution.pickle: (precision, recall)

Pickles containing LinearRegression:
- 05-ML/01-Fundamentals-of-Machine-Learning/01-Linear-Regression/tests/trained_model.pickle: (model)

Pickles containing DataFrame:
- 05-ML/01-Fundamentals-of-Machine-Learning/02-Learning-Curves/tests/reduced_dataset.pickle: (dataset)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/duplicates.pickle: (dataset)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/encoding.pickle: (dataset)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/missing_values.pickle: (dataset)
- 05-ML/02-Prepare-the-dataset/01-Preprocessing-Workflow/tests/scaling.pickle: (dataset)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/collinearity.pickle: (dataset)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/encoding.pickle: (dataset)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/missing_values.pickle: (dataset)
- 05-ML/02-Prepare-the-dataset/03-Car-Prices/tests/scaling.pickle: (dataset)
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/feature_engineering.pickle: (metadata)

Pickles containing bool_:
- 05-ML/06-Unsupervised-Learning/03-Movie-Recommendation/tests/feature_engineering.pickle: (unique_movies)

Pickles containing RandomizedSearchCV:
- 05-ML/08-Workflow/03-Tuning-Pipeline/tests/solution.pickle: (search)

@krokrob submits that it might be convenient for the content writers to test the pickled results using data science dedicated data types (pandas)
with this in mind, a possible suggested approach is to add steps around the data serialisation :

encode from package specific (pandas) data types to python standard types when saving the result
if we want to be extra safe, replace pickle with json for storage/transmission (need to evaluate the impact)
decode when loading the result

if we allow the content writers to use pandas data types (ndarray, series, index, dataframe), all the existing data-solutions pickles will be covered except from the ones storing a LinearRegression or a RandomizedSearchCV, which we should update

On year later, what's the appetite on that one @krokrob @gmanchon ?

Oh I'm sure this is not a priority, let's close this 👌


Change Detection - Table - It doesn't detect changes when I use reference.

Reproduction link
https://stackblitz.com/edit/ng-zorro-antd-ivy-jas8lq
Steps to reproduce
As mentioned in the NgZorro documentation, when I create rows with "let item of itemTable.data" as in Table 1, it does not detect data updates.
If I create "let item of data.items" as in Table 2, it detects the data change. You can check the example on stackblitz.
What is expected?
I expect it to detect the data update when I use "let item of itemTable.data".
What is actually happening?
When I create rows using "let item of itemTable.data" it does not detect data updates.

Environment
Info

ng-zorro-antd
12.1.1

Browser
Chome 97

You should always replace this.data.items instead of change this.data.items
Because NG-ZORRO changeDetection is onPush
this.data.items = this.data.items.map(item => {
    if (item.name === this.selectedElement.name) {
      return {
        ...item,
        ...this.selectedElement,
      }
    }
    return item;
});

You should always replace this.data.items instead of change this.data.items
Because the changeDetection in NG-ZORRO is ChangeDetectionStrategy.OnPush,  read docs ChangeDetectorRef get more info
this.data.items = this.data.items.map((item, index) => {
    if (index === elementIndex) {
      return {
        ...item,
        ...this.selectedElement,
      }
    }
    return item;
});


[DM-29799] Create nublado-users depending on nublado2 or nublado

Otherwise if we disable nublado, we'll stop creating this.

Okay looks like the minikube deploy went well enough, so let's try this out on data-dev


Building bare metal ubuntu images does not work anymore

What happened: Due to the lack of CSI solution compatibility in Bottlerocket, we are building our own Ubuntu images following the procedure described in the documentation.
We have successfully built many images with the same procedure in the past (last successful build was for the 1-25 release channel on the 12th of April).
But now, the build procedure fails to build an ubuntu image with the following error message during ansible run (tested both for release-channel 1-25 and for 1-26 and both are failing):
    qemu: TASK [containerd : download containerd] ****************************************
    qemu: fatal: [default]: FAILED! => {"changed": false, "checksum_dest": null, "checksum_src": null, "dest": "/tmp/containerd.tar.gz", "elapsed": 0, "msg": "The checksum format is invalid", "url": "null"}
    qemu:
    qemu: PLAY RECAP *********************************************************************
    qemu: default                    : ok=21   changed=16   unreachable=0    failed=1    skipped=182  rescued=0    ignored=0
    qemu:
==> qemu: Provisioning step had errors: Running the cleanup provisioner, if present...
==> qemu: Deleting output directory...
Build 'qemu' errored after 13 minutes 3 seconds: Error executing Ansible: Non-zero exit status: exit status 2

Full output is here:
https://gist.github.com/Cajga/80e27cc65f4067d0dca2c116cad24104
What you expected to happen:
Image builder procedure works.
How to reproduce it (as minimally and precisely as possible):
Follow the documentation to build a bare metal ubuntu image.
Environment:

EKS Anywhere Release: 0.15.2
EKS Distro Release: 1.26
image builder version:

image-builder@eks-a-image-builder:~$ image-builder version
0.1.2

Thanks for opening this issue! We recently made a change to the source of containerd tarball included in the image builds. Earlier we were depending on GitHub releases but now we're building our own containerd and baking it into the images. The change to incorporate this change went into the main branch but not the release-0.15 branch, so the containerd URL is getting set to null. We will fix this.

@Cajga we too are affected by this. We require NFS driver which did not work in bottlerocket. Looks like bottlerocket now supports CSI driver. I validated that NFS CSI driver does work in EKSA with bottlerocket images now.

@Cajga @dashkan could you try running the image build again? We put in a temp fix at aws/eks-anywhere-build-tooling#2134.

@abhay-krishna I was able to build ubuntu 1.26 image. Thank you for the quick fix...

@abhay-krishna I can also confirm that the build no longer fails for ubuntu image for the 1-26 channel. Thanks for your support.