bash-completion: can't be unpacked/built on macOS because of filenames

Describe the bug
As seen in https://github.com/NixOS/nixpkgs/pull/107736 and previously @nima2007 's https://github.com/NixOS/nixpkgs/pull/107511 , the derivation tries to download a git repo that can't be unpacked on macOS because of the filenames in it. At least on APFS.
If you just try to build it, it leads to a hash mismatch:
trying https://github.com/scop/bash-completion/archive/2.11.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   126  100   126    0     0    281      0 --:--:-- --:--:-- --:--:--   281
100  349k    0  349k    0     0   258k      0 --:--:--  0:00:01 --:--:-- -5575
unpacking source archive /private/var/folders/1c/pv33w80d0_n7w14hwrj92p30000xbj/T/nix-build-source.drv-0/2.11.tar.gz
hash mismatch in fixed-output derivation '/opt/facebook/nix/store/mcll1q71rick2kpk0pldabmfby6c9jr6-source':
  wanted: sha256:0m3brd5jx7w07h8vxvvcmbyrlnadrx6hra3cvx6grzv6rin89liv
  got:    sha256:1kyj62wp4sd8629r1gx5ggnkkjfkwvggj98d6cwlx14y3iri0l87

To Reproduce
Either: try to manually built bash-completion on macOS without using binary caches
Or run "nix-prefetch-url --unpack https://github.com/scop/bash-completion/archive/2.11.zip"
On macOS the prefetch looks like this:
mseeger@mseeger-mbp ~ % nix-prefetch-url --unpack https://github.com/scop/bash-completion/archive/2.11.zip
unpacking...
[0.8 MiB DL]checkdir error:  cannot create /private/tmp/nix-48190-0/unpacked/bash-completion-2.11/test/fixtures/_filedir/a+?
                 Illegal byte sequence
                 unable to process bash-completion-2.11/test/fixtures/_filedir/a+?/.
checkdir error:  cannot create /private/tmp/nix-48190-0/unpacked/bash-completion-2.11/test/fixtures/_filedir/a+?
                 Illegal byte sequence
                 unable to process bash-completion-2.11/test/fixtures/_filedir/a+?/g.
error:  cannot create /private/tmp/nix-48190-0/unpacked/bash-completion-2.11/test/fixtures/_get_cword/+?+?+?+?-?
        Illegal byte sequence
replace /private/tmp/nix-48190-0/unpacked/bash-completion-2.11/test/fixtures/evince/.DVI.bz2? [y]es, [n]o, [A]ll, [N]one, [r]ename:

On Linux it just works:
marc@thesilence:~> nix-prefetch-url --unpack https://github.com/scop/bash-completion/archive/2.11.zip
unpacking...
[0.8 MiB DL]
path is '/nix/store/ikhzx6piqwl5q1nrnk0drj8nlk46rpnr-2.11.zip'
0m3brd5jx7w07h8vxvvcmbyrlnadrx6hra3cvx6grzv6rin89liv

Expected behavior
Package can be built.
Additional context
APFS as the root filesystem, /nix is on a separate APFS filessytem
Notify maintainers
@peti @xfix
Metadata
these paths will be fetched (0.01 MiB download, 0.03 MiB unpacked):
  /nix/store/3i2rfbarnms7jrc6c38s2xc0lbbq1jc2-nix-info
  /nix/store/n31amcwkbk55q737qpsc1dgjw5gkjb8m-DarwinTools-1
copying path '/nix/store/n31amcwkbk55q737qpsc1dgjw5gkjb8m-DarwinTools-1' from 'https://cache.nixos.org'...
copying path '/nix/store/3i2rfbarnms7jrc6c38s2xc0lbbq1jc2-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-darwin"`
 - host os: `Darwin 19.6.0, macOS 10.15.7`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.3.9`
 - channels(mseeger): `"nixpkgs-21.03pre255633.c00959877fb"`
 - nixpkgs: `/Users/mseeger/.nix-defexpr/channels/nixpkgs`

There are two different issues.

The zip contains weird directory names, which are probably not allowed on APFS:

test/fixtures/_filedir/a$b
test/fixtures/_filedir/a&b
test/fixtures/_filedir/a'b
test/fixtures/_get_cword/ääää§
...

It also contains filenames which collide when the case is ignored:

test/fixtures/.DVI.BZ2
test/fixtures/.DVI.bz2
test/fixtures/.DVI.GZ
test/fixtures/.DVI.gz
...

I see a segfault when trying this
$ nix-prefetch-url --unpack https://github.com/scop/bash-completion/archive/2.11.zip
unpacking...
warning: Pathname cannot be converted from UTF-8 to current locale.
[0.8 MiB DL]Segmentation fault: 11

Here's the most-relevant looking fraction of the crash report:
Process:               nix [34669]
Path:                  /Volumes/VOLUME/*/nix
Identifier:            nix
Version:               ???
Code Type:             X86-64 (Native)
Parent Process:        bash [60130]
Responsible:           Terminal [59886]
User ID:               501

Date/Time:             2020-12-27 17:48:58.220 -0600
OS Version:            Mac OS X 10.15.7 (19H2)
Report Version:        12
Bridge OS Version:     4.6 (17P6610)
...
System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [34669]

VM Regions Near 0:
-->
    __TEXT<PHONE_NUMBER>89e000-0000000109a4f000 [ 1732K] r-x/r-x SM=COW  /Volumes/VOLUME/*

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_platform.dylib      	0x00007fff6ea67e52 _platform_strlen + 18
1   libc++.1.0.dylib              	0x000000010a635635 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::append(char const*) + 21
2   libnixutil.dylib              	0x000000010a503c82 nix::extract_archive(nix::TarArchive&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 530
3   libnixutil.dylib              	0x000000010a503e8f nix::unpackTarfile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 159
4   nix-prefetch-url              	0x000000010990dc26 main_nix_prefetch_url(int, char**) + 5142
5   nix-prefetch-url              	0x0000000109996c11 nix::mainWrapped(int, char**) + 833
6   libnixmain.dylib              	0x000000010a08c6b9 nix::handleExceptions(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<void ()>) + 425
7   nix-prefetch-url              	0x0000000109999b7f main + 223
8   libdyld.dylib                 	0x00007fff6e871cc9 start + 1

APFS accepts only valid UTF-8 encoded filenames for creation

https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html
[root@Petunia:/tmp]# zfs create -o mountpoint=legacy -o utf8only=on tank/local/junk-nixpkgs-107768

[root@Petunia:/tmp]# mount -t zfs tank/local/junk-nixpkgs-107768 /tmp/nixpkgs-107768/

[root@Petunia:/tmp]# cd nixpkgs-107768/

[root@Petunia:/tmp/nixpkgs-107768]# curl -L https://github.com/scop/bash-completion/archive/2.11.zip > 2.11.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --100   123  100   123    0     0    976      0 --:--:-- --:--:-- --:--:--   984
  0     0    0     0    0     0      0      0 --:--:-- --100  794k    0  794k    0     0  1316k      0 --:--:-- --:--:-- --:--:-- 3408k

[root@Petunia:/tmp/nixpkgs-107768]# sha256sum ./2.11.zip 5094faa20a2dc2dd36ffc4ed22a75020dbfbe6488319ef3dbd1e1afd0c932594  ./2.11.zip

[root@Petunia:/tmp/nixpkgs-107768]# nix-shell -p unzip

[nix-shell:/tmp/nixpkgs-107768]# unzip ./2.11.zip
Archive:  ./2.11.zip
b12639a6becec13a0a2c06173ba40fb3bbe972e1

[...snip...]

 extracting: bash-completion-2.11/test/fixtures/_get_cword/?????

Oddly, I can clone the repo itself:
mseeger@mseeger-mbp Downloads % git clone https://github.com/scop/bash-completion.git
Cloning into 'bash-completion'...
remote: Enumerating objects: 75, done.
remote: Counting objects: 100% (75/75), done.
remote: Compressing objects: 100% (65/65), done.
remote: Total 36534 (delta 25), reused 19 (delta 9), pack-reused 36459
Receiving objects: 100% (36534/36534), 5.59 MiB | 6.85 MiB/s, done.
Resolving deltas: 100% (27787/27787), done.

I wonder if there's some sort of zip issue?

ääää§  sea  seb  sec

perhaps Git's codeload service doesn't handle this weird character business correctly, and creates an invalid .zip, whereas git is faithfully handling the names?

Huh:
Not happy:
curl -L https://github.com/scop/bash-completion/archive/2.11.zip > 2.11.zip && unzip 2.11.zip

Happy:
curl -L https://github.com/scop/bash-completion/archive/2.11.tar.gz > 2.11.tar.gz && tar zfxv 2.11.tar.gz

What I'm a bit confused about:
This should use fetchFromGitHub:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/shells/bash/bash-completion/default.nix#L14
Which does seem to want to use the tar.gz:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchgithub/default.nix#L31
Maybe the tarball silently drops some files (e.g. the multiple colliding ones) and that's why we end up with a bad output hash

It's worth noting that neither ZIP or TAR are encoding aware in terms of their file names - they operate on byte level.
If I take a look at TAR archive, the file name is encoded as follows.
00170c00: 6261 7368 2d63 6f6d 706c 6574 696f 6e2d  bash-completion-
00170c10: 322e 3131 2f74 6573 742f 6669 7874 7572  2.11/test/fixtur
00170c20: 6573 2f5f 6765 745f 6377 6f72 642f c3a4  es/_get_cword/..
00170c30: c3a4 c3a4 c3a4 c2a7                      ........

This is a valid encoding for äääç§ in UTF-8.
On Linux, file names are just bytes, there is no particular encoding necessary, so the bytes are directly copied as is, and everything works.
Meanwhile, APFS uses UTF-8 for file names, so there is a possibility that tar is trying to interpret file names as some encoding that isn't UTF-8.

I tried checking it out via git instead of having an inbetween file format:
  src = fetchgit {
    url    = "https://github.com/scop/bash-completion.git";
    rev    = "refs/tags/" + version;
    sha256 = "1kyj62wp4sd8629r1gx5ggnkkjfkwvggj98d6cwlx14y3iri0l87";
  };

Hilariously, I ended up with 1kyj62wp4sd8629r1gx5ggnkkjfkwvggj98d6cwlx14y3iri0l87 as a hash, just as seen in https://github.com/NixOS/nixpkgs/pull/107736 and https://github.com/NixOS/nixpkgs/pull/107511

Also, maybe /cc @scop since it's their project and they might have a hunch :)

Ok, so it seems that we can get around it by using fetchurl. Let me know what you think :)

Also, maybe /cc @scop since it's their project and they might have a hunch :)

No clue, sorry. But I see you're not running tests, so if the unpacking routines you're using support it, excluding the problematic stuff could be an option. Something like tar --exclude="*/test/fixtures/*" -xf ../bash-completion-2.11.tar.xz in pure GNU tar terms.
I would be interested in getting this fixed, well worked around I think, at the source though in case there's someone to help out with it. We already avoid placing some files in git/tarball and create them on the fly where they're no problem, see e.g. https://github.com/scop/bash-completion/blob/fa21b5cecc0ef5885f9eb41abbefcee2b59b258c/test/t/unit/test_unit_filedir.py#L34 (don't remember offhand if there are more).
Also, mostly unrelated to the issue at hand, having peeked into the build recipe in question here:
https://github.com/rb2k/nixpkgs/blob/cc474f0ed49e1e455d0682868eb22a892358361a/pkgs/shells/bash/bash-completion/default.nix#L39
...I'd like to understand why it's a problem to "touch network", what's "impure logic" in the ls test case, and see if we can either make the chsh and screen cases work or xfail/skip gracefully. I think that's better done in a bunch of issues over at bash-completion rather than here, but then again I don't mind discussing here if others prefer that.

Also, mostly unrelated to the issue at hand, having peeked into the build recipe in question here: https://github.com/rb2k/nixpkgs/blob/cc474f0ed49e1e455d0682868eb22a892358361a/pkgs/shells/bash/bash-completion/default.nix#L39
...I'd like to understand why it's a problem to "touch network",

Narrowly, because the build sandbox blocks network access (among other restrictions), and the tests are running in the build sandbox. (This may not be satisfying if you mean it broadly/conceptually... :)

what's "impure logic" in the ls test case,

It looks like @jonringer disabled that case in #66599--not sure if he'll remember why.

I don't remember that's from 14 months ago. But most likely it was trying to do something like pull down files from the web, or write to something it wasn't supposed to.

generally, the "quick and dirty" way to fix this is to just delete all file paths which contain unicode, and then the hash should be able to be shared.
Tests within nixpkgs are usually valuable because they demonstrate that the package can work. But nixpkgs doesn't have to be as thorough as upsteam, so having to reduce some code coverage to support many platforms I think is a decent trade-off.

1. The zip contains weird directory names, which are probably not allowed on APFS:

They are allowed, but it will cause the sha to be different.

@scop
In short, NixOS is weird, we build in a sandbox that is weird, and we know is weird, usually sandbox issues aren't relevant to upstream, so they aren't reported, but if you are interested in that, here you go.
Here is test output: https://gist.github.com/xfix/4239c96f6a4e81e4d97b3340e092ed57
In summary (failing tests):

TestChsh - attempts to access /etc/shells which doesn't exist in sandbox.

TestEtherWake - attempts to access MAC addresses, sandbox environment only has localhost available which doesn't have a MAC address.

TestIfDown, TestIfup - it's using _configured_interfaces (https://github.com/scop/bash-completion/blob/fa21b5cecc0ef5885f9eb41abbefcee2b59b258c/bash_completion#L1013) and gets nothing because NixOS is not supported by this function. NixOS doesn't provide ifup or ifdown.

TestIfStat - I think that one is my fault, iproute should be added to checkInputs to make ifstat available.

TestIperf, TestIperf3, TestNethogs, TestUnitIpAddresses - I think that one is my fault, I probably should add nettools to checkinputs to make ifconfig available.

TestLs - No user in build sandbox has a valid home.

TestScreen - There are no termcap/terminfo files in build sandbox.

TestService - systemd is not accessible in build sandbox.

TestTime - I don't know why does this fail. This one is weird...

TestUnitExpand, TestUnitExpandTildeByRef - Some sort of weird confusion about home location, I don't know, may look into it later.

1. The zip contains weird directory names, which are probably not allowed on APFS:

They are allowed, but it will cause the sha to be different.

Nevermind, I was probably thinking of HFS+. Disregard anything I say regarding macOS

generally, the "quick and dirty" way to fix this is to just delete all file paths which contain unicode, and then the hash should be able to be shared.

I think the problem is that I haven't found an easy way to do that with the existing fetchfromGithub() call. Having the ability to delete things before it creates the hash would be interesting.
I guess downloading the tarball and comparing the checksum of that also gets around the issue in a similar way.

I think the problem is that I haven't found an easy way to do that with the existing fetchfromGithub() call. Having the ability to delete things before it creates the hash would be interesting.

you can.
I was about to make a PR:
diff --git a/pkgs/shells/bash/bash-completion/default.nix b/pkgs/shells/bash/bash-completion/default.nix
index 5fda5704812..5a1c7da6e48 100644
--- a/pkgs/shells/bash/bash-completion/default.nix
+++ b/pkgs/shells/bash/bash-completion/default.nix
@@ -15,7 +15,12 @@ stdenv.mkDerivation rec {
     owner = "scop";
     repo = "bash-completion";
     rev = version;
-    sha256 = "0m3brd5jx7w07h8vxvvcmbyrlnadrx6hra3cvx6grzv6rin89liv";
+    sha256 = "0kmz33aac846pwymq54714n1g02awnm9crl9svn05zbsmch3j5h7";
+    # fix sha for macOS by removing unicode files
+    extraPostFetch = ''
+      rm -r $out/test/fixtures/_filedir
+      rm $out/test/fixtures/_get_cword/ääää§
+    '';
   };

but the issue is the case insensitive part...

In short, NixOS builds packages in a sandbox to make packages more reproducible, usually build system sandbox issues aren't relevant to upstream, so they aren't reported, but if you are interested in that, I'm willing to provide more details.

Sure. I'm interested in getting whatever feasibly can be in the test suite fixed, or gracefully detecting failures and skipping related tests. By "feasibly" I mean not placing "too much" internal info about the test's or completion's working into the test conditions, and not making changes that will adversely affect the test suite's stability (which has been terrible, but is getting better) or completions on other systems.
I'll file separate upstream bug reports about these so they're better trackable, we have too much here I think.

In short, sandbox blocks network access

That's fine. I actually run one of the "upstream" test matrix combinations with docker's networking disabled to catch affected cases. This setup is invoked with NETWORK=none in the environment and will cause tests known to require networking to be xfailed/skipped gracefully. If you know your setup is like that, I recommend setting that envvar.
Aside, I wonder how much work it would be to add NixOS to our test matrix, or if it's doable in the first place. I know literally nothing about NixOS apart from that it's a Linux system.

and makes sure only derivations that the derivation requires are visible.

The word "derivation" in this context is not familiar to me. Is it related to dependencies?

* `TestChsh` - attempts to access `/etc/shells` which doesn't exist in sandbox.

That I think we should fix upstream, likely by mocking /etc/shells.

* `TestEtherWake` - attempts to access MAC addresses, sandbox environment only has localhost available which doesn't have a MAC address.

Covered by NETWORK=none in env, see your test output.

* `TestIfDown`, `TestIfup` - it's using `_configured_interfaces` (https://github.com/scop/bash-completion/blob/fa21b5cecc0ef5885f9eb41abbefcee2b59b258c/bash_completion#L1013) and gets nothing because NixOS is not supported by this function. NixOS doesn't provide `ifup` or `ifdown` commands.

For these we currently check if we're running in a container and xfail if so, but I suppose we should additionally make them do so with NETWORK=none.
But in addition to getting green from the test suite, I would be interested in adding support for whatever NixOS provides that we could use for _configured_interfaces.

* `TestLs` - No user in build sandbox has a valid home.
* `TestScreen` - There are no termcap/terminfo files in build sandbox.
* `TestService` - systemd is not accessible in build sandbox.

These sound like something we could try and address upstream.

* `TestUnitExpand`, `TestUnitExpandTildeByRef` - Some sort of weird confusion about home location, I don't know, may look into it later. Either way, it's probably an issue on our side.

If your setup changes $HOME of the user running the tests so that it's not in sync with what ~THATUSER expands to, that's something I ran into about a week ago when experimenting with [GitHub actions containers](https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#jobsjob_idcontainer, they set roots $HOME to /github/home while the passwd entry still points to /root and thus ~root expands to that too). Those were addressed with https://github.com/scop/bash-completion/commit/3bed0ddbe0aec8e4840ebe8cb3c78bd7568f3b2d and https://github.com/scop/bash-completion/commit/d420329a3df745a817699da8308b2606b646d5c8

* `TestTime` - I don't know why does this fail. This one is weird...

If the build env does not have any commands starting with set in $PATH or as a function, that'd explain it. TBH I have no idea why we'd use set as the prefix to look for, things like sh, bash or even time sound like something that would be more likely available (although we may want to not use time here as it could make sense to filter time out if completing a command for time, ditto for other similar, but that's something to look for later). Maybe better to use one of our functions as the canary, they're guaranteed to be available, no matter what's in $PATH.

* `TestService` - systemd is not accessible in build sandbox.

We address a related issue in our Alpine Linux test setup by installing a canary package in the test root. I wonder if NixOS has anything besides systemd that could be used to trick it to work, e.g. legacy init scripts in /etc/init.d in any package?
If not, would be nice if we could detect and handle this case gracefully in the test suite.

I'll file separate upstream bug reports about these so they're better trackable, we have too much here I think.

Done now, see references above. I'll refrain from posting more noise unrelated to the original issue here unless asked, and would appreciate further comments about the above mentioned issues to be added to them instead of here. TIA!

To get back to the 'macOS' matter at hand:

I think the problem is that I haven't found an easy way to do that with the existing fetchfromGithub() call. Having the ability to delete things before it creates the hash would be interesting.

you can.
I was about to make a PR:
diff --git a/pkgs/shells/bash/bash-completion/default.nix b/pkgs/shells/bash/bash-completion/default.nix
index 5fda5704812..5a1c7da6e48 100644
--- a/pkgs/shells/bash/bash-completion/default.nix
+++ b/pkgs/shells/bash/bash-completion/default.nix
@@ -15,7 +15,12 @@ stdenv.mkDerivation rec {
     owner = "scop";
     repo = "bash-completion";
     rev = version;
-    sha256 = "0m3brd5jx7w07h8vxvvcmbyrlnadrx6hra3cvx6grzv6rin89liv";
+    sha256 = "0kmz33aac846pwymq54714n1g02awnm9crl9svn05zbsmch3j5h7";
+    # fix sha for macOS by removing unicode files
+    extraPostFetch = ''
+      rm -r $out/test/fixtures/_filedir
+      rm $out/test/fixtures/_get_cword/ääää§
+    '';
   };

but the issue is the case insensitive part...

@jonringer Oh, "extraPostFetch" is neat! I don't think we are running any of the tests as part of the build at the moment.
For now: What do you think is better:

Just rm'ing the whole tests directory via extraPostFetch
Doing what the PR currently does (taking a checksum of the tarball and letting it extract in whatever way it feels like)

I think ultimately it'll do the same thing, so I don't have strong feelings about either :)

We address a related issue in our Alpine Linux test setup by installing a canary package in the test root. I wonder if NixOS has anything besides systemd that could be used to trick it to work, e.g. legacy init scripts in /etc/init.d in any package?

The build environment is minimal. Only explicitly mentioned dependencies will be available. However, we could make a NixOS test which will spin up a complete VM, and can run the tests with systemd or any other service; but this will not be ran as part of the build, but rather after.
The tests just need to verify that it works, doesn't need to be as thorough as upstream's testing which likely includes regression and other extensive testing.

@jonringer Oh, "extraPostFetch" is neat! I don't think we are running any of the tests as part of the build at the moment.
For now: What do you think is better:
1. Just rm'ing the whole tests directory via extraPostFetch

I would only delete what is necessary. Tests are still valuable.

2. Doing what the PR currently does (taking a checksum of the tarball and letting it extract in whatever way it feels like)

I don't think this is an option for mac + hashing.

I don't think this is an option for mac + hashing.

https://github.com/NixOS/nixpkgs/pull/107813 seems to work fine?

#107813 was merged, closing

my comments about testing are kind of mute, as we never ran them.

I stumbled upon the POSIX.1-2001/pax which supposedly standardizes UTF-8 filename handling, and will use the tar-pax automake option to generate such tarballs for future releases (see above commit), hopefully it doesn't break badly elsewhere.
I uploaded bash-completion-2.11-pax.tar.xz which is a local re-roll of the original 2.11.tar.xz to https://github.com/scop/bash-completion/releases/tag/2.11 more or less temporarily for testing, would appreciate of someone could check how the UTF-8 stuff looks in it in setups that have the problem with the original one.

I stumbled upon the POSIX.1-2001/pax which supposedly standardizes UTF-8 filename handling, and will use the tar-pax automake option to generate such tarballs for future releases (see above commit), hopefully it doesn't break badly elsewhere.
I uploaded bash-completion-2.11-pax.tar.xz which is a local re-roll of the original 2.11.tar.xz to https://github.com/scop/bash-completion/releases/tag/2.11 more or less temporarily for testing, would appreciate of someone could check how the UTF-8 stuff looks in it in setups that have the problem with the original one.

I tried to just quickly use builtins.fetchTarball:
  src = builtins.fetchTarball {
    url = "https://github.com/scop/${pname}/releases/download/${version}/${pname}-${version}-pax.tar.xz";
    sha256 = "17608v8z6xqfy46ayjsnfw6pmqx7lii3vnfbx974jl3ri5c445lz";
  };

but I end up with different checksums:
macOS: 17608v8z6xqfy46ayjsnfw6pmqx7lii3vnfbx974jl3ri5c445lz
Linux: 0hpxx410v3pwh70v6gp27nhcs4383lll57qhsp1460y8jjn2q8h1
Presumably because the unpack still runs into issues with case sensitivity and macOS by default not being case sensitive.
That being said, it still packages fine with the current solution.
I guess the only way around that would be to create the fixtures 'on the fly' when running the tests, but that's a pretty big change. I think the workaround (checksum the tarball instead of the unpacked contents) is probably good enough for now.

I tried to just quickly use builtins.fetchTarball:
  src = builtins.fetchTarball {
    url = "https://github.com/scop/${pname}/releases/download/${version}/${pname}-${version}-pax.tar.xz";
    sha256 = "17608v8z6xqfy46ayjsnfw6pmqx7lii3vnfbx974jl3ri5c445lz";
  };

but I end up with different checksums:
macOS: 17608v8z6xqfy46ayjsnfw6pmqx7lii3vnfbx974jl3ri5c445lz
Linux: 0hpxx410v3pwh70v6gp27nhcs4383lll57qhsp1460y8jjn2q8h1
Presumably because the unpack still runs into issues with case sensitivity and macOS by default not being case sensitive.
That being said, it still packages fine with the current solution.
I guess the only way around that would be to create the fixtures 'on the fly' when running the tests, but that's a pretty big change. I think the workaround (checksum the tarball instead of the unpacked contents) is probably good enough for now.

On macOS, both tarballs extract fine (pax and non-pax).
It seems like only the github generated one causes issues.

On macOS, both tarballs extract fine (pax and non-pax).
It seems like only the github generated one causes issues.

Case insensitivity issues are resolved for future bash-completion releases with the above mentioned two commits + a couple of additional fixes to the latter.

Case insensitivity issues are resolved for future bash-completion releases with the above mentioned two commits + a couple of additional fixes to the latter.


BaseRepository::updateRelations GetQualifiedForeignKey renamed in laravel 5.8

Hello there,
According to https://laravel.com/docs/5.8/upgrade, Laravel again changed method name getForeignKey. And it throws error in BaseRepository::updateRelations() method because of it.
The getForeignKey and getQualifiedForeignKey methods of the BelongsTo relationship have been renamed to getForeignKeyName and getQualifiedForeignKeyName respectively, making the method names consistent with the other relationships offered by Laravel.

@xiaohuilam with Laravel 5.8, we have published default base repository where we haven't added code for update relations. so this is not the case. I realized that the function has lots of problems and lots of duplicate issues are created over and over, so had planned to add it sometimes later to BaseRepository to 5.8.

Yeah, sorry i didn't know that before. I just finished reading your blog and it's more clear for me now.
But i have few words for you:
As you stopped using prettus/l5-repository now, you made your new repository backwards incompatible. I just cannot switch old BaseRepository for new BaseRepository in my code.
Also your lightweight repository does not provide caching, criterias etc...
For me it's a big minus and i will be forced to use old BaseRepository for now

Haven't seen this issue when creating PR #614 , so what do you suggest? @mitulgolakiya

@xkovacikm nope. I have still left all the files in a repo that we have with dependency. If you are migrating your code, your existing repositories which are using BaseRepository form package will be still valid.
Also, if you still want to use package while using 5.8 then you can customize your template to use repository class from the backend, so I do not see what's a problem here.


[WIP] implement CRI ListPodSandboxMetrics

implement CRI API ListPodSandboxMetrics
Fixes: #10506
Part of https://github.com/kubernetes/enhancements/issues/2371

/cc @mikebrow

@akhilerm You still working on this?

@akhilerm You still working on this?

Yepp. I am working on it. Couldnt focus for sometime, but will pickup from next week onwards.

/cc @zvonkok

@akhilerm Is this still on your roadmap?

AFAIC metrics are missing if we want to be on par with cAdvisor.

@akhilerm Is this still on your roadmap?

Yes

AFAIC some metrics are missing if we want to be on par with cAdvisor.

The ListMetricDescriptors should have every metrics and this was added over from cadvisor. I will cross check if I missed something while adding from the cadvisor code.


repetitive tutorials

rust tutorials (hello world, simple records, counter) are basically the same, only difference is that you copy a different code snippet and hello world explains the project structure. we should either
a) make distinctive tutorials by creating the dapp step by step
b) remove redundant tutorials and just point people to the examples

Since these examples also exist in dfinity/examples I think it makes sense to remove those

The latter two tutorial pages no longer exist. Closing this issue.


Update index.md

Replace issue name LAB00:QUICK_DESCRIPTION, for example "LAB01: My new issue" (or same name as linked Issue)
Related Issue
Link related Github Issue 🢂 Fixes # . (Include issue number after #)
Checklist
Mark completed with "x" between brackets, "[x]", or checking the box once the PR is created:

[ ] Has related GitHub Issue 💥 Create Issue 📝
[ ] Tested it
[ ] Read the PR collaboration guide 👓 Collaboration Guide 📝

Changes proposed in this pull request:

You should have a better look at that.


bilibili: 1.16.1-1 -> 1.16.1-2

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for bilibili is: Electron-based bilibili desktop client
meta.homepage for bilibili is: https://github.com/msojocs/bilibili-linux
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.16.1-2 in filename of file in /nix/store/1yham8gdmp43lig4a43i7s1g8kq34pf4-bilibili-1.16.1-2

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

bilibili

Instructions to test this update (click to expand)

Either download from the cache:
nix-store -r /nix/store/1yham8gdmp43lig4a43i7s1g8kq34pf4-bilibili-1.16.1-2 \
  --option binary-caches 'https://cache.nixos.org/ https://nixpkgs-update-cache.nix-community.org/' \
  --option trusted-public-keys '
  nixpkgs-update-cache.nix-community.org-1:U8d6wiQecHUPJFSqHN9GSSmNkmdiFW7GW7WNAnHW0SM=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The nixpkgs-update cache is only trusted for this store-path realization.)
For the cached download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A bilibili https://github.com/r-ryantm/nixpkgs/archive/cac0e6eaf4d98efc84fad2435f8386666b286e9f.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/cac0e6eaf4d98efc84fad2435f8386666b286e9f#bilibili

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/1yham8gdmp43lig4a43i7s1g8kq34pf4-bilibili-1.16.1-2
ls -la /nix/store/1yham8gdmp43lig4a43i7s1g8kq34pf4-bilibili-1.16.1-2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review

x86_64-linux

:white_check_mark: 1 package built:

bilibili

Maintainer pings
cc @Jedsek @kashw2 @Bot-wxt1221 for testing.

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.

@NixOS/nixpkgs-merge-bot merge


[Downloads Manager] ...

Extension
https://www.raycast.com/thomas/downloads-manager
Description
Cut latest download.
Copy latest download already exists. However, the copy feature requires returning to the downloads folder to delete the file once it's moved to the correct location.
Who will benefit from this feature?
People who use their downloads folder responsibly (i.e. empty) and move files from the downloads folder to a new destination.
Anything else?
No response

Do you want to make a PR with this new command?

Alternatively we could add a Move... action in the list, similar to how we have it in the File Search. WDYT?

Hi Thomas,
Good to meet! Wish I was able to add the comment myself. I'm a mech-eng and
substack python, self-taught guy.
On Wed, 15 Mar 2023 at 20:42, Thomas Paul Mann @.***>
wrote:

And servus :-P
—
Reply to this email directly, view it on GitHub
https://github.com/raycast/extensions/issues/5248#issuecomment-1470811895,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJMV7CDQXOTLIO7DHIH6JSTW4ISUXANCNFSM6AAAAAAVVDCSTI
.
You are receiving this because you authored the thread.Message ID:
@.***>


[Extensibility Request] Page 5114 "Contact Profile Answers" - CurrentQuestionsChecklistCode (BaseApp_20.1)

Please move variable CurrentQuestionsChecklistCode to protected var
Change:
    var
        ProfileQuestionnaireLine2: Record "Profile Questionnaire Line";
        ProfileQuestLineQuestion: Record "Profile Questionnaire Line";
        ProfileManagement: Codeunit ProfileManagement;
-       CurrentQuestionsChecklistCode: Code[20];
        ContactProfileAnswerCode: Code[20];
        ContactProfileAnswerLine: Integer;
        GoNext: Boolean;
        OK: Boolean;
        CaptionStr: Text;
        RunFormCode: Boolean;
        [InDataSet]
        StyleIsStrong: Boolean;
        [InDataSet]
        DescriptionIndent: Integer;

    protected var
        Cont: Record Contact;
        ContProfileAnswer: Record "Contact Profile Answer";
+       CurrentQuestionsChecklistCode: Code[20];
        Set: Boolean;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 41095.


Allow the user to select object types of interest

Right now it's all or nothing. The user ought be allowed to supply some manner of bitmask, though interfaces can't ever really be disabled...with our current design, anyway.

Note that we can't base this solely on which callbacks are set, since we still would want to support lookups and enumeration-on-demand even for those we don't provide callbacks for. Saying "I'm uninterested" doesn't mean "don't send a discovery request", but rather "don't deal with these at all."

I've implemented this for ifaces now (the only thing we actually cache yet). Need a unit test for this.

Unit tests added:
      Start 12: NoTrack.NoWorkIsInvalid
12/13 Test #12: NoTrack.NoWorkIsInvalid .............   Passed    0.01 sec
      Start 13: NoTrack.IfaceFailsLookup
13/13 Test #13: NoTrack.IfaceFailsLookup ............   Passed    0.01 sec


Update ntfsmounter to 0.4

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} left no offenses.
[x] The commit message includes the cask’s name and version.

Thank you for the contribution. It was merged directly as a7f0c0be47892d60a43f829ffaa772a6b9416c24. Your contribution is still credited to you.


Supported Firmware version

In #10 I see a reported firmware version of <IP_ADDRESS>.18. When I run the poc-ts command (commit 228f3370604763488bed3b6de1f2e033dd9b8a0d) I get the output Firmware Version:<IP_ADDRESS>.
What firmware version do you know that e.g. the allColor function in POC/ts/index.ts is known to work with? The reason I'm asking is that when I run allColor("#0000FF"), the keys do not all change to the same color; instead, it looks like some keys have their brightness changed, a few have the color changed, and such (picture below). So it might be the case that the keyboard firmware does not understand the data sent by the script.
EDIT: I should say that I just received my 5Q keyboard today and I did not connect it to a Windows pc or update the firmware.

I made the following changes to the POC to only run allColor("#0000FF"):
rav@novascotia:~/codes/node-lib$ git diff POC/ts/index.ts
diff --git a/POC/ts/index.ts b/POC/ts/index.ts
index d55a956..2e306ea 100644
--- a/POC/ts/index.ts
+++ b/POC/ts/index.ts
@@ -22,14 +22,14 @@ const hidDevice = keyboard.find();
  */
 keyboard.initialize();

-allColor("#FF0000");
-allColor("#00FF00");
+//allColor("#FF0000");
+//allColor("#00FF00");
 allColor("#0000FF");

 // sparkle();
 const data = keyboard.getKeyboardData();
 console.log("Firmware Version:" + data.firmware);
-keyboard.setBrightness(63);
+//keyboard.setBrightness(63);

 // sparkleBatch();

@Mortal thanks for reporting this! Truthfully, we didn't know what firmware versions we support yet. It's starting to look like there's some big updates in the latest firmware over some of these factory versions. Your version looks to be the earliest version reported so far.
I've added a note in the readme about this here: https://github.com/diefarbe/node-lib/blob/master/README.md#firmware-versions
If you can get to a windows PC, you'll find the latest firmware here: https://www.daskeyboard.io/get-started/download/

@Mortal
There were some issues with the color channel zones in the JSON a little while back. Have you pulled latest and tested?
Mine keyboard was originally displaying the 147.x for the firmware version but it is now displaying the more common <IP_ADDRESS>. That seems to match up with the numbering that DAS has in their versioning.


Test using CIrcleCI to publish to PyPI with 0.4.5rc1

The test is successful, it looks like this:


Generating an output sequeuence using a driving video and a source frame.

Is there a simple way to run the pretrained model to generate an output sequence corresponding to a driving video and a single source frame? The dataset I am working with is Voxceleb itself.

Hi, yes, have you tried following the manual in README.md?

Is there a way to not fine-tune and run it directly? It will be a fair comparison to how I am evaluating my approach.

Yes, just spend 0 iterations fine-tuning.

Got it thanks!


sdk-metrics: forceFlush on shutdown

Which problem is this PR solving?
Metric provider behave differently from traces and logs providers during shutdown process, being the only one that does not flush pending tasks.
Although it's not explicitly required (yet) by the specs for metrics and logs as it is for traces, this PR adds flushing metrics on shutdown, with the intent of eliminating a not infrequent confusion and wrong assumption among users.
This way we consistently ensure the flush for all supported signals upon shutdown.
The same approach has already been taken by java and python SDKs.
Short description of the changes
now MeterProvider#shutdown() calls MeterProvider#forceFlush()
Type of change

[x] New feature (non-breaking change which adds functionality)

Checklist:

[x] Followed the style guidelines of this project
[x] Unit tests have been ~added~ updated

Hi @macno thanks for working on this 🙂
I think the agreement on open-telemetry/opentelemetry-specification#2983 was on the PeriodicExporingMetricReader - not the MeterProvider. Would you mind adapting the code accordingly? 🙂

Hi @pichlermarc thanks for taking time to look at the PR.
I'm not sure if I read the same agreement, to me seems like an open point.
From my understanding, this is the shutdown flow:
graph TD;
    D@{ shape: procs, label: "metricCollector"}
    SDK-- shutdown() ---meterProvider;
    meterProvider-- shutdown() ---D
    D-- shutdown() ---metricReader

so calling forceFlush in the meterProvider will cause all the registered metricReader to execute forceFlush, meaning you don't have to implement the same logic in all the metricReader implementations to achieve the same behaviour.
What's the reason to move it to PeriodicExporingMetricReader ?

[...]
so calling forceFlush in the meterProvider will cause all the registered metricReader to execute forceFlush, meaning you don't have to implement the same logic in all the metricReader implementations to achieve the same behaviour.
What's the reason to move it to PeriodicExporingMetricReader ?

The equivalent of PeriodicExportingMetricReader is BatchSpanProcessor and BatchLogRecordProcessor in the other SDKs and that behavior is specified for their base interfaces. Looking into what the Java and Python SIGs implemented, their MeterProviders don't apply the effects forceFlush() on shutdown(), but the PeriodicExportingMetricReader does it.
While it is true that adding it to MeterProvider would be the easiest, it seems that this intentionally split. Though I don't know what the original reasoning behind that, the spec does not elaborate on it. Anyway, that why I'm not very confident about going against the grain on this.
I asked here again, but I'm fairly certain that my assumption is correct. 🤔
Some links:

Logs SDK Spec for LogRecordProcessor#shutdown()
Trace SDK Spec for SpanProcessor#shutdown()
Java Implementation of  MeterProvider#shutdown() does not include any forceFlush() effects
Python Implementation of MeterProvider#shutdown() does not include forceFlush() effects
Java implementation of PeriodicExportingMetricReader#shutdown() does include collection before shutdown
Python implementation of PeriodicExportingMetricReader#shutdown() does include collection before shutdown

Ok, I will proceed as requested

Looks like there are a few tests that rely on the old behavior, we'll need to remove the forceFlush in those:

https://github.com/open-telemetry/opentelemetry-js/blob/556eecf0f9d6b6728ad46d17cc2d689f496498f5/experimental/packages/opentelemetry-exporter-metrics-otlp-proto/test/OTLPMetricExporter.test.ts#L77
https://github.com/open-telemetry/opentelemetry-js/blob/484af40805e7e0d4e4ab6f2c43f14954ea1f7c3f/experimental/packages/opentelemetry-exporter-metrics-otlp-http/test/node/OTLPMetricExporter.test.ts#L217
https://github.com/open-telemetry/opentelemetry-js/blob/484af40805e7e0d4e4ab6f2c43f14954ea1f7c3f/experimental/packages/opentelemetry-exporter-metrics-otlp-http/test/browser/OTLPMetricExporter.test.ts#L95
https://github.com/open-telemetry/opentelemetry-js/blob/484af40805e7e0d4e4ab6f2c43f14954ea1f7c3f/experimental/packages/opentelemetry-exporter-metrics-otlp-http/test/browser/OTLPMetricExporter.test.ts#L55
https://github.com/open-telemetry/opentelemetry-js/blob/484af40805e7e0d4e4ab6f2c43f14954ea1f7c3f/experimental/packages/opentelemetry-exporter-metrics-otlp-http/test/browser/OTLPMetricExporter.test.ts#L95
https://github.com/open-telemetry/opentelemetry-js/blob/484af40805e7e0d4e4ab6f2c43f14954ea1f7c3f/experimental/packages/opentelemetry-exporter-metrics-otlp-http/test/browser/OTLPMetricExporter.test.ts#L55

Please also add a changelog entry in ./CHANGELOG.md, then we can merge this. 🙂


Add quick doc about connecting to hidden networks

This isn't a cover all document, but it is alright to give someone some pointers about this problem i just hit.

I tweaked a few things just in the editor but I didn't actually make sure the code works, so give it a try.
I'm not clear from these notes what the correct course of action is, though. You mention that if you do the thing in the code block, you're likely to get failures. What should someone do instead?

I guess it would be a good idea to give some sort of way to fix this issue. It's kind of dependent on what you want to do. If you know that the network you configured exists, you would just want to ignore that message, but if it's coming from an external source such as user configured, you will likely need some sort of state to figure out how to handle it.

Anyone mind if i merge this? i actually just hit this again.


Multigroup SEM constructor

I suggest to add a shortcut multiple group SEM constructor that looks like this:
model = MultipleGroupsSem(
    specification = partable
    data = dat,
    by = column
)

which creates an SEMEnsemble with groups corresponding to the number of unique entries in column column in dataset dat.

Thanks for the request, the constructor will be added to the next release.


any plan of spaceflake in rust?

rt.
thanks for great work.

Thanks for your positive feedback. :)
No plans on doing a port to Rust for the following reasons:

I never learned that language and learning a language just to make a port is not the way I enjoy learning a language
As I do not know Rust, I have no projects in Rust where I personally could make use of the library

The first purpose of Spaceflakes was in my personal projects. (Most of them are in Go.) I then decided to make publicly available in case someone could make good use of it, in Go project.


Boilerplate listable FPO data

The typographic specimens are awesome. But I also need this sorta stuff on most (if not all) projects:

List of user info (example)
List of article-like content (title, author, date, excerpt, body)

It'd be pretty great to have some of that included and ready to go, even/especially if it just gets edited to suit project needs. It might also help demonstrate that the data/ folder can incorporate as many files as one might like.

Low priority. Small.


Add support for free-threaded builds

fixes #67

@tacaswell so I tried to apply @minrk's changes and it's all red: https://github.com/hynek/argon2-cffi-bindings/actions/runs/11229954729

On Ubuntu, it tries and fails to build a CFFI wheel: https://github.com/hynek/argon2-cffi-bindings/actions/runs/11229954729/job/31216653852 (missing ffi.h!?). This might be possible to fix using cibuildwheel, but I won't bother until the next issue is fixed.

It seems like cffi is forcing the build into GIL-mode and then our logic to detect free-threading doesn't work and the build fails with "The limited API is not currently supported in the free-threaded build". I have no idea how to work around this?

cc @henryiii

Can check later, but see https://github.com/python-cffi/cffi/pull/104 in the meantime.

Can check tomorrow (ping me if I forget),

Thanks!

but see python-cffi/cffi#104 in the meantime.

That's rather discouraging and implies the py_limited_api is a NOP? :(

https://github.com/python-cffi/cffi/pull/125 Will be in the next version. Maybe try cffi from git?

python-cffi/cffi#125 will be in the next version. Maybe try cffi from git?

At the danger of embarrassing myself, I have no idea how to do that: pin a git rev in build-requires?
And even worse, I haven't found an easy way to get a free-threaded Python on my Mac? For now, I'm extremely excited for https://github.com/indygreg/python-build-standalone/pull/336 to ship.

For reference: uv's issue tracking free-threaded installs: https://github.com/astral-sh/uv/issues/8019

pin a git rev in build-requires?

Yeah, you can do that. Perhaps not appropriate for publication, but you can use a git archive URL as a dependency:
[build-system]
requires = [
    "setuptools>=45",
    "setuptools_scm[toml]>=6.2",
    "wheel",
    "cffi>=1.0.1",
    # or a specific commit instead of HEAD
    "https://github.com/python-cffi/cffi/archive/HEAD.tar.gz; python_version >= '3.13'",
]

as done here.

Due to changes in setuptools (https://github.com/pypa/setuptools/pull/4647) if you extend bdist_wheel from the wheel package the install will fail.  I'm not 100% sure what the support windows of setuptools.wheel is vs the support window that argon2-cff-bindings needs of setuptools so I went with the defensive option.  I'll also open a PR against the nogil branch.
diff --git a/setup.py b/setup.py
index 7510937..3a402e3 100644
--- a/setup.py
+++ b/setup.py
@@ -11,9 +11,12 @@ cmdclass = {}

 if platform.python_implementation() == "CPython":
     try:
-        import wheel.bdist_wheel
+        try:
+            from setuptools.command.bdist_wheel import bdist_wheel
+        except ImportError:
+            from wheel.bdist_wheel import bdist_wheel

-        class BDistWheel(wheel.bdist_wheel.bdist_wheel):
+        class BDistWheel(bdist_wheel):
             def finalize_options(self):
                 # Free-threaded CPython doesn't support limited API.
                 if sysconfig.get_config_var("Py_GIL_DISABLED"):
@@ -21,7 +24,7 @@ if platform.python_implementation() == "CPython":
                 else:
                     self.py_limited_api = f"cp3{sys.version_info[1]}"

-                wheel.bdist_wheel.bdist_wheel.finalize_options(self)
+                super().finalize_options()

         cmdclass["bdist_wheel"] = BDistWheel
     except ImportError:

The floor on setuptools is 70.1, although I think upstream added a workaround so you can probably ignore my last comment.

I'd still recommend using the correct one, the workaround is just to keep from blowing up for now, but wheel wants to get rid of it's API long term (it never had a public API, technically).


⚠ Bofrost.at has degraded performance

In 2a432a7, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1563 ms

Resolved: Bofrost.at performance has improved in da2cdce after 7 minutes.


.str.split(..., expand=True) with Int64Index triggers pandas FutureWarning

Describe the issue:
When using .str.split(..., expand=True) on a ddf (meta) with a non-RangeIndex, a pandas 1.5+ FutureWarning is raised regarding series slicing with integers changing to be label-based.
Minimal Complete Verifiable Example:
[ins] In [1]: import dask.dataframe as dd
         ...: import pandas as pd

[ins] In [2]: df = pd.DataFrame({"a": ["a\nb"]})
         ...: ddf = dd.from_pandas(df, npartitions=1)

[ins] In [3]: df["a"].str.split("\n", n=1, expand=True)
Out[3]:
   0  1
0  a  b

[ins] In [4]: ddf["a"].str.split("\n", n=1, expand=True)
Out[4]:
Dask DataFrame Structure:
                    0       1
npartitions=1
0              object  object
0                 ...     ...
Dask Name: str-split, 3 graph layers

[ins] In [5]:

[ins] In [5]: # a non-RangeIndex seems triggers a warning
         ...: df = pd.DataFrame({"a": ["a\nb"]}, index=[0])
         ...: ddf = dd.from_pandas(df, npartitions=1)

[ins] In [6]: df["a"].str.split("\n", n=1, expand=True)
Out[6]:
   0  1
0  a  b

[ins] In [7]: ddf["a"].str.split("\n", n=1, expand=True)
/Users/jacobhayes/.virtualenvs/dyno_parsing-ba1482f5-py3.10.2/lib/python3.10/site-packages/dask/dataframe/accessor.py:268: FutureWarning: The behavior of `series[i:j]` with an integer-dtype index is deprecated. In a future version, this will be treated as *label-based* indexing, consistent with e.g. `series[i]` lookups. To retain the old behavior, use `series.iloc[i:j]`. To get the future behavior, use `series.loc[i:j]`.
  index=self._series._meta_nonempty[:1].index,
Out[7]:
Dask DataFrame Structure:
                    0       1
npartitions=1
0              object  object
0                 ...     ...
Dask Name: str-split, 3 graph layers

Anything else we need to know?:
Environment:

Dask version: 2022.11.1
Python version: 3.10.2
Operating System: mac
Install method (conda, pip, source): pip

@JacobHayes thanks for the report. I'll take a look at this and your PR (and thanks for that too!)


VS Code briefly freezes when switching to/from any other window, including among VS Code windows on multiple monitors

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.71.2
OS Version: Windows 10 Home, v 21H2 (OS build 19044.2006)

Steps to Reproduce:

Launch VS Code.
Switch focus to any other open application.
Switch focus back to VS Code.
Notice the clock icon on the Explorer link in the sidebar, and the animated light blue underline that moves from left to right (visible beneath the "LO" portion of "EXPLORER" in the screenshot below):

That scan seems to be a blocking process of some sort, that lasts approximately 2 seconds on my machine. The animated underline moves smoothly, but everything else in VS Code is locked for those 2 seconds. Whatever scanning process that animation icon represents seems to freeze everything else in the application.
I've only noticed this issue for the last couple months, but it happens now every single time I switch focus back and forth between windows. And I switch focus frequently, between VS Code and Chrome, and even between multiple VS Code windows itself open on multiple monitors -- switching focus between those windows is also sufficient to trigger the 2-second freeze. Given the frequency with which I switch among application windows while working (often multiple times per minute even), the constant 2-second delay is becoming very frustrating. And again, the freezing behavior is relatively new, with it only attracting my attention the last couple months.
The issue seems to be potentially related to these cases:
https://github.com/microsoft/vscode/issues/141735
https://github.com/microsoft/vscode/issues/123257
But the steps to reproduce aren't the same -- for instance, the maximization state of the browser mentioned in #141735 is entirely unrelated, since I'm able to reproduce the issue even just when switching from one open VS Code window to another VS Code window on another monitor (experiencing the 2-second freeze each switch there and back). Opening "Help -> About" in a single running instance of VS Code is sufficient to trigger the brief freeze when the About dialog closes... reproducing the issue when grabbing the VS Code version number for entry into this bug report.
Case #123257 mentions completely uninstalling VS Code, deleting all artifacts left behind, and reinstalling everything from scratch. Others have indicated that this only provides about a day of relief from the freezing issue, and given the time put into setting up my development environment, it's been a step that I've been reluctant to take. I'm hoping the clue of the clock icon in the "Explorer" link in my screenshot in this issue might provide your team with an alternative path of inquiry, so that a better solution than "delete everything" might be found.

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.72.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

Following the triage bot's advice, I did a quick update to v 1.72.0. I can no longer reproduce the issue on my machine. I'm thoroughly impressed with the response time of your team -- hahaha! Thank you.

I can no longer reproduce the issue on my machine

Nice! Closing as it seems to be fixed.


[Feature]: add pgadmin4 command to cnpg plugin

Is there an existing issue already for this feature request/idea?

[X] I have searched for an existing issue, and could not find anything. I believe this is a new feature request to be evaluated.

What problem is this feature going to solve? Why should it be added?
At the moment, CloudNativePG doesn't have a good story to show how to connect with an application to a given database via network. The only application we show is pgbench. The psql command, indeed, runs on the same container where Postgres runs, and is executed as postgres user (so it is not exactly suitable for demonstration and evaluation).
Describe the solution you'd like
We should add the pgadmin4 command to cnpg plugin to provide an easy way to demonstrate and/or evaluate a deployed Postgres cluster using a web interface. The idea is to tie the deployment to a given cluster, and use only the app user and database to connect to the instance.
IMPORTANT: given that users can deploy pgAdmin4 in production independently, it is important to state in the documentation that this setup is for evaluation only and that security considerations must be taken before going in production.
Describe alternatives you've considered
We have considered running psql via the network in a separate container, that is tied to the kubectl command existence. But such a behaviour has been deprecated by kubectl (run command).
Additional context
N/A
Backport?
Yes
Are you willing to actively contribute to this feature?
Yes
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Not convinced this is a good idea. Adding pgadmin as a part of the plugin will make this look like a recommended way to manage the database. This will open up a large attack surface.
I agree that cnpg does not have a good story around apps connecting to clusters, but I think that is better solved by more docs and a demo app not in the plugin.

We will make sure to explain this is just for evaluation and not for production. Also, the idea is that pgAdmin is tied to the app user and app database, so it can only used to view the database. Let's wait to see the commit first. Happy to seek your input and help on the docs and messaging.

@achanda do you want to look at what we've come up with?

I'm going to merge the plugin command. I think that @achanda's concerns have been addressed in the patch. It's clearly stated that this is not intended for permanent deployment but only as a tool for a quick visual inspection of the database or demonstration purposes.


Update code style checking job to Ubuntu 20.04

Backport #761 onto stable to fix the build, as discovered in #770.
Checklist

[X] Change is covered by automated tests

Interesting failure seen by CircleCI:
[12:12]  secure_comparator_api_test:#12  "secure_comparator_append_secret failed"  FAIL
!    Type:      fail-if
!    Condition: condition
!    File:      tests/themis/themis_secure_comparator.c
!    Line:      222

--> 12 check(s), 11 ok, 1 failed (8.33%)

Not the first time I see it, but it's pretty rare.
I've restarted the CI run.


world::World doctests sometimes hangs with Miri

When running the world::World (line 56) doctest in Miri with many different seeds, it eventually runs into an endless loop. This can be avoided by using -Zmiri-disable-weak-memory-emulation. This probably indicates a bug in bevy or its dependencies, but could also be a bug in Miri.
Weak memory emulation makes it so that an atomic load can return "outdated" values, as is permitted by the concurrent memory model of Rust (and C/C++). So likely, some code somewhere reads an old value, thinks it is current, and goes into some kind of spin loop -- and stays there forever.
See here for further analysis by @cbeuw.

Possibly relevant: https://github.com/kanidm/concread

The fact that the advise to disable Stacked Borrows does not inspire a ton of confidence though. ;)
FWIW, latest Miri has -Zmiri-track-weak-memory-loads, which should help diagnose this issue. But OTOH @cbeuw pretty much figured it out already I think; someone probably needs to go fix smol-rs.

Ah looks like this is tracked in concurrent-queue: https://github.com/smol-rs/concurrent-queue/issues/17

The problem in concurrent-queue that we agreed on is that it was using x86-specific fences that Miri does not understand. This has been fixed in concurrent-queue 1.2.4 (https://github.com/smol-rs/concurrent-queue/commit/d61005fc792bcd9a20ae9f61726b6842211cc10a).

that Miri does not understand

To be more precise, using those fences is just wrong when writing Rust code, and Miri understands that correctly. They can only be used correctly when writing inline assembly.


"Select/Unselect" button for search function in selecting GFs step is not passing selected GFs to analysis

ERROR: No Genomic Features selected/uploaded.

Still does not work


ö changed in v1.3

Hello,
It's kind of strange that you changed the ö to oe in version 1.3 compared to version 1.2 where ö was o.
And nothing in the release notes about this.
Ex v1.2 "Köln" =""Koln"
v1.3 "Köln" = "Koeln"
This kind of messes up with our canonical url.

any news from this?

I just reverted to v1.2

I just accepted this PR: https://github.com/thecoderok/Unidecode.NET/pull/5
Is it wrong?

https://github.com/thecoderok/Unidecode.NET/blob/1.3.0/test/UnidecoderTest.cs
Checked Russian and German in tests only

Well i used the lib to fold the accents in our canonical url. Then from the version 1.3
v1.2 "Köln" =""Koln"
v1.3 "Köln" = "Koeln"
So in terms of SEO it generated a new url since i had
....controller/businessname_koln
and on v1.3 i had
....controller/businessname_koeln
Which was bad for us.
So either i remain to v1.2 or i need to use something else to do the job.

I just accepted this PR: #5
Is it wrong?

For Finnish, the correct (as in, expected) ASCII transliteration is 'ä' -> 'a', 'ö' -> 'o' and 'å' -> 'a'. I strongly suspect that this is also true for the scandinavian languages (Swedish, Norwegian, Islandic & Danish).
We are using Unidecode.NET to transliterate people's names to be used in usernames. The "German" transliteration of 'ä' -> 'ae' creates transliterations that simply look wrong to a Finnish person. For example, a common Finnish surname of 'Hämäläinen' is transliterated to 'Haemaelaenen', which not only looks foreign to a Finn, but would also be pronounced completely differently (the expected transliteration would be 'Hamalainen').
I'm not sure there is a simple resolution to this if 'ä' -> 'ae' etc. are the correct transliterations for German and other languages, but for us this change has forced us to revert to using version 1.2

I confirm, that problems after 1.2 version.
I will update all codes and fix it.

I updated characters from original library in 2.0.0. Maybe it is can help.


autofocus on search

Issue Type (delete the irrelevant ones)

[x] Enhancement Request

It would be nice to have an autofocus search option so the browser would automatically focus the search box.

@corbosman I think this is something I can include in our next minor release. Also open to PR's.


tweak the record page to to put images/preview/files last when present

This is a short-term stop gap solution. When we display the 2D image, 3D previews, and downloadable files on a record page, they should come after all other attributes of the record (and in that order).
For example, on these 2 record pages, the "dataset chromosome" related entity gets presented later on the page.
https://staging.facebase.org/data/record/#1/legacy:dataset/id=263
https://staging.facebase.org/data/record/#1/legacy:dataset/id=4277

I'm de-prioritizing this, because we need to address the underlying heuristic and annotation guidelines.


tests: TestServiceFile: increase timeout to 15s

TestServiceFile was previously waiting 10 seconds. In our test on Fedora
23, it was taking 11 seconds. This made the test fail. This patch
increases the delay to 15 seconds. It should be enough.
/cc @iaguis

LGTM


When I use nightfly colors the color become weird, how can I fix it ?

You have not provided any details about how to repeat this issue.
Nonetheless, it is quite likely that your issue either is:

You are not using a true-color terminal, refer here for details
Or, you may be using a broken version of Vim; for example the default version of Vim on macOS does not correctly support termguicolors, use Homebrew instead (if this is your issue)

Please check the prerequisites, if you still have an issue then provide a repeatable test case.
Best of luck.


Enso Atoms should expose structural equality through the Polyglot boundary

This task is automatically imported from the old Task Issue Board and it was originally created by Radosław Waśko.
Original issue is here.

Team
Technical Designer: ...
Implementer: ...
Acceptance/QA: ...
Why
As a library developer
I want Enso Atoms exposing structural equality when accessed from polyglot (Java) code
So that I can use Java utilities like HashSet with Enso Atoms and get expectable results
Acceptance Criteria
Scenario:
GIVEN two structurally equal Enso Atoms
WHEN they are passed through a Polyglot boundary and the Java `.equals` function is called to compare them
THEN it should return `true` as expected; moreover the exposed `hashCode` implementation should also be consistent with the `equals` method (as described in `Object` javadoc - o1.equals(o2) must imply that `o1.hashCode() == o2.hashCode()`)

Additional Notes
Once this is implemented, the pending test relying on this behaviour should be re-enabled.
The relevant test is should correctly handle distinct with custom types like Atoms located in test/Tests/src/Data/Vector_Spec.enso.
Comments:
A scenario to test this can be:
from Standard.Base import all

polyglot java import org.enso.bugrepro.helpers.App

main =
    p1 = Pair 1 2
    p2 = Pair 1 2

    IO.println "Enso ==: "+(p1 == p2).to_text
    IO.println "Java equals: "+(App.checkEquality p1 p2).to_text

    IO.println "Hash codes: "+(App.getHashCode p1).to_text+", "+(App.getHashCode p2).to_text

with the following Java file packaged into a JAR and put into /polyglot/java/ directory inside of the Enso project:
package org.enso.bugrepro.helpers;

public class App
{
    public static long getHashCode(Object o) {
        return o.hashCode();
    }

    public static boolean checkEquality(Object o1, Object o2) {
        return o1.equals(o2);
    }
}
``` (Radosław Waśko - Jan 17, 2022)

This is very related to the Hashset work. (James Dunkerley - Feb 3, 2023)

The test in `Vector_Spec.enso` you are mentioning is already fixed on my local branch in https://github.com/enso-org/enso/pull/4067/commits/548920b20a5f44d3951171c999a0f0e2c3f9adc0#diff-6a51a720e46f1443fc109ddce7fbcebfb231b4a7b9616aae879e9b8bfe1fe7e2 by introduction of the Comparator API. Moreover, #184074597 seems related (Pavel Marek - Feb 3, 2023)

If this comes out of the work on HashMap and equality would be a nice to have.

If this comes out of the work on HashMap and equality would be a nice to have.

This won't be part of https://github.com/enso-org/enso/pull/4067. It is related to https://github.com/enso-org/enso/issues/5259 and needs further discussion, because there is no simple solution for this.


Dave/chef 707

:nut_and_bolt: Description: What code changed, and why?
As a user of Automate HA, when I run following commands on bastion system of Automate HA cluster, I should see output relevant to Automate HA infrastructure:
chef-automate info

:chains: Related Resources
:+1: Definition of Done

Command should run from bastion system of Automate HA cluster for all types of Automate HA deployments
Execution of command should try to auto select nodes (as required)
CLI command should display detailed and relevant output message
Ensure that CLI documentation page mentions the command, with it’s description, relevant flags and system where it can be run clearly
Document potential problems which users can face in troubleshooting pages

:athletic_shoe: How to Build and Test the Change

build automate-cli
build components/automate-cli/
upload package
hab pkg upload results/package.hart
Download package at basion
hab pkg install <package> -bf
Run the command
chef-automate info

:white_check_mark: Checklist
All PRs must tick these:

[x] I have read the CONTRIBUTING document.
[x] All commits signed-off for the Developer Certification of Origin.

With occasional exceptions, all PRs from Progress employees must tick these:

[x] Is the code clear? (complicated code or lots of comments--subdivide and use well-named methods, meaningful variable names, etc.)
[x] Consistency checked? (user notifications, user prompts, visual patterns, code patterns, variable names)
[x] Repeated code blocks eliminated? (adapt and reuse existing components, blocks, functions, etc.)
[x] Spelling, grammar, typos checked? (at a minimum use make spell in any component directory)
[x] Code well-formatted? (indents, line breaks, etc. improve rather than hinder readability)

All PRs from Progress employees should tick these if appropriate:

[x] Tests added/updated? (all new code needs new tests)
[x] Docs added/updated? (all customer-facing changes)

Please add a note next to any checkbox above if you are NOT ticking it.
:camera: Screenshots, if applicable
https://user-images.githubusercontent.com/36985548/229863460-e5f2e754-6258-4ff8-8895-d2ac31de2cf5.mov

@kalroy  @Dmaddu and @YashviJain01  this is how it looks now as it was looking before...


etcd_svc: tolerate unready endpoints

fix https://github.com/kubernetes-incubator/bootkube/issues/599

@xiang90 @aaronlevy @diegs

looks good to me.
@aaronlevy where is the best place to track the alpha feature we use?

IIUC We should not be using alpha features by default in bootkube.

@diegs But this annotation doesn't break anything, isn't it? It is not possible to achieve this behavior for previous versions of kubernetes, so you lose nothing.

@diegs
So this has some offline discussion with @aaronlevy and @xiang90 . Let me share it on github too.
First of all, we have seen many problems with unready (or not "Running") endpoints in service:

https://github.com/kubernetes-incubator/bootkube/issues/599
https://github.com/coreos/tectonic-installer/issues/1266

We would need this feature from k8s to make the whole thing more robust. Otherwise, it's very fragile and only work under good assumptions.
Second, this is not something new in k8s world. There has been long-discussed issues (https://github.com/kubernetes/kubernetes/issues/39363, https://github.com/kubernetes/kubernetes/issues/25283) for how to handle stateful workloads on k8s. This is something that upstream is moving towards (even slowly) for stateful workloads. Many statefulset applications are using it.
Third, as mentioned above, this doesn't break anything. Further more, self-hosted-etcd is also alpha feature.

Thanks for the additional context. It also looks like despite being an alpha feature it's enabled all the time (https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go) so it seems ok.
Can you add a brief YAML comment to the template saying a little bit about why we need this alpha feature (and also maybe mention that it has been around a long time & is always enabled)? That way people who are deriving their own configs from the templates will understand why it's helpful.

Though it looks like the migration to self-hosted etcd failed in the e2e test:
Migrating to self-hosted etcd cluster...
I0705 17:59:32.627115     997 log.go:19] warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
I0705 17:59:42.630782     997 migrate.go:65] created etcd cluster TPR
Error: services "etcd-service" not found
Tearing down temporary bootstrap control plane...
Error: services "etcd-service" not found
Error: services "etcd-service" not found
services "etcd-service" not found

@xiang90 maybe start a Documentation/alpha-features.md document that tracks what alpha features are being used, and why.
Ideally we aren't using alpha features -- but we had discussed the risk in enabling this and it seemed low enough to at least try and exercise it with self-hosted etcd.

the feature is already enabled by default in the controller-manager (just opt-in with annotation)
it's used already by / relied upon by statefulsets
we still consider self-hosted etcd to be in alpha itself.

My main concern would be people upgrading their clusters in the future and this feature essentially being graduated to an actual field, which means the annotation would just silently stop working unless they knew to update to the new fields. Generally covered by release notes, but just something to consider (and why I'd like to have an alpha feature tracking list).

Also agree with @diegs that a brief note directly in the YAML would be helpful too

@xiang90 maybe start a Documentation/alpha-features.md document that tracks what alpha features are being used, and why.

sounds good.

comments added

maybe start a Documentation/alpha-features.md document

We will write this thing in etcd-operator repo.
@aaronlevy Can you write that in bootkube repo?

@aaronlevy
fixed

coreosbot run e2e

coreosbot run e2e

coreosbot run e2e

coreosbot run e2e

coreosbot run e2e

coreosbot run e2e

coreosbot run e2e

coreosbot run e2e

coreosbot run e2e

coreosbot run e2e

@hongchaodeng rebasing may help with the tests.

pass after rebase


Timed out waiting for the socket to be ready for a write

Hi,
I websocket connect to my server with: private let websocket = WebSocket(url: URL(string: "ws://<MY IP>:<MY PORT>")!)
When the connection is disconnected, I retry it with the code below (retry == true):
func websocketDidDisconnect(socket: WebSocketClient, error: Error?) {
        NSLog("ws disconnected: \(String(describing: error)) => retry: \(retry)")

        if retry {
            Timer.scheduledTimer(withTimeInterval: WSConstant.RECONNECT_INTERVAL, repeats: false, block: { [weak self] (_) in
                guard let strongSelf = self else { return }

                NSLog("reestablishing...")
                strongSelf.establish(retry: nil)
            })
        }
    }

But it raises this error:
ws disconnected: Optional(Starscream.WSError(type: Starscream.ErrorType.writeTimeoutError, message: "Timed out waiting for the socket to be ready for a write", code: 0)) => retry: true
Minimizing and maximizing app do not help (when my app is minimized, it disconnects from the server; and reconnects when it is maximized).
The connection is only restored when I kill the app and rerun.
What should I do to over come this1writeTimeoutError?
Thanks

I have the same issue, did you manage to fix it @quanguyen ?

+1, I have the same:(

I have the same issue, did you manage to fix it @quanguyen ?
As a workaround I added a small delay to reconnect and seems to be working.

Hi @nyeu maybe you can help put with PR? Thanks!

Do you have multiple web socket connections?

I'm going to close this for now. If it comes up again in v4, we can reopen it.

anybody gets solution?

@usama8981 I made it work with this workaround:

As a workaround I added a small delay to reconnect and seems to be working.


Maintenance: Integration tests for code changes

Summary
We should consider including an Integration test workflow for the project. This would help catch some of the issues/failures with new code changes. A simplified workflow could look like:

Create an EKS cluster ( or a matrix version of supported ones)
Perform the upgrade operation using the version with the PR or one merged to main
Use the successful upgrade as a gating process
Tear down the cluster

Why is this needed?
This would help reduce the manual effort in testing the upgrade process against changes.
Which area does this relate to?
Automation, Tests
Solution
No response

Working on pre-flight checks on newly created clusters.


redesign the ui of signup page

Is your feature request related to a problem? Please describe.
Currently, the sign-up page may look outdated or lack the cohesive style that aligns with the rest of the website. This mismatch can create a disjointed user experience, which is frustrating if the other pages have a modern, polished look.
Describe the solution you'd like
To give the sign-up page a fresh, cohesive aesthetic, consider these design elements:
1-Minimalistic Layout
*Use ample white space for a clean look.
*Center-align the form on the page, surrounded by subtle visual elements like soft gradients or shadows to make it stand out.
*Add rounded corners to input fields and buttons for a softer, modern look.
2-Color Scheme
*Match the form colors with the overall theme of the website.
*Use high-contrast, easy-to-read text on input fields and buttons.
*Apply a subtle background color (like a gradient or pastel tone) or a light blurred background to add visual depth.
Describe alternatives you've considered
Floating Form Design: Place the form in a semi-transparent floating container over a high-quality, blurred background image.
Additional context
Add any other context or screenshots about the feature request here.

@jahnvisahni31 ma'am pls assign me !!!!

could you go for login page as the sign up page has already been assigned to someone
if you could do login page then i will mark the labels

could you go for login page as the sign up page has already been assigned to someone if you could do login page then i will mark the labels

yes ok

okay will give level 2


add constraint for websockets

Proposed changes:

Addresses https://github.com/sanic-org/sanic/issues/2733

Status (please check what you already did):

[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[ ] reformat files using black (please check Readme for instructions)

@ancalita done, does that mean the CI will run again? 😞😨

@ancalita done, does that mean the CI will run again? 😞😨

Yeah 😬 we should probably open a ticket to include changelog changes in that category of docs changes that only trigger the docs jobs, (rather than all 100+ jobs).

@vcidst Thinking again of this poetry lock change, I still think we should add websockets with the constraint in pyproject.toml, reason being that if in the near future the lock file gets refreshed because of some other dependency we need to update on 3.5.x, this change will be lost.

You're right, I can add websockets as a direct dependency with this constraint. This will be temporary though because,

Sanic will fix this soon for sanic==21.12.0
And, we'll upgrade to sanic==22.12.0 or sanic==23.3.0

I'll make the changes right away


import model comments

Hello, @danirus
Help please, how can I import (get qeryset) the article comments model for functions in views?

Hi, @ogurec-ogurec. I don't understand your question. Could you please explain it a bit more?


Website isn't detecting browser/OS language

Issue originally created by @asdrub
This issue was created on gitlab and then migrated here. Only the original post was migrated, not the comments. Please take a look at the discussions on the original Gitlab issue before commenting here: https://repo.getmonero.org/monero-project/monero-site/-/issues/1030

The website doesn't change automatically to the language of the Browser or OS, even when there is a translated version available. The user must always select the language in the menu.

I did some research on this. Without using javascript, autodetecting a language is quite annoying. There are other solutions, like serving a cookie and use PHP, but this would be more complex from a legal point of view.
I'm by no mean an expert in this, if somebody has deeper knowledge about this or have a solution in mind, please share your thoughts.

We can look at the user's Accept-Language header in nginx and redirect appropriately. Maybe something like this:
map $http_accept_language $lang {
    default en;
    ~es es;
    ~fr fr;
    # ...
}

...

rewrite (.*) $lang/$1;

We could also store a cookie when a user selects a language, and then redirect them next time based on that.

We can look at the user's Accept-Language header in nginx

Yes, that's probably the best solution.

We could also store a cookie when a user selects a language

That would create problems from a legal point of view and will force us to show an annoying disclaimer.

I'm looking into this. I asked suggestions to pigeons.

Aren't cookie disclaimers needed by the GDPR only if cookies can uniquely identify your browser or are used for tracking reasons?
As Regulation (EU) 2016/679 of the European Parlament and of the council of 27 April 2016 states:
Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags. This may leave traces which, in particular when combined with unique identifiers and other information received by the servers, may be used to create profiles of the natural persons and identify them.

If I am correct, a language cookie cannot be used to identify you. For this reason I do not think that a disclaimer would be needed.


Scaffolds should be referenced by tag

We should add tag version of the scaffold in our scaffolds.json list so any updates to the scaffolds are reviewed.

Another option is to make a public repo and get pull requests there btw

I actually prefer people just making their own repos as scaffolds which can then be added here, I think it's better because you can also mark the repo as a template, and then it can also be reused on GitHub. We see how it goes we can always adapt.


Why are StatsD stats counters disabled?

The counter metics generated by StatsD itself are specifically filtered out of the data sent to InfluxDB. Is there a reason for this? At the very least it feels like this filter should be optional.
  for (key in counters) {
    /* Do not include statsd counters. */
    if (key.match(/^statsd\./)) { continue; }

https://github.com/bernd/statsd-influxdb-backend/blob/master/lib/influxdb.js#L137

Support for sending the statsd_metrics data from StatsD the backend interface and generating some basic InfluxDB backend metrics would be a great feature, as seen in the Graphite backend. I may consider creating a PR for this.

I think we should make this into a configurable option. However, the reason behinds it prolly is to reduce load for InfluxDB I think.

I added a includeStatsdMetrics option to the configuration. (default false)
See: https://github.com/bernd/statsd-influxdb-backend#configuration
Hope that helps.

The if (key.match(/^statsd\./)) logic does not work if the prefixStats config parameter is changed from statsd to something that does not begin with statsd

Good catch @gnjack, thanks! Commit cce20b6 should fix that.


%DEFAULTUSERPROFILE%.

This EV function doesn't seem exist in current version (Microsoft Windows [Version 10.0.18362.239])
also the registry key doesn't contain the mentioned related key, HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList [DefaultUserProfile]
At least this is the case on my machine running Windows 10 Pro.
Could be this has been removed and the page needs updating.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 72d479c1-cedc-05fc-2d73-45342b955037
Version Independent ID: 8e91c2df-8405-3ec0-210e-1b97e86073cf
Content: Recognized Environment Variables (Windows 10)
Content Source: windows/deployment/usmt/usmt-recognized-environment-variables.md
Product: w10
Technology: windows
GitHub Login: @greg-lindsay
Microsoft Alias: greglin

Well, it is not certain that you will see that Environment Variable in the wild, unless you are checking in some lab computer for user migration or test equipment for running lab courses involving user migration. Maybe it can be related to the following page?
Create mandatory user profiles
(https://docs.microsoft.com/en-us/windows/client-management/mandatory-user-profile)

@officedocsbot assign @mypil

@JoeC-7 - Thank you for submitting feedback.
I will get this issue over to the Win10 ITPro writing team for investigation.
Thank you for reporting and making the docs better. Much appreciated.
I made a note to request the team to update this when the work is complete.

strange because I have this
[image: Screenshot (7).png]
[image: Screenshot (6).png]
but I don't have a working environment variable i.e. when I type in file
explorer %DEFAULTUSERPROFILE%\Desktop
I get a file explorer windows error, that says this
[image: Screenshot (8).png]
On Thu, 25 Jul 2019 at 19:53, Trond B. Krokli<EMAIL_ADDRESS>wrote:

Well, it is not certain that you will see that Environment Variable in the
wild, unless you are checking in some lab computer for user migration or
test equipment for running lab courses involving user migration. Maybe it
can be related to the following page?
Create mandatory user profiles
https://docs.microsoft.com/en-us/windows/client-management/mandatory-user-profile
(
https://docs.microsoft.com/en-us/windows/client-management/mandatory-user-profile
)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/windows-itpro-docs/issues/4551?email_source=notifications&email_token=AAF3OTS567THCVU3EOMR3N3QBHZD3A5CNFSM4IG32MIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD22N7QQ#issuecomment-515170242,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAF3OTX6LNO2UGBMZMCZNMTQBHZD3ANCNFSM4IG32MIA
.

--
Legally privileged or confidential information may be contained in this
message and/or any attachments. It is intended only for the use of the
individual or organisation to whom the e-mail is addressed. If you have
received the e-mail in error please notify the sender by reply email. If
you are not the intended recipient be advised that you have received this
email in error and should delete this message and any attachments from your
system and that any use, dissemination, forwarding, printing, or copying of
this email is strictly prohibited.

Looks like your image links did not make it to the Github page intact.
Maybe you could repair your comment links directly on this web page?

think that should fix it

@JoeC-7 - Is your issue already fixed? If not, could you please assist us by suggesting how this content could be edited to incorporate your feedback? If you could suggest what the appropriate text and where it should be placed, I could assign one of our team members to create a Pull Request to get it added or you may also create a Pull Request to add in your suggestion and just link this issue so I can have the doc owner review and merge the content. Thank you.

Well as I see it, this page is the only source (and therefore the source other websites use) about windows 10 environment variables for a typical windows, therefore my issue was that I wanted to use %DEFAULTUSERPROFILE%\AppData\Local\Temp in a firewall rule, rather than hand code the drive and path, when I went to find info about it this page suggested that the correct format and only format is %DEFAULTUSERPROFILE% which it isn't, as you say this isn't for standard workstation or standalone installs it may not be relevant, but why is c:\users\default created if it's only for domain scenarios?, anyway I guess that not a documentation issue, on the documentation side perhaps which are relevant to standalone and standard home user installs and which are for some server/domain innstall scenarios.

Thank you for adding information about your issue. My knowledge is lacking, but from what I remember, C:\Users\Default\ is mostly only used as a place to add settings so they will be applied automatically to any new profile (local account) in Windows, not much else.

Oh. My. Goodness. I just remembered an obscure detail I learned about 5 years ago: Some Windows Registry keys can be referenced as Environment Variables only from within a script or programming language, not from the console or when typing a command manually into a cmd.exe window. Unfortunately, I don't remember the details about how to access the keys, only that they don't appear as visible environment variables in CMD when using the SET command to view the ordinary environment variables, nor in SystemPropertiesAdvanced -> [Environment Variables].
You may want to search for "hidden environment variables" or "environment variables pointing to registry keys" to find out more about this in other forums, like https://stackexchange.com/ or https://superuser.com/ .

It seems to be generally unavailable, I've tried various things inc. running a powershell script and building a small C++ program to getenvironmentstrings (as detailed here -https://docs.microsoft.com/en-us/windows/win32/procthread/environment-variables) results are below:-
%DEFAULTUSERPROFILE% : The term '%DEFAULTUSERPROFILE%' is not recognized as the name of a cmdlet, function, script
file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct
and try again.
At line:1 char:1

%DEFAULTUSERPROFILE%

  + CategoryInfo          : ObjectNotFound: (%DEFAULTUSERPROFILE%:String) [], CommandNotFoundException
  + FullyQualifiedErrorId : CommandNotFoundException

GetEnvironmentVariables:
GTK_BASEPATH = C:\Program Files (x86)\GtkSharp\2.12
VisualStudioEdition = Microsoft Visual Studio Community 2019
Path = C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\Program Files (x86)\GtkSharp\2.12\bin;c:\gradle\bin;C:\Users\joeca\AppData\Local\Android\sdk\platform-tools;C:\WINDOWS\System32\OpenSSH;C:\Program Files\dotnet;C:\Users\Administrator\AppData\Local\Microsoft\WindowsApps
VSSKUEDITION = Community
MSBuildLoadMicrosoftTargetsReadOnly = true
CommonProgramFiles = C:\Program Files\Common Files
USERNAME = Administrator
ALLUSERSPROFILE = C:\ProgramData
APPDATA = C:\Users\Administrator\AppData\Roaming
USERDOMAIN = DESKTOP
SystemRoot = C:\WINDOWS
PATHEXT = .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL
DriverData = C:\Windows\System32\Drivers\DriverData
PUBLIC = C:\Users\Public
__COMPAT_LAYER = Installer
PkgDefApplicationConfigFile = C:\Users\Administrator\AppData\Local\Microsoft\VisualStudio\16.0_ee324d38\devenv.exe.config
ProgramW6432 = C:\Program Files
VSAPPIDDIR = C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE
CommonProgramW6432 = C:\Program Files\Common Files
VSAPPIDNAME = devenv.exe
SystemDrive = C:
ProgramFiles(x86) = C:\Program Files (x86)
CAMLIBS = C:\Program Files\darktable\lib\libgphoto2\2.5.21
IOLIBS = C:\Program Files\darktable\lib\libgphoto2_port\0.12.0
PROCESSOR_LEVEL = 6
USERPROFILE = C:\Users\Administrator
OS = Windows_NT
windir = C:\WINDOWS
VisualStudioVersion = 16.0
PROCESSOR_REVISION = 0f0b
COMPUTERNAME = DESKTOP
FPS_BROWSER_USER_PROFILE_STRING = Default
ProgramData = C:\ProgramData
PROCESSOR_ARCHITECTURE = AMD64
NUMBER_OF_PROCESSORS = 4
TMP = C:\Users\ADMINI~1\AppData\Local\Temp
ThreadedWaitDialogDpiContext = -2
PROCESSOR_IDENTIFIER = Intel64 Family 6 Model 15 Stepping 11, GenuineIntel
ProgramFiles = C:\Program Files
LOCALAPPDATA = C:\Users\Administrator\AppData\Local
CommonProgramFiles(x86) = C:\Program Files (x86)\Common Files
MAGICK_HOME = C:\Program Files\darktable\lib\GraphicsMagick-1.3.31\modules-Q8\coders
TEMP = C:\Users\ADMINI~1\AppData\Local\Temp
PSModulePath = C:\Program Files\WindowsPowerShell\Modules;C:\Users\Administrator\Documents\WindowsPowerShell\Modules;C:\Program Files (x86)\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules
OCL_ROOT = C:\Program Files (x86)\OCL_SDK_Light
OneDrive = C:\Users\Administrator\OneDrive
VSLANG = 1033
ComSpec = C:\WINDOWS\system32\cmd.exe
FPS_BROWSER_APP_PROFILE_STRING = Internet Explorer
VisualStudioDir = C:\Users\Administrator\Documents\Visual Studio 2019
C:\Program Files\dotnet\dotnet.exe (process 10260) exited with code 0.

also tried the ShellScript "Get-Childitem -Path Env:* | Sort-Object Name" with similar results

OK, I see. No luck in tracing the origin of that environment variable that way either. All I can do from here, is to speculate and guess, since I have not needed or worked with %DefaultUserProfile% myself.
Could it be that it only appears when using the Migration Wizard or tools like that, but not otherwise?

I think maybe it was deprecated, give me a couple of days i'm busy just
now, i'll see what i can do.
On Tue, 30 Jul 2019 at 06:44, Trond B. Krokli<EMAIL_ADDRESS>wrote:

OK, I see. No luck in tracing the origin of that environment variable that
way either. All I can do from here, is to speculate and guess, since I have
not needed or worked with %DefaultUserProfile% myself.
Could it be that it only appears when using the Migration Wizard or tools
like that, but not otherwise?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/windows-itpro-docs/issues/4551?email_source=notifications&email_token=AAF3OTV5KA7P6DSVVMEDJBDQB7ILJA5CNFSM4IG32MIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3C23KI#issuecomment-516271529,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAF3OTRGYUGIGXFVDHRNTLLQB7ILJANCNFSM4IG32MIA
.

--
Legally privileged or confidential information may be contained in this
message and/or any attachments. It is intended only for the use of the
individual or organisation to whom the e-mail is addressed. If you have
received the e-mail in error please notify the sender by reply email. If
you are not the intended recipient be advised that you have received this
email in error and should delete this message and any attachments from your
system and that any use, dissemination, forwarding, printing, or copying of
this email is strictly prohibited.

@JoeC-7 - Do you have any updates on this?

@JoeC-7 - We appreciate any feedback that improves the content in the Microsoft docs but we haven't had any response from you so we are going to close this issue.
Please feel free to re-open this issue if there is a specific area of the docs that we can improve or make better. Thank you.

@officedocsbot close


[BEAM-8432] Move javaVersion to gradle.properties

This is done in order to be able to change this parameter easily using the command line, eg:
./gradlew clean build -PjavaVersion=11
It's needed for providing java 11 support.
@kennknowles could you take a look?

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Run Python PreCommit

Run CommunityMetrics PreCommit

@kennknowles @lukecwik pinging kindly. :)

Should this also apply to kotlins jvmTarget?
https://github.com/apache/beam/blob/686bfd25406e46eb862b4933fea3ca64f9b50fb0/examples/kotlin/build.gradle#L135


broken tests PsCookieTest, TsAuthTest, RsFlashTest

build project fails when system's default locale is ru_RU.
"java.lang.IllegalArgumentException: header line of HTTP response "Set-Cookie: foo=urn%3Atest%3A99;Path=/;Expires=Пт, 27 мар 2015 00:34:57 GMT" doesn't match "[a-zA-Z-]+:\p{Print}+" regular expression, but it should, according to RFC 7230"
Caused by using  locale specific's date format in PsCookie.exit

@davvd this is definitely a bug

@davvd this is definitely a bug

@yegor256 got it, tag bug is here

@d-molotchko it's your task now, the budget is 30 mins, go ahead

@dmzaytsev thanks for the ticket, your account was topped for 15 mins, payment 54211899

@d-molotchko thanks!

@yegor256 could you release new version of takes with this fix?

@rultor release, tag is 0.11.2

@d-molotchko I'm not sure who actually completed this task, since there were no messages from you above (correct me if I'm wrong)

@rultor release, tag is 0.11.2

@yegor256 OK, I will release it now. Please check the progress here

@rultor release, tag is 0.11.2

@yegor256 Done! FYI, the full log is here (took me 14min)

@davvd I completed this task

@d-molotchko in the future, always post a short message into the ticket, explaining where you fixed the problem (in what pull request) and what exactly was done

@davvd please add extra 30 mins to @d-molotchko

@yegor256 ok

@davvd please add extra 30 mins to @d-molotchko

@yegor256 I've paid 30 mins to @d-molotchko in transaction 54668412


fix: Removed offending MD5 crypto hash and replaced with SHA1 and SHA256 …

…crypto hashes to work with the FIPS crypto policy.

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)

node-red: v3.1.5
node: v18.18.2
OS: Rocky Linux 9.3.
npm version 9.8.1
Proposed changes
Allows node-red to run when FIPS crypto policy has been enabled on the operating system.
Remove using the MD5 hash algorithm which is outdated and non-FIPS compliant. Updated hash to FIPS compliant ones such as SHA1 for the cacheBuster and SHA256 for the others.
Checklist

[x] I have read the contribution guidelines
[x] For non-bugfix PRs, I have discussed this change on the forum/slack team.
[x] I have run npm run test to verify the unit tests pass
[ ] I have added suitable unit tests to cover the new/changed functionality

Raised issue 1049
Discussed in node-red forum 85462

Thanks for this. Whilst the cache-busting change is fine, I need to think about the other changes and what impact they could have. The hash is used to tell if the flows have changed; changing the hash algo will obviously generate a different hash implying things have changed when nothing has.
A user with the editor open when they update Node-RED will get told the flows have changed when they haven't. A quick reload of the editor will 'fix' it - something we'd expect users to do anyway after an upgrade. So whilst there is a small chance of this being seen by users, the overall impact is low.


Update IP block list

I'd like to update the IP block list on the fly, is there a good way to do this while still using it as an express middleware? Thanks

I would also like to know if this is possible somehow.

It's in the documentation:

Using a function to get Ips:

var ips = function() { return ['<IP_ADDRESS>']; };
// Create the server
app.use(ipfilter(ips, {mode: 'allow'}));
module.exports = app;
Works great.