How to manually generate a single dict from a tab-separated file?

I’m at a loss. I try to generate a single DE dictionary from a cleaned-up, tab-separated DE Wiktionary download.
The source files (I created a one-entry test) look like this:
Aal	Substantiv, m, Aal, Pl. Aa·le
IPA: [aːl]
Bedeutungen:
1. Zoologie: schlangenförmiger Süßwasser- und Meerwasserfisch aus der Ordnung der Aalartigen (Anguilliformes)
2. umgangssprachlich, und in Zusammensetzungen: Fisch oder Wassertier von länglicher Gestalt
3. Seefahrt, von U-Boot-Fahrern verwendete Bezeichnung für: Torpedo
4. Soldatensprache: neu eingezogener, unvereidigter Soldat, Soldat ohne Dienstgrad
5. junge Kalmuspflanze
Synonyme:
3. Torpedo
4. Brenner, Rotarsch, Sprutz, Zecke
Beispiele:
1. Er hatte einen sehr großen Aal gefangen.
1. Auf der Silberplatte lagen geräucherter Aal und Graved Lachs.
2. Meinst du, Freund der Tiere, man könnte Aal dazu sagen?
3. „Als mir der Torpedo endlich klar gemeldet wurde, gingen wir auf Gefechtskurs und nach Erreichen der Schussposition gab ich den Befehl ‚Torpedo los!‘ Der Aal verließ vorbildlich das Rohr, wurde dabei illegal vom I. Wachoffizier fotografiert und tauchte elegant ins Wasser.“
4.  Zu jungen Kalmuspflanzen sagte Ehrenfried immer Aal und Herta ärgerte das.

Substantiv, mf, Nachname, Aal, Pl. Aals
IPA: [aːl]
Bedeutungen:
1. unterdurchschnittlich häufig auftretender, deutscher und niederländischer Nachname, Familienname häufigstes Vorkommen in Deutschland im Kreis Kleve in Nordrhein-Westfalen); auch mit Namenzusatz de
Beispiele:
1. Herr Aal heiratete Frau Müller im Mai. Nun heißt er Aal und sie Müller-Aal.
1. Heute sind wir bei Aals zu Besuch.
1. „Aal?! Vortreten!“
1.  Frau Aal ist ein Genie im Verkauf.
1. Herr Aal wollte uns kein Interview geben.
1. Die Aals kommen heute von Wangerooge.
1.   Aal kommt und geht.
1. Aals kamen, sahen und siegten.

(There is a TAB after the initial "Aal". All other tab characters inside the content were removed.)
Since there is not much documentation, I tried the following command (on Linux):
./run.sh --lang1=DE --lang1Stoplist=data/inputs/stoplists/de.txt --dictOut=data/outputs/DE.quickdic --dictInfo="Wiktionary-based DE dictionary" --input1=data/inputs/MCH-DE-DE-test.txt --input1Name=dewikitionary --input1Charset=UTF8 --input1Format=tab_separated

(taking over the spelling error "dewikitionary")
But this gives an error:
Running with arguments:
--lang1=DE
--lang1Stoplist=data/inputs/stoplists/de.txt
--dictOut=data/outputs/DE.quickdic
--dictInfo=Wiktionary-based DE dictionary
--input1=data/inputs/MCH-DE-DE-test.txt
--input1Name=dewikitionary
--input1Charset=UTF8
--input1Format=tab_separated
lang1=de
lang2=null
normalizerRules1=:: Lower; 'ae' > 'ä'; 'oe' > 'ö'; 'ue' > 'ü'; 'ß' > 'ss';
normalizerRules2=null
dictInfo=Wiktionary-based DE dictionary
dictOut=data/outputs/DE.quickdic
Processing: data/inputs/MCH-DE-DE-test.txt

Dec 11, 2021 1:06:15 AM com.hughes.android.dictionary.parser.DictFileParser parse
INFO: count=0, line=Aal	Substantiv, m, Aal, Pl. Aa·le
IPA: [aːl]
Bedeutungen:
1. Zoologie: schlangenförmiger Süßwasser- und Meerwasserfisch aus der Ordnung der Aalartigen (Anguilliformes)
2. umgangssprachlich, und in Zusammensetzungen: Fisch oder Wassertier von länglicher Gestalt
3. Seefahrt, von U-Boot-Fahrern verwendete Bezeichnung für: Torpedo
4. Soldatensprache: neu eingezogener, unvereidigter Soldat, Soldat ohne Dienstgrad
5. junge Kalmuspflanze
Synonyme:
3. Torpedo
4. Brenner, Rotarsch, Sprutz, Zecke
Beispiele:
1. Er hatte einen sehr großen Aal gefangen.
1. Auf der Silberplatte lagen geräucherter Aal und Graved Lachs.
2. Meinst du, Freund der Tiere, man könnte Aal dazu sagen?
3. „Als mir der Torpedo endlich klar gemeldet wurde, gingen wir auf Gefechtskurs und nach Erreichen der Schussposition gab ich den Befehl ‚Torpedo los!‘ Der Aal verließ vorbildlich das Rohr, wurde dabei illegal vom I. Wachoffizier fotografiert und tauchte elegant ins Wasser.“
4.  Zu jungen Kalmuspflanzen sagte Ehrenfried immer Aal und Herta ärgerte das.

Substantiv, mf, Nachname, Aal, Pl. Aals
IPA: [aːl]
Bedeutungen:
1. unterdurchschnittlich häufig auftretender, deutscher und niederländischer Nachname, Familienname häufigstes Vorkommen in Deutschland im Kreis Kleve in Nordrhein-Westfalen); auch mit Namenzusatz de
Beispiele:
1. Herr Aal heiratete Frau Müller im Mai. Nun heißt er Aal und sie Müller-Aal.
1. Heute sind wir bei Aals zu Besuch.
1. „Aal?! Vortreten!“
1.  Frau Aal ist ein Genie im Verkauf.
1. Herr Aal wollte uns kein Interview geben.
1. Die Aals kommen heute von Wangerooge.
1.   Aal kommt und geht.
1. Aals kamen, sahen und siegten.

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
	at com.hughes.android.dictionary.parser.DictFileParser.parseLine(DictFileParser.java:164)
	at com.hughes.android.dictionary.parser.DictFileParser.parse(DictFileParser.java:102)
	at com.hughes.android.dictionary.engine.DictionaryBuilder.main(DictionaryBuilder.java:153)
	at com.hughes.android.dictionary.engine.Runner.main(Runner.java:29)

So I also tried adding lang2=DE:
./run.sh --lang1=DE --lang2=DE --lang1Stoplist=data/inputs/stoplists/de.txt --dictOut=data/outputs/DE.quickdic --dictInfo="Wiktionary-based DE dictionary" --input1=data/inputs/MCH-DE-DE-test.txt --input1Name=dewikitionary --input1Charset=UTF8 --input1Format=tab_separated

This does produce a dictionary (that can even be converted to v006 format using ./genv6.sh):
Running with arguments:
--lang1=DE
--lang2=DE
--lang1Stoplist=data/inputs/stoplists/de.txt
--dictOut=data/outputs/DE.quickdic
--dictInfo=Wiktionary-based DE dictionary
--input1=data/inputs/MCH-DE-DE-test.txt
--input1Name=dewikitionary
--input1Charset=UTF8
--input1Format=tab_separated
lang1=de
lang2=de
normalizerRules1=:: Lower; 'ae' > 'ä'; 'oe' > 'ö'; 'ue' > 'ü'; 'ß' > 'ss';
normalizerRules2=:: Lower; 'ae' > 'ä'; 'oe' > 'ö'; 'ue' > 'ü'; 'ß' > 'ss';
dictInfo=Wiktionary-based DE dictionary
dictOut=data/outputs/DE.quickdic
Processing: data/inputs/MCH-DE-DE-test.txt

Dec 11, 2021 1:09:04 AM com.hughes.android.dictionary.parser.DictFileParser parse
INFO: count=0, line=Aal	Substantiv, m, Aal, Pl. Aa·le
IPA: [aːl]
Bedeutungen:
1. Zoologie: schlangenförmiger Süßwasser- und Meerwasserfisch aus der Ordnung der Aalartigen (Anguilliformes)
2. umgangssprachlich, und in Zusammensetzungen: Fisch oder Wassertier von länglicher Gestalt
3. Seefahrt, von U-Boot-Fahrern verwendete Bezeichnung für: Torpedo
4. Soldatensprache: neu eingezogener, unvereidigter Soldat, Soldat ohne Dienstgrad
5. junge Kalmuspflanze
Synonyme:
3. Torpedo
4. Brenner, Rotarsch, Sprutz, Zecke
Beispiele:
1. Er hatte einen sehr großen Aal gefangen.
1. Auf der Silberplatte lagen geräucherter Aal und Graved Lachs.
2. Meinst du, Freund der Tiere, man könnte Aal dazu sagen?
3. „Als mir der Torpedo endlich klar gemeldet wurde, gingen wir auf Gefechtskurs und nach Erreichen der Schussposition gab ich den Befehl ‚Torpedo los!‘ Der Aal verließ vorbildlich das Rohr, wurde dabei illegal vom I. Wachoffizier fotografiert und tauchte elegant ins Wasser.“
4.  Zu jungen Kalmuspflanzen sagte Ehrenfried immer Aal und Herta ärgerte das.

Substantiv, mf, Nachname, Aal, Pl. Aals
IPA: [aːl]
Bedeutungen:
1. unterdurchschnittlich häufig auftretender, deutscher und niederländischer Nachname, Familienname häufigstes Vorkommen in Deutschland im Kreis Kleve in Nordrhein-Westfalen); auch mit Namenzusatz de
Beispiele:
1. Herr Aal heiratete Frau Müller im Mai. Nun heißt er Aal und sie Müller-Aal.
1. Heute sind wir bei Aals zu Besuch.
1. „Aal?! Vortreten!“
1.  Frau Aal ist ein Genie im Verkauf.
1. Herr Aal wollte uns kein Interview geben.
1. Die Aals kommen heute von Wangerooge.
1.   Aal kommt und geht.
1. Aals kamen, sahen und siegten.

Done: data/inputs/MCH-DE-DE-test.txt

Most common tokens:
  Aal@0(1)
Most common tokens:
  1@0(1)
  2@2(1)
  3@4(1)
  4@6(1)
  5@8(1)
  aːl@10(1)
  Aa@12(1)
  Aal@14(1)
  Aalartigen@16(1)
  Aalen@18(1)
  Aals@20(1)
  aber@22(1)
  Als@24(1)
  Altgedienten@26(1)
  Anguilliformes@28(1)
  ärgerte@30(1)
  auch@32(1)
  auf@34(1)
  Auf@36(1)
  auftretender@38(1)
  aus@40(1)
  b@42(1)
  Bedeutungen@44(1)
  Befehl@46(1)
  bei@48(1)
  Beispiele@50(1)
  Besuch@52(1)
  Bezeichnung@54(1)
  Boot@56(1)
  br@58(1)
  Brenner@60(1)
  Da@62(1)
  dabei@64(1)
  das@66(1)
  Das@68(1)
  dazu@70(1)
  de@72(1)
  den@74(1)
  der@76(1)
  Der@78(1)
  deutscher@80(1)
  Deutschland@82(1)
  die@84(1)
  Die@86(1)
  Dienstgrad@88(1)
  dir@90(1)
  du@92(1)
  Ehrenfried@94(1)
  ein@96(1)
  einen@98(1)
Writing dictionary to: data/outputs/DE.quickdic
sources start: 44
RAFList stats: 0x1 entries
pair start: 74
RAFList stats: 1x64 entries
uncompressed min 2085, max 2085, sum 2085, average 2085.0
compressed min 1120, max 1120, sum 1120, average 1120.0
text start: 1205
RAFList stats: 0x1 entries
html index start: 1212
RAFList stats: 0x64 entries
html data start: 1219
RAFList stats: 0x128 entries
indices start: 1227
RAFList stats: 1x32 entries
uncompressed min 14, max 14, sum 14, average 14.0
compressed min 20, max 20, sum 20, average 20.0
RAFList stats: 7x32 entries
uncompressed min 104, max 647, sum 3385, average 483.57144
compressed min 86, max 410, sum 2153, average 307.57144
RAFList stats: 0x1 entries
end: 5133

Only when I try to use this dictionary on my Tolino Vision 5, it makes the Tolino app crash. (I’m assuming there are differences between a) single-language, b) bilingual, and c) bilingual-reverse dictionaries?)
Is it even possible to create functioning single-language dictionaries (with HTML content, one line per entry) from TAB-separated text files?
Maybe I just made a simple mistake?

Not sure what you mean by bilingual-reverse.
The way the dictionaries work is that they have 1 or 2 indices (or more, but that has no use currently and I don't think it works).
The 2 indices ones are general translation/bilingual ones, and they work for both directions.
And it is indeed only possible to generate translation dictionaries from text file input.
It probably would not be that hard to change, but I've not really paid much attention to the single language dictionaries since I find them rather inconvenient to use UI-wise anyway (often need to click through to the HTML page instead of having the relevant information right there).

Interesting. Not having single-language dictionaries is sad—I, for example, almost never use "translation" dictionaries, but am instead interested in correct spelling, pronunciation, hyphenation, inflection and probably etymology of a word and thus regularly use Duden, Webster’s and Oxford (paper editions) or the dictionaries in my ebook reader (a Tolino Vision 5). Translation dictionaries usually don’t have that much information because they serve a different purpose.
Now your dictionary format(s) (and the Android app) have proved to be long-lived, rather small and performant, even on low-end hardware like an e-reader. Even to the point that all Tolinos include yours (albeit using v6), and on support requests tell users to go to your GitHub releases and download "more dictionaries" from here.
Unfortunately (and I can understand that!), you seem not to have time to adapt to the ever-changing Wiktionary styles, and thus, your dictionaries have sadly degraded to the point of being (almost) unusable, because they nowadays contain pages over pages of badly-formatted Wiktionary code fragments (see https://github.com/rdoeffinger/Dictionary/issues/147).
Well, we users of your code could help out and find new ways of filtering, or create our own dictionaries. I thought. But I, personally, don’t talk Java, so can’t change the code, the dictionary formats are complicated ("I like it", the programmer of pyglossary once said, "but the dictionary format is too complicated to include in pyglossary."). Thus, no third-party tools either, and we users must rely on what you give us. Which, in turn, is great and performant, but lacks documentation for the "average Joe".
So, we kindly as you for help. What can we do to assist?
For instance, the tab-separated dictionary of which I showed a part above, was created using a (rather speedy) Rexx script to convert from the Wiktionary XML dump, by user "tscho" on the E-Reader Forum, that does a great job of reducing the output to the necessary. It’s only needed to install "regina" on Linux (sudo apt install regina on Debian-derivatives), run it, then replace all TAB characters against nothing, and finally replacing all    < (3 blanks and less-than) against \t<, to make it a working input for, say, pyglossary, to create a StarDict dictionary, for instance.
Maybe that can help to make your dictionary content better again?
I’m sure many, many users of your Android app, end especially Tolino users, would be really happy for great new Wiktionary-based dictionaries, plus a documented way to create their own!

If there is a kind of "community" or just some people who are willing to invest some effort, I will give this higher priority and try to look into it.
I guess I am wondering if this tab delimited format is the most sensible one though.
I mean, if you could choose any formatting that is simple enough to describe and parse, would you choose that?
Because implementing a different format might not be more work than adjusting this one really (I would need to double-check that though).
Also for example the dictionary format to my understanding (which I would need to check against the specification) can do
Heading (with optional link to HTML page)
text across the whole page
text with left side and right side
Possibly that could be used to turn the single-language dictionaries into something even I would agree is useful.
For example instead of just the heading with the link to the HTML page have also 1-2 lines of description of the word visible right there on the main page.
Of course this doesn't all HAVE to be designed up-front, but especially if not this might need a bit of longer-term engagement to get it nice and polished.
I admit it is far too technical at the moment, but maybe a look at https://github.com/rdoeffinger/Dictionary/blob/master/dictionary-format.txt would be useful as well.
For example this part gives the different types of entries the dictionary consists of:
 means:
1: index into list_of([pair_entry])
2: index into list_of([index_entry]) (mark as "main word header" entry)
3: index into list_of([text_entry])
4: index into list_of([index_entry]) (mark as "extra info/translation" entry)
5: index into list_of([html_entry])/list_of([html_entry_data])
Your current idea would only allow you to create that last type of entry, which is the only one currently used by the single-language dictionaries. That to me doesn't feel like it's enough to get the most out of the app...
Btw I had a quick look into how single-language dictionaries are generated: you only specify a lang1 and no lang2. I don't think that will change that it's not possible with the input format you have.

That’d be great, thanks for considering! I do think there are actually two main types of usage:

Users that only want to translate a few words in a foreign-language text (i.e., from current ebook language to their native tongue).
Users that are native speakers (or rather fluent, or intense learners) that want "to know more" about a word/phrase (like etymology, longer explanation, pronunciation, hyphenation, probably sample sentences). These would use the more elaborate "single-language" dictionaries (as opposes to a shorter "translation-only" dictionary).

This may well be the reason for the Tolinos to have separate "Look up" and "Translate" functions—and I love it.
For creating "personal" dictionaries like this (maybe a technical dictionary), users might have different preferences on what they want to be included in the final output. I have ever been thinking about some kind of templates for this, since Wiktionary is quite well-structured. One could think about creating "Wiktionary output templates" (per language, because Wiktionary structure differs so much), and thus, for instance, allow creators to include/exclude things like inflection, etymology, sample sentences. This would of course be some extra work, but might pay. (Assuming you would want to document/open it more for creators.)
I should think both E-Reader Forum (German) and MobileRead (mainly English) might be good communities to find people willing to help. E-Reader forum seems the more active one, especially for Tolino.
And yes, you’re completely right—the "one line per entry, tab-separated" format was just a measure of last resort in my case, since I so much wanted "readable" EN and DE single-language dictionaries (and, well, also an EN-DE and DE-EN "translation" version, for the rare cases…).
For the input, I’d actually prefer Wiktionary XML dumps, plus kinda templates for output (predefined "standard" templates included that’d only show the important content, like maybe "Advanced Learner’s" level). Fortunately, most devices can render HTML (at least partially), so this could be easy. The "standard" output should be able to fit on a rather small device (smartphone) as well as larger devices (like 7" or 10" e-readers).
This would be for the "main" dictionaries (and I much like the way you currently work on the "complete sets"—it takes ages but produces rather nice lang1-lang2 dictionaries, and Wiktionary will evolve…).
For user-created content (like technical or neologism dictionaries), I’d prefer a simpler input format, maybe even down to "tab-separated", since these will most often be single-language anyway. I think.

I think that's already a bit too many options there.
The "Wiktionary output templates" needs to be far more concrete and something doable with a few hours of effort from my side.
For example I could probably add that the "HTML" content gets piped to some external program for formatting. But even if means it can be a Python or shell script, it still needs someone to write that.
Not sure if anyone who can do that could not almost as well go to src/com/hughes/android/dictionary/parser/wiktionary/AbstractWiktionaryParser.java and modify the HTML text there like the replaceSuperscript function does, so does that win anything?

And I forgot, the feature of using anything but purely the HTML entries is even missing from the wiktionary parser.
So that part is not even really solved.
I mean it's nice to have a long-term vision (but maybe it should be a bit more concrete), but there's also the question what a realistic first step would be, just making sure it's one that leads in the right direction.

Yeah, but it takes a vision to get inspired and motivated, and first feasible steps to get going. Maybe we’re both right. ;-)
The work you’ve done already is greatly appreciated, and your willingness to possibly invest some more time to it is even more appreciated. Possibly even by many, many Tolino users that don’t even know it. ;-)
My thoughts on "steps":

Modifying src/com/hughes/android/dictionary/parser/wiktionary/AbstractWiktionaryParser.java might be a good first-first step, thanks for the pointer!
Finding a way to more easily "filter" the Wiktionary XML dumps (like for all the odd newly-crept-in thinks like {K|, {Ü etc. groups) should maybe be step 1. At least this would allow creating "clean" versions of what you already have—a big step forward! You might get some inspiration from the quoted Rexx script here—it produces rather clean results (and is fast like hell).
Non-HTML output can be neglected—let’s scrap that for now. A step saved.
A HTML filtering hook using something external sounds a good idea. It’d basically allow people at least to modify the output. (Or probably remove new Wiktionary code stuff in case you might not be available.) A good second step, I’d think.
Templating… Well, future expansion? I guess this can probably only ever be done within your code, since only the code "knows" what "blocks" of data actually are in an entry. I got the idea because you seem to do something like that for the Italian language, and because people in the quoted forums sometimes talk about what to include in a dictionary and what not (i.e., sample sentences: Some love them, some simply hate them and consider them bloat.)

I actually just created a post in E-Reader Forum, asking people for assistance. Let’s see what comes out of it.

Not sure if the workaround works in the Tolino 5, but in the shine, you can create a translation dictionary with the same language in lang1 and lang2 and when long-tapping a word, select translate instead of lookup...

FYI: I just added support for the quickdic format (version 6) in pyglossary: https://github.com/ilius/pyglossary/pull/509

Thanks, I am glad someone made use of the specification I wrote!
I really disliked the old format for its use of Java serialization that made it very hard to use the format in other languages than Java. Luckily it was possible to reduce it to a small set of "magic bytes".
You say your implementation differs from the specification to be compatible with tolino, how? The specification is meant to document the format needed for tolino, and the dictionaries I generated using that worked on mine (I think)?
Also how did you solve the sorting issue? Using even an different version of libicu results in dictionaries that do not work properly, which is very problematic. If you only tested ASCII you might not have noticed.

I have a Tolino Shine 3. In the settings, I can download dictionaries from the "Tolino Cloud". This adds *.quickdic files to my file system in the folder .tolino/dictionaries/. I designed the pyglossary plugin to be compatible with these files both reading and writing.
After looking at your specification again, and at the Java code, I have to reformulate that statement: "The implementation agrees with the format specifciation, but it might or might not differ from the Java reference implementation in some aspects. My priority was to be compatible with Tolino ebook readers. While tweaking my implementation, I didn't always check that everything I did agreed with the Java code."
These are the things that caused most headaches for me when implementing the plugin:

Each index entry has a "start_index" and a "count" field that refer to blocks in the "rows" list. But the "count" is actually not the block size, but it is the block size minus one. Furthermore, the first "row" in the corresponding block always contains a reference back to the index entry. And the "type" of that first "row" is "1" or "3" depending on whether the index entry points to a "html" or "pair" type entry.
The sort order of the tokens in the index: Only the normalized version of "tokens" is compared, the non-normalized version is ignored during sorting. After I noticed this, everything worked well for German, Italian, French, and Spanish. But for English, I needed to enforce the collation rule &z<ȝ to reproduce the sorting of the Tolino dictionaries.

I am trying to add tests for QuickDic6 plugin in PyGlossary.
The problem is that the output QuickDic file is different every time with same input glossary. So can't verify the content/hash in tests.
Even after I supplied the same created datetime (parsed from environment variable).
That's the only place the current time is fetched (datetime.now() , and no time.time())
I tried a binary diff tool named vbindiff, but there seem to be lots of one-byte differences.
Do you guys know why the resulting binaries are different?

0x63 to 0x68 is 'c' to 'h'.
But the format spec is deterministic, if the pyglossary result is not that is likely a bug.
You'd need to start with the spec and compare which part of the written file exactly differs, which then will tell you which part of the code to look at.
It's a good idea to start with the smallest example possible, so parsing it by hand is realistic.
Or if you want me to look you should provide 2 dictionaries that are supposed to be identical but have diffs.

I don't have time to test this now, but please try to add mtime=0 to the call of GZipFile:
https://github.com/ilius/pyglossary/blob/533b32cf1d922af7ffcd12497373ca3c9a85cdbd/pyglossary/plugins/quickdic6.py#L343
Other than that, the format should be deterministic.

Adding mtime=0 solved the issue.
Thanks @tuxor1337 @rdoeffinger

I added 2 small tests.
And tested both tests on the Android app.
The HTML rendering is flawless BTW.
Also FYI @tuxor1337
I removed language options because we have command line flags --source-lang and --target-lang for all formats.
I will add them to Gtk GUI as well.
Do you use command line, Gtk or Tkinter?

I added 2 small tests. And tested both tests on the Android app. The HTML rendering is flawless BTW.
Also FYI @tuxor1337 I removed language options because we have command line flags --source-lang and --target-lang for all formats. I will add them to Gtk GUI as well. Do you use command line, Gtk or Tkinter?

Great, thanks for your work! I think my priority would be a working command line interface, GUI is optional. I remember having used the Gtk interface before though.


blender: enable wayland

Description of changes
Added proper wayland support.
Tested on kde and gnome, everything works just fine.
I've found another PR here, but it's been inactive for 2 weeks
I decided to add a switch with the waylandSupport conditional. Every change in this PR is under this conditional, so it doesn't break anything compared to the current version, just adds wayland. Also by default it is only enabled on stdenv.isLinux (so that it doesn't break darwin or anything else for now)

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 232067 run on x86_64-linux 1

2 packages built:

blender
blender-hip


[Feature Request]: Support array of maps in BigQuery StorageWrites when using Beam Rows as the PCollection input type

What would you like to happen?
Currently BigQuery table schema utility, and the implementation for StorageWrites, does not support sending Beam Rows with arrays of Maps (or Maps) as part of their schema.
Issue Priority
Priority: 2
Issue Component
Component: io-java-gcp

.take-issue


Cloudevents lib update

this PR updates the cloudevents library which removes avro bindings (and the need for a cc for snappyer, a library we don't even use) to simplify the build process.
I also fixed a small bug in the makefile found in #444

I've never seen the issue with CI here, curious if it's a temporary github issue.

Looks like the build fail issue isn't temporary :(

Implemented by #451


Support module level diff

Resolves #1, resolves #3

Much more sophisticated solution than in #6 :) all class and function definition nodes are stripped from module node, and module node is treated as a function and compared with other functions so it can find plagiarism when __main__ code was wrapped by  main() function (see test cases)
Also I found that strip prints feature was introducing regression in visit_Call- generic_visit was not called there - I've fixed it
Also when function code is short (like short module level code only) there might be some logic in print expressions - I've added a flag to be able to keep print expressions/statements (see test cases)


Refactor Codec/Serializers to reduce new object contstruction

There are three problems with the codec/serializer framework as it stands now.
First, the codecs rely upon holding "state" until the Serializer can write (or the inverse in the case of a read). The net effect of this is that for every serialization operation, there is at least 1 new object construction operation (to create a new codec), but in general it's much worse that this.
For example, consider the AsnInterledgerPreparePacketDataCodec, which is engaged for every prepare packet. Here, we have at least 5 new object instantiations for every deserialize operation. So, if we process 5m payments (prepare[5] + fulfill[2]), we would expect at least 35 million new object creations, which wastes memory, cpu cycles, etc.
Second, the current system attempts to automatically map from Java objects to ASN.1 OER. For the most part, this is a fine default setting, but creates complexity inside of the framework when we need to create custom mappings. For example, currently the CodecContextFactory maps a BigInteger to AsnUint64Codec, which makes sense for the ILP codecs. However, for some CCP codecs, BigInteger can sometimes be required to map to AsnUintCodec instead. This is not a showstopper -- we might simply try to create a more intelligent Codec perhaps, but this new codec would likely not suit all use-cases. A better system would be one where a developer can have more fine-grained control over the Codec/Serializer that should be used, despite what the default factory has configured.
Third, is clarity of code. There appears to be overlap between the Serializers and the Codecs. For example, it's nearly impossible to create a good unit test of a serializer because it's tightly coupled to its codec (a proper unit test here can be quite complicated, typically because creating a codec-mock is non-trivial). Additionally, it can be unclear which layer is responsible for what.
This ticket proposes to solve both performance and clarity issues by combining the codecs and serializers into a single layer, with an API that allows for simple encoding/decoding functionality (like the current API) but also allows for fine-grained read... and write... methods, similar to how an InputStream/OutputStream would allow by providing direct access to read/write methods.

A few thoughts to share:
Generally, the high level structs seem to have to declare exactly which codec they require for each field, so I don't really see much problem with the defaults (re: BigInteger <-> Uint64) .
Actually I was looking at using this library in an android background service for a rpc via Bluetooth Low Energy framework. I found declaring the structs to be quite repetitive and cumbersome.
I was messing around a while back with custom annotations, which allowed the use of reflection/Proxy to auto generate codecs for serializing/deserializing Immutable structs. See sketch here: https://github.com/sublimator/quilt/commit/b086d2e277b0e8c02e3ecac662aee327bde052c3#diff-e71e8189355a88ef639759191518ecfaR72
Obviously not very performant. I think the auto generated sequence codecs were around 35% slower.
Essentially:
AnInterface.builder().from(DecoderProxy.from(AnInterface.class, ...)).build()

Whatever way it's implemented, it might be nice to have a separate package of annotations one could apply that codec-framework could understand, and auto generate codecs, which could be replaced with hand written codecs as and when performance is required. It would aid in documenting the structs, and the auto generated codecs could be compared against the hand written ones.
I agree the stateful codecs are a bit less than ideal. On the topic of performance, there's a lot of concurrent map usage inside the context which perhaps could be better solved with builders/ immutable maps and copies when registering?
I think I recall @adrianhopebailie at one time expressing that a simpler Reader/Writer pattern could be more performant?

Yeh, this framework was definitely designed with "align to ASN.1" in mind.
The goal was a separation of the concepts:

Definition of data structures using ASN.1
Serializing data that conforms to those structures according to a set of encoding rules (in this case OER)

So you could have an ASN.1 modelling framework that allowed someone to write codecs to convert between business objects and an ASN.1 representation and then separately encode and decode these using a specific set of encoding rules.
Technically you should be able to write a different set of serializers for DER, BER etc if you needed an alternative encoding for the same ASN.1 defined objects.
The overhead of creating new objects each time was intentional as it made things simpler to implement (thread-safe without the need to enforce immutability) and I figured the "framework" approach was not the right tool for a high perf system anyway.
If you want performance you probably want use-case specific readers/writers that don't need to translate your business objects into an intermediate ASN.1 form. If we get to that then I think we want to look at a variety of other techniques for improving performance like re-using buffers for ILP packets and doing in place replacement of fixed length data (like amounts and expiries).
In a high perf ILP connector you could read a prepare packet, modify it, and forward it on with a new amount and expiry without ever needing to fully deserialize it. Maybe we can hold off on refactoring the codecs until we want to try something like that and ditch the framework altogether?

hold off on refactoring the codecs

Yeah, they'll rewrite themselves if something actually uses them and needs the perf right?

thread-safe without the need to enforce immutability

Not sure if I really follow :) If codecs were stateless it wouldn’t really
matter right? In and outs?


merge from template-github-action

README.md

docs(README): update from template

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 95.861%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
396

Relevant Lines:
411

💛  - Coveralls


Sonnet 3.5

Hey,
I just wanted to note that Sonnet 3.5 probably knows the whole human eval... Or it is really great at coding.
So I ran the test using https://github.com/svilupp/PromptingTools.jl
The results:

I know this isn't an issue. But it works well and could be pointed out.
Also just to see comparisons, it isn't exactly giving the reference solutions back. It actually solve literally every problem differently see:
Problem 002:
The reference solution:
function truncate_number(number::Float64)::Float64
    return number % 1.0
end

The Claude version:
function truncate_number(number)
    return number - floor(number)
end

For problem 13:
function greatest_common_divisor(a::Int, b::Int)::Int
    while b != 0
        a, b = b, a % b
    end

    return a
end

Also Claude:
function greatest_common_divisor(a, b)
    while b != 0
        a, b = b, a % b
    end
    return abs(a)
end

Test 76:
Reference:
function is_simple_power(x::Number, n::Number)::Bool
    n == 1 && return x == 1

    x, n = BigFloat(x), BigFloat(n)
    power = 1
    while power < x
        power = power * n
    end

    isapprox(power, x)
end

Claude Sonnet 3.5:
function is_simple_power(x, n)
    if x == 1
        return true
    elseif n == 1 || x < n
        return false
    end

    while x % n == 0
        x = x ÷ n
    end

    return x == 1
end

Still it's strange to me that claude-3-5-sonnet-20240620 passed all the tests.
This is the result I got with the prompt you provided above claude-3-5-sonnet-20240620.zip
The result is close to the one I provided in README with the default system prompt (I updated the leaderboard earlier this week).

I don't understand why does any test fails for you. :(
Have you tried with the system prompt I provided?

Have you tried with the system prompt I provided?

Yes, I already added the result file in my last comment https://github.com/01-ai/HumanEval.jl/issues/11#issuecomment-2294851357

I will attach mine tomorrow hopefully.
I cannot get what is different in your solution and mine. Why I got 100% then if we use the same LLM with the same system prompt?

I will attach mine tomorrow hopefully.

Thanks!

I cannot get what is different in your solution and mine. Why I got 100% then if we use the same LLM with the same system prompt?

This is also what I want to figure out;)

claude-3-5-sonnet-20240620.zip
I am attaching it.
What is your guess?

OK, I unzip that file and put it under the generations folder. Then I removed the results folder to reevaluate the results.
julia> include("src/evaluation.jl")

julia> evaluate("claude-3-5-sonnet-20240620-cvikli")

And the result is
Test Summary:                                                                       |   Pass  Fail  Error   Total     Time
HumanEval                                                                           | 112176  5366   3206  120748  1m31.7s
  test                                                                              | 112176  5366   3206  120748
...
    test/evalplus                                                                   | 110794  5302   3178  119274
...

And the final score is:
|                             model | temperature | evalplus |   basic |

String
String
Float64
Float64

claude-3-5-sonnet-20240620-cvikli
0.0
0.732
0.799

claude-3-5-sonnet-20240620-tj
0.0
0.707
0.793

So I guess the reason might be that the results were incorrectly generated.

Pff. Ok, sorry for the mistake then!
I am not sure what I did wrong then.
could you attach the results yet just for me to see which one was wrong in the tests?

Here it is results.zip


Nanoseconds being truncated in asof

match_time = get_info.index.asof(trade_time)
match_price = get_info[match_time]['Bid Price']
The actual index value:
<Timestamp: 2013-04-10 10:22:01.696815975-0500 CDT, tz=US/Central>
The error returned:
KeyError: u'no item named 2013-04-10 10:22:01.696815-05:00'
It seems to me that it is cutting off the last digits of the timestamp in this example.

IIRC this is fixed by some recent work in 0.15.0. If still not working, pls open a new issue.


Dockers should have proper Label Schema labels

This repos has a Dockerfile but it does not contain any Label Schema labels.  Following are the lables we have to have on our Dockers.

[x] name
[x] description Shoud be on user's perspective.
[x] url Should be SCoRe website
[x] vcs-url Should point to GitHub repo
[x] vcs-ref
[x] vendor Shoudl be "SCoRe Lab"
[x] version
[x] schema-version

@agentmilindu Would like to work on this issue.

@agentmilindu Kindly review the commit, before i create a pull request.
https://github.com/kmehant/OpenDF/blob/master/Dockerfile

Wonderful job! Let's set  the org.label-schema.schema-version to 1.0.
Like org.label-schema.schema-version = "1.0"

@agentmilindu Thank you for pointing it out, i addressed your changes :D

Nice! Merged.
Fixed with #90.

@agentmilindu Thank you so much for the merge :)


⚠ Cioos National CKAN has degraded performance

In d3fa9b8, Cioos National CKAN (https://catalogue.cioos.ca/) experienced degraded performance:

HTTP code: 200
Response time: 9875 ms

Resolved: Cioos National CKAN performance has improved in 37e1006 after 13 minutes.


Is the .merlin file required?

With my editor setup (Emacs, tuareg-mode, Merlin, ocaml-lsp) the .merlin file in the root of this project actually causes problems such that I e.g. see Unbound value ... errors.  After I delete the .merlin file those problems are gone.  I'm not familiar enough with the tooling to tell whether the .merlin file is really needed.  I would suggest to remove it.  As far as I undertand, .merlin is editor tooling specific file.  In general, IMO, editor specific files usually shouldn't even be part of the source code.  People who use a specific editor setup should create such files by themselves.

While on the subject, the .ocp-indent file is also 7 years old and we are using ocamlformat now.  Should we update or delete the .ocp-indent file also?

Good point, I will remove both !


s3.region and s3.endpoint containing region seems to be mandatory for accessing external AWS S3 bucket

When installing epinio 1.5.1 using external aws s3 bucket I noticed an error related to accessing my bucket when creating an app:

To reproduce the issue one can deploy epinio 1.5.1 without specifying --set s3.region=eu-central-1 and deploy any app over webUI, the result should be similar to the picture above.
helm upgrade --install epinio --namespace epinio epinio/epinio  --set skipTraefik=true --set global.domain=<IP_ADDRESS>.nip.io --create-namespace --set minio.enabled=false --set s3.useSSL=true --set s3.bucket=epinio-ci --set s3.endpoint=s3.amazonaws.com --set s3.accessKeyID=redacted --set s3.secretAccessKey=redacted

Maybe we should make this s3.region mandatory when using external AWS S3 bucket via changes somewhere in https://github.com/epinio/epinio/blob/main/internal/s3manager/s3manager.go#L42 and maybe also in our helm charts.
Note:
We don't provide --set s3.region=eu-central-1 in our tests which seems to be mandatory for commands like aws s3 ls now - otherwise the s3 endpoint is rendered as https://s3..amazonaws.com
Update 221202: seems that also s3.endpoint must contain the region in addition so --set s3.endpoint=s3.eu-central-1.amazonaws.com --set s3.region=eu-central-1.
Even with the region wise s3.endpoint the s3.region must be set. Otherwise it is failing on app push:
🕞  [stage-workspace-sample-a6b490545ef6c23371a018e1746065deccfwjf2w] download-s3-blob download failed: s3://epinio-ci/e50b91f4-392c-4edb-b983-96c908700d8e to ../workspace/source/e50b91f4-392c-4edb-b983-96c908700d8e An error occurred (AuthorizationHeaderMalformed) when calling the GetObject operation: The authorization header is malformed; a non-empty region must be provided in the credential.

I was trying these combinations:

s3.region=eu-central-1 and s3.endpoint=s3.eu-central-1.amazonaws.com -> PASS
s3.region=eu-central-1 and s3.endpoint=s3.amazonaws.com - > FAIL
without s3.region and s3.endpoint=s3.eu-central-1.amazonaws.com -> FAIL
without s3.region and s3.endpoint=s3.amazonaws.com -> FAIL

seems this is indeed related to aws cli bump to 2.9.0, since then it started to fail in CI

CI fixed, closing


Build error on version 3.6.9 - unsafe build flags

Xcode 15.0.1
macOS 13.5.1
I updated library version to 3.6.9 and got the following error when building the project:
❌ the target 'ZXingObjC' in product 'ZXingObjC' contains unsafe build flags.
I believe this issue was introduced in PR #569 by those changes:
https://github.com/zxingify/zxingify-objc/blob/510db5f6ed293bf2f9e1905d3b8268c502f3fb55/Package.swift#L69-L71
Library is linked with the exact version.
.package(
    url: "https://github.com/zxingify/zxingify-objc",
    exact: "3.6.9"
)

I know there are some workarounds like providing commit hash but it doesn't cut it for me.

Are there any updates regarding this issue?

+1

Any updates?

with Xcode 15.4 even the commit workaround does not work...
is there a way to remove the flag? would be willing to help if somebody points me in the right direction

Hi, news on this issue? Of course is still present on Xcode 16


stochastic terms

Hi,
I have been recently using pybird and thank you very much for developing this package.
I find pybird itself doesn't compute stochastic terms (eat 7 counter terms instead of 10) and stochastic terms are added to multipoles in likelihood:
if self.birdlkl is 'fastfull':
    self.bird.fullPs[0] += bval[7] / self.nd + bval[8] / self.nd / self.km**2 * self.k**2
    self.bird.fullPs[1] += bval[9] / self.nd / self.km**2 * self.k**2

I'm a bit confused about it:

why growth rate f wasn't included in this piece of code?
why window function and AP effect didn't apply to stochastic terms?

Could you explain it to me why? Thank you very much!

Hi, thank you for raising these two points.

In the code we have conveniently redefined from the original equations two new EFT parameters (absorbing f in their redefinition) such that there is only one k^2 stochastic term per multipole, see App. A.1. of 2003.07956.

You are right, in principle they should be applied. However, in practice, let's make the following observations. First, we observe that the stochastic terms are vanishing at large scales, while the window function acts mainly on large scales. Thus, in practice, we can neglect the window function on the stochastic terms. Second, plugging a power law P_\ell(k) ~ k^n in the AP formula results into the same power law, but with a different overall factor. This overall factor (which is order 1) can be absorbed into the EFT parameter in front of the stochastic contribution (since the EFT parameters are anyway cosmology-dependent).

Hopefully this answers your questions!

It helps me a lot. Thank you very much!
So it means the prior range is actually very broad and allows absorbing any O(1) factor multiplied by stochastic terms?
Another thing is, if I run MontePython in "fast marg" mode, I don't need to worry about these things because they are marginalized and are set to zero and then contribute nothing to final results. Do I understand it correctly?

Yes, O(1) factors can be absorbed given a prior encompassing the EFT parameter physical range.
Since the EFT parameters associated to the stochastic terms can be marginalized analytically over at the level of the likelihood, they do not appear explicitly in the likelihood and we don't need to vary them. Their effect, however, is present in the marginalized posterior of any other parameters, in particular the cosmological parameters. See e.g. App. C of 2003.07956 for further details. I guess as long as you are dealing with an analysis using data close to the one of BOSS (with b1 ~ 2), you don't need to worry too much about those things indeed if you run the MontePython likelihood in "fast marg" mode.

Thank you for your detailed explanation! It is much clearer to me now.


Supporting file preview

Per popular requests, let's prioritize the file preview feature.  Related issues are: #2116, #2427, and #4543.
Commonly requested file types:

txt (CSV/TSV)
parquet
avro
json
orc

We will add support to text preview and basic image preview in 1.26.0


support for 2nd Monday of the month MON#2

from the tests it doesn't look like this syntax is supported
0 0 12 ? 1/1 MON#2 *
to mean midnight on the 2nd Monday of every month.
it was generated by http://www.cronmaker.com/

Currently, I don't have the time to expand to more syntax (because it works for what we use it for and doesn't have any itches I need to scratch)
Feel free to create a PR though, I'll gladly review & merge it!

Thanks @siebertm, turns out this is solvable with current functionality
30 8 1-7 * 1
and 8-14 for second etc...


Question: Connect Token Design

Hi there, I was studying the connect token design and have a couple questions. I am just trying to learn about design intentions and use-cases.

timeout seconds - Why is this included in the connect token, instead of living in client/server as a tunable parameter? I was trying to think about use-cases for variable timeout. Is this perhaps somehow useful under DoS circumstances?
Redundant information - Did you ever consider consolidating the connect token public and secret data that have similarities? The redundancy was a bit confusing when reading the standard, and for a while I thought I was having a misunderstanding. A tidbit in the standard describing the intent of redundancy could help make it much simpler to read.
It looks like the only unique information in the encrypted section of the connect token format is the user data and the client id. Are there any use cases (besides testing) for having the public/secret server lists differ from one another (I couldn't think of any)? If not, would a format like this make any sense to you?

// -- BEGIN PUBLIC SECTION --
// --  BEGIN REST SECTION  --
[version info] (13 bytes)       // "NETCODE 1.02" ASCII with null terminator.
[protocol id] (uint64)          // 64 bit value unique to this particular game/application
[create timestamp] (uint64)     // 64 bit unix timestamp when this connect token was created
[expire timestamp] (uint64)     // 64 bit unix timestamp when this connect token expires
[client to server key] (32 bytes)
[server to client key] (32 bytes)
// --  END REST SECTION    --
[zero byte]                     // Packet type connection request
[version info] (13 bytes)       // "NETCODE 1.02" ASCII with null terminator.
[protocol id] (uint64)          // 64 bit value unique to this particular game/application
[create timestamp] (uint64)     // 64 bit unix timestamp when this connect token was created
[expire timestamp] (uint64)     // 64 bit unix timestamp when this connect token expires
[timeout seconds] (uint32)      // timeout in seconds. negative values disable timeout (dev only)
[num server addresses] (uint32) // in [1,32]
<for each server address>
{
    [address type] (uint8)      // value of 1 = IPv4 address, 2 = IPv6 address.
    <if IPV4 address>
    {
        // for a given IPv4 address: a.b.c.d:port
        [a] (uint8)
        [b] (uint8)
        [c] (uint8)
        [d] (uint8)
        [port] (uint16)
    }
    <else IPv6 address>
    {
        // for a given IPv6 address: [a:b:c:d:e:f:g:h]:port
        [a] (uint16)
        [b] (uint16)
        [c] (uint16)
        [d] (uint16)
        [e] (uint16)
        [f] (uint16)
        [g] (uint16)
        [h] (uint16)
        [port] (uint16)
    }
}
[connect token nonce] (24 bytes)
<zero pad to 744 bytes>
// --  END PUBLIC SECTION  --
// -- BEGIN SECRET SECTION --
[client id] (uint64)            // globally unique identifier for an authenticated client
[client to server key] (32 bytes)
[server to client key] (32 bytes)
[user data] (256 bytes)         // user defined data specific to this protocol id
// --  END SECRET SECTION  --
[hmac bytes] (16 bytes)

The PUBLIC SECTION (minus the REST SECTION - more on this later) can be used as the additional data in AEAD. This way the connect token is entirely protected from tampering, and the client id/userdata is still unreadable. I imagine there is no security risk, since this information is publicly knowable to authenticated clients anyways.
This design might have some benefits:

The server list and timeout is not redundantly included in both the PUBLIC SECTION and SECRET SECTION. The keys are still redundant, since they must not be publicly viewable. This might simplify the standard quite a bit, since the entire connect token only needs to be describe once, and the different sections are very clear. It is also now easy to understand why each piece of redundant information is in both the PUBLIC and SECRET sections.
The PUBLIC SECTION of the token can be read by the client, and the entire token, minus the REST SECTION, can be immediately forwarded as-is like a connection request packet to the server. When a client gets a connect token via REST call, they can trivially lop off the REST SECTION and send the remainder as a connection request packet, simplifying a lot of code in the reference implementation.

Does this consolidation/simplification make any sense? I'm probably missing some obvious problems, and am asking to better understand the netcode design.
P.S.
I moved the SECRET SECTION to the end of the packet. This way the SECRET SECTION resides at a known byte-offset, and the server can quickly validate connect tokens, just as before, without the need to parse the server address list.

I had personally assumed that one of the use cases for having redundant server addresses in the secret client data was so that a game server could verify if a client was allowed to connect to that server, without needing to access some sort of central "sessions" database.

@stellarLuminant Right, that makes total sense. I guess a better question would have been "why not make the connect token secret section a lot smaller", since that seems to remove redundancy, simplify things quite a bit, all without sacrificing any security.

The public token data is unencrypted, because it’s stuff the client needs to connect to the server, eg. The server IP address, private keys, timeout whatever.  It’s not encrypted by anything, because the token is sent down from backend to client via a secure channel, typically HTTPS.
The internal part of the token, the private stuff that you think is redundant, is encrypted by a key that the client does not have (by design). The client passes this opaque blob over an unsecure channel (UDP packets) to connect to the server.
The server uses this data to verify that yes, in fact this client is authorized by the backend and is allowed to connect to the server.
That is all. There is no redundancy. Please read the spec again.
Cheers

Thanks for the response Glenn! I appreciate you helping me to learn about your design.
There’s one thing I’m confused about. Does the typical use-case have the connect token share the same IP address list in both the private and public sections?
If so, is there an error in my proposal? I know you’re busy, and the nature of the question is complicated. You’re right, it’s an attempted optimization. This is typically how I personally learn. I understand it’s a long question though, and I apologize for not being able to shorten it any further. So feel free to ignore if you prefer :)

OK so I just figured out the problem. Not encrypting server IPs lets packet sniffers attempt to steal tokens before they are used, by knowing which server the token can be used to connect with.


SignatureVerification.verify is not documented

Problem
Given that cryptography is a one-way street, the place where Permit2 users will most frequently encounter errors is the verify function of the SignatureVerification library:
https://github.com/Uniswap/permit2/blob/bbbc92f895049ca45c4b25a450f9d3e907659284/src/libraries/SignatureVerification.sol#L21-L24
The function, unfortunately, lacks any form of guidance, leaving users on their own to source the necessary documentation.
This a problem that is compounded by Uniswap's leading position in DeFi, as the signature verification processes in Permit2 may be the first encounter with cryptography for many developers, especially front-end developers who have just made a career switch to web3.
Suggestions

Provide a high-level English description of the function's algorithm by using NatSpec comments (@notice, @dev, @param, and so forth)
Make the implicit knowledge in this function explicit, i.e., explicitly mention the fact that the signature param is an ECDSA signature, and that the typical secp256k1 curve is what powers your cryptography
Explain what the benefit of using both standard ECDSA signatures and EIP-2098 signatures is; also, explain that users can use either signing scheme, and that supporting both or just one scheme is up to them
Explain that using EIP-1271 means signatures can be signers, too
Document and explain UPPER_BIT_MASK (consider adding a EIP1271_ prefix)
For convenience, provide explicit link to EIP-2098 and EIP-1271
Consider reducing the control flow nesting depth by flipping the claimedSigner.code.length == 0 check into claimedSigner.code.length > 0, and using return to stop the function execution early. Here's a code snippet that shows what I mean.

Additional Context
We (cc @gavriliumircea, @razgraf) are currently working on integrating Permit2 into our project (Sablier V2), and have recently spent a lot of time debugging a bug that was being triggered by a revert on line 41 in SignatureVerification:
https://github.com/Uniswap/permit2/blob/bbbc92f895049ca45c4b25a450f9d3e907659284/src/libraries/SignatureVerification.sol#L41
The bug was caused by an issue in our front-end; however, it would have helped us during debugging to have a little more guidance in SignatureVerification.verify.

I do agree with @PaulRBerg and further more I think it would be helpful, for the frontend developers, to create a guideline for frontend integrations as well.( Right now there a literally almost no resources at all regarding this matter.)

create a guideline for frontend integrations

Yep. A front-end template that shows how to use Permit2 with Ethers.js would go a long way.

The permit2-sdk library already acts as some sort of a template (esp. when paired with the interfaces repository). What it does lack is documentation of a full end-to-end flow, which is the real bummer here. Suggestions on how to "extend" the interface at a contract level (calling .verify()) would go a long way as well. It would help avoid certain misconceptions.


is this project still alive?

Its been a few months since the last release. Is this project still alive?

A post from Jorge on Google Groups seems to suggest so - https://groups.google.com/forum/#!topic/securesocial/F9Se533PZaU

Even I'm curious to know the status of this project? I'm eager to use this plugin but the problem is that I use Play 2.4 and there seems to be no version at the moment that supports Play 2.4!

Yes, project is alive.  Master is compatible with 2.4 as of this writing. Docs have not been updated yet but the sample apps should get you going.


Added cookie parser to the httpIn node and dependencies list

I had the need for CookieParser so I could create sessions on HTTP nodes.

Coverage decreased (-0.1%) to 87.44% when pulling 035ffefd5a2a23e02f2bef0f79b0471cd813429f on sax1johno:master into bcff74327b09fc08ebab271aacf77292cd64fec0 on node-red:master.

Please check the Contributing (https://github.com/node-red/node-red/blob/master/CONTRIBUTING.md) file for details of the CLA that needs to be completed before pull requests can be accepted.
It is also preferred that changes should be discussed on the mailing list first to work out any issues that may arise.
A couple of points about the code change:

This should probably be optional as there could be users dependant on the current lack of session support to ensure each request is a new session.

Your changes to the package.json include the stuff added by npm post install as well as changing the indentation of the entire file, this should be reverted back.

Hey there,
My apologies - I kinda had my head down and didn't even notice the
"contributions" file until after I submitted the PR.
I'll address the mailing list now for a few pieces that I'd like to add.
I submitted a CLA last night so hopefully that'll come out. I've attached
to his message as well just in case it facilitates things.
Thanks,
John O'Connor
On Fri, Apr 29, 2016 at 2:30 AM, Ben Hardill<EMAIL_ADDRESS>wrote:

Please check the Contributing (
https://github.com/node-red/node-red/blob/master/CONTRIBUTING.md) file
for details of the CLA that needs to be completed before pull requests can
be accepted.
It is also preferred that changes should be discussed on the mailing list
first to work out any issues that may arise.
A couple of points about the code change:

This should probably be optional as there could be users dependant on
the current lack of session support to ensure each request is a new session.
2.
Your changes to the package.json include the stuff added by npm post
install as well as changing the indentation of the entire file, this should
be reverted back.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/node-red/node-red/pull/882#issuecomment-215668394

Updated the package.json file so that it only contains a new dependency on Cookie Parser.
After discussion in the mailing list it seems like this should have a minimal impact

Coverage remained the same at 87.544% when pulling a3d7d387925ba2f640b0896926e52ad2b97d9ef2 on sax1johno:master into bcff74327b09fc08ebab271aacf77292cd64fec0 on node-red:master.

Hey guys - I signed the cla a while ago and sent it over but I still haven't been accepted. Is there something else I need to do?

Hi @sax1johno - CLA received and all is good on that front.
You may have spotted I added it to the 'Current' milestone late last-night - so this is definitely on the todo list for our next release.
Whilst I was there, I did a quick review of the changes to see if I could merge, but there were a couple things that held me back. As it was nearing 1am, I closed the tab rather than add comments.
The main issue is package.json has had all its whitespace changed - would prefer if it simply added the one new dependency and left the formatting alone.
The other part, which arguably is beyond the scope of this specific PR, is thinking how we provide a message-driven way to set/delete cookies as part of the response - so we don't have to expose the functions on msg.res.
If you fix up package.json, we can get this merged. If you're interesting in proposing something for the second in a separate thread, then we'd welcome it (but equally, you don't need to feel obliged to, having opened this can of worms).

node-red-gitbot: please update CLA status

Added manually into 0.14.0 branch - including the ability to set/clear cookies using msg.cookies passed to the response node - 5f6a0141f056b28f948184213ebab7645dce5c62


Dont strip basic authentication from apiserver url

The following ensures that the following url retains its basic auth credentials.
./heapster --source=kubernetes:https://XXXXXX:XXXXXX@kubernetes.local:6443?inClusterConfig=false

This change is

@k8s-bot test this please (reconfigured Jenkins job)

ok to test

trying again.
@k8s-bot test this please

@k8s-bot test this please

Any chance to see this merged soon? Wanted to implement heapster in my k8s cluster but it fails reaching the api server because I'm using basic auth. Thanks!

LGTM, Please squash commits and sign the CLA.

Also please make sure e2e tests are passing

@nickschuch This PR is inactive for a longer period of time. If you plan to work on this, please reopen.


fix in huggingface model for extracting layer activation

let me know if you need additional details here, to verify Fedorenko2016 I need the changes on this pulll.

it looks good to me

ok can you please commit the suggestion? There's a button underneath that should let you merge it right from the browser.


[POA-2083] Add the list of sensitive data keys and regexes in config

Benchmark results: I was worried that running the list of regexps would add a lot (it has to be done for every primitive) but the single test case I wrote showed a noticeable drop in speed, but not horrible.
BenchmarkRedaction
BenchmarkRedaction-8     	   24046	     49254 ns/op
BenchmarkObfuscation
BenchmarkObfuscation-8   	   28398	     42044 ns/op

Verified that the regular expressions here line up with their source, and that we have omitted exactly those that are not supported by the regexp package and those that would match too many strings.


Gcloud::Storage::File#signed_url does not work

I tried using Gcloud::Storage::File#signed_url to generate object url with query strings, but I can't get it to work. I used the following code:
s = Gcloud.storage "my-project-name"
buk = s.find_bucket "my-bucket"
file = buk.find_file "myfile.txt"
file.signed_url

It returns the following error:
[10] pry(main)> file.signed_url
NoMethodError: undefined method `sign' for nil:NilClass
from /Users/premist/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/gcloud-0.1.1/lib/gcloud/storage/file.rb:502:in `signed_url'

I tried to investigate this and found out that signing_key on Gcloud::Credentials is not being set properly.
I used gcloud auth credential from my computer initially, and I tried to explicitly provide keyfile for authentication too, but it didn't work.

Hey @premist, the File#signed_url method expects that the Signet::OAuth2::Client object used for authenticating returns a valid object when calling it's signing_key method. According to the documentation, signing_key can return a OpenSSL::PKey (which we expect), or a string (which we don't expect yet, but I'm not sure we will see since we create a OpenSSL::PKey object). But in your case nil is being returned, which is very strange.
I am having a hard time reproducing this. Are you available to pair remotely so I can see what you are doing in hopes of creating a reliable reproduction?

I will try to replicate this on clean Linux machine soon and let you know. If this works properly on clean environment, there should be some misconfiguration on my machine which causes the issue.

@premist Any luck reproducing this?

Hi @blowmage, sorry for late reply. Will try this later today and I'll let you know how it goes.

Thanks! Looking forward to it. :)

Hey @premist, when you provide the keyfile explicitly, are you giving the absolute path to the file? e.g /Users/premist/gcloud.json and not ~/gcloud.json?

I created a clean Ubuntu (15.04 x64) VM, and did the following:

Install Ruby 2.2.2 via rbenv
Install Google Cloud SDK
Used $ gcloud auth login to sign in
Install gcloud gem

I tried the code above again and it failed. Seems like signing_key doesn't work as intended when gcloud gem authenticates using Cloud SDK credentials.

@premist Thanks! You also said:

I used gcloud auth credential from my computer initially, and I tried to explicitly provide keyfile for authentication too, but it didn't work.

I can't get it to fail when explicitly providing a keyfile, but I assume that the path you provided didn't resolve and so it fell back to the cloud SDK credentials. Any chance you can confirm?

I tried providing JSON keyfile explicitly to Gcloud.storage, and it worked. Providing an invalid path like ~/Downloads/keyfile.json raised a proper error, which is the following:
RuntimeError: The keyfile '~/Downloads/keyfile.json' is not a valid file.

I'm not sure why I was unable to get signed url when I provided keyfile explicitly, but seems like it works as intended.

It looks like that the code that derives credentials from the cloud SDK doesn't return "service account" credentials, which is missing the private key needed to sign the URLs. Right now we're trying to determine how we can convert "authorized user" credentials to "service account" credentials.

I found something that might be related to this on gsutil documentation. In order to use gsutil signurl command, you need to explicitly provide private key file regardless of Google Cloud SDK authentication.
I checked gcloud-python and gcloud-node, seems like those libraries don't support authentication by Cloud SDK. Also I couldn't find a code which converts 'authorized user' credentials to 'service account' credentials.

I'm pretty sure both node and python will pull the creds set by the cloud SDK. CC @stephenplusplus @dhermes to comment.

Yes, we have a tight integration with https://github.com/google/oauth2client and don't implement much of the signing ourselves, just proxy it out to the auth library.

Same as @dhermes said, our auth library is https://github.com/google/google-auth-library-nodejs which respects the cloud sdk session.

Ah, thanks for the correction. Then it should be possible to generate signed url using Cloud SDK credentials.

@dhermes @stephenplusplus Do your auth libraries give you credentials with the private key when authenticating with the cloud SDK? Or do you have a way to exchange the "authorized user" credentials for "service account" credentials?

The signed url documentation is clear about requiring service account authentication for signing the url. We need the private key to sign the url.

No, haven't found a way to do this yet: https://github.com/GoogleCloudPlatform/gcloud-node/issues/211

There is no way to sign with "authorized user" credentials, it must be a service account. Our code paths fail on non-service account credentials (as well as on GCE service account credentials).

@stephenplusplus Ah, thanks for the link. I had missed than in my searches. Makes sense.
@dhermes Thank you for the confirmation.
@premist I have some ideas. I'll try to get a PR up today.


fixes theme colors

Fixes #1

The next step of this will be to allow theme customization. But for now i want to fix the colors issue which makes lazysql almost unusable on some systems.


Using neovim with Astrovim

I've added Copilot using { import = "astrocommunity.completion.copilot-lua" }, in my communtiy.lua
I see suggestions in my neovim, but I can't seem to accept any suggestions. I've tried alt-l, but it's not working. I haven't configured anything, so I am assuming I'm supposed to use whatever the default mappings are. Does anyone have a youtube video or something showing how to use this in neovim? I have authenticated and it I run :Coplilot is says it is enabled for typescript which is what I am using. I just figure out how to accept.

It had nothing to do with this code or astrovim, it was that I didn't have my meta key mapped in iTerm2.


feat(openchallenges): add more unit tests to the image service

Description
Added additional unit tests for files that were not autogenerated by OpenAPI. Investigated several options for auto generating unit tests forOpen API auto generated files. None of the options I found seemed helpful and many were associated w/Open API competitors.
Related Issue
feat(openchallenges): added unit tests to the image service #2291
Add unit test for getting a single organization #1528
[Story] Write a unit test for the enpoint of the image service #1456
feat: Create unit test for the image service #1472
Fixes #(issue)
Changelog
Added additional unit tests that increased coverage to 32%:

Add

apps/openchallenges/image-service/src/test/java/org/sagebionetworks/exception/BadRequestExceptionTest.java
apps/openchallenges/image-service/src/test/java/org/sagebionetworks/exception/ControllerAdvisorTest.java
apps/openchallenges/image-service/src/test/java/org/sagebionetworks/exception/ErrorConstantsTest.java
apps/openchallenges/image-service/src/test/java/org/sagebionetworks/exception/ImageApiDelegateImplTest.java

@tschaffter Can you please take a look at the additional unit tests I added for image service last week?

Investigated several options for auto generating unit tests forOpen API auto generated files. None of the options I found seemed helpful and many were associated w/Open API competitors.

What did you investigate exactly? What options did you found?

Investigated several options for auto generating unit tests forOpen API auto generated files. None of the options I found seemed helpful and many were associated w/Open API competitors.

What did you investigate exactly? What options did you found?

I watched a YouTube video and found this PP How_to_fast_generate_your_API_Test_with_OpenAPI_Tools_and_Rest-Assured.pdf. This one didn't really help me to understand the "How" of implementing automatic test generation for auto generated modules they way I hoped it would.
This link seemed to address the autogeneration using a .mustache file (which I could not find an example of in our file structure). It could be added but I have no idea how to determine if classes are optionally included but, there is this template that may be helpful if this method is used.

I also am unsure where in the process this would be implemented in order to use it as described by the author below:

You can download that file to your local directory, and use it in the docker SDK generation command from the previous section by adding this flag:
-t ./local/REPLACE-WITH-YOUR-TEMPLATES-FOLDER$

Here are some other tools I found that seemed like they would not suit our needs:

newman
Humlix

Updated the test names in all new tests for image-service. Restructured the file order to match that specified by @tschaffter

@mdsage1 Thanks for sharing the resources. REST Assured describes an interesting framework that could make it easier to develop tests. However, this approach does not seem to apply to develop test for internal and other utility functions.
The approach that use templates could be used to write tests for the code generated by OpenAPI generator. I know that such templates do exist but it depends on the generator being used. For example, generating a REST API for Python comes with automatically generated tests. I'm not sure whether we could get the benefit for the generator we use for Java. I will have a look.
One topic we can discuss together offline is the format of the name of the test functions. The format you are using is described in the article below from Microsoft. Note that in this context, you were right to capitalize the first letter of the test. It seems that this naming convention for test takes precedence over the more general rule that function should start with a lowercase. So feel free to capitalize again the first letter.
https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-best-practices
See Google Java naming convention about function names. They do use camel case for test
For now please continue with the naming convention you have been applying so far.

The coverage of the package exception could still be increased. Is this something that you could do as part of this PR? I propose to write all the tests required to reach 50% coverage in this PR instead of creating multiple PRs. Alternatively, it would be OK to add PR package by package and open a PR for each of them. What is important is that PRs represent a logical chunk of work (e.g. "add unit tests to the package exception of the image service" or "increase the image service coverage to 50%").

Note
Feel free to request a new review from a reviewer by clicking on the "reload" button next to the reviewer's name.

@mdsage1 This is where the mustache template files for the generator spring live. Note that this folder includes the latest templates files while we may be using a different version (defined in our file openapitools.json).
https://github.com/OpenAPITools/openapi-generator/tree/master/modules/openapi-generator/src/main/resources/JavaSpring
I can confirm that the generator spring does not come with templates for test files with one exception. The template below creates a single, virtually empty test file that is part of our code base.
https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/JavaSpring/libraries/spring-boot/SpringBootTest.mustache

Generated code will be excluded from test coverage with #2332

@mdsage1 When working on a PR, e.g. when addressing feedback, please convert the PR back to Draft using the link Convert to Draft at the top-right corner of the page.

image-service unit test coverage has been improved to 59% after addingan adiitonal test to exception for controller advisor. Reformatted naming as requested.

Note Feel free to request a new review from a reviewer by clicking on the "reload" button next to the reviewer's name (if they have already left a review).

@tschaffter I don't see a reload button so I'm tagging you. Maybe we can look at this in our meeting for the next PR.

@mdsage1 What has changed since the last review you requested? Please update the original post in this thread, in particular the Preview section.

Add

apps/openchallenges/image-service/src/test/java/org/sagebionetworks/openchallenges/image/service/exception/ControllerAdvisorTest.java --added additional tests
apps/openchallenges/image-service/src/test/java/org/sagebionetworks/openchallenges/image/service/exception/BadRequestExceptionTest.java

Removed

apps/openchallenges/image-service/src/test/java/org/sagebionetworks/openchallenges/image/service/api/ImageApiDelegateImplTest.java (couldn't get it to work)


extend function calculate_fault_domain_enabled

High Level Description
set fault_domain_enabled to True if a user puts a fault domain
script in genconf/fault_domain_detect or if a user had overridden
fault_domain_detect_contents.
Related Issues

DCOS-18424
DCOS_OSS-1720

Checklist for all PR's

[ ] Included a test which will fail if code is reverted but test is not. If there is no test please explain here:
current tests coverage is sufficient for this change
[X] Read the DC/OS contributing guidelines
[X] Followed relevant code rules Rules for Packages and Systemd

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[ ] Change log from the last version integrated (this should be a link to commits for easy verification and review): example
[ ] Test Results: [link to CI job test results for component]
[ ] Code Coverage (if available): [link to code coverage report]

PLEASE FILL IN THE TEMPLATE ABOVE / DO NOT REMOVE ANY SECTIONS ABOVE THIS LINE
Instructions and Review Process
What is the review process and when will my changes land?
All PRs require 2 approvals using github's pull request reviews.
Reviewers should be:

Developers who understand the code being modified.
Developers responsible for code that interacts with or depends on the code being modified.

It is best to proactively ask for 2 reviews by @mentioning the candidate reviewers in the PR comments area. The responsibility is on the developer submitting the PR to follow-up with reviewers and make sure a PR is reviewed in a timely manner. Once a PR has 2 ship-it's, no red reviews, and all tests are green it will be included in the next train.

@mesosphere-mergebot bump-ee

The enterprise bump is just for testing so I'll give this a shipit even though that doesn't have one.

@mesosphere-mergebot label shipit

@branden @darkonie This PR is the only one on the current train and it seems that train is blowing up on upgrades. All we see is that the test SSHs to the master to run the node upgrade script, it starts, then exits 1.
One such tests has been kicked off for this PR: https://teamcity.mesosphere.io/viewLog.html?buildId=807146&buildTypeId=DcOs_Open_Test_UpgradeTest_FromStable
If that passes, then no issue.

@darkonie @amitaekbote @branden Uh-oh. Looks like this PR is the culprit indeed. This PR will be bumped back to WIP until we can get that to pass.

opened an issue for core team https://jira.mesosphere.com/browse/CORE-1314

@mesosphere-mergebot label Ready For Review

@branden updated, please review

This looks good, enterprise counterpart at https://github.com/mesosphere/dcos-enterprise/pull/1534 is passing CI, so :ship:

@mesosphere-mergebot label shipit


'fixed_template_export_macro_in_classes'

these commits make sense to me. We just became sloppy with the template exports, and it is good to reuse the definitions from itkMacro.
Just one question: the commit on replacing the dynamic cast with static ones, why is that?

@N-Dekker @ViktorvdValk

Just one question: the commit on replacing the dynamic cast with static ones, why is that?

dynamic_cast uses runtime type information (RTTI) to check that the cast can be performed properly. static_cast does a more "brute force" cast, ignoring RTTI. It appears that the use of RTTI might be troublesome "over library boundaries", at least for Clang.
Having said so, I do not know if it is correct to just ignore runtime type information, in these cases.

@N-Dekker Would you have time to take this branch, solve merge conflicts and make it into a ready PR?

these commits make sense to me. We just became sloppy with the template exports, and it is good to reuse the definitions from itkMacro.
Just one question: the commit on replacing the dynamic cast with static ones, why is that?

This was one of my tries to prevent the segfault, but I think since Matt used my fork on which I tried everything, this.commit is still in there. But since it didn't help (also didn't make things worse btw), I would say we can ignore the dynamic -> static commit.

ok, so then we make a clean pr with the template exports and the use of itkMacro, right?

The current elastix develop branch builds fine, also on mac, even though it has an example of an executable which links to the elastix library (elastix/Core/Main/GTesting test executable ElastixLibGTest). Would it be possible to have a test in SuperElastix/elastix which reproduces the clang visibility issue?

Yes! I thinks the mac issues is fixed with this.
I build a package using this elastix branch and I don't get the segfault.
The debugger still gives the visibility error though, but elastix does run normally.

With the force-push I just did, I squashed all commits from @ViktorvdValk that added the ITK_TEMPLATE_EXPORT  keywords, and dropped the other commits (which replaced some dynamic_casts by static_casts, and added a check to dynamic_cast<ObjectType *>(this)). My motivation is that dynamic_cast should just work properly. Otherwise there must be something wrong with the visibility of class types, which should have been fixed by those ITK_TEMPLATE_EXPORT  keywords!

should we also remove our redefinition of these macro's and include itkMAcro?

I'll rebase-and-merge now, as it has passed the SuperElastix/elastix CI, but I'm interested to hear if it has entirely fixed the clang visibility problem...!


restic optimize does not delete unreferenced packs

Restic version: restic 0.1.0 (v0.1.0-147-g867fc5b), compiled at 2015-11-21 09:32:12 with go1.5.1
Observed behavior: applying restic optimize to the repository with a single "pack not referenced in any index" error from #358 broke the repository: restic check now reports numerous  "pack not referenced in any index" errors. Numerous means, that the output does not fit in my screen scrollback buffer (5000 lines)!
I did not try to restore from that repo.
Expected behavior: The initial "pack not referenced in any index" is gone.
Steps to reproduce: See #358

Thanks for the report, this is clearly a grave bug, the optimize command should never break the repository.

I tried to reproduce the issue with debug logging enabled but failed (partially):

This time the "pack not referenced in any index" error did not multiply
Optimize promises, that "The optimize command [...] removes uneeded data", but the "pack not referenced in any index" error remained

Is the assumption correct, that optimize should remove the unrefered pack?

Is the assumption correct, that optimize should remove the unrefered pack?

Oh yes, that's indeed a bug. Can you please run restic dump index and have a look what that unrefenced pack contains?

$ restic -r /media/backup-intern/restic/limucs-main.repo dump indexes | grep 846c84f0035a7eee47de70379fee24280e7f09273faefb9e9609779632d2ba64
debug enabled

... returned nothing. But wasn't this expected? Doesn't the error message tell quite clearly that the pack is not referenced in any index? Think you'll have to explain in more detail...
But: The debug log attached to https://github.com/restic/restic/issues/358#issuecomment-159701504 indicates that the pack contains blobs 253c0bac and 726f46f2:
13886210:2015/11/24 22:03:31 0.000 [               Repo.savePacker] 326613 repository.go:254 save packer with 2 blobs
13886211:2015/11/24 22:03:31 0.000 [               Repo.savePacker] 326613 repository.go:268 saved as 846c84f0
13886212:2015/11/24 22:03:31 0.000 [MasterIndex.RemoveFromInFlight] 326613 master_index.go:188 removing 253c0bac from list of in flight blobs
13886213:2015/11/24 22:03:31 0.000 [MasterIndex.RemoveFromInFlight] 326613 master_index.go:188 removing 726f46f2 from list of in flight blobs
13886214:2015/11/24 22:03:31 0.000 [              Index.StoreBlobs] 326613 index.go:121 stored 2 blobs

Here are the files theses blobs belong to:
$ grep -n -C 3 -e "adding blob 253c0bac" -e "adding blob 726f46f2" restic-daten.log
13886174-2015/11/24 22:03:31 0.000 [               Repo.findPacker] 326615 repository.go:226 found packer <Packer 1 blobs, 1076406 bytes>
13886175-2015/11/24 22:03:31 1.293 [                 Archiver.Save]  55 archiver.go:158 checking size for file /snapshots/daten/snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0012.JPG
13886176-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  55 archiver.go:167   adding blob e0fa9d45, 901176 bytes
13886177:2015/11/24 22:03:31 0.000 [                 Archiver.Save]  55 archiver.go:167   adding blob 253c0bac, 1970872 bytes
13886178-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  55 archiver.go:167   adding blob db978bbe, 699815 bytes
13886179-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  55 archiver.go:167   adding blob cbf12ffd, 1076374 bytes
13886180-2015/11/24 22:03:31 1.293 [             Archiver.SaveFile]  55 archiver.go:174 SaveFile("/snapshots/daten/snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0012.JPG"): 4 blobs
--
13886240-2015/11/24 22:03:31 0.871 [                 Archiver.Save] 326611 archiver.go:77 Save(data, 253c0bac): new blob
13886241-2015/11/24 22:03:31 1.271 [                 Archiver.Save]  67 archiver.go:158 checking size for file /snapshots/daten/snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0012.JPG_original
13886242-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  67 archiver.go:167   adding blob 8241ad29, 896793 bytes
13886243:2015/11/24 22:03:31 0.000 [                 Archiver.Save]  67 archiver.go:167   adding blob 253c0bac, 1970872 bytes
13886244-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  67 archiver.go:167   adding blob db978bbe, 699815 bytes
13886245-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  67 archiver.go:167   adding blob cbf12ffd, 1076374 bytes
13886246-2015/11/24 22:03:31 1.271 [             Archiver.SaveFile]  67 archiver.go:174 SaveFile("/snapshots/daten/snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0012.JPG_original"): 4 blobs
--
13886266-2015/11/24 22:03:31 0.446 [                 Archiver.Save] 326617 archiver.go:77 Save(data, c579c121): new blob
13886267-2015/11/24 22:03:31 1.161 [                 Archiver.Save]  61 archiver.go:158 checking size for file /snapshots/daten/snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0013.JPG
13886268-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  61 archiver.go:167   adding blob 8ab44a03, 1010841 bytes
13886269:2015/11/24 22:03:31 0.052 [                 Archiver.Save]  61 archiver.go:167   adding blob 726f46f2, 2295075 bytes
13886270-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  61 archiver.go:167   adding blob c579c121, 1501380 bytes
13886271-2015/11/24 22:03:31 1.213 [             Archiver.SaveFile]  61 archiver.go:174 SaveFile("/snapshots/daten/snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0013.JPG"): 3 blobs
13886272-2015/11/24 22:03:31 1.213 [           Archiver.fileWorker]  61 archiver.go:296    processed snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0013.JPG, 3/0 blobs
--
13886515-2015/11/24 22:03:31 0.740 [                 Archiver.Save] 326616 archiver.go:77 Save(data, 726f46f2): new blob
13886516-2015/11/24 22:03:31 1.260 [                 Archiver.Save]  59 archiver.go:158 checking size for file /snapshots/daten/snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0013.JPG_original
13886517-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  59 archiver.go:167   adding blob 7e24d605, 1006698 bytes
13886518:2015/11/24 22:03:31 0.000 [                 Archiver.Save]  59 archiver.go:167   adding blob 726f46f2, 2295075 bytes
13886519-2015/11/24 22:03:31 0.000 [                 Archiver.Save]  59 archiver.go:167   adding blob c579c121, 1501380 bytes
13886520-2015/11/24 22:03:31 1.260 [             Archiver.SaveFile]  59 archiver.go:174 SaveFile("/snapshots/daten/snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0013.JPG_original"): 3 blobs
13886521-2015/11/24 22:03:31 1.260 [           Archiver.fileWorker]  59 archiver.go:296    processed snapshot.current/Users/Xxxxx/Pictures/2011/Sortieren/IMG_0013.JPG_original, 3/0 blobs
--
29888678-2015/11/25 00:03:09 0.002 [                 Archiver.Save] 526942 archiver.go:66 (data, cbf12ffd) already saved
29888679-2015/11/25 00:03:09 0.731 [                 Archiver.Save]  53 archiver.go:158 checking size for file /snapshots/daten/snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0012.JPG_original
29888680-2015/11/25 00:03:09 0.000 [                 Archiver.Save]  53 archiver.go:167   adding blob 8241ad29, 896793 bytes
29888681:2015/11/25 00:03:09 0.000 [                 Archiver.Save]  53 archiver.go:167   adding blob 253c0bac, 1970872 bytes
29888682-2015/11/25 00:03:09 0.000 [                 Archiver.Save]  53 archiver.go:167   adding blob db978bbe, 699815 bytes
29888683-2015/11/25 00:03:09 0.002 [                 Archiver.Save]  53 archiver.go:167   adding blob cbf12ffd, 1076374 bytes
29888684-2015/11/25 00:03:09 0.734 [             Archiver.SaveFile]  53 archiver.go:174 SaveFile("/snapshots/daten/snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0012.JPG_original"): 4 blobs
--
29888863-2015/11/25 00:03:09 0.012 [                 Archiver.Save] 526945 archiver.go:66 (data, c579c121) already saved
29888864-2015/11/25 00:03:09 0.839 [                 Archiver.Save]  63 archiver.go:158 checking size for file /snapshots/daten/snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0013.JPG
29888865-2015/11/25 00:03:09 0.000 [                 Archiver.Save]  63 archiver.go:167   adding blob 8ab44a03, 1010841 bytes
29888866:2015/11/25 00:03:09 0.000 [                 Archiver.Save]  63 archiver.go:167   adding blob 726f46f2, 2295075 bytes
29888867-2015/11/25 00:03:09 0.000 [                 Archiver.Save]  63 archiver.go:167   adding blob c579c121, 1501380 bytes
29888868-2015/11/25 00:03:09 0.839 [             Archiver.SaveFile]  63 archiver.go:174 SaveFile("/snapshots/daten/snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0013.JPG"): 3 blobs
29888869-2015/11/25 00:03:09 0.839 [           Archiver.fileWorker]  63 archiver.go:296    processed snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0013.JPG, 3/0 blobs
--
29889025-2015/11/25 00:03:09 0.002 [                 Archiver.Save] 526928 archiver.go:66 (data, 253c0bac) already saved
29889026-2015/11/25 00:03:09 1.055 [                 Archiver.Save]  55 archiver.go:158 checking size for file /snapshots/daten/snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0013.JPG_original
29889027-2015/11/25 00:03:09 0.000 [                 Archiver.Save]  55 archiver.go:167   adding blob 7e24d605, 1006698 bytes
29889028:2015/11/25 00:03:09 0.000 [                 Archiver.Save]  55 archiver.go:167   adding blob 726f46f2, 2295075 bytes
29889029-2015/11/25 00:03:09 0.000 [                 Archiver.Save]  55 archiver.go:167   adding blob c579c121, 1501380 bytes
29889030-2015/11/25 00:03:09 1.055 [             Archiver.SaveFile]  55 archiver.go:174 SaveFile("/snapshots/daten/snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0013.JPG_original"): 3 blobs
29889031-2015/11/25 00:03:09 1.055 [           Archiver.fileWorker]  55 archiver.go:296    processed snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0013.JPG_original, 3/0 blobs
--
29889108-2015/11/25 00:03:09 0.002 [                 Archiver.Save] 526947 archiver.go:66 (data, db978bbe) already saved
29889109-2015/11/25 00:03:09 1.308 [                 Archiver.Save]  59 archiver.go:158 checking size for file /snapshots/daten/snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0012.JPG
29889110-2015/11/25 00:03:09 0.000 [                 Archiver.Save]  59 archiver.go:167   adding blob e0fa9d45, 901176 bytes
29889111:2015/11/25 00:03:09 0.000 [                 Archiver.Save]  59 archiver.go:167   adding blob 253c0bac, 1970872 bytes
29889112-2015/11/25 00:03:09 0.000 [                 Archiver.Save]  59 archiver.go:167   adding blob db978bbe, 699815 bytes
29889113-2015/11/25 00:03:09 0.000 [                 Archiver.Save]  59 archiver.go:167   adding blob cbf12ffd, 1076374 bytes
29889114-2015/11/25 00:03:09 1.308 [             Archiver.SaveFile]  59 archiver.go:174 SaveFile("/snapshots/daten/snapshot.current/Users/Yyyyyyy/Pictures/Fotos/2011/Schweden/IMG_0012.JPG"): 4 blobs

What I'd really like to find out it why running optimize caused more unreferenced packs to appear. That's really strange, I'm not able to reproduce it.

Me neither. Sadly I deleted the repo in order for a new run with debug logging enabled.
I could start over again with a refresh repo. With some luck this allows me to reproduce the initial error...

Hm. I'll have a look at the optimize code again this weekend, to see if I can spot anything that might lead to such a bad result.

Well, I started over twice with the following results:

Existing unreferenced packs were not deleted by restic optimize
No additional unreferenced packs appeared

Conclusion: I would consider this issue as initially reported unreproduceable. You could rename the issue to "restic optimize does not delete unreferenced packs".

I looked up in my shell's history, the exact commands I executed monday morning: check, unlock, check, optimize, check. I forgot to write about the unlock. I cannot explain why the first check failed to lock the repository. I'm quite sure that no other restic instance was running, but who knows... it was early monday morning...

I've updated the title. Do you think there is a different issue that the one described in #358? Or should we close this one for now?

I think both issues are still different: #358 causes unreferenced packs, where as this issue is about optimize not cleaning them up.

Ok, I'll have a look.

As an additional note, it is a bit disturbing that "restic check" considers that having unreferefenced packs means that "the respository has errors". It does not seem like an error to me; maybe it could just point to optimize.

Good point, thanks.

The optimize command was replaced by the prune command which does not have this bug, closing.


error accessing manahge.cookies with chrome

with new version <IP_ADDRESS> I have problems accessing manage.cookies for chrome. my program works fine with firefox but produces an error with number -2147467262 because of an invalid conversion

Fixed in v<IP_ADDRESS>


README: link items in list of features to the corresponding section with details

What the title says :) Hopefully this makes it easier for people to get a quick sense of what the project is about and how it works.

Summary by CodeRabbit

Documentation

Enhanced documentation structure and formatting in the README file.
Added hyperlinks for validation capabilities for easier navigation.
Reformatted content under "Section Properties" and "Content Validation Rules" for improved clarity.

Sections such as "Section Properties" and "Content Validation Rules" have been reformatted for clarity, with well-structured YAML examples.

That's a hallucination :joy: but the first part of the AI-generated description is correct.

Thanks @waldyrious! QoL improvements make everything so much easier.


Image links are not working if URL contains / suffix

See e.g. https://paritytech.github.io/ink-docs/faq/.
This is most likely because we use relative paths for those images and those stop working if there is a / the end of the URL.
Using an absolute path to / is not possible, since the URL is https://paritytech.github.io/ink-docs/.
Using /ink-docs/ as an absolute path prefix is also not possible, since running the docs locally under http://localhost/ should continue to be possible.

I've hotfixed this with absolute paths for now.


fix(db-postgres): localized field sort count

Fixes #4889

This looks good to me. Is there a reason the PR is marked as draft?

Marked it as draft until I pushed tests.


Value: get method and/or Index Trait

example humphrey_json:

https://docs.rs/humphrey_json/latest/humphrey_json/value/enum.Value.html#impl-Index<T>-for-Value
https://docs.rs/humphrey_json/latest/humphrey_json/value/enum.Value.html#method.get

example rust_json:
https://docs.rs/rust_json/latest/rust_json/enum.JsonElem.html#impl-Index<%26str>-for-JsonElem
example serde_json:

https://docs.rs/serde_json/latest/serde_json/enum.Value.html#impl-Index<I>-for-Value
https://docs.rs/serde_json/latest/serde_json/enum.Value.html#method.get

Thank you for the suggestion! This will be in the next release. There are performance considerations for looking up a key in an object, since currently the backing for Object is just a Vec -- I wrote out my ideas for how I'm hoping to address that someday in #7.

I've released v0.2.2 with this new functionality. Thank you again for the request!


Implement APIBlockchain.yield_blocks_slice()

TODO:

Download as little block chunks as possible
Generalize and reuse thenewboston_node.business_logic.utils.blockchain_state.read_blockchain_state_file_from_source
Mock file_object.read() to unittest: read_compressed_file allow to pass open_function which can be mocked
Unittest various edge cases: all blocks in the same block chunk, slice crosses two blockchunks, slice in the incomplete last block chunk, empty slice, remote node does not contain the required slice, etc
Use /api/v1/block-chunks-meta to firgure out which chunks to be downloaded
DO NOT cache data for now. We will do it later
[DONE] To reduce number of API calls implement filtering by start_block_number on API side and use it from client. It must be done in DRF-style (similar to how we tweaked OrderingFilter). So we can get all chunks between from_block_number and to_block_number and one chunk before from_block_number (by using ordering=-start_block_number&limit=1). GET /api/v1/block-chunks-meta/ is meant in this item.
Care about edge case when time passes between API call and the actual download of the incomplete binary which manages to become finalized. Therefore we should query the API with max available limit and download all binaries, but if some of them (the last one in 99.99% cases) is no longer available then we must retry by making a new API call and downloading it by new name. Take about not duplicating downloads.

The task is needed for https://github.com/thenewboston-developers/thenewboston-node/issues/387

https://github.com/thenewboston-developers/thenewboston-node/pull/391

https://github.com/thenewboston-developers/thenewboston-node/pull/392

https://github.com/thenewboston-developers/thenewboston-node/pull/393

https://github.com/thenewboston-developers/thenewboston-node/pull/394

https://github.com/thenewboston-developers/thenewboston-node/pull/395

https://github.com/thenewboston-developers/thenewboston-node/pull/396

https://github.com/thenewboston-developers/thenewboston-node/pull/397

https://github.com/thenewboston-developers/thenewboston-node/pull/398

https://github.com/thenewboston-developers/thenewboston-node/pull/399

https://github.com/thenewboston-developers/thenewboston-node/pull/400

https://github.com/thenewboston-developers/thenewboston-node/pull/401

https://github.com/thenewboston-developers/thenewboston-node/pull/402

https://github.com/thenewboston-developers/thenewboston-node/pull/403

https://github.com/thenewboston-developers/thenewboston-node/pull/404