Initialize markdown only once

This is a small performance improvement. But it should be somewhat significant for large collections with a lot of markdown.

Notice that this is about performance for no-op builds. Otherwise, image resizing and video compression is far more important for the CPU.

Performance figures
Using master:
adrien@complexe ~/D/s/tests> perf stat -r10 -B ./sigal_py3_venv/bin/sigal build photos/ _build_prod;
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.54 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 2.03 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 2.05 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.82 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.47 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.95 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.80 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 2.06 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.83 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.40 seconds.

 Performance counter stats for './sigal_py3_venv/bin/sigal build photos/ _build_prod' (10 runs):

       2360,241115      task-clock (msec)         #    0,969 CPUs utilized            ( +-  4,24% )
               488      context-switches          #    0,207 K/sec                    ( +-  5,88% )
                11      cpu-migrations            #    0,005 K/sec                    ( +-  8,89% )
            12 157      page-faults               #    0,005 M/sec                    ( +-  0,02% )
     5 705 818 220      cycles                    #    2,417 GHz                      ( +-  0,40% )  (49,92%)
     4 917 277 168      instructions              #    0,86  insn per cycle           ( +-  0,17% )  (74,93%)
     1 036 823 766      branches                  #  439,287 M/sec                    ( +-  0,12% )  (75,09%)
        50 325 792      branch-misses             #    4,85% of all branches          ( +-  0,28% )  (74,99%)

       2,436645094 seconds time elapsed                                          ( +-  4,28% )

Applying this patch:
> perf stat -r10 -B ./sigal_py3_venv/bin/sigal build photos/ _build_prod;
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.34 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.29 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.30 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.42 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.47 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.71 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.55 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.33 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.86 seconds.
  Sorting albums  [####################################]  100%
   Sorting media  [####################################]  100%
Collecting files  [####################################]  100%
Processing files  [------------------------------------]  0/0
   Writing files  [####################################]  100%

Done.
Processed 0 images (379 skipped) and 0 videos (22 skipped) in 1.39 seconds.

 Performance counter stats for './sigal_py3_venv/bin/sigal build photos/ _build_prod' (10 runs):

       2043,372745      task-clock (msec)         #    0,963 CPUs utilized            ( +-  3,35% )
               453      context-switches          #    0,222 K/sec                    ( +- 10,21% )
                10      cpu-migrations            #    0,005 K/sec                    ( +- 13,89% )
            12 249      page-faults               #    0,006 M/sec                    ( +-  0,02% )
     5 344 997 080      cycles                    #    2,616 GHz                      ( +-  0,62% )  (50,02%)
     4 637 683 203      instructions              #    0,87  insn per cycle           ( +-  0,11% )  (75,03%)
       981 760 224      branches                  #  480,461 M/sec                    ( +-  0,16% )  (75,01%)
        47 384 136      branch-misses             #    4,83% of all branches          ( +-  0,33% )  (74,97%)

       2,120834234 seconds time elapsed                                          ( +-  3,26% )

So yes, this is a 12.9% improvement for me.
By the way, I have 221 Markdown files in my photos directory.

This PR is also based on the documentation of the Markdown library:

You should also use this method if you want to process multiple strings without creating a new instance of the class for each string.

Note that the reset() method should be called between convert(). I will push it with some fix for the failing tests.
Speaking about tests, there is no real assertions about what should be extracted from an empty file. Currently, all preprocessors are skipped, but attributes are not deleted, so the Meta attribute have the previous data. This seems a bug to me. But this is for another day.
Oh, and by the way: shame on me for not running the tests before pushing.

See Python-Markdown/markdown#671 and Python-Markdown/markdown#672.
There was a bug there, that will be solved in the next release. Until then, this has to be done manually.

Could you please run your benchmark once more with the reset changes, just to be sure that there is still a significant enough gain ? Thanks.

OK, here are more figures. For brevity, I skip output from sigal. I manually applied the patch for python-markdown. All sigal tests are still OK.
master with photoswipe theme
 Performance counter stats for './sigal_py3_venv/bin/sigal build photos/ _build_prod' (10 runs):

       5015,872216      task-clock (msec)         #    0,990 CPUs utilized            ( +-  5,22% )
               676      context-switches          #    0,135 K/sec                    ( +-  7,62% )
                 9      cpu-migrations            #    0,002 K/sec                    ( +- 15,99% )
            13 899      page-faults               #    0,003 M/sec                    ( +-  0,83% )
    12 944 448 854      cycles                    #    2,581 GHz                      ( +-  0,17% )  (49,95%)
    11 274 264 137      instructions              #    0,87  insn per cycle           ( +-  0,10% )  (74,98%)
     2 354 613 689      branches                  #  469,433 M/sec                    ( +-  0,13% )  (75,04%)
       115 908 249      branch-misses             #    4,92% of all branches          ( +-  0,17% )  (75,01%)

       5,067768353 seconds time elapsed                                          ( +-  5,26% )

This patch with photoswipe theme
 Performance counter stats for './sigal_py3_venv/bin/sigal build photos/ _build_prod' (10 runs):

       4794,963802      task-clock (msec)         #    0,963 CPUs utilized            ( +-  3,02% )
               922      context-switches          #    0,192 K/sec                    ( +- 13,65% )
                13      cpu-migrations            #    0,003 K/sec                    ( +- 26,46% )
            14 410      page-faults               #    0,003 M/sec                    ( +-  0,82% )
    12 704 923 530      cycles                    #    2,650 GHz                      ( +-  0,84% )  (50,01%)
    10 998 472 366      instructions              #    0,87  insn per cycle           ( +-  0,16% )  (75,02%)
     2 297 597 621      branches                  #  479,169 M/sec                    ( +-  0,17% )  (75,04%)
       112 298 936      branch-misses             #    4,89% of all branches          ( +-  0,38% )  (74,95%)

       4,977598037 seconds time elapsed                                          ( +-  3,55% )

This is 1.77% faster, so it is insignificant.
master branch with a custom theme
This is a theme I am currently working on. It's a derivative of photoswipe
 Performance counter stats for './sigal_py3_venv/bin/sigal build photos/ _build_prod' (10 runs):

       2219,313680      task-clock (msec)         #    0,980 CPUs utilized            ( +-  4,67% )
               428      context-switches          #    0,193 K/sec                    ( +-  5,91% )
                 8      cpu-migrations            #    0,004 K/sec                    ( +- 10,91% )
            12 130      page-faults               #    0,005 M/sec                    ( +-  0,03% )
     5 711 295 676      cycles                    #    2,573 GHz                      ( +-  0,39% )  (50,02%)
     4 948 863 961      instructions              #    0,87  insn per cycle           ( +-  0,21% )  (75,08%)
     1 044 639 102      branches                  #  470,704 M/sec                    ( +-  0,14% )  (75,13%)
        50 406 356      branch-misses             #    4,83% of all branches          ( +-  0,25% )  (74,85%)

       2,263710201 seconds time elapsed                                          ( +-  4,44% )

Here I'm starting to have a doubt.
This patch with my custom theme
 Performance counter stats for './sigal_py3_venv/bin/sigal build photos/ _build_prod' (10 runs):

       2064,062561      task-clock (msec)         #    0,970 CPUs utilized            ( +-  4,44% )
               434      context-switches          #    0,210 K/sec                    ( +-  5,50% )
                 9      cpu-migrations            #    0,004 K/sec                    ( +- 13,92% )
            12 767      page-faults               #    0,006 M/sec                    ( +-  0,72% )
     5 379 681 810      cycles                    #    2,606 GHz                      ( +-  0,35% )  (49,92%)
     4 665 004 022      instructions              #    0,87  insn per cycle           ( +-  0,07% )  (74,98%)
       985 776 218      branches                  #  477,590 M/sec                    ( +-  0,11% )  (75,06%)
        47 268 135      branch-misses             #    4,80% of all branches          ( +-  0,16% )  (75,02%)

       2,126881752 seconds time elapsed                                          ( +-  4,42% )

And we still have a 6% improvement. Not as significant as before, but still.
Conclusion
My custom theme seems to spend way less time to write files. I didn't have the time to inspect why for now. But only with it the parsing of Markdown is significant, otherwise, the time spent when writing files is much higher than parsing.
No doubt that this patch improve the markdown parsing. But it seems to be insignificant for a lot of cases. I'd like to see it merged, but I could understand that it will probably be invisible for everyone.

Thanks @Glandos for the new numbers. The difference is quite small and will not be noticeable in most cases, but I guess there is no real drawback and the additional complexity is limited, so let's go for it.


Major refactoring

Sweet .. I see a new release on the horizon.

@mosabua maybe though still a bunch more to go to simplify and modernize, though I suppose after so many years sitting its still stands up fairly well ;-)


PR: Scroll to the selected item after go next thumbnail

Description of Changes

[x] Wrote at least one-line docstrings (for any new functions)
[x] Added unit test(s) covering the changes (if testable)
[x] Included a screenshot (if affecting the UI)

Result of master

ThumbnailScrollBar is not scrolled after the go_next_thumbnail button is pressed.

Result of PR

ThumbnailScrollBar is scrolled after the go_next_thumbnail button is pressed.

Issue(s) Resolved

Fixes #
Affirmation
By submitting this Pull Request or typing my (user)name below,
I affirm the Developer Certificate of Origin
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.

I certify the above statement is true and correct: ok97465

@goanpeca, something else to add?

I just left two minor optional suggestions, otherwise, LGTM.
Thank you very much for this @ok97465 !

Thanks for the review @jnsebgosselin !
Thanks for the PR @ok97465


CentOS 7 failed installing

Getting this error when running puppet agent:

To reproduce:
class{'redis': }
all dependencies are installed.

@DiegoGallegos4 Can you try the following code for me:
class { '::redis':
  manage_repo => true,
}

Basically it's saying it can't find the redis package from the upstream repo, so you need to enable epel.

This is resolved by the above code, as the acceptance test states. Closing.


Fix setting the default value of 'specfile_path'

Because the default of 'downstream_package_name' was set in the
configuration dict to the name of the repo before setting the default
for 'specfile_path', the default for the later was always
'{downstream_package_name}.spec', regardless if
'downstream_package_name' was set by the user in 'packit.yaml' or was
set by the code to 'repo_name'.
Change the order of setting these two defaults to fix this issue.
Additionally, use 'setdefault' when appropriate in
'PackageConfig.get_from_dict()'.
Fixes #1757.
TODO:

[x] Write new tests or update the old ones to cover new functionality.
[x] Update doc-strings where appropriate.

Merge after #1727.
RELEASE NOTES BEGIN
Fixed an issue due to which the repository was never searched for a specfile if 'specfile_path' was not specified, and 'specfile_path' was always set to '<repo_name>.spec'.
RELEASE NOTES END

Note, that this needs to be merged after #1727.


Fix idSchema issue with dependencies

Reasons for making this change
This fixes an issue where an idSchema would not be generated for property fields of objects that are in a dependency. It also adds a test to validate the fix.
This PR actually fixes the same issue as a part of its changes (here), but it looks like it's been sitting idle for the past month, and I'd like to get this fix in. My PR touches much less code, so I figure it should be easy to get this through.
The change in UnsupportedField.js is just because I ran the cs-format script. I suppose it was missed before, and I figured I'd commit it.
Checklist

[ ] I'm updating documentation

[ ] I've checked the rendering of the Markdown text I've added
[ ] If I'm adding a new section, I've updated the Table of Content

[x] I'm adding or updating code

[x] I've added and/or updated tests
[ ] I've updated docs if needed
[x] I've run npm run cs-format on my branch to conform my code to prettier coding style

[ ] I'm adding a new feature

[ ] I've updated the playground with an example use of the feature

Thanks for opening the PR. The changes seem straightforward to me, but I get the sense that there's a larger problem. It seems like all calls to toIdSchema are called on schemas that have already been "retrieved" using retrieveSchema, so I would have naïvely thought that the schema in the test would "work" (i.e. without this change, toIdSchema(retrieveSchema(...), ...) would provide the same thing as the new code after this change). Is there a user-facing bug that motivates this change? I don't object to the change in principle, but I feel like I don't have a good sense of where it's coming from.

Looking through the code base, you're mostly right about the usage of retrieveSchema and toIdSchema. toIdSchema is used in the Form, SchemaField, and ArrayField components, each time being passed a schema that's been run through retrieveSchema, but it's also being called within itself recursively.
The key thing to note is that retrieveSchema does not "resolve" schema deeply. It will resolve multiple levels of dependencies and $refs with recursive calls, but it does not actually dig into the structure of schema (the sub-schemas of children objects and arrays). On the other hand, toIdSchema does navigate schema deeply through recursion, and because the sub-schemas wouldn't have had retrieveSchema called on them, that's why it needs to (possibly) call retrieveSchema before doing any work.
Now, at the time toIdSchema was added (in this PR), dependencies weren't even supported in react-jsonschema-form, so there was no check for them in that first if statement. The fact that the "dependencies" in schema wasn't added to toIdSchema when support for dependencies were later added for react-jsonschema-form as a whole was an oversight (and what this PR addresses).
You're correct about the test working if I had done toIdSchema(retrieveSchema(...), ...), but the test is accounting for all calls to toIdSchema, those from the components mentioned above and from recursive calls from within toIdSchema itself.
As far as user-facing bugs, this change was motived by one that I encountered while using the top-level onBlur handler (which returns a field's id) to keep track of which fields have been blurred. Because dependencies weren't being resolved in toIdSchema, dependent fields didn't have IDs, which led to some issues.

Thanks for the explanation!


How to AJAX search tables created by variable relational columns

Summary of problem or feature request
This is a two part problem...
I have columns on a datatable that are from a products table and other columns from an attribute_values table. There is an attributes table which is related to the attribute_values table via the foreign key attribute_id. The attributes table holds the name of each unique attribute, thus becoming the column heading.
The attributes table is flexible, allowing a variable number of attributes to be stored from a variety of different other models using attributable_id and attributable_type fields.
For instance, I might have attribute_id=1 AND attribute_id=5 AND attributable_type='Product'.
The columns can be added to the datatable using a loop on the first product result using the customAttributes relation.
Basic example of how the attribute columns will be added:
$products = Product::all();
$dt = Datatables::of($products);

foreach ($products->first()->customAttributes() as $a) {
    $dt->addColumn($a->name);
}

Problem 1: How do I make these fields searchable over AJAX?
The actual values I'm planning to have flattened into each product row by looping through them for each paginated slice.
Problem 2: How can I intercept DataTable's pagination on each AJAX data request so that I can add the required values to the main collection and do so in a resource efficient manner?
System details

Operating System: Ubuntu
PHP Version: 7
Laravel Version: 5.2
Laravel-Datatables Version: 6.x
Database solution: PostgreSQL 9.4

Unfortunately, for query builder and eloquent engine, this is not possible because added columns are not part of the query. But if you can convert this added column using a database view then I think it should work.
Currently, there is no way to intercept pagination and even count queries.

Do you think this could conceivably work if I modified the original query to chain a variable number of subqueries into the SELECT statement?

Yes, i think it would work and I think I already did something like that before but not in a loop.
Something like below:
$query->addSelect(DB::raw("'hard coded value' as custom_field"));

Thanks! I'll give it a try!


Not compiling

Stuck with the error below

sig_block.c: In function ‘sig_block’:
sig_block.c:15:3: warning: ‘sigblock’ is deprecated [-Wdeprecated-declarations]
15 |   sigblock(1 << (sig - 1));
|   ^~~~~~~~
In file included from sig_block.c:3:
/usr/include/signal.h:173:12: note: declared here
173 | extern int sigblock (int __mask) __THROW attribute_deprecated;
|            ^~~~~~~~
sig_block.c: In function ‘sig_unblock’:
sig_block.c:27:3: warning: ‘sigsetmask’ is deprecated [-Wdeprecated-declarations]
27 |   sigsetmask(sigsetmask(~0) & ~(1 << (sig - 1)));
|   ^~~~~~~~~~
/usr/include/signal.h:176:12: note: declared here
176 | extern int sigsetmask (int __mask) __THROW attribute_deprecated;
|            ^~~~~~~~~~
sig_block.c:27:3: warning: ‘sigsetmask’ is deprecated [-Wdeprecated-declarations]
27 |   sigsetmask(sigsetmask(~0) & ~(1 << (sig - 1)));
|   ^~~~~~~~~~
/usr/include/signal.h:176:12: note: declared here
176 | extern int sigsetmask (int __mask) __THROW attribute_deprecated;
|            ^~~~~~~~~~
sig_block.c: In function ‘sig_blocknone’:
sig_block.c:38:3: warning: ‘sigsetmask’ is deprecated [-Wdeprecated-declarations]
38 |   sigsetmask(0);
|   ^~~~~~~~~~
/usr/include/signal.h:176:12: note: declared here
176 | extern int sigsetmask (int __mask) __THROW attribute_deprecated;
|            ^~~~~~~~~~
./choose cl trysgact hassgact.h1 hassgact.h2 > hassgact.h
./compile sig_catch.c
./compile sig_pause.c
sig_pause.c: In function ‘sig_pause’:
sig_pause.c:14:3: error: implicit declaration of function ‘sigpause’; did you mean ‘sig_pause’? [-Wimplicit-function-declaration]
14 |   sigpause(0);
|   ^~~~~~~~
|   sig_pause

i'm also unable to compile
sig_block.c:15:3: error: implicit declaration of function 'sigblock'; did you mean 'sig_block'? [-Wimplicit-function-declaration]    15 |   sigblock(1 << (sig - 1));       |   ^~~~~~~~       |   sig_block sig_block.c: In function 'sig_unblock': sig_block.c:27:3: error: implicit declaration of function 'sigsetmask'; did you mean 'sigset'? [-Wimplicit-function-declaration]    27 |   sigsetmask(sigsetmask(~0) & ~(1 << (sig - 1)));       |   ^~~~~~~~~~       |   sigset
although i'm not getting the depracated errors.

Also, on Arch AUR:
./compile sig_pause.c
sig_pause.c: In function ‘sig_pause’:
sig_pause.c:14:3: error: implicit declaration of function ‘sigpause’; did you mean ‘sig_pause’? [-Wimplicit-function-declaration]
   14 |   sigpause(0);
      |   ^~~~~~~~
      |   sig_pause
make: *** [Makefile:413: sig_pause.o] Error 1


Emphasize Learning in Documentation

Please describe the feature.
Each module should have a clearly defined intro, body, and outro. I noticed we are not quite clear on this.

Intro: Tell them what they are going to achieve.
Body: Show them what they are achieving.
Outro: Tell them what they have achieved.

This has long been completed.


[Rancher2] Update RKE CIS Benchmark 1.6 hardening guide and verification guide for 2.6x docs

Request Summary:
Forward port the changes in the PR in 2.6.
https://github.com/rancher/docs/pull/3902

@anupama2501 please see PR https://github.com/rancher/docs/pull/3953 with the fix. I applied the hardening specs and reran a CIS scan with rke-profile-hardened-1.6, that's why the benchmarked file changed. I also regenerated both PDFs.

@anupama2501 I believe that we can close this issue, as the fix is already in master.


Optimize deepcopy

While investigating performance issues with the REST Proxy API I took a profile using py-spy, the profiler shows the cost of deepcopy is significant for the produce endpoint. This PR replaces the use of copy.deepcopy with a faster version based on ujson, at the same time replaces all usages of json with ujson.
Using stdlib's deepcopy:

Using the ujson deepcopy:

The commit also contains a small timeit script to compare the performance of the two implementations.

I'd rather see import ujson as json to avoid gazillion of lines changed just for s/json/ujson/.

Their APIs are similar but not a drop in replacement, for instance ujson.dump doesn't accept fp as a keyword argument, but only a position argument. Using ujson is more explicit this.

Other deepcopy changes are in the test code, so not relevant to benchmarking.

True, I changed for consistency.

@jjaakola-aiven , the PR had conflicts and needed a rebase. A new review is needed.

The karapace/avro_compatibility uses json and not ujson.

Yes, because ujson can not be used there, at least I didn't find in the API the position of the parser error to implement the same behavior.


List sftim as en-localization reviewer

I'd like to join the list of English language reviewers.
I'm not a native speaker (I'm from the UK :wink:) but should still be able to help.

@sftim Absolutely! You also qualify as an approver, just so you know the offer is there... 😁
/lgtm
/approve


fix #587

Issue are caused by associating file numbering with sheet number.
Files (like comments#.xml) should be numbered as they appear.

Can you please upgrade your abapGit installation so w don't get so many changed files.

I accidentally compared the same versions of abapGit and consider that was using fresh one.


Feature/extend data classes

This PR adds the functionality to allow using the extends keyword to create a class based on another.
class: BaseClass
fields:
  fieldOne: String

class: SubClass
extends: BaseClass
fields:
  fieldTwo: String

This generates a SubClass that can be used like
SubClass(fieldOne: '...', fieldTwo: '...');

It is not yet possible to add a DB-Table to a base- or sub-class, as a DB-Class extends TableRow and we can't extend two classes. To achieve this like requested in #1618 we would need to make TableRow an interface which can be implemented by the class so the extends property is free to be used for inheritance. New Issue: #2711
connected to: #1873
Review advice
I do recommend going through the changes commit by commit. I tried to make them (somewhat) bite sized.
Pre-launch Checklist

[x] I read the Contribute page and followed the process outlined there for submitting PRs.
[x] This update contains only one single feature or bug fix and nothing else. (If you are submitting multiple fixes, please make multiple PRs.)
[x] I read and followed the Dart Style Guide and formatted the code with dart format.
[x] I listed at least one issue that this PR fixes in the description above.
[x] I updated/added relevant documentation (doc comments with ///), and made sure that the documentation follows the same style as other Serverpod documentation. I checked spelling and grammar.
[x] I added new tests to check the change I am making.
[x] All existing and new tests are passing.
[x] Any breaking changes are documented below.

If you need help, consider asking for advice on the discussion board.
Breaking changes
No breaking changes

Wow! Great work @BenAuerDev, really looking forward to getting this into Serverpod! 🚀

Comment:
I was not able to add a test covering the check for external models to inheritance_test.dart.
as we create the models for the test like
var modelSources = [
        ModelSourceBuilder().withYaml(
          '''
          class: Example
          fields:
            name: String
          ''',
        ).build(),
        ModelSourceBuilder().withFileName('example2').withYaml(
          '''
          class: ExampleSubClass
          extends: Example
          table: example_sub_class
          fields:
            age: int
          ''',
        ).build(),
      ];

and when used like:
var modelSources = [
      ModelSourceBuilder().withFileName('example2').withYaml(
        '''
          class: ExampleSubClass
          extends: ServerHealthMetric
          fields:
            age: int
          ''',
      ).build(),
    ];

the previous restriction
  if (baseClass == null) {
      return [
        SourceSpanSeverityException(
          'The class "$baseClassName" was not found in any model.',
          span,
        )
      ];
    }

is executed as every class not declared within var modelSources = [...]; is null.
But I tested it manually in the command line and the restriction is executed

I noticed a vulnerability with the import path of the parent class.
previously it was declared like:
p.joinAll([
    ...parentClass.subDirParts,
    '${parentClass.fileName}.dart',
]),

which works only if

the files are in the same folder

/models/child_class.spy.yaml

/models/parent_class.spy.yaml

OR

the parent file is in a sub folder relative to the child class within models

/models/child_class.spy.yaml

/models/extra_folder/parent_class.spy.yaml

It worked NOT if we have for instance

/models/inheritance/child_class.spy.yaml

/models/inheritance/parent_class.spy.yaml

the resulting import looked like: import 'inheritance/parent_class.dart' as _i1
So I use an absolute path to import the parent and added some tests checking each case

no sub dir
child in sub dir
parent in sub dir
both in sub dir


Support for --icmp-type and icmp6-type is missing

The following lines
-A FORWARD -p ipv6-icmp -m icmp6 --icmpv6-type 1 -j ACCEPT
-A FORWARD -p icmp --icmp-type 0 -j ACCEPT

produce errors
Error: failed to parse line, skipping rest " --icmpv6-type 1 -j ACCEPT" of the line: match modules "icmp6" is not implemented
Error: failed to parse line, skipping rest " 0 -j ACCEPT" of the line: unknown flag "--icmp-type" found

yes the extension icmp6 is not implemented, but contributions welcome of course.

-A FORWARD -p icmp --icmp-type 0 -j ACCEPT

is this missing a -m icmp6? Otherwise the flag --icmp-type shouldn't be allowed (at least to what I can see in the man pages of man iptables-extensions)

The support for -p icmp6 is currently not very important, but just for the record.
The -A FORWARD -p icmp --icmp-type 0 -j ACCEPT should be allowed (and is currently deployed on hosts) according to my iptables-extensions(8)
   icmp (IPv4-specific)
       This extension can be used if `--protocol icmp' is specified. It provides the following option:

       [!] --icmp-type {type[/code]|typename}
              This  allows specification of the ICMP type, which can be a numeric ICMP type, type/code
              pair, or one of the ICMP type names shown by the command
               iptables -p icmp -h

fixed with #17


Add support for user login using FusionAuth

Description
Since we at Samagra, rely heavily on using FusionAuth for user logins, we need to implement proper

[ ] auth guards in the /auth folder (similar to what we have done in all projects till now)

I think this has been fixed alongside #16 in #33.


Codestral doesn't output correct response

What is the issue?
Example:
>>> write a python function to calculate fibonacci sequence
[control_8][control_11][control_19][control_35][control_18][control_11]▅[control_20][control_20][control_20][TOOL_RESULTS][control_27][control_32][control_20][control_8][control_11][control_19][control_35][control_18][control_11]▅[control_20][control_20][control_20][TOOL_RESULTS][control_27][control_32][control_20][control_32][control_19][control_21][control_26][control_19][/TOOL_RESULTS][control_11][control_30][control_16][control_14]

>>> /bye

OS
macOS
GPU
Apple
CPU
Apple
Ollama version
0.1.39

I see the same error with codestral:latest

me too!

>>> 10 + 10
[control_34][control_23]▅[control_25][control_8][control_14][control_15]

same here

Same here
M1
Sonoma 14.5
16GB RAM
ollama version is 0.1.39

Works fine with (M2, 64 GB RAM, Sonoma 14.5)
`ollama run codestral

write a python function to calculate fibonacci sequence
Sure, here is a simple Python function that calculates the Fibonacci sequence up to the nth term:

def fibonacci(n):
    if n <= 0:
        return "Input should be positive integer."
    elif n == 1:
        return [0]
    elif n == 2:
        return [0, 1]
    else:
        sequence = [0, 1]
        while len(sequence) < n:
            sequence.append(sequence[-1] + sequence[-2])
        return sequence

You can use this function by calling it with the number of terms you want to generate as an argument, like so:
print(fibonacci(10))

This will print the first 10 numbers in the Fibonacci sequence.

10 + 10
The expression "10 + 10" does not have any relation to the previously defined function or the concept of the Fibonacci sequence. It's a simple arithmetic operation that adds two numbers together, and in
this case, it results in 20.`

Is it only a M1 issue then?

I think it's a problem with the amount of RAM.
v0.1.40 already has a fix: "Fixed out of memory and incorrect token issues when running Codestral on 16GB Macs"

Same here also
M1
Sonoma 14.5
16GB RAM
ollama version is 0.1.39
ZanMax - thanks for the update on 1.40 will try when I can.

MacBook Pro M1 Sonoma 14.4 16 GB RAM ollama version is 0.1.39
>>> 10 + 10
[control_34][control_23]▅[control_25][control_8][control_14][control_15]

I can confirm that 0.1.40 fixes the output issue on M1 with 16 GB RAM.
Be warned, in my experience its extremely slow most likely due to the limited RAM available.

Closing as this is fixed. Thanks for the help, all!


Credentials dialog does not offer PIN

#2966 called for

The prompt is updated to show device pin/smart card. However, the credential blob obtained from those will not work on other machines.

What does CredUnPackAuthenticationBuffer returns for the pin?


Using a custom fee is not working

Reported by user and I reproduced the issue. Using the test-app, the custom fee does not change the fee being applied in the app.
Images provided by arkadiko:

@fbwoolf awesome, that was fast! I'll try again when the wallet has been released 👀


use protected type_map from super class to support Rails 5.2 update

rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb in version 5.2 has encapsulated  the type_map variable.
https://github.com/rails/rails/commit/2ec207520389f4a6acb393be45bc73cc3815ad76

@winebarrel Could you please merge my PR :)

Thank you!


Create a basic validator for Sentinel’s “init command”s

I’m planning to overload vsecm-keygen docker container for it. and maybe even rename that docker container to vsecm-ctl (as now it’s doing more than one thing

vsecmctl --create-root-keys
vsecmctl --decrypt
vsecmctl --validate-init-command

or something along those lines.

Assuming it has k8s access, we can even make it directly kubectl exec VSecM Sentinel commands.


How to use the last activity data? Feature request?

Since the last update the data from last activities is collected in a single entity using attributes.
How can I retrieve this information?
Is there a way to iterate trough the 4 sets of data?
I'm interested in the distance & duration, but only for a specific activity type (running vs cycling). Main goal is to add these values together.
[Feature request]
I think it would be much easier to use this data when they are stored in different entities. Like last activity, second-to-last activity and so on. In that case is easier to use in automations (like do something when activity ID changes for last activity)

After some testing this is what I currently use to have a sensor holding the last running distance.
It's working but not perfect (unknown after reboot)
      garmin_last_distance:
        friendly_name: "Last Run Distance"
        unit_of_measurement: km
        device_class: distance
        #state_class: measurement > via customize.yaml
        value_template: >
          {% set current_value = state_attr('sensor.garmin_last_distance','activity_id')| float(default=0) %}
          {% set prev_value = states('garmin_last_distance')| float(default=0) %}
          {% set all_attr = states.sensor.last_activities.attributes %}
          {% for attr in all_attr['last_Activities'] %}
             {%- if loop.first and attr.sportTypeId==1 %}
                {% if attr.activityId != current_value and current_value > 0 %}
                  {{ ((attr.distance)/1000)|round(1) }}
                {% else %}
                  {{ prev_value }}
                {% endif %}
              {% endif %}
          {% endfor %}

Is there anyway to have a picture taken and added to the activity in Garmin connect show up as an attribute in this section? I can see in the attributes that it pulls my Garmin profile pic, so I'm assuming this is possible in some way?

Is there anyway to have a picture taken and added to the activity in Garmin connect show up as an attribute in this section? I can see in the attributes that it pulls my Garmin profile pic, so I'm assuming this is possible in some way?

It's currently unknown how to update it from the API unfortunately.


init script was wrong netdata-init-d*.in*

This is not an issue, it's how netdata called their initscript.
https://github.com/firehol/netdata/blob/master/system/netdata-init-d.in

the .in file has unresolved variables on line 15&16
DAEMON_PATH=@sbindir_POST@ PIDFILE=<EMAIL_ADDRESS>the netdata-init-d is resolved...
But I would say this task is not necessary anyway. netdata-installer.sh creates per init/systemctl the correct startup script /etc/systemd/system/netdata.service in my case.

Ok,, let's merge!
Don't hesitate if you feel some others edits are needed as didn't looked it for some time now.


Noise-Aware Simulator Cleanup

Description
This PR relates to this PR in the Simulator.
Checklist:

[ ] The pull request only contains commits that are related to it.
[ ] I have added appropriate tests and documentation.
[ ] I have made sure that all CI jobs on GitHub pass.
[ ] The pull request introduces no new warnings and follows the project's style guidelines.

Hey 👋🏼
#444 just landed (finally). This simplified quite some code throughout the DD package.
Could you please rebase your changes on top of the current main branch?
Happy to take a look then!

Ah, and before I forget it: could you please add an appropriate PR description here so that one can roughly understand what has been changed in this PR?
Once you pushed your changes and updated the description, this should be good to go!

Note that the Python failures here are unrelated to your changes. They stem from the recent Qiskit update to 0.46.0.
I will quickly create a PR that fixes them.

Note that the Python failures here are unrelated to your changes. They stem from the recent Qiskit update to 0.46.0. I will quickly create a PR that fixes them.

See https://github.com/cda-tum/mqt-core/pull/544. Once that is in, you should be able to merge that into your branch here and everything should be working again.

Note that the Python failures here are unrelated to your changes. They stem from the recent Qiskit update to 0.46.0. I will quickly create a PR that fixes them.

See #544. Once that is in, you should be able to merge that into your branch here and everything should be working again.

Thanks!

I believe the PR can now be merged.


Feature/iql

Seems like you are missing a version bump in the sdk.
And please remember to add the version to the changelog.


Issue building 32bit LTO'd dxvk with binutils 2.35.1

This is only on 32bit, and previously worked with 2.34
build.log
This is more of an FYI, I've reported the issue with Gentoo here https://bugs.gentoo.org/745372 and I'll try and find how to report it upstream with the binutils folk

https://sourceware.org/bugzilla/show_bug.cgi?id=26675

I've also been maintaining a winelib build, this continues to build, it's only the mingw build that fails

Anything we can do to work around that? The linker error doesn't really make all that much sense to me since it involves the C++ standard library.

The work around if to not use LTO, or go back to an older version of ld.bfd, also reverting this commit https://github.com/bminor/binutils-gdb/commit/041d1c2d4f03a134cb7574e4d62d2ce4418503ff fixes it

We aren't using LTO by default though?

Then you're fine, I'll update the title

I can confirm that this binutils commit fixes this.

Thanks, do we know when the fix will make it into a release?

According to https://sourceware.org/bugzilla/show_bug.cgi?id=26675#c19 it should already be in 2.36.


App crash when opening camera on iOS

App is crashing when opening camera on iPhone 7 with software update 12.1.2 with memory issue message:
Message from debugger: Terminated due to memory issue
Code snippet:
          navigator.camera.getPicture(function (imageData) {                     files.push(imageData);                 }, function (error) {                     alert("Taking picture failed: " + error);                 }, {                     quality: 30,                     destinationType: navigator.camera.DestinationType.DATA_URL,                     encodingType: navigator.camera.EncodingType.JPEG,                     correctOrientation: true,                     targetWidth: 640,                     targetHeight: 640                 });
I am using version 4.0.3

Ios camera is working?  I have issue in ios camera  its working on android


Download buttons don't respond in "mobile view" with multi-image posts

I noticed this issue on my Android running Firefox with Tampermonkey addon. The buttons don't respond when clicked on posts that include more than 1 image. DL buttons on posts with a single image works fine. DL buttons on stories work fine. DL buttons on Reels work fine. The only way to get the buttons to respond in a multi image post is to enable "view as desktop webpage" for the post. When the page reloads as the desktop version then the buttons are responsive.
I was able to reproduce this issue on my desktop windows machine running firefox by setting the user agent as firefox mobile and by narrowing the browser window so that Instagram uses the narrow mobile UI.

Thanks for your feedback. I will try to fix this. But the priority of this issue will be low due to this script being targeted to the desktop user.

I think i narrowed down the bug a bit further. This is happening in multi-image/vid posts. But mediaIdx becomes undefined in a call to getUrlFromInfoApi().
I verified with a kludgefix and the download button works again on multi-image/vid posts. I tried single video/image posts and that still works.

Good finding, I have uploaded ver. 1.16.1 based on your feedback. Please update and help to check if the new version works.

I updated and tried on a multi-image post... that works now on narrow mobile view.
However, I did notice a bug on a multi-item post that had both multiple images AND videos in a single post.


Make docs more maintainable

Description

Adds PyBaMM modules to index.rst using the .. automodule:: directive—the toctrees have been moved to their module-specific __init__.py files.

The aim of this PR is to make future updates to toctrees for classes and methods easier
Type of change
Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

[x] New feature (non-breaking change which adds functionality)
[ ] Optimization (back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)

Key checklist:

[x] No style issues: $ pre-commit run (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
[ ] All tests pass: $ python run-tests.py --all
[x] The documentation builds: $ python run-tests.py --doctest

You can run unit and doctests together at once, using $ python run-tests.py --quick.
Further checks:

[ ] Code is commented, particularly in hard-to-understand areas
[ ] Tests added that prove fix is effective or that feature works

Is this ready for merging?

Yes and no. I had planned to add sphinx.ext.autosummary and tried to do so, but it creates multiple issues with our documentation structure at the moment—to be discussed in our meeting this week. The PR in its current state is ready to merge, however, following a discussion.

Closing this PR in favour of keeping the documentation separate instead of having it in different places, plus the fact that adding sphinx.ext.autosummary brought up more issues with differences in folder structure such as how our API is defined for discretisations and spatial methods, models and submodels, and the expression tree classes and methods—it is possible that this can be revisited during discussions around #2427 and what it entails with our public-private API demarcation.


MS Graph Explorer - Teams - send channel message endpoint bug.

Describe the bug
A clear and concise description of what the bug is.
Within the Graph explorer I was attempting to send a channel message in a team that i am a member of. I followed the Endpoint requirements being the Channel-ID and the Group-ID along with verifying the default necessary headers to be able to process the request. After sending the request I receive the following error:
Bad Request - 400 - “teamId needs to be a valid GUID”
The expected behavior would be to send the channel message that i am able to view in my desktop app.
To recreate:

open MS graph API explorer
Log in
scroll down on the left hand side to "Microsoft Teams"
click the arrow on the left of the name
scroll down to POST "send channel Message" and click

to properly send a request a Channel-ID and Team-ID need to be passed in.
for the Team-ID

log into MS Teams desktop
navigate to teams on the left hand side
click on a team and the channels in that team should drop down
hover over a teams channel and click on the three dots on the right
click get link to address
for the Channel-ID are the numbers and letters that begins and includes "19%" and ends with (including) "thread.tacv2" or similar
for the Team-ID the numbers follow "groupId=" up untill "&"

placing these respective numbers in the endpoint link properly and clicking run will reproduce the problem.
i am using:
Apple Macbook Air
SONOMA 14.5
tested on Firefox and Chrome

Hi @Hari19J,
This seems to be an issue with the teams API. Please post this on the Microsoft Q&A forum where the appropriate team will be able to help


Add retries to cypress tests

This PR makes cypress retry a test if it ever fails due to unpredictable conditions in the CI.
https://docs.cypress.io/guides/guides/test-retries

I might be overthinking this, but one possible concern here is that successful retries could obscure underlying flakiness in the actual code. If we can, I wonder if we should also add something to our CI configuration that emails or otherwise notifies us that a test failed, even if it succeeded on a retry, so that we can detect patterns over time and address real randomly-occurring bugs.
That could certainly be something for another PR, though, and need not block this.

+1 that this could make us more prone to not noticing race conditions (or other kinds of random errors that could impact end users) but I'd say for now the randomly failing tests are a bigger problem. Also, we are not writing code at the moment thtat would be prone to race conditions.


Implement the Study Flashcards Page

Implement the study flashcards page according to the UI designs. Make the edit flashcard page link to and from the new study flashcards page

I'll work on this

This is now done


Security Requirement Consistency with Server Objects

In OpenAPI 3.0 server objects can be defined at three basic levels: 1) entire JSON document (global - all resources and operations), 2) a path (all operations on a resource), and 3) an operation [on a resource], with the more specific overriding more global definitions. Security Requirement objects can be defined/specified at two levels: 1) globally and 2) on an operation, but not for an entire path.
If a path can have one or more servers distinguished for it, then the security on a resource server MAY be different than others more globally defined and in-effect [through Swagger Editor, Swagger UI].  Thus, adding security object to  a path object will provide this flexibility in definition and use. It will provide a productive shorthand to specifying the same on each operation for the same reason that servers does.

One reason to provide multiple servers (or at least URLs) it that each server supports a different authentication protocol.  Even if we have Security Requirement objects at the Path level, we still cannot associate a security requirement with the server which implements that requirement.  Perhaps what we need is to add a security requirement property to the Server object.

@stevendearborn For consistency, I would agree that having SecurityRequirement objects at the path level would be logic.
@cmheazel Can you expand a little on why multiple servers that expose the same API would have different security requirements?  It's the same data and there is no reason why one server can't support multiple security schemes, so I'm not sure I understand the use case.

It comes down to access control and level of trust.  If you authenticate using HTTP basic, I would have a rather low level of trust that you are in-fact who you claim to be.  I would allow you to see only that data which would cause little damage if it is compromised.  If you authenticate using HTTPS with client certificates, then I have a high level of trust that you are who you say you are.  Therefore, you can have access to a much larger set of data.
While it is true that one server can support multiple security schemes, that doesn't mean that we should require it.

@cmheazel It's not about "requiring" that all servers implement the same security scheme.  It's a cost/benefit argument for adding the ability to limit some servers to only use some security schemes.   Adding stuff to the spec is cheap and easy.  But requiring all the tooling folk to support it is neither.

It's about being competitive in the market.  A complex spec adds cost to the tool developers.  An inadequate spec adds cost to the system developers.  A spec which can not meet (or is cost prohibitive to meet) the end-user needs doesn't get used at all.  IMO this particular enhancement does not make the cut at this time.  But it could in the future.

Throwing a +1 on this.
In our case, different environments use cookies with different names (e.g. auth-dev & auth-prod). Being able to specify the correct cookie name per server would be ideal.

Me as well. We have different oauth servers for different resource servers.

Stumbled on this issue while searching for a resolution to the same -- in our situation, we have different OAuth2 endpoints and security requirements for dev/stage/prod environments.


Login Page - Whatsapp login page there should be country code search box

Describe the bug

in whatsapp login page there should be a country code search box like contact page


Upgrade Go to 1.11 and ditch 1.8 and 1.9, and fix the CI (again)

Coverage remained the same at 66.971% when pulling f72ac3253435cd6e008967c5c99cba8ca73b6a91 on upgrade-go-fix-ci into 98b7e5826f587400bba11d63478c3add2d244a24 on master.


[test] Disable broken-modules test while we figure out what broke.

Tracked by rdar://problem/30287833.

@swift-ci Please smoke test and merge


Where to find user id?

Hello, After launching app (with the app id encoded) it's ask an user id..? Where I'm supposed to find it?

https://github.com/AgoraIO/Flutter-RTM/issues/44


[shaders] Standard shader artifacts on web

Weird artifacts with lighting on platform web... it could related to some clamp() on shader or maybe depth buffer being only 16 bit? Need some testing...

Running example can be checked here.

I have just opened the example in a MAC using Chrome and it displays fine like on desktop platform :O!
MAC info:
OSX The Captain (version 10.11.6)
iMac (Retina 5K)
CPU: 3,3 GHz Intel Core i5
RAM: 8 GB 1600 MHz DDR3
GPU: AMD Radeon R9 M290 2048 MB
Chrome version: Versión 53.0.2785.143 (64-bit)
@raysan5 Could it be because it uses an AMD GPU and the problem is with Nvidia GPUs?

Hi @victorfisac, thank you very much for the feedback!
I got the same issue with Intel GPU (didn't expect to work better than nvidia... :P), I think some of the maths should be simplified...

@raysan5 tested adding a abs() to every 'f' parameter in pow(f, e) in shader_standard.h and it fixes the artifacts issue :D! I let you apply the changes by yourself because I don't have my fork up to date. Congratulations!

Hi @victorfisac, I've just been working in this issue. Using abs(f) didn't solve the issue for me... After some investigation I find the problem. Specular value was not obtained correctly from texture; the following line needed to be changed:
if (useSpecular == 1) spec *= normalize(texture2D(texture2, fragTexCoord).r);

by this:
if (useSpecular == 1) spec = texture2D(texture2, fragTexCoord).r;

The reason is because texture2D already returns values normalized in range 0..1, that's exactly what we need. If you check specularity now, it has more sense following texture values (i.e. skulls have some specular level but not the skin).
Updated on commit https://github.com/raysan5/raylib/commit/97e3277d58060df96cd002b1377a51ca4adcbb9e
Still thinking this shader could be optimized... ;)


[stress testing] Update docs, resiliency for deploy script

Updates the stress test docs to use commands from the stress test deploy script instead of the multi-step manual commands.
Updates the stress test script to handle different az subscription contexts in all scenarios, and a bug fix for auto-installing the powershell yaml module.

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow

/check-enforcer override


Align default channel groupings with GA4 docs

Description & motivation
Align default channel groupings with GA4 docs. The following things have been changed:

Order of statements aligned with order in GA4 docs.
Refactored test: Included more test cases and combined input with the expected channel for easier readability.
Added campaign param to default_channel_grouping() method as it was needed for some mappings.
Included GA4 descriptions to make it easier to maintain.
Updated ga4_source_categories.csv based on PDF download from https://support.google.com/analytics/answer/9756891?hl=en#list
Removed the reference to https://github.com/llooker/ga_four_block_dev as I don't think their implementation is used anymore.

Also see issue #273.
Checklist

[ ✅] I have verified that these changes work locally
[ not applicable] I have updated the README.md (if applicable)
[✅] I have added tests & descriptions to my models (and macros if applicable)
[✅ ] I have run dbt test and python -m pytest . to validate existing tests

Great! Thanks for reviewing & merging it so quickly!


Error translating config: No translator exists for variant openshift with version 4.10.0

If I follow the instructions on https://docs.okd.io/latest/installing/install_config/installing-customizing.html#installation-special-config-chrony_installing-customizing and download the latest butane binary from https://mirror.openshift.com/pub/openshift-v4/clients/butane/latest/ (Mon Jan 31 12:41:48 2022) I receive "Error translating config: No translator exists for variant openshift with version 4.10.0" when I'm trying to create the yaml configuration for latest OKD version 4.10.0-0.okd-2022-03-07-131213:
cat 99-worker-chrony.bu
variant: openshift
version: 4.10.0
metadata:
  name: 99-worker-chrony
  labels:
    machineconfiguration.openshift.io/role: worker
storage:
  files:
  - path: /etc/chrony.conf
    mode: 0644
    overwrite: true
    contents:
      inline: |
        pool *** iburst trust
        pool *** iburst
        pool *** iburst
        driftfile /var/lib/chrony/drift
        makestep 1.0 3
        rtcsync
        logdir /var/log/chrony

./butane 99-worker-chrony.bu -o 99-worker-chrony.yaml
Error translating config: No translator exists for variant openshift with version 4.10.0

Update: With "version: 4.9.0" the generation of the .yaml file works fine.

This works for me.  Are you sure you're using the Butane binary you've downloaded?  What does ./butane -V say?

Okay, yeah, you aren't running the latest release.  Please re-download and try again.


Link to python tutorials yields 404

On this page the link to the python tutorials yields a 404.
https://mycroft-ai.gitbook.io/docs/skill-development/introduction

Hey, thanks for flagging this!
The link has been fixed up and we've added some new resources at the same time. If you know of anything else that you think would be helpful to include please let us know :)


Add travis

Cherry-pick of https://github.com/kubernetes/client-go/pull/87
cc @lavalamp

This change is

Thanks!


Suggestion for documentation

Category

[x] Completion

OS

[x] Mac OS X

Shell

[x] zsh

This isn't a bug, just a suggestion for the documentation. Right now, in the settings section, you mention a solution to use ag for _fzf_compgen_path and call out the fact that ag only works on files not directories. This is great because by default path and dir globbing in homedir take a long time on my machine with tons of files, and being able to use a .ignore would greatly improve that. However, it doesn't solve the issue for dir globbing.
I would propose adding the following function definition (or a comparable ag definition) to also work for directory completion:
_fzf_compgen_dir() {
	rg --hidden --files --null "$1" 2>/dev/null | xargs -0 dirname | awk '!h[$0]++'
}

I found the suggestion in ripgrep's issues and they proposed using | uniq instead of this awk command, but uniq requires duplicates to be adjacent to one another, which results in duplicate results. I tried prepending | sort which removes duplicates but makes the command take about 3x longer. Finally I stumbled on this awk snippet which doesn't require the duplicates to be adjacent, and has the happy benefit of actually being a decent bit faster than the | uniq approach.

Note: For anybody reading this, you need to use gdirname from the coreutils homebrew package on OS X instead of dirname. I used this block to determine which dirname to use:
if [[ $(uname) == "Darwin" ]]; then
	# dirname on OS X behaves funky, get gdirname via
	# brew install coreutils
	export dirname_command="gdirname"
else
	export dirname_command="dirname"
fi

I understand this dirname business is probably too much for the fzf documentation though

Thanks for the suggestion. You're right that _fzf_compgen_dir is not mentioned on the page and I should add it.
I like how you implemented _fzf_compgen_dir with standard utilities, it's simple and concise. But it's sad to see that simplicity is undermined by the inconsistency of dirname on macOS. I'll see if there are simpler options.
Note that ripgrep may not be a good choice until a new version is released. See https://github.com/BurntSushi/ripgrep/issues/200

I posted the same snippet into a ripgrep issue and somebody responded with a zsh approach which used the h modifier but unfortunately zsh would wait for the entire list to be generated before extracting the dirname which made it infeasible. Also it consumed a huge amount of memory.
Given that you currently use ag in the path function it seems reasonable to do the same for dir which will alleviate waiting on ripgrep to release. But I haven't found a good portable alternative to dirname/gdirname yet

@junegunn Check it out, this awk command behaves just like dirname only faster. If I were smarter at awk I bet the two awk statements could be combined but for now it runs just as fast as the original dirname/gdirname approach:
_fzf_compgen_dir() {
	rg --hidden --files . 2>/dev/null | awk -F'/[^/]*$' '{print $1}' | awk '!h[$0]++'
}

@chrisjohnson How about:
_fzf_compgen_dir() {
	rg --hidden --files . 2>/dev/null | awk -F'/[^/]*$' '!h[$1]++ { print $1 }'
}

Yep that's what I meant, that's what #awk suggested. But I'm currently investigating that the number of results that comes back is the same -- it seems like it does not. I need to wrap up some work but I'll paste results later

I think in your snippet this:
h[$0]

should have a $1, not a $0.

Yep typo on my paste here. I fixed that. But it looks like the logic to de-dupe dirname doesn't match up with other approaches. See gist: https://gist.github.com/anonymous/132daebf67acf0bacaf490377713ccfa
I get a different word count only for the 2 awk approaches that use awk to get the dirname

After diffing I can see it's because the awk snippet for grabbing the dirname doesn't handle filenames with no dir before them, so .foo/bar returns .foo but .foobar just returns .foobar.
Here's an approach that #awk helped me put together, which is fast and has the correct number of results. The downside is that it's a bit long.
_fzf_compgen_dir() {
	rg --hidden --files . 2>/dev/null | awk 'function dirname(fn) { if (fn == "") return ".";  if (fn !~ "[^/]") return "/"; sub("/*$", "", fn); if (fn !~ "/") return "."; sub("/[^/]*$", "", fn); if (fn == "") fn = "/"; return fn } {$0 = dirname($0)} !a[$0]++'
}

Given your thumb I made a PR with this final version


Add option to disable triple buffering

Instead of always preferring VK_PRESENT_MODE_MAILBOX_KHR over VK_PRESENT_MODE_IMMEDIATE_KHR, add an option to choose which is preferred (aka. triple buffering option).

Many thanks, and well realized! 🤠


Incompatible requirements introduced with v1.5.1 and packaging module.

Hi, the following PR has introduced a bug on Alpine systems as the updated dependency on packaging >= 23.0 tries to uninstall the pre-installed version 21.3 which errors with
ERROR: Cannot uninstall 'packaging'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Is there scope to change the minimum version here?
Steps to reproduce:
From alpine:latest
pip3 install semversioner
See error

@Telltak , the new version should fix your incompatibility issues. Alpine doesn't install python pip so it must be related to other libraries or modules you have installed in your container. In any case, the new version should fix the issues. Thanks for the report.

@raulgomis brilliant news! Thanks for looking into this so fast. We had begun pinning affected repos.
Re:alpine, my guess would be that this comes from their root fs, but that's one I'll take a deeper look at for my own knowledge!

Thanks for the report @Telltak ! 👌


sap.ui.model.odata.v2.ODataModel :: attachMetadataLoaded

We are running a check when the server is offline to validate services and found an issue.
When in chrome if you set the network as offline and you run a server call as below.  The metadata will fail as we would expect.  Now if we turn the network back online and try to hit the same call or any other call without reloading.  The code will hit the attachMetadataLoaded but never hit the oModel.create() command.  It is as if the metadata has been put in a state and will not let me make any other calls.
We are running 1.44.8 but I did not see any fixes for this issue in the newer versions.
var oModel = new sap.ui.model.odata.v2.ODataModel(oLink, false, null, null, { "Authorization": oAuth });
                    oModel.attachMetadataLoaded(null, function () {
                            oModel.create("dataSet", searchTerm, {
                            success: function (oData, oResponse) {
                            // run the success code
                            },
                            error: function (oError) {
                             // run the failed code.
                            }
                        });
                    });

Hello @vtebano13 ,
Thank you for sharing this finding. I've created an internal incident<PHONE_NUMBER>. The status of the issue will be updated here in GitHub.
Regards,
Gery

By the way this happens only on non SSL URL's which is weird.  Port 8000 doesn't work but port 443 seems to work with no issue.

Hi @vtebano13,
I'm having issues reproducing this. Could you please go into more detail about what is executed:

before the connection is lost (e.g. the model is created, the metadataLoaded listener is attached?)
after the connection is lost (e.g. just oModel.create?)
after the connection is restored ( e.g. just oModel.create again?)

Thanks,
Merlin
SAPUI5 Core Developer

Easiest way is to have a  button.  Put the code in the button.
While on-line hit the button and it will work.
Switch to off-line and hit the button and the meta with fail.
Switch back to on-line and hit the button and nothing will happen.

I tried using a depreciated function refreshMetadata which goes out and re-gets the metadata.  This works but since this depreciated I don't know how long it will be there.  If the metadata fails on the first load I need a way to go back and get it.  Do you have a solution for this?

Just to make sure I get this right: In my repro, step number 5 does not display "metadata loaded" for you?

because no metadataloaded function will get called once a failed occurs.

In case of errors metadataFailed should be fired instead of metadataLoaded. Do you expect something different?

It does show metadata loaded.  This is because you are running 1.48.8.
When I change the library to 1.44.8 I get the error I was describing.
So I think updating to a newer version may fix this.

Nope.  I think we have solved the issue.  Thank you again for all your help.


ProxyServer class should try to find a free port by itself if nothing is specified

On our build server it would be handy if browsermob finds a free port by itself.
Since we are using the public org.openqa.selenium.Proxy seleniumProxy()-Method we are not dealing with the chosen port anyway.
I think thats a small change here:
https://github.com/lightbody/browsermob-proxy/blob/master/src/main/java/net/lightbody/bmp/proxy/ProxyServer.java#L66
Would you accept that?
I found this snipped how to find a free port. But i don't really now what it implies...
    private int findFreePort()
    {
        try (ServerSocket socket = new ServerSocket(0))
        {
            socket.setReuseAddress(true);
            return socket.getLocalPort();
        }
        catch (IOException e)
        {
            throw new IllegalStateException("Can't acquire free port", e);
        }
    }

Best Regards,
Christian.

I'm going to close this since it is basically works-as-designed and @cowwoc listed a solution to the original problem. @d0x - Feel free to reopen if you're still having issues.


workbox-precaching integration test failures in Chrome Beta

Library Affected:
workbox-precaching
Browser & Platform:
Google Chrome v72
I initially thought this was limited to the Travis CI environment, but I can reproduce a test failure in Chrome Beta (v72) for the workbox-precaching integration suite locally on my Macbook:
  1) [workbox-precaching] Precache and Update
       should load a page with service worker :

      AssertionError: expected [ Array(2) ] to have the same members as [ Array(2) ]
      + expected - actual

       [
         "http://localhost:3004/test/workbox-precaching/static/precache-and-update/index.html"
      -  "http://localhost:3004/test/workbox-precaching/static/precache-and-update/styles/index.css"
      +  "http://localhost:3004/test/workbox-precaching/static/precache-and-update/new-request.txt"
       ]

      at Context.<anonymous> (test/workbox-precaching/integration/precache-and-update.js:109:36)
      at <anonymous>
      at process._tickDomainCallback (internal/process/next_tick.js:229:7)

It doesn't fail on Chrome Stable, so... we need to see whether some of the changes introduced in v72 led to that failure.

I'm no longer able to reproduce this failure in Chrome 72 (which is now stable).


remove includes - no polyfill

What has changed and why?
How to test this change
Test in IE11

[ ] has automated tests
[ ] at least one a reviewer ran the code
[ ] tested in small screen
[ ] tested in Chrome
[ ] tested in Firefox
[ ] tested in IE
[ ] tested in Safari
[ ] i18n
[ ] tested in screen reader/contrast <-- remove this if no UX change

closes #521


Modal improvements

features various touch improvements
fixes the issue with breadcrumbs being not reachable on huge trees
fixes a bug when interrupting plugin creation was asking for incorrect plugin type removal
fixes positioning issues with modal (going out of the screen)
Videos
old
http://d.pr/v/1lH09
new
http://d.pr/v/1ciFe modal
http://d.pr/v/17m8G breadcrumbs

great work, really nice & clean code :+1:

Coverage decreased (-0.2%) to 29.029% when pulling 436880ac3d402b871804d07b6146317d756da18d on FinalAngel:feature/modal-improvements into c8a47bff310b6c481f06d0c1fbcca796669fccfc on divio:preview.


User can still enter dashboard after logging out

As long as the token is valid (24hrs), the user will not be able to switch user despite them being logged out from gitLab

@CMDRyan This is expected. If you want to logout dashboard when gitlab logout, you will need to implement oidc front channel logout or back channel logout, which is way more complicated.


Added initial content of changelog

Pull Request Test Coverage Report for Build 72

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 70:
0.0%

Covered Lines:
63

Relevant Lines:
63

💛  - Coveralls

Pull Request Test Coverage Report for Build 72

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 70:
0.0%

Covered Lines:
63

Relevant Lines:
63

💛  - Coveralls

Pull Request Test Coverage Report for Build 72

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 70:
0.0%

Covered Lines:
63

Relevant Lines:
63

💛  - Coveralls


Failing tar tests

@benjdlambert I saw you most recently helped merge some of the changes related to these failing tests. Maybe you know who might be able to help debug the issue.
Expected Behavior

Tests should pass.
Current Behavior

Tests failing for:

"./packages/backend-common/src/reading/tree/TarArchiveResponse.test.ts"
"./packages/backend-common/src/reading/GithubUrlReader.test.ts"

Possible Solution

It seems the tests are using a mock file system? If this is working properly, then no suggestions for a solution yet. Alpine works a little differently, so, if it is using the physical "/tmp" directory, then it could be an FS issue somehow. There is a subpath check in TarArchiveResponse.dir(...) that is returning false on occasion that could point to the culprit. See log below for more details.
Steps to Reproduce

If you have Docker or Podman installed, you can clone "master" and follow these steps:
https://github.com/backstage/backstage/tree/master/contrib/docker/devops#prerequisites
However, you will need to explicitly run make check-tests as the tests are excluded from the default make target because of this bug.
The above steps locally relate to an Alpine docker image running root, where a couple of tests that use the file system under "/tmp" fail.

yarn install
yarn tsc
yarn build
yarn test

*I will finalize the docker image I am using in a PR to share with you, so that you can execute the exact same tasks.
Context

Here is the log output. I added some console.log(...) statements with "TEMP TEST FILE" to see what the tar.extract(... filter => ( ... )) was doing.
PASS src/middleware/statusCheckHandler.test.ts
PASS src/service/lib/config.test.ts
PASS src/database/sqlite3.test.ts
PASS src/middleware/notFoundHandler.test.ts
PASS src/reading/tree/ZipArchiveResponse.test.ts
PASS src/scm/git.test.ts
FAIL src/reading/tree/TarArchiveResponse.test.ts
  ● Console

    console.log
      TEMP TEST FILE: mock-main/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: mock-main/mkdocs.yml

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true mkdocs.yml

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: mock-main/docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: mock-main/docs/index.md

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true docs/index.md

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: mock-main/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: false  docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3306:19)

    console.log
      TEMP TEST FILE: mock-main/mkdocs.yml

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: false mkdocs.yml docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3306:19)

    console.log
      TEMP TEST FILE: mock-main/docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true docs/ docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: mock-main/docs/index.md

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true docs/index.md docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: mock-main/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE FILTER: false

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3328:19)

    console.log
      TEMP TEST FILE: mock-main/mkdocs.yml

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE FILTER: true

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3328:19)

    console.log
      TEMP TEST FILE: mock-main/docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE FILTER: false

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3328:19)

    console.log
      TEMP TEST FILE: mock-main/docs/index.md

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE FILTER: false

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3328:19)

  ● TarArchiveResponse › should extract entire archive into directory

    expect(received).resolves.toBe()

    Received promise rejected instead of resolved
    Rejected to value: [Error: ENOENT, no such file or directory '/tmp/backstage-7hLcMp/docs/index.md']

      115 |       fs.readFile(resolvePath(dir, 'mkdocs.yml'), 'utf8'),
      116 |     ).resolves.toBe('site_name: Test\n');
    > 117 |     await expect(
          |           ^
      118 |       fs.readFile(resolvePath(dir, 'docs/index.md'), 'utf8'),
      119 |     ).resolves.toBe('# Test\n');
      120 |   });

      at expect (../../../node_modules/expect/build/index.js:134:15)
      at Object.<anonymous> (reading/tree/TarArchiveResponse.test.ts:117:11)

PASS src/reading/FetchUrlReader.test.ts
PASS src/reading/BitbucketUrlReader.test.ts
FAIL src/reading/GithubUrlReader.test.ts (5.505 s)
  ● Console

    console.log
      TEMP TEST FILE: backstage-mock-etag123/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: backstage-mock-etag123/mkdocs.yml

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true mkdocs.yml

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: backstage-mock-etag123/docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: backstage-mock-etag123/docs/index.md

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true docs/index.md

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: backstage-mock-etag123/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: false  docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3306:19)

    console.log
      TEMP TEST FILE: backstage-mock-etag123/mkdocs.yml

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: false mkdocs.yml docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3306:19)

    console.log
      TEMP TEST FILE: backstage-mock-etag123/docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true docs/ docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

    console.log
      TEMP TEST FILE: backstage-mock-etag123/docs/index.md

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3288:17)

    console.log
      TEMP TEST FILE: true docs/index.md docs/

      at Unpack.filter (reading/tree/TarArchiveResponse.ts:3340:17)

  ● GithubUrlReader › readTree › creates a directory with the wanted files

    expect(received).resolves.toBe()

    Received promise rejected instead of resolved
    Rejected to value: [Error: ENOENT, no such file or directory '/tmp/docs/index.md']

      279 |         fs.readFile(path.join(dir, 'mkdocs.yml'), 'utf8'),
      280 |       ).resolves.toBe('site_name: Test\n');
    > 281 |       await expect(
          |             ^
      282 |         fs.readFile(path.join(dir, 'docs', 'index.md'), 'utf8'),
      283 |       ).resolves.toBe('# Test\n');
      284 |     });

      at expect (../../../node_modules/expect/build/index.js:134:15)
      at Object.<anonymous> (reading/GithubUrlReader.test.ts:281:13)

PASS src/reading/AzureUrlReader.test.ts (5.104 s)
PASS src/reading/GitlabUrlReader.test.ts (5.264 s)

Summary of all failing tests
FAIL reading/tree/TarArchiveResponse.test.ts
  ● TarArchiveResponse › should extract entire archive into directory

    expect(received).resolves.toBe()

    Received promise rejected instead of resolved
    Rejected to value: [Error: ENOENT, no such file or directory '/tmp/backstage-7hLcMp/docs/index.md']

      115 |       fs.readFile(resolvePath(dir, 'mkdocs.yml'), 'utf8'),
      116 |     ).resolves.toBe('site_name: Test\n');
    > 117 |     await expect(
          |           ^
      118 |       fs.readFile(resolvePath(dir, 'docs/index.md'), 'utf8'),
      119 |     ).resolves.toBe('# Test\n');
      120 |   });

      at expect (../../../node_modules/expect/build/index.js:134:15)
      at Object.<anonymous> (reading/tree/TarArchiveResponse.test.ts:117:11)

FAIL reading/GithubUrlReader.test.ts (5.505 s)
  ● GithubUrlReader › readTree › creates a directory with the wanted files

    expect(received).resolves.toBe()

    Received promise rejected instead of resolved
    Rejected to value: [Error: ENOENT, no such file or directory '/tmp/docs/index.md']

      279 |         fs.readFile(path.join(dir, 'mkdocs.yml'), 'utf8'),
      280 |       ).resolves.toBe('site_name: Test\n');
    > 281 |       await expect(
          |             ^
      282 |         fs.readFile(path.join(dir, 'docs', 'index.md'), 'utf8'),
      283 |       ).resolves.toBe('# Test\n');
      284 |     });

      at expect (../../../node_modules/expect/build/index.js:134:15)
      at Object.<anonymous> (reading/GithubUrlReader.test.ts:281:13)

Test Suites: 2 failed, 1 skipped, 20 passed, 22 of 23 total
Tests:       2 failed, 4 skipped, 150 passed, 156 total
Snapshots:   0 total
Time:        9.264 s
Ran all test suites.
error Command failed with exit code 1.

Your Environment

NodeJS Version: 14.16.0
Operating System and Version: Alpine Linux 3.11.8
Browser Information: n/a

I have updated the "Steps to Reproduce" with steps from "master".

Expecting that this is the same issues as https://github.com/backstage/backstage/issues/5004.
Although tbh these tests should probably be moved over to using mockfs.

Thank you for the suggestion @Rugvip , but @wyattwalter's suggestion to mount "/tmp" as a volume for the other issue did not work for me in this scenario.
The tests seem to be using mock-fs, but maybe they're not working quite as expected.
👉🏻  I modified the dockerized test command to include a volume mount for "/tmp" and the tests still fail. I also tried creating a new "/tmp" directory inside of the "Dockerfile" before it is mounted as the working directory during initial setup. ("/app" is the working directory when the tests run.) Out of curiosity, I tried installing the apt "tar" package too just in case there was a surprising dependency between the "tar" npm package... there isn't (well, that's good I suppose).
check-tests: build
	@docker run --rm -it \
		--name $(docker_name_timestamp_prefix)-$@ \
		--network host \
		-v $(project_root_dir_path):/app \
		-v /tmp:/tmp \
		-w /app \
		--entrypoint "" \
		$(docker_tag) \
		yarn run test:all

Partial Test Log
PASS src/reading/AzureUrlReader.test.ts (36.181 s)
FAIL src/reading/GithubUrlReader.test.ts (37.481 s)
  ● GithubUrlReader › readTree › creates a directory with the wanted files

    expect(received).resolves.toBe()

    Received promise rejected instead of resolved
    Rejected to value: [Error: ENOENT, no such file or directory '/tmp/docs/index.md']

      279 |         fs.readFile(path.join(dir, 'mkdocs.yml'), 'utf8'),
      280 |       ).resolves.toBe('site_name: Test\n');
    > 281 |       await expect(
          |             ^
      282 |         fs.readFile(path.join(dir, 'docs', 'index.md'), 'utf8'),
      283 |       ).resolves.toBe('# Test\n');
      284 |     });

      at expect (../../../node_modules/expect/build/index.js:134:15)
      at Object.<anonymous> (reading/GithubUrlReader.test.ts:281:13)

PASS src/reading/GitlabUrlReader.test.ts (37.62 s)

Summary of all failing tests
FAIL reading/tree/TarArchiveResponse.test.ts (18.956 s)
  ● TarArchiveResponse › should extract entire archive into directory

    expect(received).resolves.toBe()

    Received promise rejected instead of resolved
    Rejected to value: [Error: ENOENT, no such file or directory '/tmp/backstage-18kxaM/docs/index.md']

      115 |       fs.readFile(resolvePath(dir, 'mkdocs.yml'), 'utf8'),
      116 |     ).resolves.toBe('site_name: Test\n');
    > 117 |     await expect(
          |           ^
      118 |       fs.readFile(resolvePath(dir, 'docs/index.md'), 'utf8'),
      119 |     ).resolveerror Command failed with exit code 1.
lerna ERR! yarn run test --coverage exited 1 in '@backstage/backend-common'
lerna WARN complete Waiting for 5 child processes to exit. CTRL-C to exit immediately.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
make: *** [makefile:175: check-tests] Error 1

I believe I have found the area of the code issue, but not the root cause.
In the failing "TarArchiveResponse.test.ts" test.
  it('should extract entire archive into directory', async () => {
    const stream = fs.createReadStream('/test-archive.tar.gz');

    const res = new TarArchiveResponse(stream, '', '/tmp', 'etag');
    const dir = await res.dir();
    await expect(
      fs.readFile(resolvePath(dir, 'mkdocs.yml'), 'utf8'),
    ).resolves.toBe('site_name: Test\n');
    await expect(
      fs.readFile(resolvePath(dir, 'docs/index.md'), 'utf8'),
    ).resolves.toBe('# Test\n');
  });

The line const res = new TarArchiveResponse(stream, '', '/tmp', 'etag'); initiates the object with an empty subpath.
Now, if I call const files = await res.files(); instead of res.dir() and iterate through the files, I see both of the expected files "./mkdocs.yml" and "./docs/index.md" inside of the archive stream.
However, if I run const files = await fs.readdir(dir); after res.dir(), I only see the "./mkdocs.yml" file and NOT the "./docs/index.md" file.
*Note: you cannot run both res.files() and res.dir() in the same test execution without resetting the stream, so I ran it twice with each section commented out.

@Rugvip and @OrkoHunter, based on the above ☝🏻  discovery, I think either TarArchiveResponse.dir() should be returning both files when the subpath is empty, or the test should only expect the root "./mkdocs.yml" file in the response. Based on the test name, it seems dir() is supposed to extract all of the files, but maybe the call to tar.extract(...) is being filtered somehow?

TIP: I added the following make target to the contrib makefile in order to make the testing cycle much faster. The initial docker container build and package dependencies will still take a moment.
# execute changed tests only
.PHONY: check-changed-tests
check-changed-tests: install
	@docker run --rm -it \
		--name $(docker_name_timestamp_prefix)-$@ \
		--network host \
		-v $(project_root_dir_path):/app \
		-w /app \
		--entrypoint "" \
		$(docker_tag) \
		yarn run test

With this new target, you can cd contrib/docker/devops and run make check-changed-tests to see the failing tests. You can edit the typescript files and then just re-run to "quickly" see the results.

@ericis I have spent some time over the past few days encountering problems running these tests inside a Docker container myself. I discovered that they fail any time they are run as root.
The 'entire archive' test exercises the extraction of this fixture, which contains at its base a directory belonging to UID 501:
$ tar --numeric-owner -tzvf packages/backend-common/src/reading/__fixtures__/mock-main.tar.gz
drwxr-xr-x  0 501    20          0 27 Oct  2020 mock-main/
-rw-r--r--  0 501    20         16 27 Oct  2020 mock-main/mkdocs.yml
drwxr-xr-x  0 501    20          0 27 Oct  2020 mock-main/docs/
-rw-r--r--  0 501    20          7 27 Oct  2020 mock-main/docs/index.md

when the process UID of the test is 0, then in this tar.extract call the preserveOwner option defaults to true. Because of this, when the mock-main/ entry is being extracted here, then [DOCHOWN](entry) is true so the tempdir into which the fixture is being extracted actually gets its owner changed to UID 501.
[side note: I believe this is a bug in node-tar, because the equivalent sudo tar --strip-components=1 invocation does not change the owner of the working directory.]
Anyway, that's not a problem until later on, when the nested docs/ directory needs to get extracted. During this procedure, eventually node-tar calls out to fs.mkdir, which will be intercepted by mock-fs since the test has registered all of /tmp as a mock directory.
So mock-fs is given the instruction to create a directory inside the tempdir (which is owned by UID 501) -- first thing it checks to see if it has write access. It notices that the current process doesn't own the file, so this check fails and we fall into this branch.
Zooming out a bit, this parent directory was actually created/managed by mock-fs in the first place, when it intercepted this mkdtemp call. When it did so, it created this fake Directory object, which starts its life with 0777 permissions. This would be encouraging, but node-tar chmodded it down to 0755 earlier in the extraction process. So this error gets thrown.
[Another side note, less carefully considered: is this a bug in mock-fs? I vaguely thought root should be able to write any directory...]
So the mkdir call fails silently, since we're using the "streaming" version of extract (no file: parameter). That's why, when it comes to the assertion in the test, it can't find the docs/ dir -- it never successfully extracted in the first place!
I can think of three possible workaround here:

(least intrusive) make sure the tests are not running as root
change the permissions on the mock-main/ folder inside the tarball to be permissive enough to pass (this affects fixtures for some other tests, like the GithuUrlReader, GitlabUrlReader and BitbucketUrlReader ones)
add a preserveOwner: false option to the tar.extract call -- this feels pretty hacky as I'm not sure there are real-world situations where backstage behaves badly when run as root. The problem may only occur when running the tests.

Thanks @jamieklassen . This is fantastic detail! I will look into converting the developer docker walk through to use rootless docker images or at least mention in the documentation that they should run rootless.


"Using parameters in a class" tutorials having inconsistent node names

This is more of a nitpick, but the default names of the nodes in the parameter class tutorials are inconsistent. The C++ tutorials uses the name parameter_node and the Python tutorial uses the name param_talker.
I think the names should either be the same or follow the same pattern (i.e. both have the suffix _node or _talker).
Note, I noticed the inconsistency when adapting the launch file from the C++ tutorial that is referenced by the Python tutorial.
Links to tutorials:

https://index.ros.org/doc/ros2/Tutorials/Using-Parameters-In-A-Class-CPP
https://index.ros.org/doc/ros2/Tutorials/Using-Parameters-In-A-Class-Python

Based on foxy version:
In C++ the node name is "parameter_node"
In Python the node name is 'minimal_param_node'
I guess this is not a big issue, but agree to make them consistent


Cannot read property 'select' of undefined

{tsv: {matches: "smth"}} return "Cannot read property 'select' of undefined"

<EMAIL_ADDRESS>has been published to npm, which should resolve this.  Feel free to re-open if you're still having any problems.


Option for randomize or fix seed absent in some nodes.

Ive noticed that  the option to randomize or fix seeds on some usefull nodes are absent when added to the controller like seen in the image. Is that something that could be fixed? Thank you in advance for your reply and keep up the great work!!

You need to enable it in Settings

Check in the options - there is a setting to show or hide that control. If
that’s not the issue let me know!
On Tue, 10 Dec 2024 at 6:21 am, YVD @.***> wrote:

Ive noticed that the option to randomize or fix seeds on some usefull
nodes are absent when added to the controller like seen in the image. Is
that something that could be fixed? Thank you in advance for your reply and
keep up the great work!!
Screenshot.2024-12-09.201706.jpg (view on web)
https://github.com/user-attachments/assets/ebe6716a-e51d-4bfb-b335-bb266c3e9224
—
Reply to this email directly, view it on GitHub
https://github.com/chrisgoringe/cg-controller/issues/427, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABBLMA2TPLDAB4D22ECDUZT2EXUUFAVCNFSM6AAAAABTJQ664OVHI2DSMVQWIX3LMV43ASLTON2WKOZSG4ZDQMBSHA4DQNY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Check in the options - there is a setting to show or hide that control.

I like this option, but now that we can hide individual widgets, I wonder if it's still worth having it.

You need to enable it in Settings

Thank you!!

Check in the options - there is a setting to show or hide that control. If that’s not the issue let me know!
…
On Tue, 10 Dec 2024 at 6:21 am, YVD @.> wrote: Ive noticed that the option to randomize or fix seeds on some usefull nodes are absent when added to the controller like seen in the image. Is that something that could be fixed? Thank you in advance for your reply and keep up the great work!! Screenshot.2024-12-09.201706.jpg (view on web) https://github.com/user-attachments/assets/ebe6716a-e51d-4bfb-b335-bb266c3e9224 — Reply to this email directly, view it on GitHub <#427>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBLMA2TPLDAB4D22ECDUZT2EXUUFAVCNFSM6AAAAABTJQ664OVHI2DSMVQWIX3LMV43ASLTON2WKOZSG4ZDQMBSHA4DQNY . You are receiving this because you are subscribed to this thread.Message ID: @.>

Cheers!