Limit number of AUX files deltas to reduce reconstruct time

Problem
After commit [840abe395413508db40d0428e30f09343c051fed] (store AUX files as deltas) we avoid quadratic growth of storage size when  storing LR snapshots but get quadratic slowdown of reconstruct time.
As a result storing 70k snapshots at my local Neon instance took more than 3 hours and starting node (creation of basecbackup): ~10 minutes. In prod 70k AUX files cause increase of startup time to 40 minutes:
https://neondb.slack.com/archives/C03F5SM1N02/p1708513010480179
Summary of changes
Enforce storing full AUX directory (some analog of FPI) each 1024 files.
Time of creation 70k snapshots is reduced to 6 minutes and startup time - to 1.5 minutes (100 seconds).
Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.
[ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
[ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

[ ] Do not forget to reformat commit message to not include the above checklist

How do we write an automated test for this?

Hmm, why is it quadratic at basebackup? AFAICS you need to reconstruct the aux-files key-value pair only once, which is O(n) where n is the number of delta records since last image.
How is this different from Postgres updating a single page very frequently? Do we have the same problem there? We do have a performance test (test_hot_page.py) for that.

In the worst cases of the #6626, I recall that we had a 30MB logical size tenant with 2TB of physical storage.  If  we're making image layers every 1024 writes, then we would be re-introducing the storage bloat issue to some extent: that tenant would still end up with a multi-gigabyte physical size in spite of having a tiny logical size.
Writing out the periodic image value is only okay if we also have a limit on the number of aux files.  e.g. if there is just one slot, then it's painless to write an image value every few thousand writes.  If there are many thousands of keys in this map, then writing out those periodic image values re-introduces the space amplification problem.
To give examples: assuming a 100 byte size of kv pairs in the map:

A) Okay: a single key in the map, written 70,000 times: that's 69000 deltas and 1000 100-byte images, total size ~6MB
B) Not okay: 70,000 unique keys, each written once: that's 69000 deltas and 1000 images that are of size ~(70000 * 100 / 2), total size 23GiB.

The only way we can ensure we're in the (A) category is to have some robust limit on the number of keys.

Hmm, why is it quadratic at basebackup? AFAICS you need to reconstruct the aux-files key-value pair only once, which is O(n) where n is the number of delta records since last image.

Each step in reconstruction deserializes and reserializes the whole map (the working BufMut carries it from one delta to the next when applying.
It can be made O(N) by changing the reconstruction code to do something smarter like building up the deltas in the BufMut in some intermediate form, then building the actual map one time.  Or teach the reconstruction code to modify the binary encoding of a map in-place rather than decoding/encoding the whole thing.

How is this different from Postgres updating a single page very frequently? Do we have the same problem there? We do have a performance test (test_hot_page.py) for that.

It isn't, really.  We lean heavily on postgres being "reasonable" in the number of overwrites of the same page: i.e. that by the time the number becomes a problem, we should have built up enough L0s that we start compacting.  I imagine that in the case Konstantin is testing by hand, there isn't enough other stuff going on to generate a deep enough stack of L0s to let compaction do its thing.

Questions:

How do we write an automated test for this?

There is test_layer_bloating.py reproducing it with pg_log_standby_snapshot() function

Is 70K writes a healthy postgres behavior, or a pathological behavior?  If the latter, how do we put limits on it on the compute side?

I am not sure whether it is normal behaviour or not, but it quite easy to reproduce:

create subscription from one database
stop this database
create subscription from other database
wait for some time (70k*15sec = 12 days)

Hmm, why is it quadratic at basebackup? AFAICS you need to reconstruct the aux-files key-value pair only once, which is O(n) where n is the number of delta records since last image.

It is not quadratic at basebackup extraction - it just has to traverse and append 70k deltas=files. This is why it taks about 7 minutes.
It is quadratic on ingesting this files: to append N+1'st delta we need first to extract aux directory, i.e. apply all previous N deltas. This is why it is N*N and applying this 7Mb of WAl takes 3 hours.

How is this different from Postgres updating a single page very frequently? Do we have the same problem there? We do have a performance test (test_hot_page.py) for that.

There are two main differences:

Checkpoint periodical force creation of FPI restricting number of delta which ash to be applied. It is actually not necessary for Neon architecture to perform checkpoints at all (because we do not have problem with non-atomic page write), but may be it is still good idea to perform checkpoints because it limits reconstruct time.
Each update produce new version, so very soon you exhaust space on the page and need to allocate new one,.  Certainly Postgrs is able to perform some "micro vacuum" in =place in case of hot updates, but still it is impossible by updating the same tuple multiple times construct arbitrary long chain of WAL records (even if do not take in account compaction and image layer generation).

In the worst cases of the #6626, I recall that we had a 30MB logical size tenant with 2TB of physical storage. If we're making image layers every 1024 writes, then we would be re-introducing the storage bloat issue to some extent: that tenant would still end up with a multi-gigabyte physical size in spite of having a tiny logical size.
Writing out the periodic image value is only okay if we also have a limit on the number of aux files. e.g. if there is just one slot, then it's painless to write an image value every few thousand writes. If there are many thousands of keys in this map, then writing out those periodic image values re-introduces the space amplification problem.
To give examples: assuming a 100 byte size of kv pairs in the map:

A) Okay: a single key in the map, written 70,000 times: that's 69000 deltas and 1000 100-byte images, total size ~6MB
B) Not okay: 70,000 unique keys, each written once: that's 69000 deltas and 1000 images that are of size ~(70000 * 100 / 2), total size 23GiB.

The only way we can ensure we're in the (A) category is to have some robust limit on the number of keys.

Why 1000 images if are are saving one image per 1024 deltas? So for 70k snapshots there will be 70 images.
7070k100=0.5Gb. Not so small, but not so fatal. Especially taken in account that we in any case will try to avoid creation of such larger number of snapshots (PR #6739)

It is quadratic on ingesting this files: to append N+1'st delta we need first to extract aux directory, i.e. apply all previous N deltas. This is why it is N*N and applying this 7Mb of WAl takes 3 hours.

Oops. Can we fix that? You shouldn't need to reconstruct the previous image to create a new delta record.

It is quadratic on ingesting this files: to append N+1'st delta we need first to extract aux directory, i.e. apply all previous N deltas. This is why it is N*N and applying this 7Mb of WAl takes 3 hours.

Oops. Can we fix that? You shouldn't need to reconstruct the previous image to create a new delta record.

To be precise, we cache AUX_DIR in DatadirModification, so w have to extract only once when Timeline.begin_modification() is called. And it is called once per WAL segment (~128kb) received from SK. In my test (creation of snapshots using pg_log_standby_snapshot()  WAL consists just of snapshot records and and so it is done several hundreds times per 70k. But looks like it is enough to make PS consume 100% CPU for three hours. In more realistic scenario - when snapshot produced each 15 seconds is "hidden" in tons of other WAL records, we will really have to read and reconstruct directory for each uploaded snapshot.

Why 1000 images if are are saving one image per 1024 deltas? So for 70k snapshots there will be 70 images.
7070k100=0.5Gb. Not so small, but not so fatal. Especially taken in account that we in any case will try to avoid creation of such larger number of snapshots (PR https://github.com/neondatabase/neon/pull/6739)

Right, my maths was wrong there, but 500MB is still a 10x amplification on a near-empty tenant with 30MB logical size that enables logical replication.  I think the reality is worse: because we saw a tenant with logical size 30MB and physical size 2TiB, I anticipate that if we merge this change, that tenant's size will go to ~2GiB.  So yes, it's 1000x less, but it's still an enormous amplification, that's unacceptable for tenants that are on the free tier.

There is test_layer_bloating.py reproducing it with pg_log_standby_snapshot() function

Yes, but it passed before this PR (and also before #6742, right?), so  I don't think its assertions are covering the storage piece of this issue.

Is 70K writes a healthy postgres behavior, or a pathological behavior? If the latter, how do we put limits on it on the compute side?
I am not sure whether it is normal behaviour or not, but it quite easy to reproduce:
create subscription from one database
stop this database
create subscription from other database
wait for some time (70k*15sec = 12 days)

I should have asked a slightly different question: how many keys can a healthy system produce?  70k writes to the same key is fine, but 1 write to 70k keys is not.  I think we urgently need to move from uncertainty over number of keys to defining a specific maximum, and enforcing that in postgres: then we can reason about the worst-case behavior for the storage.
We need maximums for:

Number of keys
max size of key

Once we have that, we can make a well informed decision on whether the cost of a change like the one in this PR is acceptable -- I think that gives us a way to move forward with this change.

Yes, but it passed before this PR (and also before https://github.com/neondatabase/neon/pull/6742, right?), so I don't think its assertions are covering the storage piece of this issue.

This PR actually checks that huge image layers are not generated. It was fixed some time ago. And now with https://github.com/neondatabase/neon/pull/6742 it is not relevant at all. But after #6742 this test is not usable at all (as I wrote - it took 3 hours to PS to ingest WAL).

I should have asked a slightly different question: how many keys can a healthy system produce? 70k writes to the same key is fine, but 1 write to 70k keys is not. I think we urgently need to move from uncertainty over number of keys to defining a specific maximum, and enforcing that in postgres: then we can reason about the worst-case behavior for the storage.

I think you asking only about AUX files.
Right now we are storing AUX files of three kinds:

snapshots
slot state
replication origins
1 and 2 are produced together - once per 15 seconds. Bit only snapshots has unique names (this is why my proposal was to remove LSN from its name and store it inside value, but you solution with deltas seems to be more elegant ... although cause this reconstruction problems).
3 also uses the same name and produced less frequently.
If we support persisting some other data files, like postal statistic or pg_preewarm case state, then they are produced only at system shutdown.

So number of AUX files is expected to be very small and frequency of their generation - not so large.This is why I could not imagine this problem with snapshots:(

Number of keys
with #6739 already merged number of snapshots is limited by 300. So total number of keys can be estimated as few hundreds. But please notice that it doesn't limit number of deltas. If we do not periodically stored full directory, then we can still suffer from 40 minutes of preparing basebackup.

max size of key
It is more difficult to estimate. Right now sizeof typical snapshot is ~100 bytes. But in case of larger number of active transaction is can be larger. And if we are going to persist statistic to shared buffers state, then it can be almost arbitrary larger (but still I do not thin that it will be larger than few Mb).

Once we have that, we can make a well informed decision on whether the cost of a change like the one in this PR is acceptable -- I think that gives us a way to move forward with this change.

Having 40 minutes for node startup is already mont acceptable.
And I see only three ways to fix it:

Produce image layer more frequently when it is needed. It will be hard to impelement and it may increase write amplification and have negative impact on total system throughput.
Periodically write "FPI". As it is done now by Postgres checkpointer. This is what I tried to implement with this PR.
As I suggested before: remove LSN from file name and store it separately. In this case number of files will be small (~10) so no need to use deltas.

I just looked again at the change in this PR, and realized it's not emitting an image on every 1024 updates, but only when the key count crosses a multiple of 1024.  So if the size of the map is somewhat steady with keys added/removed, it won't emit any image values at all.
For this to work without carrying more state about the number of delta values since last image value, one could use some stochastic approach to statistically emit an image roughly every N writes.  The storage amplification could be bounded by using the size of the map to control how frequently we emit image values.
e.g. if someone has a 1000-value map, and emits images approximately every 1000 writes, then the approximate amplification in storage is 2x.
If we decided to tolerate a 10x amplification in storage (i.e. a 30MB free tier tenant takes 300MB storage, which is still unfortunate but at least bounded), then the calculation would be something like:
// Try to keep storarge amplification below this factor
const MAX_AMPLIFICATION_FACTOR = 10;
// Never bother emitting image values more often than this
const MIN_IMAGE_VAL_PERIOD = 1000;
let image_val_period = max(MIN_IMAGE_VAL_PERIOD, dir.files.len() * MAX_AMPLIFICATION_FACTOR);
if rand::threaD_rng().gen() % image_val_period = 0 {
 // emit image value
}

That's a huge kludge, but it would be a similar improvement to this PR's intent, while avoiding a storage amplification which is unbounded with the key count.

Let's leave the statistics aside for the moment and stay on aux files. The problem we have is that I read "in case of larger number of active transaction is can be larger" as meaning "unbounded". I don't know if postgres has a limit on the number of active transactions (maybe it doesn't?).

Number of active transaction is limited by number of connections. At it is now 100 by default and very unlikely that it will be set larger than 1000 (Postgres is not scaling well for large number of active transactions, so if it is necessary to support more active connections, it is better to use pgbouncer to map them to backends).

To build a robust system there needs to be some limit on this: otherwise we have to build a backend that can support an arbitrarily large collection of keys.

So, snapshot really can not be larger than few kilobytes.
With statistic situation is more obscure - I am going to investigate it. It should depends on number of relations but I am not sure.

My primary concern is to ensure any LR-related bugs don't affect other tenants (e.g. by consuming excessive disk space). So my bar for what's acceptable is different for folks who have enrolled in the beta vs. for the system as a whole.

I wonder what is the principle difference between LR bugs and any other bugs which may cause bloating of storage.
One example of such bug we have discusses in another ticker:stairs of layers which caused x30 amplification of storage space.

So my bar for what's acceptable is different for folks who have enrolled in the beta vs. for the system as a whole.

For us the most important thing is customer's feedback.
This incident with storage bloating cause a lot of troubles for us but is almost not noticeable for the customers (the fact that we have x100 write amplification is not visible for customers 0 they are not paying for resident size).
But 40 minutes startup is show stopper for customer: from his point fire it just means that Neon doesn't work.
So th priority here seems to be obvious.

I have slightly adjusted test_layer_bloating.py test by adding wait_for_flush_lsn.
Now it is failed at main and pass in this PR  illustrating the problem.

Okay, so can we robustly count on the 300 number? Like, if I put an assert!(dir.files.len() < 301) is that valid?

Limit https://github.com/neondatabase/neon/pull/6739 is a bit softy, i.e. several minutes might pass before it takes place. Also, it limits only .snap files, but there are also heap rewrite files; typically there are not many of them, but if really wanted they can be likely abused. Plus there are some more (slots, origins), there a few of them normally, but similarly this probably can be abused. So it makes sense to me to have some additional restriction on pageserver side of value which presumably should never be reached, e.g. 10k. Violating it should be error, not panic (assert). Ensuring hard boundary on PG side is much harder.


Support opening multiple terminal apps

I don't know if there is a way to do this already (i couldn't see one), but for half my projects I need to the windows power shell open and for the other half I need Cygwin's mintty. Can it be possible to create two different commands for each?

I could foresee us adding support by allowing people to pass in terminal to run_terminal and use a fallback for TerminalSelector.get(). In your usage, it would be a new keybinding/command to invoke OpenTerminal with the different terminal:
https://github.com/wbond/sublime_terminal/blob/1.4.0/Terminal.py#L115
{ "keys": ["ctrl+shift+t"], "command": "open_terminal", "args": {"terminal": "mintty"} },

@wbond Can we add the default set of GitHub labels? I want to mark this with a "help wanted" and "enhancement" label.

This has been resolved by @PinataMostGrim in #206 and released in 1.20.0


fix lint issue with PR #123

comment / help text changes to fix lint issue with PR #123

Merging without Pinterest review because I'm just fixing a lint issue from another contributor.


[eas-cli] Fix expoCommandAsync to support the new cli bin path

Checklist

[ ] I've added an entry to CHANGELOG.md if necessary. You can comment this pull request with /changelog-entry [breaking-change|new-feature|bug-fix|chore] [message] and CHANGELOG.md will be updated automatically.

Why
Since https://github.com/expo/expo/pull/21388 was merged we've been experiencing failures in the Publish Native Component List Website & Update workflow on expo/expo due to the eas update command failing to find the module 'expo/bin/cli.js'
E.g. https://github.com/expo/expo/actions/runs/4620498308/jobs/8170689606

How
This PR updates the expoCommandAsync function to resolve the CLI path from the new path, expo/bin/cli, using expo/bin/cli.js as a fallback
Test Plan
Run the cli locally

/changelog-entry bug-fix Update expoCommandAsync to support the new cli bin path


Mobile optimalization

IMPORTANT: Please do not create a Pull Request without creating an issue first.
I optimized the case mentioned in issue #496. Also, I changed the calculation of thumbnail position which is now clamped inside player. This PR should be heavily tested on different devices.
Proposed Changes

On mobile devices, touch events should not skip video on the background (if thumbnail previews are enabled)
Previews are clamped into player view, no overflowing from player element

Relevant issues
Closes #496

Vvv

``

Good

@Hassan92Wa

طلب

طلب

👈

Closing this PR since the issue is closed, @skallet if there is still a problem can you create a new issue and open a new PR? Thanks


client fails to build with rustc stable

error[E0554]: `#![feature]` may not be used on the stable release channel
  --> stf/src/lib.rs:19:1
   |
19 | #![feature(structural_match)]
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0554]: `#![feature]` may not be used on the stable release channel
  --> stf/src/lib.rs:20:1
   |
20 | #![feature(rustc_attrs)]
   | ^^^^^^^^^^^^^^^^^^^^^^^^

error[E0554]: `#![feature]` may not be used on the stable release channel
  --> stf/src/lib.rs:21:1
   |
21 | #![feature(core_intrinsics)]
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0554]: `#![feature]` may not be used on the stable release channel
  --> stf/src/lib.rs:22:1
   |
22 | #![feature(derive_eq)]
   | ^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to 4 previous errors

makefile seems not to specify the nightly version to be used.
workaround:
rustup default nightly-2020-02-05
rustup target add wasm32-unknown-unknown
rustup target add wasm32-unknown-unknown --toolchain nightly-2020-04-07

should be resolved with the rust_toolchain file


PDOStatement::execute(): SQLSTATE[HY093]

Hi
Just had to revert back to 1.0.4 because in 1.0.5 was getting
PHP Warning:  PDOStatement::execute(): SQLSTATE[HY093]: Invalid parameter number: number of bound variables does not match number of tokens in
any ideas

Hello,
i have the same problem.
Here is my test setup
$pdo = new \PDO('mysql:host=localhost;dbname=monolog', 'monolog', 'monolog');
        $mysqlHandler = new \MySQLHandler\MySQLHandler($pdo, "log", array('channel', 'HD_CORRELATION_ID', 'errorFile', 'errorFile', 'errorLine', 'errorTrace'), \Monolog\Logger::INFO);

        $this->logger->pushHandler($mysqlHandler);

        $content = [
            'test1' => 'aaa',
            'test2' => 'bbb'
        ];

        $this->logger->error('Test', $content);

When i debug hte script i find the query for mysql.
Query: INSERT INTO log (channel, level, message, time, channel, HD_CORRELATION_ID, errorFile, errorFile, errorLine, errorTrace) VALUES (:channel, :level, :message, :time, :channel, :HD_CORRELATION_ID, :errorFile, :errorFile, :errorLine, :errorTrace)
What is wrong ?
Your sincerly
Stephan

@TwinMist @StephanKrauss   I think I'll have time to check this out next week, please wait a little while.

Reverting commit 45a4bf3168c21b13272d16651f073a62bffd3cb0 fixes this for me.
For once, the check for unused keys is using a wrong array index:
https://github.com/waza-ari/monolog-mysql/blob/45a4bf3168c21b13272d16651f073a62bffd3cb0/src/MySQLHandler/MySQLHandler.php#L210
This should probably be if(! isset($contentArray[$context])).
But the check does not seem to work correctly anyway as the insert statement is build and prepared prior to the unused keys getting removed. Thus, the statement always contains all additional fields/keys.
By (trying to) remove them after the statement has already been prepared, the parameters won't match up.

I've addressed this issue in https://github.com/waza-ari/monolog-mysql/pull/47/commits/4eb4ad35a16ecf5e16701326fb6a419bedee4552. It will be resolved in the upcomming release v2.


Enhancement: show templateId value on template settings page.

It would be convenient for API developers if you provided a way in the online https://my.textmagic.com/online/messages/templates/settings for a user to see the templateId value for a template.  Since the API uses the templateId field, displaying that property from the UI would be helpful.

Yes, it will be implemented in web app. Right now you can use client.templates.list() to get ID of template.


⚠ Bofrost.fr has degraded performance

In 3217e41, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 2065 ms

Resolved: Bofrost.fr performance has improved in 26c60a4 after 6 minutes.


Implement REST endpoint for endpoints

First implementation done, but not pushed yet. I'm still bad at working with reactor (Flux and Mono). I need to go through the implementation and identify the typical scenarios that I have and then do some digging on how they are done properly.
Also, the idea to have a CurrentUserContextSupplier won't work in a reactive setup. ThreadLocals won't work in the reactive world. I have to use reactors context for that.


enhancement for --verbose description

Output of restic version
v0.15.1 on windows and v0.14.0 on linux (not yet upgraded to latest)
What should restic do differently? Which functionality do you think we should add?
nothing today; restic and rest-server are great!
What are you trying to do? What problem would this solve?
This is a minor item. Current description of --verbose is confusing. from the commandline:
-v, --verbose n                  be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
if you use --verbose 2 to a command this is the response:
PS C:\LocalData\tools\RestServer> restic.exe --verbose 2 --repo "C:\Users\nlv08564\OneDrive - Nexperia\MPD\Nexperia" --tag "C_H" backup --files-from c:\_RESTIC\scripts_windows\W10.restic.include.C_H.txt
unknown command "2" for "restic"

Did you mean this?
        ls

I would propose to change the description of verbosity to:
-v, --verbose                  be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
as the verbose=n is already in the details and --verbose by itself is also a valid option.
If you want I can attempt a pull request as it seems defined only in /doc/manual_rest.rst.
Did restic help you today? Did it make you happy in any way?
Restic was my saviour after a rocky start with Duplicati. Restic had a steeper learning curve for me but step by step i have continued to sasfely automate, monitor, and restore my backups. Thank you!

The documentation in /doc/manual_rest.rst is just a sample of the output of restic --help. I've made the change in the proper place in #4198. Thanks for the suggestion!

thanks @MichaelEischer,
next time i try to improve my search as https://github.com/restic/restic/issues/3926 is essentially the same issue.


fixes a couple typos in the README

I guessed at what might be the best wording, obviously change it as you see fit

Thanks for the improvements :)


Upgrade the user interface

What kind of upgrade? do you intend to just change the styles? or change the spring mvc into some kind of SPA framework?

I wanted to update the style to make it a little bit more "stylish". I have no intention on changing any framework. Just some plain CSS styling.
I have attached a screenshot on how it looks right now for me. I wanted to do some updates on the table as well. It is currently just a prototype. Do you have any suggestions or input?

Best regards,
Karl Dahlgren

I like the new change. I just wanted to offer help in case you were changing to angular.

Thank you @mhewedy.
I will keep that in mind. You are the first person I will contact if there were plans on changing to Angular :)
Best regards,
Karl Dahlgren


ChatCompletionClient interface change

I wanted to let you know that we just merged a small change to the ChatCompletionClient interface https://github.com/microsoft/autogen/pull/4856
The effect on your lib is minimal. To update your code to match the new expected parameter to the OpenAIChatCompletionClient you should update the following line:
https://github.com/vballoli/autogen-openaiext-client/blob/ba307ae541d3def5b9fcdc4d320685252c65c761/autogen_openaiext_client/client.py#L82
to:
model_info: Optional[ModelInfo] = None,

Then pass this value to themodel_info param of the parent.
Let me know if you have any questions!

I went to open a PR but noticed that it conflicts with a model_info parameter you currently have so wanted to let you decide how to handle.

Sorry missed this issue before archiving: I noticed that your PR provides support to pass all the OpenAI-LLM-call arguments directly through the ModelFamily and base_url args - which kind of makes this repo redundant (I could add a few classes that inherit the 0.4.dev13 OpenAIClient version that support all the other popular LLMs out there which could be useful to others to minimize efforts), but seems like largely your PR removes the necessity of this repo. Curious if you think the same

Personally, I think there is value in what you're doing here. That is maintaining model info and providing easy to use wrappers for people choosing specific providers.


Release of improved and refreshed mC4 multilingual corpus

Hi all,
one of the contributions of UNIMAX: FAIRER AND MORE EFFECTIVE LANGUAGE SAMPLING FOR LARGE-SCALE MULTILINGUAL PRE- TRAINING is releasing "an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages".
I couldn't find the corpus in this repo. Has the corpus been released yet?
Thanks for your help!
Cheers,
Stephan
PS: Great work! :)

you can find the newer mc4 corpus here: https://huggingface.co/datasets/allenai/c4/tree/mC4_3.1.0/multilingual

Thanks for the pointer, much appreciated!


Add option to use cpprest or json for moddern c++ (json.hpp)

I have a version of the nlohmann json for moddern C++, it would be nice if there really is a way to use that instead of cpprest's json implementation.
If possible prvide steps please (I perfer that json implementation for now that I already have as I do not have issues with it yet).

You can use anything you like.  Like the readme at https://github.com/Microsoft/libHttpClient/tree/master/Include/json_cpp says, its there for games that don't have a JSON parser that don't want to take a full dependancy on cpprest, but if you have a better JSON parser go for it.  libhttpclient itself doesn't do JSON parsing.  Its just provided as an extra side header.

Ah, ok I see thanks.


Minimize logging SDK dependencies to improve compile times

We want to lower the threshold for someone to use rerun or re_sdk to just log data to a file, and then open that file in app.rerun.io. That includes minimizing the amount of dependencies one has to build. If someone does cargo add rerun and sees 200 dependencies being pulled in, they may very well abort.
So: to use the SDK to only log to a file should be possible with a minimal set of dependencies; mostly just Arrow.
We should therefore feature-gate things like WebSocket communication (connect/spawn).
In the future we could add support for the native viewer streaming and .rrd file that is being written to by the SDK, thus still supporting spawn even without WebSockets:

https://github.com/rerun-io/rerun/issues/4056

cargo tree -p rerun -e normal is good tool for working on this, as is cargo build --timings -p rerun --no-default-features

cargo build --timings -p rerun --no-default-features produces this:

What the hell is egui doing there?

One important crate to focus on is re_sdk.
cargo tree -p re_sdk reveals some deps that should be removed:

serde
env_logger
backtrace
fixed
num-derive
comfy-table
similar-asserts
time
re_memory

build-deps:

cargo_metadata
sha2
walkdir


[Bug]: svg generation

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug
Whenever the svg button is clicked a svg file is downloaded. The file is blank when opened.

Expected behavior
The svg should contain the drawing.
As

Request:I have solved this issue please assign me the issue then I will send a pull request.
Add ScreenShots
No response
On which device are you experiencing this bug?
Chrome, Windows
Record

[X] I have read the Contributing Guidelines
[ ] I'm a GSSOC'24 contributor
[ ] I'm a SSOC'24 contributor
[X] I want to work on this issue

Please assign me this issue.


Named operations

It should be possible to specify a named operation
[Operation("name")]

This is done.


Run Github CI on Ubuntu 22.04

There is a newer hosted runner Ubuntu image for github-actions - ubuntu-20.04 -> ubuntu-22.04.

/lgtm


Avoid creating small objects when checking redraw flags

Change List

avoid recursively creating small objects in deck.needsRedraw
all clearRedrawFlags are now default to false

Coverage increased (+0.02%) to 58.803% when pulling a4e7e8fc0a55a71183733804998dddd78cd0d0fd on redraw-cleanup into 307b84790dea56ace44a28f5583b05750e6d7824 on master.

Coverage increased (+0.02%) to 58.803% when pulling a4e7e8fc0a55a71183733804998dddd78cd0d0fd on redraw-cleanup into 307b84790dea56ace44a28f5583b05750e6d7824 on master.


build error in Mac M1

hey guys, I tried to build our main branch and it throws many errors related to cpu architecture
error: no rules expected the token `aarch64_apple`

Rustup Version
❯ rustup -V                                                                  ─╯
rustup 1.24.3 (2021-05-31)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.59.0 (9d1b2106e 2022-02-23)`

Cargo Version
❯ cargo -V                                                                   ─╯
cargo 1.59.0 (49d8809dc 2022-02-10)

rustup default
❯ rustup default                                                             ─╯
nightly-aarch64-apple-darwin (default)

Full Error Log
❯ make build                                                                 ─╯
cargo build --features cli --bin bindle-server
   Compiling ring v0.16.20
   Compiling crypto-common v0.1.3
   Compiling digest v0.9.0
   Compiling block-buffer v0.9.0
   Compiling block-buffer v0.10.2
   Compiling inout v0.1.2
   Compiling regex v1.5.5
   Compiling flate2 v1.0.22
error: no rules expected the token `aarch64_apple`
   --> /Users/vishnujin/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/cpu.rs:257:13
    |
165 |     macro_rules! features {
    |     --------------------- when calling this macro
...
257 |             aarch64_apple: true,
    |             ^^^^^^^^^^^^^ no rules expected this token in macro call

error[E0425]: cannot find value `AES` in module `cpu::arm`
   --> /Users/vishnujin/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/aead/aes.rs:381:65
    |
381 |         if cpu::intel::AES.available(cpu_features) || cpu::arm::AES.available(cpu_features) {
    |                                                                 ^^^ not found in `cpu::arm`
    |
help: consider importing this constant
    |
15  | use crate::cpu::intel::AES;
    |

error[E0425]: cannot find value `PMULL` in module `cpu::arm`
   --> /Users/vishnujin/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/aead/gcm.rs:315:26
    |
315 |             || cpu::arm::PMULL.available(cpu_features)
    |                          ^^^^^ not found in `cpu::arm`

error[E0425]: cannot find value `ARMCAP_STATIC` in this scope
   --> /Users/vishnujin/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/cpu.rs:235:41
    |
235 |             if self.mask == self.mask & ARMCAP_STATIC {
    |                                         ^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find value `ARMCAP_STATIC` in this scope
   --> /Users/vishnujin/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/cpu.rs:288:36
    |
288 |     static mut GFp_armcap_P: u32 = ARMCAP_STATIC;
    |                                    ^^^^^^^^^^^^^ not found in this scope

   Compiling serde_derive v1.0.136
   Compiling tokio-macros v1.7.0
   Compiling futures-macro v0.3.21
   Compiling tracing-attributes v0.1.20
   Compiling rand_chacha v0.3.1
For more information about this error, try `rustc --explain E0425`.
error: could not compile `ring` due to 5 previous errors
warning: build failed, waiting for other jobs to finish...
error: build failed
make: *** [build-server] Error 101

Building in Rust Stable works fine

Hmm...we'll have to debug this. Looks like ring is having trouble with stuff, likely due to conditional compilation

Getting the same error. Would appreciate any pointers! :)

Is this a rust-core issue? https://github.com/rust-lang/rust/issues/95267

@yenicelik my guess is also the same, for now I can say bindle builds fine in Rust Stable
❯ rustup default stable                                                      ─╯
info: using existing install for 'stable-aarch64-apple-darwin'
info: default toolchain set to 'stable-aarch64-apple-darwin'

  stable-aarch64-apple-darwin unchanged - rustc 1.59.0 (9d1b2106e 2022-02-23)

❯ make build                                                                 ─╯
cargo build --features cli --bin bindle-server
    Finished dev [unoptimized + debuginfo] target(s) in 0.47s
cargo build --features=cli --bin bindle
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s

yeah rust stable worked indeed. maybe because of the rust update since Feb 24th or something

Thanks for pointing out that issue @yenicelik! @VishnuJin You ok to close this?

@thomastaylor312 yep !!


What to use instead of SDK?

In the document you say: "The deprecation of the Azure Multi-Factor Authentication Software Development Kit (SDK) has been announced. This feature will no longer be supported for new customers.". It's not clear what should be used instead of this SDK. Also, a link to the mentioned announcement would be helpful.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 38490d95-6f68-e94e-ea57-a53ceba08102
Version Independent ID: 390e8bfb-5b29-6369-8598-ec2576734536
Content
Content Source
Service: multi-factor-authentication

Thanx for the feedback!
I have assigned this issue to the author to investigate and update as appropriate.

@mzielinski-spartez The note about the deprecation was added by the PM in the following commit
https://github.com/MicrosoftDocs/azure-docs/pull/3902/commits/bd1e95520722d72a1beffa8cf25c7ef3d71def2b
@richagi might be able to provide some additional guidance regarding the deprecation.
The recommendation now that the SDK is no longer available to new customers is to write your app to support modern authentication or place it behind App Proxy if supported.
https://docs.microsoft.com/azure/active-directory/active-directory-application-proxy-get-started

#in-progress

@MicrosoftGuyJFlo,
do you know what's with MS own products, like:
https://docs.microsoft.com/en-us/microsoft-identity-manager/pam/use-azure-mfa-for-activation
ETA Early 2018... now?

@makauf What is your question? @billmath is the writer for that topic that you mention. He may be able to provide some additional insight.

@mzielinski-spartez I would like to go ahead and close out this issue if that's ok with you. Feel free to reach out if you have any further concerns.

@makauf  There is no ETA at the moment but I can confirm the devs are still working on this.  We will be updating the article as soon as we have something.  Thanks!

@billmath thanks for the info. it calms me to hear that there is still something in progress for that gap.
@MicrosoftGuyJFlo my questionin other words; Microsoft Identity Manager rely on the Azure MFA SDK (in special the authentication package) and because MIM use it in workflows, its not feasible to switch to modern auth nor app proxy .. as its not used in the publishing process... and i wanted to know when to expect some updates on this (as the MIM MFA integration artikel - link in my first comment) brings the statement for eraly 2018 .. what i think is right now ;)

@makauf ok glad we could get you the info you need. Thanks @billmath for responding.

#please-close

Any news of how we should handle MFA in custom apps? In our case we want to use the API to check the second factor on special actions. This page https://docs.microsoft.com/en-us/azure/active-directory/authentication/concept-mfa-licensing shows that with premium it's possible to do "MFA for on-premises applications". Any documentation?

Microsoft MFA / auth solutions "is what it is". All they want to do is to close real use cases, because they work in box in my opinon. I recently began to understand this more and more when I started to work with AWS SDK:s. Basically, realize there will not be any space for community developer or partners developers in Microsoft realm regarding auth. MFA is something that is only available for MS own products, its probable a strategy. But this strategy will probable hamper consumption in the end, community developer will use AWS and other cloud prividers that provide secure, but yet functional auth solutions..
AWS:


Emulate API Gateway payload in event object

We run our lambdas behind an API Gateway. The event object in the lambda when we run locally does not have the same structure as the event object when running behind an API Gateway.
For example, if I want to post to a path with the /{model} parameter, I need to post with this wrapping my payload
{
    "pathParameters": {
        "model": model
    },
    "body": { .. json payload ...}
}

When I call my API Gateway endpoint I do not need to wrap the payload, as this is created by API Gateway as it passes the payload off to the lambda.
What I would really like is to be able to set an ENV var such as LAMBDA_MODE which when set to API_GATEWAY RIE wraps the event to mimic the event passed by the API Gateway. It would need to work the same for response objects too. If not set, then it would work in the default mode which is how it works today.

I know I could solve this problem by running a second container that actually takes the API call on another port, wraps the payload and passes it on to RIE. This could be done by a container running nginx + lua

I built this project to solve this problem (partially): https://github.com/brefphp/local-api-gateway

This also exists : https://github.com/eagletmt/aws-lambda-rie-gateway
Hoping the project doesn't die

I solved it by writing a simple reverse proxy.
I send my requests to this proxy (running as localhost:8080). The proxy will read any request headers, request body, query string parameters, etc., then create a new request in the shape of API Gateway.
Once I have the new request that is shaped like API Gateway, I make a new request from inside the reverse-proxy to the RIE endpoint running in a Docker container (as localhost:9000).
On the way back, the reverse-proxy reads the RIE response (status code, response body), and constructs a new response in the shape of what the Lambda function would be sending back. Then the reverse-proxy returns that.
It's pretty lo-fi, but it works remarkably well.
But I agree — it would be great if AWS offerings were less like IKEA furniture, and more solution-oriented for the 80–90% of use-cases that are most common. But they're just not. GCP, Cloudflare, and others are far better at this than AWS is.


test: Devnet Alloc for local testing

Description
A clear and concise description of the features you're adding in this pull request.
Tests
Please describe any tests you've added. If you've added no tests, or left important behavior untested, please explain why not.
Additional context
Add any other context about the problem you're solving.
Metadata

Fixes #[Link to Issue]

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 70.39%. Comparing base (f208a8c) to head (b5827dc).

Additional details and impacted files
@@           Coverage Diff           @@
##           master      #55   +/-   ##
=======================================
  Coverage   70.39%   70.39%
=======================================
  Files          19       19
  Lines        2810     2810
=======================================
  Hits         1978     1978
  Misses        743      743
  Partials       89       89

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Export of PointerProperty when 'export extras' is enabled

I'm writing a small Blender addon, that works with PropertyGroups on Objects.
In Blender apparently these are being stored as extra properties on the Objects in a JSON format.
The gltf-exporter actually does propertly export these, when "export extras" is checked.
However, as soon as an Object has a value (a pointer) assigned to a PointerProperty, the extras are no longer exported at all.
I would prefer the export to work with PointerProperties.
(however It is not clear, how exactly these references are supposed to be handled)

The following quick fix simply converts all PointerProperties which are set to ID types (like Objects) to their name.
This is so far working for me, but won't be suitable in general.
Beginning of gltf2_create.py:
from bpy import types

class BlenderEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, types.ID):
            return obj.name

        return super(BlenderEncoder, self).default(obj)

def is_json(data):
    """
    Test, if a data set can be expressed as JSON.
    """
    try:
        json.dumps(data, cls=BlenderEncoder)
        return True
    except:
        import logging
        logging.exception("failed to json.dump custom properties.")
        return False

at the start of gltf2_export.py add:
from .gltf2_create import BlenderEncoder

and in gltf2_export.py replace line 96:
glTF_encoded = json.dumps(glTF, indent=indent, separators=separators, sort_keys=True, cls=BlenderEncoder)


Fix typo in README.md

relase => release

Hi @vaporwavy, closing this out since that section of the README has been removed in v3. Thanks for the PR 👍.


Remove unnecessary code duplication

I'm sure to understand why you duplicated this code in the js and jvm dir.
I think that you don't need to do that.
You can just put the code in the shared dir and it'll be compiled for the two platforms.
LMKWYT

Thanks so much!
So a couple things: Due to #164, I'm not running the tests for the JS backend in CI.
Also, you'll see that on the server version of UserDetail, there's a joinTime field which is an Instant, but on the JS side, it's String.
This is because java.time.Instant doesn't exist in js.
So I think if you run the tests through the browser, as described in #164, you'll see a "get a real java time" message thrown from circe's not-java-time package.
I'm not super happy with the way I've implemented it either, so I do think this is a good thing to improve. Thanks so much for trying it out.

Also, you'll see that on the server version of UserDetail, there's a joinTime field which is an Instant, but on the JS side, it's String.

Oh. Sorry I missed that 😅

Ah not at all, I didn't mean to suggest we abandon this, do you think it's an ok way to handle the difference in field types?

There's this project that can maybe help you: https://github.com/scala-js/scala-js-java-time

Or even this one which seems more maintained: https://github.com/cquiroz/scala-java-time

The second one looks good to me. Initially I was trying to avoid adding this, but you've already noticed complexity in this project that would be removed by adding this library, and future modules will undoubtedly benefit from elegant handling of timestamps. So I think adding that sounds great!

I merged in the end-to-end testing that just got done in #164 #166, so now we see the not-java-time error from deserializing into Instant on the js side.
So yes we should proceed with this approach and adding scala-java-time
[info]   Cause: scala.NotImplementedError: Get a real java.time

[info]   at $c_s_NotImplementedError.$c_jl_Throwable.fillInStackTrace__jl_Throwable(file:///home/travis/build/clovellytech/http4s-modules/modules/auth/client/js/target/scala-2.12/scalajs-bundler/test/auth-client-test-fastopt-bundle.js:33098:14)

[info]   at $c_s_NotImplementedError.$c_jl_Throwable.init___T__jl_Throwable__Z__Z(file:///home/travis/build/clovellytech/http4s-modules/modules/auth/client/js/target/scala-2.12/scalajs-bundler/test/auth-client-test-fastopt-bundle.js:33216:10)

[info]   at scala.NotImplementedError.<init>(file:///home/travis/build/clovellytech/http4s-modules/modules/auth/client/js/target/scala-2.12/scalajs-bundler/test/auth-client-test-fastopt-bundle.js:56361:58)

[info]   at java.time.package$.unimplemented(file:///home/travis/build/clovellytech/http4s-modules/modules/auth/client/js/target/scala-2.12/scalajs-bundler/test/auth-client-test-fastopt-bundle.js:5862:40)

@zakpatterson why the tests are green while there's an error?

With my last commit:
[info] - A login should create usable session *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: 5717 was not less than 5000
[info]     Location: (AuthEndpointsProperties.scala:76)
[info]     Occurred when passed generated values (
[info]       arg0 = UserRequest(xtpaeAwebwbpdeoyrwznzUapbcQtjvucr,bxyuxCoydeqamkxsniltogotkqidxarJmudwqzlx)
[info]     )
[info] - A bad password return 400
[info] ScalaTest
[info] Run completed in 16 seconds, 787 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0

Why failed 0 ? 🤔

With my last commit:
[info] - A login should create usable session *** FAILED ***
[info]   TestFailedException was thrown during property evaluation.
[info]     Message: 5717 was not less than 5000
[info]     Location: (AuthEndpointsProperties.scala:76)
[info]     Occurred when passed generated values (
[info]       arg0 = UserRequest(xtpaeAwebwbpdeoyrwznzUapbcQtjvucr,bxyuxCoydeqamkxsniltogotkqidxarJmudwqzlx)
[info]     )
[info] - A bad password return 400
[info] ScalaTest
[info] Run completed in 16 seconds, 787 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0

Why failed 0 ?

I think it's because of parallel execution. You're seeing a summary from some other test underneath that one, but the real summary is a bit lower in the build.

@zakpatterson why the tests are green while there's an error?

The script I put up is just not returning the correct exit value, i'll put up another branch and merge it into this one for testing.

Ok, I think the build is pretty solid now. Let me know what you think.

I think the only error was from not-java-time still being included. I need this scala-time business for my next feature, so I'm going to go ahead and merge this if it works. Thanks @guizmaii !


Add LengthMin and LengthMax constraint to Map

Add the ability to validate the size of a map with min and max like Yamale already can with lists and strings. This can be particularly useful for validating a non-empty map.

This looks good, thanks for the PR. If you can implement that extra test, I'll merge it in. Thanks!

Well, it seems that travis-ci.org is closed for business as of today. Guess I should have looked at this yesterday. :)
I never got a notification. It'll take a bit to get up and running with another solution....please be patient, it may be a while.

Alrighty, thanks!

We got our brand new github actions working, and your addition has made it into version 3.0.7. Thanks for your contribution!


CTest suite fails when run in parallel

It seems that running the CTest suite in parallel can produce errors, for example in this CI run with 3 parallel jobs:
51/85 Test #17: testparser/Scenario: Check parents in valid NMODL constructs ................................***Failed    0.71 sec

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
testparser is a Catch v2.0.1 host application.
Run with -? for options

-------------------------------------------------------------------------------
Scenario: Check parents in valid NMODL constructs
-------------------------------------------------------------------------------
../test/unit/parser/parser.cpp:188
...............................................................................

../test/unit/parser/parser.cpp:195: FAILED:
  {Unknown expression after the reported line}
due to unexpected exception with message:
  NMODL Parser Error : can not open file : Unit.inc [Location : 2.25-34]

Cannot delete temporary file: No such file or directory
Cannot delete temporary file: No such file or directory
-------------------------------------------------------------------------------
Scenario: Check parents in valid NMODL constructs
-------------------------------------------------------------------------------
../test/unit/parser/parser.cpp:188
...............................................................................

../test/unit/parser/parser.cpp:188: FAILED:
due to unexpected exception with message:
  NMODL Parser Error : can not open file : Unit.inc [Location : 2.25-34]

===============================================================================
test cases:  1 |  0 passed | 1 failed
assertions: 34 | 32 passed | 2 failed

I have also seen this locally. I suspect the problem is that multiple tests use a temporary file with the same name (Unit.inc).

Currently working.


wb component experimentation

Summary
Will experimentation in building new Lit Components -- still very much in progress
cc: @brian-patrick-3
ADO Story or GitHub Issue Link
Link here (if applicable).
Notes

Detailed change notes
can go here

To Do

Anything else
left to do

Checklist

[ ] Used Conventional Commit messages as outlined in the contributing guide.

[ ] Noted breaking changes (if any).

[ ] Documented/updated all props, events, slots, parts, etc with JSDoc.

[ ] Ran the analyze command to update Storybook docs.

[ ] Added/updated Stories with controls to cover all variants.
[ ] Ran test locally to address any failures.
[ ] Added/updated the Figma link for the Story's Design tab.
[ ] Added any new component exports to the src/index.ts file

Testing Instructions
Provide guidance to reviewers/testers here.
Screenshots
(if any)

Tagging all @kyndryl-design-system/ui-devs so everyone can take a look here and possibly benefit.


Pre populate the database when a new user is created

We must prepopulate the database whenever a new user signs up to the app.
First, we need to check if the information of the user exists or not.

We need to only prepopulate two models i.e.

The SitePlugin model
The SiteExclude model


Feature: fungible tokens

We'll probably need it someday
Staging: https://frontend-pr-957-d14z.onrender.com/
FT list: https://frontend-pr-957-d14z.onrender.com/contracts/fungible-tokens/

@luixo Consider marking this page with /beta URL and merge this PR to master


Tutorial (vignette) for analysis of a real dataset using celda

Right now the vignette in celda only has example for simulated data. For first-time users this is not enough when they are trying to analyze real data. For example data filtering is not included in the current vignette. A vignette showing how to run celda on real data should be provided. Maybe subset genes and a few cells from each cell type in PBMC 3K data and use it as example data.

Agree - the bottleneck with this tool is speed on most datasets, and the data needs to be subset either on genes or cells in order to run.
Would be useful if the authors could put out some opinions on best practice.

Thanks for the suggestions. We have added a tutorials for celda and decontX on our new website:
http://celda.camplab.net/
Specifically, analysis of the PBMC 3K dataset can be found here:
http://celda.camplab.net/v1.9.2/articles/articles/celda_pbmc3k.html


Feature Request: Provide an option to configure the presentation of Arrays

Suggestion

Provide an option to configure the presentation of Arrays.
🔍 Search Terms

Array generics
Array presentation
Array<T>
T[]

This suggestion is hard to search for ☹
✅ Viability Checklist

My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
[x] This wouldn't change the runtime behaviour of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
[x] This feature would agree with the rest of TypeScript's Design Goals.

⭐ Suggestion

I would like to see an option that will allow for altering the presentation of Arrays.
📃 Motivating Example

When reading through type errors or hovering over source code in VS Code, I find it hard to easily distinguish between Arrays, Tuples, Arrays of Tuples, Tuples with Arrays in them etc.
I personally use the above linting rules so that I know that when I see a [ or ] in my source code, I am explicitly dealing with a Tuple.
For example, I find Array<[Array<T>]> far easier to quickly parse than [T[]][]. This may simply be attributed to my bias of writing code in the generic form. Though, this gets worse with increased levels of nesting.
🎨 Prior art
The tslint rule array-type can be used to enforce using the generic form of Array<T>. Secondarily, this rule was also ported as the @typescript/eslint rule array-type.
💻 Use Cases

I find hovering over my own code and seeing it presented differently a bit frustrating. Consider the following playground screenshot:

In addition toarray-types options of "array", "generic", "array-simple", a fourth option of "preserve" could be added which would address the screenshot above.
I would like to suggest the following option:

Flag
Type
Default value

--arrayType
"array" | "generic" | "array-simple" | "preserve"
"preserve"

I'm definitely plus-one on this, although for code authored by the user I think it probably doesn't even need configuration, just show the type as authored rather than regenerating a presentation. i.e. If I write string[] show string[], if I write Array<string> show Array<string>.
Obviously this doesn't neccessarily work for library code, so an option would probably be necessary for that.


Clarify how to set HOMEBREW_GITHUB_API_TOKEN

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[ ] Have you successfully run brew tests with your changes locally?

It's currently unclear how to fix the GitHub API Token, I've attempted to clarify the output message. Open to feedback!
Unfortunately I'm unable to figure out why the tests aren't passing, would appreciate some guidance.

@JBallin how about using Utils::Shell.set_variable_in_profile here?

Great idea @MikeMcQuaid!

@JBallin brew tests seems to be failing.

@MikeMcQuaid, the tests pass locally. There's an issue with test-bot that seems to be affecting other PR's as well.
GitHub::search_issues returns Validation Failed. My best guess is that it's an issue with hitting the GitHub API on Homebrew's end? Should I open an issue about this?
Here's the failure output from test-bot:
2018-10-15T04:19:31.2064790Z Failures:
2018-10-15T04:19:31.2065400Z
2018-10-15T04:19:31.2066170Z   1) GitHub::search_issues queries GitHub issues with the passed parameters
2018-10-15T04:19:31.2067140Z      Failure/Error: results = subject.search_issues("brew search", repo: "Homebrew/brew", author: "avetamine", is: "closed")
2018-10-15T04:19:31.2067900Z
2018-10-15T04:19:31.2068610Z      GitHub::Error:
2018-10-15T04:19:31.2069340Z        Validation Failed
2018-10-15T04:19:31.2071170Z      # ./utils/github.rb:237:in `raise_api_error'
2018-10-15T04:19:31.2071890Z      # ./utils/github.rb:194:in `open_api'
2018-10-15T04:19:31.2073140Z      # ./utils/github.rb:309:in `search'
2018-10-15T04:19:31.2074730Z      # ./utils/github.rb:242:in `search_issues'
2018-10-15T04:19:31.2076060Z      # ./test/utils/github_spec.rb:34:in `block (3 levels) in <top (required)>'
2018-10-15T04:19:31.2076720Z      # ./test/spec_helper.rb:134:in `block (2 levels) in <top (required)>'
2018-10-15T04:19:31.2077540Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
2018-10-15T04:19:31.2078190Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
2018-10-15T04:19:31.2078840Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
2018-10-15T04:19:31.2079480Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
2018-10-15T04:19:31.2080080Z      # ./test/spec_helper.rb:97:in `block (2 levels) in <top (required)>'
2018-10-15T04:19:31.2080690Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
2018-10-15T04:19:31.2081310Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
2018-10-15T04:19:31.2081950Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
2018-10-15T04:19:31.2082580Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
2018-10-15T04:19:31.2083250Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:37:in `block (2 levels) in setup'
2018-10-15T04:19:31.2083890Z      # ./vendor/bundle/ruby/2.3.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block (2 levels) in <top (required)>'
2018-10-15T04:19:31.2084020Z
2018-10-15T04:19:31.2084120Z Finished in 3 minutes 15.5 seconds (files took 2.51 seconds to load)
2018-10-15T04:19:31.2084230Z 507 examples, 1 failure
2018-10-15T04:19:31.2084300Z
2018-10-15T04:19:31.2084380Z Failed examples:
2018-10-15T04:19:31.2084450Z
2018-10-15T04:19:31.2084560Z rspec ./test/utils/github_spec.rb:33 # GitHub::search_issues queries GitHub issues with the passed parameters

Thanks again @JBallin!


SendAttachment samples should illustrate upload and base64 data URIs

https://github.com/Microsoft/BotBuilder-Samples/tree/master/Node/core-SendAttachment
https://github.com/Microsoft/BotBuilder-Samples/tree/master/CSharp/core-SendAttachment
These samples do a good job on the attachment side, but don't illustrate upload.
They should show:

Uploading via the attachment API (https://docs.botframework.com/en-us/restapi/connector/#!/Conversations/Conversations_UploadAttachment)
Including an attachment inline as a base64 data URI (example here, search for "base64": https://docs.botframework.com/en-us/core-concepts/attachments/)

Thanks @dandriscoll. We applying these enhancements and they will be ready to be reviewed later today.

@dandriscoll this is already merged. Feel free to close the issue if the updates are ok for you.

These are great! Thanks @ejadib and @pcostantini


bot.python.py

@theteerapon could you please describe problem description?

I'm closing this ticket because it contains no information and appears to be inactive.


Add temporary debug prints to identify devicelab failures

This will be reverted after the tests have run. There's a failure in the device lab tests but we're not getting the output because the completion is failing (because the future is already completed). This just prints the exception that's occurring and also the other locations that are completing (so we know why it's trying to complete twice).

Landing on red to try and get a better error message for the failures. This change will be reverted after the next run.


chore: update prettier targets

:tada: This PR is included in version 1.0.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Made Clock 9

I attempted to make Clock 9. :smile:
Thanks to @FantasyTeddy for the fonts, and drawText function. Although I added a size parameter to it

This is for the clock made of mini-clocks, right? I think I managed to get the numbers the wrong way round for the comments in clocks 9 and 10. Apart from that, it's a good start to the clock - I'll fic that numbering issues.
I might prefer to make a list of collaborators for each of the clocks rather than having author names at the top of the files - would you be okay if I were to remove your comment at the top after getting that sorted out?

Oh. I never knew those dots were supposed to be smaller clocks. Couldn't see it in the gif. I will begin making it like that.

Yeah - somewhat confusingly as well the posts' gifs are numbered badly (Hence my numbers) You can see a fuller size of the clocks to look at here: http://cmuems.com/2016/60212/wp-content/uploads/2016/09/maeda-10.gif

@haideralipunjabi I merged this with some formatting tweaks, let me know if you're working on the clocks!

@meiamsome Yeah I am working on them, might take a day or two


Symbolic speedup

Minor tweaks to improve the speed of evaluate that is quite slow going through all the nobs of sympy. Bypasses a lot of it via direct call to _subs and avoid very expensive evalf on expression while only needed on the coefficients.
timing improvement for 12th order 3D on my laptop for this branch
In [1]: %timeit u.laplace.laplace.evaluate
1.04 s ± 71.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [2]: %timeit u.laplace.laplace.evaluate
1.06 s ± 132 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

vs master:
In [1]: %timeit u.laplace.laplace.evaluate
6.23 s ± 1.02 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [2]: %timeit u.laplace.laplace.evaluate
7.16 s ± 2.23 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: %timeit u.laplace.laplace.evaluate
5.56 s ± 268 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

And aside, setup.cfg updated for flake8-nb if someone wants to play with it but not adding it to CI.

, I got a question: why do we override _subs instead of _eval_subs, which seems to be the stub we need here?

aresame  lot more expensive than the old is new in the one here.

merged


How can we have all errors in a single response?

Hello, I'm trying to figure out if this is included (or if planned for later on) where we get all errors in a single JSON response.
For example, if I were to send a bad POST request with two blank fields, I'd only get an error for one of them:
{
    "type": "validation_error",
    "code": "blank",
    "detail": "This field may not be blank.",
    "attr": "FIELD_ONE"
}

How can we get something like:
errors: [
    {
        "type": "validation_error",
        "code": "blank",
        "detail": "This field may not be blank.",
        "attr": "FIELD_ONE"
    },
    {
        "type": "validation_error",
        "code": "blank",
        "detail": "This field may not be blank.",
        "attr": "FIELD_TWO"
    }
]

It really helps with sign up forms for example, since the user gets to see all failing fields the first time, instead of fixing them one by one every failing request.

Hey! That's a cool idea, but I'm sure if it would work since I think DRF raises an error at first sight of a deserialization issue – though @paolodamico may know more.

Yeah... I only suggested that because the default DRF exception handling returns a list of all de/serialization errors.

Hey @duciito! That's a great point, I'll give it some thought over the weekend and we'll get back to you.

Any changes on this topic??

Hey @davidcode43w! I'm really sorry about the delay here, we've been swamped with other stuff and haven't had a time to jump at this, will probably need a couple more weeks before I can push an improvement here but any contributions are highly welcomed!


Update writer.py

fixed typos

Will leave this as-is actually! Some of these decrease performance (e.g. getting rid of "urls")
You're welcome to update these in your local installation though - that's the idea behind templates!


cargo-binstall: 1.6.9 -> 1.10.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/cargo-bins/cargo-binstall/releases.
meta.description for cargo-binstall is: Tool for installing rust binaries as an alternative to building from source
meta.homepage for cargo-binstall is: https://github.com/cargo-bins/cargo-binstall
meta.changelog for cargo-binstall is: https://github.com/cargo-bins/cargo-binstall/releases/tag/v1.9.0
Updates performed

Rust version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.9.0 with grep in /nix/store/q9niiyfyj1nsaynq02hbxmff9vk3jy06-cargo-binstall-1.9.0
found 1.9.0 in filename of file in /nix/store/q9niiyfyj1nsaynq02hbxmff9vk3jy06-cargo-binstall-1.9.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

cargo-binstall

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/q9niiyfyj1nsaynq02hbxmff9vk3jy06-cargo-binstall-1.9.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A cargo-binstall https://github.com/r-ryantm/nixpkgs/archive/3963a302ac699a3701446c998279a9779749a8d8.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/3963a302ac699a3701446c998279a9779749a8d8#cargo-binstall

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/q9niiyfyj1nsaynq02hbxmff9vk3jy06-cargo-binstall-1.9.0
ls -la /nix/store/q9niiyfyj1nsaynq02hbxmff9vk3jy06-cargo-binstall-1.9.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

cargo-binstall

Maintainer pings
cc @figsoda for testing.

Add a :+1: reaction to pull requests you find important.

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/cargo-bins/cargo-binstall/releases.
meta.description for cargo-binstall is: Tool for installing rust binaries as an alternative to building from source
meta.homepage for cargo-binstall is: https://github.com/cargo-bins/cargo-binstall
meta.changelog for cargo-binstall is: https://github.com/cargo-bins/cargo-binstall/releases/tag/v1.10.2
Updates performed

Rust version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.10.2 with grep in /nix/store/xmja247bn5pfxzsjnj4yx7yqdzw254xb-cargo-binstall-1.10.2
found 1.10.2 in filename of file in /nix/store/xmja247bn5pfxzsjnj4yx7yqdzw254xb-cargo-binstall-1.10.2

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

cargo-binstall

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/xmja247bn5pfxzsjnj4yx7yqdzw254xb-cargo-binstall-1.10.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A cargo-binstall https://github.com/r-ryantm/nixpkgs/archive/c834ad683c1055bb1053508b8ee0acdee2fad1d7.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/c834ad683c1055bb1053508b8ee0acdee2fad1d7#cargo-binstall

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/xmja247bn5pfxzsjnj4yx7yqdzw254xb-cargo-binstall-1.10.2
ls -la /nix/store/xmja247bn5pfxzsjnj4yx7yqdzw254xb-cargo-binstall-1.10.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

cargo-binstall

Maintainer pings
cc @figsoda for testing.

Add a :+1: reaction to pull requests you find important.


Make the use of accept-filters conditional.

We had a recent discussion about bad interactions between accept-filters and misbehaving clients in NetBSD. The patch makes the use of accept filters conditional under an option, the default behavior doesn't t change.

Patch looks reasonable.
I added a comment inline in the diff for something that needs to be fixed.
Would you mind also changing use-acceptfilter to bsd-acceptfilter? (and use_acceptfilter to bsd_acceptfilter).  Thanks.

Hmmm.  Giving this another look.  Why not re-use 'server.defer-accept' to enable/disable socket option TCP_DEFER_ACCEPT (Linux), or socket option SO_ACCEPTFILTER "httpready" (BSD)?  That would avoid adding another directive.

Would the following work for you?  (Might need to make one additional change to enable s->defer_accept on BSD in order to preserve existing behavior of having "httpready" SO_ACCEPTFILTER enabled by default.  However, if that is not robust and not recommended by *BSD developers due to various client incompatibilities, then let's change the default, too.)
diff --git a/src/network.c b/src/network.c
index 19364ea..43e6af3 100644
--- a/src/network.c
+++ b/src/network.c
@@ -441,14 +441,13 @@ static int network_server_init(server *srv, buffer *host_token, specific_config

                goto error_free_socket;
 #endif
-#ifdef TCP_DEFER_ACCEPT
        } else if (s->defer_accept) {
+#ifdef TCP_DEFER_ACCEPT
                int v = s->defer_accept;
                if (-1 == setsockopt(srv_socket->fd, IPPROTO_TCP, TCP_DEFER_ACCEPT, &v, sizeof(v))) {
                        log_error_write(srv, __FILE__, __LINE__, "ss", "can't set TCP_DEFER_ACCEPT: ", strerror(errno));
                }
 #endif
-       } else {
 #ifdef SO_ACCEPTFILTER
                /* FreeBSD accf_http filter */
                struct accept_filter_arg afa;

Renaming to bsd_acceptfilter is something I'm ambivalent about.
TCP_DEFER_ACCEPT is quite different and much more limited. I'm not sure if mixing the two things is really an improvement.

Rather than getting side-tracked with platform-specific comparisons, let's focus first on the functional need addressed by this patch: you stated that there is a need to have an option for enabling/disabling "httpready" SO_ACCEPTFILTER.  I agree.
What is the general opinion of BSD core developers about enabling "httpready" SO_ACCEPTFILTER by default or not in lighttpd?  If there are good reasons to have the default be disabled, then please add that to the patch.  Thanks.

I think semantically, Linux's TCP_DEFER_ACCEPT and the *BSD's "dataready" SO_ACCEPTFILTER are equivalent. The "httpready" filter is more involved as it includes a partial HTTP parser. There are users that don't want that much parsing to happen in the kernel, especially on low-load machines it doesn't buy much.

@jsonn: still waiting for answer to question:
Since this is a BSD-specific flag, do the BSD core developers recommend that the default be enabled or disabled? (Previously behavior in lighttpd was always enabled, so thank you for attempting to preserve that behavior.)

I don't know of any such recommendation, in either direction. It is kind of a special purpose feature. There have been bugs in the filter in the past, but they have also been fixed.


Improve benchmark comment

To resolve #620

Codecov Report

Merging #681 (152a3c0) into main (0d108b5) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #681   +/-   ##
=======================================
  Coverage   56.23%   56.23%
=======================================
  Files          63       63
  Lines        1700     1700
  Branches       41       41
=======================================
  Hits          956      956
  Misses        744      744

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d108b5...152a3c0. Read the comment docs.

🚀 Benchmark
 **Concurrency: 256**
 **Requests/sec: 427976.86**

I like this a lot more!


Investigate replacing long+TimeUnit with Duration in more places

Is your feature request related to a problem? Please describe.
No problem. It seems like a nice improvement to replace methods that accept a long and a TimeUnit with a single Duration parameter.
Looking through the code I see the following methods that have this pattern of long+TimeUnit:

ScannerBase.setTimeout(long timeOut, TimeUnit timeUnit)
ScannerBase.setBatchTimeout(long timeOut, TimeUnit timeUnit)
BatchWriterConfig.setMaxLatency(long maxLatency, TimeUnit timeUnit)
BatchWriterConfig.setTimeout(long timeout, TimeUnit timeUnit)
ConditionalWriterConfig.setTimeout(long timeout, TimeUnit timeUnit)
DelegationTokenConfig.setTokenLifetime(long lifetime, TimeUnit unit)
TabletLogger.suspended(KeyExtent extent, HostAndPort server, long time, TimeUnit timeUnit, int numWalogs)
ThreadPools.createFixedThreadPool(int numThreads, long timeOut, TimeUnit units, final String name, boolean emitThreadPoolMetrics)
ThreadPools.createThreadPool
MiniAccumuloCluster.stopProcessWithTimeout(final Process proc, long timeout, TimeUnit unit)
TestTicker.advance(final long value, final TimeUnit units)

There is a good chance that a lot of these should stay as they are or are not worth looking into but it seems like this could be a good improvement for some. I think each would need to be investigated to see how these values are used and if there are reasons why we might want to avoid Duration.

This ticket should probably wait until #4360 is either completed or closed.

I don't think we should make this change to the public API methods if we would deprecated the existing methods because it would introduce work for those who have written code written against these APIs w/o offering them a strong benefit for that work.

I don't think we should make this change to the public API methods if we would deprecated the existing methods because it would introduce work for those who have written code written against these APIs w/o offering them a strong benefit for that work.

Good point. I have edited the description to cross those out. There are just a few remaining now. I'll look around to see if there would be a good reason to not change these.

In #4364 I made some changes to Manager to use Duration and the new NanoTime type.  While working on that realized that AccumuloConfiguration.getTimeInMillis() could be a candidate for switching to Duration.  Or we could introduce a new AccumuloConfiguration.getDuration() method.

In #4364 I made some changes to Manager to use Duration and the new NanoTime type. While working on that realized that AccumuloConfiguration.getTimeInMillis() could be a candidate for switching to Duration. Or we could introduce a new AccumuloConfiguration.getDuration() method.

This seems like a good idea so I started looking into it and realized that all of the places that use this method expect it to be in millis so they use it as such. Initially, we would just be converting from Duration back to millis but I guess that is okay and is the first step to converting things. This made me realize that there are probably a ton of places that could benefit from a conversion to Duration but it would be a big undertaking due to all the places we use millis (or other units) currently.


Support transitive dependencies from local repository

Description:
As per SL Beta1, we only support direct dependencies from local repo. This should be extended to support transitive dependencies as well.

Fixed with #32646


Could not open input file: vendor/bin/coveralls

Could not open input file: vendor/bin/coveralls
happens in 90% of all cases, e.g.here.
See https://github.com/GotCms/GotCms/commit/999224c72c8278113d4734e9b938a1b61b1d46b7
for a possible fix.

Could be closed with https://github.com/FriendsOfCake/travis/issues/71

Yes.


Configure implement interface quick fix quote type

From https://github.com/Microsoft/vscode/issues/33476
TypeScript Version:  2.5.1
Code
interface IFoo {
    bar(): number;
}

class Foo implements IFoo {

}

Trigger the implement interface quick fix on class Foo
Expected behavior:
Users can configure if single or double quotes are used for the Method not implemented error
Actual behavior:
Double quotes are always used for the Method not implemented error
interface IFoo {
    bar(): number;
}

class Foo implements IFoo {
    bar(): number {
        throw new Error("Method not implemented.");
    }

}

Related to #13270

@rbuckton did something similar in #17750 -- should we generalize this to looking at all StringLiteral nodes in a file to determine what quote style is preferred?

I would say add a config option for users to choose quote type. in absence of that we can use @rbuckton's inference for imports still.

Yes I think it makes more sense to have an option in this case. VSCode uses both styles of quotes in its source for example, but only uses double quotes for UI strings


Helper for vimPlugins, vscode-extensions, etc

Would your feature fix an existing issue?

No but it would make it even easier to get running with nixos.
Describe the solution you'd like

I'm not sure what the best version of this idea looks like but I was thinking a flake in pkgs/vimPlugins that takes every input and turns it into a pkgs.vimPlugins package and something similar vscode-extensions.
Maybe even something more generic like where pkgs/sets/${pkgSet}/flake.nix takes all inputs and turn them into pkgs.${pkgSet}.input
so this
# pkgs/sets/vimPlugins/flake.nix

{
  description = "Vim Plugin Sources";

  inputs = {
    vim-cargo-make = {
      url = "github:nastevens/vim-cargo-make";
      flake = false;
    };

}

creates
pkgs.vimPlugins.vim-cargo-make

Describe alternatives you've considered

Additional context

I think for developers this would be quite convenient.

Interesting idea. Actually, I tried to modify the upstream script, to make packaging these plugins a lot simpler, but it didn't seem to garner much favour. I kinda let it go stale (not intentionally) as I've been hammered with work stuff.
Maybe DevOS is a better target for this kinda of thing after all.

description = "Vim Plugin Sources"
[inputs]
vim-cargo-make.url = "github:nastevens/vim-cargo-make"

see also: https://github.com/divnix/devos/blob/62f09e12627e884e4a15e9668d07e6a98d82edf8/pkgs/sources.toml

@nrdxp I'm glad you like the idea! I believe DevOS is the perfect place for this. Should I add this issue to digga?

@Th3Whit3Wolf , I'm more than happy to review a PR if you'd like to have a go at it

I will make some attempts but I am quite new to nix so I'm not sure I am entirely up for it.

@nrdxp I have made some progress! I have everything in pkgs/default.nix right now. With the move to nvfetcher I have changed my design.
pkgs/default.nix has 3 overlays.

Allows sources to be used in overlays

overlaySrc = final: prev: {
  sources = (import ./_sources/generated.nix) { inherit (final) fetchurl fetchgit; };
};

Automatically and carefully callPackage for packages in pkgs

rakePkgs = dir: with builtins;
  let
    sieve = name: val:
      if isAttrs val then
        all (x: x == true)
        (forEach (attrNames val) (v:
          hasInfix "_sources" v
        ))
      else
        (val != ./default.nix && name != "bud");
  in
  mapAttrs' (name: value:
    nameValuePair (last (splitString "." name)) (value)) (flattenTree (filterAttrs sieve (rakeLeaves dir)));

overlayLocal = final: prev: builtins.mapAttrs
  (name: value:
    let
      sources = (import ./_sources/generated.nix) { inherit (final) fetchurl fetchgit; };
      package = import (value);
      args = builtins.intersectAttrs (builtins.functionArgs package) { source = sources.${name}; };
    in
    final.callPackage package args
  ) (rakePkgs (./.));

ignores pkgs/_sources and pkgs/bud
pkgs/manix and pkgs/misc/documentation/manix both get create a pkg called manix
instead of

{ lib, stdenv, sources }:

stdenv.mkDerivation rec {
  inherit (sources.sanFranciscoMono-font) src pname version;

you do
{ lib, stdenv, sources }:

stdenv.mkDerivation rec {
  inherit (sources) src pname version;

Lastly an overlay for pkgSets (currently just vimPlugins). With the move to nvfetcher I think it makes sense to prefix pkgSets in nvfetcher and then build them.

For example in pkgs/source.toml I prefix pkgs apart of the vimPlugins package set with VimPlugins-. Then in the overlay we can collect them and strip the prefix when we build nixos. I am unsure if this will work for all package sets but I think it should work for a good portion of them, specifically text editors and programming language libraries.
[vimPlugins-ron-vim]
src.github = "ron-rs/ron.vim"
fetch.github = "ron-rs/ron.vim"

overlayPkgsSets = final: prev: {
  sources = (import ./_sources/generated.nix);
  vimPlugins = with final.vimUtils;
    let
      plugs = prev.lib.mapAttrs' (name: value: nameValuePair (prev.lib.removePrefix "vimPlugins-" name) (value)) (prev.lib.filterAttrs (n: v: prev.lib.hasPrefix "vimPlugins-" n) prev.sources);
      createVimPlugin = pname: version: src: prev.vimUtils.buildVimPluginFrom2Nix { inherit pname version src; };

      overridePkgVersionSrc = pkg: version: src: pkg.overrideAttrs (_: { inherit version src; });
      overrideOrCreateVimPlugin = plugin:
        let
          pname = prev.lib.removePrefix "vimPlugins-" plugin.pname;
          pkg = prev.vimPlugins.${pname} or { };
          version = plugin.version;
          src = plugin.src;
        in
          if pkg ? overrideAttrs then
            overridePkgVersionSrc pkg version src
          else
            createVimPlugin pname version src;

          mapVimPlugins = pluginsSet: prev.lib.mapAttrs'
            (name: src: prev.lib.nameValuePair name (overrideOrCreateVimPlugin pluginsSet.${name}))
            pluginsSet;
    in
    prev.vimPlugins // mapVimPlugins plugs;
  };

I also made a nvfetcher feature request to add vscode extensions. If anyone reading this is interested and more familiar with haskell I made a shell script (posted in issue), that uses curl and jq to get vscode extension versions and downloads, that might help get someone starteed.

I just saw the nvchecker PR was merged, do we have to wait for conformance?

@blaggacao yes and then we can throw vscode extension in sources.toml!

I added a function to make it easier to add pkgSets.
mapPkgSets = sources: pkgSet: pkgSetName: buildTool:
    let
      prefix = pkgSetName + "-";
      newPkgs = mapAttrs' (name: value: nameValuePair (removePrefix prefix name) (value)) (filterAttrs (n: v: hasPrefix prefix n) sources);
      createNewPkgs = pname: version: src: buildTool { inherit pname version src; };
      overridePkgVersionSrc = pkg: version: src: pkg.overrideAttrs (_: { inherit version src; });
      overrideOrCreatePkg = pkgFromSet:
        let
          pname = removePrefix prefix pkgFromSet.pname;
          pkg = pkgSet.${pname} or { };
          version = pkgFromSet.version;
          src = pkgFromSet.src;
        in
        if pkg ? overrideAttrs
        then overridePkgVersionSrc pkg version src
        else createNewPkgs pname version src;

      mapPackages = set: mapAttrs'
        (name: src: nameValuePair name (overrideOrCreatePkg set.${name}))
        set;
    in
    pkgSet // mapPackages newPkgs;

So that the pkgSets overlay can be created like this
  overlayPkgsSets = final: prev: {
    sources = (import ./_sources/generated.nix);

    vimPlugins = mapPkgSets prev.sources prev.vimPlugins "vimPlugins" prev.vimUtils.buildVimPluginFrom2Nix;
  };

It needs to be tested on more pkgSets. But I think it should work if anyone else is interested in trying it out my pkgs/default.nix is here and you would need to replace ./pkgs/default.nix with (import ./pkgs).overlay.
@blaggacao and @nrdxp  I have checked out the digga source code and there's no way I can craft an importer, however the first 130 lines of the pkgs/default.nix are rakeLeaves, flattenTree, and functions from lib required to make them work so I think it may not be too difficult for someone with a bit more nix talent. I could make a PR here before it gets moved to digga if that would be preferable?

I've done some intense refactoring and I think I might be able to get a PR to digga this weekend.

Nevermind, I was way overly confident. I will send a PR to devos so people can get some eyes on the code.


added executable bit to shell scripts

Deployment of chatops VM fails due to missing executable bit on .sh scripts. Problematic command is bash -c '/vagrant/validate.sh'. This pull request changes mode of all sh scripts to 0755. (Tested on 64bit Ubuntu 15.04)
...
==> chatops: WebUI at http://chatops:8080/
==> chatops: ==========================================
==> chatops:
==> chatops:           _   ___     ____  _  __
==> chatops:          | | |__ \   / __ \| |/ /
==> chatops:       ___| |_   ) | | |  | | ' /
==> chatops:      / __| __| / /  | |  | |  <
==> chatops:      \__ \ |_ / /_  | |__| | . \
==> chatops:      |___/\__|____|  \____/|_|\_\
==> chatops:
==> chatops:   st2 is installed and ready to use.
==> chatops: ==========================================
==> chatops:
==> chatops: Test StackStorm user account details
==> chatops:
==> chatops: Username: testu
==> chatops: Password: testp
==> chatops:
==> chatops: Test account credentials were also written to the default CLI config at .
==> chatops:
==> chatops: To login and obtain an authentication token, run the following command:
==> chatops:
==> chatops: st2 auth testu -p testp
==> chatops:
==> chatops: For more information see http://docs.stackstorm.com/authentication.html#usage
==> chatops: Running provisioner: shell...
    chatops: Running: /tmp/vagrant-shell20150630-3084-11njtq2.sh
==> chatops: stdin: is not a tty
==> chatops: ########## Setup St2 to use Syslog ##########
==> chatops: ########## Turn on the UDP listener in rsyslog.conf ##########
==> chatops: ########## Copy st2 Syslog config file in place ##########
==> chatops: rsyslog stop/waiting
==> chatops: rsyslog start/running, process 16529
==> chatops: Running provisioner: shell...
    chatops: Running: inline script
==> chatops: stdin: is not a tty
==> chatops: bash: /vagrant/validate.sh: Permission denied
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

Thanks!


Do not parse header files explicitly

This is done by the CDT preprocessor already which mimics the actual gcc behavior (include directive).
This needs to be discussed. We won't have findings in header files directly anymore.
Instead they would appear in the corresponding *.c or *.cpp file (wherever the header is included).

Something to keep in mind for this change is that, when using joern-scan, findings in the header file will be reported as being defined in the corresponding c file instead. There's a more detailed example of that in the issue linked above, but effectively it would be reporting something like
Result: 1.0 : Multiple returns: test_inline.c:7:vulnerability_in_inline_func

instead of

Consequently, the file and line number reported for the finding may not be where the issue occurs at all. We'd need some way of identifying whether a finding occurred in a C file or the corresponding header file for accurate reporting.

Yes, you are right. Its not that easy here. Lets close this one for now.