Mobile E2E Test Setup

Coverage remained the same at 95.833% when pulling 4899ac8aaebbfab6cd0fe2582874d3c8fec6a36a on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling f3ee42c749797e165063cd32ddd9bc9076f6ec12 on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 0fee755cc30a1796987dfde9d9f6c53aaa3e883f on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 87fed7d1b8e2dece7cbe6e0a806004d96c205d99 on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling b535df81e15381be962da5dc6f635c909fabc288 on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 63db83d9afff1df6db1d41e5a6099d7c025e1c0c on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling a468276d87cb546cf76afafc1de6ec38f340c658 on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 36b75bc2c11872901e1d162f6de25a71cbf973df on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 84ea1c95db96f41224f8d43ef3231cfb13e15f59 on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 5d2797a1c6d5c10bfef265e692ac551d3ca02d0b on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 33422983c9246d483e7161493cc57368213a2600 on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling a6f1f374614ef6fbf25c399f60d60ee3d02ea3b4 on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 108ba689171076044442f2cb3f01e5152a14740a on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling a4a6003905abcf068477e369e900cc112f9f2623 on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 6ecfabf7dd75f117ccb98c90e0c4a1279b33b275 on tests/e2e-pipeline-mobile into 6756580a4c3def827e9ed973aa2d1be977766b00 on master.

Coverage remained the same at 95.833% when pulling 8abe3353eb0c61ffd9c4d0773556d95203a20a8c on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 4afd2319182a1af365ecbeb45544bc669b87bf74 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling e4dfe621850fe83ec10934845aa5d7063f7c206e on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 4943be4f5912e9856267aae4a35d780ea9c0c8b0 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling a434f403fd60b5aa799791cd716754b81ba644c7 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 5f4e99115a2a7e636ed670dda7db51cc88bcf43a on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling f423ae1626aaac04ea1368c3b3d9a3e47e001bb2 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling aaa9addd7dc24cdbbbc5a5ae15cd1ad3a5b5a516 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 05c7db520e977b58ef5ce88bb283106cd18dfbae on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 58af94130eb0505c1f06be80dcb4baf072dfa11a on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 133174784db78ddab34faf236806b4f957d6211b on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 6e1b8d0cf91a8696dd4fa0b6d5a699685ba5c87c on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling d6fd20de684ae1ef6b30ab2273ba29c86a7a7670 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 08b6faf06b605304e0fb94ae3f8689a54b2aec4f on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 7d2ba1fd19d23af6cc220e5ac7a0ba9344f8bbbc on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 69bdd4c633ec93e6b460571c8f683bb076ef1ad9 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 84b44ae7294e8dc207db8a509446609359198e0a on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling a9d7c3de136f89c4dd161d09a059b6152e263453 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 33dbfc87bb4d7f9e44de3173058c26e0245940da on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 4b2f446183cc4c65e0e68799243eb340547cc466 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling e9db5bf0728df5774d9887cfe2e49123b4084a02 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling d40c3c41255de35c735f949b6afc49189afa9b59 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 45ea0a8345883dea1f7026af71a4fa0364fedc9b on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 54f2f15ed4921b8b1a71d8e538168fb2d5e4e9fe on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 3ce9e79133e0889987492a18d81f75a9c6281d5b on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling bf033de7190935dc97e88736bc26845a3c489279 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 8564d0dde4f4e881ebf1c235c8691967dfeed435 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 8ceb7f00c0891baad2a4f3619060e89f86e66a12 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling c7fe3732e1f09a7ebcbe767f308cb3a085c3762c on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling e6639595cc36b0b27d5507732261662ae5f4c2a1 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 02d4042d1e25c6b41318549ee575b4bb16e6f03c on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 8503bd778dd8ff7c69b79ad20c4b6c10d1b0437e on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 51d82374b7cd4853c32f71c43c7fa72cf25afb84 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.

Coverage remained the same at 95.833% when pulling 97a92f75165eae9f06da3e0aac12f845db1ebd56 on tests/e2e-pipeline-mobile into 4f4f53460c7b69ff95d0b4103973712f0ec7b3d2 on master.


Do not fail on octet stream content type

What does this PR do?

Solution 1 (this PR):
Set encoding to utf-8 when there is no encoding to avoid failure down the road.
Content might be in bytes when the service exposing the openmetrics endpoint doesn't set correctly the Content-Type and Encoding.
The openmetrics/prometheus exposition format suggest using utf-8: https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md
Solution 2:
https://github.com/DataDog/integrations-core/pull/5847
Raise an error if the HTTP Content-Type header is incorrect. Similarly to Prometheus Mixin: https://github.com/DataDog/integrations-core/blob/1bf1bcf5f4af4a54f16bf45ad76d55c4f364ce05/datadog_checks_base/datadog_checks/base/checks/prometheus/mixins.py#L204-L237
Motivation
Currently, if bytes content is received. In Python 3, the openmetrics check will fail with the following issue:
openmetrics (1.3.0)
-------------------
  Instance ID: openmetrics:idm-mhcf74:efee10b119b1d5b0 [ERROR]
  Configuration Source: file:/etc/datadog-agent/conf.d/openmetrics.yaml
  Total Runs: 13
  Metric Samples: Last Run: 0, Total: 0
  Events: Last Run: 0, Total: 0
  Service Checks: Last Run: 1, Total: 13
  Average Execution Time : 11ms
  Error: startswith first arg must be bytes or a tuple of bytes, not str
  Traceback (most recent call last):
    File "/opt/datadog-agent/embedded/lib/python3.7/site-packages/datadog_checks/base/checks/base.py", line 678, in run
      self.check(instance)
    File "/opt/datadog-agent/embedded/lib/python3.7/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 82, in check
      self.process(scraper_config)
    File "/opt/datadog-agent/embedded/lib/python3.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 378, in process
      for metric in self.scrape_metrics(scraper_config):
    File "/opt/datadog-agent/embedded/lib/python3.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 352, in scrape_metrics
      for metric in self.parse_metric_family(response, scraper_config):
    File "/opt/datadog-agent/embedded/lib/python3.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py", line 301, in parse_metric_family
      for metric in text_fd_to_metric_families(input_gen):
    File "/opt/datadog-agent/embedded/lib/python3.7/site-packages/prometheus_client/parser.py", line 138, in text_fd_to_metric_families
      if line.startswith('#'):
  TypeError: startswith first arg must be bytes or a tuple of bytes, not str

Additional Notes

Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached

Closing in favour of https://github.com/DataDog/integrations-core/pull/5847


Initial State

Hello!
Many thanks for this awesome and very minimalistic SSG Plugin for Vite!
What I'm currently missing is the ability to set an initial state when generating static HTML similar to frandiox/vite-ssr:
I am using Pinia as a store and these steps are necessary during building, so my fetched content from an API can stored in Pinia and injected into the generated HTML as an initial state to avoid unnecessary load times in production.
However, config functions like onBeforePageRender or onPageRendered do not help as I cannot get access to the app's context, or rather the pinia store from the Vue app.

I am not that familiar with those, but maybe let's see in the PR if you are interested in make one. Thanks


AttributeError: 'module' object has no attribute 'slim'

I am trying nsynth・training but I get an error
Ubuntu 16.04 LTS
magenta-gpu 0.3.12
tensorflow-gpu 1.11.0
librosa 0.6.2
＄bazel run //magenta/models/nsynth/baseline:train -- --train_path=//home/nekome/tffile.tfrecords ---logdir=//home/nekome/Logdire
WARNING: Processed legacy workspace file /home/nekome/magenta/tools/bazel.rc. This file will not be processed in the next release of Bazel. Please read https://github.com/bazelbuild/bazel/issues/6319 for further information, including how to upgrade.
INFO: Analysed target //magenta/models/nsynth/baseline:train (0 packages loaded).
INFO: Found 1 target...
Target //magenta/models/nsynth/baseline:train up-to-date:
  bazel-bin/magenta/models/nsynth/baseline/train
INFO: Elapsed time: 0.059s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/magenta/models/nsynth/baseline/train '--trINFO: Build completed successfully, 1 total action
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:99] Couldn't open CUDA library libcudnn.so. LD_LIBRARY_PATH:
I tensorflow/stream_executor/cuda/cuda_dnn.cc:1407] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
Traceback (most recent call last):
  File "/home/nekome/.cache/bazel/_bazel_root/8d760b4887a3a97dbc093bfa28201502/execroot/__main__/bazel-out/k8-opt/bin/magenta/models/nsynth/baseline/train.runfiles/__main__/magenta/models/nsynth/baseline/train.py", line 23, in <module>
    from magenta.models.nsynth import reader
  File "/home/nekome/.cache/bazel/_bazel_root/8d760b4887a3a97dbc093bfa28201502/execroot/__main__/bazel-out/k8-opt/bin/magenta/models/nsynth/baseline/train.runfiles/__main__/magenta/models/nsynth/reader.py", line 24, in <module>
    from magenta.models.nsynth import utils
  File "/home/nekome/.cache/bazel/_bazel_root/8d760b4887a3a97dbc093bfa28201502/execroot/__main__/bazel-out/k8-opt/bin/magenta/models/nsynth/baseline/train.runfiles/__main__/magenta/models/nsynth/utils.py", line 29, in <module>
    slim = tf.contrib.slim
AttributeError: 'module' object has no attribute 'slim'

How can I solve it?
Please give me a suggestion

Try to compile a version of tensorflow on your machine to accompain those kind of changes of API access.
Freeze a version with that method present on the contrib module

https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md


some minor fixes from testing

@david-martin the change to use tls stopped the server from working inside the docker container. I think we might be able to use the annotation on the route to get a generated cert from openshift

@philbrookes take a look also as it will likely effect you


ENH: Add scipy.datasets submodule

Reference PR
gh-8707
What does this implement/fix?
>>> from scipy import datasets
# Example ascent dataset loading with the new module
>>> datasets.ascent()
array([[ 83,  83,  83, ..., 117, 117, 117],
       [ 82,  82,  83, ..., 117, 117, 117],
       [ 80,  81,  83, ..., 117, 117, 117],
       ...,
       [178, 178, 178, ...,  57,  59,  57],
       [178, 178, 178, ...,  56,  57,  57],
       [178, 178, 178, ...,  57,  57,  58]])

With gh-8707, a couple of years ago, SciPy wanted to introduce the datasets submodule and move a handful of dataset functions from the current misc module to this new datasets submodule. Thanks to @WarrenWeckesser for the initial work. With this PR (indeed inspired by gh-8707) we resume those efforts after making some improvements (explained below) and move away from the scipy.misc module, finally deprecating it (in a separate PR #15901).

[x] Add scipy.datasets submodule
[x] Utilize pooch to handle the dataset downloading and caching.
[x] Enable meson support for scipy.datasets submodule
[x] Update the base URL and registry URL dataset links once we decide the approach and create the respected repos in the scipy organization.

TODO Later:

[ ] Move all dataset files (eg. scipy.stats has its own test datasets within the repo) to their respective new repos (explained below). This is something that can be done after landing this PR once we have a concrete datasets API and approach defined for adding new datasets.
[x] Deprecate the misc module (#15901)

Additional information
Why should we use Pooch?
OR
What are some other possible options (eg. intake, one of the suggestions made earlier) and why not prefer them?
There are a few good reasons for choosing Pooch:

With Pooch we can easily decouple the datasets that are currently present within the SicPy repository and move them to their own new repository. For example, see https://github.com/scipy-datasets, where each dataset has its own repository. This will lead to a lightweight SciPy wheel decreasing the download size for future releases. Keeping the datasets in individual repos or a single scipy-datasets repo is a point of discussion. Note that we need to take into account that GitHub has a 2Gb limit on the repo size. In case, in future, we plan to add new datasets which are relatively large, then this idea of keeping the datasets in the SciPy repo may not be a wise choice. In fact, keeping all the datasets in a separate single dataset repo might also break after a while. The best approach IMO would be to maybe group them by type of datasets into a few repositories or just keep them individually (i.e each dataset has its own repo).

Dependency: Pooch is an extremely light package and has only a few dependencies, so if we were to add a new dependency i.e. Pooch, we can expect it to be small and at the same time it won't add a lot of sub-dependencies. On the other hand, something like intake has a few more deps, see requirements.txt. With the basic intake package and some additional sections, it will also install packages like dask etc.(not required for scipy datasets). Both of them use a library called appdirs which is quite small and useful.

IMO Pooch has a simpler API overall and seems easy to implement/maintain for developers while also supporting the key features like caching (hash checks, multi download). Intake is slightly more complex with its design of catalog like YAML files and definitely has some extra features but we surely don't need those to solve the problem of dataset downloading and caching.

Ps. scikit-image also recently (in 2020) made a move to make use of Pooch for some of the same reasons.
cc @rgommers

/cc @grlee77 @hmaarrfk Any tips from the skimage fetcher that should apply here?

2. Dependency: Pooch is an extremely light package and has only a few dependencies, so if we were to add a new dependency i.e. Pooch, we can expect it to be small and at the same time it won't add a lot of sub-dependencies

Scikit-image doesn't depend on Pooch, it's an optional dependency (e.g., pip install scikit-image[data]). We should probably do something similar, dragging it in by default seems like a bit much. xref https://github.com/scikit-image/scikit-image/pull/5105 for the relevant discussion.

Caching and downloading parts can be done without the dependencies. Requests is quite a hefty dependence for a light weight lib in my opinion. We can simply check a file with a predefined hash and otherwise get it with some urllib machinary.
Also if scikit image is providing these then probably we should delegate to it since they are the golden standard for image related stuff.
Note however that we need a long deprecation time as we witnessed before with imresize and other related functions that haunted us for at least four major versions.

I'm with Ralf on the dependencies, I definitely wouldn't want to be adding more hard dependencies, only optional ones (if at all).
In any case, it's probably worth asking ourselves "Would we add the same files to the project nowadays?" and "Do we think it's worthwhile to add more datasets/images?". If the answer to these questions is no then why don't we just deprecate and remove them completely?

What Andrew said. If we go down to pip install scipy[data] or some such, why not go all the way to a separate scipy_datasets repo/project in the scipy organization.

In any case, it's probably worth asking ourselves "Would we add the same files to the project nowadays?" and "Do we think it's worthwhile to add more datasets/images?". If the answer to these questions is no then why don't we just deprecate and remove them completely?

That's a very fair question. I think the answer is that we do actually want these for examples, tutorials and tests, and ideally want some more.
For image data we could say why not just use skimage.data (https://scikit-image.org/docs/stable/api/skimage.data.html), and add them there? This has a few pros and cons:

Pro: less to maintain for us
Con: need to depend on scikit-image in docs and test suites, which is a heavier dependency, circular, and drags in 6 more packages (https://github.com/scikit-image/scikit-image/blob/main/requirements/default.txt) in addition to still depending on pooch anyway

In addtion, we also want:

time series data (like electrocardiagram)
stats data (like iris, and test/benchmark data like the NIST data we already ship)
optimize benchmark data, like the Netlib LP data in benchmarks/benchmarks/linprog_benchmark_files/
all the special functions reference data in scipy/special/tests/data/

@AnirudhDagar didn't really focus on the yes/no for scipy.datasets here, because the linked PR from @WarrenWeckesser got thorough and largely positive feedback (https://github.com/scipy/scipy/pull/8707#issuecomment-380372344) which still applies. In addition, I think the importance of sdist/wheel size reduction has grown since then, and there's more to gain than was pointed out in that thread. I recently counted, and came to about 18 MB of text/data files that we ship (uncompressed, didn't carefully measure how much wheel sizes reduce with all that removed).
So my impression is that:

the case for having a single place for all our datasets needs is strong,
depending on scikit-image for image data, scikit-learn for iris, etc. is not a good solution,
so the choice is between scipy.datasets and a separate scipy-datasets. for that:

functionality is the same either way
we need it to make scipy.test() work
it's more discoverable as part of the main scipy docs
release/docs/maintenance-wise it's easier to have it integrated rather than having an optional dependency on a (pinned version of) scipy-datasets.

Regarding file sizes: my impression is that >95% of users will not need scipy.datasets contents, unless it becomes much more popular in the future, so we can assume this is disk space & network bandwidth saved. I did a little calculation for PyPI:

assuming 18 MB on disk may translate to ~3 MB wheel size reduction, then with the current 35M downloads/months we save 36e6 * 12 * 3e6 / 1e15  # downloads/month * months * wheel-MB / PB = 1.3 PB/yr
PyPI bandwidth usage is about 1 PB/day (see https://dustingram.com/articles/2021/04/14/powering-the-python-package-index-in-2021/)
So it's about 0.35% of total PyPI bandwidth size (a little handwaving, but I'm reasonably sure it's >0.1% unless I made a calculation error)

I expect the sdists need to include tests, but do the binary wheels need them?

I expect the sdists need to include tests, but do the binary wheels need them?

We've always wanted this so users can actually diagnose install issues, and I personally use it a lot as well. It could be separated out as another package (e.g., scipy-testsuite), but also that is quite a bit of work. The main downside of actually including tests is data size, so if we can solve that here with a datasets package, that'd be my preference.

I think the dataset API should be abstracted away from files. Users should see data and python objects, not filenames.

Yes, I agree.

I think if you wanted to get away from requests, you might have better effect making patches to pooch and suggesting them in PRs.

There's no need to specifically avoid requests. I saw @ilayn's comment higher up, but urllib may get deprecated and itself recommends requests. If we optionally depend on Pooch, this is just something we shouldn't care about (and if we did, requests is the gold standard).

urllib may get deprecated and itself recommends requests

That's really good news for me. Miraculously I missed the whole PSF takeover. Makes many things so much simpler.

Thanks all for the feedback, completely understand why it makes sense to have pooch as an optional dependency. Does this mean that scipy should pass all the tests without pooch after build (we can somehow skip datasets module tests)? Or can we add pooch as a requirement for the tests on the CI and then naturally tests should be fine (like done in the PR at the moment)?
I guess a user won't care about tests and it is fine to hard depend on pooch for tests to pass, as we do depend in the case of matplotlib as well.
When using SciPy, the dependency still remains optional, until someone actually imports the datasets module or calls one of the datasets functions, all is good. IIUC when scipy.datasets is imported, an importerror should be raised if pooch is missing, and at the same time ask the user to install it.

I guess a user won't care about tests and it is fine to hard depend on pooch for tests to pass, as we do depend in the case of matplotlib as well. (on a second thought, not sure about this)

It's fine to have a hard dependency on Pooch for the test suite. It'll be hard to avoid, too many tests will depend on it. Matplotlib is a different story - it's literally only a handful of tests that depend on it.

Some build systems downstream might have restrictions on connecting to the internet. During their testing phase.
Finally, there can be spurious failures with downloading large datasets in CIs
We actually allow users to test without pooch install:
https://github.com/scikit-image/scikit-image/blob/725e8f6bcccc7d4e78ba0d64168f8ed76b6dc818/skimage/data/_fetchers.py#L143

i mostly wanted to point to our implementation rather than to say you should do.

Some build systems downstream might have restrictions on connecting to the internet. During their testing phase.

This is easy to avoid by downloading in the install phase in CI scripts (more efficient anyway probably). So when you do:
python dev.py --build-only   # or `pip install .`

to install SciPy, then right after do whatever the command will be to download all datasets used in CI.

to install SciPy, then right after do whatever the command will be to download all datasets used in CI.

I believe some are even restricted against doing this.
I think one example is debian.
Anyway, what we did was to skip any test where pooch failed for whatever reason.

Anyway, what we did was to skip any test where pooch failed for whatever reason.

This sounds like a good idea to me if we decide to skip only if the failure is due to a problem with pooch downloading the dataset.

I think one example is debian.

It must be allowed to run multiple commands I'd think. We already require:
git clone ...
git submodule update --init

That should be the "obtain sources" step. If needed we could make it possible to run some script straight after without first building SciPy.

Debian: They include their files in a mirror of your repo: https://salsa.debian.org/python-team/packages/scipy/-/blob/master/debian/rules
Fedora: Uses the url provided from github https://src.fedoraproject.org/rpms/scipy/blob/rawhide/f/scipy.spec#_30
Our implementation on skipping tests upon failure. https://github.com/scikit-image/scikit-image/blob/main/skimage/data/_fetchers.py#L217
The final case to consider, is that hammering raw.github is surely going to give you failures on builds due to github rejecting some of your requests. (3 platforms, 4+ python versions, staging & deploy = 24 builds)
to be honest, I think there are many ways to skip tests afterwards, and I know you are all very responsive to people raising build issues (and involved in the builds many times).
I wouldn't hold off because of the "nits" I am raising.
Users have explicitly asked that we make pooch an optional dependency.
We haven't heard too much from a testing side.

Thanks Ralf, yes I was able to reproduce this at a small isolated demo scale for pooch here. It raises on the pooch import which as you suggested can be silenced, wrapping in a context manager filtering out the warning.
Do you think something like this is fine?
import warnings

with warnings.catch_warnings():
    warnings.filterwarnings(action='ignore', category=DeprecationWarning)
    import pooch

Yes, that should do the job @AnirudhDagar.
If you can add that fix, and also have a look at if the remaining linting issues can be resolved without making the code worse, then CI should turn green.

For the lint errors:
scipy/datasets/__init__.py:33:1: E402 module level import not at top of file
scipy/datasets/_fetchers.py:85:80: E501 line too long (91 > 79 characters)
scipy/datasets/_registry.py:9:80: E501 line too long (85 > 79 characters)
scipy/datasets/_registry.py:10:80: E501 line too long (82 > 79 characters)
scipy/datasets/_registry.py:11:80: E501 line too long (82 > 79 characters)
scipy/datasets/_registry.py:15:80: E501 line too long (100 > 79 characters)
scipy/datasets/_registry.py:16:80: E501 line too long (91 > 79 characters)
scipy/datasets/_registry.py:17:80: E501 line too long (93 > 79 characters)

init.py E402 import is also something that is exactly like other modules in SciPy, so i feel it is okay to ignore.
_fetchers.py E501 is again a link in the doc which can be ignored.
All registry E501s are links which can be ignored for now.

Sorry for not getting involved sooner.  It is great to see this moving forward.
However, I still believe we should have a small collection of datasets included with SciPy, that do not have to be downloaded on demand.  We should maintain a small set of datasets that will just work in the Examples sections of our docstrings without requiring an internet connection.  I think the current datasets that we have, plus something like a snapshot of the sunspot data, and a widely used multivariate dataset (not sure what that might be), would be easy to maintain and would not increase the source distribution by much.  It also cuts down on internet use for data loading when building the docs in CI.
This wouldn't necessarily change the API in the current PR, except that one does not get an error when accessing one of the pre-installed datasets when Pooch is not installed.

@WarrenWeckesser it's worth considering if any datasets need to still be included in the repo (ascent is maybe the best candidate), but "all we have now + some more" isn't very compelling imho. Have a look at face for example: it's 1.6 MB, and is only used in 2 docstring examples. Both sdist and wheel actually get smaller by that amount, because the file is already compressed.
Excluding the three image files and building a wheel with python -m build shows the size on Linux goes from 23.5 MB to 21.6 MB, which is significant. For other data files:
$ find . -name "*.npz" | xargs du -sch
36K     ./linalg/tests/data/carex_19_data.npz
160K    ./linalg/tests/data/carex_18_data.npz
32K     ./linalg/tests/data/carex_20_data.npz
36K     ./linalg/tests/data/carex_15_data.npz
16K     ./linalg/tests/data/carex_6_data.npz
4.0K    ./linalg/tests/data/gendare_20170120_data.npz
164K    ./fftpack/tests/fftw_double_ref.npz
292K    ./fftpack/tests/fftw_longdouble_ref.npz
100K    ./fftpack/tests/fftw_single_ref.npz
12K     ./fftpack/tests/test.npz
24K     ./spatial/tests/data/degenerate_pointset.npz
576K    ./stats/_sobol_direction_numbers.npz
56K     ./special/tests/data/gsl.npz
204K    ./special/tests/data/local.npz
1.3M    ./special/tests/data/boost.npz
4.0K    ./interpolate/tests/data/bug-1310.npz
588K    ./sparse/linalg/tests/propack_test_data.npz
4.0K    ./sparse/tests/data/csc_py3.npz
4.0K    ./sparse/tests/data/csc_py2.npz
3.5M    total

$ find . -name "*.txt" | xargs du -sch
4.0K    ./spatial/qhull_src/COPYING.txt
4.0K    ./spatial/tests/data/pdist-correlation-ml.txt
4.0K    ./spatial/tests/data/pdist-hamming-ml.txt
12K     ./spatial/tests/data/random-int-data.txt
176K    ./spatial/tests/data/pdist-chebyshev-ml-iris.txt
4.0K    ./spatial/tests/data/pdist-spearman-ml.txt
8.0K    ./spatial/tests/data/cdist-X1.txt
8.0K    ./spatial/tests/data/random-bool-data.txt
52K     ./spatial/tests/data/pdist-double-inp.txt
176K    ./spatial/tests/data/pdist-jensenshannon-ml-iris.txt
16K     ./spatial/tests/data/iris.txt
180K    ./spatial/tests/data/pdist-minkowski-3.2-ml-iris.txt
176K    ./spatial/tests/data/pdist-correlation-ml-iris.txt
4.0K    ./spatial/tests/data/pdist-jensenshannon-ml.txt
12K     ./spatial/tests/data/random-uint-data.txt
180K    ./spatial/tests/data/pdist-minkowski-5.8-ml-iris.txt
180K    ./spatial/tests/data/pdist-euclidean-ml-iris.txt
176K    ./spatial/tests/data/pdist-cityblock-ml-iris.txt
4.0K    ./spatial/tests/data/pdist-euclidean-ml.txt
176K    ./spatial/tests/data/pdist-cosine-ml-iris.txt
52K     ./spatial/tests/data/pdist-boolean-inp.txt
4.0K    ./spatial/tests/data/pdist-chebyshev-ml.txt
4.0K    ./spatial/tests/data/pdist-jaccard-ml.txt
12K     ./spatial/tests/data/cdist-X2.txt
80K     ./spatial/tests/data/random-double-data.txt
4.0K    ./spatial/tests/data/pdist-cityblock-ml.txt
4.0K    ./spatial/tests/data/selfdual-4d-polytope.txt
180K    ./spatial/tests/data/pdist-seuclidean-ml-iris.txt
4.0K    ./spatial/tests/data/pdist-minkowski-3.2-ml.txt
4.0K    ./spatial/tests/data/pdist-cosine-ml.txt
4.0K    ./spatial/tests/data/pdist-seuclidean-ml.txt
4.0K    ./io/matlab/tests/data/japanese_utf8.txt
4.0K    ./ndimage/tests/data/label_strels.txt
8.0K    ./ndimage/tests/data/label_results.txt
4.0K    ./ndimage/tests/data/label_inputs.txt
1.9M    total

Other contributors:

other .dat files aside from the 3 image files: 1.8M
.mat: 0.46M total
.npy: 0.41M total
.arff: 0.26M total
.sav: 0.29M total

would not increase the source distribution by much

My goal would be to cut the wheels in size by as much as possible, since they're very much overweight, rather than "not increase much".

How about this: we merge this PR as is, and that gives us two releases to detect potential issues before the datasets in misc are removed. If it does give hiccups, we keep ascent and ecg as builtins in scipy.datasets, those have a reasonable size/utility tradeoff. And for the two signal examples that use face, we just switch those to ascent.

Eventually, it would be useful to have prepared a set of instructions for manually loading the cache of datasets, for our friends who work in remote or secure areas without continuous internet access.

Good point, this would be useful. I spent some time searching the scikit-learn issue tracker, they don't have an issue related to this (and only 40 issues related to datasets as a whole after ~7 years, which is a good sign). But it'd be a good thing to do anyway.

Eventually, it would be useful to have prepared a set of instructions for manually loading the cache of datasets, for our friends who work in remote or secure areas without continuous internet access.

I can work on adding these doc instructions as a follow-up.

Thanks @rgommers!

And those READMEs should probably point back to the SciPy docs page for the scipy.datasets submodule, which then explains the mechanics of how these datasets/repos are used, retrieved, cached, etc.

Good point about explaining the mechanics of how the datasets are fetched and cached using pooch. I have not mentioned that in this PR yet, so I'll update the docs there.
Regarding readme files of the dataset repos, should I add a future link? For example the following ascent doc link is obviously broken at the moment and will be applicable after the 1.10 release only. https://docs.scipy.org/doc/scipy/reference/generated/scipy.datasets.ascent.html#scipy.datasets.ascent
I'll send PRs on the three dataset repos.

Regarding readme files of the dataset repos, should I add a future link?

Pointing to http://scipy.github.io/devdocs/ would be good

@rgommers I've rebased the PR, updated the dataset links and added documentation on usage of datasets. I've also sent PRs to the dataset repos adding a basic description.

Fixed the merge conflict due to #16739.

Follow-ups we discussed:

[ ] add download_all
[ ] add something like a clear_cache function, so the user doesn't have to manually hunt for a platform-specific path
[ ] revisit the comment higher up about the needs of a packager needing to adhere to Debian rules: we need some separate script or package to download everything at once before scipy is built
[ ] skipping tests if Pooch is not available or there is no network connection

These can all be done after merging this PR.

I've addressed the review comments (major changes to test_data.py design); I'll also open a new issue to track all the follow-up tasks related to SciPy datasets.

Btw before this is merged (no dependency though), just a small reminder about the PRs on the dataset repos. LMK if they are fine or need any improvement:

scipy/dataset-ecg#1
scipy/dataset-ascent#1
scipy/dataset-face#1

Thanks again @WarrenWeckesser for the initial work, and thanks @rgommers, for helping me through all of this.
I'll update the release notes and get started with the follow-ups. :)

This PR seems to be the cause of a CI failure in SymPy that is somehow related to importing polygamma https://github.com/sympy/sympy/pull/23932. Not sure yet why that would be affected but @bjodah bisected to here.

Maybe the problem is that this does not install the pooch dependency:
pip install git+https://github.com/scipy/scipy@main

Maybe the dependency should be added in setup.py.

Thanks @oscarbenjamin for the report. Yes, pooch is an optional dependency, and thus we don't require it. I'm curious, does SymPy use the datasets module already? Or are you running all the SciPy tests in SymPy CI (I've not checked SymPy CI yet)? This warning should only hit if you try to import scipy.datasets or use a method from that module.
All other sections/submodules within SciPy can still be used without installing/requiring Pooch.

Ohh interesting, is it just the install from git that is triggering the issue?

No SymPy does not use the datasets module. Somehow this PR has affected SymPy importing other things from SciPy as part of lambdify. The code that does this is quite convoluted so it's not easy to say exactly what import statement is being used:
https://github.com/sympy/sympy/blob/master/sympy/utilities/lambdify.py

Ohh interesting, is it just the install from git that is triggering the issue?

Yes, somehow SymPy CI fails with SciPy main since this PR was merged. More discussion in https://github.com/sympy/sympy/pull/23932.

The file indeed is convoluted, and I need to look closely, but from the first glance, maybe it is L106 importing everything from SciPy.
https://github.com/sympy/sympy/blob/b1f35f8daf5efa9f801f905cf1037000d69b0278/sympy/utilities/lambdify.py#L106
I'll be happy to help out and dig deeper in SymPy code later in the day (slightly busy atm); I hope it's not too urgent, and is fine if I reply late evening or by tomorrow morning. Thanks!

Yes, from scipy import * will import scipy.datasets, because datasetsis added toall. I suggest fixing it by removing from scipy import *, that is _really_ bad practice and it's then expected to pull in optional components. There is also nothing much of interest in the main scipy` namespace that even makes it useful to star-import.

To be clear since I haven't managed to build scipy main locally does this PR make it an importerror to do from scipy import * without having pooch installed?

To be clear since I haven't managed to build scipy main locally does this PR make it an importerror to do from scipy import * without having pooch installed?

Importing everything will also include scipy.datasets submodule, which depends on pooch and we don't make it a requirement so yes, importing everything will prompt an import error and ask you to install pooch. But as Ralf said it's a bad practice to import * and that's not something you want to do.

I agree that the sympy code should be changed but does this not sound like a scipy regression to you?

Not really, no. There's many non-idiomatic things one can do, and in this case:

we have documented importing guidelines at http://scipy.github.io/devdocs/reference/index.html#guidelines-for-importing-functions-from-scipy,
not adding datasets to __all__ also seems wrong - it would avoid your SymPy issue, but only by completely omitting the optional submodule from the list of public objects. And if we'd try-except import pooch in scipy/__init__.py I'm sure other people will be unhappy by the extra overhead/import that they didn't ask for.

The sympy code should be changed in any case (although I would prefer not to have to do that at release candidate stage) so let's not be concerned with that here.
I imagine that from scipy import * is something that has worked just fine for a long time even if you don't consider it idiomatic and I expect that a lot of code depends on it. Obviously it's your call but to me this seems like a fairly gratuitous breakage.

Okay, let's open an issue for it, tag it with the 1.10.0 milestone, and see if we need to do something here before the next release. I don't think we want to change it right now in main, that is actually the right time to make changes and smoke out other possible issues.

It honestly will make a lot of users unhappy, not sure if there is a way to delay the import error raised.
Something like this inside datasets.__init__.py would avoid the error, and we can shift the error to _fetchers.py but it is obv not the best idea because now the user who wants to use this will need to install and reimport:
try:
    import pooch
    from ._fetchers import face, ascent, electrocardiogram  # noqa: E402
    __all__ = ['ascent', 'electrocardiogram', 'face']

except:
    warnings.warn("Pooch not found! Please install pooch before using scipy.datasets."
                  "After installing pooch, try importing scipy.datasets again .....")

from scipy._lib._testutils import PytestTester
test = PytestTester(__name__)
del PytestTester
....

After installing pooch, try importing scipy.datasets again

A second import statement for the same module is a no-op, so that may not do what you intend.

Ohhh I see, that's correct, will not work until the user restarts the process. I don't have any ideas atm to avoid this.

not sure if there is a way to delay the import error raised.

There are lots of ways that you can make this work if you want to make it work. The code here goes out of its way to raise ImportError. If you wanted the module to be importable but without some of its features when pooch is not installed then you just do something like:
try:
    import pooch
except ImportError:
    pooch = None
    data = None
else:
    data = pooch.create(...)

def fetch_data(...):
    if data is None:
        raise an error saying that pooch is not installed
    return data.fetch(...)

Then everywhere you currently call data.fetch you can just call fetch_data instead.

will not work until the user restarts the process. I don't have any ideas atm to avoid this

In general it is always necessary to restart the process after install or uninstalling things so that doesn't seem like a big deal. If you did want to avoid that though then you would just move import pooch into the fetch_data function rather than being at top-level.

Thanks @oscarbenjamin this is good and should work. Sorry for the negligence on my side, this was quite straightfwd. Moving pooch import into scipy.datasets._fetchers.py will do the trick. @rgommers do you think it is okay to not raise a warning/error if someone does from scipy import datasets? Now that I think it of it, i feel it should have been that way from the start. Got that wrong :/
If that's okay, I'll send a PR to fix this now.

I disagree that SymPy should change here. from mod import * is how lambdify works, to import all the names into the namespace (see the "how it works" section here https://docs.sympy.org/dev/modules/utilities/lambdify.html#sympy.utilities.lambdify.lambdify).
If SciPy is hard importing an external library with from scipy import *, then that library has effectively become a hard dependency of scipy. SymPy is hardly going to be the only instance doing an import * so this will likely affect more than just it.
I'm actually surprised that you can add something to __all__ without adding it to the main import scipy. I never realized __all__ worked like that.

I disagree that SymPy should change here. from mod import * is how lambdify works, to import all the names into the namespace (see the "how it works" section here https://docs.sympy.org/dev/modules/utilities/lambdify.html#sympy.utilities.lambdify.lambdify).

I can't understand from the link why from scipy import * is needed. You're not also recursively star-importing scipy.submodule, right? 99.x% of scipy's functionality is present in those submodules, there's only a handful of items in the main namespace. So it looks to me like this is not actually necessary. And it's most definitely bad practice.

It's pulling everything from scipy and scipy.special. I guess as Oscar showed at https://github.com/sympy/sympy/pull/23945, we really only need scipy.special.

And it's most definitely bad practice.

import * is bad practice is certain situations, but perfectly acceptable in others. If you really believe that it's universally bad, then why does scipy even define __all__?

then why does scipy even define __all__?

__all__ has always been used to define all public symbols, and we use it also to verify that all public functions are present in the API docs.


How to define endpoint route?

I am trying to migrate from serverless-rust to cargo-lambda, for deploying incrementally. One thing I cannot find in README is the way to define route and endpoint with cargo-lambda. How can I provider the following information?
This would be the config for serverless
functions:
  # handler value syntax is `{cargo-package-name}.{bin-name}`
  # or `{cargo-package-name}` for short when you are building a
  # default bin for a given package.
  hello:
    handler: hello
    events:
      - httpApi:
          path: /v1/hello
          method: get

The configuration in your example corresponds to two services, Lambda, and API GW. The httpApi configuration corresponds to API GW. This package is not designed to work with API GW. If you don't need the API GW endpoint, and you're ok with any http endpoint, you can enable the lambda URL for your function with the --enable-function-url:
cargo lambda deploy --enable-function-url ...

If you still need the dedicated API GW, you should continue using the Servlerless framework, or other tool like the AWS CDK.


provision: Fix bug in install-yarn script.

Fixes check_version function in install-yarn script so that it does not give an error if the user has another version of yarn installed in their system path.

Before when running the provision script if a user had already another version of yarn installed it was giving an error since it was comparing the version required for zulip with the user-installed yarn.
This fixes it by improving the logic for checking version!
Fixes: #23993

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

Looks good to me; @andersk @alexmv can one of you do a sanity check though?

But I think that in doing to, it raises the question of if we should be bothering to install into /usr/bin/yarn if we don't actually depend on that.  Which leads to two places where we do depend on yarn giving us the one we installed: tools/ci/production-build and tools/setup/optimize-svg.
If we're going to go this route, I think we also need to clean up those two places to use the full path to the yarn we isntalled, and stop installing into /usr/bin/yarn.

@alexmv Thanks for improving my commit message 😅.
Yeah I think we should not install yarn into /usr/bin/yarn, let's do this 👍.

Looks great -- thank you!  I squashed the two commits together (with a minor fixup to the commit message), since they're not really separable.


Fixed validation error when credentials_profile_name, or region_name is not passed

Summary
This PR corrects the checks for credentials_profile_name, and region_name attributes. This was causing validation exceptions when either of these values were missing during creation of the retriever class.
Fixes #7571
Requested reviewers:
@baskaryan

thanks @3coins!

The code example on this page needs to be updated to reflect the need for region_name:
https://python.langchain.com/docs/integrations/llms/bedrock


BLE sending notification after 2 or 3 seconds

I continuously send data from an IoT device to the android application at a frequency of 1 Hz. The data can be received at that frequency if fetched using the delays. But, when I try to fetch the data based on notifications, I observed that the notifications are being sent after 1 and 2 seconds iteratively while the app is able to read only 1 notification out of every 2 notifications received within 3 seconds. I checked with nRFConnect as well as the BLE scanner but the behavior remained the same. Is it a limitation with the BLE?

I have tried multiple devices (phones) but the behavior remained the same.


Question about density grid?

Hi,
in models/networks.py on line 160 should't it say s = min(2**(c - 1), self.scale) instaed of s = min(2**c, self.scale), in accordance with the comment on line 20?
If I'm wrong could you explain why?

Thanks for spotting this, you are correct. I fixed it. Currently cascade is always 1 and scale is 0.5 so I hope nothing is changed actually.


some files could be renamed to be effective and/or more standard

Hi there!
i think these files could be renamed to be effective and/or more standard :
gitignore -> .gitignore
README.MD -> README.md
Requirements.txt -> requirements.txt

Thanks for your input! I just fixed that along with making a big update that makes the program usable with both Llama.cpp and Ollama!


Fixing typo

Fixing a method name in closeSlidingItems method
closeItmes -> closeItems

This looks to already be fixed in source.
https://github.com/driftyco/ionic/blob/2.0/ionic/components/list/list.ts#L97
Any of the API docs get updated in the ionic source code.


[iOS] Cannot open image picker after using ModalSelector

On iOS, after using the ModalSelector, you cannot open other modals from different libraries. For example, this package breaks both react-native-image-picker and react-native-image-crop-picker's ability to launch a gallery for selecting images/videos.
My inclination is that maybe this library isn't correctly closing and cleaning up a modal after it has been "closed" via cancel or an option being selected.
This is logged in XCode when trying to call launchImageLibrary, after selecting an option from the modal selector, for react-native-image-picker:
Attempt to present <UIImagePickerController: 0x10b03dc00> on <RCTModalHostViewController: 0x113f8e0f0> (from <RCTModalHostViewController: 0x113f8e0f0>) whose view is not in the window hierarchy.
which is what makes me think react-native-modal-selector is doing something to RCTModalHostViewController
iOS Version: 14.1
React Native version: 0.62.2
react-native-modal-selector: 2.0.2
react-native-image-picker: 3.0.0-vnext.2

I had to workaround this by manually controlling the visibility of the modal, and only closing it after returning from image/video selection.

I've also opened an issue with react-native-image-picker here: https://github.com/react-native-image-picker/react-native-image-picker/issues/1456 that has additional information on the setup and usage. Although, since I get the same behavior from 2 different image picker packages I'm thinking this package is the true cause of the issue.


stageInitS4sdkPipeline running on slave node not working any more

I used to overwrite following node with slave node label to for this stage running in slave node, but it will fail all the time and change back to master node.
[Pre-condition]

My master node does not have web access to github,com, only my slave nodes have
My first slave node label is marked as 'slave1' for demo purpose.

[worked solution previously]
Modification in stageInitS4sdkPipeline.groovy
node('slave1') {
deleteDir()
checkoutAndInitLibrary(script: script, customDefaults: parameters.customDefaults)
...
[Current Error message] showing when running initS4sdkPipeline, it is changed back to master node.
[Pipeline] echo
10:24:37  [artifactSetVersion] Sending telemetry data is disabled.
[Pipeline] node
10:24:38  Running on Jenkins in /var/jenkins_home/workspace/PipelineJava_master
[Pipeline] {
[Pipeline] }
[Pipeline] // node
[Pipeline] echo
10:24:38  --- Begin library step of: mavenExecute ---
[Pipeline] echo
10:24:38  Unstash content: piper-bin
[Pipeline] unstash
[Pipeline] echo
10:24:38  Unstash failed: piper-bin (No such saved stash ‘piper-bin’)
[Pipeline] httpRequest
10:24:38  HttpMethod: GET
10:24:38  URL: https://github.com/SAP/jenkins-library/releases/download/v1.39.0/piper
10:24:38  Sending request to url: https://github.com/SAP/jenkins-library/releases/download/v1.39.0/piper
10:24:38  Treating UnknownHostException(github.com: Name or service not known) as 404 Not Found

Issue was fixed by reverting to curl.


Even domEvents = true, stopPropogation is not working

I have two elements. One is a parent with tap handler, the other is a child with tap handler. Tapping child should not trigger parent's tap handler. But in fact, it trigger!
I search for a while, but cannot find any solution.
Setting domtEvents: true cannot solve this problem.
Is a a bug? Or I need to set up something else?
Here is my code: http://codepen.io/lqzerogg/pen/wgKJwj

This is likely to be related to my issue over at the ember-gestures project https://github.com/html-next/ember-gestures/issues/73
... unfortunately I didn't get an answer over there but maybe we can come up with some solution here? 😃

In fact I use event.target and it's class name to avoid this problem. But I just think it's not a good solution.

Yeah, I also thought of "flagging" the already handled event and ignoring further events which are fired within the next 5ms or something like this....
but, all in all I think this is a qualified bug as it simply should not fire twice...

I think this particular issue is because the tap event is fired on the "up" event (pointerup / mouseup / touchend), and those are listened to by hammer at the window, not the dom element itself.  So, stopping propagation when the event is already at the window does not prevent the other recognizer from receiving it.
Listening for move / up events on the window is common practice, and allows gestures to still be detected as they leave the original target element.
Two potential workarounds I can think of:

stop propagation of all events on the nested target (http://codepen.io/anon/pen/vgLbzQ). This may be what you want anyway, and it works because the down events are listened to on the target itself, not the window
create the child recognizer first (gives it priority) and use stopImmediatePropagation to prevent the later recognizer from receiving the input. (http://codepen.io/anon/pen/apdMmj)  This still felt buggy (not sure why)

The first one sounds good!
Still, I hope hammer have a more simple option. Because I think a lot people would run into this trouble and waste lots of time searching for the answer.

I've also run into this issue…
For now I've worked around it by stopping mousedown and touchstart events on the child elements, but this is really suboptimal.
In my situation, I've got a child element type that is sometimes scrollable. If it's scrollable, then swipeup and swipedown need to be stopped from propagating.
There're swipeup and swipedown handlers further up the DOM tree.
There're also swipeleft and swiperight handlers further up the DOM tree, too (in a different place to the swipeup and swipedown handlers), but these should always fire whether or not the child element is scrollable.
Luckily there's some space around the child elements that'll mean that the UX is not broken, just tricky (if the users swipe left or right in the padding surrounding a scrollable child element, the swipeleft or swiperight handler will still fire. There're also buttons.
I've also implemented this scenario for the native wheel event, and it works great, feels very intuitive. It's a shame I can't get it to work with swipes (and presumably also pans, but I haven't gotten there yet).


Fetch the original request

The fetch request was using the URL instead of the original request which inadvertedly changed the request, also the addition for 'no-cors' isn't needed. By fetching the original request we do almost nothing differently if the request was to succeed.
See: https://jakearchibald.com/2016/service-workers-and-base-uris/
Description
Replaced event.request.url with event.request
Motivation and Context
Fixes: #56
How Has This Been Tested?
No tests ran locally.
Screenshots (if appropriate):
N/A
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.

Reading doesn't imply following...

Looks good. Thank you 👍
I am releasing a new version with the change.


I need nested subroutes

Describe the bug
function App() {
    return (<>

            <Router>
                <Route exact path="/login" component={LoginPage}/>
                <Route path="/*" component={MainPage}/>
            </Router>

    </>)
}

const MainPage = (props) => {
    const navigate = useNavigate();
    const OnSelectNavigate = (name, data) => {
        console.log(name, data);
        navigate(data);
    };
    return <>

            <Image src={AppLogo} layout="responsive" width="40px" height="40px"/>

my app

            {}

                <Menu class="flex-grow" activeName="1" onSelect={OnSelectNavigate}>
                    <MenuItem name="1" data={"/menu1"} icon={<Icon name="list"/>}>menu1</MenuItem>
                    <MenuItem name="2" data={"/menu2"} icon={<Icon name="clock"/>}>menu2</MenuItem>
                </Menu>

                <Router base={props.base}>
                    <Route path="/menu1" component={menu1Page}/>
                    <Route path="/menu2" component={menu2Page}/>
                </Router>

    </>;
};

export default MainPage;

Now, although you can access the correct page through the URL, the route is not switched and loaded normally when navigating through the menu.
Your Example Website or App
none
Steps to Reproduce the Bug or Issue
This is a project created using Vite.
Refer to the provided code.
Expected behavior
What I hope is that when using nested sub-routes, when switching between different sub-routes, the page skeletons such as menus and titles will not be refreshed and reloaded.  only the content of the sub-route will be updated.
Screenshots or Videos
No response
Platform

OS: [e.g. macOS, Windows, Linux]
Browser: [e.g. Chrome, Safari, Firefox]
Version: [e.g. 91.1]

Additional context
No response

yes. think you.


Quality Management Framework for Advice

Aspect
Description

Opportunities
Provide unified tools that allow better tracking and added value to each advice product. Increase in FAIR data.

Mitigating strategies and controls
This  is a long-term goal that will be linked to several packages of work.  The current most relevant activity is the accreditation of the data  centre.

legacy entry

Likelihood: Currently being managed by ACOM, ACOM to begin to implement the the QMF within the next 18 months as of March 2023
Impact: High for ICES, from a DIG perspective a lot has already been done with the CTS accreditation of the ICES Data Centre so it is set to medium.


Select2 in hasMany relation

Hi,
is there a way to add an option to the select2 that could open a modal for create a new item and select it.
For example I'm creating a new order that belongs to a new user, I can't find it in the list (select2 input) and so I will create it just there.
Of course a could create the user first but I would like to have this feature too.
Thanks! :)

I'll consider adding this feature.

Now you can try https://github.com/xiaohuilam/laravel-admin-select2


1 date multiple event/mark

Ionic version:  (check one with "x")
[ ] 2.x
[x ] 3.x
[ ] 4.x
Ion2-calendar mode:  (check one with "x")
[x] components mode
[ ] modal mode
I'm submitting a ...  (check one with "x")
[ ] bug report
[x] feature request
[ ] help me
Current behavior:
DaysConfig only accept 1 event/mark per date, when multiple event provided only first one will show end emitted through onSelect event
Expected behavior:
DaysConfig accept multiple event/mark per date, although shows only the first one, it can emit the rest.

I achieved it by doing the following. But it is just for displaying 1 event per line without links to the event.
config = {           date: date,           marked: true,           subTitle: 'abc'+'\n'+'def'+'\n'+'+2', }

Same thing happening with the cssClass property.


Danish translation not correct

Sorry. Just saw your request on twitter now. Hope you can use this. You have locked the thread, so I hope it s okay to open it as an issue?
"Content Downloader"
"Indholdshenter"
"Unlink Memory Card"
"Fjern link til hukommelseskort"
"ux0:id.dat has been deleted. This memory card can now be used on any Vita."
"ux0:id.dat er blevet slettet. Dette hukommelseskort kan nu bruges på en vilkårlig Vita"
"Failed to mount ux0. Is a memory card inserted?"
"Kunne ikke montere ux0. Er der indsat et hukommelseskort?"

Has created a pull request.


Add JSON metadata to the dist output

Can be added to this PR: https://github.com/vkottler/yambs/pull/42.
We'll want to add the version of yambs, maybe the date/time that the distribution was prepared?

The metadata should be included in the archives, not as a separate file.

https://github.com/vkottler/yambs/pull/42/commits/61bef161881823a7424d5c8e2b7d1ded64525d41.


Create a pattern that updates the cache

We need a pattern that can be called to invalidate locally cached modules. I'm thinking something like,
role:npm,cmd:update {name: 'some-module'}

which can then be aliased to
role:info,cache:update {name: 'some-module'}

for integration with the wider nodezoo system.
This pattern will update the cache but return no data.

@mcdonnelldean is this pattern implemented with update:true from the cmd:get pattern?
I think this is implemented and issue can be closed.
sample code from coveralls:
const payload = {name: 'seneca', update: true}
si.act('role:coveralls,cmd:get', payload, (err, reply) => {

@mihaidma yes, is implemented, this can be closed.


Can I use lambda functions inside pipelines?

Hello,
Right now I am using pipelines in several use cases, very often, in the first function I get info about the current user, and in the second function, I do something. Those both functions use SQL and Aurora serverless. I had to do that manually editing the cloud formation templates.
Now I would like to do this:
A pipeline with 2 functions, the first one gets the current user from Aurora Serverless using SQL, and the second function is a lambda function that using SES sends an email.
Is this technically possible? Using SQL and Lambda in the same pipeline? If that is the case, do you know any sample or website?
Here I read about the @function directive, but seems useless in my scenario, basically because when the client calls the function, I do need a SQL request to get info about the current user.
https://aws-amplify.github.io/docs/cli-toolchain/graphql#usage-1
Maybe, another option is to retrieve the current user inside the lambda function, but seems more complex and I have no idea about how to pass database access credentials to the lambda function. Seems a pipeline could be easier, moreover, I reuse my function to get the current user a lot in several pipelines.
What do you think?
When adding a lambda function, I see this:
"CRUD function for Amazon DynamoDB table (Integration with Amazon API Gateway and Amazon DynamoDB)"
Can you add the same option but for Aurora? When adding an API, we have that option.
Thanks a lot for your suggestions.
Which Category is your question related to?
APIs, databases and lambda.
Amplify CLI Version
4.13.3
What AWS Services are you utilizing?
AppSync, Aurora, Lambda, CloudFormation

Well, yes, it is possible. Here an example:
https://github.com/incr3m/aws-amplify-starter/blob/master/amplify/backend/api/testampl2/stacks/AppVersionQueryResource.json
You use Lambda resolvers, and once that is working, embed it into a pipeline.


Write to specified path?

I note this repo hasn’t been updated in over a year.. so not sure if still maintained? Is it possible to write md out to a specific file/path?
I might have a dig through file utils later but I have read the docs and it’s not immediately obvious :)

Actually I don't have time to create new features but it is open for the community to contribute.
I didn't test it but It is possible, provided that the path exists. For example, if you have created the directory duck/Mallard/ then you can write on the file_name='duck/Mallard/yourMarkDownFile'.


add intrinsic frequency bandwidth to laser pulse and to each slice

This should be a data member of the LaserPulse class and perhaps an array of values for the N slices.
To begin with, we will assume that the bandwidth does not affect wavefront propagation.

Given the good benchmarking results for redshifting, I don't think this is necessary.


oai removes xsi-namespace-URL

Describe the bug
at version 7.6
all metadata-views remove the namespace-link of xsi - so my national library rejected to harvest my metadata
Link to examples
(https://training-ir7.tdl.org/server/oai/request?verb=ListRecords&metadataPrefix=oai_dc)
https://tore.tuhh.de/dspace-cris-server/oai/request?verb=ListRecords&metadataPrefix=xMetaDissPlus
everytime the processor removes from my template-xsl-file
(xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance")
To Reproduce
Steps to reproduce the behavior:

Do this
... take a look at your 7.6-dspace-url with "/server/oai/request?verb=ListRecords&metadataPrefix=oai_dc" added
Then this...
... in all cases I know xsl-processor removes
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance

it doesn't matter if I change the url
only if I use an xsl-alias the url is ok, but my files cann't be harvested this way

Expected behavior
I expect to use my template as it is - import metadata in variables as I told, but not change mandatory fix parts.
I don't care if some atributes are twice or more times in the template, but missing parts that can't be added as unprocessable
(my first tries to work-around were like this) are a problem
Related work
Link to any related tickets or PRs here.
I didn't find one

I checked our 7.6 repository and your first example.  Each includes the namespace declaration, both in the root OAI-PMH element and in oai-dc elements.  From where is it missing?

I'm using e.g. curl https://scholarworks.indianapolis.iu.edu/server/oai/request?verb=ListRecords\&metadataPrefix=oai_dc | xmllint -format - | less and I see multiple instances of that namespace declaration.

I also see xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" included in the example instance at https://training-ir7.tdl.org/server/oai/request?verb=ListRecords&metadataPrefix=oai_dc
So, I'm unclear what the bug is?  It appears like this is working as expected.


validate new data with good tables

Use good tables from frictionlessdata to validate data file before integration into old data file

edited to access goodtables from command line


Allow plugins to register frontend and kernel comms

Issue Report Checklist

[x] Searched the issues page for similar reports
[x] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
[x] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
[x] Could not reproduce inside jupyter qtconsole (if console-related)
[x] Tried basic troubleshooting (if a bug/error)

[x] Restarted Spyder
[x] Reset preferences with spyder --reset
[x] Reinstalled the latest version of Anaconda
[x] Tried the other applicable steps from the Troubleshooting Guide

[x] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description
In Spyder4 the frontend communicators provide a clean way to push notifications from the kernel to the frontend. I would like to use this in a plugin, but for this some features are lacking:
A global signal, that is emitted every time a new kernel is created somewhere or attached to
This would allow the plugin to attach functions to the communicator of the kernel or to execute some code silently on kernel start. This signal should be emitted as well if a remote kernel is connected. The signal should pass the newly created kernel, so that the connected functions can modify it to their needs. Where would be the appropriate place for this signal?
Register handlers in the fronted
With the signal we can register communicators in the kernel, so we have to be able to register new handlers in the fronted as well. I guess that would be easiest if the BasePlugin implements some function to register_message_handler or so for this.
These would be my ideas for the design, do you have different ideas? I'll keep this description up to date. I would be willing to help to implement this as well.
Best
Christopher

@impact27, this one is for you. I think @ChrLackner is basically asking for an API to add kernel and frontend comms, so that third-party plugin creators can benefit from the architecture that you developed.

The comms are connected by calling KernelComm.set_shell at two occasions:

When creating shells: spyder/plugins/ipythonconsole/widgets/shell.py:ShellWidget.set_kernel_client_and_manager

When restarting kernels : spyder/plugins/ipythonconsole/widgets/client.py:ClientWidget.restart_kernel

So depending on what the needs are, I would say one of these three functions is a good place to emit a signal.
The frontends handlers and kernel calls are easy to do. If you have spyder_kernel_comm, you can just call register_call_handler to register a handler, and remote_call to call a function in the kernel.
Then to register something on the kernel side, you can execute get_ipython().kernel.frontend_comm.register_call_handler to register a call, and get_ipython().kernel.frontend_comm.remote_call to make a remote call.

So if your plug in has code in spyder-kernels, you can simply add a signal in spyder/plugins/ipythonconsole/widgets/shell.py:ShellWidget.set_kernel_client_and_manager. If instead you use get_ipython().kernel.frontend_comm. to inject code in the kernel at runtime, you will also need to handle ClientWidget.restart_kernel.

So basically what I want to do is to inject code at kernel startup to connect a signal to call get_ipython().kernel.frontend_comm.remote_call with the signal arguments if emitted. Then I want to handle this call in the frontend.
If I understood correctly, if no shell widget is created, no frontend communicator is connected? So this wouldn't work with the notebook plugin, which not necessarily creates a ipython console connected to the kernel. Or is the notebookplugin going to use the ipythonconsole plugin, but without creating a shell widget? Wouldn't it be cleaner, if this would be in some base class, which the notebookplugin widget can derive from as well?
The question would be as well who should hold the signal and how can created shell widgets as well as my plugin access it?

So basically what I want to do is to inject code at kernel startup to connect a signal to call get_ipython().kernel.frontend_comm.remote_call with the signal arguments if emitted. Then I want to handle this call in the frontend.

The kernel is not living in the same process as the frontend. Therefore, it can not emit or receive Qt signals.
The kernel client on the other hand is on the frontend side and can do that. You should use Qt signals to communicate with the kernel client. The kernel client can then send code to the kernel to be executed.
comms are specifically to communicate between the kernel and the frontend, for example the kernel client. I don't think this is what you want to do.
Maybe if you could describe why you want to do that, I could be of more help.

If I understood correctly, if no shell widget is created, no frontend communicator is connected?

Well if you don't have a kernel, there is nothing to connect to.

So this wouldn't work with the notebook plugin, which not necessarily creates a ipython console connected to the kernel. Or is the notebookplugin going to use the ipythonconsole plugin, but without creating a shell widget? Wouldn't it be cleaner, if this would be in some base class, which the notebookplugin widget can derive from as well?

I am not familiar with the notebookplugin. But if you are not using the shell widget, you can still use the kernel comm to communicate with spyder, provided you use a spyder-kernels. The only reason the comms need a Shell widget is for pdb integration and runcell/ runfile functions.

The question would be as well who should hold the signal and how can created shell widgets as well as my plugin access it?

Usually, what happens is that a Shell is created for each kernel client. This kernel client is then connected to a kernel. You can be notified when a new Shell is created.
I think I would need more explanation on what and why you want to do to help you. To summarise, the way things works now are:

Each ipython console is implemented by a shell widget (IPythonWidget). These shell widget are Qt widget and can communicate with Qt Signals.

The Shell widget has as attribute kernel_clients (QtKernelClient). It uses that to send messages to the kernel, such as code to execute.

In addition, the Shell widget has as an attribute spyder_kernel_comm (KernelComm). This kernel comm will create a second way of communicating with the kernel. It can be used for general purpose communication.

You don't need a Shell widget to work with kernel_clients or spyder_kernel_comm.

The kernel is not living in the same process as the frontend. Therefore, it can not emit or receive Qt signals.

I know that. I didn't mean qt signals here. My library has it's own signals, it is loaded in the kernel and I want to attach to these signals. Sorry for being unclear here.

Maybe if you could describe why you want to do that, I could be of more help.

I've written a plugin to integrate the GUI (written in PyQt) we have developed for the finite element library NGSolve into Spyder. A short presentation is here:
https://www.youtube.com/watch?v=yLyz9c_ew8w&t=15s
I've done this using quite ugly monkeypatching and it doesn't work if "Enable UMR" is set in the Python Interpreter settings. So after seeing your changes I want to start a new attempt to implement it in a clean way ;)
Basically I need to attach to Draw, Redraw and some other functions in NGSolve and pass their arguments to the frontend. The function, meshes,... can be pickled so they will be sent from the kernel to the frontend process. The frontend plugin handles the drawing then.
I got it working with the notebookplugin as well, but only when the notebookplugin opened a shell widget for the kernel, because all the communication is done in the shell widget. That's why I asked if that could be done in some base where the notebookplugin can derive from as well. Then I could have communicators with the notebookplugin as well to draw in my plugin.
Hope that clerifies my attempt a little, if something is still unclear let me know
Best

The kernel is not living in the same process as the frontend. Therefore, it can not emit or receive Qt signals.
I know that. I didn't mean qt signals here. My library has it's own signals, it is loaded in the kernel and I want to attach to these signals. Sorry for being unclear here.
Maybe if you could describe why you want to do that, I could be of more help.
I've written a plugin to integrate the GUI (written in PyQt) we have developed for the finite element library NGSolve into Spyder. A short presentation is here:
https://www.youtube.com/watch?v=yLyz9c_ew8w&t=15s https://www.youtube.com/watch?v=yLyz9c_ew8w&t=15s
I've done this using quite ugly monkeypatching and it doesn't work if "Enable UMR" is set in the Python Interpreter settings. So after seeing your changes I want to start a new attempt to implement it in a clean way ;)
Basically I need to attach to Draw, Redraw and some other functions in NGSolve and pass their arguments to the frontend. The function, meshes,... can be pickled so they will be sent from the kernel to the frontend process. The frontend plugin handles the drawing then.
I got it working with the notebookplugin as well, but only when the notebookplugin opened a shell widget for the kernel, because all the communication is done in the shell widget. That's why I asked if that could be done in some base where the notebookplugin can derive from as well. Then I could have communicators with the notebookplugin as well to draw in my plugin.
Hope that clerifies my attempt a little, if something is still unclear let me know
Best

In that case I think you just need to create a KernelComm item and connect it to the kernel_client you are using. I think the notebookplugin has a kernel_client.

Ah yes. I think I didn't fully understand the KernelComm. Thanks.
So the only thing I would need is to know when a new kernel_client is created somewhere and attach my KernelComm to it. At the kernel side can how can I check if my communicator is available and how can I retrieve it? If this is possible, the only thing I would need is to notify my plugin somehow when a new kernel_client is created. Could we have a global signal for that?

Ah yes. I think I didn't fully understand the KernelComm. Thanks.
So the only thing I would need is to know when a new kernel_client is created somewhere and attach my KernelComm to it. At the kernel side can how can I check if my communicator is available and how can I retrieve it? If this is possible, the only thing I would need is to notify my plugin somehow when a new kernel_client is created. Could we have a global signal for that?

If you have a spyder kernel, the comm is available by calling kernel.frontend_comm. Then you can just test kernel.frontend_comm.is_open(). If you want to get notified when the comm is opening, something should be added to FrontendComm._comm_open to call a callback or something.
I think I will change the name of KernelComm.set_kernel_client(self, kernel_client) to KernelComm.open_comm(self, kernel_client).

Ok I see. Then Then I guess the cleanest way would be if the notebookplugin creates a kernelcomm as well on kernel_client start. Then I guess the best point for emitting the signal would be at the end of KernelComm.set_kernel_client (or open_comm). Then plugins could add handlers for every kernel created.

Sry I haven't had much time lately, but I was able to implement what I wanted by monkeypatching the ipyconsole.process_started and ipyconsole.connect_ecternal_kernel functions in my register_plugin function and inserting all the code I needed there. It seems to be working, thanks for the help!


Rename Gdrive full sync queue.

Description
When moving Gdrive sync on two different queues (fullSync and incrementalSync), we could not rename the full sync queue easily because of an ongoing long full sync.
Task is here.

Risk

Deploy Plan

Deploy connectors
Check that no workflow is on the old queue. If there is one, re-trigger it via the connectors CLI.

Deployed. No Gdrive full sync happened in between so we are all good.


header: getting rid of exception-throwing behaviors in header files [include/ and source/common/]

Signed-off-by: Yifan Yang<EMAIL_ADDRESS>This is a part of efforts to get rid of all the exception throwing semantics in envoy header files, as discussed here https://github.com/envoyproxy/envoy/issues/12469.
Commit Message:
Additional Description:
Risk Level: Low. No new mechanics except the additional status check is introduced.
Testing:
Docs Changes:
Release Notes:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]

@yanavlasov could you take a look?

LGTM, I'll re-approve once format is fixed.

@yanavlasov do you mind taking a look? I believe the only test that fails here is a flaky one.

@yanavlasov @mattklein123


Getting tipsy element size with getBoundingClientRect

There is a problem right now with the map views tooltip, and it is due to the calculation of the size of the element. We use within tipsy:
https://github.com/CartoDB/cartodb/blob/master/vendor/assets/javascripts/jquery.tipsy.js#L48
But for any reason, over svg elements, that function doesn't work properly (it returns 0 for width and height). Using getboundingClientRect fixes the bug.
Tested in several browsers (firefox, safari, chrome) and in several places.
Before:

After:

CR: @javierarce
cc @saleiva
Fixes #7256.

Checked in ded02 that the fix is working properly :).

👍  +2 minor comments

Thanks!

Passing frontend tests, merging! :)


Allow running the plugin from github

I was trying to find a simple way to let people try out the generator and write a test script without downloading anything. This is the best method that I have found so far:
Gemfile
#gem "gapic-generator", github: "googleapis/gapic-generator-ruby", branch: "master"
gem "gapic-generator", path: "../gapic-generator-ruby/gapic-generator"

gem "bundler", "~> 1.15"
gem "rake", "~> 10.0"
gem "grpc-tools", "~> 1.18"

Rakefile
desc 'Generate showcase.'
task :generate_showcase do
  FileUtils.mkdir 'lib'

  command = []
  command << 'grpc_tools_ruby_protoc'
  command << '--ruby_out=lib'
  command << '--ruby_gapic_out=lib'
  command << '--proto_path ./proto/api-common-protos'
  command << '--proto_path ./proto/showcase'
  command << './proto/showcase/v1alpha3/echo.proto'
  full_command = command.join ' '

  sh full_command
end

The path version works fine, but the github version throws an error because of some files that are .gitignore'd.
$ bundle exec rake
<bunch of output>
loader.rb:17:in `require': cannot load such file -- google/api/annotations.pb (LoadError)

Is there a good way to ensure the github version works? Or, perhaps a better way to do this?

AFAIK the only way to allow this is to check the generated *.pb.rb files into SCM. This was recently done for the common protos gems, and I think it would make sense for us to do something similar in this repo. That way these files would exist and the gem could be referenced either by path or github project.
Is this a change we should make?

If that change enables this I'm all for it. Also, if any of those generated protos are exposed and would be considered part of our API for customizations I'd be in favor of doing it for that reason as well.

You can now add the gem by putting the following in a Gemfile:
gem "gapic-generator", github: "googleapis/gapic-generator-ruby"


Add public_network option to GCE provisioning

Add the option to expose a public IP address for a newly provisioned GCE instance.
Depends on:

[ ] https://github.com/ManageIQ/manageiq-ui-classic/pull/8388

TODO this dialog should move to the google provider plugin but I'd like to do that in a follow-up to keep backporting cleaner


Update props-system-size.md

We updated the pkey enumRange in 1809 to be more realistic for 2018.
should tooling have caught this?
(can contact antmor at microsoft dot com for more details.)

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: antmor sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

@antmor : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@mcleanbyron : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Thanks for submitting these changes!


BGM removal unusable

Which OS are you using?

OS: Colab
It stays at 0% and then consumes all RAM and finally it says Error:

 /usr/local/lib/python3.10/dist-packages/onnx2pytorch/convert/layer.py:30: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:206.)
  layer.weight.data = torch.from_numpy(numpy_helper.to_array(weight))
/usr/local/lib/python3.10/dist-packages/uvr/utils/fastio.py:46: UserWarning: PySoundFile failed. Trying audioread instead.
  signal, sampling_rate = librosa.load(path, sr=None, mono=False)
/usr/local/lib/python3.10/dist-packages/librosa/core/audio.py:184: FutureWarning: librosa.core.audio.__audioread_load
	Deprecated as of librosa version 0.10.0.
	It will be removed in librosa version 1.0.
  y, sr_native = __audioread_load(path, offset, duration, dtype)
/usr/local/lib/python3.10/dist-packages/uvr/models_dir/mdx/mdx_interface.py:254: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:278.)
  mix_part = torch.tensor([mix_part_], dtype=torch.float32).to(device)
/usr/local/lib/python3.10/dist-packages/torch/functional.py:704: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:873.)
  return _VF.stft(  # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/uvr/models_dir/mdx/mdx_interface.py:189: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return stft.inverse(torch.tensor(spec_pred).to(device)).cpu().detach().numpy()

in My cpuonly 32G RAM 5700G, the same thing happened. What could be the cause. Just in case I converted the opus audio into mp3 and ogg vorbis in my CPU and the same thing still happened.

Hi @starloreh.

Hi @starloreh. The stacktrace doesn't say errors, it just says warnings, which you can ignore.
The reason the progress is stuck at 0% is because it doesn't actually track the progress of the job, I just set it to 0% until the separation job is fully completed.
This confusion should be avoided, I should have said so explicitly in the progress message.
Actually, the model was probably working for separation, but very slowly on the CPU.
The UVR models are recommended to run on GPU, not CPU. I believe the speed difference is almost more than x10.
If you test it with just 2 or 3 seconds of audio, you will see that the job is done within 2 minutes on the CPU.
UVR models are super slow on the CPU, it's recommended to use the GPU only.
If you're using Colab, you can try T4 GPU runtime (free runtime).

But I was using cuda with T4!
Though the Japanese video lasts 1 hour 40 min. The System RAM gets full, but the GPU RAM doesn't seem to budge.

Great WebUI, btw! faster-whisper-large-v3-turbo-ct2 is pretty fast on my cpu, and good enough for Western European languages transcriptions.
I wanted to use faster-whisper-large-v2 with silero and BGM removal to improve japanese transcription as well translation to English.

I wanted to use faster-whisper-large-v2 with silero and BGM removal

There's a suspicious bug that seems to be related to this,
If you try to run really long audio ( more than 1hour, just like yours ) with VAD, it makes OOM error.

https://github.com/SYSTRAN/faster-whisper/pull/1198

So I guess it's related to the VAD, not the BGM separation.
This is now fixed in faster-whisper, but the new version hasn't really been released yet.
I'm considering about installing faster-whisper directly from the repository.

This should be fixed in #428.
Please feel free to reopen!

Actually, it still doesn't work. Just tried it on the new colab, and it runs out of RAM without touching the GPU. Even though cuda option was chosen.

@starloreh I just tried to reproduce it myself with 2 hours of video, but it was not reproducible. Everything worked fine on my end, even with 2 hours of video.
I'm not sure what caused the problem.
Just to be sure, are you sure you're using the latest version of the notebook here?

https://colab.research.google.com/github/jhj0517/Whisper-WebUI/blob/master/notebook/whisper-webui.ipynb

Yes. To compare I tried https://github.com/Eddycrack864/UVR5-UI with UVR-MDX-NET-Inst_HQ_5, and it almost finished till it crashed but it was another audio of 2h 20 m

I'll keep on trying and report back.


Plans for substantive next steps for template development

The goals of this issue are:

Make and discuss plans for the substantive next steps for template development
Supervise the progress of each related issue

I am assigning myself to address this.

@gentzkow
I have thought carefully about the next steps you've drafted in Roadmap and https://github.com/gentzkow/GentzkowLabTemplate/pull/14#issuecomment-2285161889, and I make some notes on how I envision to address those items, which are pulled into the .lyx document here in the branch. It would be great if we can converge on how to implement those in that document.
I also plan to use this issue and its branch for supervision purpose, like what we do in production. After we converged in the lyx document, we can ask labmates and ourselves to open the issue according to the document, address accordingly, and open a PR to merge to this issue branch. Then we merge back this branch to master to complete this round of development.
I'd appreciate if I can hear your thoughts on the plans and the workflow. Thanks!

@ShiqiYang2022 Thanks so much for taking the initiative on this! I think the most efficient thing would be to have a meeting w/ the whole predoc team to talk through these issues and make a plan. Rather than trying to create a plan for all of them, I'd suggest we try to make a plan for the first 5 or so -- enough to keep the team busy for a week at least -- then once those are done we revisit the next batch. (I think the first batch may change what we want to do on subsequent to dos so it's efficient to be sequential.) Maybe you can send around an email to schedule.

@gentzkow Thanks! I agree with https://github.com/gentzkow/GentzkowLabTemplate/issues/16#issuecomment-2337092177, I will send a email today to schedule some time later this week to meet, after Luca finish his onboarding and both Luca and Linxi have familiarized themselves in #15.

Hi @ShiqiYang2022, I just added some comments to your .lyx document here in https://github.com/gentzkow/GentzkowLabTemplate/commit/2d9cb1dcd84b91fcdf5e4f7401bfd18a349f9a55 per our discussion in #15.

Update: I have revised nextstep.lyx to reflect our decision from last group meeting. I seperate the part we have reached agreement into a new section in the .lyx document.
The file will be further updated to reflect our decision in tomorrow's meeting. For the places we have reached agreement, we should go ahead, open the issues, and then assign ourselves to address. Also, feel free to edit and correct me if I've written something incorrectly.
cc: @Xingtong-Jiang @lucamlouzada @linxicindyzeng

Hi @linxicindyzeng ,
Per discussion with @ShiqiYang2022 , we think it would be great if you could assist us with this issue, especially focusing on #21. Now that @ShiqiYang2022 has reviewed #19, I'll also open PRs soon for most of the other issues and then it would be great it you could help with reviewing those. But in the meantime, do you think you could take a look at what I wrote in #21, and @gentzkow 's reply, and then look into what you think is the best approach, especially for Stata as mentioned by him? Thanks!

Note: as we have merged several changes to the main branch, I have made a minor change in the Wiki to reflect the new approach of get_inputs.sh (#27). There will be more changes to the wiki and to READMEs in the future but I thought this one was important in case anyone is starting to use the template now.


docs(); bastion; update bastion notes

Update bastion docs. We are going to move this to prod and fix issues later as outlined in README
deployment: https://okgraph.xyz/?q=dmelotik%2Fbastion-protocol-aurora

wrong branch


Support newline after enum name, add test.

Fix for parser failing on:
enum Operator
{
}

but worked on
enum Operator {
}

@samuel can you take a look at? thanks!

Thanks! Looks good and awesome for having a test. Merging another change caused a conflict in the generated parser unfortunately. If you rebase/regenerate I'll get this merged right away.

Rebased and regenerated, thanks!


disk spilling support for hash probe

The added hash probe procedure works as follow:

hash probe operator waits for table built by hash build operators
to join
hash probe operator prepare for spilling: (1) setup spiller to spill
input rows which corresponding partitions have been spilled at the
build side.
hash probe operator process inputs and joins with the table
after all the hash probe operators have finished processing inputs,
the last operator will do last probe if the join type needs, and
notify the hash build operators of the next table to build from the
spill data if there is remaining spill data to restore.
Otherwise finish the probe operator processing.
Go back to step1 to start the next round of hash probe processing.

Unit tests are added in HashJoinTest to make each existing test case
to run test with spilling and different spilling levels.
This PR also fixes the following issues:

Add to close read only file descriptor otherwise, it will
run out of open file descriptors of the system when running
spilling continuously like in production. Detected this in
recursive spilling;
Add to clear the output row map when the build side table
is empty detected by left join types when running recursive
spilling.
Add to check if null key row selection vector is empty in
null-aware anti join processing.

The followup will add spilling stats for HashProbe

@xiaoxmeng This PR is very large, > 1K lines. Would it be possible to break it up into smaller pieces? Can the 3 fixed for pre-existing issues listed in "This PR also fixes the following issues:" be extracted into separate PRs? Smaller PRs are easier and quicker to review.

Go back to step1 to start the next round of hash probe processing.

Thank you for writing a description of the workflow. It is nice and helps reviewing. It will be also helpful for future readers who can access these descriptions from git log and git blame (as commit messages).
Step 5 is not exactly accurate, because step 2 applies only to the initial processing of the probe rows. The first pass is processing inputs and spill some of these while joining others. Subsequent passes read previously spilled rows and process all of these without spilling some. It would be nice to clarify.

@xiaoxmeng This PR is very large, > 1K lines. Would it be possible to break it up into smaller pieces? Can the 3 fixed for pre-existing issues listed in "This PR also fixes the following issues:" be extracted into separate PRs? Smaller PRs are easier and quicker to review.

@mbasmanova thanks a lot for the quick review and helpful suggestions. Yeah, I will split them out to help review.


Some small cleanup and optimization of Session.php

Hope it will be useful.

actually, the code is as is for performance reason. Anyway, the session management has been reworked in master now. So, I'm closing this PR. Last, but not the least, use the symfony/symfony repository instead of this one for PRs as this repo is read-only. Thanks.

Thanks!