Using application/x-www-form-urlencoded as the default request content type when getting access_token

First thank you for the awesome works!
Recently I am working on creating Open Bank Project OmniAuth strategy using omniauth-oauth, but keep hitting 401 unauthorized error during the get access token phase, which is using #get_access_token. Turned out if {} is not passed into #get_access_token as one of the arguments, code will run into this part and the Content-Type of the request would be missing, which kinda violate the rule of OAuth 1.0 according to http://tools.ietf.org/html/rfc5849#section-<IP_ADDRESS>.1, which says

The query component is parsed into a list
of name/value pairs by treating it as an
"application/x-www-form-urlencoded" string, separating the names
and values and decoding them as defined by
[W3C.REC-html40-19980424], Section 17.13.4.

This pull request fix the issue by implicitly declare the request body as "" and set the content type as application/x-www-form-urlencoded during the get access token phase.

Looks like this fails on Ruby 1.8, which we technically still support.

@sferik Tested using Ruby 1.8.7 against master branch locally. Got exact the same fail. Presumably something might be broken.
vagrant@localhost:/vagrant/omniauth-oauth$ git rev-parse HEAD
9f4a4f136c46eec96cb1e7f970200647e6a49b29
vagrant@localhost:/vagrant/omniauth-oauth$ ruby -v
ruby 1.8.7 (2013-12-22 patchlevel 375) [x86_64-linux]
vagrant@localhost:/vagrant/omniauth-oauth$ bundle exec rspec
WARNING: SimpleCov is activated, but you're not running Ruby 1.9+ - no coverage analysis will happen
Starting with SimpleCov 1.0.0, even no-op compatibility with Ruby <= 1.8 will be entirely dropped.
Support for Ruby 1.8.x in Addressable is deprecated.
/vagrant/omniauth-oauth/lib/omniauth-oauth/version.rb:3: warning: already initialized constant VERSION
.......F....

Failures:

  1) OmniAuth::Strategies::OAuth /auth/{name}/callback should exchange the request token for an access token
     Failure/Error: expect(last_request.env["omniauth.auth"]["extra"]["access_token"]).to be_kind_of(OAuth::AccessToken)

     NoMethodError:
       undefined method `[]' for nil:NilClass
     # ./spec/omniauth/strategies/oauth_spec.rb:101

Finished in 0.23534 seconds (files took 1.78 seconds to load)
12 examples, 1 failure

Failed examples:

rspec ./spec/omniauth/strategies/oauth_spec.rb:99 # OmniAuth::Strategies::OAuth /auth/{name}/callback should exchange the request token for an access token

I believe that there's a problem with Hashie::Mash
vagrant@localhost:/vagrant/omniauth-oauth$ ruby -v
ruby 1.8.7 (2013-12-22 patchlevel 375) [x86_64-linux]
vagrant@localhost:/vagrant/omniauth-oauth$ bundle console
Resolving dependencies...
/vagrant/omniauth-oauth/lib/omniauth-oauth/version.rb:3: warning: already initialized constant VERSION
irb(main):001:0> m = Hashie::Mash.new
=> #<Hashie::Mash>
irb(main):002:0> m.name = "foo"
=> "foo"
irb(main):003:0> m.name
=> nil

Closing as stale


Pop up an error when clicking "Load More..." from local emulator

OS: Windows
Build Version: 20200330.1
Repro Steps:

Create a new blob container from Local Emulator.
Open a folder in VS Code --> Upload the folder in Step1 to the new created blob container from Local Emulator.
Back to Azure Storage View --> Drill down from Local Emulator --> Blob Containers --> The new created blob container and refresh it.
Click the "Load More...".

Expect:
Succeed to load more.
Actual:
Pop up an error:
Element with id /attachedStorageAccounts/devstoreaccount1/Blob Containers/test0/favicons-c2a605fbc0e687b2e1b4b90a7c445cdd is already registered.

More Info:
There is no button on the error dialog.

@v-dantch are you using the Azurite extension here or the CLI?

Hi @wwlorey , yes, I have installed Azurite Extension and both Azurite Queue Service and Azurite Blob Service are running.

Add more info:
The blob display empty after clicking "Load More...".

Related: https://github.com/Azure/Azurite/issues/605

Closing as stale


Add Hanoi University of Science & Technology

Add Hanoi University of Science & Technology

@khoatran2k4 I am sorry, the domain sis.hust.edu.vn is removed from our trusted domains list because it was abused in the past. If you are the student of this school, please enroll to GitHub Student Developer Program. Then apply for JetBrains educational subscription with your GitHub account (GitHub tab at https://www.jetbrains.com/shop/eform/students).


Is there a way to remove unused styles?

I was wondering since the guide on the regular tailwindcss site explains how to do this but I'm not sure if this is possible with this library?
I'm asking because I use it for a regular react native project but am thinking of using it as well with react-native-web but since part of the performance on the web is the bundle size I was wondering if this would be an issue using this library or if there's a workaround.

Hi
As the package uses the Stylesheet class of react, react is responsible for handling styles.
The class handles the style very well and only ones when the app starts. Filtering will not improve performance.
If for some reason you think a slimmer version would be better. You can duplicate the class in this repository and remove the unneeded styles.
Greetings
Thomas


Can I make the time optional?

Hi, I am curious if is there a way to only make the user select the date and not the time? I kind of need the calendar only option in some places and the calendar+time option in others, but I cant seem to find a way to do this!.
Thanks :D great plugin

See minView - https://github.com/dalelotts/angular-bootstrap-datetimepicker#minview
Values are the same as startView
If there is something unclear about the documentation that makes this option hard to find, please submit a PR or add a comment to this issue so I can improve the documentation.


Designing a disambiguation model

Progress
StoryWeb can now load articles, run them through spaCy NER and store the extracted entity tags (e.g. John Doe) to a database. In that database, each tag is identified per article, ie. (article_id, tag, count_of_mentions, ...). There's also a database model that describes a link between two tags (A and B are the same, unrelated, or have some semantic link - e.g. family). Once two tags have a same link between them they are considered a cluster, i.e. they become essentially the same node in the graph.
There is also a small UI that lets users make those links manually - both between different tags in the same article, or tags with the same surface form across different articles.

The rationale for keeping tags constrained to one article is disambiguation: John Doe in article A may refer to a different individual than John Doe in article B.
Challenge
While disambiguation between different tags with the same surface form (e.g. two John Doe) is needed, doing this whole thing manually is intensely annoying and not even good for a prototype. What I'd like to do is to find a way to auto-decide the unambiguous/auto-decidable cases, then show the rest to the user and refine further merges based on their input.
In my mind, the core evidence for making these decisions is co-occurrence: John Doe A co-occurs with Jane Doe and Italy; John Doe B co-occurs with MegaCorp Ltd. and State Prosecutor. I'm aware that this would leave a lot of signal - the body of the documents - on the table. This goes back to wanting to build an interactive, human-in-the-loop system for better precision: keep it down to something that's explainable and where we can even re-compute clustering proposals as and while the user is providing input (active learning).
But I'm kind of stuck on this: how do I take the co-occurrence sets, model them into an input to some very simple machine learning model and then get both a set of judgements and a confidence score for each, so that I can then a) decide the ones the system is confident about, and b) show the most informative uncertain ones to a user to judge by hand, then c) re-train the model with these additional judgements.
Some things I've pondered:

Using tf/idf on the tags to score down the super common ones (Russia in my corpus is not signal, it's a stopword). I tried implementing that, and it leads to more interesting co-occurrence patterns - but not impressively so. Especially for common names (like "Vladimir Putin") the co-occurrence ends up generic as well.
Maybe this could be a simple bayes classifier (given A, B, C, what's the likelihood of this being X?)?

Stuff I want to avoid

I'd really like to avoid some sort of article-content-based mystery vectorisation (e.g. BERT), unless there's a really nice and reproducible way of productising this. However, that's what a lot of the literature is pushing:

Named Entity Disambiguation with Knowledge Graphs
Improving Named Entity Disambiguation using Entity Relatedness within Wikipedia
bert_ned

Very keen to avoid using an external knowledge base to disambiguate, because the entities we're most interested in are the ones that would not yet be recorded and identified in a KB like Wikidata - the people who work for oligarchs, kleptocrats, etc.

@pudo Friedrich, very interesting thoughts. I am working on something similar, and I think that using Spacy relation extraction co-ocurences can be factorized using inverted information entropy (or transferred information quantity, in other words). Which in an oversimplified way can be represented as 1/(term frequency). Just to start with, it's not an ideal solution.
Jane Doe established Fancy Corp in Italy
etablished - would be a strong link (tf is small)
in - could be light link (tf is high)

So for coocurences it may be:
Jane Doe - Fancy Corp + 1 link * high coeficient, because 'established' is really rare link type
Jane Doe - Italy +0.5 link (it's a link from a link, so we discount it) * negligible coeficient, because 'in' is really popular link type
Of course it should use synonyms and all other stuff. I'm oversimplifying idea
p.s. And I think that Bayes classifier is a great idea. I did this for Human-in-the-loop theme classification and it works as "magic". I didn't scale it up for production system, but prototypes worked well

@slavaGanzin I agree with what you're proposing, but my sense is that we need to address disambiguation/same-as before we can do other edge types. For tags that are not linked between articles, the co-occurrence count is always technically 1 at most ("in the article in which John Doe is mentioned, there's also Fancy Corp") - except if we consider all mentions of the same tag to refer to the same entity, which is sort of where this misery started :)


EE-1099: add client contracts for withdrawing seigniorage

Ref: https://casperlabs.atlassian.net/browse/EE-1099
See READMEs for usage instructions

bors r+


[FEATURE] PR auto merger

Description
Hello:)
My idea is to adda workflow that will automatically merge a pr from dependabot bot if all checks are success
Screenshots
No response
Additional information
No response

Thank you for the suggestion, but I prefer to test the project first with the new versions. Because it is more difficult to fix later and it can bring down production - which will make the project unstable.


CLI does not build when importing Chains module

Expected Behavior
Being able to include the chains module in the CLI and build it.
Actual Behavior
As reported in PR #245 (https://github.com/tektoncd/chains/pull/245#issuecomment-940585881), the CLI does not build due to a dependency issue when the chains module is added.
╭ cli on  chains-cli via 🐹 v1.17.2 ⌛ 4s
╰❯ go build ./cmd/tkn
# github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:117:3: cannot use "github.com/go-openapi/spec".Schema{...} (type "github.com/go-openapi/spec".Schema) as type "k8s.io/kube-openapi/pkg/validation/spec".Schema in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:146:11: cannot use ref("k8s.io/api/core/v1.Toleration") (type "k8s.io/kube-openapi/pkg/validation/spec".Ref) as type "github.com/go-openapi/spec".Ref in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:155:8: cannot use ref("k8s.io/api/core/v1.Affinity") (type "k8s.io/kube-openapi/pkg/validation/spec".Ref) as type "github.com/go-openapi/spec".Ref in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:161:8: cannot use ref("k8s.io/api/core/v1.PodSecurityContext") (type "k8s.io/kube-openapi/pkg/validation/spec".Ref) as type "github.com/go-openapi/spec".Ref in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:178:11: cannot use ref("k8s.io/api/core/v1.Volume") (type "k8s.io/kube-openapi/pkg/validation/spec".Ref) as type "github.com/go-openapi/spec".Ref in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:208:8: cannot use ref("k8s.io/api/core/v1.PodDNSConfig") (type "k8s.io/kube-openapi/pkg/validation/spec".Ref) as type "github.com/go-openapi/spec".Ref in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:240:11: cannot use ref("k8s.io/api/core/v1.LocalObjectReference") (type "k8s.io/kube-openapi/pkg/validation/spec".Ref) as type "github.com/go-openapi/spec".Ref in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:254:11: cannot use ref("k8s.io/api/core/v1.HostAlias") (type "k8s.io/kube-openapi/pkg/validation/spec".Ref) as type "github.com/go-openapi/spec".Ref in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:277:3: cannot use "github.com/go-openapi/spec".Schema{...} (type "github.com/go-openapi/spec".Schema) as type "k8s.io/kube-openapi/pkg/validation/spec".Schema in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:320:3: cannot use "github.com/go-openapi/spec".Schema{...} (type "github.com/go-openapi/spec".Schema) as type "k8s.io/kube-openapi/pkg/validation/spec".Schema in field value
vendor/github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1/openapi_generated.go:320:3: too many errors

Steps to Reproduce the Problem

cd cli
git pr 1440 upstream # Or use your favorite one-liner to pull PR tektoncd/cli#1440
go build ./cmd/tkn

Additional Info
@priyawadhwa suggested updating the github.com/tektoncd/pipeline dependency to v0.28.0 (https://github.com/tektoncd/chains/pull/245#issuecomment-943536873).

Updating the pipeline dependency as mentioned above produce the same error 😢.


[AMQ-9216] Remove deprecated java.lang.SecurityManager usage from activemq-client IdGenerator

Successfully tested locally without problem. Thanks !


Add an Infinispan source kamelet

Blocked by https://github.com/apache/camel-quarkus/issues/2344

This has been addressed.


[RFC] Proposal for a way to cache files in downstream libraries

This is a proposal following discussions started with the datasets team (@lhoestq @albertvillanova).
The goal is to have a proper way to cache any kind of files from a downstream library and manage them (e.g.: scan and delete) from huggingface_hub. From hfh's perspective, there is not much work to do. We should have a canonical procedure to generate cache paths for a library. Then within a cache folder, the downstream library handles its files as it wants. Once this helper starts to be used, we can adapt the scan-cache and delete-cache commands.
I tried to document the cached_assets_path() helper to describe the way I see it. Any feedback is welcomed, this is really just a proposal. All the examples are very datasets-focused but I think this could benefit to other libraries as transformers (@sgugger @LysandreJik ), diffusers (@apolinario @patrickvonplaten) or skops (@adrinjalali @merveenoyan) to store any kind of intermediate files. IMO the difficulty mainly resides in making the feature used :smile:.
EDIT: see generated documentation here.
EDIT 2: assets/ might be a better naming here (common naming in dev)
WDYT ?
(cc @julien-c @osanseviero as well)
Example:
>>> from huggingface_hub import cached_assets_path

>>> cached_assets_path(library_name="datasets", namespace="SQuAD", subfolder="download")
PosixPath('/home/wauplin/.cache/huggingface/extra/datasets/SQuAD/download')

>>> cached_assets_path(library_name="datasets", namespace="SQuAD", subfolder="extracted")
PosixPath('/home/wauplin/.cache/huggingface/extra/datasets/SQuAD/extracted')

>>> cached_assets_path(library_name="datasets", namespace="SQuAD")
PosixPath('/home/wauplin/.cache/huggingface/extra/datasets/SQuAD/default')

>>> cached_assets_path(library_name="datasets", subfolder="modules")
PosixPath('/home/wauplin/.cache/huggingface/extra/datasets/default/modules')

>>> cached_assets_path(library_name="datasets", cache_dir="/tmp/tmp123456")
PosixPath('/tmp/tmp123456/datasets/default/default')

And the generated tree:
    assets/
    ├── datasets/
    │   ├── default/
    │   │   ├── modules/
    │   ├── SQuAD/
    │   │   ├── downloaded/
    │   │   ├── extracted/
    │   │   └── processed/
    │   ├── Helsinki-NLP--tatoeba_mt/
    │       ├── downloaded/
    │       ├── extracted/
    │       └── processed/
    └── transformers/
        ├── default/
        │   ├── something/
        ├── bert-base-cased/
        │   ├── default/
        │   └── training/
    hub/
    └── models--julien-c--EsperBERTo-small/
        ├── blobs/
        │   ├── (...)
        │   ├── (...)
        ├── refs/
        │   └── (...)
        └── [ 128]  snapshots/
            ├── 2439f60ef33a0d46d85da5001d52aeda5b00ce9f/
            │   ├── (...)
            └── bbc77c8132af1cc5cf678da3f1ddf2de43606d48/
                └── (...)

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Cool thanks !
Can it even be possible to have the "datasets" folder one level higher ?
    datasets/
    ├── squad/
    │   ├── downloaded/
    │   ├── extracted/
    │   └── arrow/
    hub/
    └── models--julien-c--EsperBERTo-small/
        ├── blobs/
        ├── refs/
        └── [ 128]  snapshots/

Then there's "modules" (top level as well) which is the directory where we copy python scripts. IMO this one doesn't need to be part of this change (for now ?)
For context:
The "modules" directory is added to the python path in datasets and evaluate at run-time once. Inside "modules" we define "datasets_modules" and "evaluate_modules" that can be imported with importlib.import_module("datasets_modules") for example. And finally inside "datasets_modules" we have the datasets scripts that are imported as submodules of "datasets_modules".
I think we keep using one directory "modules" at the top level, this way any HF lib only need to add one directory in the python path. For now it can be excluded from the HFH cache since it only contains small files

We also use the modules folder in the cache to put a transformers_module containing the code of dynamic models/tokenizers/pipelines (that have their code on the Hub).

Thanks for the feedback :) To answer your questions 1 by 1:

Can it even be possible to have the "datasets" folder one level higher ?

What would be the goal of having it at the root of the cache ? The main advantage when I thought about it was that since it is a new folder in the cache, it would not conflict with any of the existing libraries (datasets, transformers and others). In any case, the downstream library shouldn't care too much where the cache it is only a path returned from hugginface_hub. But of course, I can be convinced here if it makes sense.

Then there's "modules" (top level as well) which is the directory where we copy python scripts.

I am fine with keeping the modules folder at root level. This folder will in any case be quite small, right ? (only python script so no data to delete to save space). But in next iterations, I don't see huge difference with having them under cache/extra/datasets/default/modules/ and cache/extra/transformers/default/modules/ (except not changing the existing code).

IMO this one doesn't need to be part of this change (for now ?)

In any case, as I see it, the huggingface_hub update would just propose a canonical path where to cache stuff. Since stuff is already implemented in datasets, transformers (and cie), we don't have to change everything at once. Sections of the cache can be iteratively moved to the extra/ cache but there is no rush in that.

The "modules" directory is added to the python path in datasets and evaluate at run-time

Does datasets and evaluate share the same python module folder ? If (and only if) we move them to the extra/ cache, we could point evaluate to import for the extra/datasets/modules/ folder since we maintain both librairies. In general I think it's best if each library uses only its own cache but modules is a quite specific use case.

I think we keep using one directory "modules" at the top level, this way any HF lib only need to add one directory in the python path
We also use the modules folder in the cache to put a transformers_module containing the code of dynamic models/tokenizers/pipelines

Yep, very specific use case (and already existing) so no need to change IMO.

What would be the goal of having it at the root of the cache ?

Users of datasets that played with the cache before would expect their data to be in .cache/huggingface/datasets as before I think - or at least in a directory at the same level with a clear name ("datasets-cache" or something like that).
Therefore if the main concern is to avoid conflicts I think we can just rename it "datasets-cache"

I am fine with keeping the modules folder at root level. This folder will in any case be quite small, right ?

Yup it's quite small (usually a few KB per dataset)

Does datasets and evaluate share the same python module folder ?

Both add .cache/huggingface/modules in the python path. But in practice they use different folders inside "modules": "datasets_modules" for datasets and "evaluate_modules" for evaluate. So their caches are different, and each lib handles its cache its own way. And we only need to add one directory to python path.

Therefore if the main concern is to avoid conflicts I think we can just rename it "datasets-cache"

I would also like to avoid conflicts with existing folders for the future scan-cache and delete-cache tools. If everything lives under the extra/ directory, I would not have to care about special folders that would not use the same structure like modules or doc_builder. For now I only have those 2 examples on my machine but idk if there are more. Having everything in a separate folder extra/ on which huggingface_hub is the only lib to deal with the structure (at least for the first 3 levels of depths, then under cache/extra/datasets/SQuAD/extracted you can put anything you want) would avoid edge cases.
Also, we it would allow to have a clear distinction between folders that can be scanned via huggingface-cli and the others. If we have datasets/ and datasets-cache/ together next to each other I'd find it more confusing actually. WDYT ?

Yup it's quite small (usually a few KB per dataset)
Both add .cache/huggingface/modules in the python path

Then let's not focus too much on that. It light-weight and already working so no need to touch it.

(nit) I just thought about it but renaming extra/ to assets/ sounds better I'd say (I would expect users/devs to understand it better). And cached_assets_folder as helper name.

Ok I see ! I like assets more as well. I'm ok with going for assets then (I just remembered we had a quite explicit message printed every time a dataset is loaded to show its cache location anyway, so it should do the job)

yes this structure is good for us (datasets) and it means that scripts are stored not in modules/datasets_modules but in a specific dataset folder (like squad/python)
but it would require either adding both datasets-cache and evaluate folders to python path which might lead to collisions, or higher level ~/.cache/huggingface directory, but this way it might lead to collision with transformers, though the latter seems unlikely because it shouldn't contain __init__.py, as far as I understand, so tell me if I'm wrong.

Hi @Wauplin,
Indeed yesterday we had a meeting of the datasets team and we discussed about this topic.
IMO, I totally understand your point: so that we keep current behavior of the hub cache and just add the required specificity by datasets with as less impact as possible to current implementation (in a different directory).
The request by @lhoestq and @polinaeterna of putting everything in a per-dataset directory, has an underlying need: to be able to easily find/delete all cached files of a specific dataset:

with their approach, this can easily be done by listing/removing the corresponding dataset directory
with your approach, that will be possible by using the scan-cache and delete-cache functions:

Therefore, if we agree on your proposed directory structure:

can we list/delete all cached files of a specific datasets, regardless of whether the files are in one directory (hub) or another (extra or assets)?

Additionally, for the "modules" directory, I think you were right (@lhoestq could you confirm this?): they need to be in a separate parent directory (that will be added to the import path), for all datasets: datasets_modules/. Therefore, not possible to have it in a per-dataset directory (squad/python/). Also note that these files are a copy of the previously cached files in your "hub/" directory (scripts are downloaded from the Hub).
Another remark: please note the specificity of the "downloaded" directory:

until now, we put there whatever file we downloaded, either from the Hub or from 3rd party servers
with the new implementation, the "downloaded" should only be used for 3rd party servers; files downloaded from the Hub will be in your "hub/" directory
I think this difference must be implemented downstream by datasets team.

I see, thanks @Wauplin and @albertvillanova :) sounds good to me

Additionally, for the "modules" directory, I think you were right (@lhoestq could you confirm this?): they need to be in a separate parent directory (that will be added to the import path), for all datasets: datasets_modules/. Therefore, not possible to have it in a per-dataset directory (squad/python/). Also note that these files are a copy of the previously cached files in your "hub/" directory (scripts are downloaded from the Hub).

We could also add the assets directory to sys.path, and name its subdirectory datasets-cache to avoid a module name collision with datasets, but I don't have a strong opinion on this - we can keep the modules dir if we want

We could also add the assets directory to sys.path, and name its subdirectory datasets-cache to avoid a module name collision with datasets, but I don't have a strong opinion on this - we can keep the modules dir if we want

Not so much opinionated about this but if you do so I think you should align with the transformers team as well.
Worth noticing that assets/ can be managed by any downstream library (e.g. any lib will be able to have its own directory under assets/) so maybe it's not ideal to add everything to the python path.

I see - let's keep modules as it is then

Ok everyone, thanks for the feedback and discussions. I think we reached a common agreement for this PR then.
When it comes to the tools to scan/delete those assets, we can discuss them in a separate issue. They will come once we start to integrate this assets folder with datasets and other downstream libraries :)
@LysandreJik @lhoestq I've ping you for the review but other contributors are welcomed to review if wanted. Thanks in advance !

Thanks for the review @LysandreJik , I'm merging :)

Awesome thanks :)


Conflict with nvim-ts-rainbow

Hi. I have below error when I have enabled nvim-ts-rainbow plugin for treesitter and going to reference via Glance. I don't know if it's error related to Glance or nvim-ts-rainbow.
Error:
Error executing lua callback: Vim:E315: ml_get: Invalid lnum: 1
stack traceback:
        [C]: in function 'bufload'
        vim/_editor.lua: in function 'region'
        /usr/local/share/nvim/runtime/lua/vim/highlight.lua:30: in function 'range'
        ...share/nvim/lazy/nvim-ts-rainbow/lua/rainbow/internal.lua:86: in function 'update_range'
        ...share/nvim/lazy/nvim-ts-rainbow/lua/rainbow/internal.lua:194: in function 'cb'
        ...l/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:780: in function '_do_callback'
        ...l/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:220: in function 'invalidate'
        ...l/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:921: in function '_on_detach'
        /usr/local/share/nvim/runtime/lua/vim/treesitter.lua:73: in function </usr/local/share/nvim/runtime/lua/vim/treesitter.lua:69>
        [C]: in function 'nvim_buf_delete'
        ...local/share/nvim/lazy/glance.nvim/lua/glance/preview.lua:177: in function 'close'
        ...i/.local/share/nvim/lazy/glance.nvim/lua/glance/init.lua:502: in function 'close'
        ...i/.local/share/nvim/lazy/glance.nvim/lua/glance/init.lua:443: in function 'jump'
        ...i/.local/share/nvim/lazy/glance.nvim/lua/glance/init.lua:300: in function <...i/.local/share/nvim/lazy/glance.nvim/lua/glance/init.lua:299>

Hey, i tried nvim-ts-rainbow and it works fine for me.
I noticed this plugin is no longer maintained, is this the one you're using?
What is your neovim version?

@DNLHC I was using mrjones2014/nvim-ts-rainbow. I have changed it to hiphish/rainbow-delimiters.nvim for now and looks like this issue is solved. I won't use nvim-ts-rainbow since it's deprecated.
I've had another issue from lewis6991/gitsigns.nvim while working with Glance, but I've changed loading event from BufReadPre to BufRead in lazy.nvim and issue has been solved.


Base64 encode response body if the rack response is compressed

Description
Currently Rack responses that are compressed to either gzip or br (Brotli) encodings will be passed by Lamby to AWS Ruby runtime client which fails to parse encoding with the following error. This is due to AWS Runtime Client by default, running to_json on handler response value
More info here - https://github.com/aws/aws-lambda-ruby-runtime-interface-client/blob/5e669fe30b48362cac758bc2d2c161787adb2e84/lib/aws_lambda_ric/aws_lambda_marshaller.rb#L30
Error raised from handler method
"errorMessage": "\"\\x8B\" from ASCII-8BIT to UTF-8",
"errorType": "Function<Encoding::UndefinedConversionError>",

Base64 encoding response body before handing over to AWS runtime client will solve the issue.
Note: Lamby already recommends setting X-Lamby-Base64 header to indicate rack adapter that the content requires base64 binary encoding here - https://lamby.custominktech.com/docs/asset_host_and_precompiling.
Perhaps this PR will prevent the need for setting that header altogether? @metaskills
Changes

Base64 encode response body if the rack response is gzip or brotli compressed

CONFIRMED WORKING! Thanks a ton.

CONFIRMED WORKING! Thanks a ton.

Released v2.5.3

Released v2.5.3


Premium Tier Billing Granularity

Is the Premium Tier billed at hourly or daily granularity?  Say a Premium Tier is created and deleted after 2 hours.  Would the cost be for 2 hours or 1 day (24 hours)?
Both this document and https://azure.microsoft.com/en-us/pricing/details/service-bus/ suggest daily granularity, but according to https://stackoverflow.com/questions/51144712/need-help-to-understand-service-bus-message-unit it's actually hourly and the docs need to be updated.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f8b75004-9c4b-37ee-a3f1-b58354a497b7
Version Independent ID: 5748d76c-fba0-939b-f43d-046e9738866a
Content: Service Bus pricing and billing
Content Source: articles/service-bus-messaging/service-bus-pricing-billing.md
Service: service-bus-messaging
GitHub Login: @axisc
Microsoft Alias: aschhab

Hi @mikeharder Thank you for your feedback! We will review and update as appropriate.

@mikeharder I believe hourly is how its done now and the pricing page seems to reflect the hourly price as well though the FAQ section still needs updating.
Some other docs also seem to require updating in a few places I think

Service Bus pricing and billing doc
FAQ in the pricing page
Service Bus Premium and Standard messaging tiers doc

On the bright side, Azure Calculator seems to have been updated properly

We are assigning this issue to the content author to review and update accordingly.

#in-progress
This article needs to be removed and redirected to the page: https://azure.microsoft.com/en-us/pricing/details/service-bus/

The redirection is in place now.
#please-close


Add auth for all HTTP APIs using shared secret Bearer token

Task
Add auth for all HTTP APIs using shared secret Bearer token
./minio-kv --token=$TOKEN

curl localhost:8081/get-blob/my-file.json -H "Authorization: Bearer $TOKEN"

^ I'd expect something like the following.
If --token is empty or not given, then disable auth

@alexellis working on it, what do you mean by disable the auth if --token is not given or empty? do you mean by then the user won't allow to access the resources?

PR: #5

/close: fixed in #5


vrchat update broke some of the endpoints

this message just gets spammed and moderations doesn't get updated so my guess they changed the endpoint for that

ok it was just the endpoint to check playermoderations

I can confirm this.

this message just gets spammed and moderations doesn't get updated so my guess they changed the endpoint for that

Yeah, they disabled the 'CheckPlayerModerationsAgainstYou' endpoint for added privacy.

its not as much for privacy as the web api still has access to it they either renamed or removed it for the client api. now this gets spammed to idk what they disabled this time

503 seems to occur every second attempt fetching auth/user/friends so uhhhh yup...

oof is that what they blocked

oof is that what they blocked

I doubt it, the reason player moderation was removed was because its now handled by Photon through game client making Friends handled by photon aswell would break the parts of the vrchat website

503 seems to occur every second attempt fetching auth/user/friends so uhhhh yup...

503 seems to happen whenever the socials endpoint gets overloaded. See the last few announcements in the discord:

In times where issues arise, we're able to preserve instances and ongoing sessions by temporarily reducing friends list functionality.

They've been messing with stuff lately trying to fix/compensate/upgrade, but it's turned into a mess.

It seems VRChat have fixed the 503's, for now...


diagonalizer-in-e-basis suffers numerical instability

The following Quil snippet triggers an error during compression in quilc 1.8.2-hotfix:
DECLARE ro BIT[2]
H 0
H 1
X 0
PHASE(-13.823007675781957) 0
X 0
PHASE(-13.823007675781957) 0
RZ(-17.592918860086126) 0
RZ(-22.61946710582502) 1
CNOT 0 1
RZ(22.61946710582502) 1
CNOT 0 1
X 0
PHASE(-1.2566370614347233) 0
X 0
PHASE(-1.2566370614347233) 0
RZ(-2.5132741228694466) 1
RZ(-2.5132741228694466) 0
CNOT 0 1
RZ(2.5132741228694466) 1
CNOT 0 1
X 0
PHASE(-6.283185307173617) 0
X 0
PHASE(-6.283185307173617) 0
RZ(-7.539822368608339) 1
RZ(-7.539822368608339) 0
CNOT 0 1
RZ(12.566370614347234) 1
CNOT 0 1
H 0
RZ(-1.2566370614278395) 0
H 0
H 1
RZ(-1.2566370614278395) 1
H 1
MEASURE 0 ro[0]
MEASURE 1 ro[1]

The ultimate source of the error appears to be in diagonalizer-in-e-basis, which attempts to group the eigenvectors emitted by magicl:eig into common eigenspaces. The eigenspaces in the above sequence are completely degenerate: everything ought to belong to the eigenspace of weight 1, but d-i-e-b sorts them into pairs of weight 1±eps, which causes some cascading failure further down the line (specifically: find-real-spanning-set goes off the rails).
I propose we first look into Gavin's randomized algorithm for performing this computation. If his approach only deals with instability in find-real-spanning-set, then that's no help here; but if it obviates us from having to sort the eigenvectors as well, then that could be a good approach. Failing that, I don't have an immediate idea for an approach to this.

We initially suspected that the large angle values appearing in the above snippet (e.g., -22.61946710582502) were part of the underlying cause, but Michał has since reported that he can trigger this bug with angles constrained to [-2pi, 2pi].

@ecpeterson with today's changes:
CL-USER> (let* ((p "DECLARE ro BIT[2]
H 0
H 1
X 0
PHASE(-13.823007675781957) 0
X 0
PHASE(-13.823007675781957) 0
RZ(-17.592918860086126) 0
RZ(-22.61946710582502) 1
CNOT 0 1
RZ(22.61946710582502) 1
CNOT 0 1
X 0
PHASE(-1.2566370614347233) 0
X 0
PHASE(-1.2566370614347233) 0
RZ(-2.5132741228694466) 1
RZ(-2.5132741228694466) 0
CNOT 0 1
RZ(2.5132741228694466) 1
CNOT 0 1
X 0
PHASE(-6.283185307173617) 0
X 0
PHASE(-6.283185307173617) 0
RZ(-7.539822368608339) 1
RZ(-7.539822368608339) 0
CNOT 0 1
RZ(12.566370614347234) 1
CNOT 0 1
H 0
RZ(-1.2566370614278395) 0
H 0
H 1
RZ(-1.2566370614278395) 1
H 1
MEASURE 0 ro[0]
MEASURE 1 ro[1]")
                (pp (quil:parse-quil p)))
           (quil::print-parsed-program (quil:compiler-hook pp (quil::build-nq-fully-connected-chip 3))))
DECLARE ro BIT[2]

RZ(2.3812748610449956) 1                # Entering rewiring: #(0 1 2)
RX(pi/2) 1
RZ(0.440570547736455) 1
RX(-pi/2) 1
RZ(-0.7603177924704826) 1
MEASURE 1 ro[1]
RZ(-1.9527009134848787) 0
RX(pi/2) 0
RZ(0.9775965506226627) 0
RX(-pi/2) 0
RZ(-2.92074085228763) 0
MEASURE 0 ro[0]
HALT                                    # Exiting rewiring: #(0 1 2)
NIL


Refinement of "ignore unknown fields" guideline

We currently have the general rule "Should: Prefer Compatible Extensions" and the more client specific rule "Must: Prepare Clients for Compatible API Extensions (the Robustness Principle)".
Both are partially overlapping -- I think we should specialise the first to be server specific and avoid redundancies and even inconsistencies.
For the server side, I'm not sure whether here robustness principle generally applies; there are some problems, for instance...
a) might not be compatible with semantics of operation, e.g. PUT for resource storage service should not ignore unknown input fields and do not store it without any feedback
b) ignore unknown input fields may have deferred impact to future extensions which are supposed to be compatible, however, might break clients that use a different type here
Having this in mind, I propose on server side "ignore unknown fields" is not a "should", but a "could" rule.
If robustness will not be provided, I would expect that still additional properties is true -- because I want to have the option to extend the data structure in the future (see issue #164 ) -- but a generic error code (which one?) is returned.
Yes, different services might want to take different approaches with "unknown fields" -- but the service provider should be aware of the problems mentioned above and be explicit if it is not supported.

short summary what we discussed today in Guild meeting:

client and server view must be clearly segregated
clients must ignore unknown fields in servers output data
when server ignores unknown fields in its input data then it has the following effects:

PUT and GET may not be symmetric, unknown fields can be ignored by PUT will not be visible subsequent in GET output
certain client errors cannot be recognized by server, e.g. typing error for optional fields will be interpreted as additional unknown field ignored instead of corresponding server error feedback
future extensions of the input data structure will not be compatible, i.e. break clients that already use this field but with different type

servers might want to take different approaches with "unknown fields" -- but the service provider should be aware of the problems mentioned above and be explicit in what is supported
discussed yet w/o consensus: should we recommend servers to ignore or not to ignore unknown fields

certain client errors cannot be recognized by server, e.g. typing error for optional field names will be interpreted as additional unknown field ignored instead of corresponding server error feedback

Why?

For this case @whiskeysierra, do you mean server should recognise typing errors in field names?

I understand typing errors as using the correct field name but the wrong data type. Are you talking about typos in the field names?

yes, I mean "field name typing errors" -- server cannot differ between client intensional provided additional field vs. client field name typing error where he intensionally wanted to provide an optional input field.

OK, got it.

short summary what we discussed today in Guild meeting:

client and server view must be clearly segregated

👍

clients must ignore unknown fields in servers output data

👍 . We'll want to clarify wording slightly - "must ignore" as intended means "must not crash", but it's ok for generic processing engines to handle generic json (ie it's not required that clients drop data)

when server ignores unknown fields in its input data then it has the following effects:

PUT and GET may not be symmetric, unknown fields can be ignored by PUT and will not be visible subsequent in GET output

This makes sense for POST/PATCH, but PUT means replace and HTTP's pretty clear about the default roundtrip expectations RFC7231 section 4.3.4:

A successful PUT of a given
representation would suggest that a subsequent GET on that same
target resource will result in an equivalent representation being
sent in a 200 (OK) response.  However, there is no guarantee that
such a state change will be observable, since the target resource
might be acted upon by other user agents in parallel, or might be
subject to dynamic processing by the origin server, before any
subsequent GET is received.

But as is typical with HTTP, it leaves enough room for some vagueness ("dynamic processing" 😞 ). Anyway, I think we'll want clear guidance here if PUT is allowed to be overloaded to means partial/selective storage/update.

certain client errors cannot be recognized by server, e.g. typing error for optional field names will be interpreted as additional unknown field ignored instead of corresponding server error feedback

👍 This isn't actionable client/server guidance at runtime, but useful because it clarifies who's responsible for issues.

future extensions of the input data structure will not be compatible, i.e. will break clients that already use this field but with different type

👍

servers might want to take different approaches with "unknown fields" -- but the service provider should be aware of the problems mentioned above and be explicit in what is supported

👍 This is relevant to event processing or any scenario where there's long term storage and you're expected to be able to read historical data back out. I think where this is critical we can consider supporting team with media type formats that have clearer semantics than JSON Schema and canonical json mappings (eg Avro/protobuf3)

discussed yet w/o consensus: should we recommend servers to ignore or not to ignore unknown fields

I think it's best to be clear on the tradeoffs in the guidance as per the suggestions above, and provide guidance on how to clearly signal write-side validation errors maybe with standard Problem URIs, and document must-ignore style processing for POST/PATCH/PUT, that would result in data being dropped by the server.

I totally agree, that clients should ignore unknown fields.
When it comes to the server side, I think we should introduce a rule, that server should (or event must) NOT ignore unkown fields (in PUT, POST, PATCH). IMHO the server defines the API and API enhancements are driven by the server side. So I think the server should introduce new fields first, before clients are allowed to use them. API enhancements are not driven by the clients by just introducing new fields. It is driven by developers of clients talking to the team providing the API.
Typos or scenarios where a client app thinks that certain fields are stored, but the server silently ignores these fields, are dangerous and very hard to detect. So I would argue, that unknown fields should not be ignored.

There might be one use case where the server should accept unknown fields: When the server side is a generic storage engine. But in this case the server usually makes no assumptions about the format (or semantics) of the stored data. So I would not expect that the only data type defined in the API spec is a string, that then might contain any data structure, that is opaque to the server.

fixed by PR #248


Added ProgressBar | Issue #69

I have successfully added a ProgressBar in the WebApp.
Completed #69

@CogitoKunal sir, please review and merge this

Your Contribution is accepted

Hey Please Solve Signin not Working

@CogitoKunal you haven't added gssoc-ext labels and also label of level.

@MastanSayyad sir, sorry for disturbing again, but you just have to add labels, pleaseeeeeeee.

@CogitoKunal sir you haven't added labels


Please Create use case Guides

I found this project by digging through the CCS. One of the proclaimed features is that there is

Automatic monero release upgrades, verified with SHA256SUMS

My use case is setting up a monero node on a VPS like DigitalOcean and making sure it stays up to date. Currently, to update the daemon, I have my own bash script to do so. And to ensure the daemon is always running with the a configuration, I also have a systemd configuration. This was pretty annoying to get setup, so I think this tool could be useful but the readme does not clarify what exactly it takes care of.
I also have custom configuration to set the path to the blockchain and set the bandwidth speed to unlimited.

the readme does not clarify what exactly it takes care of

The README tells you exactly what monero-bash does. Did... did you read it?

In terms of a server context.
Does it take care of ensuring the node is always running? 🤷♂. Does it ensure the node is always updated? 🤷♂. All I know is that it can install the node and use the wallets and update when told to update. This is all fine on a client but in a server, a guide would be very helpful.


Fixed the text in portuguese brazilian translation

Hey @tiagobarreto, would you resolve the merge conflicts?

@mgechev fixed. :)


Re-order rules

Is it in your plans to add a way to reorder the rules? Today I have to recreate one to be able to add them in the order I want on the WIT.

No plans yet. I have some other priorities right now. But I'll keep this issue open once I get some free time to work on it

Hi @mohitbagra - thank you for this extension, we use it frequently.  Will there be any update on the ordering issue?  We also need to recreate when wanting the buttons in a specific order on the form.


feat: support shimmer view

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[BUG] allen_cord_20um meshes are not centered

This image.sc issue led me to do some digging, and it looks like:

The allen cord atlas is (nearly) symmetrical
The annotations and reference image (nearly) align properly with the generated hemispheres image

However, the meshes don't seem be centered. I'm not 100% sure what that means TBH, as I don't know exactly what the coordinate system is that they're in. However, when the meshes are cut down the midline by vedo (via brainrender), you don't get exactly half. To reproduce:
from brainrender import Scene
from brainrender import settings
settings.SHADER_STYLE = "plastic"
scene = Scene(atlas_name="allen_cord_20um")
scene.add_brain_region("WM", hemisphere="left")
scene.render()

Gives:

This then seems to cause the downstream issues in brainrender-heatmap. Not sure if other atlas have this issue.

Currently, the left/right split is done by calculating the center_of_mass in for the root mesh in brainrender and taking that as the midpoint. Not sure why that's the case for this specific atlas. As far as I can tell it's symmetric.
See:
https://github.com/brainglobe/brainrender/blob/9cd07c8ad33fcbed0c8a5e396f9b330a5634603e/brainrender/scene.py#L287-L312
For some reason the center_of_mass doesn't align with the geometric? center. I can correct this by just taking the centre of the mesh bounds for the root mesh. That fixes this issue for the spinal cord atlas, and doesn't change the functionality for most of the other atlases.

However, this breaks this functionality for the mpin_zfish_1um atlas.
Left hemisphere:

Right hemisphere:

I noticed that the mpin-zfish_1um atlas is one of the only one that has a hemispheres.tiff. Is that because the hemispheres aren't symmetrical?
Not sure how to proceed, one idea I had was to take the geometric center for atlases that are marked as symmetric and rely on the hemispheres property for the asymmetric atlases.

I noticed that the mpin-zfish_1um atlas is one of the only one that has a hemispheres.tiff. Is that because the hemispheres aren't symmetrical?

Yep, if they're symmetrical we programmatically generate the hemisphere image, otherwise it's packaged along with the atlas. TBH I don't know if that's correct for this atlas though.

This specific issue has been closed by https://github.com/brainglobe/brainrender/pull/375, and these issues have been raised to fix specific atlases:

https://github.com/brainglobe/brainglobe-atlasapi/issues/359
https://github.com/brainglobe/brainglobe-atlasapi/issues/360


Optimized ML validator creation.

Description
Please include a summary of the change and which is fixed.

Skip  ML validator creation when no ML candidates have to checked.
concerned test refactored

How has this been tested?
Please describe the tests that you ran to verify your changes.

[x] UnitTest
[x] Benchmark

Codecov Report
All modified lines are covered by tests :white_check_mark:

Comparison is base (abc5491) 90.85% compared to head (1c1902d) 90.80%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #440      +/-   ##
==========================================
- Coverage   90.85%   90.80%   -0.06%
==========================================
  Files         125      125
  Lines        4168     4175       +7
  Branches      659      661       +2
==========================================
+ Hits         3787     3791       +4
- Misses        249      250       +1
- Partials      132      134       +2

Files
Coverage Δ

credsweeper/app.py
87.11% <100.00%> (-1.13%)
:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Outdated node checks and polyfills

Current Behavior
There are old and outdated node version checks which are not supported anymore. Also some polyfills for the unsupported node.
Expected Behavior
Remove outdated code.
Steps To Reproduce
No response
Example flow
paste your flow here

Environment

Node-RED version: v4-dev
Node.js version: 18+

@Rotzbua thanks for all the PRs. I think this issue is complete now?


Invalid acr task list-runs sample code

"az acr task list runs --name mytask --registry myregistry --output table" should be "az acr task list-runs --name mytask --registry myregistry --output table"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7e863c45-bc89-0762-7bc1-525c122cf6ec
Version Independent ID: 3937ce4a-4428-bcfc-7ea7-760705947409
Content: Schedule Azure Container Registry tasks
Content Source: articles/container-registry/container-registry-tasks-scheduled.md
Service: container-registry
GitHub Login: @dlepow
Microsoft Alias: danlep

@pengzhisun
Thanks for the feedback! We are currently investigating and will update you shortly.

@pengzhisun Thanks for reporting.
I will submit a PR.

@pengzhisun Thanks for reporting. I am able to reproduce the issue.
I will submit a PR.

@jakaruna-MSFT I've already sent a PR for this issue: https://github.com/MicrosoftDocs/azure-docs/pull/36866

@pengzhisun Nice.  But i also submitted one.
Anyway One of them will merge and it will fix the issue.
Thanks for the PR


explain on how to extend this container with own configurations

Hi,
it would be great if you could write a sentence or two on how to integrate own configurations into the container.
If managed to get DKIM working by bind-mounting over the generated exim4.conf.localmacros like
- ./00_local_macros.txt:/etc/exim4/exim4.conf.localmacros:ro
however, now I want to add a new config that would add VERP by mounting it into
- ./verp.conf:/etc/exim4/conf.d/rewrite/40_exim4-config_verp
but as the container is using a single-config file, it won't pick up this config.
I don't want to bind-mount the whole config file into the container, as then I would lose the flexibility gained by your sed-replacements (ips and so on..)
any hints / suggestions are highly appreciated!

@domdorn mounting custom confguration is not supported because it needs to be included in the custom config. PRs are always welcome to support it.
Thanks

sent a PR, please see
https://github.com/namshi/docker-smtp/pull/27
would be great if you could review it and push a new version!
thx

@domdorn the PR has been merged, would be closing this issue.
Thanks


chore(ci): update papyrus doc

Stack:

#2317 ⬅
#2316

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

This change is


types: make css(...args) allow falsy values

Closes https://github.com/chakra-ui/panda/pull/1264#issuecomment-1698394167
📝 Description
css(...args) are already filtered to only take objects, this PR makes the typings lenient with falsy values, mostly intended for component props that have optional values like cssProps?: SystemStyleObject and then later css({ ... }, cssProps)
💣 Is this a breaking change (Yes/No):
no

this is definitely an improvement but i'm running into one more usability problem.
right now css() can take multiple styles, but css.raw() cannot. So if I want to share a single style, that's easy by exporting a css.raw() system style object, but if I want to share a complex style comprised of multiple system style objects, I'm left either:

exporting a SystemStyleObject[] (downside: the API of using this style is now not the same as the API of using a single system style object)
as a convention, exporting all shared styles as SystemStyleObject[] even if there is only a single one so that I can have consistent usage (downside: it is not natural to by convention wrap single values in arrays for no obvious reason except in other modules)
updating usages of exported styles to always go through a wrapArray() function that will wrap a single style object in an array if it is not already an array of styles and then splatting that onto css() calls (downside: this is a significant amount of boilerplate littering the codebase)

It seems like a natural extension would be to have css.raw() also be able to take a list of style objects instead of just a single one and perform the same style merging. Or alternatively, a separate explicit separate composition function that isn't css() or css.raw(). Thoughts? @astahmer


Unable to clear trash when deleting dead symlink

Expected behavior
When a dead symlinked file is deleted, it should be able to be cleared from the trash directory
Actual behavior
When a dead symlink is deleted, it correctly goes into the trash directory, but the trash restore/empty commands no longer work as long as the dead symlink is there
Steps to reproduce behavior
touch test
ln -sf test test2

trash put test
trash empty

trash put test2
trash empty

The final command will result in following error
thread 'main' panicked at 'assertion failed: file.exists()', /build/trashy-2.0.0-vendor.tar.gz/trash/src/freedesktop.rs:238:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The same was happening to me.
This issue is an issue with trash, but it was corrected by this commit. So updating trash dependency version from 3.0.1 to 3.0.2 should solve the issue.

This issue is an issue with trash, but it was corrected by this commit. So updating trash dependency version from 3.0.1 to 3.0.2 should solve the issue. At least I have tried, and it solved my case.

Completed by #101.  I think we can close this issue.

Pls reopen

The issue was solved by committing this but this change is not incorporated to version 2.0.0 (as you can see here).

Any idea when this will be incorporated in a new release?


vaccination Q10d wrong influenza season

This is a question for the current influenza season, beware of the misleading text, as of today should be 2022/2023 but it's still 2020/2021:
https://github.com/influenzanet/common-study-definition/blob/8394166ed8577f527c483ccd925e5838ffcf78d5/src/studies/common/questionPools/vaccinationQuestions.ts#L431

Fixed in e301d6f563a1f068b5ca0babd539bd324b10d619, reference text is now generic


Enable Http2 Support

Changes Introduced
Enables Http2 support on both plain and TLS ports
Applicable linked issues
#29
Checklist (check all that apply)

[x] This change maintains backwards compatibility
[x] I have introduced tests for all new features and changes
[x] I have added documentation covering all new features and changes
[x] This pull-request is ready for review

CC @alexrashed @whummer . Not sure if this changes anything on your end, but kinesis-mock should now support http2 on both ports.

superseded by #557


Allow alternative panning modes

I wasn't exactly sure if this is cytoscape.js-panzoom specific but I wanted to see if there was anyway to use touchpad scrolling to pan instead of as the zoom function.
Oftentimes I would like to pan quickly at a consistent zoom level, but I wasn't sure if there were options for that in cytoscape.js itself or if it could be added here

One "UI" idea for this would be to add a button that toggles "scrolling = zoom" or "scrolling = pan"

That would need to be implemented in the core lib.

It could be good in some contexts, but it can also conflict with the touchpad swipe to go back gesture in the browser...
Maybe it would be more universal to make a keyboard shortcut using cy.panBy() with something like Mousetrap.js?


[DOCFIX] Clean up Eng Doc - remove catalog and transformation

What changes are proposed in this pull request?
Removing document for

transformation
catalog

Why are the changes needed?
As part of effort of Eng Doc clean up, we have decided to remove the document of experimental / deprecated features to avoid confusion.
Does this PR introduce any user facing changes?
Please list the user-facing changes introduced by your change, including

Eng document

@yuzhu @Rianico @binarylu my first try on doc removal, please review. Once approved, I will merge this one when i get confirmation from field. Thanks!

@apc999 hi Bin, i see that Trino official doc has info related to Alluxio catalog service too, when this pushes, we should work with them to take it out too?

alluxio-bot, merge this please

alluxio-bot, merge this please


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v2.6.30 to before v4.19.222",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=d4e0fe01a38a073568aee541a0247fe734095979",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=cc9b655bb84f1be283293dfea94dff9a31b106ac"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "d4e0fe01a38a073568aee541a0247fe734095979",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "cc9b655bb84f1be283293dfea94dff9a31b106ac",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "igbvf: fix double free in `igbvf_probe`\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v4.19.222 by commit cc9b655bb84f1be283293dfea94dff9a31b106ac, it was introduced in version v2.6.30 by commit d4e0fe01a38a073568aee541a0247fe734095979. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2021-1002653


Honour the --service-token flag properly.

Bug from #1357

why two flags for the same thing?

why two flags for the same thing?

legacy; --service-token used to be translated into -token on the probe.  Ie --probe.token.


Custom TFLite model for objection detection

I'm trying to change the default object detection to a custom one that I created using EfficientDet-Lite however it´s not working properly, does anyone have a tutorial or a indication on how to help?

Are you able to run it with the official sample?
If not, please ask it in the official community.
If you can run it in C++ but not in Unity, then please share your current code.

I would also like to know more about this. How to convert an existing .tflite model to .bytes and import it.
I have converted my model to .pb format and changed the ending to .bytes according to this post.
I then went into all the files that referred to the old model and changed the name.
grep -l -r "mediapipe/models" . *.txt
./Mediapipe/Samples/Scenes/Object Detection/object_detection_cpu.txt
./Mediapipe/Samples/Scenes/Object Detection/object_detection_gpu.txt
./Mediapipe/Samples/Scenes/Object Detection/object_detection_opengles.txt

But I get this error:
Error tflite The model is not a valid Flatbuffer buffer
Error Unity MediaPipeException: INTERNAL: Graph has errors:
Error Unity Calculator::Open() for node "TfLiteInferenceCalculator" failed: ; Failed to load model from path mediapipe/models/yolov5n6_size_320.tfliteflite_model_loader.cc:41)

Please make sure TFLiteInferenceCalculator can load your custom model first.

I would also like to know more about this. How to convert an existing .tflite model to .bytes and import it.
I have converted my model to .pb format and changed the ending to .bytes according to this post.

If you have a .tflite model file, all you need to do is just to change the extention from .tflite to .bytes.
.tflite and .pb are different format and I think .pb format won't work with TFLiteInferenceCalculator.

Thank you, using .tflite model and changing the ending worked and it was able to load the model.
But the inference result was not correct because I think it is not doing the Non-maximum Suppression (NMS) step correctly for this model.

I think it is not doing the Non-maximum Suppression (NMS) step correctly for this model.

This is out of scope for this project, so if you want to ask someone, please ask elsewhere.

@pasmai Hey, did you find any solution to this by any chance? I am facing the same issue with the custom object detection as the model is not recognising the object properly. It only works if I place the camera up close to the object, but even that fails most of the time. I have tried it the official mediapipe version on Android and the results were far better than this.
I am checking out the code if it's working properly but that is the only thing that I can do. I don't have any knowledge on how the Inference Calculator Or the Non-Max Suppression Calculator works. Would really help me a lot if you have found a fix for this.

Leaving this here if anyone else is stuck.
So I was checking out some other issues and found that there is a settings menu in the scene. On clicking that, there are 2 running modes Async and Sync modes. The object detection works properly on Sync mode although the frame rate is lower which I can understand as Inference takes time on cpu. So try switching to Sync mode, that will work out and in case you need any help for making custom model for object detection, please make sure that you are using ssd_mobilenet_v2 as that is the only supported version for object detection for mediapipe at the moment along with Tensorflow 1.15. Any model made with Tensorflow 2.0 and above would not work with mediapipe for now, hope they update this soon.
Anyways, would like to know if there is anything else I can do to increase the frame rate, that would be helpful.

@Sketh14 No updates from my side.

@UnityDev1021 I have ALMOST made it according to what you suggested - i have custom ssd_mobilenet_v2 model but I'm having a hard time converting it and using it with mediapipe. Any chance for quick call?
my email is<EMAIL_ADDRESS>I will show what I have and I would really appreciate any tips in this matter


Fix grace time not getting honored

Hey,
This pull request should resolve #7.
I applied the fix proposed by @BuonOmo verbatim.
I also added a test checking if the grace time has elapsed before busy sidekiq processes receive the shutdown signal.
Please take a look at the changes and let me know if you want to change anything.

LGTM, thank you for this contribution!
@ccyrille I'll let you merge if that suits you?

Thanks for the quick merge and gem update!


Fix a typo in Media Foundation Transforms

Nothing fancy, just a typo I noticed while browsing the docs.

@ivan100sic : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Loading web URLS (Images)

Can I use web URLS as images  instead of calling images like below  int[] sampleImages = {R.drawable.slide_1, R.drawable.slide_2};

look at issue #20

Hello, you can use Picasso:
Picasso.with(context).load(url).into(imageView);
I use that example code in the two examples of documentation link and link.
Picasso Library

Hello, you can use Picasso:
Picasso.with(context).load(url).into(imageView);
You can use that example code in the two examples of documentation link and link.
Picasso Library

How if multiple url and image?

You can use this code
public class SampleCarouselViewActivity extends AppCompatActivity {

    CarouselView customCarouselView;
    List<String> imageUrls = new ArrayList<String>();

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_sample_carousel_view);

        customCarouselView = (CarouselView) findViewById(R.id.customCarouselView);
        customCarouselView.setPageCount(NUMBER_OF_PAGES);
        // set ViewListener for custom view
        customCarouselView.setViewListener(viewListener);
    }

    ViewListener viewListener = new ViewListener() {

        @Override
        public View setViewForPosition(int position) {
            View customView = getLayoutInflater().inflate(R.layout.view_custom, null);
            ImageView myImageView = customView.findViewById(R.id.myImage);
            Picasso.with(context).load(imageUrls.get(position)).into(myImageView );

            return customView;
        }
    };

in setViewForPosition method, you can image load and it is able to multiple image load

you can simply use this Library UrlImage


update license to Jupyter

@ellisonbg what's the best way to credit Cal Poly contributors to the Data 301 starting point here?

I would just mention that this was started at Cal Poly by Brian Granger and
Jon Frederic. Originally Jon did some work on this in a different version,
but I am the sole dev of this version.
On Thu, Apr 14, 2016 at 7:19 AM, Min RK<EMAIL_ADDRESS>wrote:

@ellisonbg https://github.com/ellisonbg what's the best way to credit
Cal Poly contributors to the Data 301 starting point here?
You can view, comment on, or merge this pull request online at:
https://github.com/jupyterhub/jupyterhub-deploy-teaching/pull/1
Commit Summary

update license for JupyterHub

File Changes

M LICENSE
https://github.com/jupyterhub/jupyterhub-deploy-teaching/pull/1/files#diff-0
(4)

Patch Links:

https://github.com/jupyterhub/jupyterhub-deploy-teaching/pull/1.patch
https://github.com/jupyterhub/jupyterhub-deploy-teaching/pull/1.diff

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/jupyterhub/jupyterhub-deploy-teaching/pull/1

--
Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
<EMAIL_ADDRESS>and<EMAIL_ADDRESS>

In the LICENSE or README for that notice, then?

@minrk @ellisonbg As an FYI, I put an acknowledgment page at the end of the Sphinx docs PR yesterday. Please let me know if you want that modified. Thanks.

Merging this since the docs mention on the index page and on the acknowledgement page. I'll loop back and put in the README. We can always add to LICENSE if desired.


Could you please provide some examples?

Hi,
I've just spent some time with eww and I think it is a great project, however the documentation is extremely inadequate - especially for those (like myself) that have never done any gtk-programming.
So if you don't have the time to provide proper documentation could you at least provide some examples?
At the moment the projects you link to still use the old xml-config and so don't work at the moment while all you provide is one (ONE!!!!) example of a bar....

heres some more users configs: https://github.com/druskus20/dots/tree/master/eww/.config/eww
https://github.com/elkowar/dots-of-war/tree/master/eww-bar/.config/eww-bar
What do you find lacking in the current documentation? Some more concrete points would be helpful, so I know what to add

One can easily find dotfiles repositories using eww :P with a google search maybe, or even reddit search...

@siduck76 Isn't it what the guide already does?

its a bit confusing for normies like me :(

I'll close this for now, given that we recently added a couple more example configs to the readme!

@ghistes I have a few examples here, mostly me playing around with revealers

I am not quite sure why, but only one of the examples in your repo worked for me (Revealer Hover Module). The rest ended up just showing boxes that did not do anything. I could see fluctuation in the process manager that some sort of activity was happening when mousing over, but it still just displayed an empty box.

@ghistes I have a few examples here, mostly me playing around with revealers

I am not quite sure why, but only one of the examples in your repo worked for me (Revealer Hover Module). The rest ended up just showing boxes that did not do anything. I could see fluctuation in the process manager that some sort of activity was happening when mousing over, but it still just displayed an empty box.

same!

@elkowar the input widget doesnt work properly, I wasnt able to type anything in it

I would need to test another one to be sure, but I believe the issue is that trying to use another config folder outside of the main default is basically more problem-causing than it is useful. I took the config files I had in there if the one that worked, put them in another folder and copied in just the files from another example instead of trying to do eww -c /path/to/other/example. I guess that is probably why only part of it showed up and worked. It feels like, if the example/config is not designed with being run from another folder to begin with, it doesn't seem to work. If it doesn't work/keep track of the config path that you pass to it when it starts, then I don't really understand why it even exists? Unless there is just more to it that I am not getting? Maybe making a wrapper to automatically reuse the first config you pass to it throughout the lifetime of the current executable, and if none, then use $HOME/.config/eww/ might help, but would just have to try it out and see.
Either way, I guess, just put the files in the default place.


WIP: angular + universal module definition

Here is a WIP at making angular work with UMD (based on this format).
I did not attempt to remove angular from the window. That's a bigger project; so much of the source relies upon window.angular. Instead I just made angular export itself as a define/module.exports in addition to the window.
I still don't know how to test the umd code, since tests don't include *.prefix and *.suffix files.
I thought of moving the UMD code out of angular.prefix/module.prefix and into the code somewhere - but don't know where.  Feedback is welcome - I will add tests as soon as it's possible.
Related Issue: #4919

+1

Afaict, this has been obsoleted here: https://github.com/angular/angular.js/issues/4919#issuecomment-69672853


Upgrade Docker base image?

It seems like the Docker base image is Ubuntu 18.04. Are there any plans to upgrade? I would have expected Ubuntu 22.04, or possibly Ubuntu 20.04.

Hi @vonschultz,
we may expect to upgrade VitisAI base image to Ubuntu20.04 for VAI 3.0 release early Jan next year.


Indicate what versions of the JSII an assembly is targeting

Currently when emitting a JSII using project Python has to include a dependency on the jsii python runtime library. However there's no indication what versions of this runtime library a particular assembly (or output of pacmak at least) is intended to be targeting. This means that we can only emit an unversioned dependency on just jsii.
It would be ideal to have some mechanism to derive what the supported JSII runtime version(s) are.

The code generator has to be coded against a particular version of the runtime library, and this means it has authority to determine what version of the runtime it needs to depend on. Typically, the version will match that of jsii-pacmak.

@dstufft and @garnaat – does this resolve your concern around versioning/compatibility?  Or do you see a concern that requires deeper validation?

Confirmed with Mitch that this will do what we need.


Cloudfare POST request returns error code 500: "This domain does not exist"

Hello! I am trying to bypass a site called BSTN. They recently implemented Cloudflare and I have decided to use your library to bypass it. You can tell if you visit the link: https://raffle.bstn.com/air-jordan-1-retro-high-og-obsidian- that they have Cloudflare enabled. I tried GET requests using cloud scraper to their raffle page and that worked fine however the POST endpoint: https://raffle.bstn.com/api/register returns the error:
{"name":"GeneralError","message":"Forbidden: This domain does not exist","code":500,"className":"general-error","data":{},"errors":{}}

My code for the post request is:
var cloudscraper = require('cloudscraper');

var options = {
    method: 'POST',
    url: 'https://raffle.bstn.com/api/register',
    headers: {
        'cache-control': 'no-cache',
        Referer: 'https://raffle.bstn.com/air-jordan-1-retro-high-og-obsidian-',
        Connection: 'keep-alive',
        'Content-Type': 'application/json;charset=utf-8',
        'Accept-Language': 'en-US,en;q=0.5',
        Accept: 'application/json, text/plain, */*',
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:68.0) Gecko/20100101 Firefox/68.0'
    },
    body: '{"additional":{"instagram":"hdbfff"},"title":"male","email":"test@gmail.com","firstName":"Fake","lastName":"Name","street":"Yesfv Avenue","streetno":"3132","address2":"","zip":"60141","city":"New York","country":"US","acceptedPrivacy":true,"newsletter":false,"state":"New York","recaptchaToken":"(insertValidRecaptchaToken)","raffle":{"raffleId":"air-jordan-1-retro-high-og-obsidian-","parentIndex":0,"option":"40"},"issuerId":"raffle.bstn.com"}'
};

cloudscraper.post(options).then(console.log);`

My version of Cloudscraper is 4.1.4
This error occurs on every request using a normal cloudflare request. The way I got it to work locally is:
Code snippet
var cloudscraper = require('cloudscraper');
var request = require("request");
var rp = require('request-promise');
cloudScraper.debig = true

cloudScraper();

function cloudScraper() {
    var options = {
        method: 'POST',
        url: 'https://raffle.bstn.com/api/register',
        headers: {
            'cache-control': 'no-cache',
            Connection: 'keep-alive',
            'Accept-Language': 'en-US,en;q=0.5',
            Accept: 'application/json, text/plain, */*',
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:68.0) Gecko/20100101 Firefox/68.0'
        },
        body: '{"additional":{"instagram":"hdbfff"},"title":"male","email":"test@gmail.com","firstName":"Fake","lastName":"Name","street":"Yesfv Avenue","streetno":"3132","address2":"","zip":"60141","city":"New York","country":"US","acceptedPrivacy":true,"newsletter":false,"state":"New York","recaptchaToken":"03AOLTBLQt8oFAs7FHLdt12D2hrroUdIEexUALXv9msV_1nMwVOZ3IiIrl_-Oo4lDXxwO6hiV6zwo_3nKHlS8w16u9-R4cnn-ZcNvuLTkH04C6Fl5T6KCG1JPM0ysebufyuxsH5S9vUJNWfgP7ej1hE1nybfd9_VjqWeq8o_f-QolO45eqst3DCrZ4FYDinsNwzB3w84UaqihYfcO3HLR3fVBMcJM4C5HIBS2n1xyZKccbxeTcAr__BWrrS9VOuUidKb8RKa6fz8NcGXz0EC4a2Oxav5g1aj9DrNtOsyDo3yuma6a8h5XGVbpD8MfeStk3QqmY_LT8znnE","raffle":{"raffleId":"air-jordan-1-retro-high-og-obsidian-","parentIndex":0,"option":"40"},"issuerId":"raffle.bstn.com"}',
        JSON: true
    };
    cloudscraper.post(options).then(onResponse).catch(onError);

    async function onResponse(response) {
        await delay(4000)
        var request = await response.request.toJSON()
        var headers = await JSON.stringify(response.headers, null, 2);
        var cookies = JSON.stringify(request["headers"].cookie).split(";")

        await console.log("Clearance: " + cookies[1].replace(`"`, ""))
        await console.log("UID: " + cookies[0].replace(`"`, ""))

        submitRaffle(cookies[0].replace(`"`, ""), cookies[1].replace(`"`, ""))

    }

    function onError(error) {
        console.error(error.stack);

        if (error.cause) {
            console.log('Cause: ', error.cause);
        }

        if (error.response) {
            onResponse(error.response);
        }
    }
}

function submitRaffle(uid, clearance) {
    var cookie = `_ga=GA1.2.860778441.1565723692; cookieconsent_status=dismiss; ${uid}; __cf_bm=3892a44b19e11906234dd85e0e6a812495735938-1567037712-1800-AUyxq0xIEoeer5COx5/3mL8k659l+fBOGmFA85z+3YgE+0+zfKNkujSUPrh3cMhOejH7411YkYbcKLRTlSmKByU=; _gcl_au=1.1.1318149485.1567037735; _gid=GA1.2.728282739.1567037736; ${clearance}`
    var body = '{"additional":{"instagram":"conor.de"},"title":"male","email":"conorebeling@gmail.com","firstName":"Conor","lastName":"Ebeling","street":"Wesley Avenue","streetno":"1325","address2":"","zip":"60201","city":"Evanston","country":"US","acceptedPrivacy":true,"newsletter":false,"state":"Illinois","recaptchaToken":"03AOLTBLQUZj3k8q7ZN-lyTVzVcqI8kcLaVqHDe_7Bl3GbeV5VW4ZjrfqzSVs-YZnhYHZ2h3RYkVLRhUTtmyhBLBYVRNZY7EP_YuNL1WAP4iZbSD9fJtZDlEyfQyTM57g96Zn5wVaNPXu0roVjRfgW-kG7pARZWvkBAnmFXqtp4et6XWIvtcM3wdP2c4eU9qpCv5XvZnFeveuLKp4ZgxZHKMEVm-wHD4TMYbxkBGSoC2t31ukXfLGUJo8A_045wRMdS-NPBroyDoDvBVNbAc7k5lbJcFVfSz9LQ25xtd9Udr5bXG02WcnE8Z-JW3H6_ILh9J0EyVFneYVc","raffle":{"raffleId":"air-jordan-1-retro-high-og-obsidian-","parentIndex":0,"option":"40"},"issuerId":"raffle.bstn.com"}'

    console.log("Final Cookie: " + cookie)
    var options = {
        method: 'POST',
        url: 'https://raffle.bstn.com/api/register',
        headers: {
            'cache-control': 'no-cache',
            'Content-Length': '767',
            Host: 'raffle.bstn.com',
            'Cache-Control': 'no-cache',
            Cookie: cookie,
            Referer: 'https://raffle.bstn.com/air-jordan-1-retro-high-og-obsidian-',
            Connection: 'keep-alive',
            'Content-Type': 'application/json;charset=utf-8',
            'Accept-Language': 'en-US,en;q=0.5',
            Accept: 'application/json, text/plain, */*',
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:68.0) Gecko/20100101 Firefox/68.0'
        },
        body: body
    };

    request(options, function (error, response, body) {
        if (error) throw new Error(error);

        console.log("Status: " + response.statusCode);
        console.log("Body: " + body);
    });

}

function delay(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

I have tried using valid Recaptcha tokens and it makes no difference. Please let me know if you have any insight into this error, thanks!

Hi @ConorEB,
The error has nothing to do with Cloudscraper. It's an rest API error so if you were to simulate the exact same request with your browser, you'd get the same response. If there was an error with cloudscraper, you'd see a stacktrace unless things REALLY went bad.
The error doesn't have anything to do with Cloudflare either.
If the response was from Cloudflare, there would a Server: cloudflare header. Instead we see that the response type is JSON, in this scenario, Cloudflare wouldn't respond like that.

'Content-Type': 'application/json;charset=utf-8',

I don't know if you're misusing the server's rest API or if the server is just buggy but there's nothing we or Cloudflare can do to fix this. You'll need to determine what the problem is on your own.
Cheers

A couple of suggestions:

Don't specify the headers unless it's required.
Use the json: true option and the library will set the content-type header for you.

:+1:

Still having issues, tried a request-promise request and it failed but regular request-js works. Is there a way that I can just get the cookies to pass to request-js? Or use request-js instead of the promise version.

Yes and yes.
But first would you mind editing your issue?

Please use this template: https://github.com/codemanki/cloudscraper/issues/new?assignees=&labels=&template=bug_report.md&title=

What works with request but doesn't work with Cloudscraper?

Please share working code snippets.

Done! Github is not making all of my code formatted correctly so sorry about that :/

I'm just guessing but if it worked with request but not request-promise then the problem you're experiencing most likely bulls down to the response status code not being 2xx. The reason this makes sense is because request-promise doesn't do anything different from request, it uses request, same options, etc.
Maybe try simple: false so request-promise doesn't complain?
I'll take a more in-depth look a little later. Until then, you can try to use request instead of request-promise:
// The callback argument is required when using `request` as the requester.
const cloudscraper = require('cloudscraper').default({ requester: require('request') });

Thanks for the help! FYI: When I try using request like that that this error pops up:
throw new TypeError('Expected a callback function, got ' +
^
TypeError: Expected a callback function, got undefined instead.

@ConorEB
I've checked the code. The only difference between request and request-promise is the latter throws a StatusCodeError for non-2xx responses (500 in this case). The only different between request or request-promise and Cloudscraper is the response is Cloudflare's IUAM JS challenge response when not using Cloudscraper. When using Cloudscraper, the challenge is solved for you and the rest API error which is a user error is seen.

TypeError: Expected a callback function, got undefined instead.

You have to use a callback as I stated when I provided the example code.
My apologies but this isn't a problem with any of the libraries or Cloudflare.
You need to figure out how to use the site's API properly.
Good luck

Hey, last thing before I go off on my own.  The second request in your snippet defaults to a GET requests because the method was not explicit. I can tell that's the second request because the JSON: true (which is a typo) isn't set on the request object. It should be json: true btw.
Try correcting some of the the typos:
// cloudscraper.debig = true;
cloudscraper.debug = true; // <-- So you can see the debug information
cloudscraper.get({ url, json: true })

Fixed the typos and json:true. I verified that it has to do with GET/POST because I copied the cookies over to postman and it gave the same error when using GET. I am defining both cloudscraper.post(options) and method:"POST" in both requests so I am confused when you say my second request defaults to GET. Thanks for the help by the way... I see that this is totally my fault and just wanted to let you know that what you are doing is amazing and to keep it up :).

Thanks,
Cloudscraper never specifies or changes the request method unless you're using the reCAPTCHA helper API and you're not using that. The underlying request library might change the request method to get in the case of 3xx redirects but I forget. In any case, I can tell you that none of these libraries are playing tricks on you. If you use the verb methods e.q. request.get or cloudscraper.post the method will be overridden to GET or POST respectively. So if you use the verb methods, the method option will be ignored.
It's easy to see what requests are being made: cloudscraper.debug = true
I hope you get everything working!

The data that I showed you above was from cloudscraper.debug = true. Which is why I was confused why I was getting a GET instead of POST even though I defined
`    var options = {
method: 'POST',
url: 'https://raffle.bstn.com/api/register',
headers: {
'origin': 'raffle.bstn.com',
'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'accept': 'application/json, text/javascript, /; q=0.01',
'referer': 'https://raffle.bstn.com/air-jordan-1-retro-high-og-obsidian-',
'authority': 'raffle.bstn.com',
'x-requested-with': 'XMLHttpRequest'
},
body: body,
json: true
};
cloudscraper.post(options).then(console.log, console.error)`

Here is my entire code snippet, you can tell me if I am defining anything incorrectly. I will also include the debug details.
``var cloudscraper = require('cloudscraper');
cloudscraper.debug = true
submitRaffle();
async function submitRaffle() {
var body = await '{"additional":{"instagram":"conor.de"},"title":"male","email":"conorebeling@gmail.com","firstName":"Conor","lastName":"Ebeling","street":"Wesley Avenue","streetno":"1325","address2":"","zip":"60201","city":"Evanston","country":"US","acceptedPrivacy":true,"newsletter":false,"state":"Illinois","recaptchaToken":"03AOLTBLQUZj3k8q7ZN-lyTVzVcqI8kcLaVqHDe_7Bl3GbeV5VW4ZjrfqzSVs-YZnhYHZ2h3RYkVLRhUTtmyhBLBYVRNZY7EP_YuNL1WAP4iZbSD9fJtZDlEyfQyTM57g96Zn5wVaNPXu0roVjRfgW-kG7pARZWvkBAnmFXqtp4et6XWIvtcM3wdP2c4eU9qpCv5XvZnFeveuLKp4ZgxZHKMEVm-wHD4TMYbxkBGSoC2t31ukXfLGUJo8A_045wRMdS-NPBroyDoDvBVNbAc7k5lbJcFVfSz9LQ25xtd9Udr5bXG02WcnE8Z-JW3H6_ILh9J0EyVFneYVc","raffle":{"raffleId":"air-jordan-1-retro-high-og-obsidian-","parentIndex":0,"option":"40"},"issuerId":"raffl.bstn.com"}'
var options = await {
    method: 'PUT',
    url: 'https://raffle.bstn.com/api/register',
    headers: {
        'origin': 'raffle.bstn.com',
        'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'accept': 'application/json, text/javascript, */*; q=0.01',
        'referer': 'https://raffle.bstn.com/air-jordan-1-retro-high-og-obsidian-',
        'authority': 'raffle.bstn.com',
        'x-requested-with': 'XMLHttpRequest'
    },
    body: body,
    json: true
};

await cloudscraper(options).then(console.log, console.error)

async function onResponse(response) {
    if (error) throw new Error(error);

    console.log("Response: " + response)

    var body = response.body

    await console.log(JSON.stringify(body));
    await console.log("Email: " + email);

    if (body.message == "success") {
        await sendSuccessWebhook("You have entered into the BSTN raffle with the email: " + email + ", proxy: " + proxy + ", instagram account: " + instagram + ", and cloudfare bypass token: " + clearance, webhookLink)
        await console.log("Sucessful Entry with Email: " + email)
        await res.send("Sucessful Entry with Email: " + email)
    } else {
        await sendErrorWebhook("There was an error entering into the BSTN raffle with the: " + email + ", cloudfare bypass token: " + clearance + " and error: " + body.message, webhookLink)
        await console.log("Error: " + JSON.stringify(body));
        await res.send("Error: " + JSON.stringify(body));
    }

    await res.send("Entry Completed!")
};

async function onError(response) {
    console.log("Response: " + response)

    var body = response.body

    await console.log(JSON.stringify(body));
    await console.log("Email: " + email);
}

}``

@ConorEB
I stand corrected, you are specifying POST for both requests. The reason you're seeing the GET is because of redirects where request appears to be changing the method to GET. That behavior is to be expected with redirects.
REQUEST redirect /api/register
REQUEST redirect to /api/register

The solution:
const cloudscraper = require('.');

const xhrHeaders = Object.assign(cloudscraper.defaultParams.headers, {
  'X-Requested-With': 'XMLHttpRequest'
})

cloudscraper.debug = true;

const  options = {
  url: 'https://raffle.bstn.com/api/register',
  headers: xhrHeaders,
  json: true,
  formData: {
    foo: 'bar'
  }
};

cloudscraper.get({ simple: false, url: 'https://raffle.bstn.com/api/register' }).then(() => {
  // Now that we have the cf_clearance cookie
  cloudscraper.post(options).then(console.log).catch(console.error);
}).catch(console.error);

Hey, I think we are almost there but still getting some errors... can we chat in https://tlk.io/cloudflare to make it easier?

I really appreciate all of the help by the way, I just need to drive it home :)

I think it's preferable to handle the redirect manually in this case:
const cloudscraper = require('cloudscraper');
const errors = require('cloudscraper/errors');
const url = require('url');

const xhrHeaders = Object.assign(cloudscraper.defaultParams.headers, {
  'X-Requested-With': 'XMLHttpRequest'
})

cloudscraper.debug = true;

const  options = {
  simple: false,
  resolveWithFullResponse: true,
  url: 'https://raffle.bstn.com/api/register',
  headers: xhrHeaders,
  json: true,
  formData: {
    foo: 'bar'
  },
  followAllRedirects: false,
  followRedirect: false
};

cloudscraper.get(options).then((response) => {
  if (/^2/.test('' + response.statusCode)) {
    return response.body;
  }

  if (/^3/.test('' + response.statusCode)) {
    return cloudscraper.post({
       ...options,
       url: url.resolve(options.url, response.headers.location),
       // Don't allow future redirects, etc..
       simple: true
    });
  }

  throw new errors.StatusCodeError(response.statusCode, options, response);
}).then(console.log, console.error);

The followOriginalHttpMethod option is supported by requests and thus by Cloudscraper as well.
requests README excerpt

followRedirect - follow HTTP 3xx responses as redirects (default: true). This property can also be implemented as function which gets response object as a single argument and should return true if redirects should continue or false otherwise.
followAllRedirects - follow non-GET HTTP 3xx responses as redirects (default: false)
followOriginalHttpMethod - by default we redirect to HTTP method GET. you can enable this property to redirect to the original HTTP method (default: false)
maxRedirects - the maximum number of redirects to follow (default: 10)
removeRefererHeader - removes the referer header when a redirect happens (default: false). Note: if true, referer header set in the initial request is preserved during redirect chain.

Hey thanks that seems to do the trick (sorry for the late reply). The site or cloudflare seemed to have flagged my IP so I am working on getting good proxies to use. Do you have any recommendations as to why cloudflare seems to detect my server every time I try to run the script on it? Not sure if it has to do with specific ciphers or my IP. Anyways let me know, I will be here 🙂
Sent with GitHawk

@pro-src
Sent with GitHawk

Hey @ConorEB,
You can rule out the IP issue by tunneling your local traffic through your remote server. I recommend using Chromium (Google Chrome) for a very clear test since it uses BoringSSL. If you experience the same problem in Chromium when tunneling, it's an IP issue.
If it's not an IP issue or you're hoping for an easier way to determine the cause, here's a couple of things that you make try:

Try the suggestions mentioned in the README.md: https://github.com/codemanki/cloudscraper#recaptcha
Use a different version of Node.js, maybe try 12.x.x. Note the current version of OpenSSL that Node.js was compiled with and find a different version combination. Check with node -p process.versions.

If you need to know which ciphers are being sent, you can use https://www.howsmyssl.com/a/check
const cloudscraper = require('cloudscraper')
cloudscraper.get({ uri: 'https://www.howsmyssl.com/a/check', json: true }).then(console.log)

The TLS inspection conducted by Cloudflare goes beyond checking which cipher suite is advertised by the client and excluding ciphers might not help. AFAIK, Cloudflare's TLS checks only flag insecure TLS connections, meaning things like SHA1 being in the signature algorithms will result in a CAPTCHA.
You can reach me on IRC.


KAFKA-15552 Fix Producer ID ZK migration

This patch fixes a problem where we migrate the current producer ID batch to KRaft instead of the next producer ID batch. Since KRaft stores the next batch in the log, we end up serving up a duplicate batch to the first caller of AllocateProducerIds once the KRaft controller has taken over.

Hey I was taking a look at 3.6 builds and it looks like https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/107/testReport/junit/org.apache.kafka.clients/ClusterConnectionStatesTest/Build___JDK_8_and_Scala_2_12___testSingleIP__/ has been failing a lot more since this change.
https://ge.apache.org/scans/tests?search.names=git branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=America%2FLos_Angeles&search.values=3.6&tests.container=org.apache.kafka.clients.ClusterConnectionStatesTest
It has flaked on the build for 3.6 that it was merged, the next build, and my PR build targeted to 3.6. Can we make sure this change didn't cause issues?
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/107/
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/108/#showFailuresLink
https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14712/6/#showFailuresLink
@showuon @mumrah @cmccabe

I had a look, and think this flaky test should be the infra's issue, not our code. Like you said, in 3.6 branch, after I backported this PR, it started to fail (on Nov. 9). Now, checking trunk build, it also failed on Nov. 9 and 10.
https://ge.apache.org/scans/tests?search.names=git branch&search.rootProjectNames=kafka&search.startTimeMax=1699587001012&search.startTimeMin=1696089600000&search.timeZoneId=Asia%2FTaipei&search.values=trunk&tests.container=org.apache.kafka.clients.ClusterConnectionStatesTest#
Back to the test code, it failed at this line:
assertEquals(1, ClientUtils.resolve("localhost", singleIPHostResolver).size());
which basically our kafka logic won't impact this host resolver results.
So, I think we can keep monitoring it for now, and hope it can be self-healing. If not, we raise a issue to infra team, maybe?

@jolshan , I just created a PR to revert the change in this PR, and the test results also failed testSingleIP test. So, let's keep monitoring it, and maybe we need to change the test, because resolving localhost into multiple IPs should be fine.

Thanks @showuon for taking a look. Makes sense to me.
I also saw the conversation here. https://github.com/apache/kafka/pull/14741


keras optimizer apply_gradients arg grads_and_vars has wrong type in documentation

URL(s) with the issue:
https://github.com/tensorflow/tensorflow/blob/516c98da7b7d8526c153827c426c675a4ece9543/tensorflow/python/keras/optimizer_v2/optimizer_v2.py#L414
Description of issue (what needs changing):
grads_and_vars is documented as list but is actually passed as zip object.
Clear description
This can be problematic when writing custom optimizers that iterate over the grads_and_vars multiple times, as in that case a zip object will not give the intended behaviour.

@koenhelwegen, Since the associated PR has been merged, are you happy to close if no issue persists. Thanks!

Hi, thanks for taking this up. PR is in a separate repo where the change from list to zip object resulted in a bug in a custom optimizer. The TF documentation is still incorrect.

@koenhelwegen Are you willing to contribute through PR? Link this issue in the PR and it will automatically closes once the PR merges. Thanks!

At a closer look it seems the grads_and_vars zip is usually converted to a list before it's passed to apply_gradients (e.g. here, but not here), so I'm not sure if this is a documentation error or that this is a bug in training_eager.py.

@koenhelwegen,
Since the associated PR has been merged,please confirm if the issue can be closed. Thanks!