Add setup for Haskell server

Closes #38
This adds the necessary setup for the Haskell server we'll be relying on for the time being. Since we went with a monorepo approach, the repo structure has been rearranged and split into purescript and server subdirectories
The Haskell project has its own flake. We can add this to the main flake's inputs (i.e. to use in the devShell) later using a path URL

No idea what's happening with Hercules:
FatalError {fatalErrorMessage = "Worker failed with exit status: -6"}

No idea what's happening with Hercules:
FatalError {fatalErrorMessage = "Worker failed with exit status: -6"}

No idea what's happening with Hercules:
FatalError {fatalErrorMessage = "Worker failed with exit status: -6"}

Apparently Hercules has an issue with two flakes in the repo (there really should be only one in most circumstances, but I was hoping it would with two). Reverted to original combined flake now

Do we want to add /server/dist-newstyle and /server/dist-newstyle/ to .gitignore?

@vvtran There in the .gitignore on my estimate-fees branch, but I'll add it here too


getting error when trying to train tesseract on new font

getting error when trying to train tesseract on new font
=== Constructing LSTM training data ===
[Tue, Sep 10, 2019 1:42:29 PM] /c/Program Files (x86)/Tesseract-OCR/combine_lang_model --input_unicharset /tmp/eng-2019-09-10.lCl/eng.unicharset --script_dir langdata_lstm --words langdata_lstm/eng/eng.wordlist --numbers langdata_lstm/eng/eng.numbers --puncs langdata_lstm/eng/eng.punc --output_dir train --lang eng
Loaded unicharset of size 99 from file C:/Users/MADHUR~1.DAD/AppData/Local/Temp/eng-2019-09-10.lCl/eng.unicharset
Setting unichar properties
Setting script properties
Warning: properties incomplete for index 68 = ~
Config file is optional, continuing...
Failed to read data from: langdata_lstm/eng/eng.config
Null char=2
Invalid format in radical table at line 0: 19886 3 23 6 3
Creation of encoded unicharset failed!!
Error writing recoder!!
Reducing Trie to SquishedDawg
Error during conversion of wordlists to DAWGs!!
basename: extra operand ‘(x86)/Tesseract-OCR/combine_lang_model’
Try 'basename --help' for more information.
ERROR: Program failed. Abort.
Originally posted by @madhuri-dadhich in https://github.com/tesseract-ocr/tesseract/issues/1574#issuecomment-529825518

Invalid format in radical table at line 0: 19886 3 23 6 3

Did you try with latest version of file?

I am fetching the same problem though i am using https://github.com/tesseract-ocr/langdata_lstm/blob/master/radical-stroke.txt.
My tesseract  version is given below:
tesseract v4.0.0-beta.4.20180912
leptonica-1.76.0
libgif 5.1.4 : libjpeg 8d (libjpeg-turbo 1.5.3) : libpng 1.6.34 : libtiff 4.0.9 : zlib 1.2.11 : libwebp 0.6.1 : libopenjp2 2.2.0

tesseract v4.0.0-beta.4.20180912

Please use the latest version of tesseract with the latest version of radical-stroke.txt, so that both files are in sync.

can you give the new version link of tesseract.

can you give the latest version link of tesseract and radical-stroke.txt.

tesseract v4.0.0-beta.4.20180912

Please use the latest version of tesseract with the latest version of radical-stroke.txt, so that both files are in sync.

still getting same error. can you please explain how radical-stroke.txt works during the training time.


Consider various polling delay possibilities

It has been noticed that a 1 second poling may put a strain on the backend.
One alternative suggested is to set to a larger number - 2 would cut the API requests in half - but then for interactive use we may need an option to control this to keep it small
Another alternative is exponentially increasing delay.
Consider and implement something as the necessity of it arises.

We now do geometric polling.


Add LDAP->Django Group mapping

This PR addresses issue #11 by adding a GROUP_MAP setting which allows an LDAP group to be mapped to a list of Django groups.  The listed groups will be added or removed from a user's account at login time based on the users membership or non-membership in the LDAP group.

Oops, made a small error, will redo this request.


Error when using time.Sleep

Hello, how are you doing? I have been working on as wasm filter written in golang and I had  the following error when I used the time go package for a time.Sleep function.
{"level":"error","time":"2022-02-18T14:43:27.719183Z","scope":"envoy wasm","msg":"Failed to load Wasm module due to a missing import: wasi_snapshot_preview1.poll_oneoff"}
{"level":"error","time":"2022-02-18T14:43:27.719220Z","scope":"envoy wasm","msg":"Wasm VM failed Failed to initialize Wasm code"}
{"level":"debug","time":"2022-02-18T14:43:27.719225Z","scope":"envoy wasm","msg":"~Wasm 28 remaining active"}
{"level":"trace","time":"2022-02-18T14:43:27.720989Z","scope":"envoy wasm","msg":"Unable to create Wasm"}
{"level":"critical","time":"2022-02-18T14:43:27.721043Z","scope":"envoy wasm","msg":"Plugin configured to fail closed failed to load"}
{"level":"critical","time":"2022-02-18T14:43:27.721058Z","scope":"envoy wasm","msg":"Plugin configured to fail closed failed to load"}
{"level":"critical","time":"2022-02-18T14:43:27.721083Z","scope":"envoy wasm","msg":"Plugin configured to fail closed failed to load"}

Here is the line in goland that is giving the problem (I took it out of the code for simplicity, if I remove the delay from the code it works as is):
import (
	"time"
)
time.Sleep(300 * time.Millisecond)

I am actually translating a previous lua envoy filter that we had to a wasm filter, and I was able to implemente delays on the lua filter. We are using this for a retry strategy. Do any of you guys knows why the time package is failing when loaded into envoy (I did check that this package is supported by TinyGo)? If it can’t be loaded is there any other way of getting a small delay on the wasm code?

Sleep cannot be used on Envoy due to its async mechanism and threading model, and you will never be able to use sleep or concurrency stuff.
Instead, you could combine OnTick and StopIteration to achieve the delay: This is the example exactly doing the same thing you are trying to achieve: https://github.com/tetratelabs/proxy-wasm-go-sdk/tree/main/examples/postpone_requests

Thank you


Zulip 4.0.3 desktop is constantly reloading on MacOS

To reproduce:

Install Zulip 4.0.3 on a MacOS machine (MacBook pro, running High Sierra), after your current installation (4.0.0) was messaged as having alarming security vulnerabilities.
Attempt to use Zulip in the manner to which you have become accustomed.
3a) Expected result: Am able to use Zulip.
3b) Actual result: Zulip constantly reloads, making it impossible to write or send any messages, and can only read messages at sporadic intervals.

Zulip still works in the browser, desktop version is unusable.

Sorry about the duplicate, refound my original report.


Document automatic plugin and theme updates

Port this to the readme https://github.com/xwp/vip-go-site/wiki#plugin-and-theme-updates

Closing since we're not moving everything to the readme.


Investigate/fix test_cancel_running_job

This test has been failing for a while now - https://github.com/Qiskit/qiskit-ibm-runtime/actions/runs/5863436233/job/15896900505
It appears that when attempting to cancel a running job, calling job.cancel() can return a 204 and raise no errors while the job is actually completed.
This snippet shows that the status of job is CANCELLED but retrieving the job and status again shows DONE, which is the correct status.
I'm guessing this is some sort of race condition where the job runs too quickly and if this is the case, it should be handled appropriately on the client side.
def test_cancel_job_running(self, service):
        """Test canceling a running job."""
        job = self._run_program(service, iterations=5)
        if not cancel_job_safe(job, self.log):
            return
        time.sleep(10)  # Wait a bit for DB to update.
        rjob = service.job(job.job_id())
        print("job==rjob is " + str(job == rjob))
        print("job.job_id() = " + str(job.job_id()))
        print("rjob.job_id() = " + str(rjob.job_id()))
        print("job.status() = " + str(job.status()))
        print("rjob.status()) = " + str(rjob.status()))
        self.assertEqual(job.status(), JobStatus.CANCELLED)

For more context, here in the cancel method we are manually setting the job status to CANCELLED on line 252 even though in some cases the job is actually completed.
https://github.com/Qiskit/qiskit-ibm-runtime/blob/3d8b9c9f5a81b0c1850cbfe44625216908c41977/qiskit_ibm_runtime/runtime_job.py#L238-L252

@kt474 I believe there might be an issue both in the cancel() function race condition and test_cancel_job_running.
It appears that the job status is not updated correctly, which could be causing the problem. The Job is completing before the CANCELLATION requests goes through.
I think the following might resolve the problem:

Modify test function: Check the server-side status after using the cancel() method and wait for the state to be updated before proceeding with any assertions in test_cancel_job_running case.
Modify cancel() method:  make sure cancel() method  waits until the server-side status is updated before marking the job as "CANCELLED" on the client side.
Let me know what you think and perhaps I can try to implement this.

I think the test can be fixed quite simply by replacing the lines:
if not cancel_job_safe(job, self.log):
        return
time.sleep(10)  # Wait a bit for DB to update.
rjob = service.job(job.job_id())

by the lines:
rjob = service.job(job.job_id())
if not cancel_job_safe(rjob, self.log):
        return
time.sleep(10)

However, I think there is a deeper issue here, and that is why job and rjob are not identical. This does not have to do with the cancel itself.

Closing this for now, will reopen if it becomes an issue again


Crasch on Windows Embeded Standard 7

I'm Trying to run TcJoy .NET Part on Windows Embeded Standard 7 (Beckhoff CX2020). It's stop working after a second.

@Automatypro have you tried setting up a separate task for the controller so that the timing is maintained? You may have the ADS Rate set too low, and the system response is timing out (ADS Watchdog time exceeded). Maybe try setting up a separate task with a 10ms tick, reference that in the project, then assign it the program that is calling your FB_TcJoy instance.


fixes #12905

Q
A

Is bugfix?
yes

New feature?
no

Breaks BC?
no

Tests pass?
yes

Fixed issues
#12905

@samdark  one of the suggestions is to have a public cacheKeyPrefix that defaults to __CLASS__ and  no need for any getter as we append @app to it. This might not seem backward compatible but it's because it was private/protected.
What do you think?

Not being able to remove @app from it sounds bad to me.

is there a way to distinct different routes in different apps other than @app.
with current Yii code it's the key is a protected property that you can't change. In the new approach I allow you to change the prefix.
Another approach is to append $hash to the prefix, the downside when one changes the route, a redundant stall key will remain.
so $hash and @app I can't think of any other thing.
Btw I don't like having a setter because the cache is get/set at the initializing of yii app (loading routes) setting it later have no meaningful purpose.
If someone want to change it, he should extend the class.

Yes. A manually set prefix, for example. It's a common case when in backend you're using another instance of URLManager with the frontend rules to link to frontend.

here is the use case that you are talking about,
https://github.com/yiisoft/yii2-app-advanced/blob/master/docs/guide/topic-link-backend-frontend.md
I need to know the @app (or whatever that identifies it) of the app for which the url rules belong to.
If we are able to do this, then providing custom prefix is meaning less.
Let me read more to see how to do that.

        'urlManager' => [
            'enablePrettyUrl' => true,
            'showScriptName' => false,
            'rules' => require 'urls.php',
        ],
        'urlManagerApp2' => [
            'enablePrettyUrl' => true,
            'showScriptName' => false,
            'rules' => require 'urls2.php',
            'cachePrefix'=>'app2-urls'
        ],

I'm trying to eliminate the need of passing cachePrefix (principle of least astonishment)
I think there is there a way to get the name of the component (urlManagerApp2)
So we are left with stressing that cachePrefix should be customized.

Name of the component will be different for frontend app and backend app in this case. Rules will be the same.

in that case I prefer using the hash, but we should have ttl (public configrable ttl) to avoid stall  cache of unused hashes

How about https://github.com/yiisoft/yii2/issues/12905#issuecomment-258259468?

I've replaced this with the suggested approach,

make cacheKey public
make it expand using getAlias

https://github.com/yiisoft/yii2/pull/12926

This approach isn't good in its inflexibility. Since there's better alternative in for of https://github.com/yiisoft/yii2/pull/12926, I'm closing this one.

Agreed. Didn't see #12926 when commented this PR


If LSP doesn't return tags attempt to get tags from the tag file

At times I'm working on C projects where I don't have a compile_commands.json, and thus the LSP doesn't fully work on the file. But I do have tags file present, and I want Ctrl-] to work with them too.
So I've added logic to use the tags file if LSP didn't return any symbol.

Hey. I'm very sorry for the incredible long delay.
First of all, I would like to ask you to just return nil for this case and do not work on the tags yourself. It is just not necessary as NeoVim will do it itself when nil gets returned. Atm it doesn't work as you expect because it returns an empty list.
Second I'm thinking if this can be tweaked/configured. I'm not 100% sure if this is always the intended behavior. Especially because it will always trigger an actual tag search when the language server(s) provide no location (which can be correct). So I wonder if it would be more precise to check if the necessary location providers are available. If it does and the response is empty, accept that. If not, go ahead and use tags. Do you think this would help here? In worst case you could explicitly setup your language client so it disabled certain location provider you know of that they don't work properly.
If that should not be possible, I would at least add a configuration variable to enable the fallback to tags. So the current behavior remains as it for all users and everyone like you can configure it the way you need it.
Looking forward to your feedback.

Fixed the return value, it indeed works as expected that way.
I'm not sure what you mean by "check the location provider" - can you elaborate on that please?
I will look into adding an option for opting into this behaviour.

Added a global option for my change. Did some further testing and I noticed that when I don't have compile_commands.json clangd returns incorrect locations for tags instead of an empty list, so my change doesn't really help here. But when I turn off the LSP client (:LspStop) then my change kicks in and works perfectly.
Though maybe in such cases we need to check if there are any LSP Clients attached to the buffer, and if not return nil? Is that what you meant by "check the location provider"?

Hey,
sorry that I haven't responded. 🙈
I like the current state. Do you think we can merge it right now or would you like to add more due to your recent problem?

I'm fine with merging now. If I'll have further fixes, I'll raise another PR.


Upgrade config server together with config server host

This is a partial revert of #9176.
Details included in the corresponding vespa/hosted#7753 PR.

FYI: @mpolden


Does rails_semantic_logger support Rails 6?

Environment
Provide at least:

Ruby Version: 2.6.5
Rails Version: <IP_ADDRESS>
Semantic Logger Version: 4.6.1
Rails Semantic Logger Version: 4.4.3

On reading the documentation from rails_semantic_logger README, I see that the supported Rails version is:
Supports
Rails 3.2, 4, 5

Meanwhile from semantic_logger documentation:
http://rocketjob.github.io/semantic_logger/rails.html
Rails Support
Supports Rails 4.2, 5.0 and 5.1.

Not really match together.
However, my question is does rails_semantic_logger support Rails 6? I saw test on Rails 6 environment passed although it's 'Allowed Failures' so I'm doubting :thinking:
https://travis-ci.org/github/rocketjob/rails_semantic_logger

What problems does rails_semantic_logger might have on Rails 6? Thanks!

I need to remove the supports list from the readme and the docs. Then refer everyone to the Travis file.
The community added Rails 6 to the supported list. We only have JRuby on the allowed to fail list, even though it is successful nowadays.
We are in the process of upgrading our Rails app to rails 6 so will definitively be using Rails Semantic Logger with Rails 6.

Fixed documentation in https://github.com/rocketjob/semantic_logger/commit/e82ef88c1955b007c51e25da5e4935ff49688360

Thanks for confirming!
@reidmorrison also please update in here as well https://github.com/rocketjob/rails_semantic_logger#supports

Support
Rails 3.2, 4, 5

@reidmorrison just open a quick PR #100 to fix the above documentation


Add some Python 3.x packages

Description of changes

Add some Python 3.x packages:

cfg_diag 0.4.0 (https://github.com/storpool/python-cfg_diag) - easily build Python config dataclasses with a .diag() method that only outputs messages when requested (when the verbose flag is set)
trivver 2.2.2 (https://gitlab.com/ppentchev/python-trivver) - compare package version strings
typedload 2.19 (https://ltworf.github.io/typedload/) - load and dump data enforcing a scheme, e.g. load a dataclasses hierarchy from a JSON or TOML file
feature-check 2.0.0 (https://devel.ringlet.net/misc/feature-check/) - query a program for supported features
utf8-locale 1.0.0 (https://devel.ringlet.net/devel/utf8-locale/) - detect a UTF-8-capable locale to run child processes in
tox-delay 0.1.3 (https://devel.ringlet.net/devel/tox-delay/) - run some Tox tests after others have completed

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

All packages could do with having pythonImportsCheck.

Some packages are now stale and needs updating, plus it would be better to squash your commits and rebase to latest.


Have a way to 'balance' workloads across the cluster when new nodes added

Anything that's 'static' needs to be propagated from a good node:

Ingresses
Services
Secrets
Daemonsets
Could be best to do this at node creation. Tricky bit is knowing what to create. Safest would be a merge of what it found on the other nodes.

Then also a redeploy of deployments.
Not sure if that should be in the same action or a separate action for deployments.

fixed for service ingress in b237c60acada578c77e71eff43c5361b25683463

Next is to reschedule existing workloads.

Now cordon and uncordon is implemented, I'm leaning towards having a cluster-wide reschedule via skate cluster reconcile or skate cluster reshedule. Then probably some kind of subset of that, or even just an alias skate drain.

Added in v0.1.33


Creating container with multiple port forward only creates one

I've a bunch of containers that have multiple port forwards:
{'enp3s0f0:9900': 9900, 'zt*:9900': 9900}
{'enp3s0f0:9901': 9900, 'zt*:9901': 9900}
{'enp3s0f0:9902': 9900, 'zt*:9902': 9900}
{'enp3s0f0:9903': 9900, 'zt*:9903': 9900}
{'enp3s0f0:9904': 9900, 'zt*:9904': 9900}
{'enp3s0f0:9905': 9900, 'zt*:9905': 9900}

But when I look at the nft config generated for this:
type nat hook prerouting priority 0; policy accept;
iif "core0" mark set 0x00000001
iif "kvm0" mark set 0x00000001
iifname "zt*" tcp dport 6600 dnat to <IP_ADDRESS>:6600
iifname "enp3s0f0" tcp dport 9900 dnat to <IP_ADDRESS>:9900
iifname "zt*" tcp dport 9901 dnat to <IP_ADDRESS>:9900
iifname "enp3s0f0" tcp dport 9902 dnat to <IP_ADDRESS>:9900
iifname "zt*" tcp dport 9903 dnat to <IP_ADDRESS>:9900
iifname "zt*" tcp dport 9904 dnat to <IP_ADDRESS>:9900
iifname "zt*" tcp dport 9905 dnat to <IP_ADDRESS>:9900

so it seems that only one of the two port forward have been created each time

Seems the problems comes from https://github.com/threefoldtech/0-core/blob/4504a0260cc4fd5f7946de31814fb37b52d1a89e/apps/core0/helper/socat/socat.go#L96-L100
Since we keep a map in memory to know if a port is already used, but we only use the port number in the map not the interface:port, so we can always only have one port forward per source port.

There is no clean way to implement this without getting conflicting rules, and it was safer (and still is) is to hold the port exclusively for a port forward rule.
But we changed this so the container creation fails if u provided a port that is already in use.

But we changed this so the container creation fails if u provided a port that is already in use.

Should be good enough for now


[Event Request] Table 288 - "Vendor Bank Account" - OnBeforeGetBankAccountNo

Please add the event OnBeforeGetBankAccountNo in Table 288:
  procedure GetBankAccountNo(): Text
  var
+     IsHandled: Boolean;
+     ReturnValue: Text;
  begin
+     OnBeforeGetBankAccountNo(Rec, ReturnValue, IsHandled);
+     if IsHandled then
+        exit(ReturnValue);

      if IBAN <> '' then
          exit(DelChr(IBAN, '=<>'));

      if "Bank Account No." <> '' then
          exit("Bank Account No.");
  end;

New integration event:
[IntegrationEvent(false, false)]
local procedure OnBeforeGetBankAccountNo(VendorBankAccount: Record "Vendor Bank Account"; var ReturnValue: Text; var IsHandled: Boolean)
begin
end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Don't add julia bin dir to path

Would it make more sense to create a symlink as recommended in the julia docs? That generally seems less intrusive, and generally there seems to be a fair bit of stuff in the bin folder that doesn't need to be on the PATH.

Happy to follow the lead of someone who has experience with Julia and knows what makes things feel "Julia-thonic" (What is Pytohnic for Julia?). Something to look out for is "does it break things that currently work, even if they are 'against the proper way of doing things'". I think we should be conservative in terms of breaking things that people might be relying on.

I'm pretty positive that this wouldn't break anything. Plus, we can only do this for the Project.toml case, not the REQUIRE case, where we don't have any backwards compat issues because no one is using this right now :)

Yeah I think this sounds good. That said, I also think that there was never anything in the bin/ directory except the julia binary and in v0.6- the julia-debug binary. So i don't think this is much of a concern one way or the other. (And, to be fair, the docs you pointed to list both options as valid):

To run Julia, you can do any of the following:

Create a symbolic link to julia inside a folder which is on your system PATH
Add Julia's bin folder to your system PATH environment variable
Invoke the julia executable by using its full path, as in /bin/julia

But i don't see any harm if you think the symlinks are better for some reason?

I actually tried this, and it didn't work. For some reason it then couldn't find the libjulia anymore. I'm pretty sure one could get it to work somehow, but I think I won't try longer :)
So I'd say this is clearly very, very low priority, and we could almost certainly also just leave things the way they are.


wrap transaction with transaction.atomic instead of using the deprecated commit_unless_managed

Relates to #212

Duplicate of #197


Support for ImageDataGenerators

Hi,
I am currently putting together a talk on Image Adversaries 101 that would include the following:

Adversaries as an optimization problem
Creating perturbations that do not hurt the eyes
Create end-to-end image adversaries
Neural Structured Learning for creating robust models

I have put together a basic Colab Notebook that shows how to create basic adversaries by taking inspiration from here: https://adversarial-ml-tutorial.org/introduction. I am now preparing a notebook that would show the other half - using NSL for training adversarially robust models. The notebook is available here. For the first part, it trains a basic image classification model to distinguish between different flower species (with the Flowers-17 dataset) using the ImageDataGenerator class. So, to extend that example, I am wondering if NSL would support ImageDataGenerator class or I would need to create NumPy arrays and then convert them to tf.data.Dataset.from_tensor_slices.
I am looking forward to your views and any suggestions. All of the pointers gathered from here will be open-sourced and will be made available for the world to see :)
Edit:
I was able to turn the dataset into tf.data.Dataset objects as you'd see in the Colab Notebook. I just wanted to check in and see if you would have any suggestions for the talk proposal and its structure. I would be really happy to consider it.

Hi Sayak,
Thanks for your interest in NSL! The structure looks good in general.
Regarding ImageDataGenerator, NSL does support generator-style input, but requires each generated element to be a dict containing both input and label features. To make it work with ImageDataGenerator (which generates (image, label) tuples), you may add a converter for tuple-to-dict mapping:
def convert_to_dict_generator(image_data_gen):
  for image, label in image_data_gen:
    yield {'image': image, 'label': label}

The converter can be applied to the generators produced by ImageDataGenerator. The outcome is still a generator reading the files on the fly.
train_image_gen = ImageDataGenerator(...).flow_from_directory(...)
adv_train_image_gen = convert_to_dict_generator(train_image_gen)

And nsl.keras.AdversarialRegularization models can be trained with the converted generator:
adv_model.fit(adv_train_image_gen, ...)

Thanks a ton for this solution. I was able to train the adversarial model, though as you'd see in the Colab notebook I mentioned. If you have any recommendations that'd be great.
It was amazingly easy to plug NSL in and use it off-the-shelf :)

Thanks, Sayak.
We generate adversarial examples from the base model because it is the model in the beginning that we would like to identify vulnerability on and improve upon. Besides this kind of "white-box" attack, we also have vision to explore other adversarial methods including gray-box (transfer) attacks and black-box attacks.

Wow! Thanks for providing more explanations.
Do you mind giving a quick look at the deck I would prepare for the talk I mentioned above? That'd be really helpful. If not I can definitely respect your schedule :)


update .travis.yml for docker image build

/lgtm
Thanks for fixing the build error in PR!

Travis build error will be fixed after PR #82 is merged.

All check passed.


Arch Linux installation guild

Related #629

The arch install instructions should be under a separate header, like the ones for ubuntu.

Updated, separated the content as Arch Linux section


NVDA not supported

I've faced an issue with tests launching.
npx @guidepup/setup doesn't work for me

It looks like something wrong with my local environment, but I can't find the reason.

Hi @DeanNeal is this issue still happening for you? Going by https://github.com/guidepup/guidepup-playwright/issues/8 it sounds like it might be working locally for you, just not in a docker container?

Hi @cmorten This issue is no longer reproducible. I have only issue with docker you mentioned https://github.com/guidepup/guidepup-playwright/issues/8


Dialogue engine

Player should be able to talk to other characters, displaying text on the screen with a character portrait.
Players should be able to choose responses when prompted, affecting the course of the dialogue.
Characters should have different dialogue with the player depending on the scene_id.

After #6 and #7


Placeholder text?

Hi,
How do you add placeholder text to empty editors?
This Stackoverflow answer here shows one way to do it
Will it be a good idea to incorporate this into the library? What are your thoughts?
Thanks

+1

+1

+1

+1

Any ETA?  Thanks.

+1

+1

+1


Improve load and store interfaces with new traits from cairo 2.7.0

Which components does the task require to be changed? (think hard pls)
snforge
Description
It should be easier to construct something like those helpers, when Store trait is available for usage:
fn load<T, +Store<T>>(target: ContractAddress, base_address: felt252) -> T {
    Store::<T>::read(0, base_address).unwrap()
}

fn load_path<T>(base_address: felt252) -> StoragePath<T> {
    StoragePath::<T> { pedersen::HashState {base_address} }
}

Then the usage of the cheats would be more intuitive since we would not have to specify the sizes:
// Given storage:
struct Storage {
   a: Map<u8,u8>,
   b: MyStruct
}

// Construct load/store paths with addresses and bind to types:
let path_a = load_path::<Map<u8,u8>>(selector!("a"));
let path_b = load_path::<MyStruct>(selector!("b"));

// Retrieve values:
path_a.entry(5).read();
path_b.read();

Blocked by #2347


Tried both SendGrid & MailGun API keys with no luck..

Hi, i've recently been testing this project but i'm having issues with the email alerts as i've tried both Sendgrid and mailgun so ive been wondering where i am going wrong?
For sendgrid do you use the smtp relay or the web api?
i have tried both in the switchboard.env with no luck
Is there any guides to help?
Thanks

Hi @a6du114h,
For sendgrid, you can check how we use it here; it looks to be the API through their python client.
For mailgun, please ensure that you supply both your API key and the correct mailgun domain; if you are sure you have, you can check the logs by running docker exec -ti switchboard /bin/bash and then tail -f switchboard.log. Once you trigger the token, you should see a log line about alerting (via email).
Please let me know how it goes (also there is more documentation over here).

@jayjb @a6du114h
I have the same issue. I use SendGrid, I have supplied the API key in switchboard.env (with and without quotes) and there is no entry in switchboard.log about an email notification when the token is triggered. I see this error might be related to that: [_GenericHTTPChannelProtocol,4,IP] 'Exception occurred in switchboard dispatch: '

@ch40s that is a useful find. Ill see if i can get my hands on a Sendgrid api key and track down this issue.

Hi @ch40s, @a6du114h
Currently the Sendgrid API is working for me. I've removed all MAILGUN switchboard.env keys and left only the CANARY_SENDGRID_API_KEY=SG.xxxxxxxx-xxxxxxxxxxxxxxxxxxx. Please can you give me some more information about the Sendgrid API setup you used


it not work

I want add a icon to nav leftButton, but I found that it not work.
var app = React.createClass({
getInitialState() {
return {userIcon: {}}
},
render: function() {
Icon.getImageSource('user', 20, 'red').then((source) => {
alert(source)
this.setState({ userIcon: source })
});
return (
<NavigatorIOS
style={{flex:1}}
initialRoute={{
component: Page,
leftButtonIcon: this.state.userIcon,
title: 'home',
}}
/>
);
}
});

You cannot use async functions in render, see the TabBar example for how to do it.

thanks! it worked!
but NavigatorIOS leftButton can't reload , should be the problem of the react itself

Yes, that's a bug with the NavigatorIOS I'm afraid. You can try react-native-navbar instead!

Yeah, I just use it to replace..
------------------ 原始邮件 ------------------
发件人: "Joel<EMAIL_ADDRESS>发送时间: 2015年7月2日(星期四) 晚上8:45
收件人<EMAIL_ADDRESS>抄送: "外星人<EMAIL_ADDRESS>主题: Re: [react-native-vector-icons] it not work (#5)
Yes, that's a bug with the NavigatorIOS I'm afraid. You can try react-native-navbar instead!
—
Reply to this email directly or view it on GitHub.


finalhandler cannot 404 after headers sent +0ms

After sending response to browser it is throwing following error.
finalhandler cannot 404 after headers sent +0ms
why it is sending 404 ?
can you tell me why it is throwing error ?

provide a code snippet of what you are doing

Please find the following git repo with reproduceble code.
https://github.com/MContagious/routing-controller-sample

what will happen if you remove @EmptyResultCode(404) ?

Initially I dont have that annotation. Later I included to resolve that error but it is still throwing 404 error. But, The browser is getting the json response. Error is written in the server log.

use new @OnUndefined / @OnNull decorators to resolve this issue.

This happening to me as well. I see this in the console:
  body-parser:json content-type "multipart/form-data; boundary=--------------------------042144984244033328614529" +977ms
  body-parser:json skip parsing +0ms
  morgan log request +977ms
POST /upload/photos 200 - ms - 14
  finalhandler cannot 404 after headers sent +977ms

And it also happens when I implicitly return a value:
@Post('/photos')
  async uploadPhotos() {
    return { wee: 'wooo' };
  }

In this case the console output is:
  express:router dispatching POST /upload/photos +34s
  express:router query  : /upload/photos +0ms
  express:router expressInit  : /upload/photos +0ms
  body-parser:json content-type "application/json" +34s
  body-parser:json content-encoding "identity" +0ms
  body-parser:json read body +0ms
  body-parser:json parse body +1ms
  body-parser:json parse json +0ms
  express:router logger  : /upload/photos +1ms
  express:router jsonParser  : /upload/photos +1ms
  body-parser:json body already parsed +1ms
  morgan log request +34s
POST /upload/photos 200 - ms - 34
  finalhandler cannot 404 after headers sent +34s

Probably next() gets called even after a route match?

yes, it is

So is it an easy fix? I think it happens somewhere in the routing-controllers package. Or does it intentional?

Or is it intentional?

yes next() method call is intentional, because otherwise middleware that execute after your controllers wont work.

Yes, but if I see correctly all we have to do is to register a special last middleware after registering the normal ones not?
I see you register them in index.ts at here , isn't it enough to change this to something like
new RoutingControllers(driver)
        .initialize()
        .registerMiddlewares("before")
        .registerControllers(controllerClasses)
        .registerMiddlewares("after", middlewareClasses);
        .registerFinalMiddleware(FinalMiddleware);

where final middleware looks something like:
import {ExpressMiddlewareInterface} from "../../src/driver/express/ExpressMiddlewareInterface";
import {Middleware} from "../../src/decorator/Middleware";

@Middleware({ type: "after" })
export class FinalMiddleware implements ExpressMiddlewareInterface {

    use(request: any, response: any, next?: Function): any {
     // next() not called here
   }

}

and the registerFinalMiddleware function looks something like:
registerFinalMiddleware(class: Function): this {
        this.metadataBuilder
            .buildMiddlewareMetadata([class])
            .forEach(middleware => this.driver.registerMiddleware(middleware));

        return this;
    }

Or we can just do
new RoutingControllers(driver)
        .initialize()
        .registerMiddlewares("before")
        .registerControllers(controllerClasses)
        .registerMiddlewares("after", [...middlewareClasses, FinalMiddleware]);

but then we still have to ensure metadata is already built for this class.

Btw shouldn't we reopen this? Having "Cannot set headers after response is sent" errors for every request is not okay, and I believe using @OnUndefined / @OnNull doesn't solve the problem, but my proposal does.

Having "Cannot set headers after response is sent" errors for every request is not okay

right, its not okay. And I don't have such errors after every request. Can you show what are you doing that you have such error?

As I understood you have such error because of finalhandler you are using, right? If yes, then why are you using it?

finalhandler  is a dependency of the express framework itself. It is called as a last middleware to set the status code to500 if you call next() too many times.
On the user side it can be fixed with using a custom middleware:
import { Middleware, ExpressMiddlewareInterface } from "routing-controllers";
import { NextFunction, Request, Response } from 'express';

const debug = require('debug')('middleware:FinalMiddleware');

@Middleware({ type: "after" })
export class FinalMiddleware implements ExpressMiddlewareInterface {

  use(req: Request, res: Response, next?: NextFunction): any {
    debug('Final middleware reached!')
  }

}

and loading it as the last one:
this.express = createExpressServer({
      routePrefix: '',
      middlewares: [CorsMiddleware, FinalMiddleware],
    });

This code should not give you such error.

The only cause I can think of is that I am loading some vanilla express middleware this way:
this.express = createExpressServer({
      routePrefix: '',
      middlewares: [CorsMiddleware, FinalMiddleware],
    });
this.express.use(logger('dev'));
this.express.use(bodyParser.json());

But it should still cause the error with the above solution if this would be the root of the error. So I don't have any idea how finalHandler get reached. I dont call next() anywhere in my code except in CorsMiddleware which has the type before

There is a next() after each route execution, can your issue be because of #114 ?

I don't think so because I have no specific route handlers for OPTION requests, I just early return with a 200 code and an empty response body in the CORS middleware and. I don't call next at all in that case so my routes are not reached.


sudo must be owned by uid 0 and have the setuid bit set

The drupal4docker stack was working and I build a site on it.  After a machine reboot, the mariadb, nginx and php containers failed to start, with the "/usr/bin/sudo must be owned by uid 0 and have the setuid bit set" error.
I had changed the ownership of the /var/www/html volume on the host before the problem, so set it back to root.  The containers came up, but more fiddling with the volume caused them to fail at the next restart.  I cannot see how ownership of that volume could affect those containers.
Now I cannot get the full stack to start again.
docker info
Containers: 15
 Running: 4
 Paused: 0
Images: 16
Server Version: 18.05.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.13.0-43-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.696GiB
Name: SugarBird
ID: X7IA:V6WR:XZTW:UU4Z:X6EH:3FTD:SPPP:ZINW:YYBG:C6E4:C2GJ:FJ4E
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 <IP_ADDRESS>/8
Live Restore Enabled: false

Contents of your docker-compose.yml
version: "2"

services:
  mariadb:
    image: wodby/mariadb:$MARIADB_TAG
    container_name: "${PROJECT_NAME}_mariadb"
#    stop_grace_period: 30s
    environment:
      MYSQL_ROOT_PASSWORD: $DB_ROOT_PASSWORD
      MYSQL_DATABASE: $DB_NAME
      MYSQL_USER: $DB_USER
      MYSQL_PASSWORD: $DB_PASSWORD
    volumes:
      - db_data:/var/lib/mysql # I want to manage volumes manually.
#      - ./mariadb-init:/docker-entrypoint-initdb.d # Place init .sql file(s) here.

  php:
    image: wodby/drupal-php:$PHP_TAG
    container_name: "${PROJECT_NAME}_php"
    environment:
      PHP_SENDMAIL_PATH: /usr/sbin/sendmail -t -i -S mailhog:1025
      DB_HOST: $DB_HOST
      DB_USER: $DB_USER
      DB_PASSWORD: $DB_PASSWORD
      DB_NAME: $DB_NAME
      DB_DRIVER: $DB_DRIVER
    volumes:
      - codebase:/var/www/html

  nginx:
    image: wodby/drupal-nginx:$NGINX_TAG
    container_name: "${PROJECT_NAME}_nginx"
    depends_on:
      - php
    environment:
      NGINX_STATIC_CONTENT_OPEN_FILE_CACHE: "off"
      NGINX_ERROR_LOG_LEVEL: debug
      NGINX_BACKEND_HOST: php
      NGINX_SERVER_ROOT: /var/www/html/web
    volumes:
      - codebase:/var/www/html
    labels:
      - 'traefik.backend=nginx'
      - 'traefik.port=80'
      - 'traefik.frontend.rule=Host:${PROJECT_BASE_URL}'

  mailhog:
    image: mailhog/mailhog
    container_name: "${PROJECT_NAME}_mailhog"
    labels:
      - 'traefik.backend=mailhog'
      - 'traefik.port=8025'
      - 'traefik.frontend.rule=Host:mailhog.${PROJECT_BASE_URL}'

  redis:
    container_name: "${PROJECT_NAME}_redis"
    image: wodby/redis:$REDIS_TAG

  portainer:
    image: portainer/portainer
    container_name: "${PROJECT_NAME}_portainer"
    command: --no-auth -H unix:///var/run/docker.sock
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      - 'traefik.backend=portainer'
      - 'traefik.port=9000'
      - 'traefik.frontend.rule=Host:portainer.${PROJECT_BASE_URL}'

  traefik:
    image: traefik
    container_name: "${PROJECT_NAME}_traefik"
    command: -c /dev/null --web --docker --logLevel=INFO
    ports:
      - '8000:80'
#      - '8080:8080' # Dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  db_data:
  codebase:

Contents of your .env
### PROJECT SETTINGS

PROJECT_NAME=thecured
PROJECT_BASE_URL=thecured.co.za

DB_NAME=xxxxxxxxxxx
DB_USER=xxxxxxxxxx
DB_PASSWORD=xxxxxxxx
DB_ROOT_PASSWORD=xxxxxxxxxxxxxxxx
DB_HOST=mariadb
DB_DRIVER=mysql

### --- MARIADB ----

MARIADB_TAG=10.2-3.1.2
#MARIADB_TAG=10.1-3.1.3

### --- VANILLA DRUPAL ----
### [DRUPAL_VERSION]-[PHP_VERSION]-[STABILITY_TAG]

DRUPAL_TAG=8-7.1-4.2.6
#DRUPAL_TAG=8-7.0-4.2.6

### --- PHP ----

#PHP_TAG=7.1-dev-4.2.5
PHP_TAG=7.0-dev-4.2.5

### --- NGINX ----
### [DRUPAL_VERSION]-[NGINX_VERSION]-[STABILITY_TAG]

NGINX_TAG=8-1.14
#NGINX_TAG=7-1.13-4.0.3

### --- SOLR ---
### [DRUPAL_VERSION]-[SOLR_VERSION]-[STABILITY_TAG]

SOLR_TAG=8-6.6-2.4.0
#SOLR_TAG=8-6.5-2.4.0

### --- REDIS ---

REDIS_TAG=4.0-2.1.5
#REDIS_TAG=3.2-2.1.5

### --- POSTGRESQL ----

POSTGRES_TAG=10.1-1.3.0
#POSTGRES_TAG=9.6-1.3.0

### OTHERS

ADMINER_TAG=4.3-1.1.0
APACHE_TAG=2.4-3.0.5
ATHENAPDF_TAG=2.10.0
DRUPAL_NODE_TAG=1.0-1.0.0
MEMCACHED_TAG=1.4-2.0.0
RSYSLOG_TAG=latest
VARNISH_TAG=4.1-2.3.0
WEBGRIND_TAG=1.5-1.3.0

Logs output docker-compose logs
Attaching to thecured_nginx, thecured_php, thecured_portainer, thecured_redis, thecured_traefik, thecured_mailhog, thecured_mariadb
ESC[35mthecured_redis |ESC[0m 1:C 11 Jun 07:28:47.550 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
ESC[35mthecured_redis |ESC[0m 1:C 11 Jun 07:28:47.550 # Redis version=4.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
ESC[35mthecured_redis |ESC[0m 1:C 11 Jun 07:28:47.551 # Configuration loaded
ESC[35mthecured_redis |ESC[0m 1:M 11 Jun 07:28:47.555 * Running mode=standalone, port=6379.
ESC[35mthecured_redis |ESC[0m 1:M 11 Jun 07:28:47.555 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/so
maxconn is set to the lower value of 128.
ESC[35mthecured_redis |ESC[0m 1:M 11 Jun 07:28:47.555 # Server initialized
ESC[35mthecured_redis |ESC[0m 1:M 11 Jun 07:28:47.555 # WARNING overcommit_memory is set to 0! Background save may fail under low memory conditi
on. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for
this to take effect.
ESC[35mthecured_redis |ESC[0m 1:M 11 Jun 07:28:47.555 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will
create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
ESC[35mthecured_redis |ESC[0m 1:M 11 Jun 07:28:47.555 * Ready to accept connections
ESC[36mthecured_nginx |ESC[0m sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set
ESC[33mthecured_php |ESC[0m sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set
ESC[32mthecured_portainer |ESC[0m 2018/06/11 07:28:48 Starting Portainer 1.16.5 on :9000
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=info msg="Using TOML configuration file /dev/null"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=warning msg="web provider configuration is deprecated, you should use these op
tions : api, rest provider, ping and metrics"
ESC[34mthecured_mailhog |ESC[0m 2018/06/11 07:28:46 Using in-memory storage
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=info msg="Traefik version v1.5.4 built on 2018-03-15_01:35:21PM"
ESC[34mthecured_mailhog |ESC[0m 2018/06/11 07:28:46 [SMTP] Binding to address: <IP_ADDRESS>:1025
ESC[36;1mthecured_mariadb |ESC[0m sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=info msg="
ESC[34mthecured_mailhog |ESC[0m [HTTP] Binding to address: <IP_ADDRESS>:8025
ESC[31mthecured_traefik |ESC[0m Stats collection is disabled.
ESC[34mthecured_mailhog |ESC[0m 2018/06/11 07:28:46 Serving under http://<IP_ADDRESS>:8025/
ESC[31mthecured_traefik |ESC[0m Help us improve Traefik by turning this feature on :)
ESC[34mthecured_mailhog |ESC[0m Creating API v1 with WebPath:
ESC[31mthecured_traefik |ESC[0m More details on: https://docs.traefik.io/basics/#collected-data
ESC[31mthecured_traefik |ESC[0m "
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc4206dff80} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=info msg="Preparing server traefik &{Network: Address::8080 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc4206dffa0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=info msg="Starting server on :80"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=info msg="Starting provider *docker.Provider {"Watch":true,"Filename":"","Constraints":null,"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"unix:///var/run/docker.sock","Domain":"","TLS":null,"ExposedByDefault"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=info msg="Starting provider *docker.Provider {"Watch":true,"Filename":"","Constraints":null,"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"unix:///var/run/docker.sock","Domain":"","TLS":null,"ExposedByDefault":true,"UseBindPortIP":false,"SwarmMode":false}"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:47Z" level=info msg="Starting server on :8080"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:48Z" level=info msg="Server configuration reloaded on :8080"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:48Z" level=info msg="Server configuration reloaded on :80"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:50Z" level=info msg="Server configuration reloaded on :80"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:50Z" level=info msg="Server configuration reloaded on :8080"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:51Z" level=info msg="Skipping same configuration for provider docker"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:28:51Z" level=info msg="Skipping same configuration for provider docker"
ESC[31mthecured_traefik |ESC[0m time="2018-06-11T07:38:52Z" level=warning msg="A new release has been found: 1.6.3. Please consider updating."
ESC[34mthecured_mailhog |ESC[0m Creating API v2 with WebPath:
ESC[34mthecured_mailhog |ESC[0m [APIv1] KEEPALIVE /api/v1/events

What are permissions of /usr/bin/sudo in the container? Sounds like you've changed permissions in containers. If you don't want to figure it out, you can just kill all containers and start them again.

Thanks for this - killing the containers didn't help.  But what did work was to purge all the images and rebuild everything.
MariaDB wouldn't start due to file ownership issues in the volume, but I noticed the ownership of the log file that was being written in the volume was "systemd-timesync:systemd-journal", so set the ownership of all the files and directories in that volume to it and now it works.
Yes, I am worried about the volume having some weird file ownership... but that is a problem for another day.

I have the same problem:

sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set

Only for nginx, maria db and php containers. Traefik and mailhog are implemented smoothly.


Make etccalico mount path configurable

Current Behavior
Currently the path is hardcoded to /etc/calico here: https://github.com/tigera/operator/blob/673ecc4c4d9774dcab03942ea93fb212a26e5cb4/pkg/render/csi.go#L140
This is a problem for Talos OS that allows for writes only under /var.
Possible Solution
make path configurable, e.g. so we can set it to /var/calico
Context
I would like to deploy a talos cluster with calico
Your Environment
Talos 1.3

Hi,
I'd like to see the same! I am also on Talos.
I don't really need the mount path to be configurable, I need the hostPath to be configurable.
I think it's these lines right here (currently starting on line 216):
	{
		Name: "etccalico",
		VolumeSource: corev1.VolumeSource{
			HostPath: &corev1.HostPathVolumeSource{
				Path: filepath.Clean("/etc/calico"),
			},
		},

My current test/workaround has been this:
Edit csi-node-driver DaemonSet and set this annotation:
unsupported.operator.tigera.io/ignore: "true"
...to make the operator leave the DS alone and not reset our next change, and then to change hostPath /etc/calico in csi-node-driver DS to something else:
  - hostPath:
      path: /var/lib/calico/etc
      type: ""
    name: etccalico

This works, but I have to leave the DS 'tainted' to keep it running.

@caseydavenport Apologies for the ping, but I believe this issue blocks anyone using an OS which does not persist /etc/ from using Tigera. Unfortunately I will have to use Cilium in the meantime.

@laibe do you think you need the MountPath configurable, I think it seems more reasonable that you would need the hostPath configurable like @epleterte suggested?
@epleterte using the unsupported annotation will be an issue when you upgrade as that component won't be upgraded with the annotation.
@uhthomas are you suggesting there are OS's that will discard /etc while they are running? I'm asking because I would expect a node that was rebooted to have the CSI driver pod redeployed so no need for /etc to be persisted across reboots. (When I hear about something not persisting it is usually in reference to across reboots but maybe you have a unique case.) Also incase you are unaware, the CSI driver I believe is only needed when using the Istio integration, so if you are not using that then you can disable it by setting the KubeletVolumePluginPath to None.
@Josh-Tigera it looks like you originally added CSI to the operator, so I assume you know a decent amount about the CSI driver and its installation, would there be any concern about exposing a field to set a different path instead of /etc/calico? Would there be any other changes necessary besides creating a field and then setting the Volume hostPath?

I seem to remember there being some paths (possibly /etc/calico but not exactly sure) that needed to be kept in sync with the node-driver-registrar so in addition to the field it may or may not be necessary to propagate the new path through to the image init call. The good news is that things will explode fairly quickly if you get it wrong so it won't be easy to miss.

@tmjd I believe the issue is that on some operating systes, like Talos, /etc/ is read-only which means Calico cannot write to that path at all.

Setting the annotation to keep the operator away was just a test to see that everything starts as expected.
@uhthomas You can skip using the CSI driver altogether if you don't need it by leaving KubeletVolumePluginPath unset. CSI is needed for Application Layer Policy, required i.e. for Istio, but is not required to use Calico.

Setting the annotation to keep the operator away was just a test to see that everything starts as expected.
@uhthomas You can skip using the CSI driver altogether if you don't need it by leaving KubeletVolumePluginPath unset. CSI is needed for Application Layer Policy, required i.e. for Istio, but is not required to use Calico.

@epleterte @uhthomas you actually need to explicitly set it to None if you want CSI disabled rather than leave it unset since unset will cause it to take on the default value as documented here: https://unified-docs.tigera.io/calico/3.25/reference/installation/api#operator.tigera.io/v1.Installation

Would you be able to clarify what the Calico CSI driver does? I am using rook, would it affect that?

Would you be able to clarify what the Calico CSI driver does? I am using rook, would it affect that?

@uhthomas the Calico CSI driver merely enables standard k8s CSI functionality so that arbitrary storage interfaces can be used. You can read more about it here: https://kubernetes.io/docs/concepts/storage/volumes/#csi

Would you be able to clarify what the Calico CSI driver does? I am using rook, would it affect that?

@uhthomas the Calico CSI driver merely enables standard k8s CSI functionality so that arbitrary storage interfaces can be used. You can read more about it here: https://kubernetes.io/docs/concepts/storage/volumes/#csi

Ah :/ So definitely not compatible with rook then. I guess in that case I and many others are definitely still blocked on this.

@caseydavenport @Josh-Tigera I do have a proposed change here: #2518. I'd appreciate your feedback.

@uhthomas this shouldn't have any interference with your Rook installation. This is just a way of Calico to write some configuration to the host using a custom storage implementation. It can also be disabled if you're not using application layer policies as mentioned above.
I think we should make the etc/calico hostpath configurable for the CSI driver though - @Josh-Tigera it should be pretty straightforward no?
This is the only place it is used:
https://github.com/tigera/operator/blob/03a6123c3ba032ae85889532b4fca90792954634/pkg/render/csi.go#L200-L207
In the meantime, if you're not using ALP, you can just turn off the CSI driver altogether by setting the following config:
spec:
  kubeletVolumePluginPath: None

If it's exclusively for CSI then yeah the changes are very straightforward. @uhthomas looked at your change it looks simple enough. Would be good to run through the steps Matt mentioned on your PR. I approved the workflow so we can at least see how the tests look. If something with that path goes awry it should be pretty obvious as the CSI init should fail loudly.

We have tested the pull-request at an immutable Kubernetes installation (Sidero Talos 1.3.5)
Please go forward with this fix as it is really needed.

For anyone following along here but not part of the conversation on https://github.com/tigera/operator/pull/2518, I've suggested that the /etc/calico volume be removed. Check my message for more details.

@tmjd Please see #2654.


SDK tools pipeline evolution

This PR shows some early exploration into restructuring the SDK tools repo and associated pipelines. I've created a pipelines folder under eng like in the azure-sdk-for-net. I also restructured the source files under a tools folder with a sub-path for each discrete tool (still a little more to be done).
For now I just have one CI pipeline that builds everything. I collapsed doc-warden down into a single job build for performance reasons and overall the pipeline for pixel-server, doc-warden and the roslyn analyzers only runs for 3-4 minutes. At this point I don't think we'll get much extra by optimizing it further since all these jobs run in parallel.
I wanted to share this now so we could discuss direction here.

cc @scbedd

Looks like you moved things under a tools folder for developers working in this repo do you think it is still better to group things by language?

Looks like you moved things under a tools folder for developers working in this repo do you think it is still better to group things by language?

This repo is different than all the others. In the others we have a lot of code that all follows the same overall pattern, but in this one we've got lots of small tools/utilities, each one optimized for the particular scenario it is trying to solve. Rather than try to group by language and enforce a common set of build patterns by that language grouping I thought it would be easier just to have sub-directories under tools to allow for some flexibility - although some things will be reusable.

This repo is different than all the others. In the others we have a lot of code that all follows the same overall pattern, but in this one we've got lots of small tools/utilities, each one optimized for the particular scenario it is trying to solve. Rather than try to group by language and enforce a common set of build patterns by that language grouping I thought it would be easier just to have sub-directories under tools to allow for some flexibility - although some things will be reusable.

Ok we can try that but I do think it might be nice to be able to share some things across the langauges, for example common targets/props in .NET, etc. Lets see what patterns end up emerging.

Closing this one down as it is now way out of date.


SPXERR_GSTREAMER_NOT_FOUND_ERROR

I was working with azure speech service to convert an audio file(.mp3) to text using your sample code for an android(Java). But it is not working and throwing me the following exception:
"SPXERR_GSTREAMER_NOT_FOUND_ERROR"
The complete logcat file is attached below:
2020-06-06 15:49:38.686 24615-24615/? I/compressedinpu: Late-enabling -Xcheck:jni
2020-06-06 15:49:38.758 24615-24615/? E/compressedinpu: Unknown bits set in runtime_flags: 0x8000
2020-06-06 15:49:38.762 24615-24615/? I/compressedinpu: Reinit property: dalvik.vm.checkjni= false
2020-06-06 15:49:38.821 24615-24615/? E/libc: Access denied finding property "runtime.mmitest.isrunning"
2020-06-06 15:49:38.848 24615-24615/? D/ActivityThread: Attach thread to application
2020-06-06 15:49:38.902 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: The ClassLoaderContext is a special shared library.
2020-06-06 15:49:39.055 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: QarthPatchMonintor::Init
2020-06-06 15:49:39.056 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: QarthPatchMonintor::StartWatch
2020-06-06 15:49:39.056 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: QarthPatchMonintor::WatchPackage: /data/hotpatch/fwkhotpatch/
2020-06-06 15:49:39.056 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: QarthPatchMonintor::CheckAndWatchPatch: /data/hotpatch/fwkhotpatch/com.microsoft.cognitiveservices.speech.samples.compressedinput
2020-06-06 15:49:39.056 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: QarthPatchMonintor::CheckAndWatchPatch: /data/hotpatch/fwkhotpatch/all
2020-06-06 15:49:39.056 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: QarthPatchMonintor::Run
2020-06-06 15:49:39.056 24615-24690/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: QarthPatchMonintor::Reading
2020-06-06 15:49:39.056 24615-24690/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: QarthPatchMonintor::CheckNotifyEvent
2020-06-06 15:49:39.056 24615-24690/com.microsoft.cognitiveservices.speech.samples.compressedinput I/compressedinpu: QarthPatchMonintor::CheckNotifyEvent before read
2020-06-06 15:49:39.070 24615-24659/com.microsoft.cognitiveservices.speech.samples.compressedinput I/HwApiCacheMangerEx: apicache path=/storage/emulated/0 state=mounted key=com.microsoft.cognitiveservices.speech.samples.compressedinput#10181#256
2020-06-06 15:49:39.071 24615-24659/com.microsoft.cognitiveservices.speech.samples.compressedinput I/HwApiCacheMangerEx: apicache path=/storage/0748-1500 state=mounted key=com.microsoft.cognitiveservices.speech.samples.compressedinput#10181#256
2020-06-06 15:49:39.080 24615-24659/com.microsoft.cognitiveservices.speech.samples.compressedinput I/HwApiCacheMangerEx: apicache path=/storage/emulated/0 state=mounted key=com.microsoft.cognitiveservices.speech.samples.compressedinput#10181#0
2020-06-06 15:49:39.080 24615-24659/com.microsoft.cognitiveservices.speech.samples.compressedinput I/HwApiCacheMangerEx: apicache path=/storage/0748-1500 state=mounted key=com.microsoft.cognitiveservices.speech.samples.compressedinput#10181#0
2020-06-06 15:49:39.109 24615-24695/com.microsoft.cognitiveservices.speech.samples.compressedinput E/AwareLog: AtomicFileUtils: readFileLines file not exist: android.util.AtomicFile@f447b2a
2020-06-06 15:49:39.111 24615-24695/com.microsoft.cognitiveservices.speech.samples.compressedinput E/AwareLog: AtomicFileUtils: readFileLines file not exist: android.util.AtomicFile@a2b6d1b
2020-06-06 15:49:39.128 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput V/ActivityThread: callActivityOnCreate
2020-06-06 15:49:39.129 24615-24659/com.microsoft.cognitiveservices.speech.samples.compressedinput I/AwareBitmapCacher: init processName:com.microsoft.cognitiveservices.speech.samples.compressedinput pid=24615 uid=10181
2020-06-06 15:49:39.183 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput V/HwWidgetFactory: : successes to get AllImpl object and return....
2020-06-06 15:49:39.226 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/OverScrollerOptimization: start init SmartSlideOverScroller and get the overscroller config
2020-06-06 15:49:39.227 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/OverScrollerOptimization: get the overscroller config
2020-06-06 15:49:39.285 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput W/compressedinpu: Accessing hidden method Landroid/view/View;->computeFitSystemWindows(Landroid/graphics/Rect;Landroid/graphics/Rect;)Z (greylist, reflection, allowed)
2020-06-06 15:49:39.286 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput W/compressedinpu: Accessing hidden method Landroid/view/ViewGroup;->makeOptionalFitsSystemWindows()V (greylist, reflection, allowed)
2020-06-06 15:49:39.360 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput D/ActivityThread: add activity client record, r= ActivityRecord{5c71292 token=android.os.BinderProxy@6a571cc {com.microsoft.cognitiveservices.speech.samples.compressedinput/com.microsoft.cognitiveservices.speech.samples.compressedinput.MainActivity}} token= android.os.BinderProxy@6a571cc
2020-06-06 15:49:39.420 24615-24700/com.microsoft.cognitiveservices.speech.samples.compressedinput D/HiTouch_PressGestureDetector: onAttached, package=com.microsoft.cognitiveservices.speech.samples.compressedinput, windowType=1, mHiTouchRestricted=false
2020-06-06 15:49:39.451 24615-24691/com.microsoft.cognitiveservices.speech.samples.compressedinput I/iGraphics: [0020080c] pn: com.microsoft.cognitiveservices.speech.samples.compressedinput, p: 24615
2020-06-06 15:49:39.451 24615-24691/com.microsoft.cognitiveservices.speech.samples.compressedinput I/iGraphics: [0030080c] no spt app: com.microsoft.cognitiveservices.speech.samples.compressedinput
2020-06-06 15:49:39.472 24615-24691/com.microsoft.cognitiveservices.speech.samples.compressedinput D/mali_winsys: EGLint new_window_surface(egl_winsys_display *, void *, EGLSurface, EGLConfig, egl_winsys_surface **, EGLBoolean) returns 0x3000
2020-06-06 15:49:39.489 24615-24691/com.microsoft.cognitiveservices.speech.samples.compressedinput W/Gralloc3: mapper 3.x is not supported
2020-06-06 15:49:39.628 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/HwViewRootImpl: removeInvalidNode jank list is null
2020-06-06 15:49:41.673 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput W/Settings: Setting device_provisioned has moved from android.provider.Settings.Secure to android.provider.Settings.Global.
2020-06-06 15:49:41.674 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput V/HiTouch_HiTouchSensor: User setup is finished.
2020-06-06 15:49:41.774 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput V/AudioManager: querySoundEffectsEnabled...
2020-06-06 15:49:41.804 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput W/compressedinput: type=1400 audit(0.0:1113658): avc: granted { read } for pid=24615 name="audio-file.mp3" dev="sdcardfs" ino=78282 scontext=u:r:untrusted_app_27:s0:c512,c768 tcontext=u:object_r:sdcardfs:s0 tclass=file
2020-06-06 15:49:41.804 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput W/compressedinput: type=1400 audit(0.0:1113659): avc: granted { read open } for pid=24615 path="/storage/emulated/0/Input/audio-file.mp3" dev="sdcardfs" ino=78282 scontext=u:r:untrusted_app_27:s0:c512,c768 tcontext=u:object_r:sdcardfs:s0 tclass=file
2020-06-06 15:49:42.842 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput E/SpeechSDKDemo: unexpected Exception with an error code: 0x29 (SPXERR_GSTREAMER_NOT_FOUND_ERROR)
2020-06-06 15:49:42.850 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput I/Choreographer: Skipped 64 frames!  The application may be doing too much work on its main thread.
2020-06-06 15:49:42.870 24615-24691/com.microsoft.cognitiveservices.speech.samples.compressedinput I/OpenGLRenderer: Davey! duration=1092ms; Flags=0, IntendedVsync=104453935193010, Vsync=104455001859634, OldestInputEvent=9223372036854775807, NewestInputEvent=0, HandleInputStart=104455009897081, AnimationStart=104455010022602, PerformTraversalsStart=104455011186144, DrawStart=104455011844477, SyncQueued=104455016699685, SyncStart=104455017663748, IssueDrawCommandsStart=104455018123644, SwapBuffers=104455025674685, FrameCompleted=104455028600727, DequeueBufferDuration=404000, QueueBufferDuration=1544000,
2020-06-06 15:49:44.139 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput D/AwareBitmapCacher: handleInit switch not opened pid=24615
2020-06-06 15:54:41.706 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput W/IInputConnectionWrapper: getExtractedText on inactive InputConnection
2020-06-06 15:54:41.723 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput W/IInputConnectionWrapper: getTextBeforeCursor on inactive InputConnection
2020-06-06 15:54:42.373 24615-24691/com.microsoft.cognitiveservices.speech.samples.compressedinput W/libEGL: EGLNativeWindowType 0x7594af1950 disconnect failed
2020-06-06 15:54:42.398 24615-24615/com.microsoft.cognitiveservices.speech.samples.compressedinput V/ActivityThread: Handle window ActivityRecord{5c71292 token=android.os.BinderProxy@6a571cc {com.microsoft.cognitiveservices.speech.samples.compressedinput/com.microsoft.cognitiveservices.speech.samples.compressedinput.MainActivity}} visibility: false
2020-06-06 15:54:47.005 24615-24651/com.microsoft.cognitiveservices.speech.samples.compressedinput W/System: A resource failed to call close.
Expected behavior
I want any prerecorded audio file to get converted into text(Transcription) .
Platform, Operating System, and Programming Language

OS: Windows 10
Hardware - 8 GB Ram, 64-bit, i5 - 8th gen
Programming language: Android (Java)

Please help me with the same as soon as possible.

@Abhi55Y If you are working on windows, you need to install gstreamer on your machine. You also need to put the path C:\gstreamer\1.0\x86_64\bin  (assuming you have installed it in C:\gstreamer ) in your system path. After that it should work fine. Before shipping your product you need to look at the gstreamer license too.
If you are working on Android, please follow the the direction given in https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-use-codec-compressed-audio-input-streams?tabs=debian&pivots=programming-language-java to create libgstreamer_android.so. Put the created libgstreamer_android.so into your android app app\src\main\jniLibs\arm64-v8a\libgstreamer_android.so  (assuming you are using arm64). Compile and run the app.
Do let us know if that solves your problem.

@Abhi55Y If you are working on windows, you need to install gstreamer on your machine. You also need to put the path C:\gstreamer\1.0\x86_64\bin (assuming you have installed it in C:\gstreamer ) in your system path. After that it should work fine. Before shipping your product you need to look at the gstreamer license too.
If you are working on Android, please follow the the direction given in https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-use-codec-compressed-audio-input-streams?tabs=debian&pivots=programming-language-java to create libgstreamer_android.so. Put the created libgstreamer_android.so into your android app app\src\main\jniLibs\arm64-v8a\libgstreamer_android.so (assuming you are using arm64). Compile and run the app.
Do let us know if that solves your problem.

The documentation of cognitive speech to text service and gstreamer is so confusing that we are unable to understand the exact steps. I request you to provide a fully functional sample because no developer can spend so much time on all this things. Or else provide some proper documentation or sample videos for all this.
The error I'm getting now is:
ERROR: executing external native build for ndkBuild C:\Users\Aarambh\Desktop\cognitive-services-speech-sdk-master\samples\java\android\compressed-input\app\jni\Android.mk
Affected Modules: app
And I'm unable to understand how to make libgstreamer_android.so
Kindly help me with a functional sample project which we can directly use in our apps.

@Abhi55Y I will provide you the steps for Android. By the way were you successful with running it on windows ?

@amitkumarshukla I didn't try on windows. Can you please provide me your personal email id or email me on<EMAIL_ADDRESS>I am working on a project where I want to convert prerecorded audio file to text(transcriber). So I want to use microsoft speech to text api. I will be thankful to you if share me a working or functional project for the same. Or you can guide me for the same. Thank you in anticipation.

Just to note, we have added a work item to our backlog in order to provide a sample project that customers can use to build GStreamer for Android more easily. We will address this in a future release. Once the work is done, we will update this issue about when the sample will be available in public.

@Abhi55Y I am going to close this issue. Please try the instructions above again. We have done this internally, and it worked fine. We also now have an item on our backlog to create a sample that makes this easier. We don't know when we will have time to get to this though. When we do resolve the backlog item, we will update this issue to let you know how we resolved it, and if applicable, where to find the code sample.
Thank you for using the Speech SDK!


Xcode 14.0 'Create Symlinks to Header Folders' warning

How frequently does the bug occur?
All the time
Description
When build an app with Xcode 14.0 it produces a warning:
Run script build phase 'Create Symlinks to Header Folders' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase.

For our run script build phases it was an easy fix we just had to remove Based on dependency analysis checkmar from Run script configuration. Not sure how it is done for cocoapods though.

Stacktrace & log output
-

Can you reproduce the bug?
Yes, always
Reproduction Steps
Build a project with Realm
Version
10.29.0
What SDK flavour are you using?
Local Database only
Are you using encryption?
No, not using encryption
Platform OS and version(s)
MacOS 12.5.1
Build environment
Xcode version: 14.0
Dependency manager and version: Cocoapods 1.11.3

This is not something we have any control over; you'll need to report it to CocoaPods.

Thanks @tgoyne you are right, there is already an issue for that
https://github.com/CocoaPods/CocoaPods/issues/11477
Hence, closing this one

Workaround for now:
post_install do |installer|
  installer.pods_project.targets.each do |target|
    if target.name == 'Realm'
      create_symlink_phase = target.shell_script_build_phases.find { |x| x.name == 'Create Symlinks to Header Folders' }
      create_symlink_phase.always_out_of_date = "1"
    end
  end
end


More support for serialization & deserialization.

Describe the issue:
FastAPI is known to can't be pickled by default pickle and dill.
Click here for the details.
However, Ray pickler can.
Contradictly, dask supports alternative pickler based on this.
Why it doesn't fall back to other methods?
Minimal Complete Verifiable Example:
import asyncio
import os

import dask
import httpx
from dask.distributed import Client
from distributed import LocalCluster
from fastapi import FastAPI

if __name__ == '__main__':
    cluster = LocalCluster(threads_per_worker=1)
    client = Client(cluster)
    print(client.scheduler_info())

    async def root():
        return {"message": "Hello World"}

    queue = dask.distributed.Queue()
    for i in range(os.cpu_count()):
        app = FastAPI()
        app.i = i
        app.add_api_route("/", root, methods=["GET"])
        queue.put(app)

    def task():
        v = queue.get()
        client = httpx.AsyncClient(app=v, base_url="http://run")
        response_future = client.get("/")
        response = asyncio.run(response_future)
        print(response.json())
        queue.put(v)
        return v

    def run_dask():
        futures = [client.submit(task) for i in range(5000)]
        for future in dask.distributed.as_completed(futures):
            val = future.result()
            print(val.i)

    run_dask()

Anything else we need to know?:
Error logs:
/usr/bin/python3 /workspace/playground/two_test.py
/usr/local/lib/python3.10/dist-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 45089 instead
  warnings.warn(
1
{'type': 'Scheduler', 'id': 'Scheduler-09680619-0876-46a0-bc9c-98b24374eb6d', 'address': 'tcp://<IP_ADDRESS>:41949', 'services': {'dashboard': 45089}, 'started':<PHONE_NUMBER>.7480896, 'workers': {'tcp://<IP_ADDRESS>:33055': {'type': 'Worker', 'id': 0, 'host': '<IP_ADDRESS>', 'resources': {}, 'local_directory': '/tmp/dask-scratch-space/worker-1xu525vd', 'name': 0, 'nthreads': 1, 'memory_limit':<PHONE_NUMBER>6, 'last_seen':<PHONE_NUMBER>.8395233, 'services': {'dashboard': 39113}, 'metrics': {'task_counts': {}, 'bandwidth': {'total': 100000000, 'workers': {}, 'types': {}}, 'digests_total_since_heartbeat': {}, 'managed_bytes': 0, 'spilled_bytes': {'memory': 0, 'disk': 0}, 'transfer': {'incoming_bytes': 0, 'incoming_count': 0, 'incoming_count_total': 0, 'outgoing_bytes': 0, 'outgoing_count': 0, 'outgoing_count_total': 0}, 'event_loop_interval': 0.02, 'cpu': 0.0, 'memory': 117628928, 'time':<PHONE_NUMBER>.6350815, 'host_net_io': {'read_bps': 27891.876894116896, 'write_bps': 27891.876894116896}, 'host_disk_io': {'read_bps': 0.0, 'write_bps': 0.0}, 'num_fds': 18, 'gpu-memory-total':<PHONE_NUMBER>4, 'gpu_utilization': 0, 'gpu_memory_used': 649797632}, 'status': 'running', 'nanny': 'tcp://<IP_ADDRESS>:43315'}}}
2024-07-04 03:31:10,117 - distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 96, in loads
    return pickle.loads(x)
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  [Previous line repeated 977 more times]
RecursionError: maximum recursion depth exceeded
2024-07-04 03:31:10,119 - distributed.core - ERROR - Exception while reading from tcp://<IP_ADDRESS>:51526
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/core.py", line 900, in _handle_comm
    msg = await comm.read()
  File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 247, in read
    msg = await from_frames(
  File "/usr/local/lib/python3.10/dist-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/usr/local/lib/python3.10/dist-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 96, in loads
    return pickle.loads(x)
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  [Previous line repeated 977 more times]
RecursionError: maximum recursion depth exceeded
2024-07-04 03:31:10,134 - distributed.protocol.pickle - ERROR - Failed to serialize maximum recursion depth exceeded.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/core.py", line 900, in _handle_comm
    msg = await comm.read()
  File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 247, in read
    msg = await from_frames(
  File "/usr/local/lib/python3.10/dist-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/usr/local/lib/python3.10/dist-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 96, in loads
    return pickle.loads(x)
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  [Previous line repeated 977 more times]
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
RecursionError: maximum recursion depth exceeded while pickling an object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 24, in reducer_override
    if _always_use_pickle_for(obj):
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
  File "/usr/lib/python3.10/collections/__init__.py", line 986, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/usr/lib/python3.10/collections/__init__.py", line 978, in __missing__
    raise KeyError(key)
RecursionError: maximum recursion depth exceeded while calling a Python object

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/usr/local/lib/python3.10/dist-packages/cloudpickle/cloudpickle.py", line 1249, in dump
    raise pickle.PicklingError(msg) from e
_pickle.PicklingError: Could not pickle object as excessively deep recursion required.
2024-07-04 03:31:10,147 - distributed.protocol.pickle - ERROR - Failed to serialize <traceback object at 0x7f941009f600>.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/core.py", line 900, in _handle_comm
    msg = await comm.read()
  File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 247, in read
    msg = await from_frames(
  File "/usr/local/lib/python3.10/dist-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/usr/local/lib/python3.10/dist-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 96, in loads
    return pickle.loads(x)
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  File "/usr/local/lib/python3.10/dist-packages/starlette/datastructures.py", line 699, in __getattr__
    return self._state[key]
  [Previous line repeated 977 more times]
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 63, in dumps
    result = pickle.dumps(x, **dump_kwargs)
RecursionError: maximum recursion depth exceeded while pickling an object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 68, in dumps
    pickler.dump(x)
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 27, in reducer_override
    serialize = dask_serialize.dispatch(type(obj))
  File "/usr/local/lib/python3.10/dist-packages/dask/utils.py", line 766, in dispatch
    raise TypeError(f"No dispatch for {cls}")
RecursionError: maximum recursion depth exceeded while getting the str of an object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
  File "/usr/local/lib/python3.10/dist-packages/cloudpickle/cloudpickle.py", line 1323, in reducer_override
    is_anyclass = issubclass(t, type)
RecursionError: maximum recursion depth exceeded while calling a Python object

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/distributed/protocol/pickle.py", line 81, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/usr/local/lib/python3.10/dist-packages/cloudpickle/cloudpickle.py", line 1249, in dump
    raise pickle.PicklingError(msg) from e
_pickle.PicklingError: Could not pickle object as excessively deep recursion required.
Traceback (most recent call last):
  File "/workspace/playground/two_test.py", line 32, in <module>
    queue.put(app)
  File "/usr/local/lib/python3.10/dist-packages/distributed/queues.py", line 236, in put
    return self.client.sync(self._put, value, timeout=timeout, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/distributed/utils.py", line 358, in sync
    return sync(
  File "/usr/local/lib/python3.10/dist-packages/distributed/utils.py", line 434, in sync
    raise error
  File "/usr/local/lib/python3.10/dist-packages/distributed/utils.py", line 408, in f
    result = yield future
  File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 766, in run
    value = future.result()
  File "/usr/local/lib/python3.10/dist-packages/distributed/queues.py", line 220, in _put
    await self.client.scheduler.queue_put(
  File "/usr/local/lib/python3.10/dist-packages/distributed/core.py", line 1395, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/distributed/core.py", line 1177, in send_recv
    _, exc, tb = clean_exception(**response)
  File "/usr/local/lib/python3.10/dist-packages/distributed/core.py", line 1862, in clean_exception
    assert isinstance(traceback, types.TracebackType) or traceback is None
AssertionError

Process finished with exit code 1

Environment:

Dask version: 2024.5.1
Python version: 3.10.12
Operating System: windows 11
Install method (conda, pip, source): pip

Contradictly, dask supports alternative pickler based on this.
Why it doesn't fall back to other methods?

Dask tries pickle and cloudpickle and if both fails it requires the user to define how to (de-)serialize an object. This is outlined on the link you are pointing to https://distributed.dask.org/en/latest/serialization.html, see specifically https://distributed.dask.org/en/latest/serialization.html#id3

Contradictly, dask supports alternative pickler based on this.
Why it doesn't fall back to other methods?

Dask tries pickle and cloudpickle and if both fails it requires the user to define how to (de-)serialize an object. This is outlined on the link you are pointing to https://distributed.dask.org/en/latest/serialization.html, see specifically https://distributed.dask.org/en/latest/serialization.html#id3

Can you support more/all object types to be serialized/deserialized in default setting? If the ray library can do that, you can too.

Can you support more/all object types to be serialized/deserialized in the default setting?

Well, all is not possible since libraries can actually opt-out of pickle and in those cases we'll have to implement custom code. We're certainly happy to add serializers for commonly used libraries. fastapi is certainly popular enough for us to accept a serializer for. We have plenty of third party libraries supported this way, see https://github.com/dask/distributed/blob/main/distributed/protocol/init.py (register_lazy registers this thing if the lib is importable)
The only way Ray is able to support this is because they are also implementing an "addon", see https://github.com/ray-project/ray/blob/master/python/ray/util/serialization_addons.py
Are you interested in contributing a patch?


Identity API review

[x] consistently place tenant_id before client_id in signatures (closes #7951)
[ ] move optional tenant_id and client_id into kwargs
[ ] remove azure.core.Configuration usage

I'm creating this as a draft to get CI running early so I can see whether consumers are broken. Incidentally, packages using ClientSecretCredential to authenticate during tests should prefer EnvironmentCredential.

Can one of the admins verify this patch?

/azp run


Allow enabling other recovery methods while using Ledger

Can we remove the "You have no method to recover your account. please add a method below." when Ledger is enabled?

@Patrick1904 @kcole16 I've merged master here so that we can re-test together with other changes (as there was some Ledger related stuff). Other than that looks good so far.

@Patrick1904 not a blocker, but looks like we still have issue with page layout. Not sure if a regression as you've previously had a fix.

@Patrick1904 @mattlockyer seems like there is a problem if you mess up anything about signing transaction after entering the code from e-mail:

seems like there is a problem if you mess up anything about signing transaction after entering the code from e-mail:

@vgrichina can you please elaborate on this? Did you enter the wrong code and that caused verification to fail completely even when you entered the correct code?

seems like there is a problem if you mess up anything about signing transaction after entering the code from e-mail:

@vgrichina can you please elaborate on this? Did you enter the wrong code and that caused verification to fail completely even when you entered the correct code?

no, didn't accept transaction in time on Ledger. It all worked out ok when I sent new code. Don't think it's a blocker, but we need to improve the flow.

@vgrichina not sure if this is the same bug that you encountered but I just found one by:

Create account with Ledger
Initiate flow to setup email recovery (/setup-recovery)
Enter the email verification code
Reject the TX on Ledger
Go back to /setup-recovery
Notice that email shows as active because the key was successfully saved in CH, but not on chain.

Two issues:

Key is added to CH before it's added to chain.
We validate the security code in CH and then add the key to chain. Only adding the key to chain is confirmed on Ledger
We're not double checking on /setup-recovery that the recovery key exists on chain before, only CH. Will fix that.

Notice that email shows as active because the key was successfully saved in CH, but not on chain.

I see, this is indeed a blocker, let's postpone this PR for next release cut

@vgrichina now filtering out recovery methods that do not have key on chain

@Patrick1904 not a blocker, but looks like we still have issue with page layout. Not sure if a regression as you've previously had a fix.

@marcinbodnar is this something you can include a fix for in your other Ledger PR? Otherwise I can. Also seems it easily overflows on mobile.

@Patrick1904 all Ledger PR are closed - I can add a fix here?

@Patrick1904 all Ledger PR are closed - I can add a fix here?

@marcinbodnar that'd be awesome! Thanks

@Patrick1904 done

@Patrick1904 when I recover using e-mail link Ledger doesn't show up in the list of devices. When I "Connect" it again – it gets enabled there, but I still have a local key for signing.  Not sure if really a blocker @kcole16

@vgrichina @Patrick1904 Not having the Ledger show is borderline a blocker. If this is a quick fix, we should implement it. If not, we should aim to have a followup fix in shortly thereafter.
As for adding a localstorage key during recovery, I'd say that is intended behavior, but we do not communicate the ramifications well. There isn't much reason to enable email recovery if the user intends to enforce Ledger signing on every TX.
We should add more warning to users when they attempt to add an additional recovery method with only Ledger installed, as some may not realize that means Ledger signing will not always be enforced. Will create a separate issue for this.

@Patrick1904 when I recover using e-mail link Ledger doesn't show up in the list of devices. When I "Connect" it again – it gets enabled there, but I still have a local key for signing. Not sure if really a blocker @kcole16

@vgrichina @kcole16 - currently when user recovers their account with link (email etc) we encourage the user to 'reconnect' their Ledger when they land on /profile. I believe this is the intended behavior?
However, as @vgrichina mentioned, we currently do not delete the localStorage key if the user 'reconnects' the Ledger. I can fix that if we consider this a blocker?

@Patrick1904 I don't thin