Add sync functionality in Xero connection

Details
Add sync functionality in Xero connection.
Fixed Issues
$ https://github.com/Expensify/App/issues/41320
PROPOSAL: https://github.com/Expensify/App/issues/41320#issuecomment-2085354319
Tests

Enable accounting.
Go to accounting page.
Connect Xero.
Use the sync connection option.
Check that "Other integrations" doesn't show Xero while sync is happening.
As of now, no API call should happen when sync is done for Xero.
Do the same for Quickbooks, API call should happen.

QA Steps

Enable accounting.
Go to accounting page.
Connect Xero.
Use the sync connection option.
Check that "Other integrations" doesn't show Xero while sync is happening.
As of now, no API call should happen when sync is done for Xero.
Do the same for Quickbooks, API call should happen.

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

Android: mWeb Chrome
https://github.com/Expensify/App/assets/35566748/b6f23505-fb93-47b0-a728-c04892756e97

iOS: Native
https://github.com/Expensify/App/assets/35566748/a6368483-3fe7-403b-951c-5992efa4818c

iOS: mWeb Safari
https://github.com/Expensify/App/assets/35566748/42526c90-f86d-432c-9153-fc12e84dcf98

MacOS: Chrome / Safari
https://github.com/Expensify/App/assets/35566748/abe37704-d10f-4ded-825f-cc6ea775e41f

MacOS: Desktop
https://github.com/Expensify/App/assets/35566748/2e5d5d84-b117-4ad7-8932-bee32edb2a12

@ShridharGoel Can you fix the lint issues?

@ShridharGoel The PR  for the Sync command on the backend is on staging. Can you please add the code for sync and finish the PR?
Let me know once it's done and ready for review.

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[x] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[ ] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG)

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[ ] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native
https://github.com/Expensify/App/assets/3069065/f298c8b0-0fca-4ee1-bf18-f17412635be5

Android: mWeb Chrome
https://github.com/Expensify/App/assets/3069065/404883de-82c6-4eca-89e8-ff6170588b79

iOS: Native
https://github.com/Expensify/App/assets/3069065/17686746-49fe-46be-b6d7-c51eed4b5cfd

iOS: mWeb Safari
https://github.com/Expensify/App/assets/3069065/18ffd9b4-20f9-411e-9b1f-d36a0aa9dfa2

MacOS: Chrome / Safari
https://github.com/Expensify/App/assets/3069065/d75b9cab-ce26-4052-a558-c38cb1e4526c

MacOS: Desktop
https://github.com/Expensify/App/assets/3069065/c58a0f2d-c89f-46b5-a386-83595c663844

just double checking, have Spanish translations been approved, right?

Reviewing shortly 👀

@hungvu193, I think this PR has been already been reviewed by another C+ 🤔

@hungvu193, I think this PR has been already been reviewed by another C+ 🤔

ah yeah, I only re-verify while Manan was out.

@ShridharGoel there are conflicts, please solve them when you can. Thanks.

@lakchote Will update in a few hours.

Any update? Thanks!

@ShridharGoel I can see lint errors. Can you please fix them? Also while we might have the merge freeze? Can you check this thread on priority for potential merge conflicts? We're aiming to merge this asap as it is blocking some of our testing flows.

🚀 Deployed to staging by https://github.com/lakchote in version: 1.4.77-11 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to staging by https://github.com/lakchote in version: 1.4.77-11 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/mountiny in version: 1.4.78-5 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Expose controller kind and name to labelling rules

Relabelling rules can use this information to attach the name of the controller that has created a pod.
In turn, this can be used to slice metrics by workload at query time, ie. "Give me all metrics that have been created by the $name Deployment"
As an illustration when used with the following labelling rules:
            - source_labels:
                - __meta_kubernetes_pod_controller_kind
              target_label: kubernetes_pod_controller_kind
            - source_labels:
                - __meta_kubernetes_pod_controller_name
              target_label: kubernetes_pod_controller_name

we get:
node_disk_bytes_written{device="sr0",instance="<IP_ADDRESS>:9100",job="kubernetes-pods",kubernetes_pod_controller_kind="DaemonSet",kubernetes_pod_controller_name="prom-node-exporter",node="minikube"}

Of course, with a Deployment we only get access to the ReplicatSet, but that's enough to derive the Deployment name with an extra regex in the labelling rules:
kubedns_dnsmasq_errors{job="kubernetes-pods",kubernetes_pod_controller_kind="ReplicaSet",kubernetes_pod_controller_name="kube-dns-1301475494"}

Sorry, I missed this, thanks for assigning me. (@dlespiau for future Kubernetes SD PRs feel free to request a review from me directly)
Generally I'm skeptical whether this is the correct thing to do as there can at least theoretically be multiple owners that are controllers, in practice however I don't see how there can be multiple controllers for one Pod and given that the upstream apimachinery code uses the same method I'm ok with this.
Looking at this with fresh eyes again, I think we should probably do the same thing in kube-state-metrics, we used to have this when the created-by annotations still existed so I think it would be useful to have.
lgtm :+1:


Move assertButtonExists from STDcheck to FlexibleMink

Various methods were added to STDcheck's WebContext instead of adding them to FlexibleMink. This is one of many PRs that will move these methods so that all projects using FlexibleMink can utilize them.

Testing file changes were not found.
Paging @jose-ochoa to review the back-end work in this pull request!
Paging @fortoulrodolfo to review the testing work in this pull request!


Create user db record on signup and handle email verification states/flow

create "createServiceRoleClient" function which has permission to bypass RLS ( e.g. for user account creation)
create user record in database
create pages for email verification states (confirm, error, pending, success)

Other than the bikeshed on directory choice, I think this all looks amazing! We can merge it whenever you're ready.


[Question] Read-Host++

Like ChatGPT Web.
Is there a way, with the PS functions here or in Spectre to create an inputbox, keep it at the bottom, type text, press enter and the inputbox remains there at the bottom ready for more. Then, after enter, the text request is handled and the results are displayed in a "listbox" above?
Like ChatGPT Web.

Not with vanilla spectre.console, the built-in prompts can't be embedded in other widgets as far as I can tell.
With a bit of powershell + spectre console layouts & live rendering you can write your own input handler and manually set the text inside a widget to simulate a text entry. Example below.
This actually came out better than expected...
https://gist.github.com/ShaunLawrie/2ed8c580d685598e6faaea60e18d96ea
https://github.com/user-attachments/assets/860d06ae-063c-498f-89ad-62988c2edbed
There is a spectre.console tui framework in the works to support more dynamic terminal applications using the spectre console widgets which I think will handle this better but it's early stages.

I think to do the listbox stuff better I need to find a way to expose the height of the widgets because you'd want to measure the size of the layout where the chat messages are and only show the ones that can fit. I just did a trivial check in that demo but it's assuming every response would be one line which isn't correct.

I'll keep pondering on this. In the meantime that gist should get you pretty far for a PoC using a real AI agent.
What will make things next-level is getting the code syntax highlighting and markdown stuff working but that's going to be a tough one 😅

You read my mind!

code syntax highlighting
chart rendering
markdown rendering

Thank you, cool stuff!


problem deploy hdp cluster

Hello,
When I launch the deploy command : ./bdutil -e platforms/hdp/ambari_env.sh deploy
The deployment end with errors :
..Wed Feb 11 15:59:12 CET 2015: Command failed: wait ${SUBPROC} on line 308.
Wed Feb 11 15:59:12 CET 2015: Exit code of failed command: 1
Wed Feb 11 15:59:12 CET 2015: Detailed debug info available in file: /tmp/bdutil-20150211-155631-ezD/debuginfo.txt
Wed Feb 11 15:59:12 CET 2015: Check console output for error messages and/or retry your command.
iMac-de-:bdutil-vc a$ cat  /tmp/bdutil-20150211-154518-AeG/debuginfo.txt
When I explore the error file /tmp/bdutil-20150211-154518-AeG/debuginfo.txt
************ ERROR logs from gcloud compute stderr ************
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [1].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [1].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [1].
Problem seem come internally from bdutil generated command :
gcloud --project=xxxxxxxx --quiet --verbosity=info compute ssh hadoop-w-0 --command=exit 0 --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=europe-west1-c
Some one know how resolve this issue ?
Thanks

Are you able to use gcloud compute to ssh into those instances or others in that zone?

My mistake it was actually a combination of a number of factors, but essentially assigning a bash "readonly" variable twice.
It should be fixed in https://github.com/GoogleCloudPlatform/bdutil/pull/17.

Greet.
All is Ok now!

I am getting the same error.  gcloud compute ssh <instance> works for me though.  Here's a section of the output:
Sat Mar 21 20:39:23 CST 2015: Warning. Failed to delete all staging files.
Sat Mar 21 20:39:23 CST 2015: Invoking on workers: ./ambari-setup.sh
....Sat Mar 21 20:39:24 CST 2015: Invoking on master: ./ambari-setup.sh
.Sat Mar 21 20:39:24 CST 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
Sat Mar 21 20:39:58 CST 2015: Exited 1 : gcloud --project=xxxx --quiet --verbosity=info compute ssh hadoop-w-0 --command=sudo su -l -c "cd ${PWD} && ./ambari-setup.sh" 2>>ambari-setup_deploy.stderr 1>>ambari-setup_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a

In the debuginfo.txt, the problem area seems to be with the ssh command:
ssh: connect to host 130.211.xxx.xxx port 22: Connection refused
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

The weird thing is that this worked for me the first time I went through but now it doesn't.  Any suggestions?

That appears to be a transient error which should be solvable by retrying your deployment. If you find a case where a GCE VM never becomes ssh-able, please send your project details to a Google cloud-support team, or email<EMAIL_ADDRESS>and someone can take a closer look at the faulty VM.


Release 0.3.0 from GitHub Actions CI/CD

Don't merge main into release branches — you cause foxtrot merges this way. It should be the other way around.


Branch feature tag command

Users will now be notified if they are adding existing tags or deleting non existing tags.
This took longer than expected. :')

Codecov Report

Merging #176 into master will increase coverage by 0.42%.
The diff coverage is 94.73%.

@@             Coverage Diff              @@
##             master     #176      +/-   ##
============================================
+ Coverage     55.74%   56.16%   +0.42%
- Complexity      698      706       +8
============================================
  Files           128      128
  Lines          2854     2886      +32
  Branches        342      349       +7
============================================
+ Hits           1591     1621      +30
- Misses         1136     1138       +2
  Partials        127      127

Impacted Files
Coverage Δ
Complexity Δ

...va/seedu/tr4cker/logic/commands/CommandResult.java
60.75% <83.33%> (+4.04%)
16.00 <0.00> (ø)

.../java/seedu/tr4cker/logic/commands/TagCommand.java
90.47% <100.00%> (+3.80%)
14.00 <6.00> (+5.00)

src/main/java/seedu/tr4cker/model/task/Task.java
86.56% <100.00%> (+1.82%)
28.00 <6.00> (+3.00)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 88a5d11...ecf6931. Read the comment docs.


PROTOCOL_PACKETS_OUT_OF_ORDER

Sometimes i receive this error with a query.
{ [Error: Packets out of order. Got: 65 Expected: 216] code: 'PROTOCOL_PACKETS_OUT_OF_ORDER', fatal: true }

this is the debug log:
  serverVersion: '5.1.73',

<-- RowDataPacket
{ id_customer: 1853,
  id_sale: 4,
  id_right: 3,
  company_name: 'zzz aa ANiudad',
  address: 'la,lala>\u0000\u0000�\u00041274\u00017\u00013\u000fabc def$zzz 691,xxx,xxxH\u0000\u0000�\u00041275\u00011\u00013\u001cCIQABR' }

{ [Error: Packets out of order. Got: 36 Expected: 195] code: 'PROTOCOL_PACKETS_OUT_OF_ORDER', fatal: true }

in node 0.10.35 x64 and mysql 2.6.1

Thanks for the report! Unfortunately there is not enough information posted above to do anything with. The best way to help would be to post the full debug log, rather than just the last few lines, because there isn't much context. Another thing that can help us is if you can distill causing the error into a tiny .js file I can run, which also helps us know what methods you are calling in what order as well for this to occur.

Hi @charlitos85 any update?

Hi, i've also got this error in a earlier version (node-mysql 2.6.2) when inserting a big dataset (160.000 records). Now after an update to node-mysql 2.8.0, the same query returns an ER_NET_PACKAGE_TOO_LARGE which means the query is too big for the server to process. You may need to increase max_allowed_packet in your server config to avoid this issue.

Ah, thanks, @tvollstaedt ! If you were getting PROTOCOL_PACKETS_OUT_OF_ORDER for the same reason as @charlitos85 (it's hard to know, since there was never an update), and now the error changed to something that makes sense, it's certainly possible we've fixed the underlying error :)

Get same error here,
node:internal/process/promises:246
          triggerUncaughtException(err, true /* fromPromise */);
          ^

Error: Packets out of order. Got: 1 Expected: 2
    at Parser._tryReadPacketHeader (/Users/minhuang/Code/mysql_http/node_modules/mysql/lib/protocol/Parser.js:470:15)
    at Parser.write (/Users/minhuang/Code/mysql_http/node_modules/mysql/lib/protocol/Parser.js:33:29)
    at Protocol.write (/Users/minhuang/Code/mysql_http/node_modules/mysql/lib/protocol/Protocol.js:38:16)
    at Socket.<anonymous> (/Users/minhuang/Code/mysql_http/node_modules/mysql/lib/Connection.js:88:28)
    at Socket.<anonymous> (/Users/minhuang/Code/mysql_http/node_modules/mysql/lib/Connection.js:526:10)
    at Socket.emit (node:events:394:28)
    at addChunk (node:internal/streams/readable:312:12)
    at readableAddChunk (node:internal/streams/readable:287:9)
    at Socket.Readable.push (node:internal/streams/readable:226:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
    --------------------
    at Protocol._enqueue (/Users/minhuang/Code/mysql_http/node_modules/mysql/lib/protocol/Protocol.js:144:48)
    at Protocol.handshake (/Users/minhuang/Code/mysql_http/node_modules/mysql/lib/protocol/Protocol.js:51:23)
    at Connection.connect (/Users/minhuang/Code/mysql_http/node_modules/mysql/lib/Connection.js:116:18)
    at /Users/minhuang/Code/mysql_http/write_multi_thread.js:87:20
    at new Promise (<anonymous>)
    at connect (/Users/minhuang/Code/mysql_http/write_multi_thread.js:86:14)
    at handle (/Users/minhuang/Code/mysql_http/write_multi_thread.js:97:11)
    at Worker.<anonymous> (/Users/minhuang/Code/mysql_http/write_multi_thread.js:26:23)
    at Worker.emit (node:events:406:35)
    at ChildProcess.<anonymous> (node:internal/cluster/worker:33:12) {
  code: 'PROTOCOL_PACKETS_OUT_OF_ORDER',
  fatal: true
}


Change enabled plugins to allow block syncing

Disables API plugins by default as well for security

Good improvement, thank you! :slightly_smiling_face:


  Fix missing calls for Mansions.  Fix missing lookups for /locate command.  Fix Temple Structure generations.
Signed-off-by: Dockter<EMAIL_ADDRESS>

This pull request surely adds some nice features.
The code looks a bit unfinished to me. System.out.println calls, a method that is almost two hundred lines long, other methods consisting of just // TODO Auto-generated method stub. Blocks like this:
            catch (Exception exception)
            {
                exception.printStackTrace();
                return;
            }

Maybe it would be better to use ReflectiveOperationException instead, with TerrainControl.log methods?
It seems the code has just been copied from OpenTerrainGen. While the license allows this, it would have been nice to mention this. cc @PG85
I will merge it when I have some time to fix up the code.

I felt it wasn't necessary to mention it because he remove all mentions of OTG being a fork of TerrainControl.

I will work at cleaning up the system.out.print's and try to get more stuff back to the actual logger.
Thanks for your consideration.

Merged! I ended up rewriting the entity support. Entity(...) is now part of the block list. Because BO3 files do not care about the order, existing BO3 files from OpenTerrainGen should work fine.

Hey guys, np, borrow whatever you like :). TerrainControl as well as the names of the original devs are still mentioned on the frontpage of the OTG git btw as well as in the mod description. If you feel credits are missing somewhere then it's no problem to add them, just let me know.


Update code to use full Norris table

Currently, the import pipeline uses a stripped-down version of Table 6 from Norris et al. (2006). This lead to a bug where the location of the radio objects was imported instead of the location of the host. The pipeline should be updated to use the original Table 6 file, which can be obtained here.

Will close this once I've confirmed my fix actually works.

Fixed in ee237da.


Completely broken

error.txt
I am no Rust or Tauri dev (just trying to learn at the mo). In a fresh Tauri install I get the following:

Error Snippet
```sh
               i128
             and 261 others
note: required by a bound in `aggregate::_::_serde::ser::SerializeStruct::serialize_field`
    --> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/ser/mod.rs:1860:21
     |
1858 |     fn serialize_field(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
     |        --------------- required by a bound in this associated function
1859 |     where
1860 |         T: ?Sized + Serialize;
     |                     ^^^^^^^^^ required by this bound in `SerializeStruct::serialize_field`
     = note: this error originates in the derive macro `Serialize` (in Nightly builds, run with -Z macro-backtrace for more info)
error[E0277]: the trait bound uom::si::Quantity<dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::Kind + 'static), L = PInt<UInt<UInt<UTerm, B1>, B0>>, M = PInt<UInt<UTerm, B1>>, N = Z0, T = NInt<UInt<UInt<UTerm, B1>, B1>>, Th = Z0>, dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second>, f32>: Serialize is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:313:17
|
313  | #[derive(Debug, Serialize, Deserialize)]
|                 ^^^^^^^^^ the trait Serialize is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = ..., L = ..., M = ..., N = ..., T = ..., Th = ...>, ..., ...>
...
319  |     energy_rate: Power,
|     ----------- required by a bound introduced by this call
|
= help: the following other types implement trait Serialize:
bool
char
isize
i8
i16
i32
i64
i128
and 261 others
note: required by a bound in aggregate::_::_serde::ser::SerializeStruct::serialize_field
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/ser/mod.rs:1860:21
|
1858 |     fn serialize_field(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
|        --------------- required by a bound in this associated function
1859 |     where
1860 |         T: ?Sized + Serialize;
|                     ^^^^^^^^^ required by this bound in SerializeStruct::serialize_field
= note: this error originates in the derive macro Serialize (in Nightly builds, run with -Z macro-backtrace for more info)
error[E0277]: the trait bound uom::si::Quantity<dyn uom::si::Dimension<I = NInt<UInt<UTerm, B1>>, J = Z0, Kind = (dyn uom::Kind + 'static), L = PInt<UInt<UInt<UTerm, B1>, B0>>, M = PInt<UInt<UTerm, B1>>, N = Z0, T = NInt<UInt<UInt<UTerm, B1>, B1>>, Th = Z0>, dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second>, f32>: Serialize is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:313:17
|
313  | #[derive(Debug, Serialize, Deserialize)]
|                 ^^^^^^^^^ the trait Serialize is not implemented for Quantity<dyn Dimension<I = ..., J = ..., Kind = ..., L = ..., M = ..., N = ..., T = ..., Th = ...>, ..., ...>
...
320  |     voltage: ElectricPotential,
|     ------- required by a bound introduced by this call
|
= help: the following other types implement trait Serialize:
bool
char
isize
i8
i16
i32
i64
i128
and 261 others
note: required by a bound in aggregate::_::_serde::ser::SerializeStruct::serialize_field
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/ser/mod.rs:1860:21
|
1858 |     fn serialize_field(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
|        --------------- required by a bound in this associated function
1859 |     where
1860 |         T: ?Sized + Serialize;
|                     ^^^^^^^^^ required by this bound in SerializeStruct::serialize_field
= note: this error originates in the derive macro Serialize (in Nightly builds, run with -Z macro-backtrace for more info)
error[E0277]: the trait bound uom::si::Quantity<dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::si::marker::TemperatureKind + 'static), L = Z0, M = Z0, N = Z0, T = Z0, Th = PInt<UInt<UTerm, B1>>>, dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second>, f32>: Serialize is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:313:17
|
313  | #[derive(Debug, Serialize, Deserialize)]
|                 ^^^^^^^^^ the trait Serialize is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = ..., L = ..., M = ..., N = ..., T = ..., Th = ...>, ..., ...>, which is required by std::option::Option<uom::si::Quantity<dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::si::marker::TemperatureKind + 'static), L = Z0, M = Z0, N = Z0, T = Z0, Th = PInt<UInt<UTerm, B1>>>, dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second>, f32>>: Serialize
...
324  |     temperature_kelin: Option,
|     ----------------- required by a bound introduced by this call
|
= help: the following other types implement trait Serialize:
bool
char
isize
i8
i16
i32
i64
i128
and 261 others
= note: required for Option<Quantity<..., ..., ...>> to implement Serialize
note: required by a bound in aggregate::_::_serde::ser::SerializeStruct::serialize_field
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/ser/mod.rs:1860:21
|
1858 |     fn serialize_field(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
|        --------------- required by a bound in this associated function
1859 |     where
1860 |         T: ?Sized + Serialize;
|                     ^^^^^^^^^ required by this bound in SerializeStruct::serialize_field
= note: the full name for the type has been written to '/Users/andy/Development/Tauri/photo-importer/src-tauri/target/debug/deps/tauri_plugin_system_info-f921e2b9bc7ad894.long-type-2250701583780203999.txt'
= note: consider using --verbose to print the full type name to the console
= note: this error originates in the derive macro Serialize (in Nightly builds, run with -Z macro-backtrace for more info)
error[E0277]: the trait bound uom::si::Quantity<dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::Kind + 'static), L = Z0, M = Z0, N = Z0, T = PInt<UInt<UTerm, B1>>, Th = Z0>, dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second>, f32>: Serialize is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:313:17
|
313  | #[derive(Debug, Serialize, Deserialize)]
|                 ^^^^^^^^^ the trait Serialize is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = dyn Kind, L = Z0, M = Z0, N = Z0, T = ..., Th = ...>, ..., ...>, which is required by std::option::Option<uom::si::Quantity<dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::Kind + 'static), L = Z0, M = Z0, N = Z0, T = PInt<UInt<UTerm, B1>>, Th = Z0>, dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second>, f32>>: Serialize
...
331  |     time_to_full: Option,
|     ------------ required by a bound introduced by this call
|
= help: the following other types implement trait Serialize:
bool
char
isize
i8
i16
i32
i64
i128
and 261 others
= note: required for Option<Quantity<..., ..., ...>> to implement Serialize
note: required by a bound in aggregate::_::_serde::ser::SerializeStruct::serialize_field
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/ser/mod.rs:1860:21
|
1858 |     fn serialize_field(&mut self, key: &'static str, value: &T) -> Result<(), Self::Error>
|        --------------- required by a bound in this associated function
1859 |     where
1860 |         T: ?Sized + Serialize;
|                     ^^^^^^^^^ required by this bound in SerializeStruct::serialize_field
= note: the full name for the type has been written to '/Users/andy/Development/Tauri/photo-importer/src-tauri/target/debug/deps/tauri_plugin_system_info-f921e2b9bc7ad894.long-type-5172835510086939520.txt'
= note: consider using --verbose to print the full type name to the console
= note: this error originates in the derive macro Serialize (in Nightly builds, run with -Z macro-backtrace for more info)
error[E0277]: the trait bound uom::si::Quantity<(dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::Kind + 'static), L = Z0, M = Z0, N = Z0, T = Z0, Th = Z0> + 'static), (dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second> + 'static), f32>: Deserialize<'_> is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:315:22
|
315  |     state_of_charge: Ratio,
|                      ^^^^^ the trait Deserialize<'_> is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = dyn Kind, L = Z0, M = Z0, N = Z0, T = Z0, Th = Z0>, ..., ...>
|
= help: the following other types implement trait Deserialize<'de>:
<bool as Deserialize<'de>>
<char as Deserialize<'de>>
<isize as Deserialize<'de>>
<i8 as Deserialize<'de>>
<i16 as Deserialize<'de>>
<i32 as Deserialize<'de>>
<i64 as Deserialize<'de>>
<i128 as Deserialize<'de>>
and 295 others
note: required by a bound in next_element
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/de/mod.rs:1726:12
|
1724 |     fn next_element(&mut self) -> Result<Option, Self::Error>
|        ------------ required by a bound in this associated function
1725 |     where
1726 |         T: Deserialize<'de>,
|            ^^^^^^^^^^^^^^^^ required by this bound in SeqAccess::next_element
error[E0277]: the trait bound uom::si::Quantity<(dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::Kind + 'static), L = PInt<UInt<UInt<UTerm, B1>, B0>>, M = PInt<UInt<UTerm, B1>>, N = Z0, T = NInt<UInt<UInt<UTerm, B1>, B0>>, Th = Z0> + 'static), (dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second> + 'static), f32>: Deserialize<'_> is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:316:13
|
316  |     energy: Energy,
|             ^^^^^^ the trait Deserialize<'_> is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = ..., L = ..., M = ..., N = ..., T = ..., Th = ...>, ..., ...>
|
= help: the following other types implement trait Deserialize<'de>:
<bool as Deserialize<'de>>
<char as Deserialize<'de>>
<isize as Deserialize<'de>>
<i8 as Deserialize<'de>>
<i16 as Deserialize<'de>>
<i32 as Deserialize<'de>>
<i64 as Deserialize<'de>>
<i128 as Deserialize<'de>>
and 295 others
note: required by a bound in next_element
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/de/mod.rs:1726:12
|
1724 |     fn next_element(&mut self) -> Result<Option, Self::Error>
|        ------------ required by a bound in this associated function
1725 |     where
1726 |         T: Deserialize<'de>,
|            ^^^^^^^^^^^^^^^^ required by this bound in SeqAccess::next_element
error[E0277]: the trait bound uom::si::Quantity<(dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::Kind + 'static), L = PInt<UInt<UInt<UTerm, B1>, B0>>, M = PInt<UInt<UTerm, B1>>, N = Z0, T = NInt<UInt<UInt<UTerm, B1>, B0>>, Th = Z0> + 'static), (dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second> + 'static), f32>: Deserialize<'_> is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:317:18
|
317  |     energy_full: Energy,
|                  ^^^^^^ the trait Deserialize<'_> is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = ..., L = ..., M = ..., N = ..., T = ..., Th = ...>, ..., ...>
|
= help: the following other types implement trait Deserialize<'de>:
<bool as Deserialize<'de>>
<char as Deserialize<'de>>
<isize as Deserialize<'de>>
<i8 as Deserialize<'de>>
<i16 as Deserialize<'de>>
<i32 as Deserialize<'de>>
<i64 as Deserialize<'de>>
<i128 as Deserialize<'de>>
and 295 others
note: required by a bound in next_element
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/de/mod.rs:1726:12
|
1724 |     fn next_element(&mut self) -> Result<Option, Self::Error>
|        ------------ required by a bound in this associated function
1725 |     where
1726 |         T: Deserialize<'de>,
|            ^^^^^^^^^^^^^^^^ required by this bound in SeqAccess::next_element
error[E0277]: the trait bound uom::si::Quantity<(dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::Kind + 'static), L = PInt<UInt<UInt<UTerm, B1>, B0>>, M = PInt<UInt<UTerm, B1>>, N = Z0, T = NInt<UInt<UInt<UTerm, B1>, B0>>, Th = Z0> + 'static), (dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second> + 'static), f32>: Deserialize<'_> is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:318:25
|
318  |     energy_full_design: Energy,
|                         ^^^^^^ the trait Deserialize<'_> is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = ..., L = ..., M = ..., N = ..., T = ..., Th = ...>, ..., ...>
|
= help: the following other types implement trait Deserialize<'de>:
<bool as Deserialize<'de>>
<char as Deserialize<'de>>
<isize as Deserialize<'de>>
<i8 as Deserialize<'de>>
<i16 as Deserialize<'de>>
<i32 as Deserialize<'de>>
<i64 as Deserialize<'de>>
<i128 as Deserialize<'de>>
and 295 others
note: required by a bound in next_element
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/de/mod.rs:1726:12
|
1724 |     fn next_element(&mut self) -> Result<Option, Self::Error>
|        ------------ required by a bound in this associated function
1725 |     where
1726 |         T: Deserialize<'de>,
|            ^^^^^^^^^^^^^^^^ required by this bound in SeqAccess::next_element
error[E0277]: the trait bound uom::si::Quantity<(dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::Kind + 'static), L = PInt<UInt<UInt<UTerm, B1>, B0>>, M = PInt<UInt<UTerm, B1>>, N = Z0, T = NInt<UInt<UInt<UTerm, B1>, B1>>, Th = Z0> + 'static), (dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second> + 'static), f32>: Deserialize<'_> is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:319:18
|
319  |     energy_rate: Power,
|                  ^^^^^ the trait Deserialize<'_> is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = ..., L = ..., M = ..., N = ..., T = ..., Th = ...>, ..., ...>
|
= help: the following other types implement trait Deserialize<'de>:
<bool as Deserialize<'de>>
<char as Deserialize<'de>>
<isize as Deserialize<'de>>
<i8 as Deserialize<'de>>
<i16 as Deserialize<'de>>
<i32 as Deserialize<'de>>
<i64 as Deserialize<'de>>
<i128 as Deserialize<'de>>
and 295 others
note: required by a bound in next_element
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/de/mod.rs:1726:12
|
1724 |     fn next_element(&mut self) -> Result<Option, Self::Error>
|        ------------ required by a bound in this associated function
1725 |     where
1726 |         T: Deserialize<'de>,
|            ^^^^^^^^^^^^^^^^ required by this bound in SeqAccess::next_element
error[E0277]: the trait bound uom::si::Quantity<(dyn uom::si::Dimension<I = NInt<UInt<UTerm, B1>>, J = Z0, Kind = (dyn uom::Kind + 'static), L = PInt<UInt<UInt<UTerm, B1>, B0>>, M = PInt<UInt<UTerm, B1>>, N = Z0, T = NInt<UInt<UInt<UTerm, B1>, B1>>, Th = Z0> + 'static), (dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second> + 'static), f32>: Deserialize<'_> is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:320:14
|
320  |     voltage: ElectricPotential,
|              ^^^^^^^^^^^^^^^^^ the trait Deserialize<'_> is not implemented for Quantity<dyn Dimension<I = ..., J = ..., Kind = ..., L = ..., M = ..., N = ..., T = ..., Th = ...>, ..., ...>
|
= help: the following other types implement trait Deserialize<'de>:
<bool as Deserialize<'de>>
<char as Deserialize<'de>>
<isize as Deserialize<'de>>
<i8 as Deserialize<'de>>
<i16 as Deserialize<'de>>
<i32 as Deserialize<'de>>
<i64 as Deserialize<'de>>
<i128 as Deserialize<'de>>
and 295 others
note: required by a bound in next_element
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/de/mod.rs:1726:12
|
1724 |     fn next_element(&mut self) -> Result<Option, Self::Error>
|        ------------ required by a bound in this associated function
1725 |     where
1726 |         T: Deserialize<'de>,
|            ^^^^^^^^^^^^^^^^ required by this bound in SeqAccess::next_element
error[E0277]: the trait bound uom::si::Quantity<(dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::Kind + 'static), L = Z0, M = Z0, N = Z0, T = Z0, Th = Z0> + 'static), (dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second> + 'static), f32>: Deserialize<'_> is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:321:22
|
321  |     state_of_health: Ratio,
|                      ^^^^^ the trait Deserialize<'_> is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = dyn Kind, L = Z0, M = Z0, N = Z0, T = Z0, Th = Z0>, ..., ...>
|
= help: the following other types implement trait Deserialize<'de>:
<bool as Deserialize<'de>>
<char as Deserialize<'de>>
<isize as Deserialize<'de>>
<i8 as Deserialize<'de>>
<i16 as Deserialize<'de>>
<i32 as Deserialize<'de>>
<i64 as Deserialize<'de>>
<i128 as Deserialize<'de>>
and 295 others
note: required by a bound in next_element
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/de/mod.rs:1726:12
|
1724 |     fn next_element(&mut self) -> Result<Option, Self::Error>
|        ------------ required by a bound in this associated function
1725 |     where
1726 |         T: Deserialize<'de>,
|            ^^^^^^^^^^^^^^^^ required by this bound in SeqAccess::next_element
error[E0277]: the trait bound uom::si::Quantity<(dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::si::marker::TemperatureKind + 'static), L = Z0, M = Z0, N = Z0, T = Z0, Th = PInt<UInt<UTerm, B1>>> + 'static), (dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second> + 'static), f32>: Deserialize<'_> is not satisfied
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tauri-plugin-system-info-0.2.0/src/model/common.rs:324:24
|
324  |     temperature_kelin: Option,
|                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait Deserialize<'_> is not implemented for Quantity<dyn Dimension<I = Z0, J = Z0, Kind = ..., L = ..., M = ..., N = ..., T = ..., Th = ...>, ..., ...>, which is required by std::option::Option<uom::si::Quantity<(dyn uom::si::Dimension<I = Z0, J = Z0, Kind = (dyn uom::si::marker::TemperatureKind + 'static), L = Z0, M = Z0, N = Z0, T = Z0, Th = PInt<UInt<UTerm, B1>>> + 'static), (dyn uom::si::Units<f32, amount_of_substance = uom::si::amount_of_substance::mole, electric_current = starship_battery::units::electric_current::ampere, length = uom::si::length::meter, luminous_intensity = uom::si::luminous_intensity::candela, mass = uom::si::mass::kilogram, thermodynamic_temperature = starship_battery::units::thermodynamic_temperature::kelvin, time = starship_battery::units::time::second> + 'static), f32>>: Deserialize<'_>
|
= help: the following other types implement trait Deserialize<'de>:
<bool as Deserialize<'de>>
<char as Deserialize<'de>>
<isize as Deserialize<'de>>
<i8 as Deserialize<'de>>
<i16 as Deserialize<'de>>
<i32 as Deserialize<'de>>
<i64 as Deserialize<'de>>
<i128 as Deserialize<'de>>
and 295 others
= note: required for Option<Quantity<..., ..., ...>> to implement Deserialize<'_>
note: required by a bound in next_element
--> /Users/andy/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.203/src/de/mod.rs:1726:12
|
1724 |     fn next_element(&mut self) -> Result<Option, Self::Error>
|        ------------ required by a bound in this associated function
1725 |     where
1726 |         T: Deserialize<'de>,
|            ^^^^^^^^^^^^^^^^ required by this bound in SeqAccess::next_element
= note: the full name for the type has been written to '/Users/andy/Development/Tauri/photo-importer/src-tauri/target/debug/deps/tauri_plugin_system_info-f921e2b9bc7ad894.long-type-1585738714020904999.txt'
= note: consider using --verbose to print the full type name to the console
</details>

What version of Tauri are you using?

Sorry, was tird last night. I think it's 1.6.7
To recreate, just create a new Tauri app. This is what I did:

npx create-tauri-app
cd {project_dir}
npm i
cd src-tauri
cargo add tauri-plugin-system-info
cd ..
npm run tauri dev

Tauri version based on this section from the Cargo lock file
[[package]]
name = "tauri"
version = "1.6.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67c7177b6be45bbb875aa239578f5adc982a1b3d5ea5b315ccd100aeb0043374"
dependencies = [
 "anyhow",
 "cocoa",
 "dirs-next",
 "dunce",
 "embed_plist",
 "encoding_rs",
 "flate2",
 "futures-util",
 "getrandom 0.2.15",
 "glib",
 "glob",
 "gtk",
 "heck 0.5.0",
 "http",
 "ignore",
 "objc",
 "once_cell",
 "open",
 "percent-encoding",
 "rand 0.8.5",
 "raw-window-handle",
 "regex",
 "semver",
 "serde",
 "serde_json",
 "serde_repr",
 "serialize-to-javascript",
 "state",
 "tar",
 "tauri-macros",
 "tauri-runtime",
 "tauri-runtime-wry",
 "tauri-utils",
 "tempfile",
 "thiserror",
 "tokio",
 "url",
 "uuid",
 "webkit2gtk",
 "webview2-com",
 "windows 0.39.0",
]

Any thoughts? Can you recreate the issue?

Yeah,there is some serialization error.
Not sure why. It used to work. No dependengcy version has changed. I am working on fixing it.

Should now be fixed in v1.0.0

Wahoo! Yes, that fixed it
Thank you very much


Address recaptcha tech review feedback

This PR include the changes addressing tech review feedback.
comments are extract from PR https://github.com/firebase/firebase-admin-node/pull/1603
Discussion

Read the contribution guidelines (CONTRIBUTING.md).
If this has been discussed in an issue, make sure to link to the issue here.
If not, go file an issue about this before creating a pull request to discuss.

Testing

Make sure all existing tests in the repository pass after your change.
If you fixed a bug or added a feature, add a new test to cover your code.

API Changes

At this time we cannot accept changes that affect the public API.  If you'd like to help
us make Firebase APIs better, please propose your change in an issue so that we
can discuss it together.

cc @kevinthecheung


Remove a level of indirection from internal std.xml.ElementParser code

Removes an odd choice to keep the base string of a tag as a pointer to a string rather than the string itself.
I don't see an obvious reason why this was done, so I removed it for code simplicity and possible speed improvements.

A pointer to a string might be necessary if the caller expects the callee to modify the string (as in, replace it with a different slice of the original data).

In modern D we'd pass the string by ref, but this code may predate that, I'm not sure.

But if the autotester passes, that probably means it was completely unnecessary and we can just merge this. :-D

As best as I can tell, it looks like an attempt to avoid copying. Which is ridiculous as the string struct is tiny in reality and you end up having a pointer to a pointer for the actual data of the string, which is very bad for caches.

I say we just merge this and see what explodes. We can clean up later. This code is badly in need of a major cleanup, new std.xml candidates notwithstanding.

@quickfur Thanks. For more std.xml clean up please consider looking at https://github.com/dlang/phobos/pull/5378

This introduced a regression. See #5489.


Implement AbsGS

Abs grad for gaussian splitting criterion. See

"AbsGS: Recovering Fine Details for 3D Gaussian Splatting"
"EfficientGS: Streamlining Gaussian Splatting for Large-Scale High-Resolution Scene Representation"

AbsGS: Recovering Fine Details for 3D Gaussian Splatting

@niujinshuchong  It is really nice that you bring this up! Sorry that we missed it but actually your paper is already under my radar and is exactly the next paper I plan to read in depth (I have skimmed through it and very like it!).
That being said, I just realized there is a subtle but probably important difference between the AbsGS paper and yours. It seems like what's proposed in the AbsGS  paper is abs-sum-norm (first abs grad_x and grad_y, then sum over all pixels, and  take the norm after than). What's in your paper seems to be abs-norm-sum (please correct me if I'm wrong).
I was implementing following the AbsGS paper but since you bring this up I actually now feels your way might makes more sense. I wonder what's your thoughts on this.
+cc @jb-ye

Hi, thanks for the kind words.
We implemented it as abs-sum-sum, please check here, the l2-norm is a typo in the paper. I agree different norms could be used. But as a result, different thresholds should be chosen.
Another difference of our implementation to the other papers is that we do not disable 3DGS's original metric but instead combine them together. We first select the candidates for densification with 3DGS's metric and check how many percentage of the Gaussians are selected and then select the same number of candidates with our metric. By doing so, we don't need to tune the hyperparameter for the threshold and it is guarantee that we will get at least the same quality as the original 3DGS's method.

The original approach (sum-norm) and the new approach (abs-sum-norm) is like two extremes. What really matters here to identify how to split the gaussians into two such that variance of pixel partial gradients within each group are minimized.
Intuitively, the ideal approach is very like the one used in training a regression tree, each time we find the best split to minimize some form of errors / variances, and we do so by growing the data structure (e.g. tree) in a greedy way.

I tested this in nerfstudio by switching self.xys.grad to self.xys.absgrad and for some reason much fewer gaussians get split and the quality is worse. (screenshots attached, in particular look at the trees in the background)
Normal grads:

this PR (xys.absgrad)

Yeah from my limited experiments I also didn't find it to be a life saver. It might also because that there are many thresholds entangled together, making it hard to predict how a single one would affect the overall procedure

One thing one should keep an eye on is the large gaussian culling. This is the major difference of splatfacto and the original GS code. You can try to disable it and see if things improve with AbsGrad.

@liruilong940607 Hey, I found a bug in your implementation. See my latest commit.


Broke out encoding/decoding mechanisms

Broke out the encoding/decoding mechanism so if you want to make any further changes, it should only be in one place. Ended up going with the pseudo-json format because it gave non-buffer values priority in terms of encoding performance.

some questions/nits but overall looks good

You'll notice that I'm encoding keys as base64. This is b/c we were getting keys that don't work with sql when using utf8 or binary encoding.

we my need to just look at what leveldown does which i believe is to turn everything into a buffer before storing it, e.g. for a value
if (typeof value === 'string')
    value = new Buffer(value)

value = value.toString('base64');

keys could be similar but in hex so they could sort right

Couldn't find it in leveldown, but didnt look through the c++ code.

Superseded by #22


Adding comments.

This work is partly going on in http://github.com/praekelt/molo.comments but the TuneMe implementation bits will be reflected here.

Not ready for review yet.

aaah yeah... ready now? if so :+1:

Not not ready for review yet.

:clap: nice work

@saeedmb8 @bruskiza @miltontony ready for review.
Doesn't meet everything required in the scope yet but we're getting closer.
This builds on the work done in https://github.com/praekelt/molo.commenting

:+1: :ok_hand:

:+1:

Created #9 for anonymous commenting, will address that there.


[QUESTION] FastAPI keeps booting new workers when deploying

Hi there,
I'm trying to run two different instances of FastAPI in the same Kubernetes pod (different Docker containers), but one of the two seems to keep booting up new workers. Therefor, the container uses a lot of CPU for nothing. It's also reporting that it's waiting for application startup constantly, but it is reachable (but slow).
Here is the log that I keep getting:
Checking for script in /app/prestart.sh
Running script /app/prestart.sh
Running inside /app/prestart.sh, you could add migrations to this file, e.g.:
 #! /usr/bin/env bash
 # Let the DB start
sleep 10;
# Run migrations
alembic upgrade head
 [2019-05-24 11:51:24 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-05-24 11:51:24 +0000] [1] [INFO] Listening at: http://<IP_ADDRESS>:8081 (1)
[2019-05-24 11:51:24 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2019-05-24 11:51:24 +0000] [8] [INFO] Booting worker with pid: 8
[2019-05-24 11:51:24 +0000] [9] [INFO] Booting worker with pid: 9
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:51:33 +0000] [9] [INFO] Started server process [9]
[2019-05-24 11:51:33 +0000] [9] [INFO] Waiting for application startup.
[2019-05-24 11:51:33 +0000] [8] [INFO] Started server process [8]
[2019-05-24 11:51:33 +0000] [8] [INFO] Waiting for application startup.
[2019-05-24 11:51:37 +0000] [18] [INFO] Booting worker with pid: 18
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:51:44 +0000] [18] [INFO] Started server process [18]
[2019-05-24 11:51:44 +0000] [18] [INFO] Waiting for application startup.
[2019-05-24 11:51:45 +0000] [23] [INFO] Booting worker with pid: 23
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:51:53 +0000] [23] [INFO] Started server process [23]
[2019-05-24 11:51:53 +0000] [23] [INFO] Waiting for application startup.
[2019-05-24 11:51:54 +0000] [28] [INFO] Booting worker with pid: 28
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:52:02 +0000] [28] [INFO] Started server process [28]
[2019-05-24 11:52:02 +0000] [28] [INFO] Waiting for application startup.

This will continue untill I shut it down. This problem weirdly does not happen when I run everything locally.
A difference between local and remote deployment is probably only the port that I set.
As you can see, I use port 8081:
[INFO] Listening at: http://<IP_ADDRESS>:8081 (1)

I set the port by specifying it in my Kubernetes yaml file, but I am unsure if this has something to do with the problem.

Iirc there are no max workers set so I'd either tweak the gunicorn conf to
set your number according to the server you deploy on, or set another env
var, can't remember on top of my head but it's in the docs I'm sure
Le ven. 24 mai 2019 à 2:03 PM, SansDK<EMAIL_ADDRESS>a écrit :

Hi there,
I'm trying to run two different instances of FastAPI in the same
Kubernetes pod (different Docker containers), but one of the two seems to
keep booting up new workers. Therefor, the container uses a lot of CPU for
nothing. It's also reporting that it's waiting for application startup
constantly, but it is reachable (but slow).
Here is the log that I keep getting:
Checking for script in /app/prestart.sh
Running script /app/prestart.sh
Running inside /app/prestart.sh, you could add migrations to this file, e.g.:
#! /usr/bin/env bash
Let the DB start
sleep 10;
Run migrations
alembic upgrade head
[2019-05-24 11:51:24 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-05-24 11:51:24 +0000] [1] [INFO] Listening at: http://<IP_ADDRESS>:8081 (1)
[2019-05-24 11:51:24 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2019-05-24 11:51:24 +0000] [8] [INFO] Booting worker with pid: 8
[2019-05-24 11:51:24 +0000] [9] [INFO] Booting worker with pid: 9
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:51:33 +0000] [9] [INFO] Started server process [9]
[2019-05-24 11:51:33 +0000] [9] [INFO] Waiting for application startup.
[2019-05-24 11:51:33 +0000] [8] [INFO] Started server process [8]
[2019-05-24 11:51:33 +0000] [8] [INFO] Waiting for application startup.
[2019-05-24 11:51:37 +0000] [18] [INFO] Booting worker with pid: 18
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:51:44 +0000] [18] [INFO] Started server process [18]
[2019-05-24 11:51:44 +0000] [18] [INFO] Waiting for application startup.
[2019-05-24 11:51:45 +0000] [23] [INFO] Booting worker with pid: 23
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:51:53 +0000] [23] [INFO] Started server process [23]
[2019-05-24 11:51:53 +0000] [23] [INFO] Waiting for application startup.
[2019-05-24 11:51:54 +0000] [28] [INFO] Booting worker with pid: 28
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:52:02 +0000] [28] [INFO] Started server process [28]
[2019-05-24 11:52:02 +0000] [28] [INFO] Waiting for application startup.
This will continue untill I shut it down. This problem weirdly does not
happen when I run everything locally.
A difference between local and remote deployment is probably only the port
that I set.
As you can see, I use port 8081:
[INFO] Listening at: http://<IP_ADDRESS>:8081 (1)
I set the port by specifying it in my Kubernetes yaml file, but I am
unsure if this has something to do with the problem.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tiangolo/fastapi/issues/253?email_source=notifications&email_token=AAINSPWLEJ53IUGIRMY2MDDPW7KQBA5CNFSM4HPO4Y32YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVV7SAQ,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAINSPTKBVSZMRPIDNLUCDTPW7KQBANCNFSM4HPO4Y3Q
.

Le ven. 24 mai 2019 à 2:03 PM, SansDK<EMAIL_ADDRESS>a écrit :

Hi there,
I'm trying to run two different instances of FastAPI in the same
Kubernetes pod (different Docker containers), but one of the two seems to
keep booting up new workers. Therefor, the container uses a lot of CPU for
nothing. It's also reporting that it's waiting for application startup
constantly, but it is reachable (but slow).
Here is the log that I keep getting:
Checking for script in /app/prestart.sh
Running script /app/prestart.sh
Running inside /app/prestart.sh, you could add migrations to this file, e.g.:
#! /usr/bin/env bash
Let the DB start
sleep 10;
Run migrations
alembic upgrade head
[2019-05-24 11:51:24 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-05-24 11:51:24 +0000] [1] [INFO] Listening at: http://<IP_ADDRESS>:8081 (1)
[2019-05-24 11:51:24 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2019-05-24 11:51:24 +0000] [8] [INFO] Booting worker with pid: 8
[2019-05-24 11:51:24 +0000] [9] [INFO] Booting worker with pid: 9
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:51:33 +0000] [9] [INFO] Started server process [9]
[2019-05-24 11:51:33 +0000] [9] [INFO] Waiting for application startup.
[2019-05-24 11:51:33 +0000] [8] [INFO] Started server process [8]
[2019-05-24 11:51:33 +0000] [8] [INFO] Waiting for application startup.
[2019-05-24 11:51:37 +0000] [18] [INFO] Booting worker with pid: 18
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:51:44 +0000] [18] [INFO] Started server process [18]
[2019-05-24 11:51:44 +0000] [18] [INFO] Waiting for application startup.
[2019-05-24 11:51:45 +0000] [23] [INFO] Booting worker with pid: 23
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:51:53 +0000] [23] [INFO] Started server process [23]
[2019-05-24 11:51:53 +0000] [23] [INFO] Waiting for application startup.
[2019-05-24 11:51:54 +0000] [28] [INFO] Booting worker with pid: 28
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 11:52:02 +0000] [28] [INFO] Started server process [28]
[2019-05-24 11:52:02 +0000] [28] [INFO] Waiting for application startup.
This will continue untill I shut it down. This problem weirdly does not
happen when I run everything locally.
A difference between local and remote deployment is probably only the port
that I set.
As you can see, I use port 8081:
[INFO] Listening at: http://<IP_ADDRESS>:8081 (1)
I set the port by specifying it in my Kubernetes yaml file, but I am
unsure if this has something to do with the problem.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tiangolo/fastapi/issues/253?email_source=notifications&email_token=AAINSPWLEJ53IUGIRMY2MDDPW7KQBA5CNFSM4HPO4Y32YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVV7SAQ,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAINSPTKBVSZMRPIDNLUCDTPW7KQBANCNFSM4HPO4Y3Q
.

I'll have a look in the docs, but this isn't expected behavior.
I've deployed FastAPI with uvicorn and gunicorn before, and I never had to worry about gunicorn settings. Now it's just creating 1000+ workers for no reason and without any load at all.
For now I'll just try to set the WORKERS_PER_CORE environment variable and see if that fixes the problem. I guess it will, but it's still weird behavior because the default WORKERS_PER_CORE is 1, and I'm fairly sure I don't have a 1000+ cpu cores on my server.
Anyways, I'll report back if setting the variable fixes the problem. Thanks!

Okay, I set the WORKERS_PER_CORE to 1 to no avail.
I struggle to understand how this is happening..
I have 1 Kubernetes pod with 2 different Docker container, each containing a FastAPI instance.
One of the two always works and the other one keeps doing this looping.
Since it keeps Waiting for application startup, does it check on the wrong port for startup maybe?
What does the (1) mean on the line Listening at: http://<IP_ADDRESS>:8081 (1)?
Thanks @euri10 for the help.

WORKERS_PER_CORE is a float telling the container how many workers per core to boot.
If you have it set to 1 and you have 64 processors with 2 cores each one (128 cores), it will start 128 workers.
To make it a fixed number of workers you can use WEB_CONCURRENCY: https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker#web_concurrency
That will fix the number of workers to that specific number (independent of the number of available cores).
The image is configured to, by default, get the maximum performance available. But it might be a waste of resources and you might not really need to handle 30,000 (or more) requests per second. In that case, you can play around with those configurations as you need.
There are several combinations of configurations you can use, check the docs: https://github.com/tiangolo/uvicorn-gunicorn-fastapi-docker

Hey @tiangolo, thanks for your reply.
I tried to set the WEB_CONCURRENCY variable (to 2), but still I'm getting the following log:
[2019-05-24 14:15:29 +0000] [488] [INFO] Booting worker with pid: 488
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 14:15:37 +0000] [488] [INFO] Started server process [488]
[2019-05-24 14:15:37 +0000] [488] [INFO] Waiting for application startup.
[2019-05-24 14:15:38 +0000] [493] [INFO] Booting worker with pid: 493
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 14:15:46 +0000] [493] [INFO] Started server process [493]
[2019-05-24 14:15:46 +0000] [493] [INFO] Waiting for application startup.
[2019-05-24 14:15:47 +0000] [498] [INFO] Booting worker with pid: 498
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 14:15:54 +0000] [498] [INFO] Started server process [498]
[2019-05-24 14:15:54 +0000] [498] [INFO] Waiting for application startup.
[2019-05-24 14:15:56 +0000] [503] [INFO] Booting worker with pid: 503
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 14:16:03 +0000] [503] [INFO] Started server process [503]
[2019-05-24 14:16:03 +0000] [503] [INFO] Waiting for application startup.
[2019-05-24 14:16:05 +0000] [508] [INFO] Booting worker with pid: 508
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 14:16:13 +0000] [508] [INFO] Started server process [508]
[2019-05-24 14:16:13 +0000] [508] [INFO] Waiting for application startup.
[2019-05-24 14:16:14 +0000] [513] [INFO] Booting worker with pid: 513
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-24 14:16:22 +0000] [513] [INFO] Started server process [513]
[2019-05-24 14:16:22 +0000] [513] [INFO] Waiting for application startup.
[2019-05-24 14:16:23 +0000] [518] [INFO] Booting worker with pid: 518
WARNING:root:email-validator not installed, email fields will be treated as str.

Would you have any idea why this could be happening with the variable set to a fixed, small value?
Could an error in my Python code cause this?
I am currently executing some code in this method:
@app.on_event("startup")
def startup_event():
    ...

Could it be that the application keeps trying to start if something in here fails?
Thanks.

That's quite strange. I guess it might be related to the startup event, if there's an exception there, I think it won't necessarily be properly handled by Uvicorn (specifically there in the startup event), I think there's an issue about it somewhere, but I'm not sure.
Could you put a try there and log if there's any exception? That might help finding out the root case of the problem.

That's what I thought too.
Unfortunately I won't be able to try this out anymore today, but I'll come back on this as the first thing on monday morning.
Thanks so far!

Cool!

Hi there, I'm back.
I added a try/except, but there is no exception while running.
Now, I commented out everything in (and including) the:
@app.on_event("startup")
def startup_event():

And now it runs fine..
Things that I was doing in the startup method were:
global db_ip, db_port, x, y, z
x = load_x()   # Reads a pickled file and returns it
y = load_y()   # Reads a pickled file and returns it
db_ip = os.getenv('DB_IP', default="...")
db_port = os.getenv('DB_PORT', default="...")
z = init_z()     # Makes a connection to the DB and executes a query fairly quickly

Is any of this not possible, or not smart to do in the startup method?
Thanks.

you didn't pint the whole log
in particular the default guncorn conf spits out the number of workers
https://github.com/tiangolo/uvicorn-gunicorn-docker/blob/dea28f070ea5bfcb9dda6b6f66d8708562a1f861/python3.7-alpine3.8/gunicorn_conf.py#L32-L42
If after checking it's effectively at 2 like you configured with WEB_CONCURRENCY, what I'd do is use a custom gunicorn conf, like explained here  https://github.com/tiangolo/uvicorn-gunicorn-docker#custom-gunicorn-configuration-file
imho you're hit by https://github.com/tiangolo/uvicorn-gunicorn-docker/blob/dea28f070ea5bfcb9dda6b6f66d8708562a1f861/python3.7-alpine3.8/gunicorn_conf.py#L23
in my custom gunicorn config I use something like
max_workers = os.getenv("MAX_WORKERS", "2")
...
    web_concurrency = min(int(max_workers), int(default_web_concurrency))

I think the 1st thing is to make sure you effectively launch gunicorn with 2 workers, from the log it seems not

Are you sure that the problem is related to that?
If I comment out my startup event code, this is the (full) log that I'm getting:
Checking for script in /app/prestart.sh
Running script /app/prestart.sh
Running inside /app/prestart.sh, you could add migrations to this file, e.g.:
 #! /usr/bin/env bash
 # Let the DB start
sleep 10;
# Run migrations
alembic upgrade head
 [2019-05-27 06:50:44 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-05-27 06:50:44 +0000] [1] [INFO] Listening at: http://<IP_ADDRESS>:8081 (1)
[2019-05-27 06:50:44 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2019-05-27 06:50:44 +0000] [8] [INFO] Booting worker with pid: 8
[2019-05-27 06:50:44 +0000] [9] [INFO] Booting worker with pid: 9
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
WARNING:root:email-validator not installed, email fields will be treated as str.
To install, run: pip install email-validator
[2019-05-27 06:50:53 +0000] [8] [INFO] Started server process [8]
[2019-05-27 06:50:53 +0000] [8] [INFO] Waiting for application startup.
[2019-05-27 06:50:53 +0000] [9] [INFO] Started server process [9]
[2019-05-27 06:50:53 +0000] [9] [INFO] Waiting for application startup.

And that's all. No looping like in the previous logs.
 It looks like this:
{"loglevel": "info", "workers": 2, "bind": "<IP_ADDRESS>:8081", "workers_per_core": 1.0, "host": "<IP_ADDRESS>", "port": "8081"}

This confirms that there are 2 workers, and just 1 worker per core.
To me it seems more like there is something weird going on in the startup event. I couldn't find anything about the startup event method in this repo, so I'm not sure how it works exactly. Is there some timeout value on the startup method, maybe?
Thanks for the help.

so if I got it correctly, without a custom that
global db_ip, db_port, x, y, z
x = load_x()   # Reads a pickled file and returns it
y = load_y()   # Reads a pickled file and returns it
db_ip = os.getenv('DB_IP', default="...")
db_port = os.getenv('DB_PORT', default="...")
z = init_z()     # Makes a connection to the DB and executes a query fairly quickly

it works fine, 2 workers like you set
with just the code above it spins more workers ?

Something like that seems to be happening, yes. Maybe because of some error happening in the startup, but it doesn't throw any exception.
I can't share the full code, but I'll provide the baseline.
main.py:
import pandas as pd
import pickle
import xgboost as xgb
# import uvicorn
import psycopg2 as pg
import os
import importlib
import sys
import json
from sklearn.preprocessing import QuantileTransformer
from typing import List
from datetime import datetime
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI(title="Title", description="Description ", version="v0.1")

@app.post("/x", tags=["tag1"], summary="Summary")
def x_func(req: Request):
# couple of more endpoints and methods

def load_x():
    try:
        return pickle.load(open("filex.dat", "rb"))
    except:
        print("Can't load x file", file=sys.stderr)
        return None

def load_y():
    try:
        return pickle.load(open("filey.dat", "rb"))
    except:
        print("Can't load y file", file=sys.stderr)
        return None

def init_knpafst():
    connection = pg.connect("..........")
    return pd.read_sql("SELECT ........")

db_ip = ""
db_port = ""
knpafst = None
model = None
qt_y = None

@app.on_event("startup")
def startup_event():
    global db_ip, db_port, x, y, z
    x = load_x()   # Reads a pickled file and returns it
    y = load_y()   # Reads a pickled file and returns it
    db_ip = os.getenv('DB_IP', default="...")
    db_port = os.getenv('DB_PORT', default="...")
    z = init_z()     # Makes a connection to the DB and executes a query fairly quickly

Dockerfile:
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.6

RUN pip install pandas numpy requests fastapi pydantic datetime xgboost psycopg2 scikit-learn==0.20.3

ENV DB_IP=...
ENV DB_PORT=...

COPY ./app /app

Kubernetes yaml file:
      - name: test-name
        imagePullPolicy: Always
        image: ...image
        env:
        - name: PORT
          value: '8081'
        - name: WEB_CONCURRENCY
          value: '2'
        ports:
        - containerPort: 8081
        resources:
          limits:
            cpu: 200m
            memory: 300Mi
          requests:
            cpu: 50m
            memory: 150Mi

Is that enough of an example or do you need other more specific info?
Thanks!

Hey,
I did some further research and I found out what is causing the issue.
It seems to be some kind of timeout issue. I was messing around commenting and uncommenting lines of code, and everything worked without executing the DB query. Then I uncommented the DB query and added a LIMIT, which also worked!
So with the when executing the following query, everything works fine (and gunicorn just boots 2 workers):
pd.read_sql("SELECT ... LIMIT 1000", connection)

And when I change the query to no limit, gunicorn keeps booting workers (which has something to do with some kind of startup timeout I guess):
pd.read_sql("SELECT ...", connection)

Without the LIMIT it's fetching around 185k rows with 5 columns, which is quite a bit, but not huge.
@euri10 @tiangolo Do any of you know a fix for this problem? Can I define a timeout value in a gunicorn conf file maybe?
Thank you.

You can add a custom timeout variable to a custom Gunicorn config. But I think I should add an env var for this, it seems to be a frequent need.

Thanks a lot @tiangolo, that fixed my problem!
I added a gunicorn_conf.py file and added:
timeout = 100
graceful_timeout = 100

An env var for this would be a nice to have, but since the gunicorn conf is easy to use as well, I wouldn't give it a high priority.
Thanks again for the help, and thanks for the great FastAPI lib!

Sorry, I spoke too quickly.
I still had the LIMIT 1000 in place, so of course it worked (I haven't had my coffee yet).
Now that I removed the limit, it keeps booting new workers like earlier, sigh.
Guess I'll try increasing the timeout value, but I highly doubt that it can even take longer than 100 seconds already.

I set the timeout value to 200 seconds, but still no succes.
The weird thing is that this starts happening in way less than 200 seconds, which indicates that it

ignores the timeout (unlikely)
or, the env vars don't get passed on through Kubernetes correctly somehow

I'll do some checks to see if the env vars get passed on correctly.

For now I changed WEB_CONCURRENCY to 1, which is not ideal but at least it seems to prevent the continuous worker-booting.

It's strange that it works with WEB_CONCURRENCY 1 but doesn't with more workers, might be some other configuration in Gunicorn or something else in the environment...?
Anyway, I'm glad you solved your problem.
An alternative that might be equally feasible (or maybe even better) is to replicate the containers, taking advantage of Docker and Kubernetes. That also means that you can scale more easily in memory and processors horizontally, by not being constrained to a single container.

I had the same issue, running FASTApi server on ECS. Turns out the timeout was because by server could not access the RDS on another VPC due to Sercurity Group Rules.

I'm having the same issue as well. My api works with WEB_CONCURRENCY="1" but when I increase the number of workers, it goes into a loop of booting up new workers over and over again. I've increased the timeout configuration in gunicorn_conf.py but doesn't seem to help. For now, I will scale by replicating containers, but would be helpful if we can get a solution to this.

I ran into the same issue. I am suspecting in my case (8 cpus) with webconcurrency at None and workers per core 1 by default it would try to start 8 workers. However, the data I am loading at on_startup is probably to large for 8 workers so maybe they die and gunicorn tries to restart those who crash. Does that make sense?

Yeah, I think that could make sense @fdroessler .

Hi
I have the same issues when deploy FastAPI on GCP's k8s.
Can't figure what the problem is.
Thanks in advance

I have the same issues when deploy FastAPI on GCP's k8s.

It turns out is the pod's memory limit is too small, after increase the memory limit the booting worker issue disappear it.


add Helpers.find_relation method and maintenance/testing stuff

This PR:

adds a dedicated CollectionSpace::Client.find_relation method (mixed in from Helpers) that belongs here rather than in collectionspace-mapper.
updates the find method to set the default search field to the term for authorities/vocabularies and ID for other service types
moves some tests from examples/search.rb into helpers_spec.rb (describing find) Several of these were failing, and I had to run that file manually to know.
fixes a "raises_error" test Rspec was complaining about not being specific enough.
configures VCR to allow HTTP request when an expected cassette does not exist. I added this in order to be able to run the tests on the line dev instance, and it is likely to be useful again in the future.
refreshes all of the VCR cassettes, with the exception of the one verifying that client can retrieve all object records. I did this to verify everything still worked against the API for 7.0. The one file was not updated because (a) there are currently over 1000 objects in dev; and (b) listing all object records in an instance does not seem at all related to the errors I was trying to track down.
configures Rspec to be able to run only failures

I'm going to leave raising that error for later, as it is more complicated than I thought. Or at least the pattern is not as clear as I hoped and I need input from Ray, and I want this bug to get fixed ASAP.


Different .time granularity on different platforms?

First of all, thank you for the package!
I've used UUID7 from your package as an ID (token) for user session - I want to use time stored in the token as a creation timestamp directly. And I want to invalidate the token after some time (a day). So what I did was to fetch .time from the UUID and compare it to current time.timestamp. Problem is - on different platforms UUID.time returns different order of values. As a workaround I have to divide by certain value to make it possible to compare.
Final function currently looks like this:
def check_token_expired(uuid: UUID, session_timeout_seconds: int) -> bool:
    """Check if session has expired by uuid v7 timestamp. Returns True if session IS expired."""
    assert isinstance(uuid, UUID)
    now_unix = datetime.now().timestamp()
    uuid_time = uuid.time
    if os.name == 'posix':
        uuid_time = uuid_time / 1000
    if os.name == 'nt':
        uuid_time = uuid_time /<PHONE_NUMBER>
    return (now_unix - uuid_time) > session_timeout_seconds

Those os.name == ... comparissons look wrong to me. And it looks like the difference comes from uuid.time, not from datetime.now().timestamp(). Could you help checking this please?
Platforms I've tested on are Windows 10 and WSL with Ubuntu.

The whole .time property is somewhat hidden even in the official UUID library. https://docs.python.org/3.11/library/uuid.html
For example the UUD7 .time implementation of this library has changed the returned values from nanosecond precision to millisecond precision when the official draft made the same change.

Sorry, I forgot to update this issue. It's no longer relevant. I'm not sure what, when and where (probably Python update) was changed, but one day there was no longer a difference in readings, so I could remove that workaround.


Latest version 2.+ no longer supports setActiveTabColor

I have written some wrappers that let me programmatically change colors within my app and part of that process is when the toolbar color changes I also change the active tab color to reflect the new color. It appears in the latest version you just released you no longer support the ability to call setActiveTabColor from code and instead require that the color is set in the xml file. I would MUCH prefer to have setActiveTabColor back please.

I added support for that and all other properties for the upcoming version 2.0.2.
See the commit here.
I will release the new version during this weekend.


How do I get my native guide back?

Description
What needs to be changed in the registry so that the native explorer returns?
Steps To Reproduce
No response
Expected behavior
more stable work of the program
Files Version
2.0.34
Windows Version
Windows 10 Pro 21H1 19043.1348
Relevant Assets
No response

Hi, you have to follow the instructions here to unset as default file explorer:
https://files.community/docs/configuring/replace-file-explorer
TL;DR: download this zip and run UnsetFilesAsDefault.reg to restore windows explorer

Hi, you have to follow the instructions here to unset as default file explorer:
https://files.community/docs/configuring/replace-file-explorer
TL;DR: download this zip and run UnsetFilesAsDefault.reg to restore windows explorer

Why can't this function be added to the program settings?

Hi, you have to follow the instructions here to unset as default file explorer:
https://files.community/docs/configuring/replace-file-explorer
TL;DR: download this zip and run UnsetFilesAsDefault.reg to restore windows explorer

Why can't this function be added to the program settings?

It's already on the Experimental part of Files settings, though previous insider versions had a bug that caused that "the option to reset the default file explorer wouldn't work after reinstalling Files". This bug has been fixed on last insider version.