Modify MoneyRequestConfirmPage for Distance Request

@tgolen
Details
Display a map with the distance route on MoneyRequestConfirmPage.
Fixed Issues
$ https://github.com/Expensify/App/issues/22710
Tests / QA
Creating a distance request from a workspace chat
Test 1: Check Distance Request is only available in Workspace chats and Global Create

Video walkthrough
https://github.com/Expensify/App/assets/98560306/0fc1a59e-113a-4a5c-bb91-b08c8691a258

Distance Request should be only available for Workspace chats and Global Create. In non-workspace chats, the Distance Request tab should not be visible

Go to a workspace chat. Click the "+" button next to the text composer and click "Request money." Confirm you DO see the Distance Request tab.
Go to a non-workspace chat. Click the "+" button next to the text composer and click "Request money." Confirm you DO NOT see the Distance Request tab.
Click the Global Create Floating Button and click "Request money." Confirm you DO see the Distance Request tab.

Test 2: Create a distance request from a Workspace chat
🔔 Test steps 20 ~ 24 are not ready. Waiting for other PR.  If you don't have a workspace yet, create one.

Click the "+" button in the text field.

Click "Request money" on the popup

Click the "Distance" tab on the next screen that appears

Click the "Start" item in the list of waypoints

On the text field in the next screen, type 88 Kearny Street. Select the first option.

Click the "Finish" item in the list of waypoints

On the text field in the next screen, type "1 Telegraph Hill". Click the first option that appears.

Click the "Next" button

Confirm that the next screen (Money Request Confirm Page) looks like the screenshot below. Check "Amount" is $0.92.

Click "Show more" and confirm the screen looks like below.

Check you're taken back to the previous  (Distance Request) screen. Click "Next" button again.

Click "Date". Check you can change and save the date.

Click "Description". Check you can add and save the description.

At this time, we won't check if "Amount" will work or not. Click "Request $0.92 button at the bottom.

Confirm that the Right Hand Panel (where we were creating the distance request) is closed

Confirm you're still in the same workspace chat and a new distance request is created.

Click the distance request created.

Confirm that you're taken to the distance request details page. Confirm information displayed is the same information you inputted.

Go back to the workspace chat and create another distance request. Confirm that the map is reset to the initial state without any waypoints or route drawn.

Click "Next" and this time, click "Amount"

Try to click the dollar sign "$". Check that nothing happens when you click it (to confirm the currency is not editable)

Edit the amount and save. Confirm the amount is updated on the confirmation page. Click "Request $<new amount>" at the bottom to create the new distance request.

Check the newly created distance request has the correct amount to which you changed.

Test 3: Create a distance request through Global Create

Video walkthrough
https://github.com/Expensify/App/assets/98560306/28d78853-418e-4680-aa35-582e18d34bfd

Make sure you existing chats other than the test workspace.

Click the Global Create button (the floating "+" button).

Click "Request Money"

Repeat the steps from steps 4 - 9 from the first test.

Check the next screen appears looks like the screenshot below. On this screen, you can select  to who/where you make distance request. Make sure that only workspace(s) appear as options. Make sure that no personal account appears as selections (step 1 was for this purpose).

Select a workspace and confirm that the next screen that appears looks like the screenshot from step 10 of the first test (Confirm page).

Offline tests

Video walkthrough
https://github.com/Expensify/App/assets/98560306/138b1188-952b-4d2f-bd9d-f9af492d4a1e

Go to the Distance Request page
Go to the developer console and toggle the network to offline
Check that a placeholder offline component takes place of the map
Bring the network online and save two waypoints so that "Next" button can be clicked. Click the next button.
Toggle the network to offline again. Check that the placeholder component takes place of the map

QA Steps
PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message: https://expensify.slack.com/archives/C21FRDWCV/p1693419641253399?thread_ts=1693415114.500789&cid=C21FRDWCV

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] If we are not using the full Onyx data that we loaded, I've added the proper selector in order to ensure the component only re-renders when the data it is using changes
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
Please refer to the screen recordings for Tests #1 #2, and #3 above

Mobile Web - Chrome
https://github.com/Expensify/App/assets/98560306/ef2c2ad7-0e1e-4916-bc61-3c7ded2b1930

Mobile Web - Safari
https://github.com/Expensify/App/assets/98560306/452284e3-3e1f-42ff-b404-19079a8a3bf7

Desktop
https://github.com/Expensify/App/assets/98560306/38320ff3-3160-41a4-9e90-a532e04456fc

iOS
https://github.com/Expensify/App/assets/98560306/78061d17-a439-458d-bfec-833889f352e4

Android
My development environment is not working for Android

UPDATE: was able to display the map on the confirmation page.

This afternoon, I gonna work on:

[ ] Calculate the request amount
[ ] Only render the map when the money request is a distance request
[ ] Call CreateDistanceRequest on save

UPDATE:
I was able to grab transactions and display waypoint markers on the map. I gonna commit and push the code

I'm going to take dinner break now but I gonna work on the following when I come back:

Passing policyID down to the MoneyRequestConfirmationList
Subscribing to the policy with withOnyx
Getting the custom unit rate from the policy with a selector

Update:
Working on this part of the design doc where I need to extract the mileage rate from the policy associated with the report (where the money/distance request is create) and the distance itself from the transaction.
We need these values to calculate the expense amount to be displayed on the confirmation page.

UPDATE:

was able to display the distance route
was able to calculate the distance amount ($0.05 in the following example)
was able to create a new item (Distance) in the list

TODO

[ ] Create DistanceRequestUtils.js and move all util functions there
[ ] Reread the section in design doc and check if all requirements are met
[ ] Make the PR read for review

@rushatgabhane
I couldn't reproduce the bug you found here: https://github.com/Expensify/App/pull/25707#pullrequestreview-1594798796
could you check again and pinpoint the exact steps that lead to the crash?

Known issues

When clicking Amount, Description, Date and Category, the screen is navigated back to the confirmation immediately.
The map in the Distance Request page won't be displayed property when being navigated back from the confirmation page

Do I need to call the backend API to actually save the distance request? What method should I call? What kind of parameters do I have to pass?

@hayata-suenaga Yes please, since this is the last PR in the flow leading up to creating the distance request. We already build the action function createDistanceRequest so you just need to call that. I think we forgot to serialize some of the JSON objects when calling the API, and the params that Web-E expects might have changed.

@neil-marcellini I think I need help

@rushatgabhane
I couldn't reproduce the bug you found here: #25707 (review)
could you check again and pinpoint the exact steps that lead to the crash?

@rushatgabhane could you kindly check if the bug is still present or not? 😄

Memo
Hide distance request tab when the chat is not workspace

Issues I need to fix

[ ] Clicking on the Distance option on the confirmation page closes the Right Hand Navigator (RHN)
[ ] Clicking on the "Save" option doesn't close (RHN)

@rushatgabhane this PR is ready for review
We probably won't handle test steps 20 ~ 24 of Test #1 in this PR.

I uploaded screen recordings. I realized some navigation issues on smaller screens (mobile web and mobile native).
When I click "Save" -> the distance request screen -> list of chats
When it supports to be: "Save" -> the original workspace chat where the distance request creation flow started
I remember a navigation issue was addressed recently. @rushatgabhane could you check if this is a regression caused by this PR or if this issue is present on production / the main branch?

@rushatgabhane how is Android build going 😄 ?

@rushatgabhane
being build here

Use the links below to test this build in android and iOS. Happy testing!

To solve the issues below, I pushed this commit and this commit.

I merged the main into my branch, and I realized this new issue at step 6 of the Test 3
https://github.com/Expensify/App/assets/98560306/4dde16c8-c1fb-44e8-b30e-1408f82d34c0

There was apparently a syntax error 😭 gonna try the build again

Waiting for translation for the following here (internal link)
mile: 'milla',
miles: 'millas',
kilometer: 'kilómetro',
kilometers: 'kilómetros',

@rushatgabhane this one is ready for review again 🙇

@rushatgabhane new builds going on here

that will be covered in another PR but nice catch nonetheless 👍

on smaller screens, this navigation issue still persists
https://github.com/Expensify/App/assets/98560306/fa0878f7-ddd5-4562-8a7d-b0353a0ae10e

Or.. we could use the pattern used in Reimbursement page for connecting a bank account.
@rushatgabhane I love the idea! I'm checking with the internal team about this.

By the way, do you have idea why we're having this issue?

The amount is shown as INR 0.92. @hayata-suenaga I think we should restrict making a distance requests from workspaces that have currecy set as USD only. What do you think?

We actually don't want to restrict the feature to USD only. You can actually change the rate to any amount
https://github.com/Expensify/App/assets/98560306/7bf0e8a1-a047-4233-8940-6a5fdbef072f

:test_tube::test_tube: Use the links below to test this build in android and iOS. Happy testing! :test_tube::test_tube:

android :robot:
iOS :apple:

https://ad-hoc-expensify-cash.s3.amazonaws.com/android/25707/index.html
https://ad-hoc-expensify-cash.s3.amazonaws.com/ios/25707/index.html

desktop :computer:
web :spider_web:

❌ FAILED ❌
https://25707.pr-testing.expensify.com

The QR code can't be generated, because the Desktop build failed

@rushatgabhane test build is here: https://github.com/Expensify/App/pull/25707#issuecomment-1699841674

:test_tube::test_tube: Use the links below to test this build in android and iOS. Happy testing! :test_tube::test_tube:

android :robot:
iOS :apple:

https://ad-hoc-expensify-cash.s3.amazonaws.com/android/25707/index.html
https://ad-hoc-expensify-cash.s3.amazonaws.com/ios/25707/index.html

desktop :computer:
web :spider_web:

https://ad-hoc-expensify-cash.s3.amazonaws.com/desktop/25707/NewExpensify.dmg
https://25707.pr-testing.expensify.com

Thanks for the review @rushatgabhane! We'll get started looking into all of these except the map not loading on web. Please see if you can reproduce on main or staging and then report it as a separate issue. It feels unrelated.

solving the conflicts now 😓

🆕 Bug
When creating a distance request from Global Create, on the Money Request Participants page, you cannot type anything int the Text Input.
https://github.com/Expensify/App/assets/98560306/b3720522-aae0-4da9-a223-6f25bb36543e
I'm working on ⬆

🆕 Bug
When the height of the screen shrinks, the button to move Next is not displayed.
https://github.com/Expensify/App/assets/98560306/5981b8a7-8ee9-4d11-a947-1271e882d765

I shouldn't review since I authored some commits.

building app for easy testing in case needed: https://github.com/Expensify/App/actions/runs/6043113007\

:test_tube::test_tube: Use the links below to test this build in android and iOS. Happy testing! :test_tube::test_tube:

android :robot:
iOS :apple:

https://ad-hoc-expensify-cash.s3.amazonaws.com/android/25707/index.html
https://ad-hoc-expensify-cash.s3.amazonaws.com/ios/25707/index.html

desktop :computer:
web :spider_web:

https://ad-hoc-expensify-cash.s3.amazonaws.com/desktop/25707/NewExpensify.dmg
https://25707.pr-testing.expensify.com

reviewing based on @hayata-suenaga's request

please pull main. gonna perform Test2 & Test3 combined with https://github.com/Expensify/App/pull/25990 which was merged 3 hrs ago

@situchan thank you for the thorough offline tests. Let's address these bugs you found in follow up PRs 👍

I gonna address Neil's review now

BUG2: amount is 0. The issue is that it keeps 0 even after online

Ah yeah good catch. As per the design doc we should show TBD when the route hasn't been fetched yet and the amount can't be calculated. We also need to fetch the route when we go back online on this page.

BUG3: infinite loading on preview
https://github.com/Expensify/App/assets/108292595/f7ba7e96-5c04-428b-bbb8-68e94e1fed16

BUG4: optimistic data is not correct (not sure if known bug already)

https://github.com/Expensify/App/assets/108292595/fe4f6f77-fbf7-4b2d-a860-e17477f65a6d
BUG5: display name is duplicated. I think one should be workspace name

:test_tube::test_tube: Use the links below to test this build in android and iOS. Happy testing! :test_tube::test_tube:

android :robot:
iOS :apple:

https://ad-hoc-expensify-cash.s3.amazonaws.com/android/25707/index.html
https://ad-hoc-expensify-cash.s3.amazonaws.com/ios/25707/index.html

desktop :computer:
web :spider_web:

https://ad-hoc-expensify-cash.s3.amazonaws.com/desktop/25707/NewExpensify.dmg
https://25707.pr-testing.expensify.com

@situchan let's fix all those in the follow ups because they aren't directly related to the confirmation. With that in mind, can you complete the checklist?

@neil-marcellini I created an issue that has all the cleanup points you identified in your PR review. We might need to breakup the issue into several, but for now, this is where I'm tracking all the cleanup we have to do.

@situchan @rushatgabhane PR ready for (hopefully) the final round of review 🤞

👋 How are we doing with the checklist?

👋 How are we doing with the checklist?

in 2-3 hrs. still testing

So I tested with fresh accounts and 2 critical bugs still happening which should be blocker:

https://github.com/Expensify/App/pull/25707#issuecomment-1702140383
https://github.com/Expensify/App/pull/25707#issuecomment-1702143728

@situchan @rushatgabhane
as @neil-marcellini mentioned here, we're addressing https://github.com/Expensify/App/pull/25707#issuecomment-1702140383
https://github.com/Expensify/App/pull/25707#issuecomment-1702143728 in a separate issue 😓
could you go through the checklist as soon as possible?

Image infinite loading issue doesn't happen on QR build so looks much better

Image infinite loading issue doesn't happen on QR build so looks much better

@situchan ah that's nice

Let's tackle this as well in follow-up if caused by this PR (I didn't confirm on main)
"Pay elsewhere" button not working

BUG6: console error

https://github.com/Expensify/App/assets/108292595/8c18d658-534c-4685-afb2-6e067f19443d
BUG7: regression - #24325
https://github.com/Expensify/App/assets/108292595/d565d668-b1af-4072-b56f-13edd37fcb28

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[x] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
(QR build)
https://github.com/Expensify/App/assets/108292595/6c0b3d8e-0277-4123-baff-bd3250502984

Mobile Web - Chrome
(dev build)
https://github.com/Expensify/App/assets/108292595/c2c530dc-72d3-4ab0-aed7-beb771ae6c2a

Mobile Web - Safari
(QR build)
https://github.com/Expensify/App/assets/108292595/13ce08f3-80eb-4d31-a870-5d4e59d206f3

Desktop
(dev build)
https://github.com/Expensify/App/assets/108292595/7d1e2140-1b4f-442f-b587-8390ed7be42e

iOS
(dev build)
https://github.com/Expensify/App/assets/108292595/4c30a69e-3f40-4612-8701-b29316e53d39

Android
(dev build)
https://github.com/Expensify/App/assets/108292595/8c18d658-534c-4685-afb2-6e067f19443d

Reviewer checklist failing because of https://github.com/Expensify/App/pull/25707#issuecomment-1699783574

Reviewer checklist failing because of https://github.com/Expensify/App/pull/25707#issuecomment-1699783574

deleted that checklist because @situchan completed another one

We won't be able to merge this until @tgolen reviews again, right?

@Li357 @tgolen is OOO today and we wanna merge this by 3pm today
But this one is a big PR so I'll ask @neil-marcellini to review again 👍

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

I couldn't reproduce this bug on desktop pointed to Staging API. Might be from the optimistic data?

Here is the cleanup issue for this PR

Awesome work on this one! Wow that was tricky.

🚀 Deployed to staging by https://github.com/neil-marcellini in version: 1.3.62-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

Ok I went through almost all the bugs reported here and report them in Slack and created GH issues for them. If I missed any I'm sure we'll see them reported again eventually.

🚀 Deployed to production by https://github.com/mountiny in version: 1.3.62-4 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

I'm trying to figure out this props.iouAmount === 0 -> setMoneyRequestAmount -> props.iouAmount !== 0 cycle to review the proposals properly.

@cubuspl42  could you link the code please 🙇


⚠ AGEG website (prod) has degraded performance

In 66e03ee, AGEG website (prod) (https://www.ageg.ca/) experienced degraded performance:

HTTP code: 200
Response time: 2343 ms

Resolved: AGEG website (prod) performance has improved in 25f31b6.


Install and configure ceph

Not sure if this is part of juno release but is it possible to use ceph for Cinder and Glance backend storage.

Hi
yes it is possilbe, but the support for ceph has not been added yet.
Have a look at https://github.com/mmailand/puppetlabs-openstack/tree/topic-ceph_v3
I am working on it.


unexpected behaviour for toggle "auto hide menu bar"

Currently, "auto hide menu bar" only works for Window Mode. However, since MacOS Monterey, users are allowed to keep the menu bar visible when in Full-Screen Mode. It is quite useful for most situations to have a glance at the time when in Full-Screen Mode. But some apps' windows will just conflict with the menu bar, like Parallels Desktop. But "auto hide menu bar" in OnlySwitch does not work as expected when in Full-Screen Mode.
To Reproduce
Steps to reproduce the behavior:

Make your app enter Full-Screen Mode
Click on 'auto hide menu bar' to enable hiding menu bar

Expected behavior
the menu bar should be hidden

OS: MacOS
Version: Monterey
CPU: m1

Thank you for your feedback. Since autohide menu bar in full screen and on desktop are two different toggles in System Preferences, I think a new switch for this feature is proper. Unfortunately, I haven't find an apple script way as auto hide menu bar on desktop to implement it. I know I can use UI script, but preferences UI maybe change when macOS upgrading, it isn't an elegant approach for this. If you know some better ways, please tell me.

@alvis233 Perhaps I can build a special version via UI scripting for you. If you feel it good, I'll release it but mark the feature as beta. When I get the better way, or in the Ventura provides an original script for it, I'll replace the method.  BTW, I create a telegram group, feel free to join it: https://t.me/OnlySwitchforMac

@alvis233 How about use shortcuts action? Like this it can be easily update following macOS version upgrading. You can try this shortcut action I create: https://www.icloud.com/shortcuts/29341dfe669e4c898fd78faa6549dcbb
If you feel it good, I'll contribute it on OnlySwitch shortcuts gallery.


Baseline violations from baseline file saved in a subfolder are NOT ignored in the PHPMD report

PHPMD version: 2.10.1
PHP Version: 7.3
Installation type:composer
Operating System / Distribution & Version: CentOS 7.6

Current Behavior
The baseline violations are not ignored in the final report
Expected Behavior
The baseline violations are ignored in the final report
Steps To Reproduce:

Save the ruleset.xml file within a subfolder of the project root to not pollute it:
/home/username/src/project/build/phpmd
Generate the baseline file and save it in the same subfolder
/home/username/src/project/build/phpmd

<?xml version="1.0"?>
<phpmd-baseline>
  <violation rule="PHPMD\Rule\Design\CouplingBetweenObjects" file="/home/username/src/project/src/ClassName.php"/>
</phpmd-baseline>

Check the generated file, note that its violations contain the absolute file name paths
/home/username/src/project/src/ClassName.php
Run the check with the baseline
The check report will contain all the violations that are set to be ignored in the baseline file
The reason is the bug in the logic that reads the baseline file - \PHPMD\Baseline\BaselineSetFactory::fromFile
The absolute file name paths are prepended with a basepath calculated as $basePath    = dirname($fileName); resulting in (the part in bold is the basepath)
/home/username/src/project/build/phpmd/home/username/src/project/src.ClassName.php

Checks before submitting

[x] Be sure that there isn't already an issue about this. See: Issues list
[x] Be sure that there isn't already a pull request about this. See: Pull requests
[x] I have added every step to reproduce the bug.
[x] If possible I added relevant code examples.
[x] This issue is about 1 bug and nothing more.
[x] The issue has a descriptive title. For example:  "JSON rendering failed on Windows for filenames with space".

Thanks for the report, was struggling a bit with the basepath implementation. It shouldve been possible to move the phpmd.baseline.xml file. Added PR :)


prog: add support for riscv64

This will allow to run make TARGETVMARCH=riscv64.
h/t Munich RISC-V meetup talk by @bjoto

That was fast.

FWIW s390 port is just being added, so if you do $ git log --format="%h %ae %s" | grep Alexander.Egorenkov, this will give some approximation of what needs to be done a  new arch.

@dvyukov Awesome, thanks a lot.


How should xarray use/support sparse arrays?

I'm looking forward to being easily able to create sparse xarray objects from pandas: https://github.com/pydata/xarray/issues/3206
Are there other xarray APIs that could make good use of sparse arrays, or could make sparse arrays easier to use?
Some ideas:

to_sparse()/to_dense() methods for converting to/from sparse without requiring using  .data
to_dataframe()/to_series() could grow options for skipping the fill-value in sparse arrays, so they can round-trip MultiIndex data back to pandas
Serialization to/from netCDF files, using some custom convention (see https://github.com/pydata/xarray/issues/1375#issuecomment-402699810)

This is very exciting! In energy-economic research (unlike, e.g., earth systems research), data are almost always sparse, so first-class sparse support will be broadly useful.
I'm leaving a comment here (since this seems to be a meta-issue; please link from wherever else, if needed) with two example use-cases. For the moment, #3206 seems to cover them, so I can't name any specific additional features.

MESSAGEix is an energy systems optimization model framework, formulated as a linear program.

Some variables have many dimensions, for instance, the input coefficient for a technology has the dimensions (node_loc, technology, year_vintage, year_active, mode, node_origin, commodity, level, time, time_origin).

In the global version of our model, the technology dimension has over 400 labels.
Often two or more dimensions are tied, eg technology='coal power plant' will only take input from (commodity='coal', level='primary energy'); all other combinations of (commodity, level) are empty for this technology.
So, this data is inherently sparse.

For modeling research, specifying quantities in this way is a good design because (a) it is intuitive to researchers in this domain, and (b) the optimization model is solved using various LP solvers via GAMS, which automatically prune zero rows in the resulting matrices.
When we were developing a dask/DAG-based system for model results post-processing, we wanted to use xarray, but had some quantities with tens of millions of elements that were less than 1% full. Here is some test code that triggered MemoryErrors using xarray. We chose to fall back on using a pd.Series subclass that mocks xarray methods.

In transportation research, stock models of vehicle fleets are often used.

These models always have at least two time dimensions: cohort (the time period in which a vehicle was sold) and period(s) in which it is used (and thus consumes fuel, etc.).
Since a vehicle sold in 2020 can't be used in 2015, these data are always triangular w.r.t. these two dimensions. (The dimensions year_vintage and year_active in example #1 above have the same relationship.)
Once multiplied by other dimensions (technology; fuel; size or shape or market segment; embodied materials; different variables; model runs across various scenarios or input assumptions) the overhead of dense arrays can become problematic.

+1 for the introduction of to_sparse() / to_dense(), but let's please avoid the mistakes that were done with chunk(). DataArray.chunk() is extremely frustrating when you have non-index coords, and 9 times out of 10 you only want to chunk the data and you have to go through the horrid
a = DataArray(a.data.chunk(), dims=a.dims, coords=a.coords, attrs=a.attrs, name=a.name)

Exactly the same issue would apply to to_sparse().
Possibly we could define them as
class DataArray:
    def to_sparse(
        self,
        data: bool = True,
        coords: Union[Iterable[Hashable], bool] = False
    )

class Dataset:
    def to_sparse(
        self,
        data_vars: Union[Iterable[Hashable], bool] = True,
        coords: Union[Iterable[Hashable], bool] = False
    )

same for to_dense() and chunk() (the latter would require a DeprecationWarning for a few release before switching the default for coords from True to False - only to be triggered in presence of dask-backed coords).
e.g. ``a.to_sparse("data")``

As already mentioned in #3206, unstack(sparse=True) would be extremely useful.

As for NetCDF, instead of a bespoke xarray-only convention, wouldn't it be much better to push a spec extension upstream?

netCDF has a pretty low-level base spec, with conventions left to higher level docs like CF conventions. Fortunately, there does seems to be a CF convention that would be a  good fit for for sparse data in COO format, namely the indexed ragged array representation (example, note the instance_dimension attribute). That's probably the right thing to use for sparse arrays in xarray.

Would it be feasible to use the contiguous ragged array spec or the gathering based compression when the COO coordinates are sorted? I think this could be very helpful for read efficiency, though I'm not sure if random writes were a requirement here.

I like the indexed ragged array representation because it maps directly into sparse’s COO format. I’m sure other formats would be possible, but they would also likely be harder to implement.

That's fair. I just think it would be useful to have an assurance that indices are sorted you read them. I don't see how to express this within the CF specs while still looking like a COO array though.

Yes, it would be useful (eventually) to have lazy loading of sparse arrays from disk, like we want we currently do for dense arrays. This would indeed require knowing that the indices are sorted.

I would be interested in trying to add PDAL (https://pdal.io/) support, is this something of interest in a similar way to the support for rasterio for dense arrays?

Tagging @jeliashi for visibility/collaboration

Would it be possible that pd.{Series, DataFrame}.to_xarray() automatically creates a sparse dataarray - or we have a flag in to_xarray which allows controlling for this. I have a very sparse dataframe and everytime I try to convert it to xarray I blow out my memory. Keeping it sparse but logically as a DataArray would be fantastic.

We have a new "sparse=True" option in xarray.Dataset.from_dataframe for
exactly this use case. Pandas's to_xarray() method just calls this method,
so it would make sense to forward keyword arguments, too.
On Fri, Aug 30, 2019 at 11:53 AM firdaus janoos<EMAIL_ADDRESS>wrote:

Would it be possible that pd.{Series, DataFrame}.to_xarray() automatically
creates a sparse dataarray - or we have a flag in to_xarray which allows
controlling for this. I have a very sparse dataframe and everytime I try to
convert it to xarray I blow out my memory. Keeping it sparse but logically
as a DataArray would be fantastic
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/3213?email_source=notifications&email_token=AAJJFVTD2IWWPE6RTSWPVLDQHFUDTA5CNFSM4ILGYGP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5SPPNI#issuecomment-526710709,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJJFVWEOGEBGXV62QFYU6DQHFUDTANCNFSM4ILGYGPQ
.

I cloned the master branch and installed it using 'python setup.py develop'.
When I try to use the sparse data loading functionality as per
oo = xa.Dataset.from_dataframe( my_df, sparse=True )

I get the following error:
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-9-fce0ca6bc4c2> in <module>
----> 1 oo = xa.Dataset.from_dataframe( poly_df.iloc[:10000], sparse=True )

/mnt/local/xarray/xarray/core/dataset.py in from_dataframe(cls, dataframe, sparse)
   4040
   4041         if sparse:
-> 4042             obj._set_sparse_data_from_dataframe(dataframe, dims, shape)
   4043         else:
   4044             obj._set_numpy_data_from_dataframe(dataframe, dims, shape)

/mnt/local/xarray/xarray/core/dataset.py in _set_sparse_data_from_dataframe(self, dataframe, dims, shape)
   3936         self, dataframe: pd.DataFrame, dims: tuple, shape: Tuple[int, ...]
   3937     ) -> None:
-> 3938         from sparse import COO
   3939
   3940         idx = dataframe.index

ModuleNotFoundError: No module named 'sparse'

Any suggestions on what I need to do ?

conda install -c conda-forge sparse
Basically you need to install https://sparse.pydata.org/en/latest/ using either pip or conda.

Thanks.
That solved that error but introduced another one.
Specifically - this is my dataframe

and this is the error that I get with sparse=True

My numpy version is definitely about 1.16

I also set this
os.environ["NUMPY_EXPERIMENTAL_ARRAY_FUNCTION"]='1'
just in case
Furthermore, I don't get this error when I don't set sparse=True ( I just get OOM errors but that's another matter) ...

You will need to install NumPy 1.17 or set the env variable before
importing NumPy.
On Fri, Aug 30, 2019 at 1:57 PM firdaus janoos<EMAIL_ADDRESS>wrote:

Thanks.
That solved that error but introduced another one.
Specifically - this is my dataframe
[image: image]
https://user-images.githubusercontent.com/923438/64050831-2d061280-cb47-11e9-915b-01fe42eadefe.png
and this is the error that I get with sparse=True
[image: image]
https://user-images.githubusercontent.com/923438/64049668-91bf6e00-cb43-11e9-921f-1a044f3446a9.png
[image: image]
https://user-images.githubusercontent.com/923438/64050631-a94c2600-cb46-11e9-8653-9820b445bc86.png
My numpy version is definitely about 1.16
[image: image]
https://user-images.githubusercontent.com/923438/64050648-b701ab80-cb46-11e9-8dac-aaf2bf9e260d.png
I also set this
os.environ["NUMPY_EXPERIMENTAL_ARRAY_FUNCTION"]='1'
just in case
Furthermore, I don't get this error when I don't set sparse=True ( I just
get OOM errors but that's another matter) ...
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/3213?email_source=notifications&email_token=AAJJFVTN37AMEA6ROS7YT2LQHGCVHA5CNFSM4ILGYGP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5SYQ6Q#issuecomment-526747770,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJJFVWM2V4H3V6BJMJ7IQDQHGCVHANCNFSM4ILGYGPQ
.

I would like to add a request for sparse xarrays:  Support ffill and bfill operations along ordered dimensions (such as datetime coordinates) while maintaining the sparse level of data density.
The challenge to overcome is that performing ffill operations on sparse data quickly creates data that is no longer "sparse" in practice and makes dealing with the data challenging.
My suggested implementation (and the way I have previously done this in another programming environment) is to represent the data as rows of contiguous regions with a single (non-sparse) value rather than rows of single points.  The contiguous dimensions could be defined as any dimensions that are "ordered" such as datetime coordinates.  That is, the data then is represented as a list of values + coordinate ranges rather than a list of values + coordinates.
The idea is that you can easily compute operations like ffill without changing the sparsity of the matrix, and thus support typical aggregating functions you might like to apply to the data before you collapse the data and convert to a non-sparse form  (e.g.  perform a lag difference of the most recent value with the most recent value 20 days ago, or do a cross-sectional mean on the data along a certain dimension, using the most recent data at each given point in time).  These types of operations can be more useful when the data is "fuller" such as after a forward fill, but often not useful when the data is very sparsely populated (as the cross-sectional operations are unlikely to hit the sparse data among the different dimensions).
Care must be taken to avoid "collisions" between sparse blocks of data, that is, avoiding that the list of sparse blocks accidentally overlap.  The implementation can get tricky but I believe the goal to be worthwhile.
I am happy to expand on the request if the idea is not well expressed.

@p-d-moore what you say makes sense but it is well outside of the domain of xarray. What you're describing is basically a new sparse class, substantially more sophisticated than COO, and should be proposed in the sparse board, not here. After it's implemented in sparse, xarray will be able to wrap around it.

Thanks @crusaderky, appreciated.  Might as as well suggest it there.

So how would one change an existing dataset or dataarray to using a sparse representation?
something like
xr.Dataset.from_dataframe(data_set.to_dataframe(),sparse=True)?
seems a little silly to have to convert it to a pandas dataframe and back

Thanks for rolling out support for sparse arrays!
I think it would be great to have a sparse argument in the concat function. This will be useful with join='outer', as the resultant DataArray/Dataset can become quite sparse.
As an example, I have a use case where I want to concatenate (across a new dimension) a number of DataArrays with date indexes covering different date ranges. When I use concat the memory used by the resultant array is much greater than that used by the original list of DataArrays due to the large number of NA values that appear in it.

@oliverhiggs Ive also noticed a huge computational overhead when joining xarray datasets where the result would be sparse. Something like a minute of computation time to join two 10GB datasets, even when there are no overlapping indices. I'm not sure if a sparse representation would help but its possible we'd get a reduced memory footprint and a faster merge/concat time with this kind of support.

@El-minadero a lot of that overhead may be fixed on master and more recent xarray versions.  https://xarray.pydata.org/en/stable/io.html#reading-multi-file-datasets has some tips on quickly concatenating / merging datasets. It depends on the datasets you are joining...

@dcherian These examples seem focused on merging from disk, whereas the use-case I'm running into is joining data produced by computation in ram. I'll try updating my xarray installation and see where that gets me.

the coords, data_vars, join ,compat kwargs in that example are passed down to concat and merge, as appropriate. We do need more documentation on that ....

Thank you all for making xarray and its tight development with dask so great!
As @shoyer mentioned

Yes, it would be useful (eventually) to have lazy loading of sparse arrays from disk, like we want we currently do for dense arrays. This would indeed require knowing that the indices are sorted.

I am wondering, if creating a lazy & sparse xarray Dataset/DataArray is already possible?
Especially when creating the sparse part at runtime, and loading only the data part:
Assume two differently sampled - and lazy dask - DataArrays are merged/combined along a coordinate axis into a Dataset.
Then the smaller (= less dense) DataVariable is filled with NaNs. As far as I experienced the current behaviour is, that each NaN value requires memory.
That issue might be formulated this way:
Dask integration enables xarray to scale to big data, only as long as the data has no sparse character. Do you agree on that formulation or am I missing something fundamental?
A code example reproducing that issue is described here: https://stackoverflow.com/q/60117268/9657367

Hi fmfreeze,
> Dask integration enables xarray to scale to big data, only as long as
the data has no sparse character. Do you agree on that formulation or am I
missing something fundamental?
I don't agree. To my understanding xarray->dask->sparse works very well
(save bugs), as long as your data density (the percentage of non-default
points) is roughly constant across dask chunks.
If it isn't, then you'll have some chunks that consume substantially more
RAM and CPU to compute than others. This can be mitigated, if you know in
advance where you are going to have more samples, by setting uneven dask
chunk sizes. For example, if you have a one-dimensional array of 100k
points and you know in advance that the density of non-default samples
follows a gaussian or triangular distribution, then it may be wise to have
very large chunks at the tails and then get them progressively smaller
towards the center, e.g. (30k, 12k, 5k, 2k, 1k, 1k, 2k, 5k, 10k, 30k).
Of course, there are use cases where you're going to have unpredictable
hotspots; I'm afraid that in those the only thing you can do is size your
chunks for the worst case and end up oversplitting everywhere else.
Regards
Guido
On Thu, 13 Feb 2020 at 10:55, fmfreeze<EMAIL_ADDRESS>wrote:

Thank you all for making xarray and its tight development with dask so
great!
As @shoyer https://github.com/shoyer mentioned
Yes, it would be useful (eventually) to have lazy loading of sparse arrays
from disk, like we want we currently do for dense arrays. This would indeed
require knowing that the indices are sorted.
I am wondering, if creating a lazy & sparse xarray Dataset/DataArray
is already possible?
Especially when creating the sparse part at runtime, and loading only
the data part:
Assume two differently sampled - and lazy dask - DataArrays are
merged/combined along a coordinate axis into a Dataset.
Then the smaller (= less dense) DataVariable is filled with NaNs. As far
as I experienced the current behaviour is, that each NaN value requires
memory.
That issue might be formulated this way:
Dask integration enables xarray to scale to big data, only as long as the
data has no sparse character. Do you agree on that formulation or am I
missing something fundamental?
A code example reproducing that issue is described here:
https://stackoverflow.com/q/60117268/9657367
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/3213?email_source=notifications&email_token=ABPM4MBFBIH7EK4PPAWHRH3RCURJLA5CNFSM4ILGYGP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELUJNRQ#issuecomment-585668294,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPM4MCMRVUZXQSDYCAIP3LRCURJLANCNFSM4ILGYGPQ
.

Thank you @crusaderky for your input.
I understand and agree with your statements for sparse data files.
My approach is different, because within my (hdf5) data files on disc, I have no sparse datasets at all.
But as I combine two differently sampled xarray dataset (initialized by h5py > dask > xarray) with xarrays built-in top-level function "xarray.merge()" (resp. xarray.combine_by_coords()), the resulting dataset is sparse.
Generally that is nice behaviour, because two differently sampled datasets get aligned along a coordinate/dimension, and the gaps are filled by NaNs.
Nevertheless, thos NaN "gaps" seem to need memory for every single NaN. That is what should be avoided.
Maybe by implementing a redundant pointer to the same memory adress for each NaN?

you just need to

load up your NetCDF files with xarray.open_mfdataset. This will give
you

an xarray.Dataset,
that wraps around one dask.array.Array per variable,
that wrap around one numpy.ndarray (DENSE array) per dask chunk.

convert to sparse with xarray.apply_ufunc(sparse.COO, ds).
This will give you

an xarray.Dataset,
that wraps around one dask.array.Array per variable,
that wrap around one sparse.COO (SPARSE array) per dask chunk.

use xarray.merge or whatever to align and merge

you may want to rechunk at this point to obtain less, larger chunks. You
can estimate your chunk size in bytes if you know your data density (read
my previous email).

Do whatever other calculations you want. All operations will produce in
output the same data type as point 2.

To go back to dense, invoke xarray.apply_ufunc(lambda x: x.todense(),
ds) to go back to the format as in (1). This step is only necessary if you
have something that won't accept/recognize sparse arrays directly in input;
namely, writing to a NetCDF dataset. If your data has not been reduced
enough, you may need to rechunk into smaller chunks first in order to fit
into your RAM constraints.

Regards
On Tue, 18 Feb 2020 at 13:56, fmfreeze<EMAIL_ADDRESS>wrote:

Thank you @crusaderky https://github.com/crusaderky for your input.
I understand and agree with your statements for sparse data files.
My approach is different, because within my (hdf5) data files on disc, I
have no sparse datasets at all.
But as I combine two differently sampled xarray dataset (initialized by
h5py > dask > xarray) with xarrays built-in top-level function
"xarray.merge()" (resp. xarray.combine_by_coords()), the resulting dataset
is sparse.
Generally that is nice behaviour, because two differently sampled datasets
get aligned along a coordinate/dimension, and the gaps are filled by NaNs.
Nevertheless, thos NaN "gaps" seem to need memory for every single NaN.
That is what should be avoided.
Maybe by implementing a redundant pointer to the same memory adress for
each NaN?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/3213?email_source=notifications&email_token=ABPM4MFWF22BFFYDHV6BS2DRDPSHXA5CNFSM4ILGYGP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMCBWHQ#issuecomment-587471646,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPM4MHIUWDYX6ZFKRRBIJLRDPSHXANCNFSM4ILGYGPQ
.

Thank you @crusaderky, unfortunately some obstacles appeared using your loading technique.
As thousands of .h5 files are the datasource for my use case and they have various - and sometimes different paths to - datasets, using the xarray.open_mfdatasets(...) function seems not to be possible straight forward.
But:

I have a routine merging all .h5 datasets into corresponding dask arrays, wrapping dense numpy arrays implicitly

I "manually" slice out a part of the the huge lazy dask array and wrap that into an xarray.DataArray/Dataset

But applying xr.apply_ufunc(sparse.COO, ds, dask='allowed') on that slice then results in an NotImplementedError: Format not supported for conversion. Supplied type is <class 'dask.array.core.Array'>, see help(sparse.as_coo) for supported formats.

(I am not sure, if this is the right place to discuss, so I would be thankful for a response on SO in that case: https://stackoverflow.com/questions/60117268/how-to-make-use-of-xarrays-sparse-functionality-when-combining-differently-size)

xr.apply_ufunc(sparse.COO, ds, dask='parallelized')

Concatenating multiple lazy, differently sized xr.DataArrays - each wrapping a sparse.COO by xr.apply_ufunc(sparse.COO, ds, dask='parallelized') as @crusaderky suggested - results again in an xr.DataArray, whose wrapped dask array chunks are mapped to numpy arrays:
<xarray.DataArray 'myDataset' (cycle: 10, time: 8000000)>
dask.array<concatenate, shape=(10, 8000000), dtype=float64, chunksize=(1, 5273216), chunktype=numpy.ndarray>
Coordinates:
  * time      (time) float64 0.0 5e-07 1e-06 1.5e-06 2e-06 ... 4.0 4.0 4.0 4.0
  * cycle     (cycle) int64 1 2 3 4 5 6 7 8 9 10

But also when mapping the resulting, concatenated DataArray to sparse.COO afterwards, my main goal - scalable serialization of a lazy xarray - cannot be achieved.
So one suggestion to @shoyer original question:
It would be great, if sparse, but still lazy DataArrays/Datasets could be serialized without the data-overhead itself.
Currently, that seems to work only for DataArrays which are merged/aligned by DataArrays of the same shape.

Small comment from #3981: sklearn has just started running benchmarks, but it looks like pydata/sparse is not feature complete enough for us to use. We might be interested in having scipy.sparse support in xarray.
There are two problems with scipy.sparse for us as far as I can see (this is very preliminary): it only has COO, which is not good for us, and ideally we'd want to  avoid memory copies whenever we want to use xarray, and I think going from scipy.sparse to pydata/sparse will involve memory copies, even if pydata/sparse adds other formats.

Wrapping scipy.sparse in xarray would present two challenges:

It only supports 2D arrays, which feels awkward for a library focused on N-dimensional data.
There is no existing "duck array" compatibility layer (i.e., __array_function__) that makes scipy.sparse matrices work like NumPy arrays (in fact, they actually are designed to mimic the deprecated np.matrix).

(2) is the biggest challenge. I don't want to maintain that compatibility layer inside xarray, but if it existed we would be happy to try using it.
pydata/sparse solves both these problems, though again indeed it only has quite limited data structures.

@shoyer thanks! Mostly spitballing here, but it's interesting to know that 2) would be the bigger problem in your opinion, I had assumed 1) would be the main issue. That raises the question whether it's easier to wrap scipy.sparse in a duck array, or to make pydata/sparse a viable solution for sklearn.

@amueller have you all connected with @hameerabbasi ?  I'm not surprised to hear that there are performance issues with pydata/sparse relative to scipy.sparse, but Hameer has historically been pretty open to working to resolve issues quickly.  I'm not sure if there is already an ongoing conversation between the two groups, but I'd recommend replacing "we've chosen not to use pydata/sparse because it isn't feature complete enough for us" with "in order for us to use pydata/sparse we would need the following features".

Hi. Yes, it’d be nice if we had a meta issue I could then open separate issues for for sllearn implementations.
Performance is not ideal, and I realise that. However I’m working on a more generic solution to performance as I type.

I am not familiar with the details of the various applications people in this discussion have, but here is an approach we are taking, trying to solve variations of the problem "data scattered in multi-dimensional space" or irregular time-series data. See https://scipp.github.io/user-guide/unaligned-data.html for an illustrated description.
The basic idea is to keep data in a linear representation and wrap it in a "realigned" wrapper. One reason for this developemnt was to provide a pathway to use dask with our type of data (independent time series at a large number of points in space, with chunking along the "time-series", which is not a dimension since every time series has a different length). With the linked approach we could use dask to distribute the linear underlying representation, keeping the lightweight realigned wrapper on all workers. We are still in early experimentation with this (the dask part is not actually in development yet). It probably has performance issues if more than "millions" of points are realigned --- our case is millions of time series with thousands/millions of time points in each, but the two do not mix (not both are realigned, and if they are it is independently), so we do not run into the performance issue in most cases.
In principle I could imagine this non-destructive realignment approach could be mapped to xarray, so it may be of interest to people here.

Thanks for looking into sparse arrays for xarray. I have a use case I believe would be common:

Load a netCDF file written using ragged array representation
Extract a slice in either coordinate direction
Store back into netCDF

At least I would love such a functionality...

@pnsaevik If the approach we adopt in scipp could be ported to xarray you would be able to to something like (assuming that the ragged array representation you have in mind is "list of lists"):
data = my_load_netcdf(...) # list of lists
# assume 'x' is the dimension of the nested lists
bin_edges = sc.Variable(dims=['x'], values=[0.1,0.3,0.5,0.7,0.9])
realigned = sc.realign(data, {'x':bin_edges})
filtered = realigned['x', 1:3].copy()
my_store_netcdf(filtered.unaligned, ...)

Basically, we have slicing for the "realigned" wrapper. It performs a filter operation when copied.

Thanks so much! Appreciate it.

I would prefer to retain the dense representation, but with tricks to keep the data of sparse type in memory.
Look at the following example with pandas multiindex & sparse dtype:

The dense data uses ~40 MB of memory, while the dense representation with sparse dtypes uses only ~0.5 kB of memory!
And while you can import dataframes with the sparse=True keyword, the size seems to be displayed inaccurately (both are the same size?), and we cannot examine the data like we can with pandas multiindex + sparse dtype:

Besides, a lot of operations are not available on sparse xarray data variables (i.e. if I wanted to group by price level for ffill & downsampling):

So, it would be nice if xarray adopted pandas’ approach of unstacking sparse data.
In the end, you could extract all the non-NaN values and write them to a sparse storage format, such as TileDB sparse arrays.
cc: @stavrospapadopoulos

For ffill specifically, you would get a dense array out anyway, so there's no point to keeping it sparse, unless one did something like run-length-encoding or similar.
As for the size issue, PyData/Sparse provides the nbytes attribute which could be helpful in determining size.

I know. But having sparse data I can treat as if it were dense allows me to unstack without running out of memory, and then ffill & downsample the data in chunks:

It would be nice if xarray automatically converted the data from sparse back to dense for doing operations on the chunks just like pandas does.
The picture shows that I'm already using nbytes to determine the size.

Hi all! As part of a research project, I'm looking to contribute to xArray's sparse capabilities, with an emphasis on sparse support for use-cases in the geosciences. I'm wondering if anyone in the geosciences (or adjacent disciplines!) has encountered problems with xArray's current level of sparse support, and what kinds of improvements they'd like to see to address those issues. From playing around, it seems the current strategy of backing DataArrays with COO sparse arrays takes care of a lot of use cases, but I have the following ideas that may (or may not) be useful to implement further:

Functions/methods to convert from geopandas GeoDataframes of vector data to rasterized, potentially sparse ndarrays in an xArray Dataset/DataArray (reverse direction too); this is related to the issue of converting from sparse arrays back to multi-indexed pandas objects at the top of this issue (which I believe has yet to be solved)
Loading sparse data from a netcdf() file directly into a Dataset/Array backed by sparse ndarray(s) (seems like the only way to get sparse backings is to either unstack or call '.from_dataframe()/series()' with the sparse flag set to True?)
Support for other sparse array conventions (for ex, GCXS in the sparse package for better memory efficiency; I can't find any improvements to make on the current COO backing in terms of supported arithmetic operations, merges/joins, etc.)

I'd appreciate any feedback on these ideas, as well as any other things that would be nice to have implemented!

Hi @jdbutler and welcome! We would welcome this sort of contribution eagerly.
I would characterize our current support of sparse arrays as really just a proof of concept. When to use sparse and how to do it effectively is not well documented. Simply adding more documentation around the already-supported use cases would be a great place to start IMO.
My own exploration of this are described in this Pangeo post. The use case is regridding. It touches on quite a few of the points you're interested in, in particular the integration with geodataframe. Along similar lines, @dcherian has been working on using opt_einsum together with sparse in https://github.com/pangeo-data/xESMF/issues/222#issuecomment-1524041837 and https://github.com/pydata/xarray/issues/7764.
I'd also suggest catching up on what @martinfleis is doing with vector data cubes in xvec. (See also Pangeo post on this topic.)
Of the three topics you enumerated, I'm most interested in the serialization one. However, I'd rather see serialization of sparse arrays prototyped in Zarr, as its much more conducive to experimentation than NetCDF (which requires writing C to do anything custom). I would recommend exploring serialization from a sparse array in memory to a sparse format on disk via a custom codec. Zarr recently added support for a meta_array parameter that determines what array type is materialized by the codec pipeline (see https://github.com/zarr-developers/zarr-python/pull/1131). The use case there was loading data direct to GPU. In a way sparse is similar--it's an array container that is not numpy or dask.

@jbbutler please also see this comment et seq. https://github.com/pydata/sparse/issues/1#issuecomment-792342987 and related pydata/sparse#438.
To add to @rabernat's point about sparse support being "not well documented", I suspect (but don't know, as I'm just a user of xarray, not a developer) that it's also not thoroughly tested. I expected to be able to use e.g. DataArray.cumprod when the underlying data was sparse, but could not.
 Perhaps explicitly testing/confirming which parts of the API do/do not currently work with sparse would support the improvements to the docs that Ryan mentioned, and reveal the work remaining to provide full(er) support.

Speaking a bit to things like cumprod, it's hard to support those natively with sparse data structures in many cases (at least as things stand in the current Numba framework).
While that doesn't apply in the case of cumprod, PyData/Sparse also has a policy that if the best algorithm available is a dense one, we simply raise an error, and the user should densify explicitly to avoid filling all available RAM or getting obscure MemoryErrors.

That's a totally valid scope limitation for the sparse package, and I understand the motivation.
I'm just saying that the principle of least astonishment is not being followed: the user cannot at the moment read either the xarray or sparse docs and know which portions of the xarray API will work when giving …, sparse=True, and which instead require a deliberate choice to densify, or see examples of how best to mix the two. It would be helpful to clarify—that's all.

I suspect (but don't know, as I'm just a user of xarray, not a developer) that it's also not thoroughly tested.

Existing sparse testing is here: https://github.com/pydata/xarray/blob/main/xarray/tests/test_sparse.py
We would welcome enhancements to this!

Thank you all so much for the feedback and resources! I agree (1) testing the limits of xArray's API compatibility with sparse and (2) developing some documentation for what is/isn't supported are great places to start, so I'll get on that while I think about the other I/O issues (serialization, etc.)

A use case example: https://ncar.github.io/esds/posts/2022/sparse-PFT-gridding/


Icons: Replace current @wp-g2/icons usage with @wordpress/icons

As we're getting ready for potential integration, we should update the usage of icons to @wordpress/icons, at least for component code exported by @wp-g2/components.
This would most likely include things like checkmarks, chevrons, and arrows.
It's okay to use @wp-g2/icons (aka. react-icons/feather) for prototyping purposes.

Completed with this PR:
https://github.com/ItsJonQ/g2/pull/133


Fix i3dm matrices in ModelExperimental

Fixes https://github.com/CesiumGS/cesium/issues/10255
This PR fixes two things about i3dm model matrices in ModelExperimental:

The matrix product was missing model.modelMatrix
The axis correction was applied on the right of the node hierarchy transform, when it should have been applied on the left.

I also refactored how axis correction is handled, now it's just a matrix sceneGraph.axisCorrectionMatrix.
Here's a diagram that shows the correct matrix product order and how it compares with the glTF EXT_mesh_gpu_instancing matrix product.

Thanks for the pull request @ptrgags!

:heavy_check_mark: Signed CLA found.
:grey_question: CHANGES.md was not updated.

If this change updates the public API in any way, please add a bullet point to CHANGES.md.

Reviewers, don't forget to make sure that:

[ ] Cesium Viewer works.
[ ] Works in 2D/CV.

@j9liu this is now ready for review when you get a chance.

@j9liu updated!

@ptrgags looks good, merging now. Thanks!


Bundle Arithmetic doesn't seem to be working properly

As specified here - https://github.com/systemjs/builder#example---third-party-dependency-bundles
tried to create bundle- builder.bundle('app//* - [app//*]', 'dependencies.js', { minify: true, sourceMaps: true });
Folder structure:

Expectation : dependencies.js bundle SHOULDN'T include any *.js file contents under app
Behaviour : Includes abcFolder.component.js & abcFolder.service.js file contents. Doesn't include main.js & app.module.js. It shouldn't include even the abcFolder files

folder names case sensitive check


How to handle cancelled divisions

In 2023 Lents some of the divisions were cancelled, with the result that some divisions completed their first three races, other their first two, and other the first two and the last race.
My format already had support for a 't' code which means terminate that division.  It didn't quite handle the lents 2023 scenario, but I've changed my parsing code so that results that follow a 't' go to the correct place.  With a little tinkering to show light grey line if there a split between results, this shows what I render from the results file.

lents2023_women.txt
Is this a good suggestion of how these types of events should be handled?

Oh, that looks like a nice way to handle it.
I'm a bit rusty on the meaning of t. Does the following describe the new situation?

t by itself -> division cancelled
Normal results followed by t -> remaining crews rowed over

I think originally t meant number 2 on your list, and possibly number 1.  I'm thinking that number 2 is just a bit lazy and not very useful (and potentially confusing), so I'm suggesting it's only allowed as option 1 (so it must be the first result for a new division).

Having said that, the parsing code I checked in https://github.com/mcshane-fire/bumps/blob/master/src/bumps.py allows for both of your suggestions.

If none of the crews in that division have a results code and we see a 't' then that division is shown as not racing.
If there has been at least one other result code for that division before the 't' then I assume the whole division raced (with crews yet to have results rowing over).
In the results files that I'm generating I don't rely on that second use case, but the description of this result code in https://github.com/mcshane-fire/bumps/blob/master/readme.txt does implicitly allow both use cases.


Experiment with mixpanel for user behavior analytics

As part of #239, we're replacing fabric with something else for mobile analytics. Before we go full-on tracking everything in the app, I'd like to see how it behaves tracking data on multiple platforms.
I've already integrated mixpanel with the website. I'm going to do a quick test integration for iOS on 1.5.3 so we can see if both sets of data can live nicely along side one another.

There are still a couple of calls that use it, and the Pod. Can they be removed?

Yep, I'm still looking for a good way to get insights into how many users we have and how they use the app, but we can remove mixpanel because I'm not using it at all right now.


Add socket CAN dispatcher support

This allows application(s) to listen same CAN-IDs for the same network interface. Note that the 1st linux compat patch is from #13374 so comments to that should be done there.

This needs rebase.

Codecov Report

Merging #13375 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #13375   +/-   ##
=======================================
  Coverage   52.25%   52.25%
=======================================
  Files         307      307
  Lines       45429    45429
  Branches    10506    10506
=======================================
  Hits        23740    23740
  Misses      16899    16899
  Partials     4790     4790

Impacted Files
Coverage Δ

subsys/net/ip/net_context.c
58.5% <ø> (ø)
:arrow_up:

subsys/net/ip/connection.c
71.86% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e199ff5...175b459. Read the comment docs.

New version:

Built on top of #14193 and #14201
Tested on nucleo_l432kc

@alexanderwachter can you have a re-review?

When you send a message you should check the registered filters and if the id matches, the message should be cloned to this receiver.
What about detaching the filters after the socket is closed? I can't find this in the actual implementation.

Lets target this for 1.15

Rebased against latest master

This still needs some TLC as @alexanderwachter comments needs to be addressed. I am working on it but the progress is slow because of other pending tasks.

Fixed merge conflicts, no other changes yet.

Updated according to comments.

Updated according to comments.

@alexanderwachter can you review this, all the comments you had should be ok now.

There is still no filter detaching when the socket is closed.
The filter is still attached if an application for example opens a socket, set the filter socket opt and then closes the socket. If the application does this several times we would run out of filters very quickly. Another problem with leaving the filter attaches is that this filter causes interrupts even nobody is waiting for this frame.

Fixed the socket closing (detach filters if needed).


Fix building image

Overview

Fix the missing base_image config for openstack image building.
Rationale

Bug fix.
Checklist

[x] The charm style guide was applied
[x] The contributing guide was applied
[x] The changes are compliant with ISD054 - Managing Charm Complexity
[x] The documentation is generated using src-docs
[x] The documentation for charmhub is updated.
[x] The PR is tagged with appropriate label (urgent, trivial, complex)

Manually tested to ensure the image building works.

This is likely not needed.

This was a temp solution
The long term solution "github-runner-image-builder" charm is released.

Will close this PR once the github-runner charm is successfully deployed with github-runner-image-builder charm.

Closed due to image builder is merged to main.
This is not needed.


Make Feditest unit tests work

This should address some of the other open issues for this milestone as well.
Punch list:

[x] mastodon_api.session.json with mastodon.ubos.constellation.json
[ ] mastodon_api.session.json with mastodon.saas.constellation.json
[x] mastodon_api.session.json with wordpress.ubos.constellation.json
[ ] mastodon_api.session.json with wordpress.saas.constellation.json
[x] mastodon_api_mastodon_api.session.json with mastodon_mastodon.ubos.constellation.json
[ ] mastodon_api_mastodon_api.session.json with wordpress_mastodon.ubos.constellation.json

Cannot fix mastodon_api_mastodon_api.session.json with wordpress_mastodon.ubos.constellation.json due to #367

... and now due to #370

Due to #385, we can now also do mastodon_api_mastodon_api.session.json with wordpress_mastodon.ubos.constellation.json.


Persistent Device Claims for KubeVirt

Proposal : https://github.com/kubevirt/community/issues/254
The proposal mentions two potential solutions:

Using the Dynamic Resource Allocation (DRA) API
Implementing custom Persistent Device Claims (PDCs)

The detailed plan focuses on developing PDCs. I understand this might be because DRA is still under development. However, the initial proposal also mentioned DRA as a possibility.
Could you please give some more clarity over approaching the implementation ?

@sarthaksarthak9 please use the issue in the community repository. I'll reply to you there.


StatusCode in response & isNumeric fix

Close #27
Close #28
Close #29

Looks cool, please update the package version number

I see it breaks PWA. E.g. when it adds responseCode to empty cart. I will try to make it differently and update after some tests


Automate update image tag when PR is merged

If can use latest, use it to image tag.
Or not, refer this example

Resolved this issue by @nick-bisonai in PR #1224


Making rule input more user-friendly

While adding new parameter max_in_degree to the deletion_reduction in reduction.py,
I found that this function does not allow node names that have _ or - or start with numbers.
I think it is related to simplify_using_expression_and_negation in reduction.py.

Right now there is no documentation or warning. User won't be notified that certain node names are not allowed unless the  deletion reduction actually happens

Another issue with the input rules is that it does not allow rules that do not have an empty space between not and (.
For example A* = not(B or C) cannot be imported and there is no warning or error message that tells the user why.

The second issue, with the A*=not(B or C) should be easy enough to fix. Are you sure underscores cause an issue? I knew about the hyphen because sympy interprets it as a minus sign, but I was pretty certain we had tested with underscores. In any case, I agree that this kind of thing needs to be better documented.


[Documentation Bug]: Want to add Contribution.md Doc

Describe the bug ✍
I think the Contribution.md file is missing here. I want to add it in which I'll explain step by step guide for contributing in this project.
Provide step by step information reproduce the bug 📄
Description
Select program in which you are contributing
No response
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @silky-modi22 !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

Assign me this issue

Hello @silky-modi22, Time's Uppp!⏰
Sorry for closing your issue!
But it's more than a week since we haven't received anything from your side 😢 .
Come up with new ideas, create a new issue and make sure you finish it within a week! 🔥
All the best! 🚀
Happy Hacking! 💗

Hey @silky-modi22 ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇

@kunjgit can you please reopen this issue I think you assign this to me then after only I can raise PR. That's why I not raise the PR.
Please reopen this issue By today I raise the PR


kube explorer not working with k3d local cluster

kube-explorer does not start and shows no logs. here is my kube-explorer command
kube-explorer --kubeconfig="$HOME/.kube/config" --context=k3d-devship --http-listen-port=9080 --https-listen-port=0 --debug

When I run the command it shows nothing, even I passed the --debug flag. If I go to http://localhost:9080, it shows Site can not be reached error.
I am using kube-explorer version v0.3.1 (f4970b8)
My Machine
macOS v12.6.4 (MacbookPro Mid 2015 model)
Darwin Kernel Version 21.6.0: Thu Mar  9 20:08:59 PST 2023; root:xnu-8<IP_ADDRESS>0.8~1/RELEASE_X86_64 x86_64

Others Informations
# devship.localhost resolves to <IP_ADDRESS>
$ kubectl cluster-info

Kubernetes control plane is running at https://devship.localhost:6445
CoreDNS is running at https://devship.localhost:6445/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://devship.localhost:6445/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy

$ kubectl get nodes

NAME                   STATUS   ROLES                  AGE   VERSION
k3d-devship-server-0   Ready    control-plane,master   13m   v1.26.3+k3s1
k3d-devship-agent-1    Ready    <none>                 13m   v1.26.3+k3s1
k3d-devship-agent-0    Ready    <none>                 13m   v1.26.3+k3s1

How did you create your k3d cluster? I cannot reproduce this way.
k3d cluster create -i rancher/k3s:v1.26.3-k3s1 v1-26-3
kube-explorer --kubeconfig="$HOME/.kube/config" --context=k3d-v1-26-3 --http-listen-port=9080 --https-listen-port=0

@niusmallnan thanks for replying. Here is my k3d-cluster.yml
apiVersion: k3d.io/v1alpha4
kind: Simple # internally, we also have a Cluster config, which is not yet available externally
metadata:
  name: devship # name that you want to give to your cluster (will still be prefixed with `k3d-`)
servers: 1
agents: 2
kubeAPI:
  host: devship.localhost
  hostIP: <IP_ADDRESS>
  hostPort: "6443"
image: rancher/k3s:v1.26.3-k3s1
# network: ronsvpn
volumes:
  - volume: /Volumes/Devzone/storage/kubernetes/clusters/volumes/devship:/mnt/devship
    nodeFilters:
      - all
ports:
  - port: <IP_ADDRESS>:443:443
    nodeFilters:
      - loadbalancer
  - port: <IP_ADDRESS>:80:80
    nodeFilters:
      - loadbalancer
env:
  - envVar: DEV_SALT=MyrCUiOM1dFfmSbUVI9EWP3UAVBmRogR # for hashing secrets
    nodeFilters:
      - server:0
registries:
  create:
    name: registry.devship.localhost
    host: "<IP_ADDRESS>"
    hostPort: "5000"
    volumes:
      - /Volumes/Devzone/storage/kubernetes/k8s-clusters/registry:/var/lib/registry

# hostAliases:
#   - ip: <IP_ADDRESS>
#     hostnames:
#       - cloud.flare.dns # add cloudflare dns
options:
  k3d:
    wait: true
    timeout: 30s
    disableLoadbalancer: false
    disableImageVolume: false
    disableRollback: false
    # loadbalancer:
    #   configOverrides:
    #     - settings.workerConnections=2048
  k3s: # options passed on to K3s itself
    extraArgs:
      - arg: "--tls-san=*.devship.localhost"
        nodeFilters:
          - server:*
      - arg: "--tls-san=<IP_ADDRESS>"
        nodeFilters:
          - server:*
      - arg: "--disable=traefik" # disable traefik
        nodeFilters:
          - server:0
    nodeLabels:
      - label: "CreatedBy=Rony"
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true
  runtime: # runtime (docker) specific options
    gpuRequest: ""
    serversMemory: ""
    agentsMemory: ""
    labels:
      - label: CreatedBy=Rony
        nodeFilters:
          - server:0
          - loadbalancer

When I run the command it shows nothing, even I passed the --debug flag.

This means kube-explorer cannot talk to kube-api via kubeconfig. Could you please share more details about $HOME/.kube/config.


scm.InitNoScmBench test is failing

https://github.com/iterative/dvc-bench/runs/720445931?check_suite_focus=true#step:7:6374

@pared ?


objectstore: eagerly unmount bind-mounts

In the object store, temporary bind mounts are used when accessing the content, i.e. the individual trees. Their unmount is currently done with the --lazy flag. The use of this flag goes way back to commit da121beda1570f9b43fda75f3aefbba2c848d7f2, which sadly does not mention why the flag was introduced. Since the tree and files in the tree will be used by consequent stages it seems reasonable to do the un-mounting eagerly and thus this reverts back to that behavior.

Thanks for this! Out of interest, does it remove the need for the sync in the mount logic?

Have not verified yet, but this seems the only place that could introduce that issue.


[SPARK-40859][INFRA] Upgrade action/checkout to v3 to cleanup warning

What changes were proposed in this pull request?
Upgrade action/checkout to v3 (point ot v3.1 now).
Why are the changes needed?

https://github.com/actions/checkout/releases/tag/v3.1.0 cleanup "The 'set-output' command is deprecated and will be disabled soon."
https://github.com/actions/checkout/releases/tag/v3.0.0 since v3, use the node 16 to cleanup "Node.js 12 actions are deprecated"

According to https://github.com/actions/checkout/issues/959#issuecomment-1282107197, v2.5 also address 'set-output' warning, but only v3 support node 16, so we upgrade to v3.1 rather than v2.5
Does this PR introduce any user-facing change?
No, dev only
How was this patch tested?
CI passed

@srowen @HyukjinKwon Thanks, merged.


build: share cargo target dir per builder

Description
Previously, there was a seperate cargo target dir per builder/application tuple.
This PR relaxes this to one target dir per builder.
This should increase re-using of build artifacts, reducing build times.
edit this PR's build used 2h:14m of CI builder time, the previous PR build used 3h:20m.
Issues/PRs references

Open Questions

Change checklist

[x] I have cleaned up my commit history and squashed fixup commits.
[x] I have followed the Coding Conventions.
[x] I have performed a self-review of my own code.
[ ] I have made corresponding changes to the documentation.

I'll merge this so it is easier to dogfood. Watch out for miscompilations (which I don't expect).


User picture lost, can't create folder /storage/users/default.png

Laravel Version: 5.4
Voyager Version: v0.11.9
PHP Version: 7.1.3
Database Driver & Version: MariaDB 10.1.22

Description:
When I reinstall my OS and run Voyager project again.
The profile picture is lost.
I can't find folder /storage/users.
Have only three folders in /storage is: app, framework, logs.

You have to create a symbolic link from /storage/app/public to /public/storage.
You can run this command to do it:
php artisan storage:link
Otherwise, you can use this (always in UNIX):
ln -s /path/to/laravel/storage/app/public /path/to/laravel/public/storage
Make sure your apache has this feature enabled.

Thanks @GlezSosa


Find a way to check for file updates and notify users to download

That's why we can't have nice things. Not all servers (and certainly not Lichess or siderite.dev) support HEAD returning content-length. So the solution must be saving another file next to the one we want, with some metadata.


RFE: is it possible to start making github releases?🤔

On create github release entry is created email notification to those whom have set in your repo the web UI Watch->Releases.
gh release can contain additional comments (li changelog) or additional assets like release tar balls (by default it contains only assets from git tag) however all those part are not obligatory.
In simplest variant gh release can be empty because subiekt of the sent email contains git tag name.
I'm asking because my automation process uses those email notifications by trying to make preliminary automated upgrades of building packages, which allows saving some time on maintaining packaging procedures.
Probably other people may be interested to be instantly informed about release new version as well.
Documentation and examples of generate gh releases:
https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository
https://cli.github.com/manual/gh_release_upload/
https://github.com/jbms/sphinx-immaterial/pull/282
https://github.com/marketplace/actions/github-release
https://pgjones.dev/blog/trusted-plublishing-2023/
https://github.com/jbms/sphinx-immaterial/issues/281#issuecomment-1700933026
tox target to publish on pypi and make gh release https://github.com/jaraco/skeleton/blob/928e9a86d61d3a660948bcba7689f90216cc8243/tox.ini#L42-L58

Hi @kloczek I have added a tag-only GitHub release for 5.0.4 just now at https://github.com/hsluv/hsluv-python/releases/tag/v5.0.4 . Regarding automation ofr creation of GitHub releases, I don't see much value in that personally, so I won't be the one to build that or compaign for it. Regarding release notification, I use rss2email and the RSS feed of Repology to be notified of new releases of software I maintain  even without GitHub releases. The repology page of hsluv-python is https://repology.org/project/python:hsluv/versions . I'm unsure why Fedora is not listed there, could be a naming-difference thing. Repology works great, it helps me be a better Gentoo maintainer.
@boronine shall we close this as fixed?

Yes if you will remember to add gh release entry on next release 😋
nevertheless .. thank you 👍

Yes if you will remember to add gh release entry on next release 😋

@kloczek that was the plan at least, yes :smiley:

Seems complete, closing…


Feature request: Print time/date stamp whenever an exception is printed.

Last night when my provider was doing maintenance, the database connection to two of my delayed_job runners was lost, causing them to lock up and throw "Agent#4254: Error when fetching url: Mysql2::Error::TimeoutError: Lock wait timeout exceeded; try restarting transaction" exceptions.  The thing is, I don't know exactly when that happened (and I suspect other folks might run into this as well).  Would it be possible to print the system's time and date right before the exceptions get dumped?

Hmm not sure, I am pretty sure the background processors just log to STDOUT, when using docker the timestamp would be added by docker itself. On "manual installations" there might be an option to add the timestamp in runit.


Fix Flathub checks

https://flathub.org/apps/io.github.divverent.aaaaxy
when logged in, shows 3 failing checks:

Has primary brand colors

Next release will fix that, already taken care of by adding the missing tag

Doesn't fill too much or too little of the canvas (https://docs.flathub.org/docs/for-app-authors/metainfo-guidelines/quality-guidelines/#reasonable-footprint)

Still to be done
While at it, may also want to install the SVG file as an icon instead
Gotta check if anything uses the SVG right now and will need adjustments too (e.g. macOS or iOS)

Reasonable window size

Next release will fix that, already taken care of by downscaling the images

Seems like we need the icon to be downscaled to 13/16 its current size. It then just fits the rectangle.

Did this scaling factor in the SVG file now. Will soon send another release to flathub to try this out.

Remaining checks that fail:

Icon: good contrast on dark backgrounds

I thought I did that but there are some things that may still be possible. Keeping open.

Icon: In line with contemporary styles

This is very intentional, so, this is a wontfix.

Screenshots: reasonable window size

Again I thought this has been addressed, so I gotta figure out what that check really does


replace example/testC/TestMain.cpp by TestMain.c

For the developers' reference, C program is better reference to explain C-wrapper API.

SVGNativeCWrapper.{cpp,h}: svg_native_render_output() API tweaked.
example/testC/TestMain.c: rewrite of TestMain.cpp.
example/testC/TestMain.cpp: removed.
example/testC/CMakeLists.txt: TestMain.cpp is replaced by TestMain.c.

@mpsuzuki I think there is a misunderstanding. The Cairo function was an example how to pass the arguments. I didn’t mean to replace the text part with raster content. IMO this should stay the same to keep it simple as an example.

Oh, I'm sorry! I would retry.

I did like this:
void svg_native_get_output(svg_native_t* sn, const char** data, size_t* length)
{
    if (!length || !data)
    {
        return;
    }

    *data = NULL;
    *length = 0;

    auto _sn = dynamic_cast<svg_native_t_*>(sn);
    if (!_sn || _sn->mRendererType != SVG_RENDERER_STRING)
        return;

    auto renderer = std::dynamic_pointer_cast<SVGNative::StringSVGRenderer>(_sn->mRenderer);
    if (!renderer)
        return;

    const auto& string = renderer->String();

    *data = string.c_str();
    *length = string.size();

    return;
}

@mpsuzuki The Argument list looks good now. However, the string just exists as long as the renderer exists and gets cleared together with the renderer. You still want to allocate a limited amount of memory (still 2048 max?) and memcopy the data into the new buffer.

Without testing, this looks very nice, thanks!


How to export Enums?

Hi!
Just a simple question.
How can create a function that returns an enum from CPP land to PHP land?
For example:
     Php::Value SomeClass::getFillType()
    {
        enum rgb { red, green, blue };
        // is there a way to export rgb to PHP?
    }

@gtamas I think it may be impossible as PHP does not support enum types. Its best to use a library for enums.

@gtamas I think it may be impossible as PHP does not support enum types. Its best to use a library for enums.

Indeed.

Indeed.

OK Thanks guys!

OK Thanks guys!


integcli: change Dettach to Detach

LGTM

LGTM. Windows failure is because CI is broken, this is just renaming a test... merging.