make monitor in eclipse problem

Environment

Development Kit:      NodeMCU-32S Lua WiFi IOT Development Board v1.1
Core (if using chip or module): [ESP32S]
IDF version (git rev-parse --short HEAD to get the commit id.):
221eced06
Development Env:		[Eclipse]
Operating System:		[Windows10]
Power Supply:         [USB]

Problem Description
I have just installed esp32_win32_msys2_environment_and_toolchain-20181001.zip for Windows (W10).
No errors with "make monitor" in bash/make environment. I can makeconfig, build, flash and monitor
In Eclipce: "make all" "make flash"targets finished without errors.
python --version
Python 2.7.15
git describe --tags --dirty
v3.2-dev-1250-g221eced06
PATH in Eclipse "Environment to set" C:\msys32\usr\bin;C:\msys32\mingw32\bin;C:\msys32\opt\xtensa-esp32-elf\bin
winpty winpty.dll ... in C:\msys32\usr\bin from the box.
and the result with the error.
I tried hard reading forums with no success...
10:59:04 **** Build of configuration Default for project hello_world ****
python C:/msys32/home/Megauser/esp/esp-idf/tools/windows/eclipse_make.py -j2 monitor
stdin is not a tty
make: *** [/home/Megauser/esp/esp-idf/components/esptool_py/Makefile.projbuild:106: monitor] Error 1
Running make in 'C:/msys32/home/Megauser/esp/hello_world'
including C:/msys32/home/Megauser/esp/esp-idf/components/app_update/Makefile.projbuild...
including C:/msys32/home/Megauser/esp/esp-idf/components/bootloader/Makefile.projbuild...
including C:/msys32/home/Megauser/esp/esp-idf/components/bootloader_support/Makefile.projbuild...
including C:/msys32/home/Megauser/esp/esp-idf/components/coap/Makefile.projbuild...
including C:/msys32/home/Megauser/esp/esp-idf/components/esp32/Makefile.projbuild...
including C:/msys32/home/Megauser/esp/esp-idf/components/esptool_py/Makefile.projbuild...
including C:/msys32/home/Megauser/esp/esp-idf/components/mbedtls/Makefile.projbuild...
including C:/msys32/home/Megauser/esp/esp-idf/components/nghttp/Makefile.projbuild...
including C:/msys32/home/Megauser/esp/esp-idf/components/partition_table/Makefile.projbuild...
including C:/msys32/home/Megauser/esp/esp-idf/components/ulp/Makefile.projbuild...
Toolchain path: C:/msys32/opt/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc.exe
Toolchain version: crosstool-ng-1.22.0-80-g6c4433a5
Compiler version: 5.2.0
python C:/msys32/home/Megauser/esp/esp-idf/tools/check_python_dependencies.py
Python requirements from C:/msys32/home/Megauser/esp/esp-idf/requirements.txt are satisfied.
[ -f C:/msys32/home/Megauser/esp/hello_world/build/hello-world.elf ] || echo "*** 'make monitor' target requires an app to be compiled and flashed first."
[ -f C:/msys32/home/Megauser/esp/hello_world/build/hello-world.elf ] || echo "*** Run 'make flash monitor' to build, flash and monitor"
[ -f C:/msys32/home/Megauser/esp/hello_world/build/hello-world.elf ] || echo "*** Or alternatively 'make simple_monitor' to view the serial port as-is."
[ -f C:/msys32/home/Megauser/esp/hello_world/build/hello-world.elf ] || exit 1
winpty python C:/msys32/home/Megauser/esp/esp-idf/tools/idf_monitor.py --baud 115200 --port "COM7" --toolchain-prefix "xtensa-esp32-elf-" --make "make" --print_filter "" C:/msys32/home/Megauser/esp/hello_world/build/hello-world.elf
11:06:15 Build Failed. 1 errors, 0 warnings. (took 7m:11s.88ms)
Expected Behavior
I expect hello_world response in eclipse console window.
But when I copy/paste
winpty python C:/msys32/home/Megauser/esp/esp-idf/tools/idf_monitor.py --baud 115200 --port "COM7" --toolchain-prefix "xtensa-esp32-elf-" --make "make" --print_filter "" C:/msys32/home/Megauser/esp/hello_world/build/hello-world.elf
to bash make env window and run then that command I get the hello_world response from myboard ok.
Actual Behavior
no hello_world response from the board. Stops with the 106 error/

Eclipse build is not running from a tty, so starting "monitor" there is not possible. Since "monitor" is an interactive application, it needs some kind of console...
You can change the build target to make flash, which is able to run without a tty attached. This target will build your app and upload it to the board. Then you can either open a separate console window and run make monitor there, or open one within Eclipse.

So, you mean I cannot use "make flash monitor" or "make monitor"in Eclipse?I have to open MinGW32 console and run
winpty python C:/msys32/home/Megauser/esp/esp-idf/tools/idf_monitor.py --baud 115200 --port "COM7" --toolchain-prefix "xtensa-esp32-elf-" --make "make" --print_filter "" C:/msys32/home/Megauser/esp/hello_world/build/hello-world.elf
Is that what you meant when wrote "you can either open a separate console window and run make monitor there". But how to evoke a console within Eclipse as you said?

I guess I got it! Open Teminal -->Serial Terminal. And I got the output of my dev. board Thank you, Игорь!

Hi
I was able to build and flash successfully using eclipse IDE. But the problem is that when I checked with blinking LED example, the LED was not blinking on the board. I also tried by changing its GPIO in menuconfig. But when i flashed using msys command prompt using same examples and menuconfig the led was blinking on the board. So i want to ask that why is it not blinking using when flashed with eclipse?


Download artifacts in consistent way

ARTIFACT CONSISTENCY PROBLEM:
We have realized that containers using shared folders to store artifacts were suffering from time to time from inconsistently downloaded artifacts.
When running on orchestrator, but basically whenever downloading artifacts from containers (and not from BcContainerHelper from the host machine) and using a shared folder, it's not possible to use mutext (which is the difference between in-container download vs. download on the host using BCCH) and guarantee only one process accessing the shared resources.
The provided functionality uses file lock that controls access and guarantees only one process can access and other containers will be waiting until done (with success or not).

I have created this as a draft because I think I will wait several next days before confirming everything works as expected.

We have tested it during the last months and it works great. We haven't been experiencing any inconsistency since putting this into our container-based environments.
I will switch from draft to the regular PR. You can decide if it's worth to implement into master or not.


How to run 'ad-hoc' regex patterns or word lists in v2?

We are using presidio to let different users define different 'triggers' to classify sensitive data 'n the fly' without having to program. With v2, this not seems possible anymore? Or how is this intended?
I understand you want to go stateless, but should be easy to let the parameters that previously where part of the post to create a pattern, now be a parameter in the get where then on the fly pattern recognition is done?
How do you envision users implement this feature, which was one of the best things of v1? Flexibility is killed if for every custom pattern to have to create a python script and restart your servers, right?
Thanks!

Hi @rickgoud,
Thanks for making us aware of your issue. We are currently looking into re-introducing this capability in V2.
V2 is planned to grow iteratively, and this component is one of the highest priorities.
If you, or anyone else from the community, would like to contribute to this effort, please reach out to us at<EMAIL_ADDRESS>and we can discuss this further.
Some clarification on the recognizer store in V2: https://github.com/microsoft/presidio/discussions/584#discussioncomment-434562

Hi @omri374 ,
Thanks for the response! Do you have any ETA for adding this feature? Would be really a waste if that would limit people migrating to v2. If you let the users handle persistency by adding the required analyzers as parameters in the GET, presidio would not need a datastore (which you suggest in #584 ), right? That should make implementation much easier (large part can be copies from v2?) and not related to Kubernetes, right? I think with these 'ad-hoc' features added stateless, presidio v2 would be incredibly powerful!! Great indeed if the community could upvote this issue to support it's priority!
Thanks in advance people!

Hi @rickgoud
I'd like to make sure I understand what you mean by "If you let the users handle persistency by adding the required analyzers as parameters in the GET, presidio would not need a datastore".
Do you propose to add those regex patterns / deny-lists as part of the "analyze" request? So that I would not only send the text for analysis and the language, but also the actual patterns / lists to be used?

Hi @omri374 ,
Apologies for the unclarity. Indeed that is what I mean. I understand you want presidio to be stateless for scalability, which makes total sense. That would be possible if users can indeed add patterns in the analyze request, right?
Regards,
Rick

Having the ability to have simple regex patterns / deny lists as part of the 'analyze request'  would be a great addition indeed for v2!

It would indeed be a nice feature of v2 to be able to use 'custom recognizers' without having to change code and/or restart the analyzer application. To be able to dynamically do this in each POST /analyze request sounds nice! 😊

Hi @omri374 ; Anything you can share on if/when this issue gets picked-up? Apparently more people excitingly waiting for this! :). Also the context words in v1 were awesome!

@omri374, any updates/progress? Would be great!

Hi @rickgoud. Unfortunately I don't have an ETA for this. If you or any one else would be interested in contributing the stateless approach you mentioned in this issue, I would be happy to help with design, review or anything else needed. Feel free to reach out to<EMAIL_ADDRESS>

Hi, please see PR #677

You are a rockstar @omri374!!
With this commit up to part with v1? Are there still some 'ToDo's' on your list? Specific things you would like to have feedback on?
Looking forward to giving it a spin!!

Thanks @rickgoud! This is still WIP, pending merge and release.
As mentioned in this thread, this is not 100% in par with V1 as there is no state involved. Every ad-hoc recognizer would have to be attached to every request to the /analyze service. If you have a long list of ad-hoc recognizers, or a long list of strings in your deny list, for example, this payload could start becoming pretty large.

Nice! Great effort @omri374 ! This MR includes ad-hoc recognizers for patterns and denylist, is the contexphrase extension also supported? Could not find a reference in the updated documentation. Thank you!

@olivierpaling should be! Good question, I will add tests to verify

@omri374 great stuff! I was just testing your branch and noticed that when doing the /analyze POST without the 'entities' parameter, you get back all found recognizers including the add-hoc one. So far so good. If you define only the ad-hoc entity to narrow down the request it doesn't work. In the example I believe you use "US_DRIVER_LICENSE", "ZIP"] but if you only keep ZIP you get an error ["error": "No matching recognizers were found to serve the request."]. The other way around if you remove ZIP and keep US_DRIVER_LICENSE it still works and you get back both classifications. So it seems no need to put it in the entities parameter.
Question of the day. Why is this the case and can we narrow down the request to ONLY analyzing based on the ad-hoc recognizer (ZIP in this example).
Thank you!


Text misinterpreted as Unclosed element

Emberjs: v1.10.0
Ember-cli: v0.1.15
In the following template snippet, the <-- arrow is misinterpreted as an HTML element.
{{#link-to "tornado"}}
    <p> <-- Go Back</p>
{{/link-to}}

bdw ~/projects/tornado/src/browser ember server
version: 0.1.15
Could not find watchman, falling back to NodeWatcher for file system events
Livereload server on port 35729
Serving on http://<IP_ADDRESS>:4200/
File: browser/pods/tornado/one/template.hbs
Unclosed element `go` (on line 6).
Error: Unclosed element `go` (on line 6).
    at HTMLProcessor.nodeHandlers.Program (/Users/bdw/projects/tornado/src/browser/bower_components/ember/ember-template-compiler.js:3385:17)
    at HTMLProcessor.acceptNode (/Users/bdw/projects/tornado/src/browser/bower_components/ember/ember-template-compiler.js:3579:43)
    at HTMLProcessor.nodeHandlers.BlockStatement (/Users/bdw/projects/tornado/src/browser/bower_components/ember/ember-template-compiler.js:3405:44)
    at HTMLProcessor.acceptNode (/Users/bdw/projects/tornado/src/browser/bower_components/ember/ember-template-compiler.js:3579:43)
    at HTMLProcessor.nodeHandlers.Program (/Users/bdw/projects/tornado/src/browser/bower_components/ember/ember-template-compiler.js:3375:16)
    at HTMLProcessor.acceptNode (/Users/bdw/projects/tornado/src/browser/bower_components/ember/ember-template-compiler.js:3579:43)
    at preprocess (/Users/bdw/projects/tornado/src/browser/bower_components/ember/ember-template-compiler.js:3551:55)
    at compileSpec (/Users/bdw/projects/tornado/src/browser/bower_components/ember/ember-template-compiler.js:741:17)
    at TemplateCompiler.__exports__.default [as precompile] (/Users/bdw/projects/tornado/src/browser/bower_components/ember/ember-template-compiler.js:643:14)
    at TemplateCompiler.processString (/Users/bdw/projects/tornado/src/browser/node_modules/ember-cli-htmlbars/index.js:50:59)

According to the HTML spec, https://html.spec.whatwg.org/multipage/syntax.html#tag-open-state, <-- should be a parse error, since the < indicates the start of a tag and the - is not valid for a tag name.  HTMLBars is not handling this correctly, in that it's not throwing an error right there. Regardless, the markup shown is not valid.  You need to escape those characters (e.g. < instead of <, since they have special meaning in html.

cross-post from https://github.com/emberjs/ember.js/issues/10477

Ah, got it :-)

this should be reported and discussed here https://github.com/tildeio/simple-html-tokenizer/issues


Bug: Failing to run Install API Template run while Deploying API

Report
Hi I'm getting below issue while using APIM Resource toolkit software to deploy REST API,

FYI, I'm getting this issue rarely like once in a week. I just want to know why it is occuring and what needs to do.
Expected Behavior
To Install API Templates successfully
Actual Behavior
Install ARM Templates successfully based on config file
Steps to Reproduce the Problem
Follow the APIM Resource toolkit procedure
Component
Creator
DevOps Resource Kit Version
reskit0.7
.NET Version
NA
OS
Windows
Configuration
Configuration:
version: 0.0.1
apimServiceName: example-apim-name

API Management Configuration
No response
Logs
example

Anything else?
No response
Contact Details
<EMAIL_ADDRESS>

There is not enough information in this report to diagnose or reproduce the information.  At a minimum, we need the configuration you are using and the full logs you are experiencing.

Actually, I am seeing what the problem is.  You are using v0.7 and do not have the .NET Core runtime v3.1.0 installed.  It's right in the error message.
Install .NET Core v3.1.0 and try again.


Nested OpenApi

I have multiple and multi-layer routes and I'd like to organize the routes by nesting openapis, which maybe something like this:
pub struct RootApi {
    pub struct FatherApi {
        pub struct Son1Api;
        pub struct Son2Api;
    }
}

And i can visit the route by /root/father/son1/... and /root/father/son2/....
Is this possible? Or  is there any suggestion for it?

I tried to do something similar, but no luck. Did you figure it out?


feat(gatus): add support for env[].valueFrom and envFrom

Summary
This MR adds support for the env[].valueFrom syntax, allowing one to have Kubernetes Secrets made available to Gatus as environment variables.
Checklist

[X] Tested and/or added tests to validate that the changes work as intended, if applicable.
[X] Updated documentation in README.md, if applicable.

This PR was copied from https://github.com/minicloudlabs/helm-charts/pull/86

@Mareo Great work & thank you for the contribution!


Add user 'tags' similar to reddit flairs

A little snippet of information the user can set, such as there email, country - etc.

Progress made in https://github.com/Matt-Allen44/Quik/commit/85abbbbfc0d14e846a4d12f2ea4dace084fe8bb7

Added a menu option on top of the chat command.


lnurlwithdraw fails with BTCPay Server

When I scan a lnurlwithdraw QR code from BTCPay Mutiny shows an error message.  These QRs can be claimed with other wallets including Blue Wallet and Zeus.
BTCPay Bech32:
lightning:lnurl1dp68gurn8ghj7cn5vdcxz7fwvf5hgcm0d9hxc6t6v9exgtnwv46z7sj5gvh42j2vfe24ynp0wa5hg6rywfshwtmswqhny72dw4r9wv6nxes5z334vau4w3mjw5m5zkz32p885umwjqsg8n
Error: Failed to call on the given LNURL.
BTCPay URI:
lnurlw://btcpay.bitcoinlizard.net:443/BTC/UILNURL/withdraw/pp/2yMuFW3S6aAF5gyWGru7AXQPNzsn
Error: Invalid payment request
These QR codes have already been claimed.  I can provide you with redeemable lnurlwithdraw QR codes upon request.

@bitcoinlizard It sounds like your query would be best dealt with by the support team.
Please see the link below to our dedicated support line:
Support :Help Center
Note: Click on the live chat icon at the bottom corner of the page to start a conversation

It look like these 404, do you have an example of a working one

I can claim this one with Blue Wallet.  Here it is in text:
lightning:lnurl1dp68gurn8ghj7cn5vdcxz7fwvf5hgcm0d9hxc6t6v9exgtnwv46z7sj5gvh42j2vfe24ynp0wa5hg6rywfshwtmswqhnyvmfw9dx7wt6feaxjjrpdve42kfngdpkv5zjffn5v7n5uhnu65

Looks like this is failing from CORS issues. This is on the btc pay server's side where it needs to send back a CORS header


Problem with Bundle Identifiers

Following the tutorial you linked I couldnt just compile the project as it is. I needed to add my Team Signing everywhere and did so. Now I changed every gabrieloc in the Bundle Identifier to dendron and it still wont work. Now I just get this error and cant seem to fix it.
WatchKit App doesn't contain any WatchKit Extensions whose WKAppBundleIdentifier matches "com.dendron.giovanni". Verify that the value of WKAppBundleIdentifier in your WatchKit Extension's Info.plist matches the value of CFBundleIdentifier in your WatchKit App's Info.plist.
Checking every Bundle Identifier and it fits perfect. But I also cant seem to find CFBundleIdentifier in the plist just the WKAppBundleIdentifier under the NSExtension and I changed that too.

Dont change the Bundle Identifiers with $(PRODUCT_BUNDLE_IDENTIFIER) to com.xxxx.giovanni   Xcode seems to have a problem with that.
For all having this problem you just need to the Info.plist file from the watchOS folder and change the WKCompanionAppBundleIdentifier and in the Info.plist from the WatchKit extension under NSExtensions -> NSExtensionsAttribute -> WKAppBundleIdentifier
This should fix all problems


Increase size (or remove max limit) for _ebpf_id_table

_ebpf_id_table is a hash table for tracking object IDs. Its current size is 1024. An ID is generated for each program, map and link. If multiple programs are loaded that also have multiple maps, limit of 1024 can be exhausted very soon, and program load starts to fail after that.
Possible fixes:

Increase size of _ebpf_id_table to a very large value.
Option 1 still has possibility of hitting max limit when many programs and maps are present. An alternate solution is to update hash table implementation to optionally not have a max limit, and then not set any max limit for _ebpf_id_table.

Looks like hash table implementation already supports unlimited number of entries. The fix for this can be to just set max_entries = 0


feat: event cell text color

Add two props eventCellTextColor, allDayEventCellTextColor.
these props can override event text color.

Sorry I missed this PR completely.
Using theme sounds enough, but I understand lots of people prefers more simple way to accomplish to change these texts. Anyway this PR should be merged although some future refactoring is definitely needed.


⚠ ClamAV has degraded performance

In d5a195f, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 219 ms

Resolved: ClamAV performance has improved in 4aeda22.


build(feat): latest submissions script #trivial

This PR resolves PHIRE-712
Description

Adds a little script to easily get the latest submission tags that are needed for release processes and hot fixes:
$ ./scripts/deploys/latest-submissions
Latest iOS submission: ios-8.34.0-2<IP_ADDRESS>-submission
Latest Android submission: android-8.34.0-1674645623-submission

PR Checklist

[ ] I have tested my changes on iOS and Android.
[x] I hid my changes behind a feature flag, or they don't need one.
[x] I have included screenshots or videos, or I have not changed the UI.
[x] I have added tests, or my changes don't require any.
[x] I added an app state migration, or my changes do not require one.
[x] I have documented any follow-up work that this PR will require, or it does not require any.
[x] I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

latest release script - Brian

Need help with something? Have a look at our docs, or get in touch with us.

This PR contains the following changes:

Dev changes (latest release script - Brian)

  Generated by :no_entry_sign: dangerJS against 2aacd41c3728c2e90bf95c10fdf666d4ac2877ba


Error when used in a project targeting catalyst

Once I enable catalyst, building for Mac platform will give me the following errors:
Signing for "FontAwesome.swift-FontAwesome.swift" requires a development team. Select a development team in the Signing & Capabilities editor.
It doesn't do this for any other pod files that I use. Excluding FontAwesome (which needs me to set minimum iOS to 13) works as well, but I am wondering why this pod is so special.
Any ideas?

I have the same issue. Bumping for visibility

Same problem here

Need this as well

Same problem. In iOS 16. this won't appear until iOS 16.
I config the developer acount. But I don't know why this happens.

This issue appeared for me after updating XCode from 13.4.1 to 14.0. I can compile if I select a development team in the FontAwesome.swift-FontAwesome.swift target, but I guess it's not supposed to work like this?

suffering from this issue as well. please fix

For anybody facing this, here's the solutions (choose the one you prefer):

Untick "Automatically manage singing" for FontAwesome.swift-FontAwesome.swift target
Set your own development team for FontAwesome.swift-FontAwesome.swift target
(PREFERRED) If you're using Cocoapods and want the fix to persist (meaning you won't have to re-do above steps after each pod install), add following code at the end of your Podfile, and re-run pod install:

post_install do |installer|
  installer.pods_project.targets.each do |target|
    if target.respond_to?(:product_type) and target.product_type == "com.apple.product-type.bundle"
      target.build_configurations.each do |config|
        config.build_settings['CODE_SIGNING_ALLOWED'] = 'NO'
      end
    end
  end
end


Decreased accuracy after conversion to TensorRT

I derived a c++ app (attached) from cppapi/infer.cpp for evaluating a TensorRT model on the COCO validation set. When I convert your pretrained retinanet_rn50fpn.pth and evaluate it on a TX2, it's accuracy is reduced.  I'm expecting  IoU=0.50:0.95 score of 0.358.  Instead, I measure 0.277.
I'm using an unmodified version of this repository. According to dpkg, I'm using tensorrt                                   <IP_ADDRESS>-1+cuda10.0 on the TX2.
Conversion, evaluation parameters:
retinanet export /coco/retinanet_rn50fpn.pth --opset=9 --batch=1 --size=800 --verbose /coco/retinanet_rn50fpn_800.onnx
./export retinanet_rn50fpn_800.onnx retinanet_rn50fpn_800.plan
./infer_coco_val retinanet_rn50fpn_800.plan /mnt/data/datasets/coco_2017/list.txt detections_800.json

Thanks for the excellent repository, and thanks in advance for any ideas.
infer_coco_val.zip

Looking at some of the TX2 detection outputs, the classes look good, but sometimes the bounding boxes are shifted towards the top, left.  Possibly a problem with the anchor box decoder, or the box offsets from here? :  extras/cppapi/export.cpp

Hi JimKlingshirn,
There's a few things I noticed in your issue description:

By using --size 800 in your retinanet export cmd, you are specifying that the TRT engine produced should have a fixed input of 800x800. This input size is smaller than what was used to calculate the IoU scores (and smaller than what the model was trained with) so I would expect the accuracy to be reduced.

In general, the cppapi code is more intended as a minimal example to demonstrate the C++ workflow, not necessarily to demonstrate accuracy.

Looking through the cpp file used to evaluate accuracy, the image preprocessing used is different than what is used during retinanet infer. It is resizing the images to match the input dimensions of the TRT engine. I recommend you check out how TRT inference is done in the rest of the codebase (https://github.com/NVIDIA/retinanet-examples/blob/master/csrc/extensions.cpp#L100 and https://github.com/NVIDIA/retinanet-examples/blob/master/retinanet/data.py#L47).


Installation instruction does not contain set up with Docker

Description
Most of the developers are using python with Docker now a days and I see instructions contain only conda and pycharm. I will suggest to have the setup with Docker as well
Context
I still find it difficult to setup Kedro on localsetup, however when I did with Docker it was much simpler and reproducible, hence would like to add this feature and if needed I can do that and contribute as well
Possible Implementation
We could use a base python image or a standard image loaded with other packages and extend it
Possible Alternatives

@kuldeep-singh-duhan good idea, please have a look at the issue #265 and also the PR #268 - please see if what you have expressed above is covered partially if not fully.

Yes, it looks good and work. I will close my issue


Make Function alias to { [[call]]: any }

Proposal
Make Function alias to { [[call]]: any } instead of any

Use case
It would error if you provide non-callable things and also seems more safe, instead of outright any

Is { [[call]]: any } equivalent to { (...args: any[]): any }?
I didn't know about [[call]], are there any docs for it?  I can't seem to find docs for callable types either.

There's PR for call property https://github.com/facebook/flow/pull/7790 docs, not sure about equivalency

We now updated the docs to recommend other safer patterns: https://flow.org/en/docs/types/functions/#any-function


[SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion

What changes were proposed in this pull request?
This is the shuffle writer side change where executors can push data to remote shuffle services. This is needed for push-based shuffle - SPIP SPARK-30602.
Summary of changes:

This adds support for executors to push shuffle blocks after map tasks complete writing shuffle data.
This also introduces a timeout specifically for creating connection to remote shuffle services.

Why are the changes needed?

These changes are needed for push-based shuffle. Refer to the SPIP in SPARK-30602.
The main reason to create a separate connection creation timeout is because the existing connectionTimeoutMs is overloaded and is used for connection creation timeouts as well as connection idle timeout. The connection creation timeout should be much lower than the idle timeouts. The default for connectionTimeoutMs is 120s. This is quite high for just establishing the connections.  If a shuffle server node is bad then the connection creation will fail within few seconds. However, an overloaded shuffle server may take much longer to respond to a request and the channel can stay idle for a much longer time which is expected.  Another reason is that with push-based shuffle, an executor may be fetching shuffle data and pushing shuffle data (next stage) simultaneously. Both these tasks will share the same connections with the shuffle service. If there is a bad shuffle server node and the connection creation timeout is very high then both these tasks end up waiting a long time time eventually impacting the performance.

Does this PR introduce any user-facing change?
Yes. This PR introduces client-side configs for push-based shuffle. If push-based shuffle is turned-off then the users will not see any change.
How was this patch tested?
Added unit tests.
The reference PR with the consolidated changes covering the complete implementation is also provided in SPARK-30602.
We have already verified the functionality and the improved performance as documented in the SPIP doc.
Lead-authored-by: Min Shen<EMAIL_ADDRESS>Co-authored-by: Chandni Singh<EMAIL_ADDRESS>Co-authored-by: Ye Zhou<EMAIL_ADDRESS>

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@Victsm @mridulm @tgravescs @jiangxb1987 @attilapiros @Ngone51 Please take a look.

ok to test

Test build #130957 has started for PR 30312 at commit c658423.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35563/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35563/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35563/
Test PASSed.

Test build #130957 has finished for PR 30312 at commit c658423.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130957/
Test FAILed.

retest this please

Test build #131004 has started for PR 30312 at commit c658423.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35611/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35611/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35611/
Test FAILed.

Test build #131010 has started for PR 30312 at commit 9029993.

Test build #131013 has started for PR 30312 at commit 1c43ac0.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35616/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35616/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35616/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35619/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35619/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35619/
Test FAILed.

Test build #131004 has finished for PR 30312 at commit c658423.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131004/
Test FAILed.

Test build #131010 has finished for PR 30312 at commit 9029993.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131010/
Test FAILed.

Test build #131013 has finished for PR 30312 at commit 1c43ac0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ShufflePushBlockId(shuffleId: Int, mapIndex: Int, reduceId: Int) extends BlockId

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131013/
Test FAILed.

Test build #131242 has started for PR 30312 at commit 70844d1.

Test build #131243 has started for PR 30312 at commit a00bc38.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35846/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35846/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35846/
Test FAILed.

Test build #131242 has finished for PR 30312 at commit 70844d1.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #131243 has finished for PR 30312 at commit a00bc38.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class PushRequest(

Test build #131263 has started for PR 30312 at commit d41b34c.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131243/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131242/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35866/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35866/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35866/
Test PASSed.

Test build #131263 has finished for PR 30312 at commit d41b34c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131263/
Test PASSed.

Test build #131366 has started for PR 30312 at commit 35c4cf8.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35970/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35970/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35970/
Test PASSed.

Test build #131366 has finished for PR 30312 at commit 35c4cf8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131366/
Test FAILed.

Since #30164 is merged, can you update this PR and change remove WIP @otterc ? Thx !

Test build #131436 has started for PR 30312 at commit 18d4b93.

Test build #131437 has started for PR 30312 at commit 7c13c01.

Since #30164 is merged, can you update this PR and change remove WIP @otterc ? Thx !

Thanks @mridulm. I have updated the PR. @tgravescs @attilapiros @Ngone51 @Victsm Please take a look when you get a chance.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36042/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36043/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36042/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36042/
Test FAILed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36043/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36043/
Test PASSed.

Test build #131436 has finished for PR 30312 at commit 18d4b93.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131436/
Test FAILed.

Test build #131437 has finished for PR 30312 at commit 7c13c01.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131437/
Test FAILed.

Test build #131478 has started for PR 30312 at commit b1e7501.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36084/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36084/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36084/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131478/
Test FAILed.

Test build #131602 has started for PR 30312 at commit 002cb69.

Test build #131602 has finished for PR 30312 at commit 002cb69.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131602/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36203/

Test build #131795 has started for PR 30312 at commit faa55f3.

Test build #131809 has started for PR 30312 at commit 40fbafc.

Test build #131795 has finished for PR 30312 at commit faa55f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131795/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36393/

Test build #131809 has finished for PR 30312 at commit 40fbafc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131809/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36408/

Test build #132068 has started for PR 30312 at commit bd0c427.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36667/

Test build #132068 has finished for PR 30312 at commit bd0c427.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132068/

Current version already looks good to me.
For the changes needed for #30312 and #30513, I think we can address them outside of this PR in the other 2 PRs respectively if we are close on merging this PR.

Test build #132152 has started for PR 30312 at commit 27e7e2e.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36753/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36753/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36753/

Test build #132152 has finished for PR 30312 at commit 27e7e2e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132152/

Test build #132188 has started for PR 30312 at commit dce53ea.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36790/

Test build #132188 has finished for PR 30312 at commit dce53ea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132188/

Test build #132204 has started for PR 30312 at commit bc7eaa2.

LGTM. I left one last minor comment and please also don't forget to address https://github.com/apache/spark/pull/30312/files#r535438604.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36805/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36805/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36805/

Test build #132204 has finished for PR 30312 at commit bc7eaa2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132204/

Test build #132241 has started for PR 30312 at commit 6d0fade.

I have addressed all the review comments so far.
cc @Ngone51 @Victsm @mridulm @dongjoon-hyun

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36842/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36840/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36842/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36842/

Test build #132241 has finished for PR 30312 at commit 6d0fade.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132241/

ok to test

Test build #132252 has started for PR 30312 at commit 6d0fade.

The master branch is under investigation for those failures. It happens Jenkins sometimes and GitHub Action more frequently (almost always) for now. I'm still not sure about the root cause.

(cc @MaxGekk )

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36853/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36853/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36853/

Test build #132252 has finished for PR 30312 at commit 6d0fade.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132252/

LGTM again. Thanks for addressing my last 2 comments.

@otterc Could you address @mridulm 's comments? I think we're almost close to merge.

@otterc Could you address @mridulm 's comments? I think we're almost close to merge.

Yes, apologies for the delay. I will address them by tomorrow.

Test build #132918 has started for PR 30312 at commit 23cc502.

Test build #132918 has finished for PR 30312 at commit 23cc502.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132918/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37522/

Test build #132924 has started for PR 30312 at commit bcebb13.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37527/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37527/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37527/

Test build #132924 has finished for PR 30312 at commit bcebb13.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132924/

Test build #133044 has started for PR 30312 at commit 6aae02a.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37644/

Test build #133044 has finished for PR 30312 at commit 6aae02a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133044/

Test build #133049 has started for PR 30312 at commit 21ea881.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37649/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37649/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37649/

Test build #133049 has finished for PR 30312 at commit 21ea881.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133049/

Have addressed all the comments so far. The failing test seems unrelated.
cc @Ngone51 @Victsm @mridulm

Have addressed all the comments so far. The failing test seems unrelated.
cc @Ngone51 @Victsm @mridulm

ok to test

ok to test

Given no other comments, will merge once tests pass (have retriggered it).

Given no other comments, will merge once tests pass (have retriggered it).

Test build #133803 has started for PR 30312 at commit 21ea881.

Test build #133803 has started for PR 30312 at commit 21ea881.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38392/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38392/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38392/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38392/

test this please

test this please

Test build #133807 has started for PR 30312 at commit 21ea881.

Test build #133807 has started for PR 30312 at commit 21ea881.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38396/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38396/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38396/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38396/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38396/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38396/

Test build #133807 has finished for PR 30312 at commit 21ea881.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133807 has finished for PR 30312 at commit 21ea881.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133807/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133807/

Merged to master.
Thanks for working on this @otterc !
And thanks for all the reviews and comments @Ngone51, @dongjoon-hyun, @Victsm

Merged to master.
Thanks for working on this @otterc !
And thanks for all the reviews and comments @Ngone51, @dongjoon-hyun, @Victsm

Thanks @mridulm for merging and also reviewing. Thanks @Ngone51, @dongjoon-hyun, and @Victsm for the reviews as well.

Thanks @mridulm for merging and also reviewing. Thanks @Ngone51, @dongjoon-hyun, and @Victsm for the reviews as well.

Thank you, @otterc and @mridulm and all!

Thank you, @otterc and @mridulm and all!


feat: add support for pi5 firmware

Add support for loading pi5 firmware via the include_firmware option.
Included the firmware by adding the following to your rugpi-bakery.toml file:
include_firmware = "pi5"

Looks good. Thanks!


PropertyDDS Rebase Issue

Describe the bug
We document below the behavior (^0.57.0) that a PropertyDDS client (C1) is loosing pending (uncommitted) changes in case of competing clients (C2) modifying in parallel the same state. Notable is also the inconsistency in behavior - while first C2 commit was not reflected by C1 bound/displayed state, second C2 change & commit modifies the C1 state.
The fundamental question to answer is what should be the expected behavior when one of the clients (C1) has pending local changes whereas another client (C2), starting the transaction/updates after C1 is committing earlier than C1.
We believe that the correct client transactional model should reflect the snapshot isolation policy. That is queries are seeing a consistent and immutable snapshot of the local data (captured when the transaction has been started). Coupled with a traditional LWW conflict resolution policy we believe that the last transaction (client) to commit the changes should win the race.
To Reproduce
A reproducer is created in a forked repository in the examples section of the experimental PropertyDDS.
Steps to reproduce the behavior:
Clone Forked Repository
git clone git@github.com:dstanesc/FluidFramework.git FluidFramework.Debug

Build Forked Repository
cd FluidFramework.Debug
git switch pdds-uncommitted-behavior
npm run clean
npm install
npm run build:fast
alias fb='clear && node "$(git rev-parse --show-toplevel)/node_modules/.bin/fluid-build"'
fb --all @fluid-experimental/prop0

Usage
Terminal 1
npx tinylicious

Terminal 2
cd FluidFramework.Debug/experimental/PropertyDDS/examples/prop0
npm start

The default browser will open to http://localhost:8080/.
Click on the red number to update the value. Click on the green commit button to commit the value.
Current Behavior

Client C1 modifies the dice value. No commit.
Client C2 modifies the dice value and commits. The commit is reflected locally (C2) while the C1 state remains unchanged.
Client C1 commits. C1 local state reflects intended C1 change. C2 local state remains unchanged. Opening a third client (C3) would prove that the actual state of DDS is the one of C2 while C1 seems out of synch.
Client C2 modifies the dice value once again and commits. This time C1 state is updated with the most recent state from C2, also meaning C1 pending changes simply vanish. Notable is also the inconsistency in behavior - while first C2 commit was not reflected by C1 state update, second C2 commit actually modifies current C1 state.

A video documenting the current behavior is provided
Expected behavior
Following above reasoning (Describe the bug section), the expected behavior is:

Client C1 modifies the dice value. No commit.
Client C2 modifies the dice value and commits. The commit is reflected locally (C2) while the C1 state remains unchanged.
Client C1 commits. C1 local state reflects intended C1 change. C2 local state is updated to C1 state.

Debugging Insights
The investigation points to FluidFramework/experimental/PropertyDDS/packages/property-dds/src/propertyTree.ts. This seems to be a rebase problem. The method _applyRemoteChangeSet#565 is the last one invoked when handling the remote changes. Reads the pending changes and  reconciles them with the remote changes: local changes win. After the rebase, the pending changes are cleared up. The commit consequently cannot see local changes anymore, they are applied only locally.
Relevant code sections:
commit#196
const changes = this._root._serialize(true, false, BaseProperty.MODIFIED_STATE_FLAGS.PENDING_CHANGE);

_applyRemoteChangeSet#565
const pendingChanges = this._root._serialize(true, false, BaseProperty.MODIFIED_STATE_FLAGS.PENDING_CHANGE);
...
const changesNeeded = this.rebaseLocalChanges(change, pendingChanges, changesToTip);
...
this._root.cleanDirty(BaseProperty.MODIFIED_STATE_FLAGS.PENDING_CHANGE);

Note
The current findings build upon longer lived edit sessions and manual commits. It is our belief however that same behavior will manifest, this time as more difficult to detect race conditions, even with shorter edit sessions and automated commits.

Further debug insight.
As mentioned above, in the method :
FluidFramework/experimental/PropertyDDS/packages/property-dds/src/propertyTree.ts
_applyRemoteChangeSet#565

there is the cleanup of pending changes by calling :
this._root.cleanDirty(BaseProperty.MODIFIED_STATE_FLAGS.PENDING_CHANGE);
However, after this call there is reapply of the previous pending changes by
this._root.applyChangeSet(pendingChanges);
The execution leads to
experimental/PropertyDDS/packages/property-properties/src/properties/stringProperty.js _setValue#361
(or to any other primitive property, in the dice case, it might be some number property)
This method, adds the pending operation. However it checks, whether value changed.
var oldValue = this._dataArrayRef;
var castedValue = String(in_value);
 var changed = castedValue !== oldValue;
 if (changed) {...
The pending changes were previously applied in the _applyRemoteChangeSet#565 method at the call this._root.applyChangeSet(changesToTip); so they do not change.
That is why they also do not repopulate the _pendingChanges within the _root tree and the commit will not find them.
The further exploration could lead to an option to reapply value even if it did not change (possibly enabled by flag passed through the whole stack)

Thanks for the detailed description, we're having a look!

This sounds like a bug.
The expected behaviour you describe is correct. The C2 state should be updated.
Thanks for posting this and preparing the detailed repo case. We will investigate the bug.

Hello @ruiterr
We have an experimental fix, the changes can be found in
https://github.com/dstanesc/FluidFramework/commit/d5a8ffeff3d1c8bb4ff6947db4cb37db69f080ef
The original reproducer with this fix can be found here
https://github.com/dstanesc/FluidFramework/tree/fix-test-pdds-uncommitted-behavior/experimental/PropertyDDS/examples/prop0
Meanwhile we discovered another race condition issue between commits concerning concurrent inserting the same key into the MapProperty.  We will create ticket once we have more details.

Hi @dstanesc  @milanro
Thank you again for reporting the issue and for investigating it.
We have a fix for it in this PR, it may as well fix the issue you are having with MapProperty.

@nedalhy Thank you for the fix. I tested it and it works in my test cases.
The other problem still persist but I will create another ticket for it. It will need more complex test case so it will take some time.
Just preliminary info is that when we add into map (which in my case is under path W_0.0249997558655648".words) the same key ("1") on both clients and then commit on both clients, and then change the sub-attribute (string property) W_0.0249997558655648".words[1].person on 1st client and commit, it will broadcast to second client but then change on the second client of that sub-attribute and commit leads to exception bellow. The in_context.getPropertyContainerType() is "template" and in_context.getUserData().parallelState is undefined.
react-dom.development.js:327 Uncaught Error: PR-003: Invalid path in ChangeSet: "W_0.0249997558655648".words[1].person. Making primitive value reversible.
at _recursivelyBuildReversibleChangeSet (main.531681f2e492d4b3681d.js:161279:27)
at _traverseChangeSetRecursively (main.531681f2e492d4b3681d.js:167256:13)
at processChange (main.531681f2e492d4b3681d.js:167319:13)
at _traverseChangeSetRecursively (main.531681f2e492d4b3681d.js:167445:25)
at processChange (main.531681f2e492d4b3681d.js:167319:13)
at _traverseChangeSetRecursively (main.531681f2e492d4b3681d.js:167432:25)
at processChange (main.531681f2e492d4b3681d.js:167319:13)
at _traverseChangeSetRecursively (main.531681f2e492d4b3681d.js:167445:25)
at processChange (main.531681f2e492d4b3681d.js:167319:13)
at _traverseChangeSetRecursively (main.531681f2e492d4b3681d.js:167432:25)

@nedalhy Confirm that https://github.com/nedalhy/FluidFramework/commit/15f6542bdd6651a3c3f831e1187979fa478c94a9 fixes the bug reproducer. Thank you!


New metadata file not correctly associated with the EBSD (texture) experiment

@merrygoat has added a new metadata file named ebsd-(texture).yml, which should be automatically associated with the EBSD (texture) experiment, but it is not. For example, on the Experiments home page, there is a link to add the metadata file, when there should be a link to edit the metadata file:

I added it through the web interface using the add button but on the first attempt, was was not able to save my edited page due to not having authenticated the wiki addin to access my github. Once I had authenticated it I then added it successfully through the web interface.

Thanks for the info @merrygoat. This was an issue with parentheses in the metadata file names. The authentication thing with Prose is a bit annoying but a separate issue. It's not particularly intuitive, but using Prose.io seem to be the best available solution at the moment. The worst that can happen if you don't authenticate is that instead of directly committing you end up submitting a pull request, which you then have to approve.
Fixed with 326386cf461ab966910f3fe9c2fdb79eb2e4bf20 and 9bf348450a4f1032bd72618ee459fca76f63a9ee.


Switching nighttime theme twice moves you to last comic

Where exactly does that happen? Can you share the steps to reproduce this?

Go to any comic that is not the last
Select "night theme" in the overflow menu in appbar
Select again "night theme" in the overflow menu in appbar
You are at the last comic

Ah right, I didn't read the "twice", sorry!

Fixed in 8.0


ZMS cannot start after DDL update

SQL column name does not match. ZMS cannot start up.
https://github.com/yahoo/athenz/blob/master/servers/zms/schema/zms_server.sql#L54

I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.

Thanks @WindzCUHK. We noticed the same failure in our functional test pipeline as well, but your PR came first.


What is best practice for my application state: is MonadReader (Dynamic t State) m OK?

This is a question regarding a best practice. If there's a better place to ask this kind of stuff, I will post there and close this here.
I have an application state for my frontend:
data State = State
  { stSession :: SessionData
  , stMsg :: Maybe Message
  , stShowTOCStage1 :: Bool
  -- ...
  , stRedirectUrl :: R FrontendRoute
  }

It contains all kinds of stuff: session data, messages, and more detailed things like whether or not the table of contents of my page is currently shown in full.
Currently I make the application state available like this:
    dynState <- foldDyn appEndo defaultState $ leftmost [eLoaded, eStateUpdate]

    eUpdated <- tailE $ updated dynState

    (_, eStateUpdate) <-
        mapRoutedT (runEventWriterT . flip runReaderT dynState) $ do
            message

            subRoute_ $ \case
                FrontendRoute_Main -> -- ...

I copied this appraoch from here. Note that in the original code there is
newtype FrontendStateT t s m a = FrontendStateT
  { unFrontendStateT :: ReaderT (Dynamic t s) m a }

where I simply use ReaderT (Dynamic t State) m.
The approach is quite neat and allows me to write frontend pages like this:
signup
    :: forall t (m :: * -> *)
     . ( EventWriter t (Endo State) m
       , MonadReader (Dynamic t State) m
       -- ...
       )
    => m ()
signup = elClass "div" "auth" $ mdo
    dynState <- ask
    dyn_ $ dynState <&> \State{..} ->
        -- do something, using the application state ...

   -- ... or update the application state
    let updateState event = tellEvent $ foldMap Endo <$> event
    updateState
        $  evUserExists
        $> [ field @"stApp" . field @"stMsg" ?~ Message
                 "Error"
                 "username already exists"
           ]

I keep stumbling over the fact that this way my State is inside a Dynamic. This makes sense when looking at my app as a whole, but it seems awkward when I write within the signup page dyn_ $ dynState <&> \State{..} -> -- .... I will try and explain the  awkwardness with this in the end of this post.
An alternative would be to wrap my whole app into a dyn like this:
    dynState <- foldDyn appEndo defaultState $ leftmost [eLoaded, eStateUpdate]

    eUpdated <- tailE $ updated dynState

    evEvStateUpdate <- dyn $ dynState <&> \st -> mapRoutedT (fmap snd . runEventWriterT . flip runReaderT st) $ do
            message

            subRoute_ $ \case
                FrontendRoute_Main -> do
                    -- ...
    eStateUpdate <- switchHold never evEvStateUpdate
    blank

An intermediate appraoch would be to wrap the signup function into a dyn.
So my questions:

How does either appraoch affect performance?
Are there other important considerations?

Thanks.

And for completeness sake: the Dynamic t State in the Reader also gets me confused from time to time.
I am only starting thinking in the functional-reactive style and then there's haskell and I occasionally keep falling into some imparative thinking.
To clear things up: a Dynamic t State in a Reader just gives access to that dynamic. It doesn't mean something depends on State yet. My haskell functions that pass around dynamics and events only build the functional-reactive dependency graph.
There are exceptions though: Things like subRoute, dyn, and widgetHold really mean: depending on a dynamic or an event (or a dynamically changing route) some code gets executed and other code doesn't.


[BUG]: Facets disappear in model explorer

Describe the bug
When visiting the model explorer page, the list of facets shows up but then disappears, leaving an empty Facets sidebar.
To Reproduce
Steps to reproduce the behavior:

Go to https://app.terarium.ai/app/#/
Click on Home -> Explorer -> Models

Expected behavior
Facets should not disappear
Additional context
The problem seems to goe away when reloading the page or navigating away and back again.

Problem no longer exists so closing.


added service interface and main.py to sembench

This PR will solve the following issues

#34
#33
#31
#29
#26
#23
#69

Summary by CodeRabbit
Release Notes

New Features

Introduced a command-line interface (CLI) for the Sema benchmarking tool, allowing users to specify configurations and manage execution parameters easily.
Added new CLI commands: sema-bench, sema-harvest, sema-query, and sema-syncfs for enhanced functionality.

Bug Fixes

Improved error handling and success state tracking in the synchronization process.
Enhanced error handling in the CLI to ensure required inputs are validated.

Tests

Added unit tests for CLI argument parsing and help functionality to ensure robustness and correctness.
Enhanced test coverage for the discover_subject function and CLI commands.
Updated tests to reflect changes in function signatures and argument handling.

apparently some tests are failing...

apparently some tests are failing...

This is due to the changes in the ./tests/conftest.py file , changing localhost to <IP_ADDRESS>


Beef up test for issue 200

This PR adds more tests to cover edge cases where we may run into #200.

bors r+


Celery beat worker fails to import model / receiving mails broken

Celery or our production configuration are having an issue with our new module setup when receiving mails. The simulation job works fine.
Dec 09 07:54:45 leeway celery[64771]: [2022-12-09 07:54:45,644: INFO/MainProcess] Task leeway.celery.check_mailbox[54bd3f5d-8c62-4473-ab8e-df38492b8976] received
Dec 09 07:54:46 leeway celery[64780]: [2022-12-09 07:54:46,075: ERROR/ForkPoolWorker-3] Task leeway.celery.check_mailbox[54bd3f5d-8c62-4473-ab8e-df38492b8976] raised unexpected: RuntimeError("Model class leeway.models.LeewaySimulation doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.")
Dec 09 07:54:46 leeway celery[64780]: Traceback (most recent call last):
Dec 09 07:54:46 leeway celery[64780]:   File "/opt/leeway/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
Dec 09 07:54:46 leeway celery[64780]:     R = retval = fun(*args, **kwargs)
Dec 09 07:54:46 leeway celery[64780]:   File "/opt/leeway/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
Dec 09 07:54:46 leeway celery[64780]:     return self.run(*args, **kwargs)
Dec 09 07:54:46 leeway celery[64780]:   File "/opt/leeway/opendrift_leeway/leeway/celery.py", line 34, in check_mailbox
Dec 09 07:54:46 leeway celery[64780]:     mail_to_simulation(email.message_from_bytes(data[0][1]))
Dec 09 07:54:46 leeway celery[64780]:   File "/opt/leeway/opendrift_leeway/leeway/utils.py", line 178, in mail_to_simulation
Dec 09 07:54:46 leeway celery[64780]:     from .tasks import run_leeway_simulation
Dec 09 07:54:46 leeway celery[64780]:   File "/opt/leeway/opendrift_leeway/leeway/tasks.py", line 10, in <module>
Dec 09 07:54:46 leeway celery[64780]:     from .models import LeewaySimulation
Dec 09 07:54:46 leeway celery[64780]:   File "/opt/leeway/opendrift_leeway/leeway/models.py", line 11, in <module>
Dec 09 07:54:46 leeway celery[64780]:     class LeewaySimulation(models.Model):
Dec 09 07:54:46 leeway celery[64780]:   File "/opt/leeway/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 132, in __new__
Dec 09 07:54:46 leeway celery[64780]:     raise RuntimeError(
Dec 09 07:54:46 leeway celery[64780]: RuntimeError: Model class leeway.models.LeewaySimulation doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

When setting the app_label in the model Meta class, the error changes to:
Dec 09 08:08:57 leeway celery[65800]: [2022-12-09 08:08:57,639: ERROR/ForkPoolWorker-3] Task leeway.celery.check_mailbox[d8c5a067-3dd2-4378-9873-ae6be1c7655d] raised unexpected: RuntimeError("Conflicting 'leewaysimulation' models in application 'opendrift_leeway.leeway': <class 'opendrift_leeway.leeway.models.LeewaySimulation'> and <class 'leeway.models.LeewaySimulation'>.")
Dec 09 08:08:57 leeway celery[65800]: Traceback (most recent call last):
Dec 09 08:08:57 leeway celery[65800]:   File "/opt/leeway/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
Dec 09 08:08:57 leeway celery[65800]:     R = retval = fun(*args, **kwargs)
Dec 09 08:08:57 leeway celery[65800]:   File "/opt/leeway/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
Dec 09 08:08:57 leeway celery[65800]:     return self.run(*args, **kwargs)
Dec 09 08:08:57 leeway celery[65800]:   File "/opt/leeway/opendrift_leeway/leeway/celery.py", line 34, in check_mailbox
Dec 09 08:08:57 leeway celery[65800]:     mail_to_simulation(email.message_from_bytes(data[0][1]))
Dec 09 08:08:57 leeway celery[65800]:   File "/opt/leeway/opendrift_leeway/leeway/utils.py", line 177, in mail_to_simulation
Dec 09 08:08:57 leeway celery[65800]:     from .models import LeewaySimulation
Dec 09 08:08:57 leeway celery[65800]:   File "/opt/leeway/opendrift_leeway/leeway/models.py", line 11, in <module>
Dec 09 08:08:57 leeway celery[65800]:     class LeewaySimulation(models.Model):
Dec 09 08:08:57 leeway celery[65800]:   File "/opt/leeway/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 364, in __new__
Dec 09 08:08:57 leeway celery[65800]:     new_class._meta.apps.register_model(new_class._meta.app_label, new_class)
Dec 09 08:08:57 leeway celery[65800]:   File "/opt/leeway/.venv/lib/python3.9/site-packages/django/apps/registry.py", line 234, in register_model
Dec 09 08:08:57 leeway celery[65800]:     raise RuntimeError(
Dec 09 08:08:57 leeway celery[65800]: RuntimeError: Conflicting 'leewaysimulation' models in application 'opendrift_leeway.leeway': <class 'opendrift_leeway.leeway.models.LeewaySimulation'> and <class 'leeway.models.LeewaySimulation'>.

https://stackoverflow.com/questions/57666355/cannot-import-models-into-celery-tasks-in-django


Show software version in app

Description
Hello,
Would it be possible to display the current software version in the app?
Suggested implementation
No response
Additional information
No response
New request

[X] I have checked that a similar feature hasn't been requested yet

I'm planning on doing this. There will be an "about NeoStumbler" screen, which will include version info, links to this repo etc.


Regarding "Arabic and Hebrew languages are not supported on the Power Pages preview."

can you please clarify/correct where exactly Hebrew is not supported?
With Power Portals being renamed to Power Pages, this is difficult to follow
but
a) Power Pages is no longer in preview
b) on make.powerpages Hebrew is not offered as a language, even if Hebrew is installed on the org,
so the "not supported" statement should probably be
"Arabic and Hebrew languages are not supported on make.powerpages yet"

Thank you for your feedback.
We have recently updated our documentation to indicate that Arabic and Hebrew languages are not supported when creating a site using Power Pages or using the Power Pages design studio.
You will need to create and edit Arabic and Hebrew language sites using Power Apps and the legacy portals Studio.
See https://learn.microsoft.com/en-us/power-apps/maker/portals/configure/enable-multiple-language-support#supported-languages
Closing this issue.


fix possible data race with failure count

Signed-off-by: Victor Vieux<EMAIL_ADDRESS>

ping @dongluochen

Line 388:
     log.WithFields(log.Fields{"name": e.Name, "id": e.ID}).Infof("Engine came back to life after %d retries. Hooray!", e.failureCount)

@dongluochen PTAL

LGTM

LGTM


Add support for cmd_vel teleoperation

In Crazyswarm(1), no teleoperation (control using a joystick) was ever implemented. Now that Crazyflie_ros is deprecated, this feature should be part of Crazyswarm2.
Steps:

Update the teleop node to publish cmd_vel (feature to be enabled using a parameter). Examples are https://github.com/whoenig/crazyflie_ros/blob/master/crazyflie_demo/src/quadrotor_teleop.cpp and https://github.com/ethz-asl/rotors_simulator/blob/master/rotors_joy_interface/src/joy.cpp
Update the crazyswarm_server to subscribe to cmd_vel and to send the appropriate command to the Crazyflie, see https://github.com/whoenig/crazyflie_ros/blob/df3ce76f700953cfa2794b8fd4628ea6b93c92c4/crazyflie_driver/src/crazyflie_server.cpp#L310-L322

Interesting! I was working on doing the gazebo plugin to do the same for the simulator. Perhaps I can take a look at this.
I guess the idea is to actually make an separate package out of this right?

I actually created this one for my PhD student to learn a bit more about ROS2 and she already started:-)


Javascript filter return true when the expression result is false

Description

When use javascript filter below, the result will contains all unit_id the data source has, like the js function always return true.

{
  "type" : "javascript",
  "dimension" : "unit_id",
  "function" : "function(unit_id) { return ['89793647', '89794140'].includes(unit_id); }"
}

But change the filter like below, the result is correct, only contains two unit_id in the js function.

{
  "type" : "javascript",
  "dimension" : "unit_id",
  "function" : "function(unit_id) { return ['89793647', '89794140'].includes(unit_id) == true; }"
}

It seems standard ECMA.
toBoolean


Allow arguments name on ambient class method declaration

TypeScript Version:  2.2.1
Code
// A *self-contained* demonstration of the problem follows...
declare class HTMLCanvasElement {
    toBlob(callback: (result: Blob | null) => void, type?: string, ...arguments: any[]): void;
}

Expected behavior: The name arguments on ambient class declarations does not really do anything, it should just be allowed.
Actual behavior: This is blocked.

There already are tests for this e.g. collisionArgumentsInterfaceMembers but it seems they do not work as expected.


Does not work after update to 2023.08

Dees anyone work in this version (after update to 2023.08)? Before update was fine.
My SPA is PureSpa Greywood Deluxe v22111, firmware 1.0.300. Intex aplication is working, spa is online.
Edit
My mistake, it was a filter error causing the disconnect (E90). I'm sorry:/

Thank you for updating, enjoy your spa !


Prefilled setting for SSHAgentTask

Can we add an empty setting if you choose for SSHAgentTask?
Like RepositoryCheckoutTask, if you choose this one you also get "repositoryUrl": "" in your accompli json. If you choose SSHAgentTask you might want to add:
"keys": [
],

Do you mean with the new init command?
The init command currently only adds the required parameters to the configuration.

Yes, isn't the keys a required parameter for the SSHAgentTask?
I'm actually asking if you want a PR of this or if you don't think it's a good idea....

Changing 'keys' to a required parameter would make sense as without any keys the task is pretty useless.
I'd very much like a PR of that. :smiley:


Added support for obj/mtl to drake-visualizer.

This contains the changes to Director needed to resolve https://github.com/RobotLocomotion/drake/issues/5044.

This change is

Looks pretty good, thanks!  By the way, as I discussed last week in the slack channel, the CDash server for director is down, so CI is currently failing when it attempts to submit the test results.  I will try to get that resolved today.
I think you design might be working fine, but an alternative to consider is rather than making the logic split at createGeometry and createGeometryFromObj, you could have a single createGeometry function and put the logic split in the lower function loadPolyDataMeshes.  This function could be changed to return a list of (mesh,color,alpha,texture) tuples.
A problem with putting the split higher as you have done is that you lose some of the operations in the lower functions, like applying scaling and transforms, and the caching feature being introduced in https://github.com/RobotLocomotion/director/pull/493.

Thanks for the review! I will refactor as suggested.

For non-obj meshes, they could return None for color and alpha, in which case the values from the geometry message are used.  For obj with mtl, you have to decide whether you want to use the color from the .mtl file, or the color from the URDF (geometry message).

Also, as currently implemented, ioUtils.readObjMtl() does not make it obvious whether the returned actor's color was specified from a .mtl file, or whether the actor color is just the default color (perhaps because there was no .mtl file associated with the .obj file)

Glad to see this.  It sounds like the Drake doc (http://drake.mit.edu/models.html#texture-file-format) will have to be updated with this change. (That doc describes drake-visualizer too).  Will drake-visualizer continue to support  VTM/VTP texture files, as it does now?

Yes, vtm files should continue to be supported.  I think this feature as implemented is currently breaking support for some models, because it now preferences obj files over vtm.  But we have several models that have vtm files generated from dae+textures, and obj without mtl, and so the vtm files should be preferred.

@patmarion, please check r3 of this PR.  I still have to polish it some more but wanted to check whether it's going in the right direction. Thanks!

Review status: 0 of 1 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

definitely headed in the right direction, but the implementation has gotten really messy.  I don't think the functions should have multiple list return values.  It's time to return a list of structs.  Director has a very convenient class for this.  Try something like:
from director.fieldcontainer import FieldContainer

...

result = []

# load mesh and color and texture data
for ...:
    f = FieldContainer(polyData=polyData, color=color, alpha=alpha, texture=texture)
    result.append(f)

return result

Now you can access the fields by name:
f = result[0]
f.polyData
f.color
f.texture

Thanks @patmarion. In r5 I switched to use FieldContainer as suggested though I believe it still needs quite a bit of polishing. PTAL?

Review status: 0 of 1 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

I completed a round of polishing in r6.

Review status: 0 of 1 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

i read through it, thanks for working on this!  I will leave some in-line comments.  Overall I think there are some simplifications that could be made.  The existing logic was already messy due to its history, and this patch makes it further complicated.  I might take a pass over this, if i have time, but we might be able to merge it sooner than that, after you get a chance to address a few more review comments.  Thanks!

Thanks for the review! I will address your comments ASAP.

@patmarion, I addressed your comments. PTAL.

Review status: 0 of 1 files reviewed at latest revision, 7 unresolved discussions.

src/python/director/drakevisualizer.py, line 113 at r6 (raw file):
Previously, patmarion (Pat Marion) wrote…
style: please use camelCase, not underscores.  Maybe we can name this variable meshDataList?  this comment applies for the whole diff.

Done.

src/python/director/drakevisualizer.py, line 127 at r6 (raw file):
Previously, patmarion (Pat Marion) wrote…
style: please don't wrap lines with \.  I don't enforce line length limits in these files.  this comment applies for the whole diff.

Done.

src/python/director/drakevisualizer.py, line 205 at r6 (raw file):
Previously, patmarion (Pat Marion) wrote…
style: camelCase method name please

Done.

src/python/director/drakevisualizer.py, line 250 at r6 (raw file):
Previously, patmarion (Pat Marion) wrote…
bug: this is not the correct way to determine whether an mtl file exists.  Also, we probably want all obj files to be loaded with the ioUtils.readObjMtl function, not just obj files with .mtl material files.

Done.

src/python/director/drakevisualizer.py, line 262 at r6 (raw file):
Previously, patmarion (Pat Marion) wrote…
style: don't use name=arg style arg list, just a regular positional arg list.  this comment applies to other areas in the diff, too

Done.

src/python/director/drakevisualizer.py, line 294 at r6 (raw file):
Previously, patmarion (Pat Marion) wrote…
loop could be a list comprehension

Done.

src/python/director/drakevisualizer.py, line 295 at r6 (raw file):
Previously, patmarion (Pat Marion) wrote…
maybe just pass the struct to the ctor instead of the individual of fields?

Done.

Comments from Reviewable

thanks for working on this further! sorry for dragging out the review, thanks for your patience.

Thank YOU for all of the rounds of reviews! No worries about the latency -- I'm always happy to learn how to improve the code.

Review status: 0 of 1 files reviewed at latest revision, 11 unresolved discussions.

src/python/director/drakevisualizer.py, line 215 at r8 (raw file):
Previously, patmarion (Pat Marion) wrote…
remove semi-colon

Done.

src/python/director/drakevisualizer.py, line 244 at r8 (raw file):
Previously, patmarion (Pat Marion) wrote…
i think i need to test this, but i think there is one more remaining bug, which is the case of urdf files that specify visuals with color and mesh obj files, and those obj files do not have associated mtl files.  In this case, the color specified in the urdf should be used.  I think we have urdf files like this.  But as implemented here, the urdf color will be ignored.

Good point.  I will also test this ASAP.

src/python/director/drakevisualizer.py, line 250 at r8 (raw file):
Previously, patmarion (Pat Marion) wrote…
remove spaces around =

Done.

src/python/director/drakevisualizer.py, line 251 at r8 (raw file):
Previously, patmarion (Pat Marion) wrote…
remove semi-colon

Done.

Comments from Reviewable

Don't forget to update the Drake doc. Good to see this.

Review status: 0 of 2 files reviewed at latest revision, 11 unresolved discussions.

Comments from Reviewable

Thanks for the reminder. I'll be sure to update Drake's Doc in the same PR that bumps Director's version in Drake.

Review status: 0 of 2 files reviewed at latest revision, 11 unresolved discussions.

Comments from Reviewable

I think this looks good, thanks!  I would like to do some manual testing myself prior to merging, since there isn't any good test coverage for a lot of these cases.  Hopefully I'll find some time to do that today and then I'll merge it.

Wonderful! Let me know if there are any additional fixes I can apply.

Is there any documentation for drake-visualizer that might mention what files it supports (or a banner comment in drake-visualizer, or something)?

Review status: 0 of 2 files reviewed at latest revision, 11 unresolved discussions.

Comments from Reviewable

Docs are very sparse.  User guide information goes here and there's always room to add high quality comments to the .py files.

So if it makes sense, it would be great to have a sentence or two  somewhere, for users of drake-visualizer, especially now that versions matter for what files you want to use.  Maybe if the help for drake-visualizer  could print out what files it supports (I'm thinking especially of the non-programmers who use it)?  Or at least a banner comment in drake-visualizer.py.  So when you run drake-visualizer  with no args, it prints help.   Again, really glad to see this PR.

Review status: 0 of 2 files reviewed at latest revision, 11 unresolved discussions.

Comments from Reviewable

Drake users will probably not use drakevisualizer.py directly. Instead, they'll use a C++ executable called drake-visualizer that is a wrapper around the Python script (it's implemented in director/src/app/drakeVisualizerApp.cpp). Because of this, I'm hesitant to place the documentation within drakevisualizer.py, and will probably include it in Drake's drake/doc/models.rst document.

Review status: 0 of 2 files reviewed at latest revision, 11 unresolved discussions.

Comments from Reviewable


Use with geocoder or searchkick ?

Hello !
Do you know if there is a way to use your gem with the geocoder or searchkick gem ?
I'm trying to find Account records near to a latitude/longitude.
Do you have any idea to help me ?
Thanks !

I didn't try yet. but let me check it. The short answer is no for now.
I think the geocoder has a function to work with sqlite that don't use any of the sql function to calculate distances. May be you could try to set sqlite on geocoder and try it out.
Please, let me know if you fond something.

I found this : http://www.salesforce.com/us/developer/docs/soql_sosl/Content/sforce_api_calls_soql_geolocate.htm
It could be my answer, i test it. :)

It works with geolocation fields and salesforce DISTANCE query.
Thanks for your help :)


MINIFICPP-1736 Add MiNiFi agent C2 operation capabilities to agent manifest

Added MiNiFi supported C2 operation data to the agent manifest in the following format:
{
  ..
  "agentInfo": {
      "agentManifest": {
          ...
          "supportedOperations": [
              {
                  "properties": {
                      <OPERAND1>: {},
                      <OPERAND2>: {
                          <METADATA_KEY>: [
                            <METADATA_VALUE_OBJECTS>
                          ]
                      }
                  },
                  "type": <OPERATION>
              }
          ],
          ...
      },
      ...
  }
}

https://issues.apache.org/jira/browse/MINIFICPP-1736

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically main)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file?
[ ] If applicable, have you updated the NOTICE file?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.

The JSON structure of the supported operations was simplified in 9a29abe81ce6b918dcb1b9f8924117dc19a2be59, with some additional bugfixes/new features of the serialization:

removed merging of parent-child nodes with the same name in HeartbeatJsonSerializer.cpp that would cause a bug of losing non-collapsible JSON nodes
added keep_empty flag to Value nodes signaling to not drop empty JSON objects or arrays while serializing


Adding a new environment variable in .env.development should instantly be available in functions

Right now a restart of all docker containers is required.

solved by https://github.com/nhost/functions/pull/11


Inconsistent output type for jnp.array([])

The output type of jnp.array([]) (input: empty list or tuple) is numpy.ndarray, inconsistent with the output type of jnp.array([1]) (or other non-trivial inputs) which is DeviceArray. This seems to be set explicitly here and seems an unnecessary inconsistency, as empty DeviceArrays seem generally supported (device_put(jnp.array([])) produces what one would expect).

This was just fixed in #3049.


no examples other than streetview

trying to get lookaround working.
But it is difficult, because an example is only showed in the lookaround repo, which also uses different variable names I believe?
Also I suspect my input coordinates are sometimes the problem?
51.50056270050847, 3.885667340737007 as an example will give errors, so maybe 8 digits are the max or something?
Would be nice if such a limit is needed, it would give an text error so it is clear right away or even better, that code does the rounding

trying to get lookaround working.
But it is difficult, because an example is only showed in the lookaround repo, which also uses different variable names I believe?

Yes, the lookaround repo is slightly different. You can find the documentation for the Look Around module of streetlevel here:
https://streetlevel.readthedocs.io/en/stable/streetlevel.lookaround.html
I'll add some sample code for these functions in a few days or so.

51.50056270050847, 3.885667340737007 as an example will give errors, so maybe 8 digits are the max or something?

I tried the coordinates, and they work fine for me:
In [1]: from streetlevel import lookaround

In [2]: panos = lookaround.get_coverage_tile_by_latlon(51.50056270050847, 3.885667340737007)

In [3]: print(panos[0].id, panos[0].lat, panos[0].lon)
11129494710789177858 51.50188556473774 3.8849992191090337

What is the error message you're getting?

No I used the "Downloading imagery" example of the lookaround repo.
from lookaround import get_coverage_tile_by_latlon, get_pano_face
from lookaround.auth import Authenticator

panos = get_coverage_tile_by_latlon(46.52943, 10.45544)

auth = Authenticator()
zoom = 2
for face in range(0, 6):
    image = get_pano_face(panos[0].panoid, panos[0].build_id, face, zoom, auth)
    with open(f"{panos[0].panoid}_{face}_{zoom}.heic", "wb") as f:
        f.write(image)

I have allready found some errors:
get_pano_face     >>     get_panorama_face
panoid     >>     id  (also in lookaround.py  lines 85, 139, 141, 143 ?)

but I still get this error:
    image = get_panorama_face(panos[0].id, panos[0].build_id, face, zoom, auth)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\_PDOK\MapServerZeeland\Repos\streetlevel\streetlevel\lookaround\lookaround.py", line 85, in get_panorama_face
    panoid, build_id = _panoid_to_string(pano)
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\_PDOK\MapServerZeeland\Repos\streetlevel\streetlevel\lookaround\lookaround.py", line 141, in _panoid_to_string
    panoid, build_id = str(pano[0]), str(pano[1])
                           ~~~~^^^
TypeError: 'int' object is not subscriptable

No I used the "Downloading imagery" example of the lookaround repo.

Well there's your problem - you can't just take code for module A and expect it to work for module B. The equivalent code for streetlevel looks like this:
from streetlevel.lookaround import get_coverage_tile_by_latlon, get_panorama_face
from streetlevel.lookaround.auth import Authenticator

panos = get_coverage_tile_by_latlon(46.52943, 10.45544)

auth = Authenticator()
zoom = 2
for face in range(0, 6):
    image = get_panorama_face(panos[0], face, zoom, auth)
    with open(f"{panos[0].id}_{face}_{zoom}.heic", "wb") as f:
        f.write(image)

Which you can simplify further with the download_panorama_face function:
from streetlevel.lookaround import get_coverage_tile_by_latlon, download_panorama_face
from streetlevel.lookaround.auth import Authenticator

panos = get_coverage_tile_by_latlon(46.52943, 10.45544)

auth = Authenticator()
zoom = 2
for face in range(0, 6):
    download_panorama_face(panos[0], f"{panos[0].id}_{face}_{zoom}.heic", face, zoom, auth)


Add doc comments to check_overflow and check_signed_overflow

Problem
These were introduced in this PR https://github.com/noir-lang/noir/pull/2713 . We should add doc comments to these functions
Happy Case
.
Alternatives Considered
No response
Additional Context
No response
Would you like to submit a PR for this Issue?
No
Support Needs
No response

Reopening - #3393 documents overflow in Noir, it does not document the check_overflow or check_signed_overflow functions within the compiler

Reopening - #3393 documents overflow in Noir, it does not document the check_overflow or check_signed_overflow functions within the compiler


Add integration for Contao CMS

I'd like to request an integration for the Contao Open Source Content Management System to track the time spent on working for a website project.
Contao Demo back end: https://demo.contao.org/contao/login
Maybe you can add this feature request to the list of user requested buttons, too. Thank you.

Hi,
I've added your request to the list of requested buttons
Cheers


Multi modal support?

Are we able to add Multi modal support? Ollama now supports this and it would be great to see it added to this very lovely UI I've just come across and started using 🙏 It would be great to be able to a) answer questions asked about images added to "documents" and b) generate new images from prompts.

Hi, feature a) has implemented a long time ago! you should just be able to drag and drop images to your chat input, make sure to use multimodal models like llava. As for the feature b), generating images is not within the scopes of multimodal models and Ollama does not support diffusion models and only supports LLMs afaik, but we'll consider adding it in the future (maybe via stable-diffusion-webui apis). Thanks!

Thanks I realized the same after I posted this! Sorry 🙏


[Bug] Database does not start due to Permission denied

Current Behavior
I added pvc with following settings

pvc:
  ## @param pvc.enabled Specify whether to use persistentVolumeClaim or EmptyDir storage
  enabled: true
  ## @param pvc.annotations [object] Annotations to add to the persistentVolumeClaim
  annotations: {}
  ## @param pvc.storage Storage size to request for the persistentVolumeClaim
  storage: 5Gi
  ## @param pvc.accessModes Access mode for the persistentVolumeClaim. See: [Access Modes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims)
  accessModes:
    - ReadWriteOnce
  ## @param pvc.selector [object] Selector for the persistentVolumeClaim. See: [Selector](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims)
  selector: {}
  ## @param pvc.storageClassName [string, nullable] The name of the StorageClass for the persistentVolumeClaim. See: [Class](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims)
  # storageClassName:
  storageClassName: "managed-csi"

and cryostat does not start, issue is with cryostat-db container -
❯ k logs cryostat-dbb678dd9-g7lzd -c cryostat-db
mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

in the deployment
      volumes:
      - name: cryostat
        persistentVolumeClaim:
          claimName: cryostat

and in cryostat-db container
        volumeMounts:
        - mountPath: /var/lib/pgsql/data
          name: cryostat
          subPath: postgres

I believe some file ownership should be changed ?
Expected Behavior
Cryostat start succesfully
Steps To Reproduce
No response
Environment
- OS:
- Environment:
- Version:

Anything else?
No response

Sorry, actually the bug that my colleague spotted last week was in the downstream equivalent container build, which works a bit differently from the upstream one. The root cause is therefore different and isn't actually the same bug, it's just that both manifest as '/var/lib/pgsql/data/userdata': Permission denied.
For this case, maybe it has to do with the security context and fsGroup of the container?
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
https://github.com/cryostatio/cryostat-helm/blob/61cf6a015f182c333b87a2ed385fc787746aee10/charts/cryostat/values.yaml#L93