Repeat that there is a problem with your work

Hello, I have this problem: The size of tensor a (128) must match the size of tensor b (0) at non-singleton dimension 1, how do I solve it

Hi, I'm using the latest version of the code, but I'm running it on windows. If I run main.py directly, do I need to add any instructions? The error code is as follows. thank you
`Found rdkit, all good
Dataset smiles were found.
E:\anaconda\envs\digress\lib\site-packages\torch\nn\init.py:405: UserWarning: Initializing zero-element tensors is a no-op
warnings.warn("Initializing zero-element tensors is a no-op")
Marginal distribution of the classes: tensor([0.7230, 0.1151, 0.1593, 0.0026]) for nodes, tensor([0.7261, 0.2384, 0.0274, 0.0081, 0.0000]) for edges
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [clclcl]:54216 (system error: 10049 - �����������У�������ĵ�ַ��Ч��).
[W ..\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [clclcl]:54216 (system error: 10049 - �����������У�������ĵ�ַ��Ч��).
[2023-07-14 15:41:57,862][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:1 to store for rank: 0
[2023-07-14 15:41:57,862][torch.distributed.distributed_c10d][INFO] - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
You are using a CUDA device ('NVIDIA GeForce RTX 4060 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Error executing job with overrides: []
Traceback (most recent call last):
File "E:\DiGress-main\src\main.py", line 202, in main
trainer.fit(model, datamodule=datamodule, ckpt_path=cfg.general.resume)
File "E:\anaconda\envs\digress\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 531, in fit
call._call_and_handle_interrupt(
File "E:\anaconda\envs\digress\lib\site-packages\pytorch_lightning\trainer\call.py", line 41, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "E:\anaconda\envs\digress\lib\site-packages\pytorch_lightning\strategies\launchers\subprocess_script.py", line 91, in launch
return function(*args, **kwargs)
File "E:\anaconda\envs\digress\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 570, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "E:\anaconda\envs\digress\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 951, in _run
self.strategy.setup(self)
File "E:\anaconda\envs\digress\lib\site-packages\pytorch_lightning\strategies\ddp.py", line 164, in setup
self.configure_ddp()
File "E:\anaconda\envs\digress\lib\site-packages\pytorch_lightning\strategies\ddp.py", line 269, in configure_ddp
self.model = self._setup_model(_LightningModuleWrapperBase(self.model))
File "E:\anaconda\envs\digress\lib\site-packages\pytorch_lightning\strategies\ddp.py", line 183, in _setup_model
return DistributedDataParallel(module=model, device_ids=device_ids, **self._ddp_kwargs)
File "E:\anaconda\envs\digress\lib\site-packages\torch\nn\parallel\distributed.py", line 657, in init
_sync_module_states(
File "E:\anaconda\envs\digress\lib\site-packages\torch\distributed\utils.py", line 136, in _sync_module_states
_sync_params_and_buffers(
File "E:\anaconda\envs\digress\lib\site-packages\torch\distributed\utils.py", line 154, in _sync_params_and_buffers
dist._broadcast_coalesced(
RuntimeError: The size of tensor a (128) must match the size of tensor b (0) at non-singleton dimension 1
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
`

I used to have the same question as you, check if your torch version is under 2.0.
The problem occurs when the output dim of y is set to 0, it seems that this is not working with torch version under 2.0.


survey fhl integation into current sample

What
First integration of FHL postcall into sample code.
The following is being worked upon -

Pass in config (remove separate fetchConfig api call within survey component) and bring in checks wether SurveyForm or Noticepage is to be shown

Why
Brainstorming on ways to implement "re-join call"

CLosing this DRAFT PR


Feature/unite project with team culture

[x] add CODEOWNERS file
[x] add .pylintrc file
[x] add Black check to worflow
[x] format files with Black

[ ]  pull
[ ]  code review
[ ]  test run - 80 passed, 2 skipped, 500 warnings

From the overall review, I have noted these points (as ideas only):

Data and data_validation could be placed in the tests folder where they are used. Then, no repo structure explanation is needed in README.
Why mention workflows? The README is for users. The developer will face it in the first PR.
Did you think about a command line argument parser to avoid enabling inner features by editing main.py file?
In README in How to run, I would expect sub-chapters describing how to reach variant: two variant of similarity and column2vec
DataShow looks like three features. I saw only one line in README.
I would add some duration value. Maybe an example of the output: 80 passed, 2 skipped, 500 warnings in 1887.83s (0:31:27)
Consider a more detailed table of contents for users looking for exact parts.

Data and data_validation could be moved in the future.
Workflow note is removed from README.md
Command line will be solved #26
How to run will be solved in #30
DataShow will be solved in #30
duration value added
details table of contents will be solved in #30

I saw

2 skipped tests
500 warnings

Are you planning to zero this indicators.

I will resolve warnings in future
skipped tests will be resolved in #32


Provide annotated services a way to return a successful response with alternative status code

For example:
@Get("/hello")
public String greet() {
    return "world"; // A user wants to send a non-200 response, but how?
}

Related: https://github.com/line/centraldogma/pull/141#discussion_r173067177

I think that we need to introduce some class like Spring's ResponseEntity to make the response flexible.

@trustin suggested a name HttpResult<?>. Thank you!

Returning customized HTTP response.

@Get("/api1")
public HttpResult<SomeObject> api1() {
	final SomeObject someObject = produceSomeObject(...);
	return new HttpResultBuilder<SomeObject>().ok()
			                        		  .header("name", "value")
			                        	      .contentType(MediaType.JSON_UTF_8)
			                                  .body(someObject)	// Set an object.
			                                  .build();
}

Returning customized HTTP response after a future has completed.

@Get("/api2")
public HttpResult<SomeObject> api2() {
	final CompletableFuture<SomeObject> someObject = produceSomeObjectLater(...);
	return new HttpResultBuilder<SomeObject>().ok()
			                        		  .header("name", "value")
			                        	      .contentType(MediaType.JSON_UTF_8)
			                                  .body(someObject)	// Set a future.
			                                  .build();
}

Returning customized HTTP response with a publisher. How can we support streaming response? Should the type to be a List<SomeObject>?

@Get("/api3")
public HttpResult<SomeObject> api3() {
	final Publisher<SomeObject> publisher = ...;
	return new HttpResultBuilder<SomeObject>().ok()
			                        		  .header("name", "value")
			                        	      .contentType(MediaType.JSON_UTF_8)
			                                  .body(someObject)	// Set a publisher.
			                                  .build();
}

For JSON streaming (MIME type application/json-seq): https://tools.ietf.org/html/rfc7464

https://jira.spring.io/browse/SPR-16742

For Server Sent Events (MIME type text/event-stream): https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

https://www.baeldung.com/spring-server-sent-events

It'd be better to support @ResponseStatus as well.
We need to change the method signature of ResponseConverterFunction in order to pass HttpStatus and MediaType into the converter.


Update us-year-end.md

updated

Docs Build status updates of commit eec6bda:
:warning: Validation status: warnings

File
Status
Preview URL
Details

dynamics-gp/payroll/us-year-end.md
:warning:Warning
View
Details

dynamics-gp/payroll/us-year-end.md

Line 52, Column 5: [Warning: hard-coded-locale - See documentation] Link 'https://docs.microsoft.com/en-us/dynamics/s-e/' contains locale code 'en-us'. For localizability, remove 'en-us' from links to most Microsoft sites.
Line 52, Column 5: [Suggestion: docs-link-absolute - See documentation] Absolute link 'https://docs.microsoft.com/en-us/dynamics/s-e/' will be broken in isolated environments. Replace with a relative link.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 3d61725:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

dynamics-gp/payroll/us-year-end.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Fix TP Delivery Service form field validation for httpBypassFqdn

What does this PR do?
The TP field validation currently requires httpBypassFqdn to start with http:// or https://, but the field is an FQDN not a URL. This PR fixes that form validation to verify it's an FQDN.
Fixes #2768
Which TC components are affected by this PR?

[ ] Documentation
[ ] Grove
[ ] Traffic Analytics
[ ] Traffic Monitor
[ ] Traffic Ops
[ ] Traffic Ops ORT
[x ] Traffic Portal
[ ] Traffic Router
[ ] Traffic Stats
[ ] Traffic Vault
[ ] Other _________

What is the best way to verify this PR?
Add a bypass FQDN to a http delivery service.
Check all that apply

[ ] This PR includes tests
[ ] This PR includes documentation updates
[ ] This PR includes an update to CHANGELOG.md
[ ] This PR includes all required license headers
[ ] This PR includes a database migration (ensure that migration sequence is correct)
[ ] This PR fixes a serious security flaw. Read more: www.apache.org/security

@saoyag001 @rawlinp for future PRs,  can you describe the change in the title rather than the issue # ?   Note the issue in the description as you've done,  but give the title something that'll be meaningful without having to look up the issue a year down the road

+1, edited the title and description to be more descriptive of the issue.

Tested in staging. Works as designed.


AQ-26856 Update Newtonsoft.Json to 12.0.3

Remove Newtonsoft.json.dll from repository.
Make Newtonsoft.Json a NuGet package.
Upgrade Newtonsoft.Json to 12.0.3.

@MartinBenn-AI
can you wait to merge this until next week?
I have 3 reports lined up to check-in this week and I would need to do double merges if this one goes through first.

@kimsharpe-ai we can hold off, but as it is right now we need this to get merged to unblock running reports in our 21.1 environment.
Are these reports planned for 21.1 and 20.4?

@MartinBenn-AI not 100% sure about 20.4 vs 21.1 but if I can get these in now and do a release before your change then that will cover both cases.
Also benefit is that you will be able to add your changes to the new reports - which they need.

We already have a branch for 20.4 which any changes that are targeted for that release will need to go into that as well. I think what you are saying that is if I hold off on merging this, you can apply the same fix to 21.1 (this branch) and 20.4 (release/20.4) without making any project changes right? That makes sense to me and I think we can put up with the broken tests for a few days.

@MartinBenn-AI ok thanks!

@MartinBenn-AI
Thanks for holding off on merge - today I added 4 new custom reports which need to be updated.
That's the last of new reports to go into 21.1.
I wonder if the dlls in the folder ExternalLibraries/Aqts ought to be updated too?
They are old - Aqts version 20.3.65

I will update the AQTS libraries since we are on ServiceStack 5.10.4 now as well as re-work this PR to cover those new reports. Thanks for the reminder!


Update plz this package

warning webpack-s3-plugin > s3 ><EMAIL_ADDRESS>please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
warning webpack-s3-plugin > cdnizer ><EMAIL_ADDRESS>Package no longer supported. Contact<EMAIL_ADDRESS>for more info.

PR welcome

PR is welcome, https://github.com/webpack-contrib/s3-plugin-webpack/pull/96 is started but I don't know appveyor enough to bypass it...


GUACAMOLE-335: Use /etc/guacamole for GUACAMOLE_HOME if not overridden and ~/.guacamole does not exist.

Sounds like a good idea!


Adding option to build on save

Would be very useful to add a option to automatically build on save

any news? I'm new at both, node and atom.io.
If you could give me a clue on how this can be done, I would be able to help you ;)

I'm sorry. I've been really strapped on time lately and haven't gotten around yet.
If you'd like, you could look at the Atom API documentation and find an event which is sent every time a TextEditor is saved. Then register a listener of that event.
Then create a new config variable (Automatically build on save - boolean default off)
When the event arrives, you should check the config variable. If it is set, trigger the build command (e.g. dispatch 'build:trigger').

Hey, I'm kind of a noob in Atom too. I attempted to implement this in https://github.com/sryze/atom-build/commit/cc7d29046af8b99974da767e2956bec78e0eacbe but it's not working for some reason - the build:trigger command is not dispatched at all, I've no idea why...

@sryze Thanks for the effort!
I think it needs the actual dom element in dispatch (something like workspaceElement = atom.views.getView(atom.workspace);. You can have a look at the specs how I dispatch build:trigger there.
Also, don't forget to implement some specs for the functionality so we can guarantee the functionality.

That worked, thanks.
As for the specs, should I create a separate file for this option like it's done for keepVisible?

https://github.com/sryze/atom-build/commit/95e966dfa5365faef441122e8c940987d3eb6029
Does this look good?

@sryze The spec itself looks good.
It might be worth just putting it in build-spec.js to avoid having to duplicate beforeEach and afterEach functionality.

First of all, thanks for adding this option...
However is not working for me :(
This is my configuration file:
{
    "cmd": "{PROJECT_PATH}/build.sh",
    "sh": true,
    "cwd": "{PROJECT_PATH}",
    "build": {
        "buildOnSave": true
    }
}

I've been trying adding the builOnSave outside the build declaration, and is not working either...

@citosid It's a global option, so it goes to your Atom config.

You can also configure it in the setting panel of atom. Might be easier.


docs: Replace northeast-arrow symbol

Substitute ↗ for 🡕 that seems to work in all browsers

Thanks!


shift usage from deep_merge --> core.merge and get_path --> slice and set_path --> set_slice everywhere

do we need to do get_slice and set_slice?


Modrinth modpack

Create and publish a modpack to Modrinth, because that's where the cool Fabric kids are :)
The files are added per #62, now just needs to be updated and packed when Modrinth adds the support.
https://github.com/modrinth/labrinth/wiki/Modpack-Format
Preferably automatically: https://github.com/comp500/packwiz/issues/34
Edit: I'll probably do this whenever it's possible, release of FO 2.1.0 won't be affected by this
Edit 2: this is fully automated now, releasing the first mrpacks on Github for testing

Preview is now available: https://knossos-4imn3jq70-emmaffle.vercel.app/modpack/fabulously-optimized

Site redesign announced, so the official link is:
https://modrinth.com/modpack/fabulously-optimized

Planned launcher:

via

That looks good! Currently using it in the regular minecraft launcher unless for fabulously optimized 1.18.1 but when the modrinth modpacks are out I would use it. Even if Fabulously Optimized is fast with modrinth modpacks then it will be in the top 10 modrinth modpacks I expect.

As of May 9th, all existing pack versions should break, due to the new Curseforge API and not allowing to directly link to files.
The current Modrinth availability state as of FO 3.7.0 is 34 out of 48 mods, this is not enough to publish a half-baked pack. It might be worth it to ask the rest of the mods to support Modrinth, but I'm not yet sure this should be a requirement for future FO mod inclusion. Maybe Modrinth will support Curseforge API?
Will let you know of any updates.

Modrinth modpacks are now publicly listed (in Alpha!)
https://modrinth.com/modpacks

No official launcher yet, though they have installation instructions for MultiMC development version
They are now working on payouts to get more mods onto the platform
Still no support for downloads is provided here because it can be unstable and unsustainable for now
fofirst_place

https://blog.modrinth.com/modpacks-alpha/

The first integrity test is out! Read more at https://modrinth.com/modpack/fabulously-optimized/version/3.8.2-mr.1

Second integrity test was done as well and no major imparities were found, so continuing with a (hopefully) more regular release schedule. Do continue reporting any imparities though.

Modrinth modpack works flawlessly with ferium. (used ferium to install modpack, gdlauncher to play). But didn't work with polymc.

But didn't work with polymc.

PolyMC is sadly not supported by FO officially yet. Also, this bug is reported in the PolyMC issue tracker so it's going to be fixed soon.

Good to know. Whats the officially supported launcher? ferium was working yesterday, but today its downloading the 1.18.2 version. There's no way to specify version.

Good to know. Whats the officially supported launcher? ferium was working yesterday, but today its downloading the 1.18.2 version. There's no way to specify version.

See https://fabulously-optimized.gitbook.io/modpack/readme/install-instructions
Modrinth is currently not supported at all as it can have bugs and inconsistencies, is there just for testing.

3.10.0 and few other latest versions had Don't Clear Chat History missing due to a bug in the script. That version has been re-uploaded AS -mr.1 and the 4.0.0-beta.1 has it fixed.
On a general note, Modrinth support by FO is waiting on:

Farsight, which is waiting on author rewards, which is coming soon
A launcher with good UX and easy updates - see #297 for more info

As of 4.0.0-beta.8, Modrinth version no longer includes JARs!
This means that it is becoming becoming more sustainable as a platform because mods are cross-posting to it.
(Though obviously things can still change in the future versions if some mod does not post to Modrinth as fast as it does to other platforms.)

Another step for parity!
4.0.0-beta.11 removed Don't Clear Chat History, so CF and MR users will have near-identical set of mods now (except for Farsight, still).

Yeah!

Is this issue still needed? I'm asking because there is a Modrinth modpack for FO.

Is this issue still needed? I'm asking because there is a Modrinth modpack for FO.

Yes, because Modrinth downloads are still experimental. If I understand correctly, we're mostly waiting to add support for launchers like PolyMC (#456, waiting on PolyMC/PolyMC#60), and then we should be good to remove the experimental notice and close this issue.

OK!

Another step for sustainability: MultiMC (auto-update) now prefers Modrinth mods!
That way the downloads actually increase download count for the dev and you get more stable, potentially faster downloads!

Sounds great :)

Especially because I like Modrinth. It's open source afterall :)

Fabulously Optimized now supports Prism Launcher and Modrinth downloads! The issue will still be kept open for Modrinth Launcher news, whenever that comes.

Is there any reason Modrinth isn't supported on MultiMC? It seems to work fine

Is there any reason Modrinth isn't supported on MultiMC? It seems to work fine

Yes:

CurseForge downloads benefit me financially, and are a familiar place for people to download from
MultiMC has no modpack updater feature, so an auto-update pack has been provided in GitHub to workaround that
MultiMC stable currently has a broken modpack downloader from Modrinth, meaning you can only download the latest stable version (have a try lol)
There is still a slight imparity in CF vs MR versions (Farsight vs Hold That Chunk), so MultiMC users can get the same mods as most other launchers
Overall I want to support only one (best) source per launcher to make providing support consistent

It looks like MultiMC is getting Curseforge pack import support again soon, will that be supported?

It looks like MultiMC is getting Curseforge pack import support again soon, will that be supported?

I don't think so. Downloading from CurseForge via an API nowadays does not even increase the download count, and some mods need to be downloaded manually.

Any reason Modrinth downloads are supported on Prism, since some of those problems still apply? Is it just because they have a modpack updater?

Yes, they have a working modpack downloader and updater 😄
I hope to close the gap between CF and MR ASAP though.

Is MR support on MultiMC going to be considered once the working pack downloader is stable, since the default MultiMC pack isn't updatable anyway?

Is MR support on MultiMC going to be considered once the working pack downloader is stable, since the default MultiMC pack isn't updatable anyway?

From above:

Overall I want to support only one (best) source per launcher to make providing support consistent

There are already two sources for MultiMC (the normal and auto-updating versions), it may be easier to replace the normal version with MR (once the fixed pack downloader goes stable) since that's already supported for Prism anyway

Then vanilla users would be troubled...
There is not much difference whether I would distribute that for vanilla as just a set of three folders or one that's compatible with something 😄

Wait, why isn't this closed? Modrinth is fully supported now, isn't it?

Yes, as Prism relies on it. I think this is safe to close.

Oh

Per #627, MultiMC users are now also advised to download from Modrinth instead of CurseForge.

Modrinth App now in open beta
The Modrinth App is now in open beta! If you already have it, make sure to run accept the update prompt.
Among other things, it now allows importing instances from other launchers. In my testing, the Modrinth-based instances of FO on other launchers are also updatable on Modrinth App :tada:
Update blog post
Preliminary install/update/etc instructions for FO
Things to test for:

Was your Java detected automatically? Note that this doesn't mean it'll be installed automatically if you don't have it.
Linux users - how smooth is the installation on your distro? Are you satisfied with the provided options? Give feedback on #497

Known issues:

FO's options.txt defaults are not currently applied (#504)
If you want to change mods, you can't auto-update the modpack later (#505)
There is no way to see what modpack version you have installed (#503)
Cannot upgrade to a specified version (e.g. latest for 1.19.4 instead of 1.20.1) (#194)

Once the app is stable and the aforementioned known issues are fixed, it will be officially supported by FO :)

Modrinth App 0.5.4
Modrinth App has been updated and all previously known blocking issues have been fixed.
Are there any new ones? Try it out and let me know! If not, then we might just start supporting it soon enough.
:scroll: Changelog
FO Preliminary instructions: Install | Update | Add mods | Disable mods
Remember, if your launcher UI does not match the instructions above, you're on an older version, so update it first!
Unfortunately, you can't yet check which launcher version you're running, see #737.

and also #656 is finally planned for 5.4.0, which would bring full parity between CurseForge and Modrinth.
So, during the 5.4.0 testing period the Modrinth App can continue to get improvements and when done, it would make sense to release 5.4.0 and Modrinth App support together.

It is also possible that Modrinth App will become the no. 1 recommended launcher for FO, because it supports FO well and literally promotes it as the first on homepage :stuck_out_tongue: But I think that may need to wait a bit more while it gets a full stable release overall.

Very interesting stats:

My interpretation

Modrinth is growing by a lot lately
Modrinth-compatible launchers don't care about betas
Modrinth still lacks the archival options, which skews the download stats of older (major) versions :(

I plan to support Modrinth App when the version indicator gets released (for my support purposes), and make it the first when it moves out of beta (because some people are still having issues with logins and Linux stuff).

Fabulously Optimized now supports Modrinth App!
The app is still in beta, but I think it is in a usable state for most.


Shoelace homepage has red background in Edge

https://shoelace.style/
Haven't looked into the cause but the site looks different in Edge compared to (Safari/Chrome/Firefox)
It has a red background, making it hard to read the logo, compared to the correct white background.

I just checked it out in Edge and everything looks okay @AmirAlavi

It would help to know what versions of Edge.

@AmirAlavi in your screenshot, the text isn't the default color either. Is this local or the live website? Have you made any customizations or changes in the dev console?

I'm getting the same thing in Edge - was just heading here to ask about it.
It happens when I hit https://shoelace.style/
Microsoft Edge 40.15063.0.0
Microsoft EdgeHTML 15.15063
If I kill this bit in dev tools things look nice again
::selection {
background-color: var(--selection-bg-color);
color: var(--selection-color);
text-shadow: none !important;
}

Thanks for that, looking into it.

Thanks @mahoneyjames.
@claviska Sorry for the delay in response. To answer your questions for the record:
It was live, without any changes in the dev console.
And my versions are the exact same as @mahoneyjames

Looks like an Edge 15 bug. I created a minimal test case. https://codepen.io/claviska/pen/qXXdQZ

@chrisdemars Though the colors are correct in your screenshot, the font looks different if you look closely. Which versions do you have?

This bug has already been reported and confirmed by the Edge team: https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/11495448/

I've applied a workaround that makes shoelace.style legible in Edge 15. Hopefully the bug gets resolved soon.


why restart all service when restart fleetd process?

Hi, I found all service be restarted in fleet when i use "systemctl restart fleet".
fleetctl status hello.service
hello.service - Hello World
Loaded: loaded (/run/fleet/units/hello.service; linked-runtime)
Active: active (running) since Fri 2015-01-09 16:06:34 CST; 9s ago
Main PID: 39760 (bash)
CGroup: /system.slice/hello.service
├─39760 /bin/bash -c while true; do echo "Hello, world"; sleep 1; done
└─39793 sleep 1
systemctl restart fleet
fleetctl status hello.service
hello.service - Hello World
Loaded: loaded (/run/fleet/units/hello.service; linked-runtime)
Active: active (running) since Fri 2015-01-09 16:08:42 CST; 1s ago
Main PID: 40234 (bash)
CGroup: /system.slice/hello.service
├─40234 /bin/bash -c while true; do echo "Hello, world"; sleep 1; done
└─40251 sleep 1
I think fleet should not restart all services. because I only want to restart fleet process.
Thanks
Best Regards.

Right now this is by design because we don't distinguish between fleet shutting down permanently or temporarily, and so we gracefully stop all units that it is controlling
What are you trying to achieve by restarting the fleet process?

To add to this, you can send the SIGKILl signal to shut down the fleetd process without first unscheduling all locally-scheduled units. This simulates a catastrophic machine failure.

@jonboulle thanks for you relay. In my opinion, fleet is a management program.  When  the program is  starting or stoping , it should not affect the management unit.  For example, we use the fleet to manage  many docker containers.  But once the fleet is restarted, it will lead to the restart action of  all  containers .  I think it is a big risk for production environment.

@holmes86 To tug on this thread a little further: how should fleet distinguish between when the machine is being shut down (and hence all units under its control should be terminated), and between a temporary restart when the units should not be interrupted?
More pertinently, I am really curious to know why you are restarting the fleet daemon in production environments - is there a problem you're experiencing?

@jonboulle For production environment, whatever machine is being shut down or fleet temporary  restart. We can know fleet is not working through by monitor system.Before I modify fleet.conf config options, but I can't reload there config options. So I just restart fleet process.Then it made me got this idea.
Thanks

@holmes86 fleet already has the ability to reload its configuration without restarting completely: just send it a SIGHUP and it will re-read the configuration file. Can you give that a try instead? :-)

OK. thank you jonboulle

My fleet service restarted automatically as it was unable to talk to etcd . But it caused all app containers to restart which was totally undesirable.


Make BoneAttachment3D and Skeleton3D names consistent.

Make BoneAttachment3D and Skeleton3D names consistent.
BoneAttachment3D nodes are now named as their bone name.
Resolves cases where BoneAttachment3D nodes will have arbitrary names like BoneAttachment3D6
Reserve the name "Skeleton3D" and use this name for all generated Skeleton3D nodes.
BoneAttachment3D nodes are now named as their bone name. Resolves cases where BoneAttachment3D nodes will have arbitrary names like BoneAttachment3D6
Reserve the name "Skeleton3D" and use it for all Skeleton3D nodes.
This change will break existing imported scenes with more than one skeleton and/or bone attachments.
Tested with the godot-tests repository. (with https://github.com/godotengine/godot/pull/72158 )

Thanks!


Add image processing support for AVIF

AVIF is a new image format (based on the open AV1 video format) that has the potential to produce significantly smaller file sizes than existing formats, although with different tradeoffs. I find demos of the format to be quite compelling: https://jakearchibald.com/2020/avif-has-landed/
I realize that webp support is probably higher priority, since caniuse.com has webp at 85% support while avif is currently at 25%, but I think it is likely to gain support quickly, since Firefox already has a decent implementation (behind a flag) and Apple are a member of the group that created AV1.
UPDATE August 2021: caniuse.com now has avif at 67.25% support. The next version of Firefox will have it enabled by default, it seems.

Does a native Go decoder/encoder exist?

I don't think so, all I was able to find in a quick search was what looks like abandonware: https://github.com/Kagami/go-avif

... abandonware: https://github.com/Kagami/go-avif

And it's not native Go.  I binds to a library (libaom) with cgo.

Are you willing to call a binary program or C (libaom) or Rust (rav1e) library for AVIF encoding? Since those encoders already have significant engineering work put into them, even if someone did write a native Go encoder, I wouldn't expect it to match the existing ones in performance or compression efficiency.

Any news?

Yes.

Is this issue awaiting a native Go encoder? Because I'm not sure if that will ever get written.

No.

what about https://github.com/Kagami/go-avif?

That uses libaom, so it's not a native Go encoder.

+1

+1

Is there any eta for getting AVIF support? The global support among browsers is quite 90% (according to CanIUse).

@Nacioss See https://github.com/gohugoio/hugo/issues/7837#issuecomment-1063793230

it's not very likely unless a top quality native Go decoder/encoder pops up

So, no, there isn't an ETA, because a top quality native Go decoder/encoder hasn't popped up.

+1
avif

Because I am under the impression nobody mentioned it so far: reducing the advantages of AVIF down to better compression isn't, IMHO, entirely fair.
A notable advantage of AVIF for me is HDR support (10- 12 bit per channel compared to JPEG's 8 bit per channel). And for galleries rendered with Hugo that will really get interesting. Perhaps we aren't there yet, but this will certainly draw further attention in the (very) near future.
As I am a Golang developer myself I know about the pain that comes from integrating C dependencies... still I would wish for an alternative to not support this "unless a top quality native Go decoder/encoder pops up" (which very likely might be never, a lot of popular formats have never been properly ported to native Go). 😊

For photography JPEG XL is superior, where AVIF shines is in highly compressed images for the WEB. I'm already using it on my Hugo blog (batch conversion with ImageMagick).

For photography JPEG XL is superior, where AVIF shines is in highly compressed images for the WEB

I agree on this, but as only Safari supports JPEG XL right now, its not really worth considering for web projects in general at the moment: https://caniuse.com/jpegxl

Since introducing wasm as a "thing in Hugo", I'm planning on replacing the current libwebp bindings with wasm, which would make the threshold for adding AVIF (slightly) lower

That could be a promising approach. Perhaps, at some point in the future, this could allow for pluggable components taking care of wrapping C dependencies providing additional format support to Hugo via some simple interface.

That said, I'm an avid photographer, and I haven't been totally convinced that my processed photos needs this HDR.

I am into that topic too and so far I was sure I wouldn't need/want it because "HDR" basically referred to oddly saturated and rainbow-colored images. So the term has got a negative reputation with "professional" photographers over the last years. But now with tools like Lightroom, PhotoShop, Affinity supporting HDR in the actual sense of an increased dynamic (and not only weird tone-mapping) it becomes a totally different things. Downside: one requires a display that is actually capable of displaying this (much) wider range. And that is why I am not fully sold yet too, because only few devices are capable of properly presenting such images. But that kind of hardware is on the rise and therefore support for it should be too ;)

That could be a promising approach. Perhaps, at some point in the future, this could allow for pluggable components taking care of wrapping C dependencies providing additional format support to Hugo via some simple interface.

Yea, well, there has been done a lot of research about a suitable plugin mechanism in Hugo, and portability, execution speed and good security/sandboxing has been ... hard to find Wasm/WASI is the most promising "thing" I have seen so far. We now embed the Katex wasm in the Hugo binary, but that was a proof of concept. I would very much come to a place where we could define some plugin interfaces where you could include these WASI plugins as Hugo Modules. And some hugo install commands that took care of the "packaging".
Whether image processing would be a plugin candidate is up for debate. My motivation behind libwebp ported to WASM is mostly motivated by getting rid of the CGO build.


TypeError Exception when importing dynamic data of Synchronous Machines

The parameter inertia of the class RotatingMachineDynamics is initialized in the contructor as an integer (see here).
When importing dynamic data, cimimport is converting the values of the cim files using the default value of the cimpy classes. That means, that cimimport tries to read the parameter inertia fro the CIM files as an integer and not as a float which traslante into an exception when reading dynamic data of cim topologies (see here)
The same bahaviour can be observed in the class SynchronousMachineTimeConstantReactance. The parameters tpdo, tppdo, tpqo, tppqo, tc should be initialized as floats in the contructor of the class.

solved in #39


Translate date-fns and Quasar into Luxembourgish and include them

When I switch the language to Luxembourgish, the German locale is used for date formatting in the following pages:

Wall
Feedback
Members
History
Issues
Place: pickups, wall, feedback, history

The Luxembourgish locale is used on the following pages:

Pickups
Applications
Place: manage pickups

Both lists are non-exhaustive; For instance, I have not checked the offer feature.
I assume that a third-party library is used for date formatting, but I have not checked which one.

Indeed! We use German there, as the library is not translated into Luxembourgish.

here is the bit in our code where we make it use German
the date library we use is called date-fns (list of languages is on the front page) and it has a nice guide for adding another language

Also, our UI library (Quasar) also has a language selection, and they also don't have Lëtzebuergesch so we use German there too (they have these languages available)... but probably what you're seeing is from date-fns.
We could collaborate to get Lëtzebuergesch added to date-fns if you're up for it?

Thanks for pointing me to the wiring and the libraries used! German as an alternative has been a wise choice as both languages are quite close. This is also the reason why it looks correct on some pages.
Sure, I am up for translating both libraries. I will look into this later today.

I have created a PR for the date-fns library: https://github.com/date-fns/date-fns/pull/1900

Amazing! Good job :+1:
I had a look at another recent language addition to see how long it might take until it gets merged - that one was 9 days until merged to master. I guess it'll then need to have a new release from that too. Let's see how it goes.

Thanks! I will let you know about its progress and will translate Quasar in the next days too. I suggest to leave this issue open until we have included the updated versions of both libraries.

@nicksellen Your guess of 9 days was correct! My PR on date-fns has been merged yesterday, accompanied with a new instant release. I have created another PR here just now to change the config.

Nice! I'll do a production deployment shortly to get it live...

Nice! I'll do a production deployment shortly to get it live...

To https://dev.karrot.world/ ?

Deployment to https://dev.karrot.world happens automatically after it's merged to master, production deployment is to https://karrot.world and has a confirmation button to press, which I pressed :) ... so it should all be live and active on both sites now.

Yeah, it is looking good! Thank you to both of you @tiltec and @nicksellen for reviewing and deploying this so fast!


Clarification on pretrained weights

When pretrained: True the pretrained weights in data/backbone_ckpt/res2net50_v1b_26w_4s-3cf99910.pth are used.
Where do these weights come from? Are they pretrained on colon images by you, or they are pretrained just on ImageNet?

Hi, thanks again for your interest.
As I mentioned at the end of the readme, it is brought from the PraNet repository.
Also, it is pre-trained with ImageNet 1K dataset.


ERROR: Could not find a version that satisfies the requirement pyppeteer-box==0.0.27

ERROR: Could not find a version that satisfies the requirement pyppeteer-box==0.0.27 (from -r requirements.txt (line 20)) (from versions: none)
ERROR: No matching distribution found for pyppeteer-box==0.0.27 (from -r requirements.txt (line 20))
大佬们，啥原因啊，有个模块下载不下来，源应该没问题。

可以尝试不指定版本 可能与你的python解释器的版本有关

python3 -m pip install pyppeteer
https://pypi.org/project/pyppeteer-box/


Consider using textEdit for completion

23-Wed 01:00:04	Franciman	not sure it will be significant, but it may be useful. Do you remember the completion problem I had, and you said that you are keeping track of the offset. I found out that in the response to the textDocument/complete request, there can also be a textEdit which contains the range of text to apply the text transformation to (in layman's words write the completion :P)
23-Wed 01:01:08	Franciman	so instead of keeping track of the offset you can use the start position of the range of the text edit. This has also the advantage of being able to support, in future, the use of the textEdit, when you select a completion, which may be useful
23-Wed 01:02:04	Franciman	since insertText field is deprecated in favor of the textEdit, when the text to insert is different from the label

Could you prioritize this feature please? Along with adding support for "additionalTextEdits"?
I am using a C++ language server that provides completions and suggests the appropriate #include header file through "additionalTextEdits". This would make working in kakoune much more productive.
I don't know much rust to be helpful with an actual PR.

Yes, I think it would be useful to have this done. It should be much easier to implement now because we have textEdit counterparts in lsp.kak. The only problem which remains is that we should cancel Kakoune's autocompletion when item is selected and do our own text edit instead. I'm not sure how to do it in elegant way, going to ask around at IRC channel.

Unfortunately I still don't know how to solve bottleneck I flagged in my previous comment. I raised an issue in Kakoune repo to get some advice.

tsserver sends adtional text edits adding namespace prefixes.
If the additional edits given by a completion at the intersection of the multiselection differ in content or range they should both be applied unless they overlap.

Does anyone want to give this issue a go? I consistently find myself failing to allocate enough time to have a proper look into it. This is a non-trivial feature which should be quite rewarding once implemented. Of course, I'm keen to help as much as I can.

Waiting on https://github.com/mawww/kakoune/issues/2898

I was trying to implement this, but it seems that the argument to the InsertCompletionHide hook is empty if you use <ret> to accept the completion.

it seems that the argument to the InsertCompletionHide hook is empty if you use  to accept the completion.

@mawww do you have any ideas why is this the case?

When you use <ret> to validate the selection, you also add a newline just after it, then once the menu gets hidden, kakoune detects that the cursor is not preceeded by the completion text anymore and filters that out.
I think the existing logic in Kakoune side is too brittle, we should probably keep track of the inserted ranges instead of trying to infer them back from cursor positions and buffer text.

fc3e5ea419aa79c7adf38a9252586d867b3eb19b should have improved things, keen to get feedback.

I have a rough prototype for this in https://github.com/krobelus/kak-lsp/commit/textedit-completion
Here is how to try it locally:
(I'm using ccls, I did not test with other language servers.)
cat > test.c << EOF
void foobar(int x) {}
void main() {
        foba
}
EOF

cat > min-kak-lsp.toml << EOF
[language.c_cpp]
filetypes = ["c", "cpp"]
command = "ccls"
roots = []
EOF

cargo build
HOME=$PWD kak test.c +3:11 -e '
	source rc/lsp.kak
	set global lsp_cmd "target/debug/kak-lsp -s %val{session} -c min-kak-lsp.toml -vvv --log log"
	lsp-enable
'

What already works:

i<c-n><esc> -> line 3 will be foobar(
i<c-n>x -> line 3 will be foobar(x

What doesn't work:

i<c-n><ret> -> leftover indentation from <ret>

There are two problems I'm still facing

To make the missing case work it would be nice if InsertCompletionHide
knew which completion was inserted. Currently it receives a range which
includes characters typed when accepting the completion, for example by typing
<ret> or some other char. Observe with:
 hook global InsertCompletionHide .+ %{ select %val{hook_param} }

For this use case we don't actually care about the inserted range - we can
already compute that from the textEdits sent by textDocument/completion.  The
only necessary info is  which completion was accepted, so we can look up the
corresponding textEdit.  @mawww, maybe we should change InsertCompletionHide to
pass the raw completion text. We can probably even remove the range parameter
since there is little use for it then.
To still make the above cases work I use a really ugly workaround where I drop the
last character of the range.  Of course that doesn't work for <ret> because
of autoindentation.

When selecting a completion in insert mode we first receive a new
textDocument/completion and only after that receive the
textDocument/didChange event that tells kak-lsp that we inserted the
completion earlier. Because of this wrong ordering, the textDocument/completion
event contains a completion that is out-of-bounds in the buffer that kak-lsp knows
about. This causes a panic in the handler for textDocument/completion.

I added a horrible workaround in position.rs that seems to work as long as the
source file is ASCII only.
I'm not yet sure how to fix that. Probably the order of requests should be fixed.

Multiple selections are not supported yet, I think that's okay. Also I need to add support for competions that contain multiple lines, this should be easy.

I've implemented a different approach now, see https://github.com/kak-lsp/kak-lsp/pull/551 if you wanna try it.
Some tricky issues are remaining, e.g. if/how we should support multi-selections.
Does anyone know a server that sends additionalTextEdits, then I can test that. Thanks.

I believe rust-analyzer uses additionalTextEdits when completing an autocomplete suggestion which requires adding an additional import.
If I have the following program:
fn main() {
    println!("Hello, world!");
    let x = Hash<cursor here>
}

Then autocomplete in VS Code with rust-analyzer plugin and select the second option, "HashMap (use std::collections::HashMap)`, it results in the following program:
use std::collections::HashMap;

fn main() {
    println!("Hello, world!");
    let x = HashMap<cursor here>
}

Note, I'm pretty sure rust-analyzer also specifically disables this functionality if the client does not report additionalTextEdit support, so that might be something to watch out for.
I don't have time right now to test this myself, but I hope this info helps!

let x = Hash

yeah, rust-analyzer does not compute the additionalTextEdits eagerly, so we need to call completionItem/resolve.
But first we need to expose that capability (like VSCode), otherwise rust-analyzer won't even send the HashMap completion
diff --git a/src/general.rs b/src/general.rs
index c35ee43..0c7a362 100644
--- a/src/general.rs
+++ b/src/general.rs
@@ -103,7 +103,13 @@ pub fn initialize(root_path: &str, meta: EditorMeta, ctx: &mut Context) {
                         preselect_support: Some(false),
                         tag_support: None,
                         insert_replace_support: None,
-                        resolve_support: None,
+                        resolve_support: Some(CompletionItemCapabilityResolveSupport {
+                            properties: vec![
+                                "documentation".to_string(),
+                                "detail".to_string(),
+                                "additionalTextEdits".to_string(),
+                            ],
+                        }),
                         insert_text_mode_support: None,
                         label_details_support: None,
                     }),

Then, whenever the user confirms (not just selects!) a completion, we'd need to call completionItem/resolve and apply the resulting text edits.
Here's how Emacs' lsp-mode does it. Apparently nvim doesn't support this.
The problem is that in Kakoune (like in Vim) there is no difference between selecting and confirming a completion, so this might get tricky. The server does undo the import statements, so we'd need to clean up, or leave them around.
LSP and Kakoune have different ideas of how completion works, and I don't think it's possible to reconcile them without changing one of them.
As a workaround, we can use :lsp-code-actions Import after inserting the completion. I'm not sure however if we should lie about resolve_support, maybe we can just ask rust-analyzer to compute additionalTextEdits eagerly if the client does not support resolve. They can just send two batches of completions, unresolved and resolved.

BTW the original issue here should be solved for many practical cases by
c413929 (Use column from textEdit as completion start instead of heuristic, 2021-09-12)

Ok nevermind I believe we can use InsertCompletionHide to figure out when a completion was "accepted", so additionalTextEdits should be doable.

Gotta go now but https://github.com/krobelus/kak-lsp/commit/completion-additionalTextEdits is a quick-and-dirty fix for the rust-analyzer example.
In some scenarios it will also add the import of the selected item if you hit backspace, I'm not sure how to avoid that.
This happens when you type HashMap, select the completion, then type backspace. The backspace makes kak-lsp ask for completions again, so it triggers another InsertCompletionHide and InsertCompletionShow

2637b26ad53d60c8dfc36134a31124a71044a085 takes care of adding the import statements, thanks for the test case


PROD SUPPORT: Nyaymitra can't see the registration request of advoacte

Describe the bug
Advocate Registration flow is breaking
Request not received to nyaymitra

similar to #2422. closing this

Let's add a locatization text saying : If your registration hasn't been approved within 2 hours, don't worry! You can reach our Helpline at +91-472-2919099 or write to us at<EMAIL_ADDRESS>


[QUERY] Copy data from one storage account to another storage account across subscription

Query/Question
Copy data from one storage account to another storage account across subscription using azure java sdk.
Why is this not a Bug or a feature Request?
We need to copy a virtual directory content to another storage account through code. AzCopy provides this option by using this :
azcopy copy 'https://source-storage-account-name.blob.core.windows.net/container-name/directory-path SAS-token'
'https://destination-storage-account-name.blob.core.windows.net/container-name' --recursive
Is the same option available in azure java sdk. Currently we are using startCopy .
Limitation of this method is we need to pass files in directory one by one.
It is time consuming process. We want to avoid looping through the files and just copy all the data from virtual directory in recursive way to destination.
Setup (please complete the following information if applicable):

OS: [e.g. iOS]
IDE : STS
Version of the Library used :

com.azure
azure-storage-blob
12.6.0

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[ ] Query Added
[ ] Setup information Added

The sdk does not have a method to bulk copy a directory from one location to another, and there are not plans to add such a method. As you mentioned, azcopy includes this feature, and our recommendations in these scenarios is always to use azcopy as it is finely tuned for exactly this workload. If you want this feature within Java, it will be up to you to build the solution as you mentioned.

Closing this issue as it is not a feature we plan to add and it has been left idle for a while.


Landmine exploding in hands when you try leave interior.

When I try to leave the interior through the main entrance or the fire exit with a landmine in hands, it almost always explodes. This is happening as client only propably. Im using FairAI v1.3.5 by TheFluff.

Thanks for pointing this out! It may have been caused by the v50 update, not sure. The issue was occurring even without the FairAI mod so something must have changed.
Either way, this should be resolved in 1.3.0 from #2
Lemme know if that didn't fix it for you.


Added log line of type counter.  Will be used for alerting.

Why: The goal is to alert on worker success/failure rates.  To that, we must first log the data in a convenient KayVee format.
What: On infra's recommendation, the following data is stored in the following fields:
source: <something>
title: <failure, or success>
type: counter [optional]
function: <worker name eg. ps, json_processor>

Testing: Ran unit tests and manually verified that the appropriate log lines are generated.
https://clever.atlassian.net/browse/DATA-1121

lgtm!


Implement Google Data Catalog as under meta service

Tackles #12751
Draft PR

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@9c64bf0). Click here to learn what that means.
The diff coverage is 0.00%.

@@            Coverage Diff            @@
##             master   #13252   +/-   ##
=========================================
  Coverage          ?   43.41%
  Complexity        ?     8884
=========================================
  Files             ?     1361
  Lines             ?    78287
  Branches          ?     9505
=========================================
  Hits              ?    33987
  Misses            ?    41424
  Partials          ?     2876

Impacted Files
Coverage Δ
Complexity Δ

...main/java/alluxio/table/under/gdc/GDCDatabase.java
0.00% <0.00%> (ø)
0.00 <0.00> (?)

...va/alluxio/table/under/gdc/GDCDatabaseFactory.java
0.00% <0.00%> (ø)
0.00 <0.00> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c64bf0...08c5be5. Read the comment docs.


Negative numbers with currency symbol: incorrect format

format('$#,##0.', -100) should return '-$100', the same way as Excel does it

Hi @maxkan!
Hmm, adding this ability to the formatter would make it much more complicated. There are many currency symbols and the mask could also include a longer prefix (e.g. The amount was € #,##0.)
Maybe adding a placeholder to the mask for the positive and/or negative sign would work better. Something like this:
format('$#,##0.', 100)     // $100
format('$#,##0.', -100)    // $-100
format('{-}$#,##0.', 100)  // $100
format('{-}$#,##0.', -100) // -$100
format('{+-}$#,##0.', 100) // +$100

What do you think?

And I actually thought Excel wrapped negative numbers in parenthesis (e.g. (12.34) = -$12.34). Did that change?


7.0.0-beta6 memory issues (regression)

A test task runs over 310,000 independent test executions. In 6.14.3 this completes in about 2.5 minutes. In 7.0.0-beta6 this slows down to a crawl at 200k tests, appearing to GC thrash as small batches of tests complete.
In the past, I've seen this occur due to reporting keeping parameters in-memory. The project uses a workaround to replace test parameters with their toString version, as these are too heavy for a large run. I suspect there is some similar per-test memory grab by the reporter.
The next step is probably to take a heap dump and determine what is held. Before running too far into this, do you know what changes were made that might hint at what to look for?
./gradlew :caffeine:strongKeysAndStrongValuesAsyncCaffeineTest

@ben-manes - Thanks for taking the time to report this issue. Between 6.14.3 to 7.0.0-beta6 a lot of changes have gone in. So it would be great if you could help do one of the following

Extract out heap dump from the current execution itself so that we can fix the problem itself.
Incrementally use beta versions one after the other to isolate what is causing the problem.

It appears to be due to holding every test method invoked in a ConcurrentLinkedQueue and ConcurrentHashMap. The TestNGMethod and TestResult  did get fatter between releases.
There are also many, many strings for reporting every test method. I reduced my CI time by 10 minutes by interning the strings. I suspect you could do the same as method names, etc are identical for a parameterized test. When returning to the prior commit, turning on G1's -XX:+UseStringDeduplication delays the GC thrashing from ~200k to ~300k. Unfortunately, that also causes a lot of InterruptedException to be thrown, so I probably can't turn it on.

I was able to greatly reduce the string storage by using a short string for the context parameter if no failures had occurred. This way when a failure happens, I can obtain details but that won't be the common case. I also intern'd the TestName, which together brought it down to 4mb instead of 400mb.
The ConcurrentLinkedQueue$Node and the TestResult objects they hold are still the dominant cost. I am surprised this is all held in memory. Why not asynchronously write them to disk and reindex when writing the report? It seems very wasteful to hold it all in memory and doing so incurs more GC cycles. Something like log4j2's disruptor design for async logging might work well to avoid an I/O penalty.

This runs in 2 minutes on 6.14.3 and 4 minutes with 7.0.0. This is likely due to a lot more full GCs required, which causes thread interrupts. Thanks to fixes in  https://github.com/awaitility/awaitility/issues/134, the tests all pass fine.
The slow down is concerning and would block upgrading. I have optimized my memory usage significantly, and the cause seems to be due to TestNG. The larger TestNGMethod and TestResult classes, plus storing them in-memory using a ConcurrentLinkedQueue are the bloat in my heap dumps. These are areas that I cannot optimize in my code except to split the test suites apart even further (as millions overall).
My proposed fix would be to have the results asynchronously streamed out of memory to a file and then process the file for the report. This would drop memory usage to significantly speed up large test suites (by seconds), but might make the common case of few tests a little slower (by millis).

@ben-manes - Thanks for adding that additional context. Yes the reporting piece of TestNG has started bloating in memory and perhaps would need some revamp or a different approach.

My proposed fix would be to have the results asynchronously streamed out of memory to a file and then process the file for the report.

That was something that I was thinking too, but the problem is that TestNG also exposes some objects which are a bit more complex such as the java.lang.reflect.Method for e.g.,
But please do share any ideas that you may have in terms of an approach of how to go about doing this. We can vet it out together and then have this addressed.

the problem is that TestNG also exposes some objects which are a bit more complex such as the java.lang.reflect.Method

For instances, like the test method / class / etc, can you use a cache and store the fully-qualified name or other unique identifier? The number of test instance methods will be small in comparison, so you can keep a Map<String, Method> and similar to resolve against when deserializing. That way all usages still get the exact instance. Cases like the instantiated test class can be qualified with an additional identifier, so the exact object instance is resolved. If think all of this can be worked around, but I agree it becomes less pretty.

When optimizing memory usage again, I noticed that Gradle uses a custom testng reporter. This does not extend TestListenerAdapter and instead is much less memory hungry thanks to capturing the minimal fields it needs and using synchronized for cheaper data structures. The performance seems fine, so perhaps CLQ is overkill?
The only TestListenerAdapter that I see in a heap dump comes SuiteRunner and appears to be unused. This is forcibly added during initialization, but that listener is never inspected afterwards. It appears to be entirely unnecessary?
https://github.com/cbeust/testng/blob/518bd75212805eb5f6f6a9518616abb5fd454159/src/main/java/org/testng/SuiteRunner.java#L37
https://github.com/cbeust/testng/blob/518bd75212805eb5f6f6a9518616abb5fd454159/src/main/java/org/testng/SuiteRunner.java#L193-L196
The actual TextReporter is may be added later, but is not done so by Gradle as disabled. For those using it and others, I think you could rewrite them to not extend TestListenerAdapter and instead incrementally build the output like TestNGTestResultProcessorAdapter does, thereby allowing the heavy-weight objects to be discarded. Ideally those that write textual data could stream to disk, e.g. TextReporter prints the buffer at the end and could simply display a temp file instead.
My latest build optimizations to reduce TestNG's heavy-weight reporting seems to save 20 minutes due to not choking on full garbage collections (a 22% speedup). I believe all of my low-level hacks could be removed if TestNG would more aggressively discard its heavy result objects instead of unnecessarily keeping them throughout the execution. This would improve performance for large builds significantly, and shouldn't negatively impact small suites either.

I tried to reflectively clear the queues inside of this useless TestListenerAdapter instance. That dropped the CLQ#Node instance count by half, but did not reduce the memory usage.
I then reflectively captured TestRunner#m_invokedMethods and clear it. This seems to only be utilized by SuiteRunner#getExcludedMethods() for JUnitReportReporter, SuiteHTMLReporter, and IgnoredMethodsPanel. None of these are used in my build. That halved the CLQ#Node instances again, but did not drop memory.
This lead to SuiteRunner#invokedMethods which is similarly unused. By reflectively clearing out those results, the CLQ#Node dropped from my heap dump and the execution ran successfully.
While it dropped memory, it didn't decrease as much as desired because the TestResults are still being held by a ConcurrentHashMap. This is from ResultMap inside of TestRunner. Again these are only useful for some reporters and I can clean the results manually.
This finally dropped the memory usage and didn't affect the test reports from Gradle or Eclipse. While certainly ugly, it does prove that TestNG is holding onto the test results unnecessarily for builds, and could avoid that unless explicitly needed by a reporter. Those reporters could also be much more efficient by not requiring the full results stay in-memory, but rather capture only the details they care about (and stream them to disk if possible).
It would be really great if this memory behavior could be improved so that I could remove my narly hacks 😉.
(See CacheValidationListener for all of these hacks...)

In today's round of optimizations, I was able to drop a test suite task from 315MB (8.6M objects) down to 22MB (0.5M objects). The noticeable pauses from a full GC also disappeared.
There are roughly 30 test suites of equal size (140k tests per suite) in order to reduce the report size, as a single test task failed due to the memory overhead. Previously I had already optimized by aggressively removing test parameters and expected exceptions if successful and shortening + dedup'ing test names in the report. The GC is also tuned.
4.2M tests executing in parallel on TravisCI with 512mb heap now takes ~1 hour.

@ben-manes - Based on the details that you shared here as observations, I went ahead and created a PR https://github.com/cbeust/testng/pull/2390 that attempts at reducing the footprint.
Would it be possible for you to checkout this branch, build TestNG locally and try it out without the tweaks you added to caffeine ? That way I can know for sure if the changes are taking us forward in the right direction.

Thanks!
It would be nice if TestNG could be sourced from jitpack to simplify testing a commit. I had to build a local jar and include its dependencies, rather than merely swap coordinates.
I wrote a simple stress tester for you. It begins showing GC pauses at ~1M tests and eventually runs out of memory ~1.3M. This occurs for both 7.3 and your branch. It appears that CLQ is improved (but remains), but those captured in the CHM still cause an issue. Note my descriptions for fixing TestRunner and SuiteRunner, or look at my horrid code 😄.
testng-stress.zip
// terminal 1
./gradlew cleanTest test
// terminal 2
jmap -dump:live,format=b,file=/Users/ben/Downloads/1.hprof "$(pgrep -f GradleWorkerMain | head -n 1)"

Then open the heap dump, e.g. VisualVM.
Across runs this will be at different heap times, but offers enough to eyeball where problems are and iterate.
7.3.0

fix_reduce_memory_footprint_in_reports

It would be nice if TestNG could be sourced from jitpack to simplify testing a commit.

What do you mean? You should be able to find the build from @krmahadevan's branch here:
https://jitpack.io/#krmahadevan/testng/fix_reduce_memory_footprint_in_reports-SNAPSHOT

Yes, but it fails to produce a detectable artifact. Go to branches and see the log.

fyi, you can use the environment variable JITPACK to detect and then skip incompatible steps, e.g. this case of failing to produce a jar because it was unable to sign the archive.

I ran the stress test at 256mb heap against master, after this was closed. I still see the memory problems, so it isn't quite fixed. However maybe a new issue is appropriate since this was is getting quite verbose. I'd recommend using the testng-stress project to optimize with, since it nicely shows where there are excessive allocations.

@ben-manes - Please try using #2392 changeset. It should bring down the memory foot print further down.
The biggest of the problems is the ITestResult object which embeds a reference to ITestNGMethod object which internally embeds a reference to ConstructorOrMethod object which embeds within it a reference to Constructor (or) Method.

I did notice the duplications of Method since Class#getDeclaredMethods copies it, so there are many duplicates. It wasn't a big problem, but there are lots of little wasteful allocations that can be removed.
The new code still has the memory problem because there are many places that hold onto all of the ITestResult objects. The ITestNGMethod did drop, so that's great.

@ben-manes - In continuation with my effort to drive down the memory foot print, I have raised another PR https://github.com/cbeust/testng/pull/2393 which removes references to ITestNGMethod array from TestRunner. This should further drive down the memory foot print. Can you please help check ? I think now the only thing that would be left is to figure out how to handle the reporting aspect of TestNG (which I guess is where the ITestResult implementation is going to slim down)

@kamilszymanski I don't see ITestNGMethod in VisualVM anymore. The TestResult instances still remain and causes heap exhaustion. From my prior analysis it would be fixed if you can remove the holds in 3 and 4.

useless TestListenerAdapter instance [done]
TestRunner#m_invokedMethods [done]
SuiteRunner#invokedMethods [todo]
ResultMap inside of TestRunner [partial]

No more ITestNGMethod instances

TestResult & InvokedMethod remain

@ben-manes I guess you meant @krmahadevan

oops, sorry for fat fingering the auto-complete!

@ben-manes

SuiteRunner#invokedMethods [todo]

I have fixed this ask as well, as part of the PR https://github.com/cbeust/testng/pull/2394
Can you please try this branch ? [ Note: my previous PR has not yet been merged ]

@krmahadevan It did seem a little better by performing more tests in the stresser. Since there are still places that retain all of them, e.g. ResultMap, unfortunately you have to fix all before we feel the impact.
If you feel that you have to wait on me and get blocked, you can perform my steps:

Build the local jar (gradlew jar)
Download testng-stresser and run the project.

Update the build to point to your local jar
I added maxHeapSize = '256m' locally to fail faster.

Run gradlew cleanTest test until it slows down
Create a heap dump, e.g.
`jmap -dump:live,format=b,file=/Users/ben/Downloads/1.hprof "$(pgrep -f GradleWorkerMain | head -n 1)"
Open in your favorite tool (JVisualVM, YourKit, JProfiler). The latter two are free for OSS and are great.

@ben-manes - For now, this is the max I can do in terms of fixing this problem (atleast to some extent) because for the other areas such as ResultMap I think its a bit more involved effort which will need some serious refactoring of the codebase, because its part of exposed API interfaces and there's no way we can get rid of them without some re-engineering.
And the only reason why I was coming back to you, asking for feedback was because, I felt you could give me a quicker feedback in terms of the memory improvements, than me (since you already know what to look for and how to look for etc.,)
I will be honest here, this is the first time I am using these tools for reviewing memory footprints (please bear with me if that sounds a bit lame or childish excuse), so there's a learning curve involved for me and it means i spend more time. Trying to juggle between work priorities and testng bugs (i love working on testng coz of the learnings it gives me and that's why i keep coming back to it over and over again) and so i dont find enough time for these investigations. Hope that adds the context
All said and done, it would be really helpful for me if you could please share me some tutorial/video/blog post/book or anything else, that I can refer to so that i also start becoming acquainted with all these stuff and become a better coder.

Oh of course I don't mind. I'm sorry if you took that nudge negatively as if I do. I felt bad thinking you were making changes in the dark or might wait on me, as I know how important quick iterations are during refactorings and to stay in the zone. I'm happy to iterate with you on this, even if its delayed if requiring a major version bump, as you'll make usages better and it's your charitable labor.
This is an invasive change and I'm not sure if you'll be comfortable without a major bump, as it might impact user-facing APIs with their custom integrations. Or maybe not, you're the expert here :)
Using a Profiler seems scary and awkward at first (just like learning SQL), but it pays for itself and isn't that hard! The JVM team develops VisualVM and Java Mission Control. The former is older and can open a heap dump, while the latter also offers Flight Recorder (JFR) as a low overhead profiler that can run in production. They are unbundled in recent JDKs so you have to install them separately (brew cask install visualvm, brew cask install jmc). I'm not as familiar with JMC but its pretty well regarded. YourKit and JProfiler are excellent commercial profilers that have been around for many years, highly regarded, and generously free for open-source projects. They are all worth installing and playing with. The JVM tool jmap lets you dump a live heap rather than needing to attach the profiler against the running process, which was more convenient in this case.
When you take a heap dump it generates an hprof file. You might have seen this at work when a memory leak causes an OutOfMemoryError (where -XX:+HeapDumpOnOutOfMemoryError kindly dumps it prior to termination). The profilers can open the dump and provide insights like instance counts, memory dominators, duplicate strings, etc. In most cases when a analyzing a leak, the problems are immediately obvious and the calculated object path shows where its held. Once you know the dominators and major allocation spots, it's back to the code to determine the fix. When you have a patch, if you can reproduce the failure then you iterate and can verify in a new heap dump the success.
For example I was curious to see how JUnit 5 handles this stress test (as I've been a happy TestNG user for over a decade). It turns out to run the simple allocator tests slower and fail with fewer executions (~200k vs ~600k). The YourKit snapshot of the heap shows that LinkedHashSet in TestIdentifier and TestTemplateInvocationTestDescriptor are the dominators, and from the source code they appear their version of TestResult. Since their results are heavier objects, the stress test fails with fewer executions.
Learning a profiler just takes an issue to track down and exploration. It's not hard, just new and requires a little knowledge to set up. Afterwards you'll quickly track down the problem and maybe use the profiler for only 5% of the overall effort. The value is that it quickly uncovers the root problem, which would otherwise be a guessing game (where you are almost always wrong). So it pays off, but using the tool requires no real expertise and just a problem to be motivated by.
Perhaps @cbeust has further insights here. Really I think you'll be happy once you take the jump and learn it. Like SQL, its seems scary at first and then pays off handsomely throughout your career.


Confirm chain id in pcli requests

Currently, it's very possible to accidentally connect to the wrong chain with pcli, resulting in confusion (where are my assets? why has time gone backwards?). This should instead result in an error.
I can see 3 possible ways this could be implemented, depending on where the check occurs:

pcli transmits the id of the chain it expects to be connecting to in each request, and pd returns an error if that does not match the chain id of the chain pd is managing.
pd returns, along with any other result data, its chain id, and pcli checks this and returns an error instead of processing the rest of the result if it does not match the expected chain id.
pcli precedes each request other than ChainParams itself with a ChainParams request, and checks to make sure the chain id matches before proceeding.

I think that (3) is the best solution, because:

unlike (1) it means that pcli doesn't ever disclose to pd what chain id it is expecting to see (which could leak local development information to a public server in the case of misconfiguration), and
unlike (1) and (2) it means that this can be a transparent upgrade, because it does not change the proto types.

Aside from the changes mentioned in (3), implementing this will require a change to the wallet format to store the expected chain id. Considerations here:

This field should be optional: when it is not yet populated (as in a wallet that pre-existed prior to this update), it should be filled in the first time a chain is contacted.
When the wallet is initialized, it should also be left blank, and then only when the client connects to the chain should it be filled in, at this point effectively "locking" the client to that chain.
pcli wallet reset should remove this field, so that keys can be reused across multiple chains by resetting all the rest of the state.

Since we store the ChainParams as part of the wallet state now, I don't think we have to do anything extra to record the expected chain ID.  But we do have to figure out how to handle the case where the chain parameters haven't been downloaded yet.
I think (3) is less preferable, because it means that clients have to do some extra network roundtrips, and have to figure out when it's appropriate to do the extra requests.  Instead, I'd suggest (1'):
1'. Add a string chain_id to every request proto, and have pd return an error if the string is nonempty and doesn't match the expected chain ID; if the string is empty or if the string matches, process the request as normal.
This is also drop-in compatible, since adding fields to protos is fine, as all proto fields are supposed to be optional by default.  It also gives the behavior we'd like for grabbing the chain params initially, acting as "pin on first use".


To use cat

https://github.com/Timo9Madrid7/maliciousfl/blob/a0163b4fa1f461eee4a11da34ce10305b659152e/Crypto/s2pcSyMPC_v2.py#L76
ToDo: use cat() to concatenate cosine distances for each player. This may accelerate the l2-distance matrix in the next step.

The issue is solved by using concatenation. PR https://github.com/OpenMined/SyMPC/pull/326 is waiting to be accepted.


[curiosity] compare/contrast/synergy/etc with parallel --tmuxpane

i guess this is more of an open ended prompt, i've come across these two in the same breath of internet searching on a couple occasions, curious how/if they relate/conflict/supplement/etc?
https://www.gnu.org/software/parallel/man.html#tmuxpane

Wow, I did not know GNU parallel provides such the option.
Thank you for your information!
I've tried parallel command and I am sure that there are some design different policies and features between parallel --tmuxpane and xpanes.
But, I think, most different point is tmux-xpanes is also designed to help users operate interactive screens.
Better for tmux-xpanes
For example, it is impossible for parallel to connect multiple servers/containers and make changes to all of them simultaneously  (parallel closes panes automatically).
## Works
$ xpanes -c 'ssh {}' host1 host2 host3

## Does not work
$ parallel --fg --tmuxpane 'ssh {}' ::: host1 host2 host3

Better for parallel
In contrast, it might be better to use parallel to run the jobs more effectively.
Originally, parallel is designed to run jobs in "parallel" and it has some useful features for them. For example, it can specifies the maximum number of process.

curiosity satisfied :) thanks!


Increase test coverage

The current percentage is rather unpleasing. We should aim higher.
This issue is more of a goal, marking meta.

We double count source files from app and lib and different tests hit them both. We should figure out how to consolidate these

Do we still do that after https://github.com/mozilla/tofino/commit/9fd89a69bcb0d69f75e2c35b4044ed75f3a86a0b ?

Well, the webdriver tests are the only ones that test the electron backend and the services, in which case we want to count those lib directories

Actually I misunderstood what those options in package.json do. Since we're passing all: true, we do in fact run coverage analysis on everything. The exclude just seems to prevent nyc from rebabeling.

Also, the webrenderer tests inject the babelified code from lib into the new electron renderer process, so there's more ambiguity there as well.

https://github.com/mozilla/tofino/issues/1193


sudo devtools-terminal --install gives error

sudo devtools-terminal --install                                                                                ⏎ master ✱
/usr/local/lib/node_modules/devtools-terminal/node_modules/socket.io/lib/store.js:35
Store.prototype.__proto__ = EventEmitter.prototype;
                                        ^

TypeError: Cannot read property 'prototype' of undefined
    at Object.<anonymous> (/usr/local/lib/node_modules/devtools-terminal/node_modules/socket.io/lib/store.js:35:41)
    at Module._compile (module.js:573:32)
    at Object.Module._extensions..js (module.js:582:10)
    at Module.load (module.js:490:32)
    at tryModuleLoad (module.js:449:12)
    at Function.Module._load (module.js:441:3)
    at Module.require (module.js:500:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/devtools-terminal/node_modules/socket.io/lib/manager.js:16:13)
    at Module._compile (module.js:573:32)

node --version                                                                                master ✱
v7.0.0
OS X 10.9.5
npm install -g devtools-terminal                                                                                ⏎ master ✱
/usr/local/bin/devtools-terminal -> /usr/local/lib/node_modules/devtools-terminal/bin/devtools-terminal
/usr/local/lib
└──<EMAIL_ADDRESS>Edit:
from this , it could be an issue with dependencies with node v7.

It seems Node v7 finally removes process.EventEmitter. Practically speaking this means that Socket.IO 0.9.x no longer works. The correct solution, at least in my mind, would be to upgrade Socket.IO or probably just use WebSockets, though I am new to this codebase and am not 100% certain why Socket.IO 0.9 is here.
If you just want a workaround, you can manually patch the local Socket.IO installation. Go to the node_modules for devtools-terminal, i.e. /usr/lib/node_modules/devtools-terminal/node_modules/ and replace process.EventEmitter with require('events') globally.
Perhaps a middleground between those two would be a fork of Socket.IO 0.9.x (does such possibly already exist?) or trying to convince the maintainers of Socket.IO to release a new 0.9.x build with this fixed.

for mac users, if they have sed error for the command suggested above:
sudo find . -type f -name '*.js' -exec sed -i '' -e "s/process.EventEmitter/require('events')/g" {} +
https://stackoverflow.com/questions/19456518/invalid-command-code-despite-escaping-periods-using-sed

In case if you face any error "sed: RE error: illegal byte sequence" use export LC_CTYPE=C
export LANG=C


Live Chat Replay

How can I be sure youtube live chat replay is returning the all the comments and not just the top chat?
It definitely is excluding a lot of them.

Can you please give an example? (The command you used including the YouTube video URL)
The tool should display everything which appears on the chat replay, so if you can find a message which is displayed on the website, but not retrieved, I'll be sure to look into it further.

Hey! Thanks for coming back to me so quickly.
We would like to use this tool to retrieve the chat from
There should be over 1.5million messages but your tool is returning 11k.
I used the cmd py chat_replay_downloader.py https://www.youtube.com/watch?v=fqnnU6s_SB -output example.csv

Are you sure there are 1.5 million messages?
The video is 23 minutes long, with only 400k views. If there are 1.5 million messages, that would mean ~11k per second, which I highly doubt is correct.
If you check out the chat replay, the number of messages seems to correspond with what the tool is getting (~8 per second)

This is what YT analytics is telling me.
The youtube cap is 2000 live chat messages per second and the aim of this video was an interactive experience which required people to constantly type into the chat
Per-minute that would be 120k if we reached the maximum limit which we did on a few occasions. Youtube would frequently say "Network busy" and prevent users from typing. The game interacted with youtube via the YT Live API which was also limited to 2000 per second and i can confirm we had hit this in-game. The reason we want to use your tool is because the live video no longer has a live chat id and so we are look at other means of extracting the names of players.
Perhaps this is something you would be interested in helping us on and we can take this off git and into email? Perhaps we could hire you for a few hours to take a look to see if this is possible?

Oh that is quite strange indeed. The tool works by loading the video page and simulating http requests for the next "continuation" of the live chat replay. So, the tool will retrieve everything to be shown by the chat replay. In other words, the tool will only ever show what is visible on the YouTube video page.
I suppose YouTube implemented a cap when showing messages from a past live stream (possibly for a case such like this) to improve the viewer's experience (i.e. not loading 1.5 million messages). The tool does allow for passing a cookies file to the program, but I'm unsure whether this will fix anything.
A simple google search for missing YouTube livestream messages does not seem to yield any results, so it must be a "feature" they added in without documenting.
Excuse me for not knowing, but on the "uploader's side", can you see all the messages? Do they provide an option for downloading everything? Is there a similar UI for seeing all the messages? I understand that YouTube says that 1.5 million messages were sent, but do they provide a means of accessing it?
If you want, you can send me an email at<EMAIL_ADDRESS>for us to discuss further - this is quite intriguing!

Sorry to butt in, but pretty sure Youtube limits it on the live chat replay.


Dataset

Hi,
Awesome paper! When will the dataset be available for download?

Thank you for your interest in our work. We will release the dataset in early February, so stay tuned.

and do you plan on releasing the code too?

We plan to share the code in the near future. However, due to our current engagement in completing another task under a tight schedule, we kindly request your patience for a little more time.

The download link of AVMuST-TED has been updated.


Write Buffer Size for USB CDC Device (VCOM)

Hello.  I have a microcontroller device that is set up as a USB CDC device, that creates a virtual COM port.  I would like to write data to this device as fast as possible, and according to this StackOverflow question, I can maximize my throughput by writing 2 kB chunks in 1 ms intervals.
Is this possible using SerialPortStream?  There are two questions related to this, from what I understand.

Can Windows send data every 1 ms frame?  Or will it be more like 10 ms?
I see SerialPortStream.cs has a WriteBufferSize property.  Should I be setting this to 2 kB, to match the settings of my microcontroller buffer?  Or does the low-level driver need to have a buffer size of 2 kB?  If so, is there any way to set this from SerialPortStream?

There are properties DriverInQueue and DriverOutQueue to set the queue size of the driver itself for operating systems that support this (e.g. Windows only for now). You can set that to be 2kB or 8kB as you want. That's the size requested for the driver to allocate. Some drivers can handle large buffers, others not so, which is why this is configurable.
The "buffer" is a RAM backed buffer independent of the driver queue size. How fast it writes to the driver depends on the scheduler. Under windows, the scheduling can change from 8ms to 55ms depending on the OS and if multimedia services are currently active or not, as well as it's configurable I believe with an API. I don't test scheduling, but getting 1ms scheduling won't be possible from user space - that will depend on how good the USB CDC driver is written.
As the article you provided points out, write as much as you can in the buffer and let the OS deal with this.
So to answer your question:

This is a Windows question. I've provided what I know above, but it's outside the scope of this project.
Set the DriverOutQueue.

When the SerialPortStream writes data, it writes in chunks as big as possible, and lets the driver decide how much it should take. What it doesn't write gets sent again in the next call. This all happens in a single loop using in a single managed thread using overlapped I/O.


[GOBBLIN-1042] Fix ForkMetric incorrect return type of parent metric object and relevant unit tests

Along with some documentation enhancement, typo fixing, renaming of methods, etc.
JIRA

[ ] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-1042

Description

[ ] Here are some details about my PR, including screenshots (if applicable):

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #2882 into master will decrease coverage by 41.64%.
The diff coverage is 18.51%.

@@             Coverage Diff              @@
##             master   #2882       +/-   ##
============================================
- Coverage     45.73%   4.09%   -41.65%
+ Complexity     9112     747     -8365
============================================
  Files          1921    1924        +3
  Lines         72391   72424       +33
  Branches       7967    7967
============================================
- Hits          33106    2963    -30143
- Misses        36262   69143    +32881
+ Partials       3023     318     -2705

Impacted Files
Coverage Δ
Complexity Δ

...apache/gobblin/metrics/GobblinMetricsRegistry.java
0% <ø> (-27.59%)
0 <0> (-6)

...ache/gobblin/cluster/InMemorySingleTaskRunner.java
0% <0%> (ø)
0 <0> (?)

...g/apache/gobblin/cluster/InMemoryWuSingleTask.java
0% <0%> (ø)
0 <0> (?)

...a/org/apache/gobblin/cluster/SingleTaskRunner.java
0% <0%> (-1.86%)
0 <0> (-1)

...ava/org/apache/gobblin/metrics/GobblinMetrics.java
0% <0%> (-18.87%)
0 <0> (-15)

...he/gobblin/cluster/InMemoryWuFailedSingleTask.java
0% <0%> (ø)
0 <0> (?)

...in/java/org/apache/gobblin/cluster/SingleTask.java
0% <0%> (-81.14%)
0 <0> (-9)

...a/org/apache/gobblin/cluster/ContainerMetrics.java
0% <0%> (-100%)
0 <0> (-4)

...a/org/apache/gobblin/runtime/util/ForkMetrics.java
100% <100%> (ø)
5 <2> (-1)
:arrow_down:

...a/org/apache/gobblin/runtime/util/TaskMetrics.java
81.25% <100%> (ø)
8 <1> (ø)
:arrow_down:

... and 1121 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef3003c...c6e60b3. Read the comment docs.


HADOOP-17608.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ branch-3.2 Compile Tests _

+1 :green_heart:
mvninstall
29m 14s

branch-3.2 passed

+1 :green_heart:
compile
15m 42s

branch-3.2 passed

+1 :green_heart:
checkstyle
0m 36s

branch-3.2 passed

+1 :green_heart:
mvnsite
0m 42s

branch-3.2 passed

+1 :green_heart:
javadoc
0m 48s

branch-3.2 passed

+1 :green_heart:
spotbugs
0m 54s

branch-3.2 passed

+1 :green_heart:
shadedclient
13m 16s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 24s

the patch passed

+1 :green_heart:
compile
14m 54s

the patch passed

+1 :green_heart:
javac
14m 54s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 34s

hadoop-common-project/hadoop-kms: The patch generated 0 new + 92 unchanged - 2 fixed = 92 total (was 94)

+1 :green_heart:
mvnsite
0m 40s

the patch passed

+1 :green_heart:
javadoc
0m 33s

the patch passed

+1 :green_heart:
spotbugs
1m  3s

the patch passed

+1 :green_heart:
shadedclient
13m 24s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
3m 42s

hadoop-kms in the patch passed.

+1 :green_heart:
asflicense
0m 51s

The patch does not generate ASF License warnings.

108m 32s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2929/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2929

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux a79cd3b90546 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.2 / ec5e3e3eccae2ef3f0a112fa60bddfb7f68a7b13

Default Java
Private Build-1.8.0_282-8u282-b08-0ubuntu1~18.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2929/1/testReport/

Max. process+thread count
420 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2929/1/console

versions
git=2.17.1 maven=3.6.0 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.


example_interfaces: 0.9.1-2 in 'rolling/distribution.yaml' [bloom]

Increasing version of package(s) in repository example_interfaces to 0.9.1-2:

upstream repository: https://github.com/ros2/example_interfaces.git
release repository: https://github.com/ros2-gbp/example_interfaces-release.git
distro file: rolling/distribution.yaml
bloom version: 0.10.3
previous version for package: 0.9.1-1

Please hold for coordinated merge...


Update toLocal function to use POSIX compatible path

Fixes issue #503 . On OS X the path separator used in the toLocal function of client.js is causing issues on download. Pretty sure this fix will work on Windows since it supports both forward and back slashes.

Hi there, I've fixed the issue in a slightly different way. Which will be released in 0.9.3


[DO NOT MERGE] Fixes bug of create user without credentials

Hi everyone, I found something that I think it is a bug. The issue happen when you create a user without credentials
Steps to reproduce the issue:

Execute the following script (it is something similar of what is explained in the oktasdk documentation)

import okta
from okta.models.user import User
users_client = okta.UsersClient(os.environ['OKTA_BASE_URL'], os.environ['OKTA_API_TOKEN'])

user =<EMAIL_ADDRESS><EMAIL_ADDRESS>firstName='saml', lastName='jackson')

users_client.create_user(user, activate=False)

Verify the user created in the Okta instance.
The user should be created with status STAGED but it is created with status PROVISIONED.

I only changed one line of code on the create_user method, basically, if activate is True, it will make a request to okta without query parameters (this will create a user with status PROVISIONED as described in OKTA API documentation here), if activate is False it will create a request to Okta with the query parameter activate=false.
Please correct me if I am understanding something wrong.
Also, I marked this PR with [DO NOT MERGE] because this change breaks a test, I tried to fix it but I couldn't find a way to update the tapes, I would like some guidance/instructions on how to do that if possible.

Hi @nicolastrres! Thanks for finding this :bug:. After more investigation, it turns out the current implementation can not send a request when activate is false.
Your suggested solution is on the right track, but I think we still do not send the correct request when activate is false.
Instead, lets change our default parameter value for activate to None, then perform our check:
def create_user(self, user, activate=None):
        """Create a user

        :param user: the data to create a user
        :type user: User
        :param activate: whether to activate the user
        :type activate: bool
        :rtype: User
        """
        if activate is None:
            response = ApiClient.post_path(self, '/', user)
        else:
            params = {
                'activate': activate
            }
            response = ApiClient.post_path(self, '/', user, params=params)
        return Utils.deserialize(response.text, User)

Then, we can make the following calls:
users_client.create_user(user, activate=False)
# Status: Pending activation

users_client.create_user(user, activate=True)
# Status: Password reset

users_client.create_user(user)
# Status: Password reset

Feel free to make these updates and I'll merge it in as an obvious fix.

Hi @jmelberg-okta thanks for your answer, I can make the updates that you suggested but I just have a couple of questions before:

Did you notice that I mentioned that with this change a test is breaking?
How can I generate new tapes? or update the existing one to fix the test?
Are you using any CI/CD tool to run the tests? I am wondering if you have something like a Travis that I can access to see the tests running.

Great questions @nicolastrres.
Yes, the original proposed change would have broken the test_creates_a_user_without_credentials test. However, the updated change passes successfully.
Tapes are distributed across all of our current SDKs, so we are not releasing the ability to add/edit tapes at this time. You can record new tapes locally and test against your organization by updating the .sdk.config.json file. Then run:
$ npm run record-new-tapes

Since we are in the early stages of adding a comprehensive test suite into our SDKs, we are still building out a release process.  This includes an integration with Travis and publishing to our package managers. All of these enhancements are planned in a 1.0 release of this SDK, which our team is working to get up as fast as possible.


Deduplicate mainnet_blocks_from_rpc and optimism_mainnet_blocks_from_rpc

We can share the main flow of fetching block, setting up an RPC storage and running test_execute_alloy. Each chain can just pass in the difference like chain, rpc_url, and block numbers.

I'd like to take this up

@varun-doshi Assigned, thank you!