Adding load without consultation

Highlighted by the NPM load issue introduced in 1.7.
Going forward, can the VS Code team please set the default for all background network activity to OFF. Or, have the update ask for user input for these setting when first run. The ability for extensions to write to user settings has already been added, this (on first load) is the best use for that functionality. I'm perplexed that the VS Code team themselves wouldn't use it.
I'm an advocate for this software, and a contributor to many extensions, but I'm seriously disappointed that the team has decided on behalf of the user community to start using a third party endpoint without telling (or preferably asking) the user first. It is wholly justifiable that NPM would be upset about it as well. I'm sure they would have liked to have been consulted.

We won't have a single setting for tuning off all network activity. You already have individual settings to disable updates and telemetry.

I'm not asking for a setting to turn off all network activity.
You added NEW network activity, without any advice to the user. All I'm saying is, allow them to change the setting ("typescript.disableAutomaticTypeAcquisition") when you add the functionality. All you have to do is ask the user what they want when you first include the setting. Stop assuming everyone wants a particular thing. It's why the setting was added right? Just don't set a default - ask.
You're not instilling confidence in the product, or the company.

cc @egamma @chrisdias @waderyan @dbaeumer

Hi @HookyQR
First off thanks for the feedback here - we have for sure learnt a few things throughout this process.
Right now we are getting close to shipping an update to 1.7.1.  In an attempt to address a few of your points we are going to add a few clarifications in the release notes i.e. how to turn off, how it works.
That said I'd welcome a chance to chat w/ you about this - so if you are interested let me know I'm smcbreen at microsoft.com.  Drop me a line and we can have a quick chat/a quick call as that will probably make it easier to discuss details/ask clarifying questions.
Finally - thanks for the great work on the extensions you have created - the work on formatters in particular was a factor in our formatting related updates in 1.7.x and the marketplace!
Sean


Support overloading LUT bitstream from attributes in .eblif file format

Motivate of the pull request

[ ] To address an existing issue. If so, please provide a link to the issue.
[X] Breaking new feature. If so, please describe details in the description part.
To enhance the arithmetic support in modern FPGAs, LUTs with fracturable outputs are also used to implement adder functions. The carry input and carry output ports of these LUTs are connected in a chain, in the same concept as the 7-series look-ahead carry chain. An example can be found in the QLSOFA architecture.
However, LUT mapped to adder function (3 input and 2 output) does not comply with the conventional LUT definition, which only accept 1 output. As a result, the LUT has to be modelled as .subckt in .blif format.
The .subckt model cannot contain truth table information for the LUTs, which are required to generate bitstream to configure the LUTs. This is due to that a .subckt model is typically considered as a black box.
The truth table information can be defined through the .attr and .param annotation in .eblif file format.
For example,

.subckt adder_lut4 in[3]=$false in[2]=sum_frac_lut4_lut4_out_lut2_out_carry_follower_a_cout[7] in[1]=$false in[0]=$false lut2_out[1]=cout_frac_lut4_lut4_out_lut2_out[1] lut2_out[0]=cout_frac_lut4_lut4_out_lut2_out[0] lut4_out=cout
.param LUT<PHONE_NUMBER>100001

Describe the technical details

What is currently done? (Provide issue link if applicable)
Currently, OpenFPGA bitstream generator only support decoding LUT bitstream based on the truth table of .blif file format.
What does this pull request change?
This PR supports the LUT bitstream to overloaded from the .attr and .param lines in .blif files.
Users can customize through a bitstream setting file (see file format section in documentation) on which pb_type in the VPR architecture can be overloaded.
Also, new commands for read/write bitstream setting files are added to the openfpga shell.

Which part of the code base require a change
In general, modification on existing submodules are not acceptable. You should push changes to upstream.

[ ] VPR
[X] OpenFPGA libraries
[X] FPGA-Verilog
[ ] FPGA-Bitstream
[ ] FPGA-SDC
[ ] FPGA-SPICE
[ ] Flow scripts
[X] Architecture library
[ ] Cell library

Checklist of the pull request

[X] Require code changes.
[X] Require new tests to be added
[X] Require an update on documentation

Impact of the pull request

[ ] Require a change on Quality of Results (QoR)
[ ] Break back-compatibility. If so, please list who may be influenced.

@ganeshgore Due to the missing support on .eblif file (see issue #210), regression tests are in red now. I have manually run a test locally, the bitstream overloading is working well. So I would like to have the .eblif file support landed before merging this PR.

@tpagarani @lnsharma This is the feature to support the soft adder in SOFA architectures. I plan to use the adder benchmark you provided but I see VPR reports critical bugs on the netlist. If the bug can be solved, I will deploy the benchmark in test case.

Can update few files as follows
Add circuit_file=eblif before following line
https://github.com/lnis-uofu/OpenFPGA/blob/a4c266d59a6977f47d2d8937599285d46cae46cd/openfpga_flow/tasks/fpga_verilog/adder/soft_adder/config/task.conf#L44
replace this line with vpr ${VPR_ARCH_FILE} ${VPR_TESTBENCH_BLIF} --clock_modeling route --circuit_format ${CIRCUIT_FILE}
https://github.com/lnis-uofu/OpenFPGA/blob/a4c266d59a6977f47d2d8937599285d46cae46cd/openfpga_flow/openfpga_shell_scripts/bitstream_setting_example_script.openfpga#L3

Can update few files as follows
Add circuit_file=eblif before following line
https://github.com/lnis-uofu/OpenFPGA/blob/a4c266d59a6977f47d2d8937599285d46cae46cd/openfpga_flow/tasks/fpga_verilog/adder/soft_adder/config/task.conf#L44
replace this line with vpr ${VPR_ARCH_FILE} ${VPR_TESTBENCH_BLIF} --clock_modeling route --circuit_format ${CIRCUIT_FILE}
https://github.com/lnis-uofu/OpenFPGA/blob/a4c266d59a6977f47d2d8937599285d46cae46cd/openfpga_flow/openfpga_shell_scripts/bitstream_setting_example_script.openfpga#L3

Great. Let me try this. I can bypass the problem then.


Expose Version Number

To make it easier to catch problems caused by people running older
versions.
Using the CFBundleVersion number to indicate 1.2 beta 5, based on the answers here http://stackoverflow.com/q/19726988/35794 and Apple's documentation here https://developer.apple.com/library/ios/documentation/General/Reference/InfoPlistKeyReference/Articles/CoreFoundationKeys.html#//apple_ref/doc/uid/20001431-102364

<EMAIL_ADDRESS>

Thanks, I'll be merging this soon!

+1

+1

@JohnCoates - Please merge this!

That feature was added as part of
https://github.com/JohnCoates/Aerial/pull/501


Update to latest wgpu-native

[ ] Update the handling of strings for uncaptured error callbacks, see https://github.com/gfx-rs/wgpu-native/pull/262#issuecomment-1549279242.

Hey! I'm interested in having wgpu-py working with the latest version of wgpu-native.
I tried running the update script(s), Ran into another issue that needed fixing.
This pr https://github.com/gfx-rs/wgpu-native/pull/243/ changed the api for getting present modes.
Also its worth noting somewhere that the WGSL language has changed since the release
type became alias and top level let is disallowed in favour of const https://github.com/gfx-rs/wgpu/releases

Yeah, the update process typically comes with added complexities :) - We have the update planned after the current work on mesh morphing.

I would be interesting in helping out to some extent. I've managed to compile wgpu-native for conda forge so hopefully this should make this a "python only exercise".
https://github.com/conda-forge/wgpu-native-feedstock/
I'm happy to compile a few intermediate versions as well if that helps.
Second, there would be the challenge of updating your CIs for PyPI. I know that isn't always trivial.

Thanks. Well one thing that would be great, since the wgpu-native binaries are now on conda, is to enable wgpu-py to use these. I.e. check whether the binary is bundled or detectable on conda, and select one.

Second, there would be the challenge of updating your CIs for PyPI.

Sorry, what do you mean? We already publish binary wheels to pypi. Not sure if its related to your comment, but the wgpu-native repo has CD setup to produce binaries (using manylinux).

wgpu-py to use these

wgpu-py, as installed from conda-forge will use these today. it will pin to the specific version you use since there were some breaks when i tried to use version 0.17

Second, there would be the challenge of updating your CIs for PyPI.

I mean that there work going from version 0.14 to 0.17 and sometimes compilation fails. Not sure if it really is as easy as updating a single line in a configuration file.

You appear to misunderstand the setup here. Let me try to clarify:

wgpu-native is compiled here: https://github.com/gfx-rs/wgpu-native

the build pipeline produces binaries which are published here: https://github.com/gfx-rs/wgpu-native/releases

wgpu-py (this repo) holds a script to download those binaries, so there is no compilation step in this repo

wgpu-py is 99% python wrappers for the c++ interface of wgpu-native

So updating wgpu-py to a newer version of wgpu-native is a three step process:

Pinning to a new version of wgpu-native
Determining what portion of the wrappers need to be updated
Updating and testing the wrappers

Not sure if it really is as easy as updating a single line in a configuration file.

Hihi, I was it was 😄  But the "Determining what portion of the wrappers need to be updated" step that Korijn mentions above is a pretty big step that will cost me multiple days, especially now that we're behind wgpu-native by quite a bit.

Thanks for the explanation. In your experience, would it make sense to update from

0.14 to 0.15, then from 0.15 to 0.16, then from 0.16 to 0.17

or to just try to do the leap from

0.14 to 0.17

Is there a blocker, perhaps meshmorphing, that you still want to accomplish before this work is undertaken (potentially by others)

It depends on the situation, i.e. what changes are made in each release. But it might indeed be easier to do it in 2 steps (or more).

Just letting all the subscribers here know @almarklein is working on this now, see #370

Also see the top post in this issue, that I use for planning the update.


Create Score system UI

Description
I want the player to be able to see the active score value of the game.
Acceptance Criteria

[ ] The score value is displayed on top of the screen
[ ] The score value fades after some time after some time has passed
[ ] The score label can adjust with the view and exists as its own widget blueprint

Required

[ ] Introduce the use of view models to upgrade the score system.
[ ] Segregate the widget from the main UI widget.


Print from Desktop is available from August Update.

Print from desktop is available from 2018 August Update. https://powerbi.microsoft.com/en-us/blog/power-bi-desktop-august-2018-feature-summary/#print. But this page says  "Printing is only available in Power BI service and not Power BI Desktop"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8c738da8-eb0a-7e87-0cbc-f424d4d2f381
Version Independent ID: 1584c6a8-2152-8657-6a0b-6538f94fd1e3
Content: Printing from Power BI service - Power BI
Content Source: powerbi-docs/consumer/end-user-print.md
Service: powerbi
GitHub Login: @mihart
Microsoft Alias: mihart

Hi @v-nisvoo , thank you for your feedback. This issue has been assigned to the author for review.

Re-assigning to the doc owner.

Hi @v-nisvoo ,
I've updated the article with this information. Thanks for catching this!
The DOCS team


Fcrepo 1555

https://jira.duraspace.org/browse/FCREPO-1555

@robyj, why did you close this PR? It is very close to completion...

I had technical issues on friday and messed up my local branches, thinking if I closed the pull request, it might help me re-enable/re-create my local branch. I re-constructed the branch and the modifcations and pushed it up so hopefully its still good. I ran a build and it passed the tests.

This PR appears to be inactive or obsolete, is that correct? I am marking it inactive in advance of the 4.5.1 codefreeze.

Hi, Jared asked me to rebase the code into the latest master branch, which i'm still trying to find time to do
Thanks
Jon


Diaspoara post: Only first image visible

A diaspora post with multiple images is broken on Hubzilla. The first image is visible in network stream only,

Unable to duplicate; either with uploaded photos or linked photos.

I remember very well this bug since  friendica's times...
But I did a test today  and it work well  now, at least using 3 jpeg files  in diaspora upload.
I don't know about >3 or png or others..

I'll post a link to a post if this happens again. Usually this user posts multiple images all two or three days.

Ok - here we go.
Original post: https://sechat.org/posts/3456702
Same post on Hubzilla:

and source:

Difficult problem. Basically the photo isn't being served in HTTPS and we are being blocked from accessing the photo by proxy (mod_sslify). I suspect the server in this case is sniffing the header from our HTTPS image proxy and finding something it doesn't like. The server at http://www.fotos.goehde.com returns:
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 0
Connection: keep-alive
Keep-Alive: timeout=15
Date: Sun, 11 Feb 2018 19:36:19 GMT
Server: Apache
X-Powered-By: PHP/5.5.38
...which is not an image.
Ironically this image would work perfectly if the source image was https but since it is http it would be blocked from being viewed by your browser, and our attempt to get around this restriction by proxying the photo through an https connection on your site is being blocked by the original content server.
I don't know that there is much we can do except ask the person to turn off blocking. If the block is according to user-agent string (a common method for blocking photos) we might be able to provide a different user-agent in our ssl proxy.

okay - I see. The first image is uploaded at sechat.org but the others are served via camo server from http://www.fotos.goehde.com/start.php?vb=995019 and http://www.fotos.goehde.com/start.php?vb=995021

Correct. I cannot easily determine what software is running at www.fotos.goehde.com so it is a bit difficult to determine what blocking mechanisms may be in use and what is triggering it. The Diaspora camo server is apparently allowed to access the photos, but the Hubzilla equivalent is not. We provide a user-agent string of
Mozilla/5.0 (compatible; zot)
I also cannot easily determine what user-agent the Diaspora camo server is providing. I have no problems adding 'Mozilla' compatibility to the user-agent (most everybody does), but I think there would be ethical issues with trying to match the Diaspora camo implementation just to get around somebody's access block. We would effectively be trying to bypass their web server security.

okay. So this is a one posters issue only and I think there is no need to fix it.
Thanks for your assistance Mike.


Metrics extraction subfeature se

Here's the branch I made for SE data. I removed the inner distance calls, I also tried replacing _R1 with _raw like you suggested in your e-mail but I still get this error (note it was the same thing as before I changed it but instead of _raw being the key error it was _R1):
You can use OSD-255 to test
pull metrics
2023-11-02 15:05:29.302 | DEBUG    | dp_tools.core.files.isa_archive:fetch_isa_files:29 - Extracting ISA Archive to temp directory: /tmp/tmpxlsigc0c
2023-11-02 15:05:29.322 | DEBUG    | dp_tools.core.files.isa_archive:fetch_isa_files:29 - Extracting ISA Archive to temp directory: /tmp/tmp4n74_gyj
2023-11-02 15:05:29.341 | DEBUG    | dp_tools.core.files.isa_archive:fetch_isa_files:29 - Extracting ISA Archive to temp directory: /tmp/tmpuwpe1pmp
2023-11-02 15:05:29.344 | DEBUG    | dp_tools.core.files.isa_archive:fetch_isa_files:29 - Extracting ISA Archive to temp directory: /tmp/tmp7xsdv5ff
2023-11-02 15:05:29.363 | DEBUG    | dp_tools.core.files.isa_archive:fetch_isa_files:29 - Extracting ISA Archive to temp directory: /tmp/tmpo_cjt231
2023-11-02 15:05:29.363 | DEBUG    | dp_tools.core.files.isa_archive:fetch_isa_files:29 - Extracting ISA Archive to temp directory: /tmp/tmpkf_qohqv
Not implemented for dataframe extraction: heatmap, skipping this plot with Title: FastQC: Status Checks
/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py:186: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
self.metrics = self.metrics.append(df_updated_metrics)
Not implemented for dataframe extraction: heatmap, skipping this plot with Title: FastQC: Status Checks
/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py:186: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
self.metrics = self.metrics.append(df_updated_metrics)
/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py:186: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
self.metrics = self.metrics.append(df_updated_metrics)
/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py:186: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
self.metrics = self.metrics.append(df_updated_metrics)
/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py:186: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
self.metrics = self.metrics.append(df_updated_metrics)
/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py:186: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
self.metrics = self.metrics.append(df_updated_metrics)
/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py:186: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
self.metrics = self.metrics.append(df_updated_metrics)
/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py:282: FutureWarning: merging between different levels is deprecated and will be removed in a future version. (4 levels on the left, 1 on the right)
merged = metrics_reset.merge(
Traceback (most recent call last):
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3629, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: '_raw'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/workspace/dp_tools/multiQC_output_ASB/SE_datasets_w_all_multiQC/OSD-255/../../../extract_dataset.py", line 36, in
main()
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/workspace/dp_tools/multiQC_output_ASB/SE_datasets_w_all_multiQC/OSD-255/../../../extract_dataset.py", line 33, in main
metricsExtractor.process_metrics(assay_type=AssayType.bulkRNASeq).to_csv(f"{osd_id}_summary.csv")
File "/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py", line 901, in process_metrics
df_samplewise_raw = _process_fastqc_data(df_interim, "raw reads")
File "/workspace/dp_tools/dp_tools/core/utilites/metrics_extractor.py", line 352, in _process_fastqc_data
df_fastqc_subset = df_full.xs(
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/pandas/core/generic.py", line 3836, in xs
loc, new_ax = labels.get_loc_level(key, level=level, drop_level=drop_level)
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 2972, in get_loc_level
loc, mi = self._get_loc_level(key, level=level)
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 3005, in _get_loc_level
loc, new_index = self._get_loc_level(k, level=lev)
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 3113, in _get_loc_level
indexer = self._get_level_indexer(key, level=level)
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 3222, in _get_level_indexer
idx = self._get_loc_single_level_index(level_index, key)
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 2802, in _get_loc_single_level_index
return level_index.get_loc(key)
File "/home/gitpod/miniconda3/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3631, in get_loc
raise KeyError(key) from err
KeyError: '_raw'

Closing with the following branch replacing & working (tested on OSD-420):
https://github.com/J-81/dp_tools/tree/metrics_extraction_subfeature_SE


[Create New Issue]: Complete Concept of OOPS

Description
Description 📜
It will include the following points -

Introduction
Class
Objects
Encapsulation
Abstraction
Polymorphism
Inheritance
Dynamic Binding
Message Passing

Domain
C/CPP
Type of Contribution
Documentation
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

/assign


Missing Scratches and Consoles node in Project window

The Scratches and Consoles node is missing in the Project window for Bazel projects opened on macOS with:
IntelliJ IDEA 2018.2.7 (Ultimate Edition)
Build #IU-182.5107.41, built on November 27, 2018

This is fixed for newer plugin releases, but they're only available for 2018.3 (https://plugins.jetbrains.com/plugin/8609-bazel).


fix: broken shutdown for services on fraud proofs receival

Closes #1205

Codecov Report

Merging #1220 (24dec8f) into main (e89b628) will decrease coverage by 0.21%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1220      +/-   ##
==========================================
- Coverage   56.05%   55.84%   -0.22%
==========================================
  Files         160      160
  Lines        9692     9697       +5
==========================================
- Hits         5433     5415      -18
- Misses       3727     3749      +22
- Partials      532      533       +1

Impacted Files
Coverage Δ

nodebuilder/daser/module.go
89.47% <100.00%> (ø)

nodebuilder/header/module.go
92.18% <100.00%> (ø)

nodebuilder/state/module.go
93.93% <100.00%> (+1.08%)
:arrow_up:

fraud/subscription.go
46.15% <0.00%> (-23.08%)
:arrow_down:

fraud/proof.go
62.06% <0.00%> (-6.90%)
:arrow_down:

share/discovery.go
32.29% <0.00%> (-3.13%)
:arrow_down:

fraud/bad_encoding.go
63.00% <0.00%> (-3.00%)
:arrow_down:

fraud/pb/proof.pb.go
34.71% <0.00%> (-1.90%)
:arrow_down:

das/coordinator.go
92.64% <0.00%> (+1.47%)
:arrow_up:

ipld/get_namespaced_shares.go
90.51% <0.00%> (+2.18%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

The FraudSyncing test was failing because of https://github.com/celestiaorg/celestia-node/issues/1236. Making the explicit connection light-> full fixed the issue


Microsoft.ML.Tokenizers.Tests.TiktokenTests.TestTokenizerUsingExternalVocab failing to download gpt2.tiktoken

Build
https://dev.azure.com/dnceng-public/public/_build/results?buildId=823668
Build leg reported
Microsoft.ML.Tokenizers.Tests.TiktokenTests.TestTokenizerUsingExternalVocab
Pull Request
https://github.com/dotnet/machinelearning/pull/7252
Known issue core information
Fill out the known issue JSON section by following the step by step documentation on how to create a known issue
 {
    "ErrorMessage" : ["Microsoft.ML.Tokenizers.Tests.TiktokenTests.TestTokenizerUsingExternalVocab", "File not found"],
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

[ ] Feature changes/additions
[ ] Bug fixes
[x] Internal Infrastructure Improvements

Release Note Description
Additional information about the issue reported
It looks to me like this file no longer exists:
https://github.com/dotnet/machinelearning/blob/be1e428d41b5936903172855f7f30861ca7eb49a/test/Microsoft.ML.Tokenizers.Tests/TitokenTests.cs#L100C136-L100C149

Known issue validation
Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=823668
Error message validated: [Microsoft.ML.Tokenizers.Tests.TiktokenTests.TestTokenizerUsingExternalVocab File not found]
Result validation: :white_check_mark: Known issue matched with the provided build.
Validation performed at: 10/1/2024 5:06:40 PM UTC

Report

Build
Definition
Test
Pull Request

823907
dotnet/machinelearning
Microsoft.ML.Tokenizers.Tests.WorkItemExecution
dotnet/machinelearning#7253

823750
dotnet/machinelearning
Microsoft.ML.Tokenizers.Tests.WorkItemExecution

823744
dotnet/machinelearning
Microsoft.ML.Tokenizers.Tests.WorkItemExecution

823668
dotnet/machinelearning
Microsoft.ML.Tokenizers.Tests.WorkItemExecution
dotnet/machinelearning#7252

Summary

24-Hour Hit Count
7-Day Hit Count
1-Month Count

4
4
4

@ericstj may I ask why this is a dotnet/dnceng issue❓ shouldn't it be moved into dontet/machinelearning❓

I think it is fine to move it to the machinelearning repo.

I don't have the required rights in machinelearning either. @mmitche can you transfer the issue❓

Oh that's odd.  Maybe I clicked the wrong button when opening this.  I'll transfer.  Sorry for that.


Migrated to Poetry for easier dependency management.

I introduced the requirements.txt and personally I'm fine with going to poetry. Sound good @markokraemer?


Regeneration of pnpm-lock.yaml results in TypeScript errors during pnpm install

A general question of whether this is or should be part of any test suite?

Clone the repo
rm pnpm-lock.yaml
pnpm install

Currently, this fails due to TypeScript issues in packages/nextra/src/compile.ts. It prevents projects that integrate with source copies of ./packages to generate their own lock files.

Because we have dependencies on other code (internal and external) that we inline, and by rebuilding the lock file we can reestablish a reproducible build baseline as we go.

@robbear couldn't you use pnpm install --no-frozen-lockfile?
I think that might help. Helped me a lot

@robbearне могли бы вы использовать pnpm install --no-frozen-lockfile? (Если у вас есть оригинальный файл блокировки)
Я думаю, это может помочь. Мне очень помог

Does not help me, I can not run the project

Closing since I think there is nothing wrong with Nextra here


training question

Hello! I trained the code 'train_refine.py' without any revise using my own dataset to get the pth file.However,I find the network structure I have got is different from the pretrained model you have provide:pytorch_mobilenetv2.The following image1 is your pretrained model structure and the second one image2 is mine

image1

image2
Could you please tell me why the network structure of mine has another 10 layers than yours? and if I want to change the code to get a model like yours,how should I do ?
2.Another question is that I test the model between yours and mine on CPU,I found that your model's flops is twice smaller than mine，which using the same training code.I don’t know if it’s because my network structure has 10 layers more than yours. Like the first question said.
my English is not good,sorry And I hope you can solve my doubts,Thank you!

When you use the train script, did you set --model-backbone mobilenetv2?

When you use the train script, did you set --model-backbone mobilenetv2?

yes,I have done this,but still have the same result

When you use the train script, did you set --model-backbone mobilenetv2?

You have to understand one thing. the pytorch_mobilenetv2.pth file only contains the weights and it does not define model architecture. When you load it in python, the model architecture is defined by the code in /model, then you load the weights in pth. When you train your model, the architecture is also defined in /model. I don't know what you did wrong. You have to debug yourself.


Ceylon.io transitive dependencies uses two distinct versions of jboss logging

This dependency comes from the “ceylon.net” module that lead to conflicting jboss logging versions, undertow 1.0.0.Beta20 depends on org.jboss.logging/3.1.3.GA and org.jboss.xnio.api/3.1.0.CR7 which depends on org.jboss.logging/3.1.2.GA:
https://modules.ceylon-lang.org/modules/io.undertow.core/1.0.0.Beta20
https://modules.ceylon-lang.org/modules/org.jboss.xnio.api/3.1.0.CR7

@quintesse Any chance we can get this one finally closed for 1.2?

I had forgotten all about this issue, will take a look at it again ASAP.

Ok, so the whole reason for this was that the Undertow API was changed so the code doesn't compile anymore. Would you be willing to take a look at this @matejonnet ?
The new imports (which are all available on Herd) would be:
    import io.undertow.core "1.1.0.CR3";
    import org.jboss.xnio.api "3.3.0.Beta3";
    import org.jboss.xnio.nio "3.3.0.Beta3";

Thanks Matej!


Databricks - GCP Key Missing in Multi-Node Cluster

Hi Sam,
I ran into an interesting issue today Im not sure of the cause  but suspect its me - its just happened so need to do some deeper digging but thought to reach out to you in case you spotted anything a rookie mistake quickly and could help.
My notebook in the community edition works great and writes my dataframes out to the designated BigQuery tables as expected.  All good.
Now I've recently subscribed to a paid databricks account, I run the exact same code in a simple 2 node cluster (1 driver + 1 worker) and I get an exception that the google keyfile cannot be located.
https://www.dropbox.com/s/6kpsqsgqkb2tp2a/Screenshot 2017-05-09 19.28.39.png?dl=0
but the key is there as can be seen by this:
https://www.dropbox.com/s/yzviehngyhlz5e6/Screenshot 2017-05-09 19.29.43.png?dl=0
As mentioned its using exactly the same code in both editions - the way for prototyping I set up my key was in the notebook to just save the string of the json key to the local file system using below
dbutils.fs.put("file:/gcpkey.json",gcpKey)
and to the BQ library reference as "/gcpkey.json" - which as mentioned works fine in community editon.
The difference between community and the the proper subscription as you know is on community the driver and worker is a single physical node.  I can only guess at the moment (as this just happended) that the key file isnt on the worker in a multi-node environment, this dbutils command just saves on the driver node and I need to distribute it to the worker?  Documentation on it isnt so clear.
I assume youve had this working on a multi-node cluster on dbricks so no doubt its my rookie mistake on the platform!  Is this the issue in that it just writes to the Driver node or something else?  What are your best pracitces for distributing a senstive key like this securely?
I'm assuming there is a better mechanism for production using the Databricks REST API to bootstap a cluster with this (to be investigated), but for now was just using this simple mechanism.
Thoughts?
Thanks heaps for your help!
Cheers

not sure how to accomplish this in a notebook, but I've found I have to include my json key with the --files /path/to/key.json when running spark-submit.  that should get it on all the worker nodes where it is readable.

Hi guys
@kurtmaile you are absolutely right, the worker node cannot find the gcp file.
What you can do is create an init script which will load the credentials file when a cluster is created
Here is an example script
dbutils.fs.rm("dbfs:/databricks/init/copy-google-key-json.sh",true)
dbutils.fs.put("/databricks/init/copy-google-key-json.sh","""
#!/bin/bash
cat <<EOF > /databricks/key.json
{paste json here}
EOF
""")

I hope this helps!

Hi guys,
Awesome thanks for this info this helps greatly!
I will give the init feature a go today, much appreciated once again for the speedy response.
Cheers


Give a better documentation for noob user

Sorry if my question seem really useless.
I look to use your plugin, but i don't know how to do this.
Could you put an entire example with the html page, load of module and other needed code.
Thank you.

Any small example would be much appreciated!

A demo page would be great, which would also display how this plugin looks like

If anyone, any user of this plugin have successfully implemented this plugin and currently using it in any extent. Please provide an example and/or screenshot.

Hi @carlssonn , you need to include moment.js chart.js timeline.js in the page using this chart. The bundled version chartjs not supported. Just use this plugin like normal chart, using the config in readme and it should work.

@fanthos - I think a plunkr, jsbin, codepen, or even a sample html file would help. My console is full of errors with this.

Here is an example I came across: https://flows.nodered.org/flow/3827f07ed08826b01bd71b2c874f7fdc


chore: update references for old repository name

Related to #105
Updating references for csi-driver-image to container-image-csi-driver in the listed files/packages.

go.mod
package imports
Dockerfiles
docs
scripts

Waiting on CI builds to be fixed in #126 before we merge this PR.

Not sure why it's not getting triggered in your case. Maybe it's related to the repository name/branch changes.
Are you sure you've updated your fork correctly? For #126, the workflows are triggering.

Are you sure you've updated your fork correctly? For #126, the workflows are getting triggered.

I think the workflows got triggered for #126 because it had the changes to fix the CI. As for updating my fork, I re-created my fork after name change so I don't think the issue is related to my fork.
And now the workflows got triggered after I rebased this PR with your CI changes.

https://github.com/warm-metal/container-image-csi-driver/blob/03636ae3e6487d9096794f5f441ac4a168633a93/.devcontainer/devcontainer.json#L4
@mugdha-adhav Please change the name as well in your PR
-	"image": "ghcr.io/warm-metal/csi-driver-image/devcontainer:latest",
+	"image": "ghcr.io/warm-metal/container-image-csi-driver/devcontainer:latest",

@mugdha-adhav Please change the name as well in your PR

Not just the name, but I think we'll also need to update all the references of csi-driver-image in the devcontainer.json files. Working on it.

I tried updating all references of older repository in a single PR, but it's getting too large and hard to track.
Reverted some of the recent commits. Will create separate PRs for those changes.
@mbtamuli JFYI, your dev-container related change will also go in one of the follow up PR.


QUESTION: make:migration

Hi!,
I would like to ask where to find the file that contains the command make:migration.
My scenario is, im going to extend this, to support migration to certain path. Thanks!

Here it is https://github.com/adonisjs/adonis-cli/blob/develop/src/Commands/Make/Migration.js
Suggest to use Forums or discord channel for quick questions. Will help you in getting quick replies and helps us in focusing on main issues 😄

Ah.. i was looking at the node_modules of the framework haha
Anyway, is it located in the framework? what folder?

The adonis make:migration command is executed using adonis-cli, the code it executes is part of that package rather than the adonis-framework package.


Question pi zero w

Hello!
Congratulations on your wonderful work.
I decided to remove all my hubs and replace them with deconz and homebridge-hue.
I have pi zero w and I plan to use it, will I have problems with it or will I have to replace it with a more powerful one?

I don't have any personal experience with the Pi zero.
I'm running deCONZ and two homebridge instances on a Pi 3B+ without any issues.  One instance runs homebridge-hue, the other homebridge-otgw, homebridge-p1, homebridge-ws, and homebridge-zp.

I've been testing ConBee 2 with Homebridge-hue on pi zero w for two days and it works fine. I noticed that the processor runs between 55-75%. I had cases when I turned off the lights after about 10 minutes in the annex again showing that the lights were on, The same case I had with the ikea gateway, the bulbs are on ikea. Has anyone had such a case and what is the reason?

How did you turn off the lights?  What type of lights are you using?

I turned off the lights with the Home application.
bulb is Tradfri bulb E27 opal 1000lm

That should work normally.
The issue could be:

(with the IP network) between HomeKit and the accessory (IKEA hub, Hue bridge, or homebridge - whomever is exposing the light to HomeKit); or
(with the IP network) between homebridge-hue and the Hue bridge/deCONZ gateway; or
(with the ZigBee network) between the light and and the hub/bridge/gateway.

You want to check a native hub/bridge/gateway app to see if the hub/birdge/gateway also reports the wrong light state.  If not, you want to check the homebridge -D log to see if homebridge-hue sends the wrong state to HomeKit.  If not, the issue is with your home network or HomeKit app losing the link to the HomeKit accessory.

Thank you for the tips I will take the time to look at in detail.  by the way, I just found that when I turn off a light bulb through the wall switch in the home application and also in phoscon it remains inclusive.

You don't want to use 20th century wall switches with your smart lights.

Yes, but other accessories I own update  my tile  in the home app when I unplug and turn them on.

This behaviour is caused by ZigBee: it doesn't detect that devices are powered down.  homebridge-hue can do nothing about this.

I didn't know this, now everything became clear to me.  Once again, congratulations on the well-supported project

In the previous version, when turning off the bulb with the wall switch in the app, the home did not respond, and after restoring the power the tile did not update and constantly showed that it did not respond.  Now with the new update, I notice that the bulb is updating.  My question is whether you correct this or the problem was in my configuration

I haven't changed anything in that area recently, see https://github.com/ebaauw/homebridge-hue/releases.


URI ue instance id wildcard 0xFFFF in lieu of 0x0000

The following change adds some verbiage that states that it is valid to not set the instance_id that indicates that we do not care. This is often used when referring to services that do not have more than one instance.
This change is to help clarify the requirements after looking at the C++ UUri validators.
also minor fixes to the specifications w.r.t pattern matching for resource_id
#217

@gregmedd @PLeVasseur I've updated the any service instance id to be 0xFFFF like we talked about so it is clearer. Now the  wildcard value and default instance id are two separate values and it is simpler to code/understand. Thanks everyone for your feedback!

@stevenhartley I am generally ok with the change to the entity instance wildcard pattern. However, introducing this semantic change after I had approved the PR which was originally only about adjusting some wording seems a little off. I would appreciate if we could create a new PR for additional changes that are outside of the original scope of a PR in the future.

@PLeVasseur
My understanding is that this change means that URIs used for configuring a streamer may need to be updated. Listening for messages from a particular entity type will now require addition of the instance ID wildcard:
up://origin/A1/1/D3A2

will need to be changed to
up://origin/FFFF00A1/1/D3A2

to receive the same messages, right?

@PLeVasseur
My understanding is that this change means that URIs used for configuring a streamer may need to be updated. Listening for messages from a particular entity type will now require addition of the instance ID wildcard:
up://origin/A1/1/D3A2

will need to be changed to
up://origin/FFFF00A1/1/D3A2

to receive the same messages, right?

We will have to update these lines.
Opened issue here


Display line numbers

What I am really missing is the possibility to display line numbers.

FYI, this is now implemented in my fork: https://github.com/nikrolls/json-formatter. It's currently publishing to the Chrome store and should be available shortly.


docs: Update README.md

Changes mentioned in 1132

@isaachinman can you review it?

@isaachinman All resolved, i guess.


Merge fastq files before raw read qc

This is not an ideal solution (performance and disk usage wise), but needs to be fixed before a release. Otherwise, only one of the file is included in the MultiQC report.
Closes #211

PR checklist

[ ] This comment contains a description of changes (with reason).
[ ] If you've fixed a bug or added code that should be tested, add tests!
[ ] If you've added a new tool - have you followed the pipeline conventions in the contribution docs
[ ] Make sure your code lints (nf-core lint).
[ ] Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
[ ] Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
[ ] Usage Documentation in docs/usage.md is updated.
[ ] Output Documentation in docs/output.md is updated.
[ ] CHANGELOG.md is updated.
[ ] README.md is updated (including new tool citations and authors/contributors).

One other way of doing it is to create unique ids for each read and using that as prefix. Then when you merge the bam files belonging to the sam samples you revert back to the old ID. I think that's how it's done in the RD pipeline
Create unique read IDs: https://github.com/nf-core/raredisease/blob/c7a6f389c9b958bf5099c1f48c9b8145d755510f/subworkflows/local/utils_nfcore_raredisease_pipeline/main.nf#L83
Merging: https://github.com/nf-core/raredisease/blob/c7a6f389c9b958bf5099c1f48c9b8145d755510f/subworkflows/local/alignment/align_bwa_bwamem2_bwameme.nf#L66

I'm not sure I understand, maybe I need a walkthrough from you or Ram. Do you mean that this way, without modifying the fastq files, I can get FastQC to summarise the reads per sample in the MultiQC report?

Ah maybe I misunderstood the aim. I was thinking that it would be good to get the readQC per "read" rather than sample as you might have the same sample sequenced on different flowcells at different times. But perhaps that was what you had already and wanted to move away from 😆

We could revisit it later and I can have a look at a raredisease MutliQC-report.

Before allowing multiple input files per sample, I ran one FastQC process per input file.
After allowing multiple input files per sample, FastQC would still run one process per input file, but they would be named the same (meta.id), meaning that FastQC only grabbed and displayed the latest produced file.
This PR cats together multiple input files into one per sample, to mimic the behaviour of 1.

sample,file,family_id,paternal_id,maternal_id,sex,phenotype
HG002,https://raw.githubusercontent.com/genomic-medicine-sweden/test-datasets/nallo/testdata/HG002_PacBio_Revio.bam,FAM,XXX,YYY,0,1
HG003,https://raw.githubusercontent.com/genomic-medicine-sweden/test-datasets/nallo/testdata/HG002_PacBio_Revio.bam,FAM,XXX,YYY,1,1
HG003,https://raw.githubusercontent.com/genomic-medicine-sweden/test-datasets/nallo/testdata/HG002_PacBio_Revio_copy.bam,FAM,XXX,YYY,1,1

2.

After allowing multiple input files per sample, FastQC would still run one process per input file. Perhaps they would be named the same (meta.id), meaning that FastQC only grabbed and displayed the latest produced file (?)

Yes I think you need to create a unique meta.id for the samples that have several input files. You would use this unique meta.id for fastqc and alignment and then when you merge the bam files you would revert meta.id to be the original sample specific name,
But I agree we can discuss this later :)


Update resolution index if settings were changed

Fixes https://github.com/smartdevicelink/sdl_hmi/issues/547
This PR is ready for review.
Testing Plan
Precondition:

HMI and SDL are started
Navigation mobile app is registered

Steps to reproduce:

Mobile app starts video streaming
Go to HMI main menu
Menu item Send VIDEO_STREAMING capabilities
Update video capabilities to any preferred resolution values (for example "resolutionWidth": 320, "resolutionHeight": 200)
Press Send notification
Note: HMI shows video with the wrong resolution also in case when the mobile app starts stream with the resolution different from the default(800*480)

Expected behavior:

Mobile app starts streaming with the new resolution 320x200
HMI shows streaming with new resolution 320x200

Summary
Looks like video streaming resolution combobox is not affected by settings changed in code editor. This PR provides changes related to proper calculation of video streaming resolution index once user applies custom settings.
CLA

[x] I have signed the CLA

Internal review has passed. Submitted to Open


Looks like something went wrong while downloading the dart sdk. Can you delete the cache folder inside bin and then run flutter doctor again? What's the output?

Looks like something went wrong while downloading the dart sdk. Can you delete the cache folder inside bin and then run flutter doctor again? What's the output?
Are you by any chance behind a firewall (or using a proxy) that might be blocking the download of the dart sdk?
Originally posted by @goderbauer in https://github.com/flutter/flutter/issue_comments/339395571

this makes terminal stuck at  Building flutter tool...
Running pub upgrade...

@Mr-Perfectuz
Please provide flutter doctor -v and elaborate on what are you trying to do when you get this error. The original link you pasted above throws 404, so if you have the proper link of the issue, please provide that too.
Thanks.

Without additional information, we are unfortunately not sure how to resolve this issue.
We are therefore reluctantly going to close this bug for now.
Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact description of what happens, logs, and the output of flutter doctor -v.
All system setups can be slightly different, so it's always better to open new issues and reference related issues.


update modstofullrecord.xsl

Add clickability for links.  change some label names. add hover text for related item

This looks good!  Your alternative labels for the different types of relatedItems are more intelligible to me, and the alt texts make sense.  I added a few comments about minor changes, but otherwise this seems like it will be a benefit for users.


HDDS-8996. Remove the redundant getBucketInfo call in OMKeyCreateRequest

What changes were proposed in this pull request?
Remove the redundant getBucketInfo call in OMKeyCreateRequest
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8996

cc @DaveTeng0 @tanvipenumudy  can you please take a look?

@guohao-rosicky, Thanks for the patch.
I can see one call which try to read from cache and if cache miss then it reads from db. Other call just reads from the cache. And now you are using call which just reads from the cache. Since bucket table is FULL_CACHE, I think there should not be any problem just reading from the cache here.
Change LGTM.

The getBucketInfo method is called twice in OMKeyCreateRequest, so far it seems like it only needs to be called once, and although getBucketInfo is a relatively lightweight read from FullTableCache, it's still a good idea to remove the redundancy

Thanks for the patch @guohao-rosicky.
Changes look fine but the PR description and jira description are missing the motivation behind this change. It would be great if you could update that.
Thanks for the review @hemantk-12
Description has been added.


Fix: move pointer class to top of Chip component

Description
Allows callers to set the pointer in the chip if needed
Risk
Check chips are fine afterwards

Uncontroversial, merging. cc @Duncid @PopDaph PMRW

Actually, it was better before :sweat_smile: No issue since the front wasn't bumped but will revert; Sorry for the bother


ci: Updating k8s test suite for NPM to latest

Reason for Change:

Updates test suite being used by NPM to latest to reduce overhead of maintaining forks.
Issue Fixed:

Requirements:

[x] uses conventional commit messages

[ ] includes documentation
[ ] adds unit tests
[x] relevant PR labels added

Notes:

/azp run Azure Container Networking PR


Issue with installing the required Python packages (on both Windows 10 and Ubuntu 20.04)

Hello Stefanie,
I have recently received my copy of the second edition of your book.
Trying to follow along, I ran into an issue an page 39 with the command: (book_env) $ pip3 install -r requirements.txt
The problem seems with "wheel" or later in the requirements file:
Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /home/guido/anaconda3/envs/book_env/bin/python /home/guido/anaconda3/envs/book_env/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpu0w3a_o8
         cwd: /tmp/pip-install-ycgjf7ri/scipy_652aaa3f60914357b64375128a803e93
    Complete output (171 lines):

I have retried this on Windows 10 with Anaconda 3 and then also on a "freh install" of Ubuntu 20.04 LTS in a virtual machine.
Please see the attached traces from both OS.
Please let me know if you require additional information.
Best regards,
Guido
Windows.txt
Ubuntu.txt

Hi Guido,
Glad you got it working, and thanks for looking into this further. I have some additional questions for you on your Windows setup since pywin32 should be automatically installed with the dependencies for Jupyter:

Are you using Anaconda or Miniconda?
Also, if you take a look at the packages in the Anaconda installer, it has pywin32 included for Python 3.8 and 3.7. I remember you were using Python 3.9, but I don't see that as an option. Can you check if the installer you used has pywin32?

Hello Stefanie,
1.) I'm using Anaconda 3 (Anaconda3-2020.11-Windows-x86_64.exe)  - this is the current version that you would download from here: https://www.anaconda.com/products/individual [https://repo.anaconda.com/archive/Anaconda3-2020.11-Windows-x86_64.exe]
2.) To be honest, I have no clue how or when the Python 3.9 came into play - I must have inadvertedly installed that through Anaconda Navigator when being prompted to upgrade something and then decided to upgrade Python as well (which was obviously not a good idea in terms of compatibility)
Now, after clean up and reinstall of Anaconda, the base environment is on following versions:

  python 3.8.5
  pywin32 227
  jupyter 1.0.0
  jupyterlab 2.2.6

I have simulated the end-to-end installation process on a clean Windows 10 environment running on VirtualBox and using a freshly installed Windows 10 (Windows 10 20H2.iso).
When I create the book_env with:
conda create --name book_env python= 3.8
the book_env has the following release:  python 3.8.8.  There is no  pywin32, jupyter or jupyterlab yet.
I need to install git for windows (Git-2.31.1-64-bit.exe). After
(book_env) PS C:\Users\guido> git clone https://github.com/stefmolin/Hands-On-Data-Analysis-with-Pandas-2nd-edition.git
(book_env) PS C:\Users\guido> cd Hands-On-Data-Analysis-with-Pandas-2nd-edition
(book_env) PS C:\Users\guido\Hands-On-Data-Analysis-with-Pandas-2nd-edition> pip3 install -r .\requirements.txt
Completes successfully. Versions are:

  python 3.8.8
  pywin32 300
  jupyter 1.0.0
  jupyterlab 3.0.4

However, trying to run jupyter lab results in error:

When I then execute
(book_env) PS C:\Users\guido\Hands-On-Data-Analysis-with-Pandas-2nd-edition> conda install -c anaconda pywin32
The following indicates, that pywin32 version 227 is installed (instead of 300)

However, when verifying the pywin32 version, it still is 300! But now jupyter lab is running!

... That was a very long answer to a short question, my apologies! I do not mean to waste your time - but being atomic physicist by trade, I had to research the issue under "lab conditions". The traces show that

a) pywin32 is installed in version 300 together with python 3.8.8 but this does not help jupyter lab to work in itself
b) The installation "conda install -c anaconda pywin32 magically seems to fix the issue, although it apparently does not change the version of pywin32
c) After the installation, jupyter labs works - but we don't fully understand why.

Maybe you can read more from the traces - but don't spend to much time.
This may be an effect of "electron psychology" as my old Physics teacher would have suspected ;-)
Please let me know if there is anything else you need to have simulated or traced - happy to do so if it helps.
Best regards,
Guido

Hi Guido,
Thank you for your detailed response. From my initial research, it does seem to be an issue with the installation of the pywin32 package and is still a bug in the latest version of Anaconda (as per this comment on a similar issue). I also remember seeing something about why the command you have in b) fixes the issue: there is a setup bug, but the anaconda command runs a post install script to fix it.
I don't think we need to look into it any further since we know a fix that works consistently -- thanks for testing again! I pinned this issue for others to reference :)

how do i transfer my exercises to github?


PCA implementation

Fixes #39
Hi @ashwani-rathee ,
I added the basic implementation for PCA using numpy.
Please have a look and let me know if any changes are required.

Hi @geekquad,
Please have a look and let me know if any changes are required.
Thanks!


Why are we catching the error and not raising it?

In authenticatable.rb, we have:
        begin
          Knock::AuthToken.new(token: token).entity_for(entity_class)
        rescue
          nil
        end

Is there a reason why we catch the error here and not expose it? I thought I had an implementation error but it turned out to be just a stale token.

I had one of these *** JWT::VerificationError Exception: Signature verification raised and was expecting the API to render 401 unauthorized, instead it went straight through.

I raised the same issue here:  #122


[docs] Fix Joy UI URL to tokens

Changed in #35554, https://mui.com/joy-ui/customization/theme-tokens/ is a 404

👍 thanks!

For context, I found these two in https://search.google.com/search-console/index/drilldown?resource_id=sc-domain%3Amui.com&item_key=CAMYDSAC, pages that Google used to index but are now 404. The goal isn't to get the list to 0, I used it to find cases like these were we might have broken backlinks.
Speaking of broken backlinks, ahrefs can also be used: https://app.ahrefs.com/v2-site-explorer/broken-backlinks?anchorRules=&domainNameRules=&followType=all&grouping=one-per-domain&limit=50&mode=prefix&offset=0&refPageTitleRules=&refPageUrlRules=&sort=traffic&sortDirection=desc&surroundingRules=&target=https%3A%2F%2Fmui.com%2F&targetUrlRules=.


[CI] Allow CI_PYTEST_ADD_OPTIONS to be unbound.

This patch allows the test script to execute normally when CI_PYTEST_ADD_OPTIONS is not available.

cc @u99127 @roastduck


Add a meta description

https://developer.chrome.com/docs/lighthouse/seo/meta-description?hl=en

"Superhero is a multi-blockchain wallet to manage crypto assets and navigate the web3 and DeFi space. Powered by æternity."

Right. I agree although my idea was to emphasize that our product is not just "a wallet" (one of many) but "THE wallet" ... :)


Appstudio update mvp-test-component

Pipelines as Code configuration proposal

Pipelines as Code CI/mvp-test-component-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
9 seconds
init

✅ Succeeded
16 seconds
clone-repository

✅ Succeeded
1 minute
build-container

✅ Succeeded
20 seconds
clair-scan

✅ Succeeded
17 seconds
deprecated-base-image-check

✅ Succeeded
18 seconds
inspect-image

✅ Succeeded
58 seconds
clamav-scan

✅ Succeeded
10 seconds
sbom-json-check

✅ Succeeded
8 seconds
label-check

✅ Succeeded
8 seconds
optional-label-check

✅ Succeeded
7 seconds
show-summary


Cancel in-flight Mutations and Queries while enable: false

We have to make in opt-in by concurrency.cancelDisabled: true (or smth like it).

Because Mutations are not idempotent, it's better to just ignore its result instead of real cancel — call .finished.skip instead of .finished.success/finished.failure. Query could be canceled.

related to #189, it's going to alter the same field of factories API


Interactive mode - compile and run exercise after saving file

Is your feature request related to a problem? Please describe.
It would be nice to have the same feature rustlings has, the watch mode
Describe the solution you'd like
Our CLI should have a new command called watch. This command would start an interactive mode. Everytime a user saves the exercise being run, it should compile and run the exercise, and all other commands should consider the current context.
For exemple, If I start the watch mode with the command golings watch it would start from the next unsolve exercise, and commands like hint, list, etc

cc @lalizita


[BUG]:  Memory consumption by fp16 is not normal

🐛 Describe the bug
When i used pytorch origin amp, the gpu memory is much smaller than colossai, why?
the config is
from colossalai.amp import AMP_TYPE
from colossalai.zero.shard_utils import TensorShardStrategy
from colossalai.nn.optimizer import HybridAdam

fp16 = dict(
    mode=AMP_TYPE.TORCH,
)

optimizer = dict(
    type=HybridAdam,
    lr=0.001,
    # weight_decay=1e-2,
)

model | dataset | machine | batch | gradient accmulate size | ZeRO | speed | GPU memory | OPT | tensor_placement_policy |   |
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
ir18 | private dataset | 1 | 64 | 1 | no ZeRO | 24%|██▍       | 2089/8549 [02:51<08:39, 12.43it/s] | 8703M | HybridAdam |   | single machine + Engine |
ir18 | private dataset | 1 | 64 | 1 | no ZeRO | 19%|█▊        | 1599/8549 [02:24<10:21, 11.17it/s] | 5769M | HybridAdam |   | single machine  + wo Engine + pytorch origin fp16 |

Environment
No response

The difference between my origin pytorch implementation  and colossai is convert_to_amp API which using TorchAMPModel to decorate the origin model.
I have tested three different cases:
1  using  torch.cuda.amp.autocast(True) inside model forward function:
class model(nn.Module):
    def __init__():
        ....
    def forward(self, x, label):
        with torch.cuda.amp.autocast(True):
              .....
              .....
        return x

2 using @torch.cuda.amp.autocast()
class model(nn.Module):
    def __init__():
        ....
    @torch.cuda.amp.autocast()
    def forward(self, x, label):
              .....
              .....
        return x

3 using TorchAMPModel
class model(nn.Module):
    def __init__():
        ....
    def forward(self, x, label):
              .....
              .....
        return x

model = model()
model = TorchAMPModel(model)

The first two are normal and only need 5769M GPU memory, but the third one needs 8703M GPU memory

TorchAMPModel of ColossaiAI is  the same as @torch.cuda.amp.autocast(cache_enable=True).
You can set cache_enable=False when aligning the memory usage.

TorchAMPModel of ColossaiAI is the same as @torch.cuda.amp.autocast(cache_enable=True). You can set cache_enable=False when aligning the memory usage.

The problem now is that using the same model, using pytorch+fp16 takes less memory than colossai+fp16. （5769M vs  8703M）

I try to reproduce your problem but failed. Could you provide you code for reference?
from torch import nn
import torch
from colossalai.amp.torch_amp import TorchAMPModel

class SimpleNet(nn.Module):
    """
    In this no-leaf module, it has subordinate nn.modules and a nn.Parameter.
    """

    def __init__(self) -> None:
        super().__init__()
        self.embed = nn.Embedding(20, 4)
        self.proj1 = nn.Linear(4, 8)
        self.ln1 = nn.LayerNorm(8)
        self.proj2 = nn.Linear(8, 4)
        self.ln2 = nn.LayerNorm(4)
        self.classifier = nn.Linear(4, 4)

    def forward(self, x):
        x = self.embed(x)
        x = self.proj1(x)
        x = self.ln1(x)
        x = self.proj2(x)
        x = self.ln2(x)
        x = self.classifier(x)
        return x

class SimpleNetAMP(nn.Module):
    """
    In this no-leaf module, it has subordinate nn.modules and a nn.Parameter.
    """

    def __init__(self) -> None:
        super().__init__()
        self.embed = nn.Embedding(20, 4)
        self.proj1 = nn.Linear(4, 8)
        self.ln1 = nn.LayerNorm(8)
        self.proj2 = nn.Linear(8, 4)
        self.ln2 = nn.LayerNorm(4)
        self.classifier = nn.Linear(4, 4)

    @torch.cuda.amp.autocast()
    def forward(self, x):
        x = self.embed(x)
        x = self.proj1(x)
        x = self.ln1(x)
        x = self.proj2(x)
        x = self.ln2(x)
        x = self.classifier(x)
        return x
use_colo = True # False
if use_colo:
    model = SimpleNet().cuda()
    model = TorchAMPModel(model)
    # 5632 B
else:
    model = SimpleNetAMP().cuda()
    # 5632 B
print(torch.cuda.max_memory_allocated())

I try to reproduce your problem but failed. Could you provide you code for reference?
from torch import nn
import torch
from colossalai.amp.torch_amp import TorchAMPModel

class SimpleNet(nn.Module):
    """
    In this no-leaf module, it has subordinate nn.modules and a nn.Parameter.
    """

    def __init__(self) -> None:
        super().__init__()
        self.embed = nn.Embedding(20, 4)
        self.proj1 = nn.Linear(4, 8)
        self.ln1 = nn.LayerNorm(8)
        self.proj2 = nn.Linear(8, 4)
        self.ln2 = nn.LayerNorm(4)
        self.classifier = nn.Linear(4, 4)

    def forward(self, x):
        x = self.embed(x)
        x = self.proj1(x)
        x = self.ln1(x)
        x = self.proj2(x)
        x = self.ln2(x)
        x = self.classifier(x)
        return x

class SimpleNetAMP(nn.Module):
    """
    In this no-leaf module, it has subordinate nn.modules and a nn.Parameter.
    """

    def __init__(self) -> None:
        super().__init__()
        self.embed = nn.Embedding(20, 4)
        self.proj1 = nn.Linear(4, 8)
        self.ln1 = nn.LayerNorm(8)
        self.proj2 = nn.Linear(8, 4)
        self.ln2 = nn.LayerNorm(4)
        self.classifier = nn.Linear(4, 4)

    @torch.cuda.amp.autocast()
    def forward(self, x):
        x = self.embed(x)
        x = self.proj1(x)
        x = self.ln1(x)
        x = self.proj2(x)
        x = self.ln2(x)
        x = self.classifier(x)
        return x
use_colo = True # False
if use_colo:
    model = SimpleNet().cuda()
    model = TorchAMPModel(model)
    # 5632 B
else:
    model = SimpleNetAMP().cuda()
    # 5632 B
print(torch.cuda.max_memory_allocated())

My test code is as following,
use_colossai_engine   to  control whether to use colossai engine.
colossalai run --nproc_per_node 1 train_debug.py --config_dir PATH_TO_YOUR_CONFIG
the config file is :
from colossalai.amp import AMP_TYPE
fp16 = dict(
    mode=AMP_TYPE.TORCH,
)
gradient_clipping = 5.0

train_debug.py
from colossalai.logging import get_dist_logger
import colossalai
import torch
import os
from colossalai.core import global_context as gpc
from colossalai.nn.lr_scheduler import CosineAnnealingLR
from colossalai.amp.torch_amp import TorchAMPModel
import psutil
from colossalai.nn.optimizer import HybridAdam

def get_cpu_mem():
    return psutil.Process().memory_info().rss / 1024**2

def get_gpu_mem():
    return torch.cuda.memory_allocated() / 1024**2

def get_mem_info(prefix=''):
    return f'{prefix}GPU memory usage: {get_gpu_mem():.2f} MB, CPU memory usage: {get_cpu_mem():.2f} MB'

import torch
from torch import nn
from torch.utils.checkpoint import checkpoint
from torch.nn.functional import linear, normalize

using_ckpt = False

class CosFace(torch.nn.Module):
    def __init__(self, s=64.0, m=0.40):
        super(CosFace, self).__init__()
        self.s = s
        self.m = m

    def forward(self, logits: torch.Tensor, labels: torch.Tensor):
        # labels = labels.squeeze_()
        index = torch.where(labels != -1)[0]
        target_logit = logits[index, labels[index].view(-1)]
        final_target_logit = target_logit - self.m
        logits[index, labels[index].view(-1)] = final_target_logit
        logits = logits * self.s
        return logits

def conv3x3(in_planes, out_planes, stride=1, groups=1, dilation=1):
    """3x3 convolution with padding"""
    return nn.Conv2d(in_planes,
                     out_planes,
                     kernel_size=3,
                     stride=stride,
                     padding=dilation,
                     groups=groups,
                     bias=False,
                     dilation=dilation)

def conv1x1(in_planes, out_planes, stride=1):
    """1x1 convolution"""
    return nn.Conv2d(in_planes,
                     out_planes,
                     kernel_size=1,
                     stride=stride,
                     bias=False)

class IBasicBlock(nn.Module):
    expansion = 1
    def __init__(self, inplanes, planes, stride=1, downsample=None,
                 groups=1, base_width=64, dilation=1):
        super(IBasicBlock, self).__init__()
        if groups != 1 or base_width != 64:
            raise ValueError('BasicBlock only supports groups=1 and base_width=64')
        if dilation > 1:
            raise NotImplementedError("Dilation > 1 not supported in BasicBlock")
        self.bn1 = nn.BatchNorm2d(inplanes, eps=1e-05,)
        self.conv1 = conv3x3(inplanes, planes)
        self.bn2 = nn.BatchNorm2d(planes, eps=1e-05,)
        self.prelu = nn.PReLU(planes)
        self.conv2 = conv3x3(planes, planes, stride)
        self.bn3 = nn.BatchNorm2d(planes, eps=1e-05,)
        self.downsample = downsample
        self.stride = stride

    def forard_impl(self, x):
        identity = x
        out = self.bn1(x)
        out = self.conv1(out)
        out = self.bn2(out)
        out = self.prelu(out)
        out = self.conv2(out)
        out = self.bn3(out)
        if self.downsample is not None:
            identity = self.downsample(x)
        out += identity
        return out

    def forward(self, x):
        if self.training and using_ckpt:
            return checkpoint(self.forard_imlp, x)
        else:
            return self.forard_impl(x)

class IResNet(nn.Module):
    fc_scale = 7 * 7
    def __init__(self,
                 block, layers, dropout=0, num_features=512, zero_init_residual=False,
                 groups=1, width_per_group=64, replace_stride_with_dilation=None, fp16=False, input_size=(112,112), head=None, num_class=1000):
        super(IResNet, self).__init__()
        self.extra_gflops = 0.0
        self.fp16 = fp16
        self.inplanes = 64
        self.dilation = 1
        self.fc_input_dim = 132  # for input=(192, 168)

        if replace_stride_with_dilation is None:
            replace_stride_with_dilation = [False, False, False]
        if len(replace_stride_with_dilation) != 3:
            raise ValueError("replace_stride_with_dilation should be None "
                             "or a 3-element tuple, got {}".format(replace_stride_with_dilation))
        self.groups = groups
        self.base_width = width_per_group
        self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(self.inplanes, eps=1e-05)
        self.prelu = nn.PReLU(self.inplanes)
        self.layer1 = self._make_layer(block, 64, layers[0], stride=2)
        self.layer2 = self._make_layer(block,
                                       128,
                                       layers[1],
                                       stride=2,
                                       dilate=replace_stride_with_dilation[0])
        self.layer3 = self._make_layer(block,
                                       256,
                                       layers[2],
                                       stride=2,
                                       dilate=replace_stride_with_dilation[1])
        self.layer4 = self._make_layer(block,
                                       512,
                                       layers[3],
                                       stride=2,
                                       dilate=replace_stride_with_dilation[2])
        self.bn2 = nn.BatchNorm2d(512 * block.expansion, eps=1e-05,)
        self.dropout = nn.Dropout(p=dropout, inplace=True)
        # self.fc = nn.Linear(512 * block.expansion * self.fc_scale, num_features)
        self.fc = nn.Linear(512 * block.expansion * self.fc_input_dim, num_features)
        self.features = nn.BatchNorm1d(num_features, eps=1e-05)
        nn.init.constant_(self.features.weight, 1.0)
        self.features.weight.requires_grad = False

        if head is not None:
            assert head == 'cosface'
            self.head = CosFace(s=64, m=0.35)
            self.num_class = num_class
            self.weight_activated = torch.nn.Parameter(torch.normal(0, 0.01, (self.num_class, num_features)))
        else:
            self.head = None

        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.normal_(m.weight, 0, 0.1)
            elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)

        if zero_init_residual:
            for m in self.modules():
                if isinstance(m, IBasicBlock):
                    nn.init.constant_(m.bn2.weight, 0)

    def _make_layer(self, block, planes, blocks, stride=1, dilate=False):
        downsample = None
        previous_dilation = self.dilation
        if dilate:
            self.dilation *= stride
            stride = 1
        if stride != 1 or self.inplanes != planes * block.expansion:
            downsample = nn.Sequential(
                conv1x1(self.inplanes, planes * block.expansion, stride),
                nn.BatchNorm2d(planes * block.expansion, eps=1e-05, ),
            )
        layers = []
        layers.append(
            block(self.inplanes, planes, stride, downsample, self.groups,
                  self.base_width, previous_dilation))
        self.inplanes = planes * block.expansion
        for _ in range(1, blocks):
            layers.append(
                block(self.inplanes,
                      planes,
                      groups=self.groups,
                      base_width=self.base_width,
                      dilation=self.dilation))

        return nn.Sequential(*layers)

    # @torch.cuda.amp.autocast()
    def forward(self, x, label):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.prelu(x)
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
        x = self.bn2(x)
        x = torch.flatten(x, 1)
        x = self.dropout(x)
        x = self.fc(x)
        out_fc = self.features(x)
        norm_embeddings = normalize(out_fc)
        norm_weight_activated = normalize(self.weight_activated)
        x = linear(norm_embeddings, norm_weight_activated)
        x = self.head(x, label)
        return x, out_fc

def _iresnet(arch, block, layers, pretrained, progress, **kwargs):
    model = IResNet(block, layers, **kwargs)
    if pretrained:
        raise ValueError()
    return model

def iresnet18(pretrained=False, progress=True, **kwargs):
    return _iresnet('iresnet18', IBasicBlock, [2, 2, 2, 2], pretrained,
                    progress, **kwargs)

BATCH_SIZE = 32
NUM_EPOCHS = 20
NUM_STEPS = 2000
LOGGING_FREQUNCE = 100
OUTPUT = './work_dir/test_zero_bs_64/'
num_classes = 20000

use_colossai_engine = False

def main():
    parser = colossalai.get_default_parser()
    parser.add_argument('--use_trainer', action='store_true', help='whether to use trainer')
    parser.add_argument('--config_dir', type=str, default='./config.py', help='config for train fr')
    args = parser.parse_args()

    if use_colossai_engine:
        colossalai.launch_from_torch(config=args.config_dir)
    else:
        colossalai.launch_from_torch(config={})

    logger = get_dist_logger()

    # get dist info
    rank = int(os.environ['RANK'])
    local_rank = int(os.environ['LOCAL_RANK'])
    world_size = int(os.environ['WORLD_SIZE'])

    torch.cuda.set_device(local_rank)
    os.makedirs(OUTPUT, exist_ok=True)

    # build resnet
    model = iresnet18(head='cosface', num_class=num_classes)

    if not use_colossai_engine:
        model = TorchAMPModel(model)

    model.train().cuda()

    logger.info(get_mem_info(prefix='After init model, '), ranks=[0])

    # build dataloader
    def get_data(batch_size, num_classes, shape=(3, 192, 168)):
        datas = torch.randn((batch_size, )  + shape, device=torch.cuda.current_device())
        labels = torch.randint(0, num_classes - 1, (batch_size,), device=torch.cuda.current_device())
        return datas, labels

    # build criterion
    criterion = torch.nn.CrossEntropyLoss()

    optimizer = HybridAdam(model.parameters(), lr=0.001)

    # lr_scheduler
    lr_scheduler = CosineAnnealingLR(optimizer, total_steps=NUM_EPOCHS)

    if use_colossai_engine:
        engine, _, _, _ = colossalai.initialize(
            model,
            optimizer,
            criterion,
            )
        engine.train()
    else:
        amp = torch.cuda.amp.grad_scaler.GradScaler(growth_interval=1000)
    global_step = 0

    optimizer.zero_grad()
    for _ in range(NUM_STEPS):

        img, label = get_data(BATCH_SIZE, num_classes)
        global_step += 1
        img = img.cuda()
        label = label.cuda()
        if use_colossai_engine:
            engine.zero_grad()
            output, _ = engine(img, label)
            # output = engine(img)
            # train_loss = module_partial_fc(output, label)
            train_loss = engine.criterion(output, label)
            engine.backward(train_loss)
            engine.step()
        else:
            output, _ = model(img, label)

            train_loss = criterion(output, label)
            amp.scale(train_loss).backward()
            amp.unscale_(optimizer)
            torch.nn.utils.clip_grad_norm_(model.parameters(), 5)
            amp.step(optimizer)
            amp.update()

            optimizer.zero_grad()

        if global_step % LOGGING_FREQUNCE == 0:
                logger.info(
                f"global_step {global_step} -train loss: {train_loss:.5}, lr: {lr_scheduler.get_last_lr()[0]:.5g}",
                ranks=[0])

if __name__ == '__main__':
    main()

use_colossai_engine = True. During the training process, the gpu memory is 8200M.
use_colossai_engine = False. During the training process, the gpu memory is 5281M.

See this line. So you are training the torch version without amp. Is that what you want?
# @torch.cuda.amp.autocast()

See this line. So you are training the torch version without amp. Is that what you want?
# @torch.cuda.amp.autocast()

I have verified that  using TorchAMPModel  or directly setting @torch.cuda.amp.autocast(cache_enable=True) is the same.
So the conclusion of https://github.com/hpcaitech/ColossalAI/issues/1083#issuecomment-1150575785 is wrong.
I want  training the torch version with amp, as you can seen in this line.
if not use_colossai_engine:
model = TorchAMPModel(model)

That makes sense. Thanks for helping us to check the functionality of AMP. I close this issue.

That makes sense. Thanks for helping us to check the functionality of AMP. I close this issue.

That was not what I meant. What I want to express is that defining fp16 in config and then initializing the model with colossalai.initialize will consume about 8G of memory. Instead of using colossalai.initialize to initialize the model, training directly with TorchAMPModel only consumes 5.7G of memory.
I would like to know what is the reason for this extra memory consumption. Is it a problem with using the Engine?
You can test the code above.

As we have checked that AMP is correct, so I close the issue.
You now argue that the engine is using more memory during training right?
I suppose that you open another issue #1082 to discuss the problem? If that's not what I suppose, we just reopen the issue.

The issue https://github.com/hpcaitech/ColossalAI/issues/1082  is related to ZeRO and has been resolved.
I now argue that the engine is using more memory during training right.
Can you help to check this phenomenon？

I can not run your code. What is your config.py? How did you inspect the cuda memory?

config.py
from colossalai.amp import AMP_TYPE
fp16 = dict(
    mode=AMP_TYPE.TORCH,
)

I inspected the cuda memory using  watch -n 1 nvidia-smi

the command is
colossalai run --nproc_per_node 1 train_debug.py --config_dir $PATH_TO_YOUR_CONFIG

Hi, did you apply colossalai.amp.convert_to_torch_amp? This API will wrap model, optimizer and criterion.

@FrankLeeeee is right. In the not use colossalai engine situation, you should also wrap the optimizer. That will align the memory usage.

Hi, did you apply colossalai.amp.convert_to_torch_amp? This API will wrap model, optimizer and criterion.

colossalai.initialize will automaticly use colossalai.amp.convert_to_torch_amp to wrap model, optimizer and criterion.
As you can seen in the colossalai.initialize
 # check amp and zero
  fp16_cfg = gpc.config.get('fp16', None)

  if fp16_cfg is not None and fp16_cfg.mode is not None and use_zero:
      raise ConfigException(
          "It is not allowed to set fp16 and zero configuration in your config file at the same time")

  # clip grad norm
  clip_grad_norm = gpc.config.get('clip_grad_norm', 0.0)

  # initialize amp
  amp_mode = None
  if fp16_cfg is not None and fp16_cfg.mode is not None:
      cfg_ = fp16_cfg.copy()
      amp_mode = cfg_.pop('mode')
      if is_using_pp():
          assert amp_mode == AMP_TYPE.NAIVE, 'Pipeline only support NaiveAMP currently'
      if amp_mode == AMP_TYPE.NAIVE:
          cfg_['clip_grad_norm'] = clip_grad_norm
      model, optimizer, criterion = convert_to_amp(model=model,
                                                   optimizer=optimizer,
                                                   criterion=criterion,
                                                   mode=amp_mode,
                                                   amp_config=cfg_)

Yes, I mean in your two experiments without using colossalai.initialize, did you apply torch amp to the optimizer?

Yes, I mean in your two experiments without using colossalai.initialize, did you apply torch amp to the optimizer?

of course.
if not use_colossai_engine:
    model = TorchAMPModel(model)

without using colossalai.initialize, the training process only consume 5700M gpu memory.  If using colossalai.initialize,  it takes 8000M gpu memory.

I inserted a print func into the colossalai.initialize and it actually printed what i wants.
print(22222222222222222222222222222222222222222222, 'convert_to_amp')
model, optimizer, criterion = convert_to_amp(model=model,
                                             optimizer=optimizer,
                                             criterion=criterion,
                                             mode=amp_mode,
                                             amp_config=cfg_)

console output:;
 WARNING  colossalai - colossalai - WARNING: /usr/local/pytho
                             n/lib/python3.6/site-packages/colossalai/initialize
                             .py:304 initialize
                    WARNING  colossalai - colossalai - WARNING: Initializing an
                             non ZeRO model with optimizer class
22222222222222222222222222222222222222222222 convert_to_amp
                    WARNING  colossalai - colossalai - WARNING: /usr/local/pytho
                             n/lib/python3.6/site-packages/colossalai/initialize
                             .py:440 initialize
                    WARNING  colossalai - colossalai - WARNING: No PyTorch DDP
                             or gradient handler is set up, please make sure you
                             do not need to all-reduce the gradients after a
                             training step.
[06/10/22 03:15:42] INFO     colossalai - colossalai - INFO: train_debug.py:328
                             main
                    INFO     colossalai - colossalai - INFO: global_step 100
                             -train loss: 36.324, lr: 0.001

I think there is some misunderstanding. What I mean is that you should not only wrap your model with torch amp but also your optimzier. In AMP, we need to handle gradient with mixed precision as well. Wrapping only your model but not your optimizer will lose the gradient handling part. That's why you  see a lower memory usage when colossalai.initailize is not used. If you include your optimizer, it will be the same. I have attached a sample code below for your reference.
if not use_colossai_engine:
    model, optimizer, criterion =  colossalai.amp.convert_to_torch_amp(model, optimizer, criterion)

I do not think so, i have wraped criterion.
if not use_colossai_engine:
        criterion = TorchAMPLoss(criterion)

And the code below is the a pytorch way to wrap optimizer.
if use_colossai_engine:
    engine, _, _, _ = colossalai.initialize(
        model,
        optimizer,
        criterion,
        )
    engine.train()
else:
    amp = torch.cuda.amp.grad_scaler.GradScaler(growth_interval=1000)

global_step = 0

optimizer.zero_grad()
for _ in range(NUM_STEPS):

    img, label = get_data(BATCH_SIZE, num_classes)
    global_step += 1
    img = img.cuda()
    label = label.cuda()
    if use_colossai_engine:
        engine.zero_grad()
        output, _ = engine(img, label)
        # output = engine(img)
        # train_loss = module_partial_fc(output, label)
        train_loss = engine.criterion(output, label)
        engine.backward(train_loss)
        engine.step()
    else:
        **output, _ = model(img, label)

        train_loss = criterion(output, label)
        amp.scale(train_loss).backward()
        amp.unscale_(optimizer)
        torch.nn.utils.clip_grad_norm_(model.parameters(), 5)
        amp.step(optimizer)
        amp.update()**

        optimizer.zero_grad()

I may not have expressed clearly, as you said above, use colossalai.amp.convert_to_torch_amp to wrap the model, optimizer and criterion,
if not use_colossai_engine:
    model, optimizer, criterion =  colossalai.amp.convert_to_torch_amp(model, optimizer, criterion)

and then train normally, which also only consumes 4700M of memory.
output, _ = model(img, label)
train_loss = criterion(output, label)
optimizer.backward(train_loss)
optimizer.step()
optimizer.zero_grad()

But if you use colossalai.initialize to initialize, it will consume 7700M of memory. But we did see that by reading the fp16 parameter in config, in the initialization code of colossalai.initialize, the conversion of process  colossalai.amp.convert_to_torch_amp is performed, and then we use the Engine for training, but it needs to consume 7700M of memory at this time. This is where I get confused.
engine.zero_grad()
output, _ = engine(img, label)
train_loss = engine.criterion(output, label)
engine.backward(train_loss)
engine.step()

I see, I am writing a script to reproduce this problem. Can you open a new issue so that we can move our discussion there?

OK, i will open a new issue https://github.com/hpcaitech/ColossalAI/issues/1095


Fix menu

Issue
Some issues were found on the menu.

footer items height changing when menu is collapsed
(See https://watch.screencastify.com/v/pviI6RYinTPM5DmSeYJG)
collapse button outline border is different when viewing on laptop screen vs on standalone display

Description
This PR is composed of 2 commits, one for each issue, and I added a story to easily view the height issue on collapse

📚  View the storybook of this branch here

:tada: This PR is included in version 4.10.2 :tada:
The release is available on:

npm package (@latest dist-tag)
npm package (@latest dist-tag)
npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Karma tests fail with "System limit for number of file watchers reached"

Bug Description
Karma tests seem to fail with "System limit for number of file watchers reached" error which prevents the tests from running, for example:

See a failing job.

Expected Behaviour
Karma tests should fail only when an actual test fails.

Steps to Reproduce

Screenshots

Additional Context

Plugin Version:
Operating System:
Browser:

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

Implementation Brief

This doesn't actually cause failures, as there seem to be successful runs despite the error message.
See https://github.com/google/web-stories-wp/runs/1084952934 for example

Indeed. So it seems to be the disconnecting that causes failure.

Haven't encountered this in quite some time; closing for now


Switches don't work

Describe the issue
I upgraded the project to null-safety and upgraded the libraries to the newest versions. After that GraphQL stopped working correctly. It doesn't operate queries with switch signs (...) now.
Here is the API response I receive:
responseString: {                 "data": {                   "user": {                     "id": 409,                     "info": {                       "email"<EMAIL_ADDRESS>                      "firstName": "Yuri",                       "lastName": "Goncharenko",                       "avatar": "https://lh3.googleusercontent.com/a-/AOh14Gjv2BpAeTxGyhC9cupGeDFAjJb14taGAV1dznxZ=s96-c",                       "phone": "+79118248658"                     },                     "roles": [                       {                         "id": 186,                         "role": "client",                         "myFitnessPalId": null                       }                     ],                     "chatJwt": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImN0eSI6InR3aWxpby1mcGE7dj0xIn0.eyJqdGkiOiJTSzcyNGE0ZDM3YjEzN2FmNzI4NzkzOTVkYmFhMTlmMTEzLTE2MjU4MTg2ODgiLCJncmFudHMiOnsiaWRlbnRpdHkiOiJ1c2VyLTQwOSIsImNoYXQiOnsic2VydmljZV9zaWQiOiJJU2I2NGY2YzE4ZGQ4MjQ0ODE4NjlmZmIwNjlmZWVjZWYwIn19LCJpYXQiOjE2MjU4MTg2ODgsImV4cCI6MTYyNTgyMjI4OCwiaXNzIjoiU0s3MjRhNGQzN2IxMzdhZjcyODc5Mzk1ZGJhYTE5ZjExMyIsInN1YiI6IkFDOWNmNzA5YTgzMDliNDNhODE3MjAzYjI1ZjViNWU4NjEifQ.fwCixgymBBN99xbrbpkh5wyUx8wlmlCB9wJoasufP4w"                   }                 }               },                     statusCode: 200,
Here is the query model:
query {   user {     id     info {       ...UserInfoFragment     }     roles {       ... on Client {         ...RoleUserFragment         myFitnessPalId       }       ... on Coach {         ...RoleUserFragment         maxActiveClients         inviteCode       }     }     chatJwt   } }
And here is how it's being parsed:
{user: {id: 409, info: null, roles: [null], chatJwt: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImN0eSI6InR3aWxpby1mcGE7dj0xIn0.eyJqdGkiOiJTSzcyNGE0ZDM3YjEzN2FmNzI4NzkzOTVkYmFhMTlmMTEzLTE2MjU4MTg2ODgiLCJncmFudHMiOnsiaWRlbnRpdHkiOiJ1c2VyLTQwOSIsImNoYXQiOnsic2VydmljZV9zaWQiOiJJU2I2NGY2YzE4ZGQ4MjQ0ODE4NjlmZmIwNjlmZWVjZWYwIn19LCJpYXQiOjE2MjU4MTg2ODgsImV4cCI6MTYyNTgyMjI4OCwiaXNzIjoiU0s3MjRhNGQzN2IxMzdhZjcyODc5Mzk1ZGJhYTE5ZjExMyIsInN1YiI6IkFDOWNmNzA5YTgzMDliNDNhODE3MjAzYjI1ZjViNWU4NjEifQ.fwCixgymBBN99xbrbpkh5wyUx8wlmlCB9wJoasufP4w}}
I think the problem is somewhere in this library, correct me if I'm wrong.
To Reproduce
Steps to reproduce the behavior:

Make a query with switch logic
Try to get the response JSON
See that it's being parsed wrong

Expected behavior
Expected behaviour is to get the response parsed correctly
device / execution context
Any, mostly tested on Android
Other useful/optional fields
Flutter doctor
[✓] Flutter (Channel stable, 2.2.3, on macOS 11.3.1 20E241 darwin-arm, locale ru-RU) [✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3) [✓] Xcode - develop for iOS and macOS [✓] Chrome - develop for the web [!] Android Studio     ✗ Unable to find bundled Java version. [✓] Android Studio (version 4.2) [✓] VS Code (version 1.58.0) [✓] Connected device (2 available)     ! Error: iPhone is not connected. Xcode will continue when iPhone is connected. (code -13)
pubspec.yaml
environment:
sdk: '>=2.12.0 <3.0.0'
dependencies:
flutter:
sdk: flutter
graphql_flutter: ^5.0.0
dio: ^4.0.0
connectivity_plus: ^1.0.4
json_annotation: ^4.0.1
shared_preferences: ^2.0.6
package_info: ^2.0.2
hive: ^2.0.4
hive_flutter: ^1.1.0
collection: ^1.14.13
permission_handler: ^8.1.2
redux: ^5.0.0
flutter_redux: ^0.8.2
redux_epics: ^0.15.0
redux_logging: ^0.5.0
redux_persist: ^0.9.0
redux_persist_flutter: ^0.9.0
firebase_core: ^1.3.0
firebase_auth: ^2.0.0
firebase_auth_web: ^2.0.0
google_sign_in: ^5.0.4
sign_in_with_apple: ^3.0.0
flutter_facebook_auth: ^3.5.0
firebase_analytics: ^8.1.2
firebase_crashlytics: ^2.0.7
firebase_performance: ^0.7.0
logger: ^0.9.4
crypto: ^3.0.1
device_info: ^2.0.2
uuid: ^3.0.4
image_picker: ^0.8.1+3
rxdart: ^0.26.0
equatable: ^2.0.3
flutter_localizations:
sdk: flutter
timezone: ^0.7.0
flutter_native_timezone: 1.0.10
get_it: ^7.1.4
flutter_svg: ^0.22.0
cupertino_icons: ^1.0.0
pull_to_refresh: ^2.0.0
url_launcher: ^6.0.9
flutter_styled_toast: ^2.0.0
keyboard_actions: ^3.3.1+1
flutter_link_preview: ^1.5.6
cached_network_image: ^3.0.0
flutter_local_notifications: ^6.0.0
background_fetch: ^1.0.1
app_settings: ^4.1.0
dependency_overrides:
intl: ^0.17.0-nullsafety.2
build: ^2.0.0
path_provider: ^2.0.2
http: ^0.13.3
http_parser: ^4.0.0
gql: ^0.13.0
glob: ^2.0.0
dart_style: ^2.0.0
yaml: ^3.0.0
code_builder: ^4.0.0
args: ^2.0.0
plugin_platform_interface: ^2.0.0
xml: ^5.0.0
built_collection: ^5.0.0
firebase_auth_platform_interface: ^5.0.0
google_sign_in_platform_interface: ^2.0.1
device_info_platform_interface: ^2.0.1
url_launcher_platform_interface: ^2.0.4
permission_handler_platform_interface: ^3.6.0
html: ^0.15.0
path_drawing: ^0.5.1
image_picker_platform_interface: ^2.1.0
enum_to_string: ^2.0.1
csslib: ^0.17.0
path_parsing: ^0.2.1
flutter_local_notifications_platform_interface: ^4.0.0
async: ^2.7.0
dev_dependencies:
flutter_test:
sdk: flutter
json_serializable: ^4.1.3
pedantic: ^1.0.0
build_runner: ^2.0.2
rename: ^1.2.0
flutter_launcher_icons: ^0.9.0
gql_code_gen: ^0.2.0-alpha.1
hive_generator: ^1.1.0
flutter:
generate: true

This seems to be related to fragments. Please try and update to the latest beta version and ensure you have the right setup (see README)


can't generate a page view for Facility view and SWIS

System: macOS Big Sur 11.6.7
Broswer: Safari Version 15.5 (166<IP_ADDRESS>.9, 16613)
page url: https://opennem.org.au/facilities/wem/?status=operating
Works in Chrome and FireFox on macOS no problems.

@aleith can you check this again now please? it's working here

works for me too.

working now in a new window. but older tab still was half-done.
I suspect it's a caching issue or something on the frontend. when I found an old tab that was what it took that image from it got a bit more of the page but until I reloaded the page was stuck at this point:

one weird thing was it was working on NEM facilities only not working on SWIS facilities. maybe problems retrieving the data?
or I could just have localised gremlins, that has been very much known to happen.


[HUDI-5286] UnsupportedOperationException throws when enabling filesy…

…stem retry
Change Logs
FileSystem#getScheme is a method that must be overridden.
Impact
Fix the UnsupportedOperationException on object storage.
Risk level (write none, low medium or high below)
none
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

The failed test case: testRunHoodieJavaAppOnMultiPartitionKeysMORTable should not be affected by this patch, would merge it soon ~


problem with using datatables

i have written a piece of test code, but can't get it to run.
this is what i have so far
blade.php file

@section('title', 'Home')

@section('content_header')

Dashboard

@stop

@section('content')

Test run

                No
                Name
                Email
                Action

@stop

@section('css')
    <link rel="stylesheet" href="/css/admin_custom.css">
@stop

@section('js')

@stop

Controler file

namespace App\Http\Controllers;

use Illuminate\Http\Request;
use App\User;
use DataTables;

class UserController extends Controller
{
     /**
     * Display a listing of the resource.
     *
     * @return \Illuminate\Http\Response
     */
    public function index(Request $request)
    {
        if ($request->ajax()) {
            $data = User::latest()->get();
            return Datatables::of($data)
                    ->addIndexColumn()
                    ->addColumn('action', function($row){

                           $btn = 'View';
                           return $btn;
                    })
                    ->rawColumns(['action'])
                    ->make(true);
        }   return view('users');
    }
}

and my adminlte.php config file
        'Datatables' => [
            'active' => true,
            'files' => [
                [
                    'type' => 'js',
                    'asset' => true,
                    'location' => '//cdn.datatables.net/1.10.19/js/jquery.dataTables.min.js',
                ],
                [
                    'type' => 'js',
                    'asset' => true,
                    'location' => '//cdn.datatables.net/1.10.19/js/dataTables.bootstrap4.min.js',
                ],
                [
                    'type' => 'css',
                    'asset' => true,
                    'location' => '//cdn.datatables.net/1.10.19/css/dataTables.bootstrap4.min.css',
                ],
            ],
        ],

and this is a part of the error i get.

"message": "Class 'DataTables' not found",
"exception": "Error",
"file": "C:\\xampp\\htdocs\\test\\app\\Http\\Controllers\\UserController.php",
"line": 20,

I can't figure out what is wrong, i'm usig Laravel 7.

"message": "Class 'DataTables' not found",
"exception": "Error",
"file": "C:\xampp\htdocs\test\app\Http\Controllers\UserController.php",
"line": 20,

The previous message means that the DataTables class can't be found when you reference it inside the  UserController.php file. Using google I suspect you are trying to use the next package: yajra/laravel-datatables. If so, please refers to the Quick Installation Section. Maybe you have forgotten to include the alias on the config file:
'aliases' => [
    ...,
    'DataTables' => Yajra\DataTables\Facades\DataTables::class,
]

Or you can try replacing use DataTables; by use Yajra\DataTables\Facades\DataTables; on the controller file. There is also an example you can follow:
https://www.positronx.io/laravel-datatables-example/
However, I can't be really sure since I have never used this package before.

no i don't knwo which one is used within this package, isn't it pre installed when use this template ?

Installed through Yajra and now it's working.
In my opnion it should be written out that is makes use of the Yarja package.
Or how to use de datatables with this package.

Hi @ageurtse , for what I know Yajra is not part (or required) by this package. Maybe, in your local project someone has executed composer require yajra/laravel-datatables.... This packages includes, but only optionally the Datatables plugin for the client-side (to use with javascript/jquery). For what I have read, the Yajra package is an extension for use Datatables on the server-side (with php).

No it was not installed, i just did it manualy.
As a beginner i could not find it on this github that i have to manual install the datatables from yajra.
I found that package already, but didn't think that i had to install it.
so for now it is solved, thanks a lot.

@ageurtse Please, give a read to Plugins Section to understand how to configure the package config file in order to enable additional client-side plugins (like Datatables). Also, in case you won't use the CDN for plugins, you can use the php artisan adminlte:plugins command to see a list of what client-side plugins are available to install locally, and refer to Plugin Command Section to check how to install these plugins. Anyway, after installing a plugin locally you will still need to configure it on the adminlte.php config file.
And again, the yajra/laravel-datatables package is external and independent of this package, it provides a way to manage Datatables on the server side. The usage of that package will depend if you need that feature or not.