Add sbt-dependency-check

This PR adds sbt-dependency-check which scans the entire dependency tree of Pekko against OWASP to scan for any potential CVE's. This PR does a pretty basic addition of the plugin but there is future room for improvement, i.e. since this is a standard sbt plugin its possible to run dependencyCheckAggregate along with sourceDistGenerate when making a release. Another notable setting is dependencyCheckFailBuildOnCVSS which would cause the project to automatically fail if it finds a dependency with a certain security level.
The contents of both the generated report and the surpression file are located in the dependency folder (this can be renamed if needed). This also means that the report will be contained in the source dist distribution.
Note that the report has actually picked up CRITICAL CVE's from our dependency list which we should look into, i.e.

So I just checked the netty dependency which has the CRITICAL level but there isn't anything that can be done about it since there isn't a new proper release that solves it (see https://mvnrepository.com/artifact/io.netty/netty).
@pjfanning I can add this netty version into suppression.xml since there isn't anything we can do about it, just let me know if you want me to update the PR for this.

do we really need to commit the html file?

do we really need to commit the html file?

I don't know? The primary point of the plugin is to create this file. One thing to note is that the generation of this file takes a looong amount of time. This is due to the fact that it downloads the entire OWASP database locally, does some processing on it and then scans all of your dependencies. You can think of it as similar to compiled game assets, i.e. theoretically speaking you shouldn't commit these into git but because they are expensive to generate and rarely change its done for pragmatic/practical reasons.
I would also presume that having this committed in git would make Apache review/release process of Pekko a lot easier (you just need to open the file) and since the report (at least currently) is in the source dist package its a nice QOL feature for Pekko users. As stated before we would make sure that the report is in sync when a release is made but this also can be automated.
Also incase you haven't opened the file, I would recommend do so because its actually interactive (i.e. its not a static text). i.e. it gives you the ability to direct to the original OWASP vulnerability and you can also use the generated html file to add suppressions, i.e.

Ultimately I have no problem with removing it from git but then we kind of need to ascertain what the goals are here. At least from my experience in working with companies for legal/license/CVE's, a report is always the best way to handle this and thats why these plugins generally make reports. This plugin is also just a wrapper for the well known https://github.com/jeremylong/DependencyCheck whos entire point is also just to generate reports.

Or a weekly action can do too？

Or a weekly action can do too？

We can do this, but the point is still where do we persist the file? I guess we can upload it as part of pekko docs and then rsync it/commit it to git. @pjfanning would you prefer this?

We could rsync it to nightlies site.

I agree, having the report pre-generated is really useful for consumers (esp. corporate). But we shouldn't have big generated artifacts in git (it slows down clones, for one thing). It should go on the nightlies site, or near the docs.

So I plan on changing this so it just gets included in the docs. I have similar work with sbt-license-report which I have yet to submit.

So I plan on changing this so it just gets included in the docs. I have similar work with sbt-license-report which I have yet to submit.

If we remove the html file and possibly add the filename to .gitignore, I think we can merge this. We could do a 2nd PR that automates the publishing of the HTML.

In that case let me spend a few minutes today trying to automate this considering I already did it for sbt-license-report, I will ping when I update the PR.

So I did some work on this but due to the amount of time it takes to get the NVD database I have some concerns about how nicely it would work with github actions especially when integrated with paradox so I opened up an issue upstream at https://github.com/albuch/sbt-dependency-check/issues/305

Good news is that after finally managing to get the NVD database to download onto my system I managed to integrate sbt-dependency-check directly into our paradox docs

The bad news is that as you can tell, since I am inlining the html the styling doesn't look the best. Unfortunately sbt-dependency-check doesn't have markdown as a supported output format which leaves us with 2 options

Use a much more strict/data orientated format such as JSON/XML, parse it and then convert it into markdown so that it renders better
Try to find a css stylesheet and wrap the raw block in a div as documented here https://developer.lightbend.com/docs/paradox/current/directives/css-friendliness.html#raw-text-in-fenced-blocks so that it doesn't look so bad.

Closing this PR since https://github.com/apache/incubator-pekko/pull/366 is already solving this problem in a more simple way.


Test annotate ref app

Fixes #66

Codecov Report

Merging #67 (634a57c) into main (343d311) will increase coverage by 6.46%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main      #67      +/-   ##
==========================================
+ Coverage   63.30%   69.77%   +6.46%
==========================================
  Files          24       24
  Lines         526      526
==========================================
+ Hits          333      367      +34
+ Misses        193      159      -34

Impacted Files
Coverage Δ

...ta_vegattributes/app/annotate_reference_samples.py
88.88% <100.00%> (+88.88%)
:arrow_up:

fooddata_vegattributes/reference_samples_csv.py
92.98% <0.00%> (+14.03%)
:arrow_up:

fooddata_vegattributes/indexed_fooddata.py
95.23% <0.00%> (+19.04%)
:arrow_up:

fooddata_vegattributes/auto_indexed_fooddata.py
96.00% <0.00%> (+28.00%)
:arrow_up:

fooddata_vegattributes/fooddata.py
100.00% <0.00%> (+30.00%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 343d311...634a57c. Read the comment docs.


"Invalid request: should be object" error when deploying with Vercel

After adding the Environment Variables and clicking on the deploy button the "Invalid request: should be object" error pops up. I double-checked the entered values which are correct. I also contacted Vercel and there is no problem at their end.

A day after this the deploy worked but there were some errors in the building phase. The fix was to go to Dashboard>Project Settings > General and  and switch Node version to 16.x  and rebuild.


ArangoDB 3.1 fails with collection.all

When running the simple querry all with the method collection.all using a batch size smaller than the actual collection (i.e. the cursor api has to be used to fetch more results), results in a requests.exceptions.ConnectionError and the ArangoDB 3.1 instance shuts down. With ArangoDB 3.0 there is no issue. Using the HTTP API of 3.1 with curl is also fine.
Here is my environment:
# installed via homebrew on OS X 10.10.5
» python --version
Python 3.5.2

» pip freeze
appnope==0.1.0
decorator==4.0.10
ipython==5.1.0
ipython-genutils==0.1.0
pexpect==4.2.1
pickleshare==0.7.4
prompt-toolkit==1.0.9
ptyprocess==0.5.1
Pygments==2.1.3
python-arango==3.4.0
requests==2.12.3
simplegeneric==0.8.1
six==1.10.0
traitlets==4.3.1
wcwidth==0.1.7

Here is a reproducer:
A modified script crash.py from your README.md:
from arango import ArangoClient

# Initialize the client for ArangoDB
client = ArangoClient(enable_logging=False)

# Create a new database named "my_database"
db = client.create_database('my_database')

# Create a new collection named "students"
students = db.create_collection('students')

# Add a hash index to the collection
students.add_hash_index(fields=['name'], unique=True)

# Insert new documents into the collection
students.insert({'name': 'jane', 'age': 19})
students.insert({'name': 'josh', 'age': 18})
students.insert({'name': 'jake', 'age': 21})

result = students.all(batch_size=1)
print([student['name'] for student in result])

# Starting ArangoDB
» docker run -p 8529:8529 --name arangodb -it -d -e ARANGO_NO_AUTH=1 arangodb:3.1

# After finished launching, run the script:
» python crash.py

Results in
Traceback (most recent call last):
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 391, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 387, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 643, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 334, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 391, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 387, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "crash.py", line 21, in <module>
    print([student['name'] for student in result])
  File "crash.py", line 21, in <listcomp>
    print([student['name'] for student in result])
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/arango/cursor.py", line 32, in __next__
    return self.next()
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/arango/cursor.py", line 120, in next
    res = self._conn.put("/_api/cursor/{}".format(self.id))
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/arango/connection.py", line 211, in put
    auth=(self._username, self._password)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/arango/http_clients/default.py", line 105, in put
    verify=self._check_cert
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/sessions.py", line 546, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/adapters.py", line 473, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

And on the arango side (it is stopped):
» docker logs -f arangodb
Initializing database...Hang on...
Database initialized...Starting System...
2016-12-06T14:59:01Z [1] INFO ArangoDB 3.1.3 [linux] 64bit, using VPack 0.1.30, ICU 54.1, V8 <IP_ADDRESS>, OpenSSL 1.0.1t  3 May 2016
2016-12-06T14:59:01Z [1] INFO using SSL options: SSL_OP_CIPHER_SERVER_PREFERENCE, SSL_OP_TLS_ROLLBACK_BUG
2016-12-06T14:59:01Z [1] INFO file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2016-12-06T14:59:01Z [1] INFO JavaScript using startup '/usr/share/arangodb3/js', application '/var/lib/arangodb3-apps'
2016-12-06T14:59:02Z [1] INFO In database '_system': Found 14 defined task(s), 1 task(s) to run
2016-12-06T14:59:02Z [1] INFO In database '_system': state standalone/existing, tasks updateUserModels
2016-12-06T14:59:02Z [1] INFO In database '_system': existing cleanup successfully finished
2016-12-06T14:59:02Z [1] INFO using endpoint 'http+tcp://<IP_ADDRESS>:8529' for non-encrypted requests
2016-12-06T14:59:03Z [1] INFO ArangoDB (version 3.1.3 [linux]) is ready for business. Have fun!
2016-12-06T14:59:16Z [1] INFO created application directory '/var/lib/arangodb3-apps/_db/my_database' for database 'my_database'
2016-12-06T14:59:16Z [1] INFO In database 'my_database': No version information file found in database directory.
2016-12-06T14:59:16Z [1] INFO In database 'my_database': Found 14 defined task(s), 9 task(s) to run
2016-12-06T14:59:16Z [1] INFO In database 'my_database': state standalone/init, tasks setupGraphs, addDefaultUserOther, createModules, createRouting, insertRedirectionsAll, setupAqlFunctions, createFrontend, setupQueues, setupJobs
2016-12-06T14:59:17Z [1] INFO In database 'my_database': init successfully finished

This happens also with password for root on arangodb.

I just saw, that you use the export API. It seems, this is an arangodb 3.1 error, as I can reproduce the error with curl. I will propagate this to Arango.

This seems to be an issue on the python-arango side as well. You are using the wrong cursor, see this explanation.

Hi @tammoippen
I appreciate you looking into this. I will make the appropriate fix and push out a new release sometime this week or next.
Thanks!

Hi again @tammoippen
I've pushed out a fix in the latest release 3.4.1.
Let me know how it fares. Thanks.

This works for me.
Thank you very much!

Awesome. Closing this out.


Documentation is down

https://ipfs.github.io/helia redirects to https://helia.io
API on helia.io redirects back to the same page

This should be fixed now. Can you try again to confirm?

@oed I ran into this issue with Brave browser.. The API link was redirecting me from ipfs.github.io/helia to helia.io, and was only stopped when I toggled off the ‘auto-redirect tracking urls’ in brave

it may be similar in others? try an incognito tab or different browser to see that it is all set up properly.

I'm still seeing the issue in Firefox, but yes if I open an incognito tab ipfs.github.io/helia works 🤔

Try disabling then enabling firefox tracking protections in firefox (via extensions or other..) and hopefully that will resolve the issue?

I'd rather not disable tracking protections

https://github.com/ipfs/helia/pull/666 should fix this. Turns out removing the custom domain manually only worked until another commit to main came in.


Unable to login at votest.democracy.earth using Brave browser

I'm not able to login using metamask on the test network using the Brave browser.
I get the following error in the developer console.

Hey @ConorOkus thanks for reporting. Could you replicate and confirm:

Metamask version you are using
Brave version
do you have Metamask's privacy mode enabled?
does the error come up when you are connected to any network? Or is it in Rinkeby or a specific one

I see, can you try upgrading Metamask to the latest version?
On Sat, Nov 17, 2018 at 6:23 AM Conor Okus<EMAIL_ADDRESS>wrote:

Hi @medied https://github.com/medied thanks for the quick reply, I can
confirm the following,
Metmask Version - 4.11.1
Braver Version - 0.25.2
I don't seem to have that option in the Metamask settings so not really
sure?
The error seems to occur on all the networks Rinkeby, Ropsten etc
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/DemocracyEarth/sovereign/issues/416#issuecomment-439609458,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEiE1-9Tr0rtOhYYr864Snch2M2x_woMks5uv_G5gaJpZM4Ym7ka
.

There didn't seem to be an obvious way to upgrade Metamask in the Brave browser.
I had to uninstall then reinstall Brave and download Metamask from the Chrome web store (feels weird).
Then sign into Metamask using my seed phrase, but use the older version not the beta. The newer beta version didn't recognise my seed phrase.
So currently:
Metamask Version - 5.0.2
Brave Version - 0.56.12
I'm now able to login to votest.democracy.earth

Hey @ConorOkus
I´m having the same issue using Brave browser and would like to ask you: How did you saved your tabs before reinstalling? (If you had any)

Hi @juanguren
I didn't actually have any bookmarks as I've only been using Brave for a short time. Most of mine are still on Chrome.
Chrome allows you to export your bookmarks via the bookmarks manager I'm not sure if the previous version of Brave has that option but that would be your best bet.
That would give you a .html file that you can save somewhere and then import it when you install the new version of Brave.
Hope that helps pal.

@ConorOkus glad you were able to login after Metamask update, closing this for now but please feel free to open a new issue if something else comes up 👍

Hey @ConorOkus, there´s an option to recover tabs after restarting computer but with "reinstalling brave" has been hard to find. Will ask on Brave community. Thanks!


[6.5.x] kie-remote-jaxb-gen: upgrade jaxb2-maven-plugin from 1.6 to 2.2

the upgrade should fix random failures recently seen when
building with JDK8

similar upgrade was done on master few months back and everything seems to work there

@mbiarnes, @mrietveld could you please take a look?

The test failures are not related to this PR, but we need to fix them anayway.
CC @mariofusco kie-aries-blueprint + kie-ci strikes again :) I'll see if I can implement the same workaround as before.

+1 seems good to me


error :  "Transaction Interrupted!" in Android

Getting this error in Android:
{     "error": "Transaction Interrupted!",     "status": "FAILURE" }
It is working fine in iOS.
I tried searching for the solution everywhere but there is no proper channel where we can raise the issue of React-native PhonePe SDK.
My Code:
fetch(${BASE_URL}api/phonepe/checksum, {       method: 'POST',       headers: {         Accept: 'application.json',         'Content-Type': 'application/json',       },       body: JSON.stringify(data),     })       .then(response => response.json())       .then(res => {           if (res.body) {             const {base64RequestBody} = res.body;             const dropDownValue = '/pg/v1/pay';             // packageName = 'com.phonepe.app';             PhonePePaymentSDK.startPGTransaction(               base64RequestBody,               res.checksum,               dropDownValue,               headers,               null,               Platform.OS == 'ios' ? '' : null,             )               .then(a => {                 if (a.status === 'SUCCESS') {                   checkPaymentStatus(res.body);                 } else {                   Alert.alert(a.error);                 }               })               .catch(e => {                 console.log('141 PHONEPE ERROR', e);               });           } else {             Alert.alert('Something went wrong BD606');           }       })       .catch(error => {         //handle error         console.warn(error);       });
This is urgent for me, quick help will be appreciated.
Thank you

@AnshumanPhonePe Please, If you can look into this. if you have time.

Even i am getting the same issue , { error: "Transaction Interrupted!" , status: "FAILURE"} when made a request using production credentials

@kushalAppiness I checked with the PhonePe team over mail. They sent this response:
I am sending the package signature to them over the mail. let's see if that helps.
You can also check with them over this email: PG Support<EMAIL_ADDRESS>
Hi,
Good day! Please confirm whether you have received app id. If not then we would like to request you to share Android signature once you added the release signing key to get app id from us.
Please refer to the below link and share Android signature asap.
React Native SDK: https://developer.phonepe.com/category/v1/hybrid-sdk-android-and-ios/react-native-sdk-integration/#:~:text=Example%3A
  if(Platform.OS %3D%3D%3D 'android'){
     PhonePePaymentSDK.getPackageSignatureForAndroid().then(packageSignture %3D> {
        ToastAndroid.show(packageSignture%2CToastAndroid.LONG)
     })%0A%20%20%7D

Please note the below points:
For Production, share the package signature of the release build that will be published to the Playstore to generate the App ID.
For Production, do not share the package signature from the Debug or Test Build which won't work.
Feel free to reach out to us in case of any queries.
Regards,

follow this doc: https://developer.phonepe.com/v1/docs/react-native-sdk-integration-standard
if you still face any issue, reach out the integration team (you can find the contact email in readme)


UI - object parts It is possible to move to next page when no extra parts available

Environment info

NooBaa Version: master
Platform: dev-local

Actual behavior

opening the object parts modal for an object with exactly 25 parts
It is possible to move to the next page and see the attached screen: 2 of 1 page and empty list of parts

Expected behavior

the next page should be disabled

Steps to reproduce

read the object parts of an object with exactly 25 parts

More information - Screenshots / Logs / Other output

Also happens on folder pages which have 0 object parts but the page shows 1 of 0:


better format string injection

let num = 10
echo fmt("{num = }")
echo fmt "{num = }"
echo "{num = }".fmt
echo "{num = }".fmt()

this example does not get format string highlighting

This is a known issue. There are various places in the queries where I opted for brevity of queries over complete correctness for performance sake.
There is still a standing issue with performance of injection queries in nvim-treesitter:
https://github.com/neovim/neovim/issues/22426

Currently we run the full injection query on every edit to a buffer.

So for the time being I would only include the most common one (AFAIK), which is the "generic string literal" one.


Update the stream-cslc tutorial notebook

Notebook is updated to add more context to the codes based on PR #20 by @kdevlin525 .
I also modified the way the s3 paths (i.e. get_s3path function in cslc_utils.py) are being read from the bucket. The wildcard-way does not always work and freezes the cell while running.
Cosmetic adjustments on the figure are also made (i.e. colorbar label, -pi to pi).
I also renamed the ipynb to Create_Interferogram_by_Streaming_CSLC-S1.ipynb

Notebook updates look good and function properly. Nice work!


Minimal documentation on how to backup and restore with uwazi-docker

Document at least some very basic strategy on how to backup and restore with uwazi-docker.
See #21 and https://github.com/fititnt/uwazi-docker/issues/20#issuecomment-430202551

Will close this issue for now. We have, a very basic instructions on how to backup. Improvements could be done later


Virtual machine >> update to new format

https://github.com/cncf/glossary/blob/main/definitions/virtual_machine.md
Please convert entry to the property format. View the style guide for more details.

Hello, I would like to work on this one but just to be sure Virtual machine is a Concept rather than Software so if that is the case I'd love to help.

Thanks for your help @TakshakRamteke!
We are looking for two things:

Update the format according to the style guide.
Make sure the definitions are correct and complete.

If you're not 100% sure about the definition, consider asking someone else (a more experienced colleague/classmate/professor). You can also submit an incomplete definition but please say so when submitting the PR (e.g. I know virtual machines address X and Y but not too sure about Z). We can then fill the gaps. But please do verify that what you submit is technically accurate.
I hope this helps.


[AutoPR applicationinsights/resource-manager] Adding Workbooks resource type

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/2775

This PR has been merged into https://github.com/Azure/azure-sdk-for-go/pull/1496


Embedding Path

How to pass my own embedding path to the model parameters or possibly the processor? Thank you very much !

You can do it by calling mz.embedding.load_from_file.


Catalyst support

As per https://github.com/johnno1962/InjectionIII/issues/390#issuecomment-1108951191

👍


Install guide not correct

In README it said:
VERSION=TODO # choose the latest version
OS=Linux     # or Darwin
ARCH=x86_64  # or arm64, i386, s390x
curl -L https://github.com/google/ko/releases/download/${VERSION}/ko_${VERSION}_${OS}_${ARCH}.tar.gz | tar xzf - ko
chmod +x ./ko

But it not work, for example, the VERSION=v0.8.1, but the tar name is ko_0.8.1_Linux_x86_64.tar.gz rather than ko_v0.8.1_Linux_x86_64.tar.gz

This was going to be fixed by #325
I'll fix it now since that was blocked by CLA hurdles.
Thanks for pinging this.

Fixed https://github.com/google/ko/commit/0621cbac8e0384de9307b3698ef091aedf751b2b


CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

Conditions

CuPy Version          : 8.0.0
CUDA Root             : /hpc/apps/RHEL/3.10/x86_64/cuda/11.0
CUDA Build Version    : 11000
CUDA Driver Version   : 11020
CUDA Runtime Version  : 11000
cuBLAS Version        : 11200
cuFFT Version         : 10201
cuRAND Version        : 10201
cuSOLVER Version      : (10, 6, 0)
cuSPARSE Version      : 11101
NVRTC Version         : (11, 0)
Thrust Version        : 100909
CUB Build Version     : 100909
cuDNN Build Version   : 8000
cuDNN Version         : 8000
NCCL Build Version    : 2708
NCCL Runtime Version  : 2708
cuTENSOR Version      : None

Code to reproduce

import numpy as np
X = np.random.random_sample((600, 128, 128, 128))
Y = np.random.random_sample((600, 128, 128, 128))

import cupy as cp
# conversion functions between tensorflow and cupy
def tf2cp(x):
    dlcapsule = tf.experimental.dlpack.to_dlpack(x)
    return cp.fromDlpack(dlcapsule)
def cp2tf(x):
    dlcapsule = x.toDlpack()
    return tf.experimental.dlpack.from_dlpack(dlcapsule)

import tensorflow as tf
dataset = tf.data.Dataset.from_tensor_slices((X, Y))
train_dst = dataset.shuffle(500, reshuffle_each_iteration=True)
train_dst = dataset.batch(10, drop_remainder=True).prefetch(tf.data.AUTOTUNE)

for X, Y in dataset.take(10):
    print(X.shape)
    X2 = tf2cp(X)
    X3 = cp.copy(X2)
    X3[...,:] = X2[...,::-1]
    X = cp2tf(X3)

Error messages, stack traces, or logs

---------------------------------------------------------------------------
CUDADriverError                           Traceback (most recent call last)
<ipython-input-3-678fbfef46a0> in <module>
      3     X2 = tf2cp(X)
      4     X3 = cp.copy(X2)
----> 5     X3[...,:] = X2[...,::-1]
      6     X = cp2tf(X3)

cupy/core/core.pyx in cupy.core.core.ndarray.__setitem__()

cupy/core/_routines_indexing.pyx in cupy.core._routines_indexing._ndarray_setitem()

cupy/core/_routines_indexing.pyx in cupy.core._routines_indexing._scatter_op()

cupy/core/_kernel.pyx in cupy.core._kernel.ufunc.__call__()

cupy/core/_kernel.pyx in cupy.core._kernel.ufunc._get_ufunc_kernel()

cupy/core/_kernel.pyx in cupy.core._kernel._get_ufunc_kernel()

cupy/core/_kernel.pyx in cupy.core._kernel._get_simple_elementwise_kernel()

cupy/core/core.pyx in cupy.core.core.compile_with_cache()

/hpc/apps/pyhpc/dist/conda/x86_64/envs/cuda-11.0/lib/python3.8/site-packages/cupy/cuda/compiler.py in compile_with_cache(source, options, arch, cache_dir, extra_source, backend, enable_cooperative_groups, name_expressions, log_stream)
    333             name_expressions, log_stream, cache_in_memory)
    334     else:
--> 335         return _compile_with_cache_cuda(
    336             source, options, arch, cache_dir, extra_source, backend,
    337             enable_cooperative_groups, name_expressions, log_stream,

/hpc/apps/pyhpc/dist/conda/x86_64/envs/cuda-11.0/lib/python3.8/site-packages/cupy/cuda/compiler.py in _compile_with_cache_cuda(source, options, arch, cache_dir, extra_source, backend, enable_cooperative_groups, name_expressions, log_stream, cache_in_memory)
    391                 cubin_hash = hashlib.md5(cubin).hexdigest().encode('ascii')
    392                 if hash == cubin_hash:
--> 393                     mod.load(cubin)
    394                     return mod
    395     else:

cupy/cuda/function.pyx in cupy.cuda.function.Module.load()

cupy/cuda/function.pyx in cupy.cuda.function.Module.load()

cupy_backends/cuda/api/driver.pyx in cupy_backends.cuda.api.driver.moduleLoadData()

cupy_backends/cuda/api/driver.pyx in cupy_backends.cuda.api.driver.check_status()

CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

Update: if I take out train_dst = dataset.shuffle(500, reshuffle_each_iteration=True), the code will work...
Do I have to configure something related to the memory?

I figured out an alternative solution:
If I do the following to manually copy TF tensor to a CuPy device array in advance, then the code will work, however I don't know why the earlier code doesn't work.
X2 = cp.empty((10, 128, 128, 128)) # create a CuPy array in advance
for X, Y in dataset.take(10):
    print(X.shape)
    X2[:] = tf2cp(X) # note the ":", to manually copy results to a CuPy device array
    X3 = cp.copy(X2)
    X3[...,:] = X2[...,::-1]
    X = cp2tf(X3)

The only thing I can guess from this is some issue with the strides of the original tensor in TensorFlow and cupy.

I'm also getting this error, but from a completely different operation. My MRE looks like this:
import cupy as np
np.cuda.set_allocator(np.cuda.malloc_managed)

gigs = 8  # <== if this is 7 or greater, I get the error
x = np.random.random((gigs*1024*1024*256,), dtype=np.float32)
m = np.median(x)
print(m)

As noted in the code, the error only occurs for large data sizes. The error I get looks like
/(user home)/.venvs/bh_script/bin/python /(user home)/.config/JetBrains/PyCharmCE2022.2/scratches/scratch_5.py
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 550, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 143, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.memory.ManagedMemory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 550, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 143, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 550, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 143, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.memory.ManagedMemory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 550, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 143, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Traceback (most recent call last):
  File "/(user home)/.config/JetBrains/PyCharmCE2022.2/scratches/scratch_5.py", line 8, in <module>
    m = np.median(x)
  File "/(user home)/.venvs/bh_script/lib/python3.9/site-packages/cupy/_statistics/meanvar.py", line 34, in median
    return _statistics._median(a, axis, out, overwrite_input, keepdims)
  File "cupy/_core/_routines_statistics.pyx", line 389, in cupy._core._routines_statistics._median
  File "cupy/_core/_routines_statistics.pyx", line 428, in cupy._core._routines_statistics._median
  File "cupy/_core/core.pyx", line 922, in cupy._core.core._ndarray_base.partition
  File "cupy/_core/_routines_sorting.pyx", line 174, in cupy._core._routines_sorting._ndarray_partition
  File "cupy/_core/core.pyx", line 885, in cupy._core.core._ndarray_base.sort
  File "cupy/_core/_routines_sorting.pyx", line 44, in cupy._core._routines_sorting._ndarray_sort
  File "cupy/cuda/thrust.pyx", line 75, in cupy.cuda.thrust.sort
RuntimeError: smart_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 550, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 143, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.memory.ManagedMemory.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/runtime.pyx", line 550, in cupy_backends.cuda.api.runtime.free
  File "cupy_backends/cuda/api/runtime.pyx", line 143, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 217, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy_backends/cuda/api/driver.pyx", line 217, in cupy_backends.cuda.api.driver.moduleUnload
  File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Error in sys.excepthook:

Original exception was:

Process finished with exit code 1

(I haven't truncated the above output - nothing is printed after "Original exception was:")

When using managed memory, cudaErrorIllegalAddress often indicates out-of-memory on host.

Closing as inactive for a while. Please feel free to reopen if you still need help.


[Feat] adding sort functionality to the category page

Description
Adding sort functionality to the category page
Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

https://github.com/vuestorefront-community/prestashop/issues/129


Fix TIME_OF_VACCINATION validation

This fixes an error in the validation where it was expecting a value where it shouldn't.

@thomasleese I added a spec


AbstractFactories for PluginManager not fully compatible with 'shared' option

A plugin manager with such a config
$config = [
    'shared' => [
        'a_service' => true || false
    ],
    'abstract_factories' => [
        'MyAbstractFactory'
    ]
]

new MyPluginManager = (new Config($config));

Where MyAbstractFactory::canCreateServiceWithName tries to get the main SM will fail as that returns an instance MyAbstractFactory.
class MyAbstractFactory implements AbstractFactoryInterface
{
    public function canCreateServiceWithName(ServiceLocatorInterface $serviceLocator, $name, $requestedName)
    {
        var_dump(get_class($serviceLocator));
        // string(15) "MyPluginManager" // correct

        $serviceLocator = $serviceLocator->getServiceLocator();
        var_dump(get_class($serviceLocator));
        // string(17) ""MyAbstractFactory"  // uh? not good expected Zend\ServiceManager\ServiceManager

        return true;
    }

    public function createServiceWithName(ServiceLocatorInterface $serviceLocator, $name, $requestedName)
    {
        var_dump(get_class($serviceLocator));
        // string(15) "MyPluginManager" // correct

        $serviceLocator = $serviceLocator->getServiceLocator();
        var_dump(get_class($serviceLocator));
        // string(34) "Zend\ServiceManager\ServiceManager" // correct
    }
}

When the shared flag is defined you cannot get to the main service locator. Removing the shared flag, avoids the issue.

This issue has been closed as part of the bug migration program as outlined here - http://framework.zend.com/blog/2016-04-11-issue-closures.html


refactor(home): update home layout

Merge conflict on this one

We received some initial help on the card component format from UXD. @mcoker and @kyle gave us mockups which we implemented for these cards.
I know the most recent feedback requested a change of the hierarchy  ( https://marvelapp.com/66cj5ga/screen/60982350 ) , but I think we might have to give this more thought. Feels like a step back IMO

@ibolton336 Looks like the className changed and the cards lost styling. Updated that and added support for .pf-c-page__header background-color.
@eriknelson conflicts resolved.


silent and lucid not work when new shell loads first plugins

Describe the bug
I don't want the message "Loaded $___id" so I set my config file like this.
zinit ice silent lucid wait'!1'; zinit load _local/sandbox
zinit ice silent lucid wait'!1'; zinit load zsh-users/zsh-syntax-highlighting
zinit ice silent lucid wait'!1'; zinit load zsh-users/zsh-completions

But, "Loaded $___id" was shown on new shell.
To Reproduce
Steps to reproduce the behavior.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots

This is value of silent and lucid when zinit is started.
silent and lucid is 1 after "Loaded $___id" having already be shown.
 echo "silent${+ICE[silent]} lucid${+ICE[lucid]}"
    if [[ $___action = *load ]]; then
        if [[ $___tpe = p ]]; then
            .zinit-load "${(@)=___id}" "" "$___mode"; (( ___retval += $? ))
        elif [[ $___tpe = s ]]; then
            .zinit-load-snippet $___opt "$___id"; (( ___retval += $? ))
        elif [[ $___tpe = p1 || $___tpe = s1 ]]; then
            (( ${+functions[.zinit-service]} )) || builtin source "${ZINIT[BIN_DIR]}/zinit-additional.zsh"
            zpty -b "${___id//\//:} / ${ICE[service]}" '.zinit-service '"${(M)___tpe#?}"' "$___mode" "$___id"'
        fi
        echo "silent${+ICE[silent]} lucid${+ICE[lucid]}"
        (( ${+ICE[silent]} == 0 && ${+ICE[lucid]} == 0 && ___retval == 0 )) && zle && zle -M "Loaded $___id"
    elif [[ $___action = *remove ]]; then

Versions:

$ZSH_VERSION == ...
5.8
Operating system name and version (uname -a): ...
Linux coorie 5.8.6-1-MANJARO #1 SMP PREEMPT Thu Sep 3 14:19:36 UTC 2020 x86_64 GNU/Linux
tmux
tmux 3.1b
zinit
284784985

Additional context
Add any other context about the problem here.

Try only lucid, remove silent.
Other way to do the same is
zinit wait'1' lucid for \
    _local/sandbox \
    zsh-users/zsh-syntax-highlighting \
    zsh-users/zsh-completions \

Thank you very much. I tried only lucid again and it worked.


Sony channels not working

Please update the repository
Sony channels are not working in kodi
Thanks 🙏

https://www.apkfiles.com/apk-614249/cricfy-tv-3-5-02
Sideload this app, open chanels from tata play. All channels are working properly.

All Sony channels are working fine. Photos attached below:

https://www.apkfiles.com/apk-614249/cricfy-tv-3-5-02
Sideload this app, open chanels from tata play. All channels are working properly.

Sony sports channels are not working again

Please update the repository Sony channels are not working in kodi Thanks 🙏

https://github.com/tobalan/tobalan.github.io/issues/42#issuecomment-2151411198

All Sony channels are working fine. Photos attached below:

Please guide how to configure and share the SonyLIV playlist


Encoding IplImage.ToImage() and saving to file, using go image, creates a black image

I'm getting this strange bug where i can save the IplImage using opencv but when i try to do the same with the GoImage version, it creates a black image. I checked that the IplImage and GoImage pixels were similar using Get2D and At respectively. I need to ship the opencv image to another process so i need to get the raw data. The only way i could get this is using goimage, encoding to base64 sending it.
Basically,
buf := new(bytes.Buffer)
err := jpeg.Encode(buf, IplImage.ToImage(), nil)
bytestring_to_send := base64.StdEncoding.EncodeString(buf.Bytes())

Hi, thanks to @paydro , this seems to be fixed in #40
Please reopen if there is any concern.
Thanks.


Implement receiving new EC-compatible chains

Solves #8

Thanks for the comments! I have just addressed them.


fix: Inline parameters in inline_call if possible

Fixes #9491

Managed to make the whitespace work out in most situations(?), not much one can do about until we have our own formatter.
bors r+

bors retry

Yeah, I remeber starting to look into whitespace here yesterday, but I don't remember where this ended.
I recall we have a similar logic for USE there already, and it seems that we want to maybe split out:

figure whitespace within a single line
figure linebreaks/indentation

Ye I was trying to reuse that, but it didn't work, which was partially because of the insertion order being different here I think? This whitespace magic is really neat but something my head really doesnt like to think about it seems 😅

but I don't remember where this ended

Ah, seems that I just refactored the existing code: https://github.com/rust-analyzer/rust-analyzer/pull/9490 :)


Incorrect js code generated moving from Reason to Rescript

We had this ReasonML code:
type t;
<EMAIL_ADDRESS>external make: unit => t = "DOMParser";
<EMAIL_ADDRESS>t]
external parseHtmlFromString: (string<EMAIL_ADDRESS>"text/html"] _) => Dom.htmlDocument = "parseFromString";

Which, generated code just as expected on the call-site.
We transpiled that to Rescript. and now we're getting a weird
import * as Caml_external_polyfill from "bs-platform/lib/es6/caml_external_polyfill.js";
 Caml_external_polyfill.resolve("parseFromString")(content, new DOMParser());

on the call site. which, result in a runtime exception with some weird error saying this function isn't pollyfilled yet.
My questions are:
Where is this magic coming from?
When was this magic introduced?
This caused a runtime exception in production and was completely unexpected and impossible for us to even know that it'll happen.

Have you checked the difference in generated code after upgrade? That's the recommended way to proceed, which has been used by people successfully. The difference should be very nearly zero.

Hey @BlueHotDog, can you show me the code that is calling this external and results in the runtime problem?
I just converted your external Reason snippet to ReScript and compared the two. Both ASTs are the same, so I'm guessing that the problem is coming from the call site.

@cristianoc having to manually check each file we transform would result in so much manual, careful labor since we dont even commit the JS files to git so i dont see how this is a solution - we're running rescript on thousands of files and tens of projects in a monorepo setup, doing this for evey file migration means we'll not move to rescript anytime soon which puts us between a rock and a hard place - Can't upgrade, can't trust an robust upgrade process.

Hey @BlueHotDog, can you show me the code that is calling this external and results in the runtime problem?
I just converted your external Reason snippet to ReScript and compared the two. Both ASTs are the same, so I'm guessing that the problem is coming from the call site.

Just tried it on the playground, the result is the same as i reported:
https://rescript-lang.org/try?code=C4TwDgpgBMBQACA7CB3KEAewICdEEMAbKAW3wGsIAuKAV0QEtgoBeAPhlagCIARAeQCyABXw4Azrm4JJiACYA6MA0gAKGsACUsTNjxEoYMZIASwEoQBiOAPYkAysBwNEAcxqrxTl64A0UeHxxVW5sLAB6AAtzQm5NKAB9ePYoXjsFaIs0gGNaEghEZhYeIwkIaztHZzdpWAApcQVCG1dVMkpVeIAfDlLTGIqHbzcQ8TkxuM0AbiA

@send.pipe --> @bs.send.pipe
Note send.pipe is deprecated.

The playground seems ok with @bs.send.pipe.

Thanks for the example @BlueHotDog
Reason snippet

If you press the Res button, it will convert to ReScript snippet

Is it possible that you did the conversion manually and made a typo? It should be @bs.send.pipe(: t), not @send.pipe(: t).
Which ReScript version are you using?
I added an extra test case to test that the syntax converts everything correctly here: https://github.com/rescript-lang/syntax/pull/401

Also, send.pipe is deprecated, and will go soon.

I did the conversion automatically. so there's definitely a bug IMO.
If you do this:
type t;
<EMAIL_ADDRESS>external make: unit => t = "DOMParser";
<EMAIL_ADDRESS>t]
external parseHtmlFromString: (string<EMAIL_ADDRESS>"text/html"] _) => Dom.htmlDocument = "parseFromString";

Js.log(make() |> parseHtmlFromString("sdsd"));

you get this js code:
console.log(new DOMParser().parseFromString("sdsd", "text/html"));

which is exactly what one might expect.
But if you do this:
type t;
[@new] external make: unit => t = "DOMParser";
<EMAIL_ADDRESS>t]
external parseHtmlFromString: (string<EMAIL_ADDRESS>"text/html"] _) => Dom.htmlDocument = "parseFromString";

Js.log(make() |> parseHtmlFromString("sdsd"));

you get this code:
var Caml_external_polyfill = require("./stdlib/caml_external_polyfill.js");

console.log(Caml_external_polyfill.resolve("DOMParser")(undefined).parseFromString("sdsd", "text/html"));

now, i've three questions here:

Why this magic polyfill is a thing? and where is it documented?
What can i do to make sure this doesnt happen in production?
Where was some discussion i can read on on this pollyfill thingie

Would it be possible to send me the rescript compiler version you're using?

I'm using this:
https://reasonml.github.io/en/try.html?rrjsx=true&reason=C4TwDgpgBMDcBQBtAAgIwM4DoB2EDuAulBAB7AQBO2AhgDZQC21A1hAFxQCu2AlsFAF4AfDEFQARABEA8gFkACtQrpK4hCgyYV2ACaYwPSB2AF4pclTpQwSlQAlgDWgDEKAewYBlYBR7YA5hwAFOg+fv4ANFAaWNToEuRkAPQAFo604kQA+gCUgiKSHphpToUAxpwMENj8AhI2yhCuHt6+AWrw8ABSWLRu-kFMrEF5AD4iDfbpzV5hAUHi6DpL4jk5sEA

@cristianoc having to manually check each file we transform would result in so much manual, careful labor since we dont even commit the JS files to git so i dont see how this is a solution - we're running rescript on thousands of files and tens of projects in a monorepo setup, doing this for evey file migration means we'll not move to rescript anytime soon which puts us between a rock and a hard place - Can't upgrade, can't trust an robust upgrade process.

I am literally in the same process for a client project of mine, and did just that (1000+ files).
Create a branch, check in the build artifacts, do the syntax transformation, rebuild the project, check the changes in the JS output.
As long as you are not upgrading the compiler + changing the syntax at the same, there shouldn't be that many considerable diffs.

I just checked on the latest ReScript and can't reproduce the problem, doesn't reproduce in the playground either.
This is how you do the conversion:
% npx rescript -v
9.1.2

% cat src/test.re
type t;
<EMAIL_ADDRESS>external make: unit => t = "DOMParser";
<EMAIL_ADDRESS>t]
external parseHtmlFromString: (string<EMAIL_ADDRESS>"text/html"] _) => Dom.htmlDocument = "parseFromString";

Js.log(make() |> parseHtmlFromString("sdsd"));

% npx rescript format src/test.re
type t
@new external make: unit => t = "DOMParser"
@bs.send.pipe(: t)
external parseHtmlFromString: (string, @as("text/html") _) => Dom.htmlDocument = "parseFromString"

Js.log(make() |> parseHtmlFromString("sdsd"))

% npx rescript format src/test.re > src/test.res
% rm src/test.re

% npx rescript build
rescript: [1/3] src/test.ast

  Warning number 103
  /Users/mvalcke/Development/Praktijk/src/test.res:3:1-13

  1 │ type t
  2 │ @new external make: unit => t = "DOMParser"
  3 │ @bs.send.pipe(: t)
  4 │ external parseHtmlFromString: (string, @as("text/html") _) => Dom.htmlDo
    │ cument = "parseFromString"
  5 │

  FFI warning: This attribute is deprecated, use @send instead.

rescript: [3/3] src/test.cmj

% cat lib/js/src/test.js
// Generated by ReScript, PLEASE EDIT WITH CARE
'use strict';

console.log(new DOMParser().parseFromString("sdsd", "text/html"));

/*  Not a pure module */

Regarding your questions:

Why this magic polyfill is a thing? and where is it documented?
The snippet that you showed me contained a typo in the attribute. @send.pipe does not exist. It should be @bs.send.pipe Without the @bs.send.pipe, the compiler tried to resolve it as a global value.

What can i do to make sure this doesnt happen in production?
Externals can be rather unsafe. Thorough unit testing of externals can help here.

Where was some discussion i can read on on this pollyfill thingie
This is a consequence of how externals are modelled in the ocaml compiler. Ocaml native needs to deal with the c runtime, ReScript needs to support a js runtime across browsers/node.

Will close this issue, the conversion works ok in the playground and latest rescript on npm.


Kills enemies when using the on off switches

It would be nice if the on/off switches could kill enemies in Mario Maker physics.
`function buttonblock:change()
self.solid = not self.solid
if not self.solid then
    self.quad = self.quadoff
    self.active = false
else
    self.quad = self.quadon
    self.active = true
    for _, obj in pairs(objects["enemy"] or {}) do
        if math.abs(obj.x - self.x) < obj.width and math.abs(obj.y - self.y) < obj.height then
            if obj.shotted then
                local dir = (self.x < obj.x) and 1 or -1
                obj:shotted(dir)
            end
        end
    end
end

end
`
https://github.com/user-attachments/assets/16cc8513-fe19-4a54-805a-d4bdb010d403

This should probably take into account enemies with "resistseverything"


ValueError: Provided DataFrame (df) must be of pd.DataFrame type.

Hi there,
I've installed Predai  on my HA  which is connected to a GE inverter and Zappi charger. I've configured for 5 day interval at the moment but Predai seems to be suffering a ValueError: Provided DataFrame (df) must be of pd.DataFrame type.
Hope this proves useful but let me know if you require me to alter settings at my end.
Log below:
Configuration loaded
Processing sensor sensor.givtcp_xxx_load_energy_total_kwh incrementing True reset_daily True interval 30 days 5 subtract sensor.myenergi_zappi_xxx_charge_added_session
Getting history for sensor sensor.givtcp_xxxxx_load_energy_total_kwh
Making API call to http://supervisor/core/api/history/period/2024-04-10T15:00:00+0100
Response <Response [200]>
Getting history for sensor sensor.myenergi_zappi_xxxxx_charge_added_session
Making API call to http://supervisor/core/api/history/period/2024-04-10T15:00:00+0100
Response <Response [200]>
Processing dataset
                           ds    y
0   2024-04-10 15:00:00+01:00  0.0
1   2024-04-10 15:30:00+01:00  0.4
2   2024-04-10 16:00:00+01:00  0.3
3   2024-04-10 16:30:00+01:00  0.3
4   2024-04-10 17:00:00+01:00  0.3
..                        ...  ...
235 2024-04-15 12:30:00+01:00  0.1
236 2024-04-15 13:00:00+01:00  0.6
237 2024-04-15 13:30:00+01:00  0.1
238 2024-04-15 14:00:00+01:00  0.4
239 2024-04-15 14:30:00+01:00  0.1

[240 rows x 2 columns]
Processing subtract
                           ds    y
0   2024-04-10 15:00:00+01:00  0.0
1   2024-04-10 15:30:00+01:00  0.0
2   2024-04-10 16:00:00+01:00  0.0
3   2024-04-10 16:30:00+01:00  0.0
4   2024-04-10 17:00:00+01:00  0.0
..                        ...  ...
232 2024-04-15 11:00:00+01:00  0.0
233 2024-04-15 11:30:00+01:00  0.0
234 2024-04-15 12:00:00+01:00  0.0
235 2024-04-15 12:30:00+01:00  0.0
236 2024-04-15 13:00:00+01:00  0.0

[237 rows x 2 columns]
Subtracting data
Pruned set:                            ds    y
0   2024-04-10 15:00:00+01:00  0.0
1   2024-04-10 15:30:00+01:00  0.4
2   2024-04-10 16:00:00+01:00  0.3
3   2024-04-10 16:30:00+01:00  0.3
4   2024-04-10 17:00:00+01:00  0.3
..                        ...  ...
235 2024-04-15 12:30:00+01:00  0.1
236 2024-04-15 13:00:00+01:00  0.6
237 2024-04-15 13:30:00+01:00  0.1
238 2024-04-15 14:00:00+01:00  0.4
239 2024-04-15 14:30:00+01:00  0.1

[240 rows x 2 columns]
WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

Finding best initial lr:   0%|          | 0/213 [00:00<?, ?it/s]
Finding best initial lr:   0%|          | 1/213 [00:00<00:24,  8.77it/s]
Finding best initial lr:   2%|▏         | 4/213 [00:00<00:12, 17.41it/s]
Finding best initial lr:   3%|▎         | 7/213 [00:00<00:10, 20.01it/s]
Finding best initial lr:   5%|▍         | 10/213 [00:00<00:09, 21.20it/s]
Finding best initial lr:   6%|▌         | 13/213 [00:00<00:09, 20.47it/s]
Finding best initial lr:   8%|▊         | 16/213 [00:00<00:09, 20.95it/s]
Finding best initial lr:   9%|▉         | 19/213 [00:00<00:09, 21.05it/s]
Finding best initial lr:  10%|█         | 22/213 [00:01<00:08, 21.56it/s]
Finding best initial lr:  12%|█▏        | 25/213 [00:01<00:08, 21.93it/s]
Finding best initial lr:  13%|█▎        | 28/213 [00:01<00:08, 22.18it/s]
Finding best initial lr:  15%|█▍        | 31/213 [00:01<00:08, 21.97it/s]
Finding best initial lr:  16%|█▌        | 34/213 [00:01<00:08, 22.31it/s]
Finding best initial lr:  17%|█▋        | 37/213 [00:01<00:07, 22.59it/s]
Finding best initial lr:  19%|█▉        | 40/213 [00:01<00:07, 22.37it/s]
Finding best initial lr:  20%|██        | 43/213 [00:02<00:08, 20.73it/s]
Finding best initial lr:  22%|██▏       | 46/213 [00:02<00:07, 21.12it/s]
Finding best initial lr:  23%|██▎       | 49/213 [00:02<00:07, 21.62it/s]
Finding best initial lr:  24%|██▍       | 52/213 [00:02<00:07, 22.07it/s]
Finding best initial lr:  26%|██▌       | 55/213 [00:02<00:07, 22.52it/s]
Finding best initial lr:  27%|██▋       | 58/213 [00:02<00:06, 22.78it/s]
Finding best initial lr:  29%|██▊       | 61/213 [00:02<00:06, 22.65it/s]
Finding best initial lr:  30%|███       | 64/213 [00:02<00:06, 22.74it/s]
Finding best initial lr:  31%|███▏      | 67/213 [00:03<00:06, 22.83it/s]
Finding best initial lr:  33%|███▎      | 70/213 [00:03<00:06, 22.89it/s]
Finding best initial lr:  34%|███▍      | 73/213 [00:03<00:06, 22.87it/s]
Finding best initial lr:  36%|███▌      | 76/213 [00:03<00:05, 24.51it/s]
Finding best initial lr:  38%|███▊      | 80/213 [00:03<00:05, 25.84it/s]
Finding best initial lr:  39%|███▉      | 84/213 [00:03<00:04, 28.04it/s]
Finding best initial lr:  41%|████▏     | 88/213 [00:03<00:04, 29.77it/s]
Finding best initial lr:  43%|████▎     | 91/213 [00:03<00:04, 29.48it/s]
Finding best initial lr:  45%|████▍     | 95/213 [00:04<00:03, 29.83it/s]
Finding best initial lr:  46%|████▌     | 98/213 [00:04<00:03, 28.83it/s]
Finding best initial lr:  47%|████▋     | 101/213 [00:04<00:03, 28.83it/s]
Finding best initial lr:  49%|████▉     | 104/213 [00:04<00:03, 28.80it/s]
Finding best initial lr:  50%|█████     | 107/213 [00:04<00:03, 27.71it/s]
Finding best initial lr:  52%|█████▏    | 110/213 [00:04<00:03, 26.36it/s]
Finding best initial lr:  53%|█████▎    | 113/213 [00:04<00:04, 23.97it/s]
Finding best initial lr:  54%|█████▍    | 116/213 [00:04<00:04, 20.99it/s]
Finding best initial lr:  56%|█████▌    | 119/213 [00:05<00:04, 22.22it/s]
Finding best initial lr:  57%|█████▋    | 122/213 [00:05<00:03, 22.85it/s]
Finding best initial lr:  59%|█████▊    | 125/213 [00:05<00:03, 23.64it/s]
Finding best initial lr:  60%|██████    | 128/213 [00:05<00:03, 23.91it/s]
Finding best initial lr:  62%|██████▏   | 131/213 [00:05<00:03, 23.98it/s]
Finding best initial lr:  63%|██████▎   | 134/213 [00:05<00:03, 24.22it/s]
Finding best initial lr:  64%|██████▍   | 137/213 [00:05<00:03, 23.75it/s]
Finding best initial lr:  66%|██████▌   | 140/213 [00:05<00:03, 24.13it/s]
Finding best initial lr:  67%|██████▋   | 143/213 [00:06<00:02, 23.88it/s]
Finding best initial lr:  69%|██████▊   | 146/213 [00:06<00:02, 23.59it/s]
Finding best initial lr:  70%|██████▉   | 149/213 [00:06<00:02, 23.63it/s]
Finding best initial lr:  71%|███████▏  | 152/213 [00:06<00:02, 22.94it/s]
Finding best initial lr:  73%|███████▎  | 155/213 [00:06<00:02, 22.66it/s]
Finding best initial lr:  74%|███████▍  | 158/213 [00:06<00:02, 22.37it/s]
Finding best initial lr:  76%|███████▌  | 161/213 [00:06<00:02, 22.57it/s]
Finding best initial lr:  77%|███████▋  | 164/213 [00:07<00:02, 22.58it/s]
Finding best initial lr:  78%|███████▊  | 167/213 [00:07<00:02, 22.22it/s]
Finding best initial lr:  80%|███████▉  | 170/213 [00:07<00:01, 22.24it/s]
Finding best initial lr:  81%|████████  | 173/213 [00:07<00:01, 22.56it/s]
Finding best initial lr:  83%|████████▎ | 176/213 [00:07<00:01, 22.82it/s]
Finding best initial lr:  84%|████████▍ | 179/213 [00:07<00:01, 22.62it/s]
Finding best initial lr:  85%|████████▌ | 182/213 [00:07<00:01, 22.06it/s]
Finding best initial lr:  87%|████████▋ | 185/213 [00:07<00:01, 21.91it/s]
Finding best initial lr:  88%|████████▊ | 188/213 [00:08<00:01, 22.07it/s]
Finding best initial lr:  90%|████████▉ | 191/213 [00:08<00:00, 22.14it/s]
Finding best initial lr:  91%|█████████ | 194/213 [00:08<00:00, 22.39it/s]
Finding best initial lr:  92%|█████████▏| 197/213 [00:08<00:00, 21.90it/s]
Finding best initial lr:  94%|█████████▍| 200/213 [00:08<00:00, 22.14it/s]
Finding best initial lr:  95%|█████████▌| 203/213 [00:08<00:00, 22.25it/s]
Finding best initial lr:  97%|█████████▋| 206/213 [00:08<00:00, 22.53it/s]
Finding best initial lr:  98%|█████████▊| 209/213 [00:09<00:00, 22.04it/s]
Finding best initial lr: 100%|█████████▉| 212/213 [00:09<00:00, 22.07it/s]
Finding best initial lr: 100%|██████████| 213/213 [00:09<00:00, 23.10it/s]
WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/data/split.py:273: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat([df, future_df])

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

WARNING - (py.warnings._showwarnmsg) - /usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py:1152: FutureWarning: Series.view is deprecated and will be removed in a future version. Use ``astype`` as an alternative to change the dtype.
  converted_ds = pd.to_datetime(ds_col, utc=True).view(dtype=np.int64)

                     ds    y     yhat1     trend  season_daily
0   2024-04-10 14:00:00  0.0  0.335622  0.500582     -0.164960
1   2024-04-10 14:30:00  0.4  0.345751  0.499676     -0.153925
2   2024-04-10 15:00:00  0.3  0.348289  0.498769     -0.150480
3   2024-04-10 15:30:00  0.3  0.357080  0.497863     -0.140783
4   2024-04-10 16:00:00  0.3  0.383938  0.496956     -0.113018
..                  ...  ...       ...       ...           ...
331 2024-04-17 11:30:00  NaN  0.431648  0.596397     -0.164749
332 2024-04-17 12:00:00  NaN  0.367677  0.596713     -0.229036
333 2024-04-17 12:30:00  NaN  0.351509  0.597029     -0.245520
334 2024-04-17 13:00:00  NaN  0.372219  0.597345     -0.225126
335 2024-04-17 13:30:00  NaN  0.405630  0.597661     -0.192031

[336 rows x 5 columns]
Saving prediction to sensor.givtcp_xxxxx_load_energy_total_kwh_prediction last_update 2024-04-15 15:00:00+01:00
Making API call to http://supervisor/core/api/states/sensor.givtcp_xxxxx_load_energy_total_kwh_prediction
Response <Response [200]>
Processing sensor sensor.external_temperature incrementing False reset_daily False interval 30 days 5 subtract None
Getting history for sensor sensor.external_temperature
Making API call to http://supervisor/core/api/history/period/2024-04-10T15:00:00+0100
Response <Response [200]>
Traceback (most recent call last):
  File "/config/predai.py", line 267, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/config/predai.py", line 261, in main
    await nw.train(pruned, future_periods)
  File "/config/predai.py", line 141, in train
    self.metrics = self.model.fit(dataset, freq=(str(self.period) + "min"), progress=None)
  File "/usr/local/lib/python3.10/dist-packages/neuralprophet/forecaster.py", line 1015, in fit
    df, _, _, self.id_list = df_utils.prep_or_copy_df(df)
  File "/usr/local/lib/python3.10/dist-packages/neuralprophet/df_utils.py", line 43, in prep_or_copy_df
    raise ValueError("Provided DataFrame (df) must be of pd.DataFrame type.")
ValueError: Provided DataFrame (df) must be of pd.DataFrame type.

User error at my end.


Bevy 0.13

This is provided by #45 . Could you please review that and if it is good, release it?

@brandon-reinhart thank you

Fixed by #45


fix: Correct form button text - todoApp

Checklist:

[x] I have read and followed the contribution guidelines.
[x] I have read and followed the how to open a pull request guide.
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Closes #54589

This PR resolves all the points highlighted by @jdwilkin4 in the aforementioned issue

@Deep512
Also, you are failing a few of the automated tests.

Since you added a new step, you need to make sure that the new step has the solution from directions of the previous step.
Otherwise, the tests will fail.

Make sure to run the tests locally to make sure they pass before pushing up new changes.
Here is the command you will need to run the tests locally
Make sure the local server is running first
FCC_BLOCK='Learn localStorage by Building a Todo App' pnpm run test:curriculum

Here is the command you will need to run the tests locally, Make sure the local server is running first
FCC_BLOCK='Learn localStorage by Building a Todo App' pnpm run test:curriculum

@jdwilkin4 Can these tests be run on gitpod? I can't setup this up locally because of work laptop environment restrictions.
Error after successfully running the server, and then the test command:

@Deep512 You need to follow this manual https://contribute.freecodecamp.org/#/how-to-add-cypress-tests?id=cypress-gitpod-setup


[GH-15740] channel mentions in message attachments

Summary

This PR fix the webapp problem with message attachements for channel and internal link.
Ticket Link

Jira ticket: https://mattermost.atlassian.net/browse/MM-28980
https://github.com/mattermost/mattermost-server/issues/15740 => For the webapp part
Related Pull Requests

None
Screenshots

No UI change

Hello @jufab,
Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.

I create a message with attachments in the town-square channel => This redirect to Off-Topic

No problem, it's done!

/update-branch

@prapti Kind reminder to review for v5.29.

/update-branch

Testing this, but waiting on some answers from Alejandro about test steps. Will continue on Monday.

Cherry pick is scheduled.


[BUG] memory leak under zero.Init

Describe the bug
only when activating zero.Init the code leaks a lot per training iteration.
To Reproduce
I'm yet to be able to reduce this to a simple test. I shared with Tunji how to reproduce it in the large framework.
However we found the source of the leak and rewrote the module that was leaking and the leak was gone.
So unless you can see something that points to a bug in deepspeed, this is a post for posterity and can be closed.
Here is the original module that was leaking:
class DecoupledLinear(nn.Linear):
    # Derived from https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html#Linear
    """
    Implements a decoupling of parameters to allow freezing (or not) a subset of the parameters.
    In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `out_additional_features` > 0, then it will create `out_additional_features * in_features` additional parameters that are always trained.
    If `out_additional_features=0`, then the module defaults back to the regular behavior of `nn.Linear`.
    """

    def __init__(
        self,
        in_features: int,
        out_features: int,
        out_additional_features: int = 0,
        bias: bool = True,
        partially_freeze: bool = True,
        device=None,
        dtype=None,
    ) -> None:
        """
        out_additional_features: int. Number of additional trainable dimensions. Only makes sense when `partially_freeze=True`.
        partially_freeze:
        """
        super().__init__(in_features, out_features, bias, device, dtype)
        self.out_additional_features = out_additional_features
        self.partially_freeze = partially_freeze

        if partially_freeze:
            self.weight.requires_grad_(False)
            if bias:
                self.bias.requires_grad_(False)

        if out_additional_features > 0:
            self.additional_weight = Parameter(
                torch.empty((out_additional_features, in_features), device=device, dtype=dtype)
            )
            if bias:
                self.additional_bias = Parameter(torch.empty(out_additional_features, device=device, dtype=dtype))
            else:
                self.register_parameter("additional_bias", None)
            self.reset_additional_parameters()

    def reset_additional_parameters(self) -> None:
        """Equivalent of the `nn.Linear.reset_parameters` but only for the additional parameters."""
        init.kaiming_uniform_(self.additional_weight, a=math.sqrt(5))
        if self.additional_bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.additional_weight)
            bound = 1 / math.sqrt(fan_in) if fan_in > 0 else 0
            init.uniform_(self.additional_bias, -bound, bound)

    def forward(self, input: torch.Tensor) -> torch.Tensor:
        if self.out_additional_features > 0:
            all_weight = torch.cat((self.weight, self.additional_weight), 0)
            if self.bias is not None:
                all_bias = torch.cat((self.bias, self.additional_bias), 0)
            else:
                all_bias = None
        else:
            all_weight = self.weight
            if self.bias is not None:
                all_bias = self.bias
            else:
                all_bias = None
        return F.linear(input, all_weight, all_bias)

I think it's the concatenations of the 2 parts of the linear somehow lead to the leak. The 2 parts were needed in order to make part of the linear layer frozen.
I rewrote it as following  to deal with each part of the linear separately and the leak disappeared:
class DecoupledLinearNew(nn.Linear):
    # Derived from https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html#Linear
    """
    Implements a decoupling of parameters to allow freezing (or not) a subset of the parameters.
    In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `out_additional_features` > 0, then it will create `out_additional_features * in_features` additional parameters that are always trained.
    If `out_additional_features=0`, then the module defaults back to the regular behavior of `nn.Linear`.
    """

    def __init__(
        self,
        in_features: int,
        out_features: int,
        out_additional_features: int = 0,
        bias: bool = True,
        partially_freeze: bool = True,
        device=None,
        dtype=None,
    ) -> None:
        """
        out_additional_features: int. Number of additional trainable dimensions. Only makes sense when `partially_freeze=True`.
        partially_freeze: bool. If True, the regular `weight` will be frozen and extra parameters (if any) will be trainable. If False, default to the regular behavior of nn.Linear.
        """
        super().__init__(in_features, out_features, bias, device, dtype)
        self.out_additional_features = out_additional_features
        self.partially_freeze = partially_freeze

        self.in_features = in_features
        self.out_features = out_features

        if partially_freeze:
            self.weight.requires_grad_(False)
            if bias:
                self.bias.requires_grad_(False)

        if out_additional_features > 0:
            self.additional_fc = nn.Linear(
                in_features=in_features,
                out_features=out_additional_features,
                bias=bias,
            )

    def forward(self, input: torch.Tensor) -> torch.Tensor:
        output = F.linear(input, self.weight, self.bias)

        if self.out_additional_features > 0:
            output_2 = F.linear(input, self.additional_fc.weight, self.additional_fc.bias)
            output = torch.cat((output, output_2), -1)

        return output

as you can see I reworked it to remove torch.cat calls and the leak is gone. It's also a much more efficient code.
I still don't understand why would zero.Init be the trigger for such leak, as zero3 w/o zero.Init or zero2 did not leak.
While I have a more efficient solution and which doesn't leak I thought I would report this here for awareness and perhaps you can also see something obvious here that I'm missing.
It's also possible the the leak trigger was somehow in how this module was used.
@tjruwase

Hi @stas00, two questions:

Do believe this PR sufficiently fixes the issue?
How were you able to identify that memory was being leak?

Thanks,
Joe

Hi @stas00, two questions:
1. Do believe this [PR](https://github.com/microsoft/DeepSpeed/pull/2665) sufficiently fixes the issue?

I'm pretty sure this other leak was unrelated. As the one I fixed was only a temp leak, on forward it'd get partitioned again. This one kept on increasing the memory usage with every iteration.

2. How were you able to identify that memory was being leak?

On every iteration the memory was growing, and it only happened with zero.Init enabled.

Thanks @stas00 . I am attempting to repro this locally. Could you pass along the ds_config being used and the zero.Init call (a small repro script would work as well).

Thank you for trying to reproduce this, Joe
If I had a small repro script I probably would have found the problem, but, alas, it appeared to be hidden somewhere in the ensemble of things.
ds_config is just the staple defaults of zero3 + zero.Init as integrated into transformer's from_pretrained - no offload or anything.
I think the issue was coming from nested from_pretrained calls which triggered nested zero.Init calls.
As we overcame the leakage issue with rewriting several major parts of the code base to completely remove from_pretrained nesting and me not being able to narrow it down to a simple script, perhaps we just close this issue for now?

Sounds good. If anything similar happens again feel free reopen the issue and I will help investigate.

hello @stas00 , I'm glad I found this issue as I've been struggling to debug memory issues while loading models using huggingface transformers.
i've tested to load model using from_pretrained in both 1 gpu and 8 gpu HW setting.
        args = {
            'pretrained_model_name_or_path': model_path,
            'config': config,
            'torch_dtype': "auto",
        }
        model = model_class.from_pretrained(**args)
        gpu_memory_plot_helper(device, "after initializing model")

by using estimate_zero3_model_states_mem_needs_all_cold function, i expect below memory usage for each 1 and 8 gpus setting.
HW: Setup with 1 node, 1 GPU per node.
SW: Model with 6897M total params, 314M largest layer params.
  per CPU  |  per GPU |   Options
  173.43GB |   1.17GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1
  173.43GB |   1.17GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0
  154.16GB |  14.02GB | offload_param=none, offload_optimizer=cpu , zero_init=1
  154.16GB |  14.02GB | offload_param=none, offload_optimizer=cpu , zero_init=0
    1.75GB | 116.79GB | offload_param=none, offload_optimizer=none, zero_init=1
   38.54GB | 116.79GB | offload_param=none, offload_optimizer=none, zero_init=0

HW: Setup with 1 node, 8 GPUs per node.
SW: Model with 6897M total params, 314M largest layer params.
  per CPU  |  per GPU |   Options
  173.43GB |   1.17GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1
  308.32GB |   1.17GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0
  154.16GB |   2.78GB | offload_param=none, offload_optimizer=cpu , zero_init=1
  308.32GB |   2.78GB | offload_param=none, offload_optimizer=cpu , zero_init=0
   14.04GB |  15.62GB | offload_param=none, offload_optimizer=none, zero_init=1
  308.32GB |  15.62GB | offload_param=none, offload_optimizer=none, zero_init=0

i used below deepspeed configuration with huggingface accelerate with --zero3_init_flag true option.
to my best knowledge, accelerate call deepspeed internally, so it should consume above expected memory.
{
  "bf16": {
    "enabled": true
  },
  "zero_optimization": {
    "stage": 3,
    "offload_param": {
      "device": "cpu",
      "pin_memory": true
    },
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": true
    },
    "stage3_gather_16bit_weights_on_model_save": true,
    "memory_efficient_linear": true,
    "allgather_bucket_size": 2e8,
    "reduce_bucket_size": 2e8,

    "stage3_max_live_parameters": 3e7,
    "stage3_prefetch_bucket_size": 3e7,
    "stage3_param_persistence_threshold": 1e4,
    "stage3_max_reuse_distance": 5e8
  },
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",

  "communication_data_type": null,

  "gradient_clipping": 1.0,
  "prescale_gradients": false
}

however the memory profiler gave me following stats.
        after initializing model and tokenizer
        rank: 0 / device: cuda:0
        CPU Virtual Memory:  used = 54.74 GB, percent = 2.7%
        Allocated / Reserved: 931.07MB / 1204.00MB
        Max Allocated / Max Reserved: 931.07MB / 1204.00MB
        summary
        |===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |    931 MiB |    931 MiB |  45573 MiB |  44642 MiB |
|---------------------------------------------------------------------------|
| Active memory         |    931 MiB |    931 MiB |  45573 MiB |  44642 MiB |
|---------------------------------------------------------------------------|
| Requested memory      |    931 MiB |    931 MiB |  45573 MiB |  44642 MiB |
|---------------------------------------------------------------------------|
| GPU reserved memory   |   1204 MiB |   1204 MiB |   3004 MiB |   1800 MiB |
|---------------------------------------------------------------------------|
| Non-releasable memory | 279479 KiB | 279479 KiB |  53150 MiB |  52877 MiB |
|---------------------------------------------------------------------------|
| Allocations           |     467    |     468    |    2259    |    1792    |
|---------------------------------------------------------------------------|
| Active allocs         |     468    |     468    |    2259    |    1791    |
|---------------------------------------------------------------------------|
| GPU reserved segments |       4    |       4    |       7    |       3    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       5    |       6    |     911    |     906    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

it return similar results for both 1 gpu and 8 gpu setting.
however the memory estimated by deepspeed function consider both optimizer and model parameter, while in my case i just only initialized and loaded model parameter, so it should not comsume any gpu memory for parameter i guess.
(idk if accelerate reserved expected buffer for forward, backward tensors but)
and i checked it does not allocate any gpu memory if i did not use zero3 init flag for accelerate...
following python function is what i used for profile and i used this right after loading model parameter with from_pretrained() .
def gpu_memory_plot_helper(
    rank,
    device,
    message: str,
):
    gc.collect()

    torch.cuda.synchronize(device)

    allocated = torch.cuda.memory_allocated(device) / (1024**2)
    max_allocated = torch.cuda.max_memory_allocated(device) / (1024**2)
    reserved = torch.cuda.memory_reserved(device) / (1024**2)
    max_reserved = torch.cuda. max_memory_reserved(device) / (1024**2)

    vm_stats = psutil.virtual_memory()
    used_GB = round(((vm_stats.total - vm_stats.available) / (1024**3)), 2)

    logger.info(
        f'''
        {message}
        rank: {rank} / device: {device}
        CPU Virtual Memory:  used = {used_GB} GB, percent = {vm_stats.percent}%
        Allocated / Reserved: {allocated:.2f}MB / {reserved:.2f}MB
        Max Allocated / Max Reserved: {max_allocated:.2f}MB / {max_reserved:.2f}MB
        summary
        {torch.cuda.memory_summary(device, abbreviated=True)}
        '''
    )

    torch.cuda.reset_peak_memory_stats(device)

i would very much appreciate if you answer me!
best regards.

I have no idea how to give a confirmation from what you have shared, it's a lot of information but a lot of it is irrelevant. I think it's also more difficult to do any such evals when you offload to CPU as your config shows. Much simpler to use GPUs straight and then you get a single measurement.
Perhaps instead of using the profiler try to pass --skip_memory_metrics 0 as explained here: https://github.com/huggingface/transformers/blob/0864dd3beb238b7bec3528a3d1d6c17a28f51a51/src/transformers/training_args.py#L532
and let it do the reporting of the memory usage for you.
The memory estimators only give a suggestion of params and grads and optim states. Other allocations could be quite significant.
But if it's doing the right thing and the model is large enough you should definitely see a significant difference in memory usage between 1 and 8 gpus, regardless if you use zero.Init or not - the usage of the latter only is important if you can't load the whole model on a single GPU. The model will get sharded regardless by the time deepspeed got initialized.
Here is a practical example. Let's take t5-3b and do a translation example from HF transformers
1 vs 8 A100 (about 50GB GPU memory)
git clone https://github.com/huggingface/transformers
cd transformers

now edit tests/deepspeed/ds_config_zero3.json to turn off cpu offload (see the outcome at the end of this comment)
now run for 1 gpu:
# 1 gpu
export BS=1; rm -r output_dir; PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0,1 deepspeed --num_gpus=1 examples/pytorch/translation/run_translation.py --model_name_or_path t5-3b --output_dir output_dir --adam_eps 1e-06 --do_train --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 500 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS  --predict_with_generate --sortish_sampler --source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" --source_prefix "translate English to Romanian: " --val_max_target_length 128 --warmup_steps 50 --max_train_samples 10 --deepspeed tests/deepspeed/ds_config_zero3.json  --bf16 --skip_memory_metrics 0

now change to --num_gpus=8
gpu memory usage stats via --skip_memory_metrics 0
***** train metrics *****
  before_init_mem_gpu        =     5438MB
  train_mem_gpu_alloc_delta  =    38092MB
  train_mem_gpu_peaked_delta =     5736MB

8 gpus (per each gpu):
  before_init_mem_gpu        =      734MB
  train_mem_gpu_alloc_delta  =     4723MB
  train_mem_gpu_peaked_delta =     2213MB

you can clearly see that when 8 gpus are used ~1/8 of memory is used for the first two stats:
note that I have edited the ds config file to not offload to cpu, that is:
[...]
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "none",
            "pin_memory": true
        },
        "offload_param": {
            "device": "none",
            "pin_memory": true
        },
[...]

I'm using a 8x A100 80GB node - you might need to use t5-large or t5-base if your gpu memory is smaller.
So now you should be able to repeat this for your use case. Once proven that the code works (or leaks) without offload only then try with offload.
Hope this helps.

Thank you for your kind reply @stas00 .
First of all, I apologize for my English and my lack of explanation of the problem. I should have provided the information more clearly.
I didnt mean that using more GPUs does not bring memory improvement.
I have already verified that using zero 3 gives significant memory savings and when enabling CPU offload it is very memory saving.
My question was: "Why does initializing a model with Deep Speed (acceleration) require about 1-2 GB of GPU memory for a large GPT-2 model, even with CPU offload enabled?".
And I finally realized that this is not due to memory leaks in Deep Speed or Acceleration.
This is due to this behavior of GPT-2 class.
Whenever i intialized GPT-2 model, it creates upper triangle mask with full precision for all layers which consumes (4*((seq_len**2)+1))*num_layers / (1024**2) MB
e.g.) if seq_len=8192 and num_layers=36,  this would take 9216 MB for each device, even with CPU offload.
(torch.cuda.max_allocated() should return 0MB if cpu offload is activated when initalizing model)
You know, if i partition the model with deepspeed, paramters should be partitioned to cpu or each devices (according to the deepspeed configuration) and all param.data should be released by allocating empty tensor (this line).
But attention masks are not released in this way.
That's why I asked you "are there any memory leaks" and "where is this GPU memory going".
For now, I've decided to manage this attention mask globally, or otherwise, because I've hacked the attention passing method with xformers.
Thank you so much for your response.
It helps a lot :)

I'm glad to hear you have tracked down the source of gpu memory consumption and that it's not the framework, @SeunghyunSEO!
Thank you for the detailed notes on what you have discovered.


Engine Fees

Fee Structure
Each currency info object has default fee values to use. Edge has their own server that returns some standard fee values for currencies that overwrite the defaults. Vendors, such as Earn.com and Mempool.space, are used to get more accurate up-to-date fee values on an interval that overwrites Edge's and the default values.
These values are fetched and then cached to disk. On the next load, the engine fetches the cached values to then overwrite the currency defaults before it attempts to fetch fee values from Edge or any vendors.
Calculate Rate
The previously implemented function, calculateFeeRate, has been replaced with the most recent implementation from edge-bitcoin-plugin.

build on top of changes made during PR review


Demo site is down

demo.modernpaste.com reports "service unavailable".

My cloud VM has been constantly dying recently. Thanks for the report, I'm investigating.

Should be back up now.


Add ISBN Verifier exercise

Add ISBN Verifier exercise in Swift

@masters3d

@masters3d Any idea why the Linux Swift 4 build fails?

I guess we need to update the docker image:
https://hub.docker.com/r/norionomura/swift/tags/

We should probably use official images
https://hub.docker.com/r/library/swift/tags/
4.0.2 looks good.


Invoke close event when closing filter (flush buffer)

Closes #4

The failing test case depends on #10. We'll have to make sure to cleanly abort stream filtering for all edge cases.

The failing test case depends on #10. We'll have to make sure to cleanly abort stream filtering for all edge cases.

All done now. Merged in the relevant changes and added some additional tests.


Improve edge cases handling for Vec (de)serialize

Signed-off-by: Luminita Voicu<EMAIL_ADDRESS>Reason for This PR
Multiplication for computing Vec's size in bytes fails with overflow error.
Description of Changes
Added a check for overflow during multiplication for both serialize and deserialize functions.
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license.
PR Checklist
[Author TODO: Meet these criteria.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

[ ] All commits in this PR are signed (git commit -s).
[ ] The reason for this PR is clearly provided (issue no. or explanation).
[ ] The description of changes is clear and encompassing.
[ ] Any required documentation changes (code and docs) are included in this PR.
[ ] Any newly added unsafe code is properly documented.
[ ] Any user-facing changes are mentioned in CHANGELOG.md.

You will need to prepare another PR to bump de crate version and cargo publish after merge.


fix: try fix publish bb

Benchmark results
Metrics with a significant change:

node_history_sync_time_in_ms (5): 29,061 (+16%)
node_history_sync_time_in_ms (10): 56,691 (+16%)
note_history_trial_decrypting_time_in_ms (10): 210 (+45%)
note_successful_decrypting_time_in_ms (8): 281 (-18%)
note_successful_decrypting_time_in_ms (32): 809 (-23%)
note_successful_decrypting_time_in_ms (128): 2,908 (-25%)
note_trial_decrypting_time_in_ms (8): 82.0 (+86%)
note_trial_decrypting_time_in_ms (32): 31.0 (-74%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit 388a47ba and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
128 txs

l1_rollup_calldata_size_in_bytes
45,444
179,588
716,132

l1_rollup_calldata_gas
222,924
868,172
3,449,216

l1_rollup_execution_gas
842,011
3,595,280
22,204,585

l2_block_processing_time_in_ms
2,343 (+4%)
8,720 (+1%)
34,205

note_successful_decrypting_time_in_ms
:warning: 281 (-18%)
:warning: 809 (-23%)
:warning: 2,908 (-25%)

note_trial_decrypting_time_in_ms
:warning: 82.0 (+86%)
:warning: 31.0 (-74%)
137 (+1%)

l2_block_building_time_in_ms
14,595
57,823
230,597

l2_block_rollup_simulation_time_in_ms
13,281
52,631
209,655

l2_block_public_tx_process_time_in_ms
1,277 (-5%)
5,063 (-4%)
20,266 (-4%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
:warning: 29,061 (+16%)
:warning: 56,691 (+16%)

note_history_successful_decrypting_time_in_ms
2,444 (-4%)
4,790 (-4%)

note_history_trial_decrypting_time_in_ms
127 (+5%)
:warning: 210 (+45%)

node_database_size_in_bytes
1,631,774
1,098,783

pxe_database_size_in_bytes
27,188
54,187

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
89.2 (+8%)
61,697
18,841

private-kernel-ordering
55.0 (+9%)
24,233
8,089

base-rollup
2,245 (+7%)
656,309
811

root-rollup
97.5 (+9%)
4,072
1,097

private-kernel-inner
64.5 (+10%)
81,504
18,841

public-kernel-private-input
50.5 (+9%)
41,455
18,841

public-kernel-non-first-iteration
31.4 (+10%)
41,497
18,841

merge-rollup
0.913 (+12%)
2,592
873

Miscellaneous
Transaction sizes based on how many contracts are deployed in the tx.

Metric
0 deployed contracts
1 deployed contracts

tx_size_in_bytes
8,723
27,240 (-7%)


add skin rash as a new term?

Consider adding:
skin rash (a child of abnormal skin condition MP:0001191 ?)
DEF: an eruption in the skin which affects its appearance and/or texture; a rash may be localized to one part of the body, or affect all the skin; rashes may cause the skin to change color, itch, become warm, bumpy, dry, cracked or blistered, swell and may be painful
SYN : cutaneous eruption
SYN: skin eruption
SOURCE: NCIT:C39594
Annotations in J:166286, J:181261
NOTE: several types of skin rashes seen in COVID-19 patients, https://www.minnpost.com/second-opinion/2020/05/several-types-of-skin-rashes-seen-in-covid-19-patients/

Added:
skin rash MP:0031103


ADR to record publishing times and change note decisions

Trello: https://trello.com/c/GrVzLvnd/1468-document-the-change-note-decisions-in-content-publisher-adr-and-publishing-api-documentation

@benthorner this should be ready for another look.

Thanks @benthorner


New Release Todos

The Gu project has become somewhat bloated and needs major refactorings and code streamlining, hence I came up with the below list to detail the work that needs to be done as todos:
Todos
These are the following things I need to do:

Remove all uses of reflect
Add codegen to generate notification type and components code
Clean markup code and tree code to ensure correctnes
Attempt to debug and find why gopherjs takes time to load code.
Rewrite css, html and file parsers into a single structure
Provide css parsing with new parser to create css ruleset from all #includes
Provide html parser that turns html into static Markup structures in value form
Remove most file generators and use new generator system
Only provide component package without care for component directory
Remove component base folder, not needed anymore
create .theme file to contain tml details for themeing
Remove need of AppConfig from app file
Change trees from returning pointers to returning values but keep pointer methods
Refactor event internal removal system to be true with new changes.

Issues

Currently it takes some time for the dom from Gu app to load so we need to find a way to speed this up as well

Notification code gen has been added in commit 21fd384a328fb08efb419a370c203ef159706f5f and 7ab39bc2aedd30e676ee0632bf4a516804f2da5c.
See:

Custom Event Annotation (https://github.com/gu-io/gu/blob/master/notifications/notifiers.go#L6)
Custom Event Handler and Publisher Code generate(https://github.com/gu-io/gu/blob/master/notifications/appevent_event.go)


Question: Multi-table inheritance and different transitions

I have a model with multi-table-inheritance. I'd like to put the state field on the shared model and have some "standard" transitions on there. On the subclasses of that model, there may be additional transitions that are specific to that subclass. And subclasses may do slightly different things for transitions with the same name.
Problem is that if I use a shared state field, all the transitions of all the subclasses end up being available to all instances.
My current workaround is to put the state field on the subclass and re-define all the common transitions.
Is there a better way to handle this?

First-class inheritance support was added to viewflow.fsm  https://docs.viewflow.io/fsm/inheritance.html


GetLocalPlayer

Thank you for your release here!
Would it be possible to request further investigation on the clientstate?
Today I found at 0xF0 the PlayerSlot on Server
dwNetworkGameClient = 0x48BAB0;
int PlayerSlot = dwNetworkGameClient + 0xF0
but I am still looking for GetLocalPlayer if you happend to find it could you please share? :-) Thank you

I know there is LocalPlayerController and dwLocalPlayer but I'm looking for the ClientState Entity one

I know there is LocalPlayerController and dwLocalPlayer but I'm looking for the ClientState Entity one

Where did you find dwLocalPlayer? name file?

Added in 96a378be0b59b4bd54ab4ac034548d1aaa01f2fb