Handling 'Too Many Requests' During Link Extraction

While extracting multiple links, I encountered a situation where some of them returned a "Too Many Requests" message, but the status code was still 200.

To address this issue, how can I prevent hitting the 'Too Many Requests' error?
Additionally, should the status code change to 429 (or another appropriate error code) when some issue occurs.

Thanks!

Hi, thank you for using the library. For the benefit of others who may have similar questions, I'll provide a detailed answer that can be used for future reference.

Use Delay Between Requests:

from crawl4ai import AsyncWebCrawler
import asyncio

async def main():
    async with AsyncWebCrawler(verbose=True) as crawler:
        urls = ["https://example1.com", "https://example2.com"]

        # Method 1: Built-in delay
        results = await crawler.arun_many(
            urls,
            delay_between_requests=2.0,  # Add 2 second delay between requests
            **kwargs
        )

        # Method 2: Custom throttling
        semaphore = asyncio.Semaphore(3)  # Limit to 3 concurrent requests
        async def crawl_with_throttle(url):
            async with semaphore:
                result = await crawler.arun(url)
                await asyncio.sleep(1)  # Add delay after each request
                return result

        tasks = [crawl_with_throttle(url) for url in urls]
        results = await asyncio.gather(*tasks)

if __name__ == "__main__":
    asyncio.run(main())

Add Retry Logic with Exponential Backoff:

from crawl4ai import AsyncWebCrawler
import asyncio
import random

async def crawl_with_retry(crawler, url, max_retries=3):
    for attempt in range(max_retries):
        try:
            result = await crawler.arun(url)

            # Check content for rate limit messages
            if "too many requests" in result.markdown.lower():
                delay = (2 ** attempt) + random.uniform(0, 1)  # Exponential backoff
                print(f"Rate limited, waiting {delay:.2f}s before retry")
                await asyncio.sleep(delay)
                continue

            return result

        except Exception as e:
            if attempt == max_retries - 1:
                raise
            delay = (2 ** attempt) + random.uniform(0, 1)
            await asyncio.sleep(delay)

    return None

async def main():
    async with AsyncWebCrawler(verbose=True) as crawler:
        urls = ["https://example1.com", "https://example2.com"]
        results = await asyncio.gather(*[
            crawl_with_retry(crawler, url) for url in urls
        ])

Use Proxy Rotation:

from crawl4ai import AsyncWebCrawler
import random

PROXY_LIST = [
    "http://proxy1:8080",
    "http://proxy2:8080",
    "http://proxy3:8080"
]

async def main():
    async with AsyncWebCrawler(
        verbose=True,
        proxy=random.choice(PROXY_LIST)  # Rotate proxies
    ) as crawler:
        results = await crawler.arun_many(urls)

Regarding the status code issue:

Many sites return 200 with rate limit messages in content instead of 429
You can modify the crawler to check content for rate limit indicators:

I have two solutions for you. The success flag indicates true crawl success beyond HTTP status codes. It will be False if there are JavaScript errors, empty content, error messages, anti-bot notices, or unexpected page structures - even with a 200 response. Always check both success and error_message for accurate crawl validation.
# Enhanced rate limit and success checking
async def check_crawl_success(result):
    # Check both rate limits and general success
    is_rate_limited = await check_rate_limit(result)
    is_successful = result.success and result.markdown.strip() != ""

    if not is_successful:
        if is_rate_limited:
            return False, "Rate limited"
        return False, "Crawl failed"
    return True, "Success"

async def main():
    async with AsyncWebCrawler(verbose=True) as crawler:
        results = await crawler.arun_many(urls)

        # Filter successful results
        successful_results = []
        for result in results:
            success, message = await check_crawl_success(result)
            if success:
                successful_results.append(result)
            else:
                print(f"Failed to crawl {result.url}: {message}")

        # Continue processing only successful results
        for result in successful_results:
            # Process your successful crawls
            pass

You can also manually check for special messages in the content to detect rate limits or similar issues:
async def check_rate_limit(result):
    rate_limit_indicators = [
        "too many requests",
        "rate limit exceeded",
        "please try again later",
        "access temporarily limited"
    ]

    if any(indicator in result.markdown.lower() for indicator in rate_limit_indicators):
        return True
    return False

async def main():
    async with AsyncWebCrawler(verbose=True) as crawler:
        result = await crawler.arun(url)

        if await check_rate_limit(result):
            print("Rate limited despite 200 status code")
            # Handle accordingly

Upcoming Scraper Module:
I'm currently testing a new Scraper module for Crawl4AI. It uses graph search algorithms to intelligently crawl websites, handling everything from simple blog posts to complex nested pages. Whether you need to crawl an entire documentation site or extract data from multiple product pages, the Scraper will handle all the heavy lifting - navigation, content extraction, and optimization. Stay tuned for its release! In the meantime, the rate limiting solutions above should help with your current crawling needs.
Stay tuned for the release! In the meantime, the solutions above will help manage rate limiting and ensure reliable crawling results.

Thanks for detailed explanation I will try these!

You're welcome.

Hey! Do you have any reccomendation for a free proxy? Or is it only possible with a paid solution?
Now even one request, I get the 429 code for a certain website.

@tiago-falves You can use Crawl4AI with either paid or free proxies, but keep in mind that a 429 error is mostly about hitting the rate limit. One solution is to pair Crawl4AI with Lambda functions on the cloud. This lets you run multiple instances simultaneously with different IP addresses to gather data efficiently.
We don’t have an example for this yet, but I plan to create a tutorial and make it available soon. This approach is more suited for production-level use cases like this.


Add support for numeric metrics

@optimizely/fullstack-devs

@vraja2 will add docs to explain why they should send us floats or doubles instead of int/long


[Bug]: BCP cannot import table with time(7) column

What happened?
Using the latest bcp utility:
bcp -v

BCP - Bulk Copy Program for Microsoft SQL Server.
Copyright (C) Microsoft Corporation. All Rights Reserved.
Version: 16.0.1000.6

And the table in MSSQL like this:
create table tab1(
    col1 time(7)
)

insert into tab1 values ('00:00:00.1234567')

This table is exported from MSSQL using the native binary format:
bcp tab1 out tab1.bcp -n -S <IP_ADDRESS>,1433 -U sa -P pwd

Starting copy...

1 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 1      Average : (1000.00 rows per sec.)

The same table is created in Babelfish, the type modifier for col1 is saved in a catalog as "unspecified":
select atttypmod from pg_attribute where attname = 'col1'

atttypmod
---------------
-1

And sp_describe_first_result_set returns type modifier 6 for this column:
exec sp_describe_first_result_set 'select * from tab1'

scale
-------
6

When we try to import into Babelfish that tab1.bcp file exported from MSSQL, the following error occurs:
bcp tab1 in tab1.bcp -n -S <IP_ADDRESS>,1433 -U jdbc_user -P 12345678

Starting copy...
SQLState = 22008, NativeError = 0
Error = [Microsoft][ODBC Driver 17 for SQL Server]Datetime field overflow. Fractional second precision exceeds the scale specified in the parameter binding.

0 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 47

The bcp utility uses sp_describe_first_result_set to get column types for the specified table. Based on its return it generates the following bulk copy query:
insert bulk tab1([col1] time(6))

And the error happens on the client side in ODBC driver when it tries to bind the time(7) value from the input file to the time(6) parameter.
It is unclear how to approach this problem on server side (perhaps the unused 7 type modifier can be retained in attoptions catalog field?). And AFAIU there are no workarounds to this on client side (with bcp) for binary files exported from MSSQL which contain time(7) or datetime2(7) columns.
Version
BABEL_4_X_DEV (Default)
Extension
None
Which flavor of Linux are you using when you see the bug?
No response
Relevant log output
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct.

I too experience the same issue in which the datetime2(7) in the source DB get rounded to datetime2(6) in the target DB.

To clarify, the problem here is not that Babelfish accepts datetime2(7) column definition and then processes it as datetime(6). This is a reasonable approach to use Postgres native timestamp data type with max 6 fractional digits for seconds.
The problem is that when the DDL introspection is used on the column created as datetime2(7) , its definition is returned as datetime2(6). This breaks data migration scenarios that rely on bcp (that uses this introspection under the hood).
There is a workaround that seems to be not too disruptive - to return the type modifier 7 instead of 6 in introspection of columns that do not have the exact value set for this modifier (-1 is currently set for 7 input):
diff --git a/contrib/babelfishpg_tsql/sql/sys_views.sql b/contrib/babelfishpg_tsql/sql/sys_views.sql
index 796ecd34..f917200b 100644
--- a/contrib/babelfishpg_tsql/sql/sys_views.sql
+++ b/contrib/babelfishpg_tsql/sql/sys_views.sql
@@ -149,13 +149,13 @@ BEGIN
                WHEN 'date' THEN scale = 0;
                WHEN 'datetime' THEN scale = 3;
                WHEN 'smalldatetime' THEN scale = 0;
-               WHEN 'datetime2' THEN scale = 6;
-               WHEN 'datetimeoffset' THEN scale = 6;
+               WHEN 'datetime2' THEN scale = 7;
+               WHEN 'datetimeoffset' THEN scale = 7;
                WHEN 'decimal' THEN scale = 38;
                WHEN 'numeric' THEN scale = 38;
                WHEN 'money' THEN scale = 4;
                WHEN 'smallmoney' THEN scale = 4;
-               WHEN 'time' THEN scale = 6;
+               WHEN 'time' THEN scale = 7;
                WHEN 'tinyint' THEN scale = 0;
                ELSE
                        IF return_null_for_rest

This can be applied manually to existing DB cluster by recreating sys.tsql_type_scale_helper function replacing 6 return values with 7.


Pass state accessses / simPayload to _postCheck

These were removed as parameters in #97. However, after looking at the changes in ethereum-optimism/superchain-ops#356, it seems simpler if they are just passed in.

🟡 Heimdall Review Status

Requirement
Status
More Info

Reviews

🟡
0/1

Denominator calculation

Show calculation

1 if user is bot
0

1 if user is external
0

From .codeflow.yml

1

Additional review requirements

Show calculation

Max
0

0

From CODEOWNERS

0

Global minimum
0

Max

1

1

1 if commit is unverified
0

Sum
1


Refactor the existing cypress test to upload cover image

Description
Enhance the existing test suite to include functionality for uploading, editing, and deleting the facility cover image.
General Instruction

Code should follow the POM structure.
Reuse existing custom functions and those in commands.ts where possible. Before creating a new function, check for an existing one that matches the element's ID to avoid redundancy.
Add API request verification wherever required using cy.intercept();
Add the tests in the facilityManage.cy.ts file and any new page objects in the facilityManage.ts file.

Test Workflows
Test 1: District Admin

Log in as a district admin and navigate to any facility.
Open the facility cover image pop-up, then click the Cancel button to verify it functions correctly.
Reopen the cover image pop-up and upload an image, verifying the API response.
Edit the current cover image by uploading a new one, and verify the API response.
Delete the existing cover image, confirming the API response.

Test 2: Access Verification with Non-Admin User

Log in as the devdoctor user.
Repeat the steps from Test 1 to verify functionality consistency across user roles.

Screenshots

@nihal sir , can I work on this one also ?

@Jacobjeevan Can you assign this to me

@nihal467 sir, as in the issue description , to verify functionality consistency across user roles , can i define tests as such for multiple roles:

or create a custom function in loginpage which takes user role and logs a user

Hey @rohitlingarker are you still working on this issue?

Yes Iam @JavidSumra

Hey @nihal467 can you please assign this issue to me?

@nihal467 can you please assign this issue to me?
I am really interested to work on this issue

Hey @nihal467 shall i need to create a new test file under facility_spec?
ref https://github.com/ohcnetwork/care_fe/pull/9134#issuecomment-2488733438

@rithviknishad Made a PR for this issue can you please assign it back to me?


Implement switch to keep extractNativeLibs manifest option

Introduce the CLI option to preserve  android:extractNativeLibraries to allow re-packing apps that initially ship with this setting (and some code relies on it!)
To repack such an application with apktool successfully, one needs to:

put shared library extension (so) to doNotCompress section of apktool.yml in the decompiled app directory:
doNotCompress:
- so

build APK back using apktool b

zipalign using the zipalign tool from latest Android SDK:
zipalign -p -f 4 app-rebuilt.apk

sign APK by apksigner or uber-apk-signer skipping its own zipalign step:
java -jar uber-apk-signer --skipZipAlign

Sorry not convinced the purpose of keeping `extractNativeLibs'. AFIK extractNativeLibs is for when the library is archived in compressed format and needs to tell lib loader to extract first before loading.
During merging all native libs kept uncompressed.
I appreciate if you drop here sample apk that proves your point.
Thank you


Add 'notefinder' library function

This library will return start/end notes for use with the smartshape library (or anything else that might find it useful!
Currently there is a single function for returning a table of the starting/ending notes for each staff in a given selection.

I think it is very risky to keep entries from an eachentry loop in a table outside that loop. Depending on when Lua garbage collection runs, I believe you could end up with entries pointing to deleted data. Finale will then crash if you try to access it. This is worth running a test to see if you can break it, but I'm fairly certain it is not safe.
Instead of a Lua table, this function might return an FCNoteEntryLayers instance. This allows the storing of entries from multiple measures. The way tie.lua does this is to return the FCNoteEntryLayer instance along with the 1st and last entry taken from that instance. (But read the caveats!) Because you potentially want multiple layers you'll need to return FCNoteEntryLayers.

As a follow-on, I was not mistaken. This script crashes Finale, if you select at least three measure containing entries. (Maybe even just with 2, but 3 guarantees that Lua garbage collection has run on the first entry returned by eachentry.)

local extern
for entry in eachentry(finenv.Region()) do
    if not extern then
        extern = entry -- save the 1st entry we encounter
    end
end

-- this call crashes Finale because the entry has gone
-- out of scope in C++
local lownote = extern:CalcLowestNote(nil)

I strongly urge you to go look at the eachentry function and internalize how it works. You can find it here.

Thanks, Robert… I’ll figure out another approach!

On Aug 6, 2022, at 11:33 AM, rpatters1 @.***> wrote:
As a follow-on, I was not mistaken. This script crashes Finale, if you select at least three measure containing entries. (Maybe even just with 2, but 3 guarantees that Lua garbage collection has run on the first entry returned by eachentry.)
local extern
for entry in eachentry(finenv.Region()) do
if not extern then
extern = entry -- save the 1st entry we encounter
end
end
-- this call crashes Finale because the entry has gone
-- out of scope in C++
local lownote = extern:CalcLowestNote(nil)
I strongly urge you to go look at the eachentry function and internalize how it works. You can find it here https://github.com/finale-lua/lua-source/blob/master/built-in-functions/rgplua_built_in_functions.lua.
—
Reply to this email directly, view it on GitHub https://github.com/finale-lua/lua-scripts/pull/324#issuecomment-1207261172, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMCBUQ6P7GGZK76PURFLZK3VX2VYLANCNFSM55ZBMFDQ.
You are receiving this because you authored the thread.

Out of curiosity I just ran the script I have that uses the function I wrote (where entries are stored in a table) on a 6+ measure passage… and it did not crash. Why do you suppose that is, if using tables for noteentry storage is an issue?

On Aug 6, 2022, at 11:15 AM, rpatters1 @.***> wrote:
I think it is very risky to keep entries from an eachentry loop in a table outside that loop. Depending on when Lua garbage collection runs, I believe you could end up with entries pointing to deleted data. Finale will then crash if you try to access it. This is worth running a test to see if you can break it, but I'm fairly certain it is not safe.
Instead of a Lua table, this function might return an FCNoteEntryLayers instance. This allows the storing of entries from multiple measures. The way tie.lua does this is to return the FCNoteEntryLayer instance along with the 1st and last entry taken from that instance. (But read the caveats!) Because you potentially want multiple layers you'll need to return FCNoteEntryLayers.
—
Reply to this email directly, view it on GitHub https://github.com/finale-lua/lua-scripts/pull/324#issuecomment-1207258413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMCBUQYRE56TK3H4Z3XB5TTVX2TTVANCNFSM55ZBMFDQ.
You are receiving this because you authored the thread.

Predicting Lua gc is difficult. You could run it successfully 100 times (or a 1000) then one day it could start crashing Finale every time. Perhaps after one small change.
Also, it matters what you do with the collection after you've made it.

I'm going back to the drawing board on this one... I'm thinking maybe a library function is not the right solution for what I want to do if I can't pass notes around in tables.

You can pass them around in tables as long as you maintain a reference to their container. I recommended FCNoteEntryLayers, and I still think it would. You could take one of those as input and then return the start and end note. The problem arises when you try to take entries from the eachentry loop.

How would I got about maintaining a reference to the container? I'm not just interested in the start and end notes, that is just a jumping off point... Is there a good way to iterate through notes in FCNoteEntryLayers comparable to eachentry?
I read through the available functions and parameters for FCNoteEntryLayers and it didn't seem like the right tool for the job, but I'll go look at it again.
On 8/6/2022 4:06:15 PM, rpatters1 @.> wrote:
You can pass them around in tables as long as you maintain a reference to their container. I recommended FCNoteEntryLayers, and I still think it would. You could take one of those as input and then return the start and end note. The problem arises when you try to take entries from the eachentry loop.
—
Reply to this email directly, view it on GitHub [https://github.com/finale-lua/lua-scripts/pull/324#issuecomment-1207295567], or unsubscribe [https://github.com/notifications/unsubscribe-auth/AMCBUQ4TDWNBMJETTY2C3GDVX3VWLANCNFSM55ZBMFDQ].
You are receiving this because you modified the open/close state.Message ID: @.>

You maintain the reference by having a variable that's in scope, either a local variable in your code block or a global variable. You don't let it go out of scope until you are finished with it. Pretty much like any other Lua variable you ever use.
The problem with eachentry is that it is creating variables under the covers that are hidden from you and feeding entries from them into your loop body, but then they go out of scope when the loop exits. It's really only good for procession entries one at a time.
One way to iterate FCNoteEntryLayers is
for note_entry_layer in each(note_entry_layers) do
   for entry in each(note_entry_layer) do
      ...
   end
end

If you only want entries inside a region:
for note_entry_layer in each(note_entry_layers) do
   for index = 0, note_entry_layer.Count-1 do
      local entry = note_entry_layer:GetRegionItemAt(index)
      if entry then
         ...
      end
   end
end

Both FCNoteEntryLayers and FCNoteEntryLayer are collections, so they have all the usual collection functions for iterating them.

Don't have Finale open right now, but doesn't each note/entry have a unique ID? If so, you can get those IDs within the loop and store them even when the loop ends. You can then load a specific entry by ID.
I may be totally misremembering these classes, though.

The EntryNumber is unique. Loading each one separately requires a good deal of redundant work, though.


Adding active learning regression implementations based on greedy sampling

Hello!
I would love to contribute to modAL, I feel like there are not many examples of regression problems in the documentation.
I can try to add my touch to modAL/examples by implementing the algorithms presented in this paper:
https://arxiv.org/pdf/1808.04245.pdf as custom query strategies.
Does that look good?

Hello！I am an undergraduate majoring in automation, and recently I want to use greedy sampling method for active learning of cell data. However, after reading the paper, I am a little confused about the specific python structure of GSx and iGS methods‘s custom query strategies. Could you provide python programs for these two methods. I would appreciate it

Hello! I would love to contribute to modAL, I feel like there are not many examples of regression problems in the documentation.
I can try to add my touch to modAL/examples by implementing the algorithms presented in this paper: https://arxiv.org/pdf/1808.04245.pdf as custom query strategies.
Does that look good?
Hello！I am an undergraduate majoring in automation, and recently I want to use greedy sampling method for active learning of cell data. I think the method in the paper will be much more optimized than the original method. However, after reading the paper, I am a little confused about the specific python structure of GSx and iGS methods‘s custom query strategies. Could you provide python programs for these two methods. I would appreciate it.


Hole doesn't work as expected

This code:
import build123d as bd

with bd.BuildPart() as washer:
    bd.Cylinder(3, 2)
    bd.Hole(1)

debug(washer.part)

makes a hole on the bottom of the cylinder, with a depth of half its height:

Hole documentation clearly explains that it should go through: depth (float, optional): hole depth - None implies through part. Defaults to None.
From what I understand, either the Hole or the Cylinder doesn't start from (0,0,0).

See examples/holes.py for multiple tests.


Fix: Vale false positives for google.header

Update vale to latest 3.3.0
Change accept.txt path and update

@dipinknair - have you been able to investigate further?

I haven't been able to identify the issue. I will try again and let you know


Clarify usage of -instances_metadata fixes #259

I signed it!

that's pretty odd that you need to prefix with instance/attributes since the flag is called instances_metadata, but great catch. thank you.


Exchange online powershell - Unable to get refresh token programatically and via a site

Steps to reproduce
With all of the secure application models, we can have this trigger via a URL to grant consent and return a refreshtoken.
However this does not seem possible for exchange online and it is critical that we can collect this.
$token = New-PartnerAccessToken -Module ExchangeOnline
It requests going to a site and entering in the code
Its a bit more hands on than the current methods where we can use a URL to grant access to the application and then return that output to a url.
How can we capture the refreshtoken from the process to use in our application?
https://docs.microsoft.com/en-gb/powershell/partnercenter/multi-factor-auth?view=partnercenterps-3.0
I am hoping there is just something I have missed and there is an easy way to achieve this.
Expected behavior
Have a way for a user to click a link and grant access to the module so we can capture the refresh code.
Actual behavior
Seems this is currently manual and we need to be able to programatically collect this.
Diagnostic logs
With all of the other partner center secure application model, we have a url to click with our application id which a user can grant consent and it returns a refresh token to the website.
How can we achieve the same outcome with exchange online?
It appears that the only way to do it is to have someone manually run the commands in powershell and then provide us the refresh token. Please assist.
Environment
Partner Center powershell 3.0.2

@Shadoxity that particular command is how you generate the initial refresh token. That value should be stored somewhere secure like Azure Key Vault. After you generate that value you will use the following to get an access token using the securely stored refresh token
$customerId = '<CustomerId>'
$refreshToken = '<RefreshTokenValue>'
$upn = '<UPN-used-to-generate-the-refresh-token>'

$token = New-PartnerAccessToken -Module ExchangeOnline -RefreshToken $token.RefreshToken -Tenant $customerId

$tokenValue = ConvertTo-SecureString "Bearer $($token.AccessToken)" -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($upn, $tokenValue)

$session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid?DelegatedOrg=$($customerId)&BasicAuthToOAuthConversion=true" -Credential $credential -Authentication Basic -AllowRedirection

Import-PSSession $session

Hey @isaiahwilliams  That is my point,
The problem is generating the initial refresh token without having someone manually go into powershell and create it and then copy and paste it somewhere.
For the other partnercenter methods we can use the URL for our azure application to grant consent. How can we do a similar thing for exchange? I need to be able to programatically grant the consent and get the refresh token.
Please assist.


Undo redo buttons disabled state

Undo and redo buttons are now disabled if there is no undo or redo action available.

@burhanrashid52 Do you remember about this PR? :)

@burhanrashid52 Do you remember about this PR? :)

Nop. It just got slipped 😆. Added review comments.

@wasky I am planning to create a new 3.1.0 release in with some dependencies bump. If you can address the review comment than I can include this in the upcoming release together.


Consider migrating to Github

In GitLab by @JoerivanEngelen on Jul 7, 2023, 12:13
There are multiple benefits to migrating this repository to Github:

Better sprint boards
Templates for issues
Larger community
More knowledge within Deltares to help each other out with CI stuff
Github actions has more possibilities (I think...)

Downsides:

We have to ask our users to also make a Github account, which can be a hurdle to keep in touch.
Converting CI to github actions requires work
Some things cannot be ideally migrated. For example, I've found a tool to migrate which worked for the QGIS-Tim plugin, but it posted issues under my own account name. Not entirely ideal

Furthermore, there is still a product line move planned towards a Deltares Github enterprise, so it is probably good to gain some extra info about this first. We could migrate to Github.com first, and then to the Enterprise, but that might obfuscate things further for our users.

In GitLab by @JoerivanEngelen on Oct 31, 2023, 14:08
Deltares has decided that all products should migrate to the Deltares Github organization. This also avoids the issue that we run into Gitlab CI's limits and constantly have to prolong the Ultimate program.
Further work on this will be continued in #575

In GitLab by @JoerivanEngelen on Oct 31, 2023, 14:08
marked this issue as related to #575


natscli: 0.0.26 -> 0.0.28

Motivation for this change
Update to latest upstream release0.0.28
Change log:

https://github.com/nats-io/natscli/releases/tag/v0.0.28
https://github.com/nats-io/natscli/releases/tag/v0.0.27

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 148405 run on x86_64-linux 1

1 package built:

natscli


[This will not be merged] Testing a change to linter error reporting on PRs

This is for testing only and will not be merged.

This PR was for testing purposes only and was never meant to be checked.


Report all unresolved variables

Ssort currently stops after reporting the first unresolved variable.  Fixing these one at a time and waiting for ssort to rerun is tedious.  Ssort should report all unresolved variables before continuing to the next file.

Done


Add CODEOWNERS file to manage commit rights

Add a GitHub CODEOWNERS file to support disaggregating commit rights.
Allows the Committer process in the Community Membership docs to be
implemented.
Signed-off-by: Daniel Farrell<EMAIL_ADDRESS>

ping @tpantelis

Need one more +1 please. :pray: :smile:


Closure Compiled version of the ol package is broken

The basic map example no longer works when run through the Closure Compiler.  Running git bisect now to figure out when this stopped working.

9271df7e1d1dce81f08bac4e1193dbf09dc2dfcd is the first bad commit
So this is a duplicate of #7332.
Things that I thought about while git bisect ran:

we really need to have package-lock.json in the repo, npm install takes too long without, greenkeeper-lockfile will assist with Greenkeeper PRs
transforming the our source files into modules is tedious, it will be nice to have modules as source soon
it would be nice to release more often, #7333 was merged a month ago with the fix for this issue, reworking how the website is built, having modules as our sources, getting rid of closure-util as a dependency, and other simplifications will make publishing releases less painful
the Compiler takes a long time to run


bpf: Remove flowlabel optimization for identity

For IPv6 traffic, we use the flowlabel field to transfer the source security identity across nodes and avoid an ipcache lookup. This misuse of the flowlabel field can however conflict with legitimate uses such as https://lpc.events/event/11/contributions/955/. This commit therefore removes this optimization.

/test

/test-1.25-4.19

/test-runtime


Feature/backend seeds models routes

Fixed nav for about and contact links.  Both are working now.  Added basic code for pages so that it could be styled and worked on.

Force merging as time is short and pulling an all-nighter.  No one else is available to review.  No code breaks. Tested and good to go.


Need Help with a Softmax Warning in TensorFlow 2.16

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
source
TensorFlow version
2.16.1
Custom code
Yes
OS platform and distribution
Windows 11
Mobile device
No response
Python version
3.12.3
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
Hey everyone,
I'm running into a bit of a headache with TensorFlow 2.16 and could really use some help. I'm getting this annoying warning about a Softmax operation over an axis with size 1. This pops up when I'm using a custom TransformerBlock layer that includes MultiHeadAttention.
What I've Tried:
Debugging Dimensions:
Added print statements to check tensor shapes at different stages.
Used tf.squeeze to remove dimensions of size 1 before passing the tensor to MultiHeadAttention.
What I Need:
Is this a bug in TensorFlow 2.16? If yes, any workarounds or patches?
Best practices for handling tensor dimensions in MultiHeadAttention to avoid this?
Should I downgrade or wait for an update? If yes, which version should I try?
Additional Info:
Using LSTM and GRU layers followed by the custom TransformerBlock.
Running on Windows with Python 3.12.
Any help or pointers would be greatly appreciated! Thanks!
Standalone code to reproduce the issue
class TransformerBlock(tf.keras.layers.Layer):
    def __init__(self, t_num_heads, t_key_dim, t_ff_dim, dropout_rate=0.1, activation_function='relu',
                 initializer='glorot_uniform', **kwargs):
        super(TransformerBlock, self).__init__(**kwargs)
        self.att = MultiHeadAttention(num_heads=t_num_heads, key_dim=t_key_dim)
        self.ffn = tf.keras.Sequential([
            Dense(t_ff_dim, activation=activation_function, kernel_initializer=initializer),
            Dense(t_key_dim, kernel_initializer=initializer),
        ])
        self.layernorm1 = LayerNormalization(epsilon=1e-6)
        self.layernorm2 = LayerNormalization(epsilon=1e-6)
        self.dropout1 = Dropout(dropout_rate)
        self.dropout2 = Dropout(dropout_rate)
        self.dense_proj = Dense(t_key_dim, kernel_initializer=initializer)

    def call(self, inputs, training=None, *args, **kwargs):
        inputs_proj = self.dense_proj(inputs)
        print(f"inputs_proj shape: {inputs_proj.shape}")

        if len(inputs_proj.shape) == 4 and inputs_proj.shape[2] == 1:
            inputs_proj = tf.squeeze(inputs_proj, axis=2)
            print(f"inputs_proj after squeeze shape: {inputs_proj.shape}")

        attn_output = self.att(inputs_proj, inputs_proj)
        print(f"attn_output shape: {attn_output.shape}")
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(inputs_proj + attn_output)
        ffn_output = self.ffn(out1)
        print(f"ffn_output shape: {ffn_output.shape}")
        ffn_output = self.dropout2(ffn_output, training=training)
        return self.layernorm2(out1 + ffn_output)

    def compute_output_shape(self, input_shape):
        return input_shape

    def get_config(self):
        config = super(TransformerBlock, self).get_config()
        config.update({
            't_num_heads': self.att.num_heads,
            't_key_dim': self.att.key_dim,
            't_ff_dim': self.ffn.layers[0].units,
            'dropout_rate': self.dropout1.rate,
            'activation_function': self.ffn.layers[0].activation.__name__,
            'initializer': self.ffn.layers[0].kernel_initializer.__class__.__name__
        })
        return config

    @classmethod
    def from_config(cls, config):
        return cls(**config)

Relevant log output
UserWarning: You are using a softmax over axis 3 of a tensor of shape (None, 4, 1, 1). This axis has size 1. The softmax operation will always return the value 1, which is likely not what you intended. Did you mean to use a sigmoid instead?

I am experiencing the same issue when implementing my own transformer encoder decoder. So far, i am still missing positional encoding and some masking layers, i don't know whether those would affect it in any way.
Here is my code, showing the same warning with tf-nightly, and python 3.11.9
`import keras
import numpy as np
def possitionalEmbedding(input_dim, output_dim):  # TODO
return keras.layers.Embedding(input_dim=input_dim, output_dim=output_dim)
def model_func(encoder_vocab_len, decoder_vocab_len, encoder_maxlen, decoder_maxlen, params):
num_heads, key_dim, d_v, d_ff, d_model, n = params
encoder_input = keras.Input(shape=(None,))
decoder_input = keras.Input(shape=(None,))

# encoder part
embedded = possitionalEmbedding(encoder_vocab_len, d_model)(encoder_input) # todo possitional embedding
embedded = keras.layers.Dropout(0.1)(embedded)

encoded = embedded
for i in range(n):
    attended_encoded = keras.layers.MultiHeadAttention(num_heads,
                                    key_dim,
                                    dropout=0.1,
                                    use_bias=True,
                                    output_shape=(d_model,))(encoded, encoded, encoded)  # todo padding_mask
    attended_encoded_d = keras.layers.Dropout(0.1)(attended_encoded)
    add = encoded + attended_encoded_d
    normalised = keras.layers.LayerNormalization()(add)
    fed_f = keras.layers.Dense(d_ff)(normalised)  # feed forward 1 part
    fed_ff = keras.layers.Dense(d_model)(keras.activations.relu(fed_f))  # feed forward 2 part
    fed_ff_d = keras.layers.Dropout(0.1)(fed_ff)

    add2 = normalised + fed_ff_d
    normalised2 = keras.layers.LayerNormalization()(add2)

    encoded = normalised2  # and the loop is repeated

encoder_output = encoded  # output from encoder

# decoder part
de_embed = possitionalEmbedding(decoder_vocab_len, d_model)(decoder_input)
de_embed = keras.layers.Dropout(0.1)(de_embed)

for i in range(n):
    self_attention = (keras.layers.MultiHeadAttention(num_heads,
                                    key_dim,
                                    dropout=0.1,
                                    use_bias=True,
                                    output_shape=(d_model,))
                        (de_embed, de_embed, de_embed))
    self_attention_d = keras.layers.Dropout(0.1)(self_attention)
    add = de_embed + self_attention_d
    normalised1 = keras.layers.LayerNormalization()(add)
    cross_attention = (keras.layers.MultiHeadAttention(num_heads,
                                    key_dim,
                                    dropout=0.1,
                                    use_bias=True,
                                    output_shape=(d_model,))
                       (normalised1, encoder_output,encoder_output))
    cross_attention_d = keras.layers.Dropout(0.1)(cross_attention)

    add2 = normalised1 + cross_attention_d
    normalised2 = keras.layers.LayerNormalization()(add2)

    fed_f = keras.layers.Dense(d_ff)(normalised2)  # feed forward 1 part
    fed_ff = keras.layers.Dense(d_model)(keras.activations.relu(fed_f))  # feed forward 2 part
    fed_ff_d = keras.layers.Dropout(0.1)(fed_ff)

    add3 = normalised2 + fed_ff_d
    normalised3 = keras.layers.LayerNormalization()(add3)

    de_embed = normalised3

decoder_dense_output = keras.layers.Dense(decoder_vocab_len, activation='softmax', name='decoder_output')(de_embed)

return keras.Model(inputs=[encoder_input, decoder_input], outputs=decoder_dense_output)

if name == 'main':
params = (8, 64, 64, 256, 512, 6)
model = model_func(10000, 10000, 100, 100, params)
model.summary()
# Generate random input data with appropriate shapes
encoder_input_data = np.random.randint(0, 10000, (2, 1))  # (batch_size, sequence_length)
decoder_input_data = np.random.randint(0, 10000, (2, 4))  # (batch_size, sequence_length)

# Call the model with the random input data
output = model.call([encoder_input_data, decoder_input_data], training=False)

# Print the shape of the output
print(f'Output shape: {output.shape}')`

UserWarning: You are using a softmax over axis 3 of a tensor of shape (2, 8, 1, 1). This axis has size 1. The softmax operation will always return the value 1, which is likely not what you intended. Did you mean to use a sigmoid instead?

warnings.warn(
UserWarning: You are using a softmax over axis 3 of a tensor of shape (2, 8, 4, 1). This axis has size 1. The softmax operation will always return the value 1, which is likely not what you intended. Did you mean to use a sigmoid instead?
warnings.warn(
Output shape: (2, 4, 10000)

Hi @sp00N221 ,

Can you please check with recent TF compatibility versions? I tried with TF2.16.1 and I cannot reproduce the error.
Please check the screenshot
 here. Thanks!

Hey,
Thank you for taking the time to review my issue. I've had nothing but problems with my task over the past few days. I had a combination of a TransformerBlock and LSTM layers. Coupled with Optuna, it was probably just too many variables and possibilities, causing the model to become unstable. I have now switched to this task:
def objective(trial, features, target):
n_estimators = trial.suggest_int('n_estimators', 50, 300)
max_depth = trial.suggest_int('max_depth', 3, 15)
learning_rate = trial.suggest_float('learning_rate', 0.01, 0.3, log=True)
subsample = trial.suggest_float('subsample', 0.5, 1.0)
colsample_bytree = trial.suggest_float('colsample_bytree', 0.5, 1.0)
gamma = trial.suggest_float('gamma', 0, 5)
min_child_weight = trial.suggest_int('min_child_weight', 1, 10)
reg_lambda = trial.suggest_float('lambda', 1e-8, 10.0, log=True)
reg_alpha = trial.suggest_float('alpha', 1e-8, 10.0, log=True)
model = XGBClassifier(
    n_estimators=n_estimators, max_depth=max_depth, learning_rate=learning_rate,
    subsample=subsample, colsample_bytree=colsample_bytree, gamma=gamma,
    min_child_weight=min_child_weight, reg_lambda=reg_lambda, reg_alpha=reg_alpha,
    random_state=42
)

x_train, x_test, y_train, y_test = train_test_split(features, target, test_size=0.2, random_state=42)

numeric_features = x_train.columns
preprocessor = ColumnTransformer(
    transformers=[
        ('num', Pipeline(steps=[('imputer', SimpleImputer(strategy='mean')), ('scaler', StandardScaler())]),
         numeric_features)
    ])

x_train = preprocessor.fit_transform(x_train)
x_test = preprocessor.transform(x_test)

model.set_params(early_stopping_rounds=10, eval_metric='logloss')
model.fit(x_train, y_train, eval_set=[(x_test, y_test)], verbose=False)

predictions = model.predict(x_test)
accuracy = accuracy_score(y_test, predictions)

return accuracy

With this, I have no problems.
Have a nice day!

Hi @sp00N221 ,

Could you please confirm if this issue is resolved for you? Please feel free to close the issue.

Thank you!


Syntax Widget: Content Overview

Last update: June 21st
We've got the Syntax Lookup Widget skeleton (#114 ) in place now, so we need content writers to cover quick glance overviews over each topic.
This issue is here to collect a list of language constructs we should definitely cover. Feel free to propose more topics if something is missing. I will continuously update this issue for completeness reasons.
How to work on this?

[X] -, -.
[X] /, /.
[X] *, *.
[X] mod
[X]  := (ref value assignment)
[x] :> (type coercion)

Language Constructs:

[X] _ (placeholder, all scenarios like pipe, ignore value)
[X]  type
[X]  let (let binding)
[ ] let \"\" (let binding with invalid identifier name) (Apparently not part of the language, although I strongly disagree not including it, because it's user-facing... see #351)
[X]  if / else
[X]  external
[X]  () => {} (function)
[X]  (.) => {} (curried function)
[X] open (also open!)
[X] include (+ include module type of ...)
[X] module
[X] [ #value ] (polyvar definition)
[X] switch (pattern matching: exceptions, values, variants, when clause..)
[ ] 'a (generic placeholder; +'a for covariance)
[ ] (Item: Comparable) => {} (module functor)
[ ] <MyComponent> (JSX)
[X] "" (string)
[ ] j`` (implicit string interpolation)
[ ] `` (explicit string interpolation)
[x] ' ' (single quote = char)

Type Specifics:

[ ] {.} (empty object type)
[ ] { "key": "value" } (Object type)
[ ] {..} (including type obj<'a> = {.. "a": string } as 'a) aka Open object type
[ ] ref (mutable type)
[ ] exception (exn)
[ ] module type
[ ] module type of (see playground link)

Builtin Functions:

[ ] ignore

Extension Points:

[x] %%raw
[x] %raw
[x] %identity
[x] %debugger
[x] %re

Build System Specials:

[X] __POS_OF__(string)
[X] __LINE__
[X] __LOC__
[X] __FILE__
[X] __LOC_OF__
[X] __MODULE__
[X] __LINE_OF__

Lots of things already shipped thanks to @kevanstannard's help!
We already got some Operators / Decorators documented, so I think it's already useful enough to merge it into master.
In case you want to open a new PR to add more content, please use master as your target branch in the future.

Lots of things already shipped thanks to @kevanstannard's help!
We already got some Operators / Decorators documented, so I think it's already useful enough to merge it into master.
In case you want to open a new PR to add more content, please use master as your target branch in the future.

@ryyppy for the := operator, is there a formal name for it? I have referred to it as the ref assignment operator in a branch, will publish a draft PR shortly

@ryyppy for the := operator, is there a formal name for it? I have referred to it as the ref assignment operator in a branch, will publish a draft PR shortly

@illusionalsagacity something like ref value assignment sounds good to me.

@illusionalsagacity something like ref value assignment sounds good to me.

Short note: I reorganized / refined the item list for language constructs and introduced some Type Specifics category, just to separate the value / type space syntax little bit more

Short note: I reorganized / refined the item list for language constructs and introduced some Type Specifics category, just to separate the value / type space syntax little bit more

I found a decorator not on your list: @deprecated.

@kellydanma sure, go ahead. Just keep an eye out for any existing PRs when submitting new content 👍

I think {..} is missing from the list

Changes that can be applied to the OP:

@@warning is done
@deprecated is done
@send.pipe is actually called @bs.send.pipe in the syntax widget; it is done
open is done
include is done
module is done
#value is done
switch is done
'a is done, but +'a is not
"" is done
'' is done
{..} can be added

Tip: you can use the following GitHub syntax to create a task list with clickable checkboxes:
- [ ] Not done
- [x] Done

updated

%re() (RegExp literal) is missing from the list

I'd like to humbly suggest that such documentation shouldn't use a font with  "programming ligatures". They're something that some people choose to use in their own customised editors, but it's not appropriate for documentation, which should always show the concrete syntax of the language.

Even though it may not be complete, can we make the syntax widget more discoverable with a link from the API docs or the header itself?
I only recently found out this exists (and I've been doing ReScript for a while now) but even the current version can safe me a ton of time (e.g. to remember how to use %re or to find decorators without first finding the decorator overview page). This is an amazing tool for anyone using ReScript and should be front and center.

The list is not up to date anymore.
The following are merged but not checked:

ignore
`` (explicit string interpolation)
'a (but '+a needs a separate entry then)

The following can be added to the widget as well:

async/await
try/catch (maybe as part of exception, but it is a different syntactical element IMO)
{ ?key: value  } optional record fields

Update: I would remove j`` from the list, since it's deprecated


Negative densities / other crashes with latest master

Describe the bug
I've been trying to run the latest(ish) master of VR on some SWIFT outputs (on COSMA7), and I've been getting a couple of odd crashes.
To Reproduce
Version cb4336de8421b7bada3e158c44755c41e9fab78b.
Ran on snapshots under /snap7/scratch/dp004/dc-borr1/new_randomness_runs/runs/Run_*
Log files
STDOUT:
...
[ 528.257] [debug] search.cxx:2716 Substructure at sublevel 1 with 955 particles
[ 528.257] [debug] unbind.cxx:284 Unbinding 1 groups ...
[ 528.257] [debug] unbind.cxx:379 Finished unbinding in 1 [ms]. Number of groups remaining: 2

STDERR:
terminate called after throwing an instance of 'std::runtime_error'
  what():  Particle density not positive, cannot continue

Environment (please complete the following information):
cmake .. -DCMAKE_CXX_FLAGS="-O3 -march=native" -DVR_MPI=OFF -DVR_HDF5=ON -DVR_ALLOWPARALLELHDF5=ON -DVR_USE_HYDRO=ON
Currently Loaded Modulefiles:
 1) python/3.6.5               5) parallel_hdf5/1.8.20
 2) ffmpeg/4.0.2               6) gsl/2.4(default)
 3) intel_comp/2018(default)   7) fftw/3.3.7(default)
 4) intel_mpi/2018             8) parmetis/4.0.3(default)

@JBorrow do you have the git SHA of an earlier revision where this worked?

Nothing helpful, the version I usually use is quite old.

This is most probably the same issue underlying #37. It should in principle be the same problem, except now it's caught earlier and with a different error message (that has been added since #37 was reported to help picking up this issue earlier at runtime, and with a more meaningful explanation). Please check if the workaround described in the last messages in that ticket (disabling OpenMP-based FOF) removes the problem. Note also that this seems to be a problem only when compiling without MPI support (-DVR_MPI=OFF).
@JBorrow when you say the version you usually use is quite old, and that in itself could be helpful. If you could report here what exact version was that it would help finding out the real underlying issue.

I am not entirely sure it is a duplicate of #37 but it might. The reason I say this is that I get the code to run with fee3a9f7c472fdd2a3a42aa213ee1f9a8b167d8b but break on cb4336de8421b7bada3e158c44755c41e9fab78b. And both these are fairly recent versions.
The error message is
terminate called after throwing an instance of 'std::runtime_error'
  what():  Particle density not positive, cannot continue

If you need full info, the first one was compiled here:

works: /cosma7/data/dp004/jlvc76/VELOCIraptor/VELOCIraptor-STF/build
breaks: /snap7/scratch/dp004/jlvc76/VELOCIraptor/VELOCIraptor-STF/build

Both cases were compiled with Intel 2020. Both have VR_OPENMP switched ON and VR_MPI switched OFF. Both are run using 28 OMP threads on a single node.
Input is

snapshot: /snap7/scratch/dp004/jlvc76/SWIFT/EoS_tests/swiftsim/examples/EAGLE_ICs/EAGLE_25/eagle_0036.hdf5
param file: /snap7/scratch/dp004/jlvc76/SWIFT/EoS_tests/swiftsim/examples/EAGLE_ICs/EAGLE_25/vrconfig_3dfof_subhalos_SO_hydro.cfg

Happy to copy anything over to other places or give you more info is needed.

@MatthieuSchaller interesting... if you say that a previous build worked and now it didn't then I introduced a regression. My guess is that the diagnosis is correct; i.e., there are particles with non-negative densities, leading to -inf potentials, but that in your case these do not make it to DetermineDenVRatioDistribution where the crash reported in #37 happened. I still need to corroborate this, so for the time being it's just a guess.
I'm happy to revert the change that adds the check for non-positive densities. That would remove this regression, but OTOH it could mean we are doing wrong calculations silently.

Yes, agreed, it's not ideal. Just thought this might help track things down somehow.

@JBorrow does the latest master without OMP but with MPI give you an alternative that is fast enough?

@JBorrow @MatthieuSchaller please see https://github.com/ICRAR/VELOCIraptor-STF/issues/37#issuecomment-749419902 for a very likely potential fix for this problem. If that fix makes this issue disappear then we can close this ticket.

I'm trying this out in cosma6 at the moment. Sadly the fix mentioned in https://github.com/ICRAR/VELOCIraptor-STF/issues/37#issuecomment-749419902 does not seem to address the problem reported in this issue, although it's probably very similar in nature. At the moment I'm trying to gather more information and seeing if there's anything obvious around https://github.com/ICRAR/VELOCIraptor-STF/commit/8b0cf42a686808d14e9ede750dec57a994c14a87, 9a21987 and https://github.com/ICRAR/VELOCIraptor-STF/commit/08595f84a0a5858962320e50dfc5016b9aea4666 that could help here.

I'm trying this out in cosma6 at the moment. Sadly the fix mentioned in https://github.com/ICRAR/VELOCIraptor-STF/issues/37#issuecomment-749419902 does not seem to address the problem reported in this issue, although it's probably very similar in nature. At the moment I'm trying to gather more information and seeing if there's anything obvious around https://github.com/ICRAR/VELOCIraptor-STF/commit/8b0cf42a686808d14e9ede750dec57a994c14a87, 9a21987 and https://github.com/ICRAR/VELOCIraptor-STF/commit/08595f84a0a5858962320e50dfc5016b9aea4666 that could help here.

I'm commenting here to to leave a trail: in #60 yet another instance of this problem has been reported. Details can be found throughout the comments, including build configuration details.

Another instance of this issue with hydro runs was reported in https://github.com/ICRAR/VELOCIraptor-STF/issues/37#issuecomment-781771823 for a zoom, non-MPI execution.

@MatthieuSchaller with the recent surge of more cases of errors due to negative potentials (a check I added as part of #37), I'm starting to think maybe I'll turn the check off, or at least turn it into a warning instead of an unrecoverable error, or even make this behavior configurable. In https://github.com/ICRAR/VELOCIraptor-STF/issues/53#issuecomment-739760984 I argued that invalid results might be end up being generated due to this, but otherwise this problem seems to be affecting it too many areas. Thoughts?

That would be a worry then. If we can get to that point and find particles that have negative densities it means something went wrong.
I suppose if we now it only happens when OpenMP is on then we can have that issue here closed and taken over by the wider OMP issue.

@MatthieuSchaller do you know of any (hopefully small) dataset + configuration file that can be used to consistently reproduce this error? I went through all the reported occurrences and either the original input and configuration files are gone, or the error occurred as part of a SWIFT + VR run, so I can't find an easy way to actually trigger it.
I can try to reason a bit more about the code and try to advance like that, but having a small, reproducible example would be great.

That's unfortunate. Let me dig out some older test case that was backed up.

Sorry @rtobar; my config at /cosma/home/dc-borr1/c7dataspace/XL_wave_1/runs/Run_0 crashes. This just happened now, using the latest maser.

Could you expand on the modules used and cmake flags used?

Currently Loaded Modulefiles:
 1) python/3.6.5               5) parallel_hdf5/1.8.20      9) cmake/3.18.1
 2) ffmpeg/4.0.2               6) gsl/2.4(default)
 3) intel_comp/2018(default)   7) fftw/3.3.7(default)
 4) intel_mpi/2018             8) parmetis/4.0.3(default)

cmake -DVR_USE_HYDRO=ON -DCMAKE_CXX_FLAGS="-fPIC" -DCMAKE_BUILD_TYPE=Release ..; make -j

Can you try disabling OpenMP?

Yeah, it works fine if you run with MPI only. That's an okay fix for now - it's actually faster for my 25s - but the behaviour should not be different between this and OMP.
On 26 Feb 2021, at 13:24, Matthieu Schaller <notifications<EMAIL_ADDRESS>wrote:
[EXTERNAL EMAIL]
Can you try disabling OpenMP?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/ICRAR/VELOCIraptor-STF/issues/53#issuecomment-786645602, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB3Z6G6MUP4SENHG5W5KCOLTA6OHPANCNFSM4UNKJL6Q.

Ok, that's interesting. Some of the cases above were breaking both with and without OMP.

Okay - I was wrong - I get the negative densities in pure MPI only mode (/cosma/home/dc-borr1/c7dataspace/XL_wave_1/runs_correct_dt/Run_0, snapshot 0000).

Do we have a way of knowing that the particle is or what object it belongs to to try to understand what the offending setup is?

Yes, in hindsight I should have added more information to the error message. I'll do that and push a commit straight to the master branch.
BTW, I tried replicating the issue with the data, settings and compilation flags used by @JBorrow, but without success so far. I tried in a local system with 20 cores, so tries a few combinations of OpenMP threads and MPI ranks (1x20, 3x6, 20x1) and none of them produced the error. Maybe I miseed something obvious so I'll re-try tomorrow, but otherwise this will be a bit harder to track down than anticipated. I also tried briefly to reproduce in cosma, but the queues were full and I was trying to go for an interactive job -- next time I'll just submit a job instead.

Yes, in hindsight I should have added more information to the error message. I'll do that and push a commit straight to the master branch.
BTW, I tried replicating the issue with the data, settings and compilation flags used by @JBorrow, but without success so far. I tried in a local system with 20 cores, so tries a few combinations of OpenMP threads and MPI ranks (1x20, 3x6, 20x1) and none of them produced the error. Maybe I miseed something obvious so I'll re-try tomorrow, but otherwise this will be a bit harder to track down than anticipated. I also tried briefly to reproduce in cosma, but the queues were full and I was trying to go for an interactive job -- next time I'll just submit a job instead.

Thanks. This is small enough to run on the login nodes if that helps (though I will deny having ever said that when questioned by the admin police...)

Could it be the fact I'm using Intel 2018?

OK, some more data points after some experiments in cosma6 with the input and configuration files pointed out by @JBorrow. I tried different variations of MPI/OpenMP ranks/threads in an MPI+OpenMP enabled build, which yielded different results. This is congruent with the results of my local experiments for the same inputs, where different rank/thread combinations yielded different
outcomes, so the problem clearly depends on the partitioning of the problem.

28 ranks, 1 thread each (logfile): I reproduced the error. As mentioned earlier, I improved the error message to contain some more information (now that I actually tested it I've put it on the master branch), here's the main extract from the logfile (the rank/thread number are 0-based):

terminate called after throwing an instance of 'std::runtime_error'
  what():  Particle density not positive, cannot continue.

 Information for particle 1/1267: id=1, pid=11349692, type=1, pos=(-0.00781215, 0.000332759, -0.0124483), vel=(11.6557, -4.83983, 74.4938), mass=0.000969505, density=0
Information for execution context: MPI enabled=yes, rank=16/28, OpenMP enabled=yes, thread=0/1

1 ranks, 20 threads (logfile): The error doesn't happen anymore and the code progresses much further... until another error occurs:

[0000] [1445.514] [ info] io.cxx:1292 Saving SO particle lists to halos-2.catalog_SOlist.0
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/var/slurm/slurmd/job2998665/slurm_script: Zeile 4: 16915 Abgebrochen             builds/53/stf -i /cosma/home/dc-borr1/c7dataspace/XL_wave_1/runs_correct_dt/Run_0/eagle_0000 -C /cosma/home/dc-borr1/c7dataspace/XL_wave_1/runs_correct_dt/vrconfig_3dfof_subhalos_SO_hydro.cfg -I 2 -o halos-2

This looks conspicuously close to the work added in #57 so it might be a new problem, and hopefully easy to fix.

4 ranks, 7 threads each (logfile) and 7 ranks, 4 threads each (logfile): they both break at a very similar point in time where 1x20 case above breaks, only this time nothing gets printed to screen because the error probably happens not in the main thread (see the logfiles for details). Based on this observation I'm assuming it's the same error in all three cases.

I'm running an extra 1x28 run to see what happens there, I'll post the results after it finishes.

To avoid the issue in #71 I tried running the experiments with the latest master that didn't have the changes introduced in #53, but otherwise had all the other fixes we've introduced since we started working on this problem.
The summary is that all but the 28x1 combination ran to completion. I don't think this sheds more light other than confirming that the issue doesn't depend on the inputs only, but actually on how the domain decomposition is handled. All logs are available below, but here's a summary of the successes:
[rtobar@bolano ~]---> grep "VELOCIraptor finished in" mpi-*
mpi-1-openmp-20.log:[0000] [1310.405] [ info] main.cxx:572 VELOCIraptor finished in 21.839 [min]
mpi-1-openmp-28.log:[0000] [1341.484] [ info] main.cxx:572 VELOCIraptor finished in 22.357 [min]
mpi-4-openmp-7.log:[0000] [ 709.688] [ info] main.cxx:572 VELOCIraptor finished in 11.828 [min]
mpi-4-openmp-7.log:[0001] [ 709.720] [ info] main.cxx:572 VELOCIraptor finished in 11.828 [min]
mpi-4-openmp-7.log:[0002] [ 709.760] [ info] main.cxx:572 VELOCIraptor finished in 11.829 [min]
mpi-4-openmp-7.log:[0003] [ 709.778] [ info] main.cxx:572 VELOCIraptor finished in 11.829 [min]
mpi-7-openmp-4.log:[0000] [ 642.523] [ info] main.cxx:572 VELOCIraptor finished in 10.708 [min]
mpi-7-openmp-4.log:[0003] [ 642.540] [ info] main.cxx:572 VELOCIraptor finished in 10.708 [min]
mpi-7-openmp-4.log:[0005] [ 642.546] [ info] main.cxx:572 VELOCIraptor finished in 10.709 [min]
mpi-7-openmp-4.log:[0001] [ 642.573] [ info] main.cxx:572 VELOCIraptor finished in 10.709 [min]
mpi-7-openmp-4.log:[0002] [ 642.583] [ info] main.cxx:572 VELOCIraptor finished in 10.709 [min]
mpi-7-openmp-4.log:[0004] [ 642.581] [ info] main.cxx:572 VELOCIraptor finished in 10.709 [min]
mpi-7-openmp-4.log:[0006] [ 642.667] [ info] main.cxx:572 VELOCIraptor finished in 10.711 [min]

The failure in 28x1 is exactly the same as in the previous test, so at least is fully reproducible:
terminate called after throwing an instance of 'std::runtime_error'
  what():  Particle density not positive, cannot continue.

 Information for particle 1/1267: id=1, pid=11349692, type=1, pos=(-0.00781215, 0.000332759, -0.0124483), vel=(11.6557, -4.83983, 74.4938), mass=0.000969505, density=0
Information for execution context: MPI enabled=yes, rank=16/28, OpenMP enabled=yes, thread=0/1

mpi-7-openmp-4.log
mpi-4-openmp-7.log
mpi-28-openmp-1.log
mpi-1-openmp-28.log
mpi-1-openmp-20.log

And just another update: this can also be reproduced in a few seconds with much smaller files and less ranks (13 was the minimum for this input/configuration in cosma):
$> OMP_NUM_THREADS=1 salloc -p cosma6 -A dp004 -N 1 --exclusive -t 1:00:00 mpirun -np 13 builds/53/stf -C /cosma7/data/dp004/jlvc76/BAHAMAS/Roi_run/vrconfig_3dfof_subhalos_SO_hydro.cfg -i /cosma7/data/dp004/jlvc76/BAHAMAS/Roi_run/baham_0036 -I 2 -o halos
....
terminate called after throwing an instance of 'std::runtime_error'
  what():  Particle density not positive, cannot continue.

 Information for particle 2473/6517: id=2473, pid=656947, type=1, pos=(0.733834, 0.93133, -0.427506), vel=(-274.327, -45.4808, 313.547), mass=0.552186, density=0
Information for execution context: MPI enabled=yes, rank=0/13, OpenMP enabled=yes, thread=0/1
...

I can reproduce this locally too, so hopefully this will open the door to proper debugging and further advances.

How is it possible that the particles have negative positions?

We box-wrap when writing the snapshots so that should be clean. The numbers also look too large for a rounding error.

Yes, I've checked in the snapshots and there are no particles outside of the box.
Could it be an error in the size of the box (as this only seems to happen to me, anecdotally, for z!=0) and then an incorrect re-wrap?

The last example above is at z=0.

Ah. Sorry, you're right. The original example is at z=0 but the ones recently have all been at z=5. VR then happily runs on the z=3 snapshots from the same simulations, though.

I remember Pascal telling me once that the position field in a Particle object is not a fixed value; during the lifetime of a run they are continuously updated, for example, to be relative to the centre of gravity of the group, or to other reference points.

So that may not be the smoking gun we think it is. I suppose if the particles are put in the frame of the centre of mass then half the particles or so will have negative coordinates.

A small update: with the smaller test case mentioned in https://github.com/ICRAR/VELOCIraptor-STF/issues/53#issuecomment-791198771, I found that the particle in question has density=0 because this particle didn't have its density calculated, not because the result of the calculation was 0. This is a small, but important piece of information, and it should help finding out what's wrong.
I was also able to reproduce the error earlier in the code, so instead of it failing in GetDenVRatio I can see the same error for the same particle (same pid and type, but different id/pos/vel values because of the dynamic nature of these fields) in the same rank at the end of GetVelocityDensityApproximative, which is where densities are calculated. This brings the cause and effect much closer in time, hopefully making it easier to analyse the problem.

Interesting. Could this particle be far from the centre of its object and hence be part of the spherical over-density but not of any actual sub-structure?

The case /cosma/home/dc-borr1/c6dataspace/XL_wave_1/runs/Run_8 for snapshot 6 (z=0) is a new crashing case. This breaks for both MPI and OpenMP only runs. Vr version, config log, etc, all linked in the directory under the velociraptor_submit_0006.slurm file.

I could finally dedicate some more time today to this problem.
As mentioned previously in https://github.com/ICRAR/VELOCIraptor-STF/issues/53#issuecomment-800028795 some particles end up with density=0 because their density is not computed, not because the result of the computation is 0. Particles are not iterated directly, but instead they are first grouped into leafnode structures (I'm assuming these are the leaf nodes of the KDTree containing the particles). The code then iterates over the leafnode structures, and for each leafnode the contained particles are iterated over.
Using the smaller reproducible example mentioned in https://github.com/ICRAR/VELOCIraptor-STF/issues/53#issuecomment-791198771 I've now observed how this plays out. When MPI is enabled density calculation is a two-step process:

First each rank loops through its leafnode structures to iterate over its particles. Before calculating densities it first checks if there is some overlap with particles from a different rank, and if so the leafnode is skipped
During the second pass the overlapping is taken care of: particle information is sent across ranks so they can now compute densities for the leafnodes that were skipped in the first step

Note that the extra overlapping checks and particle communication happens only if the Local_velocity_density_approximate_calculation is ==1. but if a greater value is given (e.g., 2) then only the first step is taken and no local leafnode structures are skipped. So a workaround could be to set this value to 2, although presumably some results will change.
The problem I've reproduced happens when during the second step particle information is exchanged, but a rank doesn't receive any particle information. This sounds like a bug either in the calculation of the overlaps (there are two flavours depending on whether MPI mesh decomposition is used, I'm using it), or in the code exchanging particle information. In other words, if overlapping is found then particle information is expected to arrive. Alternatively it might be correct to not expect particle information to arrive (only sent) in certain scenarios, in which case the code should treat such leafnode structures again as purely local and process them.
I'm not really sure what's the best to do here. @pelahi, given the situation described above would you be able to point us in the right direction?
This analysis is consistent with at least the MPI-enabled crash @JBorrow reported in the comment above. In there one can see:
[0026] [ 622.342] [debug] localfield.cxx:942 Searching particles in other domains 0

The giveaway here is the 0 (as in "no overlapping particles were imported into my rank"). This is what leads then to the fatal crash in rank 26:
terminate called after throwing an instance of 'std::runtime_error'
  what():  Particle density not positive, cannot continue.

 Information for particle 1377/4476: id=1377, pid=9634564, type=1, pos=(0.0820553, -0.0173851, -0.0163657), vel=(13.4933, -32.6683, -5.64264), mass=0.000969505, density=0
Information for execution context: MPI enabled=yes, rank=26/28, OpenMP enabled=yes, thread=0/1

The crash in the non-MPI, OpenMP case might be yet a different problem.

Hi @rtobar , I'll look into this later today. I think it is likely a simple logic fix.

Thanks to @pelahi there is now a new commit on the issue-53 branch (371b685) that addresses the problem highlighted on my last comment. The underlying problem was that the code was incorrectly assuming that if during the first processing step one the leafnode structures was skipped then at least some particle data had to be received from the other ranks. This assumption was incorrect: when no data was received but there were leafnode structures that were skipped during the first step, then the skipped structures still had to be processed locally during the second step. This fix applies
With this patch I can now run the small defective test case until the end, meaning that all particles have their densities calculated.
I tried also reproducing the last crash (MPI case) reported by @JBorrow and with the patch I can get past the original problem. Later on during the execution the code crashes again though, but seems like an unrelated issue (more like #71 or #73, but haven't dug on it).
With this latest fix I feel fairly confident the underlying problem is finally gone for the MPI-enabled cases, but there still remains the MPI-disabled problem @JBorrow is still having, and that I think has happened in a few other places. So while I'll put the latest fix on the master branch, I think we can't still call it a day here.

I like the sound of this!

Thanks for your hard work!

Just a quick update: I reproduced the MPI-disabled crash that @JBorrow experimented in cosma6 with the same dataset. Here's a small backtrace:
(gdb) bt
#0  0x00007f9f04af2387 in raise () from /lib64/libc.so.6
#1  0x00007f9f04af3a78 in abort () from /lib64/libc.so.6
#2  0x00007f9f057261e5 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x00007f9f05723fd6 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:47
#4  0x00007f9f05722f99 in __cxa_call_terminate (ue_header=ue_header@entry=0x7f99d4de90a0) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54
#5  0x00007f9f05723908 in __cxxabiv1::__gxx_personality_v0 (version=<optimized out>, actions=2, exception_class=5138137972254386944, ue_header=0x7f99d4de90a0, context=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:676
#6  0x00007f9f050b5eb3 in _Unwind_RaiseException_Phase2 (exc=exc@entry=0x7f99d4de90a0, context=context@entry=0x7f9a9cbe09a0) at ../../../libgcc/unwind.inc:62
#7  0x00007f9f050b66de in _Unwind_Resume (exc=0x7f99d4de90a0) at ../../../libgcc/unwind.inc:230
#8  0x0000000000572846 in GetDenVRatio(Options&, long long, NBody::Particle*, long long, GridCell*, Math::Coordinate*, Math::Matrix*) ()
#9  0x0000000000571e4c in GetDenVRatio(Options&, long long, NBody::Particle*, long long, GridCell*, Math::Coordinate*, Math::Matrix*) ()
#10 0x00000000005b0390 in PreCalcSearchSubSet(Options&, long long, NBody::Particle*&, long long) ()
#11 0x000000000058eb50 in SearchSubSub(Options&, long long, std::vector<NBody::Particle, std::allocator<NBody::Particle> >&, long long*&, long long&, long long&, PropData*) ()

No surprises here, this is where the code has always been crashing. However with the latest changes that have been integrated into the master branch there is a double-check for density != 0 in all particles at the end of GetVelocityDensityApproximative (the function where densities are calculated) to crash earlier rather than later. This means that the problem we are facing here is different: the particles that have their densities = 0 have them as such not because they were skipped by ``GetVelocityDensityApproximative`, but because they never made it into the call. With more time I'll try to continue digging, and hopefully I'll also find a smaller, reproducible example that I can use locally to better pin down what's going on.

Another clue here, from running on COSMA-8 (this has 128 cores/node). The code was far more likely to crash when running on all 128 (rather than the 32 that I resubmitted with) cores, in the MPI-only configuration, with this problem.

Hi, new to the party, but I'm finding this behaviour for commit 8f380fc. A difference here is this is a zoom run, with VR configured with VR_ZOOM_SIM=ON (mpi off, omp on) on cosma7. Just found this issue, and still need to go through the whole thread to see if any of the suggestions fix things, but thought worth reportig here for now

terminate called after throwing an instance of 'vr::non_positive_density'
what():  Particle density not positive, cannot continue.
Particle information: id=0, pid=2604476, type=1, pos=(-0.00540459, 0.00496019, -0.00370049), vel=(-4.63392, 13.0163, -20.7466), mass=0.000121167, density=0
Execution context: MPI enabled=no, OpenMP enabled=yes, thread=0/1[ 145.889] [debug] search.cxx:2719 Substructure at sublevel 1 with 634 particles

Welcome to the party...
The current "workaround" to just get somewhere with production runs is to toggle MPI and OMP on/off. One combination might be lucky...
Apart from that, your simulation is likely quite small so could you give the directory on cosma, as well as config file?
Might be a useful small test to see what may be going on. The more problematic examples we have the more likely we are to identify the issue.

Welcome to the party...
The current "workaround" to just get somewhere with production runs is to toggle MPI and OMP on/off. One combination might be lucky...

Thanks, configuring with MPI on seems to be working fine for me now.

Apart from that, your simulation is likely quite small so could you give the directory on cosma, as well as config file?
Might be a useful small test to see what may be going on. The more problematic examples we have the more likely we are to identify the issue.

yes the zooms seem to have a high hit-rate with this bug (3/3 of the last I've tried), the last one I was using is here for now:
/cosma7/data/dp004/wmfw23/colibre_dust/runs/zooms/Auriga/h1/data/snapshot_0013.hdf5

@james-trayford since the introduction of the "no positive density" check we have found at least three different points in the code that were causing this issue in different ways:

#37 was a zoom-specific problem that has long been fixed.
In this issue we already fixed an MPI-related problem.
There still remains at least one unsolved problem, likely triggered by OpenMP-enabled builds (regardless of MPI being enabled or disabled) and somehow related to domain decomposition: it usually shows with high CPU counts and/or big domains, hence it's been difficult to track down.

I'd assume your issue is the third, and that by using MPI you reduced the number of OpenMP threads on each rank, thus avoiding the issue. An alternative workaround (I think, not fully certain) would be to set OMP_run_fof to a non-zero value in the configuration file and try again. This disables OpenMP during FOF searching, so things would run fairly slower, but execution would finish.

Hi @james-trayford, I will have a look using your snapshots. Any chance you could put them somewhere I could access them? @MatthieuSchaller should I email Adrian to see if I can access cosma now that I have changed institutions?

Do you still have access to gadi? I can copy things there, the setup is fairly small.

Sadly no, but I can request access again.
On Wed, 23 Jun 2021 at 14:40, Matthieu Schaller @.***>
wrote:

Do you still have access to gadi? I can copy things there, the setup is
fairly small.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ICRAR/VELOCIraptor-STF/issues/53#issuecomment-866572154,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC3ZASYXDYP6FHLAYH7SRFLTUF6WNANCNFSM4UNKJL6Q
.

Ok. Might be easier to revive your cosma account then. Feel free to email Adrian and cc me in.

Is there anything I can do here to help with this issue? Tests? narrowing down of a use case? etc.

Hi @james-trayford, could you provide the config options you ran with? I am not encountering the error but it could be something specific. How many MPI ranks and OMP threads did you run with?

I have

@stuartmcalpine unfortunately no new progress has been made here. I think the best summary of the situation is https://github.com/ICRAR/VELOCIraptor-STF/issues/53#issuecomment-866507052, where a workaround (not a great one, but should work) is suggested. Older comments go into all the details on how this story has unfolded...
In order to fix this problem the best would be to find a small, quickly-reproducible example. As mentioned in the comment, this seems to be a problem with domain decomposition on high CPU counts, but that's a guess. In any case having the full details of your failure would definitely be a gain.

I am doing some zoom runs, no mpi, on-the-fly with swift. For these tests, cosma 8 and 128 threads. The DMO version segfaults on the 4th invocation, and the hydro version later, like the 10th. But the DMO is consistently failing at the same place.
module load intel_comp/2018 intel_mpi/2018 fftw/3.3.7
module load gsl/2.5 hdf5/1.10.3
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-fPIC" -DVR_ZOOM_SIM=ON -DVR_MPI=OFF -DVR_MPI_REDUCE=OFF -DVR_USE_SWIFT_INTERFACE=ON ..
Config file:
vrconfig_3dfof_subhalos_SO_dmo.txt
Last bit of log:
[1726.081] [debug] search.cxx:3982 Getting Hierarchy 23
[1726.081] [debug] search.cxx:4015 Done
[1726.726] [ info] substructureproperties.cxx:5047 Sort particles and compute properties of 23 objects
[1726.726] [debug] substructureproperties.cxx:5059 Calculate properties using minimum potential particle as reference
[1726.726] [debug] substructureproperties.cxx:5062 Sort particles by binding energy
[1795.700] [debug] substructureproperties.cxx:5087 Memory report at substructureproperties.cxx:5087@long long **SortAccordingtoBindingEnergy(Options &, long long, NBody::Particle *, long long, long long *&, long long *, PropData *, long long): Average: 70.075 [GiB] Data: 72.269 [GiB] Dirty: 0 [B] Library: 0 [B] Peak: 81.744 [GiB] Resident: 68.478 [GiB] Shared: 8.734 [MiB] Size: 72.370 [GiB] Text: 4.180 [MiB]
[1795.701] [debug] substructureproperties.cxx:42 Getting CM
[1795.702] [debug] substructureproperties.cxx:320 Done getting CM in 1 [ms]
[1795.702] [debug] substructureproperties.cxx:4621 Getting energy
[1795.703] [debug] substructureproperties.cxx:4733 Have calculated potentials in 744 [us]
[1795.704] [debug] substructureproperties.cxx:5034 Done getting energy in 1 [ms]
[1795.704] [debug] substructureproperties.cxx:338 Getting bulk properties
[1795.706] [debug] substructureproperties.cxx:2194 Done getting properties in 1 [ms]
[1795.706] [debug] substructureproperties.cxx:3219 Done FOF masses in 4 [us]
[1795.706] [debug] substructureproperties.cxx:3236 Get inclusive masses
[1795.706] [debug] substructureproperties.cxx:3237  with masses based on full SO search (slower) for halos only
Line where it segfaults:


System.DivideByZeroException

i get System.DivideByZeroException when i try to compress empty txt file.
screenshot
ArchiveType = Zip
CompressType = BZip2

File: SharpCompress\Compressors\BZip2\CBZip2OutputStream.cs
Error line: 1344

My first thought is "don't do that." but I guess zip allows for empty files?
You could just set to no compression on that entry.


various russian translation fixes and cleanups

Overall mistakes and visual fixes

The indentations need to be fixed. I can't fix them all again - I don't have the time. If you look at the diff in github you'll see that the russian translations don't line up.

OK, i try.

That's because you are using tabs instead of space I think. There is a option that let you "convert" all the tabs to spaces (look for "Blank Operations" in "Edit")

I will merge this and fix any issues if you resolve the conflicts.


Application Error on the API

the API is brokey
:(

Hi @circularsprojects,
Unfortunately, this app is running on a free Heroku dyno, therefore, the API seemed to be broken and displayed the application-error. Apologies.

API uptime - checked ✔
Issue resolved for now.


[toascii] VkFlags64's enum need a diffferent way to print

Those VkFlags64s are like VkFormatFeatureFlagBits2KHR, VkPipelineStageFlagBits2KHR, VkAccessFlagBits2KHR.
VkFlags's bit is enum
typedef enum VkSubmitFlagBitsKHR {
    VK_SUBMIT_PROTECTED_BIT_KHR = 0x00000001,
    VK_SUBMIT_FLAG_BITS_MAX_ENUM_KHR = 0x7FFFFFFF
} VkSubmitFlagBitsKHR;
typedef VkFlags VkSubmitFlagsKHR;

VkFlags64's bit is a uint64
typedef VkFlags64 VkFormatFeatureFlags2KHR;

// Flag bits for VkFormatFeatureFlagBits2KHR
typedef VkFlags64 VkFormatFeatureFlagBits2KHR;
static const VkFormatFeatureFlagBits2KHR VK_FORMAT_FEATURE_2_SAMPLED_IMAGE_BIT_KHR = 0x00000001ULL;

In this case, VkFlags64's enum need a diffferent way to print.

Same issue as #612

Closing in favor of #612


Please implement dynamic tagging approach in on thank you page resources

Please implement dynamic tagging approach in on thank you page resources carousel .  Right now they are being contributed from document only but they are common resources so should be changes at one place only.
https://www.moleculardevices.com/en/assets/app-note/dd/img/high-throughput-modeling-inflammatory-bowel-disease-using-human-intestina-organoids?page=thankyou

@andreituicu has more context about it.
Thanks,
Rajneesh

What I discussed with @dev-rajneeshkumar is that what would work for him is a carousel with links to the pages that dynamically retrieves the:

image
title
description
type

and constructs the carousel with this information.
We might have this already so please check before writing the code if any of the existing carousels would work for this.

where would those "highlighted resources" be tagged then? as metadata on the page? how about tagging some resources that are files and don't have doc?


Game files do not have steam.dll

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

You copy 4  files from server to game launch game you should get error then
delete steam.dll file from game. Files
On Sun 15 Sep 2024 at 15:49, upperking @.***> wrote:

neither of you make it clear1
cOPY WHAT FILES FROM WHAT FOLDER TO WHAT FOLDER?
—
Reply to this email directly, view it on GitHub
https://github.com/d3adwolf/rustlegacy/issues/10#issuecomment-2351624526,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AK6R5T3CPB3FR3BAD3A3NPTZWWNADAVCNFSM6AAAAABNEVJ6G2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJRGYZDINJSGY
.
You are receiving this because you authored the thread.Message ID:
@.***>

i get Auth Error: invalid ticket
Auth failed: Steam auth denied user - dude

copy what copy 4  files from server to game launch ?

They are mentioned in chat by me
On Mon 16 Sep 2024 at 17:34, upperking @.***> wrote:

copy what copy 4 files from server to game launch ?
—
Reply to this email directly, view it on GitHub
https://github.com/d3adwolf/rustlegacy/issues/10#issuecomment-2353397884,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AK6R5T5ETPYCIENQD6J5AELZW4CBZAVCNFSM6AAAAABNEVJ6G2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJTGM4TOOBYGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

In the picture?


Jackson2JsonMessageConverter: New ObjectMapper instance not aware of jsr310 module

In what version(s) of Spring AMQP are you seeing this issue?
3.0.6
Describe the bug
We are transporting Pojos over AMQP. These Pojos also include Java8 Time objects like LocalDateTime. Our application ran in a ""java 8 date/time type java.time.localdatetime not supported by default" error although jackson-datatype-jsr310 was present in the classpath.
To Reproduce
Return a Pojo including Java8 Time objects in a @RabbitListener annotated method.
Expected behavior
The returned Pojo (including Java8 Time objects) should be converted to JSON without any error if  jackson-datatype-jsr310 is present in the classpath.
*Solution
The problem itself was that we provied a Jackson2JsonMessageConverter in our project using its default constructor, see:
https://github.com/spring-projects/spring-amqp/blob/c845cb7cc5d8d39224e3ea219fb3b005f35963f4/spring-amqp/src/main/java/org/springframework/amqp/support/converter/Jackson2JsonMessageConverter.java#L45C9-L45C37
This calls the construcor which creates a new ObjectMapper instance, see:
https://github.com/spring-projects/spring-amqp/blob/c845cb7cc5d8d39224e3ea219fb3b005f35963f4/spring-amqp/src/main/java/org/springframework/amqp/support/converter/Jackson2JsonMessageConverter.java#L58
This new ObjectMapper is not aware of any additional Jackson modules in the classpath. Since the mapper receives already some configuration in this constructor, I suggest to add a this.objectMapper.findAndRegisterModules() there.
I'll provide a pull request for this.

I'm converting this into an Enhancement since there is respective ctor you can use to mitigate expectations Jackson2JsonMessageConverter(ObjectMapper jsonObjectMapper).
Will provide the fix shortly for main to make the target configuration easier.


Unable to Check for Updates?

Description
After downloading the :atom: atom 1.19.2 update and restarting atom I received the following alert:

There was an error checking for updates
The command is disabled and cannot be executed

Atom seems to be working fine and the update did install, but before that I couldn't even get the update to install. I would keep having to download the update and restart atom, but the update didn't install and I would just get this alert.

I seem to be having an issue with APM, perhaps relating to this.
Can you try running the following command for atom's package manager in the terminal.
apm is the atom package manager command.
apm outdated; apm update;
let me know if this gets you your updates, however let me know if a graphical error shows up. Running apm update for seems to be all distorted on the output. Nothing shows up correctly since they make it look fancy when updating.

Thanks for reaching out!
Feel free to try @Jahhein's suggestion @jajaperson and let us know if that helps but we'll go ahead and close this issue as mentioned below.
There's a similar issue with more details so you can subscribe there for updates if you'd like /cc https://github.com/atom/atom/issues/15417.

We require the template to be filled out on all new issues and pull requests so if you can open a new issue with the template filled out, that would be helpful for us. We do this so that we can be certain we have all the information we need to address your submission efficiently. This allows the maintainers to spend more time fixing bugs, implementing enhancements, and reviewing and merging pull requests.
Thanks for understanding and meeting us half way :grinning:


Exist vs Must Exist

in the GUI, I have "Exists" and "Must Exist", looks like they do the same job.
XML is:
  <rule name="Exists" value="ID" tracktype="General" occurrence="*"/>
  <rule name="Must Exist" value="ID" tracktype="General" occurrence="*" operator="exists"/>

Is there any reason for having both?

I noted a discrepancy about it when we discusse "Must not exist". I added "Must exist" at that time but left "exists" for reverse comparability. Can drop "exists".

Compatibility is good to use, but maybe we should avoid a weird display for a beginner (2 lines for the same thing).
As we need to support old policies, XSL should not drop the old fashion. But the UI should have a "upgrade" feature so we upgrade policies to the new style when we load them. Not doable quickly so for later, let's keep it as it right now.


bug(layerdb): spawn those writes after all

We saw an error last week where some snapshots we expected weren't appearing in the database. Our theory was that perhaps we were doing some out-of-order writes. That might have been true, but the fix was that we stopped spawning tasks for each persister request - which slowed down the processing of those writes to a degree that, unless you had a very fast machine, would mean that the snapshot write hadn't been distributed by the time the rebaser hit its two second limit (and the write hadn't committed to postgres yet).
This would then cause the SDF instance running the migration to panic, which would destroy the remaining persister tasks, which would ensure the data never arrives.
This brings back spawning the write tasks in the persister, which lets us sail through the writes.

bors merge


CanError alert when going offroad

Let's fix up the transition to offroad when we do #2262.

Seems to only be an issue on PC.

I'm seeing this in-car.


Bitmap library throws error when compiled with optimizations level 2

There is an error when compiling Bao for RISC-V in sbi_ipi_handler function for gcc version 12.1.0
struct sbiret sbi_ipi_handler(unsigned long fid)
{
    if (fid != SBI_SEND_IPI_FID) return (struct sbiret){SBI_ERR_NOT_SUPPORTED};

    unsigned long hart_mask = vcpu_readreg(cpu()->vcpu, REG_A0);
    unsigned long hart_mask_base = vcpu_readreg(cpu()->vcpu, REG_A1);

    struct cpu_msg msg = {
        .handler = SBI_MSG_ID,
        .event = SEND_IPI,
    };

    for (size_t i = 0; i < sizeof(hart_mask) * 8; i++) {
        if (bitmap_get((bitmap_t*)&hart_mask, i)) {
            vcpuid_t vhart_id = hart_mask_base + i;
            cpuid_t phart_id = vm_translate_to_pcpuid(cpu()->vcpu->vm, vhart_id);
            if(phart_id != INVALID_CPUID) cpu_send_msg(phart_id, &msg);
        }
    }

    return (struct sbiret){SBI_SUCCESS};
}

The following error occurs:
In function 'bitmap_get',
    inlined from 'sbi_ipi_handler.part.0' at /media/ninolomata/Nino/bao-demos/wrkdir/srcs/bao/src/arch/riscv/sbi.c:260:13:
/media/ninolomata/Nino/bao-demos/wrkdir/srcs/bao/src/lib/inc/bitmap.h:43:16: error: 'hart_mask' is used uninitialized [-Werror=uninitialized]
   43 |     return (map[bit / BITMAP_GRANULE_LEN] & (ONE << (bit % BITMAP_GRANULE_LEN)))
      |             ~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
/media/ninolomata/Nino/bao-demos/wrkdir/srcs/bao/src/arch/riscv/sbi.c: In function 'sbi_ipi_handler.part.0':
/media/ninolomata/Nino/bao-demos/wrkdir/srcs/bao/src/arch/riscv/sbi.c:251:19: note: 'hart_mask' declared here
  251 |     unsigned long hart_mask = vcpu_readreg(cpu()->vcpu, REG_A0);
      |                   ^~~~~~~~~
cc1: all warnings being treated as errors

I  tested running Bao demos Linux + Freertos in the following scenarios:

compile bao with optimization level 3 -> didn't compiled;
compile bao with optimization level 0 and 1 -> compiled and demo worked;
switch the bitmap_get function to non inline function -> -> compiled and demo worked;
suppress the error -> -> compiled with warnings and demo worked;

Can you give for information on what exactly toolchain you are using and where to get it?

Anyways, I was able to reproduce this using riscv64-unknown-elf-gcc (g2ee5e430018) 12.2.0 downloaded from here.
The solution here is quite straightforward. We need to use the bit_get function defined in lib/inc/bit.h. However, we should hunt for other possible issues related to casting integer types do bitmaps for extracting/manipulating bits and fix all in a single go.

@ninolomata Please check #79, it should solve the issue.

@josecm Just tested it is working.


[Feature request] Suggest Longer Break after 4 Sessions

Big fan of the software thanks a lot for the effort!!!!
I do experience that after a couple of session I have a need for a longer break.
It is also suggested as by the wikipedia article https://en.wikipedia.org/wiki/Pomodoro_Technique to have a longer break after 4 sessions.
Can this be somehow integrated the popup after 4 sessions suggests  a 25min break?

Thanks!
The feature is already there but you need to enable it from the Settings -> Timer Duration section.


chore(actors-sdk): auto-determine api endpoint & environment to select region from

Changes

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#1588  👈 (View in Graphite)
#1587
#1586
#1585
#1580
#1548
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Water Bottle Scrap

Made a simple water bottle in blender to later be added as a scrap to the game.

Nice job with the texture! GTM


B-18393-INT

Agility ticket
Summary
PPMs for moves whose origin duty location does not provide service counseling are counseled by PRIME. Because PRIME does not have the authority to approve or reject AOA requests, customers will have to do this manually. As such, when a customer requests an advance while setting up their PPM, our system will have that flagged as "needing review" when it eventually makes its way to a closeout counselor. This is confusing for them.
As such it was determined that if a prime counseled customer requests an advance, and indicates it was received, they closeout counselor should see "Received" in their UI.

If a prime counseled customer requests an advance, and indicates it was not received, the closeout counselor should see "Not received" in their UI.

How to test

Access the
Login as a

Warnings

:warning:
This PR does not include changes to unit tests, even though it affects app code.

  Generated by :no_entry_sign: dangerJS against 3bb8e3f8d2a54e93dd856d7b0c677b43614e32f7


user experience distorted - back end language on scope should not be displayed to user

Not in scope of tool. this is an ODI website issue.


document isomorphic usage

The README should document how to use node-fetch in a polyfill fashion, perhaps with a Webpack exemple that shows how to load whatwg-fetch or node-fetch conditionally. This is not something obvious as node-fetch might actually need to be patched to act in a polyfill fashion vs. a simple ProvidePlugin, which would fail with external node_modules not processed by Webpack for example.
This will work properly with isomorphic-fetch because it polyfills, but I have not yet found a proper way to do it with Webpack.
node-fetch might possibly need to be updated to provide something like import 'node-fetch/polyfill so that it can be used in a truly isomorphic fashion.

node-fetch is not trying to be isomorphic, while isomorphic-fetch is, it would be much better to provide such info on isomorphic-fetch readme.

Yes, that make sense. The isomorphic question aside, node-fetch should probably provide a polyfill functionality, so it can be used with libraries that depend on fetch being available as a global.

I am closing this as decision is still to leave polyfilling to isomorphic-fetch (or you can write your simple module to expose node-fetch to global)


Enabling Nightly cron run

Enabling Nightly cron job

Looks like Nightly jobs are failing, error messages indicate missing packages but install logs don't show any issues...

/azp run Nightly

scikit-learn install issue is fixed but still a problem with coverity runs - not sure about this. Nightly runs are now running properly and first look at  sklearn 1.5. Not sure if we want to merge this now with the red CI or resolve all issues first, but at least one source of many failures is deprecation of TSNE n_iter for max_iter (https://github.com/scikit-learn/scikit-learn/pull/28471/files#diff-dec5f8e0a26a0b2f3a2200eb52b0079fc1ec90b1ad04940bf7fe00d2aad7851b)

I would say - yes, with plan for addressing problems. At least this would enable Coverity and Jupyter validation.
Coverity fails in pr because key is not passed to job


Plugin refactoring and update

This pull request contains multiple improvements for the project:

Fixed #7: upgrade liquibase version to the currently latest 3.5.3 version.
Fixed issue with multiple tasks running. Previously, if you created a task, which contains multiple liquibase tasks, all tasks after first will fail. This happens because all tasks shared the same liquibaseInstance object and after first task connection to the db will be closed (see RichLiquibase.execAndClose). To fix this, I changed the type of liquibaseInstance to TaskKey[() => Liquibase]. So, all tasks will create their own instance and DB connection. I added tests for this.
Changed type of liquibaseUrl, liquibaseUsername, and liquibasePassword from SettingKey to TaskKey to make them lazy.
Did some refactoring:

Combined RichLiquibase implicit conversion with class definition. It's slightly reduced code size.
Removed liquibaseDatabase task, because its definition is pretty big, but it is internal thing and no one should use it in their own build definition.

Updated README.md file.
Fixed #9: move from 0.1.0-SNAPSHOT to 0.2.0. This snapshot stuff is very deterrent, but this plugin is pretty stable and works well. I didn't test public publishing (I used only publishLocal).

This plugin is the best sbt liquibase plugin I found (I checked a lot), so It would be really cool if it will evolve.

@llinder Could you check this plz?


Add branch hint for _mi_os_good_alloc_size

In _mi_os_good_alloc_size, overflow caused by alignment size is rare,
and this patch added the appropriate branch hint during range checks.

Thanks!