Prisma connection doesn't recover from idle in transaction state

Bug description
Hi there,
As the title mention it, I encountered a bug when using both interactives and non-interactives transactions with prisma.
Sometimes, the prisma client get stuck into a 'idle in transaction' mode, this mainly happens under loads (eg: truncate all tables in a database) when postgres is allowed little ressources (eg: in CI, 1 CPU for postgres container).
From my preliminary investigation:

This bug is random and ressources related (with 4 CPU allowed to postgres it doesn't appears), with less query load it doesn't appears. And when it does, it doesn't appears at the same point.
It seems to be postgres "lock" related. For us, this show up during our "integrations testing" in which we truncate all the tables, then seed them with controlled data. Sometimes it's one of the "flush" (TRUNCATE) query who get stuck in the 'idle in transaction' mode, sometimes it's one of the "seed" (INSERT INTO). The common denominator seems to be the fact that some locks aren't unlocked by postgres.

The main issue is that, once it get into this state, the prisma client never get out of it. In our case, requiring to restart the webserver to instantiate a new connection to the database.
It's not very clear for me how to make a MRE from my issue since it's heavily related our code and only appears under loads.
How to reproduce
Here is a MRE reproducing this issue: https://github.com/avallete/prisma-9584-MRE
Expected behavior
Prisma could throw a "transaction timeout" error after X seconds passed in a transaction which doesn't end.
Prisma information
nothing specific
Environment & setup

OS: OSX/Debian
Database: postgres:12
Node.js version: v14.17.1

Prisma Version
prisma                : 2.30.3
@prisma/client        : 2.30.3
Current platform      : darwin
Query Engine (Binary) : query-engine b8c35d44de987a9691890b3ddf3e2e7effb9bf20 (at ../../node_modules/@prisma/engines/query-engine-darwin)
Migration Engine      : migration-engine-cli b8c35d44de987a9691890b3ddf3e2e7effb9bf20 (at ../../node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine  : introspection-core b8c35d44de987a9691890b3ddf3e2e7effb9bf20 (at ../../node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary         : prisma-fmt b8c35d44de987a9691890b3ddf3e2e7effb9bf20 (at ../../node_modules/@prisma/engines/prisma-fmt-darwin)
Default Engines Hash  : b8c35d44de987a9691890b3ddf3e2e7effb9bf20
Studio                : 0.423.0
Preview Features      : interactiveTransactions, nApi, referentialActions

Also tried with the latest 3.X version, the issue is the same.

i've been debugging a similar issue with long running transactions the past few days. I think it's likely the same root cause as this report. We wrap our unit tests in database transactions to isolate/reset state between tests. Fairly regularly, our test suite will have several tests fail due to timeouts. After adding logging, it appears the tests get stuck after we throw inside a long running transaction but the $transactions promise never rejects, instead it hangs.
I wasn't really having much luck debugging the underlying issue, but after seeing this report, I ran select * from pg_stat_activity where state='idle'; in psql. There are idle rows returned consistently with query=ROLLBACK when a test fails due to timeouts and they don't go away until the jest process ends.

i've been debugging a similar issue with long running transactions the past few days. I think it's likely the same root cause as this report. We wrap our unit tests in database transactions to isolate/reset state between tests. Fairly regularly, our test suite will have several tests fail due to timeouts. After adding logging, it appears the tests get stuck after we throw inside a long running transaction but the $transactions promise never rejects, instead it hangs.
I wasn't really having much luck debugging the underlying issue, but after seeing this report, I ran select * from pg_stat_activity where state='idle'; in psql. There are idle rows returned consistently with query=ROLLBACK when a test fails due to timeouts and they don't go away until the jest process ends.

I'm using "pgAdmin" to monitor both ressources usages / numbers of locks / opened sessions on my DB. And I have quite the same result.
However, I did try to extract that from jest environment by only running "lock intensives" operations in an infinite loop (no tests, only running code). And after a while ~(120 interations but it's random between the run 90-150), the same error show up. Basically, one of the transaction take some locks on the database, then, the session get stuck into an "idle in transaction" infinite hanging state, prisma doesn't throw an error, the session is never killed (blocking all subsequents queries who may need the locks acquired).
Here is the pseudo code I run in the loop:
async function flush() {
    const results = await prisma.$executeRaw(SELECT_ALL_DB_TABLES_QUERY);
    await prisma.$transaction(results.map((result) => prisma.$executeRaw(`TRUNCATE TABLE ${result.table_name} CASCADE`)));
}

async function seed() {
    const insert_intos = await readfile("./myseed.sql");
    await prisma.$transaction(insert_intos.map((sql) => prisma.$executeRaw(sql)));
}

async function main() {
   while (true) {
       await flush();
       await seed();
   }
}

I'm actually trying to create a MRE on another repo re-using this concept to be able to investigate further on this outside my application specific context.
ATM the only mitigation I found is to refactor the prisma.$transaction, into a for of + await loop. I do loose both the rollback ability and transaction/database integrity. But it does allow my code not to hang forever.

Possibly related: https://github.com/prisma/prisma/issues/8707

Hi. I will add my bits to it.
I am facing a similar problem in MySql 8. This works locally (or at least I didn't encounter the problem yet) on 4-core 32GB device but hangs (promise do not return) in i.e. preemptible GCE server with 1 core and low amount of RAM.
Seems like the database is not locked afterwards just the Prisma promise does not return, nor throw/rejects.
In my scenario that prevents ack'ing broker messages at some point which creates the service unresponsive:

Code:
   //called by message broker consumer for each message
    async saveMessage(input: MessageWithoutOrderInput, order?: number) {
        const uuid = randomUUID();
        const promise = this._prismaSaveMessage([input], order, uuid);

        const interval = this.checkForPending(promise, uuid);
        intervals.set(uuid, interval);

        const debug = (await promise)[0];
        this.logger.debug(debug);
        return debug;
    }

    //for logging pending promises
    private checkForPending(promise: Promise<any>, uuid: string) {
        return setInterval(() => {
            const isPending = util.inspect(promise) === 'Promise { <pending> }';
            this.logger.debug(`isPending: ${isPending}`, uuid);
            if (!isPending) {
                intervals.get(uuid)();
            }
        }, 2000);
    }

    private _prismaSaveMessage(inputArray: MessageWithoutOrderInput[], explicitOrder?: number) {
        return this.db.$transaction(async (prisma) => {
            let lastMessageOrder =
                explicitOrder ||
                ((
                    await prisma.assistantMessage.aggregate({
                        _max: { order: true },
                        where: {
                            assistantConversationUuid: inputArray[0].assistantConversationUuid,
                        },
                    })
                )._max.order || 0) + 1;

            return Promise.all(
                inputArray.map((input) => {
                    return prisma.assistantMessage.upsert({
                        where: { uuid: input.uuid || '' },
                        create: {
                            ...input,
                            order: lastMessageOrder++,
                        },
                        update: input,
                    });
                }),
            );
        });
    }

output: 15 lines of pending promise output each 2 seconds indefinitely
Changing from interactive to array-based transaction method does not change the outcome.
Creating my own timeout over this to prevent it, seems like overkill and it should be rather taken care of by the framework.

Thanks for the reproduction @avallete - this will make it much easier for us to observe what is going on here.

With the release of 3.9.0 we have reworked interactive transactions and it should solve this issue. Could you test with the latest version.
I'm going to close this ticket and we can reopen if the issue is not resolved.

Ping @awinograd @avallete @ssukienn - could you please have a look? Thanks.

@janpio Just tried to upgrade on my MRE, seems and it seems indeed the "hanging forever" issue is resolved, I now see this as an error, then Prisma properly recover and continue my script:
...
app_1       | iteration number: 751
app_1       | an error:  {
app_1       |   is_panic: false,
app_1       |   message: 'Transaction API error: Unable to start a transaction in the given time.',
app_1       |   meta: { error: 'Unable to start a transaction in the given time.' },
app_1       |   error_code: 'P2028'
app_1       | }

However, the "timeout" time of the transaction lock doesn't seem to be affected by this command on postgresql:
SET idle_in_transaction_session_timeout TO XXX
IDK if it's expected behavior and that we have to configure the "timeout time" elsewhere instead.
Anyway, thank's everyone for your work and the time you spent on this issue!

We had a similar issue occur in our production system just today. getting Unable to start a transaction in the given time errors on all prisma queries and which could only be recovered by restarting the affected instance.

Please don't comment on two year old issues, but open a new one and supply all the information it asks for - optimally including a reproduction of your problem. Thanks.


[web] Fractional line width on scaled canvas fails with html renderer

Steps to Reproduce

Run flutter create bug.
Update the files as follows: replace main.dart with the following code

import 'dart:math';
import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key? key, required this.title}) : super(key: key);

  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {

  int _counter = 0;
  final myPainter = MyCustomPainter();

  void _incrementCounter() {
    setState(() {
      _counter++;
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Expanded(child: CustomPaint(painter: myPainter, child:Center())),
            Text('You have pushed the button this many times:'),
            Text(
              '$_counter',
              style: Theme.of(context).textTheme.headline4,
            ),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _incrementCounter,
        tooltip: 'Increment',
        child: Icon(Icons.add),
      ),
    );
  }
}

class MyCustomPainter extends ChangeNotifier implements CustomPainter {

  final sprites = [
    MySprite(1, Offset(1,2)),
    MySprite(2, Offset(-1,2)),
    MySprite(3, Offset(1,-2)),
    MySprite(4, Offset(-1,-2)),
  ];

  @override
  void paint(Canvas canvas, Size size) {
    canvas.save();
    canvas.drawRect(Rect.fromLTWH(0,0,size.width,size.height),
        Paint()..color = Color(0xfffff0e0));
    final range = min(size.width,size.height);
    canvas.translate(size.width/2, size.height/2);
    var sc = range / 13.0;
    canvas.scale(sc,sc);
    for (final s in sprites) {
      s.draw(canvas);
    }
    canvas.restore();
  }

  @override
  bool? hitTest(Offset position) => null;

  @override
  SemanticsBuilderCallback? get semanticsBuilder => null;

  @override
  bool shouldRebuildSemantics(covariant CustomPainter oldDelegate) => shouldRepaint(oldDelegate);

  @override
  bool shouldRepaint(covariant CustomPainter oldDelegate) => true;

}

class MySprite {

  Offset position;
  int number;

  MySprite(this.number, this.position);

  void draw(Canvas canvas) {
    canvas.save();
    canvas.translate(position.dx,position.dy);
    var p = Paint()..color = Color(0xffffc0c0);
    var rect = Rect.fromLTWH(-0.5, -0.5, 1.0, 1.0);
    canvas.drawRect(rect, p);
    p.strokeWidth = 0.1;
    p.color = Colors.red;
    p.style = PaintingStyle.stroke;
    canvas.drawRect(rect, p);
    canvas.restore();
  }

}

Run the program with Chrome using the html renderer
flutter run -d chrome --web-renderer html

Expected results:
With the canvaskit renderer

Actual results:
With the html renderer

Logs

P:\FlutterProjects\htmlbug>flutter run --verbose -d chrome --web-renderer html
[ +158 ms] executing: [P:\flutter/] git -c log.showSignature=false log -n 1 --pretty=format:%H
[ +125 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%H
[   +1 ms] 02c026b03cd31dd3f867e5faeb7e104cce174c5f
[   +3 ms] executing: [P:\flutter/] git tag --points-at 02c026b03cd31dd3f867e5faeb7e104cce174c5f
[  +97 ms] Exit code 0 from: git tag --points-at 02c026b03cd31dd3f867e5faeb7e104cce174c5f
[   +1 ms] 2.2.1
[   +8 ms] executing: [P:\flutter/] git rev-parse --abbrev-ref --symbolic @{u}
[  +68 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{u}
[   +1 ms] origin/stable
[        ] executing: [P:\flutter/] git ls-remote --get-url origin
[  +67 ms] Exit code 0 from: git ls-remote --get-url origin
[   +1 ms] https://github.com/flutter/flutter.git
[ +191 ms] executing: [P:\flutter/] git rev-parse --abbrev-ref HEAD
[  +85 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[   +1 ms] stable
[ +132 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[   +2 ms] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[   +1 ms] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[   +7 ms] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[   +2 ms] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[   +3 ms] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[  +78 ms] executing: p:\programs\AndroidSDK\platform-tools\adb.exe devices -l
[  +72 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[   +1 ms] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[   +2 ms] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[   +3 ms] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[ +107 ms] executing: p:\programs\AndroidSDK\platform-tools\adb.exe devices -l
[  +17 ms] Skipping pub get: version match.
[ +176 ms] Generating P:\FlutterProjects\htmlbug\android\app\src\main\java\io\flutter\plugins\GeneratedPluginRegistrant.java
[  +91 ms] List of devices attached
           emulator-5554          device product:sdk_gphone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:1
[ +136 ms] Launching lib\main.dart on Chrome in debug mode...
[  +16 ms] List of devices attached
           emulator-5554          device product:sdk_gphone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:1
[ +207 ms] Updating assets
[ +124 ms] Waiting for connection from debug service on Chrome...
[ +132 ms] <- reset
[  +10 ms] P:\flutter\bin\cache\dart-sdk\bin\dart.exe --disable-dart-dev
P:\flutter\bin\cache\artifacts\engine\windows-x64\frontend_server.dart.snapshot --sdk-root P:\flutter\bin\cache\flutter_web_sdk/
--incremental --target=dartdevc --debugger-module-names --experimental-emit-debug-metadata -DFLUTTER_WEB_AUTO_DETECT=false
-DFLUTTER_WEB_USE_SKIA=false --output-dill C:\Users\brad\AppData\Local\Temp\flutter_tools.36ee024e\flutter_tool.55d2efa5\app.dill
--libraries-spec file:///P:/flutter/bin/cache/flutter_web_sdk/libraries.json --packages
P:\FlutterProjects\htmlbug\.dart_tool\package_config.json -Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts
--track-widget-creation --filesystem-root C:\Users\brad\AppData\Local\Temp\flutter_tools.36ee024e\flutter_tools.ef52edf9
--filesystem-scheme org-dartlang-app --initialize-from-dill build\e1ffba66f615b3b54e08e2104bac7ab8.cache.dill.track.dill --platform
file:///P:/flutter/bin/cache/flutter_web_sdk/kernel/flutter_ddc_sdk_sound.dill --sound-null-safety
[  +15 ms] <- compile org-dartlang-app:/web_entrypoint.dart
[+29678 ms] Waiting for connection from debug service on Chrome... (completed in 29.8s)
[   +6 ms] Synced 28.2MB.
[   +2 ms] <- accept
[        ] Caching compiled dill
[+1375 ms] [CHROME]:
[   +1 ms] [CHROME]:DevTools listening on ws://<IP_ADDRESS>:55864/devtools/browser/3b2cdb0e-70ce-4fa4-ab79-888e5028a53b
[ +496 ms] DwdsInjector: Received request for entrypoint at http://localhost:55764/main_module.bootstrap.js
[  +30 ms] MetadataProvider: Loading debug metadata...
[  +51 ms] MetadataProvider: Loaded debug metadata (sound null safety)
[  +16 ms] DwdsInjector: Injected debugging metadata for entrypoint at http://localhost:55764/main_module.bootstrap.js
[+2727 ms] ChromeProxyService: Initializing expression compiler for main_module.bootstrap.js with sound null safety: true
[ +100 ms] DevHandler: Debug service listening on ws://<IP_ADDRESS>:50874/nWzDLZnsqeM=/ws

[  +40 ms] This app is linked to the debug service: ws://<IP_ADDRESS>:50874/nWzDLZnsqeM=/ws
[  +15 ms] Debug service listening on ws://<IP_ADDRESS>:50874/nWzDLZnsqeM=/ws
[   +1 ms]  Running with sound null safety
[   +7 ms]   To hot restart changes while running, press "r" or "R".
[   +2 ms] For a more detailed help message, press "h". To quit, press "q".

[√] Flutter (Channel stable, 2.2.1, on Microsoft Windows [Version 10.0.19042.985], locale en-US)
    • Flutter version 2.2.1 at P:\flutter
    • Framework revision 02c026b03c (12 days ago), 2021-05-27 12:24:44 -0700
    • Engine revision 0fdb562ac8
    • Dart version 2.13.1

[√] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at p:\programs\AndroidSDK
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = p:\programs\AndroidSDK
    • ANDROID_SDK_ROOT = p:\programs\AndroidSDK
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

[√] Android Studio (version 4.1.0)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] IntelliJ IDEA Community Edition (version 2021.1)
    • IntelliJ at C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2018.3
    • Flutter plugin version 57.0.5
    • Dart plugin version 211.7233

[√] Connected device (3 available)
    • Android SDK built for x86 (mobile) • emulator-5554 • android-x86    • Android 8.0.0 (API 26) (emulator)
    • Chrome (web)                       • chrome        • web-javascript • Google Chrome 91.0.4472.77
    • Edge (web)                         • edge          • web-javascript • Microsoft Edge 90.0.818.56

• No issues found!

Using canvaskit:

Using html:

 flutter doctor -v
[✓] Flutter (Channel stable, 2.2.1, on Mac OS X 10.15.4 19E2269 darwin-x64,
    locale en-GB)
    • Flutter version 2.2.1 at /Users/dhs/documents/fluttersdk/flutter
    • Framework revision 02c026b03c (4 days ago), 2021-05-27 12:24:44 -0700
    • Engine revision 0fdb562ac8
    • Dart version 2.13.1

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] VS Code (version 1.55.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[✓] Connected device (4 available)
    • Darshan's iphone (mobile)  • 21150b119064aecc249dfcfe05e259197461ce23 •
      ios            • iOS 14.4.1
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729     •
      ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                    •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                                   •
      web-javascript • Google Chrome 91.0.4472.77

• No issues found!

Maybe similar / related to https://github.com/flutter/flutter/issues/77395

Also could be related to #70301.  Unlike that issue, I don't have an easy workaround for this case.


Linker failed while linking libed25519_android.so.

Hi,
I am getting the following error while attempting to build ECC-25519-Android project with Android Studio for the first time:
Error:Execution failed for task ':ecc-25519:linkEd25519_androidArmeabi-v7aDebugSharedLibrary'.

A build operation failed.
Linker failed while linking libed25519_android.so.

What am I missing? I've attached log file. Any help is appreciated. Thanks.
output.txt

@pepelepoe The error is related to not finding -lm_hard.  This flag was removed on
You are probably using Android NDK 15 or higher, change your NDK to Android NDK 14 and it should work.
The hard float ABI was removed from the NDK in r12, but it still exists on r13 and r14.  Only on r15 and moving forward it was officially removed.
Please see: https://android.googlesource.com/platform/ndk/+/master/docs/HardFloatAbi.md and https://groups.google.com/forum/#!topic/kaldi-help/cILizZfqHCk for similar discussions.


Can not run without trainning

sorry for open a new post.
But After finished the train.I dont know which cells to choose to run again that could make the last cell working by skipping the trainning part.
And I dont find how to assign model path for which model to use.

when you launch the gradio interface after the training, it will automatically use the trained model, but do not disconnect from the runtime after the training.
I'm working on making it easier to use and reuse the trained models

OK.Waiting for that then.Thank you for the work.

OK.Waiting for that then.Thank you for the work.

Now you can test your trained at anytime using just the identifier used for it's training, as long as the model is in your Gdrive, all in one click without going through the other cells


fix(ci): make docs preview unique by PR

Description

Breaking Changes

Notes & open questions

Change checklist

[ ] Self-review.
[ ] Documentation updates following the style guide, if relevant.
[ ] Tests if relevant.
[ ] All breaking changes documented.

What exactly is the problem you're trying to solve? Some explanation in the PR description would help me review this!

Thanks for jumping in, I was hoping it works for any branch you set it to (dynamically).
Anyways, the crux of it is this https://github.com/n0-computer/iroh/actions/runs/11105394334/job/30851460274?pr=2756
It seems mostly to be a race between PRs to publish their own. Maybe do a softer version of and do like up to 5 retries with back off.

what if we switch this to a command instead of auto generating it? eg /generate_docs
alternatively, maybe we just push this onto one of our CI machines and run nginx there just hosting these ourselves per branch

Seems to be a common issue with the action we're using to update the branch: https://github.com/peaceiris/actions-gh-pages/issues/1078

This was recommended as a fix to the concurrency issue: https://github.com/peaceiris/actions-gh-pages/issues/998#issuecomment-1742383037
I don't think our problem is the concurrency though. It seems the action has stopped at some point in the last month or so.

Thanks for jumping in, I was hoping it works for any branch you set it to (dynamically). Anyways, the crux of it is this https://github.com/n0-computer/iroh/actions/runs/11105394334/job/30851460274?pr=2756 It seems mostly to be a race between PRs to publish their own. Maybe do a softer version of and do like up to 5 retries with back off.

Yeah actually I think this is a separate issue. If pushing to the generated-docs-preview branch were the problem, then the stuff that's in the branch would still be served correctly, but e.g. the 2760 PR exists on the branch, but isn't served by github for some reason.

Ok, think I've figured it out. Seems like we're hitting some file size limits judging by the action outputs. I'll manually prune some of the old PRs and see if that makes things work again.

Yup that worked just fine now. I'll add a step to clean up the branch every now and then.

@matheus23 This should now also trigger a weekly job to clean up the docs branch. It does risk to remove the preview for PRs that are old but still open, but I guess if you still need or want the docs preview you can easily re-run the job to generate them.

Think this should do it, but cant be 100% sure till we merge and test it out due to the nature of workflows and permissions on GHA.

Why does netsim keep adding comments? 😅 (Is that intentional?)

It's supposed to update them but something is messed up, that's what I'm going to fix next.


ESP-IDF VS Code extension setup fails at import _ssl (IDFGH-13833)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
v5.3.1
Operating System used.
Windows
How did you build your project?
Command line with Make
If you are using Windows, please specify command line type.
None
What is the expected behavior?
I expected the setup to complete successfully.
What is the actual behavior?
Setup halts with the error message:
non zero exit code 1
Traceback (most recent call last):
File "c:\Installs\esp-idf-v5.3.1\esp-idf-v5.3.1\tools\idf_tools.py", line 43, in
import ssl
File "C:\Users\ilewis.platformio\python3\Lib\ssl.py", line 100, in
import _ssl             # if we can't import it, let the error propagate
^^^^^^^^^^^
ImportError: DLL load failed while importing _ssl: The specified procedure could not be found.
Steps to reproduce.
Follow the instructions here:
https://github.com/espressif/vscode-esp-idf-extension/blob/master/docs/tutorial/install.md
Choose 'Express Setup', and 'Find ESP-IDF in your system' point to previously downloaded esp-idf-v5.3.1 folder.
Build or installation Logs.
Using existing C:\Users\ilewis\.espressif\dist\idf-git-2.39.2-win64.zip
Installing C:\Users\ilewis\.espressif\tools\idf-git\2.39.2 ...
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\cmd\git.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\cmd\scalar.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\awk.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\b2sum.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\basename.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\basenc.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\cat.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\chattr.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\cmp.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\comm.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\cp.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\cut.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\cygwin-console-helper.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\dash.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\date.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\diff.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\diff3.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\dirname.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\echo.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\env.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\expr.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\false.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\find.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\gencat.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\getfacl.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\getopt.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\gmondump.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\grep.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\head.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\ls.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\lsattr.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\mkdir.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\mktemp.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-2.0.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-asn1-8.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-cbor-0.9.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-com_err-1.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-crypt-0.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-crypto-1.1.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-fido2-1.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-gcc_s-seh-1.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-gmp-10.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-gssapi-3.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-hcrypto-4.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-heimbase-1.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-heimntlm-0.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-hx509-5.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-iconv-2.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-intl-8.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-krb5-26.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-mpfr-6.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-ncursesw6.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-pcre-1.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-readline8.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-roken-18.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-sqlite3-0.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-wind-0.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\msys-z.dll
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\mv.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\pathchk.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\printf.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\profiler.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\pwd.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\readlink.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\rebase.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\rebaseall
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\rm.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\rmdir.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\sdiff.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\sed.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\sh.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\sort.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\ssh-add.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\ssh-agent.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\ssh.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\tail.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\test.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\touch.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\tr.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\true.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\uname.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\uniq.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\wc.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\bin\xargs.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\etc\profile.d\gawk.csh
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\etc\profile.d\gawk.sh
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\lib\ssh\ssh-keysign.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\lib\ssh\ssh-pkcs11-helper.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\lib\ssh\ssh-sk-helper.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\libexec\getprocaddr32.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\libexec\getprocaddr64.exe
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\assert.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\bits2str.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\cliff_rand.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\ctime.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\ftrans.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\getopt.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\gettime.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\group.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\have_mpfr.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\inplace.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\intdiv0.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\join.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\libintl.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\noassign.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\ns_passwd.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\ord.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\passwd.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\processarray.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\quicksort.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\readable.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\readfile.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\rewind.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\round.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\shellquote.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\strtonum.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\walkarray.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\awk\zerofile.awk
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\licenses\gcc-libs\RUNTIME.LIBRARY.EXCEPTION
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\licenses\libsqlite\LICENSE
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\licenses\openssh\LICENCE
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\licenses\openssl\LICENSE
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\licenses\zlib\LICENSE
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\share\nano\git.nanorc
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\ssl\ct_log_list.cnf
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\ssl\ct_log_list.cnf.dist
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\ssl\openssl.cnf
                                        already exists and was not updated.
Warning File C:\Users\ilewis\.espressif\tools\idf-git\2.39.2\usr\ssl\openssl.cnf.dist
                                        already exists and was not updated.
Extracted C:\Users\ilewis\.espressif\tools\idf-git\2.39.2 ...
Using existing C:\Users\ilewis\.espressif\tools\idf-python\3.11.2

Downloading GDB for Xtensa
Using mirror Github with URL https://github.com/espressif/binutils-gdb/releases/download/esp-gdb-v14.2_20240403/xtensa-esp-elf-gdb-14.2_20240403-x86_64-w64-mingw32.zip
Found xtensa-esp-elf-gdb in C:\Users\ilewis\.espressif\dist
Downloading GDB for RISC-V
Using mirror Github with URL https://github.com/espressif/binutils-gdb/releases/download/esp-gdb-v14.2_20240403/riscv32-esp-elf-gdb-14.2_20240403-x86_64-w64-mingw32.zip
Found riscv32-esp-elf-gdb in C:\Users\ilewis\.espressif\dist
Downloading Toolchain for 32-bit Xtensa based on GCC
Using mirror Github with URL https://github.com/espressif/crosstool-NG/releases/download/esp-13.2.0_20240530/xtensa-esp-elf-13.2.0_20240530-x86_64-w64-mingw32_hotfix.zip
Found xtensa-esp-elf in C:\Users\ilewis\.espressif\dist
Downloading Toolchain for 32-bit RISC-V based on GCC
Using mirror Github with URL https://github.com/espressif/crosstool-NG/releases/download/esp-13.2.0_20240530/riscv32-esp-elf-13.2.0_20240530-x86_64-w64-mingw32.zip
Found riscv32-esp-elf in C:\Users\ilewis\.espressif\dist
Downloading Toolchain for ESP32 ULP coprocessor
Using mirror Github with URL https://github.com/espressif/binutils-gdb/releases/download/esp32ulp-elf-2.38_20240113/esp32ulp-elf-2.38_20240113-win64.zip
Found esp32ulp-elf in C:\Users\ilewis\.espressif\dist
Downloading CMake build system
Using mirror Github with URL https://github.com/Kitware/CMake/releases/download/v3.24.0/cmake-3.24.0-windows-x86_64.zip
Found cmake in C:\Users\ilewis\.espressif\dist
Downloading OpenOCD for ESP32
Using mirror Github with URL https://github.com/espressif/openocd-esp32/releases/download/v0.12.0-esp32-20240318/openocd-esp32-win32-0.12.0-esp32-20240318.zip
Found openocd-esp32 in C:\Users\ilewis\.espressif\dist
Downloading Ninja build system
Using mirror Github with URL https://github.com/ninja-build/ninja/releases/download/v1.11.1/ninja-win.zip
Found ninja in C:\Users\ilewis\.espressif\dist
Downloading IDF wrapper tool for Windows
Using mirror Github with URL https://github.com/espressif/idf_py_exe_tool/releases/download/v1.0.3/idf-exe-v1.0.3.zip
Found idf-exe in C:\Users\ilewis\.espressif\dist
Downloading Ccache (compiler cache)
Using mirror Github with URL https://github.com/ccache/ccache/releases/download/v4.8/ccache-4.8-windows-x86_64.zip
Found ccache in C:\Users\ilewis\.espressif\dist
Downloading dfu-util (Device Firmware Upgrade Utilities)
Using mirror Github with URL https://dl.espressif.com/dl/dfu-util-0.11-win64.zip
Found dfu-util in C:\Users\ilewis\.espressif\dist
Downloading ESP ROM ELFs
Using mirror Github with URL https://github.com/espressif/esp-rom-elfs/releases/download/20240305/esp-rom-elfs-20240305.tar.gz
Found esp-rom-elfs in C:\Users\ilewis\.espressif\dist

Using existing GDB for Xtensa in C:\Users\ilewis\.espressif\tools\xtensa-esp-elf-gdb\14.2_20240403
Using existing GDB for RISC-V in C:\Users\ilewis\.espressif\tools\riscv32-esp-elf-gdb\14.2_20240403
Using existing Toolchain for 32-bit Xtensa based on GCC in C:\Users\ilewis\.espressif\tools\xtensa-esp-elf\esp-13.2.0_20240530
Using existing Toolchain for 32-bit RISC-V based on GCC in C:\Users\ilewis\.espressif\tools\riscv32-esp-elf\esp-13.2.0_20240530
Using existing Toolchain for ESP32 ULP coprocessor in C:\Users\ilewis\.espressif\tools\esp32ulp-elf\2.38_20240113
Using existing CMake build system in C:\Users\ilewis\.espressif\tools\cmake\3.24.0
Using existing OpenOCD for ESP32 in C:\Users\ilewis\.espressif\tools\openocd-esp32\v0.12.0-esp32-20240318
Using existing Ninja build system in C:\Users\ilewis\.espressif\tools\ninja\1.11.1
Using existing IDF wrapper tool for Windows in C:\Users\ilewis\.espressif\tools\idf-exe\1.0.3
Using existing Ccache (compiler cache) in C:\Users\ilewis\.espressif\tools\ccache\4.8
Using existing dfu-util (Device Firmware Upgrade Utilities) in C:\Users\ilewis\.espressif\tools\dfu-util\0.11
Installing package ESP ROM ELFs
Installing tar.gz package ESP ROM ELFs

Traceback (most recent call last):
  File "c:\Installs\esp-idf-v5.3.1\esp-idf-v5.3.1\tools\idf_tools.py", line 43, in <module>

    import ssl

  File "C:\Users\ilewis\.platformio\python3\Lib\ssl.py", line 100, in <module>

    import _ssl             # if we can't import it, let the error propagate
    ^^^^^^^^^^^
ImportError: DLL load failed while importing _ssl: The specified procedure could not be found.

non zero exit code 1

Traceback (most recent call last):
  File "c:\Installs\esp-idf-v5.3.1\esp-idf-v5.3.1\tools\idf_tools.py", line 43, in <module>
    import ssl
  File "C:\Users\ilewis\.platformio\python3\Lib\ssl.py", line 100, in <module>
    import _ssl             # if we can't import it, let the error propagate
    ^^^^^^^^^^^
ImportError: DLL load failed while importing _ssl: The specified procedure could not be found.

More Information.
I ran the doctor command, but the output seems to be too big to include in the issue.

Hi @IanLewis42,
I was not able to reproduce your issue.
But it looks like the PATH ("C:\Users\ilewis\.platformio\python3\Lib\ssl.py") is not from the tools directory and should not be here during the installation, can you try to disable the PlatformIO extension during the installation?

Hi, @jakub-kocka
I have tried what you suggested, disable platformio extension, but it made no difference, the error was the same, I guess because PYTHONHOME and PYTHONPATH are both pointing to C:\Users\ilewis.platformio\python3.
I tried changing both PYTHONHOME and PYTHONPATH  to C:\Users\ilewis.espressif\tools\idf-python\3.11.2
and I got a different error, see below.
Downloading from https://dl.espressif.com/dl/esp-idf/idf_versions.txt
idf_versions.txt has (1) KB
idf_versions.txt progress: 100.00% (0.08 / 0.08) KB
Using existing C:\Users\ilewis.espressif\dist\idf-git-2.39.2-win64.zip
Installing C:\Users\ilewis.espressif\tools\idf-git\2.39.2 ...
Warning File C:\Users\ilewis.espressif\tools\idf-git\2.39.2\cmd\git.exe
already exists and was not updated.
Warning File C:\Users\ilewis.espressif\tools\idf-git\2.39.2\cmd\scalar.exe
already exists and was not updated.

---------

Warning File C:\Users\ilewis.espressif\tools\idf-git\2.39.2\usr\ssl\openssl.cnf.dist
already exists and was not updated.
Extracted C:\Users\ilewis.espressif\tools\idf-git\2.39.2 ...
Using existing C:\Users\ilewis.espressif\tools\idf-python\3.11.2
"C:\Users\ilewis.espressif\tools\idf-python\3.11.2\python.exe -m pip" is not valid. (ERROR_INVALID_PIP)

Hi, @IanLewis42,
Okay, this seems to be an issue with the environment.
You can try to run python -m ensurepip in the PowerShell console for example ... or VS Code terminal and repeat the installation.
I am not sure how you got the existing ESP-IDF environment but in case this does not help, you can try to reinstall the ESP-IDF directory and environment with the IDF installer (https://dl.espressif.com/dl/esp-idf/) - I recommend using the "Universal Online Installer", it will also sets the register values and environmental variables for you. Then proceed with the configuration of the VS Code extension with the installed directory.

Hi @IanLewis42,
I am glad that you get it working 👍
However, the PYTHONPATH variable determines the global Python interpreter used.

In resulting CMD prompt, run 'python -m ensurepip'

This should not be needed because when the installation is completed, for the "resulting" terminals its environment with the interpreter is used (e.g. C:\Espressif\tools\idf-python\3.11.2;C:\Espressif\python_env\idf5.3_py3.11_env) and there it should be included.
Be aware that this might be a little issue when using Python for other use cases not just with ESP-IDF tools that the interpreter will be the one from the PATH you have specified.
Thanks for providing more information, the resolution should be complete, so we will close this issue.


Update in code-snippet on the indexing page.

Changing processor to :active_record to match the context.

Thank you :)


Boost Eidolon Spell Missing Range

Very minor data entry issue, the Boost Eidolon spell in the 11.315 compendium does not include its 100ft range.

Fast fix! :) Also, sorry for not mentioning the system module version, I was very tired and checking the wrong spot 😅


Weird logo placement while using Safari

When using MacOS: in Safari, the logo is displaced, and shown like this:

Using Google Chrome, it works fine.

Yep, can confirm that there's an issue with WebKit browsers. Tested on macOS and Linux (Epiphany)

Seems to be ok on iOS webkit.

I'm still getting this in the singleton docker container. When are these containers updated with the latest?

Probably on tuesday :)


Handling consistency issues with messages in cast threads

Problem
Take a cast thread that has 4 layers:  OP, child, gchild, ggchild. Each child cast is linked to its parent which eventually connects back to the OP. This creates a couple of problems:

If the child cast goes missing from the network then the gchild and the ggchild are left in a dangling state and can never be associated back to the thread.

If the child was written by bob, and the node has chosen not to subscribe to bob, it could not construct the thread without fetching bob's child message from his server.

Ideally, clients would always be able to reconstruct the thread and show missing messages accordingly.
Proposed Solution
We introduce a new cast sub type (reply cast) that must contain:

A URI reference to the parent (present today)
A URI reference to the thread OP
A generation clock which increments with every generation (child = 1, gchild = 2 etc.)

Nodes currently merge a thread sequentially from OP to ggchild, they can now do it in parallel after ingesting the OP.  They can also skip messages from users they don't subscribe to. Our data storage requirements go up 48 bytes a cast and we have a few more checks in our merge set.
Counter Proposal A
The alternative is to stick with the current approach where he gchild if the child cannot be provided. Messages in a thread will always propagate in a consistent state or not at all, at the expense of making the sync process less parallelizable within threads. This solves (1) and possible even reduces the occurrence of inconsistency issues which would be helpful for clients.
It does not solve (2) since bob's message would never be ingested. In this case, the node sync would have to specially fetch bob's message using the URI provided in order to construct the thread. It does not need to retain the whole message once verified, just enough to construct the thread.
The main argument against this approach is that it forces us to choose between no sharding (i.e. you can't ignore bob) which is undesirable for scaling or having dependencies on messages from users nodes don't care about, which can create more choke points when syncing data.

Regarding:

We introduce a new cast sub type

Alternatively, we could define a single cast type that includes all the necessary information so we don't need to define multiple types. An original cast has a generation of 0, the thread OP merkle root is its own merkle root, and there is no reply parent defined.
Unless we expect a lot more functionality to be present in a reply cast vs an original cast, perhaps there's no need to define a separate cast type as it would just introduce complexity in the protocol.
Were there other reasons we might want a separate type?

The primary argument for not requiring props would be space savings. Adding the generation and merkle root would is an extra 90 bytes on top level messages. If you assume each DAU has 2 OP posts per day at 10k DAU's you need (90 * 2 * 365 * 10000) / (1024^2) = ~ 625 MB per year per node.

Sorry, I wasn't being clear. What I mean is that the TypeScript type would simply include optional keys like conversationMerkleRoot,replyParentMerkleRoot, and generation. This way you still have a single "type" (from a TypeScript-centric viewpoint) but the protocol can then infer that the lack of any of these keys means the cast is an OP. There's no additional data that needs to be stored, and we're able to use the same type for both OPs and replies.

Yes, supportive of optional properties.

On further reflection, the proposed solution has more downsides that make it undesirable:

The parent (URI reference to the direct parent) and root (URI reference to the thread OP) can come into conflict if root is accidentally or intentionally pointed at a cast that is in a different branch than the parent. There are cases where we cannot deterministically know this (e.g. if an ancestor of the the parent and the conflicting root are both excluded because of sharding).

The generation clock can similarly create conflicts that are hard to check if set incorrectly by the client.

I propose closing this issue and reverting to the behavior of not rendering dangling messages in a thread. In many cases, they are irrelevant without the parent. These messages should still be synced and preserved so that the URLs to them will stay valid.

@pfletcherhill and @danromero  here is a simple example illustrating an edge case where a malicious actor can create confusion by retracting their messages from a thread in a case where the root is available.

If the root property is made available, clients will almost certainly attempt to implement rudimentary threading where floating messages are displayed. In most cases, these are fine but in some rare cases they can lead to catastrophic confusion which undermines trust in clients, and by extension, the network.
My view is that the protocol/hubs should take a stricter approach only claiming that something is part of a thread if the entire conversation is visible and there is no potential for confusion about the intent behind messages. (i.e., close this ticket and do not add the root / reference to thread OP)
The downside of this approach is that:

A missing child cast can permanently disconnect a grand-child+ reply from the parent
Hubs will have to do more work to construct threads - every time a new message appears we have to look at all disconnected messages to see if any of them have a connection to this message. With a root, we would scan a much smaller set to make that determination.

It is possible (but not certain) that there is a more elegant datastructure that could make stronger guarantees about the exact position of a cast in a tree even if all the other messages were missing. If such a data structure is feasible it would alleviate (1) and (2). But this will take a long time to get right, and we should extend the protocol if/when it is discovered.

after further discussion, we've decided to close this ticket.
the tradeoff is that hubs will not, at least for the current version, offer threading API's which must be constructed by an indexer that sits on top of a hub.


Improved formatting of docker-compose-hanoi-no-secty-arm64.yml

Till the Geneva release, the docker-compose files had a proper structure in definitions
An example would be :
version:
environment variables
volumes:
services:
consul:
redis:
system:
notifications:
metadata:
data:
command:
scheduler:
app-service-rules:
rulesengine:
device-virtual:
device-rest:
device-random:
device-mqtt:
device-modbus:
device-snmp:
networks:
But in the Hanoi release, this structure is not followed and makes the process of development tedious.

I would like to contribute to this issue

@punidramesh , the compose files are now generated from multiple compose files using the docker-compose config command. This command outputs the all the services and their properties/setting alphabetically. There is noting improper about this structure.
See the compose-builder README here: https://github.com/edgexfoundry/developer-scripts/blob/master/compose-builder/README.md

Oh I see. I shall be closing this issue now.


Hamcrest tries to access missing attribute array.is-empty

is-empty attributes has been recently moved to collections.list object.
However the lib still tries to use it in array:
# Main object for assertions
[actual matcher reasons...] > assert-that

  if. > @
    matcher.match actual
    TRUE
    sprintf
      "%s\nExpected: %s\n     but: %s"
      if.
        reasons.is-empty
...

assert-that object needs to be updated.

@mximp Indeed, thanks!

@mximp I think we can close this issue


Ask for one of the core team members coming to give a little 10-15min intro to rust talk

I reached out. I'll keep you posted.

sounds like this was taken care of


[AppConfig] Fix flaky tests

The test for accept_time is pretty flaky, it assumes the number of configuration settings on one day ago should be less than the number of configuration settings listing at current time(without passing parameter accept_time). The assumption is not always correct, for example, when there are some configuration settings created and deleted one day ago, we can still get the configuration settings in the list result with accept_time equals to one day ago, but it is no longer in the list result at current time. So it doesn't make sense to compare all_inclusive vs exclude_today. Also updated the sample test in docstring.
Updated the description of parameter accept_time in docstring to be more clearer.
Since the configuration setting's last updated time is saved in utc time, so changed to use utc time in docstring samples.

API change check
API changes are not detected in this pull request.


[Feature][Plugin] Refactor seatunnel-connector structure

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description
The sink-plugin-module and source-plugin-module under seatunnel-connectors  should be merge into a connector module

refactor before :

refactor after :

Usage Scenario
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

LGTM

From my view, I think we can:

merge source and sink to a combined connector, because the source and sink of a specific storage probably dependent on  some common libraries

keep spark and flink connectors separately because we may directly use connectors from spark, flink if possible, and users of one of spark or flink connectors do not intend to load dependencies from both spark and flink.

done


Describe your problem
docker compose down -v docker compose up
above command doesn't work

Thanks for following!

change docker/entrypoint.sh: 26: sleep 60 -> sleep 180
add a line after docker/docker-compose.yml: 109
- ./entrypoint.sh:/ragflow/entrypoint.sh
docker compose down -v
docker compose up

Thanks for following!

change docker/entrypoint.sh: 26: sleep 60 -> sleep 180
add a line after docker/docker-compose.yml: 109
- ./entrypoint.sh:/ragflow/entrypoint.sh
docker compose down -v
docker compose up

docker-compose.yml only has 30 lines. where to insert?
1 version: '2.2'
 2
 3 include:
 4   - path: ./docker-compose-base.yml
 5     env_file: ./.env
 6
 7 services:
 8   ragflow:
 9     depends_on:
10       mysql:
11         condition: service_healthy
12       es01:
13         condition: service_healthy
14     image: infiniflow/ragflow:v1.0
15     container_name: ragflow-server
16     ports:
17       - ${SVR_HTTP_PORT}:9380
18       - 80:80
19       - 443:443
20     volumes:
21       - ./service_conf.yaml:/ragflow/conf/service_conf.yaml
22       - ./ragflow-logs:/ragflow/logs
23       - ./nginx/ragflow.conf:/etc/nginx/conf.d/ragflow.conf
24       - ./nginx/proxy.conf:/etc/nginx/proxy.conf
25       - ./nginx/nginx.conf:/etc/nginx/nginx.conf
26     environment:
27       - TZ=${TIMEZONE}
28     networks:
29       - ragflow
30     restart: always

My version is 2.2. Could you tell your version?

Thanks for following!

change docker/entrypoint.sh: 26: sleep 60 -> sleep 180
add a line after docker/docker-compose.yml: 109
- ./entrypoint.sh:/ragflow/entrypoint.sh
docker compose down -v
docker compose up

docker-compose.yml only has 30 lines. where to insert?
1 version: '2.2'
 2
 3 include:
 4   - path: ./docker-compose-base.yml
 5     env_file: ./.env
 6
 7 services:
 8   ragflow:
 9     depends_on:
10       mysql:
11         condition: service_healthy
12       es01:
13         condition: service_healthy
14     image: infiniflow/ragflow:v1.0
15     container_name: ragflow-server
16     ports:
17       - ${SVR_HTTP_PORT}:9380
18       - 80:80
19       - 443:443
20     volumes:
21       - ./service_conf.yaml:/ragflow/conf/service_conf.yaml
22       - ./ragflow-logs:/ragflow/logs
23       - ./nginx/ragflow.conf:/etc/nginx/conf.d/ragflow.conf
24       - ./nginx/proxy.conf:/etc/nginx/proxy.conf
25       - ./nginx/nginx.conf:/etc/nginx/nginx.conf
26     environment:
27       - TZ=${TIMEZONE}
28     networks:
29       - ragflow
30     restart: always

My version is 2.2. Could you tell your version?

version: '2.2'
include:

path: ./docker-compose-base.yml
env_file: ./.env

services:
ragflow:
depends_on:
mysql:
condition: service_healthy
es01:
condition: service_healthy
image: infiniflow/ragflow:v1.0
container_name: ragflow-server
ports:
- ${SVR_HTTP_PORT}:9380
- 80:80
- 443:443
volumes:
- ./entrypoint.sh:/ragflow/entrypoint.sh
- ./service_conf.yaml:/ragflow/conf/service_conf.yaml
- ./ragflow-logs:/ragflow/logs
- ./nginx/ragflow.conf:/etc/nginx/conf.d/ragflow.conf
- ./nginx/proxy.conf:/etc/nginx/proxy.conf
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
environment:
- TZ=${TIMEZONE}
networks:
- ragflow
restart: always
Here is my profile. I hope it helps


@acset macro not interacting well with unique_indexed

ACSets that are created with the @acset macro can generate acset instances that violate the uniqueness condition (because it seems to be accessing random bytes of memory). E.g.
@present TheoryLabeledGraph <: TheoryGraph begin
  Label::AttrType
  label::Attr(V,Label)
end
@acset_type IndexedLabeledGraph(TheoryLabeledGraph, index=[:src,:tgt],
                                unique_index=[:label]) <: AbstractGraph

z=@acset IndexedLabeledGraph{Int} begin V=7 end
z[:label]

<PHONE_NUMBER>
          3
     296964
          3
          4
          0
          0

It should probably fail if the unique attributes are not specified.

Because integer vectors cannot be undef, I agree that the correct behavior here is probably to fail with an error that the unique-indexed attribute was not initialized.
In the original acset implementation, I was at some pains to try to deal with the subtle issues around initialization and indexing. It's tricky.

This issue seems to have been resolved by the most recent acsets refactor:
@acset_type IndexedLabeledGraph(SchLabeledGraph, index=[:src,:tgt],
                                       unique_index=[:label]) <: AbstractGraph
z=@acset IndexedLabeledGraph{Int} begin V=7 end
IndexedLabeledGraph{Int64} with elements V = 1:7, E = 1:0
┌───┬─────────┐
│ V │   label │
├───┼─────────┤
│ 1 │ nothing │
│ 2 │ nothing │
│ 3 │ nothing │
│ 4 │ nothing │
│ 5 │ nothing │
│ 6 │ nothing │
│ 7 │ nothing │
└───┴─────────┘


HB-5511: cherry-pick CHANGELOG.md updates from <IP_ADDRESS>.0 release

<IP_ADDRESS>.0 is a simple backward release (PR #35), thus we only need to cherry-pick CHANGELOG.md.

Close this backward release and cherry-pick the CHANGELOG.md changes only (PR #35).


Please upload 4.17.44 to npmjs.com

The version on npmjs.com is still 4.17.43, please upload the latest release.
https://www.npmjs.com/package/eonasdan-bootstrap-datetimepicker

done


x86 rflags appears to be disjoint from individual flags

Version and Platform (required):

Binary Ninja Version: 3.3.3996 Personal (Build ID e34a955e)
OS: macOS
OS Version: 13.4 (22F5037d)
CPU Architecture: Apple silicon

Bug Description:
Some instructions update rflags, which invalidates the results of individual flags such as flag.z. This results in unsound decompilation.
Steps To Reproduce:
Please provide all steps required to reproduce the behavior:

Load this code:

	xor eax, eax
	inc eax
	test rax, rax
	xtest
	jne skip
	ud2
skip:
	syscall

Expected Behavior:
Binary Ninja decompiles this to something like
int64_t sub_0()
{
    if (_xtest() != 0) {
        syscall();
        /* undefined */;
    }
    trap(6);
}

Additional Information:
The actual code looks like this:
int64_t sub_0()
{
    _xtest();
    syscall();
    /* undefined */;
}

Looking at the LLIL, it's clear that the condition was lifted early and rflags is just not used at all for anything.
0  eax = 0
1  eax = eax + 1
2  cond:0 = rax != 0
3  rflags = _xtest()
4  if (cond:0) then 5 @ 0xe else 7 @ 0xc

5  syscall
6  undefined

7  trap(6)

(Yes, this is from the Plaid CTF challenge.)

The dynamics of this issue:

we are missing the lifting for the xtest instruction
we fallback to lift it as an intrinsic, in which case xed provides input/output information for it
xed reports the rflag is the output, and we simply follow it

Providing a lifting for the xtest instruction should fix this issue

Well, I can provide several more instructions that you don’t provide semantics for. Obviously I’d love for you to lift them correctly but until then my 2¢ would be that you should probably update your mapping for clobbers to be more conservative.

If you would like to let us know the list of incorrectly lifted instructions, it should be relatively easy to fix them.
Like I said, the faulty/inaccurate information actually comes from xed, so it is not easy for us to update the clobbers


Update to wit-bindgen 0.20.0

Keepin' up to date.
I don't know what's going on with cargo vet and the entries here. I can't run cargo vet on main and I'm not sure why. I ran cargo vet regenerate audit-as-crates-io and it flipped some flags to false and all of a sudden I could run it, so I stopped asking questions.

That's wild, when I ran the exact same command yesterday, it flipped those to true.

Yeah I'm not sure what's going on either, its says that the crate doesn't exist on crates.io  but it does, so it may be some sort of odd scenario we're running into...


Update ru_RU.lang

Git merge isn't liking me today.  I manually updated the file from my sandbox.  If you could take a look at what I checked in to make sure it is good I would appreciate it.  https://github.com/OreCruncher/DynamicSurroundings/blob/master/src/main/resources/assets/dsurround/lang/ru_ru.lang

All is well. But can you fix line 65 on this one?
dsurround.cfg.asm.EnableSMSync=Включить синхронизацию менеджера звука

Sure!  Thanks for the translations - I appreciate it.