[MM-27132] Get user group mention keys for channel

Summary

Modify user mention keys selector to look at group mention keys associated to that specific channel
Move group mentions permission checks out of redux as certain selectors require the user to get the list of groups even if they cannot mention (eg. highlighting)

Ticket Link

https://mattermost.atlassian.net/browse/MM-27132

Related Pull Requests

https://github.com/mattermost/mattermost-redux/pull/1214

Taking over this PR for @fmunshi while he's on PTO.

@srkgupta I have fixed the issue mentioned above.

Rohitesh is off this week so I've swapped the QA reviewer to Ogi,

Hi @ogi-m, In order to test this you will need a user that belongs to two different ldap groups (both with allow_reference set to true), and a group synced channel that only has one of those two groups synced to it.
Once you have that the expected scenario can be reproduced, when the synced group is mentioned it should be highlighted and when the non synced group is @ mentioned it should just appear as a blue keyword and not have any highlight

@fmunshi
Error trying doing the automated Cherry picking. Please do this manually
+++ Updating remotes...
Fetching upstream
Fetching origin
+++ Updating remotes done...
+++ Creating local branch automated-cherry-pick-of-mattermost-webapp-#6133-upstream-release-5.27-1598633675
Branch 'automated-cherry-pick-of-mattermost-webapp-#6133-upstream-release-5.27-1598633675' set up to track remote branch 'release-5.27' from 'upstream'.

+++ About to attempt cherry pick of PR #6133 with merge commit ba300338d0aedc3a2e79fbb7244a91308ebda131.

+++ Conflicts detected:

UU package-lock.json
UU package.json

+++ Aborting in-progress git cherry-pick.

+++ Returning you to the master branch and cleaning up.


[SPARK-9319][SPARKR] Add support for setting column names, types

Add support for for colnames, colnames<-, coltypes<-
Also added tests for names, names<- which have no test previously.
I merged with PR 8984 (coltypes). Clicked the wrong thing, crewed up the PR. Recreated it here. Was #9218
@shivaram @sun-rui

Merged build triggered.

Merged build started.

Test build #45719 has started for PR 9654 at commit c102bc0.

Test build #45719 has finished for PR 9654 at commit c102bc0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45719/
Test PASSed.

Merged build triggered.

Merged build started.

Test build #45825 has started for PR 9654 at commit 21f36e0.

Merged build triggered.

Merged build started.

Test build #45826 has started for PR 9654 at commit 829893d.

Test build #45825 has finished for PR 9654 at commit 21f36e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class JavaTrackStateDStream[KeyType, ValueType, StateType, EmittedType](\n

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45825/
Test PASSed.

Merged build finished. Test PASSed.

Test build #45826 has finished for PR 9654 at commit 829893d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * case class StddevSamp(child: Expression,\n  * case class StddevPop(\n  * class TypedColumn[-T, U](\n  * class JavaTrackStateDStream[KeyType, ValueType, StateType, EmittedType](\n

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45826/
Test PASSed.

any more comment?

Test build #46380 has started for PR 9654 at commit f0419ac.

Test build #46380 has finished for PR 9654 at commit f0419ac.

This patch fails R style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46380/
Test FAILed.

Test build #46382 has started for PR 9654 at commit 7e139f4.

Test build #46382 has finished for PR 9654 at commit 7e139f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46382/
Test PASSed.

updated from comments.

@sun-rui any more comment?

@sun-rui ?

Will take a look soon

Could we please review this? I think it would be great to get matching coltype<- in Spark 1.6

Test build #46831 has started for PR 9654 at commit e399acd.

Test build #46831 has finished for PR 9654 at commit e399acd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46831/
Test PASSed.

Merged build finished. Test PASSed.

Couple of minor comments, but otherwise change looks good.

Test build #46845 has started for PR 9654 at commit d555cf8.

Test build #46845 has finished for PR 9654 at commit d555cf8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46845/
Test PASSed.

Merged build finished. Test PASSed.

LGTM. Thanks @felixcheung


any known issues with 2.6.4.rc2?

As I'd like to use the gem mime-types in version 3.x, I'd want to upgrade mail 2.6.3 to 2.6.4.rc2 (or a release, assuming it might be available shortly). The old version is restricting mime-types to < 3.
I've looked through the open issues and none specifically referenced 2.6.4. Is 2.6.4 ready for release, or are there issues that suggest not using it in production yet?
Thanks!

Haven't heard any issues! But also haven't heard from many trying the gem. We'll release shortly.

@jeremy Thanks for the info!

2.6.4 is released.

thanks!


Move dependencies to peerDependencies

To prevent shipping all the required dependencies and bloating consumers with different versions of all the dependencies, move the required dependencies to peerDependencies.
Basically defining the dependencies like styled-components in the dependencies object will require everyone to install a certain version of styled-components even though a consumer may already have a different version installed which suffices, this prevents that from happening.

I'm afraid that you don't really understand the concept of peerDependencies.
Of these dependencies, the only one that it would be reasonable to be declared as a peer-dependency is prop-types, but it turns out that it's recommended by the authors not to depend on it as a peer-dependency.
The rest of the dependencies are just not peerDependencies, they are dependencies of this package. They are not peer-dependencies. The supposed problem that you are mentioning:

a consumer may already have a different version installed which suffices, this prevents that from happening

could be addressed using yarn, the yarn.lock file will prevent that from happening, and I'm pretty confident that the latests versions of npm will also take care of that issue.

Hmm guess it is a problem on my end then, cause I am using yarn but see the dependency (styled-components) two times in my webpack bundle. (using a caret in package.json).
Then again for the prop-types etc. I understand that those are better in the normal dependencies object but isn't a library like styled-components more suited to be placed in a peerDependency since most users already have it as a dependency?

Ups!! I didn't notice that styled-components was defined as a dependency... It does make sense to have it as a peer-dependency, absolutely! Sorry, I saw that dependencies became peerDependencies and I thought that was insane... But yeah, I think that styled-components should be a peer-dependency. But, then instead of being defined like ^2.2.0 we should probably figure out the correct version range that's compatible with grid-styled... Maybe >=2.0.0? Not sure.

Yeah putting all dependencies as peerDependencies is maybe a bit overkill :sweat_smile: my bad.
I'm not sure which versions of styled-components it works with but i'm positive it works on >= 2.1.1.

Closes #64

What is the status on the release of this issue?

If you ask me, the maintainer isn't maintaining grid-styled anymore (last commit seems 4 months ago?), I myself just forked grid-styled and used a git reference in my package.json

First off, thanks for the PR!
With something like styled-components as a dependency, if your app depends on a compatible version of styled-components, an npm install should use the same instance for both your app and this package. To operate as a standalone package, grid-styled requires styled-components as a dependency. There are tradeoffs for including styled-components both as a dependency and a peerDependency, but I'd rather keep it as a dependency since I find that the tradeoffs (generally due to misconfigured build setups) better than the alternative of having a package that doesn't include it's dependencies and can appear broken.
Hope that makes sense!


Is there a way to turn off rich Status spinner

Trying to use get_by_accession_number in a loop wrapped by tqdm progress bar (or any progress bar). But the rich.Status spinner inside the loop is interrupting the progress bar causing repeated bars printed out. Is there a recommended way to avoid this or turn off the the rich.status spinner?

Also wonder what is the source of CUSIP ticker mapping file - ct.pq? Is it a static file?
Thanks!

Added a parameter show_progress=False to 3.2.4 that you can use to turn off the progress bar
get_by_accession_number does a search so can be slower that doing a filter
filings.filter(accession_number=['0001888524-24-016584', '0001888524-24-016606'])
for filing in filings:
    ....

The cusip ticker file is a data file rebuilt roughly twice a month from the SEC's failure to deliver list. It's not critical to update to get the latest unless you absolutely never want to miss a new Cusip

Fixed

Looks great thanks!


chore: validate content URL

Fix #665
Fix #688
Fix #698
Fix #700

:tada: This PR is included in version 6.42.0 :tada:
The release is available on:

GitHub release
v6.42.0

Your semantic-release bot :package::rocket:


WIP: Enforce script size limit when hashing scripts

Patch 1 is https://github.com/rust-bitcoin/rust-bitcoin/pull/2812 because the api files a a bit messed up.
WIP because I have unanswered questions please (see below), also note my claims are made from things I learned reading BIPs 16 and 141 while doing this, I do not claim authority.
Done as part of #2784
There are two limits that the Bitcoin network enforces in regard to hashing scripts

For P2SH the redeem script must be less than 520 bytes (bip 16)
For P2WSH the witness script must be less than 10,000 bytes (bip 141)

Currently we are only enforcing the p2sh limit when creating an address with Address::p2sh.
There are various ways to create addresses from script hashes and if users manually hash a script then use the ScriptHash (or WScritpHash) our APIs assume the script that was hashed is valid. This means there is the potential for users to get burned by creating addresses that cannot be spent, something we would like to avoid.

Add fallible constructors to ScriptHash and WScriptHash
Add TryFrom impls as well to both types
Remove the From impls

QUESTION:

Includes an unchecked constructor for ScriptHash but I do not know when this would be used or if we want one for WScriptHash (loosely related to the ongoing "wrapper hash types should not be general purpose" because an unchecked version allows hashing any script).

Orthogonal to this work but related because of the 520 byte limit is enforcing the maximum witness stack element size. This is explicitly left to https://github.com/rust-bitcoin/rust-bitcoin/issues/2793
Close: #2794

Going to ignore this until #2793 is resolved, so that discussion is only in the one place.

Gawd damn, the api files are sweet - so easy to check that things are as they should be.

Includes an unchecked constructor for ScriptHash but I do not know when this would be used or if we want one for WScriptHash (loosely related to the ongoing "wrapper hash types should not be general purpose" because an unchecked version allows hashing any script).

There are two usecases that I see:

You have messed things up and are trying to use rust-bitcoin to represent the mess you've found yourself in. "Forensics".
You are hashing a pre-checked or fixed vector or something that is obviously within the size limits and don't want to .unwrap().

Added patch 2 an 3, please check the argument presented in 2 in case I'm missing something (removes a test assertion).

Yeah, fine to drop this test. It tests a common sequence of API calls for constructing a p2sh-wrapped segwit scriptpubkey, but you're right that there's really no way that this test can fail if the other tests pass.

9f7ee7b15e7fd0e2989f595e739d9a22133d83d8 looks good except that the new p2wsh_from_hash functions should take the hash by value, not by reference.
Then e.g. Script::to_p2wsh which you currently have as
        let hash = self.wscript_hash()?;
        Ok(ScriptBuf::new_p2wsh(&hash))

can be just self.wscript_hash.map(ScriptBuf::new_p2wsh).

Done, had to explicitly add the ref inside the call to map though because the ScriptBuf constructors take hashes by reference. I'll leave those to be fixed up as I pull each patch out of #2404


Update documentation about snapshot tests

This bit of the README got left behind in one of our refactors.

Messages

:book:

Thanks for contributing to our documentation. We :heart: our documentarians!

:book:
:sparkles: Thanks for your contribution to Shields, @PyvesB!

  Generated by :no_entry_sign: dangerJS against 9a49b78853845e6a73324103a168d6d215d17e79


[Enum Request] Table 5407 Prod. Order Component, field 44 Calculation Formula

Please create an enum for Table 5407 Prod. Order Component, field 44 Calculation Formula.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


[web] exceptions on mouse move when window loses focus

On desktop chrome for Linux, running a flutter application in debug mode and then losing window focus causes all subsequent mouse moves to throw the assertion below:
Error:
dart_sdk.js:5312 Uncaught Error: Assertion failed: org-dartlang-sdk:///flutter_web_sdk/lib/_engine/engine/pointer_converter.dart:271:18
!state.down
is not true
    at Object.throw_ [as throw] (dart_sdk.js:4021)
    at Object.assertFailed (dart_sdk.js:3969)
    at _engine.PointerDataConverter.new.convert (dart_sdk.js:145428)
    at _engine.PointerAdapter.new.[_convertEventToPointerData] (dart_sdk.js:145120)
    at dart_sdk.js:145088
    at HTMLElement.<anonymous> (dart_sdk.js:144990)

Doctor:
[✓] Flutter (Channel master, v1.13.1-pre.94, on Linux, locale en_US.UTF-8)
    • Flutter version 1.13.1-pre.94 at /usr/local/google/home/jonahwilliams/Documents/flutter
    • Framework revision 36e599eb5d (12 hours ago), 2019-12-11 07:27:13 +0100
    • Engine revision 12bf95fd49
    • Dart version 2.7.0 (build 2.7.0-dev.2.1 8b8894648f)

[!] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at /usr/local/google/home/jonahwilliams/Android/Sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-29, build-tools 29.0.2
    • Java binary at: /opt/android-studio-with-blaze-3.5/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)
    ! Some Android licenses not accepted.  To resolve this, run: flutter doctor --android-licenses

[✓] Chrome - develop for the web
    • Chrome at google-chrome

[✓] Linux toolchain - develop for Linux desktop
    • clang++ 8.0.1
    • GNU Make 4.2.1

[!] Android Studio (version 3.4)
    • Android Studio at /opt/android-studio-with-blaze-3.4
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    ✗ Unable to find bundled Java version.
    • Try updating or re-installing Android Studio.

[!] Android Studio (version 3.3)
    • Android Studio at /usr/local/google/home/jonahwilliams/Documents/android-studio
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)

[!] Android Studio (version 3.5)
    • Android Studio at /opt/android-studio-with-blaze-3.5
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)

[✓] IntelliJ IDEA Community Edition (version 2019.1)
    • IntelliJ at /usr/local/google/home/jonahwilliams/Documents/idea-IC-191.7479.19
    • Flutter plugin version 36.1.4
    • Dart plugin version 191.7830

[!] IntelliJ IDEA Ultimate Edition (version 2018.3)
    • IntelliJ at /opt/intellij-ue-2018.3
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    • Dart plugin version 183.5912.23
    • For information about installing plugins, see
      https://flutter.dev/intellij-setup/#installing-the-plugins

[✓] IntelliJ IDEA Ultimate Edition (version 2019.1)
    • IntelliJ at /opt/intellij-ue-2019.1
    • Flutter plugin version 39.0.2
    • Dart plugin version 191.8423

[!] VS Code (version 1.39.2)
    • VS Code at /usr/share/code
    ✗ Flutter extension not installed; install from
      https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (3 available)
    • Linux      • Linux      • linux-x64      • Linux
    • Chrome     • chrome     • web-javascript • Google Chrome 79.0.3945.79
    • Web Server • web-server • web-javascript • Flutter Tools

Could be related to https://github.com/flutter/flutter/issues/46231

I am also seeing this on desktop linux using the glfw shell, not web. So I don't think this is just a web issue.

This assert in the engine is wrong: https://github.com/flutter/engine/blob/5a25a281743976f271425c44ada62f8a9f9cd73a/lib/ui/window/pointer_data_packet_converter.cc#L182
Here is the sequence of events:

Run application
Cause application to lose focus
Click on the application
=> observe assert fires

glfw delivers a mouse enter event and then a mouse release event. This is correct behavior that is coming from X11. The mouse press gives you focus but occurs before you have it. Therefore as the mouse press event happened while another window had focus your application does not receive it. It does receive the mouse release event because you now have focus.
An application should be able to handle a mouse release without a corresponding mouse press. As far as I can tell this is the only scenario where this happens, but I think the up event without a down should just get thrown away. This could also happen with keypresses presumably.
It might be possible to do something with the focus callback to try to detect this, but it looks like you get an add, a move, then an up all after you get focus. So it isn't clear exactly how you tie the next up to that regaining of focus. You could on focus set a bit that says you can get on up for free and ignore one such event and clear that bit. Then the assert can be state.isDown and that bit is not set. At least then you could still get some useful debugging info.

FYI @dkwingsmt

Related (insofar as it relates to X11 pointer events not quite meeting our expectations) to https://github.com/flutter/flutter/issues/74939.
Generally the expectation is that it's the embedders' responsibility to handle any event suppression/synthesis to ensure we're behaving in accordance with the contract between the engine and the framework.
/cc @chunhtai who did the rewrite for pointer event consistency between embedders and the framework.

this is not linux specific
i was checking documentation and stumbled upon this interactive example https://flutter.dev/docs/cookbook/forms/validation#interactive-example
this reproduces only with this specific example only (i'm on windows)
steps to reproduce:

type something to the input field
press submit
drag the snackbar down so cursor goes out of the flutter iframe

i tried it on Snacbar with a different sample - no problem

sample
import 'package:flutter/material.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    final appTitle = 'Form Validation Demo';

    return MaterialApp(
      title: appTitle,
      home: Scaffold(
        appBar: AppBar(
          title: Text(appTitle),
        ),
        body: Builder(
          builder: (context) => GestureDetector(
              onTap: (){
                ScaffoldMessenger.of(context).showSnackBar(SnackBar(
                  content: Text('drag me down'),
                ));
              },
              child: Container(color: Colors.red)
            ),
         )
      ),
    );
  }
}


KAFKA-10120: Deprecate DescribeLogDirsResult.all() and .values()

As per KIP-621. Also added some tests in KafkaAdminClientTest

@cmccabe, @omkreddy, @mimaison, @dajac, @dongjinleekr since you all voted on the KIP feel free to review.

Thats for the review @dajac. I've addressed your comments.

@dongjinleekr thanks for the review, amended. Also rebased for conflict.

@dongjinleekr ah, thank you, I hadn't noticed that that was the norm.

ok to test

@mimaison done, thanks.

retest this please

ok to test

Failures look unrelated:

org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest > shouldUpgradeFromEosAlphaToEosBeta[true] FAILED
org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest > testOneWayReplicationWithAutorOffsetSync1

@tombentley Congratulations! :congratulations: @omkreddy @mimaison Thanks again for the detailed review, as usual! :smiley:


Errors with sqitch plan between versions

I have a problem with sqitch plans (and databases) between different versions. We currently have a 2.0 branch and a master branch (which represents a future 2.1 version) and there will be changes on both branches. Changes on the 2.0 branch will be merge into master (bugfix changes) and master will have feature changes, but when merging branches it isn't possible to deploy changes with sqitch. This is an example case:
2.0 branch is created, this is the state:

2.0 branch
master branch

change A
change A

change B
change B

change C
change C

Bugfixes are created on 2.0 and new features developed on master. A database per version is build from scratch

2.0 branch
master branch

change A
change A

change B
change B

change C
change C

bugfix A
Feature A

We merge 2.0 in master (to transfer bugfixes to master). This is the result:

2.0 branch
master branch

change A
change A

change B
change B

change C
change C

bugfix A
Feature A

bugfix A

But here is where the problem occurs, the merge like this is correct for the master database. Bugfix A must be include after feature A, because if bugfix A is before feature A sqitch deploy will exit with an error that it can not find bugfix A in the plan file (quite strange error, but it's an wrong error message i think)
So this will block us from upgrading databases from version 2.0 up to 2.1
How can we solve this problem or can we find a solution in sqitch so that the order of the plan file doesn't matter but it just follows the requirements. I hope my explanation is a bit clear otherwise i will be more than happy to explain it a bit more!
Thanks in advance

By the looks of how the merge option works it should fix this issue or atleast in the case where we are at. The plan file is just a bit out of order but the content of each file referenced in the plan file is exact over all branches!
So I think we are good to go with this fix!

Excellent, thanks for the feedback, @acrobat!

Looks good @theory!

I'm not sure if this is still an open item, but here's a few thoughts about it:

I think it would be useful to have a way to scream loudly about any changes that happen in a dev environment that might not be able to deploy in prod. In order to do that, you'd need some way to publish the current state of multiple production environments, and have sqitch be able to grab that info. I'm not sure git would be the right tool for that, though there is an interesting possibility of specifying a number of git branches/tags as being "production", and looking at the sqitch.plan in each of those. But that still depends on each dev keeping git in sync, so probably better to also support pulling the info from some other source.
Right now there's still too much need to hand-edit the plan when you screw something up. #290 and #295 start to address that, but at a minimum you'd still need to be able to change dependencies of something you're working on, and possibly re-ordering things (though, ideally the order would be determined only by dependencies...)
Once there's no more reason to hand-edit the plan the door opens for having a plan that's much more like a log of actions instead of a static snapshot. That allows for the concept of "hey, just replay these actions". I suspect that's what's needed to truly support things like renames that will also work with deployed changes.

All of those sound pretty non-trivial. Maybe they're worth doing; maybe not... but if they're not done then it would be good to have specific documentation on how to handle some of these scenarios. (Maybe some of that already exists; I'm not deep enough into the Koolaid yet to have encountered most of these problems...)

FWIW, you can create object-specific templates yourself, as detailed in this blog post.


Picamera web interface (mjpeg streaming, etc)

Dave, all
thank you so much for all the amazing work.
I've been working on a security system RPi based and I'm using this code based on picamera to get motion detection and recording done (works like a charm!):
https://gist.github.com/spikedrba/aecb82d8b51e991dbd01
I'm now trying to figure out the best way to expose the stream and recorded videos over the web very much like it's being done in this project:
https://github.com/silvanmelchior/RPi_Cam_Web_Interface
The second part is easy, but exposing the stream is being an issue. H264 is very efficient, but there doesn't seem to be a good way to get that going for streaming in a browser, which I guess is why most IP cameras use MJPEG. I know picamera has support for that in start_recording, however dumping that to a network stream as described in the  Recording to a network stream basic recipe is only giving me one image and then an error (using an html img tag to load it).
I could alternatively use capture_sequence somehow and output stills to the same file name, which is what the above project does (in a ramdisk) and then the web interface reads it out again into a img src tag. The problem with this and the existing code I have is that it would be blocking the script (compared to start_recording). I could put that in a thread, but I'm not sure it's a good solution.
The last idea I had was to use a custom output that would take a h264 stream and output every X frames (depending on fps) to disk as jpg (basically becoming a non blocking capture_sequence).
Am I looking at this completely wrong? Happy to put in the time, but would appreciate some pointers/direction.
thanks,
Spike

I am having a somewhat similar issue. In my case the stream circular buffer does't work as I loose frames on resuming back to the stream (no lock especially does this) . I have this need as i want to potentially read the buffer out several times with overlapping video clips to create.
I made my own sort of sequence to 1 seconds clips (a rotating 1 minute buffer) in a RAM drive using split stream. This collection of files can be shared quite easily as a network drive, and read or other manipulated away from picamera as to whatever you need. At the expense of a 1 seconds latency! and you are abel to do so out of sequence to capturing frames in the main thread.
I suppose I could learn hoot setup multiple circularIO buffers, but that ends up being way more complex.

@paulhothersall , thanks for chiming in. What do you mean circular buffers in your case lose frames? even more interesting, how can you tell? I was looking for ways to analyze the stream (dump all the frames, their info etc, but couldn't find a lib/example to do that.
I'd love to take a look at your code if you can share it, I'm defintiely ok with 1s latency for the web stream.
thanks, Spike

http://petrkout.com/electronics/low-latency-0-4-s-video-streaming-from-raspberry-pi-mjpeg-streamer-opencv/
its not using picamera, but will give you you the idea about what is going on for the web interface part.
the buffers have an issue in the sense if you lock it it takes to long to read to even a ram drive. No locking and you can get the stream mixed up.....which is bad.

@paulhothersall I used to use mjpeg-streamer and in the end switched to raspimjpeg and RPi_Cam_Web_Interface:

https://github.com/silvanmelchior/userland
https://github.com/silvanmelchior/RPi_Cam_Web_Interface

I get a lot better performances out of those and a nice web interface. The problem is raspimjpeg has no buffer so often the video is missing the first part of the motion.
Spike

Writing a chapter on web streaming is something I need to spend some time on, but I'll just spend a moment responding to some of the points in the discussion above:
H.264 is easily the best format in terms of quality and size for streaming. However, most HTML5 browser clients expect a nicely encapsulated MP4 video stream for playing. Remember that the Pi's camera module is, in essence, a stripped out mobile phone camera. In a mobile phone firmware you don't want the camera to produce a full MP4 video - you want it to produce the raw H.264 video stream so that you can mux it with, say an AAC audio stream from the microphone to produce a nice MP4 video. So that's exactly what the Pi's camera gives you: a raw H.264 video stream. It's up to you to encapsulate it nicely for browser playback.
At the moment the best/easiest way of wrapping the H.264 output into an MP4 is with the MP4Box utility. However, this only works on files: not streams. At the moment I'm not aware of a good (read as: simple to implement) method of encapsulating a stream. If someone's aware of one, let me know and I'll add it to the docs.
On MJPEG, I'm not surprised that pointing an <img> tag at it only displays the first frame. It's probably interpreting it as a JPEG file with a load of garbage on the end. The <img> tag is generally used for static images - for example, pointing one at an MP4 file doesn't work either. You should probably try the HTML5 <video> tag instead or write some client side JavaScript to strip out the individual frames and update an image or canvas element with them.
On streaming latency (not mentioned above, but it comes up so frequently in relation to this topic I may as well stick all the info in one place): a lot people seem to be under the mis-apprehension that H.264 encoding introduces lots of latency. This isn't the case. H.264 playback typically introduces lots of latency because almost every video player is designed to buffer lots of data (several seconds worth) to guard against lost packets, network issues, etc. Therefore, getting low latency playback generally involves persuading the client to forgo its buffering (this is easier said than done, and again I've no idea how to do in the case of HTML5 browsers - any tips welcome!)
In the meantime people may want to have a look at the pistreaming demo I put together which uses ffmpeg to transcode to MPEG1, and uses an open source JavaScript based MPEG1 decoder (yes, horrid, but it works and it's near-universally supported) to provide a live stream from the camera. Feel free to fork and extend it however you like - I'm happy to accept pull requests with enhancements!

Dave,
thanks for chiming in, very much appreciated. And thanks for the pistreaming demo, it gave me a great starting point to integrate my motion detection script so that I can have a low res streaming while a full res video is recorded and a third stream is used for motion detection with another custom output (using the 1-3 splitter ports available).
Regarding the MJPEG comment, I guess I misunderstood what it was, in my mind I thought of it as "a series of jpeg compressed images" and so I could just dump it in there, but I see that's not right.
So let me ask a different question: would it work to write a custom output that parsed the h264 stream and wrote out jpeg encoded images from all i-frames? this would basically do the same thing as a capture sequence except in a non blocking fashion just like start_recording does and writing to the same file on a ramdisk that I can then serve. This would also allow me to easily do all kinds of reverse proxying and whatnot on that file to serve it to multiple clients from a different machine and leave the Pi alone.
Last but not least, in the pistreaming demo README you mention: "Better still there's plenty of applications which will happily decode and play the H.264 encoded video streams which the Pi's camera produces". Maybe my google-fu is weak, but I couldn't find much that is programmable, like a python library that reads and plays the stream. Closest I could get was gaming related libraries like pyglet. Otherwise maybe vlc which would be installed everywhere and driven with vlc.py bindings. Mplayer has something similar but looks unmaintained. Do you have any recommendation if I wanted to write a desktop client in python to play the videos of 1 or more PIs?
thanks,
Spike

Hi Spike,

Regarding the MJPEG comment, I guess I misunderstood what it was, in my mind I thought of it as "a series of jpeg compressed images" and so I could just dump it in there, but I see that's not right.

Well, you're absolutely correct that it is "a series of jpeg compressed images" - that's a pretty much precise description of MJPEG.  The reason that won't display as a video in an <img> tag is because that tag is for displaying a static image so the browser reads the first JPEG, says "job done" and ignores the rest.

So let me ask a different question: would it work to write a custom output that parsed the h264 stream and wrote out jpeg encoded images from all i-frames? this would basically do the same thing as a capture sequence except in a non blocking fashion just like start_recording does and writing to the same file on a ramdisk that I can then serve. This would also allow me to easily do all kinds of reverse proxying and whatnot on that file to serve it to multiple clients from a different machine and leave the Pi alone.

I certainly wouldn't bother with decoding H.264 and then re-encoding it as JPEG images. There's no point when you can easily grab each JPEG image out of the MJPEG stream. Here's a little gist I whipped for someone on IRC the other day which uses a custom output to extract individual JPEGs from an MJPEG stream (this is easy because JPEG has an extremely well designed "magic number" (FF D8) which due to the way JPEG is encoded cannot appear within the rest of the JPEG output.

Last but not least, in the pistreaming demo README you mention: "Better still there's plenty of applications which will happily decode and play the H.264 encoded video streams which the Pi's camera produces". Maybe my google-fu is weak, but I couldn't find much that is programmable, like a python library that reads and plays the stream....

That's referring to applications like VLC, mplayer, XBMC (aka Kodi), and ffmpeg. However, you're correct there are precious few libraries with decent Python bindings for extracting individual images. OpenCV has a VideoCapture class which can do this (and which does work with network streams if you pass it an appropriately formatted URL), but all it's actually doing under the covers is firing up an instance of ffmpeg and reading its output (in other words, nothing you couldn't do with subprocess in Python). Also, ffmpeg isn't (yet) optimized to use the GPU H.264 encoder/decoder on the Pi so it's slow as anything (in other words, there's no point going that route unless you're doing the decoding on a more hefty machine).
At the moment, your best option for high performance H.264 decoding on the Pi is to dig into omxplayer's source code to learn how to use the GPU's H.264 decoder via OpenMAX (it's probably possible from MMAL as well but I haven't had time to look into that). I can't say that's a terribly easy option though!

@spikedrba, @waveform80: I found a way to stream MJPEG in a browser, you can use a  tag and assign to it the following css class:
.ui-content-stream {
background:url('http://<your_mjpeg_stream_url>') no-repeat;
background-size: auto 100% !important;
background-position: center;
}
It works fine in Chrome on Win, OSX and Android; on iOS only Safari is actually working.
Latency is quite low on model B and B+ (about 0.5s@640x480px), even less on a B 2.

Interesting stuff (I'm not at all sure why it works though - I would've though CSS background-image would interpret things the same as <img>! Anyway I'll have a play with that when I get a sec, and see whether it's worth adding something to the doc recipes (although I'm still rather horrified we're resorting to 20 year old formats for streaming!).

@previ pretty excited that this works for you (equally puzzled why it does compared to an  tag), but it won't work for me on linux (neither FF nor chrome). In both cases I just get a blank page.
Here's my test code in case you can spot something dumb I'm doing:
https://gist.github.com/spikedrba/38a90e76cfce43c5e358
thanks,
Spike

@spikedrba looking at your code, it seems the HTTP header part is missing.
After the socket is connected, write the following:
        self.send_response(200)
        #self.send_header("Connection", "close")
        self.send_header("Max-Age", "0")
        self.send_header("Expires", "0")
        self.send_header("Cache-Control", "no-cache, private")
        self.send_header("Pragma", "no-cache")
        self.send_header("Content-Type", "multipart/x-mixed-replace; boundary=--BOUNDARYSTRING")
        self.end_headers()

and for each image (frame):
                    self.wfile.write("--BOUNDARYSTRING\r\n")
                    self.send_header("Content-type", "image/jpeg")
                    self.send_header("Content-Length", str(len(_jpegstreamers[port].jpgdata)))
                    self.end_headers()
                    #_jpegstreamers[port]._lock.acquire()
                    self.wfile.write(_jpegstreamers[port].jpgdata)
                    #_jpegstreamers[port]._lock.release()
                    self.wfile.write("\r\n")

(full source file here: https://github.com/CoderBotOrg/coderbot/blob/develop/viz/streamer.py )
@waveform80 I totally agree, MJPEG is definitely not the smartest streaming technology around; a modern and more efficient codec like H264 would lead to better performance, especially in term of bandwidth used.
The problem is, at least for me, that it seems to be quite difficult to implement.

@previ
bear with me, but I'm confused how this is streaming the actaul MJPEG stream picamera.PiCamera.start_recording gives you back.
The way I read that code is that you expect individual jpeg images, not the stream. If that's the case, or even if it wasn't, what's the advantage over doing something along the lines of this gist @waveform80 put together:
https://gist.github.com/waveform80/263b9c8bdcb1e9b79749
and simply writing to the same file and serving that? (this is what @silvanmelchior's rpi_web_interface does). So basically you pick apart the MJPEG stream and rewrite the img file (doing this in a ramdisk for less lag) and then just serve that file (with lighty or something, or even python, but you're basically just returning a static file). And on the client side you just have a piece of js with a settimeout that reloads the source of the img.
is that a bad idea?
thanks,
Spike

@spikedrba sorry I didn't link the code that does the picamera image grabbing stuff.
Yes my implementation basically grabs single jpegs (in memory, using BytesIO) and serves them through the HTTPStreamer.
On the client side, no js code is required since the browser is able to process the stream correcly, using the css in the message above (didn't try the  tag).
I didn't try to send the MJPEG stream directly to the client, but as far as I understood, this doesn't seem to work because the client is unable to process correctly the MJPEG stream.

Many thanks for all the info everyone's provided on this ticket! There'll be an MJPEG streaming demo in the Advanced Recipes section of the next picamera as a result.


How to capture the logs from the screen that is shared using ttyd

Describe your problem or question here.
i ran the below instance of ttyd on my centos machine
** ttyd --writable -p 8080 telnet  **
this resulting terminal window was shared on a web browser using machineIp:8080 ,  i wanted to capture what ever being executed on that terminal window shared over browser, i have gone through the documentation of ttyd, I did not found any such logging mechanism. how to achieve this ?

#251


Floyd County, GA

data is here: https://gis01.romega.us/arcgis/rest/services
but I cannot get past
Error: Services Directory has been disabled
Source: http://gis.romega.us
Maybe @trescube or @iandees can work their magic on this!?

There's parcels here: https://gis01.romega.us/arcgis/rest/services/Parcels/MapServer/0?f=json
I can't tell if the address-related fields there are situs or mailing, though.

Resolved in #4578


Add visual mode search

Search for word or text in selection, to circumvent the limitations of vim-words.

Done


(MAINT) Add delete_on_termination and volume_type to block_devices parameter

This PR updates and replaces https://github.com/puppetlabs/puppetlabs-aws/pull/238

Looks good.  Thank you.


Contributing.md and CoC

We've updated all our organisation repos to have a code of conduct and mentioned this in both the readme and the contributing.md files.  Please do the same for this repo.

Thank you for this @lornajane. I've addressed this issue quickly as I think it's very important.


Adds phone number validator.

@brendt - This adds a fairly hefty dependency, but I believe it might be worth it to support phone number validation natively.
Related to #3.

Nice!


doc: add release notes for 2.10

Description
add release notes for 2.10
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@raintygao Seems the following PRs were missed:
https://github.com/opensearch-project/ml-commons-dashboards/pull/238
https://github.com/opensearch-project/ml-commons-dashboards/pull/239

@raintygao Seems the following PRs were missed: #238 #239

@wanglam I saw these PRs are labeled 2.9, do we need to append them?


issues #407 To accept " ," as seperator replaced "|" with ","

issues #407

This was fixed in #483 . Closing this one.


Rancher agent cannot authenticate with ECR during upgrade.

Rancher versions:
rancher/server or rancher/rancher: 1.6.14
rancher/agent or rancher/rancher-agent: 1.2.9
Infrastructure Stack versions:
healthcheck: 0.3.6
ipsec: 0.2.3
network-services: 0.2.9
scheduler: 0.8.3-1
kubernetes (if applicable): N/A
Docker version: (docker version,docker info preferred)
17.09.0-ce
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.4"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"
3.10.0-693.21.1.el7.x86_64
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
AWS EC2
Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)
Rancher HA external DB
Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
Cattle
Steps to Reproduce:
Set AWS ECR as default registry and pre-populate ECR with Rancher images. Use ecr-updater catalog item to manage ECR authentication within Rancher and ecr-credential-helper for ECR authentication on the host (for manual pulls). Upgrade Rancher from 1.6.14 > 1.6.16. When the agent goes to upgrade from 1.2.9 to 1.2.10 it fails to pull from ECR with an ECR authentication error. All pulls successfully authenticate with ECR except this in-place agent upgrade. A workaround is to volume mount the ECR credential helper and its config into the agent container however have been advised by @chrisurwin that this may interfere with the agent and to raise this as an issue.

This is actually by design. We never intended to have the agent upgrade be able to pull credentials from a registry to grab images.
We will document best practices on how to prepare for an upgrade if you are using a registry that requires credentials. We recommend using a catalog item that will help pull all the necessary images that your hosts will need prior to the upgrade.

Tested with pre-pull images on rancher v1.6.18-rc3. Set the default registry to a private registry. Launched the catalog pre-pull images specifying the required agent image. The agent image was pre-pulled on all the hosts.


Serve images with the article file.

If we want to add an image to an article, we have to host it in a third party place then reference it from there.
It would be benefcial if the node.js server serve the images linked to the articel as well, and does not require the use  of third party to do so.

Currently we don't have an effective way to handle images. adding them to the repository will make it grow out of hands as there is no limit to the amount of articles, projects, and images in each article. hosting them on a dedicated service is much more effective. maybe we will consider this in future releases. thank you for the suggestion.

closing this because we no longer host our own articles, will re-open it once we bring back articles


Unable to install Xcode

Describe the bug
Fresh install of macOS Monterey, fully signed in my Apple ID, when trying to install Xcode 13.3.1 the app give error "Unable to install Xcode, Please file a bug report"
To Reproduce
1- start Xcodes.app
2- click on Install Xcode 13.3.1(13E500a)
3- Error "Unable to install Xcode." without any further information.
Expected behavior
It should be able to install Xcode.
Screenshots
If applicable, add screenshots to help explain your problem.
Version

OS: macOS Monterey 12.3.1
Xcodes: 13.3.1 13E500a

check my comment here
https://github.com/RobotsAndPencils/XcodesApp/issues/215#issuecomment-1117091473


Triangulation Failed

Hi, thanks for the amazing work. I just tried your code to relocalize my own dataset. But somehow, I found an error when I  performed the matcher.
Here is what I did:

I have 3D map (cameras.bin, images.bin, and points3D.bin) retrieved from the Colmap. Later, I'll use this reference model to triangulate the new SfM model
I extract the features for all database and queries, then perform pairs_from_covisibilty. It worked
Moved to the matches, I got this issue:

Loaded SuperPoint model
100%|██████████████████████████████████████████████████████████████████| 2210/2210 [01:13<00:00, 29.98it/s]
[2023/09/27 13:48:00 hloc INFO] Finished exporting features.
[2023/09/27 13:48:00 hloc INFO] Reading the COLMAP model...
[2023/09/27 13:48:02 hloc.utils.read_write_model WARNING] This SfM file structure was deprecated in hloc v1.1
[2023/09/27 13:48:02 hloc INFO] Extracting image pairs from covisibility info...
100%|█████████████████████████████████████████████████████████████████| 1105/1105 [00:01<00:00, 624.03it/s]
[2023/09/27 13:48:04 hloc INFO] Found 11050 pairs.
[2023/09/27 13:48:04 hloc INFO] Matching local features with configuration:
{'model': {'name': 'superglue',
           'sinkhorn_iterations': 50,
           'weights': 'outdoor'},
 'output': 'matches-superglue'}
Loaded SuperGlue model ("outdoor" weights)
  0%|                                                                             | 0/6360 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "./dev/Hierarchical-Localization/hloc/pipelines/library/db5.py", line 31, in <module>
    sfm_matches = match_features.main(matcher_conf,sfm_pair,feature_conf['output'],outputs) #matches from sfm model
  File "/home/nanda/dev/Hierarchical-Localization/hloc/match_features.py", line 160, in main
    match_from_paths(conf, pairs, matches, features_q, features_ref, overwrite)
  File "/home/nanda/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/nanda/dev/Hierarchical-Localization/hloc/match_features.py", line 219, in match_from_paths
    for idx, data in enumerate(tqdm(loader, smoothing=.1)):
  File "/home/nanda/.local/lib/python3.8/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/home/nanda/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 652, in __next__
    data = self._next_data()
  File "/home/nanda/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1347, in _next_data
    return self._process_data(data)
  File "/home/nanda/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1373, in _process_data
    data.reraise()
  File "/home/nanda/.local/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/nanda/.local/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/nanda/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/nanda/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/nanda/dev/Hierarchical-Localization/hloc/match_features.py", line 108, in __getitem__
    grp = fd[name0]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/nanda/.local/lib/python3.8/site-packages/h5py/_hl/group.py", line 357, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'frame000074.png' doesn't exist)"

Would you enlighten me why this happened? And I also curious about the warning of my SfM model reference. Is that also affected the whole code? Thank you!

Update, the error above because something's wrong with my prefix. But now it failed on the triangulation process as follows:
100%|███████████████████████████████████████████████████████████████| 1105/1105 [00:00<00:00, 59633.38it/s]
[2023/09/27 14:47:33 hloc INFO] Importing matches into the database...
100%|██████████████████████████████████████████████████████████████| 11050/11050 [00:02<00:00, 4029.93it/s]
[2023/09/27 14:47:36 hloc INFO] Performing geometric verification of the matches...
100%|██████████████████████████████████████████████████████████████████| 1105/1105 [00:47<00:00, 23.23it/s]
[2023/09/27 14:48:24 hloc INFO] mean/med/min/max valid matches 80.08/87.01/7.06/100.00%.
[2023/09/27 14:48:24 hloc INFO] Running 3D triangulation...
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0927 14:48:24.215937 73566 database.cc:85] Keypoint format not supported
*** Check failure stack trace: ***
Aborted (core dumped)

So, is it valid if the error above is correlated with my 3D model?

Try deleting whatever is in your SfM folder and running the triangulation step again. For some reason the database seems corrupted.

I tried your advice and it produced the same error. Seems like there is something wrong in the geometric verification process. To check it, I tried to only perform the feature extractor and and image matching and exported that into a database. And it ended like this:
100%|████████████████████████████████████| 1105/1105 [00:00<00:00, 85157.67it/s]
[2023/10/03 16:24:22 hloc INFO] Importing matches into the database...
100%|█████████████████████████████████████| 9900/9900 [00:03<00:00, 2619.60it/s]
[2023/10/03 16:24:26 hloc INFO] Performing geometric verification of the matches...
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1003 16:24:26.196774 19814 database.cc:85] Keypoint format not supported
*** Check failure stack trace: ***
Aborted (core dumped)

And can you provide me more about the information of  corrupted database? I still don't get it why it related to the keypoint format etc. Thanks!

Solved this error by remove all repo and build it again from the scratch. I'll close this issue~


util/signature: set scheme only if it is non-default

Signed-off-by: Evgenii Stratonikov<EMAIL_ADDRESS>

Fixed with https://github.com/nspcc-dev/neofs-api/pull/203 and https://github.com/nspcc-dev/neofs-api-go/pull/380


allow full ARN for API key SSM param identifier

what
This expands the validation rules for the dd_api_key_source variable to allow you to provide the full ARN for a SSM parameter.
why
In the case where the parameter in question is shared from another account (via RAM), it has to be referenced using the full ARN.
references
https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-shared-parameters.html#accessing

/terratest

@natw the tests are failing with :

Error: creating Lambda Function (eg-ue2-test-datadog-lambda-forwarder-lygfv1-logs): operation error Lambda: CreateFunction, https response error StatusCode: 400, RequestID: d2dfcadf-4a63-48ff-9199-dd7c687e672f, InvalidParameterValueException: The runtime parameter of python3.7 is no longer supported for creating or updating AWS Lambda functions. We recommend you use the new runtime (python3.12) while creating or updating functions.

Could you please update the default value for variable lambda_runtime to python3.12 :
variable "lambda_runtime" {
  type        = string
  description = "Runtime environment for Datadog Lambda"
  default     = "python3.12"
}

@gberenice as of lambda version 3.106.0, python 3.12 was giving me errors of some kind (sorry, it was a few weeks ago), but 3.11 worked fine, so I updated the default to that.

/terratest

/terratest


Add support for XML files through xmltodict library

Add support for XML file as input. Work like the YAML support.

This lgtm, but I really want to do #17 before accepting this. With all of these formats, it's starting to impact startup time. :)

I would also add this under extras inside setup.py so we could do $ pip install jinja2-cli[xml]

You are right, https://github.com/mattrobenolt/jinja2-cli/issues/17 need to be fixed before adding xml support.

Resolved with #29. Thanks. :)


Default platform libc to glibc in init script

Some Linux distributions, such as Fedora, do not include the libc in their target triple. For these cases we should default to gnu for the .platform file now that ponyup no longer assumes glibc.

This doesn't seem like a good fix to me.
I think if we can't determine the target, we should ask.

@SeanTAllen the default has been replaced with a prompt


Fixed some minor warnings + code style suggestion

Description
I fixed some warnings raised by the linter. Mainly:

'subtitle' is deprecated and shouldn't be used. This is the term used in the 2014 version of material design. The modern term is subtitle2. This feature was deprecated after v1.13.8.. Try replacing the use of the deprecated member with the replacement.
Replaced all such usages of 2014-spec with respective 2018 versions.

Close instances of 'dart.core.Sink'.
This is a false positive, as described here. I suggest enforcing people to put linter exception rule before such a safe usage of BLOC. (below):

//ignore: close_sinks
SendRequestBloc bloc = BlocProvider.of<SendRequestBloc>(context);

This way, we'll have safe, false positives suppressed, but still get lint for any future non-BLOC related usages of Dart Stream API.

Also, removed all unused imports

First code style rules
Although dartfmt does a lot of work already, we need even stricter rules to maintain consistent style in the future.
I suggest:

Column line width set to 100. 80 is simply too small.
This can be easily set in Android Studio/IntelliJ Dart code style settings tab (example below):

In VSCode you have open settings and set Dart: Line Length to 100 (example below):

Enforcing the aforementioned use of //ignore: close_sinks in context of safe usages of BLOCs

This is just a draft, but if we agree on this, I'll update README with the style guide info.
Flutter Channel:

[x] I have used the Flutter Beta channel on my local machine

Type of Change:

Code
Quality Assurance

Code/Quality Assurance Only

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
All features work fine on Pixel 3a (API 29) emulator.
Checklist:

[x] My PR follows the style guidelines of this project
[x] I have performed a self-review of my own code or materials
[x] I have commented my code or provided relevant documentation, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] Any dependent changes have been merged

Code/Quality Assurance Only

[x] My changes generate no new warnings
[ ] My PR currently breaks something (fix or feature that would cause existing functionality to not work as expected)
[ ] Any dependent changes have been published in downstream modules

warning: [options] target value 7 is obsolete and will be removed in a future release any idea why these warnings pop up?

i have never used ignore: close_sinks before so i'm not sure if they block only false positives. If this is not really big change i would keep the ignore: close_sinks commented out for now.What do you think @bartekpacia

@Techno-Disaster this probably comes from some native Android libraries (as those warnings are not present on iOS build logs).
Also // ignore: close_sinks has to be commented, because, well, it's a comment. I want this to be used only in specific use-cases like assigning BLOC to the variable for pure convenience. Having 10 false positives on Lint really makes it irritating.
I recommend reading https://github.com/felangel/bloc/issues/587 because it  describes it very well (look at comments with high reaction count).

ah kk thanks for the info, my bloc knowledge isn't that good as i have used it only 1-2 times till now, will learn it asap

@Techno-Disaster no problem, take your time :) I recommend BLOC crash course by ResoCoder, this guy has a lot of quality content.
I also believe somebody submits a PR where he will use //ignore: close_sinks incorrectly, we will see that and point it out. But it shouldn't ever happen because it's just one specific situation where such ignore comments should be put.

thanks for the resources @bartekpacia , will start there


no such file or directory

Expected Behavior
getExports is throwing an error:
{
  context: { format: 'module', importAttributes: {} },
  error: Error: ENOENT: no such file or directory, open '/Users/x/Developer/contra/gaia/node_modules/.pnpm/graphql-yoga@5.1.1_graphql@16.8.1/node_modules/graphql-yoga/esm/@graphql-yoga/logger'
      at async open (node:internal/fs/promises:633:25)
      at async readFile (node:internal/fs/promises:1242:14)
      at async getSource (node:internal/modules/esm/load:46:14)
      at async defaultLoad (node:internal/modules/esm/load:137:34)
      at async nextLoad (node:internal/modules/esm/hooks:750:22)
      at async getExports (/Users/x/Developer/contra/gaia/node_modules/.pnpm/import-in-the-middle@1.7.3/node_modules/import-in-the-middle/lib/get-exports.js:68:17)
      at async processModule (/Users/x/Developer/contra/gaia/node_modules/.pnpm/import-in-the-middle@1.7.3/node_modules/import-in-the-middle/hook.js:134:23)
      at async processModule (/Users/x/Developer/contra/gaia/node_modules/.pnpm/import-in-the-middle@1.7.3/node_modules/import-in-the-middle/hook.js:160:20)
      at async getSource (/Users/x/Developer/contra/gaia/node_modules/.pnpm/import-in-the-middle@1.7.3/node_modules/import-in-the-middle/hook.js:269:60)
      at async load (/Users/x/Developer/contra/gaia/node_modules/.pnpm/import-in-the-middle@1.7.3/node_modules/import-in-the-middle/hook.js:334:26) {
    errno: -2,
    code: 'ENOENT',
    syscall: 'open',
    path: '/Users/x/Developer/contra/gaia/node_modules/.pnpm/graphql-yoga@5.1.1_graphql@16.8.1/node_modules/graphql-yoga/esm/@graphql-yoga/logger'
  },
  url: 'file:///Users/x/Developer/contra/gaia/node_modules/.pnpm/graphql-yoga@5.1.1_graphql@16.8.1/node_modules/graphql-yoga/esm/@graphql-yoga/logger'
}

I added the following code to catch the above error.
let parentCtx;

try {
  parentCtx = await parentLoad(url, context)
} catch (error) {
  console.log({
    context,
    error,
    url
  });

  throw error;
}

The same code works if I remove --loader=import-in-the-middle/hook.mjs.
Happy to provide more context.
Specifications

Platform: Node.js v21

Hard to tell from import URLs what's happening here, but I wonder if it is related to how pnpm works.

The paths mentioned in the URLs do not actually exist, e.g.
{
  context: { format: 'module', importAttributes: {} },
  error: Error: ENOENT: no such file or directory, open '/Users/x/Developer/contra/gaia/node_modules/graphql-yoga/esm/@graphql-yoga/subscription'
      at async open (node:internal/fs/promises:633:25)
      at async readFile (node:internal/fs/promises:1242:14)
      at async getSource (node:internal/modules/esm/load:46:14)
      at async defaultLoad (node:internal/modules/esm/load:137:34)
      at async nextLoad (node:internal/modules/esm/hooks:750:22)
      at async getExports (/Users/x/Developer/contra/gaia/node_modules/import-in-the-middle/lib/get-exports.js:66:17)
      at async processModule (/Users/x/Developer/contra/gaia/node_modules/import-in-the-middle/hook.js:134:23)
      at async processModule (/Users/x/Developer/contra/gaia/node_modules/import-in-the-middle/hook.js:160:20)
      at async processModule (/Users/x/Developer/contra/gaia/node_modules/import-in-the-middle/hook.js:160:20)
      at async getSource (/Users/x/Developer/contra/gaia/node_modules/import-in-the-middle/hook.js:269:60) {
    errno: -2,
    code: 'ENOENT',
    syscall: 'open',
    path: '/Users/x/Developer/contra/gaia/node_modules/graphql-yoga/esm/@graphql-yoga/subscription'
  },
  url: 'file:///Users/x/Developer/contra/gaia/node_modules/graphql-yoga/esm/@graphql-yoga/subscription'
}

There is no such directory as /Users/x/Developer/contra/gaia/node_modules/graphql-yoga/esm/@graphql-yoga/subscription
The closest directly that exists is /Users/x/Developer/contra/gaia/node_modules/graphql-yoga/esm/.

An example of file that triggers this is:
export * from 'styled-components';
//# sourceMappingURL=index.js.map

I can see that when processModule is trying to process it, it derives wrong URL for the module.
I added this line just before processModule:
console.log({
  modFile,
  normalizedModName,
  modUrl,
  modName,
});

which tells us:
{
  modFile: 'styled-components',
  normalizedModName: 'styledComponents',
  modUrl: 'file:///Users/x/Developer/contra/gaia/packages/styled-components/dist/styled-components',
  modName: '17062121655529e604482'
}

/Users/x/Developer/contra/gaia/packages/styled-components/dist/styled-components is wrong. It should be: /Users/x/Developer/contra/gaia/node_modules/styled-components/. But there is no way of knowing that just from analyzing the URLs, which is what this module does.
https://github.com/DataDog/import-in-the-middle/blob/c3c2c52c1915b47994af59d507c59029c1f1fae9/hook.js#L155-L166
I am confused how this works for other people use cases, since this module is clearly used by others.

Another example it fails:
Object.defineProperty(exports, "TimeoutFailure", { enumerable: true, get: function () { return common_1.TimeoutFailure; } });
+__exportStar(require("@temporalio/common/lib/errors"), exports);
__exportStar(require("@temporalio/common/lib/interfaces"), exports);
__exportStar(require("@temporalio/common/lib/workflow-handle"), exports);

{
  exports: [
    '__esModule',
    'LOCAL_TARGET',
    'Connection',
    'TimeoutFailure',
    'TerminatedFailure',
    'TemporalFailure',
    'ServerFailure',
    'defaultPayloadConverter',
    'ChildWorkflowFailure',
    'CancelledFailure',
    'ApplicationFailure',
    'ActivityFailure'
  ],
  reexports: [
    '@temporalio/common/lib/errors',
    '@temporalio/common/lib/interfaces',
    '@temporalio/common/lib/workflow-handle',
    './async-completion-client',
    './client',
    './errors',
    './grpc-retry',
    './interceptors',
    './types',
    './workflow-client',
    './workflow-options',
    './schedule-types',
    './schedule-client',
    './task-queue-client'
  ]
}

Error: Cannot find module '@temporalio/common/lib/errors'
Require stack:
- /Users/x/Developer/contra/import-in-the-middle/lib/get-exports.js
- /Users/x/Developer/contra/import-in-the-middle/hook.js
    at Module._resolveFilename (node:internal/modules/cj

An example of file that triggers this is:
export * from 'styled-components';
//# sourceMappingURL=index.js.map

I can see that when processModule is trying to process it, it derives wrong URL for the module.
I added this line just before processModule:
console.log({
  modFile,
  normalizedModName,
  modUrl,
  modName,
});

which tells us:
{
  modFile: 'styled-components',
  normalizedModName: 'styledComponents',
  modUrl: 'file:///Users/x/Developer/contra/gaia/packages/styled-components/dist/styled-components',
  modName: '17062121655529e604482'
}

/Users/x/Developer/contra/gaia/packages/styled-components/dist/styled-components is wrong. It should be: /Users/x/Developer/contra/gaia/node_modules/styled-components/. But there is no way of knowing that just from analyzing the URLs, which is what this module does.
https://github.com/DataDog/import-in-the-middle/blob/c3c2c52c1915b47994af59d507c59029c1f1fae9/hook.js#L155-L166
I am confused how this works for other people use cases, since this module is clearly used by others.

So all of these failures are happening when resolving export * from ... pattern, e.g.
{ n: '* from @envelop/types' }
{ n: '* from @graphql-yoga/logger' }
{ n: '* from @floating-ui/react-dom' }
{ n: '* from playwright-core' }
{ n: '* from styled-components' }

Tagging @bengl in case you have a better sense of what's breaking.

For what it is worth, if we update the logic to resolve import paths relative to the srcUrl, it resolves to correct modules (?).
-      const data = await processModule({
-        srcUrl: modUrl,
-        context,
-        parentGetSource,
-        ns: `$${modName}`,
-        defaultAs: normalizedModName
-      })
+      let data;
+
+      try {
+        data = await processModule({
+          srcUrl: modUrl,
+          context,
+          parentGetSource,
+          ns: `$${modName}`,
+          defaultAs: normalizedModName
+        });
+      } catch (error) {
+        console.log({
+          srcUrl,
+          modFile,
+          modUrl,
+          n,
+        });
+      }
+
+      if (!data) {
+        const newModUrl = require.resolve(modFile, {
+          paths: [srcUrl]
+        });
+
+        console.log({
+          newModUrl
+        });
+
+        data = await processModule({
+          srcUrl: newModUrl,
+          context,
+          parentGetSource,
+          ns: `$${modName}`,
+          defaultAs: normalizedModName
+        });
+      }

{
  srcUrl<EMAIL_ADDRESS>  modFile: '@envelop/types',
  modUrl: 'file:///Users/x/Developer/contra/gaia/node_modules/@envelop/core/esm/@envelop/types',
  n: '* from @envelop/types'
}
{
  newModUrl<EMAIL_ADDRESS>}
{
  srcUrl: 'file:///Users/x/Developer/contra/gaia/node_modules/graphql-yoga/esm/index.js',
  modFile: '@graphql-yoga/logger',
  modUrl: 'file:///Users/x/Developer/contra/gaia/node_modules/graphql-yoga/esm/@graphql-yoga/logger',
  n: '* from @graphql-yoga/logger'
}
{
  newModUrl<EMAIL_ADDRESS>}
{
  srcUrl: 'file:///Users/x/Developer/contra/gaia/node_modules/playwright-chromium/index.mjs',
  modFile: 'playwright-core',
  modUrl: 'file:///Users/x/Developer/contra/gaia/node_modules/playwright-chromium/playwright-core',
  n: '* from playwright-core'
}
{
  newModUrl: '/Users/x/Developer/contra/gaia/node_modules/playwright-core/index.js'
}
{
  srcUrl<EMAIL_ADDRESS>  modFile: '@floating-ui/react-dom',
  modUrl: 'file:///Users/x/Developer/contra/gaia/node_modules/@floating-ui/react-dom-interactions/dist/@floating-ui/react-dom',
  n: '* from @floating-ui/react-dom'
}
{
  newModUrl<EMAIL_ADDRESS>}
{
  srcUrl: 'file:///Users/x/Developer/contra/gaia/packages/styled-components/dist/index.js',
  modFile: 'styled-components',
  modUrl: 'file:///Users/x/Developer/contra/gaia/packages/styled-components/dist/styled-components',
  n: '* from styled-components'
}
{
  newModUrl: '/Users/x/Developer/contra/gaia/node_modules/styled-components/dist/styled-components.cjs.js'
}

Now it is giving another error.
TypeError [ERR_INVALID_ARG_VALUE]: The argument 'node:internal/modules/esm/load 'load' hook's nextLoad() url' should be a URL string. Received<EMAIL_ADDRESS>    at validateArgs (node:internal/modules/esm/hooks:359:17)
    at nextLoad (node:internal/modules/esm/hooks:739:7)

I am assuming this is because we resolved to cjs instead of esm, when it is expecting esm?

Looks like depending on how modules are installed (hoisted or non-hoisted), you will need to try both variants, i.e.
const newModUrl = pathToFileURL(require.resolve(modFile, {
  paths: [srcUrl]
})).toString();

and
const newModUrl = pathToFileURL(require.resolve(modFile)).toString();

This is the patch that we ended up with.
diff --git a/hook.js b/hook.js
index aa22356f370a34cd49ae3d5ea8fd076202e71bf5..074c388edf40f10e3e4274cf38c308371f7ce4fe 100644
--- a/hook.js
+++ b/hook.js
@@ -22,6 +22,48 @@ if (NODE_MAJOR >= 20 || (NODE_MAJOR === 18 && NODE_MINOR >= 19)) {
   getExports = ({ url }) => import(url).then(Object.keys)
 }

+function shouldIgnore (url) {
+  // In short, some packages are not compatible with this hook.
+  // I have attempted to patch this hook to work with every package, but that just sent me down a rabbit hole.
+  // See attempts and related issues here:
+  // * https://github.com/DataDog/import-in-the-middle/issues/60
+  // * https://github.com/DataDog/import-in-the-middle/issues/59
+  //
+  // However, for this hook to be useful, it does not need to work with every package.
+  // It just needs to work with the packages that we want OpenTelemetry to instrument.
+  const patterns = [
+    '@envelop/core',
+    '@floating-ui/react-dom-interactions',
+    'dataloader',
+    'graphql-yoga',
+    // Does not get patched properly.
+    // Produces errors such as:
+    // > TypeError: Cannot read properties of undefined (reading 'sendCommand')
+    // > at instrumentation_1.InstrumentationNodeModuleDefinition.moduleExports [as patch] (/Users/x/Developer/contra/gaia/node_modules/.pnpm/@opentelemetry+instrumentation-ioredis@0.36.1_@opentelemetry+api@1.7.0/node_modules/@opentelemetry/instrumentation-ioredis/build/src/instrumentation.js:40:78)
+    'ioredis',
+    'openai',
+    'playwright-chromium',
+    'styled-components',
+  ];
+
+  for (const pattern of patterns) {
+    if (url === pattern || url.startsWith(pattern + '/') || url.includes('/' + pattern + '/')) {
+      return true;
+    }
+  }
+
+  // If you miss a pattern, then you will get an error along the lines of:
+  //
+  // > Error: ENOENT: no such file or directory, open '/Users/x/Developer/contra/gaia/packages/styled-components/dist/styled-components'
+  //
+  // Usually, it is easy to tell what package is causing the issue by just looking at the path (e.g. 'styled-components' in the above example).
+  // However, other times, it is not so obvious. In those cases, you can uncomment the following line to get everything that was attempted up to the point of failure.
+  //
+  // console.log('url', url)
+
+  return false;
+}
+
 function hasIitm (url) {
   try {
     return new URL(url).searchParams.has('iitm')
@@ -230,6 +272,9 @@ function addIitm (url) {

 function createHook (meta) {
   async function resolve (specifier, context, parentResolve) {
+    if (shouldIgnore(specifier)) {
+      return parentResolve(specifier)
+    }
     const { parentURL = '' } = context
     const newSpecifier = deleteIitm(specifier)
     if (isWin && parentURL.indexOf('file:node') === 0) {
@@ -264,6 +309,9 @@ function createHook (meta) {

   const iitmURL = new URL('lib/register.js', meta.url).toString()
   async function getSource (url, context, parentGetSource) {
+    if (shouldIgnore(url)) {
+      return parentGetSource(url)
+    }
     if (hasIitm(url)) {
       const realUrl = deleteIitm(url)
       const { imports, namespaces, setters: mapSetters } = await processModule({
@@ -330,6 +378,9 @@ register(${JSON.stringify(realUrl)}, _, set, ${JSON.stringify(specifiers.get(rea

   // For Node.js 16.12.0 and higher.
   async function load (url, context, parentLoad) {
+    if (shouldIgnore(url)) {
+      return parentLoad(url)
+    }
     if (hasIitm(url)) {
       const { source } = await getSource(url, context, parentLoad)
       return {

I think hard coding a bunch of module identifiers is not the correct path forward. We should be able to concoct a replication that we can solve.

This was fixed by #78

This was closed by #78.
Feel free to open another issue if this is not the case!


Fix failing test on main

This PR fixes the failing test by instead checking the current used CUDA memory vs what's there. For some reason after the modeling tests memory is no longer being freed on the updated CUDA/PyTorch version, leading to this failure.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


#63 Add JSON import and export to custom menu

Local branch 63... created from local branch 62..., which has not yet been merged into develop.
Added submenu items for JSON import and export to "File" menu item.
Using ipcRenderer module to communicate asynchronously from renderer process to main process.
Removed sidebar buttons and related functions for JSON import and export.
Addresses issue #63.

Signed-off-by: John M. Horan<EMAIL_ADDRESS>

Looks great 👍


helmwave build requires working stand

Describe the bug
For now there is no way you can run helmwave build with invalid KUBE_CONFIG or if somebody is running helwave up in parallel or if you try to test charts with incompatible changes.
With invalid KUBE_CONFIG, you will get an error:
[🤬 aka FATAL]: I can't check if release is installed
        release: tp-docs-ui-demo-tp@demo-tp-infra
        error: Kubernetes cluster unreachable: Get "https://<IP_ADDRESS>:64434/version": dial tcp <IP_ADDRESS>:64434: connect: connection refused

If somebody is updating stand in parallel you'll get:
[🤬 aka FATAL]: one of goroutines in waitgroup sent error: 8 errors occurred:
	* failed to fix "session-data-t846-0b9ed@testing-t846-0b9ed-apps" pending status: release is in pending status
	* failed to fix "security-t846-0b9ed@testing-t846-0b9ed-secr" pending status: release is in pending status
	* failed to fix "api-platform-t846-0b9ed@testing-t846-0b9ed-apps" pending status: release is in pending status
	* failed to fix "billing-t846-0b9ed@testing-t846-0b9ed-apps" pending status: release is in pending status
	* failed to fix "authentication-t846-0b9ed@testing-t846-0b9ed-apps" pending status: release is in pending status
	* failed to fix "observability-t846-0b9ed@testing-t846-0b9ed-infra" pending status: release is in pending status
	* failed to fix "appl-t846-0b9ed@testing-t846-0b9ed-apps" pending status: release is in pending status
	* failed to fix "notification-t846-0b9ed@testing-t846-0b9ed-apps" pending status: release is in pending status

For incompatible chart changes It fails with:
[💩 aka ERROR]: ❌ can't get manifests: failed to upgrade home-page-demo-tp@demo-tp-infra: rendered manifests contain a resource that already exists. Unable to continue with update: Ingress "home-page-demo-tp-dbp-home-page-ing" in namespace "demo-tp-infra" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "home-page-demo-tp"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "demo-tp-infra"

Even running with --diff-mode local does not help in all of the cases above.
** General info (please complete the following information):**

OS: verified on Linux and Windows
Helmwave Version 0.24.3

To Reproduce
Steps to reproduce the behavior:

Break contents of your KUBE_CONFIG
Try to run helmwave build --yml --diff-mode local for any of your Helmwave manifests
You will get an error: I can't check if release is installed

Expected behavior
There should be a way of running helmwave build without building diff and making any requests to K8S API.
It is useful for the sake of linting and verifying Helmwave manifests.
In our team several engineers can edit Helmwave manifests in parallel and if somebody runs helmwave up all others should wait for 20 minutes before testing their changes with linters because they can't get a successful helmwave build.

will fix in #611


Completely pause chat while hovering whenever the mouse is over the chat.

without having to move the mouse to pause the chat

Or just add keybind to pause chat

available in nightly

Yeah, but you have to move your mouse for it to pause, i think a better
idea would be to add a bind that stops the chat, just like twitch added
recently for mods
בתאריך יום ב׳, 29 ביולי 2019 ב-20:11 מאת fourtf <
notifications@github.com>:

available in nightly
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Chatterino/chatterino2/issues/1130?email_source=notifications&email_token=AJKPTVQ4OL2OO43UFXK6BILQB4QEPA5CNFSM4H6WBPX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3BL37I#issuecomment-516079101,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJKPTVQTL343T5XVSL4FMHDQB4QEPANCNFSM4H6WBPXQ
.

it is fixed


Multiple code improvements - squid:S1186, squid:MethodCyclomaticComplexity, squid:S1151

This pull request is focused on resolving occurrences of Sonar rules
squid:S1186 - Methods should not be empty.
squid:MethodCyclomaticComplexity - Methods should not be too complex.
squid:S1151 - "switch case" clauses should not have too many lines.
You can find more information about the issue here:
https://dev.eclipse.org/sonar/rules/show/squid:S1186
https://dev.eclipse.org/sonar/rules/show/squid:MethodCyclomaticComplexity
https://dev.eclipse.org/sonar/rules/show/squid:S1151
Please let me know if you have any questions.
George Kankava

Sorry, George, don't think these changes add any value to the codebase.


python3Packages.metawear: init at 1.0.8

Description of changes

python3Packages.metawear: init at 1.0.8
metawear

https://pypi.org/project/metawear/
https://github.com/mbientlab/metawear-sdk-python

dependency warble

https://github.com/NixOS/nixpkgs/pull/238628

$ nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
2 packages built:
python310Packages.metawear python311Packages.metawear

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/2351

Why are they vendoring nlohman json?

Looks like it's a dependency for their sensor firmware
https://github.com/mbientlab/MetaWear-SDK-Cpp/blob/f67699e5c7eb64401a68157f553f9585e3857a6a/src/metawear/dfu/cpp/file_operations.cpp#L10

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/2383

Looks like it's a dependency for their sensor firmware

I was more asking in the direction if we can unvendor that and use ours?

Thanks for the review!

I was more asking in the direction if we can unvendor that and use ours?

Just changed it, gcc11 -> gcc, unvendored their version of nlohmann_json and using nixpkgs's nlohmann_json.

On x86_64-linux:
$ nix run nixpkgs#nixpkgs-review -- pr 238630
2 packages added:
python310Packages.metawear (init at 1.0.8) python311Packages.metawear (init at 1.0.8)
4 packages built:
python310Packages.metawear python310Packages.metawear.dist python311Packages.metawear python311Packages.metawear.dist

Fixed, on x86_64-linux:
$ nix run nixpkgs#nixpkgs-review -- pr 238630
2 packages added:
python310Packages.metawear (init at 1.0.8) python311Packages.metawear (init at 1.0.8)
4 packages built:
python310Packages.metawear python310Packages.metawear.dist python311Packages.metawear python311Packages.metawear.dist

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/2416

@ofborg build python310Packages.metawear


Add missing badges to readme

Signed-off-by: Vvamp<EMAIL_ADDRESS>

Thank you for doing that! I was being lazy and had not cropped the rest of the badges haha.

You're welcome haha, I understand. :)


Don't import dev cert

Instead use the ASP.NET env vars to just point to the cert location.

/azp run net - eventgrid - ci


feat(resolve)!: allow removing conditions

Description
Updates the default conditions / externalConditions and remove the need for overrideConditions and other odd logics.
refs #18016
refs #18332
refs #17326
refs #9860

/ecosystem-ci run

/ecosystem-ci run

/ecosystem-ci run

If we change the default to ['module', 'browser', 'node', 'production', 'development'] (https://github.com/vitejs/vite/pull/18395/commits/713cee381e95fc9769390f50d735e37bcfbe0f67), the following CIs fail (https://github.com/vitejs/vite/pull/18395#issuecomment-2422385882):

histoire: https://github.com/histoire-dev/histoire/blob/f5833710e22b8f6b55dd46693f3c80c4ad807c80/packages/histoire/src/node/vite.ts#L100
quasar: not sure why
vite-plugin-svelte: https://github.com/sveltejs/vite-plugin-svelte/blob/52e4a4f165dea8442ec729d73cd43f1aef2eed08/packages/vite-plugin-svelte/src/utils/options.js#L350
vitest: https://github.com/vitest-dev/vitest/blob/9d9bad5b1025576bbd58cc67db39c3e4afaf7329/packages/vitest/src/node/pool.ts#L93-L97

This behavior aligns with resolve.mainFields, but it's not easy to remove default conditions.
import { configDefaults } from 'vite'

const plugin = {
  name: 'remove-conditions',
  config(config) {
    config.resolve ||= {}
    config.resolve.conditions ||= configDefaults.resolve.conditions
    config.resolve.conditions = config.resolve.conditions.filter(
      (c) => c !== 'node',
    )
  }
}

If we put the default conditions before config hooks (https://github.com/vitejs/vite/pull/18395/commits/c8ab916f5a4aa0f125564488d9ae01ab9d5867a3), it will be easier to remove the defaults. But more CI failed 😅 (https://github.com/vitejs/vite/pull/18395#issuecomment-2431278934).
I'll revert to the original one (https://github.com/vitejs/vite/pull/18395/commits/713cee381e95fc9769390f50d735e37bcfbe0f67).

Pushed a commit that removes webCompatible. Let's see if that reduces the fails.

/ecosystem-ci run

I have updated to be the behavior decided in the meeting.


Possibility to save gpx

Hi bartsaintgermain!,
Thanks for raising an issue! The file is in XML format so all you need to do is to hit Ctrl + S as save it as e.g. .gpx.
I tried to add download flied to do <a> as is recommended. Unfortunately, it is not working and I assume because the GPX file isn't on the same domain 👇

[path] that resolves to an URL on the same origin. That means the page and the file must share the same domain, subdomain, protocol (HTTP vs. HTTPS), and port (if specified). Exceptions are blob: and data: (which always work), and file: (which never works). (https://stackoverflow.com/a/18678698)

In the next release, I will try to take deeper look into :)
Matej

Hi bartsaintgermain!,
Thanks for raising an issue! The file is in XML format so all you need to do is to hit Ctrl + S as save it as e.g. .gpx.
I tried to add download flied to do <a> as is recommended. Unfortunately, it is not working and I assume because the GPX file isn't on the same domain 👇

[path] that resolves to an URL on the same origin. That means the page and the file must share the same domain, subdomain, protocol (HTTP vs. HTTPS), and port (if specified). Exceptions are blob: and data: (which always work), and file: (which never works). (https://stackoverflow.com/a/18678698)

In the next release, I will try to take deeper look into :)
Matej

Is this also possible when running locally. I did succeed in generating the png files but could not find the gpx/xml file.

Hmm, interesting. It is created as XML file so when you download it, you should get XML and not png.

I have the same question. If I download the python code and run the program locally I can generate the .png's given in the example of "The Grange". However, the code as given does not generate a .gpx file. The website generates a .gpx file so it must be possible to generate one. How can we generate a .gpx in the vein of the code example you provide. Thanks.

I think that understand your query now. Sure, the jupyter notebook / the Grange example doesn't have the .gpx output.
You can do it simply by converting networkx nodes into geo coordinates, e.g. using this function convert_final_path_to_coordinates(org_graph, final_path) and using newly added gpx_formatter.py. Putting all together, you can try something like:
from libs.tools import *
from libs.gpx_formatter import TEMPLATE, TRACE_POINT

...

# List all edges of the extended graph including original edges and edges from minimal matching
single_edges = [(u, v) for u, v, k in graph.edges]
added_edges = get_shortest_paths(graph, matched_edges_with_weights)
edges = map_osmnx_edges2integers(graph, single_edges + added_edges)

# Finds the Eulerian path
network = Network(len(graph.nodes), edges, weighted=True)
eulerian_path = hierholzer(network)
converted_eulerian_path = convert_integer_path2osmnx_nodes(eulerian_path, graph.nodes())
double_edge_heap = get_double_edge_heap(org_graph)

# Finds the final path with edge IDs
final_path = convert_path(graph, converted_eulerian_path, double_edge_heap)
coordinates_path = convert_final_path_to_coordinates(org_graph, final_path)

# Route statistics from OSMnx
ex = ox.extended_stats(org_graph, ecc=True)
center_node = org_graph.nodes[ex["center"][0]]

trace_points = "\n\t\t\t".join([TRACE_POINT.format(
    lat=lat, lon=lon, id=i, timestamp=datetime.now().isoformat()
) for i, (lat, lon) in enumerate(coordinates_path)])

gpx_payload = TEMPLATE.format(
    name="Name your everystreet route",
    trace_points=trace_points,
    center_lat=center_node["y"],
    center_lon=center_node["x"]
)

with open("gpx_output.xml", "w") as f:
    f.write(gpx_payload)

I believe this could work 😄

Thank you so much! That works great. The .xml format doesn't play nice with the mapping software I am using, but plugging it into this website : https://mygeodata.cloud/converter/mapfactor-to-gpx generates a .gpx file that works.
I also had to use datetime.now() -> datetime.datetime.now().

Glad to hear that! Oh yeah, I could name it open("gpx_output.gpx", "w") and it would work.
BTW, did you know that you can generate the GPX file on http://www.everystreetchallenge.com?

Thx.
Would be great to generate from site but quickly hit on 3km2 limit.

I agree that the 3km^2 limit is annoying, on the other hand I think you will understand it is a webserver and the algorithm is a bit "heavy" 😄
Maybe different question, I guess you have tried to find a route on larger area and you saw that it takes a while, it is basically undoable to generate larger areas such as whole cities. I tired to generate Edinburgh and after an hour I had to kill it. I believe that some optimisation (such as multiprocessing) could help. What would be a decent area size for you as a user? Or what is the area that you are trying to generate?

sure, I fully understand the limit.
my area is about 50m2 but mostly rural area. I guess the number of streets and intersections are also determining the complexity.

Yeah, the area it self isn't a problem, even the number of nodes / edges, but the issue are odd-degree nodes. I thought, it might be simpler to and easier to understand to restrict it based on the area size than based on the number of odd-degree nodes.

Hello @matejker,
Your job on that CPP is great, thanks a lot ! However, I've tried many things to save the route as GPX on www.everystreetchallenge.com (ctrl+s, save as, source code display, both on PC & MAC, etc.), but I cannot get anything else than HTML or TXT. Did I miss something ? Because that wold be SO useful !
JB

did you know that you can generate the GPX file on http://www.everystreetchallenge.com?

I think that functionality might be broken? At least just like @1J2B I can't seem to find a way to export the result.

I think that understand your question now. Sure, the jupyter notebook / the Grange example doesn't have the .gpx output. You can do it simply by converting networkx nodes into geo coordinates, e.g. using this function convert_final_path_to_coordinates(org_graph, final_path) and using a newly added gpx_formatter.py. Putting all together, you can try something like:
from libs.tools import *
from libs.gpx_formatter import TEMPLATE, TRACE_POINT

...

# List all edges of the extended graph including original edges and edges from minimal matching
single_edges = [(u, v) for u, v, k in graph.edges]
added_edges = get_shortest_paths(graph, matched_edges_with_weights)
edges = map_osmnx_edges2integers(graph, single_edges + added_edges)

# Finds the Eulerian path
network = Network(len(graph.nodes), edges, weighted=True)
eulerian_path = hierholzer(network)
converted_eulerian_path = convert_integer_path2osmnx_nodes(eulerian_path, graph.nodes())
double_edge_heap = get_double_edge_heap(org_graph)

# Finds the final path with edge IDs
final_path = convert_path(graph, converted_eulerian_path, double_edge_heap)
coordinates_path = convert_final_path_to_coordinates(org_graph, final_path)

# Route statistics from OSMnx
ex = ox.extended_stats(org_graph, ecc=True)
center_node = org_graph.nodes[ex["center"][0]]

trace_points = "\n\t\t\t".join([TRACE_POINT.format(
    lat=lat, lon=lon, id=i, timestamp=datetime.now().isoformat()
) for i, (lat, lon) in enumerate(coordinates_path)])

gpx_payload = TEMPLATE.format(
    name="Name your everystreet route",
    trace_points=trace_points,
    center_lat=center_node["y"],
    center_lon=center_node["x"]
)

with open("gpx_output.gpx", "w") as f:
    f.write(gpx_payload)

I believe this could work 😄

Hi Thanks for this fun tool.
In order to get this GPX output i had to (in addition to install scipy):

add "import datetime"
add an extra "datetime." to get the following:
lat=lat, lon=lon, id=i, timestamp=datetime.datetime.now().isoformat()

find an example (replace .txt by  .ipynb)
example.txt

Function extended_stats was deprecated for a while before it was eventually removed.
use NetworkX :
from libs.gpx_formatter import TEMPLATE, TRACE_POINT
from datetime import datetime
...
# List all edges of the extended graph including original edges and edges from minimal matching
single_edges = [(u, v) for u, v, k in graph.edges]
added_edges = get_shortest_paths(graph, matched_edges_with_weights)
edges = map_osmnx_edges2integers(graph, single_edges + added_edges)

# Finds the Eulerian path
network = Network(len(graph.nodes), edges, weighted=True)
eulerian_path = hierholzer(network)
converted_eulerian_path = convert_integer_path2osmnx_nodes(eulerian_path, graph.nodes())
double_edge_heap = get_double_edge_heap(org_graph)

# Finds the final path with edge IDs
final_path = convert_path(graph, converted_eulerian_path, double_edge_heap)
coordinates_path = convert_final_path_to_coordinates(org_graph, final_path)

eccentricity = nx.eccentricity(graph)
center = nx.center(graph)
center_node = graph.nodes[center[0]]

trace_points = "\n\t\t\t".join([TRACE_POINT.format(
    lat=lat, lon=lon, id=i, timestamp=datetime.now().isoformat()
) for i, (lat, lon) in enumerate(coordinates_path)])

gpx_payload = TEMPLATE.format(
    name="Name your everystreet route",
    trace_points=trace_points,
    center_lat=center_node["y"],
    center_lon=center_node["x"]
)

with open("gpx_output.gpx", "w") as f:
    f.write(gpx_payload)


Clingo only supports 32 bit ints even through it uses a 64 bit model

Clingo version says that it uses a 64 bit address model
clingo --version
clingo version 5.2.2
Address model: 64-bit

libgringo version 5.2.2
Configuration: with Python 3.6.4, without Lua

libclasp version 3.3.3 (libpotassco version 1.0.1)
Configuration: WITH_THREADS=1
Copyright (C) Benjamin Kaufmann

License: The MIT License <https://opensource.org/licenses/MIT>

However, large numbers start to wrap around.
$ echo 'a(2147483648).' | clingo
clingo version 5.2.2
Reading from stdin
Solving...
Answer: 1
a(-2147483648)
SATISFIABLE

Models       : 1
Calls        : 1
Time         : 0.001s (Solving: 0.00s 1st Model: 0.00s Unsat: 0.00s)
CPU Time     : 0.001s

Is this intended behavior? I couldn't find any documentation for this.

This is intended. Supporting 64 bit integers would greatly increase memory consumption.

Thank you for the clarification! Is that limitation documented somewhere? I couldn't find any mention in https://www.mat.unical.it/aspcomp2013/files/ASP-CORE-2.03b.pdf. Would it make sense to add a warning in gringo when numbers overflow or underflow?

Is that limitation documented somewhere?

I can't remember that we put it anywhere. Our guide could be a good place to add this information.

Would it make sense to add a warning in gringo when numbers overflow or underflow?

It might help, but it is not that easy to implement. A lot of places in the code would have to be adjusted. And the C/C++ language does not make it easy to check for these overflows.

It might help, but it is not that easy to implement. A lot of places in the code would have to be adjusted. And the C/C++ language does not make it easy to check for these overflows.

Okay, I thought this could be added to the parser but I thought that without knowing how things are implemented. thanks for the clarifications. For now, I clamp all numbers to max int 32 when I generate ASP programs.

For now, I clamp all numbers to max int 32 when I generate ASP programs.

Be careful when you use arithmetic in your program. Numbers might still overflow. If you do not need to use arithmetic, you could as well use strings, e.g., 1 becomes "1".

I would appreciate 64bit ints. perhaps with a switch as an argument or compilation option.

There is no way to simply increase the size of the numbers supported by clingo.

I would also like to suggest arbitrary large integers. I understand that deviating from 32 bit ints will impact performance but could there be a separate notation for bigints or a cli argument that enables big ints?

Supporting big integers on the grounder side would be possible without sacrificing (much) performance. The solving part only supports minimize/sum constraints over 32 bit. Big integer constraints would have to be supported additionally because the performance hit would be noticeable. Another difficult part would be to expose them via the API. Then, there are also license issues. The gmp is GPL but there are (slower) MIT alternatives.
I'll keep these things in mind but don't expect anything to happen soon.

@rkaminsk thanks for quick response

Hi @rkaminsk, in theory, if I/someone were to create a fork with 64 bit numbers (not biginter), could you please comment on some assumptions/questions?

Implementing this probably isn't as easy as replacing some type definitions. Are there any major obstacles?
Apart from caching disadvantages, roughly the same performance could be expected on most CPUs
The memory consumption would be doubled at most

Hi @rkaminsk, in theory, if I/someone were to create a fork with 64 bit numbers (not biginter), could you please comment on some assumptions/questions?
* Implementing this probably isn't as easy as replacing some type definitions. Are there any major obstacles?

* Apart from caching disadvantages, roughly the same performance could be expected on most CPUs

* The memory consumption would be doubled at most

@kherud: I am currently working on this. I decided to add biginter support because this has the nice side effect that operations no longer overflow.