[SPARK-25818][CORE] WorkDirCleanup should only remove the directory at the beginning of t…

What changes were proposed in this pull request?
The cleanup mechanism will clear all the eligible directories under SPARK_WORK_DIR. If the other configured paths are the same as the SPARK_WORK_DIR configuration, this will cause the file directories of other configuration items to be deleted by mistake. For example, the SPARK_LOCAL_DIRS and SPARK_WORK_DIR settings are the same.
We should add an another condition which start with 'app-'  when removing the app-* directories  in  SPARK_WORK_DIR
How was this patch tested?
manual test

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

IIUC it's not expected to share the SPARK_WORK_DIR with any other usage.

Yes, it happened in our real environment.
The scenario as follows:
Some disk corruption in the production cluster which is normal.
SPARK_LOCAL_DIRS = /data1/bigdata/spark/tmp, disk data1 is broken, the maintenance engineer modified data1 to data2  or another. Unfortunately. the config SPARK_WORK_DIR = /data2/bigdata/spark/tmp, and then we start a Thriftserver process, some temporay folder will be created at the new config path which is the same as SPARK_WORK_DIR, but when the cleanup cycle time is reached, the folder created by Thriftserver will be removed by WorkDirCleanUP, so it will cause the Beeline and JDBC query to fail.
There is a very extreme situation that the user configures the operating system directory, which will cause a lot of trouble. So i think add this condition could reduce some unnecessary risks.

I don't think that's a reasonable usage scenario. That said is there any harm to this change? would it ever miss cleaning something up that it should?

@ouyangxiaochen . Sorry, but the use case sounds like a misconfiguration.

As far as I know, when a spark program is submitted to the cluster, a directory will be created under SPARK_WORK_DIR. The directory name consists of application, timestamp, and five-digit serial number. WorkDirCleanUp should only delete expired application directories. @srowen  Could you tell me if there are other types of directories or files being created under SPARK_WORK_DIR? In addition to the application directory. Thanks!

@srowen Your suggestion is very good, but sometimes the maintenance engineers have limited skills in this area. If they configures the operating system root directory as SPARK_WORK_DIR due to disk damage, it will bring a catastrophic accident. So, I think it is necessary to  add this condition to avoid this production accidents.

I just think that if you have engineers randomly writing and reading stuff in this dir, a bunch of other stuff goes wrong. This is not a problem that Spark can reasonably solve. Certainly, you have much bigger production problems if this level of discipline can't be enforced.


Edit secrets and configmap fields

I have found headlamp recently and I am already liking it a lot, I would love to replace OpenLens with Headlamp for me and my colleagues, but there is one thing blocking the adoption.
We, as devops, have to edit env vars defined in configmaps and secrets for testing purposes and with OpenLens we can edit directly the env vars in a view similar to the read only one of Headlamp, but in Headlamp that view is read only.
I know I can edit the whole configmap and secret, but:

changing one single field is more practical;
secrets are base64 encoded, not a big deal, but I would have to encode the value and then paste it in Headlamp editor;

This is the only issue that is blocking us from adopting completely headlamp to replace both OpenLens and Kubernetes dashboard

Hi @akelge . We do want to add special edit UI for some resources, like what you are saying. I will use this issue to keep track of the feature.

Hi @joaquimrocha, if we haven't started working on this issue, I am more than happy to pick this one up


Highlight Tags not working

I have the same problem. I implemented it the same way as in the provided Test.
boolQuery := elastic.NewBoolQuery().
		Must(elastic.NewQueryStringQuery(queryValue)).
		MustNot(elastic.NewMultiMatchQuery("test", "contenttype"))

	hl := elastic.NewHighlight().
		Field("content").
		PreTags("<em>").
		PostTags("</em>")

	searchResult, err := es.Client.Search().
		Index(index).
		Highlight(hl).
		Query(boolQuery).
		RestTotalHitsAsInt(true).
		From(from).Size(size).
		Pretty(true).
		Do(*es.Ctx)
	if err != nil {
		log.Info("Error performing Elastic Search")
		return nil, err
	}

Originally posted by @claudioontheweb in https://github.com/olivere/elastic/issues/1025#issuecomment-642639662

It's hard to debug without a reproducible test case. E.g. in the example above, I have no idea what the mapping your index looks like. Please provide a complete test case, otherwise we'll be digging for hours.
Notice that there is a unit test for this in the code base, so maybe just change that and send it along.

This is the mapping of the index that I get through the Get Mapping API:
{
   "tools":{
      "mappings":{
         "tools":{
            "properties":{
               "alias":{
                  "type":"text",
                  "fields":{
                     "keyword":{
                        "type":"keyword",
                        "ignore_above":256
                     }
                  }
               },
               "content":{
                  "type":"text",
                  "fields":{
                     "keyword":{
                        "type":"keyword",
                        "ignore_above":256
                     }
                  }
               },
               "name":{
                  "type":"text",
                  "fields":{
                     "keyword":{
                        "type":"keyword",
                        "ignore_above":256
                     }
                  }
               }
            }
         }
      }
   }
}

So my goal is to highlight the search query for the property "content".
I hope this helps.

I just found the reason. I was looking at the source field instead of the highlight field from the Elastic Search response. My bad!


BUG on SSL View certificates

Where the problem was
I was trying to see the certificate that I uploaded on the remote HaProxy-Server.
I was getting the following message, despite the fact the certificate exist and I could see them both on the roxy-ui and on system also the had the correct permission:

Digging a little bit to the code I find by printing the variable "cert_id" that is used in the options.py when:
if form.getvalue('getcert') is not None and serv is not None:
Printing this value figured out that a space is added on the end so when the command is invoked will be something like the following:
openssl x509 -in "/etc/ssl/cert.key "  -text
That thing was invalid on my version of Ubuntu
lsb_release -a
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal
My fix was to alternate the function on funct.py
Before:
def checkAjaxInput(ajax_input: str) -> str:
        pattern = re.compile('[&;|$`]')
        if pattern.search(ajax_input):
                print('error: nice try')
                sys.exit()
        else:
                from shlex import quote
                return quote(ajax_input)

After:
def checkAjaxInput(ajax_input: str) -> str:
        pattern = re.compile('[&;|$`]')
        if pattern.search(ajax_input):
                print('error: nice try')
                sys.exit()
        else:
                from shlex import quote
                return quote(ajax_input.rstrip())

Thank you! Will add it


Fixed selenium exclusion treatment

@TheAngryPython  please try to follow the contributing guidelines
Thanks again, for your interest and participation.

hi @TheAngryPython , this error handling for imports is completely removed, in next version. because its no longer required.
when you will install wappdriver via pip, all python dependencies will get installed.
if somebody clones from git, that person must run
pip install -r requirements.txt


Uber hot reload

This is long term issue depending on other people work.

[ ] client actions, reducers, messages should be hot reloadable
[ ] native as well

You won't be able to hot-reload native side at this stage, mainly because of some limitations https://github.com/mjohnston/react-native-webpack-server/issues. Likely to change when Dan starts working on native tooling from next month on.

That's why I wrote "long term issue".

Sorry, I've missed that and only gone through the points. Fair enough.

Reducers are already hot-reloadable? Actions and mapStateToProps should be hot-reloadable if we replace babel plugin based hot reloading with the old "react-hot-loader" webpack loader.

Msg

@steida is this solved?

No, but there is a little we can do for that.


Add Emotion Recognition under NLP Folder

Hello!! I have an NLP-based Project Idea.
Problem statement: Emotion Recogniser using NLP
I've been working on this already would be happy to add this project here.
Here is a glimpse
I'd Like to work on this issue as part of GSSoC'21.
Thanks.

@DARK-art108  Can I start working on this issue?


1.18.2?

Does this mod work on version 1.18.2? Based on the curseforge files it looks like it just works on 1.18.1. I don’t know how different the 2 versions are so I’m not sure what to do.

Yes and no. It was supposed to but it crashes. I track that issue in #5 so I'll close this one.


Added a JWT verification view

This PR adds a verification view: verify_jwt_token, which checks if a JWT POSTed to it is valid and that the user exists.
Much of the functionality is the same as the existing refresh view and serializer. I abstracted out the majority of the shared serializer logic to maximise DRYness. There are several other ways this could be implemented, however.
I added tests for the new view and moved a test which was more relevant to this functionality from the refresh view tests.
Suggestions for improvements welcome! If the serializer implementation is ok, it could probably use some unit tests.

Also, a lot of the logic in the base serializer is the same as in the JSONWebTokenAuthentication class - apart from the errors raised. I wonder if it would be possible/desirable to abstract those out into shared functions.

@Jwpe thanks, checking this out. I think someone tried abstracting that logic from the JSONWebTokenAuthentication a while back. I wouldn't mind seeing it done. There might also be some logic in JSONWebTokenAuthentication that can be shared as well.

@Jwpe just to be clear, were you gonna tackle the base API view in this PR?

@jpadilla happy to do so, but I can also make a separate PR if you'd rather split it up.

@Jwpe seems right to me if it were part of this one.

@jpadilla refactored to add the base class. Was trying to rebase on the latest master branch, but I'm getting import errors. It seems like PyJWT is no longer in requirements.txt on master - is this intentional? Am I missing something?
===================================== test session starts =====================================
platform linux2 -- Python 2.7.6 -- py-1.4.26 -- pytest-2.5.2
plugins: cov, django
collected 0 items / 4 errors

=========================================== ERRORS ============================================
________________________ ERROR collecting tests/test_authentication.py ________________________
tests/test_authentication.py:13: in <module>
    from rest_framework_jwt import utils
rest_framework_jwt/utils.py:1: in <module>
    import jwt
E   ImportError: No module named jwt
_________________________ ERROR collecting tests/test_serializers.py __________________________
tests/test_serializers.py:8: in <module>
    from rest_framework_jwt.serializers import JSONWebTokenSerializer
rest_framework_jwt/serializers.py:1: in <module>
    import jwt
E   ImportError: No module named jwt
____________________________ ERROR collecting tests/test_utils.py _____________________________
tests/test_utils.py:6: in <module>
    import jwt.exceptions
E   ImportError: No module named jwt.exceptions
____________________________ ERROR collecting tests/test_views.py _____________________________
tests/test_views.py:15: in <module>
    from rest_framework_jwt import utils
rest_framework_jwt/utils.py:1: in <module>
    import jwt
E   ImportError: No module named jwt
=================================== 4 error in 0.05 seconds ===================================

@jpadilla fixed!

@Jwpe this looks perfect, thanks again for the great work! Only thing missing is adding a bit on it to the docs since we are adding a new feature here.

@jpadilla added a section to the documentation. Let me know what you think!

@Jwpe looks great, thanks again!

@jpadilla no worries! Thanks for making an awesome library.


DictionaryInfo-related analysis does not fully track account key and value types

It starts with DictionaryInfo instance equality ignoring those types.
This in turn makes analysis result depend on the order when various DictionaryInfo instances are analyzed and/or added to analysis sets. An excerpt from test case:
var untypedTyped = new VariableDef();
untypedTyped.AddTypes(entry, untypedDict);
untypedTyped.AddTypes(entry, intDict);
var typedUntyped = new VariableDef();
typedUntyped.AddTypes(entry, intDict);
typedUntyped.AddTypes(entry, untypedDict);

Assert.AreEqual(untypedTyped.Types.ToString(), typedUntyped.Types.ToString());

Results in: Assert.AreEqual failed. Expected:<dict()>. Actual:<dict(keys=(Class int, ), values = ()>.

Closing; this code is gone.


UVI Request

--- UVI JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v5.3 to before v5.10.76",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=bf361231c295d92a28ca283ea713f56e93e55796",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=16802fa4c33eb1a8efb23f1e93365190e4047d05"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "bf361231c295d92a28ca283ea713f56e93e55796",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "16802fa4c33eb1a8efb23f1e93365190e4047d05",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "audit: fix possible null-pointer dereference in audit_filter_rules\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.10.76 by commit 16802fa4c33eb1a8efb23f1e93365190e4047d05, it was introduced in version v5.3 by commit bf361231c295d92a28ca283ea713f56e93e55796. For more details please see the references link."
}
--- UVI JSON ---

This issue has been assigned UVI-2021-1002045


brew install not respecting tap when formula name collides with core

brew doctor output
Your system is ready to brew.

Verification

[X] My "brew doctor output" above says Your system is ready to brew. and am still able to reproduce my issue.
[X] I ran brew update twice and am still able to reproduce my issue.
[X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

brew config output
HOMEBREW_VERSION: 4.1.20-2-gd72ee37
ORIGIN: https://github.com/Homebrew/brew
HEAD: d72ee37f60e287e833ef9b1882bc6ca8ded01aea
Last commit: 2 hours ago
Core tap JSON: 13 Nov 15:42 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_BROWSER: open
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: vi
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit arm_firestorm_icestorm
Clang: 15.0.0 build 1500
Git: 2.42.1 => /opt/homebrew/bin/git
Curl: 8.4.0 => /usr/bin/curl
macOS: 14.2-arm64
CLT: <IP_ADDRESS>.1.1694021235
Xcode: 15.0
Rosetta 2: false

What were you trying to do (and why)?
I want to install terraform from hashicorp/tap:
brew install hashicorp/tap/terraform

What happened (include all command output)?
brew is preferring the formula in homebrew/core
What did you expect to happen?
I expect brew to respect the tap prefix in my command and install from there.
Note this definitely seems related to 97b411906d6faaa4713ad784998cb9d8e60c96b5 as when I checkout 4.1.20 (the previous commit) the issue goes away.
Step-by-step reproduction instructions (by running brew commands)
brew install hashicorp/tap/terraform                                                                                ⌘ /opt/homebrew/Library
==> Downloading https://ghcr.io/v2/homebrew/core/terraform/manifests/1.5.7
Already downloaded: /Users/spice/Library/Caches/Homebrew/downloads/2765dc205c0f89c57475889f81e01a98697bdbcbe4f638d40f7af02f7d4e61c4--terraform-1.5.7.bottle_manifest.json
==> Fetching terraform
==> Downloading https://ghcr.io/v2/homebrew/core/terraform/blobs/sha256:f43afa7c6970e1bc768f739829ee589e88fd2b9275f867c5d0be60369ce0772e
Already downloaded: /Users/spice/Library/Caches/Homebrew/downloads/035ebdf8caeb70e899395805af7e655cf61fcda92aaffd3f96172667b2e19081--terraform--1.5.7.arm64_sonoma.bottle.tar.gz
==> Pouring terraform--1.5.7.arm64_sonoma.bottle.tar.gz
🍺  /opt/homebrew/Cellar/terraform/1.5.7: 6 files, 69.7MB
==> Running `brew cleanup terraform`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

The workaround is to install with an abs path:
brew install --formula /opt/homebrew/Library/Taps/hashicorp/homebrew-tap/Formula/terraform.rb

or checkout 4.1.20:
cd /opt/homebrew
git checkout 4.1.20


PR: Fix validation for import as ndarray/array

Description of Changes

[ ] Wrote at least one-line docstrings (for any new functions)
[ ] Added unit test(s) covering the changes (if testable)

[ ] Included a screenshot or animation (if affecting the UI, see Licecap)

Disable array option if numpy.ndarray or numpy.array can't be imported
Issue(s) Resolved

Fixes #11634
Affirmation
By submitting this Pull Request or typing my (user)name below,
I affirm the Developer Certificate of Origin
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.

I certify the above statement is true and correct: dalthviz

Hello @dalthviz! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file spyder/plugins/variableexplorer/widgets/importwizard.py:

Line 69:1: E722 do not use bare 'except'


flukker android licence parameter error

When i try to run

flukker docker --android-licenses
it shows parameter error.
but when i try to run
sdkmanager -licenses
it asked for accepting all licenses and everything was Ok with it.

sample logs:
\AppData\Local\Android\Sdk\cmdline-tools\latest\bin\sdkmanager.bat --licenses
All SDK package licenses accepted.======] 100% Computing updates...

D:\flutter>flutter doctor --android-licenses
CreateProcessW failed 87
Android sdkmanager tool was found, but failed to run
(C:\Users\KBC-ngen\AppData\Local\Android\Sdk\cmdline-tools\latest\bin\sdkmanager.bat): "ProcessException: The parameter
is incorrect.

  Command: C:\Users\KBC-ngen\AppData\Local\Android\Sdk\cmdline-tools\latest\bin\sdkmanager.bat --licenses".
Try re-installing or updating your Android SDK,
visit https://flutter.dev/docs/get-started/install/windows#android-setup for detailed instructions.

D:\flutter>flutter --verbose doctor -v
[ +120 ms] executing: [D:\flutter/] git -c log.showSignature=false log -n 1 --pretty=format:%H
[  +95 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%H
[   +1 ms] 9b2d32b605630f28625709ebd9d78ab3016b2bf6
[   +3 ms] executing: [D:\flutter/] git tag --points-at HEAD
[  +72 ms] Exit code 0 from: git tag --points-at HEAD
[        ] 1.22.6
[   +7 ms] executing: [D:\flutter/] git rev-parse --abbrev-ref --symbolic @{u}
[  +64 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{u}
[        ] origin/stable
[   +1 ms] executing: [D:\flutter/] git ls-remote --get-url origin
[  +63 ms] Exit code 0 from: git ls-remote --get-url origin
[   +1 ms] https://github.com/flutter/flutter.git
[ +110 ms] executing: [D:\flutter/] git rev-parse --abbrev-ref HEAD
[  +76 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[   +1 ms] stable
[  +99 ms] Artifact Instance of 'AndroidMavenArtifacts' is not required, skipping update.
[   +1 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[   +2 ms] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[   +3 ms] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[  +15 ms] Artifact Instance of 'MaterialFonts' is not required, skipping update.
[        ] Artifact Instance of 'GradleWrapper' is not required, skipping update.
[        ] Artifact Instance of 'AndroidMavenArtifacts' is not required, skipping update.
[   +4 ms] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[   +2 ms] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[        ] Artifact Instance of 'FlutterSdk' is not required, skipping update.
[        ] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[   +1 ms] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FontSubsetArtifacts' is not required, skipping update.
[  +19 ms] executing: [D:\flutter/] git rev-parse --abbrev-ref HEAD
[  +65 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[   +1 ms] stable
[   +2 ms] Skipping request to fetchTags - on well known channel stable.
[        ] executing: [D:\flutter/] git tag --points-at HEAD
[  +85 ms] Exit code 0 from: git tag --points-at HEAD
[   +1 ms] 1.22.6
[  +15 ms] executing: C:\Program Files\Android\Android Studio\jre\bin\java -version
[ +132 ms] Exit code 0 from: C:\Program Files\Android\Android Studio\jre\bin\java -version
[   +1 ms] openjdk version "1.8.0_242-release"
           OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
           OpenJDK 64-Bit Server VM (build 25.242-b01, mixed mode)
[  +21 ms] executing: C:\Program Files\Android\Android Studio\jre\bin\java -version
[ +101 ms] Exit code 0 from: C:\Program Files\Android\Android Studio\jre\bin\java -version
[   +1 ms] openjdk version "1.8.0_242-release"
           OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
           OpenJDK 64-Bit Server VM (build 25.242-b01, mixed mode)
[   +6 ms] executing: [D:\flutter/] git -c log.showSignature=false log -n 1 --pretty=format:%ar
[  +64 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%ar
[   +1 ms] 4 weeks ago
[  +77 ms] java -version
[  +15 ms] executing: C:\Users\KBC-ngen\AppData\Local\Android\Sdk\platform-tools\adb.exe devices -l
[  +16 ms] [√] Flutter (Channel stable, 1.22.6, on Microsoft Windows [Version 10.0.18363.1379], locale en-IN)
[   +5 ms]     • Flutter version 1.22.6 at D:\flutter
[   +2 ms]     • Framework revision 9b2d32b605 (4 weeks ago), 2021-01-22 14:36:39 -0800
[   +1 ms]     • Engine revision 2f0af37152
[   +1 ms]     • Dart version 2.10.5
|[ +158 ms] executing: C:\Users\KBC-ngen\AppData\Local\Android\Sdk\cmdline-tools\latest\bin\sdkmanager.bat --licenses
CreateProcessW failed 87
[+2852 ms] Failed to run Android sdk manager: ProcessException: The parameter is incorrect.

                      Command: C:\Users\KBC-ngen\AppData\Local\Android\Sdk\cmdline-tools\latest\bin\sdkmanager.bat
                      --licenses
[   +3 ms] [!] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
[   +1 ms]     • Android SDK at C:\Users\KBC-ngen\AppData\Local\Android\Sdk
[   +1 ms]     • Platform android-30, build-tools 30.0.3
[        ]     • ANDROID_SDK_ROOT = C:\Users\KBC-ngen\AppData\Local\Android\Sdk
[        ]     • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
[        ]     • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
[        ]     X Android license status unknown.
[        ]       Run `flutter doctor --android-licenses` to accept the SDK licenses.
[        ]       See https://flutter.dev/docs/get-started/install/windows#android-setup for more details.
[   +3 ms] [!] Android Studio (version 4.1.0)
[   +1 ms]     • Android Studio at C:\Program Files\Android\Android Studio
[   +1 ms]     X Flutter plugin not installed; this adds Flutter specific functionality.
[        ]     X Dart plugin not installed; this adds Dart specific functionality.
[        ]     • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
[   +2 ms] [!] VS Code (version 1.49.1)
[        ]     • VS Code at C:\Users\KBC ngen\AppData\Local\Programs\Microsoft VS Code
[        ]     X Flutter extension not installed; install from
[        ]       https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter
\[   +9 ms] List of devices attached
           c831ca750408           device product:merlin_in model:M2003J15SC device:merlin transport_id:1

           * daemon not running; starting now at tcp:5037
           * daemon started successfully
[  +10 ms] C:\Users\KBC-ngen\AppData\Local\Android\Sdk\platform-tools\adb.exe -s c831ca750408 shell getprop
|[ +107 ms] ro.hardware = mt6769z
[   +1 ms] ro.build.characteristics = default
[   +5 ms] executing: C:\Users\KBC-ngen\AppData\Local\Android\Sdk\platform-tools\adb.exe devices -l
[  +62 ms] List of devices attached
           c831ca750408           device product:merlin_in model:M2003J15SC device:merlin transport_id:1
[   +3 ms] [√] Connected device (1 available)
[   +1 ms]     • M2003J15SC (mobile) • c831ca750408 • android-arm64 • Android 10 (API 29)
[   +3 ms] ! Doctor found issues in 3 categories.
[   +9 ms] "flutter doctor" took 3,942ms.
[ +261 ms] ensureAnalyticsSent: 255ms
[   +2 ms] Running shutdown hooks
[   +1 ms] Shutdown hooks complete
[   +1 ms] exiting with code 0

D:\flutter>flutter doctor --android-licenses
CreateProcessW failed 87
Android sdkmanager tool was found, but failed to run
(C:\Users\KBC-ngen\AppData\Local\Android\Sdk\cmdline-tools\latest\bin\sdkmanager.bat): "ProcessException: The parameter
is incorrect.

  Command: C:\Users\KBC-ngen\AppData\Local\Android\Sdk\cmdline-tools\latest\bin\sdkmanager.bat --licenses".
Try re-installing or updating your Android SDK,
visit https://flutter.dev/docs/get-started/install/windows#android-setup for detailed instructions.

@kbcece
As the error log suggests Try re-installing or updating your Android SDK,. Can you try the same and see if it works ?
Also check out similar solutions on how to update Android SDK and see if any of them helps.

Without additional information, we are unfortunately not sure how to resolve this issue.
We are therefore reluctantly going to close this bug for now.
Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact description of what happens, logs, and the output of flutter doctor -v.
All system setups can be slightly different, so it's always better to open new issues and reference related issues.


Add a flag to toggle -Zbuild-std, and default to using it

By default, all the debug assertions in the standard library are compiled out, and the only way to get them back is -Zbuild-std.
Even generic debug assertions can catch code that passes the sanitizers. For example, in this PR I'm trying to add more and I found a few bugs in the compiler by doing so: https://github.com/rust-lang/rust/pull/92686
Having these assertions on in general is also a very good idea, because even if the behavior would be caught eventually, catching it at the source is an aid to debugging. Example here: https://reddit.com/r/rust/comments/t87itu/how_can_i_debugreport_possible_compiler_bugs/
Turning on -Zbuild-std with debug assertions enabled introduces a performance regression. I'm trying to squint at a bunch of numbers over here, but things keep shifting around. Maybe I'm just bad at this. My best guess is that turning on -Zbuild-std is a ~20% regression. Still much better than just turning on ASan, which is a ~60% regression.
I'm also a big fan of having these on by default because these debug assertions work everywhere and with all codebases, as opposed to tools like ASan which doesn't necessarily.

I'm not convinced that we want to do this by default, mainly because of the compile time overhead rather than run time overhead.
I think having an option makes a lot of sense, though.

I agree that the compile time overhead is concerning, but cargo-fuzz already enables ASan by default, which has a dramatic effect on the compile time of large codebases.
Building an image fuzz target:
No sanitizers
real	0m52.239s
user	2m0.496s
sys	0m5.073s

ASan
real	1m47.423s
user	3m46.308s
sys	0m8.821s

-Zbuild-std, no sanitizers
real	1m8.773s
user	3m13.047s
sys	0m9.173s

Building a serde-json fuzz target:
No sanitizers
real	0m20.624s
user	0m29.311s
sys	0m2.069s

ASan
real	0m22.219s
user	0m35.924s
sys	0m2.225s

-Zbuild-std, no sanitizers
real	0m37.803s
user	1m44.005s
sys	0m4.873s

The checks added by ASan have not found all that many bugs in the Rust ecosystem, I suspect if/when the PR I linked lands, it will find more UB than the ASan option. So I think you're left making an awkward argument about the value of the UB that ASan detects to continue to justify it as a default.
I am also concerned about codebases supposedly having fuzzing but missing bugs because they didn't turn on some option. We already run into this with Miri. If you say "my code passes Miri" there are a growing number of people who will say "Well did you run RUSTFLAGS="-Zrandomize-layout" MIRIFLAGS="-Zmiri-symbolic-alignment-check -Zmiri-check-number-validity -Zmiri-tag-raw-pointers" cargo miri?" Now of course those are all opt-in for a reason, but as of today I've seen them all find real bugs that otherwise go undetected.

The sanitizer(s) are (or were in the past, at least) required for fuzzing to work at all, as the instrumentation from sanitizers was how it would gather information from the program execution.

When I say "no sanitizers" I mean --sanitizer=none. Even with that, cargo fuzz still passes a number of sanitizer-coverage-* flags.
Looks like coverage doesn't work, because of https://github.com/rust-lang/wg-cargo-std-aware/issues/63 sigh

I agree that the compile time overhead is concerning, but cargo-fuzz already enables ASan by default, which has a dramatic effect on the compile time of large codebases.

This is not an argument to add even more compile time overhead. If anything it motivates disabling ASan by default (which I would be open to! but its default enabling predates my time here, so I'd want to get input on that from @Manishearth and @nagisa).

iirc we enabled ASan because that was the only way at the time to get sanitizer tooling

Okay so my take away thus far in the discussion is that we should:

Have a --build-std flag in cargo fuzz that does -Zbuild-std and has an explanation in the flag's help text for why you might want to pass this flag (but also explain that it has some modest compile time overhead)

Switch from ASan to no sanitizer enabled by default.

Does that sound reasonable?

I think so yeah. We should make sure it's easy to turn on the sanitizers and document both these options prominently

Perhaps raise a warning when asan isn't enabled for a couple releases so people know what's going on?

I'd be happy to see -Zbuild-std integrated somehow as well.
Personally, I think extra 30 seconds to build the standard library are probably inconsequential when the fuzz target will be running for a long time. In my eyes, most of the resources in fuzzing will be spent on actually running the fuzz target itself rather than compiling them, and so optimizing for the chance to detect an issue might make more sense?

This also works for me; a slight worry i have is that the instrumented stdlib will slow down fuzzing itself, but I suspect it's not too bad.

:facepalm: cargo-fuzz already supports passing through -Zbuild-std to Cargo, all you have to do is run cargo fuzz run fuzz-target -Zbuild-std. So it's already possible but somehow I managed to not notice this until I opened up the CLI code.
I'm still not opposed to turning off ASan by default with a warning, but should there be a second way to turn on build-std?

In general I think we add generally useful rustc options as separate flags so they're more prominent in --help and such

cargo fuzz coverage and --build-std are incompatible at the moment: https://github.com/rust-lang/wg-cargo-std-aware/issues/63. I added a check that should bail out correctly if a user attempts to use a build configuration that won't work.

Assuming that

Personally, I think extra 30 seconds to build the standard library are probably inconsequential when the fuzz target will be running for a long time.

Is true and that enabling these flags would indeed find more issues then I don't really see a reason not to? Though, I am also assuming that it building the std is visible to the user and that sooner or later a better way to get these checks is available.
The reasoning for this is simple: It sounds like those 30 extra seconds actually save time in the long run as it means that people don't have to run it a second time because they forgot to enable them.
Also, if you go out of your way to fuzz your crate, you do it to find bugs. And though options are nice to sacrifice some detection in favor of speed, I think that in general people care more about the amount of bugs that get found by fuzzing then how long the fuzzing takes.
So, TL;DR: I vote for enabling both Asan and build-std by default but make it clear to the user that both are enabled and make it easy for the user to find out what their benefits and downsides are.

I am also concerned about codebases supposedly having fuzzing but missing bugs because they didn't turn on some option. We already run into this with Miri. If you say "my code passes Miri" there are a growing number of people who will say "Well did you run RUSTFLAGS="-Zrandomize-layout" MIRIFLAGS="-Zmiri-symbolic-alignment-check -Zmiri-check-number-validity -Zmiri-tag-raw-pointers" cargo miri?" Now of course those are all opt-in for a reason, but as of today I've seen them all find real bugs that otherwise go undetected.

For both fuzzing and Miri, this seems like a situation where it might be good to have "profiles" to pick from, in the same sense that Cargo offers profiles of compiler options (dev/release/test/bench). These profiles could provide different sets of “good defaults” with documented intended use cases, so that a user can choose one and get good-enough results, while still allowing customization by additional flags.
This would improve on specifying individual flags because:

Figuring out what each flag does may be intimidating to people who have heard fuzzing or running under Miri is worth doing but aren't familiar with the theory behind the things the flags talk about. ("What's a sanitizer?")
Specific functionality described by flags may be added, removed, or replaced with different flags as compiler/fuzzer/miri architecture changes, and it would be easier to know how to update an old configuration if it is primarily based on a profile of documented intent, with only customizations as separate flags.

Some possible use-cases/profiles:

"Turn on everything!"
"Lots of low-hanging fruit, so please keep compile times fast while I iterate on fixing bugs"
"I'm testing a non-unsafe algorithm for edge cases; don't bother with sanitizers because the bugs I care about will manifest as panics"

(for transparency, i was linked to this thread to leave a comment by saethlin)
One bug that shows up a fair bit in binary parsing that asan does catch is OOM.  Without, it either ignores it, or presumably just aborts on OOM. So even if asan doesn't catch any safety issues, catching OOM issues (where people read a u32 or u64 length from some file, and then do vec![0_u8; length] without validating the length actually makes sense) is very worth the compile time hit.
Though I generally do run with -snone if I know that's not an issue and I'm looking more for panics, I've not found the compile time hit of asan to be that significant.
I generally don't go out of my way to -Zbuild-std, but I've tried it now with the serde_json fuzzer and the build time is definitely what I'd call perfectly reasonable for a fuzz target, it's something I'm fine with being the default, personally.
I'd definitely appreciate some way to catch internal invariants being broken, especially since you can't fuzz with miri (imagine the executions per second :fear:), at least some debug assertions are good.

From the benchmarks above, -Zbuild-std looks negligible in terms of perf to compile time, so I think making it on by default is good. It does look to have actually a pretty significant hit to executions per second (13k or so with serde with -Zbuild-std, 25k or so without), which is what I'd be more concerned about though.
That's not too bad, but you'd definitely want to turn that off if you're fuzzing safe code where you're not looking to hit the debug assertions. So I'm going with "on by default but make it easy to turn off"
As for asan, it's much heavier a compile time hit and runtime hit (without -Zbuild-std, -snone gets around 80k, asan gets 25k as said above).
This is already easy to turn off, so I'm inclined to also say leave it on by default, making the defaults "slow but accurate" in terms of how many bugs will be caught. You'd need to leave the fuzzer on for maybe 2-8x longer than you would otherwise, but I'm personally fine with just teaching users to "pass -snone --no-build-std if you're fuzzing safe code that doesn't have a risk of OOMing".

Choosing a profile could be mandatory, so that there is no need to consider what the defaults in the absence of a profile would be.

I agree with this. No need to figure out what configuration people would most likely be using if there are no defaults. It also gives a perfect opportunity to teach the user about what options are available.

According to this Mozilla blog post, Thread Sanitizer will not detect some of the races unless -Z build-std is used. I assume this also applies to Address Sanitizer, since some allocations are not instrumented.
I made -Z build-std mandatory for Memory Sanitizer earlier because not doing so results in false positives; but I fear for other sanitizers not having it results in false negatives.
So I think what really should be discussed is whether debug assertions are enabled by default, not -Z build-std. We probably should have been enabling that all along.

This is already easy to turn off, so I'm inclined to also say leave it on by default, making the defaults "slow but accurate" in terms of how many bugs will be caught

Agreed. Fuzzing, just like cryptography, is a field where you get no feedback when you're doing something horribly wrong. It seems like everything is working, when in reality it's not. So just like cryptography, fuzzing should default to the most robust option that makes it really hard to mess up. Any optimizations should be opt-in for people who really know what they're doing.
I am reminded of CryptoCat developers infamously setting "1" as the RSA exponent because that was much faster than the alternatives, which led to the data not being encrypted at all (1 to any power is always 1).
I really like cargo-fuzz especially because it enables ASAN by default, so I can just tell anyone "use cargo-fuzz" and all the bugs will be detected, instead of saying "oh, use cargo-fuzz, and invoke it with this and that flag", and see their eyes glaze over.

I am commenting here because the motivating PR with a bunch of debug assertions in core was just merged.
I agree very strongly with @Shnatsel's point, and I think that by default we should do the most checks possible. If it were up to me, we'd have ASan + build-std + debug_assertions on by default. If people want to turn off debug assertions or build-std, they can turn off debug assertions in Cargo.toml and they can pass --build-std=false.
I sort of agree with @kpreid's suggestion, in theory. I think that usage of cargo-fuzz is easily categorized but when it comes down to picking names for the profiles and deciding exactly what will be in them I'm at a loss. ASan, TSan, and MSan all conflict with each other, so we need either at least 4 profiles, or some mixture of profile flags and sanitizer flags, at which point we don't really have fewer flags than stipulating all the individual options. And thinking back, I've talked to a few users of cargo-fuzz who don't want to turn off ASan, even if the only unsafe code in their build is in the standard library. Whether or not this decision is well-informed is somewhat besides the point; I think some people would be bothered to learn it had been toggled off for them.
In terms of rapid iteration speed, -Zbuild-std and debug_assertions are hardly impactful. From the extra assertions that were just added, I measured a 0.5% regression on the runtime of ./x.py test. The standard library is just another dependency, it basically doesn't alter your iteration speed, only your cold build time.

I sort of agree with @kpreid's suggestion, in theory. I think that usage of cargo-fuzz is easily categorized but when it comes down to picking names for the profiles and deciding exactly what will be in them I'm at a loss. ASan, TSan, and MSan all conflict with each other, so we need either at least 4 profiles, or some mixture of profile flags and sanitizer flags, at which point we don't really have fewer flags than stipulating all the individual options.

Ah, when I wrote that suggestion I wasn't aware there were conflicts. The first idea I think of to address that is: for an "everything" profile, why not compile the fuzz target with the incompatible sanitizers separately, and run all of them?
Support for running more than one fuzz target binary (whether in parallel or interleaved) would also benefit users with several fuzz targets who want to throw some CPU time at checking everything (after making an infrastructure change that potential affects multiple entry-points, or in CI). This is a feature I'd like to use, even independent of any question of flag-combinations.

why not compile the fuzz target with the incompatible sanitizers separately, and run all of them?

That sounds awesome and out of scope for this PR :sweat_smile:
Turning on various sanitizers requires requires different flags to cargo, so now we're talking about invoking cargo in parallel with different target directories, and also the fuzzing targets want to write to stdout/stderr so where are we going with that... yeah.

I changed this PR back to default to turning on -Zbuild-std. I think this is the best default because it detects more bugs, which is the whole point of fuzzing. Case in point, I had to fix one of the tests because it was UB on two accounts, and defaulting to -Zbuild-std detected one kind of UB before the UB that was supposed to be detected.

@saethlin Address Sanitizer already detects both out-of-bounds indexing and improper use of copy_nonoverlapping, and it is already enabled by default, so I don't think this provides much benefit as a default.

Address Sanitizer detects reads and writes which are out-of-bounds of the allocation, which is a significantly more permissive check than what the standard library requires. For example, this program executes UB which is caught by the standard library debug assertions but not caught by Address Sanitizer:
fn main() {
    let v = vec![1, 2, 3];
    let sli = &v[..1];
    let _ouch: &i32 = unsafe { sli.get_unchecked(1) };
}

And it is still not caught if we access through the reference.
Address Sanitizer also does not catch misuse of copy_nonoverlapping. It probably has support for memcpy, but copy_nonoverlapping lowers to whatever suits. For example, this program executes UB which is caught by the standard library debug assertions but not caught by Address Sanitizer:
fn main() {
    let mut v = vec![1, 2, 3];
    let ptr = v.as_mut_ptr();
    unsafe { core::ptr::copy_nonoverlapping(ptr, ptr, 1) };
}

Superseded by #339


[NEW ICON] Visual Basic .NET

Skill Name
Visual Basic .NET
Why?
According to Stack Overflow Survey , Visual Basic.NET is still one of the most used languages out there among the top 30
and pretty much every language here above and under VB.NET already have an icon except VB.NET
https://survey.stackoverflow.co/2023/#most-popular-technologies-language-prof
I feel like as a general idea if a language can score more than 1% on these surveys it should have it's own icon
Reference Image
https://en.wikipedia.org/wiki/Visual_Basic_(.NET)#/media/File:VB.NET_Logo.svg

Yea. We need this icon


Handle BOM on utf8 files causing JSON Parse errors

Using the TextDecoder solves the problem where an appsettings file had BOM on the front and was breaking the JSON Parsing

appsettings.WorkdayService.txt
Needed to change file ending to txt, but this is the actual json file that was causing problems, doing a raw octal output (od) shows characters 357 273 277   at the beginning of the file that needs to be stripped out before calling JSON.parse.


Zulip room not linked

The part in the README.md that would link to the zulip room is blank.

(Oops there's already a PR about this.
But it's 9 days old so an issue is fair I guess)


Update interests Postolache

👷 Deploy Preview for gladia processing.

Name
Link

🔨 Latest commit
16968860da096db1233ab1303de545f4b661f220

🔍 Latest deploy log
https://app.netlify.com/sites/gladia/deploys/63df9bfbc93e7e0008e2f9cb


Feature Request: Ability to add an argument file with exe

So you can just click the exe instead of needing to open a command line and add your arguments every time.
File in the same folder as the exe could just be something like arguments.txt with the contents being something like --vulkan_large_heap_block_size 0 --local_tank_cache="Z:\shark\SHARK\shark_tank" --use_base_vae --ckpt_dir "Z:\cust_models\PhotoMerge" --custom_vae "Z:\amby\diffusers\vae\vae-ft-mse-840000-ema-pruned_diffusers" --output_dir="Z:\shark\SHARK\output"

you can just make a shortcut to the exe and add the arguments to the start path, by right clicking and going into properties.

you can just make a shortcut to the exe and add the arguments to the target path, by right clicking and going into properties. Remember to keep the target path in quotes, then add the arguments after them.

thanks for the recommendation, but the shortcut length is too small. it cuts off " --output_dir="Z:\shark\SHARK_EXE\output", only keeping --vulkan_large_heap_block_size 0 --local_tank_cache="Z:\shark\SHARK_EXE\shark_tank" --use_base_vae --ckpt_dir "Z:\cust_models\PhotoMerge" --custom_vae "Z:\amby\diffusers\vae\vae-ft-mse-840000-ema-pruned_diffusers

huh... didn't know there was a limit. Just make a batch file in that case.

huh... didn't know there was a limit. Just make a batch file in that case.

im using a powershell script for now, but i feel like this feature would be better because then you can update the exe whenever and it'll just load from the txt.

Not a bad idea, but probably pretty low priority for us, as for most users that want advanced flags will be able to figure out a batch file or ps1 script etc.
Welcoming community contributions though!


Update jquery.js reference

jquery is no longer included at the path specified. Shoudl be in the admin directory

Current coverage is 70.23% (diff: 100%)

Merging #6100 into master will not change coverage

@@             master      #6100   diff @@
==========================================
  Files           416        416
  Lines         35711      35711
  Methods        4688       4688
  Messages          0          0
  Branches          0          0
==========================================
  Hits          25083      25083
  Misses        10628      10628
  Partials          0          0

Powered by Codecov. Last update 5a2e2b7...b2d07a1

I think it also needs to be fixed in config-form.html
This way the js-errors on the install.php page are also fixed...

We're looking at pulling admin out into its own module soon so this isn't a long term fix. Better to use Google's CDN?

Which version of jquery shoud be we be fulling from the CDN?

simple uses 1.7.2, so probably that...

thanks

@baukezwaan sorry, I missed your comment some how! Would you mind opening a PR to fix config-form.html?


[BUG] Lifecycle tags in node_group prevents nodegroup from being changed

Description
Changing instance types in node groups requires a manual workaround because lifecycle tags are incorrect
If I try to switch an instance type eg t3.small -> t3.medium or even just adding more types I cannot apply without performing a targeted delete or simply removing the node group.
The issue is https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/modules/node_groups/node_groups.tf#L84 attempts to create a replacement identically named node group before removing the old one.
⚠ Note
Before you submit an issue, please perform the following first:

Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
Re-initialize the project root to pull down modules: terraform init
Re-attempt your terraform plan or apply and check if the issue still persists

All of these steps are performed by our CI server
Versions

Terraform: 15.4
Provider(s):
Module:

Reproduction

Create cluster with node group
Change the node group definition
Attempt to apply the changes

Code Snippet to Reproduce
node_groups = [
    {
        name                    = "change-me"
        capacity_type           = "SPOT"
        instance_types          = ["t3.medium"]
        max_capacity            = "2"
        min_capacity            = "1"
        desired_capacity        = "1"
        disk_size               = 20
        additional_tags = {
            project_cost_type   =   "Shared"
        }
        k8s_labels = {
            "example.com/nodetype"   = "changeme"
        }
    }
]

Replace instance_types          = ["t3.medium"] with instance_types          = ["t3a.medium"]  after the eks cluster has been created and attempt to re-apply  (note it plans fine)
Expected behavior
Terraform should apply the change without requiring terraform destroy -target .... or similar
Actual behavior
error creating EKS Node Group (example:change-me): ResourceInUseException: NodeGroup already exists with name change-me and cluster name example
RespMetadata: {
    StatusCode: 409,
    RequestID: "< removed uuid>"
,
  ClusterName: "example",
  Message_: "NodeGroup already exists with name change-me and cluster name example",
  NodegroupName: "change-me"
```

+1  we are ranning to the same issue.

I don't understand why this bug is closed. And the whole argument like it's a normal behaviour and a limitation of the AWS API doesn't make sense to me.
The node group name must be unique. Because of this, the create_before_destroy = true , the module tries to create a node group name with the same name before delete it then it fails and that's it. Without the create_before_destroy = true it will work, there is no such thing like AWS limitation here!
I dig the history. In the first version of file, there was create_before_destroy = true but the node group name was unique because generated by a random_pet suffix to avoid name collision --> https://github.com/terraform-aws-modules/terraform-aws-eks/blob/11147e9af34054c4c4576aa00938a2c65198ca5f/modules/node_groups/node_groups.tf#L4
In this PR https://github.com/terraform-aws-modules/terraform-aws-eks/pull/1372 the random_pet suffix was removed hence the node group name is not unique on re-creation, and since create_before_destroy = true has not been removed there is now this name collision bug. Once again regression, not AWS limitation.
Btw from https://www.terraform.io/docs/language/meta-arguments/lifecycle.html#create_before_destroy

This is an opt-in behavior because many remote object types have unique name requirements or other constraints that must be accommodated for both a new and an old object to exist concurrently. Some resource types offer special options to append a random suffix onto each object name to avoid collisions, for example. Terraform CLI cannot automatically activate such features, so you must understand the constraints for each resource type before using create_before_destroy with it.

let me analyze this again

Digging a bit more I found this https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/upgrades.md#upgrade-module-to-v1700-for-managed-node-groups

After the first apply, we recommand you to create a new node group and let the module use the node_group_name_prefix (by removing the name argument) to generate names and avoid collision during node groups re-creation if needed, because the lifce cycle is create_before_destroy = true.

I haven't try it (in my next upgrade I will) but I think there might be no name collision if using name_prefix instead of name --> https://github.com/terraform-aws-modules/terraform-aws-eks/blob/a9b84558b1abb4ca2a4dd07f571439ccfff79f9d/modules/node_groups/locals.tf#L35
If my understanding is correct, when using prefix_name, the node group are suffixed by an increment to avoid name collision.

I checked and in general lifecycle hook is there to avoid a downtime  for workload when you modify node groups which are using name_prefix which is default behaviour when you dont pass name` input into node_group definition.
Unfortunately lifecycle meta arguments cannot be conditional in terraform what is causing that we must choose which issue from two is acceptable:

issue described in this thread
causing downtime for workload running on node group each time even if there is use of name_prefix used.

From my side I think we must accept issue 1, as issue 2 will make this module will be not allowing for rolling updates at all.

@damienleger  thank you for putting attention on this 🥇

Thank you @daroga0002 for the quick look. Yes I agree with you using name_prefix is the safest solution. So if I'm correct we can use (1) name, (2) name_prefix or neither of both then it (3) fallbacks to name_prefix logic. If I may, can you add in the doc the difference between (2) and (3)? From the tf code I must admit it's not clear to me 😕


Cannot Update via APT Becuase of Weak Security

Describe the bug
I have installed rtx via apt using the manual in Ubuntu 23.10, and cannot update it. The error apt update is showing me is this:
W: No Hash entry in Release file /var/lib/apt/lists/partial/rtx.pub_deb_dists_stable_InRelease
E: The repository 'https://rtx.pub/deb stable InRelease' provides only weak security information.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

To Reproduce
Install rtx by adding its repository and calling apt update.
Expected behavior
A clear and concise description of what you expected to happen.
rtx doctor output
rtx version:
  2023.12.9 linux-x64 (989f505 2023-12-06)

build:
  Target: x86_64-unknown-linux-gnu
  Features: DEFAULT, NATIVE_TLS, OPENSSL
  Built: Wed, 6 Dec 2023 04:35:08 +0000
  Rust Version: rustc 1.74.0 (79e9716c9 2023-11-13)
  Profile: release

shell:
  /usr/bin/zsh
  zsh 5.9 (x86_64-ubuntu-linux-gnu)

rtx data directory:
  /home/farhood/.local/share/rtx

rtx environment variables:
  RTX_SHELL=zsh

settings:
  {"always_keep_download": "false", "always_keep_install": "false", "asdf_compat": "false", "disable_default_shorthands": "false", "disable_tools": "[]", "experimental": "false", "jobs": "4", "legacy_version_file": "true", "legacy_version_file_disable_tools": "[]", "plugin_autoupdate_last_check_duration": "10080", "raw": "false", "trusted_config_paths": "[]", "verbose": "false", "yes": "false"}

config files:
  /home/farhood/.config/rtx/config.toml

plugins:
  dockle     https://github.com/mathew-fleisch/asdf-dockle.git#0af3e32
  go         (core)
  hadolint   https://github.com/devlincashman/asdf-hadolint.git#335e230
  java       (core)
  just       https://github.com/olofvndrhr/asdf-just.git#93771e1
  node       (core)
  python     (core)
  ruby       (core)

toolset:
  node@21, <EMAIL_ADDRESS><EMAIL_ADDRESS>
1 problem found:
new rtx version 2023.12.18 available, currently on 2023.12.9

Additional context
I don't have no other info.

I can repro with the apt testing instructions.

curl -v https://rtx.pub/deb

this actually isn't necessary, apt doesn't look for anything at this location, it just uses it as a prefix for https://rtx.pub/deb/dists/stable/InRelease which is working, however the checksums are blank which is probably the issue


Editor actions are hard to discover when tabs are enabled

Due to limited space in a tab, we put the editor actions into the context menu. This makes them much harder to use and explore compared to having them visible with icons in the title header when tabs are disabled.

Since this is a somewhat adoption blocker for people in the team to use tabs, we should look into this.

We also have an issue that the path on a tab is only accessible from the native hover (which sometimes appears late). So I wonder if maybe we could combine both issues into one and show the path of the file along with the editor actions somewhere when you hover over the tab. Or, if we decide to introduce an editor toolbar, maybe it could be in there?
I think for this issue we should just sit together and brainstorm UX ideas.

when the filename is very long, a tab takes lots of width. the name should be shortened with an hellipsis character.
Instead of 'thisisaverylongfilename.txt' it might only show 'thisisav…e.txt'.

UX Meeting Summary:
We will put local commands back where they used to be before we introduced tabs. But we don’t want to mix group-level and editor-level commands in the same area. Thus a few things need to change.  See #9159 for more info...

I pushed a first cut of this work, feedback welcome. Nothing changed for when tabs are disabled but showing it here for the sake of completeness:
Tabs Disabled
Editor actions show up in the right corner with an additional X button to close the editor:

For the diff editor:

If the group is inactive we only show the X button to close the editor:

You can right click in the title to trigger a context menu with additional commands:

Tabs Enabled
Editor actions show up in the right corner:

For the diff editor:

The overflow icon moved into the "..." and so there are always two entries in the "..." to show all opened editors and close all:

For the diff editor (which is the only one to add additional secondary actions currently):

If the editor group is inactive we show "..." with the 2 actions:

You can right click on a tab to trigger a context menu with additional commands:

There are things about this solution that are questionable:

the fact that the overflow icon is gone and moved into the "..." where it is harder to find (this can be solved by having an additional overflow indication in the tabs container itself, e.g. where the scrollbar is)
the fact that for an inactive group I only show "..." as the action violates our action contract where we say that "..." can only be there if there are other primary actions (we could just not show any "..." for inactive groups however people might want a quick way to "Close All" even for inactive groups).

Just to add more complexity there are now also menu contributions. Specifically editor/title which targets depending on the group property is either in the action bar or the overflow menu - see markdown or Bookmarks extensions.

Is this the reason why the current version of vscode is not showing command icons defined like:
      {
        "command": "i18nTag.validateTranslation",
        "category": "i18nTag",
        "title": "Validate translation file",
        "icon": {
          "light": "./media/validate_icon.svg",
          "dark": "./media/validate_icon_inverted.svg"
        }
      }

or do I have to add the media files somehow to the release output of my extension?

Just talked over standup with the PMs.  We can go ahead and test this with insiders over the next 3 weeks and monitor feedback from insiders.  If there are any big gotchas, we can revert before the July release.
When you're back we can further iterate.

@bgashler1 sounds good. I would not go as far as to revert this change entirely, imho the only questionable thing is the overflow action. Let me push a boolean flag to bring it back and instruct you guys here how to enable that in case needed!

@skolmer I depends where you let the command show. Currently, only editor/title with group=navigation shows icons. Context menus don't show icons

@jrieken thank you! I think that's it. I used a custom group and not group=navigation

I added a flag to https://github.com/Microsoft/vscode/blob/master/src/vs/workbench/browser/parts/editor/titleControl.ts#L56
@isidorn is fit enough for July to tweak this experience as needed 👍

Verified via https://github.com/Microsoft/vscode/issues/9701


feat(clis,specs): Tests marked as slow have greater t8n server timeout

🗒 Description
Allow tests marked as slow (with @pytest.mark.slow) to have a higher timeout when making a request to the t8n server.
I was writing a test to verify the maximum discount BLS multi-scalar-multiplication and it takes the t8n server 30 seconds to respond, which exceeds the 20 second timeout that is hard-coded by default.
🔗 Related Issues
None
✅ Checklist

[x] All: Set appropriate labels for the changes.
[x] All: Considered squashing commits to improve commit history.
[ ] All: Added an entry to CHANGELOG.md.
[x] All: Considered updating the online docs in the ./docs/ directory.
[x] Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
[x] Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
[x] Tests: Included the type and version of evm t8n tool used to locally execute test cases:  e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
[x] Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

That's a good flag, but not to confuse with time consuming. Some tests we have just run so many loops or have so many transactions, I marked them as time-consuming to be able to disable from test runs if needed

not to confuse with time consuming

Indeed, I think  it should be fine by now to not be so granular, but if required we could prepare separate markers for:

A single t8n call is the cause for execution delay
Test requires multiple block fills and hence the slowness

Also timeout is same for all t8ns?
This might be EELS only, but need to check with other fillers


model_pruning: Why 50%  and 90% zeros of the stripped models are the same size?

I'm trying tensorflow/contrib/model_pruning/examples/cifar10.  I use strip_pruning_vars to  remove pruning ops from the trained graph, strip_pruning_vars  shows 50% and 90% zeros respectively, but their final model are the same size.

As described in the doc，“For now, it is assumed that the underlying hardware platform will provide mechanisms for compressing the sparse tensors and/or accelerating the sparse tensor computations“
Does it mean that model pruning requires hardware or inference engine support in sparse matrices?
In the current version, can I implement inference acceleration on x86 or GPU with the model pruning?

Apologies for the delay in response.
This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there.


Fix - dotnet test on a multi-target projects logs only the last target

Related issue
https://github.com/Microsoft/vstest/issues/1603
Description
Timestamp is appended to trx filename with millisecond precision. Thus for multi-target projects, trx for both targets will be stored.
Example: dotnet test testproject.csproj --logger:"trx;logFileName=E:\results.trx" will save trx file E:\results_2018-12-24_14-01-07-176.trx where 2018-12-24_14-01-07-176 is the timestamp.

Hello! I'm using version "16.0.0-preview-20190203-03" which, if I interpret the roadmap doc correctly, should include this PR.
However, when I execute dotnet test --logger "trx;logFileName=C:\Path\File.trx" against a solution containing multiple test projects the result file is not appending the timestamp mentioned above and thus the result file is overwritten by the next test project. I also tried executing the command against a single csproj file with the same result (trx file without timestamp).
Any idea of what I could be doing wrong?

What if the timestamp is unwanted? I don't see an option to omit it for single-target projects. This breaks our CI workflow.

Thanks for breaking all our ci workflows.

This change is has some problems.

VSTest inventing directories that don't exist because the string.Replace is hitting all containing directory names, not just the file name.
The filename is unsightly and there's no way to turn this off.
Our CI scripts have to guess at the resulting filename based on the directory listing or else direct VSTest to a temp folder and copy TRX files back out with proper filenames.


Can't launch clipboard

When I hit Ctrl+Shift+V, nothing happened. When I right click the Snatch logo in the tray, only show "exit". I try to reboot my PC with "Safe Mode", but I found I can't even launch Snatch——nothing is showed in the tray. Other apps run normally in my computer. What can I do?

I am also facing same issue


Lab1-5: Fix person fields to contain a First and a Last name

"the key is the person’s name (when comparing keys first compare last names; then if last names are the same compare first names"
The Person class was previously implemented with merely a Name field: there needs to be two names and the insert, delete, and searching algorithms must re-account for this.

Fixed in pull request #11 .


Ignore alphabetical properties rule in stylelint

Purpose
Alphabetizing properties doesn't significantly improve readability. More importantly, I want to be able to add comments for related things. For example:
.MyButton {
    /* This is wacky */
    background-color: rebeccapurple;
    color: gray(30);

    /* Reset styles */
    border: none;
    margin: 0;
    width: auto;
    overflow: visible;
}

Summary of Changes
Ignore the alphabetical property rule in stylelint
Side Effects / Testing Notes
Ticket
No ticket
Reviewer Checklist

[ ] meets requirements
[ ] easy to understand
[ ] DRY
[ ] testable and includes test(s)
[ ] changes described in CHANGELOG.md

Coverage remained the same at 4.673% when pulling 8fdb045382c562ce22c082b3c5c45f1cd71edbcc on feature/ignore-bad-rule into 4bb18908b0bfebf900c473dfcdf055f8bf1928a9 on develop.

Thanks, @steve!


Mod Support

Hello! I came across your Reddit post with the link to your build planner and github  and have now been using it for, well, ever.
I wanted to know if you still take feature requests and to be more specific: mod integration requests. If so there is a recent mod called Hephas Occupations and Traits. Found Here: https://steamcommunity.com/sharedfiles/filedetails/?id=3256482412. Thank you for all your hard work regardless.

Sure, if the author allows me to re-upload their icons here, I'll add support for it.

Support for Hephas Occupations and Traits has been added.

Thank you very much! Pretty quick too!


 We absolutely should not merge this, because it increases the resting size of an ipset, and doesn't deal with most of the core problems. This PR is open only to provide a clue to a temporary patch for anyone who encounters these problems.

I think this PR can be closed down - we are never going to do this, and it's not worth keeping the PR open. @Lukasa do you agree? It might be that I'm missing a customer short term need for it.


doc(decision-tree): update operator/function names

Description: updates to the operator decision tree widget's content
Related issue (if exists): #3900

Coverage remained the same at 96.747% when pulling e56774f4ff7d05d936e7144fbbc5a0e2b6097da7 on mtaran-google:tree into f9318d898d6c017652a84202ccaebb440397016c on ReactiveX:master.

In my opinion these changes shouldn't be part of the old docs. I thought we use the /doc folder for documenting rxjs < 6? I'm not sure if we want to mix it up

@JWO719 I'm not sure what's supposed to be where, but there is already some v6-specific documentation in /doc directory: doc/marble-testing.md.

@cartant I totally share your opinion on that, but we should agree on an intention of the /doc folder

Thanks, @mtaran-google


Type error when extending component with generic prop [🐞]

Which component is affected?
Qwik Runtime
Describe the bug
What I am doing:
I want to extend a component that accepts a generic Prop object.
// Message.tsx
import { component$ } from<EMAIL_ADDRESS>
export type MessageProps<T> = {
  message: T | undefined;
  style?: string;
}

export const Message = component$(function <T>(props: MessageProps<T>) {
  const { message, ...other } = props;
  return
{message?.toString()}

})

// ColoredMessage.tsx

import { component$ } from<EMAIL_ADDRESS>import { Message, type MessageProps } from "./Message";

type Props<T> = MessageProps<T> & {
  color: string;
}

export const ColoredMessage = component$(function <T>(props: Props<T>) {
  // The TS error appears in '<Message'
  return <Message {...props} style={`color: ${props.color};`} />
})

What I expect:
No error 😄 .
What's actually happening:
I receive this error:
Type '{ style: string; message: T | undefined; color: string; }' is not assignable to type 'IntrinsicAttributes & TransformProps<MessageProps<T>> & ComponentBaseProps & { children?: JSXChildren; }'.
  Type '{ style: string; message: T | undefined; color: string; }' is not assignable to type 'TransformProps<MessageProps<T>>'.
    Types of property 'message' are incompatible.
      Type 'T | undefined' is not assignable to type 'TransformProp<T | undefined>'.
        Type 'undefined' is not assignable to type 'TransformProp<T | undefined>'.

Type 'T | undefined' is not assignable to type 'TransformProp<T | undefined>'.

Reproduction
https://stackblitz.com/edit/qwik-starter-cyipju?file=src%2Fcomponents%2FColoredMessage.tsx
Steps to reproduce
It renders correctly, the error appears on the editor.
System Info
System:
    OS: macOS 14.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 8.23 GB / 64.00 GB
    Shell: 3.5.1 - /opt/homebrew/bin/fish
  Binaries:
    Node: 20.8.0 - ~/.asdf/installs/nodejs/20.8.0/bin/node
    Yarn: 1.22.19 - ~/.asdf/shims/yarn
    npm: 10.1.0 - ~/.asdf/plugins/nodejs/shims/npm
    Watchman: 2<IP_ADDRESS> - /opt/homebrew/bin/watchman
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Chrome: 119.0.6045.123
    Firefox: 119.0.1
    Safari: 17.1

Additional Information
Thank you!

TypeScript doesn't work that way - you can't make a constant generic and all Qwik Components are constants.
Here's an approach that might work for you: https://qwik.builder.io/playground/#v=1.1.5&f=7VY9a8MwEN39K4Qo%2BQCjDh0Kjq0MpUOHbtlKIaVxITR1SpQMwdF%2Fz9PpFNly%2FkAgo8%2BW7r13d8%2FXaZrnJ5TvUbzXxmC6XCgqHbshFxfzzAnTGwq%2Bbsz6%2B3VTe1Pmisi0gjKqQnXnTGjpf1MuNPRxyvz5aCEW4iRg6fUPnGU1wxuyy3nBqwTgi9F1BB9j7Crjz6QGnA5pIpuyh%2BHQ%2FOI%2F2mg9mdByc6mWv6AN2HKhlNrCyHfgWvlFyAEM44Hsog2fWN3ysbnah9aaWr9PYW6yDKK%2FbDew6lVX%2B4w06oiTyjViaDjZ0SQh3b%2B4x51JFpwjkB8MezjrGNEJG5akJSd%2FaCmu6NHOltb1nWPW1%2F84AJMEvkxsrtKj8ry0Hkw1XKnS5eBOglBJRGSoViVpgCRATe8edYMedQY
// Message.tsx
import { component$, Component, QwikIntrinsicElements } from<EMAIL_ADDRESS>
export type MessageProps<T> = {
  message: T | undefined;
  style?: string;
} & QwikIntrinsicElements['div']

export const Message = component$<MessageProps<unknown>>((props) => {
  const { message, ...other } = props;
  return
{message?.toString()}

})

// ColoredMessage.tsx

type Props<T> = MessageProps<T> & {
  color: string;
}

export const ColoredMessage = component$((props: Props<unknown>) => {
  return <Message {...props} style={`color: ${props.color};`} />
})

export const MyColoredMessage = ColoredMessage as Component<Props<string>>

export default (()=><MyColoredMessage color="red" message="Hello" />)

Thank you for your response @wmertens! Your solution solves my immediate problem!
What I really want to accomplish is to let TypeScript infer the type of a property based on the type of the property that was passed to the component. Do you know if there's a way to do it without having the end user do a type assertion?
For example, the way it's accomplished in Headless UI radio group component (demo).
As I understand it, they are doing a type assertion on the returned component to "force it" to work with TypeScript inference. I tried doing a similar type assertion for Qwik, but doing a type assertion and the functions returning Component did not work.
I have to admit that I am very ignorant when it comes to TypeScript types, so this is probably a basic question on my part. But when you mention "you can't make a constant generic" why is it that Qwik components are constants? I thought that "React-ish" components were at their core functions with a signature of (props) => object. So I thought it would be similar to something like this:
const smth = <T>(message: T) => { console.log(String(message)) }

smth(4) // TS infers T is number
smth('hello') // infers T is string

In this new StackBlitz I added two components that use the generic component and TypeScript is able to correctly infer the types for the onClick$ parameter based on the type of a given property. Looks like the issue is when I try to passthrough the received properties into a children component.
export const ColoredMessage = component$(function <T>(props: Props<T>) {
  return (
    <Message
        // Type checking fails here
        message={props.message}
        // but not here
        onClick$={props.onClick$}
        style={`color: ${props.color};`}
     />
  )
})

Could this be related to how Qwik types are defined to ensure components props are serialized correctly?

You need to "instantiate" each generic component with the proper type, like the MyColoredMessage above, or maybe do lots of overriding.
The problem stems from the fact that you can only specify generics on a function definition, not on a function constant. And a Qwik Component is always a function constant.

@guzart I was wrong, you can do generics for component$. Here's an example:
    export const Poly = component$(
      <C extends keyof QwikIntrinsicElements>({
        as,
        ...props
      }: { as?: C } & QwikIntrinsicElements[C]) => {
        const Cmp = (as || 'div') as any as FunctionComponent<typeof props>;

        return <Cmp {...props}>hi</Cmp>;
      }
    );

    <Poly as="a" href="foo" />
    // <Poly<'a'> as="a" href="foo" /> also works

But the current types don't work for this. I made https://github.com/BuilderIO/qwik/pull/5672 to fix that


[devicelab] remove 32bit test that has never passed

Description
As far as I can tell, this test has never passed. nevertheless, it is increasing resource usage on a device that we only have one of.

@liyuqian , @jmagman has brought up that these tests have caught bugs in the past so we should consider not removing them. I cannot tell when the last time they passed was though.
If these are important please fix them, if not LMK and I will remove them

It was passing on https://github.com/flutter/flutter/commit/6cd847c767f5a49616cd5c6ae04e12d752e906b3 and it passed 8 out of 10 times in the 10 previous commits before it.

That is true, I will update the title of this PR.
Nevertheless, even 8/10 is still incredibly flaky. Are you currently working on de-flaking this test?

Yes, the flakiness issue is tracked by #67753, and Jenn is working on it #69260.

That is not the source of all of the flakes unfortunately, there are additional failures where it seems like the VM Service URI is never discovered

@jonahwilliams : can you please share a link or a commit SHA to that failure log? I'll take a look and investigate.

Lets give @jmagman 's change another day or so to bake and see what the status is


Linear regression example does not produce accurate posteriors

I've been following the development of 2.0 as I find the changes to the interface very appealing. But I have been having a hard time getting accurate posterior inference for even very simple models, like linear regression example notebook. I have tried comparing the NPE with HMC, and doing SBC, and I always find that the resulting posteriors are not calibrated. The closest I have gotten to achieving calibration is with a CouplingFlow inference network after 30+ epochs, but it is still not that accurate.
I am surprised, because when using the main branch version of BayesFlow I can get valid NPE for such models quite rapidly using bf.networks.InvertibleNetwork().

I believe this has to do with our current hyperparamter settings in the dev
branch which are not yet well set by default. before merging into main we
will make sure that this is fixed.
Stefan, what are your thoughts on this?
Erik Ringen @.***> schrieb am So., 27. Okt. 2024, 11:03:

I've been following the development of 2.0 as I find the changes to the
interface very appealing. But I have been having a hard time getting
accurate posterior inference for even very simple models, like linear
regression example notebook. I have tried comparing the NPE with HMC, and
doing SBC, and I always find that the resulting posteriors are not
calibrated. The closest I have gotten to achieving calibration is with a
CouplingFlow inference network after 30+ epochs, but it is still not that
accurate.
I am surprised, because when using the main branch version of BayesFlow I
can get valid NPE for such models quite rapidly using
bf.networks.InvertibleNetwork().
—
Reply to this email directly, view it on GitHub
https://github.com/bayesflow-org/bayesflow/issues/227, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADCW2AESYSNPIFLRCDEXZV3Z5SM5RAVCNFSM6AAAAABQVT3LAWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYYTMNBTGYZTINA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Hey Erik, indeed, the default hyperparameter settings are not the same between branches, but those for the CouplingFlow are pretty close. For instance, flow matching will be pretty shallow when using the defaults, so you may find it needs a deeper internal network (e.g., as in the Two Moons notebook). Our plan is to have a large-scale hyperparameter optimization study on multiple models before merging. Would it be possible to share the model and settings you are currently working with?

Hey, thanks for your responses. I don't know if it is useful, but here is an example with a simple poisson polynomial regression model and settings that works pretty well--but these same settings do not work well for the linear regression notebook, for some reason.
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import os
if "KERAS_BACKEND" not in os.environ:
    os.environ["KERAS_BACKEND"] = "jax"
import keras
import bayesflow as bf
from scipy.stats import rankdata
from statsmodels.stats.proportion import proportion_confint
from scipy.stats import halfnorm

def meta(batch_shape):
    N = int(10)
    return dict(N=N)

def prior():
    beta = np.random.normal([-1, 0, 0], [1, 0.25, 0.25])
    return dict(beta=beta)

def likelihood(beta, N):
    # x: predictor variable
    x = np.random.normal(0, 1, size=N)
    # y: response variable
    mu = np.exp(beta[0] + beta[1] * x + beta[2] * (x**2))
    y = np.random.poisson(mu, size=N)
    return dict(y=y, x=x)

simulator = bf.simulators.make_simulator([prior, likelihood], meta_fn=meta)

adapter = bf.adapters.Adapter()
adapter.to_array()
adapter.broadcast("N")
adapter.convert_dtype(from_dtype="float64", to_dtype="float32")
adapter.as_set(["x", "y"])
adapter.standardize(exclude=["N"])
adapter.concatenate(["beta"], into="inference_variables")
adapter.concatenate(["x", "y"], into="summary_variables")
adapter.rename("N", "inference_conditions")

epochs = 30
num_batches = 512
total_steps = num_batches * epochs

inference_network = bf.networks.CouplingFlow()

summary_network = bf.networks.DeepSet(summary_dim = 6, subnet_kwargs={"base_distribution": "normal"})

approximator = bf.ContinuousApproximator(
   inference_network=inference_network,
   summary_network=summary_network,
   adapter=adapter,
)

initial_learning_rate = 5e-4
scheduled_lr = keras.optimizers.schedules.CosineDecay(
    initial_learning_rate=initial_learning_rate,
    decay_steps=total_steps,
    alpha=1e-8
)

optimizer = keras.optimizers.AdamW(learning_rate=scheduled_lr)
approximator.compile(optimizer=optimizer)

history = approximator.fit(
    epochs=epochs,
    num_batches=num_batches,
    batch_size=64,
    simulator=simulator,
)

# Check calibration of NPE
def rank_SBC(nsims = 100, num_samples = 1000):
    rank = np.empty(nsims)

    for i in range(nsims):
        prior_sims = prior()
        N = 10
        data_sims = likelihood(beta = prior_sims["beta"], N = N)
        conditions = {
                "y": data_sims["y"],
                "x": data_sims["x"],
                "N": N
            }
        pdraws = approximator.sample(conditions=conditions, batch_size=1, num_samples = num_samples)
        ranks = rankdata(np.append(pdraws["beta"][:,:,1], prior_sims["beta"][1]))
        rank[i] = ranks[-1]

    return rank

def ecdf(data, num_samples):
    sorted_data = np.sort(data)
    return sorted_data/(num_samples+1)

def compute_ecdf_diff_confidence_uniform_sampling(n, alpha=0.05):
    x_values = np.linspace(0, 1, n)
    uniform_cdf = x_values  # For uniform data, CDF(x) = x
    lower_bounds = []
    upper_bounds = []

    for i in range(n):
        successes = i + 1
        # Compute Wilson confidence interval for the ECDF at this point
        ci_low, ci_upp = proportion_confint(successes, n, alpha=alpha, method='wilson')

        lower_bounds.append(ci_low - uniform_cdf[i])
        upper_bounds.append(ci_upp - uniform_cdf[i])

    return x_values, lower_bounds, upper_bounds

num_samples = 999
nsims = 500
rank_intercept = rank_SBC(nsims = nsims, num_samples=num_samples)
ecdf_diff = np.linspace(0, 1, nsims) - ecdf(rank_intercept, num_samples)
plt.hist(rank_intercept)

plt.figure()
plt.step(np.sort(rank_intercept)/(num_samples + 1), ecdf_diff, where="post")

x_vals, lower_ci, upper_ci = compute_ecdf_diff_confidence_uniform_sampling(n = nsims)
plt.fill_between(x_vals, lower_ci, upper_ci, color='gray', alpha=0.2, label='95% Confidence Interval')

We have now improved some of the generative architecture including affine flows and also added back the diagnostics module. Woudl you mind checking your example again? We are still playing around with the clamping so I cannot guarantee yet that the current implementation will already be as good as you and we want it to be. But we will get there.

I have run over linear regression example and got good results if initialization worked. However, on some training runs, I got NaN losses after a few iterations. So I believe we need to work a bit more at least on the part that handles initialization of the coupling flows.

I have run over linear regression example and got good results if initialization worked. However, on some training runs, I got NaN losses after a few iterations. So I believe we need to work a bit more at least on the part that handles initialization of the coupling flows.

Thanks for letting me know about the updates. Both my poisson and linear regression notebook examples look better now.
I have also run into issues with NaN losses in the past when experimenting with coupling flows. I addressed  this by setting TerminateOnNaN in the tensorflow callbacks.
Below is my updated polynomial poisson regression example.
import numpy as np
import os
if "KERAS_BACKEND" not in os.environ:
    os.environ["KERAS_BACKEND"] = "jax"
import keras
import bayesflow as bf

def meta(batch_shape):
    N = int(10)
    return dict(N=N)

def prior():
    beta = np.random.normal([-1, 0, 0], [1, 0.25, 0.25])
    return dict(beta=beta)

def likelihood(beta, N):
    # x: predictor variable
    x = np.random.normal(0, 1, size=N)
    # y: response variable
    mu = np.exp(beta[0] + beta[1] * x + beta[2] * (x**2))
    y = np.random.poisson(mu, size=N)
    return dict(y=y, x=x)

simulator = bf.simulators.make_simulator([prior, likelihood], meta_fn=meta)

adapter = (
    bf.Adapter()
    .broadcast("N", to="x")
    .as_set(["x", "y"])
    .standardize(exclude=["N"])
    .apply(include="N", forward=lambda n: np.sqrt(n), inverse=lambda n: n**2)
    .concatenate(["beta"], into="inference_variables")
    .concatenate(["x", "y"], into="summary_variables")
    .rename("N", "inference_conditions")
)

inference_network = bf.networks.CouplingFlow()
summary_network = bf.networks.DeepSet(depth=2)

approximator = bf.ContinuousApproximator(
   inference_network=inference_network,
   summary_network=summary_network,
   adapter=adapter,
)

epochs = 50
num_batches = 128
batch_size = 64
learning_rate = keras.optimizers.schedules.CosineDecay(5e-4, decay_steps=epochs*num_batches, alpha=1e-6)
optimizer = keras.optimizers.Adam(learning_rate=learning_rate, clipnorm=1.0)
approximator.compile(optimizer=optimizer)

history = approximator.fit(
    epochs=epochs,
    num_batches=num_batches,
    batch_size=batch_size,
    simulator=simulator,
)

f = bf.diagnostics.plot_losses(
    train_losses=np.array(history.history["loss"]).astype(np.float32)
)

# Set the number of posterior draws you want to get
num_samples = 2000

# Simulate validation data
val_sims = simulator.sample(300)
conditions = {k: v for k, v in val_sims.items() if k != "beta" and k != "sigma"}

pdraws = approximator.sample(conditions=conditions, num_samples=num_samples)

f = bf.diagnostics.plot_recovery(pdraws["beta"], val_sims["beta"])

f = bf.diagnostics.plot_sbc_ecdf(pdraws["beta"], val_sims["beta"], difference=True)

f = bf.diagnostics.plot_sbc_histograms(pdraws["beta"], val_sims["beta"])

p.s. I am excited about the potential to use bayesflow for power/precision analysis. If anyone is actively working on this I'd be happy to chat with them.

Great, thanks! I have also thought about power analysis as a cool bayesflow application but haven't gotten around actually implementing it. If you are interested in chatting regardless, I would be happy to chat :-)

We the latest fixed, I believe the coupling flow is now stable enough. At least for the example models we have tried. I will close this issue now but feel free to reopen it if new issues along similar lines arise.


Invariant Violation: Native Component for "AIRMap" does not exist in iOS

npm: 5.4.2
node: 8.8.1
react-native: 0.52.0
react-native-maps: 0.19.0

same issue, i can't do anything

@aniiskywalker  anyluck?

@noumanDev yeah, I opened an issue if you want to take a look https://github.com/react-community/react-native-maps/issues/2064#issuecomment-376429299 i hope it helps

@alvelig so when should I link? It has to be linked at some point right?

i followed all below steps and it worked
https://github.com/react-community/react-native-maps/issues/2188#issuecomment-395977497


Different weight units of pet in adoptable pets index vs. show

I think I found a small bug. When viewing this pet on /adoptable_pets, I see the weight in kilograms. When I click on the pet, and go to the /adoptable_pets/:id show page I see a different unit for weight (pounds).
Note the highlighted text in the two screenshots below:
Index view:

Show view:

@coalest Nice find! Feel free to investigate. Note there is an option for staff to edit/update the units of weight for a pet in pounds vs. kg. I haven't looked at what units we are saving in the DB. Any help on this is greatly appreciated!!!


fix: nil check for config

I would just do an early return, also not sure why this is happening which should probably get addressed (now it will just fail silently)