HADOOP-19201 S3A. Support external-id in assume role

Description of PR
Support external id in AssumedRoleCredentialProvider.java
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html
How was this patch tested?
tested in my env
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

I didn't know about this -yes, it could be potentially useful

the docs need updating too
our delegation token support, where credentials get passed with a job, will also need this: org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding
And ideally, a way of explicitly adding a test in our assumed role tests -even if it is something we need to manually set up (e.g. we set the constraint in the role we assume for those tests and verify things work if set, fail if a wrong one is set.

Now, first step to testing: look at the test policy. You have to run all the s3 tests against a S3 store of your choice, and as you are doing things with roles, that must include the assumed role suites. Once you've done that, tell us where and then it's time to work on the test design -which we can help with.
https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/testing.html

I didn't know about this -yes, it could be potentially useful

the docs need updating too
our delegation token support, where credentials get passed with a job, will also need this: org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding
And ideally, a way of explicitly adding a test in our assumed role tests -even if it is something we need to manually set up (e.g. we set the constraint in the role we assume for those tests and verify things work if set, fail if a wrong one is set.

Now, first step to testing: look at the test policy. You have to run all the s3 tests against a S3 store of your choice, and as you are doing things with roles, that must include the assumed role suites. Once you've done that, tell us where and then it's time to work on the test design -which we can help with.
https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/testing.html

Got it. I will do it when i am free.

@Smith-Cruise this isn't going to get into 3.4.1 but if you can pick this up again we can target 3.4.2

tested this. it works
If you can add a mention of this option in hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md
then this is ready to merge. We don't want to add features which are only visible to people who looked at the source code...

tested this. it works
If you can add a mention of this option in hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md
then this is ready to merge. We don't want to add features which are only visible to people who looked at the source code...

OK, I will add it to doc

tested this. it works
If you can add a mention of this option in hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md
then this is ready to merge. We don't want to add features which are only visible to people who looked at the source code...

I've updated the doc, PTAL

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 19s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m  2s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 17s

trunk passed

+1 :green_heart:
compile
9m  5s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
8m 12s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
checkstyle
2m  5s

trunk passed

+1 :green_heart:
mvnsite
1m 40s

trunk passed

+1 :green_heart:
javadoc
1m 15s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m 11s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 20s

trunk passed

+1 :green_heart:
shadedclient
20m 49s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 22s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 50s

the patch passed

+1 :green_heart:
compile
8m 35s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
8m 35s

the patch passed

+1 :green_heart:
compile
8m 12s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
8m 12s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m  5s

the patch passed

+1 :green_heart:
mvnsite
1m 40s

the patch passed

+1 :green_heart:
javadoc
1m 11s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m  5s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 36s

the patch passed

+1 :green_heart:
shadedclient
21m 11s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 35s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m  8s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 43s

The patch does not generate ASF License warnings.

152m  8s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6876

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint

uname
Linux b01ec6b04c99 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d443ac7d2bd02230b9db3ad1491d4eb0ef8c0673

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/2/testReport/

Max. process+thread count
3109 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 20s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 23s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
19m 54s

trunk passed

+1 :green_heart:
compile
9m  7s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
8m 16s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
checkstyle
2m  6s

trunk passed

+1 :green_heart:
mvnsite
1m 37s

trunk passed

+1 :green_heart:
javadoc
1m 18s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 16s

trunk passed

+1 :green_heart:
shadedclient
20m 47s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 22s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 54s

the patch passed

+1 :green_heart:
compile
8m 43s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
8m 43s

the patch passed

+1 :green_heart:
compile
8m  9s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
8m  9s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m  2s

the patch passed

+1 :green_heart:
mvnsite
1m 32s

the patch passed

+1 :green_heart:
javadoc
1m 10s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m  9s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 31s

the patch passed

+1 :green_heart:
shadedclient
21m 25s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 43s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m  9s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

151m 22s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6876

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint

uname
Linux 18c3a5aaec05 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / b4a6ba5553d8db7801f8298188adc1d66d889745

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/3/testReport/

Max. process+thread count
1281 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 19s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 30s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m  2s

trunk passed

+1 :green_heart:
compile
9m  3s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
8m 14s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
checkstyle
2m  7s

trunk passed

+1 :green_heart:
mvnsite
1m 40s

trunk passed

+1 :green_heart:
javadoc
1m 19s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m 11s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 21s

trunk passed

+1 :green_heart:
shadedclient
21m 24s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 20s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 53s

the patch passed

+1 :green_heart:
compile
8m 45s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
8m 45s

the patch passed

+1 :green_heart:
compile
8m  8s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
8m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m  2s

the patch passed

+1 :green_heart:
mvnsite
1m 31s

the patch passed

+1 :green_heart:
javadoc
1m  6s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m  6s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 30s

the patch passed

+1 :green_heart:
shadedclient
21m 30s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 38s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 10s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 43s

The patch does not generate ASF License warnings.

152m  7s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6876

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint

uname
Linux 1bb6d7542e49 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d68a856ed7e96ca014e12d9080131ed72016375e

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/4/testReport/

Max. process+thread count
3109 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
6m 45s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m  4s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
19m 59s

trunk passed

+1 :green_heart:
compile
9m  2s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
8m 12s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
checkstyle
2m  6s

trunk passed

+1 :green_heart:
mvnsite
1m 35s

trunk passed

+1 :green_heart:
javadoc
1m 27s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m  9s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 20s

trunk passed

+1 :green_heart:
shadedclient
20m 51s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 21s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 53s

the patch passed

+1 :green_heart:
compile
8m 35s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
8m 35s

the patch passed

+1 :green_heart:
compile
8m 16s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
8m 16s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m  6s

the patch passed

+1 :green_heart:
mvnsite
1m 38s

the patch passed

+1 :green_heart:
javadoc
1m 17s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m  9s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 33s

the patch passed

+1 :green_heart:
shadedclient
20m 48s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 33s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 15s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

157m 15s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6876

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint

uname
Linux 5fedc6b5aa18 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d75fb47ce3a40b0d70059da5f015760c0e62eb09

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/5/testReport/

Max. process+thread count
1273 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6876/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

merged to trunk and 3.4 branches; added the option to my account. lets see if any surprises show up. Though I know that not enough people are running the hadoop-aws test with the assume role options correctly set.

merged to trunk and 3.4 branches; added the option to my account. lets see if any surprises show up. Though I know that not enough people are running the hadoop-aws test with the assume role options correctly set.

Actually, we've already run it in StarRocks project, everything looks OK
https://github.com/StarRocks/starrocks/blob/main/fe/fe-core/src/main/java/com/starrocks/credential/provider/AssumedRoleCredentialProvider.java


MAISTRA-2105 Use -w flag when running iptables commands

The -w flag makes iptables wait for the xtables lock. Without this flag,
the command fails if the lock is held by another process.

This needs the okay to merge label if it's ready to go in.
I assume this affects upstream as well? Is there an upstream issue we can reference here?

I guess it's this one: https://github.com/istio/istio/pull/30394
Looks like it is out of sync with this. We can fix it when that merges upstream though I suppose, or else catch it on the next rebase.


[QUESTION] Selecting columns with spaces and uppercases

Hi,
I was just wondering if it's possible to select a column name that has a space and uppercases because at the moment I'm only able to select columns that have no spaces and are in lowercase.
$excel = $reader->get(array('First_Name', 'lastname', 'email', 'company', 'jobtitle'));
That's how I retrieve my data, as you can see I've experimented with "First_Name" instead of firstname which doesn't work either, I can't afford to edit the excel file manually everytime to remove spaces and uppercases as the excel uploads will be performed on a regular basis.
Update: I've realised that underscore does work to select columns with spaces and casing isn't important.
My only issue now is that I can't read a column that's titled "E-mail Address" because of the hyphen, any solutions?
Many thanks.

Thanks for mentioning that underscores help to select word with spaces. I was wondering whether to create an issue.


python 3 compatibility fix

zip returns an iterable under python 3, and so 'Showcase.ipynb' doesn't work

Thanks for this!
Now I'm wondering how the Python 3 notebook tests passed if this wasn't working?! Maybe the reference data was updated to something broken?

I did a hammer solution, maybe one is sufficient

Maybe the reference data was updated to something broken?

Must be in an untested cell since this PR doesn't break the tests.

Must be in an untested cell since this PR doesn't break the tests.

The only way something would be untested is if the HoloViews object is not displayed and the exception doesn't affect subsequent display. Certainly possible but definitely worth checking!
Anyway, the PR itself looks fine to me though I would go with Phillip's suggestion of just fixing dim_vals if that is sufficient.

Ok, I can reproduce it. For me the cell that fails is:
items = [(k, hv.Image(sine(grid, *k), vdims=['Amplitude'])) for k in keys]
circular_wave = hv.HoloMap(items, kdims=dimensions)
circular_wave

In other words, this should have been caught by the python 3 notebook tests with both data and display failures. We need to find out why the tests passed when they shouldn't have!

Ok, the tests are passing (not that seems to say much!) so I will merge now.

I've figured out the issue with the notebook tests. To compare display output, we look at the middle frame of the holomap because animated output formats such as gif or avi don't compare nicely bitwise (we could compare javascript if we removed the uuids appropriately).
This means that not quite the same code is exercised in the tests which is why the tests pass but the cell breaks when the notebook is actually run. The data test must have passed because the HoloMap object was created correctly.
This issue of picking display formats that can be compared bitwise and making sure the proper code is run is something to consider as part of the testing process. This is something we are trying to improve in PR #318 and we should discuss this issue there.


contrib: Generate automatically test yamls for Windows

The ansible playbook generates a template yaml which has to
be edited before it can be used to create windows containers.
This patch will generate automatically an yaml file for each
Windows minion existing in the cluster which can be used for
testing.
Signed-off-by: Alin-Gheorghe Balutoiu<EMAIL_ADDRESS>

Closed in favor of #450


Handle double quote variant of distlib entrypoint script

sqlfluff seems to have double quotes instead of single quotes in their entrypoint script, this PR is to handle that variation of entrypoint script.

Thanks for reporting this and for the PR!
I ended up implementing support for scripts using double-quotes a bit differently.


[slack] allow the user to choose the message format to send

⚠⚠⚠ Since we do not accept all types of pull requests and do not want to waste your time. Please be sure that you have read pull request rules:
https://github.com/louislam/uptime-kuma/blob/master/CONTRIBUTING.md#can-i-create-a-pull-request-for-uptime-kuma
Tick the checkbox if you understand [x]:

[x] I have read and understand the pull request rules.

Description
Fixes #3226
Type of change
Please delete any options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
User interface (UI)
New feature (non-breaking change which adds functionality)
Breaking change (a fix or feature that would cause existing functionality to not work as expected)

Checklist

[x] My code follows the style guidelines of this project
[x] I ran ESLint and other linters for modified files
[x] I have performed a self-review of my own code and tested it
[x] I have commented my code, particularly in hard-to-understand areas (including JSDoc for methods)
[x] My changes generates no new warnings
[ ] My code needed automated testing. I have added them (this is optional task)

Screenshots (if any)
Non-richt text message:

Rich text message:

@CommanderStorm To be sure: is any more action required from me?

Rich messages are not functioning on version 2.0 beta. I don't receive any notifications if I enable rich messages in Slack notifications.

Rich messages are not functioning on version 2.0 beta. I don't receive any notifications if I enable rich messages in Slack notifications.

This is fixed in https://github.com/louislam/uptime-kuma/pull/5196#issuecomment-2416740399


Made changes to serialization to allow requests of lists

This PR adds the ability for the client to accept a list of objects in a request and adds a new route to the market client: put_offers, that allows the user to upload multiple offers to the MMS in one request.

Task linked: CU-8693z4817 MMS API service


BadCertificateUriInvalid

Hi,
can anyone explain me why i receive the following after using node-opcua along with opcrouter
15:53:12.020Z :opcua_server                  :396   BadCertificateUriInvalid!
15:53:12.021Z :opcua_server                  :397   applicationUri           =
15:53:12.021Z :opcua_server                  :398   applicationUriFromCert   =  null
15:53:18.004Z :opcua_server                  :396   BadCertificateUriInvalid!
15:53:18.004Z :opcua_server                  :397   applicationUri           =
15:53:18.004Z :opcua_server                  :398   applicationUriFromCert   =  null
15:54:07.157Z :opcua_server                  :396   BadCertificateUriInvalid!
15:54:07.157Z :opcua_server                  :397   applicationUri           =
15:54:07.157Z :opcua_server                  :398   applicationUriFromCert   =  null
15:54:13.543Z :opcua_server                  :396   BadCertificateUriInvalid!
15:54:13.544Z :opcua_server                  :397   applicationUri           =
15:54:13.544Z :opcua_server                  :398   applicationUriFromCert   =  null
This can be solved only if i uncheck "use the highest security level" in the opcua client configuration.

the applicationuri in your server does not match the applicationuri in your servers certificate!?
but for more support you should open a issue in node-opcua!


Question about the pretrained model

Hello,
Sorry to bother you via this platform. When I was running the quickstart with the pretrained model, there exists some errors related to _C.so files, which is displayed as follow:
albert@albert-Precision-5820-Tower:~/hawp$ python scripts/predict.py --config-file config-files/hawp.yaml --img figures/example.png Traceback (most recent call last):   File "scripts/predict.py", line 6, in <module>     from parsing.detector import WireframeDetector   File "/home/albert/hawp/scripts/parsing/detector.py", line 4, in <module>     from parsing.encoder.hafm import HAFMencoder   File "/home/albert/hawp/scripts/parsing/encoder/__init__.py", line 1, in <module>     from .hafm import HAFMencoder   File "/home/albert/hawp/scripts/parsing/encoder/hafm.py", line 5, in <module>     from parsing import _C ImportError: /home/albert/hawp/scripts/parsing/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _Z13lsencode_cudaRKN2at6TensorEiiiii
Could you please indicate how to solve this? Thanks a lot!
Weipeng

Please refer to this issue

Thank you very much!


Swab locale

About the PR

It apparently never had locale on it, or it did and was removed at some point
Media

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog

I was about to mention when they got removed https://github.com/space-wizards/space-station-14/pull/15684/files#diff-9cf33b39f5b34f92ae26f6f8c14c8e04e78b398ca2a374f5b1e89ee1454f67b3L4


ignored folders on Windows

What is your rclone version (eg output from rclone -V)
cache beta

Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows

The command you were trying to run (eg rclone copy /tmp remote:tmp)
mount

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2017/12/21 02:49:08 DEBUG : : Re-reading directory (5m4.4962215s old)
2017/12/21 02:49:08 DEBUG : : list: warm 225 from cache for: , ts: 2017-12-21 02:44:04.2506533 -0800 PST
2017/12/21 02:49:08 ERROR : TV Shows/xxxxxxxxx: Entry doesn't belong in directory "" (contains subdir) - ignoring
2017/12/21 02:49:08 ERROR : TV Shows/xxxxxxxxx: Entry doesn't belong in directory "" (contains subdir) - ignoring

I'm moving the issue in another one cause I can't seem to replicate a problem with this one on Windows and will need additional investigation. Plus, https://github.com/ncw/rclone/issues/1904 was related to another issue related to missing files caused by the old method of caching folders.
@floriansofianos I was able to replicate the same log lines in my mount but I don't see them being a problem in this case.
Here's my result:
2017/12/22 03:01:35 DEBUG : : list: warm 13 from cache for: , ts: 2017-12-22 02:58:44.6671079 -0800 PST
2017/12/22 03:01:35 ERROR : cache: '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' became '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' from ''-'1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' with sep '\'
2017/12/22 03:01:35 ERROR : cache: '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' became '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' from ''-'1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1' became 'dir1' from ''-'dir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1' became 'dir1' from ''-'dir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1/subdir1' became 'dir1/subdir1' from 'dir1'-'subdir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1/subdir1' became 'dir1/subdir1' from 'dir1'-'subdir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1/subdir1' became 'dir1/subdir1' from 'dir1'-'subdir1' with sep '\'
2017/12/22 03:01:35 ERROR : dir1/subdir1: Entry doesn't belong in directory "" (contains subdir) - ignoring
2017/12/22 03:01:35 ERROR : cache: 'dir2' became 'dir2' from ''-'dir2' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir2' became 'dir2' from ''-'dir2' with sep '\'

Now, I was listing the root folder here: / and I can't be sure why dir1/subdir1 would be returned here and I'll take a better look at it but it is valid to be ignored cause subdir1 isn't an entry of the root. Furthermore while going in Explorer in dir1, my listing was correct and got subdir1 back.
Are your folders still missing with the latest beta and you're just getting this log or are they actually missing if you look for them?

Hello @remusb , sorry for this very late answer!
With the latest beta, I have a cache on a root directory (encrypted) that contains 6 folders. If I use windows explorer to navigate to one of those 6 folders that contains files and directories. I can at first see everything correctly.
But when I wait 5 minutes, if I go to windows explorer again and browse the same directory, all the directories inside it are missing. All the files that aren't in a directory are still there though.
The same thing happens if I try to use the command line to browse the directory: I'm missing all my inner directories.
The folder I use for testing is quite big: it contains 190 folders which probably have subfolders as well and a lot of files.
In the logs, the only issue I can see is those "Entry doesn't belong in directory "" (contains subdir) - ignoring"
Also I'm not doing anything special during the 5 minutes of waiting.

The entry does not belong in directory issue should have gone away in the latest beta.
Can you give it a try if you're still interested? https://beta.rclone.org/v1.39-099-gb3d8b7e2/
Thanks

I just tried the latest (February 11th) beta. At first there were many I/O errors for my programs, possibly due to my fault of not cleaning up the old config or cache. Then I cleaned up all cache and restarted the program. It has been running smoothly for the past 12 hours. No missing files/folders until now.
I encourage everyone who encountered this bug to give it a try!
I'm using Windows 10. --dir-cache-time and --cache-info-age both set to 3 hours now.

Awesome. So the issue can be pinned on the entry doesn't belong in directory log which was fixed by handling \ even on Windows as path separator.
I'm closing the issue but if you encounter anything else, feel free to open a new one.
Thank you

Encountered missing files on rclone mount on Windows again yesterday, using the rclone Feb 16th beta AMD64 version on Windows 10. However it's just one incident and I didn't have -v or -vv enabled therefore not sure exactly what happened. Will need to rely on others who have deeper observation on this issue.


Use LIVE TestMode by Default when Running Against Service Resources

Fixes #19660
This PR updates the default TestMode when running against service resource to be LIVE instead of RECORD.

/azp run java - search - tests

/azp run java - formrecognizer - tests

/check-enforcer override

Overriding check-enforcer as additional tests ran were failing beforehand


Add defaults to Python.Runtime

Description

Added default dllmap locations, namely in form:
    <!-- Linux dll mapping -->
    <dllmap os="!windows,osx" dll="python3.6m">
        <dllentry name="python3.6m" dll="~/anaconda3/envs/base/lib/libpython3.6m.so" />
        <dllentry name="python3.6m" dll="~/miniconda3/envs/base/lib/libpython3.6m.so" />
    </dllmap>
    <!-- OSX dll mapping -->
    <dllmap os="osx" dll="python3.6m">
        <dllentry name="python3.6m" dll="~/anaconda3/lib/libpython3.6m.dylib" />
        <dllentry name="python3.6m" dll="~/miniconda3/lib/libpython3.6m.dylib" />
    </dllmap>

Related Issue

A general fix to https://www.quantconnect.com/forum/discussion/8157/lean-install-with-python-on-linux-not-finding-python3-6m/p1/comment-26460 by adding defaults for non-windows OSes.
https://github.com/QuantConnect/Lean/issues/4902
Motivation and Context

If one has a standard installation, DllNotFound-type errors for python3.6m should be eradicated. Otherwise, one can specify their location for their Python install.
Requires Documentation Change

How Has This Been Tested?

Built and ran both a Python and C sharp algorithm on Linux (5.8.16-2-MANJARO; Mono JIT compiler version 6.13.0) without errors.
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] Refactor (non-breaking change which improves implementation)
[ ] Performance (non-breaking change which improves performance. Please add associated performance test and results)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Non-functional change (xml comments/documentation/etc)

Checklist:

[x] My code follows the code style of this project.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.
[ ] My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

A clean build revealed issues exist, will open new PR with adjustments.


Add MonoDevelop addin

Fixes #2367
A folder should be created here named Packages filled with nuget files generated by the build bot. Names of nuget need to be specified here. This is so that nuget packages don't need to be downloaded during project creation.
To actually edit the project I would recommend using MonoDevelop / Visual Studio for Mac together with: https://github.com/mhutch/MonoDevelop.AddinMaker

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Oh yea, almost forgot, do tell me what you want AssemblyInfo.cs and AddinInfo.cs filled in with, and if you want me to change project structure or naming.

Hi @cra0zy – This is awesome! Thanks for submitted these changes to provide Template support.
I'll be the one reviewing this PR, but I'm far behind schedule with some other tasks, so I might not get to the first review for a few days. I'll try my best to review before the end of the week.

Stale PR with no new activity. Closing, but we can re-open in the future if new activity.


Proper Lifetime Requirements of Buffers

I'm trying to write a liburing-based I/O library and I have a few questions.
Assuming I'm preparing a read request and supplying a buffer, how long must this buffer live?
In the happy path, the buffer must live at least as long as the SQE does, until the CQE is retrieved.
But I'm wondering about two other paths: io_uring_prep_cancel() and close(fd).
If I call io_uring_prep_cancel(my_sqe); io_uring_submit(ring); free(buffers);, am I in danger of segfaulting?
Similarly for close(fd).
i.e. should I only call free(buffers) once I have the corresponding CQE in hand?

Since a CQE will be posted regardless for the original request when you do a cancelation, yes you should keep the buffer alive until you get a CQE. There's no difference between the rules there, cancelation or not.

Wow, thank you for the quick reply!
Alright, that means I have a good bit of refactoring to do. This should be pretty fun. I'm marking this issue as resolved as I have my answer.
I appreciate your time.

Just to be clear, if you successfully cancel the request, then the buffer can get freed immediately. But that case also means that the CQE for the canceled request will be sitting in the CQ ring for you anyway. Hence I think the cleanest solution will be to always honor the rule that buffers must stay alive until that particular request has posted its completion.


Supporting ovn-northd HA colocated with OVNDB-HA

As ovn-northd parse network element between ovnnb_db and ovnsb_db,
ovn-northd need connect to ovnnb_db and ovnsb_db. OVNDB-HA feather
was implemented depend on pacemaker, ovn-northd will failover following
OVNDB-HA.
If user wants to enable ovn-northd HA colocated with OVNDB-HA depend on
pacemaker, setting parameter MANAGE_NORTHD_DEFAULT="yes" in ovndb-servers.ocf

continued from https://github.com/openvswitch/ovs/pull/176

@numansiddique

the pacth new merge request reference #179


Set maximum processing time limits for Metasmoke queries

Currently if a query is submitted to Metasmoke that requires a long time to return, Metasmoke will continue to handle the request even if the requestor has timed out (to my knowledge).  This can mean searches will return 504 Gateway Timeout errors even though the request is still being processed.
It would be prudent to set a maximum processing time limit for a request (which would correspond to HAProxy's setup currently which is a maximum of 60 minutes wait-time for a response from Metasmoke for a request) and then if that value is exceeded issue a 408 Request Timeout or similar error code from within Metasmoke and then ending the processing of the request when that time is elapsed.
This is currently most seen in Search, but probably could be implemented elsewhere.  (Arose from #705 as a separate feature discussion idea)

Could also use this gem: https://github.com/sharpstone/rack-timeout. If we wanted to, I'd be happy to test it and set it up.

This would also be resolved my switching activejob backends cc #795


How should we differentiate dangerous methods?

Optics, like most functional programming concepts, follow some principles (also called laws). FP libraries tend to have two kinds of operations:
Constructors, where users need to verify the constructed object, follow those rules, usually via property-based testing.
Combinators, where users are guaranteed properties are preserved. For example, if you have a valid optic, applying any combinator gives you another correct optic.
Now, the problem is that some methods are incredibly convenient for users, but they may break optics principles. // TODO add examples.
In Monocle 1.x and 2.x, we told users: "Sorry, we understand what you want. It is useful but potentially unsafe, so we don't put this in the library". It has been very frustrating to users. In my opinion, we should allow those combinators, but we need a way to mark them as dangerous.
Many libraries use unsafe prefix, e.g. unsafeFilter. Other languages use a ! suffix like filter!. Ideally, it shouldn't be too verbose.

One thing I've done is to use a different, unsafe interface. This means the client programmer must explicitly indicate s/he wants to use unsafe operations, which a) removes any surprise on the part of the client programmers and b) makes it easier for them to search for unsafe usage later than by searching for each unsafe method name individually.
I've written up two examples here, one than can be used to retroactively make a library safer, and one that is stricter: https://github.com/tpdi/examples/blob/master/src/main/scala/org/diffenbach/unsafely/Safe.scala

@tpdi good idea, but one thing a bit annoying with that approach is that you don't know which methods may be unsafe. For example, you could refactor your code and not need it anymore but you wouldn't know.

I'm not sure I follow. Is "you" in this case the client programmer?I'd think they would know what's unsafe, because in order to call those methods, they'd first have to call .asUnsafe():var x = someLens()x.asUnsafe().unsafeMethod()Perhaps you're saying that if they write a method that relies on an unsafe method, their method might take a reference to the unsafe interface (rather than calling .asUnsafe in that method):def clientMethod(ulens: UnsafeLens) = {     ulens.unSafeMethod()}Then they refactor:def clientMethod(ulens: UnsafeLens) = {     ulens.safeMethod()}  // whoops still passing an UnsafeLens, now unnecessarily If that's what you're referring to, my second example (disjoint interfaces) addresses it, because with that, it's also not possible to call safe operations on the unsafe interface.The cost of that is that the client programmer will have to call the interface swapping methods more often when mixing safe and unsafe operations. But that's what you want, if you want to sensitize them to which methods are unsafe.Thanks,Tom
-------- Original message --------From: Julien Truffaut<EMAIL_ADDRESS>Date: 5/30/20  12:07  (GMT-08:00) To: optics-dev/Monocle<EMAIL_ADDRESS>Cc: TP Diffenbach<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [optics-dev/Monocle] How should we differentiate dangerous
  methods? (#786)
@tpdi good idea, but one thing a bit annoying with that approach is that you don't know which methods may be unsafe. For example, you could refactor your code and not need it anymore but you wouldn't know.
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/optics-dev/Monocle/issues/786#issuecomment-636371744",
"url": "https://github.com/optics-dev/Monocle/issues/786#issuecomment-636371744",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]

@tpdi Yes, I meant the user of the library may not know which method is safe or not. I see what you mean with disjoint interfaces but I see two issues:

we shouldn't allow going from UnsafeOptic to SafeOptic
it makes the API quite verbose

The plan for now is to add documentation.


Improve release script

We should consider adding the following features to the release script:

Update gh-pages branch's index.html to new version. (mentioned in #1424)
Update READMEs even when using 1-argument form. (mentioned in #1423)

As a side note, this will only work for people who can directly push to the repo.  I think this should include all NPM package maintainers, but it currently doesn't include me.  Is there a way to only make this possible with an extra argument to git push, so that it cannot (easily) be done accidentally, except from the release script?

@edemaine you should be able to push directly even to master.
w.r.t. gh-pages we should look at other ways of handling documentation.  I'd rather have it one the main branch in the docs folder.
We might also look at hosting the site on netlify and then using a static redirect.  netlify also can serve all previous versions which would come in handy when providing versioned documentation.

With required status checks and/or required pull request reviews enabled, commits cannot be directly pushed to the master, without a pull request. And I think this should be the right behavior, since it's the master branch.
I think release workflow should involve at least two people, e.g.:

One runs the release script, which creates a branch, and opens a pull request
Another reviews the pull request, approves it, and merges it
CI deploys package to the NPM

I didn't realize that.  I guess the reason why I was able to run the script as is is b/c I'm an admin.  I'm glad we're talking about this.  Having non-admins be able to deploy is important.
@ylemkimon your plan looks good.  I'm not sure how to do step 3 though.

I'm assuming CI can not just push a tag without some sort of user intervention, right? If we're following semver rules, someone would need to specify major/minor/patch bump. Unless I'm misunderstanding.

I think the idea is that the release script would still push a developer-specified tag (including version number), but that CI would do the npm publish.  I'm not sure that's really that important.  The big thing would be to move to a PR model instead of directly touching master as the script currently does.

#1 will be easy now, as we have website on the master branch 😄

The current release script seems to support releasing from another branch. I've added detailed instructions in #1521.

I think I'll give that a try when I do the release this weekend.  @ylemkimon thanks for adding that option.  Also, I'm happy to see that updating the docs are part of the publish workflow.  I'm a little scared/excited about versioned docs though.  Nothing ventured nothing gained I guess.

Same here. It would be cool if you use the new PR approach because then we can double check that the publish script os checking in the right things. Maybe don't even need to use the beta moniker in that case, as we probably won't need to change the NPM release if something goes wrong, only the PR.

#1511 still pushes the tag and publishes to the NPM(yarn publish) and to the GitHub pages(yarn publish-gh-pages), even running on another branch.

Thanks for catching this! I assume it's just the publish-gh-pages that is incorrect?

@edemaine If we're moving to a PR model, it'd be preferable to push the tag and publish to NPM and GitHub pages after the PR gets accepted and merged. I think we can do this automatically using CI or manually using a flag.

@ylemkimon Maybe for starters we can just separate the two processes, and run both of them manually?  Then we can automatic the NPM part if we want (though it's also not hard to run it manually).
But I'm unclear how we're supposed to test the documentation before we release to NPM, unless/until we have something like #1520.

Some libraries with versioned docs provide a problem for the searcher, as follows.
An early release of the documentation gets a lot of links pointing to it (perhaps there was a popular change, or something), then the search engines prioritise that version of the docs because more links point to it. Then a user comes along searching for something, lands on old documentation, and can't understand why the suggested code does not work in their latest version. (Happened to me way too often...)
This may already be in the plan, but can I suggest something like the following:
Latest documentation: katex/documentation/blah...
All previous documentation:
katex/v0.9.1/documentation/blah...
katex/v0.9.2/documentation/blah...
etc
This way, the current documentation will get growing numbers of links over time, and most of the previous ones will get very few, eventually falling into oblivion.

@mbourne Latest release (stable) version documentation will be available at https://khan.github.io/KaTeX/docs/ and previous version, v0.9.0 for example, documentation will be available at https://khan.github.io/KaTeX/docs/0.9.0/. And if we decide to publish master documentation, they will be available at https://khan.github.io/KaTeX/docs/next/.

@ylemkimon Looks perfect!

[ ] CHANGELOG
[ ] Add check to the CI whether SRI has been changed


Bugfix/resources title adjustments 97

issue 97

The titles like auth HTTP methods and Status code` are still not aligned with boxes - can you please explain what it means in this context?

updated


package-lock.json

I wonder why the package-lock.json was removed with the commit 791a2617361657630e389b09e193ca57c237623d
Shouldn't it be keept under versioning to provide consistency?

I've opted to use yarn with this repo. Having both yarn.lock and package-lock.json causes potential problems:

We highly recommend you to delete the package-lock.json file if you decide to use yarn in order to avoid future confusion and possible consistency issues.

https://classic.yarnpkg.com/blog/2018/06/04/yarn-import-package-lock/#future-plans

Ops, I've completely missed that ... sorry

Either way, dev-only lockfiles only serve to insulate the project’s devs from breakage that affects consumers - it doesn’t actually ensure consistency, it just means you’ll have to wait for bug reports or auto-update PRs failing tests before issues in your deps reveal themselves.

@ljharb I'm sorry, English isn't my first language so it's completely possibile that I'm missing something but on the doc about package-lock.json is stated

Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies

Installing exactly the same dependencies should ensure consistency or have you some links that could help me to better understand the issue?

Teammates, deployments, and CI doesn’t include consumers (those who npm install your package)


How to get body and params and other without composables (res.params, res.body)

Environment
Node 20
h3 last
Reproduction
res.body are undefined
Describe the bug
How to get body and params and other without composables (res.params, res.body).
Or to get event, context and read from them if i have only res (I can use node res.end but not res.send)?
Additional context
No response
Logs
No response

Hello, we tried to solve the issue.
This is what we did:
Modify the NodeResponseProxy class to include references to the original H3Event and its properties, allowing access to request body, parameters, and other event-related information through the response object.
You can review changes in this commit: https://github.com/wwqwq2313/unjs-h3-934/commit/85c46586fc752dc88a29679a7c624067f89e6823.

[!CAUTION]
Disclaimer: The concept of solution was created by AI and you should never copy paste this code before you check the correctness of generated code. Solution might not be complete, you should use this code as an inspiration only.

If you no longer want Latta AI to attempt solving issues on your repository, you can block this account.

@wwqwq2313 a ha ha ha
@pi0 what you think?


Update temperature and humidity to use 2.0 API

In porting over the capsules, somehow the libtock-c parts were missed for temperature and humidity. This updates them to the 2.0 syscall API. Part of #177

I think it makes sense to call what happens when a capsule calls .schedule() an upcall. That differentiates it from a .set_client()-based callback.
I don't think that means we should automatically rename libtock-c, however. The API and terminology presented to userspace  applications using libtock-c does not need to be based on the terminology of the kernel. Callbacks in a main.c file look like any other callback in any other language, and renaming them to upcalls is I think unnecessarily adding complexity. Whether they are the direct result of a kernel upcall or come from more software layers in libtock-c (alarms are a good example of this) is not really a concern of an app developer, to them it just looks like a callback.
Changing subscribe() to take an upcall function pointer makes sense, but I don't think we should change the libtock-c APIs to upcall at the same time.

I think it's fine if the libtock-c function names say "callback" rather than "upcall". But what about the typedef? The clean way to do this is for a libtock-c library to filter the upcall into a callback. While the upcall provides 4 arguments, for example, it might be the callback only has 2. So the type you pass to a subscribe is a subscribe_upcall but the type you pass to humidity_set_callback is a function pointer with a single argument, the humidity value.

bors r+


Add options for swap discarding (TRIM)

Motivation for this change
swap on ssd will benefit from TRIM/discard so the writes to the ssd go faster with less wearing.
It works in my encrypted swap0.
lsblk -D:
sdb                    0      512B       2G         0
├─sdb1                 0      512B       2G         0
├─sdb2                 0      512B       2G         0
├─sdb3                 0      512B       2G         0
│ └─root               0      512B       2G         0
└─sdb4                 0      512B       2G         0
  └─swap0              0      512B       2G         0

Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Isn't this change the same as adding the following configuration? (unless swap partition is in a different device).
fileSystems."swap".options = [ "discard" ];
boot.initrd.luks.devices."cryptroot".allowDiscards = true;

If TRIM support is not properly implemented by the SSD vendor, there is a risk of data loss. There should be a
warning if this is merged. Also enabling TRIM on an encrypted area is a potential security concern.
This has been explained here very well: http://asalor.blogspot.com/2011/08/trim-dm-crypt-problems.html.

Isn't this change the same as adding the following configuration? (unless swap partition is in a different device).
fileSystems."swap".options = [ "discard" ];
boot.initrd.luks.devices."cryptroot".allowDiscards = true;

If TRIM support is not properly implemented by the SSD vendor, there is a risk of data loss. There should be a
warning if this is merged. Also enabling TRIM on an encrypted area is a potential security concern.
This has been explained here very well: http://asalor.blogspot.com/2011/08/trim-dm-crypt-problems.html.

Isn't this change the same as adding following configuration? (unless swap partition is in a different device)
fileSystems."swap".options = [ "discard" ];
boot.initrd.luks.devices."cryptroot".allowDiscards = true;

Only luksroot had allowDiscards, not the general 'encrypted-devices.nix'. I didn't know I could override options for swap that way either. Does that work? Maybe it should say 'swap0'?
I'm using this change this way:
  swapDevices = [ {
       device = "/dev/mapper/swap0";
       discard = true;
       encrypted.enable = true;
       encrypted.blkDev = "/dev/disk/by-id/ata-WDC_WDS120G2G0A-00JH30_1806BB803081-part4";
       encrypted.label = "swap0";
       encrypted.allowDiscards = true;
       } ];

If TRIM support is not properly implemented by an SSD vendor, there is a risk of data loss. There should be a
warning if this PR is merged. Also enabling TRIM on an encrypted area is a potential security concern.
This has been explained here very well: http://asalor.blogspot.com/2011/08/trim-dm-crypt-problems.html.

I agree. I copied the documentation from luksroot.nix

Isn't this change the same as adding following configuration? (unless swap partition is in a different device)
fileSystems."swap".options = [ "discard" ];
boot.initrd.luks.devices."cryptroot".allowDiscards = true;

Only luksroot had allowDiscards, not the general 'encrypted-devices.nix'. I didn't know I could override options for swap that way either. Does that work? Maybe it should say 'swap0'?
I'm using this change this way:
  swapDevices = [ {
       device = "/dev/mapper/swap0";
       discard = true;
       encrypted.enable = true;
       encrypted.blkDev = "/dev/disk/by-id/ata-WDC_WDS120G2G0A-00JH30_1806BB803081-part4";
       encrypted.label = "swap0";
       encrypted.allowDiscards = true;
       } ];

If TRIM support is not properly implemented by an SSD vendor, there is a risk of data loss. There should be a
warning if this PR is merged. Also enabling TRIM on an encrypted area is a potential security concern.
This has been explained here very well: http://asalor.blogspot.com/2011/08/trim-dm-crypt-problems.html.

I agree. I copied the documentation from luksroot.nix

Only luksroot had allowDiscards, not the general 'encrypted-devices.nix'. I didn't know I could override options for swap that way either. Does that work? Maybe it should say 'swap0'?

You should change the names according to your partition naming. I haven't tested it, but it should work. fsType = "swap" attribute should be valid for fileSystems option.
Is it a good idea to mount swap with discard? It may stall disk operations. I would prefer mounting / with discard and running periodic fstrim on swap area.

Only luksroot had allowDiscards, not the general 'encrypted-devices.nix'. I didn't know I could override options for swap that way either. Does that work? Maybe it should say 'swap0'?

You should change the names according to your partition naming. I haven't tested it, but it should work. fsType = "swap" attribute should be valid for fileSystems option.
Is it a good idea to mount swap with discard? It may stall disk operations. I would prefer mounting / with discard and running periodic fstrim on swap area.

On Sun, Jan 10, 2021 at 03:26:28PM -0800, bb2020 wrote:

Only luksroot had allowDiscards, not the general 'encrypted-devices.nix'. I didn't know I could override options for swap that way either. Does that work? Maybe it should say 'swap0'?
Is it a good idea to mount swap with discard? It may stall disk operations. I would prefer mounting / with discard and running periodic fstrim on swap area.

The manpage (swapon) says that the TRIM runs asynchronously, so maybe they
will not stall. In any case, I could add discard options (once, pages,
both), if you want.
--
(Escriu-me xifrat si saps PGP / Write ciphered if you know PGP)
PGP key 7CBD1DA5 - https://emailselfdefense.fsf.org/

On Sun, Jan 10, 2021 at 03:26:28PM -0800, bb2020 wrote:

Only luksroot had allowDiscards, not the general 'encrypted-devices.nix'. I didn't know I could override options for swap that way either. Does that work? Maybe it should say 'swap0'?
Is it a good idea to mount swap with discard? It may stall disk operations. I would prefer mounting / with discard and running periodic fstrim on swap area.

The manpage (swapon) says that the TRIM runs asynchronously, so maybe they
will not stall. In any case, I could add discard options (once, pages,
both), if you want.
--
(Escriu-me xifrat si saps PGP / Write ciphered if you know PGP)
PGP key 7CBD1DA5 - https://emailselfdefense.fsf.org/

How does this relate to #107728?

How does this relate to #107728?

Oh, that looks like the official way to achieve this.

Looks like changes in #107728 will work only with unencrypted swap, so discards still should be enabled in in luks.

It is good to see #107728 is merged.

Looks like changes in #107728 will work only with unencrypted swap, so discards still should be enabled in in luks.

I suppose you can use LVM to create swap and root areas on a cryptroot LUKS partition and then use this to enable discard on both. boot.initrd.luks.devices."cryptroot".allowDiscards = true;
TRIM on LUKS should not be enabled by default as there are security implications. It is explained here. It should be up to the user.

Here I added PR for this: https://github.com/NixOS/nixpkgs/pull/129408

I suppose you can use LVM to create swap and root areas on a cryptroot LUKS partition and then use this to enable discard on both.

I'd prefer not to use LVM as I don't need it and use btrfs subvolumes instead.

TRIM on LUKS should not be enabled by default as there are security implications. It is explained here. It should be up to the user.

Right, it's disabled by default via discardPolicy.

superseded by #107728 and #129408
additionally #128032 will allow people to set their own swap options


Improving Kafka Connector docs

Changes navigation and minor edits.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Add BuildkiteAgentTokenParameterStorePath

Closes #8

@moskyb thanks for the feedback! Indeed, now I see I missed a lot and it won't work automagically.
I marked PR as a draft now and I'll add the missing pieces in upcoming days.

it won't work automagically

unfortunately! my job would be a heck of a lot easier....
anyway, lemme know if you run into any problems, happy to help out :)

@moskyb, not sure what I was thinking either when I created a PR with this change only. I'm working on adding the required items: IAM role and policy, a script to inject the agent token after agent installation, etc.

Apologies @scadu, I just merged another community contributed PR that caused some merge conflicts with yours. It should be a somewhat straight forward fix, though. Let us know if that's not the case though and we can take a look.

Closing it for now as I don't plan to work on it now, and it's been a year already.


Develop soil storage structure for multiple buckets

Sw types are closed the first version is updated by PDM and PDM2 and other improvements.


Setup yalc

GS1-2667

@jd-gray Did you merge to master because this wouldn't change anything in production since this is Readme stuffs?

@jd-gray Did you merge to master because this wouldn't change anything in production since this is Readme stuffs?

@Tristan-Trainual this is actually a different repo, it's not Trainual.


Update global parameters fails for Azure Government subscription

Describe the bug
I'm receiving a 404 (Subscription not found) when the task is trying to update the global parameters.
To Reproduce
Run the pipeline to deploy/update an ADF instance to a subscription hosted in Azure Government which contains global parameters.
GlobalParam.ps1 contains a hardcoded reference to https://management.azure.com (= Azure commercial URL)
The Azure Government URL equivalent is https://management.usgovcloudapi.net
Expected behaviour
The parameters are updated without any issues/errors.
Screenshots

Duplication of https://github.com/Azure-Player/azure.datafactory.tools/issues/356


Support for access service definition

Add support of access service specification via Helm Values for multi-node (v2.x) installations.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@feikesteenbergen : This PR extends the service definition by enabling the users to specify majority of the
available service parameters. Main use-case here is to enable running the service un-exposed (ClusterIP).
Please note, that it changes the default service type from LoadBalancer to ClusterIP and also removes the
AWS specific annotation. The reason for this is to enable vanilla kubernetes installations to work without
any modification.
If this behavior change is unfeasible for any reason please let me know and I'll adjust the PR accordingly.
Thank you for your time reviewing this!

We need this as well. This is a breaking change to the service, but I think it is extremely important since exposing the database to the Internet by default is a security concern. Any chance this can be reviewed by the maintainers?

We need this as well. This is a breaking change to the service, but I think it is extremely important since exposing the database to the Internet by default is a security concern. Any chance this can be reviewed by the maintainers?


Upgrade Package.swift syntax for newest PackageDescription API

Upgraded Package.swift syntax accordingly to newest PackageDescription API and bumped swift-tools-version to 5.3

I have one error


local-path-provisioner doesn't work on clusters without internet

Fixing Helm chart is easy: add HELPER_IMAGE with env var to point to the proper address of busybox.
But additionally imagePullSecrets has to be added to the create-xxxx pods generated by the provisioner and there's no way to do it.

@yasker this would be a minor change I think by extending the podpec in provisioner.go:
ImagePullSecrets: []v1.LocalObjectReference{
				{Name: p.pullSecret}, // should be loop of spec.imagepullsecrets
			},

We have met this issue our self having to use a private docker proxy.
I can add a PR for this one, but in my simple impl (since I have a limited knowledge of the go-client) I've added a pullproxy option to the provisioner (similar to the --helper-image). (Would probably be best if it was injected from the spec.imagepullsecrets from deployment,if possible)

Seems we should add an image pull secret option in the cmdline.

If the PR https://github.com/rancher/local-path-provisioner/pull/119 is merged, this issue shouldn't be closed?


Owlapi mapper tests

Added test for axiom mappings on owlapi mapper
Fixed a few other bugs with the mapper
Updated __eq__ comparison for owl classes that have Iterable arguments to be order-insensitive

I will have a look on that


Reply Disappeared

Location
Item detail view in #Settler: https://sc-nightly.netlify.com/detail/0xCeb2F510DE0945e6540cf507d0E18ED9A7e1B3fE/0x2a2e512c8691f74b432058771be48f1674759b63f2f43f5fe7b22a5a657cdc23
Expected behavior
When you click the blue arrow in the reply, it opens the reply box so you can choose the replier to be your provider.
Actual behavior
The reply disappeared:

Steps to reproduce the problem
Browse to the item detail view. Choose the blue arrow in the reply. Reply disappears.
When I refreshed the page, the reply came back.
Operating system / hardware 2017 Macbook pro 13", OS High Sierra 10.13.3, Chrome v 68.0.3440.106

I can't reproduce, but I have seen it once myself at least too. If you have the time, it would be very useful to know the exact flow I need to follow to reproduce this. @matthewcarano
Right now just going to an existing item as Seeker and going to a reply detail works as expected.

we need more information, what exactly happened.

Desktop User created a request.
Phone User replied to the request
Desktop User tried to select the blue arrow on the item detail to see the reply.
The reply and blue arrow disappeared.
5, Upon refresh, the reply and arrow came back.

Thx @matthewcarano, I pushed a fix that I think has solved this. I it happens again, please tell it here.


Error on example "autocomplete-suggestions"

Bug report
Description / Observed Behavior
What kind of issues did you encounter with SWR?
Example autocomplete-suggestions not working on my environment. it has problem with useLayoutEffect.
I think it cause by @reach/combobox. but I did not find why it is.
I made sandbox for this issue. I works well.
Warning: useLayoutEffect does nothing on the server, because its effect cannot be encoded into the server renderer's output format. This will lead to a mismatch between the initial, non-hydrated UI and the intended UI. To avoid this, useLayoutEffect should only be used in components that render exclusively on the client. See https://fb.me/react-uselayouteffect-ssr for common fixes.
    in ForwardRef(ComboboxInput) (at pages/index.js:28)
    in div
    in Context.Provider
    in ForwardRef(Combobox) (at pages/index.js:27)
    in div (at pages/index.js:25)
    in Unknown
    in App
    in Unknown
    in Context.Provider
    in Context.Provider
    in Context.Provider
    in AppContainer

Expected Behavior
How did you expect SWR to behave here?
Repro Steps / Code Example
https://codesandbox.io/s/exciting-haze-ej4vd
Additional Context
SWR version : 0.5.4

@ChangJoo-Park I've confirmed this issue, so I've fixed it as #1183


Update to compiler v20160713

Note, this PR depends upon #420.

Thanks for the very prompt review!
PTAL

@joeyparrish Could you please take another look at this updated change?

I recommend splitting this PR up.  Most of these changes are easy to accept (fixing semicolons, annotations, etc.), so we can go ahead and land those.  I don't think we are ready to land the new compiler, though, until we find a way to address the lint check errors, and that may take more time.
If we split up the PR, we can land the less controversial parts relatively quickly and avoid having to rebase those changes many times as we continue to push changes from our end.

@joeyparrish Thanks for your feedback. I broken up this change into several commits per your recommendation hoping to make them a bit easier to merge.
This change depends upon #453, #454, #455, #456, #457, #458, #459. However all of the other changes should be independent of one another and can be reviewed/merged in any order.
PTAL

Updated to include the {width,height} string to number conversion.
PTAL

Perhaps this needs a quick rebase?

Rebased.
PTAL

Strange, this did not happen locally for me. Looking more.

Hmmm, I'm having no luck reproducing this. Do you have any hints for what might be different on the shaka-player-bot machine?

I'm getting two failures on my Linux workstation, neither of which make any sense to me:
Checking the tests for type errors...
/usr/local/google/home/joeyparrish/hacking/shaka/js/test/test/util/fake_media_source_engine.js:149: ERROR - mismatch of the bufferedAheadOf property type and the type of the property it overrides from superclass shaka.media.MediaSourceEngine
original: function (this:shaka.media.MediaSourceEngine, string, number, number=): number
override: function (this:shaka.test.FakeMediaSourceEngine, ?, ?, ?): ?
shaka.test.FakeMediaSourceEngine.prototype.bufferedAheadOf = function(
^

/usr/local/google/home/joeyparrish/hacking/shaka/js/test/test/util/memory_db_engine.js:52: ERROR - condition always evaluates to false
left : null
right: null
  return this.stores_ != null;
         ^

I believe these are both compiler bugs.  The first is a failure to infer types in an override, even though they are identical.  The second is clearly not correct.  this.stores_ != null is obviously not always false.

Replacing this.stores_ != null with !!this.stores_ seems to work for the second failure.  It shouldn't be necessary, but it works.  Maybe do that with a // TODO: report closure-compiler bug.
Still trying to figure out the FakeMediaSourceEngine failure, which the build bot seems not to have.  I think we could call the nondeterministic behavior a compiler bug, as well.

Looks like you can work around FakeMediaSourceEngine by adding explicit parameter and return annotations to bufferedAheadOf.  Again, something you shouldn't have to do.  Another TODO: report closure-compiler bug on that as well, please.

One more request: It looks like with the new compiler and these two changes, we now pass 'analyzerChecks'.  Would you please remove --jscomp_off=analyzerChecks from build/build.py?  Thanks!

I still can't have been unable to reproduce any failure. I tried 32 and 64 bit linux kernels with openjdk 7, 8 and 9.
Can you share which jdk you are seeing the failures on?

After getting much debugging help from @dbhoot I think the source of this issue might be that the build scripts use os.walk() to determine the order in which input files are passed to the compiler. As the order of files returned from os.walk() is dependent on the directory entries in the file system this leads to unpredictable results across machines.
I'll look into turning on --dependency_mode in another change in order to have the compiler pre-sort the files.
  How about we sort the files in python before passing them to the compiler?

Sorting the files list in build_raw() fixed all the compiler errors for me.

That is awesome that a fix is so simple. However, isn't this a sign of a larger problem? As build/all.py is not passing any --dependency_mode is means that NONE is used. Therefore the order of files passed to the compiler is relevant. By doing an alphabetical sort are we not just "getting lucky" that the ordering is correct? I think we need a sort which respects the dependency information.
What are your thoughts?

I was honestly not familiar with --dependency_mode until I looked it up just now.
We should fix this through --dependency_mode.  Does --entry_point support a file full of requires?  If so, we can use shaka-player.uncompiled.js as the entry point, since this already has to list all the exported classes an app might use.

Good idea about shaka-player.uncompiled.js. If we use --dependency_mode=LOOSE any file without a goog.provide or goog.module will be retained. So as long as we pass shaka-player.uncompiled.js all of the goog.require() in it will be retained as well.

As the testing code does not use goog.provide/goog.require the compiler is having problems sorting them correctly. Can we merge this PR with the files.sort() fix to land the new compiler?
I'm looking into a more complete fix for the dependency_mode. Either way, II think the command lines we build shouldn't depend on ordering returned by os.walk() so sorting the files is a good idea.

I double-checked, and as it should be, everything in lib/ has a goog.provide.
The testing code doesn't need dependency tracking, as we don't really compile it.  We just check it for type errors so we can catch more mistakes in our test code before they show up as test failures.
We can definitely also sort the files in python, though.

I agree that everything in lib/ looks good (i.e. all files have a goog.provide).
However, I'm struggling to see why the testing code does not need a dependency based ordering. It seems we have at least three options for the testing code:

dep_mode = NONE, order files based upon os.walk()
dep_mode = NONE, order files alphabetically
dep_mode = LOOSE

Option 1 is not good since the ordering varies based upon the file system directory entries. Option 2 is not good since it requires the files to be named in a specific way to ensure the compile works. Option 3 does not work since the compiler does not know how to order files which do not contain a goog.provide/goog.require dependency graph.
Can you clarify?
PTAL

The tests are not compiled.  We run the compiler over them with --checks-only to catch mistakes early.
The test runner serves up all library code, uncompiled, followed by all test code.  It uses closure to bootstrap the library through shaka-player.uncompiled.js.  The Karma config controls everything.  See the "files" array in karma.conf.js.

The tests are not compiled. We run the compiler over them with --checks-only to catch mistakes early.

True. But even --checks-only code needs the types to be declared in the correct order.  Therefore, source file ordering matters for --checks-only compiler invocations. See an example in #465.

Okay, let's discuss dependency mode in #465 as you suggested.  One more pass through the build bot, and then we'll merge this.

Thanks for all your help! I really appreciate it.

No problem.  Thank you for contributing!


Support rewriting transform to derived column

Description
Allow broker to automatically rewrite query with transform to derived column if exists
Release Notes
Add a new config enableQueryRewrite to TransformConfig to enable the rewrite

Codecov Report

Merging #8308 (410cd62) into master (a208954) will decrease coverage by 49.90%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master    #8308       +/-   ##
=============================================
- Coverage     64.06%   14.15%   -49.91%
+ Complexity     4253       81     -4172
=============================================
  Files          1590     1590
  Lines         83774    83843       +69
  Branches      12694    12709       +15
=============================================
- Hits          53668    11872    -41796
- Misses        26244    71079    +44835
+ Partials       3862      892     -2970

Flag
Coverage Δ

unittests1
?

unittests2
14.15% <0.00%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a208954...410cd62. Read the comment docs.

Discussed offline and decided to do it in a more generic way


mvsum() now fails with lme models

I believe the issue is in the new call object definition
call <- paste(deparse(model$formula), collapse = "")
Here is an example
library(nlme)
model <- lme(distance ~ age + Sex, data = Orthodont, random = ~ 1|Subject)
call <- paste(deparse(model$formula), collapse = "") #New
call
call <- paste(deparse(summary(model)$call),collapse="") #Original
call

This has been fixed now. Changed to
call <- Reduce(paste, deparse(formula(model$terms), width.cutoff = 500))
which is used for other models.


OTLP Zipkin Collector

Code + tests for OTLP encoding and collecting. Tests using OTel OTLP exporters against OTLP Zipkin Collector

marking this draft until it is mergable without conflict. helps me remember what's ready to review vs not.

when polishing this later, consider @reta recent PR which had a nice description README updates etc. basically we want to get to a point where merging this won't cause work for others to go back and polish it, add docs, etc https://github.com/openzipkin/zipkin/pull/3765

OK, I've done everything that was requested of me, I've applied all the review suggestions. I think that there's no point in delaying the review of this any further.

@marcingrzejszczak can you please close this pr as #13 has already been merged.


[racon::Polisher::initialize] error: duplicate sequence <file_name> with unequal data

Hi Duncan,
the error means that there are two sequences with the same identifier but with different lengths (i.e. they are not equal). Are you by any chance using paired end sequences where reads in a pair have the same name up to the first white space?
Best regards,
Robert

Hello Robert.
Below is an example of two header lines in my fasta file that I got as output after correcting with Canu:

1b05a25b-07ba-4a45-ac0f-4ae3ead718d9 runid=90e28bee03a6438bda9d3cb74d9e3105aa2ed89a sampleid=Barcodes read=331 ch=222 start_time=2018-10-29T13:51:43Z id=11 clr=0,2148

c50437e9-ec14-46d5-af5c-57981519079d runid=90e28bee03a6438bda9d3cb74d9e3105aa2ed89a sampleid=Barcodes read=1832 ch=390 start_time=2018-10-29T14:11:48Z id=19 clr=0,1058

In the above is the runid causing the error?

Everything after the first space is not used (i.e. only the sring before runid is stored). Make sure that all sequences in both contig and read file have unique names up to the first space.

Hi,
The sequences in both the contig and read file have unique names but I suspect may have unequal read lengths as when I used Canu it would have trimmed the sequences to output the corrected reads. Is there anyway I can change the parameters of the read lengths to be ignored by Racon and it goes ahead with the polishing?

Are you trying to polish contigs or correct reads with each other? Please provide me with your command and descriptions of input parameters.

Hi I am try to polish the sequence reads obtained from running the ONT Minion.I only have the original fastq files to work with and dont have the fast5 files and so I cannot use Nanopolish. Hence I am trying to use Racon for polishing and getting a final polished consesus I can use for comparing. Please see below my command and input parameters for Racon:
:~$ racon -t 2  <path/to/fastq_files>  <path/to/sam_file>  <path/to/correctedReads.fasta.gz > correctedReads_racon.fasta
Notes:
-The fastq files are the original fastq files from the Minion experiment

CorrectedReads were obtained from Canu using the following command:
./canu  -p Read_prefix -d <path /to/output_for_CorrectedReads>  genomeSize=2.1k -nanopore-raw  /path/to/Reads.fastq

sam file was generated using minimap2 using the following command:
./minimap2 -ax map- ont <path/to/ref.fasta>  <path/to/correctedReads.fasta.gz>  > <path/to /correctedReads_aln.sam

After running the command for Racon I get the following:
[racon::Polisher::initialize] loaded target sequences
[racon::Polisher::initialize] error: duplicate sequence 1b05a25b-07ba-4a45-ac0f-4ae3ead718d9 with unequal data
I am stuck and dont know hat to do next and the only other alternative I have is to get the fast5 files and try polishing with Nanopolish instead of racon. BUt if you can assist me with resolving this error, i will greatly appreciate

Hi,
I meant the directory path to access the reference genome fasta file

What do you need the reference genome for?

It is required as input when generating an alignment in minimap2 mapping Oxford nanopore reads

I hope you can help and perhaps a solution to debugging the error that racon gives

If you want to polish your reads with racon you should run the following:
minimap2 -ax ava-ont --dual=yes <reads> <reads> > alignments.sam
racon -f <reads> alignments.sam <reads> > polished_reads.fasta

If you want to assemble your genome with canu and afterwards polish it again with racon, run the following:
canu -p <prefix> -d <directory> genomeSize=<size of the sequenced genome> -nanopore-raw <reads>
minimap2 -ax map-ont <canu contigs> <reads> > alignments.sam
racon <reads> alignments.sam <canu contigs> > polished_contigs.fasta

thank you for the help. I will run the commands and will let you know if all works out

Thank you for your help. i manage to get polished contigs with your suggested approach.
I now want to compare the polished contigs and the unpolished contigs using Mummer to see how efficient the polishing with racon was. If you have any suggestions of other scripts that can check the efficiency of polishing with racon, I will would very much appreciate if you can share them

I have been only using dnadiff from the Mummer package.

may you please send me the command you use with Mummer for dnadiff?. I have version 4 beta and it doesnt seem to be giving me the output i expect

I am running dnadiff <reference file> <assembly file>.

I assume the reference file and assembly file is in sam format? But Thanks will try it out. thank you for the wonderful program you developed. It is very valuable in the event that one does not have immediate access to the large fast5 files generated by minion but still has a convenient way of polishing the fastq sequence files

Both files need to be in fasta format. Thank you for your kind words :)

I am running Mummer v4 beta and I keep getting 1 of two errors. This happens when I run dnadiff program. It gives the following:
Error :- multiple query file input required in SAM output format
or if I first run the nucmer program to generate the delta file and then use it as input in the dnadiff program I get the following:
Error:- could not parse delta file
error- no 400
I know you are not maintaining the mummer program but  I would appreciate any advice from your experience of running the program and are you using version 4 or another version?

I have been using this one: https://github.com/marbl/MUMmer3. Did not yet try v4. You can download it with git clone https://github.com/marbl/MUMmer3 and run make in the created directory.

Hi Robert,
I'm digging up a bit of an older thread here. I'm having a very similar problem to Duncan while trying to run a pipeline similar to that described here https://www.biorxiv.org/content/10.1101/645903v3.full.pdf (though I only have amplicon reads, ~4000bp, no UMIs). I've checked through my fastq read identifiers, and as far as I can tell they're all unique. The racon manual says to input commands in this order:

racon [options ...] <sequences> <overlaps> <target sequences>

but as per the discussion above, I also attempted:

racon [options ...] <target sequences> <overlaps> <sequences>

and got the same [racon::Polisher::initialize] error: duplicate sequence <read identifier> with unequal data error. So I'm a bit stumped.
Am I missing something?
Kind regards,
Robert H
Here's my pipline in short, starting from my base-called reads in fastq format:
I first quality and length trimmed my reads with NanoFilt (>3000bp, >qual 13):

NanoFilt -l 3000 -q 13 raw.fastq > trimmed.fastq

I next generated reference consensus reads with usearch (0.75 id, double stranded):

usearch -cluster_fast trimmed.fastq -id 0.75 -strand both -centroids reference.fa

Then I mapped the trimmed reads to the reference sequences with minimap2:

minimap2 -ax map-ont -t 5 reference.fa trimmed.fastq > mapped.sam

And finally attemped to polish the trimmed reads in racon (tried both configurations, got the same error):

racon -m 8 -x -6 -g -8 -w 500 -t 5 reference.fa mapped.sam trimmed.fastq > polished.fa
racon -m 8 -x -6 -g -8 -w 500 -t 5 trimmed.fastq mapped.sam reference.fa > polished.fa

I checked for repeat names with the following, but all names were unique:

grep '^@[a-z|0-9]*-' raw.fastq | sort | uniq -c
grep '^@[a-z|0-9]*-' trimmed.fastq | sort | uniq -c

Hi Robert,
the problem is that you have a sequence in trimmed.fastq and reference.fa that share a name. Try renaming your reference reads, rerun the minimap2 command and run racacon as racon trimmed.fastq mapped.sam reference.fa.
Best regards,
Robert V

Thanks Robert, this solved it


Improvement proposals

This was easier than rebasing.
https://github.com/ePages-de/restdocs-openapi/pull/9


WIP: Support exporting pretrained models to mace/ncnn and others via pnnx

The aim of this PR is to support exporting pretrained models in a way such that they can be used by other inference frameworks such as

[ ] mace
[ ] ncnn

and possibly others like: tnn, mnn, etc.
We can also use the quantization support from the above frameworks, e.g., using full int8 support from ncnn
As the first step, I am using pnnx as an intermediate format.

It requires
https://github.com/csukuangfj/ncnn/pull/1

In the export.py, if we just save the model.state_dict, we don't need to execute convert_scaled_to_non_scaled(model, inplace=True).

In the export.py, if we just save the model.state_dict, we don't need to execute convert_scaled_to_non_scaled(model, inplace=True).

Yes, you are right.

Closing via #571


Cache failed source map lookup attempts

I was getting terrible performance in a test suite when I added source-map-support and realized it was because it was repeatedly trying to retrieve the source map for a javascript file that has no source map.
This pull request adds logic to cache lookup failures in addition to lookup success which improved performance substantially.

Looks useful.  Any word on this?

+1

@khalsah Is it still relevant?

Looks like this problem was solved by af37071383bb87d529028ad6dfe665584c9313b8. Should no longer be needed.


Add test for silent k8s connection failure.

Description

Trying to repro https://github.com/projectcalico/calico/issues/5173 but to no avail.  This test passes; the kube-client seems to detect a black-holed connection after about 30s.
Marking as draft for now because the test flakes sometimes when running kubectl exec not sure why yet but seems unrelated to the thing I'm actually trying to test.
Todos

[ ] Tests
[ ] Documentation
[ ] Release note

Release Note

None required

The Calico team is reorganizing our code and repository structure in order to make development easier, and as such all of the code in this repository is now located at https://github.com/projectcalico/calico. We appreciate your PR and your patience as we work to improve our developer experience, and welcome you to re-open your changes against the new location. Please reach out if you have any questions!


esbuild minifies CSS in a way that can break in old browsers

What version of Remix are you using?
@remix-run/dev 1.15.0
Are all your remix dependencies & dev-dependencies using the same version?

[X] Yes

Steps to Reproduce
I found this compilation of styles.

when i use postcss-preset-env compile my style code, eg: body { position: fixed; inset: 0; }
First, it will be processed by postcss,get body { position: fixed; top:0; bottom:0; left:0; right:0; } .
Then esbuild will process my style code,This will translate into body { position: fixed; inset:0; }
Expected Behavior
Keep The postcss-preset-env dealing with me.
Actual Behavior
Now, then esbuild will process my css code after then postcss.

Hey @19Qingfeng, it would help a lot if you could provide a repository reproducing the problem.

I had a look into this. It's actually not strictly related to PostCSS as you can manually write a CSS file with your example (body { position: fixed; top:0; bottom:0; left:0; right:0; }) and get the same result.
This is more of an issue with esbuild's CSS minification logic which is quite aggressive by default. I was able to fix it for CSS file imports by setting the targets array on the build inside cssImportsPlugin. I tried setting it to the list of browser versions needed for native JS module support since that's the absolute baseline for Remix:
target: [
  "chrome61",
  "edge16",
  "safari11",
  "ios11",
  "firefox60",
  "opera48",
],

This worked for CSS file imports (i.e. import href from "./styles.css") but this threw errors during our CSS bundle build since that also bundles JS, and esbuild can't handle certain JS features when targeting these browser versions.
I can fix this issue (at least for the default remix template) by raising the browser versions to the following:
target: [
  "chrome63",
  "edge79",
  "safari12",
  "ios12",
  "firefox60",
  "opera50",
],

However, this feels a bit brittle to me since we can't decouple CSS and JS minification options and we might run into further issues with other JS features. I think the safer approach would be to look into a custom CSS minification step in our compiler (maybe using Lightning CSS) but this is a bigger job.

Is this causing an actual user-facing issue for you in terms of browser support? Are there specific browsers you're trying to support that don't support CSS inset? https://caniuse.com/mdn-css_properties_inset

Is this causing an actual user-facing issue for you in terms of browser support? Are there specific browsers you're trying to support that don't support CSS inset? https://caniuse.com/mdn-css_properties_inset

Yes,I have some user browsers that don't support color abbreviations eg: #0009 not support on chrome 70.At present, my solution is use pnpm patch '@remix-run/dev' and add target in esbuild.

Is this causing an actual user-facing issue for you in terms of browser support? Are there specific browsers you're trying to support that don't support CSS inset? https://caniuse.com/mdn-css_properties_inset

I had a look into this. It's not strictly related to PostCSS because you can manually write a CSS file containing your example code (body { position: fixed; top:0; bottom:0; left:0; right:0; }) and get the same result.
This is more of an issue with esbuild's CSS minification logic which is quite aggressive by default. I was able to fix it for CSS file imports by setting the targets array on the build inside cssImportsPlugin. I tried setting it to the list of browser versions needed for native JS module support since that's the absolute baseline for Remix:
target: [
  "chrome61",
  "edge16",
  "safari11",
  "ios11",
  "firefox60",
  "opera48",
],

This worked for CSS file imports (i.e. import href from "./styles.css") but this threw errors during our CSS bundle build since that also bundles JS, and esbuild can't handle certain JS features when targeting these browser versions.
I can fix this issue (at least for the default remix template) by raising the browser versions to the following:
target: [
  "chrome63",
  "edge79",
  "safari12",
  "ios12",
  "firefox60",
  "opera50",
],

However, this feels a bit brittle to me since we can't decouple CSS and JS minification options and we might run into further issues with other JS features. I think the safer approach would be to look into a custom CSS minification step in our compiler (maybe using Lightning CSS) but this is a bigger job.

I always think exposing the browserlist is a good op, I saw this discussion, but I don't know what the remix team is choice right now.

Unfortunately we had to revert the fix for this in https://github.com/remix-run/remix/pull/7324 as it caused some other issues with CSS features.  Going to re-open this so we can address it via browserslist support early in v2.


Update store listing -  Product detail?

It should be better that we can also update store description detail - not only can publish new files.

I wish. Unfortunately the API I use is very limited: https://developer.chrome.com/docs/webstore/api_index/