Nodes provisioned by Karpenter unable to schedule pods with custom AMI in govcloud

Version

Karpenter Version: v0.27.0

Kubernetes Version: v1.24
Expected Behavior
Pods scheduled on nodes provisioned by Karpenter (using custom built AMI) are able to have pods scheduled successfully.
Actual Behavior
When enabling Karpenter, our application does not schedule pods. When disabling Karpenter and using cluster autoscaler, kubernetes is able to schedule pods. In both scenarios we leverage a custom built AMI in an AWS govcloud environment.
This was all done via following the Migration from Cluster Autoscaler steps. My team (@zestrells) has recently switched to Karpenter on several of our production clusters across multiple regions in standard AWS, those AMIs are amazon-provided bottlerocket AMIs, and they are able to schedule pods.
Specifically, our pod pulls our relatively large ECR container image onto the node, and then it continues to pull the same image continuously until there is insufficient space on disk. We have increased our root EBS volume on our AMI 10x to 200GB.
To reiterate, we believe this issue is linked with Karpenter as we are able to leverage the custom hardened AMIs with cluster autoscaler successfully.
Any help or guidance would be greatly appreciated!!
Steps to Reproduce the Problem
In an AWS govcloud account, create a custom AMI and enable karpenter to deploy an ECR image.
Resource Specs and Logs
Events log from our pod experiencing the described behavior
Events:
  Type     Reason                  Age                    From                        Message
  ----     ------                  ----                   ----                        -------
  Normal   Scheduled               10m                    default-scheduler           Successfully assigned `[our-pod]` to ip-`[redacted]`.us-gov-west-1.compute.internal
  Normal   Pulling                 9m45s                  kubelet                     Pulling image "`[dependency-image-repo]`"
  Normal   Pulled                  8m32s                  kubelet                     Successfully pulled image "`[dependency-image-repo]`" in 1m13.182202816s (1m13.182211639s including waiting)
  Normal   Created                 8m32s                  kubelet                     Created container `[dependency]`
  Normal   Started                 8m32s                  kubelet                     Started container `[dependency]`
  Normal   Pulled                  8m21s                  kubelet                     Container image "`[dependency-image-repo]`" already present on machine
  Normal   Created                 8m21s                  kubelet                     Created container `[dependency]`
  Normal   Started                 7m43s                  kubelet                     Started container `[dependency]`
  Warning  Failed                  7m4s                   kubelet                     Failed to pull image "`[our-image-ecr]`": rpc error: code = Unknown desc = failed to pull and unpack image "`[our-image-ecr]`": failed to prepare extraction snapshot "extract-183789945-sVBg sha256:a8bdab477eec8083fa8c7403e9310fe7e56c3a7275dd27450647324ad222e082": failed to create prepare snapshot dir: failed to create temp dir: mkdir /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/new-3844284403: no space left on device
  Warning  Failed                  4m50s                  kubelet                     Failed to pull image "`[our-image-ecr]`": rpc error: code = Unknown desc = failed to pull and unpack image "`[our-image-ecr]`": failed to extract layer sha256:323417b2b3a2ec3e009da684023e11a7b3a92a7f7d7511407250f3904f63d146: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/132/fs/app/falkor/thirdparty/texrecon/build/libs/tex/CMakeFiles/tex.dir/rectangular_bin.cpp.o: no space left on device: unknown
  Normal   Pulling                 4m26s (x3 over 7m42s)  kubelet                     Pulling image "`[our-image-ecr]`"
  Normal   SecretRotationComplete  2m32s (x3 over 6m30s)  csi-secrets-store-rotation  successfully rotated K8s secret monitor-secret
  Warning  Failed                  107s (x3 over 7m4s)    kubelet                     Error: ErrImagePull
  Warning  Failed                  107s                   kubelet                     Failed to pull image "`[our-image-ecr]`": rpc error: code = Unknown desc = failed to pull and unpack image "`[our-image-ecr]`": failed to extract layer sha256:3bc1fff4b37d3e4d0274c837a59c0f5157ad31bc8d2a23c2e82b414d0913fc6c: write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/165/fs/usr/lib/x86_64-linux-gnu/libssl3.so: no space left on device: unknown
  Normal   BackOff                 93s (x5 over 7m4s)     kubelet                     Back-off pulling image "`[our-image-ecr]`"
  Warning  Failed                  93s (x5 over 7m4s)     kubelet                     Error: ImagePullBackOff

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

What's your node role policy? You need ECR permissions on the node to pull from the gov partition.

These commands are missing the gov partition
aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy

aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Thank you I appreciate the answer! Unfortunately this is not our issue - we are already using the aws-us-gov aws partition for our node role's policies and have the govcloud equivalents of the above four attached.
Our node role policies are just the above four policies, and then the trust relationship is just as suggested in the docs:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Also we do have ECR permissions - to reiterate the issue, we are pulling the large image successfully, but then upon successful pull we are attempting to pull again until there is insufficient capacity. This is a behavior we are facing only when Karpenter is enabled in solely govcloud.

Ah oops -- I completely missed the issue. That's very strange -- seems like an interaction w/ the kubelet and your image repo, but odd that it only happens w/ Karpenter. Are you able to see anything interesting in kubelet logs? You can get logs like this: https://karpenter.sh/v0.27.1/troubleshooting/#node-notready

standard AWS, those AMIs are amazon-provided bottlerocket AMIs, and they are able to schedule pods.

Are you using Bottlerocket in govcloud, too?

We are not currently using bottlerocket in govcloud because it is not FIPS compliant at the moment. Our AMI is built with the standard Amazon Linux 2 AMI as a base.
our AWSNodeTemplate manifest
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: amd-os
spec:
  amiSelector:
    Name: <<our-custom-ami>>
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        deleteOnTermination: true
        volumeSize: 200Gi
        volumeType: gp3
    - deviceName: /dev/xvdb
      ebs:
        deleteOnTermination: true
        volumeSize: 200Gi
        volumeType: gp3
  securityGroupSelector:
    karpenter.sh/discovery: <<cluster-name>>
  subnetSelector:
    karpenter.sh/discovery: <<cluster-name>>
  userData: >
    MIME-Version: 1.0

    Content-Type: multipart/mixed; boundary="BOUNDARY"

    --BOUNDARY

    Content-Type: text/x-shellscript; charset="us-ascii"

    #!/bin/bash

    set -e

    /etc/eks/bootstrap.sh <<cluster-name>> --b64-cluster-ca $B64_CLUSTER_CA
    --apiserver-endpoint $API_SERVER_URL

    --BOUNDARY--
status:
  securityGroups:
    - id: `<<redacted>>`
  subnets:
    - id: `<<redacted>>`
      zone: us-gov-west-1b
    - id: `<<redacted>>`
      zone: us-gov-west-1c
    - id: `<<redacted>>`
      zone: us-gov-west-1a

our Provisioner manifest
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: amd-os-20-provisioner
spec:
  limits:
    resources:
      cpu: '120'
  providerRef:
    name: amd-os
  requirements:
    - key: node.kubernetes.io/instance-type
      operator: In
      values:
        - m5.2xlarge
    - key: kubernetes.io/arch
      operator: In
      values:
        - amd64
    - key: topology.kubernetes.io/zone
      operator: In
      values:
        - us-gov-west-1a
        - us-gov-west-1b
        - us-gov-west-1c
    - key: kubernetes.io/os
      operator: In
      values:
        - linux
    - key: karpenter.sh/capacity-type
      operator: In
      values:
        - on-demand
  ttlSecondsAfterEmpty: 30
  weight: 20

some kubelet logs from one of our nodes
...
Apr 05 16:38:16 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: E0405 16:38:16.861411   18040 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"aws-node\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=aws-node pod=aws-node-mj8mh_kube-system(80080f9e-711e-48cb-aead-9e087b964094)\"" pod="kube-system/aws-node-mj8mh" podUID=80080f9e-711e-48cb-aead-9e087b964094
Apr 05 16:38:21 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: I0405 16:38:21.553601   18040 kubelet.go:2136] "SyncLoop DELETE" source="api" pods=[observability/datadog-59hvp]
Apr 05 16:38:21 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: I0405 16:38:21.566712   18040 kubelet.go:2130] "SyncLoop REMOVE" source="api" pods=[observability/datadog-59hvp]
Apr 05 16:38:21 `<<redacted>>`us-gov-west-1.compute.internal kubelet[18040]: I0405 16:38:21.891517   18040 kubelet.go:2120] "SyncLoop ADD" source="api" pods=[observability/datadog-rz6gk]
Apr 05 16:38:21 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: I0405 16:38:21.891582   18040 eviction_manager.go:167] "Failed to admit pod to node" pod="observability/datadog-rz6gk" nodeCondition=[DiskPressure]
Apr 05 16:38:29 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: I0405 16:38:29.861492   18040 scope.go:110] "RemoveContainer" containerID="e3cf71efb0c4662afd2219a2cbaea51aef93d999c667c55dd005df235e6020c1"
Apr 05 16:38:29 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: E0405 16:38:29.861901   18040 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"aws-node\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=aws-node pod=aws-node-mj8mh_kube-system(80080f9e-711e-48cb-aead-9e087b964094)\"" pod="kube-system/aws-node-mj8mh" podUID=80080f9e-711e-48cb-aead-9e087b964094
Apr 05 16:38:30 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: I0405 16:38:30.812379   18040 kubelet.go:1364] "Image garbage collection succeeded"
Apr 05 16:38:36 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: W0405 16:38:36.178813   18040 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {/var/lib/kubelet/plugins/csi-secrets-store/csi.sock /var/lib/kubelet/plugins/csi-secrets-store/csi.sock <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial unix /var/lib/kubelet/plugins/csi-secrets-store/csi.sock: connect: no such file or directory". Reconnecting...
Apr 05 16:38:42 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: I0405 16:38:42.861310   18040 scope.go:110] "RemoveContainer" containerID="e3cf71efb0c4662afd2219a2cbaea51aef93d999c667c55dd005df235e6020c1"
Apr 05 16:38:42 `<<redacted>>`.us-gov-west-1.compute.internal kubelet[18040]: E0405 16:38:42.861766   18040 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"aws-node\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=aws-node pod=aws-node-mj8mh_kube-system(80080f9e-711e-48cb-aead-9e087b964094)\"" pod="kube-system/aws-node-mj8mh" podUID=80080f9e-711e-48cb-aead-9e087b964094
...

We believe that the csi-secrets-store socket error isn't related to this.
Thank you again! We appreciate the help and guidance!

Can you confirm that the volumes are as big as you think they are? Just ssh to the node and check. How large is the image you're trying to pull. From your events, it looks like it fails to pull the image the first time:
  Warning  Failed                  7m4s                   kubelet                     Failed to pull image "`[our-image-ecr]`": rpc error: code = Unknown desc = failed to pull and unpack image "`[our-image-ecr]`": failed to prepare extraction snapshot "extract-183789945-sVBg sha256:a8bdab477eec8083fa8c7403e9310fe7e56c3a7275dd27450647324ad222e082": failed to create prepare snapshot dir: failed to create temp dir: mkdir /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/new-3844284403: no space left on device

And then repeatedly fails afterwards
  Normal   BackOff                 93s (x5 over 7m4s)     kubelet                     Back-off pulling image "`[our-image-ecr]`"

Here is the disk information for one of the nodes provisioned by Karpenter
(NOTE: this is not the same node as the one referenced in the previous comment - as that one has been deprovisioned by Karpenter)
sh-4.2$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  4.5M   16G   1% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/nvme0n1p1  200G  2.7G  198G   2% /
tmpfs            16G  8.0K   16G   1% /tmp
/dev/nvme2n1p1   20G   17G  1.6G  92% /var
/dev/nvme2n1p4   20G  1.4M   19G   1% /home
tmpfs            16G     0   16G   0% /var/tmp
/dev/nvme2n1p5   99G   15G   80G  16% /var/lib/docker
/dev/nvme2n1p2   20G   42M   19G   1% /var/log
/dev/nvme2n1p3   20G  661M   18G   4% /var/log/audit
tmpfs           3.1G     0  3.1G   0% /run/user/0
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/59cc9fe1c81b424ba670095c60a0f92423d7bd9c59faae7cb8e2cc8cb495beca/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/f4f87c8bca110b6df004f76299965e0d056646a7806a3f0824808aab1a1e7460/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/05ba66ace5ba72acb834e06a5db8ce22e9aa57fc45a260be35e61b4efc44cd45/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/320b43b74bd3985eff4215e3d8cbae19887c92a0eb7185ee9f28bf87a3b21f3f/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c9da7d2ed090f623a2cb2ec750e290445027afd7b38863bdbdb40db7227bfd07/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/26d42684687ee62a4da36ef59f5e2135c9d21babd77e34a6c3fe20af168884b5/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/b428cf302b2c9713f6c41e07b2f47f5fd570e21bd23fdea732fed1fec5d52d10/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/42c2c2c0a5cd3ce48149b815f243ebe5eaf4ea486bc830b213cd4a01ea290fe0/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/7c74943e77704adf3470565482d38da6195ec3cdd7cadf9e7ae5706cdaf4b330/shm
shm              64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/605e70c03f09ebde2e2919cb04e7342951468b4266217c205ff9b1daf7ad5f19/shm

Your /var partition only has 20G of space which is where it looks like your images are being pulled.  I know some of the STIGs require separate partitions, so that is likely what required creation of the 20GB /var.  However you appear to be running out of space.

Thank you for the input @tzneal. As stated above, we are aware that we are running out of disk space. This issue is NOT present when using cluster autoscaler with the same AMI and is ONLY present when enabling Karpenter.
This ECR image in question is 6499.24 MB in size, and the image is able to be successfully pulled at first, however upon successful pull the image is attempted to be pulled again - a behavior we see only with Karpenter enabled. This behavior was not captured in the logs that were posted above.
My team would love to leverage the power of Karpenter across all of our AWS stacks - including govcloud, however worst case scenario we can always revert back to using cluster autoscaler for our govcloud environments. We believe that this is an issue specifically with Karpenter due to this behavior not existing when using cluster autoscaler with the same AMI (with a much smaller volume in that case, 20GB vs 200GB).
Once again I appreciate any and all previous and ongoing help and input on this issue!

Images will be GC'd and removed if not in use and you experience disk pressure, so a subsequent pod may need to pull the same image.  Can you inspect /var and see what is using the space?

We are going recreate our AMI with a larger /var partition. We will report back with information upon increasing the /var partition size.
Thank you again for all the help, input, and guidance! 🙏


add support to override not ready taint to be ignored by the CA

Please ensure your pull request adheres to the following guidelines:

[x] For first time contributors, read Submitting a pull request
[x] All code is covered by unit and/or runtime tests where feasible.
[x] All commits contain a well written commit description including a title,
description and a Fixes: #XXX line if the commit addresses a particular
GitHub issue.
[x] All commits are signed off. See the section Developer’s Certificate of Origin
[x] Provide a title or release-note blurb suitable for the release notes.
[x] Thanks for contributing!

As described in this issue, if you scale a node pool down to zero, agent-not-ready taints can prevent the cluster autoscaler from scaling the node pool up from zero nodes because it thinks nothing will be able to run on the node (or at least the pending app pods won't be able to).
This PR adds the ability to pass a prefix that will be prepended to the agent-not-ready taint in order for it to be ignored by the cluster autoscaler when generating sanitized node templates in order to make scaling decisions. More specifically, this allows scaling node pools up from zero in GKE, where you can not customize the command line arguments of the cluster autoscaler, and where a multi-zone node pool is made up of multiple node pools under the hood, making it very easy to end up with a "node pool" with no nodes in it if you scale below 1 node per zone.
Fixes: #19241
Add config flag to add a prefix to AgentNotReadyNodeTaint value in order to enable the taint being ignored by cluster autoscaler.

let's wait until #19074 and one other PR is not open yet is merged first.

@thejosephstevens the PR https://github.com/cilium/cilium/pull/19074 was merged, can you rebase and squash this PR against master? Thank you

Thanks for the PR @thejosephstevens! I have a few comments:

As it currently stands, users will have to resort to extraConfig in Helm to specify agent-not-ready-taint-prefix, because a corresponding Helm value is currently not being added. It would be great if you could add this (I suggest agentNotReadyTaintPrefix so it matched the name of the option).
Also, the choice being given to users seems to be between the old taint key (agent.cilium.io/not-ready) and <custom-prefix>/cilium-agent-not-ready. While this is fine and works, I wonder if we should instead give users full control over the taint key (i.e., insteadl of agent-not-ready-taint-prefix, we'd support something like agent-not-ready-taint-key). It seems to me the work is essentially done, so this shouldn't imply a big change.
There is no documentation being added/updated as part of the PR. I suggest adding the Helm value and document it there, as that should be picked up by helm-docs, and at least making a mention to the new property and the reason for introducing it in this page.
I also believe that we should update CI to exercise the new code path. This should amount to updating this line (and the corresponding ones in the -1.10 and -1.11 files), and setting the new option somewhere around here.

Thanks for the review @bmcustodio ! We're working through the suggestions (they look pretty straightforward - I expect we'll have something ready later today). Just a note, @XiaoxiHuang21 is taking over lead on this PR. Feel free to tag me, but you'll likely get quicker responses tagging her directly.

Thanks for the PR @thejosephstevens! I have a few comments:

As it currently stands, users will have to resort to extraConfig in Helm to specify agent-not-ready-taint-prefix, because a corresponding Helm value is currently not being added. It would be great if you could add this (I suggest agentNotReadyTaintPrefix so it matched the name of the option).
Also, the choice being given to users seems to be between the old taint key (agent.cilium.io/not-ready) and <custom-prefix>/cilium-agent-not-ready. While this is fine and works, I wonder if we should instead give users full control over the taint key (i.e., insteadl of agent-not-ready-taint-prefix, we'd support something like agent-not-ready-taint-key). It seems to me the work is essentially done, so this shouldn't imply a big change.
There is no documentation being added/updated as part of the PR. I suggest adding the Helm value and document it there, as that should be picked up by helm-docs, and at least making a mention to the new property and the reason for introducing it in this page.
I also believe that we should update CI to exercise the new code path. This should amount to updating this line (and the corresponding ones in the -1.10 and -1.11 files), and setting the new option somewhere around here.

Thanks for the review @bmcustodio ! Updated as per comment, PTAL

Thanks @bmcustodio. I changed as per suggestion. But since my machine is arm64, cannot run this command make -C Documentation update-helm-values. I tried running it in a x86 image, but got this error:
time="2022-04-28T18:43:57Z" level=error msg="Error finding chart directories: lstat /src/install/kubernetes: no such file or directory"
Wondering if there is a better way to run this command on a arm64 machine?

Thanks @bmcustodio ! Please take another look.

/test

Only a couple nits and it should be good. However, CI seems to be failing.

Updated. Thank you @bmcustodio

@XiaoxiHuang21 @thejosephstevens just checking if you'll be able to perform the latest requested changes ☝ over the next coming days?

@XiaoxiHuang21 @thejosephstevens just checking if you'll be able to perform the latest requested changes ☝ over the next coming days?

@bmcustodio Updated, sorry about the delay.

No worries @XiaoxiHuang21 — and thank you!

@bmcustodio Thank you! Looks like the CI is still failing. Wondering how to debug this?

CI is failing on some linting checks.  Could you upgrade to Go 1.18 and try again?

 Could you upgrade to Go 1.18 and try again?

yeah, I upgraded Go and now getting this. It seems like Mac things, I'm on M1.
 Could you run the following and squash the changes into the same commit?

make -C Documentation update-cmdref
make -C install/kubernetes
Run goimports against operator/watchers/node_taint_test.go

I was able to run both make commands successfully from Linux, what else needs to happen on the goimports side? I just ran goimports operator/watchers/node_taint_test.go and it looks like it just printed the file (potentially a formatted file, but I'm not familiar with how to use goimports).

I just ran goimports operator/watchers/node_taint_test.go and it looks like it just printed the file (potentially a formatted file, but I'm not familiar with how to use goimports).

you can add -w flag to write file to disk. also cilium import convention required -local flag as well.
goimports -w -local operator/watchers/node_taint_test.go

Looks like no changes from the goimports

goimports is sometimes a little bit tricky with implicit rule, please find below the patch to make CI happy.
Also, there is small conflict with master, appreciate if you can rebase with current master.

patch
Index: .github/workflows/conformance-gke.yaml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/.github/workflows/conformance-gke.yaml b/.github/workflows/conformance-gke.yaml
--- a/.github/workflows/conformance-gke.yaml	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/.github/workflows/conformance-gke.yaml	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -151,6 +151,7 @@
             --helm-set=clustermesh.apiserver.image.useDigest=false \
             --helm-set=hubble.relay.image.repository=quay.io/${{ github.repository_owner }}/hubble-relay-ci \
             --helm-set=hubble.relay.image.tag=${SHA} \
+            --helm-set=agentNotReadyTaintKey=ignore-taint.cluster-autoscaler.kubernetes.io/cilium-agent-not-ready \
             --wait=false \
             --rollback=false \
             --config monitor-aggregation=none \
@@ -209,7 +210,7 @@
             --machine-type e2-custom-2-4096 \
             --disk-type pd-standard \
             --disk-size 10GB \
-            --node-taints node.cilium.io/agent-not-ready=true:NoExecute \
+            --node-taints ignore-taint.cluster-autoscaler.kubernetes.io/cilium-agent-not-ready=true:NoExecute \
             --preemptible

       - name: Get cluster credentials
Index: Documentation/cmdref/cilium-agent.md
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/Documentation/cmdref/cilium-agent.md b/Documentation/cmdref/cilium-agent.md
--- a/Documentation/cmdref/cilium-agent.md	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/Documentation/cmdref/cilium-agent.md	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -13,6 +13,7 @@

   --agent-health-port int                                   TCP port for agent health status API (default 9879)
   --agent-labels strings                                    Additional labels to identify this agent

 --agent-not-ready-taint-key string                        Key of the taint indicating that Cilium is not ready on the node (default "node.cilium.io/agent-not-ready")
 --allocator-list-timeout duration                         Timeout for listing allocator state before exiting (default 3m0s)
 --allow-icmp-frag-needed                                  Allow ICMP Fragmentation Needed type packets for purposes like TCP Path MTU. (default true)
 --allow-localhost string                                  Policy when to allow local stack to reach local endpoints { auto | always | policy } (default "auto")

Index: Documentation/gettingstarted/taints.rst
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
diff --git a/Documentation/gettingstarted/taints.rst b/Documentation/gettingstarted/taints.rst
--- a/Documentation/gettingstarted/taints.rst	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/Documentation/gettingstarted/taints.rst	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -33,18 +33,25 @@
on a given node to help preventing pods from starting before Cilium runs on said
node. The mechanism works as follows:
-1. The cluster administrator places a taint with key

node.cilium.io/agent-not-ready on a given uninitialized node. Depending on
the taint's effect (see below), this prevents pods that don't have a matching
toleration from either being scheduled or altogether running on the node until
the taint is removed.
+1. The cluster administrator places a specific taint (see below) on a given

uninitialized node. Depending on the taint's effect (see below), this prevents
pods that don't have a matching toleration from either being scheduled or
altogether running on the node until the taint is removed.

Cilium runs on the node, initializes it and, once ready, removes the

node.cilium.io/agent-not-ready taint.

aforementioned taint.

From this point on, pods will start being scheduled and running on the node,
having their networking managed by Cilium.

+By default, the taint key is node.cilium.io/agent-not-ready, but in some
+scenarios (such as when Cluster Autoscaler is being used but its flags cannot be
+configured) this key may need to be tweaked. This can be done using the
+agent-not-ready-taint-key option. In the aforementioned example, users should
+specify a key starting with ignore-taint.cluster-autoscaler.kubernetes.io/.
+When such a value is used, the Cluster Autoscaler will ignore it when simulating
+scheduling, allowing the cluster to scale up.
+
The taint's effect should be chosen taking into account the following
considerations:
Index: Documentation/helm-values.rst
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
diff --git a/Documentation/helm-values.rst b/Documentation/helm-values.rst
--- a/Documentation/helm-values.rst	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/Documentation/helm-values.rst	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -17,6 +17,10 @@
- Install the cilium agent resources.
- bool
- true

agentNotReadyTaintKey

- Configure the key of the taint indicating that Cilium is not ready on the node. When set to a value starting with ``ignore-taint.cluster-autoscaler.kubernetes.io/``\ , the Cluster Autoscaler will ignore the taint on it's decisions, allowing the cluster to scale up.

- string

- ``"node.cilium.io/agent-not-ready"``

alibabacloud.enabled
Enable AlibabaCloud ENI integration
bool
Index: daemon/cmd/daemon_main.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/daemon/cmd/daemon_main.go b/daemon/cmd/daemon_main.go
--- a/daemon/cmd/daemon_main.go	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/daemon/cmd/daemon_main.go	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -512,6 +512,9 @@
flags.String(option.K8sNamespaceName, "", "Name of the Kubernetes namespace in which Cilium is deployed in")
option.BindEnv(option.K8sNamespaceName)

flags.String(option.AgentNotReadyNodeTaintKeyName, defaults.AgentNotReadyNodeTaint, "Key of the taint indicating that Cilium is not ready on the node")

option.BindEnv(option.AgentNotReadyNodeTaintKeyName)

flags.Bool(option.JoinClusterName, false, "Join a Cilium cluster via kvstore registration")
option.BindEnv(option.JoinClusterName)

Index: install/kubernetes/cilium/README.md
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
diff --git a/install/kubernetes/cilium/README.md b/install/kubernetes/cilium/README.md
--- a/install/kubernetes/cilium/README.md	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/install/kubernetes/cilium/README.md	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -55,6 +55,7 @@
|-----|------|---------|-------------|
| affinity | object | {"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"k8s-app":"cilium"}},"topologyKey":"kubernetes.io/hostname"}]}} | Affinity for cilium-agent. |
| agent | bool | true | Install the cilium agent resources. |
+| agentNotReadyTaintKey | string | "node.cilium.io/agent-not-ready" | Configure the key of the taint indicating that Cilium is not ready on the node. When set to a value starting with ignore-taint.cluster-autoscaler.kubernetes.io/, the Cluster Autoscaler will ignore the taint on its decisions, allowing the cluster to scale up. |
| alibabacloud.enabled | bool | false | Enable AlibabaCloud ENI integration |
| annotateK8sNode | bool | false | Annotate k8s node upon initialization with Cilium's metadata. |
| autoDirectNodeRoutes | bool | false | Enable installation of PodCIDR routes between worker nodes if worker nodes share a common L2 network segment. |
Index: install/kubernetes/cilium/templates/cilium-configmap.yaml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
diff --git a/install/kubernetes/cilium/templates/cilium-configmap.yaml b/install/kubernetes/cilium/templates/cilium-configmap.yaml
--- a/install/kubernetes/cilium/templates/cilium-configmap.yaml	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/install/kubernetes/cilium/templates/cilium-configmap.yaml	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -889,3 +889,7 @@
{{ toYaml .Values.extraConfig | nindent 2 }}
{{- end }}
{{- end }}
+
+{{- if hasKey .Values "agentNotReadyTaintKey" }}

agent-not-ready-taint-key: {{ .Values.agentNotReadyTaintKey | quote }}
+{{- end }}
Index: install/kubernetes/cilium/values.yaml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/install/kubernetes/cilium/values.yaml b/install/kubernetes/cilium/values.yaml
--- a/install/kubernetes/cilium/values.yaml	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/install/kubernetes/cilium/values.yaml	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -2054,3 +2054,7 @@

-- Configure whether to unload DNS policy rules on graceful shutdown
dnsPolicyUnloadOnShutdown: false

+# -- Configure the key of the taint indicating that Cilium is not ready on the node.
+# When set to a value starting with ignore-taint.cluster-autoscaler.kubernetes.io/, the Cluster Autoscaler will ignore the taint on its decisions, allowing the cluster to scale up.
+agentNotReadyTaintKey: "node.cilium.io/agent-not-ready"
Index: operator/flags.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
diff --git a/operator/flags.go b/operator/flags.go
--- a/operator/flags.go	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/operator/flags.go	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -13,8 +13,8 @@
operatorOption "github.com/cilium/cilium/operator/option"
"github.com/cilium/cilium/pkg/defaults"
ipamOption "github.com/cilium/cilium/pkg/ipam/option"

ciliumio "github.com/cilium/cilium/pkg/k8s/apis/cilium.io"
"github.com/cilium/cilium/pkg/option"

pkgOption "github.com/cilium/cilium/pkg/option"
)

func init() {
@@ -327,7 +327,7 @@
flags.String(operatorOption.CiliumPodLabels, "k8s-app=cilium", "Cilium Pod's labels. Used to detect if a Cilium pod is running to remove the node taints where its running and set NetworkUnavailable to false")
option.BindEnv(operatorOption.CiliumPodLabels)

flags.Bool(operatorOption.RemoveCiliumNodeTaints, true, fmt.Sprintf("Remove node taint %q from Kubernetes nodes once Cilium is up and running", ciliumio.AgentNotReadyNodeTaint))

flags.Bool(operatorOption.RemoveCiliumNodeTaints, true, fmt.Sprintf("Remove node taint %q from Kubernetes nodes once Cilium is up and running", pkgOption.Config.AgentNotReadyNodeTaintValue()))
option.BindEnv(operatorOption.RemoveCiliumNodeTaints)
flags.Bool(operatorOption.SetCiliumIsUpCondition, true, "Set CiliumIsUp Node condition to mark a Kubernetes Node that a Cilium pod is up and running in that node")
Index: operator/watchers/node_taint.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/operator/watchers/node_taint.go b/operator/watchers/node_taint.go
--- a/operator/watchers/node_taint.go	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/operator/watchers/node_taint.go	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -12,12 +12,12 @@
"github.com/cilium/cilium/operator/option"
"github.com/cilium/cilium/pkg/controller"
"github.com/cilium/cilium/pkg/k8s"

ciliumio "github.com/cilium/cilium/pkg/k8s/apis/cilium.io"
"github.com/cilium/cilium/pkg/k8s/informer"
slim_corev1 "github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1"
slim_metav1 "github.com/cilium/cilium/pkg/k8s/slim/k8s/apis/meta/v1"
k8sUtils "github.com/cilium/cilium/pkg/k8s/utils"
"github.com/cilium/cilium/pkg/logging/logfields"

pkgOption "github.com/cilium/cilium/pkg/option"
"github.com/sirupsen/logrus"
corev1 "k8s.io/api/core/v1"
@@ -203,7 +203,7 @@
// Not Ready Node Taint.
func hasAgentNotReadyTaint(k8sNode *slim_corev1.Node) bool {
for _, taint := range k8sNode.Spec.Taints {

  if taint.Key == ciliumio.AgentNotReadyNodeTaint {

  if taint.Key == pkgOption.Config.AgentNotReadyNodeTaintValue() {
  	return true
  }

}
@@ -325,7 +325,7 @@
var taints []slim_corev1.Taint
for _, taint := range k8sNode.Spec.Taints {

  if taint.Key != ciliumio.AgentNotReadyNodeTaint {

  if taint.Key != pkgOption.Config.AgentNotReadyNodeTaintValue() {
  	taints = append(taints, taint)
  } else {
  	taintFound = true

@@ -336,13 +336,13 @@
if !taintFound {
log.WithFields(logrus.Fields{
logfields.NodeName: nodeName,

  	"taint":            ciliumio.AgentNotReadyNodeTaint,

  	"taint":            pkgOption.Config.AgentNotReadyNodeTaintValue(),
  }).Debug("Taint not found in node")
  return nil

}
log.WithFields(logrus.Fields{
logfields.NodeName: nodeName,

  "taint":            ciliumio.AgentNotReadyNodeTaint,

  "taint":            pkgOption.Config.AgentNotReadyNodeTaintValue(),

}).Debug("Removing Node Taint")
createStatusAndNodePatch := []k8s.JSONPatch{
Index: operator/watchers/node_taint_test.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/operator/watchers/node_taint_test.go b/operator/watchers/node_taint_test.go
--- a/operator/watchers/node_taint_test.go	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/operator/watchers/node_taint_test.go	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -21,9 +21,9 @@
"github.com/cilium/cilium/pkg/checker"
"github.com/cilium/cilium/pkg/k8s"

ciliumio "github.com/cilium/cilium/pkg/k8s/apis/cilium.io"
slim_corev1 "github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1"
slim_metav1 "github.com/cilium/cilium/pkg/k8s/slim/k8s/apis/meta/v1"

pkgOption "github.com/cilium/cilium/pkg/option"
"github.com/cilium/cilium/pkg/testutils"
)

@@ -60,7 +60,7 @@
Spec: slim_corev1.NodeSpec{
Taints: []slim_corev1.Taint{
{

  			Key: ciliumio.AgentNotReadyNodeTaint, Value: "Foo",

  			Key: pkgOption.Config.AgentNotReadyNodeTaintValue(), Value: "Foo",
  		},
  		{
  			Key: "DoNoRemoveThisTaint", Value: "Foo",

@@ -106,7 +106,7 @@
Path: "/spec/taints",
Value: []slim_corev1.Taint{
{

  				Key: ciliumio.AgentNotReadyNodeTaint, Value: "Foo",

  				Key: pkgOption.Config.AgentNotReadyNodeTaintValue(), Value: "Foo",
  			},
  			{
  				Key: "DoNoRemoveThisTaint", Value: "Foo",

@@ -196,7 +196,7 @@
Spec: slim_corev1.NodeSpec{
Taints: []slim_corev1.Taint{
{

  			Key: ciliumio.AgentNotReadyNodeTaint, Value: "Foo",

  			Key: pkgOption.Config.AgentNotReadyNodeTaintValue(), Value: "Foo",
  		},
  		{
  			Key: "DoNoRemoveThisTaint", Value: "Foo",

@@ -329,7 +329,7 @@
Spec: slim_corev1.NodeSpec{
Taints: []slim_corev1.Taint{
{

  			Key: ciliumio.AgentNotReadyNodeTaint, Value: "Foo",

  			Key: pkgOption.Config.AgentNotReadyNodeTaintValue(), Value: "Foo",
  		},
  		{
  			Key: "DoNoRemoveThisTaint", Value: "Foo",

Index: pkg/defaults/node.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
diff --git a/pkg/defaults/node.go b/pkg/defaults/node.go
--- a/pkg/defaults/node.go	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/pkg/defaults/node.go	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -19,8 +19,18 @@
// HostDevice is the name of the device that connects the cilium IP
// space with the host's networking model
HostDevice = "cilium_host"
+
// SecondHostDevice is the name of the second interface of the host veth pair.
SecondHostDevice = "cilium_net"
+

// CiliumK8sAnnotationPrefix is the prefix key for the annotations used in kubernetes.
CiliumK8sAnnotationPrefix = "cilium.io/"

// AgentNotReadyNodeTaint is a node taint which prevents pods from being
// scheduled. Once cilium is setup it is removed from the node. Mostly
// used in cloud providers to prevent existing CNI plugins from managing
// pods.
AgentNotReadyNodeTaint = "node." + CiliumK8sAnnotationPrefix + "agent-not-ready"
)

var (
Index: pkg/k8s/apis/cilium.io/const.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
diff --git a/pkg/k8s/apis/cilium.io/const.go b/pkg/k8s/apis/cilium.io/const.go
--- a/pkg/k8s/apis/cilium.io/const.go	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/pkg/k8s/apis/cilium.io/const.go	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -62,17 +62,8 @@
// to sync the CNP with kube-apiserver.
CtrlPrefixPolicyStatus = "sync-cnp-policy-status"

// CiliumK8sAnnotationPrefix is the prefix key for the annotations used in kubernetes.
CiliumK8sAnnotationPrefix = "cilium.io/"
// CiliumIdentityAnnotationDeprecated is the previous annotation key used to map to an endpoint's security identity.
CiliumIdentityAnnotationDeprecated = "cilium-identity"

// AgentNotReadyNodeTaint is a node taint which prevents pods from being
// scheduled. Once cilium is setup it is removed from the node. Mostly
// used in cloud providers to prevent existing CNI plugins from managing
// pods.
AgentNotReadyNodeTaint = "node." + CiliumK8sAnnotationPrefix + "agent-not-ready"
)

const (
Index: pkg/option/config.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
diff --git a/pkg/option/config.go b/pkg/option/config.go
--- a/pkg/option/config.go	(revision 26cb7ded2d8fdb69d309b84c0038a32c4c922392)
+++ b/pkg/option/config.go	(revision c42a9907cfa7b7aa81a18922fc9cc8900ee3c2ea)
@@ -657,6 +657,10 @@
// K8sNamespaceName is the name of the K8sNamespace option
K8sNamespaceName = "k8s-namespace"

// AgentNotReadyNodeTaintKeyName is the name of the option to set
// AgentNotReadyNodeTaintKey
AgentNotReadyNodeTaintKeyName = "agent-not-ready-taint-key"
// JoinClusterName is the name of the JoinCluster Option
JoinClusterName = "join-cluster"

@@ -1539,6 +1543,12 @@
// deployed in when running in Kubernetes mode
K8sNamespace string

// AgentNotReadyNodeTaint is a node taint which prevents pods from being
// scheduled. Once cilium is setup it is removed from the node. Mostly
// used in cloud providers to prevent existing CNI plugins from managing
// pods.
AgentNotReadyNodeTaintKey string
// JoinCluster is 'true' if the agent should join a Cilium cluster via kvstore
// registration
JoinCluster bool
@@ -2495,6 +2505,16 @@
return c.K8sNamespace
}

+// AgentNotReadyNodeTaintValue returns the value of the taint key that cilium agents
+// will manage on their nodes
+func (c *DaemonConfig) AgentNotReadyNodeTaintValue() string {

if c.AgentNotReadyNodeTaintKey != "" {

  return c.AgentNotReadyNodeTaintKey

} else {

  return defaults.AgentNotReadyNodeTaint

}
+}

// K8sAPIDiscoveryEnabled returns true if API discovery of API groups and
// resources is enabled
func (c *DaemonConfig) K8sAPIDiscoveryEnabled() bool {
@@ -3181,6 +3201,7 @@
c.HTTP403Message = viper.GetString(HTTP403Message)
c.DisableEnvoyVersionCheck = viper.GetBool(DisableEnvoyVersionCheck)
c.K8sNamespace = viper.GetString(K8sNamespaceName)

c.AgentNotReadyNodeTaintKey = viper.GetString(AgentNotReadyNodeTaintKeyName)
c.MaxControllerInterval = viper.GetInt(MaxCtrlIntervalName)
c.PolicyQueueSize = sanitizeIntParam(PolicyQueueSize, defaults.PolicyQueueSize)
c.EndpointQueueSize = sanitizeIntParam(EndpointQueueSize, defaults.EndpointQueueSize)

</details>

@thejosephstevens friendly ping if you have have some time for the above. In case of you are busy, I can do rebase and make CI happy if you grant me write permission on your forked repo. Thanks a lot.

I have forced pushed your commit here after resolving conflicts with master, as well as minor formatting changes to make CI happy. Thanks for your support.

/test

/test

Forced pushed after rebase with master and fixed upgrade tests.

/test

/test-runtime

/test-1.23-net-next

reviews are in, required CI tests are passed. I am marking this ready to merge, please let me know if you think otherwise.

Thanks so much for taking this across the line @sayboras !!!


AlsoLoad and NotSaved mappings for YAML

The AlsoLoad and NotSaved features are not configurable through YAML. YamlDriver should be updated to support these migration features.
Here's an idea for how they can be supported:
My\Domain\Thing:
    type: document
    inheritanceType: SINGLE_COLLECTION
    fields:
        hash:
            id:  true
        customDescription:
            type: string
            name: custom_description
            alsoLoad: old_description
        notSavedField:
            name: not_saved_field
            notSaved: true
        createdDate:
            type: date
            name: created_date
        modifiedDate:
            type: date
            name: modified_date
    lifecycleCallbacks:
        prePersist: [ onPrePersist ]
        preUpdate: [ onPreUpdate ]
    alsoLoad:
        migrateSomeField: [ someField ]

Closing due to lack of feedback. As @jmikola explained, both properties are supported by the yaml driver.


patch(update-pom): Fix version tag

:tada: This PR is included in version 1.0.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


How to add a babel plugin

Sorry for my ignorance, but after ejecting I can't seem to figure out how to add a plugin,
I'm used to the simplicity of .babelrc but it seems to not work with create-react-app
Thanks.
When I add .babelrc I get the following error:

I looked at this issue: https://github.com/facebookincubator/create-react-app/issues/1069 by gaearon where he basically says:
after ejecting, in order to use .babelrc you need to add "presets": ["react-app"] for it to work

How about without ejecting?

There is no way to add a Babel plugin without ejecting. This helps us ensure that the setup works correctly and isn't suspecticle to very common configuration mistakes or inclusion of unmaitained plugins. What plugin did you want to add?

Hmm, so there's no other option unless ejecting.
If I may suggest, how about adding babel-plugin-transform-export-extensions.
It is a stage 1 proposal but I think it a pretty solid one and it is in the official tc39 repo (https://github.com/tc39/proposal-export-default-from)

If you consider syntactic sugar to be more important than being able to get updates to underlying tools, then sure, your best option would be to eject. I probably wouldn’t do it on my project even if I liked the syntax.
I’m not aware of any news about this proposal going forward. Moving the repo could just be a logistical decision (worth checking if other proposals did the same).

Just want to add my own translation Babel plugin, but ejecting all stuff doesn't look good.

I agree we need a better story around handling i18n. A dedicated issue would be better for this though.

+1 to the internationalization issue.  I really want to eject just to get https://github.com/akameco/babel-plugin-react-intl-auto and https://github.com/akameco/extract-react-intl-messages working :/

So how can I add https://github.com/davidhu2000/react-spinners which MUST use the https://github.com/emotion-js/emotion babel plugin?

@stoyandamov I ended up using this: https://github.com/timarney/react-app-rewired

We wanted to use the babel plugin provided by styled-components [https://www.styled-components.com/docs/tooling#babel-plugin] which adds support for server-side rendering, minification of styles and gives a nicer debugging experience while using styled-components with react.
It would be great if we could add plugins by extending the babel config CRA is using.

You can accomplish this with Babel Macros support coming out in react-scripts@2. See the announcement thread.


Camera calibration is not working, giving curved images while playing the updated rosbag file ( using cameracalibrator.py for getting the parameters).)

Dear sir,
I am facing an issue regarding the camera calibration, the command to generate the parameter in the repo is not working on our system so we got the ost.yml file and the images from the ros cameracalibrator.py here is the link, I have attached a google doc file with all the steps and methods we follow, please look into it and it'd be really helpful if you could provide us with your valuable suggestions as to where are we getting it wrong.
google doc: https://drive.google.com/file/d/1xHIiCQy1X_ZurqnUTnDtCl0UAr84VXSV/view?usp=sharing
Thanks and regards
Harshal

@neurodisk23 are you able to solve ?


WIP: New site design

Description
This PR implements the fancy new site design.
Also added is the Small Horizontal card, required for the site design.
How and where has this been tested?

How this was tested: A little scrolly scroll on the deployed (internal) test site https://git.corp.adobe.com/pages/lawdavis/spectrum-css-new-design/docs/
Browser(s) and OS(s) this was tested with: Chrome 75 on macOS

Screenshots

To-do list

[x] If my change impacts other components, I have tested to make sure they don't break.
[x] If my change impacts documentation, I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.

[x] Theme/scale switcher
[ ] Bundle version switcher
[ ] Manual entry of Spectrum website URLs/slugs (currently determined from component name, which is often incorrect)
[ ] This pull request is ready to merge.

I'd like to merge this instead of #293 for the 2.14 release.

Vulnerable issue introduced by handlebars solved.


BaselineGridTextView lineHeightHint difference

Hi,
lineHeightHint in xml as 20sp - getDimensionPixelSize == 53
lineHeightHint in xml as 20sp - getDimension == 52.5
programmatically - 20 * resources.displayMetrics.scaledDensity == 52
The problem is that computeLineHeight for value 53 calculates baselineAlignedLineHeight as 62, but when I set lineHeight programmatically for value 52 the baselineAlignedLineHeight stays 52.
So in the end which should be the right value for line height? 52 or 62?

Thanks for raising this, hopefully the fix will resolve this issue for you by retaining precision for as long as possible.
Incidentally this

programmatically - 20 * resources.displayMetrics.scaledDensity == 52

…is not correct. I assume you're casting to an int at some point to get that value? If so you should always round up (thus the + 0.5f in my fix) when converting to discrete pixel values (see TypedValue for the source of truth).

Hi, yes I was using one of Anko methods:
fun Context.sp(value: Int): Int = (value * resources.displayMetrics.scaledDensity).toInt()
I applied your fix and remove casting to an int, now everything looks awesome. Thanks for your fast response.


作者您好，3.2.2版本有点问题，手机上放大手势不生效

Subject of the issue
Describe your issue here.
Your environment

version of jquery.panzoom
version of jquery
which browser and its version

Steps to reproduce
Tell us how to reproduce this issue. Please provide a working demo, you can use this template as a base.
Expected behaviour
Tell us what should happen
Actual behaviour
Tell us what happens instead

Sorry. empty issue.


Implement ValidateNamed method to support validation errors with proper names

Per @casualjim advice, a new method (ValidateNamed) should be considered to add in order to support validation errors with proper names. This is especially useful for the use case of aliased property types, in which a model can be referenced commonly (via $ref) by another property. For the sake of cleanness for the existing Validate implementation, the ValidateNamed method is therefore taken into this implementation.

I've been thinking a bit more about this and because the thing being validated would always be called from another method., that method could just rewrite the errors by adding the field name info in there.
Less invasive

@casualjim Woo... great hints! will verify it.


bluealsa-client: a new utiility for invoking D-Bus API messages from the command line.

This is a simple tool wrapping the org.bluealsa Manager1 and PCM1 D-Bus interfaces. It may be useful for selecting the A2DP codec, viewing properties, or enabling/disabling SoftVolume. The other methods and signal monitoring are also implemented, although possibly less useful in practice.

There was an email beep when I was writing a comment regarding "cmd" name :) You was faster than me.
Please, keep it "client" for now, but I've just checked on my PC how many binaries end with "*client" and there are few, e.g. "mosh-client", "rpcclient", "smbclient", which are indeed client apps. BUT there are also apps like "keepassxc-cli" or "wpa_cli" which are command line interfaces for related services (maybe bluealsa- is also more a command line interface than client). And there are also "bluetoothctl", "syslog-ng-ctl", "cec-ctl". And to be fair there is one "bccmd".
So, maybe the name should be "ctl" or "cli", but I don't know which would fit better :D

So, maybe the name should be "ctl" or "cli", but I don't know which would fit better :D

OK - its easy to change - have a think and I'll go with whatever you decide. Maybe there are others reading this who may have suggestions?
I originally had bluealsa-ctl, influenced by pulseaudio's pactl, but decided that clashed with the ALSA ctl plugin, so then went with bluealsa-cmd, based on pulseaudio's pacmd. I have no strong feelings on this, just whatever you prefer.

so then went with bluealsa-cmd, based on pulseaudio's pacmd

Ooo I see, I don't have PA on my box installed so I did not find pacmd. Also I do not have any strong feelings about the name, but once it will be merged with master changing the name would not be nice (from the user point of view). So, I'd like to define it in a way compatible with other similar tools.
Like @borine has said, any suggestions are welcomed :)

A list of potential names for this utility:
bluealsa-cli
bluealsa-client
bluealsa-cmd
bluealsa-command
bluealsa-control
bluealsa-ctl
bluealsa-dbus
bluealsa-manage
bluealsa-pcm

i propose to squash the commits in this PR once a name for the utility has been chosen - I think that should make a final review easier against a single commit.
I've used size_t instead of ssize_t for returns of read() and write(), so that needs fixing, but other than that I hope we are gettiing close to a final review now. Unless you have any new comments, we just need to choose the name. I'll continue to try to think up more candidates, but am not having any inspired ideas at present.

I'll make all the changes you request as soon as I can - have real-life priority right now. I'll also change the name to bluealsa-cli. On the subject of names, have you any thoughts on the choice of command names? For example  "softvol" could be "softvolume", or "soft-volume"; and should they all be "get-" and "set-"? or anything else ?

I'll make all the changes you request as soon as I can - have real-life priority right now.

No problem. I also have a lot of family-related real-life urgent tasks recently - that's why I had to slow down development of my open-source projects :)

On the subject of names, have you any thoughts on the choice of command names? For example "softvol" could be "softvolume", or "soft-volume"; and should they all be "get-" and "set-"? or anything else ?

I haven't thought about that yet. But since the review of the implementation part is more or less done (from my point of view) it might be a good idea to look at this. And right of the bat I've got some ideas/thoughts (not change requests :D). How about something like this:

list-pcms           <- that is fine
properties     <- for such task bluetoothctl uses "info" but properties is also OK, however, "information" might show something beyond property list alone (from the word meaning point of view)
get-codecs   <- OK, but maybe merge it with properties/info ?
select-codec    <- to this command (and other) I'd add "help" in case where "value" is missing, i.e. select-codec /path/xxx this will act as get-codecs but with emphasized currently selected codec
set-volume   <- maybe rename to simply "volume" and add volume listing in case of only path given
mute         <- show mute state when only path given
softvol          <- "soft-volume" maybe
monitor                               <- OK
open           <- OK

OK,  I will squash these commits and make final minor style changes (mixed tab-spaces indentations, double new lines, etc.) and merge it :)


error import nmslib after building from source

after building from source with the setup.py, when I tried to import nmslib got this error
Python 3.9.1 (default, Jan  8 2021, 17:17:17)
[Clang 12.0.0 (clang-1<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nmslib
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: Object of type 'Logger' is not an instance of 'module_'

Tried with python v3.9.1 and v3.7.9 on MacOS and Ubuntu 18.04. All got exactly the same error.
Not sure if it is the cause but there is a warning against a missing file similarity_search/src/main.cc, which i checked it is indeed not in this repo:
warning: no previously-included files matching 'similarity_search/src/main.cc' found anywhere in distribution
Also using pip no-binary install on all the older versions > 2.0.5 gave the same error as well, e.g.
pip install --no-dependencies --no-binary :all: nmslib==2.0.5

Does it work with older Python versions?

Does it work with older Python versions?

I'm experiencing the same. Python 3.7.9 on Ubuntu 20.04

I'm experiencing the same. Python 3.7.9 on Ubuntu 20.04

There seem to be some issues with incomplete release.

There seem to be some issues with incomplete release.

I am experiencing the same (Python 3.7.9 MacOS) but 2.0.11 didn't do the trick for me
Also I noticed that pypi failed to get the latest version when --no-binary :all:
Installed with pip
pip install --no-dependencies --no-binary :all: nmslib==2.0.11
Return the following error
ERROR: Could not find a version that satisfies the requirement nmslib==2.0.11 (from versions: 1.6.3, 1.7, 1.7.1, 1.7.2, <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, 1.8, 1.8.1, 2.0.4, 2.0.5, 2.0.6, 2.0.9) ERROR: No matching distribution found for nmslib==2.0.11

I am experiencing the same (Python 3.7.9 MacOS) but 2.0.11 didn't do the trick for me
Also I noticed that pypi failed to get the latest version when --no-binary :all:
Installed with pip
pip install --no-dependencies --no-binary :all: nmslib==2.0.11
Return the following error
ERROR: Could not find a version that satisfies the requirement nmslib==2.0.11 (from versions: 1.6.3, 1.7, 1.7.1, 1.7.2, <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, 1.8, 1.8.1, 2.0.4, 2.0.5, 2.0.6, 2.0.9) ERROR: No matching distribution found for nmslib==2.0.11

We are having multiple deployment issues. I apologize for this and I will work on resolving this next week. Unfortunately, for some reason the latest version on pypi lacks the tar ball. To install from sources, it is easiest to download the latest version (tagged 2.0.11) and just just do
pip install -r requirments.txt
python setup.py

from the python_bindings folder.

I've experienced the same issue. The problem was fixed by using the previous release of pybind11 (pip install pybind11==2.6.1).

thanks a lot @maumueller something got broken recently and I was struggling with building a proper release. That might have been it. Hold on, let's see if this was THE issue.

@zioufang @cristianmtr @DicksonK sorry about the issue, I believe it should be fixed now. Make sure you delete all the remainings of the older version when you install a new one. It should 2.1.1. Have to jump versions due to PyPi quirks. Feel free to reopen the issue or comment if it's still not working.

@searchivarius may be worth adding something to the readme temporarily noting that those versions were removed. I was able to find this issue and figure things out, but it's not going to be intuitive to someone in 2-3 months why nmslib 2.0.9 is gone and their build is failing. Thanks for your work sorting out the release issues! Greatly appreciated.

@jknix yes, it's a good thought. I am going to chart the release and I can add these notes there, but I haven't done it yet. Unfortunately, PyPi has this policy of not letting re-upload a deleted build. Overall, sorry, this release was plagued by a lot of issues. I will have to set up a test pypi account to prevent such issues in the future.

PS: @jknix what else could be a good place to mention that these releases were gone due to deployment shenanigans?

@searchivarius maybe in the quickstart docs? I feel like if it's posted there, on the readme, and PyPi, that should be plenty sufficient. At that point, if anyone can't find it then they cannot possibly be looking. I think a quick note in each place along the lines of "Note: Users who have previously installed versions 2.0.7-2.0.13 will need to uninstall and reinstall nmslib >= 2.1.1 for compatibility." Thanks!

@jknix thank you! I added a note here.
Fortunately, not so many people install nmslib from sources. Interestingly enough, it was possible to build binaries and they passed the test so apparently


Fix: perform clear action first in Onyx.clear()

Details

When we call Onyx.update(), we should perform clear action first
Related Issues

$ https://github.com/Expensify/App/issues/18686
Tests

Login to ND
Navigate to Settings > Security
Click/tap on "Close Account"
Verify the "Closed Successfully" message is below the email field.

QA Tests

Login to ND
Navigate to Settings > Security
Click/tap on "Close Account"
Verify the "Closed Successfully" message is below the email field.

Screenshots/Videos

Web

https://github.com/Expensify/react-native-onyx/assets/113963320/5da8cc32-e690-47de-a23d-63a52d7ec84d

Mobile Web - Chrome

https://github.com/Expensify/react-native-onyx/assets/113963320/5612ec46-6a74-41a0-9e42-d13b1ce93077

Mobile Web - Safari

https://github.com/Expensify/react-native-onyx/assets/113963320/13099f3c-e8d2-4e88-be54-b5da6eabadb5

Desktop

https://github.com/Expensify/react-native-onyx/assets/113963320/4f33096c-75c4-4722-8240-f3a35b73512f

iOS

https://github.com/Expensify/react-native-onyx/assets/113963320/ad6cefda-f6e4-4a52-80ba-765edcf8a213

Android

https://github.com/Expensify/react-native-onyx/assets/113963320/7c42eaa8-0e88-4fff-9ed5-85437b108a3d

Looks good! Now let's get an App PR to incorporate these changes.


Qol updates

Description
Some QOL Updates to make development easier as our dags are getting more complex.
Highlights

Parameterizes the dag schedule_interval within the manifest at the platform level while allowing for a default.
schedules.enabled added to the manifest which lets users turn off scheduling and only allow manual triggering of DAGs.
Creates models for  OpenshiftRelease, BaremetalRelease, and  DagConfig to make handling the release/dag configurations easier throughout the codebase.
Made the Baremetal installer inherit from the AbstractOpenshiftInstaller

4.8 test run http://whitleykeith-qol-updates-airflow.apps.sailplane.perf.lab.eng.rdu2.redhat.com/tree?dag_id=4.8_aws_default

@kedark3


fix(frontend): protected sensitive paths

1. Protected Sensitive Paths
2. Added user check

If the user is not logged in, render a message to log them in.

3. N/A
(Optional) 4. Slight Bug

Logging into the organization list after adding user check protection does not render the list of orgs and must reload but it shouldn't be possible to access that route anyways if not logged in.

Works well, but I think for now just have it redirect to the login page rather that display a message. That should also avoid the bug of loading in on the org list page.

I swapped the displayed text for useEffect and I was able to hack a similar fix with the logout button. But I am still figuring out how to send them to login page only if the event is private and stay on the event if public.


Variable inside Code block

Hi jffaust,
Big Thanks for this awesome plugins. Possible if we can make it happen to have the variable working in code block as well?

works for me, but not in preview mode, as stated in https://github.com/jffaust/obsidian-variables/issues/4


Add device condition support to the water_heater integration

Add device condition support to the water_heater integration.
Device conditions are part of Device Automations, a new concept that was recently launched to make automations more user friendly.
Device automations consist of three different parts: triggers, conditions and conditions. This issue is about the condition part.
To get started with this issue, you will need to set up a Home Assistant development environment. The easiest approach is to get started using a dev container.
Once you have a development environment set up, run the following and follow the instructions:
python3 -m script.scaffold device_condition

(This issue is part of #26994)

Device automations consist of three different parts: triggers, conditions and conditions.

Noticed you repeated conditions twice when looking at this issue.  Thought I'd let you know ^^

I'm working on this :)


Adding reverse geocoding feature

Assuming you have coordinates, you may want to get information from them (street address actually).
It can be easily doable by changing the ProviderInterface:

Tweak the getData() method;
Adding a reverse() method.

Adding a reverse() method in the GeocoderInterface.

done


Unable to install - possible symlink in archive?

This package looks like it would meet my needs very well, but I was unable to install it on my (Windows) system. After ignoring the platform requirements for Horizon, I am getting the following error:

Some quick review on the error message indicated that maybe there is a symlink in the package that 7z identified, causing the extraction to fail.

Hey @joyfullservice, thanks for the report! I was aware of the symlink, but never bothered to remove as I wasn't aware that it would break anything. Have just released version 1.2.1 that removes the symlink. Let me know if that fixes it. Thanks!

Yes, that appears to have resolved the issue. Thank you so much!!


Support "hooks"?

Currently there is no control over what gets executed in builder tasks (such as ods-build-go). While this is by design (to prevent customisations that are not predictable) it is probably too restrictive. For example, given we have a solution for #135, users might want to run database migrations before running the integration test suite. With the current design, this would not be possible with the official task and a custom task would be needed.
One option could be to provide more parameters that allow to run scripts before/after certain steps in the task, such as before running tests (e.g. params.script-before-tests) and after running tests (e.g. params.script-after-tests). While this would allow users to run arbitrary things in the task, it would not allow them to bypass the default behaviour without doing crazy stuff.
Thoughts?
FYI @felipecruz91 @renedupont

That could be a good starting point. I assume it will be quite simple at the beginning, by just running an entrypoint bash script, e.g. script-before-tests.sh and script-after-tests.sh where users would put their logic in there, right?

Correct. If we wanted to, we could even auto-detect such scripts instead of having many parameters. Not sure which option is better ... the params at least would provide clear documentation.

I'd rather have them explicitly via parameters


Error UnresolvableHostName

Hello,
I have a strange error. Azure says in the runs history form the icebreaker app green check marks. But on almost all icebreaker bots the first run gives an error:
UnresolvableHostName. The provided host name 'icebreaker-...-...-ttdgyml......azurewebsites.net could not be resolved.
I can click and i see: "code": "BadRequest",
"message": "Http request failed with status code 'NameResolutionFailure' and status message: 'The remote name could not be resolved: 'icebreaker-culture....-ttt.....i5lpq.azureweb.....net''."
The dots i placed because i do not want to show everything in this post. If needed i can sent it.
Please help?

+1 on this. I get the same error with the logic code.

You may need to verify allowed characters for app service, logic app, cosmosdb naming etc. Otherwise Please remove special characters from base resource name and deploy. Hope it helps.


MacOS provisioning job fails when installing docker

This one has been affecting us for a while, just noticed we don't have an issue for it.
The MacOS provisioning job (which verifies SDKs can provision engines when running natively on MacOS; only runs on main + bump engine PRs) fails more often than not before dagger is even involved when it tries to install docker.
It will just hang on Waiting for the essential requirement 1 of 2: \"ssh\"" repeatedly until it times out.
Example: https://github.com/dagger/dagger/actions/runs/9998434027/job/27637765911?pr=7978#step:3:105

It takes a few retries and eventually gets passed that step.
Clearly not a dagger issue but is deeply annoying to retry during releases especially, when it's especially crucial we verify there's no MacOS-specific provisioning issues.

This seems to have degraded further to the point where I can't find the last time it succeeded on main. It always either fails on the ssh install step or just starts hanging in such a way that I can't even seem to cancel the workflow


[zeus] tegra186-flashtools-native: tegraflash_internal fix with python 3.9.

methods getiterator have been removed on python 3.9.
Methods getchildren() and getiterator() of classes :class:~xml.etree.ElementTree.ElementTree
and :class:~xml.etree.ElementTree.Element in the :mod:~xml.etree.ElementTree module
have been removed. They were deprecated in Python 3.2.
Use iter(x) or list(x) instead of x.getchildren() and x.iter() or list(x.iter()) instead
of x.getiterator(). (Contributed by Serhiy Storchaka in :issue:36543.)
https://github.com/python/cpython/blob/master/Doc/whatsnew/3.9.rst
Signed-off-by: Jose Quaresma<EMAIL_ADDRESS>

probably tegra210-flashtools-native_32.3.1.bb can be patched too.

I think we could just backport 7bb6c3f

I don't have knowledge of the https://github.com/OE4T/meta-tegra/commit/7bb6c3f1ce8053bde6f27b73a12743d395fdc6df python 3.9 fixes.
I update this PR with the backported commit.

I've also back-ported the other Python 3 compatibility patches for the flashing tools from the dunfell branch.

Thanks


Freezing features set in preparation for v1.0.0

All the functionality I have planned for v1.0.0 is in place. From now on, I'd like to observe how it works in real life and collect (and eventually fix) all the possible issues for both the technical and UX sides.
Please feel free to report your experience and suggest minor changes to the existing functionality to make it convenient.

How long are you planning to collect issues before going live with 1.0.0?

nobody knows - as long as it takes to be certain that the latest version doesn't have any issues we know about

The current master is stable enough to become v1. It may have some minor bugs we have yet to discover, but generally, it is solid.
The api is stabilized, and no breaking change for either command line of the playbook/inventory format is planned. The code was cleaned and refactored to be easy to read and contribute to. The set of tests is comprehensive and the docs seem to be sufficient.


ZEPPELIN-1901. Output schema might be null for PigQueryInterpreter

What is this PR for?
The output schema might be null if user doesn't specify it explicitly. In this PR, I will use 'col_{pos}' to replace the column name if it is null.
What type of PR is it?
[Improvement]
Todos

[ ] - Task

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1901

How should this be tested?
Unit test is added.
Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

@felixcheung Please help review, I will add document in #1830 to encourage user to specify schema explictly.

Ping @felixcheung , CI failure is not relevant.
AngularElem
- should provide onclick method *** FAILED ***
  The code passed to eventually never returned normally. Attempted 1 times over 192.484891 milliseconds. Last failure message: 0 was not equal to 1. (AbstractAngularElemTest.scala:64)

LGTM
merging if no more comment


Data submitters want to order the datasets in a collection based on their relevance

Story
Data submitters want to order datasets in the collection detail and datasets view based on their relevance.
Background
@pablo-gar recently curated Tabula Muris Senis which contains 42 datasets. Pablo wanted to ensure that the most relevant datasets were at the top to be easily discovered by consumers.  The most relevant - both the All — A single-cell transcriptomic atlas characterizes ageing tissues in the mouse datasets are currently listed at the bottom making it harder for consumers to discover.
There is another case that is pending publication:

... the most relevant dataset is the one where data from multiple papers was integrated into a single dataset.The others are either subsets of that dataset or the original datasets from the papers. That integrated dataset is currently in the middle.

May 2022
Request from the Allen Institute to define the precise order of datasets in a collection.

Research from @tihuan on potential FE solutions:

Ooh I just did a quick search, looks like HTML5 has drag and drop API
And this article details how to do it with React!
So maybe we just need a quick POC to make sure it works with an HTML table too, then we’re good?

Closing. This has not been a priority for more than 2 years. This would be expensive to implement due to the increased complexity in the Curation UX, Discover UX, Portal API, and data store to maintain dataset ordering per collection.
I noted a simpler approach above :

One solution could be a sort by column hint set by curators for the collection detail page.

CC: @jahilton  @hthomas-czi

At present, the Portal API returns datasets without an explicitly-determined order; presumably this is being governed by default by the date of creation, or howsoever psql decides to yield the query results.
A couple concerns: the main concern appears to be ordering "by importance", which is obviously subjective (rather than some objective measure like date revised, title, etc.). To me the only reasonable approach for satisfying this feature request is to provide the curators the ability to manually drag/reorder the Datasets in the UI (in the private/revision state, of course -- is a revision that only results in reordering of Datasets truly a revision? Open question...).
We'd either...

Add an explicit positioning index column on table DatasetVersion, or
Express order through the order of their uuids in the CollectionVersion.datasets array.

To me the only reasonable approach for satisfying this feature request is to provide the curators the ability to manually drag/reorder the Datasets in the UI

Yes. That's why I recommended that CC t-shirt size this epic in its entirety.

presumably this is being governed by default by the date of creation, or howsoever psql decides to yield the query results.

The UX sorts datasets in a collection by descending order of cell count.

Yes. That's why I recommended that CC t-shirt size this epic in its entirety.


Adding track # metadata to mp3 downloads by using the playlist_index #

[x ] I've verified and I assure that I'm running youtube-dl 2017.05.29

your issue?

[ ] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[x ] Question
[ ] Other

As the title says, I'd like to download audio from youtube playlists while adding track numbers to each audio file based on its position in said youtube playlist.
I have tried tracking down the correct set of command modifiers and have managed to download perfectly healthy .mp3 files, but i have not found the proper set to add the track # metadata so i don't have to add it manually with audacity.
Thanks in advance.

is this a duplicate of #7393? yes.  has #7393 been dead for a year and remained unresolved?  also yes

@dstftw

Opened pull request #14991 which I believe may support the OP request


Re-write installation READMEs

So that they reflect the fact that most of the morphologica code is now header-only

Pretty sure I've addressed this now.


[vue3.x] Min and Max don't work. No warnings or errors in dev console.

Describe the bug
I can't get the min and max props to change the range of the slider. The slider is using the default min=0, max=100.Here is my template code:
<VueSlider
                ref="sliderRef"
                v-model="sliderValue"
                :process="proc"
                :interval="0.05"
                :min="10"
                :max="30"
                :dot-size="sliderDotSize"
                :height="sliderHeight"
                class="vslider"
                tooltip="none"
                @change="getThresholds"
                @drag-end="getThresholds"
            >

Additional context (If there is no relevant content, please delete the block)

Environment (If you feel unrelated, please delete the block)

Docker image node:20.5.0-bullseye
Vue version: ^3.4.21
Component Version: 4.1.0-beta.7

Closing this issue because min and max are working on a component a few rows higher in my code.


The adjustment of learning rate

Thanks for the nice work. I have a question for you. Your default is 8 cards, and you set the learning rate to 0.02. If I conduct the experiment from multiple cards to single card, does the learning rate need to be adjusted? I have tried to adjust the learning rate to 0.0025 and 0.005 for experiments, but the results are not satisfactory. May I ask if you have any suggestions on the adjustment of learning rate when switching from multiple cards to single cards? Looking forward to your reply.

The learning rate is set to 0.02 for a total batch size of 16 in RefineMask. If you use a batch size smaller/larger than 16, please follow the linear scaling rule to adjust the learning rate (i.e. 16/8/4 - 0.02/0.01/0.005). However, If you use a batch size smaller than 16, the performance will drop in MMDetection.

Thanks for your reply. I have tried to set samples_per-GPU to 2 on R50-Refinemask-1x with a single card and a learning rate of 0.0025 using coco data set. The result is 36.0, which is worse than the result in your paper. If I use coco dataset and set samples_per-GPU to 1 in R50-Refinemask-1x, single card and learning rate to 0.00125. Is that possible?

Thanks for your reply. I have tried to set samples_per-GPU to 2 on R50-Refinemask-1x with a single card and a learning rate of 0.0025 using coco data set. The result is 36.0, which is worse than the result in your paper. If I use coco dataset and set samples_per-GPU to 1 in R50-Refinemask-1x, single card and learning rate to 0.00125. Is that possible?

The critical thing is the total training batch size can not be smaller than 16, and you can use 16x1 or 8x2 to reproduce the results presented in the paper.


[swate-alpha] Add assay metadata sheet

Swate standalone should feature a metadata sheet.
As far as i can see this sheet can have two types of metadata.

The main case, an assay metadata sheet
A template metadata sheet #257

Must templates also contain an assay metadata sheet? I am not sure about this one @Brilator @Martin-Kuhl
If they must not contain an assay metadata sheet, i will handle these as "any of" case. And will not support adding both to one xlsx file in Swate standalon.

From my perspective, templates do not need an assay metadata sheet. I think the current process (adding them via AC) is sufficient.

If we're talking about the isa.investigation-style Assay sheet: should not be added by Swate (templates).

Just to be sure, i am talking about this assay metadata sheet:

Yes. Swate should not add this (as of now).

Swate is also relevant for isa.study (-> sheet "Study")
Few of the information (if any) would be templated.


Enable changing listen port for NextCloud

I am using a nginx reverse proxy as an SSL offloader in front of my containers.
As I already had a container using port 80 I needed the possibility to change NextCloud listen port to another one.

Hi, thanks for your interest. However, I'm not sure why this is needed.
So, if you're using nginx as a reverse proxy, why would you like to change nextcloud listen port ? All you have to do is link the nextcloud container to the nginx container and use proxy_pass http://nextcloud. If you're not using a nginx container, you can use the IP address of the container like this :
docker inspect conteneur | grep IPAddress\" | head -n1 | grep -Eo "[0-9.]+"

In a nutshell, when using a reverse proxy, there's no need to bind the container port to the host. But if you want to do so, Docker allows you, for example, to bind port 80 on the container to whatever port you want on the host.

Thank you for your fast answer.
You are right, I'm new to docker and misunderstood the requirements for the "backend" containers as I believed that I had to expose the port.
It's working now, sorry for disturbing and thanks again for your explanations.
I'll now look on how to switch memcaching to a linked redis container :)


⚠ Bofrost.com has degraded performance

In 15ccdbe, Bofrost.com (https://www.bofrost.com) experienced degraded performance:

HTTP code: 200
Response time: 1265 ms

Resolved: Bofrost.com performance has improved in 5a1073d after 8 minutes.


GOD EATER 2 RAGE BURST [PCSH00133]

App summary

App name: GOD EATER 2 RAGE BURST
App serial: PCSH00133
App version: 1.30

Vita3K summary

Version: v0.1.6
Build number: 3020
Commit hash: https://github.com/vita3k/vita3k/commit/a4ded14b
CPU backend:  Unicorn
GPU backend: Vulkan

Test environment summary

Tested by: Croden1999
OS: Windows 10
CPU: Intel Core i7-7700HQ
GPU: NVIDIA GeForce GTX 1050
RAM: 8 GB

Issues

need use the unicorn cpu backend, dynarmic does not work. only menu, no sound. After creating saved data, it is stuck.
Screenshots

Log
vita3k.log

now ge2rb use dynarmic cpu backend can running on new vesion of Vita3K

Still crashes after character creation


undefined is not iterable... on text hover

Hi
Thanks for this great extension!
I don't know if this happens to anybody more but I have the IDE's log console plenty of the stacktrace seen below.
Extension is capable of fetch my gitsts and edit it.
To Reproduce
Steps to reproduce the behavior:

Open any text file. I'd checked it with a .md (markdown)
Wait for the Intellisense suggestions
Check errors on Output panel > Log (Window) / Log (Extension Hosts)

Expected behavior
No console errors or more context about it to know about solve it
Screenshots
Desktop (please complete the following information):

VS Code:
Version: 1.62.3 (system setup)
Commit: ccbaa2d27e38e5afa3e5c21c1c7bef4657064247
Date: 2021-11-17T08:11:14.551Z
Electron: 13.5.2
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Windows
GistPad Version: v0.3.0

Additional context
[2021-11-27 16:11:38.475] [exthost] [error] [vsls-contrib.gistfs] provider FAILED
[2021-11-27 16:11:38.476] [exthost] [error] undefined is not iterable (cannot read property Symbol(Symbol.iterator)): TypeError: undefined is not iterable (cannot read property Symbol(Symbol.iterator))
    at o.provideHover (\user\.vscode\extensions\vsls-contrib.gistfs-0.3.0\dist\extension.js:51:186784)
    at d.provideHover (\vscode\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:94:170537)
    at \vscode\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:94:194656
    at $e._withAdapter (\vscode\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:94:191442)
    at $e.$provideHover (\vscode\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:94:194634)
    at n._doInvokeHandler (\vscode\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:98:13802)
   ...

I'm not able to repro this issue, so I'm going to close it for now. Please re-open it if you're still experiencing an issue, and could share any further repro steps. Thanks!


Ensure rancher-machine image is added to images.txt

The rancher-machine image was mistakenly left out of the images.txt
file. This change will include it.
Issue:
https://github.com/rancher/rancher/issues/34986

I ran make ci locally before submitting this to ensure the machine image is included in rancher-images.txt.


Using indexers ingestion / Integrated Vectorization will not apply page number

Please provide us with the following information:

This issue is for a: (mark with an x)
- [ ] bug report -> please search issues before submitting
- [x ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

By using Indexers to ingest from storage account, source page will not be added compared to using prepdocs.ps1. Is there a way to add source page with the indexers ingestion of pdf?

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

azd version?

run azd version and copy paste here.

Versions

Mention any other details that might be useful

Thanks! We'll be in touch soon.

cc @srbalakr @mattgotteiner I think I saw this as well- I was noticing my citation filenames were missing page numbers and wondered where they went to.

Also seeing this. It looks to me like it is from the difference between the skillset document chunker.
The integrated vectorization does this where it maps source page to the filename in blob
        index_projections = SearchIndexerIndexProjections(
            selectors=[
                SearchIndexerIndexProjectionSelector(
                    target_index_name=index_name,
                    parent_key_field_name="parent_id",
                    source_context="/document/pages/*",
                    mappings=[
                        InputFieldMappingEntry(name="content", source="/document/pages/*"),
                        InputFieldMappingEntry(name="embedding", source="/document/pages/*/vector"),
                        InputFieldMappingEntry(name="sourcepage", source="/document/metadata_storage_name"),
                    ],
                ),
            ],

The original  (non-integrated) maps the source page to the exact chunk  within the sourcefile:
                        "sourcepage": (
                            BlobManager.blob_image_name_from_file_page(
                                filename=section.content.filename(),
                                page=section.split_page.page_num,
                            )
                            if image_embeddings
                            else BlobManager.sourcepage_from_file_page(
                                filename=section.content.filename(),
                                page=section.split_page.page_num,
                            )
                        ),

In the index, integrated leaves it looking like this:
      "sourcepage": "file.pdf",
      "sourcefile": null,

Whereas the other searchmanager.py leaves it looking like this;
      "sourcepage": "file-4.pdf",
      "sourcefile": "file.pdf"

@jakebowles99 did you make some progress ?

Hi there @pamelafox @mattgotteiner  I'm looking for a solution to this issue. Is there a way to get the chunk page using integrated vectorization? Specifically, I'm trying to ensure that source page numbers are included in the index?
Any guidance or suggestions on how to achieve this with the integrated vectorization approach would be greatly appreciated.

Found these possible solutions, but both feel suboptimal to me

are you planning in implementing it?

I am currently exploring the first suggestion @pamelafox provided and somehow merge it with this project to enrich the pipeline to get the page number. This approach will unfortunately add a lot of overhead and complexity.
Is there more information when the page number will be available to extract with Integrated Vectorization in this solution @pamelafox ?

@DiPbas  @casperdamen123
https://github.com/Azure-Samples/azure-search-openai-demo/issues/1287#issuecomment-2394960531

@DiPbas @casperdamen123 #1287 (comment)
"Each index projection document contains a unique identifying key that the indexer generates in order to ensure uniqueness and allow for change and deletion tracking to work correctly. This key his key contains the following segments: A random hash to guarantee uniqueness. This hash changes if the parent document is updated across indexer runs. The parent document's key. The enrichment annotation path that identifies the context that that document was generated from.
For example, if you split a parent document with key value "123" into four pages, and then each of those pages is projected as its own document via index projections, the key for the third page of text would look something like "01f07abfe7ed_123_pages_2".
If the parent document is then updated to add a fifth page, the new key for the third page might, for example, be "9d800bdacc0e_123_pages_2", since the random hash value changes between indexer runs even though the rest of the projection data didn't change."
Maybe we can use the Microsoft.Skills.Util.ShaperSkill to extract "sourcepage" from "sourcefile" by string pattern "(.pages(\d+)" or using input sourcing from /document/content/pages//number after the splitter?

Thats not true. The "_pages_X" only defines the chunk number, it doesnt specify the actual page number. How I solved it creating a custom skillset which does the chunking (you can use a custom text splitter or simply langchain/llama-index for this), and then also stores the page_num. Then this function returns the chunks and the page_num for each chunks and in this way I can store the actual page_num.

You mean with „it‘s not true“ that the Ms Docs are buggy?
https://learn.microsoft.com/en-us/azure/search/index-projections-concept-intro?tabs=kstore-rest#projected-key-value

I meant that it is not true that you can solve it this way, because what they mean by "pages" here are "chunks" not actual pages of a PDF.


Collapsible section configuration not present in extracted template

I am extracting all site pages using the Get-PnPSiteTemplate command with handlers Pages,PageContents and IncludeAllClientSidePages but the XML doesn't have any reference to section being collapsible or not. It just contains  information.
Currently using the latest version of engine -1.12.0 and the schema being referred is V202209.
I can see many users reporting this issue, can someone please provide a solution.
Thanks.

Same issue here.


Make hello_world__memory run if the build configuration is modified

https://github.com/flutter/flutter/pull/142300#issue-2102312089
Either Configure post submit test Linux_android hello_world__memory to run in presubit if one of the build files is modified OR setup a test where the code is compiled but the memory performance test does not run in presubmit.

Possibly related https://github.com/flutter/cocoon/blob/main/app_dart/lib/src/request_handlers/github/webhook_subscription.dart

In the ci.yaml for the target,
--- presubmit: false
+++ runIf:
+++  - .ci.yaml
+++  - examples/hello_world/android/**


Support for Storage KMS integration

Definitely review one commit at a time. Will ping Chris with more background.
Not sure whether this will be merged as soon as it's reviewed, or whether we need to hold off.

Notes for further work:

Use SkippableFact more widely
Document the environment variables required for testing KMS-related features
Use those environment variables in BigQuery tests as well

(I'll ask someone from Storage to review this as well for semantics and API surface.)

Added another commit after further consultation - @chrisdunelm PTAL.

Heads up: A change is coming that will not allow global KMS keys and tests will need to use regional KMS keys, that match the bucket location.


how to create and add striped disks to the SQL server on a Azure VM?

we have few SQL servers on Azure VM, now the drives are above to full, we need to add new drives with striping, since we have created SQL VM's with striping disks while spinning from market place.
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d508e8ae-bbba-703d-e94e-c42520a4cad8
Version Independent ID: 2a3bcae3-b889-f498-8de8-1481a4bcb604
Content: Azure Premium Storage: Design for high performance - Azure Virtual Machines
Content Source: articles/virtual-machines/premium-storage-performance.md
Service: storage
Sub-service: disks
GitHub Login: @roygara
Microsoft Alias: rogarana

Thanks for the feedback! We are currently investigating and will update you shortly.

Hello, @selectstarfromsql . Thanks for reaching out. GitHub issues are meant only for documentation issues, not product support. Since you've a product support request, I recommend reaching out to support.
Accordingly, I'm closing this out.
#please-close


Implement CCS across node registries

💡 Description

Work with SAs to implement CCS across all DN registries

This works and is deployed.

This was done manually by the SAs (dev team or Jimmie are not allowed to do it)

Closing for now - we should revisit (testing) once more nodes have consumed data


Configure the Blazor WASM runtime

Fixes  #30320
Confirm what I called these in case I'm making up my own 🦖 Blazor framework .... again! 😆

To configure the .NET WebAssembly runtime, use the configureRuntime property with the dotnet runtime host builder.

configureRuntime ... a "property"?
dotnet ... the "runtime host builder"?

@maraf ... I took your test example for the doc coverage. If either you or Pavel have something better for the example, I'm 👂.
Finally, this is used to configure quite a few runtime-y things. I see what looks like an incomplete list at ...
https://github.com/dotnet/aspnetcore/issues/49264#issuecomment-1628483544
What do we want to say about that (or cross-link for it)?

Internal previews

📄 File
🔗 Preview link

aspnetcore/blazor/fundamentals/startup.md
ASP.NET Core Blazor startup

aspnetcore/release-notes/aspnetcore-8.0.md
What's new in ASP.NET Core 8.0

I'm going to go ahead and merge because we can't hold up the release notes topic at this time. We're on a HIGH doc churn day with RC1 🏃🏃🏃🏃😰.
I'll patch any updates into the text/example that you see fit. Please leave your comments here.

@guardrex I think the same is good enough. I should also show the other part, eg
Blazor.runtime.getAssemblyExports(...);

The Blazor.runtime gives access to Runtime API of running .NET https://github.com/dotnet/aspnetcore/blob/cdbb59f6a7ee4cc2a7727a4555c53f4578a89276/src/Components/Web.JS/src/GlobalExports.ts#L42

I added it after this PR on https://github.com/dotnet/AspNetCore.Docs/pull/30339 ...

The .NET runtime instance can be accessed from Blazor.runtime.

Live topic section ...
https://learn.microsoft.com/en-us/aspnet/core/blazor/fundamentals/startup?view=aspnetcore-8.0#configure-the-net-webassembly-runtime


raGOO & Quast

add raGOO (https://github.com/malonge/RaGOO) for the ordering of assembly contigs. Integrate output of raGOO into Pangolin.
add Quast (http://quast.sourceforge.net/quast) for visualisation of config quality

wait for PR #29, which adds polishing with varlociraptor.


bow removed background

I'm happy to see that you're trying out the feat/fix stuff! Though, you really only need to use it for making the Pull Request title. Your commits should be just short & explains what you're pushing!
i.e commits

If I updated styles on about.jsx -> "updated styles for about section"
If I added content to footer for the first time -> "created basic footer"
If I added images or made new files -> "added personal imgs & made (file)"


the documentation does not specify on how to run this in debug mode

I am not sure if the option or running this in firebase debug mode is available.
is there a way to run this on ionic appflow with "-FIRDebugEnabled" argument or is their any fork of this plugin with debug enabled?

Documentation issue

With @RodriSanchez1 we find a way to enable the debug_mode to a Android USB-connected device with the objective of using Analytics debug view.

Connect your developer Android device via USB.
Allow the connection on the device.
Open your terminal and run adb devices -l
if appears run adb shell setprop debug.firebase.analytics.app PACKAGE.NAME
Now your device is in debug Mode.
I have not already tested on IOS but you can find information here

Refer to the Firebase documentation for how to enable debug mode for the Firebase SDK on a test device.
The plugin is unable to help with this on either the iOS or Android platforms therefore it lies out of scope.


wifi-3331: Fix opensync Firmware version date format

Opensync fw version date was getting trimmed.
This patch fixes the same

Signed-off-by: Yashvardhan<EMAIL_ADDRESS>

Merged 8bec57d4, thanks