cilium dns proxy traffic doesn't make it back to the pod

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
We created a new CiliumNetworkPolicy with the following syntax, from the examples in the documentation:
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: test-fqdn-policy
spec:
  endpointSelector:
    matchLabels:
        name: debug
  egress:
  - toEndpoints:
    - matchLabels:
       "k8s:io.kubernetes.pod.namespace": kube-system
       "k8s:k8s-app": kube-dns
    toPorts:
      - ports:
         - port: "53"
           protocol: ANY
        rules:
          dns:
            - matchPattern: "*"
  - toFQDNs:
      - matchName: "google.com"

When this policy is created, we're no longer able to create dns requests at all - dig google.com returns
bash-5.1# dig google.com

; <<>> DiG 9.18.3 <<>> google.com
;; global options: +cmd
;; connection timed out; no servers could be reached

Deleting the policy re-establishes the connectivity.
We see the following logs when we run cilium monitor | grep 4000:d::17c5 (the pod I'm attempting to query from)
Policy verdict log: flow 0x465d4a1f local EP ID 331, remote ID 26603, proto 17, egress, action redirect, match L3-L4, [2001:db8:4000:d::17c5]:53991 -> [2001:db8:4000:1::7e78]:53 udp
-> proxy port 33293 flow 0x465d4a1f , identity 59577->unknown state new ifindex 0 orig-ip <IP_ADDRESS>: [2001:db8:4000:d::17c5]:53991 -> [2001:db8:4000:1::7e78]:53 udp
-> endpoint 331 flow 0xce94595a , identity 26603->59577 state reply ifindex lxc215b2ab5e197 orig-ip 2001:db8:4000:1::7e78: [2001:db8:4001::a]:53 -> [2001:db8:4000:d::17c5]:53991 udp

cilium fqdn cache list
Endpoint   Source   FQDN          TTL    ExpirationTime             IPs
1341       lookup   google.com.   3600   2022-09-20T10:15:02.677Z   <IP_ADDRESS>

Cilium Version
1.12.0
Kernel Version
Linux <> 5.15.67
Kubernetes Version
1.24.2
Sysdump
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Could you share a sysdump? How did you install Cilium? Do you have the same issue over IPv4?

Ah interesting, querying against the v4 service ip seems to get correctly redirected and proxied.
Cilium installed with Helm.
I can send the sysdump privately on slack if needed

Ah interesting, querying against the v4 service ip seems to get correctly redirected and proxied.

Could you share the cilium monitor trace equivalent to the above one for that successful request?

After running dig @(service ip v4 for kubedns):
cilium monitor | grep <IP_ADDRESS>
Listening for events on 96 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
Policy verdict log: flow 0x11f4600b local EP ID 592, remote ID 26603, proto 17, egress, action redirect, match L3-L4, <IP_ADDRESS>:54329 -> <IP_ADDRESS>:53 udp
-> proxy port 36603 flow 0x11f4600b , identity 59577->unknown state new ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:54329 -> <IP_ADDRESS>:53 udp
-> endpoint 592 flow 0x0 , identity 26603->59577 state reply ifindex lxcb91d1f86377c orig-ip <IP_ADDRESS>: <IP_ADDRESS>:53 -> <IP_ADDRESS>:54329 udp

In the IPv6 case, are you able to confirm that 2001:db8:4001::a is the VIP for the DNS service and 2001:db8:4000:1::7e78 is the IP for the backend DNS pod?

Cilium installed with Helm.

Please provide the full command.

In the IPv6 case, are you able to confirm that 2001:db8:4001::a is the VIP for the DNS service and 2001:db8:4000:1::7e78 is the IP for the backend DNS pod?

Yep, that's correct

Cilium installed with Helm.

Please provide the full command.

Apologies for the delay, on a phone.
https://gist.github.com/gpl/2601c0fee05034ff99081f510774f9ce

We enabled CiliumLocalRedirectPolicy to forward dns requests to node-local-dns. We enabled DNS-policy in CNP. Sometimes it takes several seconds to resolve DNS name. At the same time I see drops in cilium monitor: UDP packets from node-local-dns can't reach cilium agent:

That doesn't sound like the same issue. Please open a new issue.

I think I got something similar.
As soon as I add a dns matcher rule or annotate the pod with io.cilium.proxy-visibility="<Egress/53/UDP/DNS>" the dns traffic stops.
I can see the UDP packets leaving the pod, I can't see them entering the coredns pod (tcpdumped within its netns). The monitor says:
-> proxy port 33577 flow 0x0 , identity 103224->unknown state new ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:56661 -> <IP_ADDRESS>:53 udp

I have a pretty much allow-all cnp:
spec:
  egress:
  - toEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
      rules:   # adding or removing this has no effect; as long as the pod annotation is there it's already broken
        dns:
        - matchPattern: '*'
  - toEntities:
    - world
  endpointSelector:
    matchLabels: {}

cilium 1.12 via helm. values.yaml

As soon as I add a dns matcher rule or annotate the pod with io.cilium.proxy-visibility="<Egress/53/UDP/DNS>" the dns traffic stops.

I can see the UDP packets leaving the pod...

@farcaller We are also observing similar behaviour!
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchName: "api.twitter.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"

Before applying the CiliumNetworkPolicy ✅
➜ kubectl exec pod/mediabot -- curl -I -s https://api.twitter.com | head -1
HTTP/1.1 404 Not Found

➜ hubble observe -f --namespace default
Sep 27 16:58:39.732: default/mediabot:39680 (ID:42395) -> kube-system/kube-dns-b8976dfcd-qx8bt:53 (ID:7394) to-stack FORWARDED (UDP)
Sep 27 16:58:39.875: default/mediabot:39680 (ID:42395) -> kube-system/kube-dns-b8976dfcd-qx8bt:53 (ID:7394) to-endpoint FORWARDED (UDP)
Sep 27 16:58:39.886: default/mediabot:39680 (ID:42395) <- kube-system/kube-dns-b8976dfcd-qx8bt:53 (ID:7394) to-stack FORWARDED (UDP)
Sep 27 16:58:39.887: default/mediabot:39680 (ID:42395) <- kube-system/kube-dns-b8976dfcd-qx8bt:53 (ID:7394) to-endpoint FORWARDED (UDP)
Sep 27 16:58:40.017: default/mediabot:46096 (ID:42395) -> <IP_ADDRESS>:80 (world) to-stack FORWARDED (TCP Flags: SYN)
Sep 27 16:58:40.191: default/mediabot:46096 (ID:42395) <- <IP_ADDRESS>:80 (world) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Sep 27 16:58:40.191: default/mediabot:46096 (ID:42395) -> <IP_ADDRESS>:80 (world) to-stack FORWARDED (TCP Flags: ACK)
Sep 27 16:58:40.481: default/mediabot:46096 (ID:42395) -> <IP_ADDRESS>:80 (world) to-stack FORWARDED (TCP Flags: ACK, PSH)
Sep 27 16:58:40.481: default/mediabot:46096 (ID:42395) <- <IP_ADDRESS>:80 (world) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep 27 16:58:40.767: default/mediabot:46096 (ID:42395) -> <IP_ADDRESS>:80 (world) to-stack FORWARDED (TCP Flags: ACK, FIN)
Sep 27 16:58:40.821: default/mediabot:46096 (ID:42395) <- <IP_ADDRESS>:80 (world) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Sep 27 16:58:40.822: default/mediabot:46096 (ID:42395) -> <IP_ADDRESS>:80 (world) to-stack FORWARDED (TCP Flags: ACK)

➜ kubectl exec pod/cilium-qk7ps -n kube-system -- cilium monitor | grep <IP_ADDRESS>
Listening for events on 2 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
-> stack flow 0x0 , identity 42395->7394 state new ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:39680 -> <IP_ADDRESS>:53 udp
-> endpoint 3359 flow 0x0 , identity 7394->42395 state reply ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:53 -> <IP_ADDRESS>:39680 udp
-> stack flow 0xe543d10a , identity 42395->world state new ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:46096 -> <IP_ADDRESS>:80 tcp SYN
-> endpoint 3359 flow 0x0 , identity world->42395 state reply ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:80 -> <IP_ADDRESS>:46096 tcp SYN, ACK
-> stack flow 0xe543d10a , identity 42395->world state established ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:46096 -> <IP_ADDRESS>:80 tcp ACK
-> stack flow 0xe543d10a , identity 42395->world state established ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:46096 -> <IP_ADDRESS>:80 tcp ACK
-> endpoint 3359 flow 0x0 , identity world->42395 state reply ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:80 -> <IP_ADDRESS>:46096 tcp ACK
-> stack flow 0xe543d10a , identity 42395->world state established ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:46096 -> <IP_ADDRESS>:80 tcp ACK, FIN
-> endpoint 3359 flow 0x0 , identity world->42395 state reply ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:80 -> <IP_ADDRESS>:46096 tcp ACK, FIN
-> stack flow 0xe543d10a , identity 42395->world state established ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:46096 -> <IP_ADDRESS>:80 tcp ACK

After applying the CiliumNetworkPolicy ✅
➜ kubectl exec pod/mediabot -- curl -I -s https://api.twitter.com | head -1
command terminated with exit code 6

➜ hubble observe -f --namespace default
Sep 27 17:03:28.858: default/mediabot:55743 (ID:42395) -> kube-system/kube-dns-b8976dfcd-qx8bt:53 (ID:7394) policy-verdict:L3-L4 ALLOWED (UDP)
Sep 27 17:03:28.858: default/mediabot:55743 (ID:42395) -> kube-system/kube-dns-b8976dfcd-qx8bt:53 (ID:7394) to-proxy FORWARDED (UDP)

➜ kubectl exec pod/cilium-qk7ps -n kube-system -- cilium monitor | grep <IP_ADDRESS>
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init)
Listening for events on 2 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
level=info msg="Initializing dissection cache..." subsys=monitor
Policy verdict log: flow 0x0 local EP ID 3359, remote ID 7394, proto 17, egress, action redirect, match L3-L4, <IP_ADDRESS>:55743 -> <IP_ADDRESS>:53 udp
-> proxy port 44795 flow 0x0 , identity 42395->unknown state new ifindex 0 orig-ip <IP_ADDRESS>: <IP_ADDRESS>:55743 -> <IP_ADDRESS>:53 udp

I've also messaged the #kubernetes channel in Cilium & eBPF's Slack: https://cilium.slack.com/archives/C53TG4J4R/p1664289703439699
cilium-sysdump-20220927-153534.zip

@farcaller @waltfy Note the first report was specific to IPv6 (it worked over IPv4) so it doesn't look like you're hitting the same bug. I'd recommend to open new issue(s) for your specific reports if you want visibility.

@pchaigno oh, I somehow didn't make that connection - thank you for calling it out. I will do that!


Artemis: cb: Version commit for 2024.14.01

Description:

Version commit for ColterBay BIC 2024.14.01.

Motivation:

Committing the version to release new bic firmware.

Test Plan:

Build code: Pass
Get firmware version: Pass

Log:
root@bmc-oob:~# fw-util cb --version bic
CB BIC Version: cb2024.14.01

Rebase main branch


Deprecation Warning

/Users/kato/.local/share/virtualenvs/interleaving-M68ZqYhg/lib/python3.6/site-packages/pulp/solvers.py:71: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.
  'os':operating_system, 'arch':arch})

/Users/kato/dev/interleaving/interleaving/probabilistic.py:215: DeprecationWarning: `logsumexp` is deprecated!
Importing `logsumexp` from scipy.misc is deprecated in scipy 1.0.0. Use `scipy.special.logsumexp` instead.
  p_all = np.exp(p_log_all - misc.logsumexp(p_log_all))

There are two messages about deprecation.

It seems that the first warning is not ours but pulp's problem.

I also have another pulp problem on Python 3.7.
c:\users\owner\appdata\local\programs\python\python37\lib\site-packages\pulp\pulp.py:1665: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead
  self.solutionTime += clock()

The possibly happiest way is to use scipy instead of pulp...


Request: Infobar Plugin Support For The Latest Deadbeef?

Hello,
Would you be able to correct the source code in the Infobar Plugin 1.4 to work with the latest versions of Deadbeef?
The reason I am asking you, since you have done coding in Deadbeef for this project, I'm hoping you can do this, if you have the free time.
I posted an issue about this on Deadbeef's Github repo, but unfortunately no one has come forward to update this, which is really a shame.
https://github.com/DeaDBeeF-Player/deadbeef/issues/2206
If you are familiar with the plugin, the reason I want to use it, it also has Biography support besides Lyrics.
You can only use deadbeef-0.7.2 with the Infobar Plugin.
I'm attaching the original source for the Infobar plugin.
Thank you for your time and consideration, please let me know. :)
deadbeef-infobar_1.4.orig.tar.gz

Yes that is a shame, but I am unfortunately time-limited at the moment.

No worries deadbeef-infobar-ng has had some recent updates, so it's working at present.
https://github.com/loskutov/deadbeef-infobar-ng/issues
THANKS


What llvm versions can be used?

In spack we have the llvm-amdgpu package which takes it sources from https://github.com/RadeonOpenCompute/llvm-project/archive/rocm-4.2.0.tar.gz (and all other tags: rocm-3.5.0 to rocm-4.2.0)
This project (aomp) seems to pull in https://github.com/ROCm-Developer-Tools/llvm-project
Can the spack package build aomp on top of https://github.com/RadeonOpenCompute/llvm-project so that we can avoid having to build multiple versions of llvm?

The AOMP released on Git Hub is a standalone experimental compiler and the spack package for aomp relates to when openmp support for offloading was first integrated into ROCm. Initially, aomp was based on LLVM 11 while the llvm-amdgpu was ahead on LLVM 12. This was the reasoning for the aomp spack package building it's own LLVM from https://github.com/ROCm-Developer-Tools/amd-llvm-project. I am considering renaming aomp spack package to aomp-amdgpu to further differentiate.
The rocm-openmp-extras package was used to differentiate the nomenclature of AOMP (GH standalone) vs openmp support in ROCm and to get rid of the separate build of an additional LLVM inside of ROCm. rocm-openmp-extras uses the llvm-amdgpu (RadeonOpenCompute) shipped with ROCm.  The aomp spack package is no longer being updated as it has been deprecated after 3.10.0. AOMP GH dev work does eventually flow into ROCm and now uses a similar version of LLVM as llvm-amdgpu, but with additional commits that may not be in llvm-amdgpu yet.
All that being said, the rocm-openmp-extras spack package uses llvm-amdgpu and not GCC to build it's components. Hence, why llvm-amdgpu is a dependency and built after llvm-amdgpu is present. Using resource to get llvm-amdgpu source code is necessary to build openmp as it contains openmp source code. rocm-openmp-extras builds llvm-amdgpu without the openmp variant as we do not want the associated omp headers and libraries as they will be built in rocm-openmp-extras.
In my opinion, the answer to your question in regards to the aomp spack package is no as the llvm is completely different (11 vs 12).


[AutoPR servicefabric/data-plane] fix: servicefabric character encoding

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/6687

This PR has been merged into https://github.com/Azure/azure-sdk-for-go/pull/5431


Apply scripting language (Groovy) for implementing specific simulation algorithms.

First algorithm shall be implemented in pure Java 8, tested, possibly feature simplified user interface, and only then converted to Groovy. This is more simple way of implementing core contract.


feat(Topology/Compact): an infinite set has an accumulation point

Add more versions of this statement. Also remove simp from Filter.disjoint_cofinite_{left,right} as RHS is not much simpler than LHS.

Scrapped from an old branch. I don't need these lemmas for anything specific.

bors r+


Skip adding the where clause to avoid triggering overflow

Problem
#[derive(BorshSerialize, BorshDeserialize)]
enum A {
    B,
    C {
        d: Vec<A>,
    },
}

gives overflow evaluating the requirement A: BorshDeserialize or BorshSerialize
Idea
Allow skipping adding to the where clause with an attribute to allow edge cases to function.
I am told this could help https://github.com/rust-lang/rust/issues/48214 but there is currently no way to achieve the same effect "working for recursive data structure" by determining automatically which where clause to add in the proc_macro
Not complete yet as this attribute only has an effect for an enum
Related issues
https://github.com/near/borsh-rs/issues/7

Please, add tests

Sorry, i did have the file locally but forgot to add it.
Please note that I am waiting for your agreement on the broad feature before I'll add it to other types.

@Arrowana The reason I asked for the test is to see how this is expected to be used.
Reading https://github.com/near/borsh-rs/issues/7#issuecomment-765582816:

serde, can handle this case, btw

I wonder how they achieved that and whether it is possible to do here, so we don't need to have this attribute.

@Arrowana The reason I asked for the test is to see how this is expected to be used.
Reading #7 (comment):

serde, can handle this case, btw

I wonder how they achieved that and whether it is possible to do here, so we don't need to have this attribute.

Looked at this user repositories i found this https://github.com/near/borsh-rs/pull/96#issuecomment-1196652786
MachineValue is recursive
This is also my current workaround, manual trait implementation, just seems to be a lot of redo.

I would much prefer a solution that doesn't require the attribute but i just don't know how or if it is feasible at all.

It is possible, serde handles it. It's definitely more of an extensive process which I had started previously but somewhat abandoned since I didn't have the bandwidth.

lgtm - lets commit

I think this PR is stalled and is not required anymore, so I am closing it.


AES-NI

Hi Quentin,
I'm using this container for a while now on an ARM device as you probably know.
For me it's the best PIA container available.
Now I'm testing the container on an different device (HP Thin Client T620 with AES-NI).
Is it possible for the PIA container to use AES-NI?
kr.,
Frepke

I'm not sure if that can be configured on the client side. Looking at the Openvpn configuration page I cannot find any setting related to AES-NI. It might as well be used automatically as these are CPU instructions set if I'm not mistaken.
Why do you think it's not used?

I'm not sure if that can be configured on the client side. Looking at the Openvpn configuration page I cannot find any setting related to AES-NI. It might as well be used automatically as these are CPU instructions set if I'm not mistaken.
Why do you think it's not used?
EDIT: It seems to be a kernel module, the only discussion related to it with containers is here without an answer 😢

Okay thanks,
The doubt was because I expected a bit higher download speed in sabnzbd when the AES-NI instructions were used.
kr.,
Frepke
p.s.
I gonna try a different setup for the T620 without docker to see what the results are.

Maybe try this; completely unsafe if someone hacks your container, it accesses your whole host; but that might work!

Maybe try this; completely unsafe if someone hacks your container, it accesses your whole host; but that might work!

Thanks,
Looks promising.
Try it tonight and report it back.
Kr.,
Frepke

Tried the normal docker setup, afterwards the suggested way as you mentioned before and afterwards the normal installation without docker.
Download speed with sabnzbd in all the cases stuck at approx. 10MB/s.
Maybe this is the max. for the T620 :)

Yes I have a 160MByte/s download and upload internet link but PIA servers usually reaches a maximum of 10MByte/s for me as well. It's probably limited on their side I guess. AES-NI would probably be useful if you do LAN transfers on Gbit networks.

Yes I have a 160MByte/s download and upload internet link but PIA servers usually reaches a maximum of 10MByte/s for me as well. It's probably limited on their side I guess. AES-NI would probably be useful if you do LAN transfers on Gbit networks.

I think you're right. I've to be more patient when downloading ;)

I'll close the issue for now. Ahh 10MB/s is still better than the 5KB/s I would get years ago :smile:

Okay,
Thanks for your awesome container!!!


Feature: Upgrade to Cypress 12

@Haviles04 sent this over, https://docs.cypress.io/guides/references/migration-guide.
Issue number
Relevant issue number: Resolves #
Please check the following

[ ] Do the tests still pass? (see Run the Tests)
[ ] Is the code formatted properly? (see Linting (Formatting))
For New Features:

[ ] Have tests been added to cover any new features or fixes?
[ ] Has the documentation been updated accordingly?

Please describe additional details for testing this change

@itsalaidbacklife same error but we're on 12. I say upgrade and we'll continue working on debugging this issue.
Error: Error subscribing opponent to game 57: Error: Server responded with a 403 status code:

Sick, glad this one is finally getting finished


Gunicorn plumbing

This PR builds the plumbing to run QuantumLeap as a WSGI app in Gunicorn---details below.
WSGI
The building of the QuantumLeap WSGI callable object got factored out to a separate module (server.wsgi) so to have a clean WSGI interface provider.  It's a singleton object exposed through the application variable which you can use to run QuantumLeap in your WSGI container of choice---Gunicorn, uWSGI, etc. Here's an example of running the QuantumLeap WSGI app in Gunicorn:
$ cd ngsi-timeseries-api/src
$ gunicorn server.wsgi --config server/gconfig.py

The above Gunicorn config file contains settings that should be okay in most situations (more about it in the next section) but YMMV.
Also part of the server.wsgi module is a function to run the bare-bones QuantumLeap WSGI app in the Flask dev server just like we've done up to now. This function comes in handy for debugging and provides a safety net if we realise the QuantumLeap app is unstable when run in Gunicorn and we quickly have to switch back to the old Flask dev server setup---see USE_FLASK env var in the next section. Also existing QuantumLeap users can leverage it for for backward-compatibility ---I doubt this is a use case though as I can't think of any good reason for our users not wanting to upgrade to a proper WSGI container, but someone may have a good use case for sticking with Flask.
Gunicorn support
While server.wsgi gives QuantumLeap users the freedom to choose any WSGI runtime, we also provide fully-fledged support for Gunicorn in code. The rationale for that is:

we decided to use Gunicorn in production in place of uWSGI
the plumbing required to run QuantumLeap decently in Gunicorn is quite involved, error prone, and takes lots of experimentation
ditto for the ability to easily debug in your IDE/editor with the same software stack you'd use in prod

The last point is critical when running QuantumLeap in Gunicorn+gevent. In fact, while you could debug in the Flask dev server, that'd be miles away from what actually happens in Gunicorn since gevent/greenlet monkey patches Python, Gunicorn spawns several processes, etc. so it's a dramatically different runtime!
So we've got our own drop-in replacement for the Gunicorn server runner which starts QuantumLeap in Gunicorn just like you'd do yourself with a suitable command line (see example in the previous section) but is more flexible (w/r/t configuration) and can be easily debugged in your IDE/editor. In fact, the server.grunner module comes with a run function to start a fully-fledged Gunicorn server to run the QuantumLeap WSGI app. This is the function that QuantumLeap's entry point (app.py) calls now. As a result of that, now invoking the Python interpreter on app.py triggers the following sequence of actions:

The Gunicorn master process gets booted.
Our server.grunner reads in the Gunicorn config settings in server.gconfig.
Then it reads in the config file specified through the --config CLI option if given. If this file contains any setting with the same name as in server.gconfig, they'll override those in server.gconfig.
Finally it processes any other CLI arguments as specified by the Gunicorn docs. In particular any config settings given on the CLI will override previous settings with the same name in (2) and (3).
server.grunner tells the master to load the QuantumLeap WSGI app. (This can't be configured, any other WSGI callable specified in (3) or (4) will be ignored.)
The master completes the startup procedure by forking worker processes.

If you pass no CLI args as in
$ python app.py

then this is the same as running
$ gunicorn server.wsgi --config server/gconfig.py

Here's an example of overriding the default bind address in server/gconfig.py with a CLI option to make QuantumLeap listen to port 8080 instead of 8668 (default)
$ python app.py -b '<IP_ADDRESS>:8080'

You'll find an example of using a custom config file in the section below about the new Docker image. Before moving on, we should mention there's an escape hatch. If you don't want Gunicorn and you'd rather run QuantumLeap classic (i.e. exactly the way it was before this PR, with the Flask dev server) then all you need to do is set the USE_FLASK environment variable to one of the following values (case won't matter): true, yes, 1, t, y.
Gunicorn parallelism and concurrency
QuantumLeap does alot of network IO, so we configure worker processes to use multi-threading (gthread) to improve performance---see server.gconfig module. With this setting, each request gets handled in its own thread taken from a thread pool. In our tests, the gthread worker type had better throughput and latency than gevent but gevent used up less memory, most likely because of the difference in actual OS threads. So for now we go with gthread and a low number of threads. This has the advantage of better performance, reasonable memory consumption, and keeps us from accidentally falling into the gevent monkey patching rabbit hole. Also notice that according to Gunicorn docs, when using gevent, Psycopg (Timescale driver) needs psycogreen properly configured to take full advantage of async IO. (Not sure what to do for the Crate driver!)
But here's the surprise. In our tests, w/r/t to throughput gthread outperformed gevent---27% better. Latency was pretty much the same though. But the funny thing is that we used exactly the same number of worker processes and the default number of threads per process, which is, wait for it, 1. Yes, 1. We did some initial quick & dirty benchmarking to get these results. We'll likely have to measure better and also understand better the way the various Gunicorn worker types actually work. (Pun intended.)
Docker image
The Docker image got slightly updated too. It still calls python app.py but now it does that as an entry point rather than a command. This makes it easier to specify CLI args to docker run or in k8s/docker compose YAML. In particular, it gives you a convenient way to re-configure Gunicorn by mounting your config file on the container and then running the container with the following option
--config /path/to/where/you/mounted/your/gunicorn.conf.py

as in the below example
 $ echo 'workers = 2' > gunicorn.conf.py
 $ docker run -it --rm \
              -p 8668:8668 \
              -v $(pwd)/gunicorn.conf.py:/gunicorn.conf.py
              smartsdk/quantumleap --config /gunicorn.conf.py

@chicco785 the build is broken because of an OSM issue unrelated to this PR's change set---I created a separate GitHub issue for that, #358. Should we go ahead and merge or is there something else you'd like to do?

I would go ahead, but there is not even ci result to look at in my gui :/

I would go ahead, but there is not even ci result to look at in my gui :/

Yea, for some weird reason GitHub isn't displaying the link. No idea why, but here;s Travis's log:

https://travis-ci.org/github/smartsdk/ngsi-timeseries-api/builds/725742784

@c0c0n3 let's disable the failing geocoding (to have clean tests) and let's merge

disable the failing geocoding (to have clean tests)

agreed :-) just pushed the change and mentioned in #358 that we should re-enable it as soon as we have a fix.


Use a keyword argument list for options here

Basedon @aaronrenner's suggestion on https://github.com/keathley/wallaby/pull/313
@aaronrenner I'm not sure how to update the typespec - feel like helping?

@aaronrenner See update - thanks for your help!


Update version detection for GraalVM 23.0+

An update so as to detect version strings like these as well:
native-image 17.0.6 2023-01-17
OpenJDK Runtime Environment Mandrel-23.1.0-dev (build 17.0.6+10)
OpenJDK 64-Bit Server VM Mandrel-23.1.0-dev (build 17.0.6+10, mixed mode)

Version tests are failing (see #145), I am also wondering whether it would be worth bringing in the unit tests from Quarkus as well.

 Sigh.
@@ -247,7 +247,7 @@ public class UsedVersion {
             }
             LOGGER.infof("The test suite runs with Mandrel version %s %s, JDK %d.%d.%d%s.",
                     mandrelVersion.version.toString(), inContainer ? "in container" : "installed locally on PATH", mandrelVersion.jdkFeature, mandrelVersion.jdkInterim, mandrelVersion.jdkUpdate, mandrelVersion.betaBits);
-            return UNKNOWN_VERSION;
+            return mandrelVersion;
         }

Fixed now.
I'm not sure we need the unit tests from Quarkus here. I've just pushed a commit with a basic test for the 3-line version, though.

The CI doesn't seem happy with the new unit test :/

The CI doesn't seem happy with the new unit test :/

Yes, sigh. Looking into it.

@zakkak Please take another look when you get a chance. Thanks!

This debuginfo related failure seems not related to this patch:
Error:  Failures:
Error:    DebugSymbolsTest.debugSymbolsQuarkus:203


:GoRun report "go run: no go files listed"

Hi,
:GoRun doesn't work.  it reports "go run: no go files listed".
but  ":!go run %"  ,  ":GoRun %"  works well.
envionment: Centos7 , vim

Please open this issue to https://github.com/fatih/vim-go. This repo is only about the tutorial. Thanks.


Invalid "Column is ambiguous error"

Version
1.12.0
What happened?
SELECT
    "UserRecords"."Username"
FROM
    "Monitor"
    JOIN ( SELECT DISTINCT ON ("User")
            *
        FROM
            "UserRecords"
        ORDER BY
            "User",
            "CreatedAt" DESC) "UserRecords" ON "Monitor"."ID" = "UserRecords"."User"
WHERE
    "Monitor"."Monitor" = TRUE;

Gives column is ambiguous. This happens because I have a table called "UserRecords", as well as called the JOIN table "UserRecords". However the SQL is not actually ambiguous and executes fine, sqlc confuses the two when there is no reason to. Renaming the JOIN table mitigates it for now.
Relevant log output
No response
Database schema
No response
SQL queries
No response
Configuration
No response
Playground URL
No response
What operating system are you using?
No response
What database engines are you using?
No response
What type of code are you generating?
No response

Another issue for the same query, the star expansion does not seem to work here properly unfortunately. The following query gets generated:
SELECT
    "Monitor"."ID",
    "UserRecordsU"."Username"
FROM
    "Monitor"
    JOIN ( SELECT DISTINCT ON ("User")
            ID, "User", CreatedAt, Username, FullName, Biography, FollowingCount, FollowerCount, IsBusiness, IsPrivate, IsVerified, HasVideos, Category, ClipsCount, IsPotentialBusiness, HasHighlightReels, MediaCount, HasGuides, GeoMediaCount, CTAEnabled, FollowingTagCount, BiographyEntities, BiographyProductMentions, ExternalUrl, IGTVVideos, HasIGTVSeries, ProfilePicture, AREffects, UsertagsCount, ShowShoppableFeed, ShoppablePosts, CanBeReported, MerchantChecoutStyle, SellerShoppableFeedType, VisibleProducts, HasAffiliateShop, CreatorShoppingInfo, IsEligibleForSMBSupportFlow, SMBSupportPartner, DirectMessaging, AddressStreet, BusinessContactMethod, CityID, CityName, ContactPhoneNumber, Latitude, Longitude, PublicEmail, PublicPhoneCountryCode, PublicPhoneNumber, Zip, SuggestedBusinessAccType, AccountType, FundraiserInfo, AccountBadges, IncludeDirectBlacklistStatus
        FROM
            "UserRecords"
        ORDER BY
            "User",
            "CreatedAt" DESC) "UserRecordsU" ON "Monitor"."ID" = "UserRecordsU"."User"
WHERE
    "Monitor"."Monitor" = TRUE

And here only the User column gets quoted, where all columns should be quoted. Normally that works perfectly, but for this one query it seems to break.

@Davincible Could you please provide the schema for the Monitor and User tables?


Resource for PowerShellGet

It would be really nice to have an included directive in the "dsc_resource" resource to have it pull down modules from a URL and extract them to the right locations.  As it is, I've been doing these steps previous to running my dsc_resource work, but it would be much easier if it was just a directive inside of dsc_resource.

needs triage from @chef/client-windows for milestone and priority

This should not be built into dsc_resource, but we could use a powershellget resource that wraps the PowerShellGet module.  There is one in progress for the powershell cookbook which should be near release now that PowerShell 5 has re-RTM'd.

We're working on powershell package support in https://github.com/chef/chef/pull/5315.

https://github.com/chef/chef/pull/5315 has been merged and is included in 12.17.5+ in current.


storageconsumer: define api for storageconsumer and code for controller

As part of the PR below tasks are done
Define API for StorageConsumer
Add code for controller

Signed-off-by: kesavan<EMAIL_ADDRESS>

suggestion to split the commits. One for API and one for the controller.

@nb-ohad @sp98 @iamniting PTAL

The whole PR looks good to me, I would suggest that we split the first commit into 2 commits one for API changes and 1 for the generated code.

/retest

@jarrpa Please take a look

/retest

/retest

/retest


Demo: enable single file build

Because currently the demo app has tons of tiny little files https://scottplot.net/demo/

Giving up for now...

Publish has encountered an error.
Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details.

A diagnostic log has been written to the following location:
"C:\Users\scott\AppData\Local\Temp\tmp3261.tmp"

4/27/2024 3:44:39 PM
System.AggregateException: One or more errors occurred. ---> Microsoft.WebTools.Shared.Exceptions.WebToolsException: Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details.
   --- End of inner exception stack trace ---
---> (Inner Exception #0) Microsoft.WebTools.Shared.Exceptions.WebToolsException: Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details. <---

Microsoft.WebTools.Shared.Exceptions.WebToolsException: Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details.


fmt Bazel support

I bazelized fmt -> https://vertexwahn.de/2022/09/27/bazelizingfmtlib/
fmt is now also part of this bazel-central-registry. Maybe it makes sense to make use of  https://github.com/fmtlib/fmt/blob/master/support/bazel/BUILD.bazel and not creating an own build file via a patch.

/cc @mihaimaruseac

To get Bazel working with {fmt} you can copy the files BUILD.bazel, WORKSPACE.bazel, .bazelrc, and .bazelversion from this folder (support/bazel) to the root folder of this project.

Is it possible to move those files to the root directory in the upstream repo? Otherwise, we still need a patch in the BCR.

{fmt} maintainers do not want to have those files in the root dir... unfortunately...  (@vitaut)

it feels to me like an OSS project is only going to be willing to spend effort on the build system that they use themselves (e.g. adding some tests that prevent them from making changes that break the Bazel build)
so unless they are actually considering switching to Bazel the answer is to follow the typical pattern here and maintain the BUILD file independenty from the project.

I don't think putting build configs for anything other than the main build system (which is CMake for {fmt}) in the top-level directory is a good idea. If limitations of the build system don't allow putting configs in a subdirectory then I suggest creating a separate repo and having {fmt} a submodule.

I don't think this issue should have module wanted on it - fmt already exists in the registry; this issue is about upstreaming the bazel build files to the fmt project.

Ah, well I'm not even sure it makes sense to have issues on this repo which aren't resolvable by making changes to this repo. I think we ought to close it and you could have an issue on the fmt repo about accepting contributions, WDYT?

I closed this issue - just wanted to point out that I maintain https://github.com/fmtlib/fmt/blob/master/support/bazel/BUILD.bazel - if there are any changes in {fmt} I will try to adapt it


Failed to resolve: com.afollestad.material-dialogs:core:<IP_ADDRESS>

I get this gradle error while trying to build. Please advise?

See the other 3 or 4 issues that this is a duplicate of.

Really sorry to start the issue. I added the maven dependency and that resolved the issue. Thanks again for creating a great library!

The library is now on jCenter, as an alternative to JitPack.io. jCenter is Android Studio's default Maven repository.
See the README for updated dependency info: https://github.com/afollestad/material-dialogs#gradle-dependency


failed compilation of KEY-TEST on ECL

Hello,
I ran into this problem with ecl 16.1.3 and the 2018-04-30 dist of quicklisp while loading
serapeum:
Linux violet > ecl
;;; Loading "/home/psilord/quicklisp/setup.lisp"
;;; Loading #P"/usr/local/lib/ecl-16.1.3/asdf.fas"
ECL (Embeddable Common-Lisp) 16.1.3 (git:UNKNOWN)
Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
Copyright (C) 1993 Giuseppe Attardi
Copyright (C) 2000 Juan J. Garcia-Ripoll
Copyright (C) 2016 Daniel Kochmanski
ECL is free software, and you are welcome to redistribute it
under certain conditions; see file 'Copyright' for details.
Type :h for Help.
Top level in: #<process TOP-LEVEL>.
> (ql:quickload :serapeum)
To load "serapeum":
  Load 1 ASDF system:
    serapeum
; Loading "serapeum"
;;; Warning: Class TOPMOST-OBJECT-CLASS has been forward referenced.
;;;
;;; Compiling /home/psilord/quicklisp/dists/quicklisp/software/serapeum-20180430-git/sequences.lisp.
;;; OPTIMIZE levels: Safety=3, Space=0, Speed=1, Debug=3
;;;
;;; Compiling (DEFSUBST SEQUENCE? ...).
;;; Compiling (DEFUN CANONICALIZE-KEY ...).
;;; Compiling (DEFUN CANONICALIZE-TEST ...).
;;; Compiling (DEFUN KEY-TEST ...).
.
;;; Error:
;;;   in file sequences.lisp, position 1487
;;;   at (DEFUN KEY-TEST ...)
;;;   * The macro form (FBIND KEY (WITH-TEST-FN (TEST) (LAMBDA (X Y) (TEST (KEY X) (KEY Y))))) was not expanded successfully.
;;; Error detected:
;;; Detected access to an invalid or protected memory address.
Condition of type: COMPILE-FILE-ERROR
COMPILE-FILE-ERROR while compiling #<cl-source-file "serapeum" "level1" "sequences">

Available restarts:

1. (RETRY) Retry compiling #<cl-source-file "serapeum" "level1" "sequences">.
2. (ACCEPT) Continue, treating compiling #<cl-source-file "serapeum" "level1" "sequences"> as having been successful.
3. (RETRY) Retry ASDF operation.
4. (CLEAR-CONFIGURATION-AND-RETRY) Retry ASDF operation after resetting the configuration.
5. (ABORT) Give up on "serapeum"
6. (RESTART-TOPLEVEL) Go back to Top-Level REPL.

Broken at SI:BYTECODES. [Evaluation of: (QUICKLISP-CLIENT:QUICKLOAD :SERAPEUM)] In: #<process TOP-LEVEL>.

Thank you!

I haven't been able to get ECL working at all for a while -- it goes into a stack overflow while loading ASDF -- but I have a general idea where the problem is coming from, and I've pushed a tentative fix to the ecl-fix branch. If you could test that, I'll merge it into master.

It seemed to work! Thank you!


Update part3.md

Proposed changes

Unreleased project version (optional)

Related issues (optional)

Sorry, I made a mistake. This should be "five" instead of "5". The rule described in the documentation is "Write out numbers in body text and titles from one to ten. From 11 on, use numerals." I confused them.
Can I change it back?

You'll need to make a new PR. Sorry!


Support for Other doc types

Hi: Is there a plan to support other types? e.g. PDF (with images), JSON, PPTX? etc If we need to enable, what's the best way?

Hey @bazooka720 ! Thanks for submitting this request.  Yes there is - we're working on putting together a more visible roadmap w/ these changes and just FYI we'll be targeting Monday ~6AM package releases (starting tomorrow) going forward for a more predictable cadence.
All three that you mentioned were on our internal roadmap; I'll make separate issue tickets for 1) JSON, 2) PPTX and 3) PDF w/ images here to track publicly ;  we'll have the team do a little discovery and see if we can fit in these changes by tomorrow! Stay tuned

@bazooka720 just created this ticket for PDF parsing: https://github.com/DataFog/datafog-python/issues/18 and please feel free to comment there too.  do you have any example PDF types (w/ images) that might be good to keep in mind? i.e. medical record images vs textbooks .

Will do! Thanks

Thanks for the suggestion - let me give them another look.  They made some major changes end of last year (moving to API, requiring keys) so I held off then but will add this as an investigation item


Add private Nomad client option

This adds an option to have Nomad clients deployed to a private subnet
with no public IPs assigned to them. The limitations this entails have
been documented in the terraform.tfvars.template file.
Furthermore, some typos were fixed and the k8s cluster node count
amended to satisfy increased preflight check demands
Checks conducted:

Nomad clients don't have public IPs assigned to them
Jobs can still be run successfully, even if they need to pull images etc

How does the UI respond when "rerun job with ssh" is broken. I wonder what it would take to hide that option in the UI when it will be broken anyway

The job fails with a Circle Bug. To hide the option in the UI would require the frontend to be aware that the Nomad client is deployed into the private subnet and I think this is the part that would cause most of the work. Actually hiding the element in the frontend is trivial.

How does the UI respond when "rerun job with ssh" is broken. I wonder what it would take to hide that option in the UI when it will be broken anyway

The job fails with a Circle Bug. To hide the option in the UI would require the frontend to be aware that the Nomad client is deployed into the private subnet and I think this is the part that would cause most of the work. Actually hiding the element in the frontend is trivial.

I had in mind, exposing a toggle via env vars that would show/hide "rerun job with ssh".  Then we might have some flex-ability to include it as a user exposed config option when installing the app.
This is a non-default option, so I don't think we need to block this PR on a UI fix being merged, but I do think it should be blocked on deciding on a UI fix and getting it on someones roadmap (probably X-Team).

I had in mind, exposing a toggle via env vars that would show/hide "rerun job with ssh".  Then we might have some flex-ability to include it as a user exposed config option when installing the app.
This is a non-default option, so I don't think we need to block this PR on a UI fix being merged, but I do think it should be blocked on deciding on a UI fix and getting it on someones roadmap (probably X-Team).

I had in mind, exposing a toggle via env vars that would show/hide "rerun job with ssh". Then we might have some flex-ability to include it as a user exposed config option when installing the app.
This is a non-default option, so I don't think we need to block this PR on a UI fix being merged, but I do think it should be blocked on deciding on a UI fix and getting it on someones roadmap (probably X-Team).

We cannot set the env var in the terraform, though, so this would be another kots config option, I suppose?

I had in mind, exposing a toggle via env vars that would show/hide "rerun job with ssh". Then we might have some flex-ability to include it as a user exposed config option when installing the app.
This is a non-default option, so I don't think we need to block this PR on a UI fix being merged, but I do think it should be blocked on deciding on a UI fix and getting it on someones roadmap (probably X-Team).

We cannot set the env var in the terraform, though, so this would be another kots config option, I suppose?

Yes, It would have to be on the application side.  I clarified my comment

Yes, It would have to be on the application side.  I clarified my comment

Converting this to a draft until I have verified the handling of the "rerun with SSH" functionality in Server 2.x

Converting this to a draft until I have verified the handling of the "rerun with SSH" functionality in Server 2.x

Rerun with SSH working now using the private IP address

Rerun with SSH working now using the private IP address


Build improvements

This PR does a few improvements around the building:

Use gulp-tsb. It's simply a better approach at using TypeScript with gulp. We use it to develop VSCode.
Add tsconfig.json. This serves both as a configuration for the compiler used in gulp, as well as a configuration for VS Code (and other tools), when editing the project. This enables dropping the nasty /// <reference... snippets.
Simplify the gulp tasks. Instead of having multiple, interdependent tasks, just have one that does the build all at once. Gulp streams here help us out in a very nice way.
Fix npm link. Currently it is not possible to run npm link because package.json points to ./tfx-cli.js, which never exists at development time, since package.json is copied to the _build directory as part of the build step.
Add .npmignore. So we can take away all the cruft from the node module, once it's published to npm.
Add prepublish step to package.json. So gulp build gets run everything someone runs npm link or npm publish.

Thanks again, João.


test(js): use local version in assertion

Summary

I think I needed this before lerna ran, but in either case, I think it makes more sense to assert the lcoal version is used than the one from @algolia/autocomplete-core
Result

no dependencies are used within the test of autocomplete.js

That makes sense—it was done like that because autocomplete-core was the only package exporting version at first.


Feature Request: Update maxbits 31 -> 32 in src/etc/rc.initial.setlanip

Hello
Could it be possible to update the maxbits of IPv4 from 31 to 32 in src/etc/rc.initial.setlanip (see below) ?
function console_configure_ip_address($version) {
...
$maxbits = ($version === 6) ? 127 : 31;
...

Hint Maybe to update for IPv6 but not tested so it's up to you :-)
I've already tried it on command line via sed (see below) and it works since (IPv4) far gateways are already coded and ready for use:
sed -i.dist -e '/maxbits *=/ { s/31/32/; }' /usr/local/etc/rc.initial.setlanip
Thanks in advance.
Kind Regards,

@sazb thanks, looks reasonable. also fixes for IPv6. you can try the patch, but it's also likely in 18.1.5 soon:
# opnsense-patch ef45114


Configuration via YAML

The app should have the option of using a YAML configuration file. This file would be used to enable/disable tabs and features, as well as set the specific configuration of each tab. For example, the number of part search results to auto-expand can be set.
This may require a rework of how each tab is registered with the App, which is something that should be looked at anyway.

Config ideas:

check-in items confirmation dialog : bool = True. Enable or disable the pop-up confirmation dialog for the check-in items screen.
error popups : bool =True. Enable or disable the pop-ups caused by errors, instead relying on the status text to inform the user of an error.

working yaml config in latest commit


SPARQLGraph - maximum stack size exceeded

Whilst attempting to construct a query, SPARQLGraph produced the following error message:
Error: RangeError: Maximum call stack size exceeded.
url: http://rack.office.adelard.com:8080/sparqlGraph/js/belmont.js
line: 2872
undefined
column: 31
error: RangeError: Maximum call stack size exceeded.
To reproduce, drag a REQUIREMENT object onto the canvas, click on the governs attribute, and in response to the pop-up offering a choice between a new PROV-S#ENTITY and a REQUIREMENT, select REQUIREMENT.
The only way to recover from this error is to Clear the query and start again.

Interesting! I can add a bit more of my observation; hopefully @cuddihyge can help.
Everything functions correctly if you chose governs, then select new PROV-S#ENTITY. If you chose REQUIREMENT instead, then I get the "too much recursion" error (which I expect; note: not the same error that Robert reported). But when I then try to select PROV-S#ENTITY instead, I still get the "too much recursion" error.
Here's where it gets even more interesting: after I cancel out of that dialog, I "x" the REQUIREMENT object, SPARQLgraph replaces the REQUIREMENT with an ENTITY.

I think I see what's happening.   SemTK does not like circular graphs.
It looks like it is allowing you to create an edge to/from the same REQUIREMENT, and then blowing up when it tries to generate the SPARQL.
We can add a task to the next release to for SPARQLgraph to prevent them from being created in the first place, and handle these errors better.

I've implemented a partial fix.  SPARQLgraph now makes it very difficult for you to create a circular graph.
Further improvements are possible, but let's see whether this one gets us what we need.


Fix sha256 checksum for MAMP 3.3

Currently unable to install MAMP 3.3 due to a 'sha256 mismatch' error, this PR should fix it.

Thank you for the contribution. It was merged directly as b1d635f10f254a2ff262b36280740e0ef4d6280e to keep commit history cleaner. Your contribution is still credited to you.


ec_deployment: Make the kibana resource mandatory

Overview

We need to make the kibana resource mandatory,  as it's becoming part of the "platform".

well this is no more! Kibana isn't mandatory for advanced use cases!


Refactor and add white listing APIs.

This PR does the following:
Testing:

Revives the original tests, removing tests of dead routes.
Mocks storage so that during testing we can write packages to disk instead of writing them to S3.

Refactors:

Removes cloud search capability. This was causing difficulties in getting local testing set up, and is not used in the app.

Visual Studio Code Support:

Adds the .vscode folder to support VS Code.
Adds typings.json to allow typings to be installed, allowing for better code completion in VS Code.

New Routes:

Adds the white_list field to the Package schema.
Adds new routes for white listing packages.

GET /whitelist - returns all white listed packages
PUT /whitelist/:pkg_id - sets white_list = true on a package.
PUT /unwhitelist/:pkg_id - sets white_list = false on a package.

PTAL:
@pboyer

LGTM. I'd highly recommend testing this out on the staging server via Dynamo before putting into prod. Please let me know if you need any help with any of that.

@pboyer Yeah. I'm going to be banging on this for a couple of weeks before I main line it. Thanks for helping me to get all of this set up.


Blacklist specific properties for type object

What version of Ajv you are you using?
v6.1.1
What problem do you want to solve?
Can specific properties of type object be blacklisted?
Apologies if this is already addressed somewhere and I missed it. I read through the documentation and I tried looking on stack overflow without success. I know it's possible to set additionalProperties: false to blacklist all other properties not identified in the JSON schema but I'm only concerned with having a list of properties that are not valid, rather than all properties not listed in the schema. Perhaps this can already be solved with the compound keyword not?
What do you think is the correct solution to problem?
Maybe something like below?
{
  "type": "object",
  "properties": {
    "foo": { "type": "string" },
    "bar": { "type": "string" }
  },
  "invalidProperties": ["baz"]
}

valid: { "foo": "a" }, { "bar": "b" }, {}
invalid: { "foo": "a", "baz": 5 }
Will you be able to implement it?
Possibly

You can use “false” as a schematic for properties that should always fail. “Not” with “required” can be used as well.

@epoberezkin I appreciate the response. I wasn't aware of using false as a schematic for a property.
I tried out the below and it worked as expected.
{
  "type": "object",
  "required": ["foo"],
  "properties": {
    "foo": { "type": "string" },
    "bar": { "type": "string" },
    "baz": false
  }
}

However, I couldn't get an example working using a combination of not and required from looking through the docs. Could you provide a quick example for me to work with? I would greatly appreciate it


Build failure with AndroiX

 Android dependency 'androidx.***' has different version for the compile (1.0.0-beta01) and runtime (1.0.0-rc01) classpath. You should manually set the same version via DependencyResolution

I created a gist here of the logs when the command below fired.
flutter run -t main.dart --verbose

My project have a lot of dependencies, but it can be sure that flutter plugins may cause the error.
dependencies:
  flutter:
    sdk: flutter
  cupertino_icons: any
  camera: ^0.4.0+2
  barcode_scan:
    git:
      url: git@github.com:apptreesoftware/flutter_barcode_reader.git
  meta: any
  json_serializable: any
  json_annotation: any
  http_client:
    path: ../http_client
  flutter_spinkit: any
  intl: any
  cached_network_image: ^0.5.1
  url_launcher: any
  sentry: any
  curl: any
  uuid: any
  flutter_secure_storage: any
  permission_handler: ^2.1.3
  package_info: ^0.4.0+1
  device_info: ^0.4.0+1

Finally, I typed a lot of the dependencies with version for those package which causes the conflict.
But Is it correct way to fix it?

I had this same issue just now. Make sure you have AndroidX compatibility enabled for your Android app. You can follow the steps provided here and here on how to enable AndroidX in order for your packages to use the new Android support library.

Hi @fatihbalsoy, I have already enabled AndroidX support, and using it for a long time.
It worked well until this week, I think it would be caused by I upgraded somethings today.

You're right, by looking at the amount of issues relating to this topic in the last several hours, there's been a recent update that caused this problem. Your gist also says AndroidX is enabled, similar to this issue.

Seeing the same issue on my side.  Didn't want to make a new thread on this issue since there seem to be countless already being made.  I've tried all of the migration stuff; really clunky to have to open android studio when working with VSCode just to migrate to androidX and not have it work.

is the issue still relevant?

Without additional information, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact descriptions what happens, logs and the output of ' flutter doctor -v'  please.
All system setups can be slightly different so its always better to open new issues and reference related issues.


Update 2024-08-22_hosting_rshiny_applications.md

This is to create the rest of the posts if it is possible.

still, the problem exists with the Matrix but at least we have a proper way to handle the error and send it to the rest of the social media platforms.

Awesome! Great work!


[BUG] Microsoft.Azure.CognitiveServices.Language.LUIS.Authoring.LUISAuthoringClient.Apps.DownloadQueryLogsAsync does not provide an includeResponse parameter even though the API does.

Describe the bug
DownloadQueryLogsAsync does not provide an includeResponse parameter even though the API does (even in v2).
Code Snippet
public static async Task<Stream> DownloadQueryLogsAsync(this IApps operations, System.Guid appId, CancellationToken cancellationToken = default(CancellationToken))

Expected behavior
The SDK should provide all the functionality that the REST API does.
Additional context
For the sake of completeness, a model or helper method should also be included to help consume the CSV result of this call.
Information Checklist

[X] Bug Description Added
Repro Steps Added (N/A)
Setup information Added (N/A)

Hi @v-kydela, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.


Zendesk client keeps the thread occupied even after performing Zendesk.close()

I am using zendesk-java-client version 0.6.1 jar from the maven repository. The program does not terminate even after performing the close() method. The isClosed() method returns true, but the program does not terminate.  Is this intentional? Is there a work around? Thank you.

Fixed with https://github.com/cloudbees/zendesk-java-client/pull/238

@aheritier should we release 0.6.2?

yes let's go @johnou

0.6.2 released ~1h ago (waiting for central sync now) @johnou

Thank you


FR: open link with a new pane

I think it would be cool if there is an option that we could open a link using an iframe with a new pane like “search on internet”( https://github.com/HEmile/obsidian-search-on-internet )

I think it would be cool if there is an option that we could open a link using an iframe with a new pane like “search on internet”( https://github.com/HEmile/obsidian-search-on-internet )

Thank you for raising this issue, after checking the search on internet plugin, i introduced two ways to open split pane view links in Obsidian. You can try this new feature by updating the plugin version to 0.1.6 and changing the in-app view as your opening method in the plugin's settings panel.

That's exactly what I need! Thank you


Plot Bounding Boxes over 6 Camera Images and Publish to ROS Topics to be Visualized in Rviz

@tianweiy @muzi2045 From the trained model's generated co-ordinates of the 3D bounding boxes from line 197 in single_inference.py, I wanted to first transform these points into their 6 different respective camera plane and plot it using OpenCV/Matplotlib as described in the render_annotation() function in nuScenes devkit nuscenes.py at line 903. Additionally, it would great if the tracking id and object category could be labeled over these bounding boxes.
Once this is done, we can publish these images with bounding boxes to a ROS topic to be visualized in Rviz. And these changes to the single_inference.py and multi_sweep_inference.py files.

It's just a visualization problem, hope you can implement it and open a pull request.

@muzi2045 Yes, sure. How can I transform the predicted co-ordinates of the bounding boxes from lidar plane to camera plane using the calibration files, depending on the camera the object is visible the most?

@muzi2045 If add Subscribers for the 6 different cameras after line 248 in single_inference.py, do I also have to add TimeSynchronizer?

there normally has there coordinates when transform point from lidar coords to camera coords:

Lidar coords(right hand, x front, y left, z up) -->  Camera coords(right hand, z front, y left, x down)
Camera coords  --> Camera plane (u,v) coords.
you will find the calib param in those open source dataset.

If the dataset has done the time sync in hardware collection, there is no need to sync it again in reading the raw data.

As described in line 925 in nuscenes.py, before plotting the 2D bounding boxes over camera images, how do I figure out among the 6 different cameras, in which camera image the object is fully visible?

The /tf ROS topic is given out these type of data with translation and rotation. Can we use it to transform co-ordinates of bounding boxes in lidar frame to camera frame?

here it does the translation and check visibility.

@tianweiy Thanks, but I am afraid I am unable to the retrieve the cs_record dictionary data from any ROS topic. The cs_record dictionary seems to have all the transformation and rotation matrices and camera intrinsic parameters as described here and here respectively. Kindly suggest.

@tianweiy @muzi2045 The predicted bounding boxes co-ordinates are with respect to the lidar frame. Before transforming them to a particular camera, we need to first determine which camera's translation and rotation matrix we should use. The render_annotation() function from the nuScenes devkit takes as input the annotation token token with which the image path is known and the bounding boxes are plotted.
But in our case I was wondering how can we determine to which camera we transform the co-ordinates of the predicted bounding box and plot the bounding boxes after that?

it's about the Nuscenes camera FOV parameters. and compute the obstacles center_pos and dimension size locate in which camera FOV, there will be something like trunk locate in three camera image in the mean time.

@tianweiy @muzi2045 Can the box_in_image() function be used for the problem of determining the predicted bounding boxes co-ordinates corresponds to which camera and which camera's external calibration matrix (rotation + translation) should be used?

@iamsiddhantsahu  sorry for the late reply, I am really busy working on other projects. I think you can use this function. Though you need to check for every camera and then you will get a list of cameras that contain this object so you can draw boxes on these images. If you are not hurry, I can add some visualization files in a few weeks

Thanks! I am also working on it and hopefully can I can open a PR.
This is what I have decided to do.

Create an instance of the nuScenes "Box" class.
Assign the member variable like center, size, orientation, label etc. of the above created Box instance with the predicted co-ordinates of the Bounding Box in the single_inference.py file
Transform these predicted co-ordinates, first from the lidar frame to the ego vehicle frame and then from the ego vehicle to the camera frame. (Which camera's transformation needs to be applied will be answered by the box_in_image() function.)
Once we have the list of cameras, we can plot the predicted co-ordinates by using the render_cv2() function and pass the corresponding camera's intrinsic parameters to the "view" argument as well.
Then it can can be published to a ROS topic and visualized in Rviz.

Any suggestions/improvements are most welcomed!

@tianweiy @muzi2045 Thanks! I am also working on it and hopefully I can open a PR.
This is what I have decided to do.

Create an instance of the nuScenes "Box" class.
Assign the member variable like center, size, orientation, label etc. of the above created Box instance with the predicted co-ordinates of the Bounding Box in the single_inference.py file
Transform these predicted co-ordinates, first from the lidar frame to the ego vehicle frame and then from the ego vehicle to the camera frame. (Which camera's transformation needs to be applied will be answered by the box_in_image() function.)
Once we have the list of cameras, we can plot the predicted co-ordinates by using the render_cv2() function and pass the corresponding camera's intrinsic parameters to the "view" argument as well.
Then it can can be published to a ROS topic and visualized in Rviz.

Any suggestions/improvements are most welcomed!

@iamsiddhantsahu would you please open a Pull request regarding this task if your implementation was indeed successful?


Error: Element 'compiler', line 0 when importing xml file

Intro
Hi!
I am a  undergrad student at Technion, I use MuJoCo for my modeling of a robotic arm that will perform specific tasks. For now, I need to model the ur5e arm on top of the box. I have the issue of importing ur5e.xml file into scene.xml. I want to place it on top of the box, so I use
<mujoco>
   //<`-- ...Some code -->
   <worldframe>
   //<`-- ...Some code -->
       <body pos="0.3 0 0.2">
         <include file="ur5e.xml" />
       </body>
   </worldframe>
</mujoco>

but it throws the following error: XML error: Schema violation: unrecognized element. Element 'compiler', line 0. And even if I comment out the compiler in ur5e.xml, it still gives the same error but with the next tag in the xml file...
What is this issue? and how can I go around it?
My setup
Version 3.2.5, Python, ARM, MacOS.
What's happening? What did you expect?
I expect the ur5e robot to be on top of the box, but it throws an error instead. I am making a bug here, because this is google-deepmind model
Steps for reproduction

Open the scene from this DeepMind folder https://github.com/google-deepmind/mujoco_menagerie/tree/main/universal_robots_ur5e
Change scene.xml the one in minimal model for reproduction
Should appear on top of the box, but instead throws an error

Minimal model for reproduction
<mujoco model="ur5e scene">

  <statistic center="0.3 0 0.4" extent="0.8" />

  <visual>
    <headlight diffuse="0.6 0.6 0.6" ambient="0.1 0.1 0.1" specular="0 0 0" />
    <rgba haze="0.15 0.25 0.35 1" />
    <global azimuth="120" elevation="-20" />
  </visual>

  <asset>
    <texture type="skybox" builtin="gradient" rgb1="0.3 0.5 0.7" rgb2="0 0 0" width="512"
      height="3072" />
    <texture type="2d" name="groundplane" builtin="checker" mark="edge" rgb1="0.2 0.3 0.4"
      rgb2="0.1 0.2 0.3" markrgb="0.8 0.8 0.8" width="300" height="300" />
    <material name="groundplane" texture="groundplane" texuniform="true" texrepeat="5 5"
      reflectance="0.2" />
    <material name="wall" rgba="0.8 0.8 0.8 1" />
    <material name="box" rgba="0.6 0.4 0.2 1" />
  </asset>

  <worldbody>
    <!-- Ground plane -->
    <geom name="floor" size="0 0 0.05" type="plane" material="groundplane" />

    <!-- Base box -->
    <geom name="base_box" pos="0.3 0 0.1" size="0.2 0.2 0.1" type="box" material="box" />

    <!-- UR5e Robot placed on top of the box -->
    <body pos="0.3 0 0.2">
        <include file="ur5e.xml" />
    </body>

    <!-- Surrounding walls -->
    <geom name="wall1" pos="0.3 -0.3 0.5" size="0.2 0.01 0.5" type="box" material="wall" />
    <geom name="wall2" pos="0.3 0.3 0.5" size="0.2 0.01 0.5" type="box" material="wall" />
    <geom name="wall3" pos="0.1 0 0.5" size="0.01 0.2 0.5" type="box" material="wall" />
  </worldbody>
</mujoco>

Code required for reproduction
no code needed
Confirmations

[X] I searched the latest documentation thoroughly before posting.
[X] I searched previous Issues and Discussions, I am certain this has not been raised before.

The error is obvious, you are including a model that has <compiler/> at the top, so you cannot include it there.
Why are you using include instead of attach?


Remove version: SillyKoalaHugs.FerrariCreator version 1.0

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

URL: https://sillykoala.eu.org/?smd_process_download=1&download_id=351
OK
(Automated message - build 908)

URL: https://sillykoala.eu.org/?smd_process_download=1&download_id=351
OK
(Automated message - build 908)

Hi @Exorcism0666,
This manifest has the highest version number for this package. Is it available from another location? (This might be in error if the version is switching from semantic to string, or string to semantic.)
(Automated response - build 885.)

URL: https://sillykoala.eu.org/?smd_process_download=1&download_id=351
Status Code: 200
(Automated message - build 886)

For moderators or/and anyone else

[!Caution]
As a result of several personal problems I've had over the last few months + my final exams coming up very soon, I'm currently unable to look at my GitHub notifications every day and maintain my executions correctly.
If you detect a certain problem on a certain pul request:

Either you can close the pull request directly without mentioning me.
Or you can send me a message on X (Twitter): @Exorcism0666

Thank you for your understanding, and I'm sorry if some pull requests can't be approved due to the situation I'm in.

This account is automated by Github Actions and the source code was created by CoolPlayLin. If you have any questions about any pull request, don't hesitate to ping Exorcism0666.

[!warning]
With the new version of Komac, numerous Pull Requests may contain bugs or unwanted elements. If such issues arise, please tag russellbanks so that these bugs can be addressed in future updates.

[!important]
Please carefully review these Pull Request before merging. If it is a Pull Request for removing incorrect content and the URLs are issue, free upon manual checking, please close this Pull Request directly. (It is best to inform Exorcism0666 of the closure.)

良い一日をお過ごしください！


NCC-DITY002-029 Vectors initialized on the stack should specify u8 as the type of elements

Otherwise it could be interpreted as 32-bits integers by the compiler and allocate too much buffer space.

See also https://dfinity.atlassian.net/browse/FOLLOW-156


Pagination: Startpage and Max-Page Actions disabled

Actual:
If the user is on the startpage or max-page (pagination available), all icons are still available. reload is executed, even if the user ist allready on the first or last page form the pagination.
Expected behavior:
If the user ist on:

Startpage within a pagination, both arrows on the left side are disabled
max-page within a pagination, both arrows to the right side are disabled

Danke und liebe Grüsse Eva

Ready to test in v3.0.8


Change "value" property of inputs to "placeholder"

Thanks for the contribution, but I don't really want this change. Placeholders are generally considered bad for accessibility, and I actually did want to include default values so that when one value is changed and the result is updated, we don't get an error because one value is not a number.
All the best.