Pr fw acro

This PR brings in acro support on the ecl side.
Changes:

Split rate control methods into

control_bodyrate() -> pure bodyrate control
control_euler_rate() -> control desired euler_rates (calls control_bodyrate())

Implemented coordinated turn logic using airspeed rather than body velocities.
See: https://books.google.ch/books?id=N4abCgAAQBAJ&pg=PA186&lpg=PA186&dq=coordinated+turn+constraint&source=bl&ots=5oupXeXFdB&sig=1MhdC7NUF5Uv9hUc6p-FB2JPBxg&hl=en&sa=X&ved=0ahUKEwj9sdzhxfnRAhXFvBoKHXUBCH4Q6AEIGjAA#v=onepage&q=coordinated turn constraint&f=false

Better naming of variables, e.g. body_x_rate instead of roll_rate when we mean body rates.

Commented sideslip control based on lateral acceleration measurements because implementation is not yet correct. Needs an integrator!

@dagar This needs to be integrated with the changes you have prepared for the fw attitude control module in order to support acro.

Looking good. I'll do a simple acro implementation in the current fw_att_control for the sake of testing this and getting it in.
I keep wondering if having this stuff in ECL is worth the effort. Calling these in the correct order while passing around a big struct that you update between calls is a bit of an awkward api.
What are your thoughts on the different coordination methods? Long term do you think we could settle on one (after some testing and comparison)?

Should we disable coordinated turns below a certain speed?

This is the branch where I've been playing with a refactored fw_att_control (with and without ecl) - https://github.com/PX4/Firmware/compare/master...dagar:fw_att_control2?expand=1
It'll be much cleaner once all the manual_control_setpoints are pulled into stick_mapper.
I started with your ecl change, but then refactored and simplified to make them as simple as possible. I still think the entire thing would be much simpler pulled into fw_att_control, but this works.
https://github.com/PX4/ecl/compare/master...dagar:dagar_fw_att?expand=1
I had plain ACRO working, then I added attitude lock. RATTITUDE also works. I think the inverted case may need a closer look.

@dagar I'd like to get this in soon together with Acro on the firmware side. Did you mean to do this with your refactoring? I'm not sure what you meant with your comment before about "simple acro implementation".

Sorry, should have been more clear. I have a big refactor of fw_att_control where I was able to use this, but I'm guessing no one wants to deal with a huge code review and significant testing at the moment.
So for right now I can implement the bare minimum acro mode (rate setpoint from sticks) in the current fw_att_control that works with this ecl update. Sound reasonable?
Questions/Notes for this PR

Should we disable coordinated turns below a certain airspeed? FW_AIRSPEED_MIN perhaps?
is there anything specific we should validate in the logs and compare to the existing ecl? Was anyone actually using the existing coordinated turns?
do you think we actually need user configurable open or closed loop modes? Try we try both and keep one?
I think there are some problems in the control_attitude inverted case, but we can ignore it until we have something that might intentionally try control_attitude inverted. I'll do basic acro now, and acro + attitude lock later.
can you make the ECL_ControlData variable naming consistent and delete the unused fields?

Thanks. On the implementation: we obviously have the simple implementation already since we've been flying this so we could push that. So should we add a PR for this?
I would prefer to do one big change after another. The argument for doing it together the refactoring would be combined testing. Because this here needs to be tested anyhow. How is the state of your FW attctl refactoring?

Should we disable coordinated turns below a certain airspeed? FW_AIRSPEED_MIN perhaps?

@tumbili ?

is there anything specific we should validate in the logs and compare to the existing ecl? Was anyone actually using the existing coordinated turns?

As far as we know no one was using the coordinated turn method, and if it would not be good as a reference. In the previous implementation, activating the coordinated turn would have resulted in having the pitch correction twice because of the so called turn compensation which actually did pretty much the same thing. Roman, please correct me here if I got it wrong.

do you think we actually need user configurable open or closed loop modes? Try we try both and keep one?

I think it's a dev feature, and anyway not exposed to users. I remember you wanted to play with that, and possibly other methods? So this is up to you then I guess :)

I think there are some problems in the control_attitude inverted case, but we can ignore it until we have something that might intentionally try control_attitude inverted. I'll do basic acro now, and acro + attitude lock later.

We took a shortcut and limited the turn compensation to non-inverted flights. It has numerical issues near 90degree roll. I think this is fine because it is our opinion that this is not wanted in Acro. For attitude locking I'd suggest not locking into an inverted attitude ;). Then it should be fine too. Thoughs?

can you make the ECL_ControlData variable naming consistent and delete the unused fields?

Sure we can have another pass there. The only cleaner way of doing this passing values around would be to have separate structs for each of the methods. These methods have their interface and we won't get around to feed them the data.

Thanks. On the implementation: we obviously have the simple implementation already since we've been flying this so we could push that. So should we add a PR for this?

Yes that's probably fastest.

I would prefer to do one big change after another. The argument for doing it together with the refactoring would be combined testing. Because this here needs to be tested anyhow. How is the state of your FW attctl refactoring?

By the time I replaced everything with controllib and split the header, implementation, and launcher into separate files nearly every single line has changed. The hard part is making sure I didn't break the little features I don't use. I also find the incremental refactor PRs tends to be ignored (ahem https://github.com/PX4/Firmware/pull/6530). Maybe if I expanded more CI testing with thorough fw_att coverage people would be more comfortable pushing big changes through?

I think it's a dev feature, and anyway not exposed to users. I remember you wanted to play with that, and possibly other methods? So this is up to you then I guess :)

Only problem with a "dev feature" is that it shows up alongside the non-"dev feature" params that a PX4 FW or VTOL user needs to edit in the full param tree. Maybe one day that won't be the case in QGC.

We took a shortcut and limited the turn compensation to non-inverted flights. It has numerical issues near 90degree roll. I think this is fine because it is our opinion that this is not wanted in Acro. For attitude locking I'd suggest not locking into an inverted attitude ;). Then it should be fine too. Thoughs?

That's a good portion of my interest here. I'm also tempted to implement MAV_CMD_DO_INVERTED_FLIGHT
Looking forward to trying this outside of the simulator. Thanks guys!

Yes that's probably fastest.

Will do.

I also find the incremental refactor PRs tends to be ignored

At the end of the day, contributors need to go out and flight test. Especially for big refactoring. Sure, CI would help a lot initially.
The roadmap should also guide this a bit. I don't know what the plan is at the moment (@LorenzMeier ?) but from what I see going on currently we'd need to get the multirotor rework finished for a new release and then take care of fixed-wing refactoring?

Only problem with a "dev feature" is that it shows up alongside the non-"dev feature" params

Yep, UI problem ;)

That's a good portion of my interest here. I'm also tempted to implement MAV_CMD_DO_INVERTED_FLIGHT

Cool! Inverted flight should not be a problem and the turn compensation can also be easily adjusted to work. The interesting part is where you cross 90 degrees. I'm curious with what solution you come up with. My thought is still that you disable it when you are in rate control (which for rattitude would mean to temporarily disable it when the user is on the sticks).

Is this ready to go and tested, correct?

Test flown once on this state, but we've used the same for a while now. Should be good to merge here but before we move to it on PX4 (#6567) a few more flights are needed.


How to get public key to validate JWT received from "Bot Channels Registration"

Hi
I have a bot channel registration and configured my bot endpoint at
https://ukatie.com/api/v1/microsoft/message
When somebody enters a message, then I receive a request at this endpoint together with a JWT.
I would like to validate this JWT using the public key of Microsoft, but it is not clear to me how to get this public key.
I have read
https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens#validating-tokens
and I have tried the following URLs
https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration
https://login.microsoftonline.com/botframework.com/.well-known/openid-configuration
https://login.microsoftonline.com/botframework.com/.well-known/openid-configuration?appid=MY_APP_ID
and then continued with jwks_uri in order to get the public key, but I could NOT find a public key with the right/matching "kid".
Do I misunderstand something completely?
Thanks
Michael

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2471c08b-befe-8e1a-df08-f13e116c231e
Version Independent ID: 0d827d33-2906-6352-9398-13863def9354
Content: Microsoft identity platform access tokens - Microsoft identity platform
Content Source: articles/active-directory/develop/access-tokens.md
Service: active-directory
Sub-service: develop
GitHub Login: @nickludwig
Microsoft Alias: ludwignick

@michaelwechner Thank you for your feedback . We will investigate and update the thread.

Thanks!

@michaelwechner You can obtain public key by calling the public Azure AD OpenID configuration endpoint:
https://login.microsoftonline.com/{tenant_id}/discovery/keys?appid={client_id} and verify against the private key generated by Azure AD token.

Thanks for your feedback!
When I am requesting
https://login.microsoftonline.com/botframework.com/discovery/keys?appid=aaa8c4a1-d204-468f-ac6e-540b26b3a122
then the problem is none of the return values is matching my "kid" or "x5t" value "ZyGh1GbBL8xd1kOxRYchc1VPSQQ", which is part of the Azure AD token.
So, which returned public key should I use?
Here the content from the Azure AD token
{
"alg": "RS256",
"kid": "ZyGh1GbBL8xd1kOxRYchc1VPSQQ",
"typ": "JWT",
"x5t": "ZyGh1GbBL8xd1kOxRYchc1VPSQQ"
}
{
"serviceurl": "https://smba.trafficmanager.net/ch/",
"nbf":<PHONE_NUMBER>,
"exp":<PHONE_NUMBER>,
"iss": "https://api.botframework.com",
"aud": "aaa8c4a1-d204-468f-ac6e-540b26b3a122"
}

@[michaelwechner] Did you try passing GUID in tenant Id? You might get multiple keys and you need to search in it. I tried the same in my environment and able to get public key for my Azure AD token.
Azure AD token
{
"typ": "JWT",
"alg": "RS256",
"x5t": "jS1Xo1OWDj_52vbwGNgvQO2VzMc",
"kid": "jS1Xo1OWDj_52vbwGNgvQO2VzMc"
}
Public key

when I use the App ID in tenant-id, then I receive the error "invalid_tenant".
Or which GUID do you mean?
https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens

@michaelwechner Tenant Id is different from Client Id. I am not sure about bot registration.
If you register your bot using https://docs.microsoft.com/en-us/azure/bot-service/bot-service-quickstart-registration?view=azure-bot-service-4.0&tabs=userassigned then
To get your bot's app or tenant ID:
Go to the Azure Bot resource page for your bot.
Go to the bot's Configuration pane. From here you can copy the bot's Microsoft App ID or App Tenant ID.

I created a new Azure Bot and for this I received a tenant Id and I receive 4 keys, but the kid value of the Bearer token still is not contained. What else I might be doing wrong, that the "kid" values do not match?

@michaelwechner Are you validating the new bearer token issued by new Azure Bot?

you mean whether I try to use one of the 4 keys to validate the Bearer token?
Please find below a recent Bearer token
eyJhbGciOiJSUzI1NiIsImtpZCI6Ilp5R2gxR2JCTDh4ZDFrT3hSWWNoYzFWUFNRUSIsInR5cCI6IkpXVCIsIng1dCI6Ilp5R2gxR2JCTDh4ZDFrT3hSWWNoYzFWUFNRUSJ9.eyJzZXJ2aWNldXJsIjoiaHR0cHM6Ly9zbWJhLnRyYWZmaWNtYW5hZ2VyLm5ldC9jaC8iLCJuYmYiOjE2NDczNjE4NzEsImV4cCI6MTY0NzM2NTQ3MSwiaXNzIjoiaHR0cHM6Ly9hcGkuYm90ZnJhbWV3b3JrLmNvbSIsImF1ZCI6IjU3MWM2NDNhLWM1MTMtNDQxZS05Y2EyLWUwMWI0OGI5YTBkZSJ9.B93W0YumIcNQi7fz_dNh1rb3Rv0tpSMha8ym_aYPGpghn-1oy82eeI1XhuZWWrDQa83KGXgbbr6jLiMcJG6jCd2q9s0ei6OJUzkrLpfcam9QOTSJ12tz8FlYNFlnTzIm3cX4xmFVuA7Hu1QBzVRurYh1N_tao4CcmCO5jDBYdyHo6B5SXuukxv-rFGuzbNSSJ5V5bjuBfzdycz-yYwUTa6By7h2Xa1nglDbVCqryTJg253e8pjd5NCly5NVyr6cIFKVSh4nWILZNKwX8klr_OtDYY0nm9hlPF9WF5OVZWDYDuprbsQuWwFUV7bXT7XaNYK3DMqTVaRDpMpTRwLVjRA
and
Tenant ID: 1435b95f-4910-4050-b49d-888120d126ee
App ID: 571c643a-c513-441e-9ca2-e01b48b9a0de
and to get the four keys
https://login.microsoftonline.com/1435b95f-4910-4050-b49d-888120d126ee/discovery/keys?appid=571c643a-c513-441e-9ca2-e01b48b9a0de
but as you can see the "kid" values do not match.
When I send a message from within Azure Portal using the "Test Web Chat", then I receive a Bearer token with another "kid" value, but which does also not match, whereas please find below a recent token
ew0KICAiYWxnIjogIlJTMjU2IiwNCiAgImtpZCI6ICJ3c0s1UzloUHFpcXV0RFBvUm54eG
t2Q3RtZ2siLA0KICAieDV0IjogIndzSzVTOWhQcWlxdXREUG9Sbnh4a3ZDdG1nayIsDQogICJ0eXAiOiAiSldUIg0KfQ.ew0KICAic2VydmljZXVybCI6ICJodHRwczovL3dlYmNoYXQuYm90ZnJhbWV3b3
JrLmNvbS8iLA0KICAibmJmIjogMTY0NzM2MjUwNywNCiAgImV4cCI6IDE2NDczNjMxMDcsDQogICJpc3MiOiAiaHR0cHM6Ly9hcGkuYm90ZnJhbWV3b3JrLmNvbSIsDQogICJhdWQiOiAiNTcxYzY0M2EtY
zUxMy00NDFlLTljYTItZTAxYjQ4YjlhMGRlIg0KfQ.LdEEeOnREDgmNLiJjtmMmQB1XOP9AjzeHFxRfEUJ_zO72ba-N6XyJBj3isqNEOcwX5xuoAFduwNdPzSCkhl5RZMICdpb0HmxDQg1YGL2X3z3-8wsn
igG2r1ab43_rPmbXrVP7vm_K2UvgRYmAeRPtqcCKkkYxoFvJ8S6hR7Fe1ajd4zjylcuojLXj7KhhepML0nllx4L0rIzd56or9Hn6AoPxEbAPYhCtHeDI0x4DUVm9iNH1bKg9EV93Vr-VIwkGqIaf_JlJe8H
9MZ10N93Vcmoa3Kh1xzyvvxpJ5ktP1cNo9jhUFQ_mUhUU0alq2B8dRmzP5MQebluFQguI5DMMQ

@michaelwechner I am not aware how to generate the access token using Bot. I am opening another support ticket for you to help you  further.
Please send us an email on azcommunity [at] microsoft [dot] com referencing this issue along with Azure subscription Id with a subject line "ATTN:shweta" and we will help you with alternative support options on this.

Thanks! I just sent the email as you described

@michaelwechner seems like the Bot Service has its own IdP, the tokens are not issued by AAD.
You need to get the keys from their own endpoint, as below:
GET https://login.botframework.com/v1/.well-known/keys
This returns me a key with kids ZyGh1GbBL8xd1kOxRYchc1VPSQQ and wsK5S9hPqiqutDPoRnxxkvCtmgk, which matches the signatures of both of your tokens.
You can also read more here:
https://docs.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-connector-authentication?view=azure-bot-service-4.0#connector-to-bot-step-3

Hi Sergio
Thanks for your feedback!
I will give it a try shortly and will keep you posted.
All the best
Michael
Am 16.03.22 um 16:28 schrieb Sergio Correia:

@michaelwechner https://github.com/michaelwechner seems like the Bot
Service has its own IdP, the tokens are not issued by AAD.
You need to get the keys from their own endpoint, as below:
GET https://login.botframework.com/v1/.well-known/keys
This returns me a key with kids ZyGh1GbBL8xd1kOxRYchc1VPSQQ and
wsK5S9hPqiqutDPoRnxxkvCtmgk, which matches the signatures of both of
your tokens.
You can also read more here:
https://docs.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-connector-authentication?view=azure-bot-service-4.0#connector-to-bot-step-3
—
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/89481#issuecomment-1069252992,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIHNMCSZG3O45LUHW2HX7LVAH42LANCNFSM5QI6SHBA.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi Sergio
I am using Java to get the public key, whereas I took the x5c value
which is matching the kid value "ZyGh1GbBL8xd1kOxRYchc1VPSQQ"
String publicKeyPem ="MIIGGDCCBACgAwIBAgITbQDgJqLi3fOkWY1IZwACAOAmojANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwpNU0lUIENBIFoyMB4XDTIxMDYxMTExMzY0MFoXDTIyMDYxMTExMzY0MFowJTEjMCEGA1UEAxMaYm90YXBpLXByb2QtcHViLWp3dHNpZ25pbmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDYMLwR/vMAMR18nHlCPowbyaOsk8HzCQtUccSIv7f7CiGmRfqvC80IZwVpROSkg4TO2so8MgvrcE89efA6hXK+NnihOWp7IxMkUpKrW/JjseFFGYYoJ8ZbY7Bh+g4W+oFAzMRbeOjciKqyc/D/IrvIshbj9jemWsCmAmuMivo8Xnf7Lz/63VIFtwZsq/HT60m4+mmZoC96pgumGarhd5iOhJZ7hIfvmqq0VsuSubo9h/ObRVR+0UiOZsgGIVvHO7sC8K1lbHZ9Rkjeck5e9M3jg9xsmIYrDCIODXM7SGToHSZjJAhqL3Cu7W5U40o4FkcoYm2IQDl3bIDMSjfJAQTlAgMBAAGjggJPMIICSzALBgNVHQ8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCUGA1UdEQQeMByCGmJvdGFwaS1wcm9kLXB1Yi1qd3RzaWduaW5nMB0GA1UdDgQWBBSzObQplPSO2oxWYNbFks2BPm3iVzAfBgNVHSMEGDAWgBTSMoT753PGyeDd5Ju5RZlDZBgUeDCBvgYDVR0fBIG2MIGzMIGwoIGtoIGqhihodHRwOi8vY29ycHBraS9jcmwvTVNJVCUyMENBJTIwWjIoMikuY3Jshj9odHRwOi8vbXNjcmwubWljcm9zb2Z0LmNvbS9wa2kvbXNjb3JwL2NybC9NU0lUJTIwQ0ElMjBaMigyKS5jcmyGPWh0dHA6Ly9jcmwubWljcm9zb2Z0LmNvbS9wa2kvbXNjb3JwL2NybC9NU0lUJTIwQ0ElMjBaMigyKS5jcmwwgYsGCCsGAQUFBwEBBH8wfTA0BggrBgEFBQcwAoYoaHR0cDovL2NvcnBwa2kvYWlhL01TSVQlMjBDQSUyMFoyKDIpLmNydDBFBggrBgEFBQcwAoY5aHR0cDovL3d3dy5taWNyb3NvZnQuY29tL3BraS9tc2NvcnAvTVNJVCUyMENBJTIwWjIoMikuY3J0MD4GCSsGAQQBgjcVBwQxMC8GJysGAQQBgjcVCIfahnWD7tkBgsmFG4G1nmGF9OtggV2F3ulrgdSFZwIBZAIBGjAnBgkrBgEEAYI3FQoEGjAYMAoGCCsGAQUFBwMBMAoGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4ICAQATO0ipno7SRLtMa/4LnENt+Qb/x6b4uKD1MD+DYtYcaCCHbxo9PamUfDEkEgw7EOocx6xhFlDspf9Iu4c7DYdHxTOBWSQN8EBNpfvlIQ00mP+LS0djPLAZuFpeHKIkdUzH2XJwHWw+S4Vf1kAErZyt+jTWIQ9vvNXy8/EN/p/B7OG4VARSLwNWV8L7SFgJSkmXBv8upXjTr28rjVAaqhrwZ/HifStvAlSP94ST86GgwlYs1bSRUCXwDMsNfUyn2Rdju/ECElwsL34Xb0D2MOYNCyQu1G9cjkFWP0JWpM0EmggblCv485wRQcMvCWRIyqggHUlILIfcs1BxjjpcgyEdQx7NYOwM6r8Dl4q7cUPT1keaC6XdV9uDHjSVOHXOch2qaApPkPO2fZUhm6ZqOYmwgn3nL3d+FAUcXHi9ES4/7+JgXIDfMSIAOmxEXunqFACo8L6q/jAXkfMQQeVSKMvxrYey73/eiKCgFZ0BLORG/sg9Q/wVKfj5jPmGinebZ1LfvSXh5ZJWaXvx2G3OYnUMMdO0kqKi2CHuCh0cZHGhCgo5PqrgsnzB9RvihXs8YQJYzxzDSSi/ngnJrdOh3STCB9fdWRcu6OaBSYbV1QE1yuSXvCOT+S6O36Jkj1STePzVU6IsadoFXoaRf+3bMRPk/RieaSzIxqbeAVddfCCVAQ==";
byte[] keyContentAsBytes =  java.util.Base64.getDecoder().decode(publicKeyPem);
EncodedKeySpec keySpec =new X509EncodedKeySpec(keyContentAsBytes);
java.security.cert.CertificateFactory fact = java.security.cert.CertificateFactory.getInstance("X.509");
X509Certificate cer = (X509Certificate) fact.generateCertificate(new java.io.ByteArrayInputStream(keyContentAsBytes));
return cer.getPublicKey();
which works fine, but then I get the following error when trying to verify the signature:
"Unable to verify RSA signature using configured PublicKey. Signature length not correct: got 255 but was expecting 256"
Whereas I use io.jsonwebtoken.Jwts to verify the signature
Do you have an idea what I might be doing wrong?
Thanks
Michael
Am 16.03.22 um 16:53 schrieb Michael Wechner:

Hi Sergio
Thanks for your feedback!
I will give it a try shortly and will keep you posted.
All the best
Michael
Am 16.03.22 um 16:28 schrieb Sergio Correia:

@michaelwechner https://github.com/michaelwechner seems like the
Bot Service has its own IdP, the tokens are not issued by AAD.
You need to get the keys from their own endpoint, as below:
GET https://login.botframework.com/v1/.well-known/keys
This returns me a key with kids ZyGh1GbBL8xd1kOxRYchc1VPSQQ and
wsK5S9hPqiqutDPoRnxxkvCtmgk, which matches the signatures of both of
your tokens.
You can also read more here:
https://docs.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-connector-authentication?view=azure-bot-service-4.0#connector-to-bot-step-3
—
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/89481#issuecomment-1069252992,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIHNMCSZG3O45LUHW2HX7LVAH42LANCNFSM5QI6SHBA.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi Sergio
I think I was able to solve the problem :-)
If you don't mind, then I would like to keep the ticket open for some more tests to be certain that we got everything working correctly according to
https://docs.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-connector-authentication?view=azure-bot-service-4.0#step-4-verify-the-jwt-token
Btw, how did you see that our Bot Service has its own IdP, and that the tokens are not issued by AAD?
Thanks
Michael

Hey Michael,
Awesome, I am glad you were able to sort it out :)
If you decode the token (using https://jwt.ms for example), you will see the following claim:
"iss": "https://api.botframework.com"
This represents the IdP that issued the token.
When a token is issued by AAD, the issuer will be one of the below:
"iss": "https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/v2.0",
"iss": "https://sts.windows.net/fa15d692-e9c7-4460-a743-29f29522229/",
You can decode the sample tokens on this page and check the differences:
https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens#sample-tokens

Hey Sergio
Thanks again for your feedback!
I was now able to finish the implementation and it works very well now :-)
Please feel free to close the ticket in case it is still open.
All the best
Michael
Am 17.03.22 um 11:07 schrieb Sergio Correia:

Hey Michael,
Awesome, I am glad you were able to sort it out :)
If you decode the token (using https://jwt.ms for example), you will
see the following claim:
"iss": "https://api.botframework.com"
This represents the IdP that issued the token.
When a token is issued by AAD, the issuer will be one of the below:
"iss":
"https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/v2.0",
"iss": "https://sts.windows.net/fa15d692-e9c7-4460-a743-29f29522229/",
You can decode the sample tokens on this page and check the differences:
https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens#sample-tokens
—
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/89481#issuecomment-1070736403,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIHNMADS2C6XDUBGBQP4J3VAL77DANCNFSM5QI6SHBA.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>


Adding a design controls G2 option into the experiments screen in the Gutenberg plugin.

Hey Jon
What about adding an option into the Gutenberg -> Experiments screen to enable testing out G2 design control components directly in the Gutenberg plugin? As it would be a way to do some real life testing of various controls made with G2. I ask because you show so many awesome examples that I look forward to testing out in a real situation. Example typography controls. Having the experimental option inside the plugin will also give a good opportunity to get additional feedback.
Thanks!

Heya @paaljoachim ,
Interesting idea! It would be a good way to poke at the G2 Components in the project.
I'm not sure that's feasible at the moment (I may be wrong).
The G2 Components code isn't a theme/skin, that means we'd need to rebuild/replicate the existing controls with the components.
Maybe that's okay to start?
Sidenote: I'm preparing G2 to get a more formal proposal/PR for Gutenberg soon! We'll see how it plays out :)

The formal approach sounds like the best way to proceed.
Then see how things happen from there.
I will close this issue for now, and if it natural to do so we can always reopen.


Incorporate database to ci-infra.yml

We currently have an end-to-end test for the service layer .github/workflows/ci-infra-service.yml that:

Creates a new temporary workspace in the service layer (e.g. the equivalent of running something like terraform -chdir=infra/app/service workspace new t-qwerty where the workspace name is generated randomly each run
Spins up the service layer (e.g. running make infra-update-app-service APP_NAME=app ENVIRONMENT=dev)
Waits until the service is stable (e.g. running aws ecs wait services-stable ...)
Pings the service's root endpoint
Destroys the service layer infrastructure
Deletes the temp workspace

This test has been useful to catch issues with the service layer in the past that are more subtle and aren't caught with manual testing.
We currently don't have any automated tests for the database layer, which makes testing database layer changes very time consuming, especially since a database cluster takes upwards of 30 minutes to spin up.
This ticket is to add something similar for the database layer.
Notes
I had a previous attempt at this in this PR (https://github.com/navapbc/platform-test/pull/31) that got pretty close to having something working. The key aspects of that PR are:

Created a ci-infra-database.yml workflow
Split up infra/test/infra_test.go into a service_test.go and a database_test.go so that we can run service layer tests without the database layer tests and vice versa. The database layer tests would spin up the database layer in a separate workspace, run make infra-update-app-database-roles, then run make infra-check-app-database-roles
Moved the random workspace name function to infra/test/helpers.go to be reusable
Added a check database roles function to the role manager (This ended up being useful so I merged this change in a separate PR)

Stale attempt at this: https://github.com/navapbc/platform-test/pull/31

@lorenyu Could you refine this ticket some more?

@rocketnova refined!

@lorenyu Excellent! Thank you!


description with more than two lines

hi!
it's possible add description with more than two lines?
how?
many thanks!

You should be able to do this using the --description option.  If you are using the prompt then it will only accept a single line of input.
 ./gcalcli --calendar bhartvigsen add --description "This is a triumph
Making a note here
Huge success\!" --prompt


fix Lint/UnusedArgument

Summary
fix part of #1021

Hi @hotovson, please can you fix the conflicts with master? Thanks again for your effort, really appreciated, I will merge your branches as soon as possible.

@nodo rebased

Thanks @hotovson ! 👍


C++ version is difficult to build offline

I use Nix to manage dependencies and build a C++ project that uses Aeron. Nix packages are reproducible (builds can't use the network), and I can't use  the script that downloads Gradle to build the SBE/Aeron jars. Nix works on any Linux and on macOS.
Gradle isn't a problem when running the default C++ build, but it crashes when building the archive API because it downloads/builds the SBE tool to generate the archive codecs.
I managed to put together a Nix package spec (they are called derivations) to build the C++ version with the archive API by patching the CMake files to use pre-downloaded jars. If you have Nix installed and want to give it a try, put the following default.nix file on the root of the repo checked out to the 1.25.0 tag, and run nix-build.
This issue isn't currently blocking me, but I'm adding it here for future reference, or in case the maintainers are interested in making the build process "offline-friendly". Happy to help.
default.nix:
{ pkgs ? import (fetchTarball https://github.com/nixos/nixpkgs-channels/archive/37df59d1c36a5d1283f79a0cde668de9ca74fb9a.tar.gz) {} }:

let
  aeron-all-jar = pkgs.fetchurl {
    url = "https://repo1.maven.org/fromsearch?filepath=io/aeron/aeron-all/1.25.0/aeron-all-1.25.0.jar";
    sha256 = "110dd4z4bf6qphkgl57pdgbbldpzvzvdagrxg5i6d0sh8rkkm90y";
  };

  sbe-all-jar = pkgs.fetchurl {
    url = "https://search.maven.org/remotecontent?filepath=uk/co/real-logic/sbe-all/1.16.0/sbe-all-1.16.0.jar";
    sha256 = "1kp971nbflw48wfysl5cpk08zq6c415frz1wk13d0l47v3y8xnby";
  };
in

pkgs.stdenv.mkDerivation {
  name = "aeron";
  src = ./.;

  nativeBuildInputs = with pkgs; [ cmake jdk8 libbsd libuuid zlib ];

  cmakeFlags = "-DBUILD_AERON_DRIVER=ON -DBUILD_AERON_ARCHIVE_API=ON";

  patchPhase = ''
    substituteInPlace CMakeLists.txt \
      --replace 'set(AERON_ALL_JAR "''${CMAKE_CURRENT_SOURCE_DIR}/aeron-all/build/libs/aeron-all-''${AERON_VERSION_TXT}.jar")' \
                'set(AERON_ALL_JAR "${aeron-all-jar}")'
    substituteInPlace aeron-archive/src/main/cpp/CMakeLists.txt \
      --replace './gradlew -Dcodec.target.dir=''${ARCHIVE_CODEC_TARGET_DIR} :aeron-archive:generateCppCodecs' \
                'java -Dsbe.output.dir=''${ARCHIVE_CODEC_TARGET_DIR} -Dsbe.target.language=Cpp  -Dsbe.target.namespace=aeron.archive.client -Dsbe.validation.xsd=''${ARCHIVE_CODEC_SCHEMA_DIR}/fpl/sbe.xsd -Dsbe.validation.stop.on.error=true -jar ${sbe-all-jar} ''${CODEC_SCHEMA}'
  '';
}

Gradle will cache the JARs used in dependencies. You could prime the local cache with your build. Look under ``~/.gradle/caches/modules-2/files-2.1.

I'm not sure that would work because the build inputs have to be explicitly passed in order to make the build fully reproducible. I think a pair of new CMake flags with the location of the two jars would be good enough for this specific case.
If you are interest, I can help coming up with a proper solution for a reproducible build.

If you want to take a shot, we can look at it. However, if the jars are downloaded, Gradle does use them, so not sure why what @mjpt777 mentions would not work.

Thanks for waiting for my response.

However, if the jars are downloaded

That is the issue: either the build isn't reproducible (because it is affected by the machine state) or it needs to fetch from the network. If I can explicitly pass the location of the jars to CMake, the build can be reproducible offline.

Closing due to no further progress.


Amplify push with Api change overrides schema.json path

Describe the bug
I have a mono repo setup where my /amplify folder sits in the root of the project and my /graphql folder is located in a shared package like so packages/core/src/graphql/schema.json. When I'm pushing changes in the API the schemaPath setting in my .graphqlconfig.yaml file is always overwritten. The actual mutation and query file paths are respected and only the schemaPath is changed to a default schemaPath: src/graphql/schema.json
To Reproduce
Steps to reproduce the behavior:

Move your /graphql directory somewhere other than /src
Refelect the change in your .graphqlconfig.yaml
Change something in your GraphQL schema
use amplify push and confirm updates to GraphQL API and generating GraphQL statements.
Observe GraphQL statements correctly being placed in new location but a new /graphql/schema.json created in the root of your project AND in your .graphqlconfig.yaml the schemaPath changed back to schemaPath: src/graphql/schema.json

Expected behavior
I expect the schemaPath setting to be respected and not overwritten.

We don't expose the schemaPath custom configuration and is hardcoded currently. I'm adding this as an enahncement so that this schemaPath is configurable and not overwritten.

Awesome thanks!


Functional modalities CBV and phase are not represented in Entity table

It doesn't look like the Entity table was updated with the CBV and phase functional modalities when they were added to the specification. I think they just need to be listed under func with bold, sbref, and events.

Good catch. Would you care to open a PR?

Absolutely. Will do.


more compact edits in insta

LSP has a particularly awkward representation of edits in completion. This PR makes sure that we use a simple TextEdit representation interanlly, and moves all the complexity to the conv layer.
bors r+


Understanding Motion Vectors

Hello,
I am trying to understand the motion vectors returned from Method :: retrieve() described in the readme.md.

For 1, 2: I assumed to be w and h of the macroblock according to AVMotionVector. Would you mind confirming whether the size of the macroblock is for the current frame or the reference frame?
For 5, 6: I am confused about the x/y-center coordinate. Does this mean that the dst_x and dst_y represent the center of the current frame's macroblock instead of the top-left? And, do src_x and src_y represent the center of the reference frame's macroblock, as well?
For 7, 8: I am unsure how to use the equations for motion_x and motion_y. Are these equations for calculating each individual pixel within the macroblock?
For 9: Is this the current frame's macroblock scale compared to the reference frame's macroblock? For example, assume the w, h refer to the size of the current frame's macroblock and dst_x, dst_y, src_x, and src_y refer to the top-left positions of their corresponding macroblocks, if motion_scale = 2 and src_x, src_y = 1, 1 and dst_x, dst_y = 2, 3 and w, h = 16, 16, then does the macroblock of x1, x2, y1, y2 = 2, (2+16), 3, (3+16) of the current frame is reconstructed from the macroblock of x1, x2, y1, y2 = 1, (1+16/2), 1, (1+16/2) of the reference frame?

I appreciate any help, thank you!

Hello @chanwutk,
thanks for your questions. They are good ones, so it took me some research to figure out the answers. Here, they are:

Correct, fields 1 and 2 are width and height of the macroblock, respectively. For some reason, the description was deleted in the README when I rephrased the text a few years ago. I added this back in. The macroblock size in H.264 can be 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, or 4x4. Importantly, the size of the MB is the same in the current and reference frame because the encoder will subtract the predicted MB from the current MB and only consider the difference within that region for the next step (discrete cosinus transform).

I looked at the motion vectors of one frame in the MPEG-4 Part 2 encoded video for simplicity because here the macroblocks are all 16 x 16. src_x increases from 8 to 1272 and src_y from 8 to 712. Similarly, dst_x takes values between 8 to 1272 and dst_y values from 8 to 712. From this I deduce that the motion vector "base" and "tip" are centered in the macroblock and do not refer to the top-left corner. But beware that H.264 uses quarter-pixel motion estimation, i.e., the src_x and src_y coordinates are actually fractional coordinates that are calculated as src_x = dst_x + motion_x / motion_scale and src_y = dst_y + motion_y / motion_scale.

The previous answer contains the equations for the relation between source positions, reference position, motion, and motion scale. I took the equation from the documentation of the AVMotionVector struct, which is extracted by the motion vector extractor. i don't know for what reason the AVMotionVector struct stores src_x and src_y additionally as integers as this information is redundant and less accurate (integer-resolution) than the representation via motion and motion scale (quarter-pel resolution). Anyways, I noted that the formulas for motion_x and motion_y in the README are incorrect and I will update them accordingly.

I hope the previous two answer could clarify this question. If you are still unsure I recommend section 3.3.16 "sub-pixel motion compensation" in this book.


My new branch

Hey! I have added a new button to the list.
Please review it.

@Spyware007  please review this and add this button


Deconstrucoring and AoT

🐞 Bug report
Command

build

Description
When using destructuring in a module-factory, the AoT-build seems to create a bundle with the wrong value.
The following code provides {prop1:{prop1:"prop1",prop2:"prop2"},prop2:undefined} instead of {prop1:"prop1",prop2:"prop2"}:
public static forRoot(
    { prop1 = "prop1", prop2 = "prop2" }: Partial<Bug>
): ModuleWithProviders
{
    return {
        ngModule: BugModule,
        providers: [
            { provide: BUG_TOKEN, useValue: { prop1, prop2 } },
        ]
    };
}

I inspected the generated main.*.js and it seems like the AoT-build inlines the value, but it is not calculated correctly.
🔬 Minimal Reproduction
A minimal reproduction can be found here.
Steps to reproduce:

Execute ng build --prod
Open main.*.js and search for prop1. You will find the following Object:
{prop1:{prop1:"First Property",prop2:"Second Property"},prop2:void 0}
Open bug.module.ts and enable the workaround by removing the comment form line 9 and line 16 and putting line 8 and line 15 in comment instead.
Run ng build --prod again.
Open the new main.*.js and search for prop1 again. Now the Object has the correct value
{prop1:"First Property",prop2:"Second Property"}

🌍 Your Environment

Angular CLI: 7.3.8
Node: 10.14.0
OS: win32 x64
Angular: 7.2.14
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.13.8
@angular-devkit/build-angular     0.13.8
@angular-devkit/build-optimizer   0.13.8
@angular-devkit/build-webpack     0.13.8
@angular-devkit/core              7.3.8
@angular-devkit/schematics        7.3.8
@angular/cli                      7.3.8
@ngtools/webpack                  7.3.8
@schematics/angular               7.3.8
@schematics/update                0.13.8
rxjs                              6.3.3
typescript                        3.2.4
webpack                           4.29.0

Anything else relevant?
I was looking for this issue and found an article on angular.io, which says that Destructured variables are not supported in AoT.
I am not sure if this also applies for my example, but if it does, it should probably show an error instead of resulting in a wrong bundle.

Hi, would you mind re posting this in https://github.com/angular/angular
As this is a compiler bug.
Thanks


add retro-bit 2.4GHz pads

Hello, I'm trying to add 2 pads with 2 modes each at the same time, please let me know if I've done it right.

Maps already in the DB for these, but thanks !


Update internal submodule to latest commit on its main branch

@trivoldus28 @JoeStrout
The hiccup here is that we need to sync up the requirements between pyproject.toml's when we update internal versions. It's not ideal, but I haven't found a way around it yet. We want both projects to be installable independently to run independent CI's, which drives the need for two pyprojects

I'm a bit confused, why does this repo need the extra dependencies if the tests and functionalities don't need them?

Maybe put a check if internal exists, install it. If not then don't?

I'm guessing it's because neither the tests nor the rest of the CI actually import the file in question


Feature Request : match result api

I am developing the starcraft2 ladder platform based on the commandcenter and I find it it hard to get the match result.
Could you please expose the match result api? Thanks.

This is an issue with the api, not the bot. Closing it for now


Cluster Agent cannot reconcile webhook

Output of the info page (if this is a bug)
2022-01-04 15:47:36 UTC | CORE | WARN | (pkg/util/log/log.go:630 in func1) | Deactivating Autoconfig will disable most components. It's recommended to use autoconfig_exclude_features and autoconfig_include_features to activate/deactivate features selectively
2022-01-04 15:47:36 UTC | CORE | INFO | (cmd/system-probe/config/config.go:119 in Merge) | no config exists at /etc/datadog-agent/system-probe.yaml, ignoring...
Getting the status from the agent.

===============
Agent (v7.32.3)
===============

  Status date: 2022-01-04 15:47:36.933 UTC<PHONE_NUMBER>933)
  Agent start: 2022-01-04 15:46:59.953 UTC<PHONE_NUMBER>953)
  Pid: 1
  Go Version: go1.16.7
  Python Version: 3.8.11
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: INFO

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -3.602ms
    System time: 2022-01-04 15:47:36.933 UTC<PHONE_NUMBER>933)

  Host Info
  =========
    bootTime: 2022-01-03 09:07:50 UTC<PHONE_NUMBER>000)
    kernelArch: x86_64
    kernelVersion: 5.4.0-1064-azure
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 21.04
    procs: 219
    uptime: 30h39m16s
    virtualizationRole: host
    virtualizationSystem: kvm

  Hostnames
  =========
    host_aliases: [6f2277ad-0ffe-4bcc-ad0a-497915c1b7ac aks-common-76155617-vmss000000-alg-m3-test-aks]
    hostname: aks-common-76155617-vmss000000-alg-m3-test-aks
    socket-fqdn: datadog-agent-5m7l5
    socket-hostname: datadog-agent-5m7l5
    host tags:
      cluster_name:alg-m3-test-aks
      kube_cluster_name:alg-m3-test-aks
    hostname provider: container
    unused hostname providers:
      aws: not retrieving hostname from AWS: the host is not an ECS instance and other providers already retrieve non-default hostnames
      azure: azure_hostname_style is set to 'os'
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: status code 404 trying to GET http://<IP_ADDRESS>/computeMetadata/v1/instance/hostname

  Metadata
  ========
    cloud_provider: Azure
    hostname_source: container

=========
Collector
=========

  Running Checks
  ==============

    containerd
    ----------
      Instance ID: containerd [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/containerd.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 558, Total: 1,116
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 2
      Average Execution Time : 318ms
      Last Execution Date : 2022-01-04 15:47:21 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:21 UTC<PHONE_NUMBER>000)

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 9, Total: 11
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-01-04 15:47:28 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:28 UTC<PHONE_NUMBER>000)

    cri
    ---
      Instance ID: cri [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cri.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 54, Total: 108
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 13ms
      Last Execution Date : 2022-01-04 15:47:35 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:35 UTC<PHONE_NUMBER>000)

    disk (4.4.0)
    ------------
      Instance ID: disk:e5dffb8bef24336f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 712, Total: 1,424
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 56ms
      Last Execution Date : 2022-01-04 15:47:27 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:27 UTC<PHONE_NUMBER>000)

    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 5, Total: 10
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-01-04 15:47:34 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:34 UTC<PHONE_NUMBER>000)

    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 78, Total: 102
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-01-04 15:47:26 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:26 UTC<PHONE_NUMBER>000)

    kubelet (7.1.0)
    ---------------
      Instance ID: kubelet:5bbc63f3938c02f4 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 1,083, Total: 2,108
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 4, Total: 8
      Average Execution Time : 449ms
      Last Execution Date : 2022-01-04 15:47:27 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:27 UTC<PHONE_NUMBER>000)

    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 6, Total: 12
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-01-04 15:47:33 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:33 UTC<PHONE_NUMBER>000)

    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 18, Total: 36
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-01-04 15:47:25 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:25 UTC<PHONE_NUMBER>000)

    network (2.4.0)
    ---------------
      Instance ID: network:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 79, Total: 158
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 13ms
      Last Execution Date : 2022-01-04 15:47:32 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:32 UTC<PHONE_NUMBER>000)

    ntp
    ---
      Instance ID: ntp:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 1, Total: 1
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 1
      Average Execution Time : 29ms
      Last Execution Date : 2022-01-04 15:47:06 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:06 UTC<PHONE_NUMBER>000)

    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 1, Total: 2
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-01-04 15:47:24 UTC<PHONE_NUMBER>000)
      Last Successful Execution Date : 2022-01-04 15:47:24 UTC<PHONE_NUMBER>000)

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    ClusterRole: 0
    ClusterRoleBinding: 0
    CronJob: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    HighPriorityQueueFull: 0
    Job: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Role: 0
    RoleBinding: 0
    Service: 0
    ServiceAccount: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 6
    Successes By Endpoint:
      check_run_v1: 2
      intake: 2
      series_v1: 2

  API Keys status
  ===============
    API key ending with b9643: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - b9643

==========
Logs Agent
==========

    Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.com on port 443
    BytesSent: 1.416464e+06
    EncodedBytesSent: 43496
    LogsProcessed: 1333
    LogsSent: 1291

  datadog/datadog-agent-5m7l5/init-config
  ---------------------------------------
    - Type: file
      Identifier: d52221bc34ee20b008768595ae19895f09fe83c816fb54164db65dcb4eb616d1
      Path: /var/log/pods/datadog_datadog-agent-5m7l5_22ab947a-4f5a-45b8-a8a7-cc4b41c192cf/init-config/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  container_collect_all
  ---------------------
    - Type: docker
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  datadog/datadog-agent-5m7l5/process-agent
  -----------------------------------------
    - Type: file
      Identifier: 0dfeab4c2b292ce6d46e4c933819f7ee3b2022d6f19a74e36f9c053bed23ca15
      Path: /var/log/pods/datadog_datadog-agent-5m7l5_22ab947a-4f5a-45b8-a8a7-cc4b41c192cf/process-agent/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  datadog/datadog-agent-5m7l5/agent
  ---------------------------------
    - Type: file
      Identifier: 9edc900f82e3ecc7c3f876ee3a8f76a30e3ffc51998d4f653f9a02c9a7956c75
      Path: /var/log/pods/datadog_datadog-agent-5m7l5_22ab947a-4f5a-45b8-a8a7-cc4b41c192cf/agent/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  kube-system/local-nvme-provisioner-nmf42/provisioner
  ----------------------------------------------------
    - Type: file
      Identifier: 44a728ccbecbcf5d406f217608dafaa7e31401828302891084c7a74bcae5112d
      Path: /var/log/pods/kube-system_local-nvme-provisioner-nmf42_6b15fdc3-93f3-459c-ac35-4de83ea6439b/provisioner/*.log
      Status: OK
        1 files tailed out of 1 files matching
      Inputs:
        /var/log/pods/kube-system_local-nvme-provisioner-nmf42_6b15fdc3-93f3-459c-ac35-4de83ea6439b/provisioner/0.log
      BytesRead: 150625
      Average Latency (ms): 140
      24h Average Latency (ms): 140
      Peak Latency (ms): 527
      24h Peak Latency (ms): 527

  datadog/datadog-agent-5m7l5/init-volume
  ---------------------------------------
    - Type: file
      Identifier: 88ae3e02e78682babcf3e30529b9e02dfdd60d0e9d1614b504fb8a9c8d1b3c3a
      Path: /var/log/pods/datadog_datadog-agent-5m7l5_22ab947a-4f5a-45b8-a8a7-cc4b41c192cf/init-volume/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  datadog/datadog-agent-5m7l5/trace-agent
  ---------------------------------------
    - Type: file
      Identifier: 24242ad0baa82636c471e84b78148d751d31a7804f6a22e2037d261d569b1e55
      Path: /var/log/pods/datadog_datadog-agent-5m7l5_22ab947a-4f5a-45b8-a8a7-cc4b41c192cf/trace-agent/*.log
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

=========
APM Agent
=========
  Status: Running
  Pid: 1
  Uptime: 36 seconds
  Mem alloc: 17,203,264 bytes
  Hostname: aks-common-76155617-vmss000000-***
  Receiver: <IP_ADDRESS>:8126
  Endpoints:
    https://trace.agent.datadoghq.com

  Receiver (previous minute)
  ==========================
    From go 1.17.5 (gc-amd64-linux), client v1.34.0
      Traces received: 11 (4,852 bytes)
      Spans received: 11

    Default priority sampling rate: 100.0%
    Priority sampling rate for 'service:api,env:test': 100.0%
    Priority sampling rate for 'service:db,env:test': 100.0%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 5,528
  Dogstatsd Metric Sample: 438
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 2
  Series Flushed: 3,245
  Service Check: 35
  Service Checks Flushed: 32
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 437
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 73,515
  Udp Packet Reading Errors: 0
  Udp Packets: 275
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 1
  Unterminated Metric Errors: 0

=====================
Datadog Cluster Agent
=====================

  - Datadog Cluster Agent endpoint detected: https://<IP_ADDRESS>:5005
  Successfully connected to the Datadog Cluster Agent.
  - Running: 1.16.0+commit.9961689

=============
Autodiscovery
=============
  Enabled Features
  ================
    containerd
    cri
    kubernetes

Describe what happened:
After upgrading the datadog helm chart to version 2.28.11 (datadog 7.32.3, DCA 1.16.0), we're getting the following errors from the cluster agent: CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:170 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again.
Deleting and reinstalling the datadog helm chart does not fix the issue. Downgrading to version 2.22.10 (datadog 7.31.1, DCA 1.15.1) fixes the issue though.
Describe what you expected:
We expect the cluster agent to work nominally and not throw errors about the admission controller webhook.
Steps to reproduce the issue:
Deploy datadog via the helm chart in version 2.28.11 (or any 2.28 patch) with the following values:
datadog:
  kubelet:
    tlsVerify: false

  logs:
    enabled: true
    containerCollectAll: true

  apm:
    portEnabled: true

  env:
    - name: DD_CONTAINER_EXCLUDE_LOGS
      value: "image:mcr.microsoft.com/.*" # Exclude kube-proxy (mcr.microsoft.com/oss/kubernetes/kube-proxy)

  systemProbe:
    collectDNSStats: false

clusterAgent:
  admissionController:
    enabled: true
    mutateUnlabelled: true

Additional environment details (Operating System, Cloud provider, etc):
Running on AKS with kubernetes v1.21.2

Actually, downgrading to version 2.22.10 of the helm chart does not solve the issue, it makes the logs less frequent.

I believe this has happened to us, sometimes the agent just never installs the webhook. Is that the same thing here?

Hi @seddarj @tehlers320,
could you try to update to the cluster-agent v1.17.0, because we made some improvement around the webhook registration: https://github.com/DataDog/datadog-agent/pull/10171

Hi @clamoriniere, we still have the issue with 1.17.0 (used in version 2.35.0 of the datadog helm chart)

We are experiencing the same issue at the moment after updating to those versions.

I see this too after upgrading to chart datadog-2.30.7, but only on one of the two nodes in my cluster.

I am experiencing the same with datadog helm chart - 2.30.17

Experiencing this issue by installing using yamls (without helm). Image version: Agent (v7.37.1)

We're having the same problem.  Tested on chart versions 2.22.10, 2.24.0, and 2.28.something.

Same for Datadog Cluster-agent v.1.22.0 (from chart version 2.37.4)

Same for chart version v2.37.7

same issue in chart 3.1.8

I am also facing the issue with Helm chart version 3.1.9 with Cluster Agent 7.39.2.
Part of the debug logs from the cluster-agent
2022-10-13 10:05:01 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_v1.go:141 in reconcile) | The Webhook datadog-webhook was found, updating it
2022-10-13 10:05:01 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:149 in enqueue) | Adding object with key datadog-webhook to the queue
2022-10-13 10:05:01 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:177 in processNextWorkItem) | Webhook datadog-webhook reconciled successfully
2022-10-13 10:05:01 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_v1.go:141 in reconcile) | The Webhook datadog-webhook was found, updating it
2022-10-13 10:05:01 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:149 in enqueue) | Adding object with key datadog-webhook to the queue
2022-10-13 10:05:01 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again
2022-10-13 10:05:01 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_v1.go:141 in reconcile) | The Webhook datadog-webhook was found, updating it
2022-10-13 10:05:02 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:177 in processNextWorkItem) | Webhook datadog-webhook reconciled successfully
2022-10-13 10:05:02 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_v1.go:141 in reconcile) | The Webhook datadog-webhook was found, updating it
2022-10-13 10:05:02 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:149 in enqueue) | Adding object with key datadog-webhook to the queue
2022-10-13 10:05:02 UTC | CLUSTER | DEBUG | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:149 in enqueue) | Adding object with key datadog-webhook to the queue
2022-10-13 10:05:02 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again

I am also facing the issue with Helm chart version 3.1.10 with Cluster Agent 7.39.2.
Running on AKS, kubernetes version 1.24.6
Part of the debug logs from the cluster-agent:

2022-10-17 14:52:34 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again
2022-10-17 14:52:34 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again
2022-10-17 14:52:34 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again
2022-10-17 14:52:35 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again
2022-10-17 14:52:35 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again
2022-10-17 14:52:36 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again
2022-10-17 14:52:36 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again

I can confirm we are also facing the issue with Helm chart version 3.1.10 and Cluster Agent 7.39.2. We are running AKS on  kubernetes version 1.22.6. Tried deleting the MutatingWebhookConfiguration and installing Datadog cluster agent again but I am still facing the same issue.

This issue popped up when running two instances of the Helm chart in one cluster(one with APM enabled, the other disabled) and inadvertently running two instances of the cluster agent(wrong indent in YAML used for disabling the cluster agent).  Once the second cluster agent was disabled, the problem was resolved.

Modified the DataDogAgent CRD to enable the admissionsController and am seeing the same issue.

This is happening to me - AKS 1.22.11 and Helm chart version 3.1.11 on a fresh DD install.

The same problem is occurring.
AKS kubernetes version:1.24.6
cluster-agent:7.40.1
helm chart:datadog-3.3.1

Same problem here:
AKS kubernetes version:1.23.8
cluster-agent:7.40.1
helm chart:datadog-3.3.1

I ran into the same situation with a fresh deploy on AKS (1.24.6) and helm chart 3.3.3.
Using configuration values from Datadog's AKS example config fixed it: https://github.com/DataDog/helm-charts/blob/main/examples/datadog/agent_on_aks_values.yaml

Any update to this?
Same problem:

AKS kubernetes version: 1.23.8
Cluster-Agent: 7.39.0
Helm Chart: datadog-3.1.3

This was resolved here via the addition of an envar specific to AKS.
There are 2 potential workarounds:

If you are not using Admission Controller functionality you can set clusterAgent.admissionController to false https://github.com/DataDog/helm-charts/blob/main/charts/datadog/values.yaml#L858

OR

If you are using Admission Controller functionality, please  set the envar DD_ADMISSION_CONTROLLER_ADD_AKS_SELECTORS to true in the clusterAgent section of your Helm chart.

clusterAgent:
  env:
    - name: "DD_ADMISSION_CONTROLLER_ADD_AKS_SELECTORS"
      value: "true"

Either of those two options will remove those errors
We will get this updated in the documentation.

Based on https://github.com/DataDog/datadog-agent/issues/10413#issuecomment-1347204883 I think this error should no longer be happening for me but I am still seeing it.
2023-06-15 04:46:54 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again

I am using helm chart 3.25.1 which has
          {{- if .Values.providers.aks.enabled }}
          - name: DD_ADMISSION_CONTROLLER_ADD_AKS_SELECTORS
            value: "true"

and I am passing
    providers:
      aks:
        enabled: true

https://github.com/DataDog/datadog-agent/issues/10413#issuecomment-1330642566 also saw this on a recent helm chart that should have been fixed.
I am only running one instance of the chart
emilyzall@Emilys-MBP datadog-agent % helm list -n datadog
NAME         	NAMESPACE	REVISION	UPDATED                                	STATUS	CHART         	APP VERSION
datadog-agent	datadog  	3       	2023-06-13 09:30:23.615522944 +0000 UTC	failed	datadog-3.25.1	7

The release was fine on other clusters and was okay until 6/13. I don't know what happened on 6/13 to cause this.
REVISION	UPDATED                 	STATUS  	CHART         	APP VERSION	DESCRIPTION
1       	Thu May 25 09:45:40 2023	deployed	datadog-3.25.1	7          	Install complete
2       	Tue Jun 13 09:10:18 2023	failed  	datadog-3.25.1	7          	Upgrade "datadog-agent" failed: timed out waiting for the condition
3       	Tue Jun 13 09:30:23 2023	failed  	datadog-3.25.1	7          	Release "datadog-agent" failed: timed out waiting for the condition

I noticed I had two replicasets running. When I removed the one with zero replicas this error stopped. I am still seeing my helmrelease failing with the same error.
emilyzall@Emilys-MBP datadog-agent % kg rs -n datadog
NAME                                     DESIRED   CURRENT   READY   AGE
datadog-agent-cluster-agent-6976b556d6   12        12        12      22d
datadog-agent-cluster-agent-7fb8cbc449   0         0         0       3d2h
emilyzall@Emilys-MBP datadog-agent % kg deployment -n datadog
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
datadog-agent-cluster-agent   12/12   12           12          22d

Quick FYI, there is new guidance as far as which envar to set.  See this for more info
https://docs.datadoghq.com/containers/kubernetes/distributions/?tab=helm#AKS

This seems to be broken again, at least on AKS 1.25.11. Assuming you've set the env variable from the workaround above the agent creates the webhook with:
        "namespaceSelector": {
          "matchExpressions": [
            {
              "key": "control-plane",
              "operator": "DoesNotExist"
            }
          ]
        },

While the resource after Azure modifies it has
  namespaceSelector:
    matchExpressions:
    - key: control-plane
      operator: DoesNotExist
    - key: control-plane
      operator: NotIn
      values:
      - "true"
    - key: kubernetes.azure.com/managedby
      operator: NotIn
      values:
      - aks

The agent then just spins trying to reconcile the webhook several times a second forever.

The current AKS label is hardcoded here:
https://github.com/DataDog/datadog-agent/blob/4b9f448dc5ab5a01c1b45023a90131c4d29d7991/pkg/clusteragent/admission/controllers/webhook/common.go#L79

I'm getting the same error. Didn't happen before. I'm running 1.25.11 as well for AKS.

I noticed I had two replicasets running. When I removed the one with zero replicas this error stopped. I am still seeing my helmrelease failing with the same error.
emilyzall@Emilys-MBP datadog-agent % kg rs -n datadog
NAME                                     DESIRED   CURRENT   READY   AGE
datadog-agent-cluster-agent-6976b556d6   12        12        12      22d
datadog-agent-cluster-agent-7fb8cbc449   0         0         0       3d2h
emilyzall@Emilys-MBP datadog-agent % kg deployment -n datadog
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
datadog-agent-cluster-agent   12/12   12           12          22d

@emily-zall's solution is how I corrected my error in GKE Autopilot. I'm unsure how I ended up with 2 replicasets, but once I removed the one that had 0 desired, the error stopped for me.
I'm putting my exact error message so it can help others find the solution:
Couldn't reconcile Secret default/webhook-certificate: secrets is forbidden: User "system:serviceaccount:datadog:datadog-cluster-agent" cannot create resource "secrets" in API group "" in the namespace "default"


Patient resource, Display text for code 2186-5 needs to be changed

Some of the Synthetic data for Patients has a code of 2186-5 with Display text of "Non Hispanic or Latino", this needs to be changed to "Not Hispanic or Latino" to validate against US Core.
This might have been correct before issue https://github.com/synthetichealth/synthea/issues/783
The display code in the ValueSet doesn’t match whats in the CodeSystem.  The CodeSystem is the one that should be in the instances.

I opened https://jira.hl7.org/browse/FHIR-34488 to hopefully avoid this kind of confusion in the future.
In case there is any debate on which display value should be in instances, refer to https://www.hl7.org/fhir/valueset.html#designations

Any display name for a concept provided in the value set is for display to a human user. The display in the Coding that results from a user selecting a concept from the expansion must be taken from the underlying code system definition, even if a value set is referenced explicitly in the Coding (e.g. by an extension). The correct display for a code can be determined by a $lookup operation. Any alternative display specified in the value set would go in CodeableConcept.text, perhaps appended to the UI label for the matching data element.

The code as written is this:
https://github.com/synthetichealth/synthea/blob/9f5ff46eaa8e2b03651fb575ee38dc5016d24efc/src/main/java/org/mitre/synthea/export/FhirR4.java#L479-L485
There may be previously generated sample data sets that do not comply with the current code.
The "current version" of US Core (v4.0.0) says "Non Hispanic or Latino" (http://hl7.org/fhir/us/core/STU4/ValueSet-omb-ethnicity-category.json.html).
The latest ballot of US Core (v4.1.0) says Not Hispanic or Latino (http://hl7.org/fhir/us/core/2022Jan/ValueSet-omb-ethnicity-category.html and https://vsac.nlm.nih.gov/valueset/2.16.840.1.114<IP_ADDRESS>7/expansion [UMLS account required])

Great, then this should already be fixed.
My point above was that in 4.0.0, even though the valueset uses "Non", the CodeSystem still says "Not" and so that is always the right one to use in an instance.  Glad the latest draft of US Core has changed the valueset display name to avoid future confusion.

Will test again, but according to this, the issue is fixed.


ConnectionNameList

version 2.1.1
jdk openjdk 8
When failing over to a clustered mq(connection named list has more then one server in it), if the pooling property is set to true, the connections are very slow and spinning up more connections is slow.When first connecting, if the first server in the list is the passive server, the pool expands very slowly. If the active server is first, behavior is normal, unless the passive becomes the active, and then the slowness reappears.

This sounds like it's probably due to underlying behaviour of the MQ JMS classes - I believe that connections will always try the entries in the list from the first until it finds one that works. And depending on what the problem is, it may take a while to recognise a connection failure before it tries subsequent entries.
There's not much that can be done about that in this library, but it might be worth raising an RFE to see if the base interface can be enhanced to do something like automatically reorder the list on retries as it knows which connections have failed. There's no indication back to the caller that a particular conname has succeeded or failed.
This Spring Boot module could conceivably shuffle the connname elements for each connection request but that might have adverse implications such as connecting to a secondary system (not always a failover address) in preference to the primary. Which then leads to needing more configuration controls ...

Nothing to do in this project


Fixed up documentation regarding workflow support

Forgot to update the documentation regarding workflows with the changes from yesterday, here is the fixup

If this works well we should do a release?

That might be a good idea, it's been a while since the last release and I think workflows are a very nice new Jenkins feature worth supporting.


Unable to use with Snowpack due to multiple versions of React

If your site is using React 17 and you're using Snowpack, if you try to add react-media-recorder, you'll see hook errors due to multiple versions of React. Snowpack will try to include both React 16 and 17.
Solutions may be to ensure React 16 is not considered a dependency for this package.

This problem was related to Snowpack configuration. Closing issue.
If anyone else runs across this, here is the configuration fix: https://github.com/snowpackjs/snowpack/issues/3033#issuecomment-905481198


Receiver CRON network results dashboard

Fixes https://github.com/CDCgov/prime-reportstream/issues/5679
Test Steps:

In client, select Admin -> Receiver status dashboard
Verify:

[ ] Change start and end dates and verify data changes
[ ] Type in filters for Receiver and Results; verify data changes
[ ] Select Mixed Results from the Success dropdown
[ ] Click on Green time slice and Red time slice and verify Info Details dialog is displayed.

Todo
Not part of this PR but future work.

Refactor the one major rendering component into subcomponents to make it easier to test/maintain.
Cache (memo) each row component so filtering is faster

Changes

Updated the admin.yml API file to include new Admin-only API call.

Testing

[x] Tested locally?
[x] Ran ./prime test or ./gradlew testSmoke against local Docker ReportStream container?
[x] (For Changes to /frontend-react/...) Ran npm run lint:write?
[x] Added tests?

Process

[x] N/A Are there licensing issues with any new dependencies introduced?
[x] Includes a summary of what a code reviewer should test/verify?
[x] N/A Updated the release notes?
[x] N/A Database changes are submitted as a separate PR?
[x] N/A DevOps team has been notified if PR requires ops support?

Specific Security-related subjects a reviewer should pay specific attention to

Does this PR introduce new endpoints?

new endpoint A
new endpoint B

Does this PR include changes in authentication and/or authorization of existing endpoints?
Does this change introduce new dependencies that need vetting?
Does this change require changes to our infrastructure?
Does logging contain sensitive data?
Does this PR include or remove any sensitive information itself?

If you answered 'yes' to any of the questions above, conduct a detailed Review that addresses at least:

What are the potential security threats and mitigations? Please list the STRIDE threats and how they are mitigated

Spoofing (faking authenticity)

Threat T, which could be achieved by A, is mitigated by M

Tampering (influence or sabotage the integrity of information, data, or system)
Repudiation (the ability to dispute the origin or originator of an action)
Information disclosure (data made available to entities who should not have it)
Denial of service (make a resource unavailable)
Elevation of Privilege (reduce restrictions that apply or gain privileges one should not have)

Have you ensured logging does not contain sensitive data?
Have you received any additional approvals needed for this change?

@doug-s-nava Thanks for the detailed review. Super helpful. I think I've addressed most of your feedback and/or made another bug to improve if it seemed like to big of a refactor. Can you look it over and see if I missed anything?

@doug-s-nava I think I've addressed everything including remove the extra code scoping brackets. (I used the IDK hint // <editor-fold desc="Time slices for a given day render"> .. which webstorm users but I'm not sure about vscode.)
Think we're close. Thanks again for all the time spent reviewing!

@doug-s-nava I really do appreciate a detailed review and the time, focus it requires. Thank-you for being so thorough.


Fix no permission message

after https://github.com/SpongePowered/SpongeAPI/commit/b38390e1751fff802d355155c45b2eb4913f5817 when a player hasn't got permission they get a message saying the command doesn't exist, this is very frustrating when attempting to set up a new server/plugin as it appears that the plugin isn't loaded when you just don't have permission for the command(s)

I was discussing this with someone on the Nucleus Discord as to what the change was intended to do, message reproduced below.
I'll take this on over the next couple of days when I get a moment, I know how I want to fix this in a simple way, at least in the short term - mentioned below.

For background - that commit comes from this PR - https://github.com/SpongePowered/SpongeAPI/pull/2038. The idea was that it would prevent the commands appearing in /help and though tab completion - before this commit this was happening. This is something that a lot of people have been asking for and so makes sense. That method basically returns the command if the source should be able to use it - and is used during execution too.
A solution would be to pass in null as the source during execution, it would then suppress tab completion and usage in /help, but allow for it to be executed if desired. Could use a config option too.


Datum based comparison kernels (#4596)

Which issue does this PR close?

Closes #4596
Rationale for this change

Adds datum based comparison kernels, deprecating the old kernels and drastically reducing the amount of generated code.
Benchmarks show no major performance regressions, and if anything non-trivial performance improvements.
Additionally a release build on master with dyn_cmp_dict takes
________________________________________________________
Executed in   39.60 secs    fish           external
   usr time  128.19 secs  583.00 micros  128.19 secs
   sys time    2.71 secs  102.00 micros    2.71 secs

But with the changes in this PR takes
________________________________________________________
Executed in   15.87 secs    fish           external
   usr time   60.84 secs    0.00 micros   60.84 secs
   sys time    2.28 secs  643.00 micros    2.28 secs

What changes are included in this PR?

Are there any user-facing changes?

Latest benchmarks
eq Float32              time:   [7.6360 µs 7.6405 µs 7.6457 µs]
                        change: [-17.240% -16.996% -16.750%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 15 outliers among 100 measurements (15.00%)
  3 (3.00%) high mild
  12 (12.00%) high severe

eq scalar Float32       time:   [5.2382 µs 5.2407 µs 5.2433 µs]
                        change: [-2.6651% -2.3977% -2.1336%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe

neq Float32             time:   [7.3749 µs 7.3801 µs 7.3862 µs]
                        change: [-27.961% -27.771% -27.616%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
  3 (3.00%) high mild
  9 (9.00%) high severe

neq scalar Float32      time:   [5.2315 µs 5.2346 µs 5.2384 µs]
                        change: [-35.454% -35.241% -34.988%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  2 (2.00%) high mild
  8 (8.00%) high severe

lt Float32              time:   [15.455 µs 15.580 µs 15.681 µs]
                        change: [-5.7116% -4.9659% -4.2590%] (p = 0.00 < 0.05)
                        Performance has improved.

lt scalar Float32       time:   [9.8461 µs 9.8686 µs 9.8992 µs]
                        change: [+4.0063% +5.2827% +6.7502%] (p = 0.00 < 0.05)
                        Performance has regressed.

lt_eq Float32           time:   [15.896 µs 15.905 µs 15.915 µs]
                        change: [-10.347% -10.136% -10.003%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe

lt_eq scalar Float32    time:   [9.7845 µs 9.7888 µs 9.7932 µs]
                        change: [-21.407% -21.254% -21.051%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 23 outliers among 100 measurements (23.00%)
  11 (11.00%) low mild
  5 (5.00%) high mild
  7 (7.00%) high severe

gt Float32              time:   [15.894 µs 15.905 µs 15.917 µs]
                        change: [-0.3861% -0.0659% +0.2289%] (p = 0.72 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  2 (2.00%) high severe

gt scalar Float32       time:   [9.7929 µs 9.7982 µs 9.8039 µs]
                        change: [+0.1360% +0.4044% +0.6769%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

gt_eq Float32           time:   [15.868 µs 15.883 µs 15.901 µs]
                        change: [-9.9332% -9.7737% -9.5779%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 26 outliers among 100 measurements (26.00%)
  12 (12.00%) low mild
  4 (4.00%) high mild
  10 (10.00%) high severe

gt_eq scalar Float32    time:   [9.8059 µs 9.8120 µs 9.8192 µs]
                        change: [-21.359% -21.226% -21.019%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

eq Int32                time:   [7.6663 µs 7.6775 µs 7.6896 µs]
                        change: [-12.682% -12.410% -12.111%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
  6 (6.00%) high mild
  6 (6.00%) high severe

eq scalar Int32         time:   [5.1598 µs 5.1615 µs 5.1638 µs]
                        change: [-4.2256% -4.0547% -3.8551%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) high mild
  6 (6.00%) high severe

neq Int32               time:   [7.5694 µs 7.5728 µs 7.5769 µs]
                        change: [-20.136% -19.885% -19.648%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
  1 (1.00%) high mild
  11 (11.00%) high severe

neq scalar Int32        time:   [5.1674 µs 5.1699 µs 5.1726 µs]
                        change: [-36.189% -36.007% -35.816%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) high mild
  7 (7.00%) high severe

lt Int32                time:   [7.6795 µs 7.6866 µs 7.6961 µs]
                        change: [-18.855% -17.713% -16.550%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe

lt scalar Int32         time:   [5.1625 µs 5.1655 µs 5.1691 µs]
                        change: [-4.2787% -4.0476% -3.8997%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  10 (10.00%) high severe

lt_eq Int32             time:   [7.5761 µs 7.5806 µs 7.5861 µs]
                        change: [-20.731% -20.486% -20.235%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  2 (2.00%) high mild
  12 (12.00%) high severe

lt_eq scalar Int32      time:   [5.2456 µs 5.2493 µs 5.2536 µs]
                        change: [-34.425% -34.242% -34.062%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  3 (3.00%) high mild
  8 (8.00%) high severe

gt Int32                time:   [7.5842 µs 7.5915 µs 7.6007 µs]
                        change: [-11.722% -10.713% -9.6121%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 22 outliers among 100 measurements (22.00%)
  1 (1.00%) high mild
  21 (21.00%) high severe

gt scalar Int32         time:   [5.1702 µs 5.1726 µs 5.1753 µs]
                        change: [-3.9266% -3.6285% -3.3424%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 17 outliers among 100 measurements (17.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
  12 (12.00%) high severe

gt_eq Int32             time:   [7.6899 µs 7.6971 µs 7.7053 µs]
                        change: [-21.943% -21.716% -21.463%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 15 outliers among 100 measurements (15.00%)
  4 (4.00%) high mild
  11 (11.00%) high severe

gt_eq scalar Int32      time:   [5.1667 µs 5.1693 µs 5.1722 µs]
                        change: [-36.230% -36.051% -35.867%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
  9 (9.00%) high severe

eq MonthDayNano         time:   [63.182 µs 63.378 µs 63.600 µs]
                        change: [-4.9198% -4.2991% -3.5572%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

eq scalar MonthDayNano  time:   [54.143 µs 54.196 µs 54.262 µs]
                        change: [+2.4635% +2.9231% +3.3797%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

eq_dyn_utf8_scalar dictionary[10] string[4])
                        time:   [53.156 µs 53.176 µs 53.197 µs]
                        change: [-0.0663% +0.1054% +0.3198%] (p = 0.37 > 0.05)
                        No change in performance detected.
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) high mild
  2 (2.00%) high severe

gt_eq_dyn_utf8_scalar scalar dictionary[10] string[4])
                        time:   [108.95 µs 108.98 µs 109.02 µs]
                        change: [+1.3318% +1.6177% +1.9018%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe

eq dictionary[10] string[4])
                        time:   [388.52 µs 388.82 µs 389.12 µs]
                        change: [-8.7351% -8.5458% -8.4271%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe

To what do you attribute the performance improvements like the following?
...
gt Int32
                        time:   [7.5842 µs 7.5915 µs 7.6007 µs]
                        change: [-11.722% -10.713% -9.6121%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 22 outliers among 100 measurements (22.00%)
  1 (1.00%) high mild
  21 (21.00%) high severe
///
eq dictionary[10] string[4])
                        time:   [388.52 µs 388.82 µs 389.12 µs]
                        change: [-8.7351% -8.5458% -8.4271%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe

For the case of dictionaries it is likely because the process of normalizing the keys allows for unchecked indexing, which in turn helps LLVM optimise the code properly as there isn't a branch in the main loop.
For the case of primitives I don't have a very good intuition, however, these kernels historically have been extremely sensitive to inlining. It is possible that collocating more of the code allows LLVM to better optimise the code... I haven't looked in too much detail

Ran the benchmarks with the pre-existing SIMD kernels, and they are roughly twice as fast. As written this PR will therefore represent a performance regression for users with the SIMD feature enabled. I can look to bring back the SIMD versions if people feel strongly, although my preference would be to avoid this complexity

I do wonder if some of the compilation speedup is that arrow-rs no longer instantiates all the code for the different comparisons unless it is actually called

We don't make use of generics in the public interface, so it will instantiate all the variants regardless of if they're called

I recommend leaving this PR open for another day or two to allow others the chance to comment (or request more time to comment)

I will probably look to get this in Friday morning GMT to make the next arrow release

We are currently several arrow releases behind in our engine, so I can't easily test the performance impact on our internal benchmark queries. The compile-time improvements seem really nice though, and if needed we could implement comparison kernels for our limited set of datatypes using portable_simd.


[openssl-unix] Fix openssl-unix android build

Updates the openssl-unix port to properly build for Android

What does your PR fix? Fixes #9874, #11282, #8174

Which triplets are supported/not supported? Have you updated the CI baseline?
N/A. Tested w/ [arch]-android and [arch]-android-static

Does your PR follow the maintainer guide?
Yes

Thanks for your PR! Can you please correct the Version field in the CONTROL file? You can get more information from maintainer-guide.md.

/azp run

I ended up hitting the same issue as this PR : https://github.com/microsoft/vcpkg/pull/11344
Will wait and merge changes to msys code after that is merged

/azp run

@majaeger Now port signalrclient and osg build failed on Linux, they all depend on opnssl-unix.

@majaeger Can you resolve the conflict scripts/cmake/vcpkg_acquire_msys.cmake?

@LilyWangL sorry for the delay. I've undone my changes to vcpkg_acquire_msys.cmake

Thanks for the PR!


[storage][stg74] address PR comments

/azp run js - storage-blob - tests

/azp run js - storage-file-share - tests

/azp run js - storage-queue - tests

Should we update the version to have -beta.1?

Should we update the version to have -beta.1?

@jeremymeng
I thought we should use  -prerelease.1?

Should we update the version to have -beta.1?

@jeremymeng
I thought we should use -prerelease.1?

The guidance changed recently after some back and forth to have our nightly builds be -alpha and our prerelease builds be -beta. For packages that have an active -preview we're leaving that alone, but for new prereleases, we would like to snap to -beta.


Optimize document store and sync algorithm

The sync algorithm currently reads large parts of the store in to memory. E.g.
// TODO: This is hugely inefficient and needs to be optimized
// For an identity range that includes everything we allocate a vec with all entries of
// the replica here.
let local_values: Vec<_> = self
    .store
    .get_range(range.clone())?
    .collect::<Result<_, _>>()?;

this will be fine for a few 1000 entries, but it won't scale for really large documents. We should make sure that iroh provides few but load-bearing primitives, and currently iroh docs (unlike e.g. blobs) are not load-bearing. As soon as you build something big on top, they will crumble.
I think we can come up with a trait for the store that allows implementing an efficient mem store and disk store, and refactor the sync algo (process_message) to use this trait.
The trait should

make it possible for the store to cache fingerprints (the current one does this as well)
provide an interface that allows pivot selection without loading all keys and values into mem
be sync (no async here, since that will be hell. In the worst case we drive this thing from an actor that has it's own thread)
support efficient prefix replacement like in the original willow design

I think the first thing we must do is to make sure that keys are basically newtypes for &[u8]. That way we ensure that we can use a lex sorted kv store or a u8 radix tree and get the correct order.


[MWCore] Update to latest 'engine' and 'data-capture' lib versions

IMPORTANT: Where possible all PRs must be linked to a Github issue
Fixes https://github.com/opensrp/fhir-resources/issues/565
Checklist

[ ] I have written Unit tests for any new feature(s) and edge cases for bug fixes
[ ] I have added any strings visible on UI components to the strings.xml file
[ ] I have updated the  CHANGELOG.md file for any notable changes to the codebase
[ ] I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the project's style guide
[ ] I have built and run the fhircore app to verify my change fixes the issue and/or does not break the app

@LZRS  I was able to open the registration form once. After I registered someone, I was no longer able to open any form including the registration form. Could you please look into this? I keep getting this in the logs:
java.lang.NullPointerException: Parameter specified as non-null is null: method kotlin.jvm.internal.Intrinsics.checkNotNullParameter, parameter <this>
        at kotlin.collections.CollectionsKt___CollectionsKt.joinToString(Unknown Source:2)
        at kotlin.collections.CollectionsKt___CollectionsKt.joinToString$default(_Collections.kt:3515)
        at org.smartregister.fhircore.quest.ui.main.AppMainActivity.onSync(AppMainActivity.kt:105)
        at org.smartregister.fhircore.engine.sync.SyncBroadcaster$registerSyncListener$1$1.emit(SyncBroadcaster.kt:67)
        at org.smartregister.fhircore.engine.sync.SyncBroadcaster$registerSyncListener$1$1.emit(SyncBroadcaster.kt:67)
        at kotlinx.coroutines.flow.SharedFlowImpl.collect$suspendImpl(SharedFlow.kt:383)
        at kotlinx.coroutines.flow.SharedFlowImpl$collect$1.invokeSuspend(Unknown Source:15)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loopOnce(Looper.java:226)
        at android.os.Looper.loop(Looper.java:313)
        at android.app.ActivityThread.main(ActivityThread.java:8669)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:571)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1135)
    	Suppressed: kotlinx.coroutines.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@1624cd5, Dispatchers.Main.immediate]

@LZRS After doing some more testing, I've also noted this bug which needs to be resolved before we merge this PR. Will wait to hear back from the Google team.

@LZRS thanks for working on this. Please incorporate these PRs too:

https://github.com/google/android-fhir/pull/1779 (fixes the bug in the previous comment)
https://github.com/google/android-fhir/pull/1740

@LZRS I've checked out the PR. I've noticed that on single page questionnaires, there is now a submit button present in the bottom right corner above the save button. Can you please remove this. Here is a screenshot of the issue.

@LZRS I've checked it out and the submit button has been removed, thank you. However, two other issues have come up:

The new progress bar present in the questionnaires is meant to be immediately under the toolbar as shown in the FHIR sample app. However, in the Tingathe app, it seems there is some padding as shown here. If you could please look into this.
The answer expression extension is no longer working due to an update in the implementation. Please use this comment as a guide to get this working again.

@LZRS Thanks for working on this, looks good to merge.