[WIP] Middleware jms

Add middleware jms to each server and make it possible to list all jms for all servers.
When clicking on refresh middleware provider jms are loaded successfully, however listing these jms is not yet working, because of error in listicon_image method.

[x] Add tests
[x] Add preview screenshots

@miq-bot add_label providers/hawkular, ui, wip

I was unable to find suitable icon for jms so I just rotated datasources icon and used is as placeholder, but we'll have to create specific icon for this.
@pilhuhn what do you think about fetching all recursive children when fetching server entities?

I was unable to find suitable icon for jms so I just rotated datasources icon and used is as placeholder, but we'll have to create specific icon for this.

@karelhala which icon is used for this in hawkular ui?

for the icon, you should use https://github.com/ManageIQ/manageiq/blob/master/app/assets/images/100/placeholder.png
(copy + rename it to the icon that needs design)
cc @epwinchell

I'm not sure regarding the name Middleware_jms. Is it how it's called in hawkular?
as a start, jms should probably be all capitalized since it is an abbreviation.

perhaps I missed it, but where's a differentiator/type in UI & backend for queue versus topic?
For example, a user is looking at a jms summary page - how is it possible to know whether this is a queue or topic?

I was also wondering if we should separate queues and topics there. @pilhuhn @jdoyleoss ?

Queues and Topics do have some different properties/metrics to them, so I think there is value in separating them. Eventually we will have similar but distinct representations, and it's better to start that way.

What's a Jms?  Is this a generic term for all possible Middleware providers or is that a Hawkular specific thing (I generally prefer avoiding abbreviations unless it's an industry standard term).

JMS is Java Message Service, it's one of the Java EE specifications and a subsystem in EAP servers. Within JMS there are queues and topics, which are the primary objects to model.

Interesting...are all Middleware managers Java based?  If so, this is fine, but it struck me as something very specific to Java Middleware managers.

The middleware provider (Hawkular) leverages EAP as it's runtime platform. So it's specific to EAP, and thus Hawkular.

Interesting...are all Middleware managers Java based?

Not particular, it can be written in any language, but most of middlewares communicate trough messages.

So, what I'm hearing is that an abstraction of this would be to call it middleware_message_service?

Actually JMS is the Java version of a Message Oriented Middleware(MOM). Those also come in other flavors (e.g. MQTT for IoT), but the notion of Topics and Queues are the same for all MOMs. In the context of Java/JBoss JMS is certainly the term people want to use.

Queues and Topics do have some different properties/metrics to them, so I think there is value in separating them. Eventually we will have similar but distinct representations, and it's better to start that way.

Please clarify what kind of separation - completely separate to different entities? Have a single representation with different 'type' field? (I'd go with the latter)

So, what I'm hearing is that an abstraction of this would be to call it middleware_message_service?

I propose middleware_message_destination

Yes, destination is the 'superclass' for Topics and Queues.

I've fixed problems with decorator and things proposed by @abonas (thanks for pointing those flaws out).
There is still work needed on test suites and we should add more information to message detail.
Once we will agree on naming this center I'll rename it. I forgot that we will support more than just Java middleware so I went blindly for JMS, but this should be named better.
Do we agree on renaming it from Middleware jms to Middleware message destination?

Once we will agree on naming this center I'll rename it. I forgot that we will support more than just Java middleware so I went blindly for JMS, but this should be named better.
Do we agree on renaming it from Middleware jms to Middleware message destination?

@jdoyleoss please clarify what is the direction here so the work on this PR can proceed. thanks.
is there any place where jms should be mentioned at all as well as topic/queue? (imo it can be in the type field "jms queue" and "jms topic" for Java users clarity)

I think we should go for a general "superclass" of "messaging destination", which can then be sub-divided into message queue and message topic. Those can be on the MiQ-level like the generic middleware_server. This way we can support any message-oriented-middleware in the future.
Then inside of Hawkular we can subclass this for specifics of the EAP7' Artemis Queues and Topics. Here at that level we can name them JMS Queue and JMS Topic if we want.

The problem here is that JMS have ques and topics, but Python might have different name conventions. So I propose this (item highlights are broken, because I just copy pasted html on console, so provider is highlighted instead of Messaging):

Either this, or as @pilhuhn proposed (Egain, provider is highlighted instead of Messaging):

First solution will allow for combining same middleware's server messages, the other will combine message types together.
I'll let you choose @pilhuhn @abonas @jdoyleoss which is better.

My concern with "messaging" is that if the entities are named 'messages', that sounds like they are the actual 'units' (similar to events) that are sent, versus queues and topics which are the channels/destination - those that allow them to be sent.
@jdoyleoss the terminology here is your call.

Messaging is ok to me. The thing that does the work is usually called 'Message oriented Middleware' (or MoM). A message is what is sent. See e.g. https://en.wikipedia.org/wiki/Message-oriented_middleware (or https://en.wikipedia.org/wiki/Message_oriented_middleware, but the first one is better).
We could rename Topic to Fan-out or Pub/Sub. RabbitMQ seems to use the term fanout (and the example is in Pyhton https://www.rabbitmq.com/tutorials/tutorial-three-python.html )
Queue is ubiquitous.

Messaging is ok to me. The thing that does the work is usually called 'Message oriented Middleware' (or MoM). A message is what is sent. See e.g. https://en.wikipedia.org/wiki/Message-oriented_middleware (or https://en.wikipedia.org/wiki/Message_oriented_middleware, but the first one is better).
We could rename Topic to Fan-out or Pub/Sub. RabbitMQ seems to use the term fanout (and the example is in Pyhton https://www.rabbitmq.com/tutorials/tutorial-three-python.html )
MQTT calls them Topics too.
Queue is ubiquitous.
So I guess for people in the MoM space, Topic and Queue are known terms.

the question is how many people, especially existing hawkular users and those who come from java world will understand the move from topic to fanout

I think Messaging is the right top level term to use, and that we should use Topics and Queues under that. I think these are both the correct Java terms, generally understood, and even applicable outside of Java EE.

I think Messaging is the right top level term to use, and that we should use Topics and Queues under that. I think these are both the correct Java terms, generally understood, and even applicable outside of Java EE.

messaging it is.
do you want messaging, topic, queue prefixed with middleware like the rest of the entities?

Yes, let's keep it consistent.  Thanks

@karelhala please rebase this and also the icon/images needs to be replaced by the proper placeholder image

@abonas Removed all utilization settings and added messaging type to this PR. Could you please look if I missed anything? Thank you.

@karelhala the tests are failing, please check it. thanks

Ok the test gave me a lot of headache.
The problem was with refresh tests and VCR settings. What I did in this PR is that we are no longer calling children, but recursiveChildren. I tried update VCR using recording new episodes, but that was not good, since feedId changes (feedId in VCR was different thant the one on my computer).
I had to delete refresher.yml and run recording this VCR egain. I bumped into problem with not having middleware domains so I followed this guide (with @Jiri-Kremser help): https://github.com/ManageIQ/manageiq/pull/10048
So if anyone will change something in a way that it will call new URL he'll have to do the same steps.

I had to delete refresher.yml and run recording this VCR egain. I bumped into problem with not having middleware domains so I followed this guide (with @Jiri-Kremser help): #10048
So if anyone will change something in a way that it will call new URL he'll have to do the same steps.

why was there a problem with not having mw domains in regards to jms?
you mentioned some solution/steps above taken from the #10048, worth to document this with more detail, this is not very clear how to solve it from reading the comment. (at least to me :smiley: )
also, since #10148, I believe some code can be erased from controllers due to adding reusable shared code. please check it. thanks

why was there a problem with not having mw domains in regards to jms?

I had to record brand new VCR and these tests checks for mw domains and domain groups, so I had to reproduce same steps as in #10048 it's not required to use same steps when running this, just to clarify what to look at when refresher tests are failing.
The problem was that when VCR tests were recorded they used different feedId, and now when fetching resources recursive method is used so new calls were made but with bad feedId, so I removed whole VCR file, added mw domains and server groups as described in #10048 egain just to record new VCR with recursive resources fetch.
I'll check #10148 and see if I can simplify this PR.

@miq-bot assign @rubenvp8510
@rubenvp8510 please review this PR, thanks!

I tested it and works well, this looks good.

@karelhala there are a lot of commits in this PR. Some of them should not be here, such as "Fix tests" or "Fix styling" - because they are relevant internally in the PR, but meaningless once it's merged to master. The reason is that when someone will look at git history, seeing "fix tests" or "fix styling" will sound like the master branch needed fixing, while it was not the case here. So, it might be hard to fix it now in this PR, but looking forward please avoid such commits. Using 'git --amend' is very useful in cases like this.

I also validated this.
Few comments and questions. @jdoyleoss

is Messagings (in plural) a valid name? It doesn't sound like proper English to me, but I might be wrong on this.
It looks like the type (JMS queue/topic) is displayed always twice - in the name and in the type column. Do we want this duplicity? Is it always part of the name?

@karelhala fyi regarding #10461 - less code to write...

@abonas

is Messagings (in plural) a valid name? It doesn't sound like proper English to me, but I might be wrong on this.

It felt strange for me as well, but it looks like messaging's plural is messagings

It looks like the type (JMS queue/topic) is displayed always twice - in the name and in the type column. Do we want this duplicity? Is it always part of the name?

This is because name of topic/queue in wildfly is JMS queue [TypeOfMessaging] and type is JMS Queue/Topic at least this is the information I receive from inventory. We could parse the type out of the name if we want thou.

Edit tags operation yields several errors

@abonas
is Messagings (in plural) a valid name? It doesn't sound like proper English to me, but I might be wrong on this.
It felt strange for me as well, but it looks like messaging's plural is messagings

ok lets keep it :smiley:

It looks like the type (JMS queue/topic) is displayed always twice - in the name and in the type column. Do we want this duplicity? Is it always part of the name?
This is because name of topic/queue in wildfly is JMS queue [TypeOfMessaging] and type is JMS Queue/Topic at least this is the information I receive from inventory. We could parse the type out of the name if we want thou.

for now leave it as is, we can always do it later

tags work as expected now. there's some schema related failure, please have a look

ok, all tests pass, good! @rubenvp8510 do you have any comments on the PR or it can be moved to miq?

@karelhala please remove the wip signs.

@miq-bot add_label wip

@miq-bot remove_label wip

@miq-bot assign @chessbyte

@karelhala please rebase/fix conflicts as needed. thank you

@abonas rebasing was quite a challenge since domains were changing similar files.

@karelhala I'm getting this error when refreshing after applying this PR: "unknown attribute 'feed' for ManageIQ::Providers::Hawkular::MiddlewareManager::MiddlewareMessaging. "
please check

That's because I had to add feed to migration script. I've added it to creation script, but thinking about it that might be a bad idea, because when someone already did tried this he'd have to rake db:rollback STEP=someNumber and rake db:update.
What do you think? Might be better to create new db update which will add feed to db, or keeping it this way?

That's because I had to add feed to migration script. I've added it to creation script, but thinking about it that might be a bad idea, because when someone already did tried this he'd have to rake db:rollback STEP=someNumber and rake db:update.
What do you think? Might be better to create new db update which will add feed to db, or keeping it this way?

since this wasn't merged yet, I'd go with putting them in the same migration script introduced in this PR so it will be merged as "one piece". I'll rerun the necessary scripts and retry.

LGTM except for one comment.

@Fryguy please review, or, if you are overloaded, @chessbyte please assign someone for review/merge? it contains schema changes and we'd like to complete with those asap. thank you

The UI coode looks good. But this needs to be tested in the UI.
@abonas : is this tested? or can you give me access to a provider where I can test this? @karelhala? @Jiri-Kremser ?

The UI coode looks good. But this needs to be tested in the UI.
@abonas : is this tested? or can you give me access to a provider where I can test this? @karelhala? @Jiri-Kremser ?

I tested it in UI last week


Is there demand for Kaleidoscope csv format Compatibility?

In our work we have found Kaleidoscope csv outputs to be very convenient to work with since it consolidates the ability to use a powerful software to quickly look at a lot of spectrograms of an audio dataset, as well as conveniently process a large csv with all of the annotations using Pandas, R, or excel.
We modified the deprecated BirdNET-Lite repo with this functionality, and we will likely rework our pipeline a bit to use these new models, but I am curious if there would be any demand for us to add an extra "rtype" parameter 'kaleidoscope' that produces a csv in that format on this repo? Not sure what the intersection of people that use Kaleidoscope and people that use BirdNET looks like.
If this git issue gets any traction, I can submit a PR request for this functionality.

Sounds very interesting!
I use Kaleidoscope a lot, since it allows to go through a lot of spectrograms in little time.
So, I would be interested in such a possibility.
However, we also plan to use the R output format for further processing in R.
Just an idea:

would it make sense to have an rtype option to simultaneously output TWO formats: Kaleidoscope csv and an R-compatible csv ?

@DD4WH I am not all that familiar with an R-compatible format. My understanding is that R can build dataframes off of any arbitrary csv file. Which would lead me to believe that it should work just fine with a csv file format that is compatible with Kaleidoscope. Are you referring to some specific kind of R library that you use?

Sorry for not being clear enough . . .
There is already an output option by BirdNET-Analyzer with --rtype 'r'.
For my -probably very specific purposes- it would be nice to have an option to output a csv compatible with Kaleidoscope and additionally the already implemented files of the --rtype 'r'

I would be open to it, if there is a need, we should support it. @JacobGlennAyers can you move ahead and make a PR as I am not very familiar with the Kaleidoscope format? If you do, please make sure to add the format to segments.py too. Thanks!

@DD4WH From my skim over this codebase that I am still getting used to, it would probably be tricky to handle two kinds of outputs. In cases like this, I usually write a small, personal Pandas script that can convert between different formats. I have done this several times for the sake of converting from the Raven format to the Kaleidoscope format.

Feel free to write up a small template of how the ideal output would look like here in the comments, I'd be happy to include it in the code if you think it would make things easier.

I don't have an example on the machine I am on right now, though I can provide this screenshot from my PyHa repository:

Ok, so it seems like we need id, folder, in file, channel, offset, duration and manual id. Do you know if manual id is required? Can we call it BirdNET ID or similar?

@kahst It seems that there is quite a bit more information (column names) that we don't normally use that is compatible with Kaleidoscope. My educated guess is that you wouldn't get the same functionality renaming the MANUAL ID csv column to BirdNET ID.

My educated guess is that you wouldn't get the same functionality renaming the MANUAL ID csv column to BirdNET ID.

This is true in my experience.  You have to use their "names".

Support was added by #77.


Issue with document intelligence and frontend interraction

Description
Bug Summary: Since the api keys have been changed, the frontend is not able to use the analyze route.
The swagger route (/apidocs) is able to use it correctly though.
Through some modification to the backend i was able to confirm that the pdf document could be created by the blob storage and was in fact correctly created. I dont have much more information on why it doesnt work.
Steps to Reproduce:
Use the frontend to send an image to the analyze route ( upload a picture through the capture page and click on the submit button
or use this code snippet to send a picture
Expected Behavior:
The analyze route sends a json in the format of the expected.json (present in the pipeline repo)
Actual Behavior:
An error is sent

@Youllou I can't reproduce. Do you still observe this on the main branch while using the latest changes in https://github.com/ai-cfia/fertiscan-frontend/pull/220?

From the tests I've been doing, the front-end seems to fail after the analysis.
@Youllou The route that fails is POST /inspection. Can you confirm this on your end?
If that's the case it'd be helpful to see what you're sending to the backend.

From the tests I've been doing, the front-end seems to fail after the analysis.
@Youllou The route that fails is POST /inspection. Can you confirm this on your end?
If that's the case it'd be helpful to see what you're sending to the backend.

If there is a problem with /inspection, we should make another issue. And close this one once there's a clear confirmation that /analyze is working as expected.

Can't reproduce. Closing this. Please @Youllou, reopen if the issue arises again.


2 agent scenarios

Hi I was wondering if it's possible to spawn a single human and a single robot for certain scenarios to evaluate side preference for passing, overtaking and crossing? I was trying to do that using the example model and it seems that the robot does well in approaching the goal but lingers around the last 0.5m of its goal. I've set sim.human_num = 1 and spawned the robot and human directly opposite on a circle for passing.

Yes, it is possible. You can just change the human number and starting & goal positions of each agent in the simulator.
It sounds like you are in the right direction, perhaps the model just needs more training or some hyperparameter tuning (including reward shaping) to get rid of the lingering behavior.

Thanks for the recommendation. Any idea which values I should start changing? I noticed that the difference between DSRNN and other papers is that DSRNN's reward function has much higher magnitudes.

You can try things out and do experiments to test your hypothesis.
In fact, we could have normalized the reward to [-1, 1]. Feel free to try this.


Feature/add lsuffix rsuffix to join

Description and related tickets, documents
Reviewers: @nhanitvn @Huandao0812 @hai-adatao
Breaking changes & backward compatible issues
Is this PR a breaking change or has backward compatible issue (e.g: changes in API names, interfaces, signature / remove something...)?
If yes, please state out what and tag, and if it can possibly affect BA, please tag aht, tuananh, khangpham and baonguyen
How to test
Describe how this PR is tested. In case manual testing is required, describe how to do so.
PR Progress
Make sure all checkboxes below are checked before merged

[ ] Branch is in format prefix/description (see this)
[ ] Merge check has no conflicts. PR checks passed.
[ ] Code review is done.

@ducleminh Please help resolve conflicts and bring all common JoinHandler tests to ddf-test, JoinHandlerBaseSuite, unless it's spark specific


version variable gets auto resolved to latest even when set.

Hi,
I've been having issues with the version variable, whenever i specify 1.16.5 in my variable, it still resolves it to "LATEST" and corrupts my world saves. It has done this on 2 big worlds of mine and all the data is now corrupt.
This usually happens right after i update the docker image, if i start up a fully new one with the version also set to 1.16.5; it just downloads the specified version and works. is there any fix for this?
I'm running these docker images on my unraid server, using the docker tag: "latest".

I figured it out quite quickly; the variables in docker are case-sensitive in some cases. using the variable "VERSION" instead of "version" fixed my issue. I hope my mistakes help someone in the future :)


Update changelog for v1.2.0 release

This depends on #160 and #159.

This commit updates the CHANGELOG.md file for the upcoming v1.2.0 release.
Commit checklist

[X] Changelog
[X] Doc gen (make generate) - unaffected
[X] Integration test(s) - unaffected
[X] Acceptance test(s) - unaffected
[X] Example(s) - unaffected

TFTR!

I retried a second release process. The first one raced with a batch operation in production (which grabbed the cluster's lock), causing one of the tests to fail when updating the custom CA.


Modify Perfect Value function into Library function

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

@sairish2001 Please assign me this issue

Go ahead
Deadline 26 June,2021

any updates @jasminekaur-prog ?

already did it issue #5298

ok sorry for inconvenience

@arpit456jain No problem


memory overflow problem

hi, i tested your plugin and its great but i worked your example, it uses 65 mb ram in normally but when i clicked to "start service" and "start task" over and over, The memory usage increased exponentially with each click. and it reached to 250 mb. So it seems a bug. Service should only work once.

Hello @yakupbaser
Yes I see this too.
I will work on that bug
Thank you :)

I published a new version (0.1.5), from my testing it now fixed.
Let me know if you see any problem
BTW, now if you try to start the task multiple times it will throw an error, the same for stoping the task.

it seems fixed, thanks. If you want to make the plugin that ios version. I can give a teamviever/anydesk remote connection from my mac and i can plug a real ios device too. 👍

Thank you for your offer but I read about foreground services in IOS and it is not possible to run foreground service like in android,
in IOS you can run a foreground service under specific conditions like a video player service

hmm thanks again for your plugin.


Support positional and keyword arguments when adding widgets from dictionary

Pull Request Test Coverage Report for Build<PHONE_NUMBER>5
Details

3 of 3 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.04%) to 73.602%

Totals

Change from base Build<PHONE_NUMBER>3:
0.04%

Covered Lines:
987

Relevant Lines:
1341

💛  - Coveralls


FileServicesClient.SetServiceProperties does not work from around 3/8/2022

Bug Report

import path of package in question, e.g. .../services/compute/mgmt/2018-06-01/compute
SDK version e.g. master, latest, 18.1.0

Specify the exact commit if possible; one way to get this is the REVISION
column output by dep status "github.com/Azure/azure-sdk-for-go.

output of go version
go version go1.17 linux/amd64

What happened?
From around 3/8/2022, we found FileServicesClient.SetServiceProperties does not work, related code:
https://github.com/Azure/azure-sdk-for-go/blob/9dcffb6f614851ffd59ff8f5d6c9f26af0fe944e/services/storage/mgmt/2021-02-01/storage/fileservices.go#L227)

return error is like following:
GRPC error: rpc error: code = Internal desc = failed to ensure storage account: storage.FileServicesClient#SetServiceProperties: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidXmlDocument" Message="XML specified is not syntactically valid.\nRequestId:a8dbc76c-d01a-005a-7bc9-32f63b000000\nTime:2022-03-08T08:47:57.6391029Z"

You could repro the error by GetServiceProperties and then SetServiceProperties, e.g.
            prop, err := az.FileClient.GetServiceProperties(resourceGroup, accountName)
            if err != nil {
                return "", "", err
            }
            if prop.FileServicePropertiesProperties == nil {
                return "", "", fmt.Errorf("FileServicePropertiesProperties of account(%s), resource group(%s) is nil", accountName, resourceGroup)
            }
            prop.FileServicePropertiesProperties.ShareDeleteRetentionPolicy = &storage.DeleteRetentionPolicy{Enabled: to.BoolPtr(false)}
            if _, err := az.FileClient.SetServiceProperties(resourceGroup, accountName, prop); err != nil {
                return "", "", err
            }

related code snippet: https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/4e25bd8c909e639304ac249399e69759f5add2ca/vendor/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_storageaccount.go#L264-L276

What did you expect or want to happen?
How can we reproduce it?
Anything we should know about your environment.

Hi @andyzhangx , thank you for opening this issue.
Based on the error message you posted here, it is very likely that the service returns this error and the SDK is just bubbling it up. Therefore is it possible you could share some more information about the request you are making here as well as the response so that we could better identify if this is a service issue?

@ArcturusZhang thanks, do you know how to open debug mode with this go sdk? I actually looped azure file service team to take a look, while they said it's sdk issue, if we could get some back and forth debug info, that could narrow down the issue quickly, thanks.

If you enable debug logging it will dump the request body so we can see if the XML is malformed.  See here for instructions.

If you enable debug logging it will dump the request body so we can see if the XML is malformed. See here for instructions.

@jhendrixMSFT thanks, I have set AZURE_GO_SDK_LOG_LEVEL=DEBUG and below are all the go sdk debug logs:
https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_azurefile-csi-driver/939/pull-azurefile-csi-driver-e2e/1503653286425661440/build-log.txt
X-Content-Type-Options: nosniff
Date: Tue, 15 Mar 2022 09:18:43 GMT
Content-Type: application/json
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1144
{"error":{"code":"InvalidXmlDocument","message":"XML specified is not syntactically valid.\nRequestId:962f07da-001a-004d-504d-38d7e8000000\nTime:2022-03-15T09:18:43.1796898Z"}}
E0315 09:18:43.202205       1 utils.go:81] GRPC error: rpc error: code = Internal desc = failed to ensure storage account: storage.FileServicesClient#SetServiceProperties: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidXmlDocument" Message="XML specified is not syntactically valid.\nRequestId:962f07da-001a-004d-504d-38d7e8000000\nTime:2022-03-15T09:18:43.1796898Z"
I0315 09:18:43.203022       1 utils.go:76] GRPC call: /csi.v1.Controller/CreateVolume
I0315 09:18:43.203039       1 utils.go:77] GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-374e77f0-cd2a-41bd-8203-0d65d40f833a","parameters":{"csi.storage.k8s.io/pv/name":"pvc-374e77f0-cd2a-41bd-8203-0d65d40f833a","csi.storage.k8s.io/pvc/name":"pvc-pqzw9","csi.storage.k8s.io/pvc/namespace":"azurefile-1351","disableDeleteRetentionPolicy":"true","secretNamespace":"kube-system","skuName":"Standard_LRS","useDataPlaneAPI":"true"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","uid=0","gid=0","mfsymlinks","cache=strict","nosharesock","vers=3.1.1"]}},"access_mode":{"mode":7}}]}
(2022-03-15T09:18:43.2032397Z) INFO: REQUEST: GET https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts?api-version=2021-02-01
User-Agent: Go/go1.17.7 (amd64-linux) go-autorest/v14.2.1 file.csi.azure.com/e2e-31a3db16c891322ccf4811cfe6fd905860786f73 gc/go1.17.7 (amd64-linux) e2e-test
Authorization: **REDACTED**
(2022-03-15T09:18:43.3403763Z) INFO: RESPONSE: 200 https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts?api-version=2021-02-01
X-Ms-Request-Id: 6499138f-a2c1-4450-8880-18d1b5fef59f
X-Content-Type-Options: nosniff
Cache-Control: no-cache
Content-Type: application/json
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
X-Ms-Correlation-Request-Id: 68b3aa86-355e-4011-966f-8d4554da165d
X-Ms-Routing-Request-Id: SOUTHCENTRALUS:20220315T091843Z:68b3aa86-355e-4011-966f-8d4554da165d
Pragma: no-cache
Expires: -1
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11955
Date: Tue, 15 Mar 2022 09:18:43 GMT
Vary: Accept-Encoding
{"value":[{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f01838d58b72d4f45834814","name":"f01838d58b72d4f45834814","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:15:21.1146728Z","key2":"2022-03-15T09:15:21.1146728Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:15:21.1303305Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:15:21.1303305Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:15:21.0365404Z","primaryEndpoints":{"dfs":"https://f01838d58b72d4f45834814.dfs.core.windows.net/","web":"https://f01838d58b72d4f45834814.z21.web.core.windows.net/","blob":"https://f01838d58b72d4f45834814.blob.core.windows.net/","queue":"https://f01838d58b72d4f45834814.queue.core.windows.net/","table":"https://f01838d58b72d4f45834814.table.core.windows.net/","file":"https://f01838d58b72d4f45834814.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f094889c01a7e4f07891fbf","name":"f094889c01a7e4f07891fbf","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:10:38.7682132Z","key2":"2022-03-15T09:10:38.7682132Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:10:38.7839685Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:10:38.7839685Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:10:38.6744640Z","primaryEndpoints":{"dfs":"https://f094889c01a7e4f07891fbf.dfs.core.windows.net/","web":"https://f094889c01a7e4f07891fbf.z21.web.core.windows.net/","blob":"https://f094889c01a7e4f07891fbf.blob.core.windows.net/","queue":"https://f094889c01a7e4f07891fbf.queue.core.windows.net/","table":"https://f094889c01a7e4f07891fbf.table.core.windows.net/","file":"https://f094889c01a7e4f07891fbf.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f09c5b05ff6db47f3a12788","name":"f09c5b05ff6db47f3a12788","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:17:25.9287201Z","key2":"2022-03-15T09:17:25.9287201Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:17:25.9443476Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:17:25.9443476Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:17:25.8192930Z","primaryEndpoints":{"dfs":"https://f09c5b05ff6db47f3a12788.dfs.core.windows.net/","web":"https://f09c5b05ff6db47f3a12788.z21.web.core.windows.net/","blob":"https://f09c5b05ff6db47f3a12788.blob.core.windows.net/","queue":"https://f09c5b05ff6db47f3a12788.queue.core.windows.net/","table":"https://f09c5b05ff6db47f3a12788.table.core.windows.net/","file":"https://f09c5b05ff6db47f3a12788.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Standard_GRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f0c23f58ebf124605a2cb08","name":"f0c23f58ebf124605a2cb08","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:07:33.6414472Z","key2":"2022-03-15T09:07:33.6414472Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":true,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:07:33.6570802Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:07:33.6570802Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:07:33.5320714Z","primaryEndpoints":{"dfs":"https://f0c23f58ebf124605a2cb08.dfs.core.windows.net/","web":"https://f0c23f58ebf124605a2cb08.z21.web.core.windows.net/","blob":"https://f0c23f58ebf124605a2cb08.blob.core.windows.net/","queue":"https://f0c23f58ebf124605a2cb08.queue.core.windows.net/","table":"https://f0c23f58ebf124605a2cb08.table.core.windows.net/","file":"https://f0c23f58ebf124605a2cb08.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available","secondaryLocation":"northcentralus","statusOfSecondary":"available"}},{"sku":{"name":"Standard_RAGRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f0fd99eacfe964435a918b2","name":"f0fd99eacfe964435a918b2","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:14:51.0049152Z","key2":"2022-03-15T09:14:51.0049152Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:14:51.0206183Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:14:51.0206183Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:14:50.9111650Z","primaryEndpoints":{"dfs":"https://f0fd99eacfe964435a918b2.dfs.core.windows.net/","web":"https://f0fd99eacfe964435a918b2.z21.web.core.windows.net/","blob":"https://f0fd99eacfe964435a918b2.blob.core.windows.net/","queue":"https://f0fd99eacfe964435a918b2.queue.core.windows.net/","table":"https://f0fd99eacfe964435a918b2.table.core.windows.net/","file":"https://f0fd99eacfe964435a918b2.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available","secondaryLocation":"northcentralus","statusOfSecondary":"available","secondaryEndpoints":{"dfs":"https://f0fd99eacfe964435a918b2-secondary.dfs.core.windows.net/","web":"https://f0fd99eacfe964435a918b2-secondary.z21.web.core.windows.net/","blob":"https://f0fd99eacfe964435a918b2-secondary.blob.core.windows.net/","queue":"https://f0fd99eacfe964435a918b2-secondary.queue.core.windows.net/","table":"https://f0fd99eacfe964435a918b2-secondary.table.core.windows.net/"}}},{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f12840513cf524d56a82f08","name":"f12840513cf524d56a82f08","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:03:49.4200528Z","key2":"2022-03-15T09:03:49.4200528Z"},"privateEndpointConnections":[{"id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f12840513cf524d56a82f08/privateEndpointConnections/f12840513cf524d56a82f08.0b78514f-9571-4f3b-b0e0-7135c327ebde","name":"f12840513cf524d56a82f08.0b78514f-9571-4f3b-b0e0-7135c327ebde","type":"Microsoft.Storage/storageAccounts/privateEndpointConnections","properties":{"provisioningState":"Succeeded","privateEndpoint":{"id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Network/privateEndpoints/f12840513cf524d56a82f08-pvtendpoint"},"privateLinkServiceConnectionState":{"status":"Approved","description":"Auto-Approved","actionRequired":"None"}}}],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"largeFileSharesState":"Enabled","networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Deny"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:03:49.4357374Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:03:49.4357374Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:03:49.3106849Z","primaryEndpoints":{"dfs":"https://f12840513cf524d56a82f08.dfs.core.windows.net/","web":"https://f12840513cf524d56a82f08.z21.web.core.windows.net/","blob":"https://f12840513cf524d56a82f08.blob.core.windows.net/","queue":"https://f12840513cf524d56a82f08.queue.core.windows.net/","table":"https://f12840513cf524d56a82f08.table.core.windows.net/","file":"https://f12840513cf524d56a82f08.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f3503338c5857410aa79cd5","name":"f3503338c5857410aa79cd5","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:17:05.4284513Z","key2":"2022-03-15T09:17:05.4284513Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:17:05.4440369Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:17:05.4440369Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:17:05.3034903Z","primaryEndpoints":{"dfs":"https://f3503338c5857410aa79cd5.dfs.core.windows.net/","web":"https://f3503338c5857410aa79cd5.z21.web.core.windows.net/","blob":"https://f3503338c5857410aa79cd5.blob.core.windows.net/","queue":"https://f3503338c5857410aa79cd5.queue.core.windows.net/","table":"https://f3503338c5857410aa79cd5.table.core.windows.net/","file":"https://f3503338c5857410aa79cd5.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Premium_LRS","tier":"Premium"},"kind":"FileStorage","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f41161eba21a64056bd594f","name":"f41161eba21a64056bd594f","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:10:16.4868173Z","key2":"2022-03-15T09:10:16.4868173Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"largeFileSharesState":"Enabled","networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:10:16.5025016Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:10:16.5025016Z"}},"keySource":"Microsoft.Storage"},"provisioningState":"Succeeded","creationTime":"2022-03-15T09:10:16.3774545Z","primaryEndpoints":{"file":"https://f41161eba21a64056bd594f.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f454026354d704a42a1e73d","name":"f454026354d704a42a1e73d","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:05:40.7183099Z","key2":"2022-03-15T09:05:40.7183099Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:05:40.7339297Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:05:40.7339297Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:05:40.6245057Z","primaryEndpoints":{"dfs":"https://f454026354d704a42a1e73d.dfs.core.windows.net/","web":"https://f454026354d704a42a1e73d.z21.web.core.windows.net/","blob":"https://f454026354d704a42a1e73d.blob.core.windows.net/","queue":"https://f454026354d704a42a1e73d.queue.core.windows.net/","table":"https://f454026354d704a42a1e73d.table.core.windows.net/","file":"https://f454026354d704a42a1e73d.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f53b8f0131b144e7e8cf605","name":"f53b8f0131b144e7e8cf605","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:18:24.3043432Z","key2":"2022-03-15T09:18:24.3043432Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:18:24.3043432Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:18:24.3043432Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:18:24.1949546Z","primaryEndpoints":{"dfs":"https://f53b8f0131b144e7e8cf605.dfs.core.windows.net/","web":"https://f53b8f0131b144e7e8cf605.z21.web.core.windows.net/","blob":"https://f53b8f0131b144e7e8cf605.blob.core.windows.net/","queue":"https://f53b8f0131b144e7e8cf605.queue.core.windows.net/","table":"https://f53b8f0131b144e7e8cf605.table.core.windows.net/","file":"https://f53b8f0131b144e7e8cf605.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f6660ceead91147718e4ae3","name":"f6660ceead91147718e4ae3","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:17:45.5695515Z","key2":"2022-03-15T09:17:45.5695515Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:17:45.5695515Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:17:45.5695515Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:17:45.4445795Z","primaryEndpoints":{"dfs":"https://f6660ceead91147718e4ae3.dfs.core.windows.net/","web":"https://f6660ceead91147718e4ae3.z21.web.core.windows.net/","blob":"https://f6660ceead91147718e4ae3.blob.core.windows.net/","queue":"https://f6660ceead91147718e4ae3.queue.core.windows.net/","table":"https://f6660ceead91147718e4ae3.table.core.windows.net/","file":"https://f6660ceead91147718e4ae3.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f69c7d1e0b9574ac5af6176","name":"f69c7d1e0b9574ac5af6176","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:18:04.8823249Z","key2":"2022-03-15T09:18:04.8823249Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:18:04.8823249Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:18:04.8823249Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:18:04.7729447Z","primaryEndpoints":{"dfs":"https://f69c7d1e0b9574ac5af6176.dfs.core.windows.net/","web":"https://f69c7d1e0b9574ac5af6176.z21.web.core.windows.net/","blob":"https://f69c7d1e0b9574ac5af6176.blob.core.windows.net/","queue":"https://f69c7d1e0b9574ac5af6176.queue.core.windows.net/","table":"https://f69c7d1e0b9574ac5af6176.table.core.windows.net/","file":"https://f69c7d1e0b9574ac5af6176.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Standard_RAGRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f8608d75b997f4be591a203","name":"f8608d75b997f4be591a203","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:09:50.0179306Z","key2":"2022-03-15T09:09:50.0179306Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:09:50.0179306Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:09:50.0179306Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:09:49.9084912Z","primaryEndpoints":{"dfs":"https://f8608d75b997f4be591a203.dfs.core.windows.net/","web":"https://f8608d75b997f4be591a203.z21.web.core.windows.net/","blob":"https://f8608d75b997f4be591a203.blob.core.windows.net/","queue":"https://f8608d75b997f4be591a203.queue.core.windows.net/","table":"https://f8608d75b997f4be591a203.table.core.windows.net/","file":"https://f8608d75b997f4be591a203.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available","secondaryLocation":"northcentralus","statusOfSecondary":"available","secondaryEndpoints":{"dfs":"https://f8608d75b997f4be591a203-secondary.dfs.core.windows.net/","web":"https://f8608d75b997f4be591a203-secondary.z21.web.core.windows.net/","blob":"https://f8608d75b997f4be591a203-secondary.blob.core.windows.net/","queue":"https://f8608d75b997f4be591a203-secondary.queue.core.windows.net/","table":"https://f8608d75b997f4be591a203-secondary.table.core.windows.net/"}}},{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f8fe5403643e04730851d46","name":"f8fe5403643e04730851d46","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:08:02.2667762Z","key2":"2022-03-15T09:08:02.2667762Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:08:02.2823973Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:08:02.2823973Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:08:02.1574801Z","primaryEndpoints":{"dfs":"https://f8fe5403643e04730851d46.dfs.core.windows.net/","web":"https://f8fe5403643e04730851d46.z21.web.core.windows.net/","blob":"https://f8fe5403643e04730851d46.blob.core.windows.net/","queue":"https://f8fe5403643e04730851d46.queue.core.windows.net/","table":"https://f8fe5403643e04730851d46.table.core.windows.net/","file":"https://f8fe5403643e04730851d46.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Premium_LRS","tier":"Premium"},"kind":"FileStorage","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/fc1fa1cd8292b466caaa6fb","name":"fc1fa1cd8292b466caaa6fb","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:14:15.7076881Z","key2":"2022-03-15T09:14:15.7076881Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"largeFileSharesState":"Enabled","networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:14:15.7232556Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:14:15.7232556Z"}},"keySource":"Microsoft.Storage"},"provisioningState":"Succeeded","creationTime":"2022-03-15T09:14:15.5983262Z","primaryEndpoints":{"file":"https://fc1fa1cd8292b466caaa6fb.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Premium_LRS","tier":"Premium"},"kind":"FileStorage","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/fe9078d98b3c54da5b91411","name":"fe9078d98b3c54da5b91411","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"account":"azurefile-test","k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:02:50.9506514Z","key2":"2022-03-15T09:02:50.9506514Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"largeFileSharesState":"Enabled","networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:02:50.9664227Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:02:50.9664227Z"}},"keySource":"Microsoft.Storage"},"provisioningState":"Succeeded","creationTime":"2022-03-15T09:02:50.8413063Z","primaryEndpoints":{"file":"https://fe9078d98b3c54da5b91411.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}},{"sku":{"name":"Premium_LRS","tier":"Premium"},"kind":"FileStorage","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/fffba57d985164caf8e0e3b","name":"fffba57d985164caf8e0e3b","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:11:56.4874180Z","key2":"2022-03-15T09:11:56.4874180Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"largeFileSharesState":"Enabled","networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:11:56.4874180Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:11:56.4874180Z"}},"keySource":"Microsoft.Storage"},"provisioningState":"Succeeded","creationTime":"2022-03-15T09:11:56.3780097Z","primaryEndpoints":{"file":"https://fffba57d985164caf8e0e3b.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}}]}
I0315 09:18:43.342839       1 azure_storageaccount.go:233] azure - no matching account found, begin to create a new account f1a7a315d1945489ea6a4ba in resource group kubetest-eqlh8vfv, location: southcentralus, accountType: Standard_LRS, accountKind: StorageV2, tags: map[k8s-azure-created-by:azure]
I0315 09:18:43.342882       1 azure_storageaccount.go:254] set AllowBlobPublicAccess(false) for storage account(f1a7a315d1945489ea6a4ba)
(2022-03-15T09:18:43.3429571Z) INFO: REQUEST: PUT https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f1a7a315d1945489ea6a4ba?api-version=2021-02-01
Content-Type: application/json; charset=utf-8
User-Agent: Go/go1.17.7 (amd64-linux) go-autorest/v14.2.1 file.csi.azure.com/e2e-31a3db16c891322ccf4811cfe6fd905860786f73 gc/go1.17.7 (amd64-linux) e2e-test
Authorization: **REDACTED**
{"kind":"StorageV2","location":"southcentralus","properties":{"supportsHttpsTrafficOnly":true,"allowBlobPublicAccess":false,"minimumTlsVersion":"TLS1_2"},"sku":{"name":"Standard_LRS"},"tags":{"k8s-azure-created-by":"azure"}}
(2022-03-15T09:18:45.1681544Z) INFO: RESPONSE: 202 https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f1a7a315d1945489ea6a4ba?api-version=2021-02-01
Retry-After: 17
Date: Tue, 15 Mar 2022 09:18:45 GMT
Cache-Control: no-cache
Content-Type: text/plain; charset=utf-8
Strict-Transport-Security: max-age=31536000; includeSubDomains
Pragma: no-cache
X-Ms-Request-Id: cf3aa220-873b-4425-9472-95396f8df4e8
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
X-Ms-Routing-Request-Id: SOUTHCENTRALUS:20220315T091845Z:4b7e11f6-1014-4cbc-a49f-1e2daa496cad
Expires: -1
Location: https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/providers/Microsoft.Storage/locations/southcentralus/asyncoperations/cf3aa220-873b-4425-9472-95396f8df4e8?monitor=true&api-version=2021-02-01
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1187
X-Ms-Correlation-Request-Id: 4b7e11f6-1014-4cbc-a49f-1e2daa496cad
X-Content-Type-Options: nosniff
Content-Length: 0
(2022-03-15T09:18:45.1682257Z) INFO: WaitForCompletionRef: initial polling delay
(2022-03-15T09:18:45.1682323Z) INFO: DelayForBackoffWithCap: sleeping for 17s
(2022-03-15T09:19:02.1686412Z) INFO: REQUEST: GET https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/providers/Microsoft.Storage/locations/southcentralus/asyncoperations/cf3aa220-873b-4425-9472-95396f8df4e8?monitor=true&api-version=2021-02-01
Authorization: **REDACTED**
User-Agent: Go/go1.17.7 (amd64-linux) go-autorest/v14.2.1 file.csi.azure.com/e2e-31a3db16c891322ccf4811cfe6fd905860786f73 gc/go1.17.7 (amd64-linux) e2e-test
(2022-03-15T09:19:02.1985603Z) INFO: RESPONSE: 200 https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/providers/Microsoft.Storage/locations/southcentralus/asyncoperations/cf3aa220-873b-4425-9472-95396f8df4e8?monitor=true&api-version=2021-02-01
X-Ms-Routing-Request-Id: SOUTHCENTRALUS:20220315T091902Z:74bce417-2737-4654-9972-46830ce68977
Pragma: no-cache
Content-Type: application/json
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
X-Ms-Correlation-Request-Id: 74bce417-2737-4654-9972-46830ce68977
X-Content-Type-Options: nosniff
X-Ms-Request-Id: a08b1200-b48d-4e03-a2ac-c873b11d2f7d
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11954
Date: Tue, 15 Mar 2022 09:19:02 GMT
Expires: -1
Vary: Accept-Encoding
Cache-Control: no-cache
{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f1a7a315d1945489ea6a4ba","name":"f1a7a315d1945489ea6a4ba","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:18:43.6795889Z","key2":"2022-03-15T09:18:43.6795889Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:18:43.6795889Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:18:43.6795889Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:18:43.5701540Z","primaryEndpoints":{"dfs":"https://f1a7a315d1945489ea6a4ba.dfs.core.windows.net/","web":"https://f1a7a315d1945489ea6a4ba.z21.web.core.windows.net/","blob":"https://f1a7a315d1945489ea6a4ba.blob.core.windows.net/","queue":"https://f1a7a315d1945489ea6a4ba.queue.core.windows.net/","table":"https://f1a7a315d1945489ea6a4ba.table.core.windows.net/","file":"https://f1a7a315d1945489ea6a4ba.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}}
(2022-03-15T09:19:02.1989450Z) INFO: REQUEST: GET https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f1a7a315d1945489ea6a4ba?api-version=2021-02-01
User-Agent: Go/go1.17.7 (amd64-linux) go-autorest/v14.2.1 file.csi.azure.com/e2e-31a3db16c891322ccf4811cfe6fd905860786f73 gc/go1.17.7 (amd64-linux) e2e-test
Authorization: **REDACTED**
(2022-03-15T09:19:02.2384737Z) INFO: RESPONSE: 200 https://management.azure.com/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f1a7a315d1945489ea6a4ba?api-version=2021-02-01
Strict-Transport-Security: max-age=31536000; includeSubDomains
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json
Expires: -1
Vary: Accept-Encoding
X-Ms-Request-Id: ea420b5c-f77b-4e9a-95a6-f33e18b8a6ba
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
X-Ms-Correlation-Request-Id: 18e57c73-aa9e-4e29-a9b5-35f669e05298
X-Ms-Routing-Request-Id: SOUTHCENTRALUS:20220315T091902Z:18e57c73-aa9e-4e29-a9b5-35f669e05298
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11953
X-Content-Type-Options: nosniff
Date: Tue, 15 Mar 2022 09:19:02 GMT
{"sku":{"name":"Standard_LRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-eqlh8vfv/providers/Microsoft.Storage/storageAccounts/f1a7a315d1945489ea6a4ba","name":"f1a7a315d1945489ea6a4ba","type":"Microsoft.Storage/storageAccounts","location":"southcentralus","tags":{"k8s-azure-created-by":"azure"},"properties":{"keyCreationTime":{"key1":"2022-03-15T09:18:43.6795889Z","key2":"2022-03-15T09:18:43.6795889Z"},"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":false,"networkAcls":{"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:18:43.6795889Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2022-03-15T09:18:43.6795889Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2022-03-15T09:18:43.5701540Z","primaryEndpoints":{"dfs":"https://f1a7a315d1945489ea6a4ba.dfs.core.windows.net/","web":"https://f1a7a315d1945489ea6a4ba.z21.web.core.windows.net/","blob":"https://f1a7a315d1945489ea6a4ba.blob.core.windows.net/","queue":"https://f1a7a315d1945489ea6a4ba.queue.core.windows.net/","table":"https://f1a7a315d1945489ea6a4ba.table.core.windows.net/","file":"https://f1a7a315d1945489ea6a4ba.file.core.windows.net/"},"primaryLocation":"southcentralus","statusOfPrimary":"available"}}

Already start the internal conversation for this item. The issue seems to be a service problem so close the issue in GitHub.
Let me know if you have further questions.


KON-47 Use Interfaces Instead Of Inheritance

KoModifierProviderSequenceExt and KoModifierProviderSequenceExtTest are very long, but I have no idea how to meaningfully extract it.
I didn't extract KoKDocTagDeclaration and KoValuedKDocTagDeclaration to providers because in core it overrides its properties at constructor and it didn't work. I can try to get it out again, but I don't think it makes sense.

1, 2, 4 are fine.
We will get back to 3 after the weeked

lemonappdev.atlassian.net/browse/KON-47


Container Manager: unable to remove ".syno.bak" containers after updating

I successfully migrated my Container Manager to the new NVME volume this morning after the v3.0.40 update. When I ran the Update available button on one of the images with an update, it was successful at updating the container, but failed to remove the old version (with .syno.bak extension).

The container is up and running properly, I just fear that I'll start accumulating these .syno.bak containers with no way to remove them.
I see the btrfs snapshot file under /volume2/@docker/btrfs/subvolumes, and it looks like all those files are drwxr-xr-x root root permission/owner:

Do you have any idea what is preventing the removal of these snapshots? (I don't have Snapshot Replication or anything else set up on my ds920)

Container Manager has been the most difficult app to reliably move. I have experienced the issue you are having, but I thought it was fixed. Though I've not updated an image after moving Container Manager.
By coincidence right now I actually have an image with an update available. I'm currently doing a full system backup so I keep returning to the current state (with an image that has an update available) so I can do some testing.

What happens if run the following command?
docker system prune

If docker system prune doesn't work you could try:
btrfs subvolume delete /volume2/@docker/btrfs/subvolumes/f36d05e480b865b1e79b30e2960f11771b43f3fa97220ec03cd2fd3affdaa93

Assuming I typed the btrfs snapshot folder name correctly from your images.

I made sure to have all my containers running and then ran docker system prune, but it didn't come up with anything to remove, probably same as your run:
jeff@synology01:~$ sudo -s docker system prune
Password:
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - all dangling build cache

Are you sure you want to continue? [y/N] y
Deleted Images:
deleted:
deleted:
deleted:

Total reclaimed space: 0B

Only thing I noticed afterwards was that the other .bak container was now red, so it looks like it tried to remove it, but failed, and gave the same error on that container, pointing to its container/subvolume IDs as well.
I haven't had any luck with the btrfs subvolume delete yet, I get an error about "not a subvolume".
jeff@synology01:~$ sudo -s btrfs subvolume delete /volume2/@docker/btrfs/subvolumes/f36d05e480b865b1e79b30e2960f11771b43f3fa97220ec03cd2fd3affdaaf93
Password:
ERROR: not a subvolume: /volume2/@docker/btrfs/subvolumes/f36d05e480b865b1e79b30e2960f11771b43f3fa97220ec03cd2fd3affdaaf93

It exists under the /volume2/@docker/btrfs/subvolumes folder:
jeff@synology01:~$ sudo -s ls -l /volume2/@docker/btrfs/subvolumes/ | grep "\sf36.*"
drwxr-xr-x 1 root root 316 Feb  1 13:17 f36d05e480b865b1e79b30e2960f11771b43f3fa97220ec03cd2fd3affdaaf93
drwxr-xr-x 1 root root 258 Feb  1 13:16 f36d05e480b865b1e79b30e2960f11771b43f3fa97220ec03cd2fd3affdaaf93-init
jeff@synology01:~$

I ran the command to list subvolumes, but don't see that particular id in the list:
jeff@synology01:~$ sudo btrfs subvolume list -p /volume2/@docker/btrfs/subvolumes
ID 256 gen 64447 parent 5 top level 5 path @syno
ID 258 gen 64450 parent 256 top level 256 path docker
ID 259 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/65b30a711f41c80dd31bf16959c7eb12d9e9d6769e04bbf5b880d86efa02a7de
ID 260 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/497882d44e7974f8210e1bd57f7f39ab287562da242e8c7617c20dea30805638
ID 261 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/75c5d87b548520daecffcdec82b78601a01f0b1f39f77b36bfb76ce817e3ef9b
ID 262 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/3e91c330f918d8f3aa86eac30d157b8b073838856d160ad5bc9506688c47e19b
ID 263 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/0c87173cc8ac338efada4ce509981b2e0b20e697a3f9efeb8e5d159cbab566ab
ID 264 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/d7af2db11ac5a49350a40040ad42e033d3148cf6543ade4493bf64ff9988295a
ID 265 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/3611284bc24765380841f77e5131a3b50d0329d21050b19b85fcb0735837e398
ID 266 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/d7c74933f5df11f7b51d9619d3df84b62614b013657c2136e5b035e52a481f2b
ID 267 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/e13cd6d39eebe46aa451fec8c5ca0397c1b51e25d59e7fb28c4ac70afbf8775e-init
ID 268 gen 33561 parent 256 top level 256 path @docker/btrfs/subvolumes/e13cd6d39eebe46aa451fec8c5ca0397c1b51e25d59e7fb28c4ac70afbf8775e
ID 269 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/2c262e186e9fba3c88f2642554f0a96da2f242b878f14847ceabb07d81693d90
ID 270 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/ae3284cedee40b9bfcccf5fd1b3cac55c21f5c7d83572e7fae86c8bde7839b80
ID 271 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/e4f72e9b215dcdda7c2e77c8eeb5654da1d990826f3ca93c00a337e4ea9296bb
ID 272 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/b519b018b8467ba815a2ef20b1de2ae1ba4a5095a28e11fdf8e58048b9cda742
ID 273 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/1ed721906b06b99b7565398c93725ce0cc81121e815019d390ffc975753fe246
ID 274 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/900438c9c4eccf97ed0fa40614a2ba40f96c6bc60433338e58bee9a5d16b4059
ID 275 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/ae6b36adf8fa55db80d0ee222cd7d7038b57a62180f7982075d7f3d40fd89a49
ID 276 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/0fd93e24d6bf761dea53bf68219c015333c6c738f47188e0bf3b42bdb4dcd755
ID 277 gen 8316 parent 256 top level 256 path @docker/btrfs/subvolumes/75e4ff92e634818e0ff7638bbeaaf945463f7b5324af1fe6c6b3dda2d4009e8e-init
ID 278 gen 58310 parent 256 top level 256 path @docker/btrfs/subvolumes/75e4ff92e634818e0ff7638bbeaaf945463f7b5324af1fe6c6b3dda2d4009e8e
ID 279 gen 63702 parent 256 top level 256 path PlexMediaServer
ID 280 gen 50127 parent 256 top level 256 path @sharesnap
ID 281 gen 50127 parent 280 top level 280 path @sharesnap/docker
ID 283 gen 50118 parent 280 top level 280 path @sharesnap/PlexMediaServer

Not sure if any of this has to do with permissions on the @docker folder, I assume it preserved permissions when the script moved it from volume1 to volume2, but here's what I see right now:

I actually deleted them using WinSCP's gui. So when I wrote "This worked for me." I meant deleting the affected subvolumes worked for me.
Try this:
sudo rm -rf /volume2/@docker/btrfs/subvolumes/e13cd6d39eebe46aa451fec8c5ca0397c1b51e25d59e7fb28c4ac70afbf8775e-init
sudo rm -rf /volume2/@docker/btrfs/subvolumes/e13cd6d39eebe46aa451fec8c5ca0397c1b51e25d59e7fb28c4ac70afbf8775e

sudo rm -rf /volume2/@docker/btrfs/subvolumes/75e4ff92e634818e0ff7638bbeaaf945463f7b5324af1fe6c6b3dda2d4009e8e-init
sudo rm -rf /volume2/@docker/btrfs/subvolumes/75e4ff92e634818e0ff7638bbeaaf945463f7b5324af1fe6c6b3dda2d4009e8e

Ok, I've updated each of my container images, and manually removed those extraneous folders from the subvolumes, on a case-by-case basis.
I'm wondering if it has anything to do with ones that were created before the migration, and if they will be able to update next time. I'll give it a week or so and see if I can update again next time.

I'm fairly sure once the subvolumes that were moved from volume 1 have been deleted there won't be further issues when updating images or containers.
I'm going to try changing the script to not move the subvolumes. They're only snapshots and docker will create new snapshots when needed, so there's really no need for the script to move them (or copy them) to the new volume.

Also given that I have 641 items under subvolumes, and only have 11 containers, seems like there would be many that should not be needed anymore 😄
Is there any safe way of identifying subvolumes not in use under the @docker folder? Are they all fair game to remove without causing any issue for the existing / running containers?

Wow. I've only got 28 on volume 1, and 22 on volume 2.

The 28 were modified on 3 different dates: 18-03-2024, 27-02-2024 and 02-02-2024.
All 3 dates groups contain 1 or 2 subvolumes that end with -init.

I just stopped Container Manager (that has been moved to volume 2).
Deleted all 28 subvolumes on volume 1.
Started Container Manager.
Now all my containers have a red dot and the log has errors with "no file or directory" for each container's subvolume.

The missing subvolume names aren't subvolumes that exist on volume 2, except for 1 but that container is also broken.
So deleting all of them is NOT an option.
And I have to figure out why  some of my containers are looking for their subvolumes on volume 1.

There may be some practicality to compare the subvolume ids from the sudo btrfs subvolume list -p /volume2/@docker/btrfs/subvolumes command, with what is actually listed in the /volume2/@docker/btrfs/subvolumes directory. If you just completely removed everything from the subvolumes folder, that may lead to the condition you saw, with Container Manager complaining about "no file or directory".
If you can restore and try comparing the results of those, you could try removing anything that is not present on the list of btrfs subvolume list -p, as they might not be connected to anything in Container Manager.
Of course, it could be something on my side that resulted in over 600 folders being located under the subvolumes directory, since yours didn't seem to have nearly that many. Mine have dates going back to 2021, so maybe it's something that has been growing for a while on my side.

I checked my DS1821+ and it has a few containers at the moment but has 120 subvolumes. Apparently docker not cleaning up old data is a common problem.
I played around with some docker prune commands and lost some unused volumes I had in /volume1/docker
I tried shutting down container manager, deleting all subvolumes, then starting container manager but that just resulted in losing everything.
I missed the fact that your sudo btrfs subvolume list -p was actually
sudo btrfs subvolume list -p /volume2/@docker/btrfs/subvolumes
because the last half wrapped to the next line.
That command lists 11 subvolumes, and I can see which are the orphan subvolumes. I'm going to write a little test script to delete the orphan subvolumes and see what happens.

There may be some practicality to compare the subvolume ids from the sudo btrfs subvolume list -p /volume2/@docker/btrfs/subvolumes command, with what is actually listed in the /volume2/@docker/btrfs/subvolumes directory.

I've written a test script: https://github.com/007revad/Synology_app_mover/blob/test/docker_cleanup.sh
It prints 3 lists:

@docker/btrfs/subvolumes list
btrfs subvolume list
orphan subvolume list

If you run it and compare the @docker/btrfs/subvolumes list to the btrfs subvolume list the difference should be the same as the orphan subvolume list
Next I'll get the script to delete the orphans and see what happens to my containers.

Could you test the script by simply moving the orphans out, then you can validate the containers and move the folders back if something breaks.
Looking forward to cleaning up these orphans!

I've moved container manager and then updated an image which resulted in:

The image update getting stuck at 96%.
An old image with .syno.bak appended to it's name.
The same issue you had trying to delete old containers after updating them.

So I was able to test the script to delete the orphan containers, and it works great.
https://github.com/007revad/Synology_docker_cleanup

While writing the script I googled container manager "syno.bak" image only found 3 results.

Your reddit post.
Your reddit profile.
An interesting Chinese blog from 2 months ago: https://juejin.cn/post/7327353593890193459

From that Chinese blog you can find images that don't have the <latest> tag with:
sudo docker images

Then with the image id of an old image you can delete it with:
sudo docker rmi <image id>

I think I'll add this to the syno_docker_cleanup script.

v1.2.3 now also removes orphaned images
https://github.com/007revad/Synology_docker_cleanup/releases

Closing this issue, opened new issue on the Synology_docker_cleanup...
https://github.com/007revad/Synology_docker_cleanup/issues/4


external/zlib optimization

external/zlib has SSSE3 and NEON intrinsics for adler32 among other things.  Should RISC-V have similar changes?

yes. hilarious that i missed filing this one, despite the fact that i spend more time on dealing with zlib optimizations myself than with any of the other projects. (though that's probably why i missed this --- never seemed worth writing down because it was so obvious!)
but that also reminds me --- i need to hassle @pirama-arumuga-nainar about why the coverage build breaks with https://android-review.git.corp.google.com/c/platform/external/zlib/+/2359720 which would let us remove external/angle's duplicate copy of zlib which currently doesn't build for riscv64 at all...

Is it necessary to get patches into zlib? My understanding is that zlib-ng is a much friendlier target for arch-specific (e.g., RISC-V vector) optimizations.

chromium is the upstream[1] for both Chrome (unsurprisingly) and Android (so "we" [=Google] only have to deal with these optimizations once).
they in turn do sync with zlib but -- as you say -- the fact it's so hard to get arch-specific optimizations into zlib that's the cause of all these forks, so it's probably way easier to get it into chromium zlib directly. that was certainly true for the various arm64 optimizations that were the reason i moved Android over to chromium zlib.

you can check by looking at the METADATA file in an external/* project. here's zlib's for example: https://cs.android.com/android/platform/superproject/+/master:external/zlib/METADATA ... feel free to hassle me personally any time you find one that's incomplete. it's being worked on as part of SBOM concerns, but it's still a work in progress for most projects.

@enh-google Any hope of getting Chromium to switch over to zlib-ng as their upstream?

not likely, no. see https://aws.amazon.com/blogs/opensource/improving-zlib-cloudflare-and-comparing-performance-with-other-zlib-forks and https://dougallj.wordpress.com/2022/08/20/faster-zlib-deflate-decompression-on-the-apple-m1-and-x86/ for example.
@Adenilson has forgotten more than i ever knew about this.

Ideally local changes should have a merge request featured in either:

canonical zlib: https://github.com/madler/zlib/
zlib-ng: https://github.com/Dead2/zlib-ng

https://cs.android.com/android/platform/superproject/+/master:external/zlib/patches/README;l=41-43;drc=03559ccddad582c1e43af419d69d7a6f0639f8ca
This suggests to me that doing the work at zlib/-ng is a prerequisite.

Hello there, thanks for the add @enh-google.
It is a long story that happened along the last 6 years and to spare time the current position is:
a) We will keep maintaining Chromium zlib for the foreseeable future.
b) Feel free to send patches, the contribution workflow is through the Chromium process (https://chromium.googlesource.com/chromium/src/+/main/docs/contributing.md).
c) There is no hard requirement to submit patches to other forks.

A couple weeks ago, I've added a cmake build system to Chromium zlib, so it is possible today to quickly experiment (i.e. no need to checkout whole Chromium repository).
To checkout and build:
a) git clone https://chromium.googlesource.com/chromium/src/third_party/zlib/ chromium-zlib
b) cd chromium-zlib && mkdir build && cd build
c) cmake -DENABLE_SIMD_OPTIMIZATIONS=1 -DCMAKE_BUILD_TYPE=RELEASE ..
d) make -j
You should have at the end of it an executable named 'zlib_bench' that is used to collect performance data.
We generally use the snappy data corpus (https://github.com/google/snappy/tree/main/testdata).
For now the cmake build enables the optimizations only for linux@x86.
A few questions:
a) I wasn't aware that RISCV featured SIMD extensions. I wonder if our optimizations could be somehow ported to their vector instructions?
b) Is there anyone actually allocated to work on it?
c) Is there hardware available in the market to run code with such optimizations?
The 'c' part is particularly important if there is a plan to merge and keep maintaining the code in Chromium zlib.
To learn more about the optimizations, check (https://docs.google.com/presentation/d/1_XWLhF4FcTz_HGc4QqvVapkxMj2-QTOZNdyXvpy2oYU/edit?usp=sharing).
The reported gains are a bit outdated but the slide deck offers a high level overview on the subject.

Hi @adenilsoncavalcanti, nice work; thanks for the references.

a) I wasn't aware that RISCV featured SIMD extensions. I wonder if our optimizations could be somehow ported to their vector instructions?

The RISC-V vector extension (V-extension a.k.a. RVV) was ratified in November 2021:
https://github.com/riscv/riscv-v-spec
(It will be merged into the main ISA spec once the LaTeX -> AsciiDoc conversion is finished.)
We look forward to leveraging RVV in projects like Zlib.
Further uplifts for CRC-like things are anticipated from an RVV add-on (separate extension):
https://github.com/riscv/riscv-crypto/releases/tag/v20230206
The vclmul(h) instructions are of particular interest. This is still work-in-progress (not ratified), but we are confident the project is nearing completion.

b) Is there anyone actually allocated to work on it?

My team (at SiFive) is allocating resources to Zlib: we think it's important to adoption of RISC-V. Please don't hesitate to reach out over email for more information on logistics.

c) Is there hardware available in the market to run code with such optimizations?

To the best of my knowledge, there is not yet any publicly available silicon with RVV. (I recall that Alibaba/T-head released an SoC with a custom vector extension derived from a draft RVV, but I believe it's incompatible with what was eventually ratified.) My team does our performance work in (RTL/FPGA) simulation. Upstream QEMU has supported RVV for quite a while, so that's another option for developers who just care about functionality (vs. cycle accuracy).

@paul-walmsley-sifive since you asked about zlib-ng, you may want to check this:
https://github.com/zlib-ng/zlib-ng/issues/1217#issuecomment-1085339594