Synchronization timeouts (Dev Spaces)

About 4 hours ago we started receiving synchronization timeouts.
"Azds up" shows a timeout error after trying for 4 minutes.
Our k8 looks great and we have tried from two different networks with the same result.
Is there a problem with synchronization atm ?

No, we don't know of any issues that would cause synchronization to fail for you. Have any changes in your project caused more or bigger files to be synced?

Hi and thanks for the quick reply
Nothing has changed in the project but he project is kinda large (around 20000 files).
I just tried again and now it synced fine and finished the build in just 1.4 minutes.
We keep seeing this performance degregation and it always occurs around 16:30 UTC.
Our large project stops synchronizing, and the smaller ones takes alot longer,  but the next morning it work fine again.
We have tried 3 different networks from 3 different carriers with the same result.
We have run performance checks on our k8 but the node has plenty or resources left, and the application on the k8 runs smoth and responsive.
Does the synchronization happen just between our clients and our k8, or is there some other servers involved aswell?

Hi @Eneuman , it sounds like there may be a pattern of high network traffic around 16:30 in nearby resources in the data center where your cluster and Controller are hosted. Are you in the West Europe region? There are some other Dev Spaces resources involved in the sync operation, but we don't have anything scheduled to run near at that time.

@philon-msft  Yes, I am in the West Europe region.
And like clockwork, the synchronization just stoped working again :(
"Timed out after 42s trying to list azds pods"
Running "kubectl get pods" returns the pods in under a second.
At the moment, dev spaces is unusable for us after 16:30.

And now its back up and working without any changes on our side. This got to be a resource problem.

Synchornization stoped working again 6/11 14:45 UTC
Today 7/11 11:00 UTC it is still not working. Even small projects receive Synchronization Timout.
This is realy frustrating...
Cluster looks fine, no changes has been made.

Still not working :(

Still not working :(
azds down
Timed out after 1m trying to list services.

@Eneuman Sorry you're encountering this issue. :( We're having a look into this, and will keep you updated.

Adding @shpathak-msft, who is looking into this.

Thank you!
Let me know of there is anything I can do to help!
Feel free to access our cluster if you need to troubleshoot.
//Per

@Eneuman Could you please reach out to me at<EMAIL_ADDRESS>I have some logs from our backend service that I would like to share with you.

You got mail :)

Från: shpathak<EMAIL_ADDRESS>Skickat: Monday, November 9, 2020 8:00:19 PM
Till: Azure/dev-spaces<EMAIL_ADDRESS>Kopia: Per Bornsjö<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Ämne: Re: [Azure/dev-spaces] Synchronization timeouts (Dev Spaces) (#407)
@Eneumanhttps://github.com/Eneuman Could you please reach out to me at shpathak<EMAIL_ADDRESS>I have some logs from our backend service that I would like to share with you.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Azure/dev-spaces/issues/407#issuecomment-724211346, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4J5TMTFZP6MS26NAH7VKTSPA34HANCNFSM4RYWJA4A.

Thank you for reaching out and glad to know code synchronization is now working for you. I suppose you already know this, but I would suggest reaching out to AKS support if you continue to see the "could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1" error.

Thanks you for getting synchronization working again.
This has happend to use before and it's blocking us from working when it does. Are there any actions being taken to stop this issue from happen again?

@Eneuman Happy that we unblocked you! The issue you encountered is tricky to investigate because it is either transient, or it relies on us restarting the component on our side associated with your Azure Dev Spaces controller. This component didn't show specific errors itself that would help us finding the root cause of this. When this issue happened, this also didn't match to a deployment on our side or to code changes.
We know that Azure Dev Spaces relies on a complex codebase and can be unreliable, and this is part of the reason why we decided to move to another approach.
If I remember correctly, you investigated in the past transitioning to Bridge to Kubernetes but cannot do it presently because you rely on pod identity for your services. Is this correct? We're presently investigating adding support for pod identity and, if it's the only blocker for you, we would be happy to provide you with early bits as soon as we have something to share.

Yes we are interested in transitioning to Bridge to Kubernetes but two features are missing.
Pod Identity (as you said) and the ability to stop all other instances of a particular pod when we start debugging it.
We rely heavily on azure service bus and pub/sub, so when debugging a service, we don't want any other running instances, of the pod we are debugging,to receive the events.

@Eneuman Thanks for your quick reply.
All right, as of today we indeed expect the targeted service to be backed by only one pod, and fail if it's not the case. We should be able to automatically decrease the number of pods to 1 when debugging, and increase it back afterwards. I'm bumping the priority of this in our backlog so that we align it with our pod identity story.
Regarding pod identity, could you please confirm that your team members are using Windows machines to develop? From previous discussions, I believe that you're using Windows yourself.
In this case, the target OS is important as we might need for pod identity to have a specific implementation for each OS (because networking is done very differently in Windows compared to Linux, for example).

Yes, we are only using Windows machines and Visual Studio 2019 Pro.
Really looking forward to trying the new bits out :)

@Eneuman All right, thanks for confirming!
Sounds good, we'll keep in the loop as soon as we have a beta for pod identity.

@daniv-msft I think we just got this issue again :(
Can you please take a look and see if you can get it running again?
Enabling debugging...
Timed out after 1m trying to list azds pods.

Regards
Per

Hi @Eneuman, I've restarted some of the components on our side. Can you let me know if the issue keeps reproducing? Thanks!

Hi @amsoedal
Yes, now it running smooth again. Thanks you !

@amsoedal
Hi again, maby I was too fast there.
I still am having problems with atleast two of my pods.
One is giving me this error. Is this on my side ?
pods/klinikportalservice-757c5cc75b-cd6b7: BackOff: Back-off restarting failed container

Hi @Eneuman, sorry you're still having issues! Could you please run kubectl logs pods/klinikportalservice-757c5cc75b-cd6b7 -p (you might need to specify a container name with -c, but it should prompt you if necessary) to see what happened?

@amsoedal
Hmm, for some reason it seems like Azure ADO Cert is no longer valid. Maby a service issue?
kubectl logs -n dev pods/klinikportalservice-757c5cc75b-cd6b7 devspaces-build
2021-02-15T17:53:35.9003579Z | BuildService | TRACE | Starting build process.\nOperation context: <json>{"clientRequestId":"5a8260c7-6a35-4651-ba04-b960705bc226","correlationRequestId":"113b9760-35ec-4773-b952-a3733a6be493","requestId":null,"userSubscriptionId":null,"startTime":"2021-02-15T17:53:34.9226682Z","userAgent":"BuildSvc/1.0.20210206.1","requestHttpMethod":null,"requestUri":null,"apiVersion":null,"version":"1.0.20210206.1","requestHeaders":{},"loggingProperties":{"applicationName":"BuildService","deviceOperatingSystem":"Linux 5.4.0-1031-azure #32~18.04.1-Ubuntu SMP Tue Oct 6 10:03:22 UTC 2020","framework":".NET Core 4.6.28325.01"}}</json>
2021-02-15T17:53:35.9076895Z | BuildService | TRACE | Download workspace
2021-02-15T17:53:35.9646040Z | BuildService | TRACE | Log Handler started
2021-02-15T17:53:36.1176640Z | BuildService | WARNG | DownloadWorkspaceAsync failed with {"Message":"The SSL connection could not be established, see inner exception.","Data":{},"InnerException":{"ClassName":"System.Security.Authentication.AuthenticationException","Message":"The remote certificate is invalid according to the validation procedure.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequestasyncRequest)\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Net.Security.SslState.ThrowIfExceptional()\n   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)\n   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)\n   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)\n   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_1(IAsyncResult iar)\n   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233087,"Source":"System.Private.CoreLib","WatsonBuckets":null},"StackTrace":"   at Microsoft.Rest.RetryDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n   at Microsoft.Azure.DevSpaces.Common.Auth.Handlers.AuthErrorHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /app/common.auth/Handlers/AuthErrorHttpHandler.cs:line 47\n   at Microsoft.Azure.DevSpaces.Common.Auth.Handlers.ServiceClientCredentialsHttpHandler`1.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /app/common.auth/Handlers/ServiceClientCredentialsHttpHandler.cs:line 34\nat System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\n   at Microsoft.Azure.DevSpaces.ExecService.Client.WorkspaceOperations.DownloadSourceWithHttpMessagesAsync(String spaceName, String serviceName, String name, Dictionary`2 customHeaders, CancellationToken cancellationToken) in /app/execsvc.client.autogen/WorkspaceOperations.cs:line 161\n   at Microsoft.Azure.DevSpaces.ExecService.Client.WorkspaceOperationsExtensions.DownloadSourceAsync(IWorkspaceOperations operations, String spaceName, String serviceName, String name, CancellationToken cancellationToken) in /app/execsvc.client.autogen/WorkspaceOperationsExtensions.cs:line 49\n   at Microsoft.Azure.DevSpaces.Build.BuildClient.DownloadWorkspaceAsync() in /app/build/BuildClient.cs:line 154","HelpLink":null,"Source":"Microsoft.Rest.ClientRuntime","HResult":-2146233087}, retry.
2021-02-15T17:53:40.2424473Z | BuildService | ERROR | Logging handled exception: System.Net.Http.HttpRequestException: {"Message":"The SSL connection could not be established, see inner exception.","Data":{},"InnerException":{"ClassName":"System.Security.Authentication.AuthenticationException","Message":"The remote certificate is invalid according to the validation procedure.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming,Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Net.Security.SslState.ThrowIfExceptional()\n   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)\n   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)\n   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)\n   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_1(IAsyncResult iar)\n   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233087,"Source":"System.Private.CoreLib","WatsonBuckets":null},"StackTrace":"   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)\n   at System.Threading.Tasks.ValueTask`1.get_Result()\n   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n   at System.Threading.Tasks.ValueTask`1.get_Result()\n   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)\n   at System.Threading.Tasks.ValueTask`1.get_Result()\n   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n   at Microsoft.Azure.DevSpaces.Common.Auth.Handlers.ServiceClientCredentialsHttpHandler`1.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /app/common.auth/Handlers/ServiceClientCredentialsHttpHandler.cs:line 34\n  at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\n   at Microsoft.Azure.DevSpaces.Common.Logging.AzdsHttpTelemetryLogger.<>c__DisplayClass12_0.<<-ctor>b__0>d.MoveNext() in /app/common/Logging/Loggers/AzdsHttpTelemetryLogger.cs:line 44\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Threading.ThreadPoolWorkQueue.Dispatch()","HelpLink":null,"Source":"System.Net.Http","HResult":-2146233087}
2021-02-15T17:53:40.2727271Z | BuildService | ERROR | Unhandled AppDomain exception! System.Net.Http.HttpRequestException: The SSL connection could not be established, seeinner exception.
2021-02-15T17:53:40.2731763Z | BuildService | ERROR | AppDomain base exception details: System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
2021-02-15T17:53:40.2736504Z | BuildService | ERROR | Is AppDomain terminating? 'True'

Unhandled Exception: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.DevSpaces.Common.Auth.Handlers.ServiceClientCredentialsHttpHandler`1.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /app/common.auth/Handlers/ServiceClientCredentialsHttpHandler.cs:line 34
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.Azure.DevSpaces.Common.Logging.AzdsHttpTelemetryLogger.<>c__DisplayClass12_0.<<-ctor>b__0>d.MoveNext() in /app/common/Logging/Loggers/AzdsHttpTelemetryLogger.cs:line 44
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Thanks for attaching this! Can you try again now?

@amsoedal Yes, now its working again :) Thanks!

Great! Happy to hear it :)

@amsoedal Hi again. It looks like this problem has started to occure again :(
The logs shows the same problem as above.
Can you please apply your fix again?

Ater upgrading my kubernetes version, i'm now receiving this error message: This version of Kubernetes is no longer supported. Please upgrade your cluster to a supported Kubernetes version and retry when running azds up
Might be related to: https://github.com/Azure/dev-spaces/issues/416

@Eneuman Azure Dev Spaces is deprecated and is only supported till 1.18.* Kubernetes versions. After the upgrade what is the Kubernetes version of the cluster?
I'm looking into the SSL error and will update once it is resolved.

@rakeshvanga Will I be able to install a AKS with version 1.18 all the way until 2023 when Dev Spaces is being deprecated?

Hi @Eneuman ,
Yes, Dev Spaces will only support v1.18 in AKS until 2023. Dev Spaces is being retired and at this point, while it might remain used for legacy reasons, it should not be used on new clusters.
We understand some limitations prevented you and your team from moving to Bridge to Kubernetes and we are actively working on those scenarios to unblock you. Bridge to Kubernetes now supports debugging a service backed by multiple replicas and will soon support managed identities. I expect we'll have an experience for you to try in the next few weeks. Once both of those features are live, we're happy to help you and your team transition from legacy Dev Spaces to our improved offering Bridge to Kubernetes.
Please let us know if you have any further questions. Thank you.

@Eneuman, I've fixed the SSL error on our side, can you try now?

It's working fine now. Thank you!

@rakeshakkera Hi
The SSL error is unfortently back :( Can you please take a look?
2021-02-24T11:31:38.5161606Z | BuildService | TRACE | Starting build process.\nOperation context: <json>{"clientRequestId":"a588e693-177b-4c98-bcc9-6f18bef6f738","correlationRequestId":"2d811ee3-ab40-4465-8483-c3854ac93da0","requestId":null,"userSubscriptionId":null,"startTime":"2021-02-24T11:31:37.7190095Z","userAgent":"BuildSvc/1.0.20210206.1","requestHttpMethod":null,"requestUri":null,"apiVersion":null,"version":"1.0.20210206.1","requestHeaders":{},"loggingProperties":{"applicationName":"BuildService","deviceOperatingSystem":"Linux 5.4.0-1039-azure #41~18.04.1-Ubuntu SMP Mon Jan 18 14:00:01 UTC 2021","framework":".NET Core 4.6.28325.01"}}</json>
2021-02-24T11:31:38.5240909Z | BuildService | TRACE | Download workspace
2021-02-24T11:31:38.5578023Z | BuildService | TRACE | Log Handler started
2021-02-24T11:31:38.7225955Z | BuildService | WARNG | DownloadWorkspaceAsync failed with {"Message":"The SSL connection could not be established, see inner exception.","Data":{},"InnerException":{"ClassName":"System.Security.Authentication.AuthenticationException","Message":"The remote certificate is invalid according to the validation procedure.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\n   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Net.Security.SslState.ThrowIfExceptional()\n   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)\n   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)\n   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)\n   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_1(IAsyncResult iar)\n   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233087,"Source":"System.Private.CoreLib","WatsonBuckets":null},"StackTrace":"   at Microsoft.Rest.RetryDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n   at Microsoft.Azure.DevSpaces.Common.Auth.Handlers.AuthErrorHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /app/common.auth/Handlers/AuthErrorHttpHandler.cs:line 47\n   at Microsoft.Azure.DevSpaces.Common.Auth.Handlers.ServiceClientCredentialsHttpHandler`1.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /app/common.auth/Handlers/ServiceClientCredentialsHttpHandler.cs:line 34\n   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\n   at Microsoft.Azure.DevSpaces.ExecService.Client.WorkspaceOperations.DownloadSourceWithHttpMessagesAsync(String spaceName, String serviceName, String name, Dictionary`2 customHeaders, CancellationToken cancellationToken) in /app/execsvc.client.autogen/WorkspaceOperations.cs:line 161\n   at Microsoft.Azure.DevSpaces.ExecService.Client.WorkspaceOperationsExtensions.DownloadSourceAsync(IWorkspaceOperations operations, String spaceName, String serviceName, String name, CancellationToken cancellationToken) in /app/execsvc.client.autogen/WorkspaceOperationsExtensions.cs:line 49\n   at Microsoft.Azure.DevSpaces.Build.BuildClient.DownloadWorkspaceAsync() in /app/build/BuildClient.cs:line 154","HelpLink":null,"Source":"Microsoft.Rest.ClientRuntime","HResult":-2146233087}, retry.

Restarted the components again. Also I'll make sure we bump up the priority on getting this sorted -- no idea why this issue is affecting West Europe so frequently 😕

@amsoedal Yes, everything is working fine now. Thanks you!

@Eneuman > We now have beta bits to try to pod identity/managed identity. As discussed previously, would you be willing to give them a try?
(Tagging @pragyamehta who worked on this on our side)

@daniv-msft Hi, yes we are intrested in trying out the new bits. I just replied to @pragyamehta email about our current timeframe :)

@amsoedal It seems like we have a problem again. Pods are giving us theese errors:
pods/klinikportalservice-858c9b7c7-lhvlq: BackOff: Back-off restarting failed container

@Eneuman are there any logs or events on the pod? ☹ We did do a release recently but it shouldn't have changed any functionality to the controller.

@amsoedal
pods/klinikportalservice-75588487b4-vtwp4: BackOff: Back-off restarting failed container
Oops... An unexpected error has occurred. A report of the error will be sent to Microsoft.
For diagnostic information, see Azure Dev Spaces logs at 'C:\Users\PerBornsjö\AppData\Local\Temp\Azure Dev Spaces'.
Please include the following Client Request ID when contacting support: ab587787-70e2-4d20-bbe6-d69adbfd786c
2021-03-12T11:22:21.5408289Z | CLI | WARNG | Logging handled exception: Microsoft.Azure.DevSpaces.Client.Exceptions.OperationIdException: {"RequestId":null,"ClientRequestId":"ab587787-70e2-4d20-bbe6-d69adbfd786c","CorrelationRequestId":null,"Request":null,"Response":null,"Format":"The SSL connection could not be established, see inner exception.","Args":[],"Message":"The SSL connection could not be established, see inner exception.","Data":{},"InnerException":{"Message":"The SSL connection could not be established, see inner exception.","Data":{},"InnerException":{"ClassName":"System.Security.Authentication.AuthenticationException","Message":"The remote certificate is invalid according to the validation procedure.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)\r\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)\r\n   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)\r\n   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)\r\n   at System.Net.Security.SslStream.<>c.b__47_1(IAsyncResult iar)\r\n   at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233087,"Source":"System.Private.CoreLib","WatsonBuckets":null},"StackTrace":"   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)\r\n   at System.Threading.Tasks.ValueTask1.get_Result()\r\n   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   at System.Threading.Tasks.ValueTask1.get_Result()\r\n   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask1 creationTask)\r\n   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   at Microsoft.Azure.DevSpaces.Common.Logging.OperationContextRequestIdsHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\\A\\1\\55\\s\\src\\common\\Logging\\OperationContextRequestIdsHttpHandler.cs:line 45\r\n   at Microsoft.Azure.DevSpaces.Client.ServiceClients.Handlers.ClientOperationContextRequestIdsHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\\A\\1\\55\\s\\src\\client\\Client\\ServiceClients\\Handlers\\ClientOperationContextRequestIdsHttpHandler.cs:line 48\r\n   at Microsoft.Azure.DevSpaces.Client.ServiceClients.Handlers.UnsupportedRegionErrorHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\\A\\1\\55\\s\\src\\client\\Client\\ServiceClients\\Handlers\\UnsupportedRegionErrorHttpHandler.cs:line 45\r\n   at Microsoft.Azure.DevSpaces.Client.ServiceClients.Handlers.ApiVersionCheckerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\\A\\1\\55\\s\\src\\client\\Client\\ServiceClients\\Handlers\\ApiVersionCheckerHttpHandler.cs:line 62\r\n   at Microsoft.AspNetCore.Http.Connections.Client.Internal.AccessTokenHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   at Microsoft.AspNetCore.Http.Connections.Client.Internal.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\r\n   at Microsoft.AspNetCore.Http.Connections.Client.Internal.SendUtils.SendMessages(Uri sendUrl, IDuplexPipe application, HttpClient httpClient, ILogger logger, CancellationToken cancellationToken)\r\n   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()\r\n   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)\r\n   at System.IO.Pipelines.Pipe.GetReadAsyncResult()\r\n   at System.Threading.Tasks.ValueTask1.get_Result()\r\n   at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReceiveLoop(ConnectionState connectionState)\r\n   at System.Threading.Channels.AsyncOperation1.GetResult(Int16 token)\r\n   at System.Threading.Tasks.ValueTask1.get_Result()\r\n   at Microsoft.AspNetCore.SignalR.Client.HubConnectionExtensions.<>c__DisplayClass57_01.<g__RunChannel|0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Threading.Channels.AsyncOperation1.GetResult(Int16 token)\r\n   at Microsoft.Azure.DevSpaces.Client.ServiceClients.ExecSignalRServerProxy._RunAsync[T](Func1 func, CancellationToken cancellationToken, String methodName) in C:\A\1\55\s\src\client\Client\ServiceClients\ExecSignalRServerProxy.cs:line 104\r\n   at Microsoft.Azure.DevSpaces.Client.ServiceClients.ExecSignalRServiceClient.<>c__DisplayClass29_01.<<AutoRunChannelStreamUntilCanceledAsync>b__3>d.MoveNext() in C:\\A\\1\\55\\s\\src\\client\\Client\\ServiceClients\\ExecSignalRServiceClient.cs:line 402\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.DevSpaces.Client.ServiceClients.ExecSignalRServiceClient.RunAsync(String operationDescription, Func1 func, CancellationToken cancellationToken, Nullable1 maxWaitForSuccess, Boolean tolerateNotFoundResponses) in C:\\A\\1\\55\\s\\src\\client\\Client\\ServiceClients\\ExecSignalRServiceClient.cs:line 227\r\n   at Microsoft.Azure.DevSpaces.Client.ServiceClients.ExecSignalRServiceClient.AutoRunChannelStreamUntilCanceledAsync[T](String operationName, Func1 startStreamCallback, Action1 itemHandler, IProgressReporter progressReporter, TimeSpan maxWaitForConnectionSuccess, CancellationToken cancellationToken, Boolean useCatchUp, Boolean tolerateNotFoundResponses, Func2 streamCompletedCallback, Action firstConnectionEstablishedCallback) in C:\A\1\55\s\src\client\Client\ServiceClients\ExecSignalRServiceClient.cs:line 407\r\n   at Microsoft.Azure.DevSpaces.Client.ServiceClients.ExecExceptionStrategy.RunWithHandlingAsync[T](Func1 func, FailureConfig failureConfig) in C:\\A\\1\\55\\s\\src\\client\\Client\\ServiceClients\\ExceptionStrategies\\ExecExceptionStrategy.cs:line 59\r\n   at Microsoft.Azure.DevSpaces.Client.ManagementClients.ServiceManagementClientImplementation.GetBuildLogsAsync(String chartName, String requestId, IProgressReporter progressReporter, CancellationToken cancellationToken) in C:\\A\\1\\55\\s\\src\\client\\Client\\ManagementClients\\ServiceManagementClientImplementation.cs:line 748\r\n   at Microsoft.Azure.DevSpaces.Client.ManagementClients.ServiceManagementClientImplementation.DeployReleaseInnerAsync(IServiceConfig serviceConfig, DeployReleaseOption deployReleaseOption, String cwd, IProgressReporter progressReporter, IPerformanceLogger perfLogger, CancellationToken cancellationToken) in C:\\A\\1\\55\\s\\src\\client\\Client\\ManagementClients\\ServiceManagementClientImplementation.cs:line 545\r\n   at Microsoft.Azure.DevSpaces.Client.ManagementClients.ServiceManagementClientImplementation.DeployAzdsReleaseAsync(IServiceConfig serviceConfig, DeployReleaseOption deployReleaseOption, IProgress1 progress, CancellationToken cancellationToken, Boolean skipSyncCode) in C:\A\1\55\s\src\client\Client\ManagementClients\ServiceManagementClientImplementation.cs:line 224\r\n   at Microsoft.Azure.DevSpaces.Client.Extensions.OwnedExtensions.<>c__DisplayClass2_02.<<TryRunOwnedOperationThenDisposeAsync>b__0>d.MoveNext() in C:\\A\\1\\55\\s\\src\\client\\Extensions\\OwnedExtensions.cs:line 62\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.DevSpaces.Client.Utilities.AutofacUtilities.TryRunWithErrorPropagationAsync[T](Func1 func, ILog log, IOperationContext operationContext) in C:\A\1\55\s\src\client\Utilities\AutofacUtilities.cs:line 81","HelpLink":null,"Source":"System.Net.Http","HResult":-2146233087},"StackTrace":"   at Microsoft.Azure.DevSpaces.Client.Utilities.AutofacUtilities._Handle[T](Exception e, ILog log, IOperationContext operationContext, Boolean isRecurse) in C:\A\1\55\s\src\client\Utilities\AutofacUtilities.cs:line 163\r\n   at Microsoft.Azure.DevSpaces.Client.Utilities.AutofacUtilities.TryRunWithErrorPropagationAsync[T](Func`1 func, ILog log, IOperationContext operationContext) in C:\A\1\55\s\src\client\Utilities\AutofacUtilities.cs:line 87\r\n   at Microsoft.Azure.DevSpaces.Cli.Commands.Service.ServiceUpCommand.ExecuteAsync() in C:\A\1\55\s\src\cli\Commands\Service\ServiceUpCommand.cs:line 233\r\n   at Microsoft.Azure.DevSpaces.Cli.DevSpacesCliApp.RunCommandAsync(String[] args, CancellationToken cancellationToken) in C:\A\1\55\s\src\cli\DevSpacesCliApp.cs:line 190\r\n   at Microsoft.Azure.DevSpaces.Cli.DevSpacesCliApp.ExecuteAsync(String[] args, CancellationToken cancellationToken) in C:\A\1\55\s\src\cli\DevSpacesCliApp.cs:line 149","HelpLink":null,"Source":null,"HResult":-2146233088}
2021-03-12T11:22:21.5443083Z | CLI | TRACE | Event: Command.End {"properties":{"arguments":"up","result":"Failed"},"metrics":{"duration":468956.0}}

@Eneuman OK thanks. I restarted our components, could you please see if it works now?

@amsoedal  Yes, now its working again.
It did one pods/klinikportalservice-75588487b4-vtwp4: BackOff: Back-off restarting failed container but then it started to build.
Thanks!

@amsoedal Seems like the problem is back, but now I also reecived this:
pods/klinikportalservice-844fb67df4-2ct56: BackOff: Back-off restarting failed container
Cancelling...
Timed out after 14m trying to start build logs streaming operation.

@Eneuman our components restarted overnight, can you try the operation now?

@amsoedal Some services can be installed with azds up but about 50% gives us timeouts or other strange errors :(

@Eneuman that's very strange :( Can you send me your log files?

@amsoedal
Here is a new error for us:
Azds up for  failed: Helm install failed with exit code '1': Error: UPGRADE FAILED: query: failed to query with labels: Get https://aks-we-eclinic-dev-dns-a2826232.hcp.westeurope.azmk8s.io:443/api/v1/namespaces/dev/secrets?labelSelector=name%3Dazds-b75d66-dev-organisationpersonalservice%2Cowner%3Dhelm%2Cstatus%3Ddeployed: dial tcp <IP_ADDRESS>:443: i/o timeoutFailed to install Helm chart! 52s

And we are also receiving this:
Timed out after 13s trying to start deploy release streaming operation.

@Eneuman that's very strange... I'll take a look at the logs on my side. This might be a naive question, but is "helm install" for your chart working outside of an AZDS context?

@amsoedal I have not used them outside of azds up so I can't say. But azds up has not been a problem before and nothing has changes in the service.

@Eneuman I restarted the components again and looking at our backend logs in parallel, I see a lot of helm timeout exceptions as well. Not sure what is causing it. Let me know if restarting changed anything

@amsoedal After a few retries they all got deployed :)
But is seems like there are some performance issues. The deployments take alot of time.

@eneuman I'm glad you got unblocked! That gives us some more time to investigate hopefully

@amsoedal Hi again. About 10 minutes ago it stoped working again. This time we receive this error message: This version of Kubernetes is no longer supported. Please upgrade your cluster to a supported Kubernetes version and retry.
after waiting for 5 min, the error went away. It's working now.
I'm beginning to see a pattern with theese outages.
Our main project has 3700 files in it. Suddenly it decied that it wants to synchonize all thoose files at once.
This works okey the first time but after that, synchonization seems to start having performance issues.
I don't know how dev spaces handles synchonizations on your end but if it tries to handle each file in a seperate thread (or to many at once) this maby could be a case of thread starvation that leads to deadlocks. Just a thought.

@amsoedal Hi.
I seems like the services are stuck again. Can you please restart them ?

@rakeshvanga who is looking at issues this week, as @amsoedal isn't available.

@Eneuman I've restarted our components. Can you try now?

@rakeshvanga Hi
It's working again :)
Thanks!

@Eneuman, I've restarted the pods. Can you try now?

Now it’s working fine again :)
Thanks!

Från: Rakesh Vanga @.>
Skickat: Wednesday, March 31, 2021 12:05:12 AM
Till: Azure/dev-spaces @.>
Kopia: Per Bornsjö @.>; Mention @.>
Ämne: Re: [Azure/dev-spaces] Synchronization timeouts (Dev Spaces) (#407)
@Eneumanhttps://github.com/Eneuman, I've restarted the pods. Can you try now?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Azure/dev-spaces/issues/407#issuecomment-810608826, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4J5TKCO5K4RUO67WWJ34DTGJDJRANCNFSM4RYWJA4A.

@rakeshvanga
Hi
It just crashed again, can you restart it plz ? :)

I have attached todays crash logs if you want to take alook and see if you can figure out whats going on :)
2021-04-01T08-07-10.4498504.log
2021-04-01T07-20-03.4750117.log
2021-04-01T15-59-25.5680363.log

@daniv-msft It looks like the services are stuck again. Can you please ask someone to restart them ? :)

Hi @Eneuman, I restarted the usual component, can you see if it's working now?

@amsoedal Hi
It's still stuck at: Waiting for container image build.... and it never starts and timeouts after 10 min :(

@Eneuman hm.... might be something else to do with connectivity to the controller. :( I restarted a different component this time, could you try it now?

@amsoedal Still not working :(

It seems like we're not able to build the service for some reason, or the service pod isn't coming up correctly. I see on the backend lots of logs like "still no user pods found in Ready state". Could you try doing an "up", and in a separate window doing kubectl get po -n <your space> --watch? And if you see any pods crashing, do you see any events when you do kubectl describe on the pod?

It seems like the pods crashes:

@eneuman thanks. What do you see when you run "kubectl describe " in a different window under the "Events" section?

Normal   Scheduled          9m24s                   default-scheduler  Successfully assigned dev/organisationpersonalservice-67644fc776-hhjnw to aks-agentpool3-99437877-vmss000000
Warning  ErrImageNeverPull  7m13s (x13 over 9m23s)  kubelet            Container image "organisationpersonalservice:devspaces-x0178b13ba1de9152" is not present with pull policy of Never
Warning  Failed             4m18s (x27 over 9m23s)  kubelet            Error: ErrImageNeverPull
I also keep getting this error:
Waiting for container image build...Oops... An unexpected error has occurred. A report of the error will be sent to Microsoft.
For diagnostic information, see Azure Dev Spaces logs at 'C:\Users\PerBornsjö\AppData\Local\Temp\Azure Dev Spaces'.
Please include the following Request ID when contacting support: 597cfa4b-4912-4713-9f8b-1916c3fd0cb5

Ok thanks for adding these details & thanks for your patience. I'm still not totally sure what's going on, but one possibility is that we're failing to build the docker image for your service for some reason so it's not present locally. Could you please zip your %Temp%\Azure Dev Spaces files and send them to me at<EMAIL_ADDRESS>I'll let you know as soon as I have an update to share.

Thanks. You should have them in your inbox now.

@eneuman thanks so much. One last question, have any changes been made to your Dockerfile.develop recently?

Hi. No changes has been made to the docker.developer files for about 2 month.
The service was working fine about 1 week ago.

Ok thanks for adding these details & thanks for your patience. I'm still not totally sure what's going on, but one possibility is that we're failing to build the docker image for your service for some reason so it's not present locally. Could you please zip your %Temp%\Azure Dev Spaces files and send them to me at<EMAIL_ADDRESS>I'll let you know as soon as I have an update to share.

Normaly I can see in the console window that it starts to build my docker image, but now I don't get any indication that it actually has started to build it.

@amsoedal Anyting else you want me to try? We are kinda stuck atm :(

@eneuman sorry no updates yet, but one of my US colleagues is now helping to look at the issue. Working to get to the bottom of this

@Eneuman our best guess right now is that something on your cluster is preventing the container image from getting built. I also see that in the screenshot mentioned here:

It seems like the pods crashes with the error ImageNeverPull.

There are some pods that have been terminating for 25m. Are they from a previous "azds up", or were they deployed another way? Are there many pods that are stuck in terminating state on the cluster?

Just the pods that I have been trying to do a azds up for.
I tried to create a new service and start it but it also gets the same errors:
testapplication-6dcd88956b-fpsrz                                  0/1     Pending             0          0s
testapplication-6dcd88956b-fpsrz                                  0/1     Pending             0          0s
testapplication-6dcd88956b-fpsrz                                  0/1     ContainerCreating   0          0s
testapplication-6dcd88956b-fpsrz                                  0/1     ErrImageNeverPull   0          1s
What image is it trying to pull?
What happens between synchronization of my files and the build?
I don't see any init containers getting started. Can this be the problem?

@Eneuman Thanks for your reply. The way it works is that we inject an init container in the user pod on your cluster, that will build the docker image needed based on your code. This is why the image pull policy of the user pod is set to "Never": we never want to pull the image remotely, it should always be pulled locally. However, the reason why this image isn't built is unclear.
You mentioned that no init containers are started. This could indeed be the problem. Can you see one called/using as docker image "devspaces-build" or "azds-build-service"?

This is what the pods running looks like:

This is what the pods we tried to build today looks like:

I think the problem is the init containers does not start

kubectl logs azds-webhook-deployment-6f5f6f596-z9h2l --namespace azds shows this?
azdslogger: main.go:133: Error flushing logs:  adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials\r\nTrace ID: 3688cb74-cddf-416a-a2b5-283177cb2200\r\nCorrelation ID: 765fc0ec-9bb8-44c1-9c32-c3cf80a90b50\r\nTimestamp: 2021-04-08 16:11:37Z","error_codes":[7000222],"timestamp":"2021-04-08 16:11:37Z","trace_id":"3688cb74-cddf-416a-a2b5-283177cb2200","correlation_id":"765fc0ec-9bb8-44c1-9c32-c3cf80a90b50","error_uri":"https://login.microsoftonline.com/error?code=7000222"}

Webhooks pods are full of problems :(

Thanks @Eneuman for sharing these, it's helpful. We're looking into them and getting back to you once we understand better what could explain this.

Also, could you please let us know if you can get any logs from the devspaces-build container?
Finally, could you please clarify what made you look into azds-webhook-deployment? I'm curious to know if you saw a log somewhere that would point to that.

I can't find a devspaces-build container, but isn't that one that the webhook admission server also creates?
What made me look there was the fact that no build gets started and Kubernetes tries to use my "raw" charts to pull in a image that does not exist.
This points to a problem with weebhook not injecting init or build containers.
Do you have access to the logs or should I mail them to you?
It's clear that for some reason the token is expired and is not getting renewed.

@eneuman have the certs for the cluster been rotated recently? Could you please try running azds controller refresh-credentials?

@amsoedal We havn't rotated them manually. I'm not sure if AKS does this by itself.

@amsoedal Didn't work, still getting theese errors:
azdslogger: main.go:133: Error flushing logs:  adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials\r\nTrace ID: 55e05563-e9ea-4000-a43d-8558017e2600\r\nCorrelation ID: 8c63e8ad-e03d-424a-aec3-d8ffa582a80d\r\nTimestamp: 2021-04-08 16:53:41Z","error_codes":[7000222],"timestamp":"2021-04-08 16:53:41Z","trace_id":"55e05563-e9ea-4000-a43d-8558017e2600","correlation_id":"8c63e8ad-e03d-424a-aec3-d8ffa582a80d","error_uri":"https://login.microsoftonline.com/error?code=7000222"}

I guess I can recreate the pods and that would probobly do it, but I don't want to do that until you got all debug information you need :)

@Eneuman thank you :) I think we have enough info, please do try recreating the webhook pods. If that doesn't do anything, recreating the controller is probably the next step

I deleted the pods but after they were created it had this in the log:
azdslogger: main.go:459: Getting standard service principal token

azdslogger: main.go:478: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials\r\nTrace ID: 34da58a1-29da-4208-996b-43fb6f122200\r\nCorrelation ID: b0174168-a54b-4890-8452-92430b625434\r\nTimestamp: 2021-04-08 17:00:07Z","error_codes":[7000222],"timestamp":"2021-04-08 17:00:07Z","trace_id":"34da58a1-29da-4208-996b-43fb6f122200","correlation_id":"b0174168-a54b-4890-8452-92430b625434","error_uri":"https://login.microsoftonline.com/error?code=7000222"}

So I guess this is not going to work

Can this have something to do with us running 2FA on our AD domain ?

@Eneuman is that a recent change?

@amsoedal No we have used it for 6 month

@amsoedal Will az aks update-credentials work?

OK it's probably not 2FA then. And not sure about the update-credentials. Do you remember how old the controller and the cluster are?

Maby 9 months or so, probobly more.

204 days :)

Ok thanks! I was reading up on update-credentials, and it mentions checking the expiration date of your service principal:
https://docs.microsoft.com/en-us/azure/aks/update-credentials
SP_ID=$(az aks show --resource-group myResourceGroup --name myAKSCluster \
    --query servicePrincipalProfile.clientId -o tsv)
az ad sp credential list --id $SP_ID --query "[].endDate" -o tsv

I wonder if the SP could have expired already? Either way I think it will be necessary to recreate the controller to pick up the right credentials for talking to your cluster

@amsoedal Hi
I have refreshed the credentials of my AKS and SP but now im receiving this:
azdslogger: main.go:205: Calling URL:  https://azds-33acce48-dad2-41a5-b03b-d9175f16c8ba.weu.azds.io/api/config/x0178b2a17b9206df

azdslogger: main.go:214: Response status code :  403

azdslogger: main.go:223: Could not find exec version for request Id  x0178b2a17b9206df

azdslogger: main.go:286: Calling URL:  https://azds-33acce48-dad2-41a5-b03b-d9175f16c8ba.weu.azds.io/api/userpod

@amsoedal My AKS looks liek this (and has been for awhile)

If I disable Dev Spaces, will I be able to enable it again? (I dont know if it checks the kubernetes version or the node version for compability)

In my new cluster, I now recive theese errors:

@Eneuman I just wanted to let you know that we're still investigating this and trying to understand what's happening. We don't have anything new to share yet, but we'll keep you updated once we find something.

Just to be sure, the errors you're seeing with the webhook deployment are still:
azdslogger: main.go:133: Error flushing logs: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials\r\nTrace ID: 55e05563-e9ea-4000-a43d-8558017e2600\r\nCorrelation ID: 8c63e8ad-e03d-424a-aec3-d8ffa582a80d\r\nTimestamp: 2021-04-08 16:53:41Z","error_codes":[7000222],"timestamp":"2021-04-08 16:53:41Z","trace_id":"55e05563-e9ea-4000-a43d-8558017e2600","correlation_id":"8c63e8ad-e03d-424a-aec3-d8ffa582a80d","error_uri":"https://login.microsoftonline.com/error?code=7000222"}

for your new cluster, right?

For my new cluster I didn't get any logs in the webhook pods, they keept crashing when i did a azds up.
The cluster was created using this:
az aks create --subscription $Subscription.Name --resource-group $ResourceGroup.ResourceGroupName --location $DefaultLocation --max-pods 100 --name $AksClusterName --node-count 1 --node-vm-size $AksNodeSize --network-plugin azure --enable-aad --aad-admin-group-object-ids $AdminGroupObjectId --kubernetes-version $KubernetesVersion
Now im recreating it again but now with aad disabled and Kubenet as network type just to see if it makes any differense.

Thanks for your reply. Would it be possible to get the logs of these pods after crash by running kubectl logs <podname> --previous?
Thanks for your patience, and apologies for the disruption.

I did try it and it only had one line in them
azdslogger: main.go:451: Getting managed identity token nothing else.

In the cluster I just created, the weebhooks pods does not crash so the weebhooks pods might not be compatible with azure network plugin för AKS i guess?
Now the logs look like this:
azdslogger: main.go:451: Getting managed identity token
azdslogger: main.go:499: Acquired token was saved in '/root/.adal/accessToken.json' file
azdslogger: main.go:113: Running webhook service using port  5050

Interesting, thanks for sharing. I assume that you didn't start using the Azure Network plugin recently, right? Maybe they did an update on their side?
Now that the webhooks pods seem to be fine, does azds up look better?

No, we are back to the problem with the init pods not beeing initilized.
When I do a azds up, it creates the services and deploy the helm charts, but then it waits for the build process to begin, but it never starts. The weekhooks logs does not update ither so something is not triggering it. What component is calling the webhook pods? Is that component in my cluster? Can I see its configuration?

Did you just change something? It started working !

@Eneuman No, we were still digging into the logs we have on our side to try to understand what was happening. We didn't restart anything manually. The only automatic restart script we have (restarting Traefik pods automatically every couple of hours) that we added a couple of weeks ago didn't run in the last minutes.
We confirm that we can now see on our side that the build did happen: Pushing build logs for space 'dev' and service 'testapplication' and requestId x0178b3bb0f35249d

Can it be some type of cache that got evicted?
I'm gonna recreate the cluster again just to see if Azure network plugin was a problem or not.

I'm gonna have to call it a night. Been at it for 20 hours straight and I can deploy services to my cluster now so I'm unblocked, even though we don't know why it started working I say it's a win :)

@Eneuman Makes sense, it's super late in Sweden right now! Yes, we share your frustration. :( Happy that you're unblocked at least!

@daniv-msft Hi
I have done alot of research today and this is what I have found so far.
My old cluster worked with Dev-Spaces and Pod-Identity, I never upgraded Pod-Identity so we don't know what version was actually running.
From what I can tell, today version av Dev-Spaces and Pod-Identity is not compatible so It looks like I can't setup a new cluster and we are stuck.
My findings so far:
1) If I install only dev space controller, I can do azds up and it works
2) If I install dev space controller first then AAD-Pod Identity: When I do a AZDS then the build pod craches with the error below
An error has occurred. Please contact Azure Dev spaces team for troubleshooting.
2021-04-09T11:41:15.7706931Z | BuildService | TRACE | AssemblyLoadContext unloading\nOperation context: <json>{"clientRequestId":"2ccf90f3-f2e4-43a3-ae94-6a6d81615450","correlationRequestId":"1f4d7edc-4e9a-4cc0-9138-918ccd629f63","requestId":null,"userSubscriptionId":null,"startTime":"2021-04-09T11:39:53.2813578Z","userAgent":"BuildSvc/1.0.20210206.1","requestHttpMethod":null,"requestUri":null,"apiVersion":null,"version":"1.0.20210206.1","requestHeaders":{},"loggingProperties":{"applicationName":"BuildService","deviceOperatingSystem":"Linux 5.4.0-1043-azure #45~18.04.1-Ubuntu SMP Sat Mar 20 16:16:05 UTC 2021","framework":".NET Core 4.6.28325.01"}}</json>
2021-04-09T11:42:15.7698385Z | BuildService | TRACE | Process exiting...
2021-04-09T11:42:15.7704239Z | BuildService | TRACE | Process exited

3) If I install AAD-Pod identity first and then Dev Space controller: The init pods does not get initialized and the build process does not start.
I have tried all the existing AAD-Pod identity versions available with the same problem :(
We need AAD-Pod identity to work for our software to run.
You can reproduced this behavior by installing a AKS like this:
 az aks create --subscription $SubscriptionName --resource-group $ResourceGroupName --location $DefaultLocation --max-pods 100 --name $AksClusterName --node-count 1 --node-vm-size $AksNodeSize --kubernetes-version $KubernetesVersion --output none
and then add AAD-identity
kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/v1.6.3/deploy/infra/deployment-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/v1.6.3/deploy/infra/mic-exception.yaml

and then create a simple .net core service and do a azds up.

@eneuman thanks for these details. We're still working to get to the bottom of this, but it helps to know that pod identity is the determining difference

@amsoedal Sounds great :)
I think one of the problem is that Azure Pod Identity is intercepting the Manage Identity call that the init build container does and that breakes the build.
I noticed that if I add a AAD Exception like this:
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzurePodIdentityException
metadata:
  name: azure-dev-spaces-exception
  namespace: azds
spec:
  podLabels:
    app: azds-injector-webhook

then the webkhook containers starts to work.
From NMI logs:
I0409 13:10:26.424302       1 main.go:86] starting nmi process. Version: v1.7.5. Build date: 2021-04-02-21:14.
I0409 13:10:26.526716       1 crd.go:447] CRD lite informers started
I0409 13:10:26.526742       1 main.go:113] running NMI in namespaced mode: false
I0409 13:10:26.526751       1 nmi.go:53] initializing in standard mode
I0409 13:10:26.526756       1 probes.go:41] initialized health probe on port 8085
I0409 13:10:26.526760       1 probes.go:44] started health probe
I0409 13:10:26.526819       1 metrics.go:341] registered views for metric
I0409 13:10:26.526867       1 prometheus_exporter.go:21] starting Prometheus exporter
I0409 13:10:26.526879       1 metrics.go:347] registered and exported metrics on port 9090
I0409 13:10:26.526981       1 server.go:102] listening on port 2579
W0409 13:10:26.736008       1 iptables.go:123] flushing iptables to add aad-metadata custom chains
I0409 13:12:17.988699       1 server.go:362] exception pod azds/azds-webhook-deployment-78d598464d-xh749 token handling
I0409 13:12:17.988720       1 server.go:305] fetching token for user assigned MSI for resource: https://azds.io/
I0409 13:12:17.998412       1 server.go:196] status (200) took 9791388 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>

@Eneuman are you able to debug your app after adding this exception?

No, but adding the exception makes the webhookspods work by the look of it. Building the images does not work. The build init container never finishes and the proxy init containers stays in Pending state.

@Eneuman are you still using user assigned managed identity?

Yes

Have you been able to reproduce this behaviour in your cluster?

@Eneuman We're trying this, and will let you know how it goes.

@Eneuman I confirm we do reproduce the issue on multiple clusters. We're still looking into what goes wrong. This issue is challenging.

@Eneuman Something strange for us is that we don't see the webhook issue you were mentioning above. On our side, the webhook seems to work fine (without adding AAD Exception), and we can even see the request to start building the image locally showing in their logs:

Also, we can see the Build service exiting because of an error:
λ kubectl logs -n dev webfrontend-76d68d944c-frtpm -c devspaces-build
An error has occurred. Please contact Azure Dev spaces team for troubleshooting.
2021-04-09T15:42:14.7684021Z | BuildService | TRACE | AssemblyLoadContext unloading\nOperation context: <json>{"clientRequestId":"a3e1d09a-a1ef-446a-868b-3b769e2126e7","correlationRequestId":"9423774f-a445-463e-839f-a683ec44dc0e","requestId":null,"userSubscriptionId":null,"startTime":"2021-04-09T15:33:09.3853715Z","userAgent":"BuildSvc/1.0.20210206.1","requestHttpMethod":null,"requestUri":null,"apiVersion":null,"version":"1.0.20210206.1","requestHeaders":{},"loggingProperties":{"applicationName":"BuildService","deviceOperatingSystem":"Linux 5.4.0-1043-azure #45~18.04.1-Ubuntu SMP Sat Mar 20 16:16:05 UTC 2021","framework":".NET Core 4.6.28325.01"}}</json>
2021-04-09T15:43:14.7748874Z | BuildService | TRACE | Process exiting...
2021-04-09T15:43:14.7754218Z | BuildService | TRACE | Process exited

Unfortunately, the logs don't show much as of today, but we're working to rebuild an image of this with more logs and see the new info we can get. Do you see something similar on your side?

Yes, I got the exact error as you.
About the webhook, it depended on in witch order I installed Dev spaces and pod identity. See my post higher up. If I installed Dev spaces first and then add, webhooks stoped working for me.

Från: daniv-msft @.>
Skickat: Friday, April 9, 2021 6:13:45 PM
Till: Azure/dev-spaces @.>
Kopia: Per Bornsjö @.>; Mention @.>
Ämne: Re: [Azure/dev-spaces] Synchronization timeouts (Dev Spaces) (#407)
Also, we can see the Build service exiting because of an error:
λ kubectl logs -n dev webfrontend-76d68d944c-frtpm -c devspaces-build
An error has occurred. Please contact Azure Dev spaces team for troubleshooting.
2021-04-09T15:42:14.7684021Z | BuildService | TRACE | AssemblyLoadContext unloading\nOperation context: {"clientRequestId":"a3e1d09a-a1ef-446a-868b-3b769e2126e7","correlationRequestId":"9423774f-a445-463e-839f-a683ec44dc0e","requestId":null,"userSubscriptionId":null,"startTime":"2021-04-09T15:33:09.3853715Z","userAgent":"BuildSvc/1.0.20210206.1","requestHttpMethod":null,"requestUri":null,"apiVersion":null,"version":"1.0.20210206.1","requestHeaders":{},"loggingProperties":{"applicationName":"BuildService","deviceOperatingSystem":"Linux 5.4.0-1043-azure #45~18.04.1-Ubuntu SMP Sat Mar 20 16:16:05 UTC 2021","framework":".NET Core 4.6.28325.01"}}
2021-04-09T15:43:14.7748874Z | BuildService | TRACE | Process exiting...
2021-04-09T15:43:14.7754218Z | BuildService | TRACE | Process exited
Unfortunately, the logs don't show much as of today, but we're working to rebuild an image of this with more logs and see the new info we can get. Do you see something similar on your side?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Azure/dev-spaces/issues/407#issuecomment-816793157, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC4J5TKVCRBCNZY4YTZVX5TTH4RTTANCNFSM4RYWJA4A.

@Eneuman, Can you check the nmi and mic pod logs in the default namespace? What do they say?

Also, I opened this issue on the pod-identity page. Hopefully they'll be able to shed some clarity. https://github.com/Azure/aad-pod-identity/issues/1039

Working on getting you the logs @amsoedal.
Just wanted to let you know that I have tried both using Azure CNI and Kubernet as network and both have the same errors.

My setup:
New clean AKS using Kubenet as Network.
Installed AAD like this:
helm install aad-pod-identity aad-pod-identity/aad-pod-identity --set nmi.allowNetworkPluginKubenet=true
Installed Dev Spaces after AAD..
Tried to azds up a service.
C:\Users\PerBornsjö>kubectl logs aad-pod-identity-mic-5b5f866bfc-mpn6w -n default
I0409 19:07:04.272950       1 main.go:114] starting mic process. Version: v1.7.5. Build date: 2021-04-02-21:14
W0409 19:07:04.273008       1 main.go:119] --kubeconfig not passed will use InClusterConfig
I0409 19:07:04.273017       1 main.go:136] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I0409 19:07:04.273186       1 main.go:144] running MIC in namespaced mode: false
I0409 19:07:04.273200       1 main.go:148] client QPS set to: 5. Burst to: 5
I0409 19:07:04.273219       1 mic.go:139] starting to create the pod identity client. Version: v1.7.5. Build date: 2021-04-02-21:14
I0409 19:07:04.297870       1 mic.go:145] Kubernetes server version: v1.18.14
I0409 19:07:04.298183       1 cloudprovider.go:122] MIC using user assigned identity: a8e2##### REDACTED #####b15f for authentication.
I0409 19:07:04.364540       1 probes.go:41] initialized health probe on port 8080
I0409 19:07:04.364563       1 probes.go:44] started health probe
I0409 19:07:04.364677       1 metrics.go:341] registered views for metric
I0409 19:07:04.364756       1 prometheus_exporter.go:21] starting Prometheus exporter
I0409 19:07:04.364764       1 metrics.go:347] registered and exported metrics on port 8888
I0409 19:07:04.364772       1 mic.go:244] initiating MIC Leader election
I0409 19:07:04.364788       1 leaderelection.go:243] attempting to acquire leader lease  default/aad-pod-identity-mic...
I0409 19:07:04.378490       1 leaderelection.go:253] successfully acquired lease default/aad-pod-identity-mic
I0409 19:07:04.580434       1 mic.go:317] upgrading the types to work with case sensitive go-client
I0409 19:07:04.785861       1 mic.go:321] type upgrade completed !!
I0409 19:07:05.271023       1 pod.go:72] pod cache synchronized. Took 300.611305ms
I0409 19:07:05.271054       1 pod.go:79] pod watcher started !!
I0409 19:07:05.471439       1 crd.go:456] CRD informers started
I0409 19:07:05.471482       1 mic.go:402] sync thread started.

C:\Users\PerBornsjö>kubectl logs aad-pod-identity-mic-5b5f866bfc-xrlwn -n default
I0409 19:07:04.365311       1 main.go:114] starting mic process. Version: v1.7.5. Build date: 2021-04-02-21:14
W0409 19:07:04.365358       1 main.go:119] --kubeconfig not passed will use InClusterConfig
I0409 19:07:04.365364       1 main.go:136] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I0409 19:07:04.365543       1 main.go:144] running MIC in namespaced mode: false
I0409 19:07:04.365560       1 main.go:148] client QPS set to: 5. Burst to: 5
I0409 19:07:04.365580       1 mic.go:139] starting to create the pod identity client. Version: v1.7.5. Build date: 2021-04-02-21:14
I0409 19:07:04.390443       1 mic.go:145] Kubernetes server version: v1.18.14
I0409 19:07:04.390701       1 cloudprovider.go:122] MIC using user assigned identity: a8e2##### REDACTED #####b15f for authentication.
I0409 19:07:04.394529       1 probes.go:41] initialized health probe on port 8080
I0409 19:07:04.394549       1 probes.go:44] started health probe
I0409 19:07:04.394628       1 metrics.go:341] registered views for metric
I0409 19:07:04.394712       1 prometheus_exporter.go:21] starting Prometheus exporter
I0409 19:07:04.394722       1 metrics.go:347] registered and exported metrics on port 8888
I0409 19:07:04.394728       1 mic.go:244] initiating MIC Leader election
I0409 19:07:04.394744       1 leaderelection.go:243] attempting to acquire leader lease  default/aad-pod-identity-mic...

C:\Users\PerBornsjö>kubectl logs aad-pod-identity-nmi-brvk4 -n default
I0409 19:07:02.821400       1 main.go:86] starting nmi process. Version: v1.7.5. Build date: 2021-04-02-21:14.
I0409 19:07:03.023889       1 crd.go:447] CRD lite informers started
I0409 19:07:03.023917       1 main.go:113] running NMI in namespaced mode: false
I0409 19:07:03.023931       1 nmi.go:53] initializing in standard mode
I0409 19:07:03.023937       1 probes.go:41] initialized health probe on port 8085
I0409 19:07:03.023942       1 probes.go:44] started health probe
I0409 19:07:03.024023       1 metrics.go:341] registered views for metric
I0409 19:07:03.024092       1 prometheus_exporter.go:21] starting Prometheus exporter
I0409 19:07:03.024102       1 metrics.go:347] registered and exported metrics on port 9090
I0409 19:07:03.024195       1 server.go:102] listening on port 2579
W0409 19:07:04.268479       1 iptables.go:123] flushing iptables to add aad-metadata custom chains
W0409 19:08:26.542949       1 client.go:208] list pod error: pod list is empty. Retrying, attempt number: 1
W0409 19:08:26.595143       1 client.go:208] list pod error: pod list is empty. Retrying, attempt number: 1
W0409 19:08:26.843583       1 client.go:208] list pod error: pod list is empty. Retrying, attempt number: 2
W0409 19:08:26.895699       1 client.go:208] list pod error: pod list is empty. Retrying, attempt number: 2
E0409 19:09:47.159672       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-g6gx8, error: getting assigned identities for pod azds/azds-webhook-deployment-7567578c7f-g6gx8 in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0409 19:09:47.159736       1 server.go:196] status (404) took<PHONE_NUMBER>2 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:09:47.204851       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-tbbpm, error: getting assigned identities for pod azds/azds-webhook-deployment-7567578c7f-tbbpm in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0409 19:09:47.204897       1 server.go:196] status (404) took<PHONE_NUMBER>1 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:09:58.070757       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-tbbpm, error: context canceled
I0409 19:09:58.070797       1 server.go:196] status (404) took<PHONE_NUMBER> ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:10:13.202240       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-g6gx8, error: context canceled
I0409 19:10:13.202283       1 server.go:196] status (404) took<PHONE_NUMBER>0 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:11:18.382253       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-tbbpm, error: getting assigned identities for pod azds/azds-webhook-deployment-7567578c7f-tbbpm in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0409 19:11:18.382288       1 server.go:196] status (404) took<PHONE_NUMBER>0 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:11:28.061983       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-tbbpm, error: context canceled
I0409 19:11:28.062023       1 server.go:196] status (404) took<PHONE_NUMBER> ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:11:33.466788       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-g6gx8, error: getting assigned identities for pod azds/azds-webhook-deployment-7567578c7f-g6gx8 in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0409 19:11:33.466829       1 server.go:196] status (404) took<PHONE_NUMBER>8 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:11:43.244913       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-g6gx8, error: context canceled
I0409 19:11:43.244948       1 server.go:196] status (404) took<PHONE_NUMBER> ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:12:48.337972       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-tbbpm, error: getting assigned identities for pod azds/azds-webhook-deployment-7567578c7f-tbbpm in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0409 19:12:48.338009       1 server.go:196] status (404) took<PHONE_NUMBER>2 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:12:58.074291       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-tbbpm, error: context canceled
I0409 19:12:58.074347       1 server.go:196] status (404) took<PHONE_NUMBER> ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:13:03.524754       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-g6gx8, error: getting assigned identities for pod azds/azds-webhook-deployment-7567578c7f-g6gx8 in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0409 19:13:03.524794       1 server.go:196] status (404) took<PHONE_NUMBER>9 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:13:13.211651       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-g6gx8, error: context canceled
I0409 19:13:13.211694       1 server.go:196] status (404) took<PHONE_NUMBER> ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:14:18.418035       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-tbbpm, error: getting assigned identities for pod azds/azds-webhook-deployment-7567578c7f-tbbpm in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0409 19:14:18.418073       1 server.go:196] status (404) took<PHONE_NUMBER>5 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:14:28.056407       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-tbbpm, error: context canceled
I0409 19:14:28.056446       1 server.go:196] status (404) took<PHONE_NUMBER> ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:14:33.485128       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-g6gx8, error: getting assigned identities for pod azds/azds-webhook-deployment-7567578c7f-g6gx8 in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0409 19:14:33.485168       1 server.go:196] status (404) took<PHONE_NUMBER>7 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:14:43.236099       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-g6gx8, error: context canceled
I0409 19:14:43.236163       1 server.go:196] status (404) took<PHONE_NUMBER> ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>

As you can see the webhook gets intercepted by nim and failes.  So no init pods gets created.
But if I add my exceptions from above I can get around this (after recreating the mic and nim pods).
Now the nim loggs look like this:
I0409 19:30:00.804890       1 server.go:362] exception pod azds/azds-webhook-deployment-7567578c7f-tbbpm token handling
I0409 19:30:00.804908       1 server.go:305] fetching token for user assigned MSI for resource: https://azds.io/
I0409 19:30:01.255704       1 server.go:196] status (200) took 450852054 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0409 19:30:14.749637       1 server.go:375] failed to get matching identities for pod: azds/azds-webhook-deployment-7567578c7f-g6gx8, error: getting assigned identities for pod azds/azds-webhook-deployment-7567578c7f-g6gx8 in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0409 19:30:14.749703       1 server.go:196] status (404) took<PHONE_NUMBER>3 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
I0409 19:30:16.750779       1 server.go:362] exception pod azds/azds-webhook-deployment-7567578c7f-g6gx8 token handling
I0409 19:30:16.750799       1 server.go:305] fetching token for user assigned MSI for resource: https://azds.io/
I0409 19:30:16.761309       1 server.go:196] status (200) took 10586120 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>

And now the init pods starts, but the BUILD Init does not complete. The state is always Running

Hey again,
@pragyamehta and I have been following up with the pod identity team. A couple things:

We followed this doc, and part of it is setting up an exception for azds infrastructure. So that might help.
The other thing is, after we followed this, when we ran kubectl describe AzureIdentityBinding my-cluster-mi-binding -n dev, we saw the following error:

Events:
  Type     Reason               Age                  From                   Message
  ----     ------               ----                 ----                   -------
  Warning  binding apply error  113s (x2 over 113s)  aadpodidentity.k8s.io  failed to apply binding dev/my-cluster-mi-binding node aks-nodepool1-87199437-vmss000000 for pod dev/bikes-58b465dd5d-ts6md, error: failed to update identities for aks-nodepool1-87199437-vmss in MC_redacted_westeurope, error: compute.VirtualMachineScaleSetsClient#Update: Failure sending request: StatusCode=403 -- Original Error: Code="AuthorizationFailed" Message="The client '<redacted>' with object id '<redacted>' does not have authorization to perform action 'Microsoft.Compute/virtualMachineScaleSets/write' over scope '/subscriptions/<redacted>/resourceGroups/MC_redacted_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-87199437-vmss' or the scope is invalid. If access was recently granted, please refresh your credentials."

Could you please run this command on your own AzureIdentityBinding?

@eneuman Our contact also told us to look out for https://azure.github.io/aad-pod-identity/docs/#v16x-breaking-change. If you follow the AZDS doc I linked above, you have to change the fields to camelcase.

@amsoedal Yes, the casing in my bindings should be correct, I have dbl checked them.
Running kubectl describe AzureIdentityBinding eclinic-pod-identity-binding -n dev says this:
Name:         eclinic-pod-identity-binding
Namespace:    dev
Labels:       <none>
Annotations:  <none>
API Version:  aadpodidentity.k8s.io/v1
Kind:         AzureIdentityBinding
Metadata:
  Creation Timestamp:  2021-04-09T20:19:38Z
  Generation:          1
  Managed Fields:
    API Version:  aadpodidentity.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:azureIdentity:
        f:selector:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-04-09T20:19:38Z
  Resource Version:  14237
  Self Link:         /apis/aadpodidentity.k8s.io/v1/namespaces/dev/azureidentitybindings/eclinic-pod-identity-binding
  UID:               4d4bfbe6-3b3b-4636-8882-85b33ebd20eb
Spec:
  Azure Identity:  eclinic-pod-identity
  Selector:        eclinic_pod
Events:            <none>

but I think thats okey since my testpod does not use managed identities yet (I was going to add this after I get it running).

Hi @Eneuman,
We're still investigating, and there is one thing that we would like to validate: could you please confirm that pod identity (independently of Azure Dev Spaces) works in a cluster as configured above? Would you have an easy way to test that?

@daniv-msft I have not been able to test this yet but Im working on it.
I did however found one problem on my side.
When I created the exceptions för AAD i followed this throubleshooting guide:
https://docs.microsoft.com/en-us/azure/dev-spaces/troubleshooting#error-no-azureassignedidentity-found-for-podazdsazds-webhook-deployment-id-in-assigned-state
However, PodLabels should be podLabels otherwise it won't work. It's probobly best to update that page to reflect the latest breaking changes in Pod Idetity 1.7.5. After finding this, webhooks got excluded fine and the build process worked.
Now I'm only stuck with Pod Identity not working because of a Identity Not Found issue. Tracking it here https://github.com/Azure/aad-pod-identity/issues/1040
I'm not sure if this is related to Dev-Spaces and Pod-Identity interceptions. So we are making progress :)

@eneuman thanks for the update! Glad you're able to get past the build issue now at least!

We have migrated to Bridge For Kubernetes so we will close this issue.


get parameter class

how to know the parameter class if the type is complex type?

Not sure what you mean.
If you still got this issue please let me know.


Task re-try strategy

I should allow task to be re-executed if it fails (maybe via retry_count?)

Implemented retry option


FR - clicking on items in Kanban board, should open the page. or atleast have a setting for it

What would you like to be added?
clicking on items in Kanban board, should open the page. or atleast have a setting for it
Why is this needed?
good UX

If you press Shift and click the link, it'll open the note instead of editing it. Is that what you're looking for?

Yes, that works,
i'm not sure what should be the default behavior if you do normal click.
In notion normal click just opens the page

It works in table view - and as default for me - but not in Kanban view. I'd LOVE it in Kanban view.

Sorry, it should be Ctrl (or Cmd) instead of Shift. My bad.

Thanks. I swear I tried that last night and it didn't work... but it is this morning, thanks.

i'm not sure what should be the default behavior if you do normal click.

My reasoning is that when you're in Projects, you're planning and managing content. The default action should reflect that.
Making things configurable is always an option, but I'd prefer to solve things by improving the UX rather than defaulting to configuration options.
Regardless of which action is default, it should be easy to discover the other action. For example, would a tooltip that says something like "Press Ctrl and click to open note"?

I just wanted to mention that I'm a new user and almost uninstalled the plugin again because of this behavior. I'm used to just having to click on a link in Obsidian to get to the corresponding note, and this being not the default behavior in the Kanban board causes enough friction / cognitive overhead for me almost not wanting to use the plugin. I know this sounds pretty drastic but it's small things like this that makes me choose one tool over the other when I'll have to use it everyday.
I'd really appreciate if you could reopen this issue and hopefully consider implementing this as a setting in the future.
Thanks a lot for the plugin.

Mark this as resolved by #620


Add sveltekit routeStyle

Description
Add sveltekit's route style (https://kit.svelte.dev/docs/routing), in which only files called +page.svelte are taken as a page, so the route is defined by the directory structure only and components meant only for some pages can live along each other.
example:
pages/
  admin/
    AdminNavbar.svelte (this is just a component used in admin routes, it's not taken as a route)
    +page.svelte (this is `/svelte` page)
  +page.svelte (this is `/` page)

[X] Am willing to create a PR

Suggested solution
sveltekit routeStyle option and it's logic would be added into vue resolver (possibly other frameworks, I'm not really familiar with solid / react)
Alternative
No response
Additional context
No response
Validations

[X] Follow the Code of Conduct
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

What needs to be discusseed is the inclusion of the catch all route [...id], as sveltekit solves those with layouts and error pages.
In my opinion catch all routes should work just like in next route style.
https://kit.svelte.dev/docs/routing#layout
https://kit.svelte.dev/docs/routing#error

Today I've realised it's possible to add +layout by transfoming parent's rawRoute inside the parent check. The only problem there is right now is +layout in pages root
src/pages
  about/
    +layout.vue // this does work
    +page.vue
  +layout.vue // this does not work
  +page.vue


Add function for keep alive with arbitrary duration

Hello,
Duration of KeepAlive() is hard-coded so we cannot change that. Accordingly, I added a new function KeepAliveBy(sec time.Duration) to configure duration.
Related: #12

@moznion very sorry about my delayed responses!


test: Fix release e2e tests

Description
Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.
HACBS-1108
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Checklist:

[ ] I have performed a self-review of my code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added meaningful description with JIRA/GitHub issue key(if applicable), for example HASSuiteDescribe("[test_id:01] DEVHAS-62 devfile source")
[ ] I have updated labels (if needed)

/ok-to-test

/retest

was targeted to main