StructureMap Child Container - being disposed?

Hello,
I am using the per tenant depenency injection, I have followed the example.
The first request to my application works just fine.
However the second request this happens:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'StructureMap Child/Profile Container'.
   at StructureMap.Container.assertNotDisposed()
   at StructureMap.Container.GetNestedContainer()
   at SaasKit.Multitenancy.StructureMap.Internal.MultitenantContainerMiddleware`1.<Invoke>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at SaasKit.Multitenancy.Internal.TenantResolutionMiddleware`1.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Gluon.Module.Migrator.Middleware.MigrationsCheckerMiddleware.<Invoke>d__6.MoveNext() in D:\Repos\gluon2\src\Gluon.Module.Migrator\Middleware\MigrationsCheckerMiddleware.cs:line 103
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Gluon.Module.Migrator.Middleware.SystemSetupMiddleware.<Invoke>d__9.MoveNext() in D:\Repos\gluon2\src\Gluon.Module.SystemSetup\Middleware\SystemSetupMiddleware.cs:line 48
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore.MigrationsEndPointMiddleware.<Invoke>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore.DatabaseErrorPageMiddleware.<Invoke>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore.DatabaseErrorPageMiddleware.<Invoke>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.<Invoke>d__6.MoveNext()

I'm not sure where I am going wrong. Does the child container get disposed and re-initialised on every request?

Ok, so it seems this is the problem:
https://github.com/saaskit/saaskit/blob/b761ef9295331f9758c5347d8b50b9def7b3589b/src/SaasKit.Multitenancy.StructureMap/Internal/MultitenantContainerMiddleware.cs#L28
I think Context.RequestServices get's disposed anyway, automatically at the end of the request. Asp.net core must have some middleware in the pipeline to handle initialisation and dispose. This was causing it to be disposed() twice.
It could take a snapshot of the existing RequestServices, then after the dispose, restore the old version?

This change fixes the issue:
  var requestContainer = tenantContainer.GetNestedContainer();
                {
                    // Replace the request IServiceProvider created by IServiceScopeFactory
                    var snapshot = context.RequestServices;
                    context.RequestServices = requestContainer.GetInstance<IServiceProvider>();
                    await next.Invoke(context);
                    context.RequestServices = snapshot;
                }

I got the same problem and solve it by modifying TenantContext.cs as such:
`protected virtual void Dispose(bool disposing)
{
if (disposed)
{
return;
}
   if (disposing)
   {
        var deferRemove = new List<String>(Properties.Count);
         foreach (var key in Properties.Keys)
         {
              if (TryDisposeProperty(Properties[key] as IDisposable))
                  deferRemove.Add(key);
         }
         for (int i = 0; i < deferRemove.Count; i++)
             Properties[deferRemove[i]] = null;

          TryDisposeProperty(Tenant as IDisposable);
   }

    disposed = true;

}
private bool TryDisposeProperty(IDisposable obj)
{
if (obj == null)
{
return false;
}
   try
   {
        obj.Dispose();
         return true;
   }
   catch (ObjectDisposedException) { return false; }

}`

Thanks, I'll give that a whirl.
I got as far as discovering that the tenant container is getting disposed, and that it was stored in properties, but hadn't quite got as far as figuring out what disposes it.

Glancing at your workaround suggests that it's disposed when the tenant is disposed.. thanks for sharing your fix I will give it a try tomorrow.

@ozinet - I think my issue might turn out to be slightly different. Here is what I am seeing.

First HttpRequest arrives - TenantContext is resolved --> MemoryCacheTenantResolver adds it to memory cache.
TenantContainer is created and added to TenantContext Properties.
`Middleware creates a nested container, and wraps it using statement.
Middleware disposes of nested Container.

At this point something is disposing of the TenantContext which in turn, disposes of the Properties, and the now Disposed TenantContaner is left in the TenantContext properties.
Start of next request
Middleware resolves tenant context --> disposed TenantContext comes from memory cache.
Middleware gets TenantContainer from TenantContext - which is disposed.
Middleware attempts to create nested container  - exception.
I am just working through now, trying to figure out what is disposing my TenantContext.

Ok so this is the call stack - it shows what is happening. When the nested container is disposed() - it's disposing of my TenantContext - presumably because somehow TenantContext has got into the nested container!
 	SaasKit.Multitenancy.dll!SaasKit.Multitenancy.TenantContext<Gluon.Module.Tenant.Data.TenantDomainApplication>.Dispose() Line 24	C#
 	StructureMap.dll!StructureMap.BasicExtensions.SafeDispose(object target)	Unknown
 	StructureMap.dll!StructureMap.Pipeline.LifecycleObjectCache.DisposeAndClear.AnonymousMethod__10_1(object object)	Unknown
 	StructureMap.dll!StructureMap.StringExtensions.Each<object>(System.Collections.Generic.IEnumerable<object> enumerable, System.Action<object> action)	Unknown
 	StructureMap.dll!StructureMap.Pipeline.LifecycleObjectCache.DisposeAndClear.AnonymousMethod__10_0()	Unknown
 	StructureMap.dll!StructureMap.Pipeline.ReaderWriterLockExtensions.Write(System.Threading.ReaderWriterLockSlim rwLock, System.Action action)	Unknown
 	StructureMap.dll!StructureMap.PipelineGraph.Dispose()	Unknown
 	StructureMap.dll!StructureMap.BasicExtensions.SafeDispose(object target)	Unknown
>	StructureMap.dll!StructureMap.Container.Dispose()	Unknown

I think that's my issue. I'll go back through my code and figure out why TenantContext is getting into the nested container.

@benfoster - I can now replicate this problem, would be interested to know if you see this as a bug:
The exception occurs if you are injecting / resolving TenantContext<AppTenant> in your services.
For example, if you change the LogTenantMiddleware in the structure map sample, to this:
 public LogTenantMiddleware(RequestDelegate next, ILoggerFactory loggerFactory, TenantContext<AppTenant> tenantContext)
        {
            this.next = next;
            this.log = loggerFactory.CreateLogger<LogTenantMiddleware>();
            this.tenantContext = tenantContext;
        }

What happens is that this forces the TenantContext to be resolved from the nested container which is wrapped by the middleware in a using statement.
When the nested container is disposed, it then disposes of the TenantContext. However the TenantContext is still left in the MemoryCache and so is re-used again on the next request - however it's Properties are now all disposed i.e the tenant container is now disposed. This results in tenantContainer.GetNestedContainer() blowing up on subsequent requests as the tenant container instance has been disposed.

I'll raise a seperate issue which is more concise.

Replaced by #68


Clarifying the LIGO contract tutorial

These are my suggestions for clarifying the "Originate your first smart contract tutorial." Per discussion with Tezos folks, I'm changing the contract from using a single main function to separate functions for each endpoint because the main syntax is more complicated and soon to be deprecated.

Yo, I thought I will provide my comments here (for the original one):

[ ] ALL links to be on new tab
[ ] Prerequisites are missing octet-client
[ ] I’m not sure if we need Want to use SmartPy? section
[ ] would be good to understand why do need those dependencies and what is their role
[ ] Note Make sure you have **installed** the above CLI tools before getting started. does not make much sense IMO
[ ] would be good to provide a link for JsLigo and CameLigo, and OCaml (some people don’t know what it is)
[ ] I would swap ligo version with ./ligo version - the second one does not work for me
[ ] Run the following command to generate a local wallet with octez-client, making sure to replace <my_wallet> with a name of your choosing: — in the provided command it’s local_wallet not <my_wallet>
[ ] Use Ligo to create the contract is not mentioning in which file we should write the main code (my assumption is increment.mligo but yeah… hard to say from the tutorial)
[ ] Originate to the Testnet - no mention of updating the my_tez_address to anything, make sure to delete < and >
[ ] again Calling the entrypoints don’t mention to update the local_wallet value to your value
[ ] Ok, it would be cool to have the end values of the storages provided as expected outcomes in each entrypoint: 10, 9, 0 - my assumption was that the reset was the initial value of 10, so yeah - might be useful
[ ] Missing conclusion (but I see that you have the Summary part already.

ALL links to be on new tab

The only way to do this with vanilla markdown is to use an html <a> link. I will make a note to see how we could do this with a markdoc extension to flag links to open in new tabs.

I would swap ligo version with ./ligo version - the second one does not work for me

It should work if LIGO is installed and in the user's PATH. Is that the problem?

would be good to provide a link for JsLigo and CameLigo, and OCaml (some people don’t know what it is)

I'll work on this when I add a version for jsLIGO.

ALL links to be on new tab

The only way to do this with vanilla markdown is to use an html <a> link. I will make a note to see how we could do this with a markdoc extension to flag links to open in new tabs.
Would be good to have it, not super necessary, but very useful when you're going through a tutorial that has internal links :) so thanks for taking care of it

I would swap ligo version with ./ligo version - the second one does not work for me

It should work if LIGO is installed and in the user's PATH. Is that the problem?
I think keeping just one method of checking the version is more than enough (the way you did that in the your text).

would be good to provide a link for JsLigo and CameLigo, and OCaml (some people don’t know what it is)

I'll work on this when I add a version for jsLIGO.
Great stuff, thank you!


module 'minisam' has no attribute 'DiagonalLoss'

I am using PyICP github repo. I built the Sophus from this commit (commit a0fe89a323e20c42d3cecb590937eb7a06b8343a) Reference.
I am using Ubuntu 22 and used virtual environment (venv with python version 3.7.14) to build the PyICP repo, minisam and Sophus repo.
Finally when I run the command  python3 main_icp_slam.py it runs in this error.

I am not sure is it caused by Sophus, Eigen, Venv or minisam.
Let me know if anymore information is needed.
Thanks, for any and all the help.

Could you solve it? Hope you solved it already.


Go tests share one client causing rate limits to be exceeded

The Go end to end tests share one client and this often causes the rate limits to be exceeded and the end to end tests to fail (the shell tests have a client per invocation of kubectl). We should change the Go tests to use a fresh client for each test. This would also help to identify individual tests that exceed rate limits. Related to #3667 @bgrant0607 @filbranden @zmerlynn @roberthbailey

@satnam6502 has this been done? Are the rate limits applied via the Go clients or on the server side?

The rate limits are applied by the server. So, actually, using separate clients does not make a difference (w.r.t. limits).

But yes, I changed the tests in #3720 to use a fresh client. We can keep doing this -- or revert to a shared client. I don't think it makes much of a difference.

I don't think the tests should recycle the client - the existing way is fine.


Iss192 v1

Properly overwrites (deletes, then creates) the session persistence when transitioning from one vip type to another.

Commented.  Update and I'll plusone it.


scene where you give notepad some chips

label .notepad_chips:
    p "_" # posty does indeed have some chips for notepad!
    show generichips
    "You handed over the {b}Generi-Chips{/b}!"
    $ item.chips = False
    $ gave_chips = True
    hide generichips
    notepad "_" # notepad is grateful, and gives you a piece of their art in return
    show badpainting
    "You got the {b}napkin \"painting\"{/b}!"  No!
np: I am at my wits end: starving in the middle of the food exhibit.
np: I should've stayed back home, then I wouldn't have ended up here.
np: Time after time I put so much heart into this, only to be thrown out like a stray cup.
np: I don't know why I bother.
p: You want a snack?
np: What?
p: I got a bunch of chips if you wan-
np: OOOHH PLEASE GIVE IT TO ME THANK YOU SO MUCH!!!
p; You need it more than I do, you gone crazy.
You handed over the {b}Generi-Chips{/b}
np: My god this tastes soooo amazing.
np: First meal in a week, you are my shining saviour.
p: Glad to be of help!
np: For assisting me in my direst hour, you shall have my greatest work.
np: It distills my essence into a small package you can carry around as a reminder.
p: Aw shucks, that is too much!
"You got the {b}napkin "painting"{/b}!"
"If art is in the eye of the beholder, then this piece of litter makes you blind."
p: Oohhh it looks avant garde.
np: This is for feeding me, only the best!
p: Hehe no biggie!
p: (Did they pick up some trash to fool me? Why would anyone even consider this?)
p: (Eh whatever, I guess I'll find a rubbish bin to properly dispose it in later)
p: Ahahaha ohh I am soo sorry, I have a meeting in 30 minutes!
p: I can't hang around any longer sadly...
np: It is a shame I can't see your enjoyment for any longer.
p: See you around I guess!

Mostly good, not sure if the sarcasm lands

p: Oh my, are you alright?
np (Notepad): Can't you tell?!  No!
np: I am at my wits end: starving in the middle of the food exhibit.
np: I should've stayed back home, then I wouldn't have ended up here.
np: Time after time I put so much heart into this, only to be thrown out like a stray cup.
np: I don't know why I bother.
p: You want a snack?
np: What?
p: I got a bunch of chips if you wan-
np: OOOHH PLEASE GIVE IT TO ME THANK YOU SO MUCH!!!
p; You need it more than I do, you gone crazy.
You handed over the {b}Generi-Chips{/b}
np: My god this tastes soooo amazing.
np: First meal in a week, you are my shining saviour.
p: Glad to be of help!
np: For assisting me in my direst hour, you shall have my greatest work.
np: It distills my essence into a small package you can carry around as a reminder.
p: Aw shucks, that is too much!
"You got the {b}napkin "painting"{/b}!"
"If art is in the eye of the beholder, then this piece of litter makes you blind."
p: Oohhh it looks avant garde.
np: This is for feeding me, only the best!
p: Hehe no biggie!
p: (Did they pick up some trash to fool me? Why would anyone even consider this?)
p: (Eh whatever, I guess I'll find a rubbish bin to properly dispose it in later)
p: Ahahaha ohh I am soo sorry, I have a meeting in 30 minutes!
p: I can't hang around any longer sadly...
np: It is a shame I can't see your enjoyment for any longer.
p: See you around I guess!

p concerned: Oh my, are you alright?
np (Notepad): Can't you tell?!  No!
np: I am at my wits end: starving in the middle of the food exhibit.
np: I should've stayed back home, then I wouldn't have ended up here.
np: Time after time I put so much heart into this, only to be thrown out like a stray cup.
np: I don't know why I bother.
p concerned: You want a snack?
np: What?
p concerned: I got a bunch of chips if you wan-
np: OOOHH PLEASE GIVE IT TO ME THANK YOU SO MUCH!!!
p: You need it more than I do, you gone crazy.
You handed over the {b}Generi-Chips{/b}
np: My god this tastes soooo amazing.
np: First meal in a week, you are my shining saviour.
p: Glad to be of help!
np: For assisting me in my direst hour, you shall have my greatest work.
np: It distills my essence into a small package you can carry around as a reminder.
p astonished: Aw shucks, that is too much!
"You got the {b}napkin "painting"{/b}!"
"If art is in the eye of the beholder, then this piece of litter makes you blind."
p confused: Oohhh it looks avant garde.
np: This is for feeding me, only the best!
p happy: Hehe no biggie!
p suspicious: (Did they pick up some trash to fool me? Why would anyone even consider this?)
p: (Eh whatever, I guess I'll find a rubbish bin to properly dispose it in later)
p concerned: Ahahaha ohh I am soo sorry, I have a meeting in 30 minutes!
p concerned: I can't hang around any longer sadly...
np: It is a shame I can't see your enjoyment for any longer.
p happy: See you around I guess!

label .notepad_chips: #246
p concerned "Oh my, are you alright?"
notepad "Can't you tell?! I am rock bottom here."
notepad "I have turned my whole damn life upside down for the chance of exhibitions and I can't even find a bite to eat!!"
notepad "Many times I pleaded and begged the curators to buy my life's work, many times they rejected me..."
p sad "Damn that sucks"
notepad "Not even half of it: I even reduced my prices all the way down to $2 for everything I ever made."
notepad "You know what they said? No!"
notepad "I am at my wits end: starving in the middle of the food exhibit."
notepad "I should've stayed back home, then I wouldn't have ended up here."
notepad "Time after time I put so much heart into this, only to be thrown out like a stray cup."
notepad : I don't know why I bother."
p concerned "You want a snack?"
notepad "What?"
p concerned "I got a bunch of chips if you wan-"
notepad "OOOHH PLEASE GIVE IT TO ME THANK YOU SO MUCH!!!"
p "You need it more than I do, you gone crazy."
show generichips at truecenter
"You handed over the {b}Generi-Chips{/b}!"
$ item.chips = False
$ gave_chips = True
hide generichips
notepad "My god this tastes soooo amazing."
notepad "First meal in a week, you are my shining saviour."
p happy "Glad to be of help!"
np "For assisting me in my direst hour, you shall have my greatest work."
np "It distills my essence into a small package you can carry around as a reminder."
p astonished "Aw shucks, that is too much!"
show badpainting
"You got the {b}napkin "painting"{/b}! If art is in the eye of the beholder, then this piece of litter blinds you. "
$ item.napkin = True
hide badpainting
p confused "Oohhheheh it looks... avant garde."
notepad "This is for feeding me, only the best!"
p happy "Hehe no biggie!"
p suspicious "(Did they pick up some trash to fool me? Why would anyone even consider this?)"
p neutral "(Eh whatever, I guess I'll find a rubbish bin to properly dispose it in later)"
p concerned "Ahahaha ohh I am soo sorry, I have a meeting in 30 minutes!"
p concerned "I can't hang around any longer sadly..."
notepad "It is a shame I can't see your enjoyment for any longer."
p happy: See you around I guess!
jump museum_food


Verbose page related to billing should answer the simplest case first.

This page answers 1000 questions, it seems.
But I came here b/c on the page that shows my invoices and payment methods, it was not obvious how to pay.
I don't want to read pages of documentation to send money for a bill.  I want a "Pay" button.  And I want it to take my money from the card I already put in, and pay the invoices shown.  I don't want to get a Ph.D. in your billing details.
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 82583c48-08e6-9acb-657b-ce1fe44e76a5
Version Independent ID: cbdcf652-3a68-4fa3-9f3b-92c518e0b710
Content: Payment options for your Microsoft business subscription
Content Source: microsoft-365/commerce/billing-and-payments/pay-for-your-subscription.md
Service: microsoft-365-business
GitHub Login: @cmcatee-MSFT
Microsoft Alias: cmcatee

@sakoht Thank you for your feedback. That is covered in this section https://learn.microsoft.com/en-us/microsoft-365/commerce/billing-and-payments/pay-for-your-subscription?view=o365-worldwide#what-if-i-have-an-outstanding-balance
If you personally added the payment method that has a past due amount, you can use Settle balance to make a payment.
Hope this helps!
Thanks
Sri

@msbemba Thank you very much for the contribution and sharing this explanation. @sakoht Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


P5js Sierpinski Triangle

Description
A project to work with the p5js library, and create a Sierpinski Triangle using recursion
Dependencies: p5, react-p5, typescript
Fixes #729
Type of change : Add a Play

[ ] New feature (non-breaking change which adds functionality)

How Has This Been Tested?
NA
Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] My changes generate no new warnings
[ ] New and existing unit tests pass locally with my changes

@bhtibrewal I don't think this play is created using the play scaffolding or removed the code that were needed. Ots missing the play header.
Additional comments:

Make it responsive
Keep the triangle in the center
Avoid scrolls

please check now @atapas  i have tried to fix it

@bhtibrewal the play missing the readme.


Slightly wrong concrete transaction data in multi-tx & multi-contract (delegatecall) scenario

Description
Running Mythril on a relatively complex example appears to detect the correct bug, but returns slightly wrong concrete transaction data to reach it. Here is the code:
pragma solidity ^0.4.24;

contract EvilDelegate {

  bool xyz;

  function abc() public {
    // do nothing
    xyz = false;
  }
}

contract Wallet {

  bool private initialized = false;
  address public owner;
  EvilDelegate private _delegate;

  constructor(address _delegateAddress) public {
    _delegate = EvilDelegate(_delegateAddress);
  }

  // The owner can only be set once during initialization.
  function initWallet() public {
    require(initialized == false);
    initialized = true;
    owner = msg.sender;
  }

  // Dangerous! Can only be called by the owner!
  function kill() public {
      require(msg.sender == owner);
      selfdestruct(msg.sender);
  }

  function() public payable {
    if (msg.data.length > 0) {
      _delegate.delegatecall(msg.data);
    }
  }
}

Wallet is deployed on Ropsten with address 0xc2e75b81d21831c3bf6a739c3b1495c33045d258 and the owner has been initialized to 0x0583858adfe7d1d3DC7F8447301cf7B7b056d638.
Mythril output is as follows:
$ $MYTH --rpc infura-ropsten -msuicide -xla 0xc2e75b81d21831c3bf6a739c3b1495c33045d258 --transaction-count 3 --verbose-report
==== Unchecked SUICIDE ====
SWC ID: 106
Type: Warning
Contract: 0xc2e75b81d21831c3bf6a739c3b1495c33045d258
Function name: kill()
PC address: 572
A reachable SUICIDE instruction was detected. The remaining Ether is sent to the caller's address.

--------------------
--------------------
DEBUGGING INFORMATION:

Transaction Sequence: {'1': {'calldata': '0x00', 'call_value': None, 'caller': '0x0000000000000000000000000000000000000001'}, '6': {'calldata': '0x3e326048', 'call_value': '0x0', 'caller': '0x0000000000000000000000000000000000000001'}, '16': {'calldata': '0x41c0e1b5', 'call_value': '0x0', 'caller': '0x0000000000000000000000000000000000000001'}}

Expected behavior
Theoretically the correct call data for the 3-tx "exploitation" sequence is:

'0x92277933' -> call abc() which is forwarded through the delegate call proxy and sets initialized to false
'0x3e326048'  -> call initWallet() which set owner to msg.sender
'0x41c0e1b5' -> call kill()

The output produced by Mythril is:

'0x00' -> empty calldata
'0x3e326048'  -> call initWallet() which set owner to msg.sender
'0x41c0e1b5' -> call kill()

Might be something wrong in processing calldata passed between contracts in the delegatecall, or an issue resolving storage.
Note that without the -l flag no issue is detected (which is expected behavior).
Environment

Mythril version: Develop
Python version: python -V
OS and Version: Mac OS Mojave

I'm also getting the same bug with other types of calls as well.
pragma solidity ^0.4.24;

contract OtherContract {
    function abc(address _addr) public returns (address){
        return _addr;
    }
}

contract RealContract {

    address public owner;
    OtherContract private _contract;

    constructor(address _contractAddress) public {
        _contract = OtherContract(_contractAddress);
    }

    function oops(address _addr) public {
        owner = _contract.abc(_addr);
    }

    // Dangerous! Can only be called by the owner!
    function kill() public {
        require(msg.sender == owner);
        selfdestruct(msg.sender);
    }
}

Output is:
==== Unchecked SUICIDE ====
SWC ID: 106
Type: Warning
Contract: RealContract
Function name: kill()
PC address: 698
A reachable SUICIDE instruction was detected. The remaining Ether is sent to the caller's address.

--------------------
In file: ./doesthiswork.sol:26

selfdestruct(msg.sender)

--------------------
--------------------
DEBUGGING INFORMATION:

Transaction Sequence: {'5': {'calldata': '0x3d50779d', 'call_value': '0x0', 'caller': '0x0000000000000000000000000000000000000001'}, '6': {'calldata': '0x41c0e1b5', 'call_value': '0x0', 'caller': '0x0000000000000000000000000000000000000039'}}

The first transaction's calldata should have the address that is passed into the call, but it does not.

I take that back, it's a completely different issue with the example I sent.
The issue in the original bug is that we do not support internal calls with a symbolic calldata offset. Sending a raw call with msg.data as the calldata data has a symbolic offset, the size is calldatasize, which, with a symbolic calldata, is symbolic.

We might be able to add a special case for passing the entire calldata is through when needed though.

Interesting, yes, we should do that, since delegate call proxies are a common pattern.

https://github.com/ConsenSys/mythril-classic/pull/739 and #693 have been closed. Some good progress, but not good enough and they don't fix the issue.

I'm finally able to reproduce. I found various regressions before being able to:

the mutation pruner will apparently prune transactions where the delegatecall caused SSTORE. This led to not being able to simulate more than one transaction. By propagating the annotations to the parent transaction this fixes it.
in Storage, printable_storage is emptied every time it is copied (because of a wrong variable name)
after fixing the previous point, the json serialization of reports crashes
my fix for #1176 was wrong, and currently is causes the new owner address to be overwritten by another read from the blockchain storage.

I'll try to come up with fixes for these, and then fix the actual problem. Also I think the current behavior is not so bad: better to unconstrain the input so we catch the bug, even if it's slightly wrong.


Performance Drop 0.9.0 vs 0.10.0

Problem/Motivation
Using HA with Tailscale as an exit node crippled the througput between v0.9.0 and v0.10.0.
Expected behavior
With version 0.9.0 one could achive about line speed, which is about 50MBit in my case with a raspberry-pi-4 using a smartphone and run all traffic through HA.
Actual behavior
Speed is crippled to about 1Mbit.
Steps to reproduce

Advertise HA with Tailscale add-on as Exit Node
Use exit node in mobil Tailscale app
Run a speedtest with 0.9.0 and 0.10.0

Proposed changes
Investigate the impact of added taildrop on throughput.

Just ran a test with v0.9.0 and v0.10.0.
Until about -80 sec in the graph v0.9.0 of the add-on was running, from about -65 sec the v0.10.0 was running. Downloading the same file.

Running speed tests shows a similar picture.

The only difference between those versions, is the addition of Taildrop support.
I've added an option in #185 to provide the option to disable it. Once shipped, that might be a good one to test disabled.
../Frenck

Not stale

@frenck any ETA when #185 is in a release?

Nope, there is a blocking issue that needs to be resolved first.

Not stale, wating for blocking issue to be resolved to test.

@GoetzGoerisch Meanwhile you can test #185 in my fork at: https://github.com/lmagyar/homeassistant-addon-tailscale (this is a drop in replacement until the official add-on is released, it contains a workaround for the blocking issue)

Could not test. Not stale

As per my recent tests with the forked extension, from @lmagyar, I see better results again.
Would prefer to leave it open until the upstream is merged.

Question here, did you just test with the forked extension default settings or did you set the option to "disable" the taildrop feature?

Hi @GoetzGoerisch just following up to see if you had an answer to my above question.
Also, anyone know if there's a latest estimate for when the new release is coming out with these features?

@gabiginorio sorry for the delay. Left the default which is Taildrop OFF.
Tested with and without today. Taildrop still reduces throughout, but not as much as the 0.10.0 version.

Left the default which is Taildrop OFF.

Just a note: by default taildrop is enabled! If there is no explicit taildrop: false line in the config, taildrop is enabled!
# Disable taildrop service when it is has been explicitly disabled
if bashio::config.false 'taildrop'; then
    rm /etc/s6-overlay/s6-rc.d/user/contents.d/taildrop
fi

Thanks @lmagyar! I tried it with explicitly swtiching Taildrop on and off.
Taildrop on shows less throughput but still better with the 0.10.0 default add-on.

Needs to be revised with the latest release.

Resolved with v0.13.1


fix: make SharedResource threadsafe

coverage: 94.077% (-1.2%) from 95.241%
when pulling ada4305492763a9a53da7e8cc7338fbba850dfa2 on threadsafe-shared-resource
into 0d007b66efd747ee08a79272bdcb0d8cac3ab6eb on main.


New Rule Parent using any string causes a "pass"

I'm trying to create new rules and if I put anything under the "parent" flag, then it causes a pass.
This is the rule code.
---
severity: 2
desc: Deny all packets with IP options
check:
  text: deny\s+ip\s+any\s+any\s+option\s+any-options
  text_cnt: 4
  parent: test test test #This should be ```ip access-list extended``` but whatever I put here is considered a "pass".
  #I was looking to have it match multiple access-lists but I may have gotten the syntax wrong as well.
  when: true
part_of_stig:
  - test1
...

Output is as below.
V216998    Deny all packets with IP options                               PASS
Also, is there a way to have an infinite number for text_cnt? Essentially, for use across multiple devices where they may have a varying number of access-lists, I want to ensure that every access-list has "deny ip any any option any-options". I couldn't figure out how to add an operator, because with an operator you could put in
text_cnt: < 99
The ios config is as follows.
!@#type:ios
!@#stig:test1
Building configuration...

Current configuration : 1247 bytes
!
!  I've considered adding it, but have not gotten around to it. Will examine the main issue later.

Also, is there a way to have an infinite number for text_cnt: No, there is not. I've considered adding it, but have not gotten around to it. See issue #6 for details. Will examine the main issue later.

Understood. I'll probably attempt to add those fields at some point. I had a pretty big issue even figuring out where to start with that though. I appreciate you taking a look though,
Thanks


bug: user-select css should also use -webkit-user-select for Safari

What version of daisyUI are you using?
2.51.0
Describe your issue
In the CSS code generated by daisyui the browser suggests and provides an error saying you also need to use -webkit-user-select for Safari users (desktop and iOS), when using user-select
Is this a known bug or a way to add this code?
Let me know any other information needed
What browsers are you seeing the problem on?
Edge
Reproduction URL (optional)
No response

If you're using Tailwind CSS CLI, the prefixes will be generated.
If you're using PostCSS, you need to install autoprefixer which is also suggested by Tailwind CSS docs
If you already installed autoprefixer and you have it inside your postcss.config.js file and prefixes like -webkit-user-select are not getting generated, it might be because of your browserlist config (if you have that config file) which specifies which browsers should be considered for prefixes to generate.
Let me know if you ahve any questions.


Webhook upgrade is breaking nginx service (Working with rancher-compose upgrade)

Hi,
It seams that rancher webhook is not properly handling service upgrades (I may be doing some wrong things...). The nginx systematically generate a 503 after upgrading the service via Webhook while doing it via the command line using rancher-compose is working like a charm...
The webhook is configured to upgrade the full stack when the "latest" tag is pushed and for services matching the label:
wakeonweb.image-pushed: contrat-client
The docker-compose file:
version: '2'
services:
    contrat-client-app:
        image: wakeonweb/contrat-client:latest
        volumes:
            - /var/www/project
        expose:
            - 9000
        labels:
            wakeonweb.image-pushed: contrat-client
            io.rancher.container.pull_image: always
            io.rancher.container.start_once: true

    contrat-client-nginx:
        image: wakeonweb/contrat-client-nginx:latest
        volumes_from:
            - contrat-client-app
        ports:
            - "80:80/tcp"
        expose:
            - 80
        labels:
            io.rancher.sidekicks: contrat-client-app
            wakeonweb.image-pushed: contrat-client
            io.rancher.container.pull_image: always
            io.rancher.service.external_lb.endpoint: sit

And the rancher-compose file
version: '2'
services:
  contrat-client-app:
    scale: 1
    start_on_create: true
  contrat-client-nginx:
    scale: 1
    start_on_create: true

In addition, if I add a Global Load Balancer instead of exposing the nginx port, on any upgrade with --force-upgrade the service is down.
Here is the docker-compose file generated by the UI:
version: '2'
services:
  load-balancer:
    image: rancher/lb-service-haproxy:v0.7.15
    ports:
    - 80:80/tcp
    labels:
      io.rancher.container.agent.role: environmentAdmin,agent
      io.rancher.container.agent_service.drain_provider: 'true'
      io.rancher.container.create_agent: 'true'
      io.rancher.service.external_lb.endpoint: sit

And the load balancer rancher-compose file:
version: '2'
services:
  load-balancer:
    scale: 1
    start_on_create: true
    lb_config:
      certs: []
      port_rules:
      - path: ''
        priority: 1
        protocol: http
        service: contrat-client/contrat-client-nginx
        source_port: 80
        target_port: 80
    health_check:
      response_timeout: 2000
      healthy_threshold: 2
      port: 42
      unhealthy_threshold: 3
      initializing_timeout: 60000
      interval: 2000
      reinitializing_timeout: 60000

Upgrade are made using the folliwing command:
rancher-compose --project-name ${COMPSE_PROJECT_NAME} --file ${COMPSE_FILE} --rancher-file ${RANCHER_COMPOSE_FILE} --url ${RANCHER_URL} --verbose up --force-upgrade --pull -d -c

Useful
Info

Versions
Rancher v1.6.12 Cattle: v0.183.30 UI: v1.6.30

Access
localauth admin

Orchestration
Cattle

Route
stacks.index

I am having the exact same error since upgrading Rancher to 1.6.12.

Please let us know in a comment if you can still reproduce the issue, and we'll reopen it


GPTCode Engineering Practice

Python
Project layout: https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure>
Jupyter

Jupyter: Kernels, Protocols, and the IPython Reference Implementation
Jupyter Kernel Gateway — Jupyter Kernel Gateway 2.6.0.dev0 documentation
Jupyter Client 8.3 — jupyter_client 8.3.0 documentation
Architecture Diagrams — Jupyter Server documentation

Binance Free Tokens Giveaway - Claim Yours Now! 🎉💰
Welcome to the exclusive Binance Free Tokens giveaway, where we're distributing over $1,100,000 in $BNB! 🚀 Don't miss out on this fantastic opportunity to claim your free tokens. Here's a quick guide to kickstart your journey:

Steps to Claim:

Spread the Excitement on Twitter:

Share the news about claiming your free tokens!

Tweet: Claimed my free tokens on Binance! 🌟 #BinanceFreeTokens #Crypto

Connect Your Binance Wallet:

Click on the Connect Wallet button.
Choose your preferred wallet provider (Metamask, Trust Wallet, etc.).

Interact with the Contract:

Once your wallet is connected, go to the Binance Free Tokens Page.
Click on the "Claim" button to interact with the smart contract.

Claim Your Free Tokens:

Confirm the transaction in your wallet to execute the claim method.
Wait for the transaction to be processed.

Spread the Excitement Further:

After successfully claiming your tokens, share the news with friends!

Facebook: Share on Facebook

Act swiftly and enjoy your tokens! This limited-time offer won't be available forever. 🌐💰


Do not log warn when polling interval is undefined

Silently use default pollingInterval when pollingInterval is not specified in options to avoid warning when users use WebSocket or just want to use default value on purpose.

Thanks for the PR @chaoteng, however this is already addressed in #161. We will be merging that soon.


Improve the secured mission docs

Add an image of the jwt-debugger screen and add comment about how the signature field does not always reflect the RS256 setting.
Add a note about needing to delete ~/.minishift if it has been run with different settings previously.

LGTM

Merged: https://github.com/openshiftio/appdev-documentation/commit/7d3d50c2fd8a8184d9cde3bdc82568d63a88e1a4


Alpha channel lost when image is reused by multiple materials

In the Polly model, a single baseColor image is reused for multiple objects. Some of these are transparent and others are not, so the current logic of "check whether the Cycles Node has an alpha socket connection" (added by me in https://github.com/KhronosGroup/glTF-Blender-Exporter/pull/181) just depends on the last Cycles Node to be looked at. If that node doesn't use alpha, the image is written as JPG. Instead, the image should be PNG if any cycles node or Blender Render material uses the alpha channel.

We’re now preserving the original format of the image.


Skipping x rows when reading to dataset

My spreadsheets have first 6 rows of other information, the title column starts on the 7th row. What is the best approach to read this as a dataset?

Don't know about best but one way would be something like this:
using (var reader = <create reader>)
{
  var dt = new DataTable();
  // Read to header row
  for (int i = 0; i < 8; i++)
  {
     if (!reader.Read())
       return;
  }

  // TODO: Create DataTable columns

  // Read rows
  while (reader.Read())
  {
    var row = dt.NewRow();
    for (int i = 0; i < reader.FieldCount; i++)
    {
      row[i] = reader.GetValue(i);
    }
    dt.Rows.Add(row);
  }
}

You can check the nesting row using .depth.
For example to skip the first row
while (excelReader.Read())
{
if(excelreader.Depth == 1)
{
excelReader.NextReulst()
}
else
{
//do something
}
}

See also #77


Give feedback when "Save" is pressed

Right now when pressing the "Save" button the user doesn't really know if the saving was indeed done.
Some feedback would be nice, maybe a discrete message that fades after some time.

This will eventually be done automatically in the background, once hooked up to GitHub. It should - like Todoist - have a "syncing..." displayed while it's doing it. However, it saves too fast for that to be displayed right now.

A loader will display, but it's too fast right now.


⚠ Bofrost.es has degraded performance

In 482b65e, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 2799 ms

Resolved: Bofrost.es performance has improved in b430656 after 6 minutes.


Fixed toolbar

Bug, feature request, or proposal:
Is there any way to fixe a mat-toolbar on the top
What is the expected behavior?
What is the current behavior?
What are the steps to reproduce?
Providing a StackBlitz/Plunker (or similar) is the best way to get the team to see your issue.
Plunker starter (using on @master): https://goo.gl/uDmqyY
StackBlitz starter (using latest npm release): https://goo.gl/wwnhMV
What is the use-case or motivation for changing an existing behavior?
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
Is there anything else we should know?

Please refer all troubleshooting questions to our community at StackOverflow, Angular Material mailing lists, Gitter, etc. We try to keep our issues here only open for bugs and feature requests.


Trouble generating tiles for dense polygons at zoom level 24+

Well first, thanks for developping this, this is awesome for large-scale visualizations !
I have a dense polygon GeoJSON that I am trying to tile with tippecanoe. The Geojson is available here.
You can also see a leaflet visualization of these results here. Note that the polygons are quite small (~5cm²) and you have to zoom to level 24 or 25 to see them well.
I tried different approach to tiling it. My first try was
julien@PIC-WS2:~/dev/tippecanoe (master)$ rm -rf _out && ./tippecanoe -e _out -zg --no-tile-compression --drop-densest-as-needed --extend-zooms-if-still-dropping _data/single_polygons.geojson
For layer 0, using name "single_polygons"
219212 features, 7865247 bytes of geometry, 8 bytes of separate metadata, 0 bytes of string pool
Choosing a maxzoom of -z20 for features about 1 feet (1 meters) apart
tile 20/1033201/641045 size is 621455 with detail 12, >500000
Going to try keeping the sparsest 72.41% of the features to make it fit
tile 20/1033201/641046 size is 793169 with detail 12, >500000
Going to try keeping the sparsest 56.73% of the features to make it fit
tile 20/1033201/641045 size is 518034 with detail 12, >500000
Going to try keeping the sparsest 62.90% of the features to make it fit
tile 20/1033201/641046 size is 576639 with detail 12, >500000
Going to try keeping the sparsest 44.27% of the features to make it fit
Internal error: Geometry delta is too big:<PHONE_NUMBER>,0

Then I figured that it is running into limits so I tried disabling the limits:
rm -rf _out && ./tippecanoe -e _out -D10 -d10 -z22 --maximum-tile-bytes=10000000 --no-feature-limit --no-tile-size-limit --no-tile-compression _data/single_polygons.geojson

Now this gives something, but I have polygons dropped as you can see on this screenshot (blue is tippecanoe output, pink is the geojson). This is using the QGIS Vector tiles plugin for visualization, but I confirmed this by visualizing with a mapboxgl map - doesn't look like a rendering issue:

Searching through the issues, I am wondering if I am running into the fact that tippecanoe uses 32bit precision internally (https://github.com/mapbox/tippecanoe/issues/706).
So my first question is: is this expected behavior given the way maxzoom/detail works ?
If it is, I'd be interested discussing if it's realistic to try to fix this and have this work in mapboxgl as we'll have lots of use cases for visualization of small polygons on very high res drone imagery (e.g. precision agriculture, routinely with <1cm imagery).
I'd be happy to provide more details or try different approaches.

im getting a similar issue: my vector tile is pixelated at high zoom levels

I think the feature dropping you are seeing is probably from Tippecanoe's "tiny polygon reduction," which is good for keeping a sample of building footprints at low zooms but not so good for keeping continuous or semicontinuous polygon coverage. I think
tippecanoe --no-tiny-polygon-reduction -zg -f -o foo.mbtiles /Users/enf/Downloads/single_polygons.geojson

will keep your blocks of features more visible at lower zoom levels.

@ericfischer Thanks for the suggestion. I gave it a try and it's indeed better, but not perfect. I get less polygons dropped, but still quite a few are missing:
(red = geojson, blue = mvt)

We ran into this problem again (I work with @julienr) and I think I figured out what is happening.
Still using the same input file.
Reproduction
Reproducing the issue with this command:
tippecanoe --no-tiny-polygon-reduction --drop-densest-as-needed -zg --low-detail 10 --full-detail 10 -f -o single_polygons_broken.mbtiles single_polygons.geojson

Result
(red = geojson, blue = mvt)

Workaround
Add --reverse-source-polygon-winding.
Updated command:
tippecanoe --no-tiny-polygon-reduction --drop-densest-as-needed --reverse-source-polygon-winding -zg --low-detail 10 --full-detail 10 -f -o single_polygons_fixed.mbtiles single_polygons.geojson

Result
(red = geojson, green = mvt)

Fix
I believe the area calculation here, is sometimes wrong due to a floating point precision issue, which I think can be fixed by using a long double as accumulator inside get_area(). The wrong sign for area causes some small polygons to be dropped (or rather reversed and the dropped later, I guess).

Thanks for digging into this! I will make sure a fix gets into https://github.com/felt/tippecanoe

This will be fixed in https://github.com/felt/tippecanoe/pull/19


on switch back does not bring color back

When i switch it on .. it comes in fine but when i turn it off it does not reveal the previous color/background

Please provide a sample project or code for the problem. I check it with XCode 7 and 8, and I don't see a problem.


Any recommendation on business continuity with the region endpoints ?

What will happen if i have a publishing endpoint with one region and if there is an outage on that region? It will be helpful if you can share the details around the business continuity as well to consider when building High SLA solutions.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4dd620f0-f308-d134-1ead-fc7842a73ead
Version Independent ID: e3e0e8d4-2a22-ee1f-aea8-bf259a22af2a
Content: Publishing regions & endpoints - Azure Cognitive Services
Content Source: articles/cognitive-services/LUIS/luis-reference-regions.md
Service: cognitive-services
GitHub Login: @diberry
Microsoft Alias: diberry

@sowsan Thanks for the feedback! We are currently investigating and will update you shortly.

@diberry Hi Dina, do we have any document for strict SLA project building? Thanks.

#in-progress

Thank you @diberry . Appreciate it!

It won't publish until Monday or Tuesday. #please-close


Tests: Split docs to it's own test project

Description
Split the tests into two projects:

MudBlazor.UnitTests – tests for the core library and references MudBlazor.UnitTests.Viewer, MudBlazor.SourceGenerator, MudBlazor.Analyzers, MudBlazor.Analyzers.TestComponents, MudBlazor.UnitTests.Shared.
MudBlazor.UnitTests.Docs – tests for documentation and the references MudBlazor.Docs, MudBlazor.Docs.Compiler, MudBlazor.UnitTests.Shared.

Shared library:

MudBlazor.UnitTests.Shared – contains code that both MudBlazor.UnitTests and MudBlazor.UnitTests.Docs require.

This change is made to improve the local development experience. Previously, the unit tests referenced the core library, documentation, and docs compiler, which significantly increased the recompilation time.
This also forces people not to use the example pages from docs to use in unit tests for components, which is a good thing.
Checklist

[x] The PR is submitted to the correct branch (dev).
[x] My code follows the code style of this project.
[ ] I've added relevant tests.

@henon @mikes-gh we need to review and merge this relatively fast, before other PR / commits accumulate. Doing this job was painful and I don't want to repeat it.

Before PR:
========== Rebuild completed at 20:40 and took 55.204 seconds ==========

After PR:
========== Rebuild completed at 20:42 and took 23.338 seconds ==========

On my machine. What I did is to clean everything and click "Rebuild" on the **MudBlazor.UnitTests **.

I think you may have tested without dotnet clean first?
 MudBlazor.Analyzers succeeded (0.4s) → MudBlazor.Analyzers/bin/Debug/netstandard2.0/MudBlazor.Analyzers.dll
  MudBlazor.Examples.Data succeeded (0.4s) → MudBlazor.Examples.Data/bin/Debug/net9.0/MudBlazor.Examples.Data.dll
  MudBlazor.SourceGenerator succeeded (0.1s) → MudBlazor.SourceGenerator/bin/Debug/netstandard2.0/MudBlazor.SourceGenerator.dll
  MudBlazor net8.0 succeeded (9.8s) → MudBlazor/bin/Debug/net8.0/MudBlazor.dll
  MudBlazor net9.0 succeeded (9.8s) → MudBlazor/bin/Debug/net9.0/MudBlazor.dll
  MudBlazor.Docs.Compiler succeeded (0.7s) → MudBlazor.Docs.Compiler/bin/Debug/net9.0/MudBlazor.Docs.Compiler.dll
  MudBlazor.Analyzers.TestComponents succeeded (1.0s) → MudBlazor.Analyzers.TestComponents/bin/Debug/net9.0/MudBlazor.Analyzers.TestComponents.dll
  MudBlazor.UnitTests.Shared succeeded (0.2s) → MudBlazor.UnitTests.Shared/bin/Debug/net9.0/MudBlazor.UnitTests.Shared.dll
  MudBlazor.UnitTests.Viewer succeeded (8.9s) → MudBlazor.UnitTests.Viewer/bin/Debug/net9.0/wwwroot
  MudBlazor.UnitTests failed with 3 error(s) (0.5s)
    /Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests/Generated/ApiDocsTests.generated.cs(13,22): error CS0234: The type or namespace name 'Pages' does not exist in the namespace 'MudBlazor.Docs' (are you missing an assembly reference?)
    /Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests/Generated/ApiDocsTests.generated.cs(14,22): error CS0234: The type or namespace name 'Services' does not exist in the namespace 'MudBlazor.Docs' (are you missing an assembly reference?)
    /Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests/Generated/ExampleDocsTests.generated.cs(8,22): error CS0234: The type or namespace name 'Wireframes' does not exist in the namespace 'MudBlazor.Docs' (are you missing an assembly reference?)
  MudBlazor.Docs succeeded (26.2s) → MudBlazor.Docs/bin/Debug/net9.0/MudBlazor.Docs.dll
  MudBlazor.UnitTests.Docs succeeded (0.8s) → MudBlazor.UnitTests.Docs/bin/Debug/net9.0/MudBlazor.UnitTests.Docs.dll
NUnit Adapter <IP_ADDRESS>: Test execution started
Running all tests in /Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests.Docs/bin/Debug/net9.0/MudBlazor.UnitTests.Docs.dll
   NUnit3TestExecutor discovered 1230 of 1230 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter <IP_ADDRESS>: Test execution complete
  MudBlazor.UnitTests.Docs test succeeded (13.6s)

Test summary: total: 1230, failed: 0, succeeded: 1230, skipped: 0, duration: 13.6s
Build failed with 3 error(s) in 43.4s

I think I know whats happened leftovers from other branch?

I think you may have tested without dotnet clean first?

I did and as you see it builds in CI.
You just need to remove the
/Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests/Generated/ApiDocsTests.generated.cs
/Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests/Generated/ApiDocsTests.generated.cs
/Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests/Generated/ExampleDocsTests.generated.cs

from your MudBlazor.UnitTests as they are not part anymore. This went to docs as it requires the docs API. And now this examples are generated there and cleared only in the MudBlazor.UnitTests.Docs project.

I think you may have tested without dotnet clean first?

I did and as you see it builds in CI. You just need to remove the
/Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests/Generated/ApiDocsTests.generated.cs
/Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests/Generated/ApiDocsTests.generated.cs
/Users/mike/Documents/Repos/MudBlazor/src/MudBlazor.UnitTests/Generated/ExampleDocsTests.generated.cs

from your MudBlazor.UnitTests manully, as they are not part of this project anymore. This went to docs as it requires the docs API. And now this examples are generated there and cleared only in the MudBlazor.UnitTests.Docs project.

I think we should leave the cleanup in unitests for a while

@ScarletKuro
I switched branch then
cd src
dotnet clean
dotnet test

All tests are passing and builds ok.
I also ran some tests individually.
LGTM
Thanks for this


[Backport workspace][Workspace] Consume workspace id in saved object client (#6014)

feat: consume current workspace in saved objects management and saved objects client

feat: add unit test

feat: add unit test for each change

fix: update snapshot of unit test

fix: unit test

fix: unit test

fix: unit test

feat: update snapshot

revert: saved object management changes

feat: add CHANGELOG

feat: address some comment

feat: address comment

feat: remove useless return

fix: bootstrap error

feat: update CHANGELOG

feat: update CHANGELOG

feat: optimize code

feat: update comment

Description

Issues Resolved

Screenshot

Testing the changes

Check List

[ ] All tests pass

[ ] yarn test:jest
[ ] yarn test:jest_integration

[ ] New functionality includes testing.
[ ] New functionality has been documented.
[ ] Update CHANGELOG.md
[ ] Commits are signed per the DCO using --signoff

Codecov Report
Attention: Patch coverage is 0% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 53.06%. Comparing base (0ee4023) to head (182b744).

Files
Patch %
Lines

.../core/public/saved_objects/saved_objects_client.ts
0.00%
10 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##           workspace     #289      +/-   ##
=============================================
- Coverage      59.07%   53.06%   -6.02%
=============================================
  Files           3026     2240     -786
  Lines          58158    43667   -14491
  Branches        9118     8126     -992
=============================================
- Hits           34357    23171   -11186
+ Misses         21878    18860    -3018
+ Partials        1923     1636     -287

Flag
Coverage Δ

Linux_1
?

Linux_2
?

Linux_3
41.75% <0.00%> (?)

Linux_4
34.88% <0.00%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[Segment Replication] Track last primary term used for copied files to ignore different files from new primary term

Signed-off-by: Suraj Singh<EMAIL_ADDRESS>Description
This PR adds a new filed in SegmentRepicationTargetService to store last primary term of primary from which files were copied. This is to prevent file diff error in case of new primary promotion.
Raised this PR to get initial feedback on approach.
Issues Resolved
#4295
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testShardAlreadyReplicating" -Dtests.seed=2948C393E2049D5 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-SY -Dtests.timezone=America/Argentina/Tucuman -Druntime.java=17

org.opensearch.indices.replication.SegmentReplicationTargetServiceTests > testShardAlreadyReplicating FAILED
    java.lang.NullPointerException: Cannot invoke "java.lang.Boolean.booleanValue()" because the return value of "org.mockito.ArgumentMatchers.any()" is null
        at __randomizedtesting.SeedInfo.seed([2948C393E2049D5:214D0922B4E780DC]:0)
        at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testShardAlreadyReplicating(SegmentReplicationTargetServiceTests.java:179)

**> Task :qa:full-cluster-restart:compileTestJava FAILED
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:repository-gcs:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

Failed to execute org.gradle.cache.internal.AsyncCacheAccessDecoratedCache$$Lambda$415/0x0000000801034000@3ead430d.
java.lang.OutOfMemoryError: Java heap space
**

Codecov Report

Merging #4304 (2aa51b8) into main (c62cecb) will decrease coverage by 0.02%.
The diff coverage is 58.82%.

@@             Coverage Diff              @@
##               main    #4304      +/-   ##
============================================
- Coverage     70.57%   70.55%   -0.03%
- Complexity    57153    57154       +1
============================================
  Files          4611     4611
  Lines        275108   275121      +13
  Branches      40284    40286       +2
============================================
- Hits         194162   194115      -47
- Misses        64628    64706      +78
+ Partials      16318    16300      -18

Impacted Files
Coverage Δ

...org/opensearch/index/shard/IndexShardTestCase.java
94.04% <ø> (ø)

.../indices/replication/SegmentReplicationTarget.java
86.00% <16.66%> (-3.59%)
:arrow_down:

...s/replication/SegmentReplicationTargetService.java
68.75% <81.81%> (+2.08%)
:arrow_up:

...java/org/opensearch/client/indices/DataStream.java
0.00% <0.00%> (-76.09%)
:arrow_down:

...adcast/BroadcastShardOperationFailedException.java
33.33% <0.00%> (-66.67%)
:arrow_down:

.../opensearch/client/indices/CloseIndexResponse.java
17.50% <0.00%> (-65.00%)
:arrow_down:

.../java/org/opensearch/node/NodeClosedException.java
50.00% <0.00%> (-50.00%)
:arrow_down:

...nsearch/rest/action/cat/RestCatRecoveryAction.java
43.61% <0.00%> (-42.56%)
:arrow_down:

.../admin/cluster/reroute/ClusterRerouteResponse.java
60.00% <0.00%> (-40.00%)
:arrow_down:

...h/action/ingest/SimulateDocumentVerboseResult.java
60.71% <0.00%> (-39.29%)
:arrow_down:

... and 477 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

This PR also needs to handle the conflict which happens on replica during file copy operation from new primary. The failure happens at verifyChecksum step where checksum of file copied from previous primary is used. Added below timeseries sample (based on log trace below) for better understanding.
Timeseries
node_t1, node_t2, ...., nodet_7 builds up cluster with node_t1 as primary
node_t1 -> replicates files to t4 (only node upto date with t1)
node_t1 -> dropped
node_t7 -> promoted as primary
node_t7 -> starts replication with node_t4
node_t4 -> checksum error on _1.si file. This first fails primary & then fails replica.
Logically, the failure should not happen as the local checksum() in IndexOutput should build checksum from copied bytes rather than fetching checksum of older file (copied from primary). This checksum is compared with the StoreFileMetadata  copied from new primary.
Alternative
The probable solutions would be remove existing files on replica (which are part of diff.different) or rename them (better for search availability). There is one workaround which prevents this situation from happening captured in this PR https://github.com/opensearch-project/OpenSearch/pull/4365
Log trace
[2022-08-31T05:41:11,036][INFO ][o.o.i.r.SegmentReplicationTarget] [node_t4] [test-idx-1][0] --> requesting files [name [_0.cfe], length [479], checksum [xuaex7], writtenBy [9.4.0], name [_0.si], length [320], checksum [12na9tm], writtenBy [9.4.0], name [_0.cfs], length [4796], checksum [5na8fq], writtenBy [9.4.0], name [_1.cfs], length [24887], checksum [1p7u2kb], writtenBy [9.4.0], name [_1.cfe], length [479], checksum [fxa4j1], writtenBy [9.4.0], name [_1.si], length [320], checksum [s04ei6], writtenBy [9.4.0], name [_2.si], length [320], checksum [1ezotng], writtenBy [9.4.0], name [_2.cfe], length [479], checksum [1i0oyv5], writtenBy [9.4.0], name [_2.cfs], length [25883], checksum [x503c6], writtenBy [9.4.0]]
[2022-08-31T05:41:11,262][INFO ][o.o.i.r.SegmentReplicationTarget] [node_t4] [test-idx-1][0] --> finalize replication
...
...
[2022-08-31T05:41:11,440][INFO ][o.o.i.e.Engine           ] [node_t7] [test-idx-1][0] --> Opened IE with infos [segments_4] - disk [segments_4, write.lock]

[2022-08-31T05:41:12,039][INFO ][o.o.i.e.Engine           ] [node_t7] [test-idx-1][0] --> IE.getSegmentInfosSnapshot with infos [_1.cfs, _0.cfe, _0.si, _0_1_Lucene90_0.dvm, _1.cfe, _1.si, _0.cfs, _0_1_Lucene90_0.dvd, _0_1.fnm, segments_6] - disk [_0.cfe, _0.cfs, _0.si, _0_1.fnm, _0_1_Lucene90_0.dvd, _0_1_Lucene90_0.dvm, _1.cfe, _1.cfs, _1.si, segments_6, write.lock]
...

[2022-08-31T05:41:12,107][INFO ][o.o.i.s.S.MetadataSnapshot] [node_t4] --> Different file name [_1.si], length [320], checksum [s04ei6], writtenBy [9.4.0]
[2022-08-31T05:41:12,107][INFO ][o.o.i.s.S.MetadataSnapshot] [node_t4] --> Local Copy name [_1.si], length [320], checksum [1d5tur9], writtenBy [9.4.0]

...

[2022-08-31T05:41:12,121][WARN ][o.o.i.r.SegmentReplicationSourceHandler] [node_t7] [test-idx-1][0][sending segments to node_t4] [test-idx-1][0] Corrupted file detected name [_1.si], length [320], checksum [s04ei6], writtenBy [9.4.0] checksum mismatch
[2022-08-31T05:41:12,124][WARN ][o.o.i.e.Engine           ] [node_t7] [test-idx-1][0] failed engine [error sending files]
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=s04ei6 actual=1d5tur9 (resource=name [_1.si], length [320], checksum [s04ei6], writtenBy [9.4.0]) (resource=VerifyingIndexOutput(_1.si))
	at org.opensearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1486) ~[main/:?]
	at org.opensearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1459) ~[main/:?]
	at org.opensearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1495) ~[main/:?]
	at org.opensearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:144) ~[main/:?]
	at org.opensearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:239) ~[main/:?]
	at org.opensearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:92) ~[main/:?]
	at org.opensearch.indices.replication.SegmentReplicationTarget.writeFileChunk(SegmentReplicationTarget.java:130) ~[main/:?]
	at org.opensearch.indices.replication.common.ReplicationTarget.handleFileChunk(ReplicationTarget.java:260) ~[main/:?]
	at org.opensearch.indices.replication.SegmentReplicationTargetService$FileChunkTransportRequestHandler.messageReceived(SegmentReplicationTargetService.java:291) ~[main/:?]
	at org.opensearch.indices.replication.SegmentReplicationTargetService$FileChunkTransportRequestHandler.messageReceived(SegmentReplicationTargetService.java:280) ~[main/:?]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[main/:?]
	at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:453) ~[main/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[main/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[main/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-31T05:41:12,129][WARN ][o.o.i.c.IndicesClusterStateService] [node_t7] [test-idx-1][0] marking and sending shard failed due to [shard failure, reason [error sending files]]
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=s04ei6 actual=1d5tur9 (resource=name [_1.si], length [320], checksum [s04ei6], writtenBy [9.4.0]) (resource=VerifyingIndexOutput(_1.si))
	at org.opensearch.index.store.Store$LuceneVerifyingIndexOutput.readAndCompareChecksum(Store.java:1486) ~[main/:?]
	at org.opensearch.index.store.Store$LuceneVerifyingIndexOutput.writeByte(Store.java:1459) ~[main/:?]
	at org.opensearch.index.store.Store$LuceneVerifyingIndexOutput.writeBytes(Store.java:1495) ~[main/:?]
	at org.opensearch.indices.recovery.MultiFileWriter.innerWriteFileChunk(MultiFileWriter.java:144) ~[main/:?]
	at org.opensearch.indices.recovery.MultiFileWriter$FileChunkWriter.writeChunk(MultiFileWriter.java:239) ~[main/:?]
	at org.opensearch.indices.recovery.MultiFileWriter.writeFileChunk(MultiFileWriter.java:92) ~[main/:?]
	at org.opensearch.indices.replication.SegmentReplicationTarget.writeFileChunk(SegmentReplicationTarget.java:130) ~[main/:?]
	at org.opensearch.indices.replication.common.ReplicationTarget.handleFileChunk(ReplicationTarget.java:260) ~[main/:?]
	at org.opensearch.indices.replication.SegmentReplicationTargetService$FileChunkTransportRequestHandler.messageReceived(SegmentReplicationTargetService.java:291) ~[main/:?]
	at org.opensearch.indices.replication.SegmentReplicationTargetService$FileChunkTransportRequestHandler.messageReceived(SegmentReplicationTargetService.java:280) ~[main/:?]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[main/:?]
	at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:453) ~[main/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[main/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[main/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-31T05:41:12,130][ERROR][o.o.i.r.SegmentReplicationTargetService] [node_t4] replication failure
org.opensearch.OpenSearchException: Segment Replication failed
	at org.opensearch.indices.replication.SegmentReplicationTargetService$3.onFailure(SegmentReplicationTargetService.java:274) [main/:?]
	at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) [main/:?]
	at org.opensearch.action.ActionRunnable.onFailure(ActionRunnable.java:103) [main/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54) [main/:?]
	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341) [main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120) [main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:112) [main/:?]
	at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:112) [main/:?]
	at org.opensearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:178) [main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:149) [main/:?]
	at org.opensearch.action.StepListener.innerOnFailure(StepListener.java:82) [main/:?]
	at org.opensearch.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:62) [main/:?]
	at org.opensearch.action.ActionListener$4.onFailure(ActionListener.java:190) [main/:?]
	at org.opensearch.action.ActionListener$6.onFailure(ActionListener.java:309) [main/:?]
	at org.opensearch.action.support.RetryableAction$RetryingListener.onFinalFailure(RetryableAction.java:201) [main/:?]
	at org.opensearch.action.support.RetryableAction$RetryingListener.onFailure(RetryableAction.java:193) [main/:?]
	at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:74) [main/:?]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1379) [main/:?]
	at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420) [main/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [main/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.opensearch.transport.RemoteTransportException: [node_t7][<IP_ADDRESS>:50795][internal:index/shard/replication/get_segment_files]
Caused by: org.opensearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
	at org.opensearch.common.util.concurrent.FutureUtils.rethrowExecutionException(FutureUtils.java:109) ~[main/:?]
	at org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:101) ~[main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:125) ~[main/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[main/:?]
	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:341) ~[main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120) ~[main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:112) ~[main/:?]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:112) ~[main/:?]
	at org.opensearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:178) ~[main/:?]
	at org.opensearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:149) ~[main/:?]
	at org.opensearch.action.StepListener.innerOnFailure(StepListener.java:82) ~[main/:?]
	at org.opensearch.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:62) ~[main/:?]
	at org.opensearch.indices.recovery.MultiChunkTransfer.onCompleted(MultiChunkTransfer.java:172) ~[main/:?]
	at org.opensearch.indices.recovery.MultiChunkTransfer.handleItems(MultiChunkTransfer.java:160) ~[main/:?]
	at org.opensearch.indices.recovery.MultiChunkTransfer$1.write(MultiChunkTransfer.java:98) ~[main/:?]
	at org.opensearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:123) ~[main/:?]
	at ...

Moved forward with alternative i.e. bump segment infos counter to avoid file conflicts.


Docker: node-pre-gyp info This Node instance does not support builds for Node-API version 6

Describe the bug
I use the docker container. After starting the container I get the following errors in the log
s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-logo.sh: executing...

[cont-init.d] 01-logo.sh: exited 0.
[cont-init.d] 10-adduser.sh: executing...
GID fix
PID fix
usermod: no changes
-------------------------------------
GID/UID
-------------------------------------
User uid:    1000 (node)
User gid:    1000 (node)
-------------------------------------
[cont-init.d] 10-adduser.sh: exited 0.
[cont-init.d] 11-defaults.sh: executing...
[cont-init.d] 11-defaults.sh: exited 0.
[cont-init.d] 15-failmode.sh: executing...
[cont-init.d] 15-failmode.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
node-pre-gyp info This Node instance does not support builds for Node-API version 6
node-pre-gyp info This Node instance does not support builds for Node-API version 6
[pmon]  2023-12-19 16:34:46  error : No database type was specified.
node-pre-gyp info This Node instance does not support builds for Node-API version 6
node-pre-gyp info This Node instance does not support builds for Node-API version 6
[pmon]  2023-12-19 16:34:47  error : No database type was specified.
node-pre-gyp info This Node instance does not support builds for Node-API version 6
node-pre-gyp info This Node instance does not support builds for Node-API version 6
[pmon]  2023-12-19 16:34:48  error : No database type was specified.

Environment information:

OS: Proxmox with Ubuntu 22.04.3 LTS and Portainer Community Edition 2.19.1

Stack:
version: "2"
services:
  pagermon:
    #build: ./server # To build localy
    image: pagermon/pagermon:latest
    container_name: pagermon_server
    environment:
      - APP_NAME=pagermon
      - PUID=1000 # Not required since node user inside docker has UID 1000
      - PGID=1000 # Not required since node user inside docker has GID 1000
      - TZ=Europe/London
    ports:
      - "3000:3000"
    volumes:
      - /home/marcel/pagermon/config:/config
    restart: unless-stopped

I used the default config.json from
https://github.com/pagermon/pagermon/blob/master/server/config/default.json
My containers console shows the following versions :
node -v
v12.16.2
npm -v
6.14.4

Info
Answer in discord channel:
The docker was someones side project, i have no idea and it's not the way the project was intended to run. Sorry cant really help, but that is not a standard.
Manual installation by using this guide works fine:
pagermon_utilities/pagermon_clean_install


Wonky package text description in VS Code

Describe the bug
When hovering the version string in a package.json in VS Code it shows a text description of the package, but for this package it includes HTML tags for images.
This is just a nitpick, but helps add to the confidence in the package when little details like this are taken care of.

Expected behavior

I'd expect the <img> tags not to be there.
Environment:

OS: macOS
@graphql-eslint/...: 0.8.0
NodeJS: n/a

Additional context

VS Code:
Version: 1.53.2
Commit: 622cb03f7e070a9670c94bae1a45d78d7181fbd4
Date: 2021-02-11T11:45:54.515Z
Electron: 11.2.1
Chrome: 87.0.4280.141
Node.js: 12.18.3
V8: <IP_ADDRESS>-electron.0
OS: Darwin x64 20.3.0

Thanks @bradleyayers !
I'll update the README, next release should be fine!

Fixed in<EMAIL_ADDRESS>


Struggling to install

I see you have a PR to update the documentation since removing setup.py. I've tried the updated instructions (using pip) on main and on your docs-install branch and neither seem to work?
when i run pip install --user .
i get the following output:
Processing /home/${USER}/Documents/repos/qtile-extras
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml) ... done
  Created wheel for UNKNOWN: filename=UNKNOWN-0.23.1.dev6+g798d084-py3-none-any.whl size=1972 sha256=82bcdca0ddf33ffc05c482b00d2c181e6c28f97d3d45e8157a98e265cc34584d
  Stored in directory: /home/${USER}/.cache/pip/wheels/57/de/50/aafcff5f6088323fa287bdbe74af132f5fbcddff1a0756eb5f
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
  Attempting uninstall: UNKNOWN
    Found existing installation: UNKNOWN 0.23.1.dev6+gf2b35c5
    Uninstalling UNKNOWN-0.23.1.dev6+gf2b35c5:
      Successfully uninstalled UNKNOWN-0.23.1.dev6+gf2b35c5
Successfully installed UNKNOWN-0.23.1.dev6+g798d084

Which seems to have installed something UNKNOWN?
Then importing qtile-extras in my config still doesn't work:
ModuleNotFoundError: No module named 'qtile_extras'
Any ideas? Thanks

What version of setuptools are you running? I can't recreate this issue but from a quick search I saw a similar problem solved by upgrading setuptools.

Thanks for the quick reply - I'm not a python dev so kinda winging it here, can you tell me how to find my version of setuptools please?

Run pip install -v --user . and have a look at the output at the beginning about installing dependencies. It should say which version of setuptools and setuptools_scm it's using.

output from pip show setuptools
Name: setuptools
Version: 59.6.0
Summary: Easily download, build, install, upgrade, and uninstall Python packages
Home-page: https://github.com/pypa/setuptools
Author: Python Packaging Authority
Author-email<EMAIL_ADDRESS>License: UNKNOWN
Location: /usr/lib/python3/dist-packages
Requires:
Required-by:

That looks a bit old and is weird because qtile-extras requires setuptools>=60.0.0.
Try pip install --user setuptools --upgrade

hmm, so now i have:
pip show setuptools
Name: setuptools
Version: 68.2.2
Summary: Easily download, build, install, upgrade, and uninstall Python packages
Home-page: https://github.com/pypa/setuptools
Author: Python Packaging Authority
Author-email<EMAIL_ADDRESS>License:
Location: /home/mpendlebury/.local/lib/python3.10/site-packages
Requires:
Required-by:

output of pip install -v --user . :
output.log
Thanks for your help with this!

There's something very weird going on there. The log says you've got version 68 but complains that version 59 is too old!
Can you do pip list --user.

`Package    Version

cairocffi  1.6.1
cffi       1.16.0
dbus-next  0.2.3
pycparser  2.21
qtile      0.23.0
setuptools 68.2.2
UNKNOWN    0.23.1.dev6+gf2b35c5
xcffib     1.5.0`

I've no idea why that's happening but you're actually going to have another problem: if you're using the release version of qtile (v0.23.0) then you'll need to stick to the release version of qtile-extras too. That won't solve this issue though...

ah yeah i read that somewhere too - i think because i originally installed qtile via apt. no issue removing it though and building from git if that would be better?

You may just be able to do pip install https://github.com/elParaguayo/qtile-extras/releases/download/v0.23.0/qtile_extras-0.23.0-py3-none-any.whl

Sorry, you meant installing latest git version of qtile...

You may just be able to do pip install https://github.com/elParaguayo/qtile-extras/releases/download/v0.23.0/qtile_extras-0.23.0-py3-none-any.whl

This worked perfectly thanks!

Great - although that just gets you the v0.23 release and doesn't solve the original issue!

Hey!
I just ran into the same problem and found this thread. I am running Ubuntu 22.04 at it seems that currently you get setuptools version 59 if you have installed pip using apt install python3-pip
Solved by uninstalling python3-pip and reinstalling pip with the get-pip.py script referenced to in the pip installation guide. https://pip.pypa.io/en/stable/installation/


Definition of submodelDescriptor to support parquet files

Description

Acceptance Criteria

[ ] [criteria 1]
[ ] [criteria 2]
[ ] [criteria 3]

Additional Information

Child of Feature: https://github.com/eclipse-tractusx/sig-release/issues/721

This might work similarly to the PCF use-case where the submodelDescriptor.endpoint.interface property is used as discriminator how to access the "submodel". Afaik, parquet is a compression technology so there'd have to be a very precise spec

how to obtain the file (http/s3/?) and how to parameterize that call
how to decompress it
how to parse the plaintext payload into a domain-model (like a SAMM aspect)

I'm interested in contributing here.

Four possible solutions (which need to be evaluated) to deliver parquet files:

Define the parquet file in aspect as BLOB-type and add the the payload(This could be very large)
Define the parquet file in aspect as File-type and add the path to the parquet file where the requester can download it. (Two steps needed to get file downloaded)
Define only the meta information like the link to the FTP Server etc. in aspect
Use API [/submodel/submodel-elements/{idShortPath}/attachment which delivers zip

Is there an explicit requirement

to have the submodelDescriptor point to a S3/BlobStorage resource?
to have (bidirectional) transformation rules from a nestable format (like SAMM) to a tabular format?

I'm inclined to reuse as much of the href/subprotocolBody mechanism from the SUBMODEL-3.0 interface as possible assuming that access will always be negotiated via DSP catalogs.

Hi @arnoweiss ,

there is no explicit requirement to use File transfer via Bucket. But for larger files, it make sense to use the transfer.
We assume to create a new model which maps the flatten hierarchy to represent the columns in the parquet file. (Rules like using "_" or "."). Therefore we have the second story: https://github.com/orgs/eclipse-tractusx/projects/40/views/7?sliceBy[value]=tunacicek&pane=issue&itemId=68272934
I added a md file which includes two solutions on how to define the SubModelDescriptors.
definition_submodeldescriptors_parquet.md

See also md file in description Possible solutions: definition_submodeldescriptors_parquet.md
Definition of Submodel Descriptors for Parquet Data
Solution 1: File transfer via S3 bucket
Introduction to File Transfer process in EDC

Provider uploads the parquet file to an S3 bucket.
Provider creates an edc-asset. The dataAddress includes information about the S3 bucket and the file name:{
  "edc:dataAddress": {
    "edc-type": "AmazonS3",
    "edc:bucket": "<provider-bucket>",
    "edc:keyName": "<file-name>"
  }
}

See also full example here: edc-asset for file transfer
Consumer starts negotiation/transfer process and provide their own bucket information:{
  "edc:dataDestination": {
    "edc-type": "AmazonS3",
    "edc:bucket": "<consumer-bucket>",
    "edc:keyName": "<file-name>",
    "edc:accessKeyId":"<access-key-id>",
    "edc:secretAccessKey":"<secretAccessKey>"
  }
}

Provider retrieve the request and download the requested file from bucket and upload it to the consumer bucket.
Consumer can download the file from their bucket.

SubmodelDescriptor for Parquet Data (via File Transfer)
For the file transfer via S3 bucket the submodelDescriptor can be defined like:
{
...
  "submodelDescriptors": [
    {
      "idShort": "quality-data",
      "id": "<uuid>",
      "endpoints": [
        {
          "interface": "productDescription",
          "protocolInformation": {
            "href": "not used for transfer via s3 bucket",
            "endpointProtocol": "HTTP",
            "endpointProtocolVersion": [
              "1.1"
            ],
            "subprotocol": "",
            "subprotocolBody": "id=<edc-asset-id>;dspEndpoint=<controlplane-url>",
            "subprotocolBodyEncoding": "application/octet-stream;type=parquet-snappy",
            "securityAttributes": [
              {
                "type": "NONE",
                "key": "NONE",
                "value": "NONE"
              }
            ]
          }
        }
      ],
      "semanticId": {
        "type": "ExternalReference",
        "keys": [
          {
            "type": "Submodel",
            //TBD: New Model which map flatten hirarchy to model (like with "_" or ".")
            "value": "urn:samm:io.catenax.vehicle.product_description:3.0.0#FlattenedProductDescription"
          }
        ]
      },
      "description": [
        {
          "language": "en",
          "text": "submodel-descriptor for quality data which will be transferred via S3 bucket."
        }
      ]
    }
  ]
}

Parameter
Value
Description

href
""
not used for transfer via s3 bucket

subprotocolBody
id=;dspEndpoint=
Includes information about provider edc controlplane and edc-asset ID

subprotocolBodyEncoding
application/octet-stream;type=parquet-snappy
format of file. The file will be transfer via bucket.

semanticId.keys[].value
urn:samm:io.catenax.vehicle.product_description:3.0.0#FlattenedProductDescription
Aspect model which map flatten hirarchy to model

description[].text
submodel-descriptor for quality data which will be transferred via S3 bucket.
Further description for the consumer.

Solution 2: File transfer via HTTPS
If the file size is not too large, the file can be transferred via HTTPS in the usual way. The submodelDescriptor for this case can be defined like:
{
...
  "submodelDescriptors": [
    {
      "idShort": "quality-data",
      "id": "<uuid>",
      "endpoints": [
        {
          "interface": "productDescription",
          "protocolInformation": {
            "href": "<provider-edc-dataplane-url>/<path-to-download-file>",
            "endpointProtocol": "HTTP",
            "endpointProtocolVersion": [
              "1.1"
            ],
            "subprotocol": "",
            "subprotocolBody": "id=<edc-asset-id>;dspEndpoint=<controlplane-url>",
            "subprotocolBodyEncoding": "application/octet-stream;type=parquet-snappy",
            "securityAttributes": [
              {
                "type": "NONE",
                "key": "NONE",
                "value": "NONE"
              }
            ]
          }
        }
      ],
      "semanticId": {
        "type": "ExternalReference",
        "keys": [
          {
            "type": "Submodel",
            //TBD: New Model which map flatten hirarchy to model (like with "_" or ".")
            "value": "urn:samm:io.catenax.vehicle.product_description:3.0.0#FlattenedProductDescription"
          }
        ]
      },
      "description": [
        {
          "language": "en",
          "text": "submodel-descriptor for quality data which will be transferred via HTTPS."
        }
      ]
    }
  ]
}

As discussed with Birgit, the submodel interface value 'productDescription' is not according to IDTA standards. We need to use interface value as 'SUBMODEL-3.0'.
We need to create a simple submodel/aspect, like we are using before. That new aspect model will contain the reference to the Parquet file as a property. It will also contain the semantics of that Parquet file and the conversion fields/values required.
Need to check the mechanism that is used currently through EDC to exchange the consumer Bucket credentials. Through this new defined submodel, consumer won't be able to provide the bucket credentials. So need to find out a way to make async call to provider and get the response for Parquet file location.
Example of submodel will be attached.

See also md file in description Possible solutions: definition_submodeldescriptors_parquet.md
Definition of Submodel Descriptors for Parquet Data
Solution 1: File transfer via S3 bucket with EDC AWS extension
Introduction to File Transfer process in EDC

Provider uploads the parquet file to an S3 bucket.

Provider creates an edc-asset. The dataAddress includes information about the S3 bucket and the file name:
{
  "edc:dataAddress": {
    "edc-type": "AmazonS3",
    "edc:bucket": "<provider-bucket>",
    "edc:keyName": "<file-name>"
  }
}

See also full example here: edc-asset for file transfer

Provider creates a shell with shell descriptors and and the edc Asset with controplane URL.

Consumer calls the DTR via EDC and read the submodeldescriptor from the shell.

Consumer starts negotiation/transfer process and provide their own bucket information:
{
  "edc:dataDestination": {
    "edc-type": "AmazonS3",
    "edc:bucket": "<consumer-bucket>",
    "edc:keyName": "<file-name>",
    "edc:accessKeyId":"<access-key-id>",
    "edc:secretAccessKey":"<secretAccessKey>"
  }
}

Provider retrieve the request and download the requested file from bucket and upload it to the consumer bucket.

Consumer can download the file from their bucket.

SubmodelDescriptor for Parquet Data (via File Transfer)
For the file transfer via S3 bucket the submodelDescriptor can be defined like:
{
...
  "submodelDescriptors": [
    {
      "idShort": "quality-data",
      "id": "<uuid>",
      "endpoints": [
        {
           // TODO: Check which interface can be used here. See :https://github.com/admin-shell-io/questions-and-answers?tab=readme-ov-file#id47
          "interface": "SUBMODEL-3.0",
          "protocolInformation": {
           //TODO: href is required. Check if insert NONE is possible.
            "href": "not used for transfer via s3 bucket",
            "endpointProtocol": "HTTP",
            "endpointProtocolVersion": [
              "1.1"
            ],
            "subprotocol": "",
            "subprotocolBody": "id=<edc-asset-id>;dspEndpoint=<controlplane-url>",
            "subprotocolBodyEncoding": "application/octet-stream;type=parquet-snappy",
            "securityAttributes": [
              {
                "type": "NONE",
                "key": "NONE",
                "value": "NONE"
              }
            ]
          }
        }
      ],
      "semanticId": {
        "type": "ExternalReference",
        "keys": [
          {
            "type": "Submodel",
            "value": "urn:samm:io.catenax.vehicle.product_description:3.0.0#ProductDescription"
          },
           {
              "type": "Submodel",
              //TBD: New Model which map flatten hirarchy to model (like with "_" or "."). Outcome of the issue: https://github.com/eclipse-tractusx/sldt-semantic-models/issues/762
              "value": "urn:samm:io.catenax.vehicle.product_description:3.0.0#FlattenedProductDescription"
           }
        ]
      },
      "description": [
        {
          "language": "en",
          "text": "submodel-descriptor for quality data which will be transferred via S3 bucket."
        }
      ]
    }
  ]
}

Parameter	Value	Description
href	""	not used for transfer via s3 bucket
subprotocolBody	id=;dspEndpoint=	Includes information about provider edc controlplane and edc-asset ID
subprotocolBodyEncoding	application/octet-stream;type=parquet-snappy	format of file. The file will be transfer via bucket.
semanticId.keys[].value	urn:samm:io.catenax.vehicle.product_description:3.0.0#FlattenedProductDescription	Aspect model which map flatten hirarchy to model
description[].text	submodel-descriptor for quality data which will be transferred via S3 bucket.	Further description for the consumer.
Solution 2: File transfer via S3 bucket without EDC AWS extension
Introduction to File Transfer process in EDC

Provider uploads the parquet file to an S3 bucket.
Provider creates a shell with shell descriptors and the edc Asset, controlplane URL, S3 Bucket Link and credentials to access s3 bucket of provider.
Consumer calls the DTR via EDC and read the submodeldescriptor from the shell.
Consumer reads the credentials and starts donwloading the file from provider s3 bucket.

SubmodelDescriptor for Parquet Data (via File Transfer)
For the file transfer via S3 bucket the submodelDescriptor can be defined like:
{
...
  "submodelDescriptors": [
    {
      "idShort": "quality-data",
      "id": "<uuid>",
      "endpoints": [
        {
           // TODO: Check which interface can be used here. See :https://github.com/admin-shell-io/questions-and-answers?tab=readme-ov-file#id47
          "interface": "SUBMODEL-3.0",
          "protocolInformation": {
           //Link to Provider AWS S3 Bucket (S3 URI)
            "href": "https://example-bucket.s3.us-east-2.amazonaws.com/productDescription. parquet",
            "endpointProtocol": "HTTP",
            "endpointProtocolVersion": [
              "1.1"
            ],
            "subprotocol": "",
            "subprotocolBody": "id=<edc-asset-id>;dspEndpoint=<controlplane-url>",
            "subprotocolBodyEncoding": "application/octet-stream;type=parquet-snappy",
             // TODO: Check credentials for S3 bucket can be used here
            "securityAttributes": [
              {
                // TODO: Clarify which enum can be used
                "type": "NONE",
                // S3 securityKey
                "key": "SecurityKey",
                // S3 accessKey
                "value": "AccessKey"
              }
            ]
          }
        }
      ],
      "semanticId": {
        "type": "ExternalReference",
        "keys": [
          {
            "type": "Submodel",
            "value": "urn:samm:io.catenax.vehicle.product_description:3.0.0#ProductDescription"
          },
           {
              "type": "Submodel",
              //TBD: New Model which map flatten hirarchy to model (like with "_" or "."). Outcome of the issue: https://github.com/eclipse-tractusx/sldt-semantic-models/issues/762
              "value": "urn:samm:io.catenax.vehicle.product_description:3.0.0#FlattenedProductDescription"
           }
        ]
      },
      "description": [
        {
          "language": "en",
          "text": "submodel-descriptor for quality data which will be transferred via S3 bucket."
        }
      ]
    }
  ]
}

Solution 3: File transfer via HTTPS
If the file size is not too large, the file can be transferred via HTTPS in the usual way. The submodelDescriptor for this case can be defined like:
{
...
  "submodelDescriptors": [
    {
      "idShort": "quality-data",
      "id": "<uuid>",
      "endpoints": [
        {
          "interface": "productDescription",
          "protocolInformation": {
            "href": "<provider-edc-dataplane-url>/<path-to-download-file>",
            "endpointProtocol": "HTTP",
            "endpointProtocolVersion": [
              "1.1"
            ],
            "subprotocol": "",
            "subprotocolBody": "id=<edc-asset-id>;dspEndpoint=<controlplane-url>",
            "subprotocolBodyEncoding": "application/octet-stream;type=parquet-snappy",
            "securityAttributes": [
              {
                "type": "NONE",
                "key": "NONE",
                "value": "NONE"
              }
            ]
          }
        }
      ],
      "semanticId": {
        "type": "ExternalReference",
        "keys": [
          {
            "type": "Submodel",
            //TBD: New Model which map flatten hirarchy to model (like with "_" or ".")
            "value": "urn:samm:io.catenax.vehicle.product_description:3.0.0#FlattenedProductDescription"
          }
        ]
      },
      "description": [
        {
          "language": "en",
          "text": "submodel-descriptor for quality data which will be transferred via HTTPS."
        }
      ]
    }
  ]
}

Hi @thomas-henn : Could you please review the solutions?

Issue for asynch. call was created:
https://github.com/admin-shell-io/aas-specs-api/issues/347


Re-evaluate need to always return tablet in tablet mgmt iterator

The TabletManagementIterator will always return tablet when certain conditions are met in this code.  The code changes in #4255, but still has similar behavior.
This behavior is a carry over from the original code in main.  However, the code in elasticity has dramatically changed since main.  Wondering if it is still needed and if it is,  is this the best location for this code.  Ideally TabletGroupWatcher and TabletManagementIterator use the same code to come to decisions about tablets, both using code like TabletGoalState.compute and TabletState.compute.  This code may be specific to TabletManagementIterator.

Some of the conditions handled in the code linked in the description are also handled in more detail in lower level parts of the code.  For those cases it may be better to always rely on the lower level conditions.
https://github.com/apache/accumulo/blob/0739f7ee6d4db2931d855abefea00e70a32ada86/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletGoalState.java#L176
https://github.com/apache/accumulo/blob/0739f7ee6d4db2931d855abefea00e70a32ada86/server/base/src/main/java/org/apache/accumulo/server/manager/state/TabletGoalState.java#L97-L99

Would be useful to try removing the specialized code and only calling  computeTabletManagementActions(tm, actions); AND running all ITs to see if anything breaks.  However need to get all of the ITs reliably working in elasticity before doing this experiment.

I removed the special logic in #4363 and kicked off a full IT build. I'm not sure that it's necessary any more.

Full IT build succeeded. Closing this by merging #4363