Simplify writing custom authorization attributes

As originally reported by @nathan-alden in #5532:
The forced authentication abstraction (requirements, policies) eliminated a very useful shared custom authorization attribute I had written that allowed this: [CustomAuthorize(Operator.And, Permission.Create, Permission.Read)]. My attribute allowed for a single authorization algorithm to be implemented (within the attribute's code) that would check various JWT claims, with each action method choosing what parameters fed into that algorithm.
See also http://stackoverflow.com/questions/31464359/custom-authorizeattribute-in-asp-net-5-mvc-6/31465227

I read the whole discussion on StackOverflow, and frankly, I dont undestand the polemic. Why having all conditions hardwired in the code within an AuthorizeAttribute should be more flexible than defining dynamically requirements that might take their data from a DB for instance?
With the new approach you have just to group all action methods having the same permissions in equivalence classes and for each such class you need to create a policy. This is the only stuff that will remain hardired in the code. In the worst case (impossible in my opinion) you will have a policy for each action method, not for all permutations of requirements.
Everything else may be changed dynamically.
Obviously one doesn't need to create a different Requirement class for each policy, since each class may have several parameters. Since each policy is an implict "and" of several requirements, one might object there is no way to have an "Or" of existing requirements types. However this is false since you might create an OrRequirement that accepts a list of requirements and is satidfied if at least one requirement is satisfied.
Try to switch the way you think to policies. Everything appears logic if one thinks to policies as equivalence classes of action methods not as sets of requirements. Where several action methods belong to the same equivalence class iff they have same requirements, or better if there is a reason implicit in the business rules why they MUST have the same requirements..
In other terms policies are an abstraction over "actions" not over "requirements".

One of the big reasons is the fact that the attribute doesn't scale past the mvc controller action. We're trying to move the actual meat of the authorization logic into an imperative m service that can be called from anywhere in application code across frameworks and non-action callsites. When SignalR is done for example, it should be possible to both use the same attribute and logic. Baking mvc specific logic goes against that.
The biggest pain point is the fact that it's hard to flow state from the callsite (in this case the attribute) to your authorization logic. In mvc the resource policy is an mvc specific context, not much more you can flow from that beside the policy name itself.

With the new approach you have just to group all action methods having the same permissions in equivalence classes and for each such class you need to create a policy.

That really is the issue ... policies are great for defining discrete types of authorization needs that are computed dynamically (e.g. options.AddPolicy("CanBuyAlcohol", policy => policy.Requirements.Add(new Authorization.MinimumAgeRequirement(21))); ), but the current implementation seems limited for variations of the same class of requirement.
The bottom line is, people want the ability to define a claim type (e.g. "_http://schemas.mycompany.com/ws/2016/12/identity/claims/permission_") with different values for separate classes of actions ("CanManageOrders", "CanManageUsers", etc.) and declare an attribute which indicates a requirement for the specific claim (e.g. [Authorize(Policy ="HasClaim", ClaimType = MyClaimTypes.Permission, "CanManageOrders")] or better [Permission("CanManageOrders")]
) without having to register a new policy for each value of MyClaimTypes.Permission.
Sure, if your application has 450 different types of permissions that can be assigned to a given user then you can have a Startup.cs which has something like the following:
options.AddPolicy("CanDoOperation001", policy => policy.Requirements.Add(new Authorization.ClaimRequirement(MyClaimTypes.Permission, "CanDoOperation001")));
options.AddPolicy("CanDoOperation002", policy => policy.Requirements.Add(new Authorization.ClaimRequirement(MyClaimTypes.Permission, "CanDoOperation002")));
options.AddPolicy("CanDoOperation003", policy => policy.Requirements.Add(new Authorization.ClaimRequirement(MyClaimTypes.Permission, "CanDoOperation003")));
options.AddPolicy("CanDoOperation004", policy => policy.Requirements.Add(new Authorization.ClaimRequirement(MyClaimTypes.Permission, "CanDoOperation004")));
...
options.AddPolicy("CanDoOperation450", policy => policy.Requirements.Add(new Authorization.ClaimRequirement(MyClaimTypes.Permission, "CanDoOperation450")));

... but I hope you can see how developers are seeing an issue with this approach.

With the new policy approach one should adopt a different perspective. You look at all policies as a "bag" of predefined authorizations each action method designer must choose one.
The right perpspective is that there is no predefined "bag" of policies, but policies are designed together with action methods. Namely, each time you define a new "action type" you define a policy name for it where,  "action types" are an abstraction over action methods. For instance, suppose you have an ERP where "user actions" are classified depending on the "area" they affect (marketing, strategic planning, etc). For instance you might put in the same class all actions that compute values based on the company balance sheet, since for sure they'll have the same permissions. Now you may deine a policy for each "area" and for each action type (read, update, create, delete). So for instance you'll have a policy called: "BalanceSheetRead" that will be applied to all action methods that show values computed from the "official" company balance sheet.
Thus, policies names are decided in a way that is completely independent from the claims defining them: they are created starting from action methods classification. So in the worst case (pratically impossible) you will have a different policy for each action method which is completely equivalent to putting all claims in the autorize attribute itself, with the difference that now you may modify policy definitions without recompiling the code, since permission definition is not harwired in action method definition.
Policies may be defined in an ad hoc class that is called in the startup class and this class may read all definitions from an xml, or json file, or from a database.

I believe I understand the intent of the policy design and it seems to provide a suitable solution for complex authorization needs.  What it doesn't do is provide a convenient solution for simple authorization needs where there is a one-to-one mapping between a claim and what you are wanting people to define as a "policy".  I would submit that 80% of customers simply need a way to easily express that a given action requires a given claim.  In this case, the claim IS the abstraction.  Sure, creating a policy named "BalanceSheetRead" provides a physical abstraction from a claim type of "http://.../claims/permission" with a value of "BalanceSheetRead", but when you're never going to need that level of abstraction, it's unnecessary complexity.
Perhaps it's the message, rather than the solution that is in need of adjustment.  The message being sent out from the ASP.Net Core team and those advocating acceptance of this design is: "You shouldn't be implementing your own Authorize attribute" (i.e. "You're doing it wrong if you are").  It seems one of the goals for the new design was to provide an out-of-the-box solution for those writing custom solutions, but it still seems more sensible to use a solution like the following when the overhead of creating policy abstractions isn't needed:
    public class ClaimRequirementAttribute : TypeFilterAttribute
    {
        public ClaimRequirementAttribute(string claimType, string claimValue) : base(typeof(ClaimRequirementFilter))
        {
            Arguments = new object[] {new Claim(claimType, claimValue) };
        }
    }

    public class ClaimRequirementFilter : IAsyncActionFilter
    {
        readonly Claim _claim;

        public ClaimRequirementFilter(Claim claim)
        {
            _claim = claim;
        }

        public async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
        {
            var hasClaim = context.HttpContext.User.Claims.Any(c => c.Type == _claim.Type && c.Value == _claim.Value);
            if (!hasClaim)
            {
                context.Result = new UnauthorizedResult();

            }
            else
            {
                await next();
            }
        }
    }

    [Route("api/resource")]
    public class MyController : Controller
    {
        [ClaimRequirement(MyClaimTypes.Permission, "CanReadResource")]
        [HttpGet]
        public IActionResult GetResource()
        {
            return Ok();
        }
    }

/cc @HaoK @blowdart

@derekgreer ,
Now I understand your point. We should need at least an out of the box ClaimRequirement attribute, for the simplest cases. I agree.

So we do have an extensibility point for this today in the PolicyProvider, granted its not as polished as I'd like for something like this scenario:
        public class DynamicPolicyProvider : IAuthorizationPolicyProvider
        {
            public Task<AuthorizationPolicy> GetDefaultPolicyAsync()
            {
                return Task.FromResult(new AuthorizationPolicyBuilder().RequireAuthenticatedUser().Build());
            }

            // policyName = encoded claims type/values, i.e. "Permission:Read+Write"
            public Task<AuthorizationPolicy> GetPolicyAsync(string policyName)
            {
                return Task.FromResult(new AuthorizationPolicyBuilder().RequireClaim(decodePolicy(policyName)).Build());
            }
        }

The general problem that exists today is the policyName is the only thing that really flows from the Authorize attribute to the AuthZ service...

Yes, parameters could be encoded within the policy name, but surely this wouldn't be the team's recommended way of addressing this issue.  I think the best answer at this point really is: The current infrastructure isn't suited for simple claim-based authorization, so we recommend creating custom Action Filters for such needs.

@derekgreer ,
After some thinking about the problem I realized that "simple cases" you are referring to might use roles. Written this way it appears king dirty hack but actually isn't. In fact, in simple cases where the Authorization attribute simply needs just one claim, this claim usually doesnt map to any "immediate user property" (such ase age....or similar) but is already the result of some processing performed on the user data. In a few word such claim has been computed "ad hoc" to perform authorization. Thus, it is kinda user grouping for authorization purpose, ie a role!
As a matter of fact the main advantage of claims is that one may define authorization criteria depending on several "native" user properties. If one doesn't use several "native" user properties it is simple using roles! On the other isde if one needs a complex user property processin the simplest way is "Policy".
In both cases we avoid to hardcode specific property values next to the action methods, but, instead, we put "identifier" that are computed somehow from such values, thus decoupling the overall authorization logic, from specific property values. In the case of roles, they are computed either at user registration time, or when the auth cookie is created by using user data. In the case of policies they are computed in each policy record.

There's nothing left to be said by me that others haven't already said. My original complaint was that ASP.NET Core removed/replaced useful abstractions that allowed for flexible designs in Web API 2. The new policy system is an example of catering to the 20%, not the 80%. We are supposed to accept magic strings (i.e., the policy name appearing in an attribute) in place of custom authorization attributes that were refactoring-friendly (see my example at the top). The design of the authorization system seems to force one of three paths: 1) a complete re-implementation of authorization using filters or middleware, 2) needless class bloat (basically, almost a 1:1 between action method and policy), or 3) the selection of a competing framework (in my case, NancyFx).

@nathan-alden , I dont see any difference between the magic strings of policy names and the magic strings of Claims name. Defining policies with full .Net code power for sure is more flexible that putting them into an AuthorizeAttribute that may accepts just strings.
My only complaint is that one can't add several Policies separated by commas like for Roles. This from one side forces a better abstraction but on the other side somehow prevent incremental implementation of complex systems since incremental changes might break the Action Method equivalence classes classification (ie classes with the same policy).

Enums are not magic strings. My custom attribute hid the underlying serialization logic from developers working on action methods. The enums are discoverable by IntelliSense and ReSharper. Changes to the shared package that contains the attribute and the enums will bubble up as compiler errors should developers consume a non-backward-compatible new version of the shared package.

@nathan-alden , in the general case you cant avoid magic strings usage in the Authorize attribute. True, if you have just one Claim per attribute with a primitive type value you can avoid string, but what if you need and/or combination of claims (which is quite common)?  Attributes do not allow complex structures so either you put the whole and/or expression in a string or you use something like "Policies" ie place the complex expression elsewhere an just put its "name" in the Attribute.
Moreover, imagine you somehow place Claims expressions in attributes. What if you need to change authorization Criteria? You need to review ALL attribute instances manually! On the contrary, if you use policies, you just review policies (which should be sensibly less than attribute instances), or maybe just one policy. Then all Action methods using it will work properly.
That said I agree that current implementation need to be improved. Namely:

Possibility to provide several policies within the same attribute (whose semantic is a lofical or). This way like in case of Roles we may specify alternative authorization "reasons". In fact the same action may be allowed because it belongs to different partially overlapping set of user actions. True that overlapping sets may be transformed into not overlapping sets by computing all possible intersections, but this would lead to a combinatorial explosion.

Simplified syntax for the single claim simple case, ie avoiding policies definition and placing it directly in the attribute.

My original post addressed that. My attribute's constructor was declared like this:
public AuthorizationAttribute(Operator @operator, params Permission[] permissions)

This handled authorization rules for most of my endpoints. For anything more custom than that, I had another constructor that takes a Type implementing a specific interface. The interface would be passed the set of claims and allowed to do whatever checks it needed.

@nathan-alden ,
public AuthorizationAttribute(Operator @operator, params Permission[] permissions) is smart, but you cant specify claims values either, just claims names. For sure, it is better than standard string based roles but doesnt add expressive power to it, just a better implementation.  It doesnt go in the direction of declarative rules based on native user properies (represented by claims).
Your more general solution of implementing an interface to define complex and/or combinations improve expressibe power but appears heavier than the Policy solution. Sincerely, I prefer the Policy solution where each and/or combination may be defined with a lambda, ie with the minimum code required to define the and/or combination, also if it has the disadvantage of relying on "magic strings".
In any case an extension point for customizing Authorization in a not trivial way should be provided.

@derekgreer I don't have enough words to express how right you're about this issue (I could upvote your comments one million times).
Please, to the people developing the asp.net core authorization system: Don't force us to code one requeriment per permission, IT'S A COMPLETE OVERKILL. Please, provide a way to pass parameters to the Authorize attribute...
@derekgreer The example of ClaimRequirementAttribute you shown above, is functional code or just pseudocode to expose the functionality?

Yes, the ClaimRequirementAttribute code should be functional.

@danroth27 @HaoK - I'm not sure how to sum up the issues here, but this really seems like it's not an MVC concern.
The currency that MVC operates on is IAuthorizeData - https://github.com/aspnet/Security/blob/dev/src/Microsoft.AspNetCore.Authorization/IAuthorizeData.cs It sounds like in order to satisfy some of the requests that are being discussed here, you'd have to have a different metadata interface. Then we could teach MVC how to deal with this new API.
What do y'all intend to do about this?

@blowdart do you want to migrate this / xform this issue into something in security?  Its kinda related to https://github.com/aspnet/Security/issues/917

Not really, no. It's Dan's issue :)

It's not. The problem is that you can't flow context through. I don't think we should build in anything first class for permissions, we should try to address the issue generally so that other frameworks (like SignalR) get the benefit.

AuthZ improvements have been consolidated into this issue: https://github.com/aspnet/Security/issues/670


fix: handle change on the input currentFile

Input can change at anytime.
Currently when currentFile value change, the tree is not updated accordingly as it's handled only in the ngOnInit.
I propose to handle that in the ngOnChanges so now the tree is updated accordingly.
We should not emit clickFile in the ngOnChanges else we will enter in a loop as clickFile will change the value of currentFile. (Angular will complain about changes applied after the view check)

Thank you so much, I have no comments to make, it’s perfect, I’ll merge and deploy :)


cuyhoik=

If this is a bug report, please fill out the following:

Version of Ruby:
Version of Kafka:
Version of ruby-kafka:

Please verify that the problem you're seeing hasn't been fixed by the current master of ruby-kafka.
Steps to reproduce
kafka = Kafka.new(...)

# Please write an example that reproduces the problem you're describing.

Expected outcome
What you thought would happen when running the example.
Actual outcome
What actually happened.

OMG, I am so sorry again, same issue, I did not lock my computer and I am guessing my cat pawed  at the keyboard 🙈🙈🙈 This sounds really lame, I know...


Login with QR code AND cross-signing in one shot

We currently have QR code cross-signing. You will typically perform the cross-signing immediately after you login on a new device.
Discord currently allows users to login with QR codes if they are already logged in on another device.
It seems like the obvious next step in terms of better UX is to kill two birds with one stone. I suggest that there be an option to login and cross-sign a new device simultaneously with one QR code.
In 99% of cases wherein you are verifying a new device, you will have just recently logged in anyway, so why not make it simpler & quicker?

Login with QR means if someone temporarily has access to one of your devices they can instantly give themselves access on one of their devices.

@t3chguy I suppose one could also perform a memory dump of the application and gain access to your encryption keys and/or token. If someone has physical access to a running Matrix implementation... it seems like it's already game over.

Neither of those things let the user spawn a new token though.

Multiple devices using the same token and device keys would break

Hmm.. perhaps this feature could be password-gated? But at that point, it's not that much of a UX improvement from just logging in on the other device.
Also, pardon my ignorance of the spec, but would it not be possible to simply imitate another device? If you have a memory dump of the application, presumably you would have the device key which could just be placed into HTTP queries on a different device?

Shoulder surfing would also be an issue. As easy as it would be to tell people not to log in to their account in insecure environments, someone is inevitably going to have a giant QR code on a library computer.

Sure, such a memory dump would work but crypto would get wedged due to two devices pushing OTKs claiming to be each other.

@turt2live sure but that would log that user in on the qr code bearing device.
The qr code on discord shows on the unauthenticated machine.
That raises the question, how do you know what server to use in this environment as an unauthenticated client showing a qr code.

@t3chguy Presumably, the server information could be encoded in the QR code.
But could the QR code even originate from the unauthenticated machine? At the time of the QR code being created, you wouldn't even know which user it's being created for. I am not sure how such an anonymous QR code could contain the necessary information for cross-signing.
I was thinking that it would have to originate from the device that was already logged-in. @turt2live I think shoulder surfing could be addresses like it currently is now with cross-signing, in that you make sure you have a green check mark on both devices before finalizing the login.

Also @t3chguy, going back to your physical access concern, wouldn't the user just notice that they now suddenly have a new device and unverify it right away?
If you have physical access, you already have all the past messages. In order to get the future messages, you'd need to make sure your victim does not unverify your device.
Perhaps there could be some kind of persistent notice that a new device has recently been added that lasts a few days. It would act as a heads up in case someone tried to add a new device whilst you left your phone unattended.

In the case of Discord, the way it works is the QR code showed on my logged-out Discord Desktop app and I used my logged-in Discord Mobile app to scan it.
The server to be used has to be the one the account exists on which only the logged in app knows.

I am not sure how such an anonymous QR code could contain the necessary information for cross-signing.

It could contain a shared key which could be used to send any data back, encrypted.

I was thinking that it would have to originate from the device that was already logged-in.

Typically people tend to have their mobile devices always connected so this would be a case of having to show your phone's QR code to your laptop's webcam.

wouldn't the user just notice that they now suddenly have a new device and unverify it right away?

Possibly, as it happens, a lot of users just hit Later on every prompt that they are ever shown.

FWIW, we'll be looking into changing logins such that the server does not see the password, which will make it safe for users to use the same password for logging in and unlocking the cross-signing keys, so that users don't need to scan QR codes.  So solving the "log in with password and then verify" issue by removing the verify step rather than removing the "log in with password" step.
Logging in using QR codes is something that could be done, but as already mentioned, would require care in making sure that we're not opening up a huge hole.  (Perhaps allowing users to choose whether they want it or not?)

server does not see the password

not always possible, like in LDAP backed logins


I-ALiRT - Process Ephemeris

Change Summary
Overview
Create the process_ephemeris file, meant to take in location data for a specific ground station partner and output pointing data (azmuth, elevation, doppler). Related issue can be found here.
New Files

process_ephemeris.py

finds azimuth, elevation, and doppler

test_process_ephemeris.py

unit tests for process_ephemeris.py

Updated Files

spice/time.py

addition of spice wrapper for str2et method, used in process_ephemeris
addition of spice wrapper for et2utc method, used in process_ephemeris

test_time.py

unit tests for new spice wrappers

conftest.py

updates the external_kernel function (previously named _download_de440s) to download an additional external kernel from NAIF

pyproject.toml

adds convrt to the list of ignored words for codespell, in order to get precommit hooks to pass

Testing
There are new unit tests for every new method added in this PR, except for the updated fixture in conftest.
Note
The code to calculate the doppler shift is not part of this PR, since it requires some more research. It'll be saved for a separate issue/PR.

One more thing that I thought of. If we DO end up needing pck00011.tpc make certain to include it in gitignore similar to de440.bsp:
Ignore specific SPICE kernels that get downloaded from NAIF automatically for tests
marked with @pytest.mark.external_kernel
*/de440.bsp

How soon do you guys think you will be able to finish this PR? The reason I ask is, I have some works that's waiting for this feature. I don't think I will get to it this month but love to get to next month(January 2025). It would be very helpful if we can have this code merged and included in December release so that it can be used later. These are the three functions I need for January work.

met_to_J2000ns()
J2000ns_to_et()
et_to_UTC()

I think last two are works of this PR. Is that right @torimarbois @subagonsouth?

How soon do you guys think you will be able to finish this PR? The reason I ask is, I have some works that's waiting for this feature. I don't think I will get to it this month but love to get to next month(January 2025). It would be very helpful if we can have this code merged and included in December release so that it can be used later. These are the three functions I need for January work.

met_to_J2000ns()
J2000ns_to_et()
et_to_UTC()

I think last two are works of this PR. Is that right @torimarbois @subagonsouth?

@tech3371
The first two are already implemented. See:

https://github.com/IMAP-Science-Operations-Center/imap_processing/blob/dev/imap_processing/spice/time.py#L17
https://github.com/IMAP-Science-Operations-Center/imap_processing/blob/dev/imap_processing/spice/time.py#L47

Note that J2000 seconds is the same as ET.
Yes, the third is in this PR.

Since MET is sort of the standard time definition for this mission, I can also implement a met_to_UTC function. I'll write a ticket for that and make sure to get it done this month. Should be an easy one.

How soon do you guys think you will be able to finish this PR? The reason I ask is, I have some works that's waiting for this feature. I don't think I will get to it this month but love to get to next month(January 2025). It would be very helpful if we can have this code merged and included in December release so that it can be used later. These are the three functions I need for January work.

met_to_J2000ns()
J2000ns_to_et()
et_to_UTC()

I think last two are works of this PR. Is that right @torimarbois @subagonsouth?

@tech3371 the goal is to get it merged sometime next week!

Thank you @torimarbois @subagonsouth for help on this! Really appreciate it

Also, I created plots to make sure that this new code created similar results to the sample code that Laura wrote. It does! Here are the plots produced by the sample code:

And here are the plots produced by the code in this PR:


Specific motor choice?

As you know, I am using the Waveshare General Robot Driver board like you mention.
I thought I would ask you here what specific motor you have used for a "beginner" robot? I am thinking about the general specs you mention, ~30cm ~2kg. So I think we need a 12v motor.
Which one did you use?

Please search Amazon, Temu or AliExpress,12V DC encoder gear motor 150rpm wheel. You may also check robot chassis kit. The 6 pins sequence may be different from the board.

https://github.com/hippo5329/linorobot2_hardware/wiki#sample-robot-build
These chassis were bought from Amazon. Motors with encoder are included.


Allows for string keys in instructions

Coverage increased (+0.2%) to 86.396% when pulling 19c4731e7de7391b9b85c85ad0071246f2fe1ffe on feature/symbolize_keys into 7f366947732b638a1bbc1ebda805ef25ad09e97e on master.


Make CMAC properly fail if cipher is not CBC mode one

Also avoid misleading users in the demo by use of the alias name.
Checklist

[x] documentation is added or updated
[x] tests are added or updated

I think it is a bug that we allow non CBC mode ciphers with CMAC as that is completely insecure use of the crypto primitive. So I propose this should be also added to 3.0.

Heh, we misuse CMAC in evp_extra_test.c. Fixup commit added.

@beldmit still OK?

Yes, still OK if tests passed

we allow non-CBC-mode ciphers with CMAC

Is this in the RFC? So CFB/OFB modes, for example, are not ok? I'm wondering how breaking this change might end up being if we backport to 3.0...

CI failure looks relevant

Is this in the RFC? So CFB/OFB modes, for example, are not ok? I'm wondering how breaking this change might end up being if we backport to 3.0...

This is by definition of the CMAC.  https://en.wikipedia.org/wiki/One-key_MAC
And yes, it is also in the AES-CMAC definition in RFC 4493.

Is this in the RFC? So CFB/OFB modes, for example, are not ok? I'm wondering how breaking this change might end up being if we backport to 3.0...

In theory one could use the CMAC MAC with other mode cipher but the result would be interoperable and with completely bogus security properties - potentially even completely insecure one.

In theory one could use the CMAC MAC with other mode cipher but the result would not be interoperable and with completely bogus security properties - potentially even completely insecure one

That may be the case - but my bet is that this will break people. Not a decision to make without some OTC input IMO.

** OTC Question: Should we fix this, and if so in which branches? **

This pull request is ready to merge

OTC: This is ok for 3.1 and above, not for 3.0.

Actually this needs re-approval from @beldmit

Still OK

This pull request is ready to merge

Merged to master and 3.1 branches. Thank you for the reviews.


Update ruby-tutorial.md

this command is already here further down, adding the one that was missing

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/rest/ruby-tutorial.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Thanks for fixing this silly typo :). I changed the base branch to master (instead of live). We don't take pull requests directly into live.


improve logging in interop client and server

Codecov Report

Merging #2495 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2495   +/-   ##
=======================================
  Coverage   85.80%   85.80%
=======================================
  Files         123      123
  Lines        9442     9442
=======================================
  Hits         8101     8101
  Misses        994      994
  Partials      347      347

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ae1a13...fe313fb. Read the comment docs.


Symlinks are now converted to absolute symlinks on cp('-R').

Previously they would be copied verbatim, still as relative links, which
may no longer work.
Includes a test. Fixes #69.

closing as it looks abandoned, please reopen if not!


drop new RegExp() in favor of generating the javascript regex

Description
Instead of
// RegexMatchCallbackHandler.php

        return sprintf("new RegExp('%s')", trim(mb_substr($matchCallback, 6)));

// use
        return trim(mb_substr($matchCallback, 6));

When the regex is wrong, you'll see the error right away.  Plus, you get flags (which we don't currently support with regex: ) and it's easier for PHP programmers to understand, even though it's technically a javascript regex.
Some musings at https://github.com/Spomky-Labs/phpwa-doc/issues/12
Example
No response

Hi,
If you just set a string without any recognized pattern, the raw string is used as it is.
I do not know the advantages of the new RexExp('foo') compared to just 'foo'.

I think the only advantage of creating a RegExp is when the string is dynamic;
const myRegex = 'ab+c';
new RegExp(myRegex);

but since the bundle is generating javascript, we don't need to do that, we can simply generate the regex as javascript.
One big advantage to this approach is that if the user makes an error in the pwa.yaml file, they will see the error immediately in the javascript console.  Otherwise, we should compile the expression and tell the user.
I found several errors in my pwa.yaml file, mostly because I was confused about the regex pattern.

Unlike PHP, it appears that you can't change the regex markers, and since we're dealing a lot with slashes, maybe it's easier to use RegExp().  We can add something for the flags if we have a good use case (e.g. matching JPG and PNG files instead of .jpg and .png).
Instinctively, seeing regex: /article looks like a bad regex to me.   What if this key were named RegExp instead?  That would make it clear it was a javascript RegExp and not php.

The simpler, the better.

We can add something for the flags if we have a good use case

Nothing prevent users to directly use JS
match_callback: 'new RegExp("/foo/bar", "g")'

Maybe I should simply drop it in favor of raw values.

I'm still a bit confused, some examples will help.  I've got a few projects using pwa-bundle, I'd like them all to be working by the time you do your talk.
Running a Symfony app on your phone without an internet connection and without writing a single line of javascript sounds like science fiction, but it's sooo close!

I added a section in the documentation.
I hope it will clarify the match_callback option.
Also, I removed the regex: handler as it is ultimately useless.

Running a Symfony app on your phone without an internet connection and without writing a single line of javascript sounds like science fiction, but it's sooo close!

I hope this will allow users to adopt the PWA standard. Developers will focus on the application and not on the implementation details.


Events are not being triggered

I'm trying to use openIAB unity plugin but the events are not being triggered. I have the following code equal to the examples and none of the events are called:
private void OnEnable()
{
// Listen to all events for illustration purposes
OpenIABEventManager.billingSupportedEvent += billingSupportedEvent;
OpenIABEventManager.queryInventorySucceededEvent += queryInventorySucceededEvent;
OpenIABEventManager.purchaseSucceededEvent += purchaseSucceededEvent;
OpenIABEventManager.purchaseFailedEvent += purchaseFailedEvent;
OpenIABEventManager.consumePurchaseSucceededEvent += consumePurchaseSucceededEvent;
OpenIABEventManager.consumePurchaseFailedEvent += consumePurchaseFailedEvent;
}
Can someone tell me what I might be doing wrong?

Hello,
Could you please check that there is event listener in the scene and post some more code and logs from device?

Nevermind. This was dumb. Yes the event listener was missing.


CodeScanning.getAlerts() returns wrong data type

Currently CodeScanning.getAlerts() (octokit/codescanning.py on line 94) returns a list[dict] instead of the expected list[CodeAlert]. This causes any usage that expects CodeAlert objects to fail. For example, when Policy as Code attempts to  access CodeAlert properties in checks.py starting on line 89, an exception is raised since a 'dict' object does not have those properties.
It looks like this may have been introduced with the changes to CodeScanning.getAlerts() in commit 88195cc.
Example of the error when this happens:

@Pradoxzon you are absolutely right, let me create a PR for this now.

Release 0.6.1 should fix the issue now
https://github.com/GeekMasher/ghastoolkit/releases/tag/0.6.1

Thank you for the prompt fix!


pull in all the upstream/github issue fixes vetted during the 4.0/4.1…

… dev cycle
/hold
(want to see if recent openshift/release fixes are doing what we expected in e2e-gcp)
but am considering merging this if the tests do run/pass given all the non-bugzilla fixes vetted during 4.x

tests still not picked up ... see https://github.com/openshift/release/pull/3735 for next iteration of efforts there

/test e2e-gpc

/test e2e-gcp

ok ..... FINALLY the 3.11 PR tests are running the ginkgo e2e's out of openshift/origin

/test e2e-gcp

/test e2e-gcp

gcp provision failure on last run

/retest

fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/tmp/openshift_gcp_provision.sh"], "delta": "0:02:10.682157", "end": "2019-05-10 21:01:04.824796", "msg": "non-zero return code", "rc": 1, "start": "2019-05-10 20:58:54.142639", "stderr": "WARNING: mode is deprecated. Please use subnet-mode instead.\nERROR: (gcloud.compute.networks.create) Could not fetch resource:\n - Quota 'SUBNETWORKS' exceeded.  Limit: 450.0 globally.", "stderr_lines": ["WARNING: mode is deprecated. Please use subnet-mode instead.", "ERROR: (gcloud.compute.networks.create) Could not fetch resource:", " - Quota 'SUBNETWORKS' exceeded.  Limit: 450.0 globally."], "stdout": "", "stdout_lines": []}

/retest


Build Error: "Building for iOS simulator but linking in dylib built for iOS"

I just upgraded to react-native v0.63.3
My dev environment
Xcode: v12.0.1
React-native: v0.63.3
react-native-agora: ^3.0.1-rc.4
after installing pods, and cleaning build folder, and building app, I get this error:

Showing Recent Errors Only
Building for iOS Simulator, but linking in dylib built for iOS, file
'/Users/Desktop/Development/AppName/ios/Pods/AgoraRtcEngine_iOS_Crypto/Agora_Native_SDK_for_iOS_FULL/libs/AgoraRtcCryptoLoader.framework/AgoraRtcCryptoLoader' for architecture arm64

I would imagine theres an issue on android as well but haven't gotten into it yet.

you are using Xcode 12, you should remove valid archs from your build setting

I have no VALID_ARCHS build setting. ion either my project OR target. the ONLY existence of VALID_ARCHS is in AgoraRtcEngine_iOS_Crypto.xcconfig

maybe you can try 3.1.3 version

same error with react-native-agora v3.1.3

Samme error

which version of your cocoapods?

@LichKing-2234 i'm using cocoapods v1.10.0 and getting same error

have you tried our quickstart?

@LichKing-2234 i have followed this quick start guide https://docs.agora.io/en/Video/start_call_react_native?platform=React Native

when i use latest react-native version (0.63.3) it crashes on ios run, but when i use 0.61.5 version (which used on quickstart sample) it works well

https://zoom.us/j/97334922301?pwd=YVNFVWIyTnR3RHplTUlNT2dzbUN1Zz09
Elgiz Hacıyev<EMAIL_ADDRESS>8 Kas 2020 Paz, 16:29 tarihinde
şunu yazdı:

when i use latest react-native version (0.63.3) it crashes on ios run, but
when i use 0.61.5 version (which used on quickstart sample) it works well
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/AgoraIO-Community/react-native-agora/issues/288#issuecomment-723570921,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACEQ4XR2XFNW4LOHJN6SEW3SO2FLXANCNFSM4TL3QDTA
.

@eleviven I means run quickstart on your simulator

Xcode12 may cause Building for iOS simulator but linking in dylib built for iOS
you should use a physical device because the simulator can't capture video and render the remote video as a green picture (because of OpenGL).


[BUG] Unable to build 0.8 from tarball - get_target_property() called with non-existent target "source-archive"

Description
I am trying to build projecteur in the open build service from opensuse - https://build.opensuse.org . This is a quite locked down environment with no network access and a minimal set of dependencies installed.
I have managed to kick off the build, but it now fails at the cmake step
[    5s] + cmake ..
[    5s] -- Setting build type to 'Release' as none was specified.
[    5s] -- The CXX compiler identification is GNU 10.2.1
[    5s] -- Detecting CXX compiler ABI info
[    5s] -- Detecting CXX compiler ABI info - done
[    5s] -- Check for working CXX compiler: /usr/bin/c++ - skipped
[    5s] -- Detecting CXX compile features
[    5s] -- Detecting CXX compile features - done
[    5s] -- Compiling without Qt5::X11Extras.
[    5s] -- Compiling without Qt5::DBus.
[    5s] -- Could NOT find Git (missing: GIT_EXECUTABLE)
[    5s] -- Version-Info: Git not found. Possible incomplete version information.
[    5s] -- Version-Info: Failure during version retrieval. Possible incomplete version information!
[    5s] -- Version info for 'projecteur': 0.8
[    5s] -- Found PkgConfig: /usr/bin/pkg-config (found version "1.7.3")
[    5s] -- Checking for module 'udev'
[    5s] --   Package 'udev', required by 'virtual:world', not found
[    5s] -- Could NOT find Git (missing: GIT_EXECUTABLE)
[    5s] -- Cannot add 'source-archive' target, git not found.
[    5s] -- Could not get linux distribution id, defaulting to 'linux'
[    5s] -- Could not get linux version, defaulting to 'unknown'
[    5s] CMake Error at cmake/modules/LinuxPackaging.cmake:179 (get_target_property):
[    5s]   get_target_property() called with non-existent target "source-archive".
[    5s] Call Stack (most recent call first):
[    5s]   cmake/modules/LinuxPackaging.cmake:165 (_makepkg_packaging)
[    5s]   CMakeLists.txt:218 (add_dist_package_target)

I am not sure what the problem is here, hence this issue. Note that I am interesting in building the binaries and configs only, as I will then assemble the RPM. If I can disable parts of the build that generate
To Reproduce
Build on OBS:

project home
build log for Tumbleweed x86_64

Expected behavior
The build succeeds.
Desktop/Linux Environment (please complete the following information):
Not applicable
Screenshots
Not applicable.

Thanks for reporting this.
Well in this case you would need to skip the call to add_source_archive_target(projecteur) and add_dist_package_target(... - probably with a patch file for the main CMakeLists.txt in your case.
I can include a CMake option to disable these two steps if set (e.g. cmake .. -DPACKAGE_TARGETS=OFF or similar), but that will only affect new versions and won't help you with 0.8 of course.
With your patch file you could wrap the mentioned calls in an if, e.g.
if(FALSE)
  add_source_archive_target(projecteur)
  add_dist_package_target(...
endif()

Thanks for the quick reply @jahnf . I'll add the change you suggested as a local patch and see if that unblocks the build.

Alright, let me know if it works. You can keep this issue open even if the build succeeds, I'll close when a build option to disable the generation of these package targets has been added.

With that change, I was able to start compilation. Compilation failed due to a GCC complaining about an unused parameter
[   16s] /home/abuild/rpmbuild/BUILD/projecteur-v0.8/src/linuxdesktop.cc: In member function 'QPixmap LinuxDesktop::grabScreenWayland(QScreen*) const':
[   16s] /home/abuild/rpmbuild/BUILD/projecteur-v0.8/src/linuxdesktop.cc:125:50: error: unused parameter 'screen' [-Werror=unused-parameter]
[   16s]   125 | QPixmap LinuxDesktop::grabScreenWayland(QScreen* screen) const
[   16s]       |                                         ~~~~~~~~~^~~~~~
[   16s] cc1plus: all warnings being treated as errors

After appending -Wno-unused-parameter to CXXFLAGS compilation succeeds. Now I just have to figure out what to install where, but it should be easy given that you already create packages.
Thanks!

Current develop and future versions now on support disabling of creating the package build targets in CMake, example:
cmake .. -DPACKAGE_TARGETS=OFF


alsaUtils: fix fftw ref

It wants fftw for float. For alsabat analysis.

Motivation for this change
Fix 'alsabat' analysis; it was built as without fftw, because we supplied fftw for double.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 116476 at 90cf12ff run on x86_64-linux 1

1 package marked as broken and skipped:

mikutter

2 packages failed to build (new failures):

dwm-status
python39Packages.notify-py

11 packages built successfully:

MMA
alsaUtils
i3blocks-gaps
linuxband
python38Packages.notify-py
python38Packages.py3status
python38Packages.toggl-cli
python39Packages.py3status
rakarrack
roon-server
solfege

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 116476 run on x86_64-linux 1

1 package marked as broken and skipped:

mikutter

2 packages failed to build and already failed to build on hydra master:

dwm-status: log was empty
python39Packages.notify-py: log was empty

11 packages built:

MMA
alsaUtils
i3blocks-gaps
linuxband
python38Packages.notify-py
python38Packages.py3status
python38Packages.toggl-cli
python39Packages.py3status
rakarrack
roon-server
solfege

The following issues got detected with the above build packages.
Please fix at least the ones listed with your changed packages:

MMA:

warning: unclear-gpl
gpl2 is a deprecated license, check if project uses gpl2Plus or gpl2Only and change meta.license accordingly.
Near pkgs/applications/audio/MMA/default.nix:66:5:
   |
66 |     license = lib.licenses.gpl2;
   |     ^

See: https://github.com/jtojnar/nixpkgs-hammering/blob/master/explanations/unclear-gpl.md
warning: patch-phase
patchPhase should not be overridden, use postPatch instead.
Near pkgs/applications/audio/MMA/default.nix:15:3:
   |
15 |   patchPhase = ''
   |   ^

See: https://github.com/jtojnar/nixpkgs-hammering/blob/master/explanations/patch-phase.md


Knife bootstrap don't refresh vault anymore

Ohai chefs !
I'm running into a bootstrap / vault issue (probably a regression) :

I'm using knife 12.3.0 gem install with rbenv :

> knife -v
Chef: 12.3.0

> ruby -v
ruby 2.0.0p643 (2015-02-25 revision 49749) [x86_64-linux]

I'm trying to bootstrap a bare metal server who got a fresh ubuntu 12.04.5.
I'm using my own Chef account to make the bootstrap
bootstrap command :

knife bootstrap node1.acme.org -N node1 -E prod -r 'role[uk],role[my-server],role[my-metrics-agent]' -x root

I'm using a validator certificate to bootstrap the node (old fashioned way)
My account is admin of the vault : (vault_keys content)

admins:
  bvessemont
  [...]

my vault is configured to search_query:                *:* (yeah it's for some root password and shit ...)
node1.acme.org is not registred in the vault (before or after the bootstrap).. not even after any other successive bootstrap operation.
I'm using vault cookbook 1.3.0 : depends 'chef-vault', '~> 1.3.0'
and it used to work before 12.3.0 afaik

Got this :
node1.acme.org   ================================================================================
node1.acme.org   Recipe Compile Error in /var/chef/cache/cookbooks/my-system/recipes/default.rb
node1.acme.org   ================================================================================
node1.acme.org
node1.acme.org   ChefVault::Exceptions::SecretDecryption
node1.acme.org   ---------------------------------------
node1.acme.org   teamops/vault is not encrypted with your public key.  Contact an administrator of the vault item to encrypt for you!

Classic and verbose log available here (I changed node/role/server/... name before putting it on gist):
https://gist.github.com/BarthV/a2e960e228392f719d67
As a workaround, I must refresh the vault after the first bootstrap attempt (so client cert is created and available in Chef server). So the vault can be loaded  using the node private cert and the bootstrap operation can go on.

well this isn't the same issue as #3121 since you're not using the new bootstrapping code.  you might try using the new bootstrapping code and pass in the --chef-vault-item's that you want to grant to the client on the command line and that might actually work better.  the old workflow didn't change though.  are you sure you're getting the old workflow?  gist of the full output might be useful since there's nothing here which looks incorrect.

It works with --bootstrap-vault-item teamops:vault , it is somewhat more restrictive since I've put vault search especially for this kind of job...
Now i've to specify all the needed vault item in my bootstrap, or bootstrap an empty node (if it works)

Only thing I can think of is the problem with the old bootstrapper uploading a blank client.pem now.

So ^^^ that statement is incorrect.  The problem is with the new bootstrapper, when -N is not passed, it uploads a blank client.pem.  As far as I can tell the code in the template is correctly protected for the old bootstrapper to not have changed.  Still have no idea what is going on with your issue.  Would still be useful to get a gist of the full output.

I will make a simple vault test next week :)

Closing due to lack of response


How to get test tear down in extended DaggerMockRule ?

Is it possible to get a method callback to the customized DaggerMockRule?
(Usage : I want to close the mocked db)
Current implementation
    @get:Rule
    val mockDbRule = DaggerMockDbRule()

    @After
    fun tearDown() {
        mockDbRule.db.close()
    }

DaggerMockDbRule.kt
class DaggerMockDbRule : DaggerMockRule<AppComponent>(AppComponent::class.java, DatabaseModule()) {

    var db: AppDatabase = Room.inMemoryDatabaseBuilder(
        ApplicationProvider.getApplicationContext(),
        AppDatabase::class.java
    ).build()

    private val app =
        InstrumentationRegistry.getInstrumentation().targetContext.applicationContext as App

    init {
        customizeBuilder<DaggerAppComponent.Builder> {
            it.baseNetworkModule(BaseNetworkModule(App.BASE_URL))
                .contextModule(ContextModule(app))
        }

        set {
            it.inject(app)
        }

        provides(AppDatabase::class.java, db)
    }
}

Right now there aren't any callbacks, however if you need it you can try to override the apply method and add your call at the end of the Statement (after invoking base.evaluate())

Okay. I'll give it a try and update here.


dibs: implement protein-translation

Hi! This is my first time ever making a pull request. Thought I would start learning to contribute to projects, and porting some code looks like a fun place to start. I'm not very experienced with github.
I'm trying to follow the instructions for contributing code here:
https://github.com/exercism/x-common/blob/master/CONTRIBUTING.md#porting-an-exercise-to-another-language-track
Please let me know if I'm doing it right, and if not, what I need to change. Thanks :)

👍 If you have any questions don't hesitate to ask or commit non working code so we can have a look and help you.

Yes I've been looking at the ruby tests and implementing those.

So I'm running into a problem: I can't get the tests to work. This is my first time doing unit testing as well. I understand the theory but I keep getting errors. This isn't just with my code either; it's the code in the other python tests.
For example, from the xpython/exercises/linked-list/ directory:
python linked_list_test.py
or
py.test linked_list_test.py
gives me huge AttributeErrors for everything, e.g.:
_____________________________ LinkedListTests.test_all _____________________________

self = <linked_list_test.LinkedListTests testMethod=test_all>

    def test_all(self):
>       self.list.push(10)
E       AttributeError: 'LinkedList' object has no attribute 'push'

linked_list_test.py:35: AttributeError

This is the same thing that happens with my own tests. Can you shed any light?

Sure, that happens because you are testing the stub file and not the example implementation.
You could either rename example.py or use the check-exercises.py script that automatically (with a tmp file) does it for you. It's also mentioned in the README.md.
$ cat ./exercises/linked-list/linked_list.py
class Node(object):
    def __init__(self):
        pass

class LinkedList(object):
    def __init__(self):
        pass

$ python ./exercises/linked-list/linked_list_test.py -f
E
======================================================================
ERROR: test_all (__main__.LinkedListTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./exercises/linked-list/linked_list_test.py", line 35, in test_all
    self.list.push(10)
AttributeError: 'LinkedList' object has no attribute 'push'

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

$ python ./test/check-exercises.py linked-list
#  linked-list
.ss....
----------------------------------------------------------------------
Ran 7 tests in 0.000s

OK (skipped=2)

TestEnvironment: Python2.7

SUCCESS!

Ok well that worked perfectly. Thank you! I guess the lesson is to always read the README.md. And now I have some working solutions, I'll push them up shortly.

I think that's it. Do I need to do anything else?

As you can see here the travis-ci build looks good: https://travis-ci.org/exercism/xpython/builds/237577149
That basically means that your code/tests work on all the different Python versions (2.7, 3.4-6) and that they meet the style requirements (flake8). Nicely done!
Now, I will try to find some time to do a review. Maybe today, but I can't promise you anything.

Oh I'm not in a rush. Maybe I'll start another one.

You can find unimplemented exercises here: http://exercism.io/languages/python/todo

The tests look good to me. I personally favor the context manger style of testing exceptions because it let's you make the call yourself. But this is just a question of taste, as both ways do the same thing.
def test_invalid_codons(self):
  strand = 'CARROT'
  self.assertRaises(InvalidCodonError, self.translate.ofRNA, strand)

def test_invalid_codons(self):
  with self.assertRaises(InvalidCodonError):
    self.translate.ofRNA('CARROT')

My feeling is that the class based solution doesn't really add any value here and a functional approach would be clearer.
Some examples where exercises have been moved from a class to functions: pull/85/files, pull/93/files
In general in Python snake case of_codon(param) is preferred over camel case ofCordon(param) for variables, methods and functions. Class names instead use camel case.
Maybe have a look at: http://pep8.org

Thanks for the feedback! I used the class-based solution because that's what the Ruby code I based it on does, but it's easily switchable. Also I think the exception is KeyError rather than ValueError. Will revise and update.

That looks much better. Only, some small details and than we should be able to merge this into master.
A) shebang
I don't really know why, but the only Python file with #!/usr/bin/env python is test/check-exercises.py. So I would remove that line to be consistent.
Maybe we should open an issue to investigate why we don't have it and/or if maybe it should be added.
B) codons dict
The name of a constant like codons should be upper case and the comment # Lookup table is redundant because thats basically what a Python dictionary is.
C) exception
I think the ValueError is better than KeyError as it's more flexible because we don't force the user to use a dictionary to implement this, even though I think it's the best option.
D) extract the chunk logic
So, this is up for debate. Do you think it helps to understand this, if we extract the logic to cut the string into chunks into its own function and by this are able to give it an explanatory name?
def of_codon(codon):
    if codon not in CODONS:
        raise ValueError('Invalid codon: %s' % codon)
    return CODONS[codon]

def of_rna(strand):
    proteins = []
    for codon in map(of_codon, _chunkstring(strand, 3)):
        if codon == "STOP":
            break
        proteins.append(codon)
    return proteins

def _chunkstring(string, n):
    return (string[i:n + i] for i in range(0, len(string), n))

Fixed A) and B), and C).
As for D), I guess it could go either way. I think I like your logic though. It does make it a bit more straightforward.

Thank you for your work. It will be live with the next update.


openapi-generator-cli: 6.4.0 -> 6.5.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/openapi-generator-cli/versions.
meta.description for openapi-generator-cli is: Allows generation of API client libraries (SDK generation), server stubs and documentation automatically given an OpenAPI Spec
meta.homepage for openapi-generator-cli is: https://github.com/OpenAPITools/openapi-generator
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 6.5.0 with grep in /nix/store/rsb6hnbsya07p01z2pkih8i4qjn1znbk-openapi-generator-cli-6.5.0
found 6.5.0 in filename of file in /nix/store/rsb6hnbsya07p01z2pkih8i4qjn1znbk-openapi-generator-cli-6.5.0

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
openapi-generator-cli

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/rsb6hnbsya07p01z2pkih8i4qjn1znbk-openapi-generator-cli-6.5.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A openapi-generator-cli https://github.com/r-ryantm/nixpkgs/archive/05f87ea0ee0a90436202a885a3687525d0bcfec0.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/rsb6hnbsya07p01z2pkih8i4qjn1znbk-openapi-generator-cli-6.5.0
ls -la /nix/store/rsb6hnbsya07p01z2pkih8i4qjn1znbk-openapi-generator-cli-6.5.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

openapi-generator-cli

Maintainer pings
cc @Shou for testing.

Result of nixpkgs-review pr 226411 run on x86_64-linux 1

1 package built:

openapi-generator-cli

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/910


Write proper unit tests

Write proper unit tests with mocha and chai to avoid any issues now and in the future.

Eventually a C# test project should also be added, so this can properly be tested against the latest Refit library, using the latest version of Mono etc.


[portmidi] build failure (download fails because project moved to GitHub)

port midi moved to https://github.com/PortMidi/portmidi on 2022-01-04
Failure logs
[cmake] https://sourceforge.net/projects/portmedia//files/portmedia-code-r234.zip/download:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[cmake]                                  Dload  Upload   Total   Spent    Left  Speed
[cmake]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
[cmake] curl: (22) The requested URL returned error: 404

...

Thanks for posting this issue. Did you install the latest version of portmidi failed? Could you please provide more infomation of this issue? I can install latest portmidi succeed in CMake project, and I can download portmidi succeed from https://sourceforge.net/projects/portmedia/files/portmidi/.
1> [CMake] -- Running vcpkg install
1> [CMake] Detecting compiler hash for triplet x64-windows...
1> [CMake] The following packages will be built and installed:
1> [CMake]     portmidi[core]:x64-windows -> 2.0.2
1> [CMake]   * vcpkg-cmake[core]:x64-windows -> 2021-12-20
1> [CMake]   * vcpkg-cmake-config[core]:x64-windows -> 2021-12-28
1> [CMake] Additional packages (*) will be modified to complete this operation.
1> [CMake] Restored 0 packages from C:\Users\v-lilywang\AppData\Local\vcpkg\archives in 676.2 us.  Use --editable to skip cleaning for the packages you specify.
1> [CMake] -- Extracting source E:/Lily/1114/vcpkg/downloads/PortMidi-portmidi-v2.0.2.tar.gz
1> [CMake] -- Using source at E:/Lily/1114/vcpkg/buildtrees/portmidi/src/v2.0.2-bd516840b2.clean
1> [CMake] -- Found external ninja('1.10.2').
1> [CMake] -- Configuring x64-windows
1> [CMake] -- Building x64-windows-dbg
1> [CMake] -- Building x64-windows-rel
1> [CMake] -- Fixing pkgconfig file: E:/Lily/1114/vcpkg/packages/portmidi_x64-windows/lib/pkgconfig/portmidi.pc
1> [CMake] -- Using cached msys-mingw-w64-i686-pkg-config-0.29.2-2-any.pkg.tar.zst.
1> [CMake] -- Using cached msys-mingw-w64-i686-libwinpthread-git-<IP_ADDRESS>06.c9a21571-1-any.pkg.tar.zst.
1> [CMake] -- Using msys root at E:/Lily/1114/vcpkg/downloads/tools/msys2/aa5af7b2aa7e90e8
1> [CMake] -- Fixing pkgconfig file: E:/Lily/1114/vcpkg/packages/portmidi_x64-windows/debug/lib/pkgconfig/portmidi.pc
1> [CMake] -- Installing: E:/Lily/1114/vcpkg/packages/portmidi_x64-windows/share/portmidi/copyright
1> [CMake] -- Performing post-build validation
1> [CMake] -- Performing post-build validation done
1> [CMake] Stored binary cache: C:\Users\v-lilywang\AppData\Local\vcpkg\archives\2f\2f5f32065adcade637bf65dcf29f515f3948c968fc8040b63a5fb134094e6fdf.zip
1> [CMake] Installing package portmidi[core]:x64-windows...
1> [CMake] Elapsed time for package portmidi:x64-windows: 6.897 s
1> [CMake]
1> [CMake] Total elapsed time: 14.09 s
1> [CMake]
1> [CMake] The package portmidi provides CMake targets:
1> [CMake]
1> [CMake]     find_package(PortMidi CONFIG REQUIRED)
1> [CMake]     target_link_libraries(main PRIVATE PortMidi::portmidi)
1> [CMake]
1> [CMake] -- Running vcpkg install - done

Thanks for checking so fast.
My mistake, I was on baseline 63e935d967a3410e26bf4a708efa39d8384d2bbb (December)
After adding
find_package(Threads REQUIRED) (should that be implicit?)
and linking to PortMidi::portmidi everything configures and builds fine now. (tested on macOS and Win10)
Thanks!

This issue has been resolved, I an closing this issue for now.


Add Brocfile.ts TypeScript support

This PR adds support for loading a typescript Brocfile and running it through ts-node.
If one provides a Brocfile.ts it will be auto picked up and run through ts-node.
An optional tsconfig.json can exist alongside the Brocfile.ts, however defaults taken from https://github.com/stefanpenner/fs-tree-diff/pull/89/files#diff-e5e546dd2eb0351f813d63d1b39dbc48 are provided
I think I've provided a type definition for the BrocfileOptions, but I may have gotten this wrong, plz inform.
Tests added.

@stefanpenner @rwjblue have you had a chance to check this out? Am I on the right path?

@chriskrycho Would you be interested in having a look at this given your work on ember-typescript?

@mike-north Would you be interested in having a look at this given your work on ember-typescript?

Ok, so here's an update.
I've removed the ts-node and typescript dependencies, and now throw an exception if they're not installed, prompting the user to install them.
However, I have some questions on how we want ts-node to work by default, and if we should provide default options.

Should type definitions be defined for imported modules? For example, doing import merge from 'broccoli-merge-trees', TS complains that there's no type information for this, and it's defaulting to any type. noImplicitAny: false is required to fix this
How should importing from CommonJS modules work? This is affected by esModuleInterop: true config option if we want it to work out of the box for import merge from 'broccoli-merge-trees'
Should we allow importing a JS file from a TS file? If so, allowJs: true is required.
if allowJs: true is used, the tests fail when there is more than 1 test that loads a TS brocfile (a single one passes, multiple fail):

     TSError: ⨯ Unable to compile TypeScript:
Brocfile.ts(2,24): error TS2683: 'this' implicitly has type 'any' because it does not have a type annotation.
Brocfile.ts(2,32): error TS2683: 'this' implicitly has type 'any' because it does not have a type annotation.
Brocfile.ts(5,23): error TS2304: Cannot find name 'exports'.
Brocfile.ts(6,43): error TS2580: Cannot find name 'require'. Do you need to install type definitions for node? Try `npm i @types/node` and then add `node` to the types field in your tsconfig.
Brocfile.ts(8,1): error TS2304: Cannot find name 'exports'.

😞

Should type definitions be defined for imported modules? For example, doing import merge from 'broccoli-merge-trees', TS complains that there's no type information for this, and it's defaulting to any type. noImplicitAny: false is required to fix this. The downside of requiring this is that no broccoli plugins currently have type definitions, so this would be painful for early adopters.

this is a pretty general issue having to do with typescript. Consumers should either define their own types locally, PR types into the dependency (through a declaration file or a JS -> TS conversion) or add a package to DefinitelyTyped

How should importing from CommonJS modules work? This is affected by esModuleInterop: true config option if we want it to work out of the box for import merge from 'broccoli-merge-trees'

the way to do this without forcing consumers to set this flag is
// import
import foo = require('foo');

// export
export = 'foo';

Thank you to everyone who contributed feedback on this PR 🎉


Got cutlass error: Error Internal at: 171

Hello, I am currently trying to reproduce your training instance in a Windows PC. The development setup is mentioned as below:
OS: Windows Pro
CUDA Toolkit: 11.0
Graphics Card: Quadro P620 (Pascal Architecture - sm_61)
CUDNN Version: 8.2.0
Python: 3.8.3
Torch: 1.7.1+cu110

**The paths for Windows are adjusted in setup.py script for all cuda extensions. To address the graphics card architecture I replaced the using SmArch = cutlass::arch::Sm75 with using SmArch = cutlass::arch::Sm61 in source code file niti\pytorch\cutlass-extension\cutlassconv_kernel.cu (7 occurences)
I can see the installed packages in this location after successful make: C:\Users*Username*\AppData\Local\Programs\Python\Python38\Lib\site-packages
When running make architecture is confirmed by the following line at the end of the compile command. (-gencode=arch=compute_61, code=sm_61)

When training I came across "Got cutlass error: Error Internal at: 171" error.

Can you please check the issue and propose an appropriate solution.

Pascal architecture is too old and doesn't have tensor cores.

Is there python function (instead of CUDA extension functions) to avoid using CUDA and Tensors at all? If there is, can you say if the training time is increased significantly?


Strip out the prefix 'docker.io' when loading images in kind

If an image starts with docker.io/, docker strips it from the result of
docker images. Then those images were not being loaded into kind cluster.

@jwendell: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

lint_istio
d1983104d8a93a18a66e683bc2654250bc8bb550
link
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@jwendell: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

lint_istio
d1983104d8a93a18a66e683bc2654250bc8bb550
link
/test lint_istio

integ-istioio-k8s-tests_istio
d1983104d8a93a18a66e683bc2654250bc8bb550
link
/test integ-istioio-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-istioio-k8s-tests_istio

@jwendell: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-istioio-k8s-tests_istio
ff83a9ab418bfe5b43368aefb6c39050c3e64566
link
/test integ-istioio-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-istioio-k8s-tests_istio


run attack_iter.py

run attack_iter.py and get an error like that " UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte ".

Does the error occur when loading images? Please make sure that your images can be loaded by scipy.misc.imread.

@gmt710
Hi!
Have you solved it? I meet the same error?

@gmt710
Hi!
Have you solved it? I meet the same error.

@WeifaGan ,hello.
I vaguely remember that it might be the scipy version that caused this problem.

Hello, your method is very good. I am repeating your method recently. I'm a novice. I don't understand many problems. 1. I'd like to ask V3, V4 and other models to train and save with your code (such as V3_ test.py Training is not successful), or download it on the official website? 2. About attack_ iter.py File directory configuration problems in, such as checkpoint_ path_ ens3_ adv_ inception_ How to initialize the path of V3?Sorry, there are a lot of questions. I hope to understand. I'd like to reproduce your methods to help my research. If possible, I can send QQ email<EMAIL_ADDRESS>, you can reply and view at any time, thanks

run attack_iter.py and get an error like that " UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte ".

Hello, are you still studying this field? How is your reproduction of the author's method?

Well, I have graduated, but I will focus on this field. I'm a little busy, so I don't have to devote a lot of energy to studying Adversarial examples as before.

This code is not hard to run.If you have more questions, you can solve them.If you're new, I suggest you take a look at foolbox. Then run a little demo of the MNIST dataset.

Well, I have graduated, but I will focus on this field. I'm a little busy, so I don't have to devote a lot of energy to studying Adversarial examples as before.

This code is not hard to run.If you have more questions, you can solve them.If you're new, I suggest you take a look at foolbox. Then run a little demo of the MNIST dataset.

Oh, congratulations on graduation. I just entered school. The teacher arranged to study this direction and asked me to run the author's code as soon as possible. Now I have downloaded the dataset. But there are many models in the article, I don't have them. So attack_ iter.py Unable to run. How can I solve it? (you can reply slowly if you are busy) thank you

Thanks.
What problem do you have met?May You be able to search on your website.

@gmt710
Hi!
Have you solved it? I meet the same error.

Hi, have you finally solved it? I have a few questions for you. Do you have time?

Hi, sorry to reply late. This code is used to attack pre-trained models. So you don't need to train those models. We have provided a link to those models. You can also download them at the official repo.

Hi, sorry to reply late. This code is used to attack pre-trained models. So you don't need to train those models. We have provided a link to those models. You can also download them at the official repo.

Hello, thank you for your reply. The previous problem has been solved. Now there is a question about the role of files under nets in the project (such as inception_ V1_ test.py , it doesn't seem to be used to test against images, so what's its purpose?

Hi, sorry to reply late. This code is used to attack pre-trained models. So you don't need to train those models. We have provided a link to those models. You can also download them at the official repo.

Hello, thank you for your reply. The previous problem has been solved. Now there is a question about the role of files under nets in the project (such as inception_ V1_ test.py , it doesn't seem to be used to test against images, so what's its purpose?

Right. There is no need to keep such files. We just copt the nets folder from https://github.com/tensorflow/models/tree/master/research/slim.

Hi, sorry to reply late. This code is used to attack pre-trained models. So you don't need to train those models. We have provided a link to those models. You can also download them at the official repo.

Hello, thank you for your reply. The previous problem has been solved. Now there is a question about the role of files under nets in the project (such as inception_ V1_ test.py , it doesn't seem to be used to test against images, so what's its purpose?

Right. There is no need to keep such files. We just copy the nets folder from https://github.com/tensorflow/models/tree/master/research/slim.

Thank you. How can I solve this problem：(Here is the problem code)
for filepath in tf.gfile.Glob(os.path.join(input_dir, '*.png')):
with tf.gfile.Open(filepath) as f:
image = imread(f, mode='RGB').astype(np.float) / 255.0    (The picture cannot be read correctly here，The error is ：'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte)
# Images for inception classifier are normalized to be in [-1, 1] interval.
images[idx, :, :, :] = image * 2.0 - 1.0
filenames.append(os.path.basename(filepath))
I have looked at many materials and tried to modify them, but none of them have been solved.What should I do? Can you help me? Thank you very much

Hi, sorry to reply late. This code is used to attack pre-trained models. So you don't need to train those models. We have provided a link to those models. You can also download them at the official repo.

Hello, thank you for your reply. The previous problem has been solved. Now there is a question about the role of files under nets in the project (such as inception_ V1_ test.py , it doesn't seem to be used to test against images, so what's its purpose?

Right. There is no need to keep such files. We just copy the nets folder from https://github.com/tensorflow/models/tree/master/research/slim.

Thank you. How can I solve this problem：(Here is the problem code)
for filepath in tf.gfile.Glob(os.path.join(input_dir, '*.png')):
with tf.gfile.Open(filepath) as f:
image = imread(f, mode='RGB').astype(np.float) / 255.0 (The picture cannot be read correctly here，The error is ：'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte)
Images for inception classifier are normalized to be in [-1, 1] interval.
images[idx, :, :, :] = image * 2.0 - 1.0
filenames.append(os.path.basename(filepath))
I have looked at many materials and tried to modify them, but none of them have been solved.What should I do? Can you help me? Thank you very much

I think it's due to the scipy version. You can try to use early versions of scipy, such as v1.2. Or you can try to use other image loading functions, such as cv2.

Hi, sorry to reply late. This code is used to attack pre-trained models. So you don't need to train those models. We have provided a link to those models. You can also download them at the official repo.

Hello, thank you for your reply. The previous problem has been solved. Now there is a question about the role of files under nets in the project (such as inception_ V1_ test.py , it doesn't seem to be used to test against images, so what's its purpose?

Right. There is no need to keep such files. We just copy the nets folder from https://github.com/tensorflow/models/tree/master/research/slim.

Thank you. How can I solve this problem：(Here is the problem code)
for filepath in tf.gfile.Glob(os.path.join(input_dir, '*.png')):
with tf.gfile.Open(filepath) as f:
image = imread(f, mode='RGB').astype(np.float) / 255.0 (The picture cannot be read correctly here，The error is ：'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte)
Images for inception classifier are normalized to be in [-1, 1] interval.
images[idx, :, :, :] = image * 2.0 - 1.0
filenames.append(os.path.basename(filepath))
I have looked at many materials and tried to modify them, but none of them have been solved.What should I do? Can you help me? Thank you very much

I think it's due to the scipy version. You can try to use early versions of scipy, such as v1.2. Or you can try to use other image loading functions, such as cv2.

Hello,"filepath" already refers to a specific picture, but here (with... As) is also used, resulting in imread () can not read the picture successfully, so the (with... As) line of code should be deleted, so it can work normally.
code：for filepath in tf.gfile.Glob(os.path.join(input_dir, '*.png')):
with tf.gfile.Open(filepath) as f:    (should be deleted)
image = imread(f, mode='RGB').astype(np.float) / 255.0  (replace   it with  image = imread(filepath).astype(np.float) / 255.0)


fix: potential typo in resnet comments

Hi,
Could this be a potential typo in the comments for the Resnet implementation? Thought I'd check with you. Really appreciate your work here! Amazing stuff!

Thank you!


Feat: Add specific recevier support

Describe the solution you'd like
The ability send message to specific recevier
telegramService, _ := telegram.New("your_telegram_api_token")
notify.UseServices(telegramService)

fooReceiver := telegram.Receiver(-1234567890)
barReceiver := telegram.Receiver(-0987654321)

// Send a message to fooReceiver .
_ = notify.To(fooReceiver).Send(
	context.Background(),
	"Hello",
	"The message to fooReceiver",
)
// Send another message to barReceiver
_ = notify.To(fooReceiver).Send(
	context.Background(),
	"Hello",
	"The message to barReceiver",
)

Hey @KaptinLin, thanks for sharing your idea/problem.
Have you considered using separate notify instances? For example like this:
// Notifier for receiver 1
service1, _ := telegram.New("")
service1.AddReceivers(-1234567890)

n1 := notify.NewWithServices(service1)
_ = n1.Send(context.Background(), "Hello", "The message to fooReceiver")

// Notifier for receiver 2
service2, _ := telegram.New("")
service2.AddReceivers(-9876543210)

n2 := notify.NewWithServices(service2)
_ = n2.Send(context.Background(), "Hello", "The message to barReceiver")

I can see that your solution would add more flexibility but I want to consider other solutions first before agreeing or declining your idea.

@nikoksr yes, it can works.
I was look for a way not initialize the service when send to different receivers.

hi @nikoksr @KaptinLin has this feature been implemented? I think I want to start contributing by solving this issue, would you mind?

nope, if you can contribute that would be awesome

@rafimuhammad01 welcome to the project. I appreciate your willingness to support the project! We'd all be excited for your contributions in any way shape or form.
@KaptinLin could you please clarify why this feature would be beneficial for the project? This seems like a subtle pattern change and I don't want to accept changes that unnecessarily change Notify's usage patterns.
Also, I'm not happy with the linguistic choice here. Keeping in mind that chained methods should sound as close as possible to a actual english sentence, the signature is not logically sound. Notify to <receiver> send <message> is far from being gramatically correct. Something like Notify send <message> to <receiver> would be cleaner. Obviously this does not work, since the send method returns an error type and thus does not allow for further method chaining.
To summarize, I'd more than happily accept your guys' idea and contribution but we need a clear explanation of what problem this exactly solves, plus, the signature has to change. I'm not okay with notify.To(...).Send(...).
cc @svaloumas

Closing until further notice.


Update README.md

Revised the readme to follow a more technical format
Removed 2nd person point of view
Reduced the amount of pronouns (them, it) to increase clarity

@CarlosFromNewYork thank you for your contribution.
I allowed myself to modify this PR a little bit in order for it to fit recent changes that has been made to README file.


Masking In the Middle of the Number/String

Initailly thanks for the great plugin.
while iam trying to mask digits  in the middle of the string I was facing a problem . i.e
Example:
I was trying to mask the below number in this format 9999xxxx99
input value :<PHONE_NUMBER>
Output: 1234XXXX567890
Expected Output : 1234XXXX90
As iam checking the js file , I was unable to figure out the issue. Can anyone help to fix the issue??

How did you define the mask?


For scvmm, set the default security protocol to ssl

The Validate button for scvmm provider goes active even though the Security Protocol is not populated.
The changes in this PR ensure that the Security Protocol for scvmm is prepopulated with ssl -- this will ensure that Validation takes into account the security protocol value and does proper validation.
Before :
Validate button is active even though no Security Protocol is selected.
Also, Flash message appears to be a Rails error more than the actual Validation error

After:
Security Protocol is prepopulated. Flash message shows the actual Validation error

https://bugzilla.redhat.com/show_bug.cgi?id=1383222

LGTM 👍

@simaishi Please clone https://bugzilla.redhat.com/show_bug.cgi?id=1383222 for Euwe

Euwe Backport details:
$ git log -1
commit e42e425cb732c768ecbd9f3057e38566b127006d
Author: Milan Zázrivec<EMAIL_ADDRESS>Date:   Tue Oct 18 17:28:12 2016 +0200

    Merge pull request #11965 from AparnaKarve/bz1383222_security_protocol_mandatotory

    For scvmm, set the default security protocol to ssl
    (cherry picked from commit 2f44353da52391d1edf2e562ac7d52ea2aed00ac)

    https://bugzilla.redhat.com/show_bug.cgi?id=1383222

Euwe BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1387409


Pipeline stuck, impossible to update, do not launch

What happened?:
We have a small data architecture based on Pachyderm. One of the pipeline is totally stuck, and can not process the last commit it has on a new datum.
Behavior is quite weird :

When starting pipeline : "pachctl start pipeline metadatas_one", the pods are created, but the pipeline is still set as paused in "pachctl list pipeline" :

pachctl list pipeline
NAME                          VERSION INPUT                                                                                CREATED      STATE / LAST JOB DESCRIPTION
metadatas_one                 4       ((compl_metadata:/(*) ⋈ dump_metadata:/(*)) ∪ (dump_metadata:/(*) ⋈ image_annotated:/(*))) 14 hours ago paused / killed  Merge entre dump_metadatas && compl_metadatas -> metadatas_one

Every one of the three pods for this pipeline has the same error log :

kubectl logs pipeline-metadatas-one-v3-f844t -c user
{"pipelineName":"metadatas_one","workerId":"pipeline-metadatas-one-v3-f844t","master":true,"ts":"2022-01-05T09:22:31.477528338Z","message":"errored transform spawner process: jobs metadatas_one@af9893153b1e41b1b01faac4f235d155 not found"}
{"pipelineName":"metadatas_one","workerId":"pipeline-metadatas-one-v3-f844t","master":true,"ts":"2022-01-05T09:22:31.481270069Z","message":"master: error running the master process, retrying in 17.73980232s: jobs metadatas_one@af9893153b1e41b1b01faac4f235d155 not found\ngithub.com/pachyderm/pachyderm/v2/src/internal/grpcutil.ScrubGRPC\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/internal/grpcutil/error.go:16\ngithub.com/pachyderm/pachyderm/v2/src/client.APIClient.InspectJob\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/client/pps.go:211\ngithub.com/pachyderm/pachyderm/v2/src/server/worker/pipeline/transform.(*pendingJob).load\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/server/worker/pipeline/transform/pending_job.go:107\ngithub.com/pachyderm/pachyderm/v2/src/server/worker/pipeline/transform.(*registry).startJob\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/server/worker/pipeline/transform/registry.go:134\ngithub.com/pachyderm/pachyderm/v2/src/server/worker/pipeline/transform.Run.func1\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/server/worker/pipeline/transform/transform.go:30\ngithub.com/pachyderm/pachyderm/v2/src/client.APIClient.SubscribeJob\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/client/pps.go:391\ngithub.com/pachyderm/pachyderm/v2/src/server/worker/pipeline/transform.Run\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/server/worker/pipeline/transform/transform.go:19\ngithub.com/pachyderm/pachyderm/v2/src/server/worker.runSpawner.func2\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/server/worker/worker.go:174\ngithub.com/pachyderm/pachyderm/v2/src/server/worker/logs.(*taggedLogger).LogStep\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/server/worker/logs/logger.go:190\ngithub.com/pachyderm/pachyderm/v2/src/server/worker.runSpawner\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/server/worker/worker.go:173\ngithub.com/pachyderm/pachyderm/v2/src/server/worker.(*Worker).master.func1\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/server/worker/worker.go:139\ngithub.com/pachyderm/pachyderm/v2/src/internal/backoff.RetryUntilCancel\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/internal/backoff/retry.go:129\ngithub.com/pachyderm/pachyderm/v2/src/internal/backoff.RetryNotify\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/internal/backoff/retry.go:118\ngithub.com/pachyderm/pachyderm/v2/src/server/worker.(*Worker).master\n\t/Users/avigil/Pachyderm/Pachyderm_releases_2.x/pachyderm/src/server/worker/worker.go:128\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"}

An interesting point is that this job id (af9893153b1e41b1b01faac4f235d155) does not appear in the pachctl list job pipeline@master

When trying to launch a whole reprocess of the pipeline, we have a very frightening error :

pachctl update pipeline -f=pipeline.json --reprocess
more than one matching row found in pipelines on index version with value metadatas_one@00000004

Which looks like there is some conflict in the postgres Database

The postgres pod show some errors too :

2022-01-05 09:04:31.218 GMT [873095] STATEMENT:  insert into collections.branches (key, proto, idx_repo, version) values ($1, $2, $3, $4) on conflict (key) do update set (key, proto, idx_repo, version) = ($1, $2, $3, $4)
2022-01-05 09:04:31.793 GMT [873091] ERROR:  could not serialize access due to read/write dependencies among transactions
2022-01-05 09:04:31.793 GMT [873091] DETAIL:  Reason code: Canceled on identification as a pivot, during write.
2022-01-05 09:04:31.793 GMT [873091] HINT:  The transaction might succeed if retried.
2022-01-05 09:04:31.793 GMT [873091] STATEMENT:  update collections.repos set version = $1, key = $2, proto = $3, idx_name = $4, idx_type = $5 where key = $6

The pipeline specification is as follows :

{
    "pipeline": {
      "name": "metadatas_one"
    },
    "description": "XXXX",
    "parallelism_spec" : {
      "constant":3
    },
    "input": {
        "union" : [
          {
            "join" : [
              {
                "pfs": {
                  "repo": "dump_metadata",
                  "branch": "master",
                  "glob": "/(*)",
                  "join_on": "$1",
                  "outer_join": true
                  }
              },
              {
                "pfs": {
                    "repo": "compl_metadata",
                    "branch": "master",
                    "glob": "/(*)",
                    "join_on": "$1"
                }
              }
            ]
          },
          {
            "join" : [
              {
                "pfs": {
                  "repo": "dump_metadata",
                  "branch": "master",
                  "glob": "/(*)",
                  "join_on": "$1"
                  }
              },
              {
                "pfs": {
                    "repo": "image_annotated",
                    "branch": "master",
                    "glob": "/(*)",
                    "join_on": "$1"
                }
              }
            ]
          }

        ]
    },

    "transform": {
      "cmd": [ "/pipeline/start.sh" ],
      "image": "IMAGE"
    }
  }

What you expected to happen?:
At least, the reprocess should create a new "clean" version of the pipeline
How to reproduce it (as minimally and precisely as possible)?:
No idea at this point, deeply sorry about that
Anything else we need to know?:
There may be some bad practices we have, as we :

Often stop and start pipelines
Dumped in the input repository (here, dump_metadata) multiple times the same folder

Environment?:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:10:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5-gke.1302", GitCommit:"639f3a74abf258418493e9b75f2f98a08da29733", GitTreeState:"clean", BuildDate:"2021-10-21T21:35:48Z", GoVersion:"go1.16.7b7", Compiler:"gc", Platform:"linux/amd64"}

Pachyderm CLI and pachd server version (use pachctl version):

COMPONENT           VERSION
pachctl             2.0.4
pachd               2.0.4

Cloud provider (e.g. aws, azure, gke) or local deployment (e.g. minikube vs dockerized k8s):

GKE

If you deployed with helm, the values you used (helm get values pachyderm):

USER-SUPPLIED VALUES:
deployTarget: GOOGLE
global:
  postgresql:
    postgresqlPassword: PASS
    postgresqlPostgresPassword: PASS
pachd:
  clusterDeploymentID: ID
  enterpriseSecret: ""
  externalService:
    enabled: true
  oauthClientSecret: ""
  rootToken: ""
  storage:
    google:
      bucket: exxacts-dataprod-v2
      cred: |
        JSON

OS (e.g. from /etc/os-release):
Debian
Others:

We investigated if there was any duplicate row in the postgresql database, does not look like the case. Hope it helps :
pachyderm=> select idx_name,idx_version from "collections"."pipelines" where idx_name='metadatas_one';
   idx_name    |      idx_version
---------------+------------------------
 metadatas_one | metadatas_one@00000001
 metadatas_one | metadatas_one@00000002
 metadatas_one | metadatas_one@00000003
 metadatas_one | metadatas_one@00000004

@quematech - My first suspicion here is that there may be an open commit which was never finished for some reason.  I say that because I've seen that happen before and the "job not found" was a side effect of it.  Can you run pachctl list commit --all --expand and then post the results here?  That will tell us definitively if that's the issue.
In case that's not the issue, though, can you also run pachctl debug dump debug-dump.tar.gz and upload the resultant file here?  That will give me a complete dump of the current state of the system to look through.
Thanks,
Dale

@dgeorg42 : Thank you a lot for your answer.
Looks like you're right, we got an open commit in the "meta" of metadatas_one, with the corresponding id (that was given as a "job id" in the pod logs) :
compl_metadata         master af9893153b1e41b1b01faac4f235d155 42 hours ago 0B       ALIAS
dump_metadata          master af9893153b1e41b1b01faac4f235d155 42 hours ago 13.2MiB  ALIAS
image_annotated        master af9893153b1e41b1b01faac4f235d155 42 hours ago 0B       ALIAS
metadatas_one.spec     master af9893153b1e41b1b01faac4f235d155 42 hours ago 0B       ALIAS
metadatas_one.meta     master af9893153b1e41b1b01faac4f235d155 -            ≤ 0B     AUTO

What is interseting is that :

this commit does not exist on metadatas_one@master.
all commits are ALIAS or this very AUTO
all commits are finished except from this one :

pachctl inspect commit metadatas_one.meta@af9893153b1e41b1b01faac4f235d155
Commit: metadatas_one@af9893153b1e41b1b01faac4f235d155
Original Branch: master
Parent: b88d8c5b912e4ea39e3c7107506a3117
Started: 42 hours ago

I can list files but not datums, and find this "job not found" error while trying to check the datum :

pachctl list file metadatas_one.meta@af9893153b1e41b1b01faac4f235d155
NAME   TYPE SIZE
/meta/ dir  87.98KiB
/pfs/  dir  26.21MiB
pachctl list datum metadatas_one.meta@af9893153b1e41b1b01faac4f235d155
ID FILES STATUS TIME
jobs metadatas_one.meta@af9893153b1e41b1b01faac4f235d155 not found
I don't want to mess it up, so what would you recommend for this case ? I'm not even sure to understand what this "meta" is. Trying to squash/delete the whole commit ? Moreover, we have finished commits on this repo after this one commit :
pachctl list commit --all --expand | grep metadatas_one.meta
metadatas_one.meta master b77a7db9755e49babdabd8a664e09a20 20 hours ago 26.3MiB  AUTO
metadatas_one.meta master 91e8a468c11a47578d4724b9e2c9bb64 24 hours ago 26.3MiB  AUTO
metadatas_one.meta master 689e14ab15c74f10922cb6af045889bf 38 hours ago 26.3MiB  AUTO
metadatas_one.meta     master 389e2a0b5b014133b19b914cd51a2932 38 hours ago 26.3MiB  AUTO
metadatas_one.meta     master a3b8810324f84b6993cfaed14a08b637 41 hours ago 26.3MiB  AUTO
metadatas_one.meta     master bc2662bdda104d1d98cd42c6eed7f668 42 hours ago 26.3MiB  AUTO
metadatas_one.meta     master af9893153b1e41b1b01faac4f235d155 -            ≤ 0B     AUTO
metadatas_one.meta     master b88d8c5b912e4ea39e3c7107506a3117 42 hours ago 26.3MiB  AUTO

Thank you a lot for your time and your help, and best regards

@quematech - OK, so this is exactly the same problem that I saw with one of our Enterprise customers.  Fortunately, it is easy to resolve.  You just need to force that commit to finish with the following command:
pachctl finish commit metadatas_one.meta@af9893153b1e41b1b01faac4f235d155
Everything should start working properly after that.
To answer your other question about what precisely the meta repo is - Every pipeline has a couple of "system repos" associated with it, which typically are hidden (which is why we used the --all flag so we could see them).  One of these is the "spec" repo where we store each version of the pipeline spec whenever you update it; another is the "meta" repo, where certain metadata about the pipeline executions (such as provenance information) gets stored.
I'm going to bring this to Engineering's attention since this is now the second time we've encountered an unfinished commit on a meta repo, and we haven't yet identified the cause.
Thanks,
Dale

@dgeorg42 : I'm sorry, but the finish commit command does not look like it works :
pachctl finish commit metadatas_one.meta@af9893153b1e41b1b01faac4f235d155
cannot finish a pipeline output or meta commit, use 'stop job' instead

I tried a stop job but no luck :
pachctl stop job  metadatas_one.meta@af9893153b1e41b1b01faac4f235d155
error from StopJob: jobs metadatas_one.meta@af9893153b1e41b1b01faac4f235d155 not found

Would you have any idea ?
Thank you a lot

I guess reading from the documentation that we could use the -force parameter, but I would prefer to be sure before forcing anything :)

Ok, we recreated the usecase on another pipeline and tried the -force option. It does work perfectly :)
Thank you a lot @dgeorg42 for your time.
If you want, we can look into a minimal reproducable scenario.
Best Regards

Unfortunately, yes, -force was necessary there as you found. As a result of the issue, no job was ever created, so StopJob wouldn't work. That also means that using -force here isn't too risky.
I'm not sure how valuable it is to you to try reproducing the issue, but it would be useful for us to know a little bit more about your pachyderm workflow besides frequent stopping and starting - do use triggers or manual branch movement (basically anything other than just putting data into an input repo)?

Hi.
(Quite) Basically, we have some input repositories that receive the data and some pipelines that process it sequentially. We use a shell script to stop/start pipelines while dumping data, as we have a few number of nodes.
For more details, it would be easier to have a private exchange. We had talked with some people from your company in 2021 and they had suggested we exchange with your engeneering team. However, confidentiality was, if I remember properly, an issue.
Best Regards


Long revision names fixed

Fixes #13
For some cases previous fix can not work.

Thanks for raising and the PR @norbertsuski