We need Not to properly express some parts of Intersections

TBD, see #16
Edit: This issue is currently incomplete and is being used to transparently post components of it for review. Please hold comments on the following topics for now:

Too complex
Unneeded
Type-checking implementation detail vs specification

Please do comment if you can directly find fault in any labeled resource that is a fault arising from an internal self-consistency problem.
Please hold comments that think something isn't relevant. Many of these are either to help illustrate later details or to provide helpful resources, but may not become part of the pep itself. The goal is to start by being correct, then to refine our definitions to be as simple and succinct as possible from there, while having the proof of that correctness available as a resource. We can cut sections for irrelevance when we have the full picture, not before.
If there is an issue with this after all is presented, please point it out then. I am not spending my time translating abstract logic to Python's type system and to a form that people are more familiar with as simply an exercise in doing so.  I'm doing so because Python doesn't have a formalized type system, so we need to put it into the language people here are all familiar with and use so that we can look for any fault in it from that shared point of understanding.
Intersections get extremely into the weeds of type theory when considering a language with no formalized type system, only a set of not fully internally consistent rules. Please do not expect that proofs be simple when the situation itself is not. We can come up with simple procedures after we have correctness shown.

Not is needed to decouple ordering from overload sets, allowing Intersections to be unordered
Not and XOR are invalid on Gradual types with current knowledge and theory

I assume @mikeshardmind can fill in the rest based on other discussion, please do not wait on me feeling like explaining this.

@mniip and @DiscordLiz both had things that arose from abstract logic in discord. I understand them, but I'm still wrangling some of how to best succinctly express them. There has been a lot of discussion today, so I may not get to explain all of this today/tonight. If people would like to read through the details on Discord, they can, but it is a lot to preparse through, and I do not mind taking the time to come up with an accurate formulation of the ideas requiring this, but need some time to do so.

I'm not putting the whole thing tonight, I'm exhausted.

The order of overloads currently matters
We do not want the order of Intersection operands to matter (ie. A & B should equal B & A)
Because A and B can each have overloads, 1 conflicts with 2
To resolve this, we can determine which overloads have disjoint parameter lists and which are specializations of each other, and synthesize new overloads which do not rely on the ordering to be correct. This requires type negation to do.
In the case that 4 cannot be resolved, it is either unsafe or ambiguous as any arbitrary order would make A & B != B & A
By 5, we need to resolve 4 to ensure a use is safe given the information available to static analysis.

I will not elaborate with python examples or with a formal proof of this tonight.

There has already been prior art on this done in Rust in an accepted RFC https://github.com/rust-lang/rfcs/blob/master/text/1210-impl-specialization.md#permitting-overlap
For those who do not want to read the whole thing, this is the opening to the section I have linked:

The goal of specialization is to allow overlapping impls, but it's not as simple as permitting all overlap. There has to be a way to decide which of two overlapping impls to actually use for a given set of input types. The simpler and more intuitive the rule for deciding, the easier it is to write and reason about code -- and since dispatch is already quite complicated, simplicity here is a high priority.

If you believe that you can show a simpler method than this, you may want to reach out to the rust team as well 😃
While overloads and intersections do not fully map to rust's impls, and dispatch on type to an implementation is different to overloads, the static typing concerns are highly related, and it can still help us ensure we pick the most accurate return type from available implementations.

one additional quote from the Rust RFC:

One nice thing about this approach is that, if there is an overlap without there being an intersecting impl, the compiler can tell the programmer precisely which impl needs to be written to disambiguate the overlapping portion.

Which means we have a path towards helpful error messages with this as well. This may be more a concern of type checkers than this PEP, but the fact that the approach provides such a nice path is a big plus.

I don't know if this was clear to everyone else from the partial summary.

Negation is needed to synthetically remove the ordering of overloads, but not for any of the steps above which are listed after it.
Removing the ordering of overloads is necessary for many of the things we've all "agreed" on such distribution rules working properly.

I have not composed all of my thoughts on this into one post yet. I can show that type negation is necessary to remove the ordering via both type theory and topology, but I would prefer to stick to type theory given the problem domain. I would not mind being proven to have missed an alternative option for doing this, but it isn't as simple as "intersect the first on one side with the first on the other", and I believe it will become more clear why later. A hint to why this is the case is that it is possible to have a "smaller" compatible minimum bound than this in some cases, and that this also needs to interact with comparing which implementations can be seen as specializations of others after.
I would highly appreciate that people coming into this discussion be aware that a lot of this has been discussed elsewhere already, and, at least for why type negation is necessary to remove the ordering of overloads, it is only a matter of forming a "to the point" post explaining all of what has already been explored and the ramifications of it. Independently of what I have done, there's at least two other people who have each started some kind of machine verifiable formal proof of some of these conclusions. I do not want to discourage other contributors from voicing issues with what is later shown, to the contrary if I and others have been wrong on some of this show us. I am happy to engage in constructive contradictions. But I want them to actually be constructive and not a chore to respond to because enough detail was not included to refute something which claims to. When this happens, it can feel like people aren't even reading what each other are saying.
I also would want to encourage other people to come talk with us less formally about those things. @mikeshardmind has already committed to ensuring all of what is relevant gets summarized for at minimum an appendix to the PEP, so you will not be missing out after the fact if you do not, but people should be empowered to be a part of the precursor discussions as well. The discord link has been shared multiple times already, but here it is again

Because A and B can each have overloads, 1 conflicts with 2

Can I request this be expanded on?  It seems untrue to me, or more specifically that it presupposes the need to synthesise the interface in a concrete way, and if that is the case I'd request for that to be expanded on.

I agree with @diabolo-dan here, in python A&B will only exist statically, hence there is no need for a synthesized interface. If a user wants to implement a subtype AB of A&B, they are responsible themselves for creating a list of overloads for AB.foo that is compatible with both A.foo's overloads and B.foo's overloads. The type-checker can verify that this is compatible with (A&B).foo without ever needing a concrete realization of (A&B).foo other than the abstract definition of intersection.

Again, checking that a type is consistent with A&B (assignment) is different from checking that if all you know is A&B and not the type that will be assigned to A&B, what you concretely know about the use
The difference is here:
def f(x: A & B):
    x.foo(...)  # Method foo
    # or
    x(...)  # Callables A & B
    # We only know foo by the minimum bound provided by A & B here.
    # We need a procedure to correctly determine the minimum bound.

This isn't about if some type is A & B, but if some use is A & B.
When it comes to methods of classes or callables in general, the way parameter lists and return types interact with substitution is slightly more complicated. While parameters may be more specific, return types must not be for the same original parameter types (Yes, I spelled out variance of callables without naming each one, intentional to avoid another point of confusion for people)
If I need to figure out some more contrived examples to show it is necessary to do more than check consistency with each individually, I will later. The need for it was not shown by a direct example originally, and I'm more focused on summarizing correctly what was shown first as I have time.

This isn't about if some type is A & B, but if some use is A & B.

A use is A & B iff the use is A or the use is B.
A type checker can check those independently to validate the use is valid at all.
If neither are valid, then the use is invalid.
If exactly one is valid, the return type will be the return type of the valid use.
If both are valid, the return your will be the intersections of the return types.

If people don't have anything additional to add that wasn't already argued, wait until this is fully explained. You're just creating the same arguments that end up wasting time having to be addressed when they could be being spent on detailing more of the problem space.

Generalized specialization from known type information
This is single, position only, required parameters for the sake of the building blocks for later
General forms (Table A)

Intersection
Result

(A) -> A & (A) -> B
Never

(A) -> A & (B) -> B
(T) -> T, T: A | B

(A) -> C & (B) -> D
(A &~B) -> C & (B & ~A) -> D & (A & B) -> C & D

(A & ~B) -> A & (B & ~A)
(T) -> T, OneOf(A, B)

(A) -> A & (B) -> B  & (C) -> C
(T) -> T, T: A | B | C

(A & ~(B | C)) -> A & (B & ~(A | C)) -> B & (C & ~(A | B)) -> C
(T) -> T, OneOf(A, B, C)

General form, with B as a subtype of A and D as a subtype of C (Table B)

Intersection
Result

(A) -> A & (A) -> B
(A) -> B

(A) -> A & (B) -> B
(B) -> B

(A) -> C & (B) -> D
(A &~B) -> C & (B) -> D

(A & ~B) -> A & (B & ~A)
(A) -> A

(A) -> A & (B) -> B  & (C) -> C
(T) -> T, T: B | C

(A & ~(B | C)) -> A & (B & ~(A | C)) -> B & (C & ~(A | B)) -> C
(T) -> T, OneOf(A, C)

General form, with B as a subtype of A and D is not a subtype of C (Table C)

Intersection
Result

(A) -> C & (B) -> D
(A &~B) -> C

General forms, A, B, C, D are mutually incompatible (Table D)

Intersection
Result

(A) -> A & (A) -> B
Never

(A) -> A & (B) -> B
(T) -> T, T: Oneof(A, B)

(A) -> C & (B) -> D
Never

(A & ~B) -> A & (B & ~A)
(T) -> T, OneOf(A, B)

(A) -> A & (B) -> B  & (C) -> C
(T) -> T, T: OneOf(A, B, C)

(A & ~(B | C)) -> A & (B & ~(A | C)) -> B & (C & ~(A | B)) -> C
(T) -> T, OneOf(A, B, C)

These are some of the simplest examples here to use as a baseline reference with more complex things discussed later, this set follows entirely from already agreed-upon basic rules. The specializations here could be seen as type-checker implementation details only if not for how we need to build on them to resolve more complex cases.

How to decouple Ordering from Overloads
Note: This is the how portion, not the why portion
given a set of overloads, the overloads can be decoupled from having an ordering concern by finding those that are disjoint and those which are specializations of each other, then adding type negations to prevent the less specialized ones from being used for more specialized cases. Note that we only add the negations for parameters, not for return types.
An example given using the subtyping relationship B <: A
(overload) (A, B) -> A
(overload) (B, A) -> B
(overload) (B) -> B
(overload) (A) -> A
(implementation, not checked) (T, U=None) -> T

In this case, (B) -> B is more specific than (A) -> A
We can remove the ordering between these two by negating the more specific from the less specific
(A & ~B) -> A & (B) -> B

in the 2-ary cases of (B, A) -> B and (A, B) -> A, we have
(B, A) -> B & (A & ~B, B) -> A

This makes the full set
(A & ~B) -> A
(B) -> B
(B, A) -> B
(A & ~B, B) -> A

but we can now freely re-order these, make other simplifications on them, and intersect them without an ordering concern as they have been made fully disjoint by specificity.
Now, lets explore the same, but without a subtype relationship
(overload) (A, B) -> A
(overload) (B, A) -> B
(overload) (B) -> B
(overload) (A) -> A
(implementation, not checked) (T, U=None) -> T

In this case, we need to ensure that  (A, B) -> A takes priority over (B, A) -> A and (B) ->B takes priority over (A) -> A without the ordering mattering. You may notice that earlier, the subtype relation gave us a naturally higher specificity, and this wasn't required.
This can be accomplished in a similar manner by assuming a total ordering between parameter lists of the same arity representing an equivalent relation
for the lower priority, we negate all higher priority types.
So we keep (A, B) -> A and (B) - > B untouched as the highest priority in each arity
this makes the full set
(A, B) -> A
(B & ~A, A & ~B) -> B
(B) -> B
(A & ~B) -> A

but these are now freely reorderable, and do not need to be treated with the care of an overload
For one more case of this showing how this resolves with more types to order:
(overload) (A) -> A
(overload) (B) -> B
(overload) (C) -> C

Absent a subtype relationship to order on most specific, we can allow this to become unordered while retaining the same priority with:
(A) -> A
(B & ~A) -> B
(C & ~(A | B)) -> C

The cases that simplify to (something) -> Never which need preservation are important because if we just removed them, we would reach incorrect results for (something) on an intersection with this. We can only completely remove anything which resolves as never when we are completely done intersecting.
An example of a slightly more interesting case for Types A,B,C and an unbound TypeVar T.
((A) -> A & (A) -> B) | (C) -> C) & (T) -> T
((A) -> Never) | (C) -> C) & (T) -> T
This is now (T) -> T, Where T is not A
Now it should be noted that (A) -> Never is compatible with (T) -> T,
so we can allow this case, but should we when it can never not error?
This one is more philosophical, and not born of necessity, necessity is the (de)ordering of overloads in parameters.
This is just one such example in which negation arises in the return type out of just logically possible things.

Once the ordering of overloads is removed, they can be treated as if they are just an intersection. This comes into play with correctly making other simplifications and selections, and with allowing the same logic to be applied to multiple parts of the problem. The ability for users to negate types also allows them to escape any ordering rules that exist currently. It also means type-checkers can have extremely efficient checking, but I've been convinced to save that part for an appendix.

I definitely see the usefulness of being able to write overloads with pairwise disjoint arguments, as they make overloads order-independent.
However, the necessity is still not clearly established. In https://github.com/CarliJoy/intersection_examples/issues/16#issuecomment-1662234504 it was alluded to that the interface A&B must be specified, which I take is meant in the sense that reveal_type((A&B).foo) would give a list of overloads that correctly describe the behavior of (A&B).foo instead of simply showing <overloads of A.foo> & <overloads of B.foo>.
This would be nice-to-have since it would aid users in implementing correct overloads.
However, the claim that without this spec inconsistencies between type-checkers could arise still seems unsubstantiated to me, since all the type-checker needs to do in order to do its job is:

When is C a subtype of A&B.
When is A&B a subtype of C.
What type A&B returns if it is a callable and called with arguments args.

Which doesn't require knowledge of a list of overloads that correctly implement A&B.

However, the necessity is still not clearly established.

Quoting myself:

Note: This is the how portion, not the why portion

There has been a lot to type up, and as much as I would like to get to all of this in a timely manner, life has had other plans.

However, the claim that without this spec inconsistencies between type-checkers could arise still seems unsubstantiated to me, since all the type-checker needs to know in order to do its job is:

You can decide if you want to take someone else at face value on this or wait until someone who has done the logic on this has time to explain it in full.

How to establish what type A&B returns if it is a Callable and called with arguments args.
(incl. case whenA,B,C are overloaded Callable)

None of which require knowledge of how to establish a list of overloads that correctly implements A&B.

Where do you think the return type comes from? From A, and from B. Now if we don't have this in the spec, when one type checker actually calculates the minimum bound, and another naively intersects, you have divergent behavior between type checkers. If A library uses a type checker that is more specific, and exposes the return type to a user expecting the more specific, and some of their users have a type checker that opted for the naive approach, this has diverging exposed types.
This needs specification, and Not is necessary for this specification to be consistent.
I will provide further details and examples as my time allows, including an efficient procedure for checking which follows this.

when one type checker actually calculates the minimum bound, and another naively intersects, you have divergent behavior between type checkers

So can we specify that the behaviour is to naively intersect? And does that still require Not?

when one type checker actually calculates the minimum bound, and another naively intersects, you have divergent behavior between type checkers

So can we specify that the behaviour is to naively intersect? And does that still require Not?

No. Absolutely not. Why in the world would we say a type checker must give less specific and less useful information?
The tools to do better are right here.
We have to have this in the spec, we should do it right or not at all.

And there's an even more basic answer: can you prove that the naive intersection is the minimum bound in all cases? you may find there are surprising edge cases if you don't have negation.

Two answers, one in terms of technical correctness and one of not forbidding type checkers to work with the type information they have because you find it inconvenient.

No. Absolutely not. Why in the world would we say a type checker must give less specific and less useful information?

Because it keeps the spec for intersections much simpler. And is unnecessary. It would maybe become necessary to have it if one wants to compile python code that has intersections (e.g. with mypyc), but that can be done in a separate PEP as it is not needed for static type checking.

Given the choice between type checkers being mandated to provide more accurate information or less, and a means to do both, more information.

Definition We say X is >a< greatest lower bound of both A and B if and only if

X ≤ A and X≤B (X is a lower bound)
If Z≤A and Z≤B then Z≤X (X is an upper bound for all common lower bounds of A and B)

Lemma: If >a< greatest lower bound to A and B exists, it is unique.

Proof: Assume X and Y are both greatest lower bound of A and B. Then by ②, X≤Y and Y≤X, ergo X=Y.

This isn't a correct expression of subtyping behavior
Even if it was, you have logical issues here.
An actual proof of this not being the case exists and is a lot longer, and something I've asked @mikeshardmind to help me put into the language the python community uses. I even mentioned the existence of the proof above.

It was asked that people who don't have more to say that wasn't already hashed out wait on all the details here

This is basically never going to get done, because people are badgering with the same points without actually waiting on more details that were promised above, and it is absolutely draining to have to deal with that.
If you're so impatient and rushing this to a worse outcome, I'll supply a quicker one of "Just don't add intersections, the community clearly doesn't care enough about the quality of them"
So with very little left to say constructive at this moment, I don't want to see another notification from this thread that repeats things people have said on the matter before without further elaboration that has value towards a correct specification. We can work on ensuring it is simple and easy to understand starting from correctness.
I'm at the point where it seems appealing to pick all of this up and independently work on a pep for intersections without all of the "proccess transparency" and just drop a full pep down when it's actually ready for review with proof of correctness.
As it is, I'm trying to actually detail step by step in digestible parts with examples at each chunk because this isn't actually simple, as much as people want it to be, while at the same time dealing with other obligations.

Okay, I think we may have a disagreement on terms that caused the "We can just reduce to the intersection or not quite"
The problem here is that while Never is a valid substitute for a runtime type, the other way around is not true. This makes detectable Never a more specific known where we can determine it as the only option.
I think if we resolve this, we can leave the proof aside for the formalization efforts because it really comes down to "is being more specific important"
To me the answer here is "While being more specific would be nice, it should be left to type checkers" However, the problem with this is that this needs is a cause for divergent behavior if we don't mandate one or the other.
So let's make the case for the extra complexity here:

There have been numerous issues with overloads in type checkers. This may give a path for users to disambiguate and assist with providing enough information to the type checker. Just checking mypy for what is accepted as a bug, not even user misuse, there are currently 52 open issues labeled as both bugs and pertaining to overloads, the oldest going back to 2017, with an additional 49 with the same labels closed.
Giving users the ability to be more expressive about their intent may actually simplify the job of type checkers.

If something can never work, and detectably so, and I still try to do it, I'd want a type checker to tell me.

Lets focus on this question for deciding what's next for this issue.

There have been numerous issues with overloads in type checkers. This may give a path for users to disambiguate and assist with providing enough information to the type checker

Sorry if this has been answered somewhere in this thread, but are you sure that negation needs to be in the intersection PEP? This last comment sounds a lot like the motivation for a new PEP?

I think it does personally, mostly because of this point:

To me the answer here is "While being more specific would be nice, it should be left to type checkers" However, the problem with this is that this needs is a cause for divergent behavior if we don't mandate one or the other.

Comment about incompatibilities (not the only one, just the first I remembered where it was) https://github.com/CarliJoy/intersection_examples/issues/16#issuecomment-1658763300
Talk at the typing summit (Pycon 2022) which mentions an issue of divergent behavior in type checkers with unions: https://youtu.be/BNTkWQfqP_c?t=8465
I don't want this PEP to create the next one of those laments about incompatibility between tools, as this hurts library authors disproportionately, which causes cascading effects in the ecosystem if it happens with a larger library. Imagine if numpy worked differently in mypy and pyright upon utilizing intersections.

-If we have to pick one of the two, the latter is more specific, and there are other benefits to it. If we stick to what is simpler, we lock off the ability for type checkers to do more. Since I view Static analysis as the tool that checks if what you are doing makes sense and catch errors before they happen, I want it to be able to use the info available.
-I think the two are rather closely related from the perspective of theory and use, and that we already have clear answers on what the behavior of not should be if it is added.
The overloads case in the way we are resolving here is simple conceptually but will take some computation. The good news it that it only needs to be done to callables at use and not prior. (Intersecting 3 classes with 30 methods each, but only using 1 method of the intersection, you're only checking that singular method to this detail)

This is still slightly incorrect. There are cases where Not is needed arising from intersections. I'll try and formulate a python example later.

There are also cases where the difference between adding & (T) -> Never is different to negating T
Not might need it's own PEP, but intersection should then depend on it.
Sequence[str] & ~str is not the same as List[str]
An api allowing a sequence of string-like objects but not treating a singular string as a sequence of strings has no way to do this except adding a runtime check for this.

Skipping the complex case with overloads, it is insufficient to only treat this as adding an intersection with (A&B) -> Never in the case of an incompatible set of parameters. If the use itself is incompatible then the use itself is an error, not something which will always raise (it may not raise if people have written code prior to needing to think about this, it might just behave incorrectly)

Putting all of this together, Adding intersections without Not would be negligent and shoving the problem onto other people for later, for predictable problems
Here's a hypothetical example based on the very real way software evolves over time.
There's a library that does sentiment analysis on written text.
somewhere in the bowels of it, it has a function that looks like this:
    def process_token_stream(self, token_stream: list[str]) ->Sentiment:

Later on, they stop using lists in some cases. On top of that, sometimes it isn't even a string, but something that behaves like a string, with extra information encoded. This function doesn't use the extra encoded information, but list can't even be used as a lying duck-type because of invariance. , they are interacting with extension code to speed things up, but it still acts like a list.
Someone realizes this means checking for unprocessed strings, and adds a check for it
    def process_token_stream(self, token_stream: Sequence[str]) ->Sentiment:
        if isinstance(token_stream, str):
            raise RuntimeError("raw strings should be tokenized before this function")

This later is a check in a hot path, and someone reasonably decides, "this is only used in our library by our users, lets remove the check and document this"
    def process_token_stream(self, token_stream: Sequence[str]) ->Sentiment:
        """
        Process a token stream

        token_stream: Sequence[str]
            note: it is an error to pass an a string here. tokenize it first with `Tokenizer`

Now intersections some along. process_token_stream and it's type information may escape the library's direct users without the corresponding warning.
It won't error to pass it a string, it just won't work as intended.
Even without considering the more nuanced issues with intersection, it would be negligent to not give library authors a way to handle this before intersections are added.
Who takes the blame when someone intersects this with something that should handle str and list[str], with the result being list[str]
The function author for caring about performance of their users, and (rightfully at the time) allowing documentation to suffice? If it's their fault, they now have a performance regression to fix this.
The intersection author? Well, we didn't give them the tools to express the correct intent.
The end user of the code the intersection author wrote? How?
The buck has to stop with us, the people arguing that Intersection should be added, when it comes to making sure that this won't cause harm. We need to ensure cases like this have a way to be resolved, and even without exploring overloads, we can find cases that need Not for that to happen.

Now intersections some along. process_token_stream and it's type information may escape the library's direct users without the corresponding warning.

Could you please clarify this?  I'm struggling to understand how an intersection type can cause escape in a way that was not already possible by re-exposing, aliasing, unioning, etc.

It won't error to pass it a string, it just won't work as intended.

But it'll still return a Sentiment? So it's correct from a types perspective?

The intersection author? Well, we didn't give them the tools to express the correct intent either.

I'm general there is lots of things that can't be expressed in the type system, including most of the actual behaviour of code, and currently some unsupported input that matches the type. I similarly can't express that an index is within the length of a Sequence (at least not yet). When an intermediary library author expose the intersection type, presumably as part of some exposed function, then they should absolutely be expected to document any limitations, in the same way the original authors documented those expectations.  Alternatively, they could handle the performance limiting assertion themselves (though I'd still expect that behaviour to be documented).
To be clear I'm not saying that a Not type wouldn't be useful in this context to better express the interface.  Having a Not type operator would be useful in general (although probably only really if there is an intersection operator too). Just that I don't think it's a prerequisite for an intersection type.  But again, I'm not convinced I've fully understood your example, so if you can elaborate on how the intersection exposes the function I may find it more convincing.

The str vs Sequence[str] is a well known issue (https://github.com/python/typing/issues/256), but can be somewhat alleviated by adding an overload of the form
@overload
def process_token_stream(self, token_stream: str) -> Never: ...
@overload
def process_token_stream(self, token_stream: Sequence[str]) -> Sentiment: ...

Which will raise unreachable warnings in code following a obj.process_token_stream("abc"). I recently opened an issue in mypy to get some better error messages in edge cases involving functions returning Never. (https://github.com/python/mypy/issues/15821#issuecomment-1666669403)
I agree with @diabolo-dan that I do not really see how this shows NOT to be a prerequisite for intersection.

Since I forgot to unsubscribe from the thread, you can have a singular additional reply.
@overload
def process_token_stream(self, token_stream: str) -> Never: ...
@overload
def process_token_stream(self, token_stream: Sequence[str]) -> Sentiment: ...

is not the same as
def process_token_stream(self, token_stream: Sequence[str] & ~str) -> Sentiment: ...

@randolf-scholz If you can't understand that the type not being handled as a parameter, and the type always erroring as a parameter are two distinct things and only appear compatible in a single direction, this might be too hard a problem for you to be working on.
The second example using type negation of the type it doesn't expect to need to handle would be compatible with a subclass that handled str. Attempt to use overloads instead to support that would not, and would also break compatible intersections.
You can't treat things that aren't equivalent as equivalent and think they suddenly solve all the same problems.
@diabolo-dan extrapolate from the difference between having the type marked as erroring when it doesn't and not marking the type as handled and you should find why it's a needed thing here. I lack any inclination to keep working on this.

To anyone still working on this PEP, (@randolf-scholz, @diabolo-dan, and anyone else?), should we close this then?

I've linked this discussion to python/typing so that it can be used in a future PEP.  Is there any objection to closing this?

Other than that the concerns raised here about this needing to exist prior to intersections, not the other way around not having been suitably addressed yet? No, but that's reason enough not to. Don't assume that just because people have stepped away as an active participant that the concerns they raised suddenly aren't concerns.

Don't assume that just because people have stepped away as an active participant that the concerns they raised suddenly aren't concerns.

I'm not trying to assume anything.  But how are we supposed to move forward with a proposal if people don't want to participate in discussion, but still want to insist that the proposal reflect their perspective?  How can this be resolved?

I'm not trying to assume anything. But how are we supposed to move forward with a proposal if people don't want to participate in discussion, but still want to insist that the proposal reflect their perspective? How can this be resolved?

The perspective has already been expressed, including concrete examples that show that overloads are not a substitute for intersections with negation, and why intersections without Not would be harmful. As long as those concerns are addressed rather than just sweeping them to the side as if they aren't important points that stand with or without someone actively arguing for them, the argument has already been shown.
You might also want to look at what caused several contributors to step back from active involvement.

You might also want to look at what caused several contributors to step back from active involvement.

Sorry, but I really don't understand this comment.  The people in this discussion have been extremely patient with an enormous amount of disrespectful language.
Never in my professional life have I been talked to this way:

I don't know why you are still arguing this @NeilGirdhar but type theory is pretty clear here and you keep asking for things you've already been provided by people being far too patient with you.

(And what makes this even worse was that in the end, the argument I made seems to be what we've accepted.)
And I don't know how @randolf-scholz feels, but I certainly wouldn't like to be talked to like this either:

If you can't understand that the type not being handled as a parameter, and the type always erroring as a parameter are two distinct things and only appear compatible in a single direction, this might be too hard a problem for you to be working on.

I know that's not you, but I think that someone does need to call this out.  There is nothing that kills open source participation like negativity and condescension.
I know you're also getting frustrated.  We all know what it feels like.  Maybe you're right, and people don't have the right background to make sense of your good arguments, or maybe you're not right, and you don't want to invest the energy to figure out why that is.  Both are perfectly fair, and both are good reasons to walk away.
But we all want to see intersections in Python.  And for that to happen, we need to make put a PEP together, which means deciding what goes into the PEP.  Do you still want to participate in making the PEP?  If so, let's try to figure out what goes in the PEP together.  If you're not participating, we can't have a discussion with your ghost.  Even if your arguments are right, we haven't yet found them convincing, and this issue is already consuming an enormous amount of energy that could be spent writing the PEP.
@CarliJoy @randolf-scholz what do you think we should do?

Sorry, but I really don't understand this comment. The people in this discussion have been extremely patient with an enormous amount of disrespectful language.

Sorry, I thought you had seen some of the frustration that had been expressed in the discord about non-productive discussion, and ignoring what others had said. I also commented on not thinking this was resulting in a productive process here

I know that's not you, but I think that someone does need to call this out. There is nothing that kills open source participation like negativity and condescension.

I know you're also getting frustrated. We all know what it feels like.

I'm not going to speak on behalf of anyone else, but it was observed in Discord that the person you quoted as being "disrespectful" had been ignored multiple times.
I don't think that excuses it, but it wasn't a one-off occurrence and I can see some of that occurring as a result of frustration.
I've personally gotten frustrated with the direction of some of the discussions where people seem more concerned about "well what if people lie to the type checker" than "What's the correct behavior". Frankly, what happens if someone lies to the type checker should be between them and their type checker, and the PEP shouldn't care except to try and minimize the number of cases where people feel that is necessary to do by providing good tools. People are often going to lie to the type-checker when they encounter shortcomings in typing; That shouldn't be our concern, instead, our concern should be creating a type system that as more and more is added to it, requires fewer lies as we gain the ability to express things more accurately about code.

Maybe you're right, and people don't have the right background to make sense of your good arguments, or maybe other people have good counterpoints, and they don't convince you, or you don't want to invest the energy to make sense of them. All of these are perfectly fair, and are understandable reasons to walk away.

I don't want to place blame on anyone for the frustration, if I had infinite time available, I'd not find this frustrating in the same manner, but there's a large time commitment in explaining things and reconciling differences when we are all starting from a different set of perspectives and experience, and I don't think there's been constructive discussion on certain topics. It can feel like a no-win situation when there's no progress on something like this because it's "spend time that could be spent on other tasks" vs "risk a feature getting accepted in a way that compromises its long-term usefulness"

But we all want to see intersections in Python. And for that to happen, we need to make put a PEP together, which means deciding what goes into the PEP. Do you still want to participate in making the PEP? If so, let's try to figure out what goes in the PEP together. If you're not participating, we can't have a discussion with your ghost. Even if your arguments are right, we haven't yet found them convincing, and this issue is already consuming an enormous amount of energy that could be spent writing the PEP.

If the choice right now was to have intersections without type negation or to not have intersections, I'd actually favor waiting to have intersections. In my mind, it's better not to add features that aren't properly explored for all of their potential issues and interactions with existing and predictable future cases. I wholeheartedly believe that without type negation, intersections involving anything other than protocols will be problematic long-term. Maybe that's a better argument for saying we should start by supporting protocols only, then look at what is left for a follow-up, maybe it's an argument not to add intersections, or maybe we can figure out how to best argue the benefits of type negation and why this prevents issues long-term; Which of the three it is probably requires more discussion.

Do you still want to participate in making the PEP?

If we can find a way to have a constructive discussion that doesn't circle the same arguments without anything new being added and to keep the discussion in-bounds for what is useful to the PEP (that is, defining the specification, not the type-checker behavior) then yes. However, it really is conditional on that for me, and I'm disappointed to even feel the need to say this.

I don't think that excuses it, but it wasn't a one-off occurrence and I can see some of that occurring as a result of frustration.

No.  There is no excuse for it, and it's inappropriate for you to make allowances for the disrespect of others.
(And you have the chronology backwards anyhow.  The disrespect came first.)

I don't think that excuses it, but it wasn't a one-off occurrence and I can see some of that occurring as a result of frustration.

No. There is no excuse for it, and I think it's inappropriate for you to make allowances for the disrespect of others.
(And you have the chronology backwards anyhow. The disrespect came first.)

Considering I did call it out in the referenced thread in an immediate response, you're barking up the wrong tree here. I'm not making allowances for anyone here, only noting the observed points of frustration. You're right, that did happen first, but this point of clarification isn't even productive as the person you quoted walked out of all discussion on this, and left the discord server for it as well.
Now can we find a way to get to the part where we have a productive discussion on a technical level about the PEP?

There is a new PEP in draft about TypeGuards.
It has a paragraph related to Intersections

The definition of A & R and A & ~R would be left up to  the type checker itself as these concepts don't yet exist in Python typing.

I guess we really shouldn't hand in a PEP without not.

I guess we really shouldn't hand in a PEP without Not.

I'm only interpreting this as "doesn't exist yet" not "cannot exist", so any PEP that included Not could just work within existing definitions or clarify them more clearly, but I think that the need for Not goes away in 2 possible outcomes of the discussion we needed to export to python/typing with regard to Never, so in either of those outcomes, I would split off the work here for any future Not PEP and possibly just have a note in the intersections pep on why Intersections was deemed to not need it (conversely if we determine we do need it, we'll have language summarizing what we came to for that too)

preemptively closing this, other decision-making outcomes that weren't necessarily clear before appear to remove the need and only make this something that would be nice and can be done separately, will reopen only if we discover this isn't actually the case anymore.

This needs reopening, the reasons I thought we didn't need Not only applied to a specific formulation and from the recently accepted pep 742:

Formally, type NP should be narrowed to A∧R, the intersection of A and R, and type NN should be narrowed to A∧¬R, the intersection of A and the complement of R. In practice, the theoretic types for strict type guards cannot be expressed precisely in the Python type system. Type checkers should fall back on practical approximations of these types. As a rule of thumb, a type checker should use the same type narrowing logic – and get results that are consistent with – its handling of isinstance(). This guidance allows for changes and improvements if the type system is extended in the future.

Type checkers already need to be able to handle this now, so this is not a new burden.

I didn't get into all of the details when re-opening this, but we need Not to handle Callables. It may also help with other aspects of intersections with Any, especially with constructors, which was a major concern that had been attempted to be handled with some special cased ordering before.
https://elixir-lang.org/blog/2023/09/20/strong-arrows-gradual-typing/ covers the theory here, as well as another gradually typed language with an Any-like construct.
When evaluating an intersection of callable, the callable only are comparable where they have an overlapping input domain. For instance,
The callables:
(int) -> int
(str) ->str
Have entirely disjoint domains, the resulting intersection would end up equivalent to
((int & ~str) -> int) & ((str & ~int) -> str) &  ((int & str) -> int & str)
Note that the return types are only intersected when the input types are, and while we can say "int and str can't ever intersect", the same isn't true of all types like this.
However, this also gives us an answer for gradual callables and Any, which strikes a much better balance between some of our original options without compromising on being theoretically sound.
Considering this intersection:
((int) -> int)  & Callable[..., Any]
We get:
((int) -> int & Any) & ((Any & ~int) -> Any) & ((...) -> Any)
where ... is a stand-in for an unknown number of potential args or kwargs (gradual callable)
This is possible to retain because any callable consistent with ((int) -> int), even if it can also handle other things, must return something that is at least as int when given just an int.
However, for users to be able to denote function domains, and for type checkers to be able to display meaningful errors, diagnostic messages, or informational messages (reveal_type) relating to function domains, as well as for this to be testable (assert_type), these domains must be denotable.
There are multiple ways to express the function domains, I believe negation to be the appropriate one for python at the current point in time.
The one major theoretical issue we can run into is:
A negated gradual type must either be treated as equivalent to Never or disallowed.
Any other interpretations would violate either consistency or the principle of gradual typing that a gradual type should not result in false positive errors when standing in for another valid type.
There is no fully consistent interpretation for a negated gradual type (See this issue comment for details), but the only places where a negated gradual type should ever show up would be synthetic in the internals of type checkers (users won't write ~Any, and if they do, Never is a reasonable interpretation in a vacuum) Type checkers warning on an annotation that is a negation of a gradual type directly in source form, and not as a result of type variables is a reasonable opinionated option available to type checkers here.
~Never should likely also be warned against if ever written directly, this has two possible interpretations (Any and object). I believe the correct option for python here is object, and the consequences of a double negation effectively casting Any to object to be acceptable, but this would need further exploration to be sure of.
@mark-todd I believe this may help address your concerns with the few things you thought that retaining ordering would help with, but I believe we still probably have more to work through on reaching a consensus toward something more closely following pure subtyping rules as @superbobry and @DiscordLiz both expressed a preference for.

To check that I'm understanding the point you are getting at here, suppose I have input parameters P1 P2 and returns R1 R2, am I right that this generalizes so that
Callable[P1, R1] & Callable[P2, R2]

can be expanded out to:
Callable[P1 & P2, R1 & R2] & Callable[P1 & ~P2 , R1] & Callable[P2 & ~P1, R2]

And in the Case where P2 is a more specific subtype of P1 (This language avoids that Any has bidirectional subtyping behavior currently and we don't have a consistent subtyping rule that applies to Any, I think we need to address that elephant in the room for these rules, and that specificity should be the criteria used), we can simplify further to
Callable[P2, R1 & R2] & Callable[P1 & ~P2 , R1]

Because P2 & P1 is trivially P2, and P2 & ~P1 is trivially empty?
If this all correct, can it be extended to handle constructors by treating a type T as having a constructor (P) -> T that is consistent with the type checker's understanding of the successive underlying functions that would be chained (type call, type new, and init), or does this still not work because of the allowed LSP violations for these? I don't fully understand the current rules for what type checkers actually check for constructors.

To check that I'm understanding the point you are getting at here, suppose I have input parameters P1 P2 and returns R1 R2, am I right that this generalizes so that [...]

The examples there all look fine to me, yeah. That's projecting in -> out types based on function domain.

This language avoids that Any has bidirectional subtyping behavior currently and we don't have a consistent subtyping rule that applies to Any, I think we need to address that elephant in the room for these rules, and that specificity should be the criteria used

Yeah... Any, Callable[..., Any], and tuple (bare or tuple[Any, ...]) need this language. Would be easier if we were doing this in a post-consistent subtyping world, but that sidesteps that and leaves that possible to further clear up with consistent subtyping.

can it be extended to handle constructors by treating a type T as having a constructor (P) -> T that is consistent with the type checker's understanding of the successive underlying functions that would be chained (type call, type new, and init), or does this still not work because of the allowed LSP violations for these?

Yes, but only because of a recent clarification to constructors that was made. This should also help clear up the current rules there.

above interacts with TypeIs as well, see https://github.com/microsoft/pyright/issues/8109

Copying some stuff that was discussed elsewhere that will need resolution of some kind.
class Sequence(Protocol[T]):
    ...

class SequenceMutationMethods(Protocol):
    ...

type MutableSequence[T] = Sequence[T] & SequenceMutationMethods
type NeverMutableSequence[T] = Sequence[T] & ~SequenceMutationMethods

Would create a problem where negated types can't ever be guaranteed unless the requirement for type checkers become total consistent use evaluating entire visible code graph
The short version below, erasing type information by assigning to a less specific structural type avoids the negated methods from being checked. Indirection through function calls possible to avoid a type checker only checking local use.
x: list = []
y: Sequence = x
z: NeverMutableSequence = y
x.append(1)

My understanding of negation types was that they required positive proof to correctly use, so assigning a Sequence to a NeverMutableSequence would result in a type error (absent other locally available proofs, such as an assertion or branch on SequenceMutationMethods).

My understanding of negation types was that they required positive proof to correctly use, so assigning a Sequence to a NeverMutableSequence would result in a type error (absent other locally available proofs, such as an assertion or branch on SequenceMutationMethods).

Requiring positive proof for something that should be possible to analyze statically is rather ugly, and results in worse code being written.
I'd rather find a better way here if possible. The obvious one that removes this requirement is that type checkers should check that the total use is consistent across the entire visible code graph.

I'm gonna open a discourse thread about typeis's negation, maybe we can put the issue to rest by just fixing negation where it exists.

I dont know how we're supposed to continue. many people in that discussion were fine with the idea of TypeIs remaining unsound and also keeping the definitions that it has. Those definitions say that TypeIs should behave like theoretic Not and Intersections would, but the unsound nature of TypeIs not having limitations based on theory and people not wanting to change that puts this in a bind.

The way forward is clear, it just precludes the use cases you and I had and discussed. The broader community doesn't care about soundness, so:

Ignore the dependence on Not being denotable
Anywhere Not would be needed for users just isn't denoatable.
An intersection containing a gradual type must be at least as gradual as the gradual type unless another operand removes or otherwise narrows capability.
An intersection containing an intersection of generics must not have conflicting variance.

number 4 above involves a related type soundness hole that isn't actually fixable without breaking more users than seems like anyone is going to go for, so if can be ignored as a constraint too.
I'd suggest closing this unless someone else decides to actually fix the issue with TypeIs and the impacts the definitions it was accepted with have here.


[Feature] Allow fine tuning T5 models?

🚀 Feature
I'd like to experiment with fine tuning small T5 based models, but it looks like there are some assumptions made in the code about the type of model and configuration class required. I haven't investigated yet, so I don't know how big of an ask this is.
Motivation
I'm training one-off models for specific tasks rather than instruction or chat-tuned models, and think the T5 family of models would be faster and more appropriate. But I like the boilerplate and config in LLM studio.

Heyy @psinger I would like to take a crack at this !

Hi @psinger , I figured out that we can add models by adding model preset here :
https://github.com/h2oai/h2o-llmstudio/blob/be2f2b155dd79ab6fd1f9fd85c5b410c58b33475/llm_studio/python_configs/text_causal_language_modeling_config.py#L492-L499
and then LLM studio loads the backbone with AutoModel.from_pretrained(<preset-name>). Hence

I added t5-small in this ConfigProblemBase class and ran llm studio.
I created an experiment with default dataset and ran experiment
following error was thrown

 Error
2023-07-04 22:41:46,555 - INFO: Number of observations in validation dataset: 83
Downloading (…)lve/main/config.json: 100%|███████████████████████████████| 1.21k/1.21k [00:00<00:00, 4.44MB/s]
2023-07-04 22:41:47,043 - INFO: Using float16 for backbone
2023-07-04 22:41:47,086 - ERROR: Exception occurred during H2O LLM Studio run:
Traceback (most recent call last):
  File "/home/shivance/Desktop/h2o-llmstudio/train_wave.py", line 106, in
    run(cfg=cfg)
  File "/home/shivance/Desktop/h2o-llmstudio/train.py", line 594, in run
    model = cfg.architecture.model_class(cfg)
  File "/home/shivance/Desktop/h2o-llmstudio/llm_studio/src/models/text_causal_language_modeling_model.py", line 141, in __init__
    self.backbone, self.backbone_config = create_nlp_backbone(
  File "/home/shivance/Desktop/h2o-llmstudio/llm_studio/src/utils/modeling_utils.py", line 526, in create_nlp_backbone
    backbone = model_class.from_pretrained(
  File "/home/shivance/anaconda3/envs/h2o/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 487, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class  for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, CodeGenConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, ElectraConfig, ErnieConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, LlamaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MvpConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, TransfoXLConfig, TrOCRConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig.

My thoughts, the models we are using as of now are AutoModelForCausalLM base class while T5 doesn't implement that, the class it has is T5ForConditionalGeneration .
So how should I tell llmstudio that for t5 model use this class to extract backbone !

Thanks @Taytay -
we will need to introduce an option for https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForSeq2SeqLM
Also need to check how the rest of the API for it matches the current implementations.

Oh just saw, that's what you mentioned above

Thanks @psinger!


Add support for STRING_MAP attributes.

This change is

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@043bc41). Click here to learn what that means.
The diff coverage is 95.91%.

@@            Coverage Diff            @@
##             master     #271   +/-   ##
=========================================
  Coverage          ?   84.07%
=========================================
  Files             ?       49
  Lines             ?     2826
  Branches          ?        0
=========================================
  Hits              ?     2376
  Misses            ?      427
  Partials          ?       23

Impacted Files
Coverage Δ

cmd/client/main.go
0% <ø> (ø)

pkg/attribute/bag.go
100% <100%> (ø)

pkg/attribute/rootBag.go
100% <100%> (ø)

pkg/attribute/mutableBag.go
100% <100%> (ø)

cmd/client/util.go
86.42% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 043bc41...3bb0e42. Read the comment docs.

@qiwzhang This PR adds support to send attribute of type StringMap map<string,string> which we expect request.headers / response.headers to be.
The mixer client code will need to support this data type.

@mandarjog I will change the client to use it.

PTAL.


TinyJit of TinyJit gives AssertionError: didn't JIT anything!

example
from tinygrad import Tensor, TinyJit

@TinyJit
def f(t): return t * 3

@TinyJit
def g(t): return f(t) + 5

for i in range(5):
  print(g(Tensor([i])).item())

  File "/Users/chenyu/code/tinygrad/tinygrad/engine/jit.py", line 162, in __call__
    assert len(self.jit_cache), "didn't JIT anything!"
AssertionError: didn't JIT anything!

I think this can work if we ignore any inner jit with a context var, or at least it needs a more comprehensible error message

asserting is good enough now


Block update cluster's attributes

As part of the terraform plan command, we would like to block updates for several attributes.
The subsystem tests couldn't catch planModifier of attributes so this mechanism cannot be tested by UT or subsystem tests

@nirarg please re-review


Support the new 0.17 syntax for declaring a module

module Queue (..) where

is now written as
module Queue exposing (..)

in Elm 0.17.  This PR supports the new syntax (whilst keeping support for the old syntax too).

Thanks a lot! :-)

Any chance of publishing this to the extensions marketplace?


Docker login & pull extremely slow on WSL2

Version
Microsoft Windows [Version 10.0.22000.778]
WSL Version

[X] WSL 2
[ ] WSL 1

Kernel Version
5.10.16
Distro Version
Ubuntu 20.04.4 LTS
Other Software
Docker Desktop 4.11.1 (84025)
Repro Steps
I recently moved my development environment from an old desktop to a new Dell Inspiron 14 7420 2-in-1 laptop. I've been using WSL2 & I copied it over using the export/import WSL tools for that matter. Everything went fine there, but then after installing Docker Desktop on the new machine, I noticed that certain commands (e.g. docker login & docker pull) take much, much longer when run from Linux side compared to running them from Windows side.
This can't be a matter of volume mounts, since these commands don't involve any volumes anyway. I haven't even tried a docker build yet to be honest, as those other 2 commands are more important to me.
Also, I've allocated enough resources to the WSL2 VM via the .wslconfig file, so that shouldn't be a problem either:
[wsl2]
memory=8GB
swap=0
localhostForwarding=true
processors=4

Expected Behavior
docker login & docker pull execution times on WSL2 to be similar to the ones from Windows side.
Actual Behavior
For example, here's a comparison for the docker login command:

Powershell: 4.117 s
Linux: 13 m 40.795 s

Diagnostic Logs
strace log for docker login: https://gist.github.com/ionel-sirbu-crunch/7c3a11caee41c4241ed067e88d5cd1a8
WslLogs-2022-08-17_12-08-55.zip

Am I missing something in reporting this ticket? It says 1 of 2 tasks at the top...

I have the same problem. Is there anything I can do?

docker login: 1 min 55 secs. 1 min 15 secs to see the message "Login succeded", 40 secs to get the prompt back
docker pull alpine : 40 secs

I can not use docker login command because I have no docker registry setup. But here is the time from docker pull alpine.
$ time docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
213ec9aee27d: Pull complete
Digest: sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest

real    0m8.013s
user    0m0.048s
sys     0m0.000s

8 seconds, I guess. I am not use Docker Desktop. Just a simple docker installation in Ubuntu 22.04 in WSL2 in Windows 10.

I started to look into this problem and noticed my firewall denies UDP requests from the dns server. I simply uninstalled the crap security software I got with my new computer and now everything works as it should! 2 seconds to pull the alpine image.

Well, that can't be it for my case though, as I already have both Inbound & Outbound rules allowing traffic into the WSL2 network interface.

On the same laptop

in a virtual machine, docker login is very fast (less than 1 second ?).
on WSL2 :

➜  time docker login nexus.xxxxx.com:4432 -u=xxxxxxxx -p=xxxxxxxxx
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
docker login nexus.xxxxx.com:4432 -u=xxxxxxxx -p=xxxxxxxxx 0.08s user 0.05s system 0% cpu 21.810 total

Docker logout is very slow too.
It take ~15sec until the line disapear from $HOME/.dicker/config.json

➜  time docker logout nexus.xxxxx.com:4431
Removing login credentials for nexus.xxxxx.com:4431
docker logout nexus.xxxxx.com:4431  0.36s user 0.18s system 0% cpu 1:24.87 total

In wsl removing "credsStore": "desktop.exe" from ~/.docker/config.json temporarily fixed the slow docker commands for me.
But the line gets re-added after a Docker Desktop restart
~/.docker/config.json:
{
  "auths": {},
  "credsStore": "desktop.exe"
}

Credits: https://github.com/docker/for-win/issues/10247#issuecomment-1126350051 @emiller-tunein
Manually installing docker-credential-wincred.exe seems to be a better workaround:
https://github.com/docker/for-win/issues/10247#issuecomment-1152865542

In wsl removing "credsStore": "desktop.exe" from ~/.docker/config.json temporarily fixed the slow docker commands for me.
But the line gets re-added after a Docker Desktop restart
~/.docker/config.json:
{
  "auths": {},
  "credsStore": "desktop.exe"
}

Credits: docker/for-win#10247 (comment) @emiller-tunein
Manually installing docker-credential-wincred.exe seems to be a better workaround: docker/for-win#10247 (comment)

Thanks for the recommendation @cimabe ! I tried the docker-credential-wincred.exe workaround, but unfortunately it didn't change anything for me, it still takes ages for a simple docker login. 😭

Is there any team moderating these issues? Shall I leave this one open, in case anyone wants to have a look at the logs, or shall I close it, now that I sorted it out with the recommended workaround?

This has been fixed for me in the new Docker Desktop 4.16.0 release
https://docs.docker.com/desktop/release-notes/#4160:

For Windows

Fixed docker build hanging while printing “load metadata for”. Fixes docker/for-win#10247.


Consider putting struct enum variants on one line

-    Switch { adt_def: AdtDef<'tcx> },
+    Switch {
+        adt_def: AdtDef<'tcx>,
+    },

     // test for equality
-    Eq { value: Literal<'tcx>, ty: Ty<'tcx> },
+    Eq {
+        value: Literal<'tcx>,
+        ty: Ty<'tcx>,
+    },

     // test whether the value falls within an inclusive range
-    Range { lo: Literal<'tcx>, hi: Literal<'tcx>, ty: Ty<'tcx> },
+    Range {
+        lo: Literal<'tcx>,
+        hi: Literal<'tcx>,
+        ty: Ty<'tcx>,
+    },

I'm not sure if the original is better or not, tbh, We should have an option for this at the least.

Should be quite easy when #407 is merged.

Is this issue specifically about struct variants, or does it also cover non-enum structs? Since the two are semantically close, and use the same syntax for literals, I would guess the policy should be the same.

Specifically variants. Although the two are the same in some sense, variants tend to have fewer fields with shorter names, so it's often nice to format them more tightly than struct items.


feat(wallets): introduce new feature

#3404  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @gregfromstl and the rest of your teammates on  Graphite


[Enhance] Add support for sequence input in --search-args option

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
mim gridsearch mmcls $CONFIG --work-dir $WORK_DIR --gpus $GPUS \
    --search-args --train_dataloader.dataset.pipeline.0.key val1 val2 ...

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/mim/commands/gridsearch.py", line 288, in gridsearch
    arg_value = get_config(cfg, arg)
  File "/opt/conda/lib/python3.7/site-packages/mim/utils/utils.py", line 515, in get_config
    assert item in cfg, f'attribute {item} not cfg{suffix}'
AssertionError: attribute 1 not cfg.train_dataloader.dataset.pipeline

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/mim", line 8, in <module>
    sys.exit(cli())
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/mim/commands/gridsearch.py", line 146, in cli
    other_args=other_args)
  File "/opt/conda/lib/python3.7/site-packages/mim/commands/gridsearch.py", line 298, in gridsearch
    raise AssertionError(highlighted_error(msg))
AssertionError: Arg train_dataloader.dataset.pipeline.0.key not in the config file.

Modification
I added logic to handle the case when cfg is instance of Sequence in get_config and set_config functions
BC-breaking (Optional)
Does the modification introduce changes that break the back-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

Hi @KKIEEK, thanks for you contribution!
This PR looks great to me, and the CI need to be fixed:

@ice-tong I fixed it.

Please @zhouzaida have a look~


maxParallel stalls when items are processed out of order

Say I would like to process some SQL tables two at a time.  Each table is mapped into() a stream, and each of those streams will take variable time to complete.  If table B ends before table A, then table C wont start processing until A also completes.  Here's some code!
// A and B immediately start streaming.
// If B finishes streaming first, I would expect C to start streaming.
// Instead C doesn't start streaming until A also completes.

await DataStream.from(['A', 'B', 'C'])
    .setOptions({ maxParallel: 2 })
    .into(
        async (out, tableName) => await out.pull(streamTable(tableName)),
        new DataStream()
    )
    .run();

Hi @devinivy,
I moved this issue here. I'm making a number of small changes for a new minor release (that I promised won't happen before v5 ;) ).
So let's make it clear what we need here:

Streams A and B and C are piped one after another to a single output, but until A or B is drained we don't want to start pulling (for instance: requesting an url) C.
Output order has no value at all and items are streamed from A and B in parallel.
If B runs dry before A, C starts flowing.

The output could be:
a1
b1
b2
b3
a2
b4 <--- B end
a3
c1
c2
a4 <--- A end
d1
c3
...

Is that your expected outcome?

That sounds and looks right to me!

Ok. This should be achievable using the MultiStream class, but let me think about this a little, as it may be sensible to add a new feature here.

Ok. I need to polish my code a little before I post it, but what you want to do can be achieved now with the following code (pseudo):
stream.use(s => {
  const out = new MultiStream();
  s.each(item => {
    const str = mkStreamFrom(item);
    out.add(str);
    return str.whenEnd(); // this keeps the backpressure
  });
  return out.mux();
})

Thank you for putting in the thought and effort!  I will need to process that a little but, but I think I see the core of the idea.  Do you have any insights as to where to maxParallel constant would come into play?  Is it an option set on stream (per your example)?

Exactly there. :)

@devinivy did you perhaps have any success in running this? I'm going to test this as a principle and put some name to such a method.

Hey, I tried this out and didn't get the results I would have expected.  It seems to behave identically, except the stream may end before all items (in my original example, "tables") are processed.  Here's a script I'm using to test this out, visualizing stream progress using progress bars.  You'll see at the top of the file a constant to switch maxParallel behavior based upon your snippet.
https://github.com/devinivy/scramjet-experiments/blob/master/max-parallel.js
I can imagine this being a common "issue" when muxing, since it's hard to know when one is done adding streams to a MultiStream, and if the sub-streams complete before another one is added, then the muxed stream may end "prematurely."  I put this all in quotes because I acknowledge it's the intended behavior!  Even so, I didn't see the parallel behavior that I expected.  I will let you know if I come-up with a solution!

If you'd like we can create an issue to document and/or alter this behavior of promiseRead(), but I think the original issue resolved—thanks! 👍

I'll get on this anyway. Version 5 is round the corner and brings some major changes. We'll still support the v4 interface (via adapter) and will add some documentation on this too.


Where did Lex.createAcmeResponder go in 2.x?

Hi,
In letsencrypt-express 1.x, I used the function Lex.createAcmeResponder. What's the equivalent in letsencrypt-express 2.x?

lex.createAcmeResponder() is now lex.middleware(require('redirect-https')()) or lex.middleware(app)

Awesome, thanks for your help.


Minimal example of running an intel realsense depth sensor live

What
Applies the stored intrinsics and extrinsics and logs the depth and rgb frames.
To run:
python examples/python/live_depth_sensor/main.py

Resolves:

https://github.com/rerun-io/rerun/issues/2379

Checklist

[ ] I have read and agree to Contributor Guide and the Code of Conduct
[ ] I've included a screenshot or gif (if applicable)

PR Build Summary: https://build.rerun.io/pr/2541

Docs preview: https://rerun.io/preview/afb479d/docs
Examples preview: https://rerun.io/preview/afb479d/examples

I don't think this will auto-close the related issue:

You need to write the magic string Closes #2379 🙄


Dropping PHP 5.4 support

@staabm, do you have any issues with dropping PHP 5.4 for a 1.3 release?

Actually I have :-(. Can we postpone upgrading php to the next release?

Alright, next release it is!

Its the price we have to pay for our lts boxes :-/


click menu button, didn't  show up sidebar panel.

comfyui & n-sider is the lastest version.

after click

edge is ok, chrome is broken.

Really weird, can you try to go in Application->LocalStorage->ComfyUI Address and delete the variable pinnedItems?
Also clear the cookies and then retry!

It works!
I clearned up Local Storage, session Storage and cookies.

chrome is my main browser, edge is backup, so they are less data in local storage.


Missing data-t-name ends in not helpful error log

I'm submitting a ...

[ ] Regression (behavior that used to work and stopped working in a new release)
[ ] Bug report
[x] Feature request
[ ] Documentation issue or request

Current behavior
If you miss to add the data-t-name attribute you got an error like this one:
TypeError: Cannot read property 'replace' of null
Expected behavior
It would be great to show an more helpful error.
For Example:
No "data-t-name" found
Please tell us about your environment

TerrificBridge version: 1.2.10

Browser:
- [x] Chrome (desktop) version 63

closed due to inactivity


Customizing Notebooks for 3.10

We changed our notebooks to match the period's agreed-upon theme. I also added borders (as per Xavier's request), and customized them to match the author of the page. For instance, a green border indicates the first author (but I kept it purple for my personal page since there was only one author). This allows our audience and graders to better understand who worked on what and who is sharing which part of the lesson.
This is the updated repo after the bad pull request.

Highlights look good.  But ### markdown is not working in how this is designed.   You would probably need to change to  and  tags with this style.


[request]: Pre-sign DeleteObject URLs

A note for the community

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue, please leave a comment

Tell us about your request
I want to be able to pre-signed a URL for a DeleteObject command.
Tell us about the problem you're trying to solve.
Allowing users without access keys to delete certain objects from my buckets with pre-signed URLs
Are you currently working around this issue?
Delete operations are routed through my backend.
Additional context
No response

we're working on it thanks for reporting!

fix was merged, you can now use this in the next branch:
[dependencies]
aws-config = { git = "https://github.com/awslabs/aws-sdk-rust", branch = "next" }
aws-sdk-s3 = { git = "https://github.com/awslabs/aws-sdk-rust", branch = "next" }

This was released with v0.9.0.


Updating PC via link does not work

See this example:
instr jump_to_bootloader_input X = bootloader_inputs.access X, pc' ->;

This does not seem to un-constraint the PC in the next row. So Witgen fails in this example.

Hopefully closed by #1221.

Yes, it does, thanks!


added options to be passed on to Html->css

Hi,
like discussed in #8, CakePdf needs full urls. So I added a parameter options_for_css
to be passed to Html->css.
I'm not this familiar with php, so I can't provide any tests.
regards
dieter

Coverage remained the same at 69.84% when pulling 769e519127d092dd44966664994fc384e2dfd90d on dspaeth-faber:master into 1259754b18968f75cdec6111485ee8454aa4e0e6 on elboletaire:master.

Ok I'll take a look whenever I can

I was gonna merge this but after making the required minor changes I've remembered that you can pass a 'tag' => false option to the helper to return the css route and make your own css tag:
$less = $this->Less->less('less/styles.less', ['tag' => false]);
echo $this->Html->css($less, ['fullBase' => true]);

This was one of the reasons why I made this and that's why I close this PR.
Sorry for not remembering until now.


As an instructor I can add additional instructors to a course so that multiple users can have instructor access to a course

User Story (Epic)

As an instructor
I can add additional instructors to a course
So that multiple users can have instructor access to a course

Fine grained user stories

As a developer, there is a react component I can use to list multiple instructors for a course so that I have an easy way to display multiple instructors for a course, seeing name and email address.
As a user, on any course where I have access to the course page, I can see a list of the instructors for that course right under the name of the course, so that I know who the instructors are for the course.
As a developer, there is a method

Acceptance Criteria

[ ] There should be a way to list all instructors for a course (this could be factored out into it's own story, and tested by manually adding additional instructors via the rails console (rails c) for testing purposes.
[ ] All instructors (including the one that created the course) should have full instructor permissions on the course; the "first instructor" (the one that created it) should not be a special case.  (Look for code that does a .first when looking up the instructor for a course, or methods that return "the" instructor for a course, and change them so that they return a list of instructors.   Make sure consumers of those methods process the return value correctly.)
[ ] There should be method on the course model object to add an instructor.  It will be up to the implementor to decide whether it makes more sense to take email address or user id as a parameter; probably user id, since detailed error handling (e.g. if the email address doesn't exist) should perhaps be handled in controller logic?   But that's up to the implementor; the only thing for sure is that error handling needs to be considered.
[ ] There should be a way in the UI for an instructor (on the course) or any admin to add an additional instructors to a course by entering the users email address.  Incorrect email addresses need to be handled gracefully.
[ ] In the first cut, it is ok if there is not a way to remove an instructor; that can be a separate story.

...
Implementation Todos
Back end:

[ ] Add a method (or refactor an existing method) on the course model object to list all instructors of a course (e.g. @course.instructors)
[ ] Add an api method to list all instructors of a course (e.g. by returning the value of @course.instructors as JSON.)
[ ] Create an api method to add an instructor to a course.  It should be available only to admins, and current instructors of the course.   It should consume the email address of a user, look up that user in the users database, and add that user as an instructor of the course.   Note that the user may or may not be an "instructor" in the sense that they have the capability to create other courses; we need to review how the "roles" work in this instance.

Front end:

[ ] Add a react component that lists all instructors of a course; it should just take the list of instructors as a parameter (and for the time being, have no interaction with the backend api.  This could be factored out into it's own story, with jest tests and a storybook entry.)
[ ] Incorporate that react component
...

Testing:

[ ] Jest coverage for all new react components added
[ ] if possible, minitest Rails coverage for all rails methods touched

Another implementation suggestion:
The first step might be to go to app/views/courses/_show_admin.html.erb and add a React Component at the top of the page called CourseInfo.  That component could at first just take a @course object, and perhaps also a list of the instructors, and it would render a table showing for each instructor, just their name and their email address.
Later on, all the other stuff that's in the header of the course page could be moved into the React component as well.

As a remidner, when you create a new react component that you are going to insert into an .erb file with react_component, you have to register it inside this file:
app/javascript/packs/react-bundle.js


Switching the 'Content-Disposition' to 'inline' cause a 403 error with Amazon.

Hi Nathan,
When left to it default setting in the S3UploadsController (line 30):
{'Content-Disposition': 'attachment'},
...uploads happen without a problem.  If I switch to:
{'Content-Disposition': 'inline'},
...or remove the line completely in the hopes that by default my upload will be 'inline', Amazon returns with following error:
Amazon S3 returned an error: HTTP status 403.
Do I need to change something else in the controller?
Thanks,
Abel.

Has anyone come up with fork with this feature. I agree with mixonic. It would be perfect if it weren't for having to have  {'Content-Disposition': 'attachment'}


RBEL failed: The given URL is invalid.

Since upgrading tiger-standalone-proxy  from version 0.15.0 to 0.18.1 I've been getting the following errors
ERROR 1 --- [ver-Scheduler13] d.g.t.tiger.proxy.ForwardProxyCallback   : RBel FAILED!

de.gematik.rbellogger.exceptions.RbelConversionException: The given URL is invalid. Please check your configuration.
	at de.gematik.rbellogger.data.RbelHostname.checkIfUrlIsValid(RbelHostname.java:104) ~[rbellogger-0.22.0.jar!/:na]
	at de.gematik.rbellogger.data.RbelHostname.generateFromUrl(RbelHostname.java:83) ~[rbellogger-0.22.0.jar!/:na]
	at de.gematik.test.tiger.proxy.MockServerToRbelConverter.convertUri(MockServerToRbelConverter.java:69) ~[tiger-proxy-0.18.1.jar!/:na]
	at de.gematik.test.tiger.proxy.MockServerToRbelConverter.convertRequest(MockServerToRbelConverter.java:53) ~[tiger-proxy-0.18.1.jar!/:na]
	at de.gematik.test.tiger.proxy.ForwardProxyCallback.handleResponse(ForwardProxyCallback.java:56) ~[tiger-proxy-0.18.1.jar!/:na]
	at de.gematik.test.tiger.proxy.AbstractTigerRouteCallback.handle(AbstractTigerRouteCallback.java:114) ~[tiger-proxy-0.18.1.jar!/:na]
	at org.mockserver.mock.action.http.HttpForwardObjectCallbackActionHandler.lambda$handleLocally$0(HttpForwardObjectCallbackActionHandler.java:70) ~[mockserver-core-5.11.9.jar!/:5.11.9]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) ~[na:na]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

I haven't changed any configurations and it was working perfectly prior to the upgrade.
The configuration it self is very basic and I could not find any hints in the release notes regarding needed migrations:
server.port: 8001
tiger-proxy:
  port: 8000
  proxyRoutes:
    - id: route1
      from: "http://route.to.service:8000"
      to: "http://forwardserivce:8000"
      disableRbelLogging: false
  activateRbelEndpoint: true
  keyFolders:
    - "/default-keyfiles/"

The problem is a more restrictive URL-Validation which was built into Rbel recently. The used URLs have to pass https://commons.apache.org/proper/commons-validator/apidocs/org/apache/commons/validator/routines/UrlValidator.html.
Since this seems to not work correctly on intranet or fantasy-urls (we have seen errors with custom domains) we will get rid of this validation in an upcoming release (hopefully in time for 0.19.0). This issue has been mirrored as RBEL-41 internally.
As a workaround you could alter your domain-names so they pass the commons-test.

Thank you for your quick reply. Indeed we are using domains for routing which do not pass this test, but as we are running our services inside a k8s cluster, we can't change this easily. We will be waiting for the 0.19.0 release then...

fixed, will be shipped with 0.19.0


jd-gui: 1.4.0 -> 1.6.5

Motivation for this change
Automatic update fails, upstream updated the build process, and got rid of a gradle 2 dependency in the process.
According to https://github.com/NixOS/nixpkgs/pull/58528#issuecomment-478302487 this could mean gradle_2 can be removed (And probably 3 as well).
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Notify maintainers
cc @

Built and tested, works as expected.


Fix error and warnings on KafkaIO javadoc generation

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

R: @aviemzur

For some context the current master is broken because of javadoc generation referring to a method that does not exist in the current version of Kafka (0.10.x). Yesterday some changes to support Kafka 0.11.x were introduced. This is a quickfix for this.

It looks like the @see used instead of the @link that existed before doesn't seem to work in this context (inside a @param).
Since the problem is that the @link is broken when the client is not dependent on Kafka 0.11+ consider instead of the original code based @link to link instead to Kafka GitHub/Javadoc

Changes Unknown when pulling 83197b3bb5a8d5ed5b22627819ba9ad1d3ba0f62 on iemejia:beam2 into ** on apache:master**.

Good point, Changed to a link to the external documentation.

SGTM
CC: @rangadi

Retest this please.

@rangadi I suppose just moving the version up would have fixed it too but didn't want to do that until I was sure it was the intended idea. We probably need a way to ensure that KafkaIO passes the tests with all the different versions. Extra question why kafka-clients is not provided or if we just move to version 0.11 it would cover the backward compatibility, e.g if I want to connect with a 0.9 instance ?

@iemejia, Kafka versions was supposed to be upgraded to <IP_ADDRESS> in BEAM-2720 that went in yesterday. But looks like root pom changes got dropped just before merging (I must have dropped it during rebase and conflict resolution). I will sending a fix for that now.
KafkaIO support all the versions from 0.9 (that's missing pom update didn't cause build failure). May be better to move it to provided scope. Right now, if a user is building Beam manually they can set kafka.client.version (e.g. mvn -pl sdks/java/io/kafka test -Dkafka.clients.version=<IP_ADDRESS>).

Sent pr #4001 to update kafka-clients version and to fix the warning with @param. mvn javadoc:javadoc -pl sdks/java/io/kafka looks good. Thanks.


Refactor aln.cpp

This renames:

align_SE → align_single
align_PE → align_paired
align_SE_read → align_or_map_single
align_PE_read → align_or_map_paired
joint_nam_scores → nam_pairs (since we now use the NamPair struct)
rescue_mate → rescue_align (could never remember what it does differently from rescue_read)
Rename get_mapq to proper_pair_mapq (since this is only used to compute MAPQ for uniquely mapping proper pairs)

Also, InsertSizeDistribution is moved to a separate file.
The main change is to factor out an output_aligned_pairs() function. The rescue_read() and align_paired() functions previously did two things:

Compute a list of paired alignments
Output them to SAM

When working on random alignment selection for #367, I noticed that the code outputting to SAM was duplicated (and had slightly diverged already). This commit moves that part to a
new function output_aligned_pairs(). Now rescue_read() and align_paired() just
return a vector of ScoredAlignmentPairs.
This should fix at least two bugs:

The details for R1 and R2 were swapped when R1 was rescued
MAPQ of secondary alignments was inconsistently set to 0 or 255 depending on whether it was a rescued alignment or not.

I haven't had the time to look at all the changes in this PR in detail. But I agree with the name changes and with what you describe, without reservation.
If this PR is holding up any other development, or if you think its mostly non-controversial changes, I suggest a merge without a careful check.

Do you want to review this PR more closely or shall I just merge it?

Won't have time anytime soon unfortunately . Yes please go ahead and merge.


i cant chat False ban/mute

I cant chat in sm64js then,
I was in my email and I found a message that said this

Yes, I did by saying "So long, Gay Bowser" and having had posted this image in the discord at a bad time.

Reply to this email directly, view it on GitHub or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android. (It said luphoria sent this to me)

I'm not banned
I never got muted before so I never evaded mute
I never made suicide jokes

Plz unmute me I never did anything wrong
I have to ignore when people ask me questions and it makes me think they think I'm mad at them for whatever reason

This was a conversation on github. You were muted for saying the N word.


model loader should treat Java declarations of type int or float as annotated 'small'

@FroMage it would be great if from an IDE perspective we could distinguish that Java fields/methods/parameters of type int or float are actually small. WDYT?

Can you remind me how small works @tombentley ?

No, because I've forgotten. See https://github.com/ceylon/ceylon/issues/6038 for the gory details though. https://github.com/ceylon/ceylon/commit/f4a9c9e6a7f687c75d51c2066c0669073e7a02cc suggests I did the model loading bit though, so I don't understand what/why it's not working for @gavinking

@tombentley it's working fine. What I'm saying is that we should "infer" small for native Java code.

Ugh. Sorry. Yes, this is already implemented. Just the IDE wasn't displaying this information. Sorry!!


Bug fixin and test app improvements

Purpose
Bug fixes
Type of PR

[x] Bug Fix
[ ] Feature Work
[ ] Revert
[ ] Refactor / Code Cleanup
[ ] Other (fill in...)

Short Description

Fixed a few request queue related bugs discovered today
Changelog / Code Overview

1 - Flush queue on activity stop, rather than pausing. This inherently fixes the issue that we weren't restarting the queue on resume, because i'm never stopping it.
2 - Reset the flush interval after 429s are resolved -- shoulda done that a while ago
3 - Add "unsent" request back to the queue if attempted while there's no internet connection (we were losing them)
4 - Fix obnoxious JSON parsing error due to android embedded org.json package.
Test Plan

Unit tests and test app
What to look for

Related Issues/Tickets

TY I will merge this one, I have another coming up that will add the type-specific json decoding, changes a bunch of the API files so I will add the comment about resetting the timer in there too.


VCamera preview plugin

When selecting VCamera node, plugin opens preview of what the camera sees in bottom right dock. This preview will stay open even if user selects other nodes, so setting up transform modifiers will give feedback from camera's own pov (well, for some modifiers until #22 gets implemented). Preview can be closed via close button (or if target VCamera gets deleted or scene changed).
As a bonus, I made VCamera widget to accurately represent Godot's own Camera widget with respecting fov and small up arrow. It doesn't get too much in way on preview.

This is super cool and a feature I wanted to add but had no clue where to start. 🎉


qgis: 3.16.9 → 3.16.10

Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Relase notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Hi,
I do not have access to my dev machine so I cannot test this, but it seems reasonable to me.  If someone else can check the build, it is :+1: for me.
Thanks.


Expose hints as JSON

Currently, hints are exposed to eww in the custom format "(key1, value1)(key2, value2)...". Since eww has JSON support, it might be easier to expose them as a JSON object.

Sort of implemented in #5, although the format is not optimal. The format rework is part of #12.


[WIP] fully asynchronous solving

@vchuravy @jpsamaroo just playing around with a driver script. It looks like KA doesn't support ranges?


firebase-dsym-upload 2.0.0

This PR updates the step's default location of where upload-symbols is stored from its Fabric location to its FirebaseCrashlytics location. This change is required due to the fact that from November 15th 2020 Firebase will be deprecating Fabric in favour of FirebaseCrashlytics.
What to do if the build fails?
At the moment contributors do not have access to the CI workflow triggered by StepLib PRs. In case of a failed build, we ask for your patience. Maintainers of Bitrise Steplib will sort it out for you or inform you if any further action is needed.
New Pull Request Checklist
Please mark the points which you did / accept.

[x] I will not move an already shared step version's tag to another commit
[x] I read the Step Development Guideline
[x] I have a test for my Step, which can be run with bitrise run test (in the step's repository)
[x] I did run bitrise run audit-this-step (in the step's repository - note, if you don't have this workflow in your bitrise.yml, you can copy it from the step template.)
[x] I read and accept the Abandoned Step policy

Hi @andrewmarmion  👋
Thank you again for your contribution! If you have a minute, please give us feedback on the process through the following minisurvey.


tools support

Simple addition to the package.json to allow for BUILD.tools support

Nope, features like autocomplete and navigate to file and diagnostics functionality works here.
This just lets VScode know that it should run our code on files that are named BUILD.tool.


Not working

zsh: segmentation fault how do I fix?

When does the it occur.?!
Kindly include a screenshot so that we can fix any problems.. thank you


Gracefully style point clouds with missing properties

Previously if you wrote a style for a point cloud that used some non-existent property CesiumJS would throw a runtime error

This PR changes that behavior so that non-existent properties are now treated as undefined which is exactly the same behavior that CPU styling has. This is essentially a continuation of https://github.com/CesiumGS/cesium/pull/8621.
If you know your tileset might be missing a property, it's a good idea to add === undefined guard like the examples below. If a property is undefined it is not assigned a default value, so any math you might have won't work.
I tested with three datasets: point cloud tileset without classification, point cloud tileset with classification, and b3dm tileset. Yellow means the property is undefined, green means the property is defined. Previously the first example would throw an error.
Example 1: Sandcastle

Example 2: Sandcastle

Example 3: Sandcastle

@mramato @hpinkos could this fix be relevant to ion in any way?

Thanks for the pull request @lilleyse!

:heavy_check_mark: Signed CLA found.

Reviewers, don't forget to make sure that:

[ ] Cesium Viewer works.
[ ] Works in 2D/CV.