A formal policy to govern how packages enter and leave the production universe

For when https://production.r-multiverse.org goes live.

With respect to production, a package has many failure modes:

The package is new and not yet in production.
The new version in QA is broken, but the older production version is still okay.
The new version in QA passes its own tests, but it breaks reverse dependencies. Again, let's say the old version is fine.
Whether because base R changed or a reverse dependency changed, a previously healthy package breaks.

On first glance at scenarios (2) and (3), my first thought was to keep the old version in production indefinitely even if the new QA version has problems. But that is a bad idea because:

If we keep an old version in production but the new one in QA, it puts prd and QA out of sync. We want prd to be a simple subset of QA, not some strange old stale version of QA.
We want to rely exclusively on the checks in QA without having to check the checks in prd. Version mismatches would cause all sorts of problems there.
If the old version stays in QA and the new version is in production, then the maintainer incentives are all wrong. It's easy to get too relaxed in prd still succeeds. Maintainers should have incentives to keep the most current versions of their packages working correctly.

Other options:
A. Keep the old version in production for a grace period of 2-4 weeks, just to protect users against breaking changes while problems get fixed. Then remove the package from production if those problems have not been fixed in those 2-4 weeks.
B. Same as (A), but put the new version in production during the grace period to keep QA and prd synchronized. This could help us avoid subtle disagreements between QA and prd.

On a theoretical level I agree with all your analysis above.
I'd be in favour of B, however we could now have a package in production that breaks other packages in production.
So I think in practice, if we're to avoid an even more complicated system with a 'staging' repo, we have to accept the new version in QA, and then it only migrates into the prod universe when all checks pass.
If things don't get sorted then there is no need to remove the old version from the prod universe, as by definition everything works with everything else in that universe. Things will just persist. This is the only way to prevent the whole auto-archival mess.
I think this means we'd have to entertain requests for breakage - which would be a job for moderators.

Reflecting on this more, I vote for option B.

I'd be in favour of B, however we could now have a package in production that breaks other packages in production.

I think this is the lesser of two evils. Not only does it keep the universes synchronized, it also incentivizes maintainers to favor the new version and upgrade the revdeps. By contrast, if we have the new version in QA and the old version in prod, then revdeps have to be compatible with both, which is not always possible. This is exactly the sort of thing that incentivizes maintainers to give up and skip all their tests (c.f. https://github.com/rstudio/rmarkdown/blob/ee69d59f8011ad7b717a409fcbf8060d6ffc4139/tests/testthat.R#L2).
When I shared R-multiverse (back when we called it "R-releases") in that R Consortium meeting a few months ago, Jeroen made a similar point. He said we should favor incentives that create change, improvement, and innovation. I think moving to production but then taking it and revdeps down after a 3-week or 4-week delay makes sense for this.
Also, option B is so much simpler to understand and implement because it does not rely on individual package versions. I feel like there are fewer unknown unknowns with option B.
In addition, https://github.com/r-universe-org/help/issues/149 may help mitigate the tradeoffs of choosing option B.

"fewer unknown unknowns" I'd say that's incredibly important as a governing principle!
I guess the decision here affects how users perceive the Community and Production repos, and the choice of one versus the other.
Community always contains the latest release made by the maintainer, irrespective of revdep checks. Our decision here doesn't affect this.
Option B basically tries to replicate this in Production immediately and takes everything down if this is non-compliance in [3-4 weeks] time.
Alternatively, Production just doesn't upgrade until such a time everything complies (like CRAN does now).
You're right option B is cleaner. But if we take a snapshot at any one time of the Production repo, not everything will pass revdep checks. I just want to confirm we want that here.
Can we create a 'verified snapshot' for each package in Production of all recursive dependencies that work - so there is an audit trail of why a package is in production. Just thinking out loud...

But if we take a snapshot at any one time of the Production repo, not everything will pass revdep checks.

I don't think option A (sticking with the old version) would entirely prevent this. Suppose package Y depends on package X 1.0.0. If X 2.0.0 breaks Y, then revdep checks will still fail regardless of A vs B.

And this doesn't provide the iron-glad guarantees for the user - they can install a package which happens to be broken from Production

Whether we go with A or B, we still have a 3-4-week delay before the package gets kicked out of production. That itself means broken packages will be up there some of the time. Except for the nominal length of the delay, it's no different with CRAN.

Suppose package Y depends on package X 1.0.0. If X 2.0.0 breaks Y, then revdep checks will still fail regardless of A vs B.

Instead of "then revdep checks will still fail regardless of A vs B", I should say "there still exist likely failure modes in both A and B." It just depends on the version of X that Y breaks. I would prefer to use the latest Y for this because of the incentives..

It just depends on the version of Y that X 2.0.0 breaks.

In (A), the versions of X and Y in Production are compatible, so neither are broken as version X 1.0.0 remains in that repo. Sure both get taken down after the grace period, but I thought the Production repo is meant to provide the strongest guarantees to the user i.e. installing broken repos should not be an option.

Except for the nominal length of the delay, it's no different with CRAN.

CRAN implements a mixture really. It does the revdep checks first before the new version is allowed to enter (A), and if this is manually overidden by a CRAN admin, then we get the possibility that a new build may break downstream possibilities, which is then (B).

I would prefer to use the latest Y for this because of the incentives.

Would you mind repeating the logic here? I am seeing that in both A and B, the package gets removed from production after a grace period, providing the incentive to correct the error.

Just to be clear, at the moment I'm not in favour of one way or another or possibly a third way. I'm just trying to state the facts as I understand them.

I think that technically (A) would be feasible either by pinning the tags at Production, or by taking snapshots (of the affected packages?) whenever an issue occurs.
I should think that it's cleaner to use pinning. The infrastructure would then generate the latest 'passing' tag for each package in Community, and use this to write the packages.json for Production.

Would you mind repeating the logic here? I am seeing that in both A and B, the package gets removed from production after a grace period, providing the incentive to correct the error.

Suppose QA and PRD can host different versions of the same packages, even on a temporary basis like in option (A).
Our current plan is to make decisions solely based on the checks in QA. Thus, maintainers are incentivized to fix problems in QA, but ignore what ultimately happens in PRD. That's fine if PRD is a true subset of QA. But if there is any possibility that QA and PRD host different versions of dependencies, then packages that pass in QA could silently and mysteriously break in PRD. Even though a package breaks in PRD, it stays in PRD because it still works in QA. And because it stays in PRD, the maintainer sees no reason to take action.
So, if there is any possibility that QA and PRD host different versions, we need to use the checks from the PRD universe somehow.
Can we make decisions based on just PRD and ignore the checks in QA? Probably not. We are relying on R-universe to run the checks, and R-universe always runs those checks within the universe where the package belongs. We are not equipped to check the package as if it were in PRD before we send it there. We can only check in QA and then move to PRD afterwards. Repurposing R-universe check infrastructure this way would be another big and niche request to Jeroen.
In this scenario where PRD is not a subset of QA (because of versions), I think the best we can hope to do is:

Observe that checks pass on QA.
On-board the package to PRD.
Confirm that tests still pass in PRD after (2).

But this creates the same frustrating loop that you recently experienced trying to keep nanonext published: R-hub/WinBuilder tests passed, then the package was accepted, then CRAN ran checks that you don't have access to, and then nanonext got flagged for problems you had trouble reproducing.
In addition, if we use checks from both QA and PRD, the incentives could be mutually exclusive. Suppose package Y depends on package X. X 2.0.0 is in QA, X 1.0.0 is in PRD, and it is impossible for Y to be compatible with both versions of X simultaneously. This situation happens all the time these days, and it causes maintainers to just give up and suppress all their tests (certainly not the incentive we want to create).

So, if PRD which is not a subset of QA, I think we would have to either ask for huge changes in R-universe, or drop R-universe in favor of completely custom infrastructure. I am not in favor of either option.

Now, let's consider option (B). In that option, we can trust that for each package in PRD,

Its version in QA is the same.
Its entire dependency graph in QA is the same (because of https://github.com/r-multiverse/multiverse.internals/pull/26)

Thus, it is enough to rely entirely on the checks from the QA universe only. Check results should be exactly the same between QA and PRD (except for intermittent non-reproducible failures that don't matter anyway). Package maintainers are incentivized to fix one and only one set of checks, which is completely achievable.
Of course, the downside is that a careless maintainer has the power to temporarily put a botched version of an existing package in PRD. I would argue this is not as bad as it seems. If I understood @llrs correctly when the three of us last spoke, Bioconductor already operates under this model. There is an initial review process that establishes trust, and then that trust is extended to future versions of the package. In our case, passing QA grants PRD privileges, and failing QA revokes PRD privileges. That's how I would look at it. This seems much more desirable and much more transparent than how things work today (accept the package initially, then reject it for surprising reasons you could not have prevented).

So, if there is any possibility that CMTY and PRD host different versions, we need to use the checks from the PRD universe somehow.

Yes I'm with you to here. There is the possibility of divergence if checks only happen in CMTY.

I'm with you to here.

I'll try to be a bit clearer. To think through option (A), it may help to think in terms of "promotion" and "demotion":
Promotion
Let's say "promotion" is when a new version of a community package goes to production. I guess I can summarize https://github.com/r-multiverse/help/issues/57#issuecomment-2161025664 by saying this: for promotion in option (A) to work properly, we would need to check each community release against the dependencies from production. This might be feasible from the R-universe side, but we would need to ask Jeroen.
Demotion
Let's call "demotion" the process of removing a package from production if its checks persistently fail. For this, we would need to run record_issues() on the production repo as well as the QA one. This is actually doable. We would need a clone of https://github.com/r-multiverse/checks for production, but that's straightforward. (We may even want to combine https://github.com/r-multiverse/checks and https://github.com/r-multiverse/multiverse back together for community and take a similar approach for production.)
And coming back to https://github.com/r-multiverse/help/issues/57#issuecomment-2127770925:

my first thought was to keep the old version in production indefinitely even if the new QA community version has problems.

If checks always use dependencies from the production universe, this actually becomes feasible. No need to interfere with a locked production version if the community version fails.

To summarize: it all depends on whether community packages can be checked using production dependencies. If this is possible, we could end up with something like (A), but better.

What about reverse dependency checks? For Option B, https://github.com/r-universe-org/help/issues/369 is equivalent to revdep checks, and there is nothing more to do on the R-multiverse side. For Option A, we have at least these obstacles:

To keep community and production from diverging, we would need an additional request to R-universe: rerun a package's community checks whenever a production dependency in updates. This may or may not be feasible from the R-universe side.
Can we run a reverse dependency check during the promotion phase, just like CRAN does? Maybe, but we would need to run that check ourselves, which is a lot to ask from an R-multiverse standpoint. Ideally we would want to rely completely on R-universe-side checks.
Should we run a reverse dependency check during the promotion phase? This may not be necessary or desirable. Dropping revdep checks during promotion might create the incentives we want (upstream packages should be free to improve, downstream packages should modernize and adjust).

I propose something a little different. Just to explain where I'm coming from: I see the biggest risk being the user experience. We are going to be maintainer-friendly however we structure the rules, even if we try on purpose not to be so!
But if, as a user, downloading packages from our shiny new Production repo (and everything we claim that to be) we kind of expect it to be 'production'. It may be OK, if one time by chance some hiccup occurs and we download a cohort of packages that are somehow broken (think something like install.packages("devtools")). But if this should happen twice or three times, then I think we've lost all credibility. As a user I'm not going to know (or want to know) how the system operates, or how this impacts maintainer incentives etc. I just know that Production guarantees everything to work, and Community to have additional packages if I can't find them in Production.
Option C
The production universe tracks specific versions of packages using branch: "SOME_GITHUB_SHA" in packages.json.
Checks are performed only on the Community repository.
A new package or new package version enters into Community to be checked.
If it passes, it remains in Community and the fixed SHA is promoted to Production.
If it fails, it is removed from  Community immediately, and Community rolls back to the fixed SHA from Production.
We would need to create a third ‘staging’ repo to then track the branch: release of these packages with issues - but this can be kept as an internal implementation detail. Then when the next version is released, this again enters Community for checking.
This mechanism ensures that Community and Production versions are aligned, and more importantly that you can’t download a broken package from Production.
Revdeps checks:
If the new version in Community and Production causes revdep check issues in Community, then take down the affected packages from Production immediately.
Community moves to the fixed SHA for these packages and they are added for monitoring in the staging repo.

Welcome your thoughts / modifications on this.

To keep community and production from diverging, we would need an additional request to R-universe: rerun a package's community checks whenever a production dependency in updates. This may or may not be feasible from the R-universe side.

I actually don’t think we need this because https://github.com/r-universe-org/help/issues/369 is really about demotion. By contrast, CRAN-like revdep checks are all about promotion. When a new version reaches community, it is checked against the deps in production, and then the promotion decision is simply made.

Option C seems like an interesting way to check candidates before they go to production, and to make sure production is a subset of community most of the time. But a few things give me pause:

We would need to create a third ‘staging’ repo to then track the branch: release

Each universe will have a lot of packages when scaled up, and that places demand on R-universe. Three universes are a lot to ask.

If it fails, it is removed from Community immediately, and Community rolls back to the fixed SHA from Production.

What if you release two packages together and the downstream one is rejected because of a timing issue? Or, what if the failure is spurious and you need to try again? Would the maintainer need to create another release version to trigger this, or does community keep pulling from staging until it works? The former seems like a lot of manual work for the maintainer, and in the latter case, the community package could loop between staging and production versions indefinitely, which seems like a lot of churn.

This mechanism ensures that Community and Production versions are aligned, and more importantly that you can’t download a broken package from Production.

What if a previously healthy version starts to break production for reasons unrelated to the package itself? E.g. updates to dependencies, the version of R, the GitHub Actions runner OS, etc.

I see the biggest risk being the user experience.

Unless I am missing something about option C, think the revised option A from https://github.com/r-multiverse/help/issues/57#issuecomment-2161401793 achieves this equally well (as long as community checks can happen with production dependencies). In the latter, new versions are checked against production dependencies before being promoted to production, which ensures the same quality from a user perspective. This makes me prefer A at this point.

Each universe will have a lot of packages when scaled up, and that places demand on R-universe. Three universes are a lot to ask.

The third repo is small as it just tracks the packages with issues. There may be a clever way to not need one, but having a 3rd repo seems to fit the bill.

What if you release two packages together and the downstream one is rejected because of a timing issue? Or, what if the failure is spurious and you need to try again?

It's on the maintainer to correct things (which he can do immediately if he chooses). It's lowest risk as if the package is rejected, Community and Production still retain the previous version. A new version is only accepted when checks pass.

Would the maintainer need to create another release version to trigger this, or does community keep pulling from staging until it works?

Create a new release, which gets picked up by Staging. Then when Community sees the new version in Staging, pulls it in for checking.

What if a previously healthy version starts to break production for reasons unrelated to the package itself? E.g. updates to dependencies, the version of R, the GitHub Actions runner OS, etc.

The check would fail in Community. IIRC they are being re-run periodically. Then it gets taken down from Production and placed on watch in staging.

I see the biggest risk being the user experience.

Unless I am missing something about option C, I think the revised option A from #57 (comment) achieves this equally well (as long as community checks can happen with production dependencies). In the latter, new versions are checked against production dependencies before being promoted to production, which ensures the same quality from a user perspective. This makes me prefer A at this point.

You identified at least 3 obstacles to A in https://github.com/r-multiverse/help/issues/57#issuecomment-2161419398. This option C is an attempt to overcome them. Did I miss something?

The third repo is just an implementation detail for want of a better solution - it doesn't even have to be public and no one will be downloading from there. We just need a way to (i) track latest release, (ii) pin the existing working release at Community and Production, (iii) when the latest release SHA changes (i.e. maintainer has made a new release), push it back to Community (or Community pulls it in).

Just to make sure I've addressed your questions:

Unless I am missing something about option C, I think the revised option A from https://github.com/r-multiverse/help/issues/57#issuecomment-2161401793 achieves this equally well (as long as community checks can happen with production dependencies)

Community checks with Production dependencies would be an ask of R-Universe, perhaps difficult right?

In the latter, new versions are checked against production dependencies before being promoted to production, which ensures the same quality from a user perspective.

If not promoted, this new version that fails checks remains in Community and causes divergence.
Option C is just my attempt at making option A work, and also tries to incorporate the most appealing feature of option B, which is that there is no divergence between the 2 repos.

I was tagged in relation to Bioconductor. Bioconductor trust upon first review, then maintainers can vandalize the packages (as long as they pass a minimal check), so old packages do not need to comply with newer requirements. I wouldn't recommend to take this path, as it has lead to "vignettes" being only a link to a book that it is out of sync with the package (Even if Bioconductor allows to publish books) and other issues.
I think you are reaching now the key issues that CRAN handled so far very well. I am not sure I followed the whole discussion well but some thoughts related to the points discussed:

To track if a Community package promotes to Production a simple file in a repository could be used. I don't see the need of a "full" r-universe R repository.
I think so far the assumption have been made that package maintainers are on top of things and following "the events of the multiverse" (sounds like a film). But that won't be the case in the long term. Unless all decisions are made automatically notifying maintainers will be required at some point.
I thought the Production repository was all about doing the reverse dependencies and only those that pass the checks are included there. If a machine has all the packages installed it should be relatively easy to check if they work with the new package (I'm thinking in symlinks and a specific library for each reverse dependency check or using the tools::check_packages_in_dir).
I am not sure how solving the discrepancy when a package pass checks for Community but not for Production will be resolved. In Bioconductor, discrepancy between Production and Community is allowed and even encouraged: new features shouldn't come to the production repository, aka. the released version, they should only appear on devel so that maintainers can adapt. In CRAN, there shouldn't be, at the cost of accepting for a window of time, 2 weeks, broken packages.

I think Gabe Becker presented these problems more organized in a meeting a year ago in the R repository working group (I am not sure if it was recorded or written down).
On a related note, I'm not sure if you are aware that CRAN now displays the deadline they set to a maintainer to fix a package. So now this is public and other maintainers can start preparing themselves if something goes amiss. See for example this post from Dirk Eddelbuettel. I think with time this will help to measure in how much of a discrepancy state is CRAN, but also maintainers and users.

Thanks @llrs. I think the following points are great takewaways:

It's important not to lend too much 'trust' and let things get stale in Production
Notifying maintainers is going to be important (the role of the RSS feed etc.), and frequency of moving in and out of Production

I think it's also important to note that we're not still having the same discussions as the RCWG (although it may look like it). I think either of the existing plans already deliver at least a CRAN-like level of service to users. We're really looking to see how far we can push things and go above and beyond.
For example CRAN allows a 2 week grace period, extended by another 2 weeks if there are strong-dependencies. During this period, any user can download 'broken' packages. Similarly, only a couple checks are run when an update is accepted into CRAN, and the rest run afterwards. We're playing with whether we can improve on this - so although it may sound alarming when we say 'broken' packages above, we're already within these parameters and it is not worse than this.
The option C I proposed above has the benefit that Production remains stable 100% of the time, but perhaps this is too ambitious. I think we can get close to it with a modified version of option B. I'll post something later on.

@wlandau to summarize where we now are, I think C is the logical conclusion of A so that package versions between Community and Production remain aligned 100% of the time.
I am also onboard with your comment here: https://github.com/r-multiverse/help/issues/57#issuecomment-2161538055
So if we focus on B and C:
Option B

Package maintainers are incentivised to fix problems are they affect Production (+)
Grace period allows packages not passing checks to enter into and stay in Production, but not beyond current CRAN practice (-)

Option C

Packages in Production are 100% safe (+)
Packages move in and out of Production with no notice when checks fail (-)

So Option C is user-friendly, but probably impossible actually for maintainers (surprising result!).
Option B seems a good compromise in comparison, and all else being equal is preferred due to its simplicity.

Community checks with Production dependencies would be an ask of R-Universe, perhaps difficult right?

Yes, and potentially difficult. Haven't asked Jeroen yet what he thinks.

I think C is the logical conclusion of A so that package versions between Community and Production remain aligned 100% of the time.

On reflection, we don't necessarily need versions to align. We just need the promotion checks in community to be identical to the demotion checks in production. Otherwise, maintainers will get blindsided by production-level failures they cannot reproduce. That's what this is really all about. Version alignment is one way to reach check alignment, e.g. with Option B, but it's not the only way. Another way is to make both sets of checks use production dependencies. That's why I prefer A over C.

On reflection, B is no longer my favorite. Even the most diligent maintainer can make mistakes that cause checks to fail. If I am a maintainer and I make a mistake, I don't want that mistake to automatically go to production. It's extra pressure to get the release right on the first attempt.

To summarize: if we go with Option A, my questions are:

Can we check community packages with production dependencies? I will ask Jeroen.
Do we need reverse dependency checks?

For (2), revdep checks have already been proposed for R-universe (https://github.com/r-universe-org/help/issues/362, https://github.com/r-universe-org/help/issues/364), and we might consider running a revdep check on a new community version before promoting it to production. Advantages:

Prevent breaking changes to downstream packages.
Ensure the same guarantees as CRAN.

Disadvantages:

Revdep checks can cause deadlock. If package X 2.0 breaks package Y 1.0, then it is impossible to promote X 2.0 and Y 2.0 together. Without revdep checks, you can promote X and then immediately promote Y.
Revdep checks are computationally demanding and might only become available on a manual on-demand basis in R-universe.
https://github.com/r-universe-org/help/issues/369 is already a less heavy-handed way to catch revdep issues early, albeit in production.

I've been thinking about a solution that works like this:

Community runs checks on Community packages
Production runs checks on Production packages

Production is just a subset of Community (same package versions at all times). Hence a new package that is pushed to Community gets reflected in Production straight away.
Only checks in Production matter. If a package in Production starts failing checks in Production - due to an update to itself, or a dependency - it gets put on notice.
If on notice from Production:

Remove in 2 weeks if no revdeps
Remove in 3 weeks if revdeps (revdeps get notified after 1 week)

The notice period is cancelled as soon as a new version passes checks in Production.
Community checks would be for information, but as we allow all packages in the repo, it leads to no action.
I'm very keen to keep the mechanism as simple as possible for it to gain easy acceptance by the community.

Your idea keeps the simplicity of Option B while actually using checks from production. It does keep things simple, and unlike B, it does not make assumptions about how checks translate from one universe to another. That part is appealing. However, I don't think it will work because checking in production requires manual work in many cases.
Consider the isoband incident from last year, when thousands of packages were on the brink of removal. Suppose isoband breaks in R-multiverse, takes down all its revdeps, and then gets fixed.

The more I think about it, the more I like revised Option A from https://github.com/r-multiverse/help/issues/57#issuecomment-2161401793. That's the one where we use checks from both community and production, but make them both use production dependencies. It has compelling advantages:

It's the most CRAN-like out of all the options we have considered so far, and that's a good thing. CRAN is a model that the R community already understands, and it is generally good at keeping broken packages out of production. I think borrowing the best parts of a tried and true model will help R-multiverse keep the ecosystem healthy, gain acceptance, and avoid criticism.
An isoband incident would fix itself automatically. After the patched version of isoband reaches production, the community checks on the revdeps will start passing again, and those thousands of packages would migrate back to production automatically.
As a maintainer, I am never sure if a package works until the important checks run (or at least the approximation that Option A provides). Suppose I create a broken GitHub release and then head out for a holiday. It will be hard to rest knowing that I just broke one of my packages in production and can't fix it until I come back.

Here is a quick visualization, to check if I understood correctly the different proposals to include packages.
In blue the repositories of r-multiverse.
Charlie's last proposal:
flowchart LR

repo[maintainer repository] --> |initial setup| A
A(Community) -->|automatic| B(Production)
B --> C{checks}
C -->|pass| F[Remains on Production]
C -->|checks fail with revdeps| D[Removed from Production in 3 weeks]
C -->|checks fail with no revdeps| E[Removed from Production in 2 weeks]

style repo fill:#34eb4f
style A fill:#34d5eb
style B fill:#34d5eb
style C fill:#eba834

Will's last proposal:
flowchart LR

repo[maintainer repository] --> |initial setup| A
A(Community) --> C{checks in production}
C -->|pass| B[Production]
C -->|checks fail| D[not included in Production]

style repo fill:#34eb4f
style A fill:#34d5eb
style B fill:#34d5eb
style C fill:#eba834

What I still don't understand is which checks are performed in each case:

For Charlie's approach will they be performed in several R versions, or only on released branch? How will R or other (system) dependencies be managed?
For Will's approach in addition to the previous questions, I am missing when packages on that scenario leave Production (besides when being replaced by newer versions). Will checks be on inclusion and on schedule (each day/week, ...) ?

The second approach is easier to understand as it is more similar to CRAN and Bioconductor inclusion criteria.

Right let's move to convergence.

The key insight from the system I proposed is that only Production checks matter. I've confirmed for Option A that this is also the case. The checks between Community and Production (= new version as if in Production) always agree, except for when a dependency has different versions between Community and Production. As soon as a passing version of that dependency is uploaded and promoted then the check discrepancy will sort itself out. But no matter what the Community checks are, only the Production ones matter for promotion.

I agree with fixing the SHA for Production as proposed in Option A (think this is good practice). This allows temporary divergence of versions, but keeps bad builds out of Production and helps with stability of the overall system.

It then becomes clear that Production checks for strong dependencies should be run whenever a new package is promoted - this is https://github.com/r-universe-org/help/issues/369#issue-2164948647 This is the answer to @llrs question above.

This system has the advantage that packages with no dependencies passing checks automatically promote, forcing dependencies to update. This is a cleaner and more transparent practice than the current manual intervention / decision for CRAN.

Open question: how to handle when checks fail in Production. The package should be put on notice for removal from Production. This necessitates also notifying strong depends. We could follow the CRAN 2 weeks / 2 weeks policy or something different. But structurally there should just be a notice period, otherwise packages will move in and out too frequently.

This system requires running a check on a package in Community as if in Production. This is the same as actually putting it in Production, run the checks, and then roll back immediately if it fails. I think this then makes it technically feasible (in case it is not possible for R-universe to implement this feature) and the time window is short enough not to likely cause issues for users (and they can always download again).
I also do not see a role for the CRAN-style revdeps checks if the above is put into place.
Anything important I've missed out?

@jeroen, thanks for your willingness to use production (+ CRAN) dependencies for community-level checks. With that on the agenda, I think are good to go with revised Option A as described above. I will begin implementing this soon, hopefully next week.

I have some prework at https://github.com/r-multiverse/community/pull/10.

@wlandau you may already have an idea in mind, but I thought we could tackle the Production universe implementation in broadly 3 steps:

GitHub action to write the fixed SHA of the Community repo packages (to populate the Production universe)
Implement the logic in terms of check results used as precondition for the above
Implement the logic for checks triggered when a package gets promoted to Production (for recursive revdeps) and actions to put failing packages on notice for removal from Production

In terms of actual checks to use, I am in favour of using R-release and R-oldrel (which fortuitously matches with R-universe's current build system), and not use R-devel (until perhaps [2] months from the next annual release date for a transition period). The current debacle with constant changing of the API including forcing errors on packages is neither desirable nor necessary. Production by definition should not be running pre-release software.

Yeah, I think that makes sense in terms of 3 layers for the implementation of a new multiverse.internals::write_production() function (or whatever we name it). And I agree to use R-release, and if available, R-oldrel (avoiding R-devil 👿 )
FYI, to update this thread: @jeroen mentioned it might not be feasible to use production dependencies for community checks after all. This is because R-universe combines the "check" step with the "build" step. So if we want install.packages("tidypolars", repos = "https://community.r-multiverse.org") to work, then polars needs to be in production, which it's not. So we're thinking about a third "Staging" universe, where packages are checked with production dependencies, but where the builds are not necessarily available to users.

An alternative is to add extra checks to Community and have those checks use production deps.

In an offline discussion, @jeroen said it may be possible to filter based on checks at the R-universe level: if a check fails in the production universe, do not publish the build. I would like to know if the following details around this are also feasible:

Beforehand, we make a list of "important" checks for the purpose of filtering.  For the purposes of getting started, I it seems reasonable to include all R-release and R-oldrel checks that R-universe provides. But creating a good list of checks is highly nontrivial, and a serious discussion belongs in a separate thread like https://github.com/r-multiverse/help/issues/58.
For the production universe only: if an "important" check fails, then no build is published for that round of checks. For example, if package X version 2.0 fails on Windows but succeeds on Linux, then the Linux build is still not published.
https://production.r-multiverse.org/api/packages shows the failed results from X 2.0.
The most recently successful build of X remains installable. For example, if X version 1.0 passed on all platforms, then install.packages("X", repos = "https://production.r-multiverse.org") installs version 1.0, and https://production.r-multiverse.org/src/contrib/PACKAGES.json lists version 1.0.
None of this applies to the community universe. In Community, every existing build gets published, regardless of checks.

If all the above is feasible, this opens new possibilities for R-multiverse.

Given the above, I would like to propose a new model for production:
Option D
Universes

Community: every supported build of the latest release is always published, regardless of checks. The R-universe packages.json has "branch": "*release" so that each new GitHub/GitLab release automatically gets published in the Community universe.
Production: a build is only published if checks succeed, as described in https://github.com/r-multiverse/help/issues/57#issuecomment-2188995766. The R-universe packages.json has "branch": "SPECIFIC_GITHUB_SHA", where SPECIFIC_GITHUB_SHA is the GitHub SHA of a specific release version.

Check types
multiverse.internals can get results from any of the following:

R CMD check checks: R CMD check results from the R-universe API.
Version checks: check that the version number monotonically increases with each release.
Description checks: check issues discoverable from the repo's src/contrib/PACKAGES.json (basically available.packages() metadata). For example, packages in production should not have Remotes:.

Promotion
Promotion is the process of putting a specific release version of a package in production. In Option D, the process is happens as follows:

Check the version checks and description checks in Community. If they pass, move to the next step. If either fails, the package version is withheld from production.
Include the new package version in the Production packages.json. If the important R CMD check checks succeed, then R-universe published the build to https://production.r-multiverse.org. If any important check fails, then the new build is not published to any platform in Production, but the most recent existing build is still available from install.pacakges() and src/contrib/PACKAGES.json.

Demotion
Demotion is the process of removing a build from Production if a check starts failing. This is accomplished with a "reset" mechanism (step 3 below).

Consider the "important" R CMD checks, version checks, and description checks that happen in Production. If any of these checks fail, put the package on notice for 2-4 weeks.
If the package is fixed on time, then the notice is removed, and any new notice starts from day 0.
If the package is not fixed on time, then it is removed and blocked from Production's packages.json until all builds disappear from src/contrib/PACKAGES.json. After the build is gone, then it can be unblocked so the automation for promotion can add it back at an appropriate time.

A much simpler version of Option D (let's call it Option E) is to

Completely remove the notice period.
Do not keep the old build available in Production if checks fail.

This makes the production model a whole lot simpler (no "reset" mechanism for demotion). And it may not actually be that bad for maintainers. Suppose my package crew version 0.9.4 was previously succeeding in production, but then I release broken version 0.9.5. In Option E, crew immediately vanishes production, and install.packages() fails. If that happens, I can simply go to https://github.com/wlandau/crew/releases/edit/0.9.4 and set version 0.9.4 to be the latest release, then have all the time in the world to troubleshoot:

On the other hand, downgrading doesn't always solve problems. If a reverse dependency is the cause of the trouble, it can take everything down with no notice. So Option E probably won't work.

Agree option E doesn't work - as a dependency can suddenly disappear at any time.
I see you've fleshed things out in option D, but fundamentally it seems to be the same as what we had agreed upon before? I can see that it technically incorporates the build/check step in Production, and only gets published if it passes. This is similar to the backup plan we had before - put in Production, check then roll back if it fails. Now we don't need to roll back, right?

I see you've fleshed things out in option D, but fundamentally it seems to be the same as what we had agreed upon before?

Yes, just a couple implementation details are different.

I can see that it technically incorporates the build/check step in Production, and only gets published if it passes. This is similar to the backup plan we had before - put in Production, check then roll back if it fails. Now we don't need to roll back, right?

Exactly, we don't need a rollback because the "roll-forward" is filtered. The only downside is that now we need the extra reset mechanism for demotion, and that seems complicated: remove and block from packages.json, then wait for the build to disappear, then unblock. Willing to go with that, but I wish it were simpler.

So if I understand correctly, if a package v2 gets submitted and fails, v2 stays in Community and v1 stays in Production indefinitely? That means that there must be a record somewhere that v2 was checked so that it is not constantly re-checked for promotion. Then only when v3 is submitted, is the new version checked.
Conversely, if we have this, that means that if a package is actually taken down e.g. V2 was in production, fails as a dependency is no longer in Production, is removed from packages.json - it eventually disappears from Production. As V2 was recorded checked, it shouldn't somehow end up in Production again.
With the exception being - we can require that every time a package is promoted, all its dependencies in Community get rechecked, and promoted if necessary. This implements the auto-healing isoband situation.


[interop] removeArrayElement not resizing arrays

If I understand removeArrayElement correctly, it should remove the element and resize the array. In GraalPython, it does the right thing. In Graal.js, however, the array is not resized. Also, the removed slot is no longer readable.
GraalSqueak repro:
((Polyglot eval: 'python' string: '[1, 3, 4]') primitiveRemoveArrayElement: 2) "ForeignObject[arraySize=2,memberSize=50]"

((Polyglot eval: 'js' string: '[1, 3, 4]') primitiveRemoveArrayElement: 2) "ForeignObject[arraySize=3]"

((Polyglot eval: 'js' string: '[1, 3, 4]') primitiveRemoveArrayElement: 2) at: 2. "FAILS (not readable)"

Tested with GraalVM CE 19.2.1 on macOS.
/cc @chumer

Hi @fniephaus
We are probably just doing what JavaScript itself would do:
> var a = [0,1,2,3]
> a
(4)[0, 1, 2, 3]
> delete a[3]
true
> a
(4)[0, 1, 2, empty]
> a.length
4

The element is deleted, but the length is not changed.
The JavaDoc of removeArrayElement is not very explizit on that: Remove an array element from the receiver object - from a JS perspective, we have fulfilled that (the property with that index is gone). Also This method does not have not observable side-effects other than the removed array element (sic!) hints to not manipulating the length. But I know other languages would disagree.
@chumer do you think this should be unified? At least, we should mention what "removing" the object means (i.e., whether that strictly needs to make the array one element shorter).
-- Christian

In the current implementation, a deleted slot is no longer readable, which violates the array interop concept (must be readable from 0 to getArraySize if I understand correctly, otherwise they cannot be iterated correctly. Maybe @chumer can confirm this).
In Javascript, the following works fine:
> a[3]
undefined

This is also true for a[4] etc. So I'd say isMemberReadable might always return true in Javascript, but may return undefined (an interop null value) in the case the element was deleted or if the index is out of bounds.
The following example demonstrates how the current behavior affects interop with Python:
$ polyglot --shell --jvm
GraalVM MultiLanguage Shell 19.2.1
Copyright (c) 2013-2019, Oracle and/or its affiliates
  JavaScript version 19.2.1
  Python version 3.7.3
js> let x = [1, 2, 3]
js> Polyglot.export('x', x)
(3)[1, 2, 3]
js> delete x[1]
true
js> x
(3)[1, empty, 3]
js> python>
python> import polyglot
python> x = polyglot.import_value('x')
python> x
[1] # may be `[1, None, 3]`, certainly not just a list with `len(x) = 1`
python> len(x)
3 # This is correct!
python> x[2]
3 # works!
python> x[1]
IndexError: invalid index 1 # may return `None`
python> for i in x:
      +   print(i)
      +
1 # `None` and `3` are missing, list iteration broke
python>

It's a bit worse than that, as this doesn't just seem to affect removeArrayElement - it also carries over if you use Value#as to cast the array to something like java.util.List.
The remove method on the List interface is clearly documented as shifting subsequent elements to the left rather than leaving a hole; if an implementation cannot do this, the documented alternative is to throw an UnsupportedOperationException.  The current graal.js behavior violates this contract in a very unexpected way.

Hi,
@woess is currently working on this, a fix will be merged shortly. We will ensure that all delete operations on foreign objects actually shift the higher indices, instead of leaving a hole. This will also be clarified in the Javadoc for removeArrayElement.
Best,
Christian

This issue should be fixed in master (will be released in GraalVM 20.1.0).
removeArrayElement now removes the element and shrinks the array, and holes in arrays are now readable and return undefined.

Thanks, https://github.com/graalvm/graaljs/commit/e45a671be53fe0210b1014bbb683c09ed5a2b48c seems to be the corresponding change. I will double check when the next dev build is out and close this if everything WAI.

I can confirm this is working now (tested with 20.1.0-dev-20200226_0309). Thanks!


Outdated cask: Sonic Pi

Sonic Pi
New version can be found here: http://sonic-pi.net/files/releases/v3.0/Sonic-Pi-for-Mac-v3.0.dmg

https://github.com/caskroom/homebrew-cask/pull/36772


Add GCC to Prerequisites

While setting up an amazon linux ec2 instance I noticed that GCC needs to be installed to compile/run from code.

GCC is needed if you want to compile TP, in addiction to other Bitcoin core dependencies. But in the getting-started we reference the binaries, so I'm not completely sure about adding this to the guide. Or maybe I'm missing something (?)


Website gallery of Cupertino widgets

Similar to how we have a gallery of material widgets https://flutter.io/widgets/ we should find a way to make it very obvious what cupertino widgets we have, and when/how to use them.
Some ideas:

Link https://docs.flutter.io/flutter/cupertino/cupertino-library.html to a widget gallery

Update https://flutter.io/widgets/ to include Cupertino widgets

Link https://docs.flutter.io/flutter/material/material-library.html over to Cupertino as a "See Also"

@sethladd says @InMatrix is on this.

Going to close this in favor of https://github.com/flutter/flutter/issues/9531


Helm chart doesn't follow best practices for specifying namespace

The Helm chart found in the deployments dir configures namespaces through --set or a values.yaml. This is not how Helm charts are recommended to be written. The recommendation is that namespace: be entirely omitted from the metadata for each resource. Operators should specify a namespace when installing a chart by using the --namespace flag.

I can't assign this to @krancour yet, but he's working on this one.

Thanks for the suggestion on Helm best practices! As I was learning Helm I discovered the recommendation to use --namespace flag and not use namespace: clauses in the chart.
Strangely enough, in another branch of the initial seed code we needed a Helm chart which would deploy into multiple namespaces (default and catalog) which is not possible otherwise. Of course, that use cases doesn't apply to this incarnation of the chart so it is possible to clean it up.


added MOCHA_MIRROR_PORT option to start the server on a custom port

+1

What's it going to take to get this merged? This request has been open for a month now :(

I would like to see this added as well.


Add some frontend tests for lookup workflow

This PR tests some behaviors of the frontend with respect to the API lookup workflow. It adds a simple end-to-end test, but doesn't actually set up the fake matching endpoint, and only really covers the case where the endpoint returns no results.

@manthey
More tests could be added in the future to properly set up the fake matching API endpoint. I'm not sure I'll have time to get that working for this PR. The testing endpoint refers to a file either in /conf or ~/.girder. I don't know exactly how to do this in tox. Does the tox working directory act as the root, and could the file be added to TOX_WORK_DIR/conf?

Maybe we could change the line filepath = '/conf/fake_matches.json' to filepath = os.environ.get('WSI_DEID_FAKE_MATCHES_PATH', '/conf/fake_matches.json') and then set that to a tox relative directory?

This is a great start.  We can merge this as it stands or you can add to it if you have time.


Fix #3616

Taking the suggestion from @TWal of adding zeta to the steps in reveal_opaque.
Note, this can cause issues when the function being revealed is recursive and not guarded by a pattern match. E.g.
let f () : Dv int = f ()

This will send the normalizer into a loop which will stack overflow. I don't think these are usual occurrences and this is a separate problem of the normalizer itself, so I think adding zeta is not too bad. When the recursion is under a match (almost always) this will block the unfolding and work OK.

https://github.com/mtzguido/FStar/actions/runs/12165596509


0.11.0 -> v0.11.1

Part of #1951
This is about a tiny path release which is mainly just for the update of the contracts to v.37.1 to be compatible with the Python client v1.1.1.

I'm confused why the tests do fail. This PR does not include any code changes.

Please, bump also raiden-cli, and change the script to bump it together with the other packages. The failure may be some small flatness, and should go away on a new run

As asked, I rebased on master to include Talel his fix. I also extended the prepare release script to include the CLI bumping. The CLI also now got its first version of a CHANGELOG file.

Okay this is pretty bad that the test are flaky. Anyways it should be fine I guess. Starting to prepare the GitHub release text now.


Update README.md

Changes
Fix installation command in README.md

@leandropineda I fixed the tests.
There was a comparison with an error message that didn't match because it referenced an old version of the Pydantic docs.


release: v1.0.0

Initial release of this plugin.
@AllanOricil - Does this PR/merge commit need to meet some message standard to trigger the semantic release at 1.0.0? Or does just merging this into main trigger a release and automatically update the package.json version?
Pending Actions before Acceptance

@scolladon offered to add initial updates to make the decompose/compose asynchronous, allowing them to process multiple files at once. I'd like to give him some time to do this (no rush!) since this will be beneficial for large directories.

Other Actions that might be pushed till after 1.0.0

Implement logging with the CLI (currently I have log points commented out to prevent ES Lint warnings, but these should be updated to use the logger class once that's in)
Implement @AllanOricil 's suggestion to add CLI hooks for post-retrieval and pre-deploy
Implement better test classes that easily show that both functions work and all files are generated successfully with no changes against the baseline

Locally, add a blank commit in beta saying "feat: first release", then run npx semantic-release --dry-run and check if it creates v1.0.0-beta. If it does, push beta to github and wait it to publish v1.0.0-beta to npm. Then finally merge beta to main using merge commit or ff, and again run npx semantic-release --dry-run locally and verify if it creates v1.0.0. If it does, merge your PR to main using ff or merge commit, and just wait v1.0.0 to appear in npm.
obs: I'm not sure because I never don't anything with a beta branch yet. That I recommend running semantic-release locally with --dry-run to simulate what is going to happen on remote.

@mcarvin8 here is their official tutorial https://github.com/semantic-release/semantic-release/blob/master/docs/recipes/release-workflow/pre-releases.md

Thanks @AllanOricil.
Running semantic release locally with --dry-run still enforces the NPM_TOKEN and GH_TOKEN variables, which is a little annoying. I have my NPM token saved locally, but the repo's GH_TOKEN isn't something that can be accessed and adding a personal access token doesn't do the trick either. Not sure if there's a trick to bypass the GH_TOKEN check locally.

@mcarvin8 remove github and npm plugins from .releaserc, and then try again. you are only interested in the version bump which comes from the commit analyzer, I think

Thank you @AllanOricil , removing NPM/Git from the .releaserc did the trick locally.
Note: With the commit message feature: beta release, since I have a v0.1.0 created already, semantic release created the beta release as 0.2.0-beta.1, not 1.0.0.
To trigger the 1.0.0 beta build, I needed to add BREAKING CHANGE: to the commit message to trigger a major release.
feat: beta release

BREAKING CHANGE: first release of this plugin

Yikes, semantic release is failing due to the commit message passing a char limit....

@AllanOricil - can you take a look at this log and let me know if something looks off ? https://github.com/mcarvin8/sfdx-decomposer-plugin/actions/runs/7718592396/job/21040197432?pr=15
keep running into a char limit on the commit message. The BREAKING CHANGE: needs to be on a newline to register a major release, not just a minor release. I had to force-push beta to remove the commits I made to trigger the beta release, but this shorter message isn't doing the trick.

Creating a release candidate build (beta-2) prior to releasing this.


fix-null-exception

The following SVG throw an exception (when using flutter_svg) : https://sweden.a.bigcontent.io/v1/static/10000199.
This SVG contain an empty  tag : <polygon xmlns="http://www.w3.org/2000/svg" fill="#0a287d" points="" id="polygon7"/> which lead to the issue.
Checking as this PR did if path is null avoid the issue.

Hi there, The Flutter team is now maintaining these packages in the flutter/packages repository at packages/vector_graphics and packages/vector_graphics_compiler. We'd appreciate it if you could direct this PR to those locations instead. Thanks!

PR open https://github.com/dnfield/vector_graphics/pull/254


[BUG]:  Rendering is not supported in JSON when the key value is empty.

Issue description
For example, this json string, but the result image is confusing.
{
"": {
"v1": "1",
"v2": "2",
"v3": "3"
},
"x_1": {
"v1": "4"
},
"x_2": {
"v2": "5"
}
}
Media & Screenshots
No response
Operating system

OS: [e.g. iOS]:

Browser [e.g. chrome, safari]:

Any other details...

Priority this issue should have
Low (slightly annoying)

I want to work on this, Can you assign this to me?


Fix lightning-cli messing up with logs

lightning-cli: Malformed response '2019-09-04T18:34:56.936Z INFO plugin-probe.py RPC method 'probe' does not have a docstring.
'

Fixed by https://github.com/ElementsProject/lightning/pull/3062


Clarity on VirtualNetwork service tag inclusion for pod CIDR

Within the Network Security Groups section of this doc, there are guidelines made for supporting Azure CNI Overlay access rules.
It is unclear whether the VirtualNetwork service tag (doc link) includes the pod CIDR natively.
I would assume it does not; can this be confirmed and if so, added into the document as a clear statement?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 64f0ba71-573f-b860-1c2c-802a62cae120
Version Independent ID: 28990db8-fce7-c356-da6f-c052a9c201ee
Content: Configure Azure CNI Overlay networking in Azure Kubernetes Service (AKS) - Azure Kubernetes Service
Content Source: articles/aks/azure-cni-overlay.md
Service: azure-kubernetes-service
Sub-service: aks-networking
GitHub Login: @asudbring
Microsoft Alias: allensu

@jeffwmiles
Thanks for your feedback! We will investigate and update as appropriate.

@jeffwmiles
I'm going to assign this to the document author so they can take a look at it accordingly.
@asudbring
Could you Please review this and update as appropriate.

Thank you for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner, and we sincerely apologize for the delayed response. The requested updates have not been made since the creation of this issue, so we've created an internal work item to incorporate your suggestions. We are closing this issue for now, but feel free to comment here as necessary.
#please-close


chore: pim-dam to 1.0.21

chore: Promote pim-dam to version 1.0.21

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


The button should have the other link to transfer a user

Steps:
Bring a peer up
When peer is ready, there will be a notification with the peer description
Expected behavior: if you click it, you should be transferred to peers page.
Current behavior: user gets transferred to war room.
Comment from slack:

https://optdyn.slack.com/archives/C76FFHJRH/p1540541257047800

Disclaimer.
The master cc was not updated.
Now the button Bazaar is renamed to Peer and it leads to the peers page on Bazaar
Fixed in master

Repeated the steps with prod cc.
Fixed


Draw 'IKEA' style instructions

Draw 'Ikea' style instructions for the  following places of the application:

[x] Drawing instructions
[x] Scanning instructions
[x] Marker selection instructions
[ ] Turbine placement

(Bonus) improve text for all instructions (these have been written in quite a hurry without proper consideration of the current app)

[ ] place images in instruction pages

is placement in AR also necceary?


Error in config ``: invalid theme config

Recently I have been running into a reported zod error during validation of my custom theme config. The structure of my file is the following:
// @ts-check
/** @type {import('nextra-theme-docs').DocsThemeConfig} */
const config = {
  banner: { text: '...' },
  editLink: { component: () => null },
  feedback: { content: () => null },
  footer: { component: () => null },
  head: (<>{/* ... */}</>),
  logo: (<>{/* ... */}</>),
  navigation: { next: false, prev: false },
  search: { placeholder: '...' },
  titleSuffix: '...',
  toc: { float: true },
  useNextSeoProps: () => ({ titleTemplate: '...' })
};

module.exports = config;

Do you have any suggestions how I might debug this to deduce what's wrong with the config? To clarify, the offending key in the error message is literally an empty string:

The titleSuffix property I had was superseded by the useNextSeoProps as far as I can tell and was the cause for the error. Keeping this issue open though since the error message isn't really pointing out the correct key.

Yeah, error message should be fixed


Image preprocessing and normalization for training!

Hi,
Thanks for this incredible collection of abdominal data.
I was wondering what kind of preprocessing is used in experiments in paper (normalization, resolution or image size, if patchwise then input size). This information will greatly help everyone to relate their results to the one in the paper.
Thanks,
Sukesh

Hi @adigasu ,
Thanks for your interest.
All these settings are the default settings in nnUNet without any changes.
Best,
Jun


Add readme about debugging using pry remote

Description
Small readme change to include info about debugging.
Type of change

Documentation update

How Has This Been Tested?
Viewed the documentation.
Screenshots

@dorner good to go, thanks for feedback.

Thanks!


[Question]: T8520 Firmware 1.3.8 Support in the near future?

Ask your question
My eufy T8520 smart lock was recently upgraded to he <IP_ADDRESS> firmware (unwillingly).  I'm using the homeassistant integration, but noticed ever since the firmware upgrade the lock is unresponsive and continually logging an error: Tried all hosts, no connection could be established.

I also accidentally updated my T8520 to the latest firmware.. Surprisingly, I couldnt find a way to downgrade the firmware.
Hi @bropat , let me know if you need access to my door lock. I can try my best to help you get the access. Thanks.

To debug the cause i need access to the device. Could you please share your device with me (i need admin rights):

<EMAIL_ADDRESS>(for EU)
<EMAIL_ADDRESS>(for US)

Thanks!

To debug the cause i need access to the device. Could you please share your device with me (i need admin rights):

<EMAIL_ADDRESS>(for EU)
<EMAIL_ADDRESS>(for US)

Thanks!

My account is in "Australia" region. Am I still be able to share it?

I also have this issue, below is my log with some text replaced with xxxxxx
2022-08-08 13:44:36.311  INFO  Eufy Security server listening on host <IP_ADDRESS>, port 3000
2022-08-08 13:44:41.600  INFO  MQTT connection successfully established
2022-08-08 13:44:41.836  INFO  Successfully registered to MQTT notifications for lock T8520xxxxxxx
2022-08-08 13:44:43.836  INFO  Connected to station T8520xxxxxxxx on host xxxxxxxx and port xxxx
/usr/src/app/node_modules/eufy-security-client/build/http/station.js:315
return ((_a = this.rawStation.devices[0]) === null || _a === void 0 ? void 0 : _a.device_sn) === this.rawStation.station_sn;
^
TypeError: Cannot read properties of null (reading '0')
at Station.isIntegratedDevice (/usr/src/app/node_modules/eufy-security-client/build/http/station.js:315:50)
at EufySecurity.onStationCommandResult (/usr/src/app/node_modules/eufy-security-client/build/eufysecurity.js:1351:25)
at Station. (/usr/src/app/node_modules/eufy-security-client/build/eufysecurity.js:351:72)
at Station.emit (node:events:527:28)
at Station.onCommandResponse (/usr/src/app/node_modules/eufy-security-client/build/http/station.js:579:14)
at P2PClientProtocol. (/usr/src/app/node_modules/eufy-security-client/build/http/station.js:32:56)
at P2PClientProtocol.emit (node:events:527:28)
at P2PClientProtocol.handleData (/usr/src/app/node_modules/eufy-security-client/build/p2p/session.js:929:34)
at P2PClientProtocol.parseDataMessage (/usr/src/app/node_modules/eufy-security-client/build/p2p/session.js:891:26)
at P2PClientProtocol.handleMsg (/usr/src/app/node_modules/eufy-security-client/build/p2p/session.js:718:26)

To debug the cause i need access to the device. Could you please share your device with me (i need admin rights):

<EMAIL_ADDRESS>(for EU)
<EMAIL_ADDRESS>(for US)

Thanks!

My account is in "Australia" region. Am I still be able to share it?

Yeah.

@bropat I've sent the share request. Thanks!

@zuohaot
What is not working for you or what error are you getting?

@zuohaot
What is not working for you or what error are you getting?

I'm sorry Bropat. I think I was just dumb.. I thought <IP_ADDRESS> was not supported according the wiki page of known-working-device list. Now, I realised that my error was simply caused by low acoount permission rights..
Maybe we can update the list as <IP_ADDRESS> is supported now?
Anyhow, it is working now. Thanks for your excellent project!!

I can only get the baterry level... I'm not seeing the lock status. Are you seeing your lock status @zuohaot

I can only get the baterry level... I'm not seeing the lock status. Are you seeing your lock status @zuohaot

Yes, I can see the lock status, battery level as well as control the lock. I use Home Assistant with it btw. I also tried on Homebridge. It also worked.

Ahhh mine is working too now... Had to update to the latest version of the integration 🙂

@zuohaot
I have found an error in the emitting of the locked event. I will try to fix it tonight.
Can you keep the sharing going until tonight so I can test it?

@zuohaot
I fixed the locked event in the develop branch (will be released in the next version). You can remove the device sharing.
Thanks!

@zuohaot

Maybe we can update the list as <IP_ADDRESS> is supported now?

I have also added the firmware version <IP_ADDRESS> as tested in the documentation.
Note: The column Tested version means what firmware version was tested by me and certified as working. It does not necessarily mean not supported. ;) Not supported is when you find a red cross under the column Support status.


[SUPPORT] Best method to have Hudi process single stream from multiple source tables

We have a work flow that publishes data changes (CDC) from multiple tables to a single Kinesis stream. Each Kinesis record contains the schema and field values. Our Spark application is using Spark structured stream to query new data changes from Kinesis. The structured stream is using foreachBatch, which contains logic some logic to retrieve the schema and field values from the Kinesis records and determine the S3 output path. Within a single foreachBatch of a Spark structured stream, we are processing data from multiple source tables. Towards the end of foreachBatch, we use Hudi and save the Hudi-generated parquet files on S3.
Here is a general idea of how this works:

Spark queries Kinesis and retrieves a batch of data
Batch of data (Dataframe) enters foreachBatch block
foreachBatch logic will group the Dataframe by source tables. For each source table, then we retrieve schema information and create a new dataframe that represents the table being processed. This table dataframe is then processed by Hudi. The for-loop continues until all grouped/distinct tables within the batch is done.
Repeat steps 1-3 for next Spark batch.

Assuming it takes 2 minutes to process changed data from a single source table, if the Spark batch contains changed data from 10 tables, then it may take 20 minutes for the Spark batch to complete as the logic provided above will only process changed data 1 table at a time.
Another method I've read about is to publish changed data from a single table to a single Kinesis stream, but this approach is difficult to manage as we have thousands of tables. This approach would mean if we have 10,000 source tables, then we'll need to push to 10,000 Kinesis streams, then have separate Spark structured streams for each Kinesis stream. Basically a 1:1:1 of Source table : kinesis stream : spark stream.
Is there another way to speed up the process we have currently? Is it a good idea to implement threading in Spark foreachBatch and have it process multiple tables at the same time?
I'm open for thoughts :)

@WTa-hash : This sounds like a general spark streaming question. Is it possible to create separate threads within your foreatchbatch after you grouped dataframes and concurrently write to separate hudi tables ?
Another option is one spark-stream for each sink table but use the same kinesis stream. The spark stream has to filter records landing to the same sink table. This will be inefficient w.r.t network I/O but it will break the synchronization with other table writes.


read/write operations from disk from GPU/device arrays ?

Hi
I have a question or a feature request evaluation.
Consider we have a text file / json that can be read into a array of a structure.
Is there a way to read/write from this file directly into/from GPU arrays ? Is this a viable request ?
If this is possible, Can Kompute be developed to handle such a functionality ?
What I am asking is the ability to handle FILE IO using parallel operations into GPU/device arrays.
Kindly provide feedback.

I'm not sure I fully understand the usecase, if what you mean is to load the contents of a file and load it into GPU memory, however I would assume the contents of the file would include contents that could be mapped into a respective array, so this would require parsing into an int / float array, and then a kp::Tensor can be created with that data, which can then be copied into GPU memory. Does that make sense?

Ya. Sorry for not giving full details. Based on what you say, I still need CPU to load from disk to RAM before it can put to GPU array.
Either ways, thank you for the response. I will try some variations at my end and ask more relevant question.


feat: implement html minifier

Changes
Minify HTML output of @snowpack/plugin-webpack.
Testing
Tested locally.

Awesome! I love this.
Happy to merge this, but would you mind adding an option to the webpack plugin to disable this, in case it causes problems for someone? Something like
plugins: [
  ["@snowpack/plugin-webpack", { minifyHTML: false }]
]

Also be sure to add it this option to the README.md.

Awesome! Thanks so much for listening to feedback. This is looking good—just have one question before we merge on whether this is still able to be disabled.


sync to live

OPS Build status updates of commit 311b6cd:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

windows-iotcore/Troubleshooting.md
:white_check_mark:Succeeded

windows-iotcore/connect-your-device/SSH.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


bug: Invalid mode: 'normal'. Falling back to 'base' mode.

Did you check docs and existing issues?

[X] I have read all the flow.nvim docs
[X] I have searched the existing issues of flow.nvim

Neovim version (nvim -v)
0.10.3
Operating system/version
Ubuntu 24.10.1
Describe the bug
Transparency not working.
Steps To Reproduce
My config:
return {
  "0xstepit/flow.nvim",
  lazy = false,
  priority = 1000,
  tag = "v1.0.0",
  opts = {
    theme = {
      style = "dark", --  "dark" | "light"
      contrast = "default", -- "default" | "high"
      transparent = true, -- true | false
    },
    colors = {
      mode = "default", -- "default" | "dark" | "light"
      fluo = "orange", -- "pink" | "cyan" | "yellow" | "orange" | "green"
    },
    ui = {
      borders = "light", -- "theme" | "inverse" | "fluo" | "none"
      aggressive_spell = false, -- true | false
    },
    config = function(_, opts)
      require("flow").setup(opts)
    end,
  },
}

Expected Behavior
Background should be transparent.
It was working before flow updated to v2.0.0.

Hey @vaskark! I see you are using the new config of v2 but forcing Lazy to upload the v1. If you update the tag to v2.0.0 everything should work fine :)

Thanks for responding.
Unfortunately changing the tag to v2.0.0 didn't work.

I had to change border from light to fluo (for example). Then it worked.

 The "light" version is not available. As you can see it is not listed in the same line of possible options. Thanks for catching it

Issue fixed here: https://github.com/0xstepit/flow.nvim/commit/41a7810cdc357c22a723ba46aabc09f0ded8e2de thanks again!


Programming Exercises: Allow to publish example solutions for programming exercises on a specific date

Checklist
General

[ ] I tested all changes and their related features with all corresponding user types on a test server.
[ ] This is a small issue that I tested locally and was confirmed by another developer on a test server.
[ ] Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
[ ] I chose a title conforming to the naming conventions for pull requests.

Server

[x] I followed the coding and design guidelines.
[ ] I added multiple integration tests (Spring) related to the features (with a high test coverage).
[x] I added @PreAuthorize and checked the course groups for all new REST Calls (security).
[x] I implemented the changes with a good performance and prevented too many database calls.
[x] I documented the Java code using JavaDoc style.

Client

[x] I followed the coding and design guidelines.
[ ] I added multiple integration tests (Jest) related to the features (with a high test coverage), while following the test guidelines.
[ ] I added authorities to all new routes and checked the course groups for displaying navigation elements (links, buttons).
[ ] I documented the TypeScript code using JSDoc style.
[ ] I added multiple screenshots/screencasts of my UI changes.
[ ] I translated all newly inserted strings into English and German.

Changes affecting Programming Exercises

[ ] I tested all changes and their related features with all corresponding user types on Test Server 1 (Atlassian Suite).
[ ] I tested all changes and their related features with all corresponding user types on Test Server 2 (Jenkins and Gitlab).

Motivation and Context

Continuation PR of https://github.com/ls1intum/Artemis/pull/4328 to solve https://github.com/ls1intum/Artemis/issues/16. Allows students to download example solution repositories after a specified date.
Description

This allows the students to download the solution repository of a given programming exercise as a zip file when the Example Solution Publication Date is set and before now. Example Solution Publication Date is present in the BaseExercise class (and in the database) since https://github.com/ls1intum/Artemis/pull/4328 but it was not present for the Programming Exercise detail and editing page at the client so it is added to the Programming Exercise Lifecycle Component Different from the repositories downloadable by the instructors at the course detail page, student zip files will have their .git directories removed in order to not leak unintended information to the students.
Steps for Testing

Prerequisites:

1 Instructor
1 Student
1 Programming Exercise

Instructor:

Log in to Artemis
Navigate to Course Administration
Go to the Edit page of the prerequisite Programming Exercise.
Ensure current Example Solution Publication Date is empty or after current time.

Student:

Log in to Artemis
Navigate to the prerequisite Programming Exercise's detail page.
Ensure example solution is not visible.

Instructor:

Change Example Solution Publication Date of the exercise to a time before current time.

Student:

Ensure example solution is visible.
Ensure the term "Example solution" is used instead "Sample solution".

Instructor:

Import an existing Programming Exercise with Example Solution Publication Date set to a non-null date.
Ensure new exercise has Example Solution Publication Date set to null.

Review Progress

Code Review

[ ] Review 1
[ ] Review 2

Manual Tests

[ ] Test 1
[ ] Test 2

Test Coverage

Screenshots

Tested at testing session on ts3. I confirm that the feature works correctly.  👍🏻
No publication date added or it is set at a later date --> example solution not shown to participants
Date set to a previous time --> solution is shown

Thanks for the suggestion @canberkanar, I am already planning a follow-up PR about adding constraints to the example solution publication dates, I will include this change there 👍


trying to upgrade from tsd

Hello
I'm tring to migrate from tsd to typings.... I can't figure the workflow.
Is there a way to install typings definition using a bower.json file or should I need to maintain both bower and typings files each time I install a new dependency ?
Thanks

You need to maintain both, similar to TSD. Feel free to ask more questions.


Brainstorm: cw-storage-plus support when key can be derived from stored object

Based on a PR comment: https://github.com/CosmWasm/cosmwasm-plus/pull/109#discussion_r502891010
If we think in typical DBs, the primary key is part of the object being saved. We don't think of these separately. If we define objects like this, then we don't have to pass in the key in save, but can derive it from the saved object.
Thinking more, it is just that one method - load, may_load, update and remove all require the full key in any case.
@maurolacy if you still like this idea, can you develop it more. Once I started to flesh it out, I think it doesn't really have much use.

I don't think this is a good idea in the end, because the pk must be unique. Relying on the first data item either wouldn't guarantee uniqueness, or would force you to introduce an id or pk field at the beginning of it. Which will not only take space as data, but will also be equally or even more difficult to deal with, than specifying the pk directly.
I think the only way this could be an improvement, is if we implement automatic numerical ids, like the AUTO_INCREMENT feature of numerical fields, in relational databases. That would probably make sense, but it will require a good level of redesign. Perhaps even a different data structure.
So, I am of the opinion of closing this without changes.

Sounds good to me.
This could make a nice API, but would be a much different design than we have now.
Someone could build a storage-plus alternative if they wished to really make something like SQL indexes


Document features available in API version 2018-11-01-preview

According to GET https://management.azure.com/providers/?api-version=2017-05-10 there's a new resource version for the createSubscription operation:  2018-11-01-preview. So far the docs only refer to 2018-03-01-preview
                {
                    "resourceType": "CreateSubscription",
                    "locations": [
                        "Central US"
                    ],
                    "apiVersions": [
                        "2018-11-01-preview",
                        "2018-03-01-preview"
                    ]
                },

Can you update the documentation? How can I find out what has changed?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 371189c2-154f-2716-bbc0-8496a76c5ac9
Version Independent ID: bb5cd103-ff51-1753-f269-e2c1aa49a8c8
Content: Programmatically create Azure Enterprise subscriptions
Content Source: articles/azure-resource-manager/programmatically-create-subscription.md
Service: cost-management-billing
GitHub Login: @jureid
Microsoft Alias: jureid

Thanks for the comment. Feedback has been shared with the content author to review and revise the documentation as appropriate.
@jureid  - fyi.

#assign:bandersmsft

#assign:anuragdalmia

@JohannesRudolph We recently updated the article for the latest APIs.
#please-close


[BUG] Xdebug does not work

Describe the bug
Xdebug does not work when trying to debug the php application.
which Branch / PHP Version are you using ?
php8
Steps to reproduce
Steps to reproduce the behavior:

Clone sample.env to .env
Run in terminal: docker-composer up -d
Open VS Code
Install the Official PHP Debug Extension by Felix Becker
Open docker-compose-lamp directory in VS Code
Add the following launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Listen for Xdebug",
            "type": "php",
            "request": "launch",
            "port": 9003,
            "pathMappings": {
                "/var/www/html": "${workspaceFolder}/www"
            }
        },
        {
            "name": "Launch currently open script",
            "type": "php",
            "request": "launch",
            "program": "${file}",
            "cwd": "${fileDirname}",
            "port": 0,
            "runtimeArgs": [
                "-dxdebug.start_with_request=yes"
            ],
            "env": {
                "XDEBUG_MODE": "debug,develop",
                "XDEBUG_CONFIG": "client_port=${port}"
            }
        },
        {
            "name": "Launch Built-in web server",
            "type": "php",
            "request": "launch",
            "runtimeArgs": [
                "-dxdebug.mode=debug",
                "-dxdebug.start_with_request=yes",
                "-S",
                "localhost:0"
            ],
            "program": "",
            "cwd": "${workspaceRoot}",
            "port": 9003,
            "serverReadyAction": {
                "pattern": "Development Server \\(http://localhost:([0-9]+)\\) started",
                "uriFormat": "http://localhost:%s",
                "action": "openExternally"
            }
        }
    ]
}

Run and debug using these settings

Expected behavior
If you set a breakpoint, the debugger should stop at execution.
Desktop (please complete the following information):

OS: Windows 10
Browser: Chrome (version 96.0.4664.110 (Official Build) (64-bit))
Docker Version:

docker -v
Docker version 20.10.11, build dea9396

Additional context
The problem will still persist even if the port 9003 is explicitly exposed through docker-compose.yml

I have same problem with php 7.4, i solution was add lines in
Install xdebug
RUN pecl install xdebug-2.8.0
&& echo 'zend_extension='$(find /usr/local/lib/php/extensions/ -name xdebug.so)'\n
xdebug.mode=debug\n
xdebug.remote_enable=1\n
xdebug.remote_host=host.docker.internal\n
xdebug.remote_enable=1\n
xdebug.remote_autostart=1' >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
docker-php-ext-enable xdebug
in dockerfile from bin/php74
and folder config/php
memory_limit = 1G
post_max_size = 1G
upload_max_filesize = 1G
max_file_uploads = 500
max_execution_time=600
xdebug.remote_enable=1
xdebug.remote_host=<IP_ADDRESS>
xdebug.remote_port="9000"
xdebug.remote_autostart=1
xdebug.remote_connect_back=1
xdebug.idekey = 'xDOCKERx'
xdebug.var_display_max_depth = -1
xdebug.var_display_max_children = -1
xdebug.var_display_max_data = -1
in php.ini.
and you launch.json
"pathMappings": {
"/var/www/html/": "${workspaceFolder}"
}
maybe change sintax xdebug for xdebug 3

Hi,
we just added new code and better documentation to the project regarding xdebug.
Please let me know if your issue is still a thing, or if I can close this one :) Thank you

on php version 7.* the problem still persist on php 8 seems to work properly

I also have a problem with Xdebug.
I tried on PHP74 and PHP8 - not work.
Any advice?

I also have a problem with Xdebug. I tried on PHP74 and PHP8 - not work. Any advice?

Workaround - I jumped to the old commit 52defc2. With this version XDebug works.

The main issue with xdebug is in docker-compose file:
      XDEBUG_CONFIG: "client_host=host.docker.internal remote_port=${XDEBUG_PORT}"
    extra_hosts:
      - "host.docker.internal:host-gateway"

host.docker.internal - it is docker gateway IP, but VSCODE server listens 9003 port on webserver - not on gateway.
To fix it you should:
docker-compose down
remove an image of webserver (e.g lamp82-webserver):
docker rmi -f id_your_webserver_image
Comment string XDEBUG_CONFIG: "client_host=host.docker.internal remote_port=${XDEBUG_PORT}"   in bin/phpXX/Dockerfile
Change xdebug.client_host value in config/php/php.ini
xdebug.client_host=localhost
And rebuild stack:
docker-compose up -d


zbxtg in zabbix 5

Hi
I just tested new Zabbix media in telegram 5.0
it's awesome, thanks
I have a question, How can we use proxy in new media type? I mean in previous version, in zbxtg setting we could use proxy..how we can use this feature in new format?

So, you confirm that Zabbix-in-Telegram addon works on Zabbix 5 without issues? :)

As of now you cannot send images.
But Telegram via the webhook works like a charm for text messages as far as I can tell.

Will wait for the addon to be updated before jumping from Zabbix 4.0 to 5.0 as i like graph integration on telegram and its very useful to me, thanks!

@hardwareadictos no one mentioned that before.

@hardwareadictos no one mentioned that before.

Sorry @ableev , but mentioned what?

So, you confirm that Zabbix-in-Telegram addon works on Zabbix 5 without issues? :)

Yeah, for Text messages, It's Okay

So, you confirm that Zabbix-in-Telegram addon works on Zabbix 5 without issues? :)

Yeah, for Text messages, It's Okay

I mean about this add-on. I know zabbix has a new webhook for text messages. Are you referring to it?

Well guys, just updated my 4.0 LTS installation to 5.0 LTS and this addon is still working. I still receive my triggers on telegram with its graphs perfectly. Didn't have to reconfigure anything. Just implemented new server config.

Hm... that's strange, coz I get a "Timeout while executing a shell script" and don't know how to fix this.

Hm... that's strange, coz I get a "Timeout while executing a shell script" and don't know how to fix this.
Using 5.0 LTS.
Python3 (created alias and symlink to "python")
Any ideas?)

Only thing I did was update zabbix server and agents. Rear it I just adapted zabbix_server.conf just to have it inline with the conf I had on the old 4.0 LTS. It's not a false positive or coincidence. It's working fine for 5 hours now. Even I restarted all my server environment to test it. Working fine.
With the error you provided maybe it's something local on the server you are executing the python scripts.
Feel free to ask me whatever you need.

Oh.. the problem isn't so deep, as I think)
Previously I set up IPv6 for my Zabbix server, so when zbxtg.py was tried to resolve api.telegram.org, it's got a v6 address, instead of v4, but server drops any v6 connection except HTTP and HTTPS because I configured that.
So I just do:
ip6tables -I INPUT -s api.telegram.org --dports 88 -j ACCEPT
ip6tables -I INPUT -s api.telegram.org --dports 8443 -j ACCEPT
And all works great :)


Security Center ImportError: No module named 'httplib'

Objective:
Log in to Tenable Security Center 5.3.2
Using Python 3.4
Start a Nessus Scan
Python Script:
#from securitycenter import SecurityCenter5
#import securitycenter
#from securitycenter import Client
#sc = SecurityCenter5('https://777.777.7.77:8834')
#sc.login('goat','start^&+fGh--!end')
#response = sc.get('status')
#print('response=%s'%(response))
from securitycenter import SecurityCenter
sc = securitycenter.SecurityCenter('https://777.777.7.77:8834','goat','start^&+fGh--!end')
print('Finish')
Run python script
File "E:\tenable\sc.py", line 8, in
from securitycenter import SecurityCenter
File "C:\Python34\lib\site-packages\securitycenter_init_.py", line 1, in
from .sc4 import SecurityCenter4 as SecurityCenter
File "C:\Python34\lib\site-packages\securitycenter\sc4.py", line 6, in
import os, sys, ssl, random, httplib, logging, mimetypes, urllib2, calendar
ImportError: No module named 'httplib'
What do I need to correct in this script?
And do I need to add mod: from nessrest import ness6rest to start a Nessus Scan after I log into Security Center?
Thank You

There is no httplib in Python3. It's http.client. https://docs.python.org/3.4/library/http.client.html


Handle bad characters (ex. '?') in song name on Windows

I appreciate your PR but this won't allow us to create new directories using the file format option.
For example, when the file format is set as {artist}/{track_name}, earlier it would create a folder named as the artist's name and save the track in that folder but your PR would replace the / with a space, so this won't work now.
I am closing this in favor of #265 that would allow us to create new directories depending on the file format and fix special characters in the file name too.
Thanks once again!