Pubsub: Re-subscribe after reconnect

Currently, we store a list of channels but automatic re-subscribe after reconnect is not implemented.
For backward compat: If the user subscribes explicitly after reconnect, we can be OK'd without sending the command to Redis.
We need to store the type of subscribe for each channel (SUBSCRIBE or PSUBSCRIBE) to be able to re-issue the same command.

Hi, is there any progress on this issue?  We are still discussing but this is probably a must-have for us, because modifying the clients to consciously re-subscribe on reconnect is not preferred.

I haven't started working on it yet. Have you? :-)

Now when Redis 7.0 is released the SSUBSCRIBE is an new animal at the zoo.
I guess this is not used by you yet @e-kmisk ? or is it planned?

I'm starting working on resubscribe now.
@bjosv I guess sharded pubsub (SSUBSCRIBE, SPUBLISH) will have to be a different feature, in eredis or in eredis_cluster. It has not been requested by anyone so far, AFAIK.

Now when Redis 7.0 is released the SSUBSCRIBE is an new animal at the zoo.

The main use case for that is high throughput. Currently the components that use PubSub have very little messages/sec traffic, so sharding for those is not needed.
So far I could not find any good reports online about performance of PubSub and Sharded PubSub when it comes to messages/sec throughput and CPU/memory/network load. If we wanted to run something with high throughput using PubSub we would first need to benchmark regular PubSub internally, and then if that is too slow then also benchmark the sharded version. I don't think that will happen soon.

I guess sharded pubsub (SSUBSCRIBE, SPUBLISH) will have to be a different feature, in eredis or in eredis_cluster.

It would have to be in eredis_cluster since it's related to sharding and slots, I imagine. An SSUBSCRIBE may get a MOVED response, like regular DB reads/writes. Additionally if the shard/slot migrates later at runtime after initial subscription it would also need to be handled by the client.

Regarding the behaviour: Is it OK that messages on the form {subscribed, Channel, ClientPid} are delivered to the controlling process after reconnect? Currently, there is nothing that intercepts these messages. If we want seamless resubscribe, we'd have to intercept and silence those messages, i.e. have some flag for each channel we are auto-resubscribing to.
Regarding subscribing multiple times (e.g. automatic + manual) there is currently code and tests for this. The SUBSCRIBE and PSUBSCRIBE command is sent to the server, but the channel is not added to the state if it's already there. We could change that too if you prefer.
Any preferences here?

Is it OK that messages on the form {subscribed, Channel, ClientPid} are delivered to the controlling process after reconnect?

Looks like our current code handles {subscribed, ...} only by invoking
eredis_sub:ack_message(Client)

where Client is the Pid of the eredis_sub_client gen_server. Atm. I don't know if this behavior makes sense in our code, I don't understand fully what it's trying to do. If calling ack_message like that causes no problems then there's no problem to us if these are re-sent on reconnect, I guess.

Regarding subscribing multiple times (e.g. automatic + manual) there is currently code and tests for this. The SUBSCRIBE and PSUBSCRIBE command is sent to the server, but the channel is not added to the state if it's already there. We could change that too if you prefer.

The question is whether Redis itself ignores duplicate SUBSCRIBE commands. I could see two things happening - either the published messages arrive in duplicates too (because we subscribed N times), or Redis itself ignores such re-subscriptions for us as nonsense.
I don't think this is documented on redis side what happens. I would assume the latter is what happens, because the former would be pretty baffling. If the Redis side ignores duplicates, I guess there is no need to discard it on client side and we can rely on thir behavior.

It's good that your client does ack_message(Client) for {subscribe, _, _} messages. eredis_sub will not send any pubsub message until it is acked. All messages on channels and all subscribe messages need to be acked, otherwise they're queued in the eredis_sub_client. Only messages like eredis_connected are sent directly and don't need to be acked, but it does no harm to ack them too.
However, I wonder if it's OK for other users. Is it a backward-incompatible change? If it is, then maybe it's better to silently catch the subscribed messages. On the other hand, it may be good for the client to know that it may have missed some messages from the time it got {eredis_disconnected, Client} until it gets {subscribed, Channel, Client} after reconnect.
Another option is to add configuration to disable auto-resubscribe. But I don't know if it's needed. The messages sent by eredis_sub are not very well documented at all. I'm leaning towards letting these messages be forwarded to the client and perhaps document it.
Redis doesn't send duplicate messages, but also doesn't fail if you send another SUBSCRIBE, so it does no harm. However, we could avoid sending it from eredis sub client.

However, I wonder if it's OK for other users. Is it a backward-incompatible change?

Maybe not strictly, but I wonder how many users were even prepared properly for the reconnect + re-subscribe scenario so far? I know our code wasn't at least (ie. it doesn't re-subscribe on reconnect, it even discarded the {eredis_connected, ...} messages). I don't think it was intuitive how eredis behaved, so I also don't think it's bad to change the behavior.

Redis doesn't send duplicate messages, but also doesn't fail if you send another SUBSCRIBE, so it does no harm. However, we could avoid sending it from eredis sub client.

I doubt redis side will change this behavior so I suppose it's fine to rely on it.


스타일 가이드

https://github.com/ssafy-vue/kr.vuejs.org/blob/6df59ee23a7a90de973600aeab1dc2cf7afb53e2/src/v2/style-guide/index.md?plain=1#L12

May I try to translate this issue?

Sure!

May I try to translate this issue?

Sure!


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target/pull/976 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target.git

Docs Build status updates of commit 7c4f7bd:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-GitBranchDeletedOrForcePushed] Cannot sync git repo to specified commit because branch Release_Archive_master_2021-08-05-02-23-39 has been deleted or has been force pushed

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Refinement on argument based on tag not working properly: v0.69.0

Example
/* @flow */

type A = {
  type: { name: "String" },
  value: string,
};

type B =  {
  type: { name: "Number" },
  value: number,
};

function foo(x: A|B): string {
  if (x.type.name === "String") {
    return x.value;
  } else if (x.type.name === "Number") {
    return String(x.value);
  }
  return "uh oh";
}

Fails with the following error:
    15:     return x.value;
                   ^ Cannot return `x.value` because number [1] is incompatible with string [2].
        References:
        10:   value: number,
                     ^ [1]
        13: function foo(x: A|B): string {
                                  ^ [2]

But it's my understanding that if (x.type.name === "String") should refine the type in this case so that x.value must be a string.

expected, since these refinements only goes one-level deep


TypeError at /payment-url/

Hey, I'm getting this error:
TypeError at /payment-url/
'bool' object is not callable
/anaconda3/envs/myDjangoEnv/lib/python3.7/site-packages/paypal/pro/models.py in init, line 105
line 105:  if hasattr(request, "user") and request.user.is_authenticated():
any idea? :) thanks.

This is fixed on master, not sure when I'll get round to a release.


Issue precompiling Mosek v1.2.2

Dear Mosek developers,
I have recently encountered some issues when precompiling Mosek v1.2.2. More specifically, precompilation never stops (unless I interrupt it manually) and the compiled folder remains empty (i.e., no .ji file is generated). I can compile Mosek v1.1.3 without any problem, however.
Please see this issue for a more detailed discussion.
Best regards,
Mathias Berger

That is a bit strange. I will try to reproduce it, but I am not really sure that the issue might be.

I have not been able to reproduce the issue. Mosek.jl takes quite a long time precompiling, but certainly not hours.


Make vertical slider height responsive?

Describe the feature
I have the slider in a flexbox and would like it to scale vertically depending on the size of the container its inside. Is this doable?
Describe the solution you'd like
I would like to just be able to set style="height: 100%"
Additional context (If there is no relevant content, please delete the block)
Sorry, I didn't know whether to report this as a bug or feature. If it can't be done, could an explanation be added to the docs?

use height="100%"


Update config to enable docfx v3

AB#275107
This Pull Request is for docfxv3 migration, after the Pull Request merged, the repository will be built with docfxv3 instead of docfxv2.

Docs Build status updates of commit 3df6a9b:
:warning: Validation status: warnings

File
Status
Preview URL
Details

:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatHistoryQueryOptions.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoom.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatHistoryResult.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomFileDownloadJob.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomFileTransferProgressEventArgs.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomFileUploadJob.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomParticipant.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomSession.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomSettings.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomSessionStateChangedEventArgs.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/FormattedOutboundChatMessage.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat.Management/ChatRoomInformation.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/MessagePartEmoticon.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ParticipantAttendanceChangedEventArgs.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/MessagePartHyperLink.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat.Management/PersistentChatUser.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/PersistentChatServices.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat.Management/PersistentChatUserServices.xml
:warning:Warning

Details

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatMessage.xml
:warning:Warning

Details

.openpublishing.publish.config.json
:white_check_mark:Succeeded

Line 0, Column 0: [Warning-ECMA2Yaml_Moniker_EmptyAssembly] W:\sa34-s\officedocs-dev-lync-persistent\xml\FrameworksIndex\lync-persistent.xml have empty assembly.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatHistoryQueryOptions.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoom.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Open'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'null'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Private'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Hidden'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Open'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Private'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Hidden'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Private'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatHistoryResult.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomFileDownloadJob.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomFileTransferProgressEventArgs.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomFileUploadJob.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomParticipant.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomSession.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomSettings.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'null'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'null'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'False'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Inherit'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatRoomSessionStateChangedEventArgs.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/FormattedOutboundChatMessage.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat.Management/ChatRoomInformation.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Open'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Private'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Hidden'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'False'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'Inherit'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/MessagePartEmoticon.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: '-)'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ParticipantAttendanceChangedEventArgs.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/MessagePartHyperLink.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: '//www.microsoft.com'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat.Management/PersistentChatUser.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/PersistentChatServices.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'matchExactPhrase'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat.Management/PersistentChatUserServices.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.

officedocs-dev-lync-persistent/xml/Microsoft.Rtc.Collaboration.PersistentChat/ChatMessage.xml

Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'true'.
Line 0, Column 0: [Warning-xref-not-found] Cross reference not found: 'false'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Example proposal. Procedural sky as a post process.

Would you be interested in a procedural sky as an example. I can supply the shaders which are pretty simple and require almost no setup on the cpu only clip quad rendered at the far plane with the camera's view and proj matrices and with a less equal depth test and pretty much thats it, results in something like this:
https://cloud.githubusercontent.com/assets/1499411/22755670/d6cf794e-ee4c-11e6-92cd-e29ae30eefc8.png
https://cloud.githubusercontent.com/assets/1499411/22755669/d6cd60fa-ee4c-11e6-9ff2-ebf0e3535e39.png
shader code:
https://github.com/volcoma/EtherealEngine/blob/master/engine_data/shaders/vs_atmospherics.sc
https://github.com/volcoma/EtherealEngine/blob/master/engine_data/shaders/fs_atmospherics.sc

:+1: Yes, examples are always welcome.

My proposal is that someone else write it since I have a lot on my plate right now. I am providing the shader code and explaning the steps.
Step 1. Render your scene to a fbo with a depth buffer(may be the window one or a seperate) and then light it. A deferred way or forward.
Step 2. Bind the fbo and depth buffer and view, projection matrices from your camera and draw a fullscreen clip quad at the far plane, z = 1 with depth test = less equal, and the shaders i provided.
And thats basicly it.

Your link to vs_atmospherics.sc  seems to be broken.

https://github.com/volcoma/EtherealEngine/blob/master/engine_data/shaders/vs_clip_quad_ex.sc
Sorry i renamed it

You can link it at specific commit, like:
https://github.com/volcoma/EtherealEngine/blob/df979db5536ad2518ab2de8e77a579505a31f1be/engine_data/shaders/vs_atmospherics.sc

@volcoma Why do you have those .io files everywhere? Technically you should have one varying.def.sc for all shaders to define your varying semantics. Having multiple means you're changing varyings names. While there are no performance side effects, you might want to consolidate those to be easier to maintain it.

@bkaradzic In my case they are easier to maintain this way. This way it is a seperate entity not connected wtih anything so one can import a shader to his project without having to go and modify the varying.def.sc residing there. Its just internal reasons, no big deal :). I would even prefer it if they are defined in the shader file itself but... I really don't see the benefit from having every input output in the same blob file. For me it is definately not easier to maintain having multiple directories with shaders in it.

https://ebruneton.github.io/precomputed_atmospheric_scattering/
https://github.com/ebruneton/clear-sky-models

36-sky:
https://bkaradzic.github.io/bgfx/examples.html#sky


Internal: Create rider and customer users in Firestore

To be discussed with demo app development team, developers and QA, as relates closely to development workflows and test cases.
┆Issue is synchronized with this Jira Uncategorised by Unito

Document how these can be created manually via the Firebase Firestore Web UI.
Perhaps also document a proposed standard for usernames, aligned with typical QA scenarios.
See this internal Slack message, from discussion with @davyskiba.


Add ability to Pin Dev Environments

Summary of the pull request
Adding support to DevBox for pinning to the Start Menu and Taskbar
References and relevant issues
Detailed description of the pull request / Additional comments
This adds the ability to pin DevBox to the Start Menu or Taskbar.  The actual pinning is done by WindowsApp via a protocol association call from Devhome.
Validation steps performed
PR checklist

[ ] Closes #xxx
[ ] Tests added/passed
[ ] Documentation updated

Consider changing the title of the PR.

Just need to resolve the conflicts.


Use this inside a UICollectionView Controller?

Having a tough time trying to get the transition to work correctly when its in a UICollectionViewController .
I am calling the following perform segue within the collectionview "didSelectItemAt" method
 self.performSegue(withIdentifier: "SqrDetails", sender: self)

Just not able to to get the circle transformation to grow from the cell selected from the collection view!?

Hey! I'll take a look at this later today.

Also your calculation
    let centerX = UIScreen.main.bounds.width * 0.5
    let centerY = UIScreen.main.bounds.height * 0.5
    let centerOfScreen = CGPoint(x: centerX, y: centerY)

This doesn't take in account the navigation bar correct?

Hey! Sorry for the late reply. I think I might have a solution for you.
        // In viewDidLoad
        let tap = UITapGestureRecognizer(target: self, action: #selector(handleTap))
        self.collectionView?.addGestureRecognizer(tap)

    @objc func handleTap(recognizer: UITapGestureRecognizer) {

        if let cv = recognizer.view as? UICollectionView {

            let tapLocation = recognizer.location(in: cv)

            // To get a reference to the tapped cell, you can do this
            //let indexPath = cv.indexPathForItem(at: tapLocation)
            //let cell = cv.cellForItem(at: indexPath)

            self.performSegue(withIdentifier: "Segue", sender: tapLocation)
        }
    }

Note, I don't implement didSelectItem here at all, since it won't get called due to the gesture recognizer. That's why I also handle the cell inside the handleTap method.
Regarding your question about the navigation bar. Yeah, I think you are right. If you have a navigation bar in your project, you might need to offset the y-position with the height of it.


Filename of LMIC_node.h mixed case

In LMIC_node.cpp the file is included by #include "lmic_node.h", but it is actually named LMIC_node.h.
Might work on systems insensitive to file name case, but not on Linux.I had to change the line to #include "LMIC_node.h"

Good catch!
I missed it because I'm using Windows which has a case insensitive file system.
B.t.w. LMIC-node.cpp and LMIC-node.h have hyphens in their name (not underscores),
the Board Support Files use underscores to match the board-id's.
The issue has been fixed.
Thanks for your feedback!


Add support for printing the DAG

Usage: helmfile build --dag
Resolves:

https://github.com/roboll/helmfile/issues/1013
https://github.com/roboll/helmfile/issues/1916

@justindriggers please fix DCO.

Apologies, fixed

@justindriggers please fix the lint error.

@yxxhero Done. I'm not particularly experienced with Golang so please let me know if there's anything I can improve

@mumoshu ping

see: https://github.com/helmfile/helmfile/pull/1477


backend-defaults: refactor rootHttpRouterServiceFactory to define options on its own

Hey, I just made a Pull Request!
In preparation for #25570, this maintains the ability to create a factory using options, but doesn't do it via createServiceFactory. Since you won't be able to pass callback-based features to backend-like things anymore, this also provides an instance of the service factory with the default options merged into the factory function itself.
This is a competing solution to #25586 which avoids the breaking change. Perhaps the main downsides of this approach is that the definition of the service factory is a bit awkward, and it also doesn't encourage exporting building blocks for easy re-implementation. The main different between this and startRootHttpServer for that last point is that when using startRootHttpServer you have access to dependency injection, and the lack of that remains the biggest downside with this approach.
:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Shipping this through alongside #25585 so that we can have all service factories declared the same way in the release.


Move /partners to /partners/referral-sign-up

Description
Moved the existing partners page to be /partners/referral-sign-up and updated the background.
Related Issue(s)
closes #1059
Checklist

[x] I have read the contribution guidelines
[x] I have considered the performance impact of these changes
[ ] Suitable unit/system level tests have been added and they pass
[ ] Documentation has been updated

Can you update the footer link @Yndira-FlowForge?


fix: remove non-breaking-space

Replace more non-breaking-spaces with regular spaces

Thanks!
bors r+


feat: add TRY_TO_{DECIMAL,...}

Adds transformation for TRY_TO_DECIMAL and friends, basically the same as to_decimal except returned node is TryCast instead of Cast.
Moved the actual node creation to another function to reuse it in both *to_decimal transformation, but not 100% sure with what I end up with.
Keeping the same logic in both functions makes it more explicit and more isolated but prone to failures in a case that is not covered by tests, would like to hear what you think about it.

[!NOTE]
I'm working on a wrapper dbt adapter so users can write unit tests and run locally. Thank you for this awesome project as it really sped things up and kept my focus solely on query transformations.
I have a couple more transformation but not sure if all fits in the scope of the project as some are quite opinionated.  I wanted to open a separate PR for each transformation to keep it reviewable but it might get annoying for a maintainer. Please let me know your thoughts, like if you prefer them to be in a single PR.

Looks good to me!
And thank you for splitting each transformation into its own separate PR, it makes it a lot easier for me to review. Plus, because the changelog is generated automatically from merged commits, it ends up being much cleaner.


Support for CI Testing using Azure Pipelines/Docker

Summary

This PR is a feature
It adds support for CI testing of Axom using Azure Pipelines/Docker.
Currently enables testing on Linux, Windows, and Mac.
Linux testing uses following compilers: Clang 4,5,6 and GCC 6,7,8
Dockerfiles located in axom/scripts/docker were used to generate docker images stored on Docker Hub.  See README.

RE: docker in the host-config name.
@gzagaris -- I am pretty sure this is the name that uberenv generates.
In the short term, it is probably best to keep things as they are so we can just copy them without renaming. Longer term, perhaps we can modify uberenv to omit the docker- prefix in the generated host-config.

At this point, the only unresolved issue is that the sidre_lulesh tests are timing out on the gcc docker configurations, which I think is due to the interaction between MPI and OpenMP on a constrained azure resource (our node only has 10 threads, and the sidre_lulesh task is being called on 8 ranks with 2 threads each.
Oddly enough:

the slam_lulesh test is being called with 8 ranks and 4 threads each and is passing on the gcc configs
the same sidre_lulesh tests are passing on the clang configs

See: https://dev.azure.com/axom/axom/_build/results?buildId=877&view=ms.vss-test-web.build-test-results-tab
Since this isn't currently showing up as a test failure, I think we should merge the current PR and resolve this separately.  And also figure out how to get azure to indicate a "check" failure when some of the tests are failing.
Any objections to merging as is?

I created a ticket to track the open issues and suggested improvements for our CI plan: https://github.com/LLNL/axom/issues/162
Please add to it if I missed anything.

Thanks for creating the ticket @kennyweiss -- I agree we should merge this as is now and fine tune things in a subsequent PRs.

Thanks @keithhealy!

A couple of comments.

Uberenv adds the hostname to the beginning of the host config file name when it is created.  When you run an instance of a Docker image the hostname that is assigned is a random 12 digit alpha-numeric string.  I removed these and added 'Docker' because it was easier to remember (and wouldn't require changing the test scripts each time).  Removing it should not be a problem.
Azure does recognize that the sidre-lulesh tests fail and reports this fact in the published results.  It does not, however, mark the CI jobs as failed when there are failed tests. The rationale seems to be that the jobs have succeeded (in running the tests and publishing the results) regardless of the contents of those results. I believe there are two options to remedy this.
A. Fail the 'Build and Test' job when tests fail.  Then set the 'Publish Test Results' job to always run.  Normally a job failing will prevent subsequent jobs from running.  In this case we want the test results to be published even if the previous job fails due to failed tests.  Otherwise we get a notification of a failure but no record of which tests failed other than what's in the log.
B. Fail the 'Publish Test Results' job when tests fail.  This is probably the easiest solution even if it is slightly misleading.  That is, the job succeeded in publishing the test results but has been marked as failed due to a failed result.
Let me know what you think.

@keithhealy , if there is a test failure, we'd want to make sure that merging of the branch is not going to be allowed. Would this be the case with what you propose in 2.B?

@keithhealy -- I agree with @gzagaris -- We want "check" jobs to fail when one or more unit tests fail, or at the very least, we'd want the github "check" reason to indicate that there were failed tests.  Right now, the only way to see that a test failed is to click on the link that says "All tests passed" and click through a "Tests" tab.
Do we have flexibility over the name "Publish Test Results" ?  Or can we add a task after that named something like "Verify Test Results" ?

There is a way to define checks that must be passed before a pull request can be merged. This is located under Settings/branches of the repo.  I believe it should be possible to check for failing CI tasks.  I'll investigate.
As for flexibility over the tasks itself, the existing ones have a lot of optional settings.  For example, the PublishTestResults task has an option called 'failTaskOnFailedTests' which will fail the task if any tests fail.  For the most flexibility we could just add a script task which finds and parses the results file.  I don't think we could rename this though.

For example, the PublishTestResults task has an option called 'failTaskOnFailedTests' which will fail the task if any tests fail.

Thanks @keithhealy -- That sounds like the one we want.

It seems the docker run is not returning the resulting return code?
It may simplify things to have azure use the docker container to run the job, instead of running docker ourselves if there are are issues with it not propagating the return codes.
Some tests not failing is only one example, what about when a compile fails?
Note - for each script block, azure pipelines will only check the return code of the last executed task. This caught me off guard a bit back, so I wanted to mention it.
In my experience -- A ctest azure step should stop if any tests fail.  (ctest will return non-zero)
In fact if you look at Ascent,  we wanted to make sure to post test results even when ctest failed.  To do this, we had to use a condition: always() specifier to the PublishTestResults.
      - script: |
           #################################
           # test
           #################################
           # find spack installed cmake
           export ROOT_DIR=`pwd`
           export CMAKE_BIN_DIR=`ls -d ${ROOT_DIR}/uberenv_libs/spack/opt/spack/*/*/cmake*/bin`
           export PATH=${CMAKE_BIN_DIR}:$PATH
           which ctest
           cd build
           # run ctest
           ctest -T test --output-on-failure -V
        displayName: 'Run Unit Tests'
      - task: PublishTestResults@2
        inputs:
          testResultsFormat: 'cTest'
          testResultsFiles: '**/Test.xml'
        displayName: 'Unit Test Results'
        condition: always()

Yes -- having an indicator to easily know which actual stage has failed (build or test) is useful.
I took a look at the ascent setup. I like how it gives a summary of each of the jobs(or configurations) in the pull request. That way we can know whether the Windows or OS X build/ or tests have failed in the Pull Request page, without having to click on the "Details" to go to the azure-pipelines page.  Would that be easy to setup for Axom?

It should be straight forward, you just use top level stages + jobs.
Another nice thing to fall out of this is that you don't need a bunch of optional steps. The per platform steps are simply written for each platform specific job.

The compile steps are run through the 'or_die' function which will cause the script to exit with a non-zero return code if the compile fails.  The 'make test' step is not run through 'or_die' because I wanted to make sure the permissions was set correctly on the test results and that the 'PublisTestResults' task was always run.
As Cyrus mentioned, setting the 'always()' condition on PublishTestResults is one way to fix this.  Another is to set the PublishTestResults task to fail if any tests fail.
You can add checks that must be passed when merging by going to Settings/Branches for the repo and selecting 'Require status checks to pass before merging'.  Next click 'Add Rule' and click the checks you want to require before a merge is allowed.  You should get something like this (Scroll down to the merge pane):
https://github.com/keithhealy/MyAxom/pull/1

Sorry, that should be "going to Settings/Branches for the repo and selecting 'Add Rule' under Branch Protection Rules"

Thanks -- I think doing the same for Axom would be useful.


Setup RTONCipher

How can I setup RTONCipher for PopStudio.Console tool in Windows? it appears that PopStudio GUI support it but not console one, it doesn't has a input field for RTON Cipher.

Console version has not been tested. You can use GUI version.

Can you support it in the later version of the tool console?


Support learn the best direction for missing value

Refer to #122
will try following three treatments for missing values, and auto choose the best one.

min(feature_values)
0
max(feature_values)

How to use:

The zero feature values (-1e-20f < value <= 1e-20f) will be treated as missing values
You can just keep zero and missing value as zero at the same time. It will not have much effect on result.
If you really want to separate zero values and missing values, please convert zero values manually ( absolute value should greater than 1e-20f) .

@Laurae2 can you re-benchmark base on this ?

@guolinke I can try when I get time on my servers. I think you can expect some preliminary results during the weekend.
btw, xgboost taking half the time now for my benchmarks (not ended yet but most are very fast). I have to check later if LightGBM is slower now. I think only LightGBM v1 is potentially faster. v2 is slower.

@guolinke I was testing the new handling of missing values.  I could create test cases where missing was treated as 0 and as max(feature value), but not min(feature value).
In this example I have a single predictor x and a label y.  I fit a single tree LightGBM model and append the model predictions.  Based on the description, it seems like NA should be treated as min(feature) but it appears to be treated as 0.
Data:

x
y
prediction

NA
1
0.1192029

-9
1
0.7913915

-8
1
0.7913915

-7
1
0.7913915

-6
1
0.7913915

-5
0
0.7913915

-4
0
0.1192029

-3
0
0.1192029

-2
0
0.1192029

-1
0
0.1192029

lgb.dump:
"name":"tree",
"num_class":1,
"num_tree_per_iteration":1,
"label_index":0,
"max_feature_idx":0,
"feature_names":["x"],
"tree_info":[{"tree_index":0,"num_leaves":2,
"shrinkage":1,
"has_categorical":0,
"tree_structure":{
"split_index":0,
"split_feature":0,
"split_gain":6.6666666666666607,
"threshold":-4.5,
"decision_type":"no_greater",
"default_value":2.2250738585072014e-308,
"internal_value":0,
"internal_count":10,
"left_child":{
"leaf_index":0,
"leaf_parent":0,
"leaf_value":1.3333333333333324,
"leaf_count":6
Code:
library(lightgbm)

df <- data.frame(x = as.numeric(c(NA,-9:-1)), y = c(rep(1, 5), rep(0, 5)))
lgb.data <- lgb.Dataset(
  data = as.matrix(df["x"])
  ,label = df$y
  ,max_bin = 255
)
lgb.model <- lgb.train(
  data = lgb.data
  ,nround = 1
  ,num_leaves = 2
  ,min_data = 1
  ,learning_rate = 1
  ,obj = "binary"
)
writeLines(lgb.dump(lgb.model))
df$prediction <- predict(lgb.model, as.matrix(df))
df

@rgranvil in your case, treating as 0 is the same as treating as min, since they have no difference when comparing with threshold. (only have positive feature values.)
The only case that NA will be really treated as 0 is, there are negative feature values. In that case, treating as 0 is different with treating as min.

@rgranvil
From your model, the NA will be treated as 2.2250738585072014e-308, this actually is zero. And the prediction of NA is correct as well.

@guolinke I agree, but why would the missing value logic not treat NA as min(feature) in this case? The NA observation has a y value of 1, so I would think the "best" choice would be to assign NA to min(feature) with a prediction ~0.79 instead of zero with a prediction ~0.12.

@rgranvil Fixed in the latest commit. It has a bug for the -inf conversion.

Thanks @guolinke.  I reran my tests and it appears to be working correctly now.

Putting some results here coming from https://github.com/Laurae2/gbt_benchmarks/issues/1

Algorithm
Time (s)
Perf (AUC)
Best Iteration

xgboost depthwise
694.043
0.7194797
603

xgboost lossguide
696.006
0.7194797
603

LightGBM master
655.582
0.7180479
796

LightGBM v2
675.591
0.7146346
636

LightGBM v1
670.116
0.7167827
716

Speed with MinGW, not VS. There are issues with MinGW threading.

Hello. After this commit it's impossible to load old models and my trees have very strange behaviour. Is it possible to turn of default values or turn them all to zeros?

@qashqay654
you can disable it by set use_missing=false. However, The old model file cannot be loaded any more.

Is it possible to disable it in python?

@qashqay654
The same. You can set it to the parameters when using lgb.train or sklearn interface.


fix nb2md

A @call_parse function cannot currently call another @call_parse function. The function's inputs end up getting mangled when there are nested call parse functions.  This is what is causing the issue reported in #381
This fixes that by reorganizing the relevant code such that @callparse is only a wrapper around the functionality we want for this particular piece of code.   cc: @jph00
I have tested this to make sure it resolves the issue.

Another way to fix this would be to fix @call_parse somehow.  I had a go but got stuck on that - but fixing fastcore's callparse is another option.  @jph00 WDYT?


add oxidebpf description

I placed the description under aya, since that's another Rust library. But per the discussion in that PR (https://github.com/zoidbergwill/awesome-ebpf/pull/58#issuecomment-953790583), there may be a better place to put it.

I tightened up the description a bit. If it looks acceptable, I'd like to merge it in.


Fixes to PSRemoting and adding Get-LAPSPasswords Module

Fixed issue documented in issue #827.  When doing PSRemote from an non-administrator the check to identify Remote OS fails and caused an invalid CheckIn format.  On error this check now returns "Windows ?"

PSRemoting utilized Invoke-Command without the -AsJob flag, this caused the agent executing the module to hang until the new agent dies.

Added some obfuscation flags to PSRemoting

Added Get-LAPSPasswords Module from https://github.com/kfosaaen/Get-LAPSPasswords

Closed PR, just noticed -AsJob is not a PSv2 thing, working on a workaround.


weird Rewind issue

Bot works well but if you pause for a bit it will try to catch up
https://github.com/codetheweb/aoede/assets/3948904/d2a08add-a5b8-4489-a6b6-eaf796cdd1cf

Interestingly if i run it from cargo directly it does not seem to have the issue

I'm not experiencing it "rewinding" but there is quite a long period of Aoede continuing to play after it's been paused which seems like a bug.


⚠ Turnos 24x7 has degraded performance

In 49cfdcd, Turnos 24x7 (https://reservas-dev.alternativasinteligentes.com/) experienced degraded performance:

HTTP code: 200
Response time: 955 ms

Resolved: Turnos 24x7 performance has improved in 644653a.


Clean commit of character sequence encoder

I was originally holding off merging this until we sorted out the problem with character-level decoding (offer users full char-to-char MT).  However, that's an issue with existing code that we can work out separately.  I've gotten a couple of requests to try this with char-to-bpe which runs fine.  (In short, the issues aren't with this code, so nothing should block a merge)
The rest should look familiar to people who have seen this code before:
This is an initial implementation of the character-level encoder from Lee at al. (https://arxiv.org/pdf/1610.03017.pdf)  It uses convolutional, pooling, and highway layers to encode a sequence of character embeddings into a shorter sequence of segment embeddings.  It can be seen as an extra transformation on the input embedding sequence before the bidirectional RNN runs.  The output segment embeddings become the "real" source sequence that subsequent encoders, decoders, and attention mechanisms operate on, making it compatible with other architectures including the recently published convolutional encoder and transformer.
Other implementation notes to follow.

Regarding @davvil question to bucketing: as far as I can tell this should not be a problem since the INPUT to the graphs is still given by the input to the encoder and the modified sequence lengths / seq_len values are a function of the original input size.
So bucketing should not be affected here.

The bucketing is not "technically" affected, but if the inputs are characters and the output is bpe, the ratios will greatly change.

Yes, but this is not related to this PR and already handled by the recent bucketing update.

I updated the arguments and have get_encoder to take a ModelConfig instead of a bunch of args.  This should be easy to update further when we finish splitting up config objects as in Felix's branch.
It looks like the consensus is that having encoders return triples of (data, lengths, seq_len), effectively transformations of the three inputs, is a good idea.  More commits to follow.

The encode() method now always returns a triple corresponding to the encoded versions of the inputs: data, lengths, seq_len.  I ran some tests on toy data to make sure perplexity and output were identical to the master branch when not using the character sequence encoder (length propagation effectively a no-op with standard encoders).
This branch should now be a candidate for merging.

The last commit adds a unit test and addresses some of the cosmetic comments.

Thanks for iterating!


The command line for Windows 10 that is provided does not appear to do anything

When I run this locally, and then click on the Docker icon in my system tray, the "Switch to..." menu item does not change:
& $Env:ProgramFiles\Docker\Docker\DockerCli.exe -SwitchDaemon .
Having commands that allow you to identify the current daemon that the CLI is pointing to along with a working command that allows you to point the CLI to a different daemon is essential for automation. If you could provide these details in this article, it would be really helpful.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e09e1b22-1344-d7b9-9462-dc4c2e160e86
Version Independent ID: f6411e97-4419-1209-56ad-9140ceb37329
Content: Windows and Linux Containers on Windows 10
Content Source: virtualization/windowscontainers/quick-start/set-up-environment.md
Service: windows-containers
Product: windows-containers
GitHub Login: @cwilhit

That looks like a bug in Docker CLI: https://github.com/docker/for-win/issues/1269. That issue got stale and was closed by a bot, but I don't think it's been fixed. I'd suggest poking there.
As for telling which daemon the docker client is currently pointing at, you can determine this by querying docker version and looking for the OS/arch displayed under the server print out. It'll say whether it's Windows or Linux.

Thanks @cwilhit, that resolves this issue, and I've moved my discussion over to the stale/closed issue and re-opened it.


Feature: ability to set a release variable ("##vso[release.setvariable variable=varname;]$value")

General availability to set a release variable from within a powershell script path/ inline script would appeal to many.
Overview: I want RM to leverage a dynamic number of test agents for automated testing. The test agents are always contained by a known resource group, but quantity and names may change. For cost saving, the test agent VMs do not need to run outside of the testing that is orchestrated by VSTS RM.
My use case:

hosted phase 1: task to get list of vms from a resource group and set the list to a release variable
hosted phase 2: configure phase with Parallelism - Multi-configuration and set multipliers to vm list from phase 1. Phase creates "n" number of agents based on parsing the CSV vm string list. Phase passes 1/n vm names to each agent. Agent processes subsequent tasks per each target vm in parallel (like start/stop/configure).

@inverted360 - we currently do not support output variables across phases within an environment. As a workaround, could you instead store the list of VMs within release using REST API? This variable will then be available to the subsequent phases.

This blog talks about how you can update a variable in a release.
https://blogs.msdn.microsoft.com/aseemb/2016/05/23/how-to-update-release-level-variables-using-rm-rest-apis/

@bansalaseem any plan to support this?

Please enable this feature in Azure DevOps or equivalent in Github.

'support output variables across phases within an environment.'

Please enable this feature in Azure DevOps or equivalent in Github.
'support output variables across phases within an environment.'


disable mpi build for osx arm64

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

@conda-forge-admin, please restart ci

I don't think that changing the readme will make the changes that you want.
You need to add some more if statements to the build recipe. I'll look at these and see if I can make something sensible.

@douglowe Sorry for the confusion. I add the whole changes but I cannot get the pipeline to run. So I strip back to only include the readme.

@douglowe This PR shall now contain all the relevant files, but I cannot get the pipeline to work though.

I think you should get your solution to work if you use osx_arm64, rather than osx and arm64. Try that, and request a rerender.
The one issue there might be is that rerendering can often mangle the select statements just above this selection. So after rerendering do check to make sure that they look the same as before.

Ahh! I'd assumed that they would use osx_arm64, not just arm64 to indicate OSX on arm64 architecture.
So, ignore my first suggestion. Instead can you try deleting and osx (as this is redundant information), and only use mpi == 'openmpi' and arm64. It might have been that the logic engine was getting confused by two (possibly conflicting) OS select statements.

@conda-forge-admin, please rerender

@douglowe I guess it didn't make much difference. I wonder if you mind merge this PR? This PR shall be merged into Mark's PR so not to the main yet. I guess we can check if the pipeline works in Mark's PR.

That is strange. But yes, I agree that we should merge this into the other PR, and continue trying to debug it there.


ERA-5 remapping weights files flip longitude and latitude axes

My first test run with the https://github.com/COSIMA/1deg_era5_iaf repo has winds (and probably all the forcing fields) flipped in the longitude and latitude direction.
Jan 1981 zonal wind stress for ERA-5 (left) and JRA-55 (right);

This is likely a remapping weights issue, perhaps with the https://github.com/COSIMA/esmgrids/blob/master/esmgrids/era5_grid.py script?

Actually, it appears from the phantom Australia in the North Atlantic and Africa in the Pacific that the data is flipped only in latitude, and offset (not flipped) in longitude.
ERA5 files, e.g. /g/data/rt52/era5/single-levels/reanalysis/10u/1979/10u_era5_oper_sfc_19790101-19790131.nc, have the latitude running in reverse from 90 to -90, which this np.flipud is apparently intended to deal with, but that doesn't seem the right thing to do if the data is not also being flipped.
https://github.com/COSIMA/esmgrids/blob/514b560adf7b9094081408da49ecb3e19ef0353b/esmgrids/era5_grid.py#L17
ERA5 longitude starts at -180 (whereas JRA55-do starts at 0), which seems about right to explain the offset.

It looks like some reengineering of make_remap_weights.py and era5_grid.py is needed. Nic must have a version of make_remap_weights.py with the ERA5 options sitting around somewhere - but I can't find it. The changes look reasonably straightforward though - I can have a bash if no-one else is working on this?
For the latitude issue I could try just not flipping the latitude in era5_grid.py. For the longitude changes I could try modifying x_t in era5_grid.py. Otherwise understanding the ESMF_RegridWeightGen script will be neccessary.

@nichannah's weights in /g/data/ik11/inputs/access-om2/input_20210915/common_1deg_era5 appear to have been generated by /g/data/v45/nah599/access-om2/tools/make_remap_weights.*.
I've pushed these into this PR: https://github.com/COSIMA/access-om2/pull/258

CI tests failed on the PR. https://github.com/COSIMA/access-om2/runs/5964911440?check_suite_focus=true
I guess Nic didn't commit it because it had bugs?
I'll see if this is easily fixed.

fixed bug that caused CI to fail.

there are some tips on remapping here https://github.com/COSIMA/access-om2/wiki/Technical-documentation#creating-remapping-weights
and here https://github.com/COSIMA/access-om2/wiki/Tutorials#3-changing-the-oasis-remapping-files

might need to be careful what version of ESMF_RegridWeightGen we use - see https://github.com/COSIMA/access-om2/issues/216

Thanks @aekiss.
I've successfully generated remapping weights using module load esmf (which I think according to that issue should be ok for anything but the 1/10th-degree) without changing anything and am running a test (TBC).
If I try to remove the np.flipud on the latitude then a run into a check made in the make_corners routine in base_grid.py - it appears that the corner finding routines need latitude to be ordered south -> north.
We obviously don't want to derive new ERA-5 forcing files with latitude flipped. So it seems like the next step would be to rewrite the corner finding routines so that they work in both directions?

thanks @rmholmes, that sounds sensible.
FYI I've put an executable using Russ' tripole bug fix here:
/g/data/ik11/inputs/access-om2/bin/ESMF_RegridWeightGen_f536c3e12d
and updated make_remap_weights.* in https://github.com/COSIMA/access-om2/pull/258 to use it (as yet untested!)

Just a recap from my progress today:
If I remove the np.flipud in era5_grid.py and comment out the assert checks in make_corners (in base_grid.py) then the code succesfully creates remapping weight files, but nothing is changed. Similarly, if I add 90, 180 or 360 to x_t in era5_grid the files are created successfully but nothing is changed.
The code in esmgrids/base_grid.py seems apriori designed for dealing with latitude ordered south to north. Without this ordering, dy is negative which doesn't seem to be what is wanted. However, I don't have a good enough understanding of what the code is doing to figure out what should be done to fix things.
It is frustrating that the ERA-5 latitude data is ordered this way - doesn't seem to serve any practical purpose except to unneccesarily make things difficult.

Thanks for all these experiments. I'll raise it at today's TWG meeting.

Try this one
/scratch/v45/aph502/era5_mom10_wts.nc
I've tested it, and it gives the correct orientation.

Thanks @aidanheerdegen!!
That file has different variable names to the ones I've been working with (e.g. /g/data/e14/rmh561/access-om2/input/ERA-5/lat_flipud/ERA5_MOM1_patch.nc), so not sure it'll work out of the box?

Fixed now.

I've applied the same ncrename command to my weights file. Give it a burl.

Ok thanks. In setting this up I've found this issue, which I think invalidates all my checks yesterday.
But I can get around that now I know what it is. So I'll get back to you.

No luck, that gives me exactly the same result, again!

The run is at /home/561/rmh561/access-om2/1deg_era5_iaf if you want to check...
What did your testing consist of @aidanheerdegen? I feel like I'm missing something basic here...

Hey Ryan
The remapping file we're altering is for the OASIS coupler, i.e. it is the one defined in namcouple
$ grep INPUT namcouple
../INPUT/ERA5_MOM1_conserve.nc dst
../INPUT/ERA5_MOM1_conserve.nc dst
../INPUT/ERA5_MOM1_conserve.nc dst
../INPUT/ERA5_MOM1_conserve.nc dst
../INPUT/ERA5_MOM1_patch.nc dst
../INPUT/ERA5_MOM1_conserve.nc dst
../INPUT/ERA5_MOM1_patch.nc dst
../INPUT/ERA5_MOM1_patch.nc dst
../INPUT/ERA5_MOM1_patch.nc dst
../INPUT/ERA5_MOM1_patch.nc dst

The INPUT directory referred to there is, rather confusingly, the one in work/INPUT, which is the input directory referred to in the access-om2 model, i.e. the top level
# Model configuration
name: common
model: access-om2
input: /g/data/ik11/inputs/access-om2/input_20210915/common_1deg_era5/

Looking at the git diff of your control directory compared to the previous run:
$ git diff HEAD^
diff --git a/manifests/input.yaml b/manifests/input.yaml
index f3653c6..4c7ed1c 100644
--- a/manifests/input.yaml
+++ b/manifests/input.yaml
@@ -30917,20 +30917,30 @@ work/atmosphere/INPUT/2021/sp_era5_oper_sfc_20211101-20211130.nc:
     binhash: 87a70249955c71776c9759a701b0f6f3
     md5: ba7376809074b024f3cb8f92177365ee
 work/atmosphere/INPUT/ERA5_MOM1_conserve.nc:
-  fullpath: /g/data/ik11/inputs/access-om2/input_20210915/common_1deg_era5/ERA5_MOM1_conserve.nc
+  fullpath: /g/data/e14/rmh561/access-om2/input/ERA-5/ERA5_MOM1_conserve.nc
   hashes:
-    binhash: 9af5ca3052f6d5ed5d9139a23f121cf5
-    md5: 1bb0abf04369ea4ffe1d9da343d20f1f
+    binhash: b9d28101ccc3821fab8320579626812b
+    md5: e5c1c2ca736b4d394fdac397c46e7a21
 work/atmosphere/INPUT/ERA5_MOM1_patch.nc:
-  fullpath: /g/data/ik11/inputs/access-om2/input_20210915/common_1deg_era5/ERA5_MOM1_patch.nc
+  fullpath: /g/data/e14/rmh561/access-om2/input/ERA-5/ERA5_MOM1_patch.nc
   hashes:
-    binhash: 2bab214c23d2eebc6d368e003fb641e6
-    md5: 8352ce46dfb3cf6f1cc6b118b8b5dde2
+    binhash: e450e9fdae1056617f2363522d41c38e
+    md5: e5c1c2ca736b4d394fdac397c46e7a21
 work/atmosphere/INPUT/PET0.RegridWeightGen.Log:
   fullpath: /g/data/e14/rmh561/access-om2/input/ERA-5/PET0.RegridWeightGen.Log
   hashes:
     binhash: ffd3c47e6c1417cc8595e470ddae1fe3
     md5: 08c82ec3a08e47ccc8b2b5b29117d3bc
+work/atmosphere/INPUT/common_1deg_era5/rmp_jrar_to_cict_CONSERV.nc:
+  fullpath: /g/data/e14/rmh561/access-om2/input/ERA-5/common_1deg_era5/rmp_jrar_to_cict_CONSERV.nc
+  hashes:
+    binhash: 6563b5e196a4ccb9a189dc533893bbb2
+    md5: 10f0b5ea6b8102a03ad1140e5163a0f7
+work/atmosphere/INPUT/era5_mom10_wts.nc:
+  fullpath: /g/data/e14/rmh561/access-om2/input/ERA-5/era5_mom10_wts.nc
+  hashes:
+    binhash: a881ab6afd66ab659457c62abd66a7b9
+    md5: e5c1c2ca736b4d394fdac397c46e7a21
 work/atmosphere/INPUT/lat_flipud/ERA5_MOM1_conserve.nc:
   fullpath: /g/data/e14/rmh561/access-om2/input/ERA-5/lat_flipud/ERA5_MOM1_conserve.nc
   hashes:

given the checksum of the updated weights file:
$ md5sum /scratch/v45/aph502/era5_mom10_wts.nc
e5c1c2ca736b4d394fdac397c46e7a21  /scratch/v45/aph502/era5_mom10_wts.nc

The only place the updated weights file shows up is in work/atmosphere/INPUT.
So you'll need to change
input: /g/data/ik11/inputs/access-om2/input_20210915/common_1deg_era5/

to another location that has the updated remapping weights files.
I guess none of the changed remapping weights were used if the same error was made in your other runs. Maybe those remapping weights files were ok? Possibly worth checking.
As an aside, the ERA5 documentation states:

When you download ERA5 data you have the option to interpolate the data to a custom grid and horizontal resolution (eg. 'grid':'0.5/0.5'). The default interpolation method is bilinear for continuous parameters (e.g. Temperature) and nearest neighbour for discrete parameters (eg. Vegetation).

If you request data in NetCDF format ('format':'netcdf'), interpolation to a regular grid is mandatory, because ECMWF's NetCDF implementation only supports regular grids. Specify the desired lat/lon grid with the 'grid' keyword, for example 'grid':'0.25/0.25'.

the ECMWF interpolation software does not conserve area integrals

So the data we're using isn't conservatively interpolated from the native ERA5 model output, so there seems little point in us doing so, in which case I would recommend changing all those entries in namcouple to ERA5_MOM1_patch.nc.

I need a face palm emogy... Although I'm not taking the full blame here since that reference to common_1deg_era5 under atmosphere (on top of being in common) came from Nic's config files...
Thanks Aidan. I'll get on it.

Thanks @aidanheerdegen - interesting point about the non-conservative interpolation. The patch interpolation gives much smoother (so more physically reasonable) results, but I wonder whether it's justifiable to add to the conservation error by using a second non-conservative interpolation. Not sure.

It is justifiable to use conservation on fields that were never conserved? I think patch is more intellectually honest, and does not give a false impression.

My thinking here is that ideally we'd want global integrals of fields like precip to be identical to the ERA5 native output to have the same water balance (likewise for radiation). The non-conservative interpolation to NetCDF introduces an error we can't avoid, but an additional patch interpolation could take it even further from native ERA5. I don't have a feel for the size of the non-conservation error in patch though. Maybe it's negligible, in which case the smoothness of patch is certainly appealing. I wince every time I look at the blocky conservative JRA55-do forcing we apply to the 0.1deg model.

Ok I now have success:

This is using the patch remapping weights file at /g/data/e14/rmh561/access-om2/input/ERA-5/ERA5_MOM1_patch.nc. I generated this with ../../tools/make_remap_weights.py --accessom2_input_dir ../ --atm_forcing_file /g/data/rt52/era5/single-levels/reanalysis/2t/1980/2t_era5_oper_sfc_19800101-19800131.nc --atm ERA5 --ocean MOM1 --npes 1 in that directory. I used the access-om2 tools code in this branch, that do not include Russ's updated ESMF code described at https://github.com/COSIMA/access-om2/pull/258 - but I don't think that will be an issue until we go to the 1/10th.
@aidanheerdegen your file did not work for me, spitting out some CICE errors which I didn't investigate any further.
All I had to do was remove the np.flipud command in era5_grid.py and the assert statements in base_grid.py that checked the corners were correctly ordered as described in this PR. Is it worth altering these checks to do something sensible when latitude is flipped?
I am now proceeding with a 3 year run to compare with JRA55.


CLI docker help

I'm trying to run subliminal in a docker on my synology
This is the command I use:
docker run --rm --name subliminal -v subliminal_cache:/subliminal/cache -v /tvshows:/tvshows -it diaoulael/subliminal download -l nl "/video/Serie/Grimm/Season 06/Grimm.S06E01.720p.HDTV.x264-AVS.mkv"
This is the response:
docker: Error response from daemon: stat /tvshows: no such file or directory.
But I have no idea were the /tvshows:/tvshows stands for.
Generally: What is the command to run subliminal from a docker?

This is the correct command but you need to mount the appropriate volume inside the docker container. That's the -v /tvshows:/tvshows part. This will mount /tvshows on your NAS to /tvshows inside the docker container. I don't think this is appropriate in your case.
See the relevant help section: https://docs.docker.com/engine/tutorials/dockervolumes/
Given your command, I think this would work:
docker run --rm --name subliminal -v subliminal_cache:/subliminal/cache -v /video/Serie:/tvshows -it diaoulael/subliminal download -l nl "/tvshows/Grimm/Season 06/Grimm.S06E01.720p.HDTV.x264-AVS.mkv"

Thanks, this did the trick.
It will create a seperate container every time this command is called.
I'm now finetuning this to close the container and add the age and provider info.

This is not a problem to create a container each time as you replace the old one with the same name anyway (--rm). You can add the rest of the command at the end.
docker run --rm --name subliminal -v subliminal_cache:/subliminal/cache -v /video/Serie:/tvshows -it diaoulael/subliminal

Is just a replacement for plain old subliminal CLI, try --help ;)

I'm used to working with version 0.7.5 so this is a huge upgrade. (this is a compliment :-) )
For your information I found out that the -it arg works in cli, but you need the -i when using this in a cron

This issue can be closed it is working as it schould.
I have a task scheduler running that calls the script.
The command looks like this:
sudo docker run --rm --name subliminal -v subliminal_cache:/subliminal/cache -v /path/to/video:/tvshows -i diaoulael/subliminal download -l nl -a 2w -v -v "/tvshows/"


Fail installation script when there is issue with App Reg creations

Is your feature request related to a problem? Please describe.
Fail installation script when there is issue with App Reg creations
Describe the solution you'd like
Fail installation script when there is issue with App Reg creations
Describe alternatives you've considered
Fail installation script when there is issue with App Reg creations
Additional context
Add any other context or screenshots about the feature request here.

Hi, I can work on this, can you please share some details about in what phase or with what error did the script failed for you?

Even when the secerts are failed to create


Thanos not reading newly uploaded blocks

Thanos, Prometheus and Golang version used:
Thanos: 0.23.1
Prometheus: 2.30.3

Object Storage Provider:
Swift
What happened:
Thanos sidecar is uploading blocks to object storage, but I cannot query any new blocks uploaded after this past September. I can see newly uploaded blocks in our swift interface directly but thanos store does not seem to be indexing them or allow me to query them. I cannot see them with thanos tools either.
What you expected to happen:
Newly uploaded blocks to be indexed and be queryable, or there to be at least some sort of error in the logs pointing to an issue.
How to reproduce it (as minimally and precisely as possible):
Full logs to relevant components:
Last few days of thanos-sidecar logs
level=info ts=2021-12-06T03:00:28.110203518Z caller=shipper.go:337 msg="upload new block" id=01FP6TY3XN05B1CVJXR6M7421D
level=info ts=2021-12-06T05:00:28.109532504Z caller=shipper.go:337 msg="upload new block" id=01FP71SV5NVXDEFD1T9J0WQEF7
level=info ts=2021-12-06T07:00:58.109686297Z caller=shipper.go:337 msg="upload new block" id=01FP78NJE3BH8E466YHH913X3C
level=info ts=2021-12-06T09:00:58.108385896Z caller=shipper.go:337 msg="upload new block" id=01FP7FH9NNR3TF8BCXFXCF94YV
level=info ts=2021-12-06T11:00:58.117311758Z caller=shipper.go:337 msg="upload new block" id=01FP7PD0XNMWPW2K7XSVHMN4N4
level=info ts=2021-12-06T13:00:28.310070192Z caller=shipper.go:337 msg="upload new block" id=01FP7X8R5NP3AEBBBV53GCFNN0
level=info ts=2021-12-06T15:00:28.108354385Z caller=shipper.go:337 msg="upload new block" id=01FP844FDNN0EHSV37JYC77FE6
level=info ts=2021-12-06T17:00:28.113764457Z caller=shipper.go:337 msg="upload new block" id=01FP8B06NS55PR45G4S9X4ACA9
level=info ts=2021-12-06T19:00:58.10643592Z caller=shipper.go:337 msg="upload new block" id=01FP8HVXXNNWRHKGD03KVN4RWR
level=info ts=2021-12-06T21:00:28.109912427Z caller=shipper.go:337 msg="upload new block" id=01FP8RQN5NXM14X7KV8BEAKGF3
level=info ts=2021-12-06T23:00:58.117373136Z caller=shipper.go:337 msg="upload new block" id=01FP8ZKCDN60YGW0SM4V8N65PJ
level=info ts=2021-12-07T01:00:58.1994395Z caller=shipper.go:337 msg="upload new block" id=01FP96F3NN42G3Z8HQWSAA56ZB
level=info ts=2021-12-07T03:00:28.10884307Z caller=shipper.go:337 msg="upload new block" id=01FP9DATXMWAPWWJSJRPMQ2Y33
level=info ts=2021-12-07T05:00:58.108401396Z caller=shipper.go:337 msg="upload new block" id=01FP9M6J5NPX5NFTSE83EQHEQ5
level=info ts=2021-12-07T07:00:28.109938466Z caller=shipper.go:337 msg="upload new block" id=01FP9V29DRQJ69DFBGBK19X8TM
level=info ts=2021-12-07T09:00:28.112379167Z caller=shipper.go:337 msg="upload new block" id=01FPA1Y0NP7PGW74HRT5S2A2WT
level=info ts=2021-12-07T11:00:28.110892707Z caller=shipper.go:337 msg="upload new block" id=01FPA8SQXMW5S5A1GGWZWR9J7G
level=info ts=2021-12-07T13:00:58.212321497Z caller=shipper.go:337 msg="upload new block" id=01FPAFNF5PNMHY2FJSB7ZK52DR
level=info ts=2021-12-07T15:00:58.126902387Z caller=shipper.go:337 msg="upload new block" id=01FPAPH6DNE4RXJ40BN4MN11BE
level=info ts=2021-12-07T17:00:58.107641604Z caller=shipper.go:337 msg="upload new block" id=01FPAXCXNM2EKDP4DPGVFWV9NZ
level=info ts=2021-12-07T19:00:58.107148304Z caller=shipper.go:337 msg="upload new block" id=01FPB48MY1WASDFWMEZF8KJ5QC
level=info ts=2021-12-07T21:00:28.11156975Z caller=shipper.go:337 msg="upload new block" id=01FPBB4C5NZGXFQ19J01NFHZQP
level=info ts=2021-12-07T23:00:58.120359257Z caller=shipper.go:337 msg="upload new block" id=01FPBJ03DPPRAH11RXTQDYMYBD
level=info ts=2021-12-08T01:00:58.215836714Z caller=shipper.go:337 msg="upload new block" id=01FPBRVTNPPHFXGSXTHS4TA9XD
level=info ts=2021-12-08T03:00:28.108025135Z caller=shipper.go:337 msg="upload new block" id=01FPBZQHXNX2AD0ZD4RFQRDERG
level=info ts=2021-12-08T05:00:28.108162267Z caller=shipper.go:337 msg="upload new block" id=01FPC6K95N6J8CE369P7KP7MP2
level=info ts=2021-12-08T07:00:58.10767824Z caller=shipper.go:337 msg="upload new block" id=01FPCDF0DS17DPSTJWQFTED5HK
level=info ts=2021-12-08T09:00:28.107857791Z caller=shipper.go:337 msg="upload new block" id=01FPCMAQNNXFW1187704D7HJ32
level=info ts=2021-12-08T11:00:28.118519438Z caller=shipper.go:337 msg="upload new block" id=01FPCV6EXNPH7YD8HPTSNMCEPZ
level=info ts=2021-12-08T13:00:58.38082914Z caller=shipper.go:337 msg="upload new block" id=01FPD2265NYJKNZS1B6HK0ZY2N
level=info ts=2021-12-08T15:00:58.108467506Z caller=shipper.go:337 msg="upload new block" id=01FPD8XXDNG0GD41SED6883D1V

thanos-store logs repeat block metadata sync with same number of blocks
level=info ts=2021-12-03T17:24:26.2140222Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=541.723393ms duration_ms=541 cached=206 returned=206 partial=0
level=info ts=2021-12-06T18:12:26.210296479Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=538.04282ms duration_ms=538 cached=206 returned=206 partial=0
level=info ts=2021-12-07T19:45:26.235772458Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=560.988617ms duration_ms=560 cached=206 returned=206 partial=0
level=info ts=2021-12-08T15:45:26.12836438Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=456.659276ms duration_ms=456 cached=206 returned=206 partial=0

"thanos tools bucket inspect" shows no new blocks since September:
level=info ts=2021-12-08T14:51:11.951671219Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2021-12-08T14:51:12.566868659Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=494.223249ms duration_ms=494 cached=206 returned=206 partial=0
| 01FEJHHR8JTWEX9XAFVJ6FMH6K | 19-08-2021 00:00:00 | 02-09-2021 00:00:00 | 335h59m59.979s | -295h59m59.979s | 3,427,072 | 96,755,734,160 | 806,988,550 | 4          | false       | environment=Production,prometheus_server=thanos-a-1.phl,service=Prometheus | 0s         | compactor |
| 01FEJPM4JV0ZEY3SPPXH8Y0ABW | 19-08-2021 00:00:00 | 02-09-2021 00:00:00 | 335h59m59.979s | -95h59m59.979s  | 3,427,054 | 11,947,448,239 | 86,376,862  | 4          | false       | environment=Production,prometheus_server=thanos-a-1.phl,service=Prometheus | 5m0s       | compactor |
| 01FEJX79VYMM5C5MG0AXZ81VG3 | 19-08-2021 00:00:00 | 02-09-2021 00:00:00 | 335h59m59.979s | -               | 3,427,054 | 997,332,376    | 12,297,715  | 4          | false       | environment=Production,prometheus_server=thanos-a-1.phl,service=Prometheus | 1h0m0s     | compactor |
| 01FEJNZQDVB592GEV253NEVD0W | 19-08-2021 00:00:02 | 02-09-2021 00:00:00 | 335h59m57.86s  | -295h59m57.86s  | 7,851     | 165,276,144    | 1,412,738   | 4          | false       | prometheus_server=thanos-a-1.phl-rule                                      | 0s         | compactor |
| 01FEJWXDQY7DQ63XH136YCKTPN | 19-08-2021 00:00:02 | 02-09-2021 00:00:00 | 335h59m57.86s  | -95h59m57.86s   | 7,851     | 16,578,072     | 122,515     | 4          | false       | prometheus_server=thanos-a-1.phl-rule                                      | 5m0s       | compactor |
| 01FEJZ74B5HE0C6ZXM3KNXNAPN | 19-08-2021 00:00:02 | 02-09-2021 00:00:00 | 335h59m57.86s  | -               | 7,851     | 1,420,543      | 20,132      | 4          | false       | prometheus_server=thanos-a-1.phl-rule                                      | 1h0m0s     | compactor |
| 01FFPJZ84RRF2GCDD5CMNJ64FP | 02-09-2021 00:00:00 | 16-09-2021 00:00:00 | 335h59m59.979s | -295h59m59.979s | 3,400,870 | 97,098,335,993 | 809,920,519 | 4          | false       | environment=Production,prometheus_server=thanos-a-1.phl,service=Prometheus | 0s         | compactor |
| 01FFPR0VMMDR6TDMTVSZYBAMZ5 | 02-09-2021 00:00:00 | 16-09-2021 00:00:00 | 335h59m59.979s | -95h59m59.979s  | 3,400,852 | 12,002,665,414 | 86,740,615  | 4          | false       | environment=Production,prometheus_server=thanos-a-1.phl,service=Prometheus | 5m0s       | compactor |
| 01FFPZ6C55PK4YTQAE4ZCY5R3W | 02-09-2021 00:00:00 | 16-09-2021 00:00:00 | 335h59m59.979s | -               | 3,400,852 | 1,001,912,047  | 12,322,824  | 4          | false       | environment=Production,prometheus_server=thanos-a-1.phl,service=Prometheus | 1h0m0s     | compactor |
| 01FFPQBTDGV4VT5NMQSHKA5PM5 | 02-09-2021 00:00:02 | 16-09-2021 00:00:00 | 335h59m57.86s  | -295h59m57.86s  | 7,020     | 165,228,294    | 1,409,521   | 4          | false       | prometheus_server=thanos-a-1.phl-rule                                      | 0s         | compactor |
| 01FFPYWMW0WXFNJ1HPR5CH0QKP | 02-09-2021 00:00:02 | 16-09-2021 00:00:00 | 335h59m57.86s  | -95h59m57.86s   | 7,020     | 16,582,692     | 121,682     | 4          | false       | prometheus_server=thanos-a-1.phl-rule                                      | 5m0s       | compactor |
| 01FFQ1A69PAM9QXK3ZAR3F5GHD | 02-09-2021 00:00:02 | 16-09-2021 00:00:00 | 335h59m57.86s  | -               | 7,020     | 1,417,131      | 19,269      | 4          | false       | prometheus_server=thanos-a-1.phl-rule                                      | 1h0m0s     | compactor |
| 01FGTMZ87NGDD1Y58K2VRQAVC4 | 16-09-2021 00:00:00 | 30-09-2021 00:00:00 | 335h59m59.979s | -295h59m59.979s | 3,412,627 | 97,466,563,770 | 812,981,973 | 4          | false       | environment=Production,prometheus_server=thanos-a-1.phl,service=Prometheus | 0s         | compactor |
| 01FGTV1TDPMMQ483C9SXBGTVSH | 16-09-2021 00:00:00 | 30-09-2021 00:00:00 | 335h59m59.979s | -95h59m59.979s  | 3,412,609 | 12,052,502,447 | 87,094,627  | 4          | false       | environment=Production,prometheus_server=thanos-a-1.phl,service=Prometheus | 5m0s       | compactor |
| 01FGV2N00S9JQ5WMB61JNBVAT1 | 16-09-2021 00:00:00 | 30-09-2021 00:00:00 | 335h59m59.979s | -               | 3,412,609 | 1,006,234,608  | 12,368,039  | 4          | false       | environment=Production,prometheus_server=thanos-a-1.phl,service=Prometheus | 1h0m0s     | compactor |
| 01FGTT9N9J2G68H8MTNR0N7S9C | 16-09-2021 00:00:02 | 30-09-2021 00:00:00 | 335h59m57.86s  | -295h59m57.86s  | 26,302    | 542,236,317    | 4,558,766   | 4          | false       | prometheus_server=thanos-a-1.phl-rule                                      | 0s         | compactor |
| 01FGV2A8C2VVTED6RFDB2R6979 | 16-09-2021 00:00:02 | 30-09-2021 00:00:00 | 335h59m57.86s  | -95h59m57.86s   | 26,302    | 54,289,362     | 395,947     | 4          | false       | prometheus_server=thanos-a-1.phl-rule                                      | 5m0s       | compactor |
level=info ts=2021-12-08T14:51:12.585586767Z caller=main.go:160 msg=exiting

root@thanos-a-1:~/thanos-0.23.1.linux-amd64# ./thanos tools bucket inspect --objstore.config-file=/etc/prometheus/thanos_objstore.yml | grep 10-2021
level=info ts=2021-12-08T14:52:45.600288241Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2021-12-08T14:52:46.361983507Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=463.397051ms duration_ms=463 cached=206 returned=206 partial=0
level=info ts=2021-12-08T14:52:46.384084705Z caller=main.go:160 msg=exiting

root@thanos-a-1:~/thanos-0.23.1.linux-amd64# ./thanos tools bucket inspect --objstore.config-file=/etc/prometheus/thanos_objstore.yml | grep 11-2021
level=info ts=2021-12-08T14:53:20.663422996Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2021-12-08T14:53:21.279779032Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=494.714997ms duration_ms=494 cached=206 returned=206 partial=0
level=info ts=2021-12-08T14:53:21.301678007Z caller=main.go:160 msg=exiting

root@thanos-a-1:~/thanos-0.23.1.linux-amd64# ./thanos tools bucket inspect --objstore.config-file=/etc/prometheus/thanos_objstore.yml | grep 12-2021
level=info ts=2021-12-08T14:53:51.612662217Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2021-12-08T14:53:52.347299755Z caller=fetcher.go:476 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=492.484166ms duration_ms=492 cached=206 returned=206 partial=0
level=info ts=2021-12-08T14:53:52.36651175Z caller=main.go:160 msg=exiting

Newly uploaded blocks in my swift interface

Anything else we need to know:
Prometheus and all thanos services run on one host
Command strings:
Prometheus
/usr/bin/docker run --rm --name prometheus --network host --volume /etc/prometheus/:/etc/prometheus/ --volume thanos-data:/opt/bitnami/prometheus/data bitnami/prometheus:2.30.3 --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.max-block-duration=2h --storage.tsdb.min-block-duration=2h --web.enable-lifecycle --web.enable-admin-api --web.external-url=http://thanos-a-1.phl/prometheus --web.route-prefix=/

Thanos Sidecar
/usr/bin/docker run --rm --name thanos-sidecar --network host --volume /etc/prometheus/thanos_objstore.yml:/thanos_objstore.yml --volume thanos-data:/data bitnami/thanos:0.23.1 sidecar --tsdb.path=/data --prometheus.url=http://localhost/prometheus --grpc-address=<IP_ADDRESS>:19100 --http-address=<IP_ADDRESS>:19200 --objstore.config-file=/thanos_objstore.yml

Thanos Rule
/usr/bin/docker run --rm --name thanos-rule --network host --volume /etc/prometheus/:/etc/prometheus/ --volume thanos-data:/data bitnami/thanos:0.23.1 rule --data-dir=/data/thanos/rule --grpc-address=<IP_ADDRESS>:19104 --http-address=<IP_ADDRESS>:19204 --objstore.config-file=/etc/prometheus/thanos_objstore.yml --rule-file=/etc/prometheus/alerts/thanos_rule/*.yml --rule-file=/etc/prometheus/rules/thanos_rule/*.yml --alert.query-url=http://thanos-a-1.phl/thanos --alertmanagers.url=http://localhost:9093 --label=prometheus_server="thanos-a-1.phl-rule" --alert.label-drop=prometheus_server --web.external-prefix=/thanos-rule --web.route-prefix=/thanos-rule --query=localhost:19090/thanos

Thanos Store
/usr/bin/docker run --rm --name thanos-store --network host --volume /etc/prometheus/thanos_objstore.yml:/thanos_objstore.yml --volume thanos-data:/data bitnami/thanos:0.23.1 store --data-dir=/data/thanos/store --grpc-address=<IP_ADDRESS>:19101 --http-address=<IP_ADDRESS>:19201 --objstore.config-file=/thanos_objstore.yml

Thanos Query
/usr/bin/docker run --rm --name thanos-query --network host bitnami/thanos:0.23.1 query --query.replica-label=prometheus_server --grpc-address=<IP_ADDRESS>:19102 --http-address=<IP_ADDRESS>:19090 --web.external-prefix=/thanos --store=localhost:19100 --store=localhost:19101 --store=localhost:19104

Thanos Compact
/usr/bin/docker run --rm --name thanos-compact --network host --volume /etc/prometheus/thanos_objstore.yml:/thanos_objstore.yml --volume thanos-data:/data bitnami/thanos:0.23.1 compact --data-dir=/data/thanos/compact --http-address=<IP_ADDRESS>:19203 --objstore.config-file=/thanos_objstore.yml --retention.resolution-raw=548d --retention.resolution-5m=548d --retention.resolution-1h=548d --wait

Is sidecar uploading new blocks to the same bucket that Thanos store is using?

Yes, sidecar (and rule) is uploading new blocks to the same object store that thanos store is looking. All thanos services that require an object store configuration share the same configuration file to ensure that they are all pointed to the same place.
Additionally, I managed to find a workaround that got thanos store accepting new blocks again, but it may not be viable for some setups. I created a new object storage container and pointed our object storage configuration at it. I created a new docker volume for all prometheus and thanos data and copied all prometheus/thanos data over to it while ommitting all "thanos.shipper.json" files so that thanos-sidecar would upload all existing prometheus and thanos-rule blocks to the new swift container. Essentially this created a fresh installation of thanos while retaining all data from the last 15 days. I could see that thanos-sidecar uploaded all available blocks and thanos-store indexed them without issues. I then spun up a second instance of thanos store that was pointed at the old swift container and added it to the thanos-query configuration in order to retain as much of our historical data as possible for queries. This new setup has run well for the past few days.
Based upon this, I suspect that thanos-store is choking on something in our old object storage container, but I cannot see what the problem is from the available logs. Here's the full initialization sequence with the log level in debug mode from the store pointed at our old data. It should be finding many more blocks than it is.
store.log

We had a similar issue. Our metrics are stored in a ceph-rgw and accessed by thanos-store via swift. We observed that the block count according to thanos-store was stuck at 808 while we found that new blocks were still uploaded to the bucket (accumulating to ~3000 blocks). Thanks to @lpashos's hint we added some debug output to fetchMetadata in fetcher.go to find the "choke point", but to no avail. After switching to a S3 based connection to the same ceph-rgw, thanos-store fetched all blocks as expected.
TL;DR: Switching from Swift to S3 solved the issue for us.
thanos-store v0.26.0
(We don't know the version of the remote ceph-rgw, unfortunately)
CC: @expektorans


formal v11 support

The module works in v11 (for me) but the module.json still has the verified compatibility version on v11.
What's needed to be confident enough to up that to v11?

Works for me. v1.7 should support Foundry 11.

Would it be possible to update the module manifest so that the Foundry website and The Forge's Bazaar shows v11 as a supported version?

The module config already has that information...

But I updated the admin info (that's being deprecated), so maybe that'll fix it.

That fixed the issue! Thanks!


Missed nightly feature proc_macro_byte_character

Crate: 1.0.80
Rustc: 1.79.0-nightly (8df7e723e 2024-03-30)
error[E0658]: use of unstable library feature 'proc_macro_byte_character'
   --> /Users/u/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.80/src/wrapper.rs:871:21
    |
871 |                     proc_macro::Literal::byte_character(byte)
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: see issue #115268 <https://github.com/rust-lang/rust/issues/115268> for more information
    = help: add `#![feature(proc_macro_byte_character)]` to the crate attributes to enable
    = note: this compiler was built on 2024-03-30; consider upgrading it if it is out of date

I suppose there should be crate attribute:
#![cfg_attr(not(no_literal_byte_character), feature(proc_macro_c_str_literals))]

Probably this is not a problem because proc_macro_c_str_literals was stabilised nearly, so 🤷🏻♂.

This being missing makes me unable to use arduino-hal templates with cargo generate because their dependencies use this crate through quote, and using stable is not an option for this. I'm trying to figure out how to force transitive dependencies to use 1.0.35 of quote or 1.0.79 of proc-macro2, but simply specifying it for cargo with a cargo update -p proc-macro2 --precise 1.0.79 does not do it.

Actually, thinking about it, the version in rust-toolchain.toml is probably infered from current latest installed nightly, and I updated nightly only after generating the template today. Sorry for the confusion.

@dtolnay Just to provide some context, the reason we pin older nightlies for Rust on AVR is that there are compiler regressions in the AVR backend from time to time.  This has two implications:

When a regression is present, there simply is no way to use latest nightly.
When no known regressions are present, we still want to advertise a known-to-work compiler version so unsuspecting users aren't the first ones to run into new problems.

Such is the life of a tier 3 target...

In general, old nightly compilers are not supported. The expectation is that if you are using a nightly toolchain, then you are remaining on top of recent nightly changes.

This is a bit unfortunate, as for the stated reason, we A) cannot always upgrade nightly and B) this makes regression-hunting quite difficult if certain crates break with not-that-old compiler versions.

then you would need to use corresponding old crate versions, too (such as via a lockfile).

This is a good pointer though, I guess we can try doing that for crates that are AVR-specific.  This of course won't work as soon as any non-platform-specific crate in the dependency tree pulls in later versions...

Actually, thinking about it, the version in rust-toolchain.toml is probably infered from current latest installed nightly

@mikkelens No, it is not.  It is hard-coded here and we update it when we have a new known-to-work compiler version.


While auditing the code to make my own wrapper I came across a typo in the code itself that was repeated.

This has been addressed in the 2.0.0 release. Thank you.


Added resources section

Added section for resources, changed line that contained a link in the code section that you could not previously click and was out of place.

[ x] I have read freeCodeCamp's contribution guidelines.
[ x] My pull request has a descriptive title (not a vague title like Update index.md)
[x ] My pull request targets the master branch of freeCodeCamp.
[ x] None of my changes are plagiarized from another source without proper attribution.
[ x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Thank you for opening this pull request.
This is a standard message notifying you that we’ve reviewed your pull request and have decided not to merge it. We would welcome future pull requests from you.
Reason:  The latest version of the master branch has a Resources section for this article.
Thank you and happy coding.


Allow to show historical data as vertical time series

This feature was added to allow to do:
Feature request: Provide Vertical time-series chart for TWD & TWS #24

updated to chart.js 3.5.1
added a "Show vertical graph" checkbox to the widget options
modified code to swap y/x axis accordingly
minor code refactoring

Hi,you did a great job!  I like the B&G stuff (and thought about coding it by myself, but you already did the most), but replacing all displays is quite an expensive thing and in any case I have a couple of pads on my boat. So the far more flexible (and cooler) solution is to use signal k and kip!I also plan to either write a widget/prepare a config that mimics the b&g TWA/TWS view.I hope the changes are ok for you, I am an old c/c++/c# programmer, so it's my "first" ng contact.Best regardsAlexAm 04.10.2021 12:07 schrieb mxtommy @.***>:
Nice! Will review/merge shortly and release new version! :)
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.


New package: playinginthedark.topspeed version 3.0.3

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.5 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Manual Validation ended with:

12:24:04 PM:  Faulting application name: TopSpeed.exe, version: <IP_ADDRESS>, time stamp: 0x4d443396
Faulting module name: TopSpeed.exe, version: <IP_ADDRESS>, time stamp: 0x4d443396 Exception code: 0xc0000005
Fault offset: 0x00041ceb
Faulting process id: 0x9a4
Faulting application start time: 0x01da0dc20a00164b
Faulting application path: C:\Program Files (x86)\Playing in the dark\Top Speed 3\TopSpeed.exe
Faulting module path: C:\Program Files (x86)\Playing in the dark\Top Speed 3\TopSpeed.exe
Report Id: 5710f423-9026-49a3-8d54-d3a9fac3e613
Faulting package full name:
Faulting package-relative application ID:

Hi @Mudb0y,
This error might be due to my validation VM not having good OpenGL drivers. But it's not certain since the error doesn't mention a video issue specifically. Is this expected - do you know if this application runs in other VMs?

https://github.com/microsoft/winget-pkgs/issues/107639


404 on refresh post

Describe the bug
Hello,
That's my problem:
When I go to my blog, the list of articles is displayed well. Once I go to an article and refresh the page I find a 404 error.
Looking more ready the networks I realize that the .JSON normally called has two points after refreshment
(Myurl.com/_nuxt/content/name..json)
Someone will have an idea where it might come from?
Thx for your rep :+1:
To Reproduce

Go to 'https://influx-agence.netlify.com/blog/'
Click on 'My second blog'
Look the web console network ( 2018.premier-posts.json )
Refresh page
The page is 404
Look web console network ( 2018.premier-posts**..**json )

Screenshots
3. Before refresh

6. After refresh

Applicable Versions:

Nuxt : Last
Netlify CMS : Last
Node.JS : 8.12.0

CMS configuration
backend:
  name: github
  repo: influx-agence/Nuxtlify
  branch: master
media_folder: "static/uploads"
display_url: https://influx-agence.netlify.com
collections: # A list of collections the CMS should be able to edit
  - name: "blog" # Used in routes, ie.: /admin/collections/:slug/edit
    label: "Blog" # Used in the UI, ie.: "New Post"
    folder: "content/posts"
    extension: "json"
    format: "json"
    create: true
    slug: "{{year}}-{{slug}}"
    fields: # The fields each document in this collection have
      - {label: "Title", name: "title", widget: "string", tagname: "h1"}
      - {label: "Publish Date", name: "date", widget: "datetime", format: "DD-MM-YYYY"}
      - {label: "Cover Image", name: "image", widget: "image", required: false, tagname: ""}
      - {label: "Body", name: "body", widget: "markdown"}
    meta:
- {label: "SEO Description", name: "description", widget: "text"}

Additional context
https://github.com/influx-agence/Nuxtlify

@influx-agence If it is happening on the site itself, rather than the CMS interface, it is a Nuxt issue, not a CMS bug. I'd suggest filing an issue on their GitHub repo: https://github.com/nuxt/nuxt.js/issues.  Thank you!


Add support for 301 redirect

Is your feature request related to a problem? Please describe.
It would be nice if the app can do a 301 redirect to preserve the title description image meta info on a link. Currently all links show the mnml meta info.
Describe the solution you'd like
Links to be redirected via 301 instead
Describe alternatives you've considered
Additional context
If it's a server needed, any advice? Maybe firebase cloud functions

I afraid that feature isn't available on free endpoint services such as jsonstore.io. Will try to implement them in future development.
Watch #26 and #24 for more info.


update online demo to use func instead of function

https://bytecodealliance.github.io/wit-bindgen/

Hi @attacker0211. Thanks for reporting this. wit-bindgen recently changed the function keyword to func, but it appears the online playground wasn't updated.
#238 should fix this shortly.

This should now be resolved. Thanks!


Make the first paragraph of each post "regular" font?

Hi,
It looks like the first line/paragraph of each page or post is bold w/italics.  Can that be disabled?
Thanks,
Brian

Just go to /assets/css/custom.min.css file and remove this line of code:
.postContent>p:first-child{font-size:1.2rem;font-style:italic;color:#444}


Need synopsis flow

Add synopsis tools from OpenRAM flow, change to have single makefile

RAM from OpenRAM needs to be synthesized properly, ensure timing passing with APR before transitioning to full verilog move. Also, floorplan.tcl needs to be changed to place arrays of RAM so we don't have to do them all by hand.


Tests companion for responders

I wonder if there are helper methods or companion projects to help with functional (requests) tests, i.e. to dry up functional tests too.
Currently my tests looks like this (from sample app)
it 'creates resource' do
  order = Order.create amount: 100
  new_payment = Payment.new amount: 100

  api.create_resource order, new_payment
  expect(api).to have_responded_with_resource_created order, new_payment
end

def has_responded_with_resource_created?(*models)
  resource = resource_for models
  # TODO: why 200?
  # expect(response).to have_http_status(:created)
  expect { JSON.parse(response.body) }.to_not raise_error, response.body
  expect(response.body).to be_json_eql resource.payload
  expect(response.parsed_body).to include(
    "id" => a_kind_of(Integer)
  )
end

Thanks.

We believe that dry and those kind of helper hurts the readability of tests. This is why we don't include any helpers on this gem. Thank you for the issue.


test_fiber is really insufficient for testing

It uses a straight fiber, and the conditions aren't nearly strict enough. Ideally they would have ~1E-11 agreement on all terms. Tests should be on a bent fiber to break known problematic symmetries. They should also test all the different boundary conditions.

examples/stokes_tests now have much better tests.