TextureFX

Adds a bunch of TextureFX ported from beta (most DX11, a few DX9) and rudimentary "Help Patches".
The patches have no help flag set because of this:

There are issues with some of the effects, I think most of them can be attributed to the sRGB "intricacies".
In the beginning I separated  each technique into a single shader/file and then used interfaces for selecting the techniques (did not want to end up with a waterfall as seen in Blend).  Later on I put the techniques inside methods – all in one file – and used a switch statement for selecting them. It is Ia lot less hassle and easier to understand in my opinion, no idea how it impacts performance though.
I am also not quite sure of the categories aka subfolders (Distortion, Math etc.) I put the files in.
Below a list of the nodes and some notes I made during "development":
Anaglyph
As far as I can tell the node works as it supposed to,
But not 100% sure because there is no help patch in beta.
Source
BackgroundSubstraction
Source
Directional Blur
Source
BlurGlow
Result looks slightly different compared to beta/DX11

Source
BlurPerfector
Results differ from beta/DX11 with same paramters/settings

Source
BubbleNoise
Results are "brighter" than beta/DX11 with same paramters/settings.

Source
Bump
Results differ from beta/DX11 with same paramters/settings.
Maybe due to input from NormalGlow which already looks different
Using an interface for selecting the techniques.

Source
ChangeFormat
Module to change textureformat and add mips
ChannelKey
Keys according to Alpha, Red, Green, Blue, Luma, Saturation.
Was originally called Keying.tfx but since there was already a KeyingBase I named it ChannelKeying
Maybe the files should be put into Keying Subdirectory?
The results differ from from beta/DX11, especially when the input texture is Srgb.
Using an interface for selecting the techniques.
Source
Colorize
Results are different compared to beta/DX11, especially if the input texture is sRGB.

Source
ColorMap
Needs first input texture set to RGB and second to sRGB to yield results similar to beta/DX11.
Using an interface for selecting the techniques
Source
ColorConvert
Was ConvertColor, decided to rename to be somewhat consistent with ColorMap, ColorRamp.
Results are different (compared to beta/dx11) for: Hue, Saturation, HSVtoRGB and RGBtoHSV
Using an interface for selecting the techniques
Source
CrossStitch
Source
DeInterlace
Not properly tested since there seems to be no way to play interlaced video in gamma currently.
Mediafoundation automagically deinterlaces (but the video stutters).
With OpenCV asImage/toTexture the resulting texture is broken - most likely due to the interlacing.
Help patch is just a placeholder.
Source
Dissolve
Source
Dither
Results are different compared to beta/dx11.

Source
Dots
Currently PixelSize input is of type int2, maybe make it float2 instead?
Source
DropShadow
The results differ from from beta/DX11, especially when the input texture is sRGB.

Source
Edge
Sobel/edge detection
The results differ from from beta/DX11

Source
Electricity
Isn't working even though there are no errors. I guess either the volume texture isn't loaded correctly
or only one slice is accessible inside the shader.

Source
Halo
The results differ from from beta/DX11. Maybe due to the different texture format?
DX11 uses R16G16B16A16_Float which is not available in Stride.
Uses Switch statement for different "Techniques".
Source
Hatch (Cross)
Results differ slightly when input texture is RGB and there is a big difference when using a sRGB texture as input.
Maybe could be combined with Hatch (Dot), Hatch (FromTexture) and Hatch (Line).

Source
Hatch (Dot)
Results differ slightly when input texture is RGB and there is a big difference when using a sRGB texture as input.
Maybe could be combined with Hatch (Cross), Hatch (FromTexture) and Hatch (Line).
Source
Hatch (FromTexture)
Maybe could be combined with Hatch (Dot), Hatch (Cross) and Hatch (Line).
Source
Hatch (Line)
Maybe could be combined with Hatch (Dot), Hatch (Cross) and Hatch (FromTexture).
Has its own implementation of r2d which differs from the one in Effects/Common/Utils.sdsl
Source
Kaleidoscope
Has its own implementation of r2d which differs from the one in Effects/Common/Utils.sdsl
Source
LinearDepth
Converting a depth buffer from a renderer into a linear format.
Isn't working correctly. Don't know why exactly.
Maybe something with the texture format.
The Depthbuffer format in DX11 is R32_Typless which isn't available in (VL.)Stride
It is currently not possible to use the depthbuffer directly from the renderer,
but also RenderTexture can't use D24_UNorm_S8_Uint.
Source
Liquish
Colors are "brighter" compared to beta/DX11

Source
Lomograph
"hipster texture filter" unc
Seems way off the original. Most likely due to RGB/sRGB.
Uses switch statement for different "Techniques".

Source
Magnify
Uses its own definition of PI couldn't "access" the one in Effects\Common\Utils
Source
Mandelbrot
Renders a Mandelbrot fractal.
Colors again are different and there seems also to be a lot more banding.
Also some other parameters have to be set differently to get similar results.

Source
Median
This one has some code-formatting that seems to be off but I left it the way it was.
Source
Metallica
Metal filter.
Colors are off. Also some other parameters have to be set differently to get similar results.
Has its own implementation of r2d which differs from the one in Effects/Common/Utils.sdsl

Source
Muffy
Uses an explicitly declared Sampler since there seems to be no pre-defined one with Mirror.
Source
NormalGlow
Results are "brighter" than beta/DX11 with same paramters/settings.

Source
Palette
Colors are off.
Uses Switch statement for different "Techniques".

Source
PolarCoordinates
Polar coordinates basically converts an incoming texture's mapping coordinates from cartesian space to circular (or polar) space or vice versa. The effect is the same as found in  the photoshop or gimp filter of the same name.
Textures seem to be a bit blurry compared to beta/dx11.
Maybe could be two separate nodes instead?
Source
Ripple
Source
Sharpen
"Enhance the edges off a picture"
Parameters have to be set differently to achieve results similar to beta / DX9

Source
Swizzle
Rearranges channels in a texture.
Source
Transform
Applies a transform to a texture
Transform pin needs to be connected to a transformation.
Source
Tricolor
Colorizing filter.
Source

That the examples look different compared to beta is understandable due to the linear color space. We have a help patch which shows how to switch the device to the Gamma color space, pasting the examples in there and things looks the same as in beta.
Reasoning what looks more correct is a little hard I must admit. For example the blur examples look different because of the generated mip maps. On the poles the level 6 mip map is a little brighter in Linear than in Gamma. Probably because of the filtering when downscaling it (averaging color values).
The edge examples look more correct to me - the colors get preserved, in Gamma space it all turns dark. But not sure whether or not that was the intention.
The palette example puzzles me. I read in some other thread that color palettes look different when generated in Linear space and that's how it's supposed to be. Or in other words in doesn't look linear on the eye, while it looked linear when it was created in Gamma space. So I wonder whether in such a case we must return with ColorUtility.ToLinear(col) - but then one immediately wonders where else that must be done. Little clueless here.
Good articles about this whole topic are https://medium.com/game-dev-daily/the-srgb-learning-curve-773b7f68cf7a and https://developer.nvidia.com/gpugems/gpugems3/part-iv-image-effects/chapter-24-importance-being-linear however after reading them three times now over the last couple of months I'm sometimes still puzzled. Especially when "comparing" results of "older" programs.

That the examples look different compared to beta is understandable due to the linear color space. We have a help patch which shows how to switch the device to the Gamma color space, pasting the examples in there and things looks the same as in beta.
Reasoning what looks more correct is a little hard I must admit. For example the blur examples look different because of the generated mip maps. On the poles the level 6 mip map is a little brighter in Linear than in Gamma. Probably because of the filtering when downscaling it (averaging color values).
The edge examples look more correct to me - the colors get preserved, in Gamma space it all turns dark. But not sure whether or not that was the intention.
The palette example puzzles me. I read in some other thread that color palettes look different when generated in Linear space and that's how it's supposed to be. Or in other words in doesn't look linear on the eye, while it looked linear when it was created in Gamma space. So I wonder whether in such a case we must return with ColorUtility.ToLinear(col) - but then one immediately wonders where else that must be done. Little clueless here.
Good articles about this whole topic are https://medium.com/game-dev-daily/the-srgb-learning-curve-773b7f68cf7a and https://developer.nvidia.com/gpugems/gpugems3/part-iv-image-effects/chapter-24-importance-being-linear however after reading them three times now over the last couple of months I'm sometimes still puzzled. Especially when "comparing" results of "older" programs.

Wow, big one. Thanks for the great work!
I left a few comments in the code itself. Mostly about readability. Most of the shaders also have formatting issues, would be nice to fix them first.

You mean the indentations? This didn't show up locally. I copied the code out of the beta code editor and used VS Code and VS to edit. Just cloned it here at home and looking at it now, the editors must have had different settings for indentations, tabs / spaces ... Will fix this tomorrow at the office -  using spaces.

At a lot of places it's also hard to understand what's going on due to missing comments and explanations. But I guess that's because the code was copied over and the original had no comments either?

Exactly. Some of it looked pretty freestyle to me and to be honest most of the time I didn't go through the code line by line trying to understand everything, much less trying to improve it. So for example when Targetsize was used I just added float2 R = 1/Texture0TexelSize;   Will go through your comments and try to fix those.

Reasoning what looks more correct is a little hard I must admit.

Same here, I don't know and wouldn't know how to make it right either way :)

Wow, big one. Thanks for the great work!
I left a few comments in the code itself. Mostly about readability. Most of the shaders also have formatting issues, would be nice to fix them first.

You mean the indentations? This didn't show up locally. I copied the code out of the beta code editor and used VS Code and VS to edit. Just cloned it here at home and looking at it now, the editors must have had different settings for indentations, tabs / spaces ... Will fix this tomorrow at the office -  using spaces.

At a lot of places it's also hard to understand what's going on due to missing comments and explanations. But I guess that's because the code was copied over and the original had no comments either?

Exactly. Some of it looked pretty freestyle to me and to be honest most of the time I didn't go through the code line by line trying to understand everything, much less trying to improve it. So for example when Targetsize was used I just added float2 R = 1/Texture0TexelSize;   Will go through your comments and try to fix those.

Reasoning what looks more correct is a little hard I must admit.

Same here, I don't know and wouldn't know how to make it right either way :)

Ok thanks, I'll merge this PR now so it's part of the next release. I guess the issues will pop-up once we all start using the nodes.
In the future it would be nice to initiate such a huge PR right at the moment when beginning working on it (marking the PR as a draft), so we all see the progress and can give hints and suggestions early on.

I didn't know it would turn out like this. I was bored during the holidays/isolation and started to port some effects planning to post them in the wip forum, then my ocd took over.

Ok thanks, I'll merge this PR now so it's part of the next release. I guess the issues will pop-up once we all start using the nodes.
In the future it would be nice to initiate such a huge PR right at the moment when beginning working on it (marking the PR as a draft), so we all see the progress and can give hints and suggestions early on.

I didn't know it would turn out like this. I was bored during the holidays/isolation and started to port some effects planning to post them in the wip forum, then my ocd took over.

I'd revert the merge until we discussed it internally. I think we should properly evaluate the consequences and identify what needs to be done now and what can be done later.
So please wait a bit before we have more clarity about the involved issues.

I'd revert the merge until we discussed it internally. I think we should properly evaluate the consequences and identify what needs to be done now and what can be done later.
So please wait a bit before we have more clarity about the involved issues.

Ok just recognized that we have a node name clash as well (ColorMap), turning many help patches red. Reminds me that we (as was done in Stride lately) need to setup TeamCity so it also builds and runs tests of pull requests.
Some nodes also don't dispose the internal state properly (like ColorMap). The internally used interface (like IColorMap) should inherit from IDisposable which should be called by the wrapper node on its own Dispose. This can be quite problematic as it leaves the game per entry point open and things don't properly shutdown on F8 or app close.
@bj-rn sorry for that, I'll revert the merge of this PR and would kindly ask you to open a new one as there's no way to re-open this one. Then @tebjan can also add his comments and we can discuss the naming conflicts as well as the dispose issues.

Ok just recognized that we have a node name clash as well (ColorMap), turning many help patches red. Reminds me that we (as was done in Stride lately) need to setup TeamCity so it also builds and runs tests of pull requests.
Some nodes also don't dispose the internal state properly (like ColorMap). The internally used interface (like IColorMap) should inherit from IDisposable which should be called by the wrapper node on its own Dispose. This can be quite problematic as it leaves the game per entry point open and things don't properly shutdown on F8 or app close.
@bj-rn sorry for that, I'll revert the merge of this PR and would kindly ask you to open a new one as there's no way to re-open this one. Then @tebjan can also add his comments and we can discuss the naming conflicts as well as the dispose issues.

Hi, cool stuff here. Seems like most of the TFX are about to be ported. Just tried VL a bit and I found that Blend Texture is missing and here also. @bj-rn  are you planning to port that one too ?

Hello,
Blend is already available in Stride:

btw this pull request has been closed and reopned as https://github.com/vvvv/VL.Stride/pull/393
The FX are / will be separated into : core, practical and gimmick. See: https://github.com/vvvv/VL.Stride/pull/393#issuecomment-758811569
Not not sure about the current state of affairs but afaik @tebjan is currently adding the core part

Strange I can't find that node but there is "BloodShader" which seems to be a textureFX. I'm using a fresh install by the way.

Found it. Had to toggle experimental nodes in nodebrowser.


How do I retrieve the cell data from another column in rendererContext

PlutoColumn(
        title: 'A',
        field: 'expiry_text_field',
        type: PlutoColumnType.date(
          format: "dd-MMM-yyyy",
        ),
        enableEditingMode: false,
        renderer: (rendererContext) {

          return Text(
            rendererContext.cell!.value.toString(),
            style: TextStyle(
              color: textColor,
              fontWeight: textFontWeight,
            ),
          );
        },
      ),

Can I get cell data from another column other than the current?

stateManager has various methods and properties.
stateManager.columns : You can get the dereferenced columns.
stateManager.refColumns : You can get the referenced columns.
...

You can also get the value from the row property, and access the cells Map, in the example bellow, you I am getting a color value that is not displayed in the row, but is configured to be there, so I can change the status color
renderer: (rendererContext) {
  Color textColor;

  try {
    textColor = rendererContext.row!.cells['Color']!.value;
  } catch (_) {
    textColor = Colors.black;
  }

  return Text(
    rendererContext.cell!.value.toString(),
    style: TextStyle(
      color: textColor,
      fontWeight: FontWeight.bold,
    ),
  );
},

hope it helps


chore: updated example with-ts to V3

Tyvm for this! I converted this issue to a draft because of course v3 isn't out yet and I've marked this PR as the one in which we'll do the full upgrade. Once V3 has been released we can finish this PR.

OP is MIA. PR has been superseded by #95


add error message if a mutable field is initialized with a reference type

would fix #6

ugh it didn't commit my changes to core.attribute nvm don't merge this yet

alright should be good now

This should be one commit

I'm actually surprised how many things this is reporting in dmd and phobos.... some of them are functions, we might want to relax it on that too because functions point to immutable data (the code segment) too.

I tried it on phobos and it compiled without complaints... weird

I tried it on phobos and it compiled without complaints... weird

nevermind I'm silly, I confused phobos with druntime

I got it working as part of my monster megacommit, the monorepo has the fixes. but we can prolly make it work ebtter in some cases like if you're assigning a struct of just strings and function pointers that's basically still a value type. still meh it is fine
i kinda worry about pulling changes out of that huge monorepo thing. there were big commits and small commits mixed together and i prolly broke stuff
but meh we can fix it as it happens from here on out

Windows' HANDLE is another one that probably shouldn't trigger it but does.

I'm gonna hack that out by making void* an exception to the rule too.


Add Alchitry Au board definition

Copy-Paste from https://github.com/m-labs/nmigen-boards/pull/44 follows:
Notes:

Alchitry Au requires a loader program from Alchitry. I tried to follow a pattern similar to what
nmigen uses to locate Yosys/Vivado but I'm not sure if that's the right approach
DDR3 is untested. The pins are copied from the schematic so I may have gotten something wrong there.
There's also an IO shield for the Au with a few buttons, some 7 segment displays, and a bunch of
LED's. I was debating adding a resource for that, similar to how the icebreaker pmod's are defined, but ultimately decided I was gambling whether I'd finish that pin definition before running out of motivation. Let me know if you'd like that pin definition in this PR and I'll try to add it.

Alchitry Au requires a loader program from Alchitry. I tried to follow a pattern similar to what
nmigen uses to locate Yosys/Vivado but I'm not sure if that's the right approach

Looks good to me.

DDR3 is untested. The pins are copied from the schematic so I may have gotten something wrong there.

 For now I think it doesn't make sense to be super strict about it, especially given that the effort to fully functionally verify the pinouts can be quite large.

There's also an IO shield for the Au with a few buttons, some 7 segment displays, and a bunch of
LED's. I was debating adding a resource for that

We have a few missing peripherals on a few boards, too. It's not great, but it's not a blocker, especially given that there are quite a few boards with an absolutely massive number of peripherals and pins used.

Thanks!


Rename host and hostPattern

Host is usually: jane.transloadit.com but since host in our case contains a full url with many more components (protocol, path, etc), I'd vote to name it endpoint.
hostPattern could than be named allowedOrigins which could either take (an array for a list of flat origins, a function to be executed on a string which returns true or false, or) a RegExp.
The proposals are up for discussion, but I do think a change is needed.
For BC-compat, we could likely still accept host and throw a Deprecation warning at the console for a release or two.

The only issue with endpoint is that we use it in uploader plugins, and that might contribute to a popular misunderstanding that Uppy Server is an upload target. Not an end of the world, just noting.

That's a fair point. How about url then?

I like url better. We never used serverHost, maybe no need for serverUrl either.

on second thought, serverUrl and serverPattern tells it just right, and avoids any future confusion whatsoever. So I'll go with those instead https://github.com/transloadit/uppy/pull/905

For BC-compat, we could likely still accept host and throw a Deprecation warning at the console for a release or two.

@kvz do we really need to? Since we are still pre-1.0 and breaking changes are still permitted-ish?

That’s fair! I expect some people will upgrade and forget to change, so maybe we can just fail fast and hard when no serverUrl is set. Perhaps this is already the case (for host, currently). Then we can just rename, note Breaking in the changelog and be done with it 👌
Sent from mobile, pardon the brevity.

On 14 Jun 2018, at 04:08, Ifedapo .A. Olarewaju<EMAIL_ADDRESS>wrote:
For BC-compat, we could likely still accept host and throw a Deprecation warning at the console for a release or two.
@kvz do we really need to? Since we are still pre-1.0 and breaking changes are still permitted-ish?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

done here https://github.com/transloadit/uppy/pull/905


OSS-Fuzz integration

My project (https://github.com/guidovranken/cryptofuzz) fuzzers cryptographic libraries and it has support for cifra.
We can run a cifra fuzzer on OSS-Fuzz (https://github.com/google/oss-fuzz) but this requires that the project is being maintained. Is this the case?

This project is passively maintained: I don't have immediate plans to add new features, but will fix bugs.

This project is passively maintained: I don't have immediate plans to add new features, but will fix bugs.

That's great to hear; new features are not necessary, only bugfixes. Are you interested in running cifra on OSS-Fuzz? You don't need to do anything, except address bugs.
The first bug I found is memory corruption when using a SHA3 family hash with HMAC (or PBKDF2). This is also what https://github.com/ctz/cifra/issues/14 seems to be reporting. I can provide a compilable proof of concept if you like. Are you willing to address this bug?
Thanks

That's great to hear; new features are not necessary, only bugfixes. Are you interested in running cifra on OSS-Fuzz? You don't need to do anything, except address bugs.
The first bug I found is memory corruption when using a SHA3 family hash with HMAC (or PBKDF2). This is also what https://github.com/ctz/cifra/issues/14 seems to be reporting. I can provide a compilable proof of concept if you like. Are you willing to address this bug?
Thanks


Add well documented readme to show all algos list.

@D3v3sh5ingh assign me this issue and tell me what all you wan in the read me

@hnishal Assigned!
You can add a bulleted list with all the algorithms (with links to respective files) we have in the repository.

ohkay im on it

Resolved by #8
Closing..


Update rubocop-govuk to 4.0.0.pre.1

Trello: https://trello.com/c/pqXbKZiC/215-test-rubocop-govuk-400pre1-prior-to-400-release
This bumps to the 4.0.0 pre release of rubocop-govuk which is part of trialling the new rubocop-govuk release. The gem pin to 4.0.0.pre.1 is intended to be removed once the final release of rubocop-govuk 4.0.0 is out. This is the last app in the trial 🎉

Thanks @benthorner, I've updated the tests for the tags validator to show the items accepted. After a bit of thought I decided against adding a UUID directly (felt a bit fishy since tags shouldn't be these as that what links should be) and went for a few that covered the regex.


Unable to connect a caddx air unit already programed

Hi ,
My caddx airunit are already activated wtfos .
I want to update the system : it works on the Google but i’m unable to connect the caddx air unit :
When i connect the air unit i get this messages :
Device connected - checking for adb
Failed abd
When i root device  : no device.
what can i do ?
thanks

I noticed that when i connect the air unit , an usb file opens .
The usb cable is ok because i can connect the googles
Thanks

ok found the solution with zadig


Add onContentCommitted as a param to CupertinoTextField.borderless

@tneotia do this before Flutter PR!!!

Issue created by BlueBubbles Bot in the BlueBubbles commnity on behalf of @tneot01 in #android-bugs.

Fixed @tneotia https://github.com/BlueBubblesApp/flutter/tree/zach/image-insertion
That's the branch it's on ^


Add support for commit cross references

Fixes #22628
This PR adds cross references for commits by using the format owner/repo@commit . References are rendered like go-gitea/lgtm@6fe88302.

I think it'd be really cool if the commit sha would be in a code span like in GitHub:
go-gitea/lgtm@6fe8830276a06b01a2f8782f55d9c05d6fc54b47
Markdown source:
go-gitea/lgtm@6fe8830276a06b01a2f8782f55d9c05d6fc54b47

I saw that too but what's the benefit of the code span?

Looks cute

Actually, I don't really understand why only the commit should get a code-span, and not the whole link…

Then lets have a vote:
🚀 go-gitea/gitea@abcd1234
👍  go-gitea/gitea@abcd1234
😄 go-gitea/gitea@abcd1234

Honestly I would almost prefer:
go-gitea/gitea@abcd1234
but also changing the issue/pr one to go-gitea/gitea#1234
I think it makes the important part more obvious imo
But with the vote at hand, I just vote for consistency.

Codecov Report

Merging #22645 (3e692bd) into main (cc1f8cb) will decrease coverage by 0.25%.
The diff coverage is 48.44%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main   #22645      +/-   ##
==========================================
- Coverage   48.22%   47.97%   -0.25%
==========================================
  Files        1050     1061      +11
  Lines      142937   144576    +1639
==========================================
+ Hits        68925    69364     +439
- Misses      65772    66939    +1167
- Partials     8240     8273      +33

Impacted Files
Coverage Δ

models/auth/token.go
45.18% <ø> (ø)

models/git/lfs.go
17.37% <0.00%> (-3.96%)
:arrow_down:

models/issues/issue_project.go
31.29% <0.00%> (+0.23%)
:arrow_up:

models/issues/pull.go
50.63% <ø> (+0.01%)
:arrow_up:

models/organization/team.go
58.00% <ø> (-7.45%)
:arrow_down:

models/organization/team_user.go
66.66% <ø> (-7.91%)
:arrow_down:

models/packages/package_blob.go
50.00% <0.00%> (-6.82%)
:arrow_down:

models/repo/repo_list.go
74.29% <0.00%> (-0.17%)
:arrow_down:

models/system/setting.go
53.80% <0.00%> (-2.13%)
:arrow_down:

models/user.go
38.33% <0.00%> (+3.21%)
:arrow_up:

... and 123 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Sometime image is in RGBA format which has 4 channel and generate will break for 2.5

As title.

https://blog.csdn.net/WhiffeYF/article/details/118253518
I converted the images to RGB format directly, but for vl-2.0 it is not necessary to do that


Hard restart after entering password

Describe the bug
After boot from provided EFI for sonoma getting Hard restart after entering password

Same issue here. after entering my password, my thinkpad reboots. This issue persists in both 1.5.1 and 1.6.0 versions of the Sonoma EFI. I have not tested other versions. My thinkpad doesn't do that when using the "EFI - Intel" efi, that supports Ventura, but the Wi-Fi does not work


String#succ with partially valid UTF-8 does not handle emoji unicode

CRuby:
[2.6.0] > "💪\xFF".succ
=> "💫\xFF"

mruby built with CFLAGS="-Wall -DMRB_UTF8_STRING" rake:
mirb - Embeddable Interactive Ruby Shell

> "💪\xFF".succ
 => "\x01💪\x00"

Found this while working on GH-4564.


Support emitting both output formats

Any chance we can have a switch to output both text and XML output for the same test run?  e.g. text on stdout and XML to a file.
Alternatively, a PS script to convert the XML output back to the text output would be useful -- the on-screen format of the text output is file is much easier to read than the tab-separated file emitted by Process-Diskspd.ps1.
Thanks

Allowing multiple outputs seems like a nice & fairly feasible idea; zero or one to stdout, but allow optional file specification.


Lower max override from 100 to 90

Change from 100 to 90 as it is meant to be a minimum bound in case of user entry error. In this case a user (me) is trying to set it to 90 on purpose which is a reasonable value.

I'm fine with this.  At the time we set these limits, we thought there was some value to having a "range" instead of a single target.  Since then we've been running with min_bg = max_bg, so I have no problem setting the max_bg = min_bg = 90 as a lower bound.
Does anyone see any reason not to merge this?

I am also fine with this given what we've seen across the user base.
On Mon, Mar 28, 2016 at 2:11 PM, Scott Leibrand<EMAIL_ADDRESS>wrote:

I'm fine with this. At the time we set these limits, we thought there was
some value to having a "range" instead of a single target. Since then we've
been running with min_bg = max_bg, so I have no problem setting the max_bg
= min_bg = 90 as a lower bound.
Does anyone see any reason not to merge this?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/openaps/oref0/pull/92#issuecomment-202582296

--
Dana Lewis | http://diyps.org | @DanaMLewis
http://www.twitter.com/danamlewis | http://www.linkedin.com/in/danalewis
"Doing something for someone else is more important than anything you would
do for yourself."


Convert 'components/post_view/post_info' module and associated tests to TypeScript

Summary
Converted 'compoents/post_view/post_info' module, related tests and snapshots to typescript.
Ticket Link
Fixes https://github.com/mattermost/mattermost-server/issues/15466
Related Pull Requests

Has server changes (please link here)
Has redux changes (please link here)
Has mobile changes (please link here)

Screenshots

Hello @vraravam,
Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.
Per the Mattermost Contribution Guide, we need to add you to the list of approved contributors for the Mattermost project.
Please help complete the Mattermost contribution license agreement?
Once you have signed the CLA, please comment with /check-cla and confirm that the CLA check is green.
This is a standard procedure for many open source projects.
Please let us know if you have any questions.
We are very happy to have you join our growing community! If you're not yet a member, please consider joining our Contributors community channel to meet other contributors and discuss new opportunities with the core team.

/check-cla

@vraravam That for the PR :tada:
There is one type error remaining:
components/post_view/post_info/post_info.tsx:226:18 - error TS2740: Type '{ post: any; commentCount: number | undefined; isFlagged: boolean | undefined; handleCommentClick: (event: MouseEvent<Element, MouseEvent>) => void; ... 4 more ...; enableEmojiPicker: boolean; }' is missing the following properties from type 'Pick<InferProps<{ post: Validator<object>; teamId: Requireable<string>; location: Validator<string>; commentCount: Requireable<number>; isFlagged: Requireable<...>; ... 14 more ...; canDelete: Validator<...>; }>, "isMenuOpen" | ... 11 more ... | "handleAddReactionClick">': components, actions, channelIsArchived, isLicensed, and 4 more.

226                 <DotMenu
                     ~~~~~~~

I will still queue your PR for an early round of feedback

@vraravam That for the PR 🎉
There is one type error remaining:
components/post_view/post_info/post_info.tsx:226:18 - error TS2740: Type '{ post: any; commentCount: number | undefined; isFlagged: boolean | undefined; handleCommentClick: (event: MouseEvent<Element, MouseEvent>) => void; ... 4 more ...; enableEmojiPicker: boolean; }' is missing the following properties from type 'Pick<InferProps<{ post: Validator<object>; teamId: Requireable<string>; location: Validator<string>; commentCount: Requireable<number>; isFlagged: Requireable<...>; ... 14 more ...; canDelete: Validator<...>; }>, "isMenuOpen" | ... 11 more ... | "handleAddReactionClick">': components, actions, channelIsArchived, isLicensed, and 4 more.

226                 <DotMenu
                     ~~~~~~~

I will still queue your PR for an early round of feedback

This thread on MM's community server is relevant, and I think when we get and answer there, it would apply here as well

tx @abdusabri & @hanzei - I am not too proficient in react - and so, I dont follow the thread in the community channel that you had pointed me to. I will try to follow the thread- but, if there's a fix for this, please do commit onto this PR and take it forward.

Throwing Work in Progress on this PR until the type check errors are resolved.
@vraravam Feel free to give me an @-mention when the PR passes type check. If you run into any issues making that happen, let us know and we will try to help :)

Tagging @devinbinnie - have made certain properties as non-mandatory in DotMenu.

/update-branch

/update-branch

Not srue what this done (I assume a git merge), but, I have already rebased from master and force-pushed just now.

@devinbinnie - can we please re-review and merge this? Both make check-tasks and make test works - and I have rebased from master as of now.

/update-branch

/update-branch

@devinbinnie - not sure if the update-branch is needed. As I said above, I have been rebasing from the master branch and keeping this fork/branch completely on top of it on a daily basis.

@devinbinnie - not sure if the update-branch is needed. As I said above, I have been rebasing from the master branch and keeping this fork/branch completely on top of it on a daily basis.

@vraravam The checks were failing so I did that to kick the CircleCI checks, I don't assume anything is wrong but it's worth keeping it updated.
Also, replying to a comment with update-branch on it appears to trigger it again :P

@devinbinnie - if the checks fail once again, how do you propose we proceed? Based on one of your past comments, I hope we can cast and move onto merging this PR?

@devinbinnie - all checkes passed! Can we please merge?

@devinbinnie - all checkes passed! Can we please merge?

@vraravam Once my review is done another developer and a member of the QA will need to review as well. Once all that is done then we can merge :)

@vraravam I'm noticing a lot of files being changed that are unrelated to this PR. Not sure if that's a result of rebasing and force-pushing (which we actually try to avoid here since it causes issues with the review history).
Also I'm seeing that the type checking has failed again, can you have a look?

@devinbinnie - please see now. I try to squash commits for a cleaner history even beyond the PR.
The previous changes were related to import statements that look at src folders for types. These have now been removed from this PR.

@devinbinnie - please see now. I try to squash commits for a cleaner history even beyond the PR.
The previous changes were related to import statements that look at src folders for types. These have now been removed from this PR.

We squash the PRs once they're merged so no need to do any squashing yourself. Make as many commits as you want!

@devinbinnie - can we move forward now?

I have also rebased from master just now @devinbinnie

A few type errors in the pipeline that will need to be resolved before submitting. Otherwise, looks good to me! Thanks @vraravam!

@nickmisasi - Need some help fixing the failing tests. Its been a long time for this PR - and I'm just about ready to give up.

@vraravam looks like a large portion of the tests are failing for the same reason: missing state.
I was able to fix a bunch of the tests in new_post.test.js by tweaking the testState for them:
Adding the general entity, with a blank or dummy value config object (this will quell Cannot read 'config' of undefined' that's being thrown here according to the test logs
general: {
                        config: {
                            test: true,
                        },
                    },

Adding the preferences entity, with a blank myPreferences object

                    preferences: {
                        myPreferences: {},
                    },

And then adding the channels object to the channels entity, with a channel entry that maps to the existing var channelId:

                        channels: {
                            [channelId]: {id: channelId},
                        },

Here's my full testStore (including my suggestions from above) for the setChannelReadAndViewed test that I have passing locally:
const testStore = await mockStore({
                entities: {
                    channels: {
                        currentChannelId: channelId,
                        manuallyUnread: {},
                        myMembers: {
                            [channelId]: {channel_id: channelId, last_viewed_at: 0, roles: ''},
                        },
                        channels: {
                            [channelId]: {id: channelId},
                        },
                    },
                    posts: {
                        posts: {
                            [post1.id]: post1,
                        },
                    },
                    users: {
                        currentUserId,
                    },
                    general: {
                        config: {
                            test: true,
                        },
                    },
                    preferences: {
                        myPreferences: {},
                    },
                },
            });

Easiest way to fix these sorts of errors is to go to the line that's failing and look what is being accessed. You can then provide a value for that (even empty) to the test state. Repeat, until success!
If you're still having trouble, feel free to reach out to me on the Mattermost Community Server! and I can try to help you through them. I'm @nick.misasi on there.

@abdusabri - I think your approval is still pending - others have approved.

@abdusabri - I think your approval is still pending - others have approved.

Thanks for your work on this! As mentioned earlier, I'm just a fellow contributor 😊. My review doesn't impact merging PRs, and I can't merge PRs.
You can probably mention Nick to assign a QA reviewer (needed before a PR is merged)

@abdusabri - I think your approval is still pending - others have approved.

Thanks for your work on this! As mentioned earlier, I'm just a fellow contributor 😊. My review doesn't impact merging PRs, and I can't merge PRs.
You can probably mention Nick to assign a QA reviewer (needed before a PR is merged)

The stats on the top right of the page show that your approval is still pending. @nickmisasi has already approved. I guess, the process has to be followed.

@jgilliam17 - thanks - i have committed a fix to handle the null checks - but, am not able to see the logs for why the new server was not created. Could you please post the url here?

The test server is failing to update on:
{"level":"info","ts":1604106411.7408109,"caller":"app/server.go:226","msg":"Server is initializing..."}
{"level":"info","ts":1604106411.7409978,"caller":"app/web_hub.go:83","msg":"Starting websocket hubs","number_of_hubs":4}
{"level":"warn","ts":1604106411.7411802,"caller":"app/server.go:235","msg":"Sentry reporting is enabled, but SENTRY_DSN is not set. Disabling reporting."}
{"level":"debug","ts":1604106411.7412515,"caller":"app/web_hub.go:413","msg":"Hub is starting","index":1}
{"level":"debug","ts":1604106411.7413144,"caller":"app/web_hub.go:413","msg":"Hub is starting","index":0}
{"level":"debug","ts":1604106411.7413328,"caller":"app/web_hub.go:413","msg":"Hub is starting","index":2}
{"level":"debug","ts":1604106411.7414694,"caller":"app/web_hub.go:413","msg":"Hub is starting","index":3}
{"level":"info","ts":1604106411.756888,"caller":"utils/i18n.go:90","msg":"Loaded system translations","for locale":"en","from locale":"/mattermost/i18n/en.json"}
{"level":"debug","ts":1604106411.7614229,"caller":"utils/html.go:30","msg":"Parsing server templates","templates_directory":"/mattermost/templates"}
{"level":"info","ts":1604106411.7651207,"caller":"sqlstore/supplier.go:235","msg":"Pinging SQL","database":"master"}
{"level":"info","ts":1604106411.787166,"caller":"sqlstore/supplier.go:235","msg":"Pinging SQL","database":"replica-0"}
{"level":"error","ts":1604106412.0081158,"caller":"sqlstore/supplier.go:909","msg":"Failed to create index","error":"pq: relation \"idx_threads_channel_id\" does not exist","error":"pq: column \"channelid\" does not exist"}

I'm going to re-create it

@jwilander - I had to rebase from master again because of conflicting files from other PRs that got merged. Once this current push finishes, can we please move forward and unblock the merge into master?

Thank you @vraravam
Unfortunately, few errors are still shown + white screen.

Thank you @vraravam
Unfortunately, few errors are still shown + white screen.

Sorry @jgilliam17 - those are not changes introduced by me. I came onto this codebase as a newbie, and trying to help with a "good first issue". Unfortunately, doesn't seem like this will end anytime soon. Please go ahead and get someone else to contribute. I'm going to abandon this and other PRs

Hi @vraravam, sorry to hear that your contribution experience wasn't as you expected.
Would you be open to sharing more on how we can improve? We are always looking to make the experience better for our open source contributors.
I'd also be happy to guide this PR to completion with you, given two other contributions you submitted are dependent on this being merged.
@jgilliam17 Are you open to testing if the errors you're seeing reproduces on the master branch?

Thank you @jasonblais -  I'll check if this issue reproduces on master branch first thing tomorrow.

thanks @jasonblais .
What I would like as a first step is for someone to take "ownership" (I know this is OSS, etc) - but, for the contributor's perspective, especially a newbie to the tech stack and codebase, a mentor can be really useful.
I do want to call out that different folks have helped unblock me as time went by, and I am really appreciative to them.
But, the problem remains that I see and experienced other PRs that are being merged, which cause a conflict on my PR, and it just keeps dragging on. I live in India, and even if some of the contributors are in the US (the exact other opposite end of the globe), there could be some manageable turn around time. In some cases, for a couple of my PRs, that has been in the matter of days - with multiple prompts from my side. - and that is not very conducive to growing a contributor's list - imo.
There have been multiple weekends (and since this is OSS, I assume that folks are contributing during their non-work time - which means weekends might be the best time to get traction) - and I have waited patiently for multiple weekends, but things dont seem to move forward (or more at a very slow pace).
Due to the other PRs, and due to their conflicting changes, I keep rebasing and fixing those merge conflicts - but, there's too much work going on there.
If you are willing to pair with me on taking this forward, I would gladly appreciate it - and am willing to stay the course, but if this keeps dragging on further, I would leave the PR open for someone else to continue, and will not contribute any more.

@vraravam and @jasonblais I could reproduce the errors on the master branch. Creating a ticket for follow up.

@vraravam and @jasonblais I could reproduce the errors on the master branch. Creating a ticket for follow up.

thanks @jgilliam17 - does that mean that this PR can be approved and move forward?

Unfortunately, due to present errors, even though they were found to be unrelated to your PR, I haven't had a chance to test your changes on the test server. That is a necessary step before we can approve and move forward.
I will bring attention to the issue and let you know what the status of the fix is.
Thank you for your patience.

I assume that I should continue rebasing from master branch till that time?

I assume that I should continue rebasing from master branch till that time?

@vraravam We would ask that you instead merge master instead of rebasing to preserve history.
There's a great article written by our own Jesse Hallam that talks about how we like to do our PRs: https://developers.mattermost.com/blog/submitting-great-prs/

@vraravam re your earlier comment - feel free to reach out to me directly on the community server  if you have any questions. My user name is ashish.bhate. Since we're in the same timezone, hopefully I can respond quicker.

@vraravam thank you for the candid feedback. It sounds like much of the frustration has come from a) continuously having to merge master against your branch given dependent changes, and b) not having someone assigned to help guide/mentor throughout the process.
As a next step, @ashishbhate has offered to help partner with you to ensure your contributions are merged smoothly. He messaged you on https://community.mattermost.com/ and this will be a good medium to discuss next steps. He is in the same timezone as you, which will help with more timely responses.
We have also queued an item to discuss ownership of the various Mattermost repositories, and how we may be able to more effectively redistribute ownership.
Thank you again for your feedback! It's much appreciated. Don't hesitate to reach out to @ashishbhate (or myself) for any further questions, and hopefully we can smoothly get your contributions to the finish line now.

/update-branch

@vraravam It looks like there are changes in this PR that aren't part of the PR, likely due to rebasing.
Can you update your branch so that those changes are no longer present?
Thanks again for your hard work, we're almost there :)

@devinbinnie - any hint on what's extraneous? I can try to remove/revert to the old state.


Waiting for CNI driver delays Spegel startup - mirror not used

Hi,
Spegel looks great! But I'm on a test cluster and have noticed that it's not starting up quickly enough on a fresh node to get any benefit from it - by the time Spegel has started and registered the mirrors the pods that I want to pull from Spegel have already started pulling from the default registry
The startup process seems to be blocking on my CNI driver (Calico) starting up:
Warning  NetworkNotReady    16s (x12 over 38s)  kubelet             network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

This more or less happens in parallel to my pods starting up, so by the time Spegel is running my pods have pulled from the default registry and bypassed Spegel.
My workaround so far is to enable hostNetwork: true in the Spegel daemonset spec - this stops it having a dependency on Calico, and it starts more or less immediately (before my pods at least)
I assume not everyone uses Calico so changing Spegel to use hostNetwork by default may not be desired? Could a flag be added to the helm chart to enable this instead? Bear in mind that service.registry.port needs setting to 30020 for this to work, but otherwise it does seem to work in some limited testing..

So I have observed this in some AKS clusters that I am running that are also using Calico. I think the issue is derived from the fact that the priority class is too high, resulting in Spegel being scheduled before the Calico DS. Then it experiences exponential backoff as the node is not ready. A good solution would be to make sure that the priority class is lower than what ever calico has set. Checking my AKS cluster I can see that both are set to system-node-critical. I will find a default priority class that is lower than this, which will hopefully solve your problems.
It is interesting that setting host network allows you to ignore the dependency towards the CNI plugin. I think that in the future it would be interesting to see if it would be possible to have a solution that does not rely on the CNI. I do not think enabling host networking is the solution though as it comes with other implications.

Interesting about the priorityClassName - I just tried to set it to system-cluster-critical though (which according to the docs is higher priority than system-node-critical) and saw the same behaviour as before.
Reading the docs I'm not sure priorityClassName really controls the order of scheduling, it just seems to control whether scheduling is guaranteed.  In my testing my own regular-priority pods get scheduled at exactly the same time as critical ones, because my nodes have enough space for them all I assume.
In fact I've not really been able to find anything that controls the order of the scheduling of pods to the degree that I suppose Spegel would ideally want - to come up before any other pod on the node. It seems the only way would be to have the mirror config on the nodes already, before kubernetes even starts - which implies some platform-specific bootstrap/init magic..
Furthermore I suppose that you might want to use Spegel as the source for your CNI images, in which case hostNetwork would be your only option I assume?
What implications could hostNetwork: true have? My test environment isn't used much but I can't say I've spotted anything bad happening with this setting, any ideas for things I could look for?

Its been a while since I read through the scheduling docs in depth so you may be right.
I think it will be an impossible challenge to mirror the CNI image now when I think about it. The reason I have not looked to far into it is due to the fact that the Calico images are pre pulled on nodes when running AKS so it has been less of an issue for me. Currently Spegel relies on poth container host ports and node port services to work. The host port creates a port map on the node to the local container. This is how Containerd is able to reach onto the container. While the node port service does the same but on a separate port that is load balanced onto a random Spegel instance in the cluster. There are two separate ports to allow for the use of Spegel even when the local Spegel instance is down.
I have been forced to think about the implications of this a bit more thanks to this issue which is great. Both of these node to container port mappings are implemented by the CNI. Which explains the dependency on the CNI which other applications may not have. Now enabling host network would in turn solve some of these problems as it would skip the CNI all together and use the hosts networking namespace, with the downside of basically being able to intercept all networking on the node. I am not really sure if this is something that i want to allow as it pushes things even further into the realm of privileged Pods, but still something that I can consider to be an option. This would most likely require a refactor of some of the logic as Spegel would need to avoid the container networking all together and instead proxy on the Node IP level, which would have some benefits.

You could include a node taint that is removed when spegel reaches healthy status. Cilium uses this model to ensure the daemonset is up before workloads can be scheduled.

I will amend the toleration to avoid scheduling. I thin k that fixing #22 could close this issue in the future. Having a single prefer local service would remove the need for host ports while offering the same feature.


There are some bugs around repeated fields

In some cases repeat fields can't be added
In some cases distinct autocomplete states aren't being preserved between different repeated questions.

Don't understand this. So we will close this until someone has come up with the exactly what this is.


Add a fluent-bit connector

This will add fluent-bit loaders and savers.
This PR is in a very early stage right now, the actual functionality is not implemented yet.
We include a custom fluent-bit-minimal.h from https://github.com/fluent/fluent-bit/issues/7165#issuecomment-1544711546 because the public API of fluent-bit is currently unusable. This will hopefully be included upstream in the near future.
TODO:

[ ] static libraries are currently not installed and scattered across many archive files. We should try to build a fused archive and copy it to the output manually.
[ ] The few dependencies that are not vendored need to be linked in manually for the static build.

That build system work is quite a feat!
I had started something local using crtp_operator. Is saver/loader the right abstractions? Aren't we going from void to events directly? I haven't studied the fluent-bit API in depth, perhaps there's a MsgPack parser/printer here as well.
struct operator_args {
  located<std::vector<std::string>> args;

  template <class Inspector>
  friend auto inspect(Inspector& f, operator_args& x) -> bool {
    return f.object(x)
      .pretty_name("operator_args")
      .fields(f.field("args", x.args));
  }
};

class fluent_bit_operator final : public crtp_operator<fluent_bit_operator> {
public:
  fluent_bit_operator() = default;

  explicit fluent_bit_operator(operator_args args) : args_{std::move(args)} {
  }

  auto operator()(operator_control_plane& ctrl) const
    -> generator<table_slice> {
    // TODO: implement
  }

  auto name() const -> std::string override {
    return "fluent_bit";
  }

  auto detached() const -> bool override {
    return false;
  }

  auto location() const -> operator_location override {
    return operator_location::local;
  }

  auto optimize(expression const& filter, event_order order) const
    -> optimize_result override {
    (void)order;
    (void)filter;
    return do_not_optimize(*this);
  }

  friend auto inspect(auto& f, fluent_bit_operator& x) -> bool {
    return f.apply(x.args_);
  }

private:
  operator_args args_;
};

class plugin final : public operator_plugin<fluent_bit_operator> {
public:
  auto signature() const -> operator_signature override {
    return {.source = true};
  }

  auto parse_operator(parser_interface& p) const -> operator_ptr override {
    auto parser = argument_parser{
      name(),
      fmt::format("https://docs.tenzir.com/docs/connectors/{}", name())};
    auto args = operator_args{};
    parser.parse(p);
    return std::make_unique<fluent_bit_operator>(std::move(args));
  }

  auto name() const -> std::string override {
    return "fluent-bit";
  }

private:
  record config_;
};

I had started something local using crtp_operator. Is saver/loader the right abstractions? Aren't we going from void to events directly? I haven't studied the fluent-bit API in depth, perhaps there's a MsgPack parser/printer here as well.

I just assumed that we would go to bytes first, but looking at the included minimal API it seems to make more sense to go to directly to events.
This is an implementation choice that you should make as the main implementer or the feature.

This is an implementation choice that you should make as the main implementer or the feature.

Alrighty, can I take it from here?

@tobim I've split the outstanding todos in two tasklists for clearer responsibilities. In particular, please review da417f10e5f1eb11aca7ce825c160f90b2463836, which I needed to get things to link on macOS.
I'm surprised that the Docker build fails, as the script that installs the dev deps runs through just fine, and the plain Debian CI build works. Here I have no intuition how to proceed and would probably not invest any time myself. Any help would be appreciated.

I managed to get rid of any custom Fluent Bit plugins. This now works with a stock Fluent Bit package that you get on your system.
The consequence is that we now have to go through JSON when using Fluent Bit as sink (Fluent Bit input), because the library function flb_lib_push only supports JSON. We're also restricted to JSON when using Fluent Bit as source (= Fluent Bit output), although here Fluent Bit already provides us with the option of parsing the raw MsgPack.
Net effect: we have the potential to tune performance as source operator, but not as sink operator.
I've asked @edsiper in Slack:

[E]duardo, would you accept a patch that enhances in_lib and flb_lib_push to accept MsgPack in addition to JSON? It's already possible to consume MsgPack with out_lib, but not yet in_lib. This would make the library API symmetric.
I'm envisioning no API change, just making this work:
in_ffd = flb_input(ctx, "lib", NULL);
flb_input_set(ctx, in_ffd, "format", "msgpack", NULL);
flb_lib_push(ctx, in_ffd, msgpack_buf, msgpack_buf_len);

With a bit of luck, this will be a welcome future contribution as it doesn't change anything about the public API.

I'm surprised that the Docker build fails, as the script that installs the dev deps runs through just fine, and the plain Debian CI build works. Here I have no intuition how to proceed and would probably not invest any time myself. Any help would be appreciated.

The regular debian CI job does not build any plugins except for web. Plugins are built in a dedicated job matrix to which I just added fluent-bit.
The issue in the docker build was quite hairy and took me longer to fix than the static build. Behold: https://github.com/tenzir/tenzir/pull/3461/files#diff-4f95d2e348b638a037c80301070df229e817aaba51d9cd3b868976c7ae96c695R37-R69

CI error:
CMake Error at CMakeLists.txt:29 (dependency_summary):
  Unknown CMake command "dependency_summary".

@tobim IIRC we don't have that function in plugins. @dominiklohmann once told me to just remove it.

@tobim IIRC we don't have that function in plugins. @dominiklohmann once told me to just remove it.

This has been bugging us often enough. I refactored the functionality so it can be reused for standalone plugins in the latest commit.

We got it all green!

This PR is now feature complete and ready for testing. First tests showed that it works, e.g.,
fluent-bit random

Produces an infinite stream of messages. I was expecting that this works as well:
yes | head | tenzir 'fluent-bit stdin'

However, it takes too long for the Fluent Bit engine to ramp up before the input is already exhausted:
❯ yes | head | tenzir 'fluent-bit -X log_level=INFO stdin'
[2023/09/23 07:00:08] [ info] [fluent bit] version=2.1.9, commit=b1bf014171, pid=10812
[2023/09/23 07:00:08] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checks[2023/09/23 07:00:08] [ info] [ctraces ] version=0.3.1
[2023/09/23 07:00:08] [ info] [input:stdin:stdin.0] initializing
[2023/09/23 07:00:08] [ info] [ctraces ] version=0.3.1
[2023/09/23 07:00:08] [ info] [input:stdin:stdin.0] initializing
[2023/09/23 07:00:08] [ info] [input:stdin:stdin.0] storage_strategy='memory' (memory only)
[2023/09/23 07:00:08] [ info] [sp] stream processor started
[2023/09/23 07:00:08] [ warn] [input:stdin:stdin.0] end of file (stdin closed by remote end)
[2023/09/23 07:00:09] [ info] [engine] service has stopped (0 pending tasks)

I don't have an answer for that yet.

For the record, I reported a bug to the Fluent Bit devs that comes up when destroying an un-started engine. Then we crash in:
239           if (ctx->config->is_running == FLB_TRUE) {
-> 240               flb_engine_shutdown(ctx->config);
   241           }

Which comes from flb_output.c:
   479       params = FLB_TLS_GET(out_flush_params);
   480       if (params) {
-> 481           flb_free(params);   <---------- segfault 💥
   482       }
   483   }

This thread-local state never got initialized. The internal do_start function is the place where this state gets initialized in flb_output_prepare, via pthread_once(&flb_lib_once, flb_init_env). However, in our scenario the output got never correctly initialized and the the user hasn't invoked flb_start yet, so this state is never initialized. That is, we free a pointer that is never allocated, which is exactly the crash output:
tenzir(63774,0x16c21b000) malloc: *** error for object 0x16c21b000: pointer being freed was not allocated

We worked around this by calling flb_init_env that initializes the thread-local state as side effect.

CI is not looking good:
/__w/tenzir/tenzir/_build/lib/tenzir/plugins/libtenzir-plugin-fluent-bit.so /usr/lib/fluent-bit/libfluent-bit.so: cannot allocate memory in static TLS block

@tobim we may have to trouble-shoot this together.

/__w/tenzir/tenzir/_build/lib/tenzir/plugins/libtenzir-plugin-fluent-bit.so /usr/lib/fluent-bit/libfluent-bit.so: cannot allocate memory in static TLS block

Looks like you have to add something like:
export LD_PRELOAD=/usr/lib/fluent-bit/libfluent-bit.so

maybe together with /__w/tenzir/tenzir/_build/lib/tenzir/plugins/libtenzir-plugin-fluent-bit.so to the CI machine setup.
Or maybe even just your tests.sh script?

So this is just an error that occurs when loading the shared lib? I would imagine that this requires adjusting some CMake that does the linking properly so that this is not needed.
I would argue that the LD_PRELOAD monkey-patching is just a band-aid for an incorrectly linked shared lib. If CI runs into this, our users might as well. So we should fix it properly.

I've found that adding -ftls-model=local-dynamic might be a more appropriate fix.

The main issue is that libfluent-bit.so contains an embedded jemalloc with static thread local storage, and that is not supported with dlopen().
We could switch to build libfluent-bit.so ourselves on linux, but that would set this PR back several days, and I would also expect a few hours per month of maintenance that would have to go into this. Let's see if we can work around the problem in our CI first and try to push a fix upstream directly.

@tobim only with the dynamically linked library we get some errors:
ERROR: ld.so: object '/usr/lib/gcc/x86_64-linux-gnu/12/libasan.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/fluent-bit/libfluent-bit.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

The error cannot open shared object file hints that the file doesn't exist.


user12986714: Watch nascleaners.com(?:.au)?

user12986714 requests the watch of the watch_keyword nascleaners\.com(?:\.au)?. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

nascleaners\.com(?:\.au)? has been seen in 2 true positives, 0 false positives, and 0 NAAs.

Is there some evidence that the spammer will use the domain "nascleaners.com"? That domain does not appear to be registered.

The company appears quite local, so the use of nascleaners.com by the same company appears to be a relatively low probability.


Linux VMs unreachable through Guacamole

Describe the bug
I deployed the most recent release (v0.6.0) of AzureTRE from the AzureTRE-Deployment repo. Any Linux VM - I've tested both the available images - I create in a workspace is unreachable via Guacamole. Attempts to connect are simply rejected with the message "The remote desktop server is currently unreachable". The Linux VMs are definitely up and running. In contrast, Windows VMs function as expected - no issues connecting.
Steps to reproduce

Create any Linux VM in a workspace
Attempt to connect to it through Guacamole

Hi @craddm . We use cloud-init scripts to install packages - such as the desktop - to the Linux VMs. I imagine these are either taking a while to run, or have failed. In production we see people using custom VM images which means these scripts can often be avoided.
Is nexus configured in the environment? The Linux VMs have a dependency on Nexus for apt. Thinking about it we used to install it by default so might need adding to the docs.
Otherwise I would check the cloud-init logs via serial console - https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-overview
This blog is useful for debugging cloud init - https://blog.gripdev.xyz/2019/02/19/debugging-cloud-init-on-ubuntu-in-azure-or-anywhere/

Ah, so yes - Nexus wasn't fully set up. Turns out the Nexus deployment isn't working. From the cloud-init logs from the Nexus VM I can see that it's failing to import the MS signing GPG key -
('apt-configure', ValueError("Failed to import key 'BC528686B50D79E339D3721CEB3E94ADBE1229CF' from keyserver 'hkp://keyserver.ubuntu.com:80' after 3 tries: Unexpected error while running command.\nCommand: ['gpg', '--no-tty', '--keyserver=hkp://keyserver.ubuntu.com:80', '--recv-keys', 'BC528686B50D79E339D3721CEB3E94ADBE1229CF']\nExit code: 2\nReason: -\nStdout: \nStderr: gpg: keyserver receive failed: No data",))

Hello @craddm - I've had this same issue and it was transient, seems the keyserver was unreachable. Could you try the following steps to fix your Nexus instance and let me know how you get on?
Run the following in the Nexus VM's shell:

sudo cloud-init clean --logs
sudo cloud-init init --local
sudo cloud-init init
sudo cloud-init modules --mode=config
sudo cloud-init modules --mode=final

This should re-run all stages of cloud-init and hopefully you won't hit the gpg failure (I've just ran this successfully so fingers crossed!)

Hi @jjgriff93
This indeed worked, and I could successfully create accessible Linux VMs afterwards. But the original failure isn't due to intermittent unreachability of the keyserver. I'd deleted the service and reinstalled it several times before reporting the issue, and it failed the same way every time. So, just to check, I deleted the service and redeployed it, right after having it working, and it failed again in the same way. Possibly firewall rule changes haven't occurred when cloud-init is first running?

Thanks for investigating @craddm - I think that's a very likely theory in that case. Let me try to replicate on my end to confirm

Morning @craddm - just an update on this, I was able to replicate and think I have a fix. You're spot on that the firewall rules aren't being applied before the installation starts which is a recent regression. Shall update you when a fix is PR'd


Savitzky Golay filter should have a module to account for error covariance

The module scipy.signal.savgol_filter (Savitzky-Golay filter) implementation in scipy does not account for the presence of errors or error covariance in the data, a very common case for any scientific use. The errors seem to have been neglected in the scientific literature often as well.
A separate code which implements these cases can be found here:
https://github.com/surhudm/savitzky_golay_with_errors
It would be good to incorporate this in scipy.signal. Please advise me on whether this should be separate from savgol_filter or included in it as a special case.

@surhud is your code based on a paper? Maybe @WarrenWeckesser (the original author of savitsky_golay) understands what you mean, but it's not immediately obvious to me what your code is doing.

One nice thing about savitzky golay is that its just a filter and we know how to make that fast.  Doing an explicit polynomial fit to each segment is equivalent but extremely slow.
In [37]: timeit xscipy = signal.savgol_filter(xm, 7, 3)
1000 loops, best of 3: 923 µs per loop

In [38]: timeit xnew1 = savgol_filter_werror(xm, 7, 3, np.ones_like(xm))
1 loops, best of 3: 6.9 s per loop

In [63]: np.allclose(xscipy, xnew1)
Out[63]: True

Here xm has 50,000 elements and there is a factor of ~7500 difference in run time.
@surhud, do you have some motivating example using real data?  And how did you decide that this was the appropriate approach?

Sorry guys, I did not get notifications for your comments. I am attaching a pdf file of the document I wrote about this. The moderators on www.arxiv.org are giving me hard time to get this submitted. But I am attaching it here. See figure 1 in the attached file, where I sampled a simple sine curve with unequal errors on data points, and compare the modified filter to the traditional Savitzky-Golay filter. As expected, the modified filter does a better job by accounting for the errors and covariance.
I developed this code when I was facing a problem for using Savitzky Golay filter on noisy data with covariance (http://arxiv.org/abs/1601.06063v1 and submitted to the Astrophysical Journal).
paper.pdf

@ewmoore Once you start accounting for errors or covariance, the implementation will be slow, but it definitely performs better (as shown in the attached image from the paper uploaded in the previous comment). In my use case I did not have to worry about time, as I had to do the filtering just once on a fairly small data set.
The traditional SG filter works by convolving with a fixed set of coefficients, which makes it really fast, but it is not useful when the errors are unequal as the outliers dominate the filtering (which is what I wanted to avoid).

Just a general idea (I was playing a bit similarly with fft smoothing of data that is robust to outliers and to missing values. A home made algorithm based on some hints in the literature.)
Doing weighted regression (WLS) would not work with a convolution because of the different weights. However, it is possible to adjust the observations that are identified as outliers and shrink them towards the predicted/smoothed line, and then iterate a few times.
In my case, I used local MAD (median absolute deviation) to get the local variance estimate. With constant variance the standard error of the smoothing/prediction would just be constant, AFAICS.
(IIRC, I needed the median filter from scipy.ndimage or somewhere.)

@josef-pkt That idea may be ok for the case when errors are not known. However with known errors or covariance, doesn't it make sense to just use them instead? I guess you are making the case for speed versus accuracy. I would like to suggest that the option should be present in the module so that the user can make this choice rather than the software deciding on it.

@surhud   If you already know the standard errors/weights, then you can skip the step to estimate the error variance.
I'm not a scipy.signal person, so here is just IMO:
IMO, the Savitzky-Golay filter is useful because it reduces to a fast convolution because we have a fixed equal spaced grid as explanatory variable. Without this feature it is just a special case of local polynomial estimation. (which is in statsmodels under kernel regression for general x, and is slow because of the explicit python loop over all observations that hasn't been moved to cython.)
If changing an option in a function of a long signal increases the computation time by a very large amount, will be at least an unexpected surprise for users, unless they read and remember the big warning in the docstring.
My impression is that if we adjust the observations instead of using weights we get something very close (I didn't try the math) to your WLS version, but can be implemented using a few iterations of the fast convolution.

Phrased in another way: Doing the local polynomial fit with weights is useful, however I doubt merging it into Savitzky-Golay is a good idea.

I would agree with that assessment.

Although that requires explicit coding of the windowing used in the Savitzky Golay algorithm which this code does for you. Having a convenient function in a widely used library as scipy helps both to raise awareness (as I said many people use the filter even when the errors are known). I do not mean that this should be the standard used, it should be reverted to only when errors are known.
I should also add that it seems Matlab allows its Savitzky Golay filter to account for weights, although not the covariance.
As I said while raising the issue, I was seeking guidance as to whether to incorporate this here. If you feel it is not the right place I will keep the code separate.
Thanks for suggestions.

matlab specifies a weighting vector w with length f where f is frame, sounds like it's using a weighted polynomial, but weights relative to the window and not for given observations.
That's a change that doesn't affect the ability to use convolution.

Thanks for checking that. It makes much more sense then how they manage to get it to run fast In that case.
If everyone agrees to keep this separate then please feel free to close the issue.

Okay.  Let's close this then.

(I was asked in #6493) @surhudm
I need to learn and think about it . The first objection that comes to mind is that Savitzky-Golay requires equally spaced xdata so it sort of leans towards efficient. but restrictive usage (what others already said).
If we can figure out efficient implementation that I believe it can (and even should) be included. If not --- I tend to agree with others that it's a bit clunky "know-how" thing, which shouldn't be mixed with a classic efficient signal processing algorithm.

@nmayorov Even without equally spaced xdata, once you have unequal covariant errors, the efficiency is reduced. I am not at all advocating adding this in savgol_filter. But whether this can be morphed in to a separate function. As I said previously, I (and I can easily imagine many other astronomers who use savgol_filter) have to routinely deal with data with unequal covariant errors.
If there is an efficient implementation, then that is definitely great to know about and include. But shouldn't usability and accuracy trump efficiency?

Maybe be can try to include it into optimize/curve fitting, as piecewise_polyfit (or similar). What do you think?

Given that it wont be returning parameters, I am a little bit surprised
that you think optimize/curve fitting would be a better place for this. I
guess as long as we advertise this new function from savgol_filter page, it
should be ok.
On Wed, Aug 24, 2016 at 2:51 AM Nikolay Mayorov<EMAIL_ADDRESS>wrote:

Maybe be can try to include it into optimize/curve fitting, as
piecewise_polyfit (or similar). What do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scipy/scipy/issues/5767#issuecomment-241816603, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEMGuU2D6m6OyETbLIQvOCZvMWJmjzvlks5qizLxgaJpZM4HNGJg
.

Maybe it's a bad idea indeed. I just thought that it wasn't really approved in signal too.
By the way, can't this problem be solved using stats-models or scikit-learn?

I am not familiar with statsmodels. I am guessing one will need something like this and use the wrapper in my implementation:
http://statsmodels.sourceforge.net/devel/examples/notebooks/generated/gls.html
I can try to think hard about how to avoid matrix inversions. :)


use:directive doesn't work in HTML literal templates

Describe the bug
When use:funcName is used in HTML templates, it sets an HTML attribute with that name, instead of calling funcName.
Check the demo to see this in action.
Your Example Website or App
https://playground.solidjs.com/anonymous/423708dc-5d7c-4539-b110-4933ce95e4f0
Steps to Reproduce the Bug or Issue
Run the demo. Inspect the generated HTML in the dev console - funcName is set as attribute in CompHTML
Expected behavior
use:directive should have the same behavior as in JSX templates
Screenshots or Videos
No response
Platform

OS: [e.g. macOS, Windows, Linux]
Browser: [e.g. Chrome, Safari, Firefox]
Version: [e.g. 91.1]

Additional context
No response

In JSX use the key to turn it into an in-scope variable. In tagged template literals the compiled scope doesn't have access outside of the interpolated expressions ${}. I had assumed directives wouldn't work here and it is better to just use them as refs.
I suppose we could consider if this is fine. This would require some more parser code to get this pattern to match.
html`<div use:${directive}=${() => "text"} />`

But where we are right now it is a yet to be implemented feature.

I think the example you gave is fine, although not having use in template literals is not that big of a deal, as long as there's an alternative. When you say "use them as refs", I guess you mean something like this, right?
html<div ref=${(e) => directive(e, "text")}/>``

Yes exactly. I think you need to wrap the argument in a function to match the API but yeah.
html`<div ref=${(e) => directive(e, () => "text")}/>`


Unable to analyze lazy_static?

use lazy_static::lazy_static;
lazy_static! {
   static ref VEC: Vec<u8> = vec![0x18u8, 0x11u8];
}

fn main() {
   let b = VEC.last();
}

b will be analyzed as Option<{unknown}>.
Using the lastest version from compiling lastest master branch. Is RA unable to  analyze lazy_static yet?

Seem like it is because we do not support #[macro_export(local_inner_macros)] yet.
lazy_static! {
   static ref VEC: Vec<u8> = vec![0x18u8, 0x11u8];
}

The code above expanded to: (By command Expand macro recursively)
__lazy_static_internal!(()static ref VEC:Vec<u8>  = vec![0x18u8,0x11u8];
);

And __lazy_static_internal! macro is defined as #[macro_export(local_inner_macros)].

Fixed by #4305
OT: About 1 year ago, I started using rust-analyzer, but found that it cannot analyze lazy_static, but now it is finally done! 🎉

If I understand correctly, this has been released in 2020-05-11, right?
Rust Analyzer: Show RA version:

rust-analyzer version: 2020-05-18 (31611da)

I'm still getting all the {unknown}s.
Code sample: https://users.rust-lang.org/t/immutable-object-design-pattern/42772/13

Looks like I should use https://docs.rs/once_cell/1.4.0/once_cell/#lazy-initialized-global-data instead :smile:

Lol yes, at least partially,  the motivation for once_cell  was that IDEs at that time had troubles with lazy_static! :D

Yeah, @matklad is the best one to answer this question.

works on master

Image is broken

Image is broken

Yes, the "Uploading image.png..." indicates the post was submitted before the image finished uploading.
In any case, lazy_static! in global scope seems to work, but when you have a lazy_static! block in a function, e.g. to keep implementation details inside, it still shows {unknown}.
I've tested this on nightly RA ("rust-analyzer.updates.channel": "nightly"), since the last comment was "works on master".
Compare:

Code for easy copy-pasting for those trying to repro:
Global:
use lazy_static::lazy_static;

lazy_static! {
    static ref VEC: Vec<u8> = vec![0u8, 1u8, 2u8];
}

fn main() {
    let x = VEC.last();
}

In function:
use lazy_static::lazy_static;

fn main() {
    lazy_static! {
        static ref VEC: Vec<u8> = vec![0u8, 1u8, 2u8];
    }

    let x = VEC.last();
}

That problem is presumably #1559


I can't change the admin password after complete the installation

I can't change the admin password after complete the installation.

I added tool that allow to init root user after application instalation.
Please, check 6th step of https://github.com/yeesoft/yii2-yee-cms/blob/master/README.md
Please, check if you have this new file https://github.com/yeesoft/yii2-yee-cms/blob/master/console/controllers/InitAdminController.php in your application in /console/controllers/ folder


gsasl: 1.10.0 -> 2.0.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/gsasl/versions.
meta.description for gsasl is: "GNU SASL, Simple Authentication and Security Layer library"
meta.homepage for gsasl is: "https://www.gnu.org/software/gsasl/"
meta.changelog for gsasl is: ""
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.0.1 with grep in /nix/store/96sp573hfnz469jsyb4adp8bwy0a9208-gsasl-2.0.1
found 2.0.1 in filename of file in /nix/store/96sp573hfnz469jsyb4adp8bwy0a9208-gsasl-2.0.1

Rebuild report (if merged into master) (click to expand)

15 total rebuild path(s)

15 package rebuild(s)

First fifty rebuilds by attrpath
automysqlbackup
curlFull
dico
gobby
gsasl
gsignondPlugins.sasl
haskellPackages.gsasl
libinfinity
libjreen
mailutils
mpop
msmtp
pokerth
pokerth-server
vmime

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/96sp573hfnz469jsyb4adp8bwy0a9208-gsasl-2.0.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A gsasl https://github.com/r-ryantm/nixpkgs/archive/608c8f2ff1fe121a005096d0299ddd11579d83c5.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/96sp573hfnz469jsyb4adp8bwy0a9208-gsasl-2.0.1
ls -la /nix/store/96sp573hfnz469jsyb4adp8bwy0a9208-gsasl-2.0.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package failed to build:

vmime

13 packages built:

automysqlbackup
curlFull
dico
gobby (gobby5)
gsasl
gsignondPlugins.sasl
libinfinity
libjreen
mailutils
mpop
msmtp
pokerth
pokerth-server

Maintainer pings
cc @shlevy for testing.

On nixos x86_64:
builder for '/nix/store/jn8ghlmwvc2q4ks81qn79123kmzh3inn-vmime-0.9.2.drv' failed with exit code 2; last 10 log lines:
        |              ^
  /build/source/src/vmime/security/sasl/SASLContext.cpp:193:15: error: 'GSASL_KERBEROS_V5_INTERNAL_ERROR' was not declared in this scope
    193 |         ERROR(GSASL_KERBEROS_V5_INTERNAL_ERROR)
        |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /build/source/src/vmime/security/sasl/SASLContext.cpp:158:14: note: in definition of macro 'ERROR'
    158 |         case x: msg += #x; break;
        |              ^
  make[2]: *** [CMakeFiles/vmime.dir/build.make:2291: CMakeFiles/vmime.dir/src/vmime/security/sasl/SASLContext.cpp.o] Error 1
  make[1]: *** [CMakeFiles/Makefile2:135: CMakeFiles/vmime.dir/all] Error 2
  make: *** [Makefile:156: all] Error 2

This patch should fix it: https://github.com/kisli/vmime/commit/c750e899b8b3a76379167148468e164b7e52f6e2.patch

Result of nixpkgs-review pr 184190 run on x86_64-darwin 1

3 packages marked as broken and skipped:

libjreen
pokerth
pokerth-server

9 packages built:

curlFull
dico
gobby
gsasl
haskellPackages.gsasl
libinfinity
mailutils
mpop
msmtp

nixpkgs-review behaves weirdly, it doesn't seem to build anything but then lists packages as built (it finishes in seconds):
 nixpkgs-review pr 184190
$ git -c fetch.prune=false fetch --no-tags --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/184190/head:refs/nixpkgs-review/1
$ git worktree add /Users/smancill/.cache/nixpkgs-review/pr-184190-2/nixpkgs 676cc82ff770ffd0e905e19840134757a9722d06
Preparing worktree (detached HEAD 676cc82ff77)
Updating files: 100% (31410/31410), done.
HEAD is now at 676cc82ff77 Merge pull request #186627 from r-ryantm/auto-update/grafana_reporter
$ git merge --no-commit --no-ff d45af719e7c7d0afcbde55ce0db89306805888db
Automatic merge went well; stopped before committing as requested
$ nix --experimental-features nix-command build --no-link --keep-going -f /Users/smancill/.cache/nixpkgs-review/pr-184190-2/build.nix --log-format bar-with-logs --max-jobs 1

Link to currently reviewing PR:
https://github.com/NixOS/nixpkgs/pull/184190

3 packages marked as broken and skipped:
libjreen pokerth pokerth-server

9 packages built:
curlFull dico gobby gsasl haskellPackages.gsasl libinfinity mailutils mpop msmtp

error: build log of '/nix/store/bn810llycji5jzi0sgbf42mc8zf6zn3i-curl-7.84.0.drv' is not available
error: build log of '/nix/store/qfmflx4ahgmrq405af6691ag4g8mis49-gsasl-0.3.7.drv' is not available
error: build log of '/nix/store/mmxldigc8n4p730815ax3naqg7i640xq-msmtp-1.8.20.drv' is not available
$ /nix/store/f9vkqzzr2y4s1cw914qd1mcr0adnr3id-nix-2.9.1/bin/nix-shell /Users/smancill/.cache/nixpkgs-review/pr-184190-2/shell.nix

Superseded by https://github.com/NixOS/nixpkgs/pull/186893.


Update zeebe.decorator.ts

Hey,
The company I worked for payk that owned this repo went down. I've created a new maintained fork
https://github.com/danshapir/nestjs-zeebe
Please create your PR there.

I also posted this problem there.
The fact is that nestjs-zeebe does not indicate transport. And you have to rewrite ZeebeWorker. Since when transport is not specified, it is suitable for everyone, including Kafka and others.


Updating entry with attach approach is not working if new value of property is null

To update db entry I want to use attach approach to avoid fetching entry from the db.
Like described here https://gist.github.com/tombowers/6f99eb132cfec185b951#file-update-without-select-cs
All works well besides the case when the new field value is null. See the example below.
Steps to reproduce
Assume we have a db entity
public class School
{
    public int Id { get; set; }
    public string Name { get; set; }
}

In the db, we already have an entry of School with Id = 1 and Name = "School Name"
In next case, update works as expected
...
var schoolToUpdate = new School {
    Id = 1
};
_dbContext.Attach(schoolToUpdate);
schoolToUpdate.Name = "New School Name";
await _dbContext.SaveChangesAsync();
...

But, if new school name is null, it is not updating the db, because original value is also null and IsModified flag is not changed to true. (School.Name in the db is not null)
...
var schoolToUpdate = new School {
    Id = 1
};
_dbContext.Attach(schoolToUpdate);
schoolToUpdate.Name = null;
await _dbContext.SaveChangesAsync();
...

Is it a bug, or is it by design? Do you have any suggestions on how to make it works?
Mark field manually as modified will not work because we have about 20 fields and it will uglify our code. So now we use AutoMapper null substitute approach and setting string.Empty if the value is null:
cfg.CreateMap<SchoolEditModel, School>()
                    .ForMember(dest => dest.Name, opts => opts.NullSubstitute(string.Empty));

In this case OriginalValue is null but NewValue is "" and IsModified flag is true; But it results to "" in the db instead null.
Further technical details
EF Core version: 2.2.3
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Windows 10
IDE: Visual Studio 2017 15.9.7

@m-demydiuk Indeed, this approach is a definite anti-pattern and should never be used.
To elaborate. The fundamental decision point here is whether you want to use partial updates (that is, only sending updates for properties where the value has actually changed) or always send updates for all properties in the entity. Of course, partial updates sounds better, but it is predicated on being able to know what has changed. Otherwise how can you know what updates to send? This is where the pattern above falls down--it basically makes the assumption that all the original values were null, and any update will be from null to non-null, and never non-null to null. This is clearly a false assumption in general.
So, what to do? The easiest thing is to not do partial updates. That is, have all the correct values in the entity instance and then call Update and SaveChanges.
To retain partial updates, either:

If you know statically which properties have changed, then Attach the entity and then explicitly set those properties as modified using something like context.Entry(foo).Property(e => e.Bar).IsMdified = true.
Make sure the original values of the entity are maintained so that it can be attached with all the correct values. This usually means sending them to and from the client.
Query for the entity (e.g. using DbSet.Find) so that it is populated with the values already in the database before making updates to it.


Reference time from pcap header rather than keyframe

Hi,
I'm looking at using Arista ASIC timestamps but right now the data I have has no key frames... It's not really relevant to me as I'm primarily interested at the delta of specific packets, but I need <10ns resolution. This means any reference time, good or bad, will work for me as the asic counter alone will retain accurate deltas packet after packet.
I'd like to implement an option to take pcap header as reference time. It would help if you could provide sample pcaps (just a couple of packets will be sufficient) with and without keyframes so I could use it for development and testing. Having sample pcaps could also be great to add automated tests.
Regards,
Thomas

Hi Thomas,
Just to confirm I understand this right, are you looking for a pcap with
timestamped data packets and no keyframes? Any preference on the
timestamping being performed before-FCS or by replacing FCS?
Thanks,
Andrei
On Wed, Mar 11, 2015 at 4:16 PM, Thomas Guyot-Sionnest <
<EMAIL_ADDRESS>wrote:

Hi,
I'm looking at using Arista ASIC timestamps but right now the data I have
has no key frames... It's not really relevant to me as I'm primarily
interested at the delta of specific packets, but I need <10ns resolution.
This means any reference time, good or bad, will work for me as the asic
counter alone will retain accurate deltas packet after packet.
I'd like to implement an option to take pcap header as reference time. It
would help if you could provide sample pcaps (just a couple of packets will
be sufficient) with and without keyframes so I could use it for development
and testing. Having sample pcaps could also be great to add automated tests.
Regards,
Thomas
—
Reply to this email directly or view it on GitHub
https://github.com/arista-eosext/PCAPDecoder/issues/1.

The switch writes (or can write) ASIC timestamps, and i'm looking at getting the high resolution timestamp into the pcap header (i.e. re-writing them). The pcap header is fairly accurate too (hw timestamping, ~10ns resolution), but not precise enough and more importantly it's taken later down the line, not at the switch port.
A reload may be needed for the keyframes to work, and with it or not I'm worried the first capture packets before the keyframe may end up missing their timestamp. Since I care only about the delay between specific packets and not the actual time accuracy I think the timestamp from the pcap packet headers could be used for synchronization. It could give a very imprecise time compared to keyframes, but the relative time between packets would be retained as long as there is no gap in the flow (~6.15 sec from what I read earlier...)
Actually going even further, I think the pcap timestamps could also be used to calculate the number of rollovers on long gaps so the script could remain perfectly synchronized... But that's not my priority right now.

I attached a few pcaps - simply remove the '.png' extension from the file - I hope that helps.
Can you please clarify where you got the ~10ns resolution number from? As far as I know (https://wiki.wireshark.org/Development/LibpcapFileFormat) you can get either microsecond precision or nanosecond precision with libpcap (nanosecond using the latest version on GitHub).
And just to clarify if I understand this right - you wish to use the pcap timestamp in the first packet as a reference and then replace the following pcap timestamps by ones generated based on the reference timestamp (the one from the first packet) and the difference between the ASIC time in the packets and the ASIC time in the reference packet. Is that right?
If you need more pcaps or more specific ones, please unicast me<EMAIL_ADDRESS>Thanks!

The file attached above is actually ZIP archive - please rename it accordingly.

Thanks.
The 10ns resolutions comes from the capture box. It's saved as ns resolution but the last digit is always 0.
And yes, I wast to use the first pcap header as reference time.
Regards,
--
Thomas

Hey, for the heads up, the python library used here doesn't support pcap-ng... this is a problem since I would have to convert pcaps beforehand and then on write-out I'd loose the ns resolution, which isn't very useful.
It seems it'll be easier to just add Arista counter support to our scripts instead... I guess you can either close this bug or make it depend on a new one for pcap-ng support....
Regards,
--
Thomas

At the time when we started this project there was no Python library for
pcap-ng (as far as I can remember).
On Thu, Mar 19, 2015 at 12:00 PM, Thomas Guyot-Sionnest <
<EMAIL_ADDRESS>wrote:

Hey, for the heads up, the python library used here doesn't support
pcap-ng... this is a problem since I would have to convert pcaps beforehand
and then on write-out I'd loose the ns resolution, which isn't very useful.
It seems it'll be easier to just add Arista counter support to our scripts
instead... I guess you can either close this bug or make it depend on a new
one for pcap-ng support....
Regards,
Thomas
—
Reply to this email directly or view it on GitHub
https://github.com/arista-eosext/PCAPDecoder/issues/1#issuecomment-83523794
.

Hello guys,
Did you end up getting anywhere with adding pcap-ng support for either the pcap decoder or the tsdump scripts? It'd be great to be able to use this with nanosecond-accurate timestamps in our pcap files

I don't know much about doing that with the modules used in this tool, nor pcap-ng support, but nseclibpcap is very similar to normal pcap format with nanosecond support... in fact it would be trivial to transcode plain pcap format to nseclibpcap using pack/unpack (we do it in Perl but this is proprietary code). If you need other formats you can also convert to/from nseclibpcap using Wireshark or its command-line tools.
That said, I found out for latency calculation over very short periods (ex. packet forwarding latency or app response time) are best done with the raw timestamps directly, unless you have more advanced tooling such as Corvil appliance that does it for you. The thing with conversion to pcap format is that you have to make time adjustments based on keyframes to find wall-clock time, and if you calculate latencies based on this resolved timestamp it will introduce jitter. The most accurate way to calculate latencies it to therefore decode raw timestamps directly and only consider the delta between packets (Arista ticks * 20 / 7 -- "20/7", or if you prefer "1000/350", being the exact interval in ns between each Arista tick).
When doing so you have to of course handle counter rollovers happening every ~6.1s... if you need to get deltas over 6s you'd probably best keep an internal 64bit representation and use keyframes to stay in sync.


add molecular-dynamics

Thank you for contributing! Please confirm this pull request meets the following requirements:

[x] I followed the contributing guidelines: https://github.com/github/explore/blob/master/CONTRIBUTING.md
[x] I have no affiliation with the project I am suggesting (as a maintainer, creator, contractor, or employee).

Which change are you proposing?

[ ] Suggesting edits to an existing topic or collection
[x] Curating a new topic or collection
[ ] Something that does not neatly fit into the binary options above

Editing an existing topic or collection
I'm suggesting these edits to an existing topic or collection:

[ ] Image (and my file is *.png, square, dimensions 288x288)
[x] Content (and my changes are in index.md)

Please replace this line with an explanation of why you think these changes should be made.

Curating a new topic or collection

[x] I've formatted my changes as a new folder directory, named for the topic or collection as it appears in the URL on GitHub (e.g. https://github.com/topics/[NAME] or https://github.com/collections/[NAME])
[x] My folder contains a *.png image (if applicable) and index.md
[x] All required fields in my index.md conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

The molecular dynamics is a very important computational method in physics, chemistry, biology, and material science. It is also a bridge between the computer science and the natural science.

Please note: we will close your PR without comment if you do not check the boxes above and provide ALL requested information.

Thanks so much for your first contribution!  You rock, @njzjz!

CI went wrong, but it's not related to my PR.

CI went wrong, but it's not related to my PR.

👍🏻 Looks like @MikeMcQuaid fixed that in #2406. If you merge in main we should be green and good to merge 😄


Independently process LoRa adapters and base model

Hello,
As I understand, during inference, the current library merges the base model weights with the learned LoRa adapters to avoid increasing in latency, as in h=(W+BA)x. However, I'm wondering if there's a way to process the base model and the learned adapters independently, as in h = Wx+BAx. The Wx computation is easy to implement (just load the original base model and perform inference) but I'm stuck on how to perform the BAx computation.
For my particular use case, I would like to benchmark the runtime of the BAx computations relative to Wx on the Whisper model.
Thank you for your help!

Thanks for the question. If you just load the base model and then the LoRA adapter, it is not merged into the base weight. Merging has to be done explicitly by the user. As such, the computation is separate as you described. For instance, for lora.Linear, this happens in this line:
https://github.com/huggingface/peft/blob/58afb34ea03ae1166fc331adee0b4646e46db6dc/src/peft/tuners/lora/layer.py#L556
So you should already be able to benchmark this.

Thank you for the explanation!