VC++ Redistributables Fail to install on images with a Windows Server 2019 datacenter host

Please fill out all the sections below for bug issues, otherwise it'll be closed as it won't be actionable for us to address.
Describe the bug
Opening this ticket as the result of a bug report and discussion in the dotnet-framework-docker repository (https://github.com/microsoft/dotnet-framework-docker/discussions/1017)
I am Unable to install Latest Visual C++ Redistributables (found here: https://aka.ms/vs/17/release/vc_redist.x64.exe) into windows containers. When the executable vc_redist.x64.exe runs inside the container either during a build or during runtime, nothing happens. This seems to be particularly an issue with Windows Server 2019 hosts, as when I tried this on a 2022 host VM, and upgraded the original host to 2022, the problem was resolved.
To Reproduce

Build the following dockerfile docker build -t redist_test:latest .

# escape=`

# docker build -t redist_test:latest .
# docker run --rm -it --entrypoint powershell redist_test:latest

ARG BaseRepo=mcr.microsoft.com/dotnet/framework/aspnet
ARG BaseRepoTag=4.7.2

FROM $BaseRepo:$BaseRepoTag as base
SHELL ["Powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; $verbosePreference='Continue';"]

RUN [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; `
    Invoke-RestMethod -Uri https://aka.ms/vs/17/release/vc_redist.x64.exe -OutFile vc_redist_x64.exe; `
    Start-Process -filepath C:\vc_redist_x64.exe -ArgumentList "/install", "/passive", "/norestart", "'/log C:/redist_log.txt'" -Passthru | Wait-Process;

Run the image docker run --rm -it --entrypoint powershell redist_test:latest
Notice no log file from executable was created
Search for signs that the redistributable was installed in the container, such as Get-Item -Path HKLM:\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0\VC\Runtimes\x64, and you won't find anything.
Any assemblies that require the redistributables to run will fail to load.

Expected behavior
The executable runs and installs the redistributables as expected. The results of Step 4 in the reproduction steps would look the below indicating installation success:
Hive: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0\VC\Runtimes

Name                           Property
----                           --------
x64                            Version   : v14.32.31332.00
                               Installed : 1
                               Major     : 14
                               Minor     : 32
                               Bld       : 31332
                               Rbld      : 0

Additionally, assemblies that rely on the redistributables as a dependency successfully load.
Configuration:

Edition: Windows Server 2019 Datacenter
Base Image being used: Windows Server core
Container engine: docker
Container Engine version: master-dockerproject-2022-03-26

Additional context
Original Bug Report in other repo: https://github.com/microsoft/dotnet-framework-docker/issues/1014
Original Discussion where the fix was found: https://github.com/microsoft/dotnet-framework-docker/discussions/1017

Hey @Juarezhm, any thoughts on this?

I have access to a Windows Server 2022 host and ran the repro steps there, and confirmed that the problem does not reproduce on Server 2022.
@Jeb135 If you force the container to run in HyperV isolated mode using the "--isolation hyperv" switch, does the problem still occur?  I am wondering if it is related to the container mode.
Of course, Server 2022 has been on the market for over a year, so I would recommend that you move up to Server 2022 in any case.

I would recommend that you move up to Server 2022 in any case

No problem, just send us licenses.
MS claims Server 2019 is supported until Jan 9, 2029.

@michbern-ms I cannot confirm this as the Server 2019 machine has long been upgraded to resolve this issue, but I do remember attempting hyperv isolation with no success.

@Jeb135 Thank you for getting back to me.   I took the time this weekend to try this out - I set up a Server 2019 machine and ran your instructions exactly.
Unfortunately, I was not able to reproduce the problem.
I used the following versions:

And your Dockerfile and commands copy-pasted into the machine.
At the end, I see:

I am sure that you tried all this and documented it carefully.  I am wondering if we had a transitory failure, or if some issue not immediately related to Containers caused this, but is now fixed.   In any case, there is nothing further I can do here.  As far as I can tell, Server 2019 customers should be good to go at this point.
Thanks,
Michael

@michbern-ms I booted up a snapshot from 2 years ago of the offending machine to try this out and also did not run into this problem. There must be another factor or configuration issue involved that I am missing that occurred between this snapshot and when I ran into the problem a few months ago. In any case, looks like this one has gone cold. Thanks for looking into it.

Thanks very much, @Jeb135 , for trying the old snapshot!


Page outline synthesis & planning

User story
As a user, I want to find what I need quickly and understand it easily so I can do my job.
Background & context
The core model provides a way to anchor content plans around our goals, user needs, and the user journey, creating guidance for consolidating redundant content and IA development. With more specific page outlines/flows for the main content categories created, this is to any wrap up and synthesize between different team members work to prepare for the next phase.
Open questions

How much this is necessary, depending on how synced our work will already be
What approval might be needed for implementation plans

Working links
Core model workshop deck
Acceptance criteria

[ ] Update service blueprint with relevant learnings
[ ] Update the questions pool with any questions that came up
[x] Work synthesized and put into plans
[x] Summary and plans shared with WebX design

Sheet with links to outlines and tentative plans


Customize verbosity

good job. this is a pre-requisite step toward integrating https://lib.rs/crates/tracing-log.
and we then need to add liberally many more logging information across both lychee-lib and lychee-bin.

@MichaIng, that's another one, which could impact the output of your pipelines. Would you mind testing the change before the next release? The PR got merged to master, so you should be able to build master and see if everything still works as expected now. Thanks! 🙏

Will do that.


adding SNA method to build meaning VCs

WIP - do not merge

@jamescwu Here's the pull request - looks like @zemlni merged


Make list of components that other can use from our codebase

Out of the box, people could incorporate the following into their projects:

CrunchyDB (already being used)
ClamAV File Scanner for S3
Metabase deployment

With some modifications, the following could be incorporated into others project:

ZAP Scan
Trivy Scan
Schemaspy

With some work, we can likely turn our repo into a template for future applications based on NextJS, GraphQL, Postgres, etc. Essentially remove the "portal functionality" and leave a bare project that would include by default:

A CI/CD Pipeline like ours
Jest testing
e2e tests
DB unit testing
Security testing (Trivy + ZAP)
Static code analysis (CodeQL + SonarCloud)
Helm deployed (might be beneficial to turn it into GitOps based Helm)
BCGov OpenShift based for HA


CSS conflict issues

Currently when uptaking seeds in core, we are having CSS style and variable conflict issues with ui-components which include:
Our spacing variables are named the same thing but have different values
We are doing a CSS "reset" of some styles like removing all background color / border from buttons

looks like this is closed @emilyjablonski - pls correct if I'm wrong


TwentyTwenty has height issues when used with Owl Carousel

This has always been a bit of an on-going issue since I started using these two items together. I know that I definitely want to use them both, and there must be a way to do it, as I've seen it used on other sites.
Anyway, I've tried using the 20/20 on it's own, and it seems to work fine when static. I've also used Owl Carousel 2 on it's own with static images, and that seems to work fine too.
But when I try to combine them, so I have multiple instances of 20/20 as 'slides' in Owl Carousel, there is an issue regarding the height of the 20/20 container, when resizing the windows.
This leads to issues when rotating mobile devices, and changing browser sizes. Usually, it sometimes gets fixed by resizing and after letting go of the mouse buttons, it may change. But sometimes it does not.
I've tried using the window.resize in the past, but that doesn't seem to make any difference.
A good example of this can be found at https://www.bellavou.co.uk/face/facelift/ - Try resizing the browser window when viewing the 20/20 slider, and after a while, you'll start to see gaps making the slide container appear huge, and not resizing with the browser.
I'm not sure what I can do, but I'm open to all suggestions and all help is appreciated! Sorry for the wordy issue...

Hey @leecollings I see the issue. Seems like both Owl and TwentyTwenty make use of the window resize event. It appears that Owl updates the width of the carousel container during the resize event AFTER TwentyTwenty has already performed its resizing calculations.
But there is hope! I found a hack to make it work for like you would expect. What it essentially does is listens for the Owl resized.owl.carousel event which indicates the Owl carousel container has finished resizing itself. At this point we can trigger another window resize event which TwentyTwenty looks for and it will base its calculations on the new container width. I tried it on your site and it appears to fix the issue:
jQuery('.owl-carousel').on('resized.owl.carousel', function(event) {
  jQuery(window).trigger("resize");
});

There are probably better solutions out there ... but I hope this helps you out. Cheers!

Hey there, thanks for your comment, that's been really helpful.
Although maybe I'm doing it wrong, but although the resize is working, it seems everytime the window is resized, I'm just adding another instance of 2020 ontop of my existing one (I can tell this from the slider line getting a thicker shadow everytime).
// load TwentyTwenty after the page has loaded
$(window).load(function(){
    $(".twentytwenty").twentytwenty();
});

// reload TwentyTwenty on page resize/orientation
$('.owl-carousel').on('resized.owl.carousel', function(event) {
    $(window).trigger("resize");
    $(".twentytwenty").twentytwenty();
});

Am I able to rather than initialise 2020 for another time, just re-initialise it?

If I'm not mistaken I don't think it's necessary to re-initialize TwentyTwenty. You could rewrite the code as follows:
// load TwentyTwenty after the page has loaded
$(window).load(function(){
    $(".twentytwenty").twentytwenty();
});

// reload TwentyTwenty on page resize/orientation
$('.owl-carousel').on('resized.owl.carousel', function(event) {
    $(window).trigger("resize");
    // Commenting this code out
    //$(".twentytwenty").twentytwenty();
});

However it's been a little while since I've looked at the code. If that doesn't work can you post a link to your site? Thanks!

Hmm it still doesn't seem to be resizing if I emulate orientation changes on iPad viewing.
It's fine when going from portrait to landscape, as the slider scales up, but when going back to portrait, the slider scales down but it's container height remains, leaving an ugly white box of space underneath.
You can view the page at: http://development.bellavou.co.uk/face/facelift/ but there's authentication applied to this subdomain. Are you able to send me a private message or email requesting this? or add me on Skype


Remove unused model diagram

We no longer use tags for our Module, unlike Person in addressBook. Lets fix it!

Codecov Report
Base: 40.01% // Head: 40.01% // No change to project coverage :thumbsup:

Coverage data is based on head (a21735e) compared to base (d759384).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #177   +/-   ##
=========================================
  Coverage     40.01%   40.01%
  Complexity      471      471
=========================================
  Files           122      122
  Lines          3149     3149
  Branches        313      313
=========================================
  Hits           1260     1260
  Misses         1781     1781
  Partials        108      108

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Add sync command api

Hello, thanks for this project
I mostly use this in repl so I end up writing a lot of await $c1 await $c2, I feel like those awaits are really unneeded and verbose. It would be nice if ax supported a sync api using Deno.spawnSync

I've been thinking about this and it wouldn't be much better because instead of adding await  on the front you would have to remember to type .spawnSync() at the end or something like that to kick it off. Additionally, it blocking synchronously would prevent some scenarios from being possible... the library would need to throw if the user did certain things.
So, overall I don't think this should be done; however, I thought of what I think is a better alternative. See here: https://github.com/denoland/deno/issues/15209

I don't get the typing of spawnSync part, the user would just type $'ls' and it would internally use it.  (for example that's what I do here https://github.com/sigmaSd/ds)

Additionally, it blocking synchronously would prevent some scenarios from being possible... the library would need to throw if the user did certain things.

I propose to not replace the current api with this but to add it as an alternative api
I guess https://github.com/denoland/deno/issues/15209 might work as well but personally I'm not fan of adding repl specific behavior, so the code you write in the repl you can't just copy paste it into a file (you'll need to  add back those await)

There's builder steps per command, so it would probably need to be something like:
$`echo $MY_VAR`.env("MY_VAR", "test").spawnSync();
// or
$.env("MY_VAR", "test").spawnSync(`echo $MY_VAR`);

So this would be just as verbose as adding await.

I guess https://github.com/denoland/deno/issues/15209 might work as well but personally I'm not fan of adding repl specific behavior, so the code you write in the repl you can't just copy paste it into a file (you'll need to add back those await)

I don't think this is that big of a deal because if someone wants to be able to copy and paste then they can add await keywords.
Anyway, I'm marking this as something that won't be done because adapting this API would be just as verbose and a synchronous API leads to limitations where things like only synchronous writers can be provided to pipe or a potential deadlock could occur if a process is waiting on stdin via a writer that executes in JavaScript (so the library would need to throw in this scenario... just leads to a lot of code to maintain for not much benefit with way more considerations).


sumneko_lua fails to install

Problem description
When installing the sumneko_lua server, the installation fails.
Neovim version (>= 0.6)
0.7.0
Operating system/version
Ubuntu 18.04
I've recently downloaded the latest plugin version of both nvim-lsp-installer and nvim-lspconfig

[ ] Yes

Affected language servers
sumneko_lua
Actual behavior
The server fails to install
Expected behavior
The server should install successfully
LspInstallInfo output
Pending servers (1)
    ◍ sumneko_lua (failed)
        ...ler/lua/nvim-lsp-installer/core/managers/github/init.lua:28: Failed to fetch latest release from GitHub API.

Installation log
[INFO  Fri 13 May 2022 17:28:41 BST] ...ed/nvim-lsp-installer/lua/nvim-lsp-installer/ui/init.lua:732: Starting install server_name="sumneko_lua", requested_version=""
[ERROR Fri 13 May 2022 17:28:41 BST] ...im-lsp-installer/lua/nvim-lsp-installer/core/process.lua:153: Failed to spawn process. cmd="gh", err="ENOENT: no such file or directory"
[INFO  Fri 13 May 2022 17:28:42 BST] ...r/lua/nvim-lsp-installer/core/managers/github/client.lua:76: Failed to find latest release. repo="sumneko/vscode-lua", opts={
  include_prelease = false
}
[ERROR Fri 13 May 2022 17:28:42 BST] ...installer/lua/nvim-lsp-installer/core/installer/init.lua:79: Installation failed, name="sumneko_lua", error="...ler/lua/nvim-lsp-installer/core/managers/github/init.lua:28: Failed to fetch latest release from GitHub API."
[INFO  Fri 13 May 2022 17:28:42 BST] ...ed/nvim-lsp-installer/lua/nvim-lsp-installer/ui/init.lua:751: Installation completed server_name="sumneko_lua", success=false

Healthcheck
nvim-lsp-installer: require("nvim-lsp-installer.health").check()
========================================================================
## nvim-lsp-installer report
  - OK: neovim version >= 0.7.0
  - WARNING: **Go**: not available
  - WARNING: **cargo**: not available
  - WARNING: **Ruby**: not available
  - WARNING: **RubyGem**: not available
  - WARNING: **Composer**: not available
  - WARNING: **PHP**: not available
  - WARNING: **julia**: not available
  - OK: **sh**: `Ok`
  - OK: **bash**: `GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)`
  - OK: **tar**: `tar (GNU tar) 1.29`
  - OK: **gzip**: `gzip 1.6`
  - OK: **wget**: `GNU Wget 1.19.4 built on linux-gnu.`
  - OK: **python3**: `Python 3.6.9`
  - OK: **node**: `v16.15.0`
  - OK: **curl**: `curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3`
  - OK: **java**: `Ok`
  - OK: **javac**: `javac 11.0.7`
  - OK: **npm**: `8.5.5`
  - OK: **pip3**: `pip 19.1.1 from /home/cdimitroulas/.local/lib/python3.6/site-packages/pip (python 3.6)`

Screenshots

Seems like it's trying to use a gh command which I don't have installed

This is also happening to me for the omnisharp server

Hello! You're probably being rate limited by GitHub's API. I just merged https://github.com/williamboman/nvim-lsp-installer/pull/697 which helps diagnose this. Could you try pulling the latest version and run :checkhealth nvim-lsp-installer again 🙏 ?

Happens to me as well. The problem seems to be in this statement
https://github.com/williamboman/nvim-lsp-installer/blob/245f1bece5b906bd7f2a11e684c3619055b6ce1d/lua/nvim-lsp-installer/core/managers/github/client.lua#L66

Happens to me as well. The problem seems to be in this statement

Ah! That was changed earlier today - I reverted it just now so should work now. I did not think through the logic well enough 😅

Fixed in #699.

Thanks for the quick turnaround!


feat(blockchain-tree): fix is_block_inside_chain canonical fork check

Should close #5927.

as @rakita commented on the PR, we should panic on non-existing chain id (or return an error) rather than silently ignore it. the bug in #5927  was caused by failed database insert (can be seen in preceding logs)


Move operations docs to /operations

Move the pages in the dAPIs doc set to the Operation docs set (/operatons).

@kolenic-martin No real test here other than see if https://docs.api3.org/operations comes up.


Include a field's AliasChoices in the model's JSON schema

Initial Checks

[X] I have searched Google & GitHub for similar requests and couldn't find anything
[X] I have read and followed the docs and still think this feature is missing

Description
from pydantic import BaseModel, Field, AliasChoices

class MyModel(BaseModel):
    my_field: str = Field(validation_alias=AliasChoices("my_field", "field_alias"))

print(MyModel.model_json_schema())

prints {'properties': {'my_field': {'title': 'My Field', 'type': 'string'}}, 'required': ['my_field'], 'title': 'MyModel', 'type': 'object'}
when (considering the alias(es)) the schema should include the property "field_alias": { "$ref": "#/properties/my_field"} (another property whose schema is just the same as the field it is aliasing).
Affected Components

[ ] Compatibility between releases
[ ] Data validation/parsing
[ ] Data serialization - .model_dump() and .model_dump_json()
[X] JSON Schema
[ ] Dataclasses
[ ] Model Config
[ ] Field Types - adding or changing a particular data type
[ ] Function validation decorator
[ ] Generic Models
[ ] Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
[ ] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

Some (likely doesn't-handle-all-the-cases-but-handles-all-of-mine) code I'm using inside of a custom GenerateJsonSchema.model_schema looks like:
fromc copy import deepcopy

class MyGen(GenerateJsonSchema):
    def model_schema(self, schema):
        json_schema = super().model_schema(schema)

        for fieldname, fieldschema in schema["schema"].get("fields", {}).items():
            validation_alias = fieldschema.get("validation_alias", None)
            if validation_alias:
                if isinstance(validation_alias, list):
                    for aliases in validation_alias:
                        for alias in aliases:
                            if alias != fieldname:
                                # @TODO: Actually make it a ref, and not just a copy!
                                json_schema["properties"][alias] = deepcopy(json_schema["properties"][fieldname])

        return json_schema

I poked at the code and I think to actually make it a ref is a bit tricky on two fronts:

We could do some kind of self.get_cache_defs_ref_schema(schema['ref'])[1]["$ref"], but that gets the "mangled" ref name. (But let's say that's not a problem and we either unmangled it or just tacked on f"/{fieldname})
in generate we call get_json_ref_counts which chokes on the ".../<fieldname>" ref. Which tells me there's likely a lot of baked in assumptions about not having $refs be to properties of objects.


Listeners in Passes and Functional Patterns

Based on what I have discussed with @nicolasvasilache and @ftynse and from what I've seen in this repo, I have 3 proposals, in order of increasing complexity/difficulty, of infra changes that might be useful. I'll just dump them here to start the discussion.
Pattern groups and registration
There should be some additional level of organization of patterns between individual patterns and the RewritePatternSet class, which is more of an unordered collection of patterns than a "set". A pattern "group" would be a set of patterns (perhaps uniqued by name) that can be combined with others or registered to a name.
E.g. mlir-opt --apply-greedily="group_a,group_b"
Passes and Rewrite Drivers should accept listeners
Passes and importantly the greedy rewrite driver should accept listeners so that the caller can monitor certain events, e.g. notifyOpReplaced. This should at least be implemented in the most common passes in MLIR. Rewrite driver should accept a listener directly, whereas listeners can be passed into passes via the pass manager. On a side note, there should be more rewrite driver "primitives" e.g. applyOnce.
There is currently a applyOpPatternsAndFold driver that applies a set of patterns "locally" on either a single operation or a set of operations. It provides some feedback as to whether an operation was erased. Does this not satisfy some of your requirements? Perhaps we can iterate on it to start.
Functional Patterns
Functional patterns (and by extent functional pattern groups) would enable structured composition of patterns. Alex's Linalg Transform interpreter is moving in this direction, by having a pattern "sequence" return operations that are passed into other patterns, although this is orchestrated by tagging operations with an attribute, finding these operations, and then mapping them to an SSA value in a map.
Functional patterns could be achieved in C++ as just a paradigm: a bunch of functions that return FailureOr<...> and accept a PatternRewriter. Implemented as
template <typename PatternT, typename ...Args>
auto applyOnce(PatternT pattern, PatternRewriter &rewriter, Args &&...args) {
  decltype(pattern(declval<Operation *>(), rewriter, args...)) result{};
  module->walk([&](Operation *op) {
    auto patternResult = pattern(op, rewriter, args...);
    if (failed(patternResult)) return WalkResult::advance();
    result = std::move(patternResult);
    return WalkResult::interrupt();
  });
  return result;
}

But I think it would be more interesting to implement this in PDL by

Allowing patterns to return values (PDLValue)
Allow patterns to "call" other patterns or pattern groups. The "call" would be a rewrite driver primitive, e.g. pdl.apply_once to apply a pattern at most once or pdl.apply_until to apply the pattern/pattern group until no more successes, and different pattern groups can define their own semantics. E.g. an ordered pattern group is just a list of patterns in decreasing benefit.

@Mogball thanks for looking into this and starting the discussion.
On our end we have been so far iterating on a top-down breakdown of "the transformations + state via attribute encoding" that we already know we want to apply. This is maybe more of a "try and reduce technical debt and see what principles appear".
What you propose makes sense and feels like a nice and clear "bottom-up" way of providing the components that would enable what we are trying to achieve here. It feels like you already have good ideas about the core principles that we are still discovering.
More manipulable pattern set, groups and names sound like a no-brainer and I imagine would play nicely with the upcoming PDLL work.
Re drivers and listeners, my issue has been that the existing abstractions are very load bearing so rather than try to extend something that I do not understand deeply, experimenting with something new that I understand made sense to me. But now that you seem engaged and confident about the core technical direction, I'd happily follow what you propose! As usual with top-down/bottom-up we need to make sure the abstractions connect so it is prob. still worthwhile to continue the prototype until it is well connected and we lower a bit the level of abstraction from the current level and make sure the python level still connects to e2e execution. But if you get something working and connected faster, I'll happily adopt early.
Re. functional patterns, @apaszke also had a bunch of comments and ideas on the direction. One thing to note is that most transformations we are talking about are implemented in the compiler return the "IR handles of interest" that the transformation produced to make the APIs chainable and composable.
Thanks for your proposals, I am looking forward to playing with some of this!

the transformations + state via attribute encoding
I'm not principally against using attributes to "mark" operations, encode state, or otherwise store information to be used by other patterns. In fact, I think this can lead to some interesting capabilities, e.g. using patterns to analyze operations and store analysis results as attributes. The discussion around how passes and transformations should handle attributes does not seem to have been resolved.

An approach I have in mind that would not require reworking attribute handling entirely would be to have important attributes implement an interface. I think the current mechanism for this is getDialectAttrs and setDialectAttrs, although it's not widely used.
In any case, Alex already has "transform-aware" passes. Perhaps those can be a starting point for upstream changes that implement "listeners" to passes.

The discussion around how passes and transformations should handle attributes does not seem to have been resolved.

The general discussion sounds a bit hard to reconcile with all possible interactions in the compiler, it seems a bit unlikely to converge..

An approach I have in mind that would not require reworking attribute handling entirely would be to have important attributes implement an interface. I think the current mechanism for this is getDialectAttrs and setDialectAttrs, although it's not widely used.

I have not thought about this particular part enough. I was wondering, whether we could take a pattern set + a drivers and register a list of attributes / interface that are known to "just propagate" ? It seems like propagation + cleanups could be implemented directly in the driver hooks alleviating the need for much deeper spreading, for now?

I have not thought about this particular part enough. I was wondering, whether we could take a pattern set + a drivers and register a list of attributes / interface that are known to "just propagate" ? It seems like propagation + cleanups could be implemented directly in the driver hooks alleviating the need for much deeper spreading, for now?

I like this idea, but it might be difficult to implement generically. I don't think it's practical to track which ops are getting erased/replaced, and which new ops their attributes should be propagated to in the driver.
It might have to be the responsibility of the canonicalization pattern itself to propagate attributes. This can be assisted with various new helper functions, e.g. rewriter.replaceOpWithNewOpAndPropAttrs (one could probably invent a better name for such a function).

Answering the original post first.
Thanks, these ideas look interesting!
The pattern group and registration idea sounds like it could replace the boilerplaty -convert-X-to-Y passes. Some downstream projects create their own uber-conversion passes that to -convert-A-and-B-and-C-and-D-and-E-and-F-to-Y, for example. The caveat is making this work cleanly with pass management + dialect registration. Passes must declare the dialects they can produce, and it's hard for me to see how to do that for a generic -apply-pattern-groups pass unless each group/pattern lists the dialects they produce.
Accepting listeners is a good idea in general IMO. We need to be wary of the overhead though. Currently, GreedyPatternRewriterDriver itself is an OpBuilder::Listener and a PatternRewriter. Figuring out the mechanism for chaining listeners is probably a good start, today we can only have one.

There is currently a applyOpPatternsAndFold driver that applies a set of patterns "locally" on either a single operation or a set of operations. It provides some feedback as to whether an operation was erased. Does this not satisfy some of your requirements? Perhaps we can iterate on it to start.

These are still recursive, i.e. they will keep applying the same pattern to the ops produced by applying this pattern. This recursion is undesirable. Consider a pattern that tiles an operation: we don't want it to keep tiling indefinitely, it has to apply once or a specified fixed number of times and stop. There are also performance considerations of not maintaining a worklist and avoiding spurious checks.
That being said, we may have a use case for "local" canonicalizations as we move forward with finer-grained transformation targeting.

Functional patterns could be achieved in C++ as just a paradigm: a bunch of functions that return FailureOr<...> and accept a PatternRewriter

This is quite interesting. Do you think such patterns could be compatible with the "main" pattern rewriter infrastructure (just discarding the non-failure result)?

I'm not principally against using attributes to "mark" operations, encode state, or otherwise store information to be used by other patterns. In fact, I think this can lead to some interesting capabilities, e.g. using patterns to analyze operations and store analysis results as attributes. The discussion around how passes and transformations should handle attributes does not seem to have been resolved.
An approach I have in mind that would not require reworking attribute handling entirely would be to have important attributes implement an interface. I think the current mechanism for this is getDialectAttrs and setDialectAttrs, although it's not widely used.

I am opposed to using attributes :) Well, not really, but relying on attributes is currently not composable with how most "core" transformation work. That is, they can drop non-inherent attributes at any point, so relying on such attributes is a liability that I'd like to minimize at worst and avoid completely at best. (That being said, putting attributes as debugging aid for humans is still interesting). Until there is some mechanism to propagate attributes across transformations (presumably, a core IR change + an interface that describes how exactly an attribute is propagated in cases other than 1->1 rewrite; IMO the generic propagation is the hardest part), the only failproof way of communicating through attributes is rolling your own passes, or regularly auditing third-party passes you run, that treat attributes in the way you expect. Not to scare you away from thinking about it, but my first email thread about non-discardable dates back to July 2019, and the issue is still there.

Passes must declare the dialects they can produce
Passes that are just wrappers around pattern groups declare the dependent dialects for those patterns anyways. No reason why patterns can't also declare dependent dialects.

This is quite interesting. Do you think such patterns could be compatible with the "main" pattern rewriter infrastructure (just discarding the non-failure result)?

I don't see why not (as I've written them). There are a lot of options to customize and define patterns now, beyond just extending matchAndRewrite endlessly.

Passes that are just wrappers around pattern groups declare the dependent dialects for those patterns anyways. No reason why patterns can't also declare dependent dialects.

Technically, they can. Practically, we are talking about changing hundreds (thousands?) of pattern classes to list dependent dialects. These things have a cost. If all patterns were declarative, we would have just inferred the dependent dialects from the declaration.

I don't see why not (as I've written them). There are a lot of options to customize and define patterns now, beyond just extending matchAndRewrite endlessly.

There are always pesky little C++ template and overloading things :) At a glance, this may work, but I haven't tried to code that.

Practically, we are talking about changing hundreds (thousands?) of pattern classes to list dependent dialects.
We don't have to change existing patterns, but just give patterns the option to list dependent dialects. Most patterns will continue to be used the same way they always have.

I also put up two patches to add listeners to the CSE and canonicalizer passes. River has some reservations about it, and I wasn't able to sync with him before the holidays, so that will be blocked upstream until afterwards. I've thought about how one would go about implementing functional patterns in PDL, and while I'm confident it can be done and be done nicely, it's not a trivial effort.
I'll try building "functional patterns" in C++ for the time being and see if that's the sort of thing that would solve the application control problem.

We don't have to change existing patterns, but just give patterns the option to list dependent dialects. Most patterns will continue to be used the same way they always have.

Then it creates two implicit kinds of patterns: those that correctly list dependent dialects and therefore can be used with the wrapper pass, and those that don't. I expect River and other to have reservations about it too, I would myself :)

I also put up two patches to add listeners to the CSE and canonicalizer passes. River has some reservations about it, and I wasn't able to sync with him before the holidays, so that will be blocked upstream until afterwards.

Yeah, that is expected and one of the reasons why I forked these things instead. The upstream change would be to just make the greedy rewriter class visible in the header so one could derive it.

I'll try building "functional patterns" in C++ for the time being and see if that's the sort of thing that would solve the application control problem.

Thanks! I suggest to implement a quick prototype here and see if that helps before trying to put it upstream. There are two long design discussions to be had: (1) evolving pass management to support more granularity (FWIW, I don't see a conceptual difference between a pass that runs PDL interpreter and a pass that runs another interpreter on my Linalg transform dialect) and (2) dealing with dependent dialects in patterns, more specifically in canonicalization patterns that became cross-dialect after splitting the standard dialect.

Hi everyone! Yeah, I'd love to see more functionally-inspired rewriting infrastructure to appear in MLIR. I started working on a prototype similar to the one that @Mogball proposed, but I hit one fundamental difficulty: mutability.
As it is today, single pattern applications are perfectly composable, because one gets a match method that lets you verify the match without any side-effects. But, consider what happens when you try to build a larger pattern out of two smaller patterns (let's call them a and b). How do you write a match for that? If a.match(op) succeeds then you can't follow up with b.match(op), since that b.match invocation would not see the IR changes that a.rewrite(op, rewriter) would apply. But, if we opportunistically perform a.rewrite(op, rewriter) and then b fails to match then we have a problem, since we have already mutated the IR and it doesn't seem trivial to invert that (barring excessive copying).
I'm sure that if we can work out the compositionaly issues, we can make a very nice rewriting framework and even embed the pattern application in MLIR. And compositionality is the critical feature of functional rewriting systems IMO. IR traversals can be expressed neatly using failable recursive computations (as in e.g. ELEVATE), but managing with failability is the part that has stopped me so far.
Any ideas as to how such rollbacks could be implemented? I would imagine that it might be doable since the dialect conversion framework does maintain two parallel copies of the IR, but I don't know enough details to actually have a good understanding of how this works.

Any ideas as to how such rollbacks could be implemented? I would imagine that it might be doable since the dialect conversion framework does maintain two parallel copies of the IR, but I don't know enough details to actually have a good understanding of how this works.

The dirty but foolproof way is to clone on the first pattern application and later discard the part we don't want anymore.
This of course won't work in the general case of non-scoped transforms but for composition of locally scoped transforms this has worked for me in the past.

What's a locally scoped transform? The difficult I see is that the rewrite method can have side effects that are potentially unbounded (e.g. it can replace all uses of a value in the whole program)

I don't have a good definition for it but basically if you have a set1{ops} that get transformed to set2{ops} ... setn{ops} and at every step the only modifications to the IR is to set1{ops} and their operands + results then you are good.
Something like CSE which escapes outside of "set1{ops}+ operands + results" is much more annoying and requires genuflexions to track (see some of the recent @ftynse commits re CSE).

also @michel-steuwer and @Bastacyclop once they accept the invite

(e.g. it can replace all uses of a value in the whole program)

maybe 1-hop is a better analogy; we can do "1-hop replacements of operands and results" with a simple clone; but not more.
The cleanup procedure is usually along the lines of:

replace the current list of results by the currently valid one
if failed to apply, you may want to rewriter.eraseOp in reverse order of creation yourself to avoid surprises if you introduced side effects that the basic DCE + fold cannot clean up.

Note that this needs to be done at the "transformation" level and transformations compose (but not patterns) since you'll need better APIs. The pattern itself is then a simple wrapper around a composition of transforms.

Right, I think that if you limit the scope of what the patterns can do then you might be able to implement reasonable rollbacks. It would be an interesting exercise to try to enforce those restrictions syntactically in the body of the handler, but an assumption which causes UB might be a good place to start 🤷

Dialect conversion postpones the replace of all uses until the end of the entire conversion process, so you always match on the original IR (unless patterns do in-place modifications), but that comes with its own set of issues. I have a pretty good understanding of how these things currently work if you want to discuss it in more detail, but this will have to wait until the end of holidays.

Yeah, that is expected and one of the reasons why I forked these things instead. The upstream change would be to just make the greedy rewriter class visible in the header so one could derive it.
That wouldn't solve the problem of tracking changes through CSE though, or any other non-pattern pass. Although I would assume that even a patch just exposing the greedy rewriter will face objections. I think once the holidays are over, we'll have a design discussion about how to support more advanced codegen strategies in MLIR.

That being said, among other things I think should be exposed publicly is the save/revert mechanism behind dialect conversion. Dialect conversion does unwind in-place modifications and it does apply patterns to generated ops, otherwise A -> B and B -> C where only C is legal would not work, and that's why conversion patterns accept an array of operands, so that they can match across the original IR and the "pending" IR. That being said, there are restrictions in conversion patterns e.g. block manipulation has to be done through the conversion rewriter's APIs.

Nice, that gives me more confidence that we could actually get this to work, for as long as people obey the rules around IR mutation. Perhaps the rewrite patterns would never get access to any real Operation instance, but would always be expected to work with views/adaptors such that they wouldn't ever be able to tell if the adaptor corresponds to a real operation or not, and would have to go through the limited and invertible rewriter API. How about we schedule some time in Jan and try to sketch a prototype together?

Perhaps the rewrite patterns would never get access to any real Operation instance, but would always be expected to work with views/adaptors such that they wouldn't ever be able to tell if the adaptor corresponds to a real operation or not, and would have to go through the limited and invertible rewriter API.

I've been wanting an "OperationView" class for a while for this purpose. It's essentially just a class one step above op adaptors.
The other option is to clone the scope of the transformation and swap it with the original if the chain of patterns succeeded, otherwise just drop it. We don't have to clone the entire IR -- passes are already scoped to certain operations and their regions e.g. FunctionPass, the contract being that passes can only modify within their scope, cannot query sibling operations (due to threading), but can query parent operations. This sidesteps the problem (for now) of trying to replicate dialect conversion's mechanisms, which get a little bit hairy.
Currently, scopes are limited to the regions of a single operation, but we can probably expand that to be a set of operations and their nested ops.

How about we schedule some time in Jan and try to sketch a prototype together?

SGTM.

@ftynse @Mogball side note: I think I saw some code fly by that was trying to limit the scope of pattern application to ops nested under another op. Rather than build this into the driver, it is likely much easier to add a matching constraint that checks isProperAncestor.

I'm not principally against using attributes to "mark" operations, encode state, or otherwise store information to be used by other patterns. In fact, I think this can lead to some interesting capabilities, e.g. using patterns to analyze operations and store analysis results as attributes. The discussion around how passes and transformations should handle attributes does not seem to have been resolved.
@Mogball I don't see that discussion converging, the proposal is way too broad.
How about instead we have some simpler TransformationTrackableOpInterface with an attribute that must always be set.
The attribute is elided from parsing/printing if it is the default value but needs to always be cloned, otherwise verification fails.
But I am unclear that leaving attributes around is what we should strive for though as I don't find it to scale.
Tracking the IR handles we want explicitly feels like a nicer approach.
In principle, this isn't very different from transformations/passes that update an analysis (e.g. for efficiency reasons).
In practice, in our cases of interest, updating the analysis (i.e. tracked IR handles) is much much easier than rediscovering them on transformed IR.

Very interesting discussion here, and thanks @nicolasvasilache for tagging us.
In ELEVATE we achieve composability by seeing a rewrite as a function. Applying this function constitutes performing the rewrite, and you get back either the rewritten program or a failure. As @apaszke pointed out, this works well for us because of the immutable setting we worked in.
I would be very curious to see how a similar design could be achieved in MLIR.
I need a bit more explanation to understand the difference between a transformation and a pattern in the discussion here.
One thing that is very clear in MLIR is the separation between the RewritePattern (i.e. the matching and the rewrite) and the PatternRewriter that is driving the application of patterns. In ELEVATE, all of these peices are combined into a single function.
To achieve composability, we will need to break the separation in MLIR up. That's why I like the "Allow patterns to "call" other patterns".
Which raises the question, when do you want what mode of pattern application?
Last year, we had started playing around with an ELEVATE inspired interface in MLIR:
https://github.com/rise-lang/mlir/blob/elevate/mlir/include/mlir/Elevate2/core.h
We opted to create a new subclass of RewritePattern that is composable, as it performs the rewrite directly and returns the rewritten IR directly. (I don't think we properly address the mutability challenges that @apaszke raised in this early prototype). You can compose these types of patterns, with operators resulting in new rewrite patterns that you can compose again. We didn't pursue this prototype super far, as @martin-luecke, the PhD student working on this, is currently on a break.

I need a bit more explanation to understand the difference between a transformation and a pattern in the discussion here.

Transformation is a very loose term referring to any modification of the IR. Pattern is a RewritePattern or something similar. A transformation can be expressed by a pattern, but it can also be significantly more complex.

To achieve composability, we will need to break the separation in MLIR up. That's why I like the "Allow patterns to "call" other patterns".
Which raises the question, when do you want what mode of pattern application?

I wouldn't necessarily require merging the patterns and the driver as a precondition for composability. MLIR patterns are essentially (stateless) functions. The driver contains the state including the IR being rewritten. It is possible to play the conceptual of treating the "pattern function" as accepting the driver state and returning a new driver state. Being able to "call" patterns sounds the most important part.

Being able to "call" patterns sounds the most important part.

Would "calling" a pattern mean then invoking the driver, and if so, which one (if there are multiple)?

and you get back either the rewritten program or a failure. As @apaszke pointed out, this works well for us because of the immutable setting we worked in.

This actually sounds like a decent idea for a problem I brought up in https://github.com/google/iree-llvm-sandbox/issues/149 where @apaszke introduced his strategy dialect and this would work well with "scoped transformations". I.e., the result of a pattern application is a failure or a mutated copy of the nearest scope.
The mutability problem could be solved by keeping a copy of the scope from before the strategy was applied and reverting back to it. Since the strategy is in control of the IR scope, then the problem of invalidating IR references in any drivers with clone/revert is avoided.

Would "calling" a pattern mean then invoking the driver, and if so, which one (if there are multiple)?

This could be as simple as directly invoking the pattern on an operation pattern.matchAndRewrite(op, rewriter) or invoking some driver (e.g. apply the pattern once to the IR, apply it N times, or greedily, although greedy only makes sense for pattern sets).

This could be as simple as directly invoking the pattern on an operation pattern.matchAndRewrite(op, rewriter)

In our experiments we basically did that. Then we use the ideas of ELEVATE, which explain how to compose rewrites. The key here to making that work is that a "call" to apply a pattern has to return a handle to the modified IR.
We use for example the seq rewrite pattern for composition of two rewrite patterns which are to be applied after another:
RewritePattern s1 = RewritePattern1();
RewritePattern s2 = RewritePattern2();

SeqRewritePattern(s1, s2).matchAndRewrite(op, rewriter);

with
RewriteResult SeqRewritePattern::matchAndRewrite(Operation *op, PatternRewriter &rewriter) {
    RewriteResult result1 = s1.matchAndRewrite(op, rewriter);
    if (isSuccess(result1)) {
        return s2.matchAndRewrite(result1.op, rewriter);
    }
    return result1 //failure
}

First the rewrite s1 is applied to op and, if successful, s2 is applied to the resulting modified IR.
Following this design and with some small syntax helpers it is easy to compose more complex rewrites:
seq(topdown(fuseReduceMap()), normalize(betaReduction())).matchAndRewrite(op, rewriter);

Here for instance, starting with op we traverse the program topdown until we can apply the fuseReduceMap rewrite. Then, we apply the betaReduction rewrite at all possible occasions using normalize.
Note: topdown and normalize here are RewritePatterns as well, which control how and where the rewrites supplied to them are to be applied.
Ergo this system is based on RewritePatterns being able to call other RewritePatterns.


runtime: Improve error reporting if DeoxysII unsealing fails

Previously, if the CPU changed between runs of the Oasis node, the error
reported was a cryptic "ciphertext is corrupted" (because the sealed SGX
secrets were invalidated).
Now we add a bit more context to make it easier for the end-user.

Alternative, we could also verify if the CPU has changed on startup and not during execution (when the policy or something else is read for the first time). This way we could also move long message (instructions) from the key manager runtime to the host node.

Alternative, we could also verify if the CPU has changed on startup and not during execution (when the policy or something else is read for the first time).

That was my first plan, but it's too complicated to implement properly without breaking compatibility. Perhaps in a later version.

Are you sure this is the case in production SGX mode? I don't think we do.

Are you sure this is the case in production SGX mode?

On SGX machine this does not work.
SGX:
{"level":"info","module":"runtime","msg":"Runtime is starting","runtime_id":"c000000000000000ffffffffffffffffffffffffffffffffffffffffffffffff","runtime_name":"test-runtime","ts":"2023-11-24T10:22:39.233649225Z"}
{"level":"info","module":"runtime","msg":"Establishing connection with the worker host","runtime_id":"c000000000000000ffffffffffffffffffffffffffffffffffffffffffffffff","runtime_name":"test-runtime","ts":"2023-11-24T10:22:39.245834246Z"}

Local (non-SGX):
{"level":"info","module":"runtime","msg":"Runtime is starting","runtime_id":"c000000000000000ffffffffffffffffffffffffffffffffffffffffffffffff","runtime_name":"test-runtime","ts":"2023-11-24T10:06:41.72290283Z"}
{"level":"info","module":"runtime","msg":"Establishing connection with the worker host","runtime_id":"c000000000000000ffffffffffffffffffffffffffffffffffffffffffffffff","runtime_name":"test-runtime","ts":"2023-11-24T10:06:41.725810805Z"}
{"level":"warn","module":"runtime","msg":"thread 'tokio-runtime-worker' panicked at 'persisted state is corrupted (if running in SGX mode, check if the CPU had changed; if yes, wipe 'worker-local-storage.badger.db' and resync the node): InvalidTag', keymanager/src/crypto/kdf.rs:785:14","runtime_id":"c000000000000000ffffffffffffffffffffffffffffffffffffffffffffffff","runtime_name":"test-runtime","ts":"2023-11-24T10:06:41.747484087Z"}
{"level":"warn","module":"runtime","msg":"stack backtrace:","runtime_id":"c000000000000000ffffffffffffffffffffffffffffffffffffffffffffffff","runtime_name":"test-runtime","ts":"2023-11-24T10:06:41.747527309Z"}
{"level":"warn","module":"runtime","msg":"note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.","runtime_id":"c000000000000000ffffffffffffffffffffffffffffffffffffffffffffffff","runtime_name":"test-runtime","ts":"2023-11-24T10:06:41.748084437Z"}

It is important the proper error is shown for production SGX runtimes :)


Manual Upload On Button Click

I saw a demo of fine uploader without react where one can first store all file on client side and then manually upload them on to the server using the upload button.
I want to implement the same logic in react fine uploader.
I am disabling the autoUpload in options object, but I am unable to find a way to manually trigger the upload button.
Thanks.

uploader.methods.uploadStoredFiles()

In react fine uploader I am not getting any such methods attached to uploader.methods

please show all of your code

@rnicholus, how do I resume upload if I have to manually trigger the upload?
Thanks,

It will auto resume after you start the upload if resume is enabled and a record for the file exists

Hello,
How can I use uploader.method.uploadStoreFiles().
Do you have the documentation for that I want to manually upload the files with custom upload button.Is it Possible to do that? @rnicholus.
Thanks!


Can't add device

Hello,
I used this integration time before, now my HA discovered device but i can't add it. I got error without any explanation.

Hi @mariusnauj,
Ow that it weird. We did have it one time earlier and then it was caused by a duplicate IP I think. Have you tried adding it manually?

Yes, I tried add it manualy, but the same. I pinging this wf-rac IP from HA it respons. I do not do anythink, but after 10min I can add it without errors. I don't know why, but sometimes it drops connection. Inside and outside temperatures stuck, and need restart HA.

Ok, when your Airco is not responding this a known issue (see #15 and #9).
A few minutes ago I did make a new release if you do use Hacs than there should be a update.
This update has some general fixes and also some connection improvements. Maybe this will help.
Otherwise restarting HA is for now the only solution. Till we find the fix for #15 and #9

Ok, thank you


Add infoBox method to let people know that we don't bypass cheating sites

          We definitely should use the infoBox method to let people know that we don't bypass this website because it's video game cheating,

Originally posted by @0xc60f in https://github.com/FastForwardTeam/FastForward/pull/921#discussion_r1151354672

I smell forks where it bypasses to cheating sites. Weird how it allows pirating sites (which is what i use it for lol) but not cheating, when cheating software isn't really even illegal, lol.

Feel free to fork :)

Closing this since we changed our stance: we will not develop fixes to these sites, but we will accept changes that someone else makes.


xargs: exec ... Argument list too long

toybox 0.7.1
On some Android devices<EMAIL_ADDRESS>using a chain of find .... | xargs .... stat ...., yields the error Argument list too long.
Some googlfu shows that this is likely related to the argument exceeding the kernels ARG_MAX value.
Androids ARG_MAX according to this 131072.
Manually limiting the xargs argument length using -s bytes fixes this for me.
I've looked at the code for xarg in toybox but could not figure out yet how the max argument length is determined.
Busybox doesn't seem to run into this issue using the same command on the same device. Busybox seems to have some sanity checks, adjusting the argument length if necessary.
Could there be an issue with how toybox's xarg determines the default argument length on Android?

On 08/10/2016 07:44 AM, Matthias Urhahn wrote:

On some Android devices<EMAIL_ADDRESS>using a chain of |find .... | xargs
.... stat ....|, yields the error |Argument list too long|.
Some googlfu shows that this is likely related to the argument exceeding
the kernels ARG_MAX value.
Androids ARG_MAX according to this 131072
https://github.com/android/platform_bionic/blob/master/libc/kernel/uapi/linux/limits.h.

The Linux kernel removed that limit in 2007
(https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b6a2fea39318
went into the 2.6.22 release).
That said, let's cd to a directory with a lot of files in it:
$ find . | wc
68971   69024 3159459
$ find . | toybox xargs echo
xargs: exec echo: Argument list too long
And I'm seeing a limit here too. Hmmm...

Manually limiting the xargs argument length using |-s bytes| fixes this
for me.

The kernel should not be enforcing this? Maybe something else is? There
isn't an obvious ulimit for this... Ah, in include/uapi/linux/limits.h:
#define ARG_MAX    131072 /* # bytes of args + environ for exec() */
And that's used in fs/exec.c:
/*

We've historically supported up to 32 pages (ARG_MAX)
of argument strings even with small stacks
*/
if (size <= ARG_MAX)
return page;

/*

Limit to 1/4-th the stack size for the argv+env strings.
This ensures that:

the remaining binfmt code will not run out of stack space,

the program will have a reasonable amount of stack left

to work from.
*/
rlim = current->signal->rlim;
if (size > ACCESS_ONCE(rlim[RLIMIT_STACK].rlim_cur) / 4) {
put_page(page);
return NULL;
}

Ok, so that's still there as a MINIMUM, and the maximum is 1/4 the
current stack size ulimit. According to ulimit -s that defaults to 8192
kilobytes (at least on xubuntu 14.04), so 1/4 of 8 megs is 2 megs. And...
$ find . | head -n 40000 | wc
40000   40002 1853620
$ find . | head -n 40000 | toybox xargs echo | wc
1   40002 1853620
Yup, the actual default limit is way more than 131072, and that
default is arbitrarily adjustable even for normal users:
$ ulimit -Hs
unlimited
$ ulimit -s 9999999
$ find . | toybox xargs echo | wc
1   69023 3159436
(Huh, what happens if I set the stack size limit to less than 131072
and then try to exec stuff? Not trying it on my work netbook just now,
thanks...)

I've looked at the code for xarg in toybox but could not figure out yet
how the max argument length is determined.

It didn't enforce a default maximum length limit because as far as I
knew the kernel hadn't required one for 9 years. I worked out the math
for enforcing such a limit back when I first did xargs:
http://landley.net/notes-2011.html#17-12-2011
So it's not hard to add a default, but what the default should be isn't
obvious. Back when I noticed the kernel had changed its logic I
checked that I could feed it more than 131072 bytes, and I could, so I
removed the limit. (Not just from xargs, but from find in commit
aa784b09a9fb and possibly elsewhere.)

Busybox doesn't seem to run into this issue using the same command on
the same device. Busybox seems to have some sanity checks
https://git.busybox.net/busybox/tree/findutils/xargs.c#n529, adjusting
the argument length if necessary.

The sanity checks are there (-s is implemented), toybox just doesn't
have a default value if you don't specify one.

Could there be an issue with how toybox's |xarg| determines the default
argument length on Android?

See above.
There's a sysconf(_SC_ARG_MAX), which does appear to be calculating this
properly (at least built against glibc, musl is giving me 131072 and I
still haven't got a host bionic toolchain because building bionic
outside of AOSP was nontrivial when I looked at it).
Oh well, use the sysconf and then file a bug against musl. (Elliott will
probably notice this, and if not he'll notice the commit.)
Thanks for the heads up,
Rob

Thanks for the details, very intresting.

Oh well, use the sysconf and then file a bug against musl.

I'm not sure I understood this. Should I file this somewhere else?

(Elliott will probably notice this, and if not he'll notice the commit.)

What commit?

On 08/11/2016 12:25 AM, Matthias Urhahn wrote:

Thanks for the details, very intresting.
Oh well, use the sysconf and then file a bug against musl.

I'm not sure I understood this. Should I file this somewhere else?

No, I meant that's how I should fix it in toybox.

(Elliott will probably notice this, and if not he'll notice the
commit.)

What commit?

One I haven't made yet, fixing this. (Busy day, sorry.)
Rob

thread restarted with patch for xargs (but not find): http://lists.landley.net/pipermail/toybox-landley.net/2017-September/009167.html

i think this can be closed now... as of the end of 2019 this is working for bionic, glibc, and macOS.

The magic -4096 is still a little creepy, but yes it seems to be working.


It shouud be able to describe things like:

Organic waste is usually collected on the 1st and 3rd Wednesday of the month
Metall is collected on the 31.3.2024 and 30.09.2024
Instead of 01.01.2024 the waste is collected on the 30.12.2023

The basic idea is only have minimal changes to the rules each year, generate an iCal file for the upcoming year and load it using the existing scripts.

Antother approach would be to use a librars like recurrent and define a YAML-like structure
waste:
  a:
    when: weekly every Tuesday
  b:
    when: weekly every Wednesday

organic:
  when: weekly every Friday

exceptions:
   - 2024-01-01
   - 2024-01-02
   - 2024-05-01
   - 2024-08-01
   - 2024-12-25
   - 2024-12-26


no-object-mutation conflicts with setters

class Foo {
thing: string;
set something (d) {
this.thing = d;
}
}
f = new Foo();
f.something = 'abc'; <-- Modifying properties of existing object not allowed. tslint(no-object-mutation)

I'm not sure I follow. The goal of the no-object-mutation rule is to enforce use of immutable objects. Having setters make objects mutable, thus defeating the goal of the no-object-mutation rule. If you want to change the object, I would recommend making a function that returns a modified copy of the object instead.

I understand the rule but got somehow confused around the project I was working on and came to my mind that the class is immutable but the data within some properties can be :)
Thanks for the time!


ERR_STREAM_PREMATURE_CLOSE

Hello,
I'm using Node.js Version 18.17.0
I've updated today :

@hono/node-server from 1.0.2 to 1.1.0
hono from 3.2.7 to 3.3.2
When I start my app I'm getting the following error:

Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
    at new NodeError (node:internal/errors:405:5)
    at ServerResponse.onclose (node:internal/streams/end-of-stream:154:30)
    at ServerResponse.emit (node:events:526:35)
    at emitCloseNT (node:_http_server:996:10)
    at Socket.onServerResponseClose (node:_http_server:278:5)
    at Socket.emit (node:events:526:35)
    at TCP.<anonymous> (node:net:323:12) {
  code: 'ERR_STREAM_PREMATURE_CLOSE'
}

The app continues to work but once in a while this error pops up without stopping the app.

Hi @LebCit , thanks for reporting.
I've noticed the error you mentioned, but I believe it's already been fixed. Does this error occur consistently? If possible, could you share the piece of code or a snippet that triggers this bug?

Hello @yusukebe
I've updated today Hono from 3.3.2 to 3.3.3 and the error still occurs.
I've finally managed to spot when it occurs.
I'm not using any kind of back-end hot reload, so if I stay a little bit (a minute or maybe a little bit longer) on the front-en on any page the visit another route the error pops up in the back-end's console.
Regarding the code, the project is in the following GitHub repository: https://github.com/LebCit/blog-doc/stargazers
I've just seen that you've starred it, thank you very much for your support.

Hi @LebCit
Thank you for sharing this. While this doesn't directly concern Hono core, it pertains to @hono/node-server. I have encountered this error a few times as well. I'll create an issue in the github.com/honojs/node-server repo and we will investigate and address it later.

@LebCit Hi, I downloaded your codes, and use node 18.17.0 / hono 3.3.3 / node-server 1.1.0.
But I couldn't reproduce your issues.
The ERR_STREAM_PREMATURE_CLOSE error means that, there is a user abort event from the use side.

Hello @yusukebe @tangye1234
Thanks again both for your time, efforts and starring Blog-Doc.
@tangye1234 If you have installed the app and run node index or npm run fire and tried the app, the error will pop up if you leave the app on a page like Documentation for a minute or maybe a little bit longer.
N.B: Today I've worked on the app and the error showed only one time in a couple of hours.
I'm really sorry to bother you both with this error that I can only spot with the described step.

This error is actually a normal tcp termination error. Usually we don't need to pay attention to it. If it really bothers, I can filter this error in node server, or just ignore the error stack. The latter thing is exactly what nextjs or express dose.

Hi @tangye1234
Thanks for investigating! Ignoring the error stack in the node server is good for me. Can you create the issue?

Hey @tangye1234 !
I too am facing this problem now. I want to make it ignores the error, to do so, do I make it does nothing in the catch statement?
https://github.com/honojs/node-server/blob/7f5dd58e1bb32747263bb89798d97c82e390a227/src/listener.ts#L85-L91

yes, just don’t print console error, and PR is welcome.
The error may be thrown either from down stream in nodejs server response or from upstream in hono stream. The premature error is thrown from nodejs down stream when user abort a http request. Most frameworks just print error message instead of error stack.
Nuxt / Next do not swallow this error, but only print message instead.

@tangye1234
Thanks for the explanation, would be great if you could make PR!

I think neither outgoing.end nor ignoring thedestroy would make any sense.
The error thrown from the pipeline for kinds of situation:

User aborts the client request. In fact, somehow the socket is closed before response's end is called normally. Such that ERR_STREAM_PREMATURE_CLOSE error is thrown. In that, we just finish the response by destroying it, because this would effectively prevent any writing action on the outgoing.
Upstream reading error while pipeline is processing, thus we can stop the response to avoid waiting on the client. If we use end instead of destroy, the client my think we get a correct response, but fact is the response is corrupted, examinated, like the response length dose not meet the Content-Length header, so destroying will give a proper hint for the client. Or if we just ignore the post processing, then the response dose not know when to close, which may lead a long tcp hangout.

The anoying stacktrace is carried by the console.log https://github.com/honojs/node-server/blob/7f5dd58e1bb32747263bb89798d97c82e390a227/src/listener.ts#L87C10-L87C10
We can just test the error info, if it is ERR_STREAM_PREMATURE_CLOSE, just console.info('use abort a request')

@tangye1234
Thanks for the detailed explanation despite you are busy . I've corrected the PR https://github.com/honojs/node-server/pull/83. Please take a look if you like.

This was closed by https://github.com/honojs/node-server/pull/83.
Thanks!


fix(ephemerides): fix houses computations

Codecov Report

Merging #118 (49f8460) into master (2d4634f) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master      #118   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           16        16
  Lines          626       626
=========================================
  Hits           626       626

Impacted Files
Coverage Δ

kanon_api/core/ephemerides/houses.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d4634f...49f8460. Read the comment docs.


5 threads (for the default singleton) can be created even when SlackConfig#statsEnabled is false

Developers can disable API clients' metrics management by setting SlackConfig#statsEnabled to false. Even in this case, the default SlackConfig object creation (as a static field in the class) starts 5 threads and those threads are unused.  See the repro steps below for details.
Reproducible in:
any versions
The Slack SDK version
any versions
Java Runtime version
any versions
OS info
any versions
Steps to reproduce:
Running the following code and checking the thread dump.
SlackConfig config = new SlackConfig();
config.setStatsEnabled(false);
Slack slack = Slack.getInstance(config);

Expected result:
No threads under a "slack-api-metrics" thread group are created.
Actual result:
5 threads for the default singleton SlackConfig instance are created.

Also running into this issue. I'm working on a slack app using Quarkus, where I hope the native compilation will allow the app to start up fast enough for the 3-second limit on processing the Events API.
Running native compilation requires exclusion of these default classes because of the threads started (which Quarkus doesn't allow)

@jfbenckhuijsen Thanks for sharing this. However, even when this issue is resolved, this SDK still does not support native builds. Refer to https://github.com/slackapi/java-slack-sdk/issues/1009 for more details. We are not planning to add native supports as long as gson project is ready for it.


Check FTP behavior in logs

Are the following issues reported at data-repository still relevant and/or tackled by vptstools?

[ ] https://github.com/enram/data-repository/issues/60
[ ] https://github.com/enram/data-repository/issues/59

@stijnvanhoey Note, I closed the issues in the data-repository repo, but did not investigate if they are tackled by vptstools.

We'll evaluate this when running the service in uat.


Question: running rendora docker together with a docker nginx container

First of all kudos for this awesome project. It solves an actual paint point while working with SPAs.
I have a custom setup I am not able to make work and I am wondering if it is even possible.
The VPS looks like this:
A docker container built from a custom nginx image that serves the react files.
The custom nginx image has a custom ngnix.conf copied to /etc/nginx/nginx.conf and the react dist files copied to /var/www/html/myapp at build time.
docker run --name my-nginx-app -d --network my-network -p 80:80 my-custom-nginx-image
Everything works but when I try to force SSR.
Just running rendora and the chrome headless images do nothing although I am able to render the SSR'ed content through the API, when enabled, posting to /render
What am I doing wrong?

Hey @ecerroni and thanks for the nice words, as for your issue, it's really hard to solve it all without seeing your rendora config file, it maybe because you didn't add any whitelisted user agent keywords since rendora works by default as a transparent reverse HTTP proxy and thus it only transmits initial response as is. You need to add something like
filters:
    userAgent:
        defaultPolicy: blacklist
        exceptions:
            keywords:
                - bingbot
                - yandex

this will whitelist all useragents containing "bingbot" and "yandex" with ignoring case sensitivity. If you try it with curl for example
curl -A "BingBot" http://<IP_ADDRESS>:3001

You will get the SSR'ed page this time, note that rendora is listening by default to the port 3001.
But tbh I believe that your problem maybe due to something else, do you have a backend server or your entire app is hosted by nginx itself (i.e. your entire app is contained within the react javascript files)? If this is the case I suggest you set rendora to be listening to the port 80 and make your nginx as your backend. You can then map your host port to another number such 4000 for example
docker run --name my-nginx-app -d --network my-network -p 4000:80 my-custom-nginx-image

and then in your rendora config file you set the backend and target urls to http://<IP_ADDRESS>:4000 , so you can change your rendora yaml config file to something like
target:
    url: "http://<IP_ADDRESS>:4000"

backend:
    url: "http://<IP_ADDRESS>:4000"

and btw, it's much easier to use docker-compose instead of multiple docker run with having all your configurations in one place
version: "3"
services:
    headless:
        image: rendora/chrome-headless:latest
        ports:
            - <IP_ADDRESS>:9222:9222
        networks:
            - my-network
        tmpfs:
            - /tmp
    rendora:
        image: rendora/rendora
        network_mode: "host"
        volumes:
            - {YOUR_CONFIG_FILE}.yaml:/etc/rendora/config.yaml
    nginx:
        image: my-custom-nginx-image
        ports:
            - <IP_ADDRESS>:4000:80
        networks:
            - my-network

networks:
    my-network:

and now you just run your entire stack with one command
docker-compose -f YOUR_DOCKER_COMPOSE_FILENAME up

Hey @geokb thanks the very detailed response.
Yes my use case is exactly how you wrote your entire app is contained within the react javascript files.
Your proposed solution made me understand better how rendora works :+1:

Following the setup listed above I cannot get Rendora to render the application.
I have a createReactApp being served through and NGINX container which can be accessed and loads through port 4000, however the content returned is not server rendered.
Steps I have completed

Copied and pasted the docker compose file and all services are running

react app loads through port 4000
Configuration loads ok (the configuration listed above with the backend / target point to port 4000)
Rendora is running and connected to headless chrome

Curl the site with bingbot user agent  (url -A "BingBot" http://<IP_ADDRESS>:4000)


Testing Multithreaded Code

How should one test multithreaded code with a mongomock database? Say I have a thread-safe function for a single document:
def doc_func(doc, mock_coll):
       mock_coll.update({'_id': doc['_id'], {'$set': {'field_b': doc['field_a'] +1}}})

I wish to go at the whole collection with a work queue or something. How should I share the mongomock database object between multiprocessing processes? I've messed around with proxies and custom managers, but no luck yet.

one possibility would be to add locks to all mongomock APIs - I don't think there would be anything more sophisticated than that....

Would that do it? I am under the impression that due to the GIL, other processes can't touch variables from other processes without some sort of proxy.

First these are threads, not processes - and secondly I'm not sure what you mean. The GIL only protects the internal interpreter state, so locks are only required when threads touch or modify shared data structures....

Oh I see - I said multithreading in the issue title. I should have said "testing parallel code". I'm using the multiprocessing library to take advantage of multiple cores when mapping an operation over a collection (e.g. the function I gave in the issue description.) I'd like to get true parallelism, and hence use "multiprocessing" rather than multithreading. This entails sharing the mock database across processes. Is there a better way to do what I'm after?

No. Multiprocessing is definitely not compatible with mongomock.

Ok. So (I realize this discussion now departs from pertinence to your library) is it the case that I should not bother with the kind of test I'm describing? Or is using multiprocessing a bad idea with this kind of database operation in general? Can you advise on a better way to achieve the parallelism I describe?

I'm not saying not to test it, just that mongomock is not enough to support it. I would use an actual mongodb process running locally for testing. Assuming you don't want to use the system default, you can have one started as a part of the test setup.

Ok. Thank you for your help, Rotem.

No problem. Good luck.


Cluster DNS not configured properly?

Deployed 3x3x3 master/infra/node configuration.  Can login and deploy basic pod apps, however, deploying things like example CI/CD pipelines w/ Jenkins integration...
https://raw.githubusercontent.com/kenthua/nodejs-ex/demo/bluegreen-pipeline.yml
...fails to work properly.  Build pipeline does not work. Looking at the log for the Jenkins pod, it's unable to reach openshift.default.svc. Issue seems to point to cluster DNS? Or is there some other basic configuration that is missing that I need to complete post deployment of the ARM template to Azure?
Saw similar behavior when trying to deploy this CoolStore demo: https://github.com/alezzandro/coolstore-microservice
The components of the demo can not deploy correctly, because once started, none of the builds can lookup the nexus svc.
Here is kubernetes/DNS configuration on OpenShift 3.5 on Azure:
oc get service kubernetes -n default -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2017-09-25T05:36:27Z
labels:
component: apiserver
provider: kubernetes
name: kubernetes
namespace: default
resourceVersion: "170"
selfLink: /api/v1/namespaces/default/services/kubernetes
uid: 79c495ad-a1b3-11e7-ba76-000d3a931eae
spec:
clusterIP: <IP_ADDRESS>
ports:

name: https
port: 443
protocol: TCP
targetPort: 8443
name: dns
port: 53
protocol: UDP
targetPort: 53
name: dns-tcp
port: 53
protocol: TCP
targetPort: 53
sessionAffinity: ClientIP
type: ClusterIP
status:
loadBalancer: {}

[user@host]$ oc project huatest1
Already on project "huatest1" on server "https://obfuscated-mstr01.region.cloudapp.azure.com:8443".
[user@host]$ oc get pods
NAME              READY     STATUS    RESTARTS   AGE
jenkins-1-xkj7h   1/1       Running   0          2h
mongodb-1-s01pz   1/1       Running   0          2h
[user@host]$ oc rsh jenkins-1-xkj7h
sh-4.2$ curl openshift.default.svc:443
curl: (6) Could not resolve host: openshift.default.svc; Name or service not known
sh-4.2$ exit
exit
Compared to kubernetes/DNS configuration on OpenShift CDK running minishift locally...
oc get service kubernetes -n default -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2017-09-15T22:01:01Z
labels:
component: apiserver
provider: kubernetes
name: kubernetes
namespace: default
resourceVersion: "620"
selfLink: /api/v1/namespaces/default/services/kubernetes
uid: 5c877e73-9a61-11e7-96f8-525400ab5a26
spec:
clusterIP: <IP_ADDRESS>
ports:

name: https
port: 443
protocol: TCP
targetPort: 8443
name: dns
port: 53
protocol: UDP
targetPort: 8053
name: dns-tcp
port: 53
protocol: TCP
targetPort: 8053
sessionAffinity: ClientIP
type: ClusterIP
status:
loadBalancer: {}

[user@host]$ oc project huatest1
Now using project "huatest1" on server "https://<IP_ADDRESS>:8443".
 [user@host]$ oc get pods
NAME                                   READY     STATUS      RESTARTS   AGE
jenkins-1-pqzxr                        1/1       Running     1          1h
mongodb-1-8q3z7                        1/1       Running     1          1h
nodejs-mongodb-example-1-build         0/1       Completed   0          1h
nodejs-mongodb-example-green-1-tkqm9   1/1       Running     1          1h
  [user@host]$ oc rsh jenkins-1-pqzxr
sh-4.2$ curl openshift.default.svc:443

sh-4.2$ exit
exit

Setting openshift_use_dnsmasq=true for each occurrence in scripts/deployOpenShift.sh has resolved the issue.

I am fixing this in the next release of the templates that will deploy OpenShift Container Platform 3.6.  In 3.6, openshift_use_dnsmasq=true is required.

Now using dnsmasq=true.
Closing.


Ensure discovery can handle invalid command ids

What was wrong?
This error came up during beam sync.
   ERROR  09-05 12:01:05               asyncio  Exception in callback UDPTransport._on_read_ready
handle: <Handle UDPTransport._on_read_ready>
Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 69, in uvloop.loop.Handle._run
  File "uvloop/handles/udp.pyx", line 64, in uvloop.loop.UDPTransport._on_read_ready
  File "/home/ubuntu/trinity/p2p/discovery.py", line 448, in datagram_received
    self.receive(address, cast(bytes, data))
  File "/home/ubuntu/trinity/p2p/discovery.py", line 463, in receive
    remote_pubkey, cmd_id, payload, message_hash = _unpack_v4(message)
  File "/home/ubuntu/trinity/p2p/discovery.py", line 1261, in _unpack_v4
    cmd = CMD_ID_MAP[cmd_id]
KeyError: 9

It's a bit surprising to me that this didn't came up earlier but I've seen it a lot today.
Relates to #1062
How was it fixed?
Catch the KeyError and raise a InvalidCommandID error that is then handled among the other DefectiveMessage errors.
To-Do

[x] Clean up commit history

[x] Add entry to the release notes

Cute Animal Picture

@pipermerriam it's already being handled here (DefectiveMessage is the base type):
https://github.com/ethereum/trinity/blob/c91e2394c0d8ea7a241ead9804e9927ab7316a08/p2p/discovery.py#L462-L466


Can't change only one platform's name

My current project will run only in iOS so I have delete the ./android project file after react-native init so when I try to use react native rename, it says project should created with react-native init
Expected behaviour
Just change the iOS files name.

You could keep the android folder, it will not interrupt your build in iOS, alot of packages will say you need the android folder. Just ignore the android folder and you will be happier.

Thank you. Actually I solve the problem,
-> react-native init samenameApp in another directory
-> Move the android folder to app.
-> Now renaming works perfect.
-> After that you can remove android again 😃


Second/Third Organisations

I can't get any emails sent through second or third organisations added to actually send, they remain in Pending status. The primary organisation I added first works with no issues. Is there something I've missed that I need to setup in the config for sending to work with other organisations or mail servers in the front-end?

Have you set up any potentially unnecessary IP pools?

Nope, all organisations are using the same IP Pool. Unless that could be the issue?

If you aren't using IP Pools, you shouldn't enable the settings.
If thats not the issue, have you looked at the logs?

Sorry we are using IP Pools, we have one pool which is used by all organisations. Could that be the issue?
There doesn't seem to be anything in the logs relating to why they're stuck in Pending status.

It might be but I'm afraid I don't know lots about Postal uses IP Pools.
Did you create the pool after creating the organisations or before?

Interestingly I removed all the IP pools and let it rotate through the IP's itself, and this stopped the issue with sending through other organisations and also stopped a huge delay we were having in connections. We've gone from as long as 5 seconds per connection to around .10ms (which is what we were expecting). It seems there may be some issues in using the IP Pools from both an organisational point of view and an optimisation of speed. Or I'm doing something wrong, which is also highly possible.

There were only 5 addresses in the pool. It's almost as if only one organisation can use a pool at a time, which may be the case. The bigger issue is the change in connection speed.


Update package.json with create-react-class with 15.6.3

It fixes -https://github.com/day8/re-frame-template/issues/123
Problem Desc-
After creating a new project
lein new re-frame "project_name" +test +10x and lein dev
Was getting following error

shadow-cljs - watching build :app
[:app] Configuring build.
[:app] Compiling ...
[:app] Build failure:
The required namespace "create-react-class" is not available, it was required by "day8/re_frame_10x/inlined_deps/reagent/v0v8v1/reagent/impl/component.cljs".
The namespace was provided via :foreign-libs which is not supported.
Please refer to https://shadow-cljs.github.io/docs/UsersGuide.html#cljsjs for more information.
You may just need to run:
npm install create-react-class

Thanks for the PR @gnuman :+1: :smile: However due to recent releases of reagent/react/re-frame no longer requiring create-react-class it is no longer an issue so closing.


chromedriver-helper breaks projects not using it

I have a project not using chromedriver-helper. I'm running my tests via:
bin/rails test:system
this passes. But once i bundle install a project which uses chromedriver-helper or i gem install chromedriver-helper i can no longer run my tests.
gem install chromedriver-helper
$ bin/rails test:system
Run options: --seed 13360

# Running:

E

Error:
CustomersTest#test_login:
Selenium::WebDriver::Error::WebDriverError: unable to connect to chromedriver <IP_ADDRESS>:9515
    test/system/customers_test.rb:5:in `block in <class:CustomersTest>'

Running the chromedriver from irb shows the reason for the error:
$ bundle exec irb
2.4.0 :001 > system('chromedriver -v')
.../rubygems_integration.rb:458:in `block in replace_bin_path': can't find executable chromedriver for gem chromedriver-helper. chromedriver-helper is not currently included in the bundle, perhaps you meant to add it to your Gemfile? (Gem::Exception)

Have you tried adding the gem to your bundle? I believe this will address your issue, which I suspect is based in your $PATH containing an executable script chromedriver but your bundle exec environment does not.

Have you tried adding the gem to your bundle? I believe this will address your issue, which I suspect is based in your $PATH containing an executable script chromedriver but your bundle exec environment does not.

Yes the 2 workarounds are:

add chromedriver-helper to your Gemfile
remove chromedriver-helper gem, from the machine:gem uninstall chromedriver-helper

The point is that if you're working on multiple projects, either all of them or non of them should have chromedriver-helper which is less than ideal.

Sorry, to be clear I meant to only install into your bundler environments. gem install will put an executable into your global path. If you avoid the global install and only install into your projects, you shouldn't have a problem.

Sorry, to be clear I meant to only install into your bundler environments. gem install will put an executable into your global path. If you avoid the global install and only install into your projects, you shouldn't have a problem.

Even if i install through bundler i still end up with a chromedriver executable in my PATH. I've never did gem install chromedriver-helper, only bundle install.
Bundler work by installing the gem globally and then only selecting the ones in the Gemfile when you do bundle exec.
$ which chromedriver
/usr/local/bin/chromedriver
$ mkdir -p /tmp/chtest ; cd /tmp/chtest
$ bundle init
$ echo 'gem "chromedriver-helper"' >> Gemfile
$ bundle install
Fetching gem metadata from https://rubygems.org/..............
Resolving dependencies...
Using io-like 0.3.0
Using archive-zip 0.11.0
Using bundler 1.16.1
Using mini_portile2 2.3.0
Using nokogiri 1.8.4
Fetching chromedriver-helper 1.2.0
Installing chromedriver-helper 1.2.0
Bundle complete! 1 Gemfile dependency, 6 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
$ which chromedriver
~/.rvm/gems/ruby-2.4.0/bin/chromedriver

@flavorjones Not true, at least if using rbenv - could be different with rvm  ---  Adding chromedriver-helper to any project ends up with $HOME/.rbenv/shims/chromedriver being installed which ends up being in the PATH at all times (even when not in that project), and prevents access to any system version of chromedriver installed, often leading to failures.

Given that no one really runs chromedriver manually (so no shim is really needed) -- I wonder what would happen if chromedriver was specified as an executable in the gemspec - would that prevent a shim being added for it, but still provide it in the path when in the bundle environment?

Another thing to try would be using a different shim name (chromedriver-helper) for example. Then use Selenium::WebDriver::Chrome.driver_path= to make capybara use that.

@mfazekas In that case you'd probably just be better off using the webdrivers gem rather than chromedriver-helper

@mfazekas In that case you'd probably just be better off using the webdrivers gem rather than chromedriver-helper

@twalpole can you ellaborate? The point is changing chromedirver-helper so that it won't break projects not using it. I don't want to use any of those gems, i just don't want to have my projects broken just because i did bundle install on a project using them.
I've suggested chaging the shim name to chromedriver-helperfrom chromedriver in chromedriver-helper gem and also  adding a minimal code to chromedriver-helper helper gem to tell  selenium-webdriver to default to this executable. That way if the gem will not shadow the system isntalled chromedriver. AFAIK all uses of chromedriver ends up being through selenium-webdriver gem,  definetely the case with capybara.

To summarize: I think the problem here is when someone starts using / contributing to A_PROJECT already using chromedriver-helper, then they find that ANOTHER_PROJECT not using chromedriver-helper suddenly is broken.
I think: This won't happen if you're using rvm with separate gemsets for the two projects (and maybe other ruby management mechanisms) for A_PROJECT. It also won't happen if you aren't using bundler for either project (but that seems rare).
If I have understood correctly, the problem is that the chromedriver bin file is generated with bundler, and is then placed in the global path but shadows the original chromedriver binary wherever that path is availble (ie for that ruby) but the bundle may not be available .
So...
Is there a way to customize the failure of the bin when if fails to find the gem to revert to ? (I think that that wrapper bin is created by bundler/rubygems rather than the chromedriver, but I may be wrong here)
I'm guessing this might happen to someone new to ruby and would be totally mystifying.

FWIW: https://makandracards.com/makandra/47846-chromedriver-helper-gem-in-gemfile-might-break-you-selenium-tests-of-other-projects

Is there a way to customize the failure of the bin when if fails to find the gem to revert to ? (I think that that wrapper bin is created by bundler/rubygems rather than the chromedriver, but I may be wrong here)

Yes, wrappers are generated by rubygems and n there seems to be no way to customize binstub generation there.
This is the code in rubygems where the scripts are generated:
https://github.com/rubygems/rubygems/blob/af9a1a0f4afaf35f712ff1c3b7928150541cdd18/lib/rubygems/installer.rb#L480
https://github.com/rubygems/rubygems/blob/af9a1a0f4afaf35f712ff1c3b7928150541cdd18/lib/rubygems/installer.rb#L736

@flavorjones  can you explain this note on #58:

Part of the value of chromedriver-helper is that it shadows the system-installed version (and calls it if it's present).

My understanding is that 99% of usecases is just adding the gem to Gemfile and running ui tests, (and this works fine with #58, as it tells capybara the chromedirver path to use). I assume small number of users are starting chromerdirver directly from command line, and i'd argue that for those advanced uses a change in docs is just fine. Or am i missing something?

Hi! I wanted to report an issue for this ages ago but I never got to it.
I asked the bundler team about this and there's no really a good solution, but a good workaround is to use bundle config --path .bundle so that gems are installed locally to each project and never shared.
But I hadn't thought about the solution proposed by @mfazekas in https://github.com/flavorjones/chromedriver-helper/issues/57#issuecomment-411356786. That's backwards incompatible, but I think it would make the best behaved chromedriver-helper.
This bug is quite hard to troubleshoot and makes contributing to some projects very painful... :disappointed:

This issue seems pretty intractable without making a breaking change.
Note that chromedriver-helper is included by default in every new Rails project generated by rails new, so it is very hard to avoid installing this gem on your system at some point. This particularly affects people who are learning Rails, as they are more likely to be following tutorials and doing other experiments involving rails new. Once the gem gets installed, removing it entirely (and keeping it removed) is tedious.
The bundle config --path .bundle solution works, but it is pretty brute-force. That would mean every project using its own copies of gems, which slows down bundle install for developers that switch between many projects. I don't like the idea of having to change my global bundler config to work around one gem problem.
I think #58 is a great approach.

I fully agree with you @mattbrictson, #58 is the way to go. And as mentioned by @mfazekas, it can be considered backwards incompatible, but it's going to be a transparent change for most users who only use this gem for UI tests. Just note that the bundle config --path .bundle will be the default in bundler 2. I think it's a good move.

Note that chromedriver-helper is included by default in every new Rails project generated by rails new

:scream: :fire_engine: I did not know that.
I've re-opened #58 and will take another look. I appreciate everyone's comments here!

:scream: :fire_engine: I did not know that.

Me neither, I just found out too!
Thanks so much for your work @flavorjones! :heart:


Static ports for NFS/RPC

Hi,
Is it possible to configure the ports Dinghy listens on for NFS and RPC (and others too if there are more)? My firewall is going haywire constantly popping up that a new port was opened again (both the 19xxx for NFS and ports below 1024 which I think are RPC).

Is it possible to configure your firewall to always allow the unfsd binary? If so that would be the quickest solution. Otherwise:
There's not currently any way to configure the NFS port, but it wouldn't be too hard to add. Basically add a new preference in cli.rb and plumb it through to the Unfs constructor. A better alternative might be to select the first port >= 19000 that isn't currently being used, rather than randomly selecting a port, but that's a bit more complex.
I'm not sure what the ports below 1024 would be, there's no other incoming connections. Or does your firewall prompt for outgoing connections as well? At what point do those connections occur? If it is an outgoing conn, it could be the fsevents daemon opening the SSH connection to the VM. Or potentially something docker-machine is doing.

I've looked into it allowing a specific program but no luck so far.
From a security perspective I prefer to have control over which ports are open on my machine and not have an application pick one itself (whether that's random or looking for the first free port). You could perhaps still have "pick a random port" as the default, and only bind to a static port if one is configured. I don't really see the benefit of looking for the first available port, to me that's not much different from a random port and I don't really see the added benefit. For people who don't care about the port, selecting one at random is fine. People who do care about the port probably won't like it if despite configuration the application still binds to a different port.
Or am I missing something here? :)
As for the other ports below 1024, I'm starting to have doubts it's dinghy. I'm going to try to setup it up again from a clean MacOS somewhere in the next couple weeks to make sure and see if I can figure out more precise where the ports are coming from.

Hm it looks like I may be able to remove the weird random port selection now. I originally did that because I ran into issues with unfsd failing on dinghy restart trying to bind to the same port again, and I didn't want to dig into the unfsd source to try and fix it. But I'm not seeing that issue on restarts anymore, so it appears to have been fixed sometime in the last 2 years or so. I'm going to try that out for a while and see if anything crops up.

So far I haven't seen any restart issues like I originally saw 2 years ago, so I think this is good to include in the next release. I've added a nfs_port preference too for those who might need to override the default of 19091


updated the cmake verison from 2.8 to 3.6.3

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Codecov Report

Merging #748 (dd5bda7) into develop (00ba9cf) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #748   +/-   ##
========================================
  Coverage    69.47%   69.47%
========================================
  Files           32       32
  Lines         1661     1661
========================================
  Hits          1154     1154
  Misses         507      507

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00ba9cf...dd5bda7. Read the comment docs.