Client implementation breakdown tasks

Breaking down execution engine implementation tasks here (not related to specific client which we will do further down the road).
This is just general requirements per Bedrock specification.
Assumptions / Questions
After reading a lot of documentation, I assume we basically fork one of the Eth clients (L1) and modify it to be a Bedrock compliant L2 client, right? Just thinking logically, wouldn't there be an OpenSource L2 client supporting Bedrock then as well very soon? Or why don't we just use what Optimism is using, since they need to do these modifications anyway? What is the general motivation behind switching the ETH client?
From the docs:
The second part of the Optimism node, the Optimism client software, is an almost completely vanilla version of Geth (opens new window). This means Optimism is close to identical to Ethereum under the hood. In particular, Optimism shares the same Ethereum Virtual Machine (opens new window), the same account and state structure (opens new window), and the same gas metering mechanism and fee schedule (opens new window).
Differences
Some differences from prev OVM and Bedrock OVM:

L1BLOCKNUMBER opcode doesn't exist anymore.
BASEFEE opcode now supported (since Bedrock includes London fork upgrade)
TIMESTAMP will now be updated every two seconds (every new block).
BLOCKNUMBER will advance every two seconds because we'll create a new block every two seconds, regardless of the amount of transactions.

Source: https://community.optimism.io/docs/developers/bedrock/
JSON RPC differences:

eth_sendMessage and eth_getAccounts now supported. Non-standard eth_getBlockRange not supported anymore.

@wsdt and
[@Michael Montour]
are basically leading this now since you both started, albeit from different angles!
the way I see it - the ultimate goal is to have erigon l2 , after it’s successfully modified, to run with optimism docker-compose: https://github.com/ethereum-optimism/optimism/blob/develop/ops-bedrock/docker-compose.yml#L24
but instead of https://github.com/ethereum-optimism/optimism/blob/develop/ops-bedrock/Dockerfile.l2
we’d put in our erigon and prove they both work with integration tests for bedrock: https://github.com/ethereum-optimism/optimism/tree/develop/packages/integration-tests-bedrock
altho I don’t see a reason why https://github.com/ethereum-optimism/optimism/tree/develop/integration-tests wouldn’t work too work!

This is a somewhat complicated project since we will be tracking two upstreams which are both in active development themselves. We will need to be careful about versioning, and should be prepared to have chunks of our work clobbered by restructuring or git rebasing in the upstream.
I would start by getting to a state where we can build and run Docker containers for upstream Optimism and/or upstream Erigon (as a private network) on a local machine. We can test them both independently, and tag particular revisions to use as a basis for development. I have started a branch which uses this approach to replace the Hardhat L1 in our Boba repository with an Erigon-based L1 as a proof of concept (not yet working).
Erigon supports the Engine API to work with an Eth2 beacon chain. We could set up a private Eth2 dev network (e.g. https://lighthouse-book.sigmaprime.io/setup.html#local-testnets ) to get some experience with this API for regular transactions, without the complexity of also handling the Optimistic Rollup transactions.
For the Optimism stack, I would not start much development work in our repo until the upstream system is able to run well enough to support basic round-trip functionality including deposits from L1->L2, local L2 transactions, and standard exits back to L1 with a fraud proof window. I don't know if the current upstream repo is at this stage now or (if not) how far away it is.
When we do have an upstream Optimism stack running with all of its pieces working, we will then be in a position to try redirecting its Engine API requests to a local Erigon node and then start fixing whatever is broken or missing (e.g. support for Optimism's new transaction type for L1->L2 deposits).
We can do other work in parallel to port our custom Boba features into Erigon (the ability to use an ERC20 as the fee token, the Hybrid Compute logic, etc).
Another task is to isolate Erigon's transaction processing into a standalone equivalent to Optimism's "minigeth", for use with a Cannon-style challenge protocol. Once it's working on a regular desktop, there will be additional work to ensure that it can run on a MIPS Linux system and then on the minimal Solidity-based MIPS VM.

[X] Erigon Docker containers running
[X] Optimism Bedrock containers running
My next steps would be to create a Bedrock version that runs with Erigon. What do you think @mmontour1306 ?

Next steps - there are two items which can be done in parallel:

Set up a local Erigon development network which operates in the post-Merge configuration, with Lighthouse or similar as the consensus layer and Erigon as the execution layer. This will allow us to develop and test extensions to the Engine API in a simpler environment than the full Optimism stack.

Modify the Dockerfiles and ops-bedrock/entrypoint.sh script in the Optimism repo to use Erigon (from a local image) in place of geth. The script includes logic to create keys and a custom genesis block before launching geth. Erigon shares some configuration parameters with geth but differs in other areas. We can do the work to get Erigon plugged into this script, and get the stack to a point where we can launch the services and see the errors which will result from not having the Optimism code implemented yet in Erigon.

Some specific items for Erigon:

Supporting the Deposit transaction type, including the cross-chain gas stipend
Deploying the L2 precompile contracts
Minting L2 ETH for Deposits
Collecting gas fees into SequencerFeeVault

Oh this is sweet, nice progress @mmontour1306 and @wsdt !
I have a question @mmontour1306 :

I have started a branch which uses this approach to replace the Hardhat L1 in our Boba repository with an Erigon-based L1 as a proof of concept (not yet working).

Why are you interested in this POC?

The l1-erigon branch was to figure out how to build Erigon and how to integrate its Docker container into our stack (including steps like generating a custom genesis block and configuring it to run as a private dev net). It was also a check on how stable and functional the code was at the time (e.g. finding the bug regarding a zero estimateGas result). That branch is no longer needed.

A couple of items based on the preliminary integration work:

Op-node is sending a GetPayload right after the ForkChoiceUpdated message, before Erigon has finished processing the block. For now I've worked around this with a fixed delay in op-node but we need some explicit sequencing in Erigon.
Erigon is expecting to extract a ChainID from the transaction, but this is not part of an Optimism deposit tx.

Also, we've discussed it before but I don't see it in this thread - we will need to implement eth_getProof support in Erigon:
ops-bedrock-op-node-1  | ERROR[08-19|20:46:30.053] failed to get contract proof             rpc=node err="the method is currently not implemented: eth_getProof"
ops-bedrock-op-node-1  | WARN [08-19|20:46:30.053] Served optimism_outputAtBlock            conn=<IP_ADDRESS>:58170 reqid=304 duration="885.931µs"   err="the method is currently not implemented: eth_getProof"

Small hint from the official Erigon team regarding this method:
it is too costly to implement with the current data model. It may be possible with erigon2.3, but to me it seems like a very niche method```

Could be something we could contribute to Erigon.

@wsdt can you figure out why and for what it's being used in Bedrock. Once you have that, let's review.
If we need to implement it - lets get it done asap.

Sure, that's what I've found:
Besides test cases, there seem to be basically two major uses of the getProof function that utilizes the call eth_getProof.
Both uses seem to be on the op-node.

The first one seems to be highly relevant for withdrawals, if the proof cannot be generated then the function returns an error.
FinalizeWithdrawalParameters queries L2 to generate all withdrawal parameters and proof necessary to finalize an withdrawal on L1.

The second seems especialy relevant for the L2ToL1MessagePasser, but this also seems to be related to withdrawals.

Looks like this definitely something we need. Happy for feedback!
Hope that helps.

https://erigon.substack.com/p/erigon-2-three-upgrades
Erigon 2 consists of 3 major upgrades.
Erigon is at 2.1 (upgrade 1) at the moment. 2.2 is at the integration phase, and 2.3 is at the prototyping stage.
--> Keeping up-to-date with their Repo becomes even more vital looking at this.

I'm closing this issue as obsolete. Please refer to the "Basic Erigon Implementation" epic in Zenhub.


Update build scripts to allow for arbitrary VS versions

Overview
This change allows a user to pass an arbitrary VS path instead of it being hardcoded.
Also updates to use Visual Studio 2019.
@MaxWang-MS, do you think this will affect the docs binary generation scripts?
Changes

Fixes: https://github.com/microsoft/MixedRealityToolkit-Unity/issues/6996

/azp run

I don't think the changes here will affect the docs binary pipeline but am wondering if you have validated the changes locally. My understanding is that the PR validation does not really validates this kind of change.

I don't think the changes here will affect the docs binary pipeline but am wondering if you have validated the changes locally. My understanding is that the PR validation does not really validates this kind of change.

Yep! I've run this script locally with success, but I'll spin up a feature branch to run CI on and ensure nothing breaks there.

@MaxWang-MS https://microsoft.visualstudio.com/Analog/_build/results?buildId=31795222&view=results succeeded! I also went ahead and reverted the default back to VS 2017 for now, since that's what the pipeline still uses (and I tried updating that to 2019 this morning without success).

/azp run


Add docs for typical use cases

i.e. how to construct a fiber, how to convert between fibers. Also, 3 big matmuls maybe? Getting data into and out of Finch, etc.

Some ideas for documentation structure attached
structure1.md
structure2.md
structure3.md
structure4.md
structure5.md


Windows / Python 3.9 not working

The following works if you use -3.8 instead of -3.9
C:\Users\ericf>cd %TEMP%

C:\Users\ericf\AppData\Local\Temp>py -3.9 -m venv zmq_env

C:\Users\ericf\AppData\Local\Temp>zmq_env\Scripts\activate

(zmq_env) C:\Users\ericf\AppData\Local\Temp>pip install --upgrade pip wheel
Collecting pip
  Using cached pip-20.3-py2.py3-none-any.whl (1.5 MB)
Collecting wheel
  Using cached wheel-0.36.0-py2.py3-none-any.whl (34 kB)
Installing collected packages: pip, wheel
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\ericf\\AppData\\Local\\Temp\\pip-uninstall-l68ui8yb\\pip.exe'
Consider using the `--user` option or check the permissions.

(zmq_env) C:\Users\ericf\AppData\Local\Temp>pip install pyzmq
Collecting pyzmq
  Using cached pyzmq-20.0.0-cp39-cp39-win_amd64.whl (699 kB)
Installing collected packages: pyzmq
Successfully installed pyzmq-20.0.0

(zmq_env) C:\Users\ericf\AppData\Local\Temp>py -c "import zmq"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\ericf\AppData\Local\Temp\zmq_env\lib\site-packages\zmq\__init__.py", line 50, in <module>
    _load_libzmq()
  File "C:\Users\ericf\AppData\Local\Temp\zmq_env\lib\site-packages\zmq\__init__.py", line 28, in _load_libzmq
    from . import libzmq
ImportError: DLL load failed while importing libzmq: The specified module could not be found.

Thanks for the report! How have you installed Python 3.9? Does C:\Users\ericf\AppData\Local\Temp\zmq_env\lib\site-packages\zmq\libzmq.cp39-win_amd64.pyd exist?
Can you show the output of:
import sysconfig
print(sysconfig.get_config_var('EXT_SUFFIX'))
print(sysconfig.get_config_var('SOABI'))

problem is pyzmq with python3.9 wheel is served without "concrt140.dll", "msvcp140.dll".

problem is pyzmq with python3.9 wheel is served without "concrt140.dll", "msvcp140.dll".

This is fixed in pyzmq 21 by #1470, though there's another issue that will be fixed shortly in 21.0.1 (#1473). Should be out later today.

This is fixed in pyzmq 21 by #1470, though there's another issue that will be fixed shortly in 21.0.1 (#1473). Should be out later today.


Autocompletion not working inside template literals of TypeScript

VSCode Version: 1.12.1
OS Version: macOS 10.12.2

Steps to Reproduce:

Write following code in typescript

let name = 'foo';
let result = `${name}`;

No autocompletion poped for name variable.

Try setting:
	"editor.quickSuggestions": {
		"other": true,
		"comments": false,
		"strings": true
	}

Closing as duplicate of #23962

@mjbvz I think for string interpolation case, like in ES6, C#, Typescript, they'd better not be categorized into "String", since we always need autocompletion for them


Unable to generate "base" skeleton for R-3.4.0, windows, 64-bit.

Failed to generate a valid skeleton for 'base'. Please file a ticket under https://github.com/holgerbrandl/r4intellij/issues
java.lang.Throwable
at com.intellij.openapi.diagnostic.Logger.error(Logger.java:132)
at com.r4intellij.packages.RSkeletonGenerator.lambda$updateSkeletons$1(RSkeletonGenerator.java:247)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I ran "R -f skeletonize_package.R --args base base.skeleton.R.txt > skeletonize_package.R.stdout.txt
If I take the resulting skeleton file, rename it, and place it in the r_skeletons directory, I am able to again resolve references in the base package in the IDE.
skeletonize_package.R.stdout.txt
base.skeleton.R.txt

I see the same behavior with R 3.4.1 on x64 Windows

I can confirm changing from R 3.4.1 to 3.3.3 also fixed this issue for me.

duplicate of #120


Automate zppy tests

Automate zppy tests. Resolves #516.

@chengzhuzhang This is an initial attempt at automating the steps from https://e3sm-project.github.io/zppy/_build/html/main/dev_guide/release_testing.html -- specifically just the a. test dev steps, since that is more useful in the near-term. Once I get that working, I'm planning to make a different script for the other two cases.
There's a number of issues in fully automating this workflow. See the #TODO in-line comments.

Made some adjustments post-zppy/zppy-interfaces split. Can't run these changes yet because 1) need to get around issues with conda activation in a script and 2) automating the waiting for multiple jobs to finish.


export warn for Vue 2

Vue 3 exports the warn function. VueDemi should also export the warn function.
Fixes https://github.com/posva/vue-promised/issues/233

See https://github.com/antfu/vue-demi/pull/19
always search existing issues and pull requests first 😄

Ups. Sorry.
^^


Add typescript transpile only (skip typecheck)

Hi team!
For now typescript work great, but it have problem: compile time in large project pretty slow because typescript do typechecking (it sample like we run compile via cli tsc). In ts-node they build flag --transpile-only which allow skip typechecking and compile code only.
Since almost people working project use IDEs like VSCode with full support typecheck when writing code , really no need second type checking step when compile. Please support new option skipTypechecking to speed up compile time
Thankyou!

Hi @TrungRueta! Thanks for the feedback 😁 An option to skip type checking really makes sense. I'll implement it whenever I get some free time.


Implement stocksnap popularity and popularity documentation

Fixes

Fixes #182 by @zackkrida
Description

Update: StockSnap again modified their API for us to include 'raw' counts of downloads, views,
and favorites. This PR now uses the downloads as the popularity metric for StockSnap.
This PR adds StockSnap to our popularity calculations. We're using download count as the metric with which to calculate popularity.
Old details + historical context:

Add documentation for popularity
Add stocksnap "favorites" field to popularity calculations

Sadly, this needs to wait, as there's a problem with StockSnap. StockSnap's api is focused on the presentation of this data, so numbers are actually automatically converted to strings like "1k" in the back-end, instead of on the client! Our popularity code expects numbers so sadly this fails. "favorites" is the only StockSnap popularity metric that is used so infrequently that it has all numbers < 1000, so it's not yet an issue there. We'd have to ask Adrian at StockSnap to stop passing formatted numbers through the API and instead format on the client-side of StockSnap.
Still, this was a good exercise.

Technical details

The popularity code is pretty clever, and it was fun to learn how it works! Calculating the 85th percentile for each provider seems to find good results. I hope we can extend it to work with multiple metrics for each provider in the future.
Tests

just recreate
Observe the presence of StockSnap in the image_popularity_metrics and image_popularity_constants DB tables
Visit airflow at http://localhost:9090
Enable the tsv_to_postgres_loader and stocksnap_workflow DAGs with the toggles.
Wait.
When stocksnap is done, and the tsvs have been added to the DB, enable the recreate_image_popularity_calculation workflow.
Wait.
When that's done, observe the calculated val and constant in the image_popularity_constants table for the StockSnap row.
Spot check a few of the images in the image_view, looking for values in the standardized_popularity column.

[x] My pull request targets the default branch of the repository (main or master).
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[x] I added or updated tests for the changes I made (if applicable).
[x] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

GitHub closed my original PR (#183) after a rebase—really odd!

Sadly, this needs to wait, as there's a problem with StockSnap

Should this PR be a draft? Or is this comment out of date?

@sarayourfriend I just amended the description with more info, this is good to go 😄

image_popularity_constants

Just a note, but I don't see this table locally:
openledger=# \dt
                 List of relations
 Schema |           Name           | Type  | Owner
--------+--------------------------+-------+--------
 public | audio                    | table | deploy
 public | audio_popularity_metrics | table | deploy
 public | image                    | table | deploy
 public | image_popularity_metrics | table | deploy
 public | old_image                | table | deploy
(5 rows)

Is that an issue?

@sarayourfriend you likely have it. Both image_view and image_popularity_constants are MATERIALIZED VIEWS and as such not visible with \dt. Plain \d should show them.

Ahh thanks @zackkrida! \d did the trick.


Updated duplicate-image-remover.py

Please fill all the details and checkbox, so we can merge PR faster.
Pull Request Template
script name - duplicate-image-remover.py
What have you Changed
what you changed in the codebase. write here
I have added few more functions which can help the program to work more efficiently.
Issue no.(must) -
Self Check(Tick After Making pull Request)

[ ] This issue was assigned to me.
[x] One Change in one Pull Request
[x] My file is in the proper folder
[x] I am following clean code and Documentation and my code is well-linted with flake8.
[ ] I have added README.md and requirements.txt with my script

If the issue was not assigned to you Please don't make a PR. It will mark as invalid.
Join Us on Discord:- https://discord.gg/JfbK3bS
acknowledgment

[x] I have tested the script and distributed it under an MIT license.

Please could you explain the code and how your function will be more efficiently?


Update init_poolparty.sh

needs libc6-dev libc6 installed deps, otherwise:
go: downloading github.com/opencontainers/go-digest v1.0.0
runtime/cgo
_cgo_export.c:3:10: fatal error: stdlib.h: No such file or directory
3 | #include <stdlib.h>
|          ^~~~~~~~~~

Can you provide more info on your OS? We've tested on MacOS and alpine linux - neither of which had libc6 installed, yet were able to run successfully

It was on Ubuntu 20.04
12 июля 2022 г., 06:36 +0300, sampocs @.***>, писал:

Can you provide more info on your OS? We've tested on MacOS and alpine linux - neither of which had libc6 installed, yet were able to run successfully
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

I've faced the same issue. You can reproduce it on Debian 11 server (I'm using Hetzner):
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

This PR solves the issue, so I'd recommend merging it to the main branch.


Add WebSocket Close Event

Reference: https://developer.mozilla.org/en-US/docs/Web/API/CloseEvent

LGTM. Build failures are unrelated.

@facebook-github-bot shipit


Fix for issue 176

https://wwwin-github.cisco.com/netascode/nac-sdwan/issues/176

Hello @danischm
Kindly review and approve.

Fix bug with loss_threshold_percentage under traffic data.

Hello @danischm
Can you review and approve


etcd_http_failed_total: how find each error

Hello. Can someone please help me figure out how to determine the cause/source of each failure/error for the etcd_http_failed_total metric?
(I know this probably isn't a bug in etcd - please let me know if there's somewhere else I should redirect this.)
In AWS I'm running a Kubernetes cluster (for masters & agents) and an etcd cluser. I'm running kube-prometheus, and an alert is firing for the HighNumberOfFailedHTTPRequests rule - reference info:

the "High number of failed requests" section on the Developing Prometheus alerts for etcd coreos blog.
github repos: coreos/etcd/etcd3_alert.rules and coreos/prometheus-operator/kube-prometheus/etcd3.rules.yaml.

In alertmanager I get the following (which shows 0.327586206896552 as the value

In prometheus I did the following 2 queries:

sum(rate(etcd_http_failed_total{job="etcd"}[5m])) BY (method) = 0.07037037037037037 for QGET
sum(rate(etcd_http_received_total{job="etcd"}[5m])) BY (method) = 0.21481481481481482 for QGET
(0.07037037037037037 / 0.21481481481481482 = 0.327586206896552)

And when I simply query for etcd_http_failed_total:

):
How do I figure out what is causing the 123058 occurrences of etcd_http_failed_total on <IP_ADDRESS>? (That etcd node is the leader of the 5 etcd nodes.)

More info:
I use custom terraform cloud-config scripts to setup the etcd service on each etcd node - here's a snippet:
            Environment="ETCD_IMAGE_TAG=v3.3"
            ExecStart=/usr/lib/coreos/etcd-wrapper \
              --name ${InstanceName} \
              --listen-client-urls=http://${InstanceIP}:2379 \
              --advertise-client-urls=http://${InstanceIP}:2379 \
              --listen-peer-urls=http://${InstanceIP}:2380 \
              --initial-advertise-peer-urls=http://${InstanceIP}:2380 \
              --initial-cluster=${EtcdInitialCluster} \
              --initial-cluster-state=new \
              --debug

On the etcd leader node, here's the beginning of the output from running journalctl -u etcd-member:
-- Logs begin at Wed 2017-04-05 15:54:37 UTC, end at Fri 2018-04-20 16:22:35 UTC. --
Mar 25 01:44:13 ip-192-168-225-40 systemd[1]: Starting etcd (System Application Container)...
Mar 25 01:44:13 ip-192-168-225-40 etcd-wrapper[937]: ++ id -u etcd
Mar 25 01:44:13 ip-192-168-225-40 etcd-wrapper[937]: + exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid --trust-keys-from-https --mount volume=coreos-systemd-dir,target=/run/systemd/system --volume coreos-systemd-dir,kind=host,source=/run/systemd/system,readOnly=true --mount volume=coreos-notify,target=/run/systemd/notify --volume coreos-notify,kind=host,source=/run/systemd/notify --set-env=NOTIFY_SOCKET=/run/systemd/notify --volume coreos-data-dir,kind=host,source=/var/lib/etcd,readOnly=false --volume coreos-etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume coreos-usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume coreos-etc-hosts,kind=host,source=/etc/hosts,readOnly=true --volume coreos-etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=coreos-data-dir,target=/var/lib/etcd --mount volume=coreos-etc-ssl-certs,target=/etc/ssl/certs --mount volume=coreos-usr-share-certs,target=/usr/share/ca-certificates --mount volume=coreos-etc-hosts,target=/etc/hosts --mount volume=coreos-etc-resolv,target=/etc/resolv.conf --inherit-env --stage1-from-dir=stage1-fly.aci quay.io/coreos/etcd:v3.3 --user=232 -- --name etcd-1 --listen-client-urls=http://<IP_ADDRESS>:2379 --advertise-client-urls=http://<IP_ADDRESS>:2379 --listen-peer-urls=http://<IP_ADDRESS>:2380 --initial-advertise-peer-urls=http://<IP_ADDRESS>:2380 --initial-cluster=etcd-1=http://<IP_ADDRESS>:2380,etcd-2=http://<IP_ADDRESS>:2380,etcd-3=http://<IP_ADDRESS>:2380,etcd-4=http://<IP_ADDRESS>:2380,etcd-5=http://<IP_ADDRESS>:2380 --initial-cluster-state=new
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.502148 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.556851 W | pkg/flags: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.556863 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.556873 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.3
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.556902 I | etcdmain: etcd Version: 3.3.2
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.556910 I | etcdmain: Git SHA: c9d46ab37
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.556916 I | etcdmain: Go Version: go1.9.4
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.556923 I | etcdmain: Go OS/Arch: linux/amd64
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.556931 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.557002 W | etcdmain: found invalid file/dir data under data dir /var/lib/etcd (Ignore this if you are upgrading etcd)
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.557153 I | embed: listening for peers on http://<IP_ADDRESS>:2380
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.557238 I | embed: listening for client requests on <IP_ADDRESS>:2379
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.625289 I | etcdserver: name = etcd-1
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.625314 I | etcdserver: data dir = /var/lib/etcd
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.625328 I | etcdserver: member dir = /var/lib/etcd/member
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.625335 I | etcdserver: heartbeat = 100ms
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.625341 I | etcdserver: election = 1000ms
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.625347 I | etcdserver: snapshot count = 100000
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.625360 I | etcdserver: advertise client URLs = http://<IP_ADDRESS>:2379
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.625368 I | etcdserver: initial advertise peer URLs = http://<IP_ADDRESS>:2380
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.625401 I | etcdserver: initial cluster = etcd-1=http://<IP_ADDRESS>:2380,etcd-2=http://<IP_ADDRESS>:2380,etcd-3=http://<IP_ADDRESS>:2380,etcd-4=http://<IP_ADDRESS>:2380,etcd-5=http://<IP_ADDRESS>:2380
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.643802 I | etcdserver: starting member caee3cd26c1e54ba in cluster 8ec3f139e70bd26a
Mar 25 01:44:21 ip-192-168-225-40 etcd-wrapper[937]: 2018-03-25 01:44:21.643844 I | raft: caee3cd26c1e54ba became follower at term 0

To give an e.g., from a slightly different angle:
core@ip-192-168-225-40 ~/k $ curl http://ip-192-168-225-40:2379/config/local/log -XPUT -d '{"Level":"DEBUG"}'

core@ip-192-168-225-40 ~/k $ journalctl -u etcd-member > etcd-log-1.txt

core@ip-192-168-225-40 ~/k $ curl --silent -l http://ip-192-168-225-40:2379/metrics | grep 'etcd_http_failed_total{code="404",method="GET"}'
etcd_http_failed_total{code="404",method="GET"} 2

# How to cause an error:
core@ip-192-168-225-40 ~/k $ curl http://ip-192-168-225-40:2379/v2/keys/does_Not_Exist
{"errorCode":100,"message":"Key not found","cause":"/does_Not_Exist","index":3769}

# Verify the previous curl cmd caused the error count to go up by 1:
core@ip-192-168-225-40 ~/k $ curl --silent -l http://ip-192-168-225-40:2379/metrics | grep 'etcd_http_failed_total{code="404",method="GET"}'
etcd_http_failed_total{code="404",method="GET"} 3

core@ip-192-168-225-40 ~/k $ journalctl -u etcd-member > etcd-log-2.txt

core@ip-192-168-225-40 ~/k $ diff etcd-log-1.txt etcd-log-2.txt | egrep "Logs begin at|does_Not_Exist"
< -- Logs begin at Fri 2018-04-13 22:49:48 UTC, end at Fri 2018-04-20 16:44:31 UTC. --
> -- Logs begin at Fri 2018-04-13 22:49:48 UTC, end at Fri 2018-04-20 16:44:58 UTC. --
> Apr 20 16:44:49 ip-192-168-225-40 etcd-wrapper[897]: 2018-04-20 16:44:49.019552 D | etcdserver/api/v2http: [GET] /v2/keys/does_Not_Exist remote:<IP_ADDRESS>:52486

But regarding the diff command above, the challenge I'm trying to solve is (even with the etcd log level set to debug) I can't find anything to tell me which http requests resulted in an error (I did also try looking in the logs of the rkt container that is started by etcd-wrapper, but I didn't find anything there either):
Any suggestions would be greatly appreciated. thanks!

How do I figure out what is causing the 123058 occurrences of etcd_http_failed_total on <IP_ADDRESS>? (That etcd node is the leader of the 5 etcd nodes.)

One idea is to use something like wireshark or tcpdump to monitor the traffic and get a better idea of the source payload etc.  Being that you have http traffic wireshark would give you a very good idea of what is going on. I hope that helps.

How do I figure out what is causing the 123058 occurrences of etcd_http_failed_total on <IP_ADDRESS>? (That etcd node is the leader of the 5 etcd nodes.)

I would first see if etcd v2 client request has ever failed (those metrics are from v2 request failures) -- probably logged somewhere in Kubernetes API server. And then look at etcd server logs, to see if it prints out any warnings.

On the etcd leader node, here's the beginning of the output from running

If that metrics still grows, I would also collect it over time and try to find the timestamps when error spikes happen, and see if server prints out any errors or warnings.

@jolson490 Any updates?

@jolson490 Sure.
QGET is quorum read request, which may fail from various reasons. Server logs from <IP_ADDRESS>:2379 would be helpful. That node might have been isolated.

Follow-up for anyone who finds this later:  we used Wireshark to trace the "failures" to Calico polling etcd for network metadata, some elements of which didn't exist (eg bgp as_num), which produced a legit 404.  So, if you have an app that legitimately creates 404s when looking for data that may or may not exist, you may need to tailor this alert to ignore that normal behavior.
> GET /v2/keys/calico/bgp/v1/host/[redacted]/as_num?quorum=true&recursive=false&sorted=false HTTP/1.1
> Host: <IP_ADDRESS>:2379
> User-Agent: Go-http-client/1.1
> Accept-Encoding: gzip

< HTTP/1.1 404 Not Found
< Content-Type: application/json
< X-Etcd-Cluster-Id: [redacted]
< X-Etcd-Index: 1118
< Date: Thu, 14 Jun 2018 20:19:33 GMT
< Content-Length: 125
<
< {"errorCode":100,"message":"Key not found","cause":"/calico/bgp/v1/host/[redacted]/as_num","index":1118}

@gyuho @jolson490 can you reopen please? Havin same issue with
etcd_version = "3.2.17"
k8s_version = "1.10.2"

This is reproducible across multiple clusters in diff regions.
(I also was asking on SO)

super helpful! many thanks!!
Yes we're on calico and etcd3, I'm happy to replace these alerts


Add pull request template

Applicable spec: NA
Overview

Add template for future pull requests.
Generate docs with src-docs
Publish charm on push to main

Rationale

Simplify the process of PR creation.
Simplify docs generation.
Automate publish of charm.

Checklist

[x] The charm style guide was applied
[x] The contributing guide was applied
[x] The changes are compliant with ISD054 - Managing Charm Complexity
[x] The documentation is generated using src-docs
[ ] The documentation for charmhub is updated.
[x] The PR is tagged with appropriate label (urgent, trivial, complex)

The removal of from __future__ import annotations is needed as lazy-docs crashes with it.
The line can be removed anyways, as we are only supporting jammy (python 3.10) and up now.


Revert "Update builder-base image tag in Prowjobs"

Reverts aws/eks-distro-prow-jobs#551

/approve

/lgtm


Add float parser, generalise one_of/.optional() a bit

add chars::float to parse floats
add chars::case_insensitive_eq for case insensitive token matching.
generalise one_of and .optional(); allow expressions passed to either to return Options or bools, not just Options.

@Easyoakland just FYI :)

Why implement these in an extension trait instead of the same way that a user of the library would implement these (free functions)?
Would it not make more sense for some of these functions to use a stack allocated buffer with a maximum size since there are a maximum number of digits? For example u8 max is 256 so 3 chars is sufficient.

Why implement these in an extension trait instead of the same way that a user of the library would implement these (free functions)?

I started with freestanding functions, but the extension trait felt ergonomically superior in the end. One example is for the functions parse_f32 and parse_f64; as freestanding functions you need to call parse_f32::<_, String>(toks) (ie need 2 generics, one for tokens and one for buffer.. granted, I could elide the tokens using impl syntax but that doesn't feel right in a proper library because one may want to specify it). With an extension crate it's toks.parse_f32::<String> which is just nicer and the one generic is easier to explain.
Subjective I'm sure, but I tried both and preferred this!

Would it not make more sense for some of these functions to use a stack allocated buffer with a maximum size since there are a maximum number of digits? For example u8 max is 256 so 3 chars is sufficient.

I didn't add any functions like that; only ones for f32/f64?

(ie need 2 generics, one for tokens and one for buffer.. granted, I could elide the tokens using impl syntax but that doesn't feel right in a proper library because one may want to specify it)

I'm not sure why that's preferable to impl syntax. If you want to specify a type why not do that earlier?
fn parser<B>(tok: &mut impl Token...){...}
let a: StrTokens<'static> = "1234".into_tokens();
parser::<String>(&mut a)

If the syntax that user's use is different from the library I think that user's defining new combinators (the main thing) becomes a second class citizen. I'm of the opinion that combinators defined in the library should be useful as a big tutorial.

I didn't add any functions like that; only ones for f32/f64?

Oh, apparently floats with excessive digits of precision are just rounded so the input can be arbitrarily long.

I also think the naming parse_f32(), parse_f(), float(), ...  is worse than f32(), float(), skip_float().
Another advantage of free functions is then if it is actually advantageous to have both parse and skip versions of things to make two modules named parse and skip.
I'm not sure it is though. Why not only have the skip versions and allow the user to .parse() if they want to?

Another alternative could be making these things Tokens. Then you could do f32(&mut toks).as_iter().count() !=0 (or make consume return a bool) for the bool case and f32(&mut toks).parse() for the parse case. For this to work the macro for defining token newtypes should probably be made first. Or it could be a function returning a particular Tokens that is parameterized by its parse function.
At this point it might make sense to impl<T: Tokens> Tokens for &mut T and take everything by value like in core::iter then you can f32(toks).parse().

If the syntax that user's use is different from the library I think that user's defining new combinators (the main thing) becomes a second class citizen. I'm of the opinion that combinators defined in the library should be useful as a big tutorial.

I don't think it's a huge deal either way, but I do agree with the tutorial perspective and promoting freestanding funcs as a good way to do these things. I might revert to freestanding funcs with impl syntax.

I also think the naming parse_f32(), parse_f(), float(), ... is worse than f32(), float(), skip_float().

This is just opinion but I prefer the current names;  parse_f32 is nicely consistent with Tokens::parse so I prefer that. float is consistent with methods like Tokens::token and Tokens::tokens in that it will return a boolean whether it's seen the thing in question. parse_f is not public anyway.

I'm not sure it is though. Why not only have the skip versions and allow the user to .parse() if they want to?

I was going to just have float(), but these are already helper utils, and I feel like most of the time somebody would actually want to get the float value back. I was originally going to not expose float at all and just parse_f32/parse_f64 and may still go that path until there is some use case for exposing float

Another alternative could be making these things Tokens. Then you could do f32(&mut toks).as_iter().count() !=0 (or make consume return a bool) for the bool case and f32(&mut toks).parse() for the parse case.

I guess right now I don't see the point so I'd rather avoid that extra complexity. Would somebody really want to iterate over the tokens that are a part of a float and do something with only a subset of them? I doubt it. (if I kept float() then somebody could do as parse_f and take a slice to the tokens anyway if they really want, but I don't see why really, so it doesn't compel me to keep it really).


Fix comment postId field name

Change postid to postId according to the payload http://jsonplaceholder.typicode.com/posts/1/comments

thank you :)


Error while running Metrics collection of inbox and outbox

While going through the example on the website I encountered an issue in combination with MS SQL Server.
This can be reproduced with the InMemoryMediator example running with a MS SQL Server.
I don't know if this is different with other databases, but while starting the application I receive the following error while it's polling to read the incoming/outgoing messages.
https://github.com/JasperFx/wolverine/blob/4ba7b42fb927803b1223d69a504b61d2426b74c6/src/Persistence/Wolverine.SqlServer/Persistence/SqlServerMessageStore.cs#L69-L97
A potential fix could be to combine both queries with a union all ?
If you want I could create a PR with this fix.
fail: Wolverine.Runtime.WolverineRuntime[0]
      Error while running Metrics collection of inbox and outbox
      System.InvalidOperationException: There is already an open DataReader associated with this Connection which must be closed first.
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.ValidateConnectionForExecute(SqlCommand command)
         at Microsoft.Data.SqlClient.SqlCommand.ValidateCommand(Boolean isAsync, String method)
         at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
         at Microsoft.Data.SqlClient.SqlCommand.BeginExecuteReaderInternal(CommandBehavior behavior, AsyncCallback callback, Object stateObject, Int32 timeout, Boolean inRetry, Boolean asyncWrite)
         at Microsoft.Data.SqlClient.SqlCommand.BeginExecuteReaderAsyncCallback(AsyncCallback callback, Object stateObject)
         at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)
         at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
      --- End of stack trace from previous location ---
         at Wolverine.SqlServer.Persistence.SqlServerMessageStore.FetchCountsAsync() in C:\Users\timde\dev\playgrounds\wolverine\src\Persistence\Wolverine.SqlServer\Persistence\SqlServerMessageStore.cs:line 95
         at Wolverine.SqlServer.Persistence.SqlServerMessageStore.FetchCountsAsync() in C:\Users\timde\dev\playgrounds\wolverine\src\Persistence\Wolverine.SqlServer\Persistence\SqlServerMessageStore.cs:line 100
         at Wolverine.SqlServer.Persistence.SqlServerMessageStore.FetchCountsAsync() in C:\Users\timde\dev\playgrounds\wolverine\src\Persistence\Wolverine.SqlServer\Persistence\SqlServerMessageStore.cs:line 100
         at Wolverine.RDBMS.Durability.MetricsCalculator.ExecuteAsync(IMessageDatabase database, IDurabilityAgent agent, IDurableStorageSession session) in C:\Users\timde\dev\playgrounds\wolverine\src\Persistence\Wolverine.RDBMS\Durability\MetricsCalculator.cs:line 31
         at Wolverine.RDBMS.Durability.DurabilityAgent.processActionAsync(IDurabilityAction action) in C:\Users\timde\dev\playgrounds\wolverine\src\Persistence\Wolverine.RDBMS\Durability\DurabilityAgent.cs:line 196

I'm generally leery of the UNION thing here. I think I'd vote to just be more aggressive with closing the first data reader instead of depending on the reader.DisposeAsync() to close it at the end of the method.
Good catch, and thank you. But I'd very happily take a PR to address this if you're up for it, or I'm constantly in the code these days.

That's a fair point 👍
I'm happy to contribute where I can, I should be able to open a PR later this week.

@timdeschryver Just let me know, happy to take a PR, and happy to deal with it if you can't get to it


NFS shares not mounted after reboot

Description
Since version 2605.5.0 NFS shares are not mounted after reboots.
Impact
NFS share is not mounted after reboot and upgrade.
Environment
flatcar-install -d /dev/sda -i ignition.json -o vmware_raw
NFS share: https://docs.flatcar-linux.org/os/mounting-storage/#mounting-nfs-exports
steps to reproduce
cat /etc/systemd/system/mnt-nfs.mount

[Unit]
Before=remote-fs.target
[Mount]
What=<ip_of_nfs_server>:/nfs
Where=/mnt/nfs
Type=nfs
[Install]
WantedBy=remote-fs.target

ls -l /etc/systemd/system/remote-fs.target.wants/

total 4
lrwxrwxrwx. 1 root root 33 Sep 22 11:29 mnt-nfs.mount -> /etc/systemd/system/mnt-nfs.mount

systemctl status mnt-nfs.mount

● mnt-nfs.mount - /mnt/nfs
     Loaded: loaded (/etc/systemd/system/mnt-nfs.mount; enabled; vendor preset: disabled)
     Active: active (mounted) since Tue 2020-09-22 11:50:06 UTC; 31min ago
      Where: /mnt/nfs
       What: :/nfs
      Tasks: 0 (limit: 28807)
     Memory: 356.0K
     CGroup: /system.slice/mnt-nfs.mount

reboot
systemctl status mnt-nfs.mount

● mnt-nfs.mount - /mnt/nfs
     Loaded: loaded (/etc/systemd/system/mnt-nfs.mount; enabled; vendor preset: disabled)
     Active: inactive (dead)
      Where: /mnt/nfs
       What: :/nfs

sudo systemctl start mnt-nfs.mount
systemctl status mnt-nfs.mount

● mnt-nfs.mount - /mnt/nfs
     Loaded: loaded (/etc/systemd/system/mnt-nfs.mount; enabled; vendor preset: disabled)
     Active: active (mounted) since Tue 2020-09-22 12:38:00 UTC; 6s ago
      Where: /mnt/nfs
       What: :/nfs
      Tasks: 0 (limit: 28807)
     Memory: 344.0K
     CGroup: /system.slice/mnt-nfs.mount

Expected behavior
NFS share is mounted after reboot and upgrade.

The upgrade to 2605.5.0 meant upgrading to systemd 245. It's likely that there's some difference there causing your unit not to start automatically. Could you check the status of remote-fs.target?

@marga-kinvolk
systemctl status remote-fs.target
● remote-fs.target - Remote File Systems
     Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; disabled; vendor preset: disabled)
     Active: inactive (dead) since Tue 2020-09-22 12:50:50 CEST; 2h 18min ago
       Docs: man:systemd.special(7)

Right, it's disabled.  It's a consequence of this change:
        * During package installation (with `ninja install`), we would create
          symlinks for<EMAIL_ADDRESS>systemd-networkd.service,
          systemd-networkd.socket, systemd-resolved.service,
          remote-cryptsetup.target, remote-fs.target,
          systemd-networkd-wait-online.service, and systemd-timesyncd.service
          in /etc, as if `systemctl enable` was called for those units, to make
          the system usable immediately after installation. Now this is not
          done anymore, and instead calling `systemctl preset-all` is
          recommended after the first installation of systemd.

(From https://github.com/systemd/systemd/blob/master/NEWS)
We solved this for getty in flatcar-linux/coreos-overlay#400, I guess we need to solve it for remote-fs.target as well.
In the meantime, you can enable the unit yourself to solve the issue for your instances.

On a fresh (qemu) Stable-2605 installation I see:
localhost ~ # systemctl status remote-fs.target
● remote-fs.target - Remote File Systems
     Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: disabled)
     Active: active since Tue 2020-09-22 13:06:28 UTC; 8min ago
       Docs: man:systemd.special(7)

@a2yp just to narrow down the symptoms, did you upgrade or deploy a fresh instance?

@t-lo The instance automatically upgraded from 2512.5.0 to 2605.5.0.

I installed a qemu instance with 2512.5.0 and upgraded and I could reproduce this issue.  With 2512:
core@localhost ~ $ systemctl status remote-fs.target
● remote-fs.target - Remote File Systems
   Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; disabled; vendor preset: disabled)
   Active: active since Tue 2020-09-22 13:54:52 UTC; 32s ago
     Docs: man:systemd.special(7)

Notice how it says it's disabled, but still active. And there are no events (it's not that I forgot to copy them, there were no events).
After upgrading and rebooting:
core@localhost ~ $ systemctl status remote-fs.target
● remote-fs.target - Remote File Systems
     Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; disabled; vendor preset: disabled)
     Active: inactive (dead) since Tue 2020-09-22 14:03:22 UTC; 2min 28s ago
       Docs: man:systemd.special(7)

Sep 22 14:03:22 localhost systemd[1]: Reached target Remote File Systems.
Sep 22 14:03:22 localhost systemd[1]: Stopped target Remote File Systems.

Now it's disabled and inactive, but we do see a couple of events. So, maybe something else is at play.

This is relevant, I think.
In the previous stable:
Flatcar Container Linux by Kinvolk oldstable (2512.5.0)
core@localhost ~ $ find /usr/lib/systemd/ | grep remote-fs
/usr/lib/systemd/system/remote-fs.target
/usr/lib/systemd/system/multi-user.target.wants/remote-fs.target
/usr/lib/systemd/system/remote-fs-pre.target
/usr/lib/systemd/system/remote-fs.target.wants
/usr/lib/systemd/system/remote-fs.target.wants/var-lib-machines.mount

In the current stable (after upgrading):
Flatcar Container Linux by Kinvolk stable (2605.5.0)
core@localhost ~ $ find /usr/lib/systemd/ | grep remote-fs
/usr/lib/systemd/system/remote-fs.target.wants
/usr/lib/systemd/system/remote-fs.target.wants/var-lib-machines.mount
/usr/lib/systemd/system/remote-fs-pre.target
/usr/lib/systemd/system/remote-fs.target

So, /usr/lib/systemd/system/multi-user.target.wants/remote-fs.target is missing.

And, for completion, on a freshly installed machine:
Flatcar Container Linux by Kinvolk stable (2605.5.0)
core@localhost ~ $ find /usr/lib/systemd/ | grep remote-fs
/usr/lib/systemd/system/remote-fs.target.wants
/usr/lib/systemd/system/remote-fs.target.wants/var-lib-machines.mount
/usr/lib/systemd/system/remote-fs-pre.target
/usr/lib/systemd/system/remote-fs.target

This is of course the same, BUT we have this file which is not present in the upgrade machine:
core@localhost ~ $ find /etc/systemd/ | grep remote-fs
/etc/systemd/system/multi-user.target.wants/remote-fs.target

I'm not sure why multi-user.target.wants/remote-fs.target ended up in /etc/ instead of /usr, but I think that's what we should fix. It should be in /usr so that it reaches all users, not just the fresh installs.

Thanks for this got my mounts fixed now! I'll update as soon as the permanent fix get's pushed.


feat(junit): Implement automatic saving of traces and screenshots via fixtures

Resolves #1526.  This PR also adds saving screenshots.
I tried to keep the API of the fixture-generated artifacts as close to node as possible.
Traces:
By default, if no outputDir is specified in the Options, the default output path will be projectDir/test-results/fully.qualified.ClassName.testMethodName-browserName/trace.zip.  If an outputDir is specified in the Options then the artifacts will go in userSpecifiedOutputDir/fully.qualified.ClassName.testMethodName-browserName/trace.zip.
Screenshots:
Screenshots follow the same nomenclature as the traces with the exception that screenshot files are named test-finished-1.png.  Depending on the number of pages a BrowserContext has, the 1 will be incremented per Page.
I added tests to test this feature.  Since traces and screenshots are created after the test is finished, I needed to add a new test dependency (junit-platform-testkit).   This is JUnit's preferred way to test extensions.
@Transient annotation was created because the new tests should not be run by themselves and only run through the EngineTestKit.  I updated the GitHub actions to exclude these tests.
There is some weird threading issues happening when the new tests run with the existing suite so I added a new step in the CI actions to run the new tests separately.

I don't think the failures here are related to the PR.  could we try re-running?

I don't think the failures here are related to the PR. could we try re-running?

Restarted. But it looks like at least some of them are in the new tests.

@yury-s the threading issues, I believe, are resulting from tests running via the engine kit and the normal way at the same time.
I believe the threading issues are due to use using threadlocal in our extensions. Somewhere, playwright is getting closed by the ExtensionContext but there are still tests running on that thread.
We should eventually move away from using threadlocal and store objects in the ExtensionContext (this is what Junit recommends). However, I'm not sure how we can scope to threads while doing that. We can scope to method or class easily but we want to reuse playwright and keep it open across test classes.
I don't think users will be affected by this but will verify by running the whole suite except engine kit tests.

@yury-s i believe i have addressed all the comments.  I also verified that the threading issues are not user-facing and should not affect users.  After this PR, I can start looking into how we can store playwright fixtures in the ExtensionContext and scope them to specific threads

will work on test failures this evening

I'm sorry for the delay life got in the way recently.  Hopefully will get to this this weekend

No problem at all, hope everything is alright.

@yury-s was wondering if it's okay to land this feature without tests.  The JUnit Engine Test Kit isn't playing well.  I have confirmed that the issue is not in the feature but how the engine test kit runs the tests.

@yury-s was wondering if it's okay to land this feature without tests. The JUnit Engine Test Kit isn't playing well. I have confirmed that the issue is not in the feature but how the engine test kit runs the tests.

Yes, if it proves too difficult to test it, we can land it without tests. Would be nice to have some description of the problem here, so that later on if we wonder why it's difficult to write such tests we wouldn't need to repeat all the debugging and investigation that you've already done.

looking into failures

@yury-s All tests should be passing now


Decompression Error: No mode was passed in or automatically detected
06 Nov 2021 15:43:58 -00: ERROR   : 'None' extraction of /var/tmp/catalyst/builds/default/stage3-amd64-openrc-latest to /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest failed.
06 Nov 2021 15:43:58 -00: NOTICE  : --- Running action sequence: unpack_snapshot
06 Nov 2021 15:43:58 -00: NOTICE  : Unpacking portage tree (this can take a long time) ...
06 Nov 2021 15:44:02 -00: NOTICE  : --- Running action sequence: setup_confdir
06 Nov 2021 15:44:02 -00: NOTICE  : --- Running action sequence: portage_overlay
06 Nov 2021 15:44:02 -00: NOTICE  : --- Running action sequence: base_dirs
06 Nov 2021 15:44:02 -00: NOTICE  : --- Running action sequence: bind            06 Nov 2021 15:44:03 -00: NOTICE  : --- Running action sequence: chroot_setup    06 Nov 2021 15:44:03 -00: NOTICE  : Setting up chroot...
06 Nov 2021 15:44:03 -00: ERROR   : Exception running action sequence chroot_setup                                                                                Traceback (most recent call last):
  File "/usr/lib/python3.9/shutil.py", line 265, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
IsADirectoryError: [Errno 21] Is a directory: '/var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/etc/'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/catalyst/base/stagebase.py", line 1515, in run
    getattr(self, x)()
  File "/usr/lib/python3.9/site-packages/catalyst/base/stagebase.py", line 1063, in chroot_setup
    shutil.copy('/etc/resolv.conf', self.settings['chroot_path'] + '/etc/')        File "/usr/lib/python3.9/shutil.py", line 426, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)                            File "/usr/lib/python3.9/shutil.py", line 293, in copyfile
    raise FileNotFoundError(f'Directory does not exist: {dst}') from e
FileNotFoundError: Directory does not exist: /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/etc/
06 Nov 2021 15:44:03 -00: NOTICE  : Cleaning up... Running unbind()
06 Nov 2021 15:44:03 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/var/tmp/portage is not a mount point. Skipping

mkdir -p /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/etc/portage

catalyst -f stage1.spec
06 Nov 2021 15:52:28 -00: NOTICE  : Loading configuration file: /etc/catalyst/catalyst.conf
06 Nov 2021 15:52:28 -00: NOTICE  : conf_values[options] = {'bindist', 'kerncache', 'autoresume', 'seedcache', 'pkgcache', 'snapcache'}
06 Nov 2021 15:52:28 -00: NOTICE  : Processing spec file: stage1.spec
06 Nov 2021 15:52:28 -00: NOTICE  : Using target: stage1
06 Nov 2021 15:52:28 -00: NOTICE  : Source file specification matching setting is: strict
06 Nov 2021 15:52:28 -00: NOTICE  : Accepted source file extensions search order: ['tar.xz', 'tpxz', 'xz', 'tar.bz2', 'bz2', 'tbz2', 'squashfs', 'sfs', 'tar.gz', 'gz', 'tar']                                                                     06 Nov 2021 15:52:29 -00: NOTICE  : Source path set to /var/tmp/catalyst/builds/default/stage3-amd64-openrc-latest
06 Nov 2021 15:52:29 -00: NOTICE  : stage1 stage path is /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/tmp/stage1root
06 Nov 2021 15:52:29 -00: NOTICE  : Resume point detected, skipping target path setup operation...
06 Nov 2021 15:52:29 -00: NOTICE  : --- Running action sequence: unpack          06 Nov 2021 15:52:29 -00: NOTICE  : Referenced SEEDCACHE does not appear to be a directory, trying to untar...
06 Nov 2021 15:52:29 -00: NOTICE  : Resume: "chroot" is valid...
06 Nov 2021 15:52:29 -00: NOTICE  : Resume: "seed source" hash matches chroot...
06 Nov 2021 15:52:29 -00: NOTICE  : Resume: Valid resume point detected, skipping seed unpack operation...
06 Nov 2021 15:52:29 -00: NOTICE  : --- Running action sequence: unpack_snapshot 06 Nov 2021 15:52:29 -00: NOTICE  : --- Running action sequence: setup_confdir   06 Nov 2021 15:52:29 -00: NOTICE  : --- Running action sequence: portage_overlay 06 Nov 2021 15:52:29 -00: NOTICE  : --- Running action sequence: base_dirs
06 Nov 2021 15:52:29 -00: NOTICE  : --- Running action sequence: bind
06 Nov 2021 15:52:29 -00: NOTICE  : --- Running action sequence: chroot_setup
06 Nov 2021 15:52:29 -00: NOTICE  : Setting up chroot...                         06 Nov 2021 15:52:29 -00: WARNING : Overriding certain env variables may cause catastrophic failure.
06 Nov 2021 15:52:29 -00: WARNING : If your build fails look here first as the possible problem.
06 Nov 2021 15:52:29 -00: WARNING : Catalyst assumes you know what you are doing when setting these variables.
06 Nov 2021 15:52:29 -00: WARNING : Catalyst Maintainers use VERY minimal envscripts, if used at all.
06 Nov 2021 15:52:29 -00: WARNING : You have been warned.
06 Nov 2021 15:52:29 -00: NOTICE  : Writing the stage make.conf to: /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/etc/portage/make.conf
06 Nov 2021 15:52:29 -00: NOTICE  : STICKY-CONFIG is enabled
06 Nov 2021 15:52:29 -00: NOTICE  : --- Running action sequence: setup_environment
06 Nov 2021 15:52:29 -00: NOTICE  : --- Running action sequence: run_local
copying make.conf to /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/tmp/stage1root/etc/portage
Copying stage1-chroot.sh to /tmp                                                 copying stage1-chroot.sh to /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/tmp
copying chroot-functions.sh to /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/tmp
Ensure the file has the executable bit set
Running stage1-chroot.sh in chroot:
    chroot /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest /tmp/stage1-chroot.sh
chroot: failed to run command './tmp/stage1-chroot.sh': No such file or directory
06 Nov 2021 15:52:29 -00: ERROR   : CatalystError: cmd(['/usr/share/catalyst/targets/stage1/stage1-controller.sh', 'run']) exited 1
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/var/tmp/portage is not a mount point. Skipping
06 Nov 2021 15:52:29 -00: ERROR   : CatalystError: Stage build aborting due to error.
06 Nov 2021 15:52:29 -00: ERROR   : Exception running action sequence run_local
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/catalyst/base/stagebase.py", line 1409, in run_local
    cmd([self.settings['controller_file'], 'run'],
  File "/usr/lib/python3.9/site-packages/catalyst/support.py", line 53, in cmd
    raise CatalystError('cmd(%r) exited %s' % (args, ret),
catalyst.support.CatalystError: cmd(['/usr/share/catalyst/targets/stage1/stage1-controller.sh', 'run']) exited 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/catalyst/base/stagebase.py", line 1515, in run
    getattr(self, x)()
  File "/usr/lib/python3.9/site-packages/catalyst/base/stagebase.py", line 1418, in run_local
    raise CatalystError("Stage build aborting due to error.",
catalyst.support.CatalystError: Stage build aborting due to error.
06 Nov 2021 15:52:29 -00: NOTICE  : Cleaning up... Running unbind()
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/tmp/kerncache is not a mount point. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/var/cache/binpkgs is not a mount point. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/tmp/stage1root/proc is not a mount point. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/run is not a mount point. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/dev/shm does not exist. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/dev/pts does not exist. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/var/tmp/portage is not a mount point. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/var/cache/distfiles is not a mount point. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/var/db/repos/gentoo is not a mount point. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/dev is not a mount point. Skipping
06 Nov 2021 15:52:29 -00: NOTICE  : /var/tmp/catalyst/tmp/default/stage1-amd64-openrc-latest/proc is not a mount point. Skipping


[SPARK-19617][SS]Fix the race condition when starting and stopping a query quickly (branch-2.1)

What changes were proposed in this pull request?
Backport #16947 to branch 2.1.
How was this patch tested?
Jenkins

Test build #73091 has started for PR 16979 at commit 7a0b199.

Test build #73091 has finished for PR 16979 at commit 7a0b199.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73091/
Test FAILed.

Test build #73101 has started for PR 16979 at commit 1d776c3.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73101/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #73117 has started for PR 16979 at commit 1d776c3.

Test build #73117 has finished for PR 16979 at commit 1d776c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73117/
Test PASSed.

Merged build finished. Test PASSed.

LGTM.

Thanks. Merging to 2.1.


Errors when deleting items.

Describe the bug
Some items produce errors whewn deleted.
This is due to previous changes in the database

** Tasks **

[ ] Go through all Entities and check deleting actions.


Immutable type allows Dates to mutate.

[x] Issue: When using the Immutable<T> where T contains some Date, it is possible to call methods like setFullYear() which will mutate the initial date.

Version: 2.0.0
Expected behavior: There should be an error when trying to mutate an Immutable date
Observed behavior: No errors

We're currently using Typescript in JSDoc (with VS Code) since it's not possible to switch completely to Typescript at the moment. I've tried making the state and props of components deeply immutable so I've used 'ts-essentials' DeepReadonly<T> type to do this and it worked quite well.
Below is an example of what I've managed to achieve using .d.ts files:
export type ComponentStateMutable = {
   something: number;
  modifiedAt: Date;
}

export type ComponentState = DeepReadonly<ComponentStateMutable>;

Everything worked well with produce() except when trying to pass an draft object that contained a Date into a function that expected a ComponentStateMutable;
Property '[Symbol.toPrimitive]' is missing in type '{ toString: () => string;
toDateString: () => string; toTimeString: () => string; toLocaleString:
{ (): string; (locales?: string | string[], options?: DateTimeFormatOptions): string; };
toLocaleDateString: { ...; }; ... 37 more ...; toJSON: (key?: any) => string; }'
but required in type 'Date'

I didn't managed to figure out how to fix the problem with Symbol.toPrimitive.
Fortunately I've looked through immer's source code and found Immutable<T>. After switching everything from DeepReadonly to Immutable the errors disappeared but I realised that Dates are not actually immutable (because of the AtomicObject type).
DeepReadonly's implementation seems quite good because it also maintains the inner types on destructuring and I was thinking about somehow extending the IProduce so that when a object of type DeepReadonly<Something> is passed as the initial state, the draft should have a Something type.
Unfortunately my knowledge of Typescript is not very big and with the current situation of using JSDoc and .d.ts it's even harder to find documentation and other people that have the same issue. Any suggestion would be immensely appreciated.

The DeepReadonly type you mentioned only adds the readonly attribute to every property, which is pointless in the case of Date objects since they have no mutable properties to begin with (only methods that mutate their internal data). For this reason, the Immutable type provided by Immer avoids touching the Date type at all.
We could use a "mapped type" to remove any Date methods that perform mutations. This would be a breaking change, but might be worth it. If we did this, you wouldn't be able to pass the "immutable" Date object to any function that expects a normal Date type, which may or may not be a good thing. This Immutable<Date> type would be "leaky" in the sense that any function that expects a Date argument (yet never plans to mutate it) would need to use Immutable<Date> instead if you ever wanted to use that function from inside an Immer producer, for example. Or you would first need to cast the Immutable<Date> to a normal Date type, which would defeat the point.
IMO, the best route is to always clone a Date object before you perform mutations, and avoid using functions that mutate a Date object without first cloning it.

As a compromise, you can still achieve runtime safety by using a function like this:
const mutators = Object.getOwnPropertyNames(Date.prototype).filter(name => name.startsWith('set'))

function frozenDateError() {
  throw Error('Cannot mutate a frozen date. Try cloning it first.')
}

export function freezeDate(date) {
  mutators.forEach(name => {
    date[name] = frozenDateError
  })
  return date
}

The only caveat is you'll need to wrap all of your Date objects with it.

You're right, I've missed the fact that you can still call mutable methods when using DeepReadonly. I think I confused this because there was an error when trying to pass a DeepReadonlyObject<Date> to moment which expected a Date.
Regarding the freezing, I was more interested in having some type of "compile-time" guarantees.
As a side note, I think it would be beneficial to include something like an inner DeepReadonlyObject (maybe called ImmutableObject like the ImmutableTuple one) so that when hovering over the definition of an immutable it could show something like:
(property) MyComponent.state: DeepReadonlyObject<MyComponent.MyComponentStateMutable>

instead of:
(property) MyComponent.state: {
    readonly pages: ReadonlyArray<{
        readonly _id: string;
        readonly something: string;
        readonly type: PageType;
        readonly createdAt?: string;
        readonly modifiedAt?: Date;
        readonly lastVisitedAt?: Date;
        ... 22 more ...;
        readonly where?: string;
    }>;
    ... 24 more ...;
    readonly forceUpdate: boolean;
}

I'm not sure if my last message required a new issue. I could try to work on it tomorrow if it seems like a reasonable problem to other people.

I've managed to temporarily solve my problems by using:
/**
 * @template T
 * @param {import('ts-essentials').DeepReadonly<T>} initial
 * @param {(draft: T) => void} func
 */
const produceDeep = (initial, func) => produce(initial, func);


add documentations for calculations

Add calculations docstrings to documentation

This has been added in tables branch

Still need to add how to contribute but some progress in branch pattern


login prompt login information

Hey @felixmosh
I want to add a prompt that the password is entered incorrectly on the login page, but I have tried multiple methods without success. Do you have any good suggestions?

Did you used the express auth example?

I used the express auth example function and it works normally
But I want to give a friendly prompt when the user enters the wrong account or password

I used the res.send() and res.json() functions and did not get the desired result

It is mot related to bull-board,  we should read on how to pass failure to the view with passport

Read this docs https://www.passportjs.org/docs/downloads/html/
You can redirect with query string, which will render the error.
Or provide custom callback, (check in docs)

I used the method in the document to integrate into the bull-board to respond to the error message but it did not take effect

Can you prepare an example repo?

I still haven't solved the problem

Can you prepare a small repo with your setup?
It will help me to understand the issue.

Review this commit https://github.com/felixmosh/bull-board/commit/e77950706c8f04c1723c81b3876846ddc8a4833a which shows an error message when username / password are invalid.


How to use it?

First of all, thanks timnka for your effort. It looks like very useful. However, I couldnt find any examplo of how to compute the flops. There many functions related.
In my case I just want to compute the flops when coompute de frobenius norm of the la palcian of two exponential matrices.
Thanks in advance.
JohannM.

Scroll down to see the README file.  Click on the links in the README to see examples of flop counting.

I didnt notice that. Thank so much. I'll check it out and comeback then if I need it. Best regards tminka

Hi,
Does it work on MATLAB R2018a with MSVC 2017?


Type of mean is not set correctly

Describe the bug
The return type of mean() is set as builtins.float where in practice it's numpy.float64.
To Reproduce
typing.py
from typing_extensions import reveal_type
import pandas as pd

val = pd.Series([True, False, True]).mean()
reveal_type(val)

mypy typing.py
typing.py:5: note: Revealed type is "builtins.float"
In the interactive terminal:
In [1]: import pandas as pd
In [2]: val = pd.Series([True, False, True]).mean()
In [3]: type(val)
Out[3]: numpy.float64

Please complete the following information:

OS: Debian
OS: 11
python version: 3.9.7
version of type checker: mypy
version of installed pandas-stubs: <IP_ADDRESS>704

Additional context
This becomes a problem when using the round function on float, pd.Series([True, False, True]).mean().round()
For which mypy gives this message: error: "float" has no attribute "round"

Feels more like a pandas bug than a typing bug to me.  I would think that scalar return values should be either float or int, as NumPy tries to do.

It's a reasonable example to allow pd.Series([1.2, 3.4, 5.6]).mean().round() .
So a PR that changes the result of Series.mean() from float to np.float64 would be OK

this has been long standing to convert to python scalars
so -1 on changing

Based on comment from @jreback, I'm going to close this, as we don't document that Series.mean() returns numpy.float64

The quick responses are much appreciated.
The python code is valid and the return type of float is wrong in the stubs.
Reading the comment that Series.mean() is not documented as returning numpy.float64 does seem to be a reason to broaden the bug and update the documentation, not closing the issue.
What am I missing here?


[Bug]: Code from one entry point is injected into all entry points

🐞 Bug Report
I have multiple entry points, and each of them imports a different css file.
Describe the bug
Styles from one entry point's css file is injected into all other entry points.
Is this a regression?
I believe not.
To Reproduce
I've made a little codebase that reproduces the issue:
git clone https://git.sr.ht/~whynothugo/vue-multi-page-style-reproducer
cd vue-multi-page-style-reproducer
npm exec -- vite build

Upon inspecting dist/assets/three-39ee0a8d.js you'll see that it includes the styles from the other entry points -- even though three.js includes no styles at all.
Expected behaviour
Each file should only includes styles that it actually imports.
Media prove
n/a
Your environment

OS: Alpine Linux x86_64
Node version: v18.18.0
Npm version: 10.1.0
Browser name and version: n/a

Additional context
I'm using this plugin to work some complexity around how styles are split into chunks:  https://github.com/vuejs/core/issues/9347

Note the three entry points defined in vite.config.js. They are one.ts, two.ts and three.ts. Only the first two actually import any stylesheets.

Hi, thank you for the report.
It sounds like you want to enable the relativeCSSInjection feature, you can find the documentation in the README, let me know if it solve your problem.
Hope it helps let me know


build: upgrade material to latest version of angular-cli

And remove the unnecessary parts.

Next step is going to add a new Angular2App for e2e.

LGTM

FYI: @jelbourn @robertmesserle @kara


skimage missing

ModuleNotFoundError: No module named 'skimage'
I think that must be part of pip install bia-bob

I'd say no, as bia-bob does not call any function from skimage. This issue is related to #42 . We should provide guidance in the installation instructions

Ideally the list that we provide internally to chatGPT would be auto-generated.

Yes, check what this function does: https://github.com/haesleinhuepf/bia-bob/blob/4bb525d13fe7209d440841b511776bda3aa33231/src/bia_bob/_utilities.py#L150
It filters out libraries which are not installed fron the given list. Also see our discussion in #30 and #28

This is one option:
import pkg_resources

Not that pkg_resources is deprecated. https://setuptools.pypa.io/en/latest/pkg_resources.html

What about this then?
from importlib.metadata import distributions

installed_packages = [(dist.metadata['Name'], dist.version) for dist in distributions()]

It would be a matter of trying...
I would however like to have the option to have a blacklist or a whitelist of packages. For example, code using OpenCV is pretty useless for my work...

What about this then?

Can you also phrase again the goal of the modification you are propsing? I'm not sure what's the issue.

I would like to provide chatGPT dynamically with the context of the currently available libraries.
Maybe I am misunderstanding the current use of Context.libraries...could you elaborate what this is list is supposed to be used for?

understood!


Add CI check back

I removed this CI check: https://github.com/sigp/anchor/blob/unstable/.github/workflows/test-suite.yml#L129
Because of multiple lighthouse dependencies. It was required because I feature gated slog out of the task executor, but other lighthouse crates still use slog.
Once lighthouse transitions from slog, this won't be necessary and we can add this CI check back.

This is added in the mega PR - https://github.com/sigp/anchor/blob/f35765ec08639c2ae9c5be273aa850750412d951/.github/workflows/test-suite.yml#L131


Question: "Reverse" permission check

Hello!
I faced a problem which is trivial at a first glance, but I still didn't find any elegant way of dealing with it.
When creating relations between definitions, is it possible to make "parent" permission depend on "child" permission?
I'll give an example:
I have a site with many sections. Site has admin and moderator roles, where moderator roles are given only to separate sections, and admin role is global.
Here is simplified schema:
  definition site {
    relation site_admin: user

    permission is_site_admin = site_admin
    permission can_view_admin_panel = site_admin
  }

  definition section {
    relation site: site
    relation section_moderator: user

    permission can_view_admin_panel = site->can_view_admin_panel + section_moderator
    permission can_edit_article = site->is_site_admin + section_moderator
  }

From the docs I know that I can inherit permissions from site in the section (to give edit right to admin on any entity, for example). But I don't know if there is a way to inherit permissions backwards?
For example, site has 100 sections and moderator has access to just 1 of them. Now to check if user can enter admin site I'm making 1 check for site + 100 checks (1 per section) for sections, to see if user has can_view_admin_panel permission in at least a one of them.
It seems so temptious to have a way of just checking site->can_view_admin_panel based on "has at least one can_view_admin_panel permission in linked sections"
Is it possible to do this check simple by authzed? Or maybe I designed this schema wrong from the very beginning?
Can you recommend me a way of idiomatic resolution for this kind of problem, please?

Hmm, understood!
I'll try doing it in the way you proposed before reverse relations will be added (if they will eventually).
Thank you very much for fast and clear response, I'm closing the issue :)


Show an indicator when data is being loaded

What is this feature about? (1 sentence)
This will show a loading spinner or some other indicator everytime data is being fetched in the frontend.
Why is it needed? What is the value? For whom do we build it?

This is is connected to the problem "Load times of assets and txs".
Even though this feature won't accelerate loading times, we should make the user aware about when data is being loaded at least.

High-level overview of the feature

There is a loading indicator in the following places:

Transaction list

History + queue
When update the entire list and when loading next pages via scrolling

Assets
When opening a tx review modal, something about the advanced params is loaded.
Resolving ENS names in all address fields
Fetching the owner list when loading a Safe.
Updating Safe settings

Contract version
Safe threshold
Safe owners
Safe modules
Safe nonce

Fetch ABI
Add custom Safe app via URL
Load List of Safe apps
Open Safe app
Load list of set Spending limits

Open questions

Are all areas equally important?
Should we display the last update time?

Should we use skeletons? https://blog.iamsuleiman.com/stop-using-loading-spinner-theres-something-better/

Nice idea, thanks for sharing, I added this to the things to discuss. In e.g. Safe apps we already use this it seems.


Refresh Token and retry old request

Hello, i want to implement refresh token with Moya and RxSwift but i don't solve my issue
but i don't understand, when my response is 401, i want to make another request ( refresh token ) and after that i want to retry my old request but i don't understand how to do that, someone can help me
i implement this code in this issue https://github.com/Moya/Moya/issues/744 but i don't know how and when i make my refresh request before to execut my old request

Hey @mrachid. This is not the easiest thing to do, so I understand your pain.
Could you please share what have you tried? The best would be code that you wrote with comments around places you had problems with. This way we could figure out the easiest way to help you in your specific scenario (configuration, abstraction layers etc.)

MY CONTROLLER
//Get User info action button
@IBAction func requestUser(_ sender: UIButton) {
    getUser().subscribe(onNext: { (user) in
        print(user)
    }, onError: { (error) in
        print("ERREUR MERDE ", error)
    }, onCompleted: {
        print("Completed")
    }) {
        print("Disposed")
    }.addDisposableTo(disposeBag)
}

//Get user with fail token for simulate error request for to do my refresh token request and retry my old request
func getUser() -> Observable<User?>  {
    let providerUser = myRxProvider<UserEndPoints>(plugins: [AccessTokenPlugin(token: "FailToken")])

    //I don't understand why i need to implemente the callback
    providerUser.authenticationBlock = { (_ done: () -> Void) -> Void in
        print("DONE")
        done()
    }

    let response = providerUser.request(.ReadCurrent()).mapObjectOptional(type: User.self)
    return response
}

// AUTHENTIFICATION PROVIDER #744
//I implemente the issue #744 i don't understand the callback first and how and where i need to make my refresh request for after execute my old request
public typealias AuthenticationBlock = (_ done: () -> Void) -> Void

public enum Error: Swift.Error {
    case missingAuthenticationBlock
    case invalidCredentials
}

public class myRxProvider<Target>: RxMoyaProvider<Target> where Target: TargetType {

    private let disposeBag = DisposeBag()
    public var authenticationBlock: AuthenticationBlock?

    public override init(endpointClosure: @escaping EndpointClosure = MoyaProvider.defaultEndpointMapping,
                     requestClosure: @escaping RequestClosure = MoyaProvider.defaultRequestMapping,
                     stubClosure: @escaping StubClosure = MoyaProvider.neverStub,
                     manager: Manager = MoyaProvider<Target>.defaultAlamofireManager(),
                     plugins: [PluginType] = [],
                     trackInflights: Bool = false) {

        super.init(endpointClosure: endpointClosure, requestClosure: requestClosure, stubClosure: stubClosure, manager: manager, plugins: plugins, trackInflights: trackInflights)
    }

    public override func request(_ token: Target) -> Observable<Response> {
        return _request(token)
    }

    private func _request(_ token: Target, isSecondTryAfterAuth: Bool = false) -> Observable<Response> {
        return super.request(token)
            .flatMap { [unowned self] response -> Observable<Response> in

                //check the response status code if error make a refresh token
                if response.statusCode == 401 || response.statusCode == 403 {

                    //Need to make a request for resfresh my token and after i have my response,
                    //change my header authorisation token to my old request and execute old request

                    //Check if me second try fail return error and move to login view
                    if isSecondTryAfterAuth {
                        return Observable<Response>.error(Error.invalidCredentials)
                    }

                    //I don't understand the callback ...
                    guard let authenticationBlock = self.authenticationBlock else {
                        throw Error.missingAuthenticationBlock
                    }

                    //i think is here where my old request is execute
                    return Observable.create { observer in
                        authenticationBlock {
                            self._request(token, isSecondTryAfterAuth: true)
                                .subscribe { event in
                                    switch event {
                                    case .next(let element):
                                        print("test element", element)
                                    case .error(let error):
                                        observer.on(event)
                                    case .completed:
                                        print("test completed")
                                    }
                                }.addDisposableTo(self.disposeBag)
                        }
                        return Disposables.create()
                    }
                } else {
                    return Observable.just(response)
                }
        }
    }

    //Refresh token request
    //We suppose the params is valid
    func refreshTokenAction() -> Observable<Token?> {
        let providerToken = myRxProvider<TokenEndPoints>()
        providerToken.authenticationBlock = { (_ done: () -> Void) -> Void in
            print("TOKEN IS REFRESH")
        }

        let response = providerToken.request(.Update(refreshToken: "Riox8RWVm8lmxxxMaMLF4PMz65MMZp4_oaq5_sjEK8c=")).mapObjectOptional(type: Token.self)
        return response

    }
}

@sunshinejr or someone else, if you have any exemple with refresh token and retry old request after the refresh you can show me thanks you so much

@mrachid this conversation may help #748 (comment)

@AndrewSB Thanks i will go to see asap

I read the issue but i don't understand all, if you have any idea for help me in my code, this will be great!!
i take look to #748 i need more time for understand, i'm just begin with RxSwift And Moya

Well i don't solve my issue @sunshinejr @AndrewSB any idea?

@mrachid  have you solve the issue at last?

@mrachid @asasdasasd any luck with this?

@rlam3  retrywhen operator can handle this

@rlam3  retrywhen operator can handle this

@asasdasasd do you have an example? Thanks!

I manage to fetch and refresh the token using the requestClosure from MoyaProvider.
See this link for code example

@rlam3
I dont know what's your scenario, so I will say my first,
when some network request found that  access token is expired, I should request backend to refresh access token. and if other request found access token is expired at the same time, they should wait, and all request should retry when the new access token come back. the refresh token request should be only invoked once.
here is my solution :
static func request(target: API) -> Observable<Response> {
     return self.provider.request(target).retry(1)
         .observeOn(ConcurrentDispatchQueueScheduler.init(qos: .default))
         .flatMap { (response) -> Observable<Response> in

             if (the access token expired) {
                 throw TokenError.TokenExpired
             } else{
                 return Observable.just(response)
             }

         }.retryWhen({ (error: Observable<TokenError>)  in
             error.flatMap{ error -> Observable<()> in
                 switch error {
                 case .TokenExpired:

                     return RefreshTokenObservable.shareReplay(1).flatMap({ (result) -> Observable<()> in
                         switch result {

                         case .RefreshSuccess:
                             return Observable.just("").asObservable().map{_ in return () }

                         case .RefreshFailure:
                             DispatchQueue.main.asyncAfter(deadline: .now() + 1, execute: {
                                 // here is logout
                             })
                             throw error
                         }

                     })
                 }
             }
         })

I don't know if this is all right for you, but works for me. and I know this is not the best or elegant code, anyone can review?
@pietbrauer @sunshinejr @ashfurrow 🙏

public class myRxMoyaProvider<Target>: RxMoyaProvider<Target> where Target: TargetType {

    private let disposeBag = DisposeBag()
    private var refreshToken = ""
    private var authenticationBlock = { (_ done: () -> Void) -> Void in
        print("Execute refresh and after retry")
        done()
    }

    public override init(endpointClosure: @escaping EndpointClosure = MoyaProvider.defaultEndpointMapping,
                         requestClosure: @escaping RequestClosure = MoyaProvider.defaultRequestMapping,
                         stubClosure: @escaping StubClosure = MoyaProvider.neverStub,
                         manager: Manager = MoyaProvider<Target>.defaultAlamofireManager(),
                         plugins: [PluginType] = [],
                         trackInflights: Bool = false) {

        if let dictionary = Locksmith.loadDataForUserAccount(userAccount: "myKeychain") {
            if let value = dictionary["refresh_token"] {
                refreshToken = value as! String
            }
        }

        super.init(endpointClosure: endpointClosure, requestClosure: requestClosure, stubClosure: stubClosure, manager: manager, plugins: plugins, trackInflights: trackInflights)
    }

    public override func request(_ token: Target) -> Observable<Response> {
        return _request(token)
    }

    private func _request(_ token: Target, isSecondTryAfterAuth: Bool = false) -> Observable<Response> {
        return super.request(token)
            .flatMap { [unowned self] response -> Observable<Response> in
                if response.statusCode == 401 || response.statusCode == 403 {
                    if isSecondTryAfterAuth {
                        try? Locksmith.deleteDataForUserAccount(userAccount: "myKeychain")
                        return Observable<Response>.error(Error.invalidCredentials)
                    }
                    if !self.refreshToken.isEmpty {
                        return Observable.create { observer in
                            self.authenticationBlock {
                                let providerToken = myRxMoyaProvider<TokenEndPoints>()
                                providerToken._request(.Update(refreshToken: self.refreshToken), isSecondTryAfterAuth: false).mapObjectOptional(type: Token.self)
                                    .bind(onNext: { (newToken) in
                                        do {
                                            let param = ["refresh_token": newToken?.refreshToken, "access_token": newToken?.accessToken, "scope": newToken?.scope, "token_type": newToken?.tokenType]
                                            try Locksmith.updateData(data: param, forUserAccount: "myKeychain")
                                        } catch let error {
                                            print("ERROR SAVE TOKEN INTO KEYCHAIN : ", error)
                                        }
                                        self._request(token, isSecondTryAfterAuth: true).subscribe{ event in
                                            observer.on(event)
                                        }
                                    })
                            }
                            return Disposables.create()
                        }
                    } else {
                        return Observable<Response>.error(Error.invalidCredentials)
                    }
                }
                else {
                    return Observable.just(response)
                }
            }.retry(1)
    }

I will hope this code can help you, i don't know if is the best practice with moya and rx to or elegant code but it's work for me :)

Hi there,
My teammates found one more solution to the problem of refreshing session token of Auth0 with RxSwift and Moya
We wrote it using pure RxSwift approach and we return a classic error in case of fail. Hope it will help!

@asasdasasd 请问这块到代码还有吗，万分感激🙏


RC test: Atomic Interactions 1.2.0-rc.1

@KatieWoe , @ariel-phet, @arouinfar, and @kathy-phet - atomic-interactions/1.2.0-rc.1 is ready for RC testing.  While its "parent" sim (states-of-matter) is having both the phet and phet-io brands released, we are only doing the phet brand for this one.
For the most part, this sim is a subset of states-of-matter, but it has a feature where heterogeneous atoms can be chosen, which can't be done in SOM, so it would probably be best to test SOM thoroughly, then focus on this feature that is unique to atomic-interactions for additional testing.
Assigning to @ariel-phet for prioritization.

General RC Test

What to Test

Click every single button.
Test all possible forms of input.

Test all mouse/trackpad inputs.
Test all touchscreen inputs.

If there is sound, make sure it works.
Make sure you can't lose anything.
Play with the sim normally.
Try to break the sim.
Test all query parameters on all platforms. (See QA Book
for a list of query parameters.)
Download HTML on Chrome and iOS.
Make sure the iFrame version of the simulation is working as intended on all platforms.
Make sure the XHTML version of the simulation is working as intended on all platforms.
Complete the test matrix.
Don't forget to make sure the sim works with Legends of Learning.
Test the Game Up harness on at least one platform.
Check this LoL
spreadsheet and notify AR or AM if it not there.
If this is rc.1 please do a memory test.
When making an issue, check to see if it was in a previously published version
Try to include version numbers for browsers
If there is a console available, check for errors and include them in the Problem Description.
As an RC begins and ends, check the sim repo. If there is a maintenance issue, check it and notify developers if
there is a problem.

Focus and Special Instructions
Issues to test:
[ ] - Pinned and Moving strings very small

Link(s)

Simulation
iFrame
XHTML
Test Matrix
Legends of Learning Harness

FAQs for QA Members

There are multiple tests in this issue... Which test should I do first?
Test in order! Test the first thing first, the second thing second, and so on.

How should I format my issue?

We typically assign the developer who opened the issue in the QA repository.

My question isn't in here... What should I do?
You should:

Consult the QA Book.
Google it.
Ask Katie.
Ask a developer.
Google it again.
Cry.

@KatieWoe currently marking this test as low since it is not necessary as part of iO deliverables

I'm going to close this, since I'm about to publish SOM 1.2.0-rc.2, and it would be better to roll a new release of AI as well so that SOM and AI are using as much of the same code as possible.


No autocomplete of Component paths in .Vue files and no go to definition in ES6 imports

VSCode Version:Version 1.10.1 (1.10.1)
OS Version: Sierra ( 10.12.3 (16D32) )

Steps to Reproduce:

clone https://github.com/gkatsanos/gplus (branch : redesign)
there's no autocomplete, can't go to Components with cmd+click.

I tried adding a jsconfig.json with this:
{
    "compilerOptions": {
        "module": "es6"
    }
}

no change

Also, there is no auto completion of exports from a module.

We don't support Vue out of the box. Try using the https://marketplace.visualstudio.com/items?itemName=octref.vetur plugin which should add support for these features

Thanks Matt. VSCode is nice but I decided to just go back to Jetbrains
which has all those out of the box. Too many missing features, needs a bit
more work till it's something it can replace a professional tool. If one
needs a simple cool editor there's always sublime atom or brackets.
On 23 May 2017 02:26, "Matt Bierner"<EMAIL_ADDRESS>wrote:
We don't support Vue out of the box. Try using the https://marketplace.
visualstudio.com/items?itemName=octref.vetur plugin which should add
support for these features
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vscode/issues/21992#issuecomment-303255853,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABr9god7lPVe4qpKHAAXBey0Zk_jlD5hks5r8ifFgaJpZM4MTQmt
.


[0.4] Install terraform from new package

To avoid the download of untrusted packages I have updated the
Dockerfile to use a package provided by alpine testing repository.

@bradrydzewski Maybe you want to merge it? Simplifies the image...

LGTM


Flutter Web has issues!

There is no mention of google ads in flutter web, though its under review. This plugin cause flutter web to crash.
Says this.
Error: MissingPluginException(No implementation found for method MobileAds#updateRequestConfiguration on channel plugins.flutter.io/google_mobile_ads)     at Object.throw_ [as throw] (http://localhost:50001/dart_sdk.js:5041:11)     at platform_channel.MethodChannel.new._invokeMethod (http://localhost:50001/packages/flutter/src/services/system_channels.dart.lib.js:943:21)     at _invokeMethod.next (<anonymous>)     at http://localhost:50001/dart_sdk.js:37403:33     at _RootZone.runUnary (http://localhost:50001/dart_sdk.js:37274:59)

Google Mobile Ads plugin does not support web platform. So that's an intended behavior.