Content addressability

This PR changes the Docker engine to use content addressable storage for images and layers. This means that image IDs are no longer arbitrarily assigned - now they are computed based on the filesystem contents and the image configuration. Images can securely share their underlying layers without duplicating data on disk.
This work involves a big refactoring of image-related code in the engine. The graph package is entirely removed, and is replaced by a few new packages:

layer: Layer store interface and implementation. This manages filesystem layers without being aware of image configurations. The layer store maintains reference counts for layers and removes unreferenced layers.
image: Image store interface and implementation. This manages image configurations. Image configurations include runtime configuration and reference the underlying filesystem data.
tag: The tag functionality from the graph package has been separated out and moved here. Tags now use types from the distribution/reference package instead of plain strings.
distribution: Push and pull code that originally was part of the graph package has been separated out into its own package.

The first time a version of the engine with these commits is started, it will migrate old graph metadata to the new format. This involves calculating content hashes for the existing data, but it does not move underlying graphdriver filesystem data. It doesn’t remove old graph metadata, so the migration process is not destructive.
The new data model does not have a one-to-one relationship between images and layers. A single image can have many layers. Existing versions of Docker create an image for each layer, and use the parent chain to link them together. That means that pulling a specific image requires pulling all the artifacts from the original build process of that image. With this PR, when an image is pulled from a registry, only a single image is created, to match the new data model. The history is preserved through a list of commands, dates, etc.
Summary of UI Changes:

Pull/push do not transfer the entire parent chain.
Full-length image IDs have a sha256: prefix. This prefix is hidden for truncated image IDs for convenience.
Images don't have the concept of VirtualSize (because there’s no special meaning for a top layer anymore).

Future work:

Add support for new manifest format for push/pull: https://github.com/docker/distribution/pull/1068
Move graphdriver initialization/shutdown out of daemon.
Move some layer, image, and distribution code out of docker/docker (remove dependencies on Engine code as necessary).
Additional changes needed for managing the Windows base layer.

Unit test coverage for the new packages:
ok  	github.com/docker/docker/image	0.051s	coverage: 85.8% of statements
ok  	github.com/docker/docker/layer	0.223s	coverage: 76.0% of statements
ok  	github.com/docker/docker/migrate/v1	0.033s	coverage: 71.1% of statements
ok  	github.com/docker/docker/tag	0.017s	coverage: 86.1% of statements

For the full design document, see https://gist.github.com/aaronlehmann/b42a2eaf633fc949f93b

@swernli - Stefan can you review from the Windows side please?

The first time a version of the engine with these commits is started, it will migrate old graph metadata to the new format. This involves calculating content hashes for the existing data, but it does not move underlying graphdriver filesystem data. It doesn’t remove old graph metadata, so the migration process is not destructive.

Curious; will repeated upgrades / downgrades work? (e.g., Used 1.10 for testing, then continue on 1.9 and run 1.10 again)

Curious; will repeated upgrades / downgrades work? (e.g., Used 1.10 for testing, then continue on 1.9 and run 1.10 again)

New images built or pulled in 1.10 wouldn't be visible to 1.9. But the original images would still be there, unless you manually deleted some of them.

Reviewing for Windows...

ping @docker/maintainers please review and test this one. It's a huge change, but really exciting stuff.
also:
please try to avoid merging large PRs while this PR isn't merged yet to avoid rebase hell

\o/
On Thu, Nov 12, 2015 at 12:56 PM, Sebastiaan van Stijn <
<EMAIL_ADDRESS>wrote:

ping @docker/maintainers
https://github.com/orgs/docker/teams/maintainers please review and test
this one. It's a huge change, but really exciting stuff.
also:
please try to avoid merging large PRs while this PR isn't merged yet to
avoid rebase hell
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/17924#issuecomment-156230950.

A difference I see with this one (probably expected, but just to verify)
Dockerfile
FROM ubuntu:14.04
ENV foo=bar
RUN echo hello > /foo

docker build -t foo .

Docker 1.9:
docker history foobar
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
408581b4301b        7 seconds ago       /bin/sh -c echo hello > /foo                    6 B
012645df8321        7 seconds ago       /bin/sh -c #(nop) ENV foo=bar                   0 B
e9ae3c220b23        2 days ago          /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
a6785352b25c        2 days ago          /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.895 kB
0998bf8fb9e9        2 days ago          /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   194.5 kB
0a85502c06c9        2 days ago          /bin/sh -c #(nop) ADD file:531ac3e55db4293b8f   187.7 MB

With this PR:
docker history foobar
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
a12448c314a7        6 minutes ago       /bin/sh -c echo hello > /foo                    2.048 kB
e39f86d96664        6 minutes ago       /bin/sh -c #(nop) ENV foo=bar                   0 B
4b1e42b414f6        2 days ago          /bin/sh -c #(nop) CMD ["/bin/bash"]             1.024 kB
<missing>           2 days ago          /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   4.608 kB
<missing>           2 days ago          /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   208.9 kB
<missing>           2 days ago          /bin/sh -c #(nop) ADD file:531ac3e55db4293b8f   196.8 MB

Differences:

sizes reported are different
images from the parent image are reported as <missing> (I expect that to be by design, but just checking)

@thaJeztah The sizes are currently from tars so they have the added padding. We are exploring ways to revert this if possible. <missing> is shown if the parent image is not locally available. This is because you probably did a fresh pull and got a flat image without parents for ubuntu.

@tonistiigi correct, it was a fresh pull

Currently, these changes break the use of alternate ID files in layers necessary for Windows containers to work.  I'm looking for a fix that can be applied to these changes to unblock us.

Not in doc review but this could have documentation impact that is pretty big depending on the implementation. So, engineering hours could be required in terms of reviewing at the least. I'd like to include this information early in the PR lifecycle.

The image discussion recently added along with the storage driver material: https://github.com/docker/docker/blob/master/docs/userguide/storagedriver/imagesandcontainers.md  --- minor changes I think likely in graphics and in discussion of how images are built. (Not impact necessarily to choosing a storage driver).
Re @thaJeztah's demo of the output of a command --> we have a lot of examples of that show image lists in the docs.  If the format of the output around images changes significantly we should make sure to update all the examples across all the projects --- these are things readers notice.

This is because you probably did a fresh pull and got a flat image without parents for ubuntu.

This part of this proposal seems really strange to me, and I'm trying to understand the rationale for making this change unilaterally for all images that get pushed/pulled -- right now, if I pull two images that have several layers with the same content and parents, they'll nicely share that layer (one of the main advertized advantages of Docker's layered image storage model) and I won't incur the space taken up by that layer twice.  With this change, overlapping layers won't matter anymore, and so if I docker pull golang:1.4 and then docker pull golang:1.5, I get to incur the bandwidth and storage cost of both the debian:jessie they're based on and the layers which are the same up until the point where the two Dockerfiles actually diverge?  https://imagelayers.io/?images=golang:1.4,golang:1.5 illustrates what I mean pretty well -- in this example, this behavior basically this means ~425MB of extra space taken up by the second image on my local system, plus whatever the compressed version of that is in bandwidth and time waiting for the pull.  Have I misunderstood the feature?  Having this as an option sounds really neat, but applying it unilaterally by default with no opt-out feels like walking backwards IMO.

if @tianon is correct about us no longer sharing layers then I'd prefer to not do this, we'd lose one of the biggest benefits that Docker offers the community.  Would it be possible for someone to explain why we can't keep the current layering/sharing but just use SHAs as the IDs instead of the random ones we use today?

Layer sharing is still happening afaict, as evidenced by doing docker pull golang:1.4 && docker pull golang:1.5 && docker pull debian:jessie
The 1.5 pull will show several "already exists" layers, and the debian pull will be all "already exists".

@tianon @duglin All images can share their underlying layers freely. Flat image means that it doesn't need a parent to run, not some kind of isolation of layers. Sharing of layers is even better because we now identify individual layers based on content, not on arbitrary image IDs that they were connected to. So you only get new layers when layer content actually changes. As the title says everything is content addressable - so you don't get any duplication. Also we take out empty layers on build  that would have previously taken you out of sharing. Other changes include having a mapping to compressed blobsums, so we can actually detect layer sharing and skip a download even for different compressed blobs.

Ahh, so we still store them separately, we just don't have a way to reference them individually?  I think that's still the part that's confusing me.  What situations cause layers to be <missing>, and would those layers (and that filesystem space) be reclaimable later since we would then have the metadata for the individual layers?  Are they still usable for build cache, etc?

@tianon Please see the design-doc about image config: https://gist.github.com/aaronlehmann/b42a2eaf633fc949f93b#new-image-config
In the flat model history is just array of text in the config that your builder can produce. There is no strict relationship between a history item and a layer. To become compatible with docker history UI we try to match them when generating that view. If the user has a parent image on disk that we can also match, we will show the parent ID as well.
If you build an image you will get a new image(still flat) for every dockerfile command. This means that if you run docker history on self-built image you should always have the image IDs there until you delete something. Also this allows us to have no modifications for the build cache.

Right, I understand the way we're handling that part -- what I'm fuzzy on is the rootfs section of that doc (https://gist.github.com/aaronlehmann/b42a2eaf633fc949f93b#rootfs).  It sounds (from the design spec) like we do still store each layer's contents separately on-disk, but from the discussion here it's fuzzy whether docker pull golang:1.5 will give me a single rootfs blob, or whether I'll still get one rootfs blob for each layer.  If we're still getting individual layers that are just missing the "image metadata", then this sounds fine.

If we're still getting individual layers that are just missing the "image metadata", then this sounds fine.

Yes, that's the right way to think about it. The layers are shared whenever they have identical filesystem contents. When you pull something, you only get one image configuration instead of one for each layer.

Cool, thanks for confirming! :+1:

@thaJeztah The layer size reporting has been switched back to file content based algorithm. This should fix the size differences in docker history output.

FYI I did an early review before PR was submitted, so count that as half an LGTM from me ;-) Awesome work @tonistiigi @aaronlehmann @dmcgowan :+1:

This seems really nice. I have a few questions/comments:

The design doc says the parent key will be removed from image configuration and it will be "stored out-of-band". What exactly will the 'parent' of an image mean (maybe the id of the image used in FROM)? Where will it be used? It doesn't seem to me that the concept of a 'parent image' has much meaning any more.
It seems unfortunate that there's no way to build on top of a specific sub-chain. Taking https://github.com/docker/docker/pull/17924#issuecomment-156234758 as an example, there appears to be no way to build on top of the first ADD layer.
Related to 2, are you planning to add (or are open to adding) plumbing apis to do useful things like: a) export individual layers, b) import individual layers, c) create an image from an image configuration (list of layers etc). Now that layers are completely divorced from image configuration this seems like a particularly useful thing to do.
IMO you need big warnings in the changelog (and any other publicity) that the daemon will be totally unavailable until migration has happened and this may take a while if you have a lot of large images.
Why does FSMetadataStore create a bunch of files and directories (under image/aufs/distribution, i.e. blobsum-{lookup,storage}/*) rather than just maintaining a single json/sqlite file for lookup (e.g. image/aufs/distribution.json)? It uses a single mutex so race conditions aren't an issue.

The design doc says the parent key will be removed from image configuration and it will be "stored out-of-band". What exactly will the 'parent' of an image mean (maybe the id of the image used in FROM)? Where will it be used? It doesn't seem to me that the concept of a 'parent image' has much meaning any more.

The idea here is to allow an image to stand on its own, instead of requiring all the layers that were part of its build history. If the parent ID was part of the image configuration, the parent information would affect the hash of the configuration and therefore the image's ID.

It seems unfortunate that there's no way to build on top of a specific sub-chain. Taking #17924 (comment) as an example, there appears to be no way to build on top of the first ADD layer.

This is correct in the context of push/pull, which only move around one image configuration. However, when you build something locally, it's still possible to build on top of a specific sub-chain.

Related to 2, are you planning to add (or are open to adding) plumbing apis to do useful things like: a) export individual layers, b) import individual layers, c) create an image from an image configuration (list of layers etc). Now that layers are completely divorced from image configuration this seems like a particularly useful thing to do.

I imagine you're talking about HTTP APIs here, since these things are already available at the Go package level. I'm certainly open to exploring this, but what we had in mind longer-term was splitting the layer/image code out of the docker/docker repo, and building tools on top of it to do things like this.

IMO you need big warnings in the changelog (and any other publicity) that the daemon will be totally unavailable until migration has happened and this may take a while if you have a lot of large images.

Yes, this needs to be made clear. We should also provide rough estimates of how long the migration takes as a function of the amount of layer data in the graph. BTW, since this is only hashing the data on disk, and not actually moving it, the migration goes relatively quickly.

Why does FSMetadataStore create a bunch of files and directories (under image/aufs/distribution, i.e. blobsum-{lookup,storage}/*) rather than just maintaining a single json/sqlite file for lookup (e.g. image/aufs/distribution.json)? It uses a single mutex so race conditions aren't an issue.

My concern about a single JSON file is that it might grow quite large, and making any update to it will involve significant I/O and parsing overhead. sqlite may be a good approach, but I'm not sure whether sqlite use is encouraged (I believe there's a PR changing links to not use sqlite). I see that boltdb is already vendored, so that would be another option.
Since the metadata store is defined as an interface, it should be easy to experiment with other backends.

Thanks for your response.

The idea here is to allow an image to stand on its own, instead of requiring all the layers that were part of its build history. If the parent ID was part of the image configuration, the parent information would affect the hash of the configuration and therefore the image's ID.

I wasn't clear enough. I agree that all of this is good - my question is, why can't you do away with parent image ID completely (i.e. don't even store it 'out-of-band')? Is it just for a nicer output of docker history? I'm trying to figure out what people will now use it for/whether it's useful any more.

This is correct in the context of push/pull, which only move around one image configuration. However, when you build something locally, it's still possible to build on top of a specific sub-chain.

Ok, given https://github.com/docker/docker/pull/17924#issuecomment-156234758, how would you build on top of the subchain consisting of just the /bin/sh -c #(nop) ADD file:531ac3e55db4293b8f layer?

I imagine you're talking about HTTP APIs here, since these things are already available at the Go package level. I'm certainly open to exploring this, but what we had in mind longer-term was splitting the layer/image code out of the docker/docker repo, and building tools on top of it to do things like this.

Wouldn't you need a HTTP api level anyway? Any tool would need root permissions to fiddle around with docker graph directory. Happy to hear that this is planned though.

Yes, this needs to be made clear. We should also provide rough estimates of how long the migration takes as a function of the amount of layer data in the graph. BTW, since this is only hashing the data on disk, and not actually moving it, the migration goes relatively quickly.

It'll also be much slower if you have many many small files, but that's an edge case.
My /var/lib/docker is ~7GB (which is pretty tiny for my standards) and it took a number of mins. It's not too long, but people generally don't expect 5 mins downtime when restarting a service post-upgrade.

My concern about a single JSON file is that it might grow quite large, and making any update to it will involve significant I/O and parsing overhead. sqlite may be a good approach, but I'm not sure whether sqlite use is encouraged (I believe there's a PR changing links to not use sqlite). I see that boltdb is already vendored, so that would be another option.

I don't mind, boltdb sounds good. At present you're just implementing your own a kv-store with the filesystem.

why can't you do away with parent image ID completely (i.e. don't even store it 'out-of-band')?

When you do a build you will still get a separate image for every Dockerfile command. You can start/push/tag any of those images and they all make up a parent chain(view them with docker images -a). The chain is mainly used for builder cache. History also uses the parents to provide mapping if possible, but this probably isn't a high priority use-case.

@tonistiigi needs a rebase :cry: hope it's a simple one

Just adding that I find it really useful while testing and developing
Dockerfiles to be able to jump into a container made from one of the
intermediate instructions, so thanks for making sure that's still included.
:+1: :heart:

I pulled an image from a v1 registry, rmid it and then tried to pull it again. Result:
Pulling repository company.internal:5000/image
8637682a033d: Error pulling image (1.0) from company.internal/image, too many non-empty layers in History section 00/v1/, too many non-empty layers in History section
47d44cb6f252: Pulling metadata
6a7b54515901: Pulling metadata
[...]
Get https://company.internal:5000/v2/: EOF
endpoint does not support v2 API
unable to ping registry endpoint https://company.internal:5000/v0/
v2 ping attempt failed with error: Get https://company.internal:5000/v2/: EOF
 v1 ping attempt failed with error: Get https://company.internal:5000/v1/_ping: EOF
Error pulling image (1.0) from company.internal:5000/image, too many non-empty layers in History section

it may be related to the comment I've just made, not yet sure.

@aidanhs: This was indeed caused by the bad loop conditions. Thank you for pinpointing it. I've updated the commit to fix the problem. This didn't show up in my original testing because the problem only appears if you already have some, but not all, of the layers for a v1 image.

@swernli I managed to get the windows baselayer support working. Tested pull/push/run/build/history/save/load in TP3. Patched tar-split with a crazy hack atm, you can have a better solution with some changes to your storage layer. My suggestion is that you should probably only put the files that never change in the tar and generate other metadata.
Base layer in config is now text based like we discussed, so if you tested my earlier patch that used 64 byte hex you may need to clear your graph.
"rootfs":{
  "type":"layers+base",
  "diff_ids":["sha256:c438aa1198fb8574036eeae2ff3864b3ab06a8b1b6dd707c82a199413ef75e4f",...],
  "base_layer":"CN=Microsoft_WindowsServerCore_10.0.10514.0"
}

Here is the change without squash: https://github.com/docker/docker/commit/d537d85c232e6c2db87ec4aa52597a3762182455

OHMAN

@tonistiigi I was able to pull up the thread I had from last time we encountered problems with tarsplit.  Unfortunately, our conclusion from last time was that tarsplit wasn't really compatible with the way Windows layers are exported and imported, since the content of the layer changes significantly as part of that process.  We can discuss more on IRC, but it may be that we'll need to rely on a hack for a while there...

testing looks good on upgrade and push/pull
LGTM

Made a PR onto the branch - https://github.com/aaronlehmann/docker/pull/2
Additionally, why is func ReleaseAndLog(ls Store, l Layer) { not a method on Store? (I would make this a comment on the actual piece of code, but the diff is too big :( )

I'm still kicking this changeset around, but so far it's looking good to me

I've seen some panic calls in layer/layer_store.go. Are those on purpose? they look very dangerous.

yeah, I saw those panics as well... I traced them and really they should be panics if it got there as it really shouldn't ever... it's still a bit weird and scary.

@calavera These should be error cases that can only happen if there is a internal algorithm bug in layerStore. We could just ignore them (and let code crash in random point). cc @dmcgowan

@tonistiigi @aaronlehmann Are we waiting on merge @aidanhs PRs?

@LK4D4: No need to wait for them. They are cleanups that can easily be submitted as followup PRs.

I would like to repeat the suggestion to use some kind of real db rather than throwing loads of files onto disk, as once it's merged this PR is merged I can't see it being changed (you'll need a second migration process).
If it's agreed that this is desirable then I can look at doing it, but I think there should be a decision one way or another.

I'm fine with the idea of using a database, but I don't think that the current approach is problematic. In general, it creates one or two files per layer, which is not very many files compared to the number of files in the layers themselves. Before this PR, there were similar files in the graph directory, such as checksum.
Also, I think it would be relatively easy to transition to a different storage scheme, because the files in the distribution directory just cache information about pushes and pulls. The worst case from changing the way they are stored is some unnecessary pushes and pulls for people who have run the intermediate version. If we switch to a database, I think the right way to do it is in a followup PR pre-1.10.

Actually it's pretty sad, that migration process is so slow :/ Seems like we can't do anything.

I'm hoping I don't need to worry about lines like
ERRO[0012] migration failed for<PHONE_NUMBER>e7c9439478f0c762993fdf044d0921b4d210acb1bd6ae6f36ab6e0, err: open /var/lib/docker/aufs/diff/5630927819e7c9439478f0c762993fdf04
4d0921b4d210acb1bd6ae6f36ab6e0/etc/group: no such file or directory
ERRO[0012] migration failed for ee6677401abbf3578319ed0268d6cbcfd57a312289a16868b047e835f4fe55d4, err: open /var/lib/docker/aufs/diff/5630927819e7c9439478f0c762993fdf04
4d0921b4d210acb1bd6ae6f36ab6e0/etc/group: no such file or directory

?

@aidanhs can you send me more info about that case, the /var/lib/docker/graph/5630927819e7c9439478f0c762993fdf044d0921b4d210acb1bd6ae6f36ab6e0/tar-data.json.gz file and ls -l /var/lib/docker/aufs/diff/5630927819e7c9439478f0c762993fdf044d0921b4d210acb1bd6ae6f36ab6e0/etc/group. Most likely something your tar-split file didn't have required data(that meant that you should get same error if you pushed that image with old version). If this is the case we can generate new tar-split file on error, but we should also figure out what causes these errors(one of them is #15643).

@tonistiigi sorry, I unfortunately can't share those files because I nuked my docker data directory about 5 mins after the error and now can't reproduct. It had an accumulation of junk from other dev branches and compiling on master and docker versions back a fair way. I'll raise an issue if I see it happen again after this PR is merged.

Migration is really slow.  Worried that systemd will kill daemons in the middle of migration because it doesn't appear started yet, as it has a 10s timeout.
Starting my daemon with a handful of small images (debian:jessie, redis, ubuntu, busybox) took 12s.
Wonder if we can parallelize some of that work.

@cpuguy83 was 5 minutes for me :)

@cpuguy83: I've addressed your latest comments in three new commits. These will be squashed before merge.

There's a number of TODO and FIXME comments. I'm not sure if these are meant to be resolved before we're ready to merge or not.

I have a feeling migration is going to cause a lot of problems, particularly on hosts with large images (and as such longer migration times).
I'm assuming the majority of time during migration is spent summing layers?
I wonder if we can refer to these by v1 ID's internally and kick off a job to a worker queue to get the sums. While sums are not populated the image can't be pushed/exported or used by content addressable ID.
This should also be an improvement for commit we don't really need the checksum until it needs to get pushed/exported (or do we???).

@cpuguy83 we were actually discussing some option last week, e.g. calculate the checksums in a queue, or even using a separate migration tool, so that the migration can be performed before upgrading docker. General opinion was that a solution has to be found before the final 1.10 release, but possibly as a separate PR

My opinion, that it should be tried in this PR. Because what if it'll be very hard or impossible to do? Revert will be not so easy.

There's a number of TODO and FIXME comments. I'm not sure if these are meant to be resolved before we're ready to merge or not.

We aren't planning to address any of these before the merge. Many are just from moved code.

For the download needs we can create a migration tool to perform the migration live on a running daemon before upgrade.
@tonistiigi is working on making the migration work concurrent and there is around a 25% performance boost but for others I think the migration tool will be the best thing to do.
This can be merged as is and we can have smaller prs for optimizations and a migration tool.  Its not like @tonistiigi and @aaronlehmann are going to stop working on this after this pr is merged.

LGTM.
I agree with @crosbymichael that we should not try to land all the migration work in this PR. Otherwise we'll end up merging this very close to the release with very little time to more exhaustive testing in real environments.

@tonistiigi said that migration-tool is not an option. I want to hear plan for migration stuff at least and not "okay, all need to have 25 minutes downtime without user-facing benefits" at release time.

I'll have to side with @LK4D4 on this. The patch isn't ready if we don't have a safe way to migrate.

@LK4D4 Sorry, some miscommunication here. Migration-tool is definitely an option and work with parallel migration is already designed with that possibility in mind.

@tonistiigi cool :) then we have enough lgtms and you can merge PR :P

do we need docs changes as part of this PR, or will we address those in a follow-up? We discussed that there's some change in behavior, and we may need to address those in the docs

there should not be any change of behavior, this is all internal refactoring and there is no way we are going through a docs review in this pr

There are some docs changes needed(mostly for clarity, and some command outputs). They are already being worked on https://github.com/tonistiigi/docker/commit/fc5ba5e3832f748b34c69255cc0d9be69c2e719 and will be posted/reviewed with another PR.

Including a notice/FYI that anyone daring enough to crawl /var/lib/docker as an API, will likely be broken by this.
I have only been able to go through behavioral reviews of this PR, not nity grity code review. It still LGTM.

Thank you all for your feedback. I'm merging it :metal:

Does this change have any effects on the build cache? So can the build cache reuse pulled images now as well? Or would that be a separate issue? Could you elaborate on that @tonistiigi?

@bracki Build cache should work as expected. Cache is for remembering the data from previous builds. It isn't based on checksums but a comparison of image configurations.

I imagine you're talking about HTTP APIs here, since these things are already available at the Go package level. I'm certainly open to exploring this, but what we had in mind longer-term was splitting the layer/image code out of the docker/docker repo, and building tools on top of it to do things like this.

Could you tell me what's the name of the package? Thank you.

I imagine you're talking about HTTP APIs here, since these things are already available at the Go package level. I'm certainly open to exploring this, but what we had in mind longer-term was splitting the layer/image code out of the docker/docker repo, and building tools on top of it to do things like this.

@aaronlehmann  Could you tell me what's the name of the package? Thank you.

@dbdd4us: The packages I was talking about are called image and layer. They are in the top-level directory of this repository.

The packages I was talking about are called image and layer. They are in the top-level directory of this repository.
@aaronlehmann Thanks.

@tonistiigi Hello, Is there any tools that I can used to calculate the content-hash of a tar file?
I have tried tar -xOf layer.tar | sha256sum, but the hash value I got is different from the hash value that calculated from docker daemon.
Or is there anything special in the go tar-split package?

@dbdd4us The content hash of a layer is the hash of the compressed content, as sent to the registry.

@stevvooe layer hashes in image config are the hashes of uncompressed content.
@dbdd4us I'm not sure what the layer.tar is in your example. Checksum is for all the bytes in tar, there is no extraction step involved. Easiest to get those tars is with docker save. Note that these do not match the image IDs that are checksums of image config + layer checksums. Make sure to also read the design doc in first post.


[backport] [FLINK-6183]/[FLINK-6184] Prevent some NPE and unclosed metric groups

Backport of #3610 for 1.2 .

I think we can merge this


Fix/pythran config nopython mode

Supersedes #944

@diorcety : please tell me if you're ok with the change!

:+1:


dh group generation

added dh group generation, secured nginx agains POODLE attacks and whitelisted ciphers
this hardens nginx against this: https://weakdh.org/sysadmin.html
And against POODLE attacks

@bpieck Thank you for starting ciphers for nginx and even putting dh group generation on top! This is an awesome addition :+1:

(The travis lint issues will be fixed separately; This MR was tested in kitchen on all OS)


Better template-coq printers

This PR improves the pretty-printers in MetaCoq.Template.Pretty : they now handle whitespace and indentation much better (i.e. the output adapts to the screen width), the output syntax is much closer to actual Coq syntax, and pretty-printing is now configurable using a set of options.
This does bring in a dependency on the external library coq-pprint, which is a general-purpose pretty-printing combinator library developed and maintained by myself.

Can you provide a pointer to a file where the improvement becomes evident? Alternatively, maybe add something to the test-suite which prints a couple of things?

There is no such file : the template-coq printers are used nowhere in metacoq at the moment. I am personally making great use of them in my own branches, which will not be integrated into metacoq before a couple weeks.

Yes I can add something to the test-suite that quotes some medium-sized terms and prints them. Are you asking this so that people have a simple example of how to use the new printers ? The test-suite might not be the best place for this :)

I think this would fit perfectly in the examples directory. And yes, I'm asking so people see how to use them, but also to have a place to check whether things broke when we change e.g. the printers in the future


How are we handling wide screenshots?

Do we need a width max? Or the ability to click on them and have them expand? Or a wider embed?

I thought about potentially implementing a fullscreen/maximize button next to the close × which would resize the entire modal to fit the window. This, coupled with Medium-style image styles which make grow images wider than the content column might be enough for what you’re trying to achieve here.
That being said, we want this to be responsive and be usable on iPads or other small screens, so we should probably strive for screenshots which are cropped well or use magnification within them to show the relevant details.

Also where did you find that example?

Wait never mind I see it’s in the copy branch.

Just to provide one example, the following image from https://github.com/EagerIO/EmbedBox/pull/4/commits/9bdefa368e6908143055186a4bdf51d147e89c85:

Could easily be cropped to:

Additionally we could show images wider within the content space.

e.g. Adding the following CSS:
.markdown figure {
  margin: 1.5em -2em 2em -4em
}

Would yield this transformation:

Implemented full width figure images as described above. Thoughts?

Seems fine to me, especially as a trial. If we do keep it as the “final” v1 design though we should probably eventually implement it as pulling in the margins on non-figure elements rather than using negative margins on the figures.


Feature/field group

Introduces the feature described in #6.

Please see
I think the current way of data structure with grouped fields is not efficient. It makes iteration through fields quite expensive and not versatile at all.
Expected behavior

Iterating through fields should be as easy as fields.map(field => ...)
Fields presence in the fieldGroup is really important only when serializing the form. This is the whole purpose of introduction of fieldGroup - to have data grouping, which would make form submission more flexible
Form.fields is one-level deep to make it simple and efficient to iterate

Steps to perform

Leave expensive recursion only for fieldUtils.serializeFields() method
Re-think the way grouped fields are stored in Form.state

Technical note
<Field.Group name="groupA">
  <Field.Input name="username" />
</Field.Group>
<Field.Group name="groupB">
  <Field.Input name="username" />
</Field.Group>

May be presented in the state as:
fields {
  'groupA.username': { ...fieldProps },
  'groupB.username': { ...fieldProps }
}

This way state updates would still use a fieldPath composition, but as a string instead of an array. Fields with the same names but in different groups would not conflict with each other, since their props are stored in the different keys (those have fieldGroup name prefix.

Update
The mentioned performance improvement was implemented under https://github.com/kettanaito/react-conform/pull/23/commits/09f6a3579ad3f6183a7ad476bb41d3fd59b696aa.
Changes

Grouped fields now persist as fieldGroup.fieldName in the Form.state and context
Form.state has only one level of depth
It's possible now to iterate through the fields simply by Form.state.fields.map(field => ...), no has-group/doesn't-have-group logic needed
Expensive recursion was removed even from the fieldUtils.serializeFields() method. By using Immutable.Map it's easy to set nested properties in the serialized fields objects by simply map.setIn(fieldProps.get('fieldPath').split('.'), value)
This change means that it is not allowed to have nested <Field.Group nodes. We may add this support in the future


07_Inception_Model.ipynb give different results ?

I went through this notebook with given introduction. But when I use classify(image_path) function in the notebook it doesn't give correct predictions all the time.
If I test it on the given panda image it will first predict correct labels. But if I run in again in will give results like  website, web .. etc

I have just now tested this, and it works fine on my system which is Ubuntu 16.04, Python 3.6, TensorFlow 1.0. It gives the exact same outputs each time I call the function.
How do you expect me to help, when you don't give details about the system and software versions you are using, and the output / error messages you are seeing?

I am sorry. My tensorflow version is 1.01. Ubuntu version is 16.04 and python version is 3.5.
When I run the notebook for several times in each instance it gives different predicted results. No error messages.
This is the original result in the tutorial

This is what I got as correct results

This is what I got after running the same cell which classify panda again

OK, that was a better error-report.
Unfortunately, I don't know what is wrong as it works fine for me. You will have to debug to find out what the problem is. It will be a good exercise for you to learn TensorFlow better. A wild guess might be that you have too little RAM, although that should probably give an error somewhere. You could try running it on a friend's PC and see if it works there.
If you find the solution then please report it here so it may help others in the future.

Well no errors at all. Thanks a lot for the support. I will surely update if I found a solution. Well I think you are correct. I tried this with tensorflow tutorial on CNN. It happened the same. Different probabilities and answers.

i pretrained the inception model in tutorial #09 on my dataset but i want to classify my images like the one in tutorial#07

classify(myimage)

@shamanez It is strange that the error happens when you rerun the entire Notebook and apparently gets worse when you just rerun a part of the Notebook. Try and download the Notebook and this entire repo again, perhaps it has been corrupted. Anyway, I can't help you so I'm closing this topic, but please leave a comment if you find the solution so it may help others in the future.
@manselag123 Your question is off-topic for this thread and I cannot help you with a question like this. You will have to do it yourself or ask on StackOverflow.

@Hvass-Labs  Well it's not a problem with repo or the notebook. I tried the same with the tensorflow tutorial on image classification with tensorflow. (https://github.com/tensorflow/models)
It gave me the same errors. I think it's something to do with tf version I am using and the ram. Thanks a lot for responses.


DocsArchive-<IP_ADDRESS>Auto archive content

Auto archive content from https://github.com/OPS-E2E-PPE/docs-archive-test-source

Docs Build status updates of commit 4dbf002:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error]

Cannot sync git repo to specified commit because branch Archive_master_2019-11-04-06-55-26 has been deleted or has been force pushed: fatal: Couldn't find remote ref Archive_master_2019-11-04-06-55-26

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Inquiry Regarding Temporal Coherence in Extended Video Generation

Dear Vlogger Development Team,
I hope this message finds you well. I am reaching out to express my commendations on the remarkable strides you have made with the Vlogger system in generating minute-level video blogs. The innovative approach of integrating a Large Language Model as a director, alongside foundation models as vlog professionals, is truly impressive.
However, I have a query pertaining to the temporal coherence in the extended video generation process. As you are aware, maintaining continuity and a seamless narrative over longer durations can be quite challenging, especially when transitioning between diverse scenes in a vlog.
Could you kindly elaborate on the mechanisms that Vlogger employs to ensure temporal coherence and narrative consistency throughout the entirety of a minute-level vlog? Additionally, are there any specific strategies in place to handle potential discrepancies that might arise during scene transitions?
I am particularly interested in understanding how the system manages the storyline's continuity, considering the complexity of human-like planning and execution that is required for extended video content.
Thank you for your time and consideration. I eagerly await your response, as I believe it would greatly benefit the academic community and industry practitioners alike.
Best regards,
yihong1120

Thank you for your attention to our work.
Regarding your questions, I'll summarize them: How do we ensure the consistency between multiple video snippets generated by Vlogger when creating long videos?
Firstly, for a complete story, it should logically be coherent so that we have a reason to expect the generated video content to be consistent.
Then, for such a story, in the planning phase, we ask LLM to preserve the plot of the original story as much as possible, without changing the timeline and development path. This way, we approximately assume that the plot in the obtained script is coherent and acceptable.
Next is how to ensure that the vlog generated from the script is visually consistent. We use reference images to some extent to address this issue. If users can see the same or similar main characters or scenes, they often perceive it as a video shot for the same story. However, this method currently has limitations, as it may not consider very fine-grained consistency since reference images provide more of a global reference.
In fact, we initially considered using a transition model to generate transition videos between videos that should have coherence, such as SEINE. However, the results did not meet our expectations entirely, so we have not included this part in our open-source code for now. I believe this could be an interesting direction to explore further.
I'm glad someone has thought of this. I hope the information above is helpful to you.
Shaobin Zhuang
------------------ 原始邮件 ------------------
发件人:                                                                                "zhuangshaobin/Vlogger"                                                                                @.>;
发送时间: 2024年1月22日(星期一) 下午2:51
@.>;
@.***>;
主题: [zhuangshaobin/Vlogger] Inquiry Regarding Temporal Coherence in Extended Video Generation (Issue #4)
Dear Vlogger Development Team,
I hope this message finds you well. I am reaching out to express my commendations on the remarkable strides you have made with the Vlogger system in generating minute-level video blogs. The innovative approach of integrating a Large Language Model as a director, alongside foundation models as vlog professionals, is truly impressive.
However, I have a query pertaining to the temporal coherence in the extended video generation process. As you are aware, maintaining continuity and a seamless narrative over longer durations can be quite challenging, especially when transitioning between diverse scenes in a vlog.
Could you kindly elaborate on the mechanisms that Vlogger employs to ensure temporal coherence and narrative consistency throughout the entirety of a minute-level vlog? Additionally, are there any specific strategies in place to handle potential discrepancies that might arise during scene transitions?
I am particularly interested in understanding how the system manages the storyline's continuity, considering the complexity of human-like planning and execution that is required for extended video content.
Thank you for your time and consideration. I eagerly await your response, as I believe it would greatly benefit the academic community and industry practitioners alike.
Best regards,
yihong1120
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


docs: fix typos

H in Hierarchical is not silent, so an is incorrect (a higher-archical rather than an eye-archical).
Missing we between Suppose & configured.

Hmm. I wouldn't have bothered for the first change because an hierarchy is legit.
But thank you for the second; therefore I'll take them both ;-)


byte where error occured

I need this, and will probably add it if you do not.

Hi Shawn,
Thanks for your interest in this code. Locating where error occurs is a useful feature. I have the plan, maybe in this month, cannot guarantee.
You are welcomed to implement it (and even greater if you can contribute your code :)
Yibo

Commit https://github.com/cyb70289/utf8/commit/1b52b9699947674be4d0e997baba0928ddd6f780
added returning index of first error char to naive.c and range-sse.c.
define RET_ERR_IDX to 1 to enable it. See https://github.com/cyb70289/utf8/blob/master/range-sse.c#L81.


Add codec config

What was changed
Add codec endpoint support in config
Why?
For remote data converter

do you want to expose this setting to the UI too? in this case it would go here so the UI can consume https://github.com/temporalio/ui-server/blob/9561fc064d011423f1e9d9ba2630e06754cbe950/server/routes/api.go#L129


Current master fails to link in plugin directory

Getting a link error for parallel I/O builds:
libtool: link: mpicc -shared  -fPIC -DPIC  .libs/H5Znoop.o .libs/H5Zutil.o   -lhdf5_hl -lhdf5 -lm -ldl -lz /usr/lib/x86_64-linux-gnu/libcurl.so  -g   -pthread -Wl,-soname -Wl,libnoop.so -o .libs/libnoop.so
libtool: link: mpicc -shared  -fPIC -DPIC  .libs/H5Znoop1.o .libs/H5Zutil.o   -lhdf5_hl -lhdf5 -lm -ldl -lz /usr/lib/x86_64-linux-gnu/libcurl.so  -g   -pthread -Wl,-soname -Wl,libnoop1.so -o .libs/libnoop1.so
libtool: link: mpicc -shared  -fPIC -DPIC  .libs/H5Zmisc.o .libs/H5Zutil.o   -lhdf5_hl -lhdf5 -lm -ldl -lz /usr/lib/x86_64-linux-gnu/libcurl.so  -g   -pthread -Wl,-soname -Wl,libmisc.so -o .libs/libmisc.so
/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:571: libnoop.la] Error 1
make[3]: *** Waiting for unfinished jobs....
/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:574: libnoop1.la] Error 1
/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:568: libmisc.la] Error 1
make[3]: Leaving directory '/home/ed/netcdf-c/plugins'
make[2]: *** [Makefile:483: all] Error 2
make[2]: Leaving directory '/home/ed/netcdf-c/plugins'
make[1]: *** [Makefile:707: all-recursive] Error 1
make[1]: Leaving directory '/home/ed/netcdf-c'
make: *** [Makefile:552: all] Error 2

The ,problem is in plugins/Makefile.am:
# Put Together AM_CPPFLAGS and AM_LDFLAGS
include $(top_srcdir)/lib_flags.am

LDFLAGS = -module -avoid-version -shared -export-dynamic -no-undefined -rpath ${abs_builddir}

Instead of setting LDFLAGS, we must set AM_LDFLAGS in the Makefile.am itself. The same is true of all the special *FLAGS variables (See liblib/Makefile.am for an example.)
I will have a PR up with a fix shortly.
What's disturbing is that this was not caught by CI. @WardF I thought you had started parallel I/O tests in Travis - was I mistaken?

Thanks for catching this. I remember that I found the set of flags for the plugins
off the net, but did not consider LDFLAGS vs AM_LDFLAGS. Is there a description
anywhere of why AM_LDFLAGS is preferred?

https://www.gnu.org/software/automake/manual/html_node/Program-Variables.html

The reason it fails is because you are overwritting the existing value. So if it was:
LDFLAGS += whatever
That would probably have worked. But AM_LDFLAGS is provided for this purpose, and then libtool etc. know how to handle things.

I see. I need to look around and make sure I did not do that elsewhere.

Well I think if you did I would have just found it. ;-)


Is it possible route UDP to Serial

I am using ESP8266 AP station on the drone to receive telemetry data.
I want to receive telemetry data on rPi and route it to a serial device so I can plug in this http://tengriuav.com/ unit. So it can communicate with the fc.
I tried using
mavlink-routerd -e /dev/ttyS0:921600 <IP_ADDRESS>:14550
but ended up with the error
Could not open conf file '/etc/mavlink-router/main.conf' (No such file or directory)
Error binding socket (Cannot assign requested address)
Could not open <IP_ADDRESS>:14550
Key Info
ESP8266 is the AP station. It output the mavlink2 data to <IP_ADDRESS>:14550
I can receive it directly to mission planner/gcs without any problem
I failed to route the UDP endpoint back to serial. Any help is welcome

Forwarding from UDP to serial works just fine at our setups.
The error having no config file should be no issue even though I think using the config file is easier to read than the command line arguments. You have to use an UDP endpoint in Eavesdropping mode then. That mode will listen for incoming UDP messages on a defined port.
The second message looks like you tried to bind to an IP which is not assigned to any of the network interfaces. Have you checked, that the device running mavlink-router has the IP <IP_ADDRESS>? But you could also just listen on all interfaces with <IP_ADDRESS> when the device does not have a static IP.

I'll close this because of inactivity for now. Just comment, if you have any updates.

I'll close this because of inactivity for now. Just comment, if you have any updates.


Runtime error in Node.FS.Sync.stat (and likely more in that module)

I was trying to use purescript-benchotron to compare some performance before & after optimization, and I ended up hitting a runtime error. Here's a minimal reproduction:
module Main where

import Prelude

import Effect (Effect)
import Effect.Console (log)
import Node.FS.Stats (isFile)
import Node.FS.Sync (stat)

main :: Effect Unit
main = do
  stats <- stat "spago.dhall"
  if isFile stats then
    log "File."
  else
    log "Not a file."

The error is:
  return s[m]();
             ^

TypeError: s[m] is not a function

Inspecting the generated code, I see
// output-es/Node.FS.Stats/foreign.js
function statsMethod(m, s) {
  return s[m]();
}

which looks fine - however, I also see this suspect line:
  const a$p = (v) => statSync("spago.dhall")();

Indeed node-fs does some unsafeCoerce to get things into effects: https://github.com/purescript-node/purescript-node-fs/blob/v8.1.0/src/Node/FS/Internal.purs#L9-L10
mkEffect :: forall a. (Unit -> a) -> Effect a
mkEffect = unsafeCoerce

I suspect that node-fs should be updated to not do that, but it made me wonder if something in the purs-backend-es printer was off. What do you think?

// output-es/Effect.Console/foreign.js
var log = function(s) {
  return function() {
    console.log(s);
  };
};

// output-es/Foreign/foreign.js
var isArray = Array.isArray || function(value) {
  return Object.prototype.toString.call(value) === "[object Array]";
};

// output-es/Node.FS.Stats/foreign.js
function statsMethod(m, s) {
  return s[m]();
}

// output-es/Data.Bounded/foreign.js
var topChar = String.fromCharCode(65535);
var bottomChar = String.fromCharCode(0);
var topNumber = Number.POSITIVE_INFINITY;
var bottomNumber = Number.NEGATIVE_INFINITY;

// output-es/Data.String.CodePoints/foreign.js
var hasArrayFrom = typeof Array.from === "function";
var hasStringIterator = typeof Symbol !== "undefined" && Symbol != null && typeof Symbol.iterator !== "undefined" && typeof String.prototype[Symbol.iterator] === "function";
var hasFromCodePoint = typeof String.prototype.fromCodePoint === "function";
var hasCodePointAt = typeof String.prototype.codePointAt === "function";

// output-es/Node.FS.Sync/foreign.js
import {
  renameSync,
  truncateSync,
  chownSync,
  chmodSync,
  statSync,
  lstatSync,
  linkSync,
  symlinkSync,
  readlinkSync,
  realpathSync,
  unlinkSync,
  rmdirSync,
  rmSync,
  mkdirSync,
  readdirSync,
  utimesSync,
  readFileSync,
  writeFileSync,
  appendFileSync,
  existsSync,
  openSync,
  readSync,
  writeSync,
  fsyncSync,
  closeSync
} from "fs";

// output-es/Main/index.js
var main = () => {
  const a$p = (v) => statSync("spago.dhall")();
  if (statsMethod("isFile", a$p)) {
    return log("File.")();
  }
  return log("Not file.")();
};

// <stdin>
main();

Ah, I see. The associativity of
(v) => statSync("spago.dhall")();

Is intended to be
((v) => statSync("spago.dhall"))();

The reason I don't handle it is that there is no language semantics that defines a PureScript function as being an Effect. Effect is necessarily foreign, and Effect binding on a PureScript function does not make sense semantically. My opinion is that node-fs is just wrong, and probably was written before EffectFn was introduced. EffectFn solves the use case of JavaScript function calling convention. Alternatively, mkEffect could be foreign instead of unsafeCoerce.

That makes sense - thanks for confirming.
I'll go ahead and close this & move the conversation over there.

Note, I think it's the case that codegen should probably still be fixed, because at a syntactic level it's an issue of incorrect precedence, and I think that will be fixed by #9.


No support for alpha numeric input. Keyboard type is always number values only.

By default keyboard type is 'Phone-pad' (numbers) only. But, for alpha-numeric (letters+numbers) OTP  not able to use the react-native-otp-input.

I will implement this later today :)

@NazifaNajish01 Published in v0.7.0

Thanks, Damian Sznajder! It's really helpful. ☺
Appreciate your efforts. 👏
Regards,
Nazifa Najish
On Sun, Sep 2, 2018 at 2:33 AM, Damian Sznajder<EMAIL_ADDRESS>wrote:

@NazifaNajish01 https://github.com/NazifaNajish01 Published in v0.7.0
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dsznajder/react-native-otp-inputs/issues/4#issuecomment-417886856,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AlKiUDWzpKCNCK8Up-BAE7V_EOu5-eBbks5uWvY8gaJpZM4WTIC3
.


This is a pretty big merge

Lots of changes covered here.  The beginnings of support to move VLAN information OUT of the manufacturers block.  The reason to do that is so that we can automatically allocate VLANs.
Also, now support for same-manufacturer and manufacturer should work.  There were several things wrong: there was an error in a dictionary name, and that code wasn't actually getting invoked.  Now it is.

This addresses Issue #12

Additional commits address Brian's comments.


chore: Pin Ralph image version

This version has the fix for the startup hang when ClickHouse is down, but we should be managing versions explicitly now that the ClickHouse implementation is stable.

@Ian2012 sure, let me know how this looks. Not sure if it will impact any of your k8s work, though.


Fixed typos in challenges description.

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Closes #47738

This is my PR for hacktoberfest2022.

Hello there,
Please edit this PR title to match our recommended title format: https://contribute.freecodecamp.org/#/how-to-open-a-pull-request

Hello there,
Please edit this PR title to match our recommended title format: https://contribute.freecodecamp.org/#/how-to-open-a-pull-request

Perfect!
Give me a moment.

I have updated the required changes.

did you make the requested changes? it's useful if you comment to say you did :)

Closing in favor of #47759


Wheel build fails due to circular dependency with dimod

Building dwave-preprocessing from source (via PEP-517 build system), requires dimod~=0.10.0, which, in turn, in versions >=0.10.5 requires dwave-preprocessing>=0.3,<0.4. As a result, pip wheel fails to install the (yet unreleased) version of dwave-preprocessing, and falls back to installing the latest for which it can find a wheel:
See for example this build of 0.3.1:
Created temporary directory: /tmp/pip-ephem-wheel-cache-e6b6j2px
Created temporary directory: /tmp/pip-req-tracker-ecez82nl
Initialized build tracking at /tmp/pip-req-tracker-ecez82nl
Created build tracker: /tmp/pip-req-tracker-ecez82nl
Entered build tracker: /tmp/pip-req-tracker-ecez82nl
Created temporary directory: /tmp/pip-wheel-lh7m08dq
Processing /dwave/preprocessing
  Created temporary directory: /tmp/pip-req-build-h47germ4
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  Added file:///dwave/preprocessing to build tracker '/tmp/pip-req-tracker-ecez82nl'
  Created temporary directory: /tmp/pip-build-env-xmkbzxsj
  Created temporary directory: /tmp/pip-standalone-pip-9ifxkzl3
  Running command /opt/_internal/cpython-3.9.7/bin/python3.9 /tmp/pip-standalone-pip-9ifxkzl3/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xmkbzxsj/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=46.4.0' 'wheel>=0.30.0' 'Cython>=0.29.21,<3.0' numpy==1.19.4 'dimod>=0.10.0.dev9,<0.11.0'
  Using pip 21.2.4 from /tmp/pip-standalone-pip-9ifxkzl3/__env_pip__.zip/pip (python 3.9)
  Collecting setuptools>=46.4.0
    Using cached setuptools-58.2.0-py3-none-any.whl (946 kB)
  Collecting wheel>=0.30.0
    Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
  Collecting Cython<3.0,>=0.29.21
    Using cached Cython-0.29.24-cp39-cp39-manylinux1_x86_64.whl (1.9 MB)
  Collecting numpy==1.19.4
    Using cached numpy-1.19.4-cp39-cp39-manylinux1_x86_64.whl (13.4 MB)
  Collecting dimod<0.11.0,>=0.10.0.dev9
    Using cached dimod-0.10.7-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (10.6 MB)
  Collecting pyparsing<3.0.0,>=2.4.7
    Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
  Collecting dwave-preprocessing<0.4,>=0.3
    Using cached dwave-preprocessing-0.3.1.tar.gz (159 kB)
    Installing build dependencies: started
    Running command /opt/_internal/cpython-3.9.7/bin/python3.9 /tmp/pip-standalone-pip-9ifxkzl3/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-z40lauk3/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=46.4.0' 'wheel>=0.30.0' 'Cython>=0.29.21,<3.0' numpy==1.19.4 'dimod>=0.10.0.dev9,<0.11.0'
    Collecting setuptools>=46.4.0
      Using cached setuptools-58.2.0-py3-none-any.whl (946 kB)
    Collecting wheel>=0.30.0
      Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
    Collecting Cython<3.0,>=0.29.21
      Using cached Cython-0.29.24-cp39-cp39-manylinux1_x86_64.whl (1.9 MB)
    Collecting numpy==1.19.4
      Using cached numpy-1.19.4-cp39-cp39-manylinux1_x86_64.whl (13.4 MB)
    Collecting dimod<0.11.0,>=0.10.0.dev9
      Using cached dimod-0.10.7-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (10.6 MB)
    Collecting pyparsing<3.0.0,>=2.4.7
      Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
    Collecting dwave-preprocessing<0.4,>=0.3
      Using cached dwave-preprocessing-0.3.1.tar.gz (159 kB)
    ERROR: Exception:
    Traceback (most recent call last):
      File "/tmp/pip-standalone-pip-9ifxkzl3/__env_pip__.zip/pip/_internal/cli/base_command.py", line 173, in _main
        status = self.run(options, args)
      ...
      File "/tmp/pip-standalone-pip-9ifxkzl3/__env_pip__.zip/pip/_internal/req/req_tracker.py", line 97, in add
        raise LookupError(message)
    LookupError: https://files.pythonhosted.org/packages/9d/6e/51d3f843bafea371f3d96cf35a2e5c85e6d17c264cf3799ad6c11043c0c4/dwave-preprocessing-0.3.1.tar.gz#sha256=360e886a2e89f3f4aa0d3ddf448c71a2eaab74183000687bcafb1175682f8775 (from https://pypi.org/simple/dwave-preprocessing/) (requires-python:>=3.6) is already being built: dwave-preprocessing<0.4,>=0.3 from https://files.pythonhosted.org/packages/9d/6e/51d3f843bafea371f3d96cf35a2e5c85e6d17c264cf3799ad6c11043c0c4/dwave-preprocessing-0.3.1.tar.gz#sha256=360e886a2e89f3f4aa0d3ddf448c71a2eaab74183000687bcafb1175682f8775 (from dimod<0.11.0,>=0.10.0.dev9)
    Installing build dependencies: finished with status 'error'
  WARNING: Discarding https://files.pythonhosted.org/packages/9d/6e/51d3f843bafea371f3d96cf35a2e5c85e6d17c264cf3799ad6c11043c0c4/dwave-preprocessing-0.3.1.tar.gz#sha256=360e886a2e89f3f4aa0d3ddf448c71a2eaab74183000687bcafb1175682f8775 (from https://pypi.org/simple/dwave-preprocessing/) (requires-python:>=3.6). Command errored out with exit status 2: /opt/_internal/cpython-3.9.7/bin/python3.9 /tmp/pip-standalone-pip-9ifxkzl3/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-z40lauk3/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=46.4.0' 'wheel>=0.30.0' 'Cython>=0.29.21,<3.0' numpy==1.19.4 'dimod>=0.10.0.dev9,<0.11.0' Check the logs for full command output.
    Using cached dwave_preprocessing-0.3.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (770 kB)

As a short-term fix, we can use dimod<0.10.5 for building, but longer-term we should revisit the solution in https://github.com/dwavesystems/dimod/pull/999 to https://github.com/dwavesystems/dimod/issues/989.

Closed by https://github.com/dwavesystems/dimod/pull/1056


WARNING: An illegal reflective access operation has occurred

Expected Behavior

Current Behavior

Context

I hope your can resolve it
Steps to Reproduce (for bugs)

Your Environment

Build scan URL:

https://github.com/gradle/gradle/issues/3118#issuecomment-415960625

Duplicate of #3118


libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /root/.djl.ai/sentencepiece/0.1.97-0.20.0/libsentencepiece_native.so)

Greetings,
I'm having an issue that I'm not very sure I understand. I'm using the ai.djl.sentencepiece package in a class I wrote in Java and I get this error below:
/root/.djl.ai/sentencepiece/0.1.97-0.20.0/libsentencepiece_native.so: /root/.../webapps/prediction/WEB-INF/lib/ml/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /root/.djl.ai/sentencepiece/0.1.97-0.20.0/libsentencepiece_native.so)"

I have added the package dependency to the pom.xml of my project such as:
<dependency>
	<groupId>ai.djl.sentencepiece</groupId>
	<artifactId>sentencepiece</artifactId>
	<version>0.20.0</version>
</dependency>

Is there something I'm missing ? Any help os greatly appreciated.
Thank you.

@wavymazy
Are you running centos 7?
It seems you are using an old version of libstdc++.so.6 version. You need upgrade to a newer version:
sudo yum -y install libstdc++

@frankfliu Yes I am. Thank you very much, I'll try that.

After further digging it seems to not be trivial. CXXABI_1.3.9 is associated with gcc version 5.1. CentOS 7 has gcc 4.8.5.

@wavymazy
We have centos 7 nightly test: https://github.com/deepjavalibrary/djl-demo/blob/master/.github/workflows/canary-centos.yml#L92

@wavymazy
Sorry, I can reproduce your issue. I checked our code, we don't support centos:7. We only support amazonlinux:2 and ubuntu:18.04+
I'm trying to figure out why our canary test passed.

Thank you @frankfliu for the explanation. This makes sense.

I created a PR to add centos 7 support: https://github.com/deepjavalibrary/djl/pull/2402
Will be available soon in 0.21.0 release.

Thank you @frankfliu ! Do you have an estimate of when the version 0.21.0 will be available ?

0.21.0 is published on maven central. You can try it now.


Update circe-core, circe-generic, ... to 0.12.0-RC2

Updates

io.circe:circe-core
io.circe:circe-generic
io.circe:circe-testing

from 0.12.0-RC1 to 0.12.0-RC2.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention @scala-steward in the comments below.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [{ groupId = "io.circe" }]

labels: semver-pre-release

Codecov Report

Merging #32 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master     #32   +/-   ##
======================================
  Coverage    37.8%   37.8%
======================================
  Files           3       3
  Lines          82      82
======================================
  Hits           31      31
  Misses         51      51

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acfb639...2a3fa52. Read the comment docs.


Bug: possible to construct a transaction with an output equal to an input

I'm submitting a…

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request

Expected Behavior
Should not be able to construct a transaction with an output address equal to an input address.

Current Behavior

It's possible to construct a transaction with an output equal to an input
See also the gif and screenshot:

Note that in the gif I don't sign the transaction but it is possible to do so, see https://blockstream.info/testnet/tx/861c1201d3600adc2b17bc4b1cdb0764aaf1c0d891ca581e2080044a2f33e6c1?expand
Possible Solution
Prevent the above and show an error message.

Steps to Reproduce (for bugs)

Setup a funded wallet
Copy an address and note the balance
Paste the address into the output text field
Set the amount to the same amount as the address balance
Click preview transaction
Note that one of your output addresses is the same as one of your input addresses

Environment

Where are you running caravan:  localhost
Operating system: Linux
Browser and version: Chrome

<EMAIL_ADDRESS>/home/walden/code/caravan
├──<EMAIL_ADDRESS>└─┬<EMAIL_ADDRESS>  └──<EMAIL_ADDRESS>
npm: 6.14.4
node: v12.16.2
Linux 5.6.13-arch1-1 #1 SMP PREEMPT Thu, 14 May 2020 06:52:53 +0000 x86_64 GNU/Linux

Migrated to new repo: https://github.com/caravan-bitcoin/caravan/issues/58


Running multiple tracing implementations simultaneously

Continuing the discussion from https://github.com/opentracing/opentracing.io/issues/66
cc: @yurishkuro @bensigelman

Higher Order Tracers (HOT) & Migrating Between Vendors
I think there's a case for Higher Order Tracers i.e. Tracers that combine other tracers (at construction time) e.g.
import ot.hot
...

zipkinTracer = ZipkinTracer()
logTracer = LogTracer()
proprietaryTracer = ProprietaryTracer()

opentracing.tracer = hot.All(zipkinTracer, logTracer, proprietaryTracer)

Any in-band keys should be namespaced in order to enable >1 tracers working in parallel. Obviously there could be others like hot.Any or router-like i.e. depended on implementation specific run-time flags, span Attributes etc.
Something like this would allow one to switch vendors without downtime since it allows a transition period (could be days) when both the old and the new tracing systems are running. As far as I can see, it should be close to impossible to switch without downtime without such a feature.

Hopefully different tracers would be using different header keys anyway, so I would punt on the namespacing, especially since introducing namespacing is also incompatible with the existing deployment.

Yeah - in the sense of changing on the spec "A set of zero or more key:value Span Tags. The keys must be strings." ->"A set of zero or more key:value Span Tags. The keys must be strings that is highly recomended to be tracer-namespaced or otherwise made unique."

@yurishkuro - I agree... no "formal" namespacing. Just that people should be aware of aliases when they design their protocols. One interesting corner-case is baggage where I can imagine different implementations normalize the key and push it directly to the carrier. A naive impl that doesn't change the value, might be ok. Generally speaking bad things could happen.

From the previous discussion...

yes, the attributes is the only read operation in the whole API, so I think the composite tracer will have to pick one of the delegates, i.e. always the first, and the end users can configure delegate tracers in the proper order.

I think the only mathematically correct way (i.e. functional) to do this, is to not discard any information i.e. something like span.get_baggage_item(key) should return {'value1':['ZipkinTracer', 'LogTracer'], 'value2': ['ProprietaryTracer']} and let the app do whatever seems fit. Personally I would be more than happy with that but I'm sure most people would freak out. Could it return the first (non-null) majority and call an error callback-function mentioning "discarded value XYZ from that tracer?". Of course it could return first non-null but first doesn't sound very well defined :) Another option is to pass an (optional) lambda iteratable:  ... to get_baggage_item() that lets the app hand-craft its policy (this makes more sense in C++).
The same thinking applies on the return/error codes.

if you assume that the tracers already use prefixed keys, then encoding of the baggage is no different from encoding of the span context.
Errors are harder. I would go with a configuration option for the multiplexing tracer to either fail fast or to do best effort and aggregate all errors into one.

Let's assume that I upgrade from a tracer that doesn't support baggage propagation to one that supports. I think this use case, could easily hint towards a good solution.

I may have misread your point about baggage - so no issues for encoding/decoding, but just to return from MultiplexinsSpan.getBaggage - I would go with a simple solution and return whatever the first tracer returns, and a more complicated solution is again internally configurable strategy in the tracer on what to do in case of conflicts (I wouldn't worry until there's a real need).

No - you are correct. There were two issues on baggage, one is the one you said above, about the prefixed-keys. i.e. if someone sets "hello": 12 as baggage, e.g. X-B3-hello:12 should go to the Carrier. There's now the second issue of what MultiplexinsSpan.getBaggage() returns (which should be the same with Span.getBaggage() in terms of type, otherwise we break the O(1) replace implementation rule).  There are are also subtleties like what happens if we have 3 tracers and the second throws KeyError ... I think the Mutiplexer should mark it but continue and try to set on the third tracer. Similar interesting cases exist on the Tracer.extract() and Tracer.inject(). We might be able to restore a MultiplexinsSpan with Mux'ed Spans for tracer 1 and 3 but not for 2 (e.g. due to UnsupportedFormatException). MultiplexinsSpan should optimistically survive, but the partial error shouldn't go completely un-noticed.

If the mux tracer can be configured with error handling strategies like FAIL_FAST and ACCUMULATE, I think it mostly solves the issue with extract(). I don't think there's a way to define the "correct" behavior of the mux, so give users a choice to pick the behavior and do best effort.
There's probably a need for a 3rd error strategy - to succeed if at least one tracer succeeds, because the user code might be checking the error and not using the returned span context as parent in case of an error.

What are your thoughts on ACCUMULATE?

There's probably a need for a 3rd error strategy - to succeed if at least one tracer succeeds, because the user code might be checking the error and not using the returned span context as parent in case of an error.

I agree on this a lot. Overall you want to be able to upgrade/downgrade gracefully in terms of Tracer features. If baggage or a specific type of carrier wasn't supported and it's supported with the new tracing system I guess you want to print an DEBUG/INFO log and continue using as many Tracer features as possible. If none of the tracer supports a feature, then an ERROR should be logged and someone look at it ASAP... because e.g. you are using a baggage value that is now e.g. null.

Bringing https://github.com/opentracing/specification/issues/28#issuecomment-270272213 here...

I think this approach will help upgrade or switch systems, though in practice it might require complete deployment across the entire architecture before you switch off the former.

Yes exactly.

Key concerns will be around aggregate (in and out of process) propagation and trace context provisioning I think.

I think that the first concerns mostly what we talk about above. I'm not quite clear on the second one...

Error handling strategies should be the very important thing in multi-tracer-implementations. We should choose them carefully. If use the wrong strategy, a trace-implementation will failure, because of other implementation.
Such as: extract in tracerA.extract() -> tracerB.extract() -> tracerC.extract(), if tracerB.extract() fails, the strategy should make sure tracerC.extract() must be invoke. Otherwise,  tracerC.context() will be missed from there. And the trace-C will be broken.
This bug will be hard to find out from the tracer-user-side or tracer-deveploer-side.

enable >1 tracers working in parallel, I think parallel is not the accurate word, maybe. parallel means execution can be truly simultaneous, also almost means run in multi-threads, this is dangerous choice.

Such as: extract in tracerA.extract() -> tracerB.extract() -> tracerC.extract()
Most likely, multi-tracers execute like a chain as I said above.

@yurishkuro , I prefer the strategies can isolate the tracer-implementations, we could provide listener to notify user, there are errors or exceptions in some of the multi-tracers.

@lookfwd sorry for the delay, having a crazy week.
The primary value prop around interoperation of Tracers is not actually so that an org can necessarily switch from one tracing vendor to another with zero hiccups, although that's another nice possible benefit :)
The main advantage I see is actually for 3rd-party OSS/etc projects that are used by many organizations and need to provide tracing instrumentation without making assumptions about the environments they're integrated into. E.g., take DropWizard: there are companies using DropWizard and Jaeger; companies using DropWizard and Zipkin; and companies using DropWizard and LightStep. By instrumenting DropWizard with OpenTracing, the vendor-neutrality is preserved for at the DW level and the vendors don't all need to independently instrument DW.
As for the idea of informally namespacing headers: sure. Though IMO this is redundant to discuss at the OT level since that's true for any possible HTTP header (given that HTTP headers are a shared namespace to begin with). I don't strongly object to making the docs longer / more complex to accommodate this advice, but I am skeptical that it's needed to achieve your desired result :)

The main advantage I see is actually for 3rd-party OSS/etc projects that
are used by many organizations and need to provide tracing
instrumentation without making assumptions about the environments they're
integrated into. E.g., take DropWizard: there are companies using
DropWizard and Jaeger; companies using DropWizard and Zipkin; and companies
using DropWizard and LightStep. By instrumenting DropWizard with
OpenTracing, the vendor-neutrality is preserved for at the DW level and
the vendors don't all need to independently instrument DW.

That would be possible if people could rely on the api. Currently, it is
blind.. features are added without an inventory of which systems support
them. It is limited to whoever is paying attention which usually is several
people at most (out of hundreds of people actively designing tracing
systems). If the goal is for people to rely on this abstraction for
frameworks, users will need to know which features are risky and which
aren't. In other words, that would be a completely independent issue which
would track the viability of apis, for example encouraging use of more
stable or more often implemented features when attempting to design generic
framework instrumentation. Until then, the promise of instrumentation with
no assumptions about environments is unlikely to be credible.

The main advantage I see is actually for 3rd-party OSS/etc projects...

@bensigelman Awesome - that wasn't clear for me

without making assumptions about the environments they're integrated into. E.g., take DropWizard: there are companies using DropWizard and Jaeger

Aren't they making the assumption there's only a single tracing framework? How can one use DropWizard with both Jaeger and Zipkin?

Aren't they making the assumption there's only a single tracing framework?

It's a pretty reasonable assumption, but it's not a concern of Dropwizard instrumentation. If someone manages to make a Multiplexing tracer, as discussed on this thread, then a Dropwizard user can use two tracers simultaneously, and DW doesn't need to change anything in its instrumentation.

How can one use DropWizard with both Jaeger and Zipkin?

I don't think Ben was suggesting that. Unless OpenTracing 2.0 comes up with a common in-band format for context propagation, even if you use multiple tracers each one of them will still see only the slice of architecture that directly runs with that tracer.

If someone manages to make a Multiplexing tracer

Exactly - the question is - does OT impose some constraints that prevent such a tracer from being possible? My guess is that we wouldn't like that.
For example, as we were talking above, span.get_baggage_item(key) is assumed to either succeed or fail but not partially fail. I agree that in general this is Multiplexing tracer's concern mainly but it's possible that e.g. DW would be interested on the distinction between span.get_baggage_item(key, mode=strict) which fails if two tracers disagree for an "important" baggage item and the default span.get_baggage_item(key) that returns the first non-null baggage from the child-tracers. Similarly for partial success/failure of inject() etc.

P.S. I realize that it's mostly talking and thinking here. I think I will try to plug a mixing tracer next time I have time to play with Zipkin-Python-OT

There's a first ugly throw-away C++ draft of some concepts here.


topic: Return JsonableError for race condition in topic mute.

Issue #21011
To avoid IntegrityError return 500 status in race condition,
we catch the integrity error and return JsonableError with status 400.

Merged after a bit of work on both the comment and commit message for clarity.  You will likely benefit from reading the version I merged.  Thanks for fixing this @madrix01!
I opened #21084 for a very low importance follow-up.


Support for x-displayName for tags

Allow to change the display name of tags without changing the "name" property. It's the same property/behavior as in redoc.

@j3rem1e Thanks for the PR. Apologies for the delay in accepting it. We are resuming our work on this library


'gdb' missing ?

Hi !
First, a few words to thank you, it is an impressive and very useful work ! I am a total newbie regarding docker, so I apologize by advance if this question is not pertinent, or if the fix is evident.
I just succeeded in compiling a kind of C++ 'hello world' inside the container using a cmake project.
Now I want to use 'rxpc gdb' as a client for my Eclipse CDT environment in order to remotely debug a program which runs on the raspberry target using 'gdbserver'.
But it seems that gdb is not present by default in the cross-compilation tool flow.
Do you think it could be possible to add it by default ? Or give me an hint to add it by myself ?
Thanks !

Hi @yba35 - thanks for the kind words!
For some reason, the official Raspberry Pi toolchain that I bundle has two separate directories of cross-compile tools.
In the rpxc docker image, one gets mapped to /rpxc/bin with arm-linux-gnueabihf- prefixes, and the other to /rpxc/arm-bcm2708-linux-gnueabi/bin with no prefixes.
The first one is in the default rpxc path, and has a whole lot more in it than the second, including arm-linux-gnueabihf-gdb.
I've never actually tried remote debugging before today, but it seems to work. I followed these instructions. It may be enough to just specify rpxc arm-linux-gnueabihf-gdb to Eclipse.
Here's what I did to test things:
On the raspberry pi, run gdbserver host:2345 ./hello-world
On the dev box, run rpxc arm-linux-gnueabihf-gdb ./hello-world, and then when inside gdb target remote your-raspberry-pi:2345. You may need to specify the ip address, depending on your local DNS setup. You don't need to use 2345 - pick whatever unused port you like.
Let me know how it goes. I'd be interested to see if it all works with CDT, or if there's any other tweaks required.

Closing this issue due to lack of activity. Please create a new issue if necessary.


Changed the contact form color

@sunny0625
Issue #984
I've changed the contact form color according to the theme of the webpage...along with that I've changed the colors of submit button as they overlapped with the background.
Fixed a small bug caused due to an extra semi colon.
Here's a screenshot of before and now:
Before:

After:

Thank you for assigning me this issue under GSSOC'24. Please add suitable labels with this PR.
Thank you!

@kashvi9 i think the color used earlier is better.


TUT 3 - IntelOneAPI

Clarify this instruction, the path looks broken to the .bashrc file

added or, not broked, separate paths


Improve visibility and display toggling

Currently, if no styles are changed, after the transition completes oldContainer has display: none (it should) but newContainer has visibility: hidden. Shouldn't it be visible after this.done() is called?
Also, changing visibility is good for switching between but using display is better for replacing content. I think it'd be handy to use a flag to determine which property to be using so it doesn't have to be done manually each time of setup.

Hi @ZachSaucier ,
Thank you for your observation!
In the transition, when .done is called the oldContainer is removed from the DOM:
done: function() {
    this.oldContainer.parentNode.removeChild(this.oldContainer);
    this.deferred.resolve();
}

I can add visibility: visible in BaseTransition.done, but actually should be the custom Transition responsible to make to hide the old Container and show the new one.
So, will change nothing, right?
In visibility: hidden, instead of display: none you can access to some property like height/width of the elements, and it's super useful;
You may start a slider or do other JS operation when the newContainer is attached to the DOM and the transition is not yet finished (event onEnter in Views or newPageReadyin Barba.Dispatcher).
Also, with visibility: hidden the browser (at least Chrome) starts preloading the images, so, it's quite handy.
Hope it reply your answer, if yes, feel free to close the issue.

I understand your points and see the value in how it currently runs, but my original points still remain I think 😛
It's not a big deal either way, just what I see as an improvement

So, basically, your idea is to make the .donefunction like this:
done: function() {
    this.oldContainer.parentNode.removeChild(this.oldContainer);
    this.newContainer.style.visibility = 'visible';
    this.deferred.resolve();
}

Am I right?
If so, the default Transition HideShowTransition can be rewritten like:
var HideShowTransition = BaseTransition.extend({
  start: function() {
    this.newContainerLoading.then(this.finish.bind(this));
  },

  finish: function() {
    document.body.scrollTop = 0;
    this.done();
  }
});

Yes, that is one of my proposed ideas. It keeps things simpler

Yo'!
Will add this in the next release!
Thank you


Wav2Vec 2.0 pretraining stuck

❓ Questions and Help
What is your question?
Why loss and accuracy plots look like this?

Code
Running hydra train as it's shown in readme.md for w2v2 base.
Changes to config:
restore_file: /root/wav2vec_small.pt
reset_dataloader: true
reset_lr_scheduler: true
reset_meters: true
reset_optimizer: true
num_workers: 24
max_tokens: 1500000
ddp_backend: no_c10d
I've parsed hydra_train.log file for my pretraining then plotted it.
What have you tried?
.
What's your environment?

fairseq Version (e.g., 1.0 or master): master
PyTorch Version (e.g., 1.0): 1.9.0a0+c3d40fd
OS (e.g., Linux): Ubuntu 20.04 LTS
How you installed fairseq (pip, source): source
Build command you used (if compiling from source): pip install --editable ./
Python version: 3.8.10
CUDA/cuDNN version: 11.3
GPU models and configuration: RTX 3090
Any other relevant information: It seems like this strange shape occurs once loss_1 and loss_2 become 0

Went on and checked stable version 0.10.2
Process seems to be the same
Loss freezes at 6.658 in training phase
Although validation is quite nice and validation loss goes to low values as well as accuracy (about 0.02 and 0.99)
Can someone explain this strange thing?
Is it ok to use checkpoint like this?
P.S. valid set to be 0.01 of original dataset

One more test to prove there's somethings wrong
valid.tsv == train.tsv
Loss at train goes to 6.658 and right after that valid loss becomes almost 0 and valid accuracy goes to 1
Have anyone else encountered something like that?

@jubick1337 I'm also getting same problem.

@shiva1393 do you have any idea how to solve this problem?

Hi,
decrease learning rate it will help, this suggestion i got from previous issues.

@shiva1393 to which values?

lowering lr to 0.0001 helps

I have the same problem here trying to train a base model from scratch. Also loss shoots up to exactly 6.658 for me, so I think that is some kind of degenerate solution. Accuracy is close to 0 (on train). Will try to lower the learning rate, as suggested by @jubick1337 @shiva1393. I am training on a single GPU, maybe that is making the difference?

@jubick1337 Hi, I know it's an old issue, but how did you solve it? Reduced learning rate, replicated GPUs, or both at the same time?

Hi @Etoye,
I managed to solve it. I had to reduce the learning rate and also introduce gradient accumulation (simulating multiple GPUs). Specifically, in the optimization part of the base model config I have:
optimization:
max_update: 400000
lr: [0.0002]
update_freq: [8]
I was able to train a base model successfully with this on just one GPU. I think the default learning rate was 0.0005.

@bmilde Thank you for the details, it's very helpful! I am wonder if I have 8 GPUs, will it be enough to just replicate 64 GPUs by using update_freq: [8] or reducing learning rate is critical?

Unfortunately its trial and error. If you have 8 GPUs, I'd try update_freq: [8] first. Maybe you can also just lower the learning rate without setting update_freq (this would give you exactly my config).
For me, with just one GPU, setting update_freq: [64] didn't work for whatever reason, but basically simulating 8 GPUs with lower learning rate did converge.


Can it be used in multi-DMA

Hello everyone：
if this driver is a single DMA hook, I have succeeded.Can it be used in multi-DMA?
This is my device tree.
axi_dma_0: dma@80010000 {
#dma-cells = <1>;
clock-names = "s_axi_lite_aclk", "m_axi_sg_aclk", "m_axi_mm2s_aclk", "m_axi_s2mm_aclk";
clocks = <&clk 71>, <&clk 71>, <&clk 71>, <&misc_clk_0>;
compatible = "xlnx,axi-dma-1.00.a";
interrupt-names = "mm2s_introut", "s2mm_introut";
interrupt-parent = <&gic>;
interrupts = <0 89 4 0 90 4>;
reg = <0x0 0x80010000 0x0 0x10000>;
xlnx,addrwidth = <0x20>;
xlnx,sg-length-width = <0x10>;
dma-channel@80010000 {
compatible = "xlnx,axi-dma-mm2s-channel";
dma-channels = <0x1>;
interrupts = <0 89 4>;
xlnx,datawidth = <0x100>;
xlnx,device-id = <0x0>;
};
dma-channel@80010030 {
compatible = "xlnx,axi-dma-s2mm-channel";
dma-channels = <0x1>;
interrupts = <0 90 4>;
xlnx,datawidth = <0x100>;
xlnx,device-id = <0x0>;
};
};
axi_dma_1: dma@80020000 {
#dma-cells = <1>;
clock-names = "s_axi_lite_aclk", "m_axi_sg_aclk", "m_axi_mm2s_aclk", "m_axi_s2mm_aclk";
clocks = <&clk 71>, <&clk 71>, <&clk 71>, <&misc_clk_0>;
compatible = "xlnx,axi-dma-1.00.a";
interrupt-names = "mm2s_introut", "s2mm_introut";
interrupt-parent = <&gic>;
interrupts = <0 91 4 0 92 4>;
reg = <0x0 0x80020000 0x0 0x10000>;
xlnx,addrwidth = <0x20>;
xlnx,sg-length-width = <0x10>;
dma-channel@80020000 {
compatible = "xlnx,axi-dma-mm2s-channel";
dma-channels = <0x1>;
interrupts = <0 91 4>;
xlnx,datawidth = <0x100>;
xlnx,device-id = <0x1>;
};
dma-channel@80020030 {
compatible = "xlnx,axi-dma-s2mm-channel";
dma-channels = <0x1>;
interrupts = <0 92 4>;
xlnx,datawidth = <0x100>;
xlnx,device-id = <0x1>;
};
};
axi_dma_2: dma@80030000 {
#dma-cells = <1>;
clock-names = "s_axi_lite_aclk", "m_axi_sg_aclk", "m_axi_mm2s_aclk", "m_axi_s2mm_aclk";
clocks = <&clk 71>, <&clk 71>, <&clk 71>, <&misc_clk_0>;
compatible = "xlnx,axi-dma-1.00.a";
interrupt-names = "mm2s_introut", "s2mm_introut";
interrupt-parent = <&gic>;
interrupts = <0 93 4 0 94 4>;
reg = <0x0 0x80030000 0x0 0x10000>;
xlnx,addrwidth = <0x20>;
xlnx,sg-length-width = <0x10>;
dma-channel@80030000 {
compatible = "xlnx,axi-dma-mm2s-channel";
dma-channels = <0x1>;
interrupts = <0 93 4>;
xlnx,datawidth = <0x100>;
xlnx,device-id = <0x2>;
};
dma-channel@80030030 {
compatible = "xlnx,axi-dma-s2mm-channel";
dma-channels = <0x1>;
interrupts = <0 94 4>;
xlnx,datawidth = <0x100>;
xlnx,device-id = <0x2>;
};
};
};
axidma_chrdev@0 {
compatible = "xlnx,axidma-chrdev";
dmas = <0x34 0x0 0x34 0x1>;
dma-names = "axidma0_tx_channel", "axidma0_rx_channel";
dma-id = <0x0>;
chrdev-name = "axidma0";
minor-num = <0x0>;
num-devices = <0x1>;
};
axidma_chrdev@1 {
	compatible = "xlnx,axidma-chrdev";
	dmas = <0x35 0x0 0x35 0x1>;
	dma-names = "axidma1_tx_channel", "axidma1_rx_channel";
	dma-id = <0x1>;
	chrdev-name = "axidma1";
	minor-num = <0x1>;
	num-devices = <0x1>;
};

as far as I know, you should assign each channel of each dma an unique device-id. please change accordingly and let me know in case of any further problem.

I made the configuration with the axi_dma_0 (xlnx,device-id = <0x1>;) and axi_dma_1(xlnx,device-id = <0x0>;),
From the axidma_benchmark  -help i got this:
-t :                    The device id of the DMA channel to use for transmitting the data to the PL fabric.
-r :                    The device id of the DMA channel to use for receiving the the data from the PL fabric.
When I  run this,
./axidma_benchmark -t 0 -r 1
the test will complete successfully.
All other combination will fail. I think those inputs have no effect on device id. But only on selecting the proper tx or rx channel.
Is there another way to select the axi_dma_1?

I think it has something to do with this part of the kernel tree:
axidma_chrdev: axidma_chrdev@0 {
compatible = "xlnx,axidma-chrdev";
dmas = <&axi_dma_0 0 &axi_dma_0 1>;
dma-names = "tx_channel", "rx_channel";
};
What should this be?

Is there a reason of this restriction?
https://github.com/bperez77/xilinx_axidma/blob/master/driver/axidma_of.c#L241
If i change my kernel tree to:
axidma_chrdev: axidma_chrdev@0 {
compatible = "xlnx,axidma-chrdev";
dmas = <&axi_dma_0 0 &axi_dma_0 1 &axi_dma_1 2 &axi_dma_1 3>;
dma-names = "tx_channel", "rx_channel" ,"tx_channel1", "rx_channel1";
};
I got this error message:
[   43.443851] axidma: axidma_of.c: axidma_of_parse_dma_nodes: 243: Device tree node axidma_chrdev: Phandle 2 in the 'dmas' property has an invalid channel (argument 0).

I have tried updating:

increased the number of channels
https://github.com/bperez77/xilinx_axidma/blob/master/driver/axidma_of.c#L241
I have changed the check, so 4 channels could pass:
https://github.com/bperez77/xilinx_axidma/blob/master/driver/axidma_of.c#L130

But now I get this error:
[   44.789393] axidma: axidma_dma.c: axidma_request_channels: 651: Unable to get slave channel 2: tx_channel1

Each DMA has two channels - 0 & 1. You can't give them 2 & 3 as in your device tree. Try this:
dmas = <&axi_dma_0 0 &axi_dma_0 1 &axi_dma_1 0 &axi_dma_1 1>;
Keep the rest the same.
Also, notice in your device tree, the entry for axi_dma_1: dma@80020000 has 2 channels, both with  xlnx,device-id = <0x1>; You'll need to change one of them to  xlnx,device-id = <0x0>;
in your system-user.dtsi. Similar to this :
&axi_dma_1 {
dma-channel@80020000 {
xilnx,device-id = <0x0>;
};
};

Thanks for the tip. Everything works fine now.
I added the following items to my device tree:
&axi_dma_0 {
dma-channel@a0000030 {
xlnx,device-id = <0x1>;
};
};
&axi_dma_1 {
dma-channel@b0000000 {
xlnx,device-id = <0x2>;
};
dma-channel@b0000030 {
xlnx,device-id = <0x3>;
};
};
And I was able to run the command for the second dma:
sudo ./axidma_benchmark  -t 2 -r 3

Excellent.

Excellent.


Allow specifying run script and copy files phase ordering

Fixes #400.

Allows specifying run scripts to run immediately after the Compile Sources phase (which should be used instead of postbuildScripts 99% of the time).
Allows specifying copy files phases to be run before the Compile Sources phase

Adjusts modulemap and static library heady copy files phases to be run pre Compile Sources
Adjusts the Carthage copy-frameworks to be run sooner

Even with these changes the new build system is broken. That doesn't make these changes not needed though. Eventually Xcode 10 will be fixed, but being fixed will still warrant these changes.

@brentleyjones

which should be used instead of postbuildScripts 99% of the time

Question: What do you expect for remaining 1% of them?
Also I think sources's buildPhase structure is getting too complicated. It should look as same as we see on Xcode.
Ideally I expect a format like this.
        buildPhase:
          copyFiles:
            destination: productsDirectory
            subpath: include
          compile:      // <- This simply makes the processing order clearer.
          runScript:
            path: my-script.sh

Question: What do you expect for remaining 1% of them?

When you want to post-process something that has been copied into the products folder (like adjusting application icons for debug versus releases).

@yonaskolb Care to give this another look?

Hi @brentleyjones. Would you be able to get this branch up to date resolve the conflicts?

@yonaskolb Yeah, I'll try to get to it today 😄.

@yonaskolb Assuming tests pass, I believe this is all merged up. I also removed the ability to specify copy files phase ordering.


[Request] Add autodismiss option

Hello, good package! First of all, thank you for your work.
I think it would be interesting to include a auto-dismiss option.

Hey, thanks! Glad the package is coming in handy.
I have thought about this too, and would like to see this implemented. I'll see what I can do here but my time is a bit limited. My thoughts:

should be vanilla Javascript
should be able to get basic POC via conditionally adding a setTimeout for calling @this.dismiss() inside the message component.

Some nice-to-have:

would like to have some sort of a visual indicator on the message itself that it will disappear... progress bar or something.
would also be nice if hovering over the alert would pause the auto-dismiss

If you have any suggestions, or even a quick PR, that would be great. Thanks!

@sat-on-site Naturally I ended up poking at this today to satisfy my curiosity.
I think I have a rough solution that checks all the boxes, but could probably use some tweaking / cleaning up.
If you would like to play around, please feel free to use the feature/auto-dismiss-v2 branch. Syntax should be:
flash('My Message')->dismissAfter(4)->info()->livewire($this);
My biggest concern is figuring out how to get the additional JS and CSS that are needed loaded in properly. But this should at least be a starting point.

did you get this working? would be nice to have this in the main branch

Hey @MACscr - I will need to see if I can find some time to review what I hacked together. I remember feeling OK-but-not-great about what I ended up doing.
Have you been using the feature branch? If so, any feedback?

Hey @mattlibera thanks for the package!
For a quick and dirty auto dismiss here, what I use:

Maybe helps for others until you publish the new version.


Update static image file

Update Image of Create a new ASP.NET Core web application from ASP.NET Core 3.0 to 3.1

Thanks!


haxelib git faxe error

when i try doing the command it just shows this

i have installed it and as far as i see it doesnt specify what the path is

have you installed git scm? go to this link and make sure to download that .


revert patchEvent field change

the change merged in the https://github.com/textileio/go-threads/pull/416#discussion_r481080485, breaks the backward compatibility with the existing db's records.
This PR aims to revert this or have some other fix to mitigate it

I've just opened it by mistake in the textile repo instead of the forked one