Repository branching and tagging, versioning

Hello @marvinroger,
Somewhere in comments we already discussed that branching in this repository should be addressed. I want to restart that discussion. Here's my first suggestion, which is open for discussion:

Introduce a strict versioning system - Homie is a convention to be followed by other development projects. It is hence important to add unambiguous version numbers for every convention revision. A developer needs to be able to state "The app conforms with the Homie convention v2.1.3". I suggest to implement semantic versioning: http://semver.org (should be a perfect fit)

Work in the main branch only - Currently there is an outdated master branch, an unclear v2.0 branch and a redesign branch. That is neither intuitive nor visible to a reader. As the convention is steadily evolving, we should present the latest revision to the interested developer/user. (With the option to switch to other versions, see 5)

Use tags to mark versions - A homie version is a fixed state of the document, hence tags are perfectly fine for the job. (Other branching models suggest branches in order to allow e.g. hotfixes)

Version every commit - Almost every commit will change details of the Homie convention, it is therefore needed to increment the version number as the MAJOR.MINOR.PATCH system of semver suggests. (Exceptions are of course commits with language changes only)

GitHub releases for stable tags - GitHub offers the releases tab to publish versions and add a release note. That should be used to announce and document the stable and noteworthy releases (e.g. v1.5.0, v2.0.0, v2.1.0)

Link the "Tags" tab in the first part of the convention - For readers to quickly find all available homie versions, the tags tab should be highlighted in the first part of the convention document.

Everyone, please add your comments, criticism and feedback!
Best! Thomas 🍻

Hi @ThomDietrich
Thanks for that! Actually, this was already planned (we do exactly all of that on https://github.com/marvinroger/homie-esp8266). I wanted to "clean everything" as soon as we all agreed about the v2.1.0, which will be the last release that breaks semver (the v2.0.0 is not really backward compatible, but I'd say this is fine as the v2.0.0 was never released officially).
The only thing I did not think about was the "Tags" tab. Great idea! 👍

https://github.com/marvinroger/homie/commit/e0e3bad3c659c12146819d53b2612098acedc176 ✨

@marvinroger just a thought before it's too late. After all recent additions and changes I wonder if 2.1.0 is the correct version to choose. How about 3.0.0 or 2.5.0?

I agree that v3.0.0 would be semver-compliant. I don't know, because the v2 was actually never released. But yes, the versioning was chaotic. Let's follow semver and go for v3. 😈
We'll have it ready very soon, that'll be a thing of the past. 😉

Hi @marvinroger!
What's left for the release of the official Homie 3.0 version? All PR's are merged and all big issues are closed, what else needs to be done?
Many thanks for all your hard work! :+1:

@timpur did you miss the mentioning?
What do you think? If you are not disinclined, I'd go and apply the plan tomorrow.

Yes, just stretched for time atm ....

@timpur @marvinroger
I might have overestimated my free time back in the last comment :D
Just so we are all on track: Do we want to go forth with the plan to mark version 3.0 now, then address all existing issues? A simple 👍 would be enough to show your approval. 😉 I'll do the needed work then.

What are some of the big issues atm, tbh not following them all ... but yest i think were ready for 3.0

Definitely!
Sorry for my tiny presence here, I have been preparing my internship
defense, which is tomorrow! 😊
Le lun. 19 févr. 2018 à 13:56, Thomas Dietrich<EMAIL_ADDRESS>a
écrit :

@timpur https://github.com/timpur @marvinroger
https://github.com/marvinroger
I might have overestimated my free time back in the last comment :D
Just so we are all on track: Do we want to go forth with the plan to mark
version 3.0 now, then address all existing issues? A simple 👍 would be
enough to show your approval. 😉 I'll do the needed work then.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/marvinroger/homie/issues/46#issuecomment-366685752,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA8eNYO7LWvql-JmbkTdi60Ec3Vd55DZks5tWW-KgaJpZM4PjPFh
.

@marvinroger Good luck!

@ThomDietrich, Do what you can and ill help when i can :)
Thank for contributing.

Hope that everything went fine @marvinroger!
Thanks for all your work guys!

Question, how do you guys prefer to do implementation versioning? eg ESP implementation? Should it align with Homie version?

Yes, IMHO it should be clear what versions of both match

yes but what :P ?

H3.0ESP23.04

Damn

Could be:
c = Homie Convention
i = Homie Implmentaton version
c.c.i.i
so for Homie ESP currently that would look like <IP_ADDRESS> think @nerdfirefighter said this.
but not sure about it, i think they should be separate, just need to document these version better, and display them more !!!

@marvinroger need you to remove v2 branch, and set master as default. I have merged and cleaned up branch v2 so you can remove.
Master sits at v2.0 now and is ready for redesign branch (3.0) we will work on redesign and master will reflect most recent release always.
We now need to get 3.0 ready for release.

@timpur I have to apologize. I ran into some unforeseen problems in my personal endeavors and didn't get around to continue here. Thanks for taking matters into your hands.
One thought I want to mention: You've changed "2.1.0" to "3.0" but it should read "3.0.0". Is this a mistake? Please compare: http://semver.org/

Didn't know what existed .... Good to know. My mistake, will fix

Hi guys!
Thanks for the work being done here. I did what you've asked @timpur. I just made some cleanup to my notifications (had a few hundreds), so I hope I'll be able to be more responsive.

thanks @marvinroger, sorry to bombard you ...

Don't be sorry! 😇

Marvin: Thanks!
@timpur I'm confused about the current state. The idea was to merge redesign into master. Are you going to do so? See point 2 of my list in OP...

IMHO, I think master should reflect the latest release and we should be working from a device branch... This is how most repos work so why change it here ?

I thought to have argued about that in the first post.
The important thing is to remember that the Homie Convention is destined to be revised constantly. That's why Marvin and I agreed to go with a stringent version scheme. Every addition or change to the convention will potentially be picked up by developers interested in this new aspect of the convention. Hence a version for this addition/change is needed.
To cope with those requirements we agreed to develop in master and pull requests only. To make sure every developer is still able and encouraged to access specific versions, we've introduced this link:

Im totally fine with that route. But i just have one question.
What if were half way in releasing a new version and working from master and a new dev comes in and starts to use what they see in master assuming that that is the latest release (as most repos use that method) and then later realises what they have implemented is half baked....
Will we need to make sure people only use versions from the release tab? Like we should only be using versions from releases but you know....
I just wonder what is the benefits and what do you risk.

A good thought. Let's presume that the master branch only evolves through PRs. Half baked changes shouldn't find their way into the master branch because they are fine-tuned in the PR first. The goal is to have quick and direct iterations of the outwards-facing convention. Every non-trivial PR merge will be accompanied by a version increment in the form of a tag.
Now how is a developer of an application implementing the convention supposed to select a Homie Version he/she wants to conform to? (This is related to #51 )
I'd say first and foremost they should always refer to the latest revision if possible. I'm saying "if possible" because it really depends on the other side, the controller or node the dev wants to interact with. He/she will need to select a specific version if in doubt. In this moment a conservative master branch wouldn't do much good as we can't predict which Homie convention version the other side is working with. In the end the developer needs to compare versions (semver already addresses compatibility) and the changelog in order to select the right version to implement.
Looking at the above I can't make out any advantages of a dedicated develop branch, on the other side I am seeing additional delays and overlaps growing out of it.
I'm not trying to undermine your critique. I may be utterly wrong and be missing something :) Let me know if you are still not convinced.

TBH, same outcome so i dont mind, if you think it will work then i trust you. If it doesnt we can always go back to master dev... so lets give it a shot. I like what your trying to achieve.

I agree with @ThomDietrich.
However, never say never. So if one day there is a need for a larger redesign or a new complex "feature" there should be a dev branch again.
The advantage of such a branch is that it is easier to go "backwards".
If something was added to master (and tagged) and later reverted because someone had a better idea, you get incompatible official versions.

I'm no git/repo expert so this might be completely left field, but i'm going to throw it in the ring away.
if i understand the constraints and intentions correctly i see the following:

once a major version is released, then changes to that version are mainly

bug fixes (particularly in the documentation of the version)
addition of non mandatory topics

things that make it easier to do something or standardise something but not something that dictates how to use the framework. e.g. defining an "standard" attribute of $unit for version 2.0 for a property so that we can make a standard connector in Home Assistant for it, but there is no need to use it if you dont want to.

other cosmetic changes.

So as i sort of see already, we need/should be able to maintain older versions with out affecting the latest development or main version. so in this case, there are things that would be nice to to with V2 with out the need to change over all the code just to implement V3 when V3 is not in a released state any way.
Now i see a few options

A new Repo per major version - This would get messy and people would not know which repo to use. - Not worth it.
A new branch pair(dev and master) per version. e.g. master-v2, develop-v2. this would allow independent development, bug fixes, etc with out  the need to worry about updates from other versions of the framework being released before they are ready or being affected if we need to rollback. its not much extra overhead as we dont create new major versions that often. also master would be more of a pointer branch that directed people to the branch of the version they want
Or, we restructure the layout and put each version in a top level folder, minor versions in a sub folder of that version and have two branches, dev (things that we are not in release state) and master (latest stable version). this has the same issue as what we have now and option 1 in promoting one version to master may push up changes in a different version of the framework while it is still in development and not in a stable state.

just my two cents for thinking about.

@ThomDietrich #68 is an example of why we normally have master and dev(another branch), while we're working on 3.0 in dev additions can be made to the current release.... this is hard with one branch.

If there is a need to make changes to an old release, then you can create a branch from the old release tag. There is no need to have a branch before a change is needed.

@euphi What do you do with the branch you create from an older tag, you can merge that back into the master branch with newer changes..... sounds messy...

For #68:
Variant a) Do the change for v2.0.1 in master branch and tag it as 2.0.1. Later merge "redesign" into this.
Variant b) Create a v2.x-branch right now, do the change in that branch and tag it as 2.0.1. No need the merge "redesign" into this.
I prefer b).

My question is after tagged how do you get it back into master when master has 3.0.0 in it ... We don't want to leave branches hanging around the....

Interesting. In my whole concept proposal I did not incorporate the need to change later versions. If you need $nodes I'd say you should use a version that supports it. Why the extra effort to patch older versions? You'll in any case need to adapt the controller and the node to support $nodes. Why not just go with v3.0.0 then?
@euphi's proposal would be how I would do it... however I wouldn't do it 😄

Naaah don't say that, we are all equals here and decisions based on diverse needs and perspective are a great thing. As I said before, my opinion is as it is but I'm more than willing to find a compromise everyone* is happy with.
I'd say we can't compare the repository to any other, because it is not a software we are developing, it's a standard.
My fear with either dev and/or version branches is the confusion and work arising out of them. Looking back at the last halve year, neither @marvinroger nor I and others had much time to improve the convention in all directions that were discussed. Still we were able to address the most serious issues with regard to Discoverability - the most important requirement wrt the Homie convention.
To drive Homie forward it is my impression, that we need to clear out uncertainties and reach a level of definition and documentation of a latest version that is sufficient for users of all levels.
I am not against your proposal in #68, I would only give to reason that your contribution would have a higher return if we all worked towards a better version 3 to reach the above. Now once again, if you think #68 is meaningful to you and (!) others, feel free to shoot away ;)

Also guys, one thing: we talked with @timpur about moving the convention to https://github.com/homieiot
I can understand people might not be confortable with the idea of working with something (e.g. Homie) that's tied to an individual. What do you think?

Since you asked...
I've found nothing comparable to Homie in terms of convention and reference implementation. It's a great piece of work, and setting it free is a powerful statement from the original artist @marvinroger. It's a great statement about your commitment to open source, and I couldn't agree more with moving it to https://github.com/homieiot.
Now that I'm typing, I should add that as an author of a fairly successful collaborative open source project, branches and version numbering are important for clients of the specification (and implementations).
The first impression github page should be of the stable code (or convention in this case). Master is for stable production use, and branches are for research & development. The signal that something is production grade is that it's merged into master. Of course that's just a convention, but the reasoning behind it is sound.
As for versioning, the semver spec is a great guide and arbitrator. It's a simple way to communicate complex concepts.
Thank you for the hard work that everyone's putting into this. I've been following updates and will update the NodeJS implementation as the specification stabilizes.
...back to lurking

@marvinroger as you might remember I proposed something similar in one of our mails. Hence fine with me ;)

One more try to settle a few comments regarding branches:

PRs are effectively branches. No unstable "code" will find it's way into master
The idea to constantly have a dev branch is founded on the believe, that there will be one stable version we should present on master - I am not convinced this is true.
Users will want to browse specific versions of the convention (not any single one). That's why the reference to the tags/releases tab and the previously proposed list of significant versions is important.

@lorenwest did this scatter your fear?
Concluding: I would proposed to make a clean cut now. Reduce to master, add tags for major versions, and continue with the improvement discussions. If at a later point bigger changes in the form of multiple additions are meaningful, I'll personally create a branch named develop 😃

I do like GitHub Flow. It's nice and easy to understand, and it's yet what has worked the best for me.
The idea is simple: master should contain the latest "stable" (in our case, the latest released convention). Whenever we want to make a new version, we create a new specific branch. This allows for small and localized increments. If the change only contain tiny fixes (typo, graphical changes), then it can be merged to master without bumping version. Otherwise, the version would need to be bumped prior to merging.
I am also against maintaining old versions, and having a develop branch. From my experience, having two parallel branches is a true nightmare to manage on GitHub. Just one extremely annoying thing: do a PR from develop into master. You'll have a nice commit history. Once merged, continue to work on the develop branch, and once again, do a PR against master. You'll not only have the newest commits, but also the ones prior to the first merge. So it gets worse on every merge. Plus, after each merge you have to rebase your develop to the tip of master. I've found this pattern hard to follow.

@ThomDietrich - Yes to everything.
In my opinion, branches are good for:

Collaborating on future major versions (transient - soon to be master)
Archiving past major versions (long lived)
Ad-hoc collaboration on an issue (transient - for the life of the feature)

1 being a subset of 3 means there's only transient issue branches and major release branches.
Having a single long lived dev branch doesn't have much value, and tagging important commits within a major release can be valuable to communicate a point in time.

@marvinroger not sure if we are on the same page here. The GitHub flow workflow is basically what I am proposing from the beginning 😃 you just have to remember that PRs automatically mean that there is a dedicated branch for the change.

Yeah, this is also something I wanted to write: Each PR is based on a branch. So the proposal of @ThomDietrich includes to have (short-living) branches for feature implementation.
For more complex features I already (yesterday) proposed to create the branch manually (but later merge it back to master).
Furthermore, for "patching" old releases, you must create a branch. So, for #68 , you need to create a branch the branches off at the commit with the v2.0.0 tag. We should agree on a convention how to name this branch, e.g. release-v2.x .
@marvinroger it seems that both Github Flow and Git Flow don't handle this "Number #68"-case that you want to apply "patches" to old revisions.
So, my proposal (based on this discussions here):

Work on master branch
Alternative: Create development branch and merge back to master only for releases
Releases shall be on master branch with Semver-Compliant Tag for each release.
-> this is a specification/convention, so there should be no need for "beta" versions.
Fixes or compatible improvements to old versions may be created and if so they shall be on a branch that is named release-v{majorversion}.{minorversion}.x..
--> There shall be no incompatible changes to old versions, so the {majorversion}.{minorversion}.x scheme is always sufficient to be semver-compliant.
--> Fixes on these branches may be merged back to master (or develop)

I hope we all agree with 2. and 3. - at least it seems that they are not affected by the decision regarding 1.

Translating what seems to be coming out of this is we should be creating a branch for a new feature or change. Up the version on that branch, create a PR for that branch to master. When approved, merge and delete branch, and create release?

@timpur yes, but github alllows you to directly create a PR. The necessary branch is created automatically and github asks to delete the branch, if the PR is merged.
So, work in master branch, but always use PRs for changes (even if you have the rights to directly edit in the master branch) is exactly what you describe in your first paragraph.
Every new "feature" can be released immediatly, so as long as there no new features that are too large to handle it in one PR, there is no need for a development branch.

Makes sense. So for bigger changes that require multiple people make a feature branch... And remove after release back to master.

To summarize, the next steps would be:

I'll merge redesign into master, sorting out leftover mess.
I'll check and complete tagged versions (some intermediate are missing)
I'll propose a PR adding details regarding the different versions of the Homie convention as described in point 3 here
You @timpur can implement #68
We can ping all stalled issues to restart discussions based on the v3.0.0 state of the convention.
We can finally continue working on the actual convention instead of discussing technical details.
When a time for bigger refactoring comes, we will use a dedicated feature branch for that.

I don't understand how I can do #68 when you've merged 3.0.0 into master. I require 2.0.0, which I can branch off yes but I'll never be able to get it back into master when you've merged 3.0.0 ?

Yes, but what's your problem with a v2.0.0 branch? There is no need to merge it back.
That's why branches were invented... :-)

Exactly. You will see that everything falls into place once you start working on it.
Maybe this way of thought helps: In the end not your branch is the important artifact, your tag is. You will branch off of the v2.0.0 tag, add your changes in one or multiple commits, then create a tag "v2.0.1".
This tag is the important thing and is what will be available to users later. At this point no one cares on which branch it is based.
@euphi regarding the branch name. After consideration, I wonder if we should prefer patch-v2.0.x over release-v2.0.x?

Okay, if you dont thing git will garbage collect the unmerged commits in a deleted branch with a tag then sure. Im all in.

@timpur
Who wants to deleted the branch?
There is absolutely no reason to do this.

@ThomDietrich
I wanted to avoid the "hotfix-v2.0.x" branch name. As the "release-v2.0.x" name is proposed in the "git flow"-branching model, but we don't follow this model*, I'm fine with another name.

in that model, the release-branch is created for developing the release and then merged back into master for the release. The git flow model does not offer a solution how to maintain old releases. (Not even for hotfix?!).

@marvinroger the master branch is protected and I can't push destructive changes. (Good so far! 😉)
However it would be meaningful to remove the latest commit on master to end up with a clean commit graph. Could you please do the following? Thanks!
git checkout master
git pull
git reset --hard 3bcbb01ef31205a5c2a19522fc9b6dd892d1c274
git merge --ff-only origin/redesign
git push

@ThomDietrich done 😉

Wait I thought the whole point of this was to only have one branch master and use feature branches only to make changed but they eventually get merged and deleted ? So won't 2.0.1 be the same. I'll tag it and relase it and then delete the branch ?

No, that's just not possible.
If you want to maintain a old release, you must have a branch for it.
The github flow and git flow models don't consider maintaining an old release, so we can't use them.

So develop in master or feature branches.
Release (tag) on master
create release branches only to develop and tag a maintenance version of an old release

Are we ready to move to @homieiot? No immediate concerns about that?
I am just wondering, should we name it homieiot/homie or homieiot/convention? One thing, I'd like to create a website that would render the convention markdown, and having it as https://homieiot.github.io/convention sounds cleaner than https://homieiot.github.io/homie

Oh and yes @ThomDietrich, I remember that we already discussed about moving the convention to its own org. I was then looking for a job, and my GitHub weighted more than my CV. I am now in a comfortable position, so now that my career is "bootstrapped", I have no problem moving the convention. 😉

I also prefer "convention".

Regarding moving to homieiot? Should we fork the repository to keep history or should we create a new Repo?
For creating a website, the directory structure will change, so maybe a new repo is better?
So before moving the contents, we should clarify:

Which site generator shall be used?
When and how do we move content (the actual convention?)

@euphi we don'T have to worry about history. Moving a repo just means it will be shifted into another namespace. Btw. GitHub will auto-redirect the current repo URL to the new one so that links and users will still find their way.
@marvinroger we are in total sync :D I was also thinking about https://homieiot.github.io/convention 😉

Let's all welcome the new homieiot/convention repo 😄

Wonderful. You might want to adapt the short description of the project to include "Homie".
I'd say the issues of this issue are settled. However let's leave it open as a reference.

Just did. Thanks to everyone! 👏

Just noticed the NodeJS implementation merged on Sept. 21 2017 didn't make it to the Home Page v2.0 or the new Implementations page.
Probably an artifact of repository movement - although if it were an accident, I wonder what other commits fell through the cracks...

@lorenwest Done :)

Would you mind also adding back to https://github.com/homieiot/convention/tree/release-v2.0.x ?
That's the version it was developed against. I'll update as soon as 3.x moves from WIP to LIVE.

@lorenwest Done, sorry should of thought about. Also you might want to define a version and link to one of the releases of Homie. https://github.com/homieiot/convention/releases :)


Colorization of output

Are there any possibilities to add colorization to this tool? Everything is brilliant but absent of colorized stdout is real issue when copying/moving multiple files in bulk.

Everything is brilliant
Thanks !

Well, this may be a good opportunity to use python-cli-ui.
Somehow I feel I can trust its author

Ask and you shall receive.
The pull request is a bit hard to review, sorry (lots of refactorings), but if you wanna give it a try, you're welcome.
If you want to skip the review, feel free to do so :). I'll merge it and publish a new release "soonish"
Cheers :)

I know shit about Python so i can't "review" it properly, sorry.

no problem :)

You can still give your opinion about what the colored output look like:

Brown on dark background doesn't look very "eye friendly", but at all looks sweet

Here you go:

That's a lot nicer

Ok, the v.8 is out. But there are serious performance problems (sorry)
See #20 for the details.
May be best to wait for 8.1 ...


Bump sane-pdf-reports docker image

Related PRs:

https://github.com/demisto/sane-reports/pull/233

Docker Image Ready - Dev
Docker automatic build at CircleCI has deployed your docker image: devdemisto/sane-pdf-reports:<IP_ADDRESS>453
It is available now on docker hub at: https://hub.docker.com/r/devdemisto/sane-pdf-reports/tags
Get started by pulling the image:
docker pull devdemisto/sane-pdf-reports:<IP_ADDRESS>453

Docker Metadata

Image Size: 729.36 MB

Image ID: sha256:a4b1ef2fc51686d4873d92701f4679c18cc4bf0569d7dbd209308f0bd7b5571c

Created: 2022-11-15T12:50:55.647009492Z

Arch: linux/amd64

Command: ["python3"]

Environment:

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D
PYTHON_VERSION=3.10.8
PYTHON_PIP_VERSION=22.2.2
PYTHON_SETUPTOOLS_VERSION=63.2.0
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/5eaac1050023df1f5c98b173b248c260023f2278/public/get-pip.py
PYTHON_GET_PIP_SHA256=5aefe6ade911d997af080b315ebcb7f882212d070465df544e1175ac2be519b4
DOCKER_IMAGE=devdemisto/sane-pdf-reports:<IP_ADDRESS>453

Labels:

org.opencontainers.image.authors:Demisto<EMAIL_ADDRESS>org.opencontainers.image.revision:0dd276f6b620092cf4387053d06ac4b7a3d92c44
org.opencontainers.image.version:<IP_ADDRESS>453

Docker Image Ready - Production
Docker automatic build at CircleCI has deployed your docker image: demisto/sane-pdf-reports:<IP_ADDRESS>454
It is available now on docker hub at: https://hub.docker.com/r/demisto/sane-pdf-reports/tags
Get started by pulling the image:
docker pull demisto/sane-pdf-reports:<IP_ADDRESS>454

Docker Metadata

Image Size: 729.36 MB

Image ID: sha256:1ba19f2fa7629ef46197a64cde57acd61ffc5a5279c386716303ca47a1f986d9

Created: 2022-11-15T12:58:03.253437829Z

Arch: linux/amd64

Command: ["python3"]

Environment:

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D
PYTHON_VERSION=3.10.8
PYTHON_PIP_VERSION=22.2.2
PYTHON_SETUPTOOLS_VERSION=63.2.0
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/5eaac1050023df1f5c98b173b248c260023f2278/public/get-pip.py
PYTHON_GET_PIP_SHA256=5aefe6ade911d997af080b315ebcb7f882212d070465df544e1175ac2be519b4
DOCKER_IMAGE=demisto/sane-pdf-reports:<IP_ADDRESS>454

Labels:

org.opencontainers.image.authors:Demisto<EMAIL_ADDRESS>org.opencontainers.image.revision:9d5324679ee1e5f2fae07aa06b012f1964e36ee1
org.opencontainers.image.version:<IP_ADDRESS>454


Next-Pylint for Azure-Core

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-core


Change various CMDs to use std::optional

Desired behavior
LinearVelocityCmds are reset to zero at every time step instead of being removed. This means that once a user sets a Link to have a certain predefined velocity, then it is assumed in all subsequent updates that  the link will have zero velocity (https://github.com/ignitionrobotics/ign-gazebo/blob/77dd8551e942c6f2a9dfbd6360a51cab9185ebab/src/systems/physics/Physics.cc#L3068-L3073). This is somewhat counter intuitive as it is not documented. Furthermore it also prevents users from setting an initial velocity on a certain link which may be useful at certain times for testing purposes (particularly with systems like LiftDrag and Hydrodynamics).
Alternatives considered
Removing the component altogether. This however would have a performance penalty as it may lead to memory deallocation.
Implementation suggestion
Recommend making the following section: https://github.com/ignitionrobotics/ign-gazebo/blob/77dd8551e942c6f2a9dfbd6360a51cab9185ebab/include/ignition/gazebo/components/LinearVelocityCmd.hh#L38-L42
Into:
using LinearVelocityCmd = Component<
  std::optional<math::vector3d>, class LinearVelocityCmdTag>;
  IGN_GAZEBO_REGISTER_COMPONENT(
      "ign_gazebo_components.LinearVelocityCmd", LinearVelocityCmd)

Alternatives considered
Removing the component altogether. This however would have a performance penalty as it may lead to memory deallocation.

I'm actually not sure if this is true - when I worked on #856, I created a "removed components" mechanism which is used internally by the ECM to mimic components being added/removed without actually removing them. The reason why I did this is because I found that the pre-existing approach of actually removing components from memory was a costly operation.  Then, if the user adds this component again, the component data is updated accordingly and is no longer ignored. So, from the user's perspective, the component has been added/removed, but internally, the component is never actually deallocated from memory.
@arjo129 have you tried removing the *Cmd components you don't need after applying the command to see if there's any noticeable performance impact?

when I worked on https://github.com/ignitionrobotics/ign-gazebo/pull/856, I created a "removed components" mechanism which is used internally by the ECM to mimic components being added/removed without actually removing them.

Ah, this is already solved then. That's great! We just need to update the behavior of the physics system to remove the components instead of resetting them. I think that's feasible for Garden.

Ah, this is already solved then. That's great! We just need to update the behavior of the physics system to remove the components instead of resetting them. I think that's feasible for Garden.

this is similar to (if not a duplicate of) #1926 and an implementation attempt is in #2228


removed line coeffs from constraint accumulation

This change is

#643  👈
dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @ohad-starkware and the rest of your teammates on  Graphite

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 92.22%. Comparing base (8d16c06) to head (95cfb98).

Additional details and impacted files
@@            Coverage Diff             @@
##              dev     #643      +/-   ##
==========================================
- Coverage   92.64%   92.22%   -0.42%
==========================================
  Files          69       69
  Lines        8995     8980      -15
  Branches     8995     8980      -15
==========================================
- Hits         8333     8282      -51
- Misses        590      626      +36
  Partials       72       72

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Auto from all sides, started buttons and pneumatics

Wrote 3 different autonomous modes: one for starting in the middle, ones for starting on left/right of alliance wall. They don't work perfectly yet - PIDs need to be tuned. Also updated joystick inversion - teleop should work perfectly. Started joystick buttons and pneumatics code but did not finish. Pneumatic code is commented out.

I'm not sure why you've put all the pneumatics in a single subsystem. It looks like PneumaticsBase is really the gear holder subsystem, so I think you could (and should) rename it to make it less confusing.
Why did you add encoders to the AutoLeftSide command group? They're not being used.

Just for the record, this was moved to #5


I cant search block height

If I use the search criteria "2" it doesn't work.
I receive "Invalid search entry!" message.
As we have one block minute, it'll be a nightmare to search the initial blockchain records.
=)

"block 2" it is.


Improve extension engine compatibility

Currently, an extension is deemed outdated when there's a newer version published in the Marketplace. Installing an extension requires O(published_versions) request roundtrips: starting from the latest version, the engine field is checked for compatibility with the current Code, and it goes on backwards from there.
We should:

[x] Publish extensions with the compatibility value as a property
[ ] Use that property to figure out which version to install
[ ] Use that property to correctly figure out whether the extension is outdated

--
VSIX manifest format:
<Properties>
      <Property Id="abc" Value="def" />
      <Property Id="ghi" Value="jkl" />
</Properties>

Response:
"version": "0.1.15",
              "flags": "validated",
              "lastUpdated": "2016-09-06T04:41:03.74Z",
              "properties": [
                {
                  "key": "abc",
                  "value": "def"
                },
                {
                  "key": "ghi",
                  "value": "jkl"
                }
]

@rebornix just hit the bug that this will fix: an extension which has a new version published, which isn't compatible with the user's version of Code, always appears out of date in the product.

I wonder if this explains what @gerane was seeing in issue #12281?

Probably not, since this is about update notifications.

Not sure. My issue is that code is not able to connect some of my installed extensions to to the ones in the marketplace any longer. I don't get update notifications for them, but not sure if those mechanisms are linked in anyway.
Also, I'm pretty sure this issue was introduced a while ago, but it's not an easy one to catch.

Outdated check for an extension is done as follows:
If the extension has defined engine property in the VSIX manifest, it is used to check its compatibility. Otherwise falls back to old way.


Sanic no longer raises ConnectionClosed on Python ≥ 3.8

Hi,
I've been using this library via Sanic and I've noticed that the connection errors that use to be raised in easier versions are no longer being thrown on calling awaiting websockets.protocol.WebScoketCommonProtocol.recv.
I've not been able to figure out exactly why, as it all "looks correct" from the time I spent looking at it. I'm also not sure if this is an issue from Sanic's side or from websockets, so I've cross posted here - hope that's OK.
Here's a description of problem: https://github.com/huge-success/sanic/issues/1871

This looks like a ramification of https://github.com/huge-success/sanic/issues/1214, which could get solved in a maintainable way once #676 is done.

Hey @aaugustin,
Thanks for the response. Yeah, I can see getting huge-success/sanic#1214 done first makes sense.
I guess I wasn't sure if this issue was actually related to how sanic integrates websockets or something that could be reproduced in just websockets itself.
That it only happens in Python 3.8, and not Python 3.7, is strange. Though if it is a kinda async race condition, then maybe the order of when the coroutines are yielded to changes between 3.7 and 3.8. That is to say that, maybe the integration in is the problem but it happens to work by chance in 3.7.
I'll maybe see if I can better understand what's going on over the weekend.
Thanks for your work on the library.

Clearly, you're onto something, but I'm not sure what yet ;-)
I'm not aware of any changes between Python 3.7 and 3.8 that may explain this.
Based on your reports here and in the sanic repository, I can't tell on which side the bug is.
I would expect that kind of bug to be caught by the test suite of websockets, but who knows!
Needs further investigation.

Could this be caused by CancelledError being changed in 3.8 not to inherit
from Exception?
https://bugs.python.org/issue32528
If you look at one of the referenced issues in this report, it mentions
CancelledError being raised in 3.8 when it wasn’t before.
On Sun, Jul 26, 2020 at 2:04 PM Aymeric Augustin<EMAIL_ADDRESS>wrote:

Clearly, you're onto something, but I'm not sure what yet ;-)
I'm not aware of any changes between Python 3.7 and 3.8 that may explain
this.
Based on your reports here and in the sanic repository, I can't tell on
which side the bug is.
I would expect that kind of bug to be caught by the test suite of
websockets, but who knows!
Needs further investigation.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/aaugustin/websockets/issues/803#issuecomment-664040159,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACW33STDHXX47PSDHJWFWDR5SK4TANCNFSM4PFYEFSQ
.

@aaugustin Independent of this issue, I would recommend auditing your code for places where except Exception is used without separately handling CancelledError. I found such occurrences e.g. in server.py and protocol.py. Those spots will behave differently in 3.7 and 3.8 due to the inheritance change I mentioned above, so it would be good to be explicit in how you want the code to behave in those places.

Thanks you very much Chris!
There's a very good chance this will explain the problem we're seeing here: a CancelledError catched and re-raised as ConnectionClosed on Python 3.7, while it bubbles up on Python 3.8.
Cancellation in asyncio is a gift that keeps giving!

Looking at this from the perspective of the target behavior, where CancelledError is a "special" exception (inheriting BaseException rather than Exception, it seems to me that:
try:
    ...
except Exception:
    ...
    raise

is fine as CancelledError will be re-raised eventually.
I went through all except Exception clauses and didn't find any location where the exception wouldn't get re-raised.
So... 🤔

There's a good chance this will go away with https://github.com/sanic-org/sanic/issues/2000.


Update AZ400_M03_Sharing_Team_Knowledge_using_Azure_Project_Wikis.md

Related Issue
No related issues
Checklist
Mark completed with "x" between brackets, "[x]", or checking the box once the PR is created:

[ ] Has related GitHub Issue 💥 Create Issue 📝
[x ] Tested it
[x ] Read the PR collaboration guide 👓 Collaboration Guide 📝

Changes proposed in this pull request:

Lab 3, Ex. 1, Task 1, Step 3: Changed "main" to "master"
Lab 3, Ex. 1, Task 1, Step 5: Changed "/Documents" to "/"
Lab 3, Ex. 1, Task 2, Step 1: Changed "main" to "master"
Lab 3, Ex. 1, Task 2, Step 2: Changed "/Documents" to "/"
Lab 3, Ex. 1, Task 2, Step 6: Changed "main" to "master"

Submitted by Tattedgeek on behalf of MOC Labs Review

will close PR and open another  one with unrelated issues.


[BEAM-14163] Fix typo in single core per container logic.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @chamikaramj @tvalentyn
CC: @youngoli

LGTM. Thanks.


Add BearSSL extra

Changes

Based on #284 by @stschake, resolved merge conflicts
Update subrepository to latest BearSSL version
Use full certificate chain verification
Add Let's-Encrypt root certificate (IdenTrust cross-signed) to support howsmyssl's trust anchor
Add workaround for missing date & time (use time of compilation for SSL for now as start date)
Add updated certificates for tls_server_bearssl example from BearSSL server example,
previous ones were expired
Verified both examples to work

Testing
Client verification
Example output on serial console at 115200 baud:
ESP-Open-SDK ver: 0.0.1 compiled @ Feb 13 2017 02:55:12
phy ver: 273, pp ver: 8.3

SDK version:0.9.9
mode : sta(a0:20:a6:00:ff:ab)
add if0
scandone
add 0
aid 3
cnt

connected with OpenBeacon.org, channel 3
dhcp client start...
ip:<IP_ADDRESS>,mask:<IP_ADDRESS>,gw:<IP_ADDRESS>
Initializing BearSSL... done.
Connected
HTTP/1.1 200 OK
Content-Length: 1727
Access-Control-Allow-Origin: *
Connection: close
Content-Type: application/json
Date: Mon, 13 Feb 2017 03:05:39 GMT
Strict-Transport-Security: max-age=631138519; includeSubdomains; preload

{"given_cipher_suites":["TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384","TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384","TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDH_RSA_WITH_AES_128_CBC_SHA","TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDH_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA","TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA","TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA","TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA","TLS_RSA_WITH_3DES_EDE_CBC_SHA"],"ephemeral_keys_supported":true,"session_ticket_supported":false,"tls_compression_supported":false,"unknown_cipher_suite_supported":false,"beast_vuln":false,"able_to_detect_n_minus_one_splitting":false,"insecure_cipher_suites":{},"tls_version":"TLS 1.2","rating":"Improvable"}

free heap pre  = 12836
free heap post = 13128
successes = 1 failures = 0
10...
9...
8...
7...
6...
5...
4...
3...
2...
1...
0...
Starting again!

Server verification
Use OpenSSL s_client to verify connection on command line:
> openssl s_client -host <IP_ADDRESS> -port 800 -verify_hostname localhost

Example output on the terminal:
CONNECTED(00000003)
depth=0 C = CA, CN = localhost
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = CA, CN = localhost
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=CA/CN=localhost
   i:/C=CA/CN=Intermediate CA
 1 s:/C=CA/CN=Intermediate CA
   i:/C=CA/CN=Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDPDCCAiSgAwIBAgIUWNq6Ns3toNpcEDNzjgxkknmSrwMwDQYJKoZIhvcNAQEL
BQAwJzELMAkGA1UEBhMCQ0ExGDAWBgNVBAMTD0ludGVybWVkaWF0ZSBDQTAeFw0x
MDAxMDEwMDAwMDBaFw0zNzEyMzEyMzU5NTlaMCExCzAJBgNVBAYTAkNBMRIwEAYD
VQQDEwlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDU
eh0nuis6Z7KRavvng0TK7Rx1rd1Ng2LWqmiVsiQhexWuKploFe1m8LhY59P1LsbZ
Kl7nDi7n/GdZwMhhfUukb92f2ciFh2THuhoPKdSWqHiaa2IgqTLQ7qmMKGFHolAq
Y/Yh3trY1fB/xQCCcOajv1yJJ09RkncDw7DMLjvsI/IvU0GviZP/0oCxQ5fe1hmg
khJ6PWZ54cG84Xdwoos9RoRTP+ROQkE3kh4f/Tiz9++HOYDTVs/04BPeZLBypAOE
xEHtb/o+4soEINLX3CyCK3ribaEcSNvPiU80lz0oqFPa58HhcxWjMHZ/jyNCFD1R
NNJarTyby8j+f26OQPO9AgMBAAGjZjBkMB8GA1UdIwQYMBaAFMUBrXzmY8mcF1/F
oqfhUF/o9ajGMB0GA1UdDgQWBBTFAa185mPJnBdfxaKn4VBf6PWoxjAMBgNVHRMB
Af8EAjAAMBQGA1UdEQQNMAuCCWxvY2FsaG9zdDANBgkqhkiG9w0BAQsFAAOCAQEA
cbNdIcIO19DG+Epzh00iAifQx/j9Gm1iWIIIdiAHwEiS8+mYWusNTlaVY2hNq9QA
duA3zwsRYVlc3valFFnZJZ9Z2dNehqwdpiwyQhkyE0ALVM1nJra9tJakyh9/N9ao
des6gVEwuflKAW/R1u1P3z8wYAZnko5hhV8atYyzD2Gp+t9dxGQA6oexM199y6OF
JG4sZTvqcz+G0/3o5ALGYWomF1IBJVx/qM5pH6xhLLcEr/2kepnLJhVM/3TUcwxX
DCbr1yrcXMNBu8Lzzha9jnv76d+rIQ2Rs43Yz8j0SbnQ4xZwP7Pe1Acl+kZEUolN
icjiyrUzf8chvSjv/mZ0Aw==
-----END CERTIFICATE-----
subject=/C=CA/CN=localhost
issuer=/C=CA/CN=Intermediate CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2143 bytes and written 327 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: EF82CE24AE7A7C8ED88A98A38F395136F997C9A0C41D7A43D19330B54CA0840F
    Session-ID-ctx:
    Master-Key: E269116E9FE6F12862FE2C6A45C9D48BFC2B426367A9604B0329660543E5439DCE5E850FCEB0EE53C7ECBF646CDF1F00
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time:<PHONE_NUMBER>
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
O hai, client <IP_ADDRESS>:52404
Free heap size is 16676 bytes
closed

Thank you, nice contribution!
As @sheinz already said in #284 it's always good to have options to choose from :)

@UncleRus I extended the timer code to continuously progress time. From a functional perspective I believe the code is ready to be merged now.


Fixed a few broken images and removed a few redundant entries

Fixed broken images for:

Rayquaza by Divyansha Sharma
Skarmory by Itsaadarsh

Removed redundant code for:

Pikachu (with mask) by Alok
Mimikyu by chaitanya


Ported token mapping function

Ported mapping function over from previous repo. Made modifications so it works with the HF Dataset classes.
For issue #8.

test looks good!
left some minor comments
please move token_map.py to eval folder and the test to tests/eval/test_token_map.py
then we need a script in scripts/ that takes HF dataset name and produces a pickle
click "ready for review" when that's done and ping me please

@jettjaniak I've updated most things, but still a little confused about some things:

What should I do about the typing on the prompt? Should I ignore the type with # type: ignore, should I keep the cast, or should I do something else?
I only added a basic script that can take any hf dataset string and make a pickle in data/pickle_name.pkl, but should it be more specific? Should it do something else?

let's discuss at check-in


On macOS, opener uses Safari where as the open command uses Firefox

When running code from cargo run in VS Code, the opener::open() call when provided with a URL will open that URL in Safari. If I run open from the command line in the same shell, it opens in Firefox.

That's strange. On mac all this library does to open a path just boils down to this:
Command::new("open")
    .arg(path_you_provide)
    .stdin(Stdio::null())
    .stdout(Stdio::null())
    .stderr(Stdio::piped())
    .spawn()?;

Therefore I don't see how this issue could be caused by this library. You should experience the same behavior manually running the open command like this in your code.

Can't reproduce this now, anyway. Sorry for the noise!


QoL improvements

Unified the path and the filename
Fixed the elapsed time thing
Changed the name changing
Made the selection at the start loop if an invalid option is inserted

Lgtm!

Lgtm!


Unexpected error: Failure("Bound term variable not found (after unmangling)")

The following code results in an unexpected error:
module Bound_termvar

module UInt = FStar.UInt
module U64 = FStar.UInt64

type t = { low: U64.t; high: U64.t }

assume val v: t -> n:nat{n < pow2 128}

let logand (a b: t) : Pure t
  (requires True)
  (ensures (fun r -> v r = UInt.logand (v a) (v b))) = a

output of fstar.exe bound_termvar.v
Unexpected error; please file a bug report, ideally with a minimized version of the source program that triggered the error.

I'm using a recent commit of F* on OS X 10.11:
$ fstar.exe --version
F* <IP_ADDRESS>
platform=Darwin_x86_64
compiler=OCaml 4.04.1
date=2017-06-20T15:49:16+0100
commit=9f13542f9 (dirty)

(the working tree was dirty at compile time due to Homebrew patches)

Seems like a common blow up message, maybe similar to what @cpitclaudel  reported in this comment https://github.com/FStarLang/FStar/issues/933#issuecomment-303765370
Let's use this new issue to track the problem.

A different repro:
$ cat B.fst
module B

let f =
    let x = let rec __f u : unit = __f u in
    __f
    in x
$ ./bin/fstar.exe B.fst
Unexpected error; please file a bug report, ideally with a minimized version of the source program that triggered the error.
Bound term variable not found (after unmangling): uu___#149443 in environment: false_elim, ignore, __proj__Mkdtuple4__item___4, __proj__Mkdtuple4__item___3, __proj__Mkdtuple4__item___2, __proj__Mkdtuple4__item___1, uu___is_Mkdtuple4, Mkdtuple4, dtuple4, __proj__Mkdtuple3__item___3, __proj__Mkdtuple3__item___2, __proj__Mkdtuple3__item___1, uu___is_Mkdtuple3, Mkdtuple3, dtuple3, dsnd, dfst, __proj__Inr__item__v, uu___is_Inr, __proj__Inl__item__v, uu___is_Inl, Inr, Inl, either, invertOption, allow_inversion, inversion, all_trivial, all_null_wp, all_assume_p, all_assert_p, all_close_wp, all_stronger, all_if_then_else, all_bind_wp, all_return, all_ite_wp, all_wp_h, all_post_h, all_pre_h, raise, lift_div_exn, ex_trivial, ex_null_wp, ex_assume_p, ex_assert_p, ex_close_wp, ex_stronger, ex_if_then_else, ex_ite_wp, ex_bind_wp, ex_return, ex_wp, ex_post, ex_pre, ex_pre, __proj__Err__item__msg, uu___is_Err, __proj__E__item__e, uu___is_E, __proj__V__item__v, uu___is_V, Err, E, V, result, st_trivial, st_null_wp, st_assume_p, st_assert_p, st_close_wp, st_stronger, st_ite_wp, st_if_then_else, st_bind_wp, st_return, st_wp_h, st_post_h, st_pre_h, __proj__Mktuple8__item___8, __proj__Mktuple8__item___7, __proj__Mktuple8__item___6, __proj__Mktuple8__item___5, __proj__Mktuple8__item___4, __proj__Mktuple8__item___3, __proj__Mktuple8__item___2, __proj__Mktuple8__item___1, uu___is_Mktuple8, Mktuple8, tuple8, __proj__Mktuple7__item___7, __proj__Mktuple7__item___6, __proj__Mktuple7__item___5, __proj__Mktuple7__item___4, __proj__Mktuple7__item___3, __proj__Mktuple7__item___2, __proj__Mktuple7__item___1, uu___is_Mktuple7, Mktuple7, tuple7, __proj__Mktuple6__item___6, __proj__Mktuple6__item___5, __proj__Mktuple6__item___4, __proj__Mktuple6__item___3, __proj__Mktuple6__item___2, __proj__Mktuple6__item___1, uu___is_Mktuple6, Mktuple6, tuple6, __proj__Mktuple5__item___5, __proj__Mktuple5__item___4, __proj__Mktuple5__item___3, __proj__Mktuple5__item___2, __proj__Mktuple5__item___1, uu___is_Mktuple5, Mktuple5, tuple5, __proj__Mktuple4__item___4, __proj__Mktuple4__item___3, __proj__Mktuple4__item___2, __proj__Mktuple4__item___1, uu___is_Mktuple4, Mktuple4, tuple4, __proj__Mktuple3__item___3, __proj__Mktuple3__item___2, __proj__Mktuple3__item___1, uu___is_Mktuple3, Mktuple3, tuple3, snd, fst, __proj__Mktuple2__item___2, __proj__Mktuple2__item___1, uu___is_Mktuple2, Mktuple2, tuple2, __proj__Some__item__v, uu___is_Some, uu___is_None, Some, None, option, assert_norm, normalize, normalize_term, string_of_int, string_of_bool, abs, min, pow2, pow2, op_Division, op_Modulus, nonzero, nonzero, pos, pos, nat, nat, cut, _assert, admitP, unsafe_coerce, magic, admit, _assume, as_ensures, as_requires, LexTop, lex_t, lex_t, returnM, decreases, smt_pat_or, smt_pat, pattern, pattern, __proj__Cons__item__tl, __proj__Cons__item__hd, uu___is_Cons, uu___is_Nil, Cons, Nil, list, strcat, array, exn, exn, op_disEquality, op_Equality, op_LessThan, op_GreaterThanOrEqual, op_GreaterThan, op_LessThanOrEqual, op_Minus, op_Addition, op_Subtraction, op_Multiply, op_Negation, op_BarBar, op_AmpAmp, int, int, guard_free, pure_wp, pure_post, pure_pre, pure_pre, range_of, labeled, string, string, range_0, range_0, range, range, l_Exists, __proj__Mkdtuple2__item___2, __proj__Mkdtuple2__item___1, uu___is_Mkdtuple2, Mkdtuple2, dtuple2, prop, prop, l_Forall, has_type, precedes, l_not, l_iff, l_imp, l_or, __proj__Right__item___0, uu___is_Right, __proj__Left__item___0, uu___is_Left, Right, Left, c_or, l_and, __proj__And__item___1, __proj__And__item___0, uu___is_And, And, c_and, b2t, eq3, uu___is_HRefl, HRefl, h_equals, eq2, uu___is_Refl, Refl, equals, l_False, l_False, l_True, l_True, squash, unit, unit, uu___is_T, T, c_True, c_True, c_False, c_False, bool, bool, eqtype, eqtype, hasEq, cps, cps, attribute, attribute

(current master, commit 3a7e06292504, F# build)

Another example with the same error:
``
let secTry (s:string): Tot Type0 =
match s with
| _ -> string
type uList (s:string) = (secTry s)
let refUL (#s:string) (tl:uList s) = ST.alloc tl
``

(Surely we noticed before, but note this is an encoding bug)
Yet another example is this one, from #1066
module A

assume val m : Type -> Type
assume val return_m : #a:Type -> a -> m a
assume val bind_m : #a:Type -> #b:Type -> m a -> (a -> m b) -> m b
assume val push_m : #a:Type -> #b:(a -> Type) -> (x:a -> m (b x)) -> m (x:a -> b x)

val push_sum : #a:Type -> #b:(a -> Type) ->
                  dtuple2 a (fun (x:a) -> m (b x)) -> m (dtuple2 a b)
let push_sum (#a:Type) (#b:(a -> Type)) (p : (dtuple2 a (fun (x:a) -> m (b x)))) =
    match p with
    | Mkdtuple2 x y ->
        bind_m #(b x) (* #(dtuple2 a b) *) y (fun (y' : b x) ->
        return_m (* #(dtuple2 a b) *) (Mkdtuple2 x y'))

Notably, uncommenting any or both of the implicit argument annotations cause the example to verify properly.

I've been working on this and made some progress, but it seems nasty.
The main problem is that we're feeding VCs with free variables to the encoding, which blows up. (So it's not an encoding bug, it's a pervasive ill-formed-VCs bug). This happens even for some current regressions tests that pass, just because the encoding doesn't go into the actual problematic subterm.
There are potentially many points that introduce these bad variables. It could be some mistake about the locally nameless repr (I indeed fixed one such issue, but it's not the most common), mistaking the environments for guards (I think this happened in tc_abs), and, most scaringly, at generalization time.
When generalizing, we break the invariant that uvars are closed, for just a few lines, and reduce/compress them away at the end to regain the invariant. The way we do that is by replacing the Tm_uvar node with its solution, which (AFAIU) is mostly needed since substitutions (and hence opening/closing) will not go into uvars, so we cannot keep the Tm_uvar node.
But this is not so easy, as the uvar might have popped up in many many places, and it needs to be replaced by its solution in all of those for our trick to work. One such example is the typechecking guard after generalizing, as fixed in commit 4a1d442801dd580ddc0ff5f810d6bac2d6c25633 (which is ugly, but fixes my B.fst example).
That's not the only one, however, and some other guards still contain some of this offending uvars (Tej's example and Cristoph's in #1215 are of this kind). I couldn't prove it without doubt, but I feel this invariant breaking uvars might be the main cause for this.
An idea to fix this is to have two kinds of uvars, one with are affected by substitution and one which is not. We never use the first kind except for generalization. I'll investigate further and give this a try tomorrow. Curious about what @nikswamy thinks about that, too.
I'll push some progress in branch guido_1091, which might be rebased so don't count on it.

Seems at least A.fst is blowing up due to a failure in type-checking dependent matches, and not related to generalization.

While not related to generalization, it's the same problem for A.fst, even without generalization: uvars resolved to terms with free variables.
I made it a hard failure to assign uvars to free terms, and things started failing very early (prims). After some fixes, I now get much further but it still blows up when typechecking A.fst, seemingly because the imitation solution contains free variables (as Nik noted in #291?). At a high-level I can see the problem scope doesn't correspond to the scope that imitation_sub_probs gets passed as argument. I'm having a hard time making sense out of it though, since I'm not sure what a im_or_proj_t represents.
I'm stopping for a while to do some other stuff.

Thanks for the work on this one Guido. I'll put this one high on my list for next week.

This is now appearing in Vale, after moving from F* commit 9c3cc3626 to b53c191fb427a2d5989ce9963cd761be2f9fe3ce

If this is indeed related to generalization, what happens if we annotate the lemma_poly_reduce with a val declaration?
That should disable generalization for it.

It has a val in the interface, but the problem is not just generalization. I didn't trace this one down, but push_sum is also not generalized and presents the same behaviour.

I'm looking at some of the simpler examples on this thread first, starting with Abhishek's

Makes sense. You might find some commits in guido_1091 useful. Some are fixes and some are about aborting early when things go wrong.

One aspect of the problem here, as Guido points out, is in the inference of logical guards when solving unification problems.
It seems clear that we break the closed-term invariant there in some hard-to-diagnose places.
Now, as for fixing it, I'm a bit reluctant to dive into that code because that is precisely the code that should disappear once c_fstar-build_two_phase_checker branch lands.
In particular, with two phases, we no longer need to compute these logical guards together with solving the unification constraints. Indeed, with two phases, it should be the case that we never have to introduce these unification variables for the guards, since we will only be comparing fully elaborated types to each other.
Note, here is a minimalistic version of Abhishek's example:
module Bug1091
assume val preorder : Type0 -> Type0
assume type uList (s:string) : Type0
assume val mref (a:Type0) (rel:preorder a) :Type0
assume val alloc: #a:Type -> #p:preorder a -> v:a -> mref a p
let refUL (s:string) (t:uList s) = alloc t

It fails in the c_fstar-build_two_phase_checker branch also, but here you can see clearly what went wrong. In the 2nd phase, it's trying to type-check the following term:
let  Bug1091.refUL :
    (#uu___#156968:(Bug1091.preorder (Bug1091.uList s#156944)) //<--- clearly a variable that's not in scope
  -> s#156969:Prims.string
  -> t#156970:(Bug1091.uList s@0:Tm_fvar: Prims.string)
  -> Tot (Bug1091.mref (Bug1091.uList s@1:Tm_fvar: Prims.string) uu___@2:Tm_app)) =
  (fun #uu___#156971 -> ((fun s#156972 t#156973 -> ((Bug1091.alloc #(Bug1091.uList s@1:Tm_fvar: Prims.string) #uu___@2:Tm_app t@0:Tm_delayed) $$ (residual) Prims.Tot (Bug1091.mref (Bug1091.uList s@1:Tm_fvar: Prims.string) uu___@2:Tm_app))) $$ (residual) Prims.Tot (s#156974:Prims.string -> t#156975:(Bug1091.uList s@0:Tm_fvar: Prims.string) -> Tot (Bug1091.mref (Bug1091.uList s@1:Tm_fvar: Prims.string) uu___@2:Tm_app))))
in
()

This second phase is going to catch many such bugs for us. @aseemr what's prognosis for getting this to master?

@nikswamy, working on it, will try to make it usable asap.
To clarify, the current 2-phase mechanism would catch this sort of bug, but the fix would come later when we actually simplify the compiler code to disentangle unification and logical guards etc.

Here's another one from @karthikbhargavan  this morning:
module Spec.Lib.IntBuf
open FStar.HyperStack
open FStar.HyperStack.ST
let deref m a = sel m a

I have a partial fix in nik_1091 that I'm trying to merge into master now.
At least part of the problem is in generalization, where we were incorrectly generalizing variable whose type may not be closed.
I see generalization as being there mainly in support of ML-style polymorphism, which allows implicitly generalizing over variable of type Type only, not variables of arbitrary type. I've put in a restriction now that allows generalization only over variables whose type is a closed refinement of Type, i.e., impliict generalization over Type, eqtype` etc. is supported, but nothing else.
This fixes Tej's original report; Abhishek's report; and Karthik's report.
But, it doesn't fix Guido's module B and A.

I managed to fix B.fst by also removing uvars in the generated guard, and also found a couple bugs in Rel. Nik: should I merge those (as time permits)? Maybe we don't care about the first one due to moving to 2 phases, but the other are needed, I think.
Further I think it would be good to show big fat warnings when some invariants are broken. Maybe even have a --defensive flag, which fails loud and early when things go eerie (such as having open terms). I can take care of adding it if we agree on it.

Merging those changes sounds good, Guido.  And so does --defensive.
Also, @aseemr, as you say, the 2-phase merge will mainly detect bad inference results in its current forms. However, it may also fix (aka mask) some problems in type inference related to the inference of guards. Since, the 2-phase checker will effectively throw away the guard generated at the first phase and recompute guards once a program has been elaborated with all implicits and type annotations filled in.

FYI, from CHANGES.md:

A revision to implicit generalization of types
(Since [commit FStar@e15c2fa4eb4cd7a10dd74fc5532b6cac8e23b4f1])
F* has for a while supported implicit generalization of types in
support of ML-style, let-polymorphism. E.g., let id x = x would
have the implicitly generalized type #a:Type -> a -> Tot a.
However, F* would incorrectly also allow implicitly generalizing
over variables of types other than Type. For example, this program
would type-check, at a rather counter-intuitive type.
type empty (x:False) =
let rec t (#x:False) (n:nat) : empty x = t #x n
let f () = t 0

where, f would be given the type #x:False -> unit -> empty x.
Worse, sometimes F* would generalize over types with free variables,
leading to crashes as in bug #1091.
We now restrict implicit generalization to variables whose type is a
closed refinement of Type, e.g.,
let id x = x has the same type as before;
let eq = op_Equality has the type #a:eqtype -> a -> a -> bool;
etc.
This restriction is a breaking change. For a sampling of the changes
needed to accommodate it see:
 [commit mitls/hacl-star@c93dd40b89263056c6dec8c606eebd885ba2984e]
 [commit FStar@8529b03e30e8dd77cd181256f0ec3473f8cd68bf]

Here's another case of it:
mitls-fstar@df553757de900044c22e302b41d20c5aff554cb9
make -C src/tls IK.fst
The bug goes away with this patch:
$ git diff
diff --git a/src/tls/IK.fst b/src/tls/IK.fst
index 041765f..937cc37 100644
--- a/src/tls/IK.fst
+++ b/src/tls/IK.fst
@@ -270,17 +270,23 @@ val derive:
   ST (derived_key u i v a)
   (requires fun h0 -> True)
   (ensures fun h0 r h1 -> True)
-
+
+let secret_ideal
+  (#u: usage info ii)
+  (#i: ii.t)
+  (#a: info)
+  (t: secret u i a{ ii.honest i}): table u i a = t
+
 let derive  #u #i #a k v =
   let (| pkg, derived_alg |) = u v in
   let a' = derived_alg a in
   if ii.honest (*safe*) i
   then
-    match MonotoneMap.lookup k v with
+    match MonotoneMap.lookup (secret_ideal k) v with
     | Some dk -> dk
     | None ->
       let dk = pkg.create (Derived i v) a' in
-      MonotoneMap.extend k v dk;
+      MonotoneMap.extend (secret_ideal k) v dk;
       dk
   else
     let raw =

The problem seems to be that implicit arguments for the applications for MonotoneMap.lookup and extend are not properly inferred and then it blows up. Indeed, in this case, we should not expect them to be inferred without something like my secret_ideal annotation; but, it should of course fail properly rather than exploding.

I ended up adding the --defensive option, tri-valued with either "no", "warn" or "fail". The default is "no", which doesn't perform any check, since they take noticeable time (CI would run in about 30mins).
It currently just checks that some terms are closed at various points. Running CI with --defensive warn shows hundreds of warnings, some of which seem relatively minimal, so there's some more data we could use.

Are there known small repros for this bug (@protz, is there a small repro for the recent manifestation of it that you saw)?. I tried the examples in the threads above, and they all seem to be either fixed, or giving different errors.
I would basically like to see if 2-phases actually helps in some of those cases.

Here's a relatively minimal one of a uvar getting set to an open term. It doesn't cause the error during encoding, but I think it's just getting lucky.
module FStar.Algebra.Monoid

unopteq
type monoid (m:Type) =
  | Monoid :
    unit:m ->
    mult:(m -> m -> m) ->
    monoid m

let intro_monoid (m:Type) (u:m) (mult:m -> m -> m)
  : Pure (monoid m)
    (requires True)
    (ensures (fun mm -> Monoid?.unit mm == u /\ Monoid?.mult mm == mult))
=
  Monoid u mult

let lift_monoid_option (#a:Type) (m:monoid a) : monoid (option a) =
  let mult (x y:option a) =
    match x, y with
    | Some x0, Some y0 -> Some (m.mult x0 y0)
    | _, _ -> None
  in
  intro_monoid (option a) (Some m.unit) mult

Running with --defensive warn gives:
ulib/FStar.Algebra.Monoid.fst(21,15-21,22): (Warning 289) Internal: term is not closed (solve_prob').
t = (fun a m x y x0 y0 -> (*?u65*) _ a m x y /\ Prims.l_True)
FVs = ((y:FStar.Pervasives.Native.option a), (x:FStar.Pervasives.Native.option a), (m:FStar.Algebra.Monoid.monoid a), (a:Type))

ulib/FStar.Algebra.Monoid.fst(22,14-22,18): (Warning 289) Internal: term is not closed (solve_prob').
t = (fun a m x y uu___2832 uu___2833 uu___2834 -> (*?u182*) _ a m x y /\ Prims.l_True)
FVs = ((y:FStar.Pervasives.Native.option a), (x:FStar.Pervasives.Native.option a), (m:FStar.Algebra.Monoid.monoid a), (a:Type))

Verified module: FStar.Algebra.Monoid (275 milliseconds)
All verification conditions discharged successfully

The two phase checker doesn't seem to make a difference.

A smaller one:
module A

let f : option unit -> option unit  =
  let aux =
    function
    | Some () -> Some ()
    | x -> None
  in
  aux

A.fst(7,11-7,15): (Warning 289) Internal: term is not closed (solve_prob').
t = (fun uu___12 x uu___111 -> (*?u1568*) _ uu___12 /\ Prims.l_True)
FVs = ((uu___12:FStar.Pervasives.Native.option Prims.unit))

Verified module: A (353 milliseconds)
All verification conditions discharged successfully

It seems to stem from the variables in the patterns, if the last one is changed from x to None, we don't get the warning. Variable uu___12 is being abstracted but with a different index, so this might be that we're not closing the terms properly. I can take a look later.

I get the same error on the following code. I was originally trying to prove mem x xs <==> mem x (reverse xs), and managed to minimize the code to this while still getting the error:
module Bug
open FStar.Tactics

val mem: #a:eqtype -> a -> list a -> Tot bool
let rec mem #a x xs =
	match xs with
	| [] -> false
	| hd :: tl -> if x = hd then true else mem x tl

let mem_sanity #a x xs =
	assert_by_tactic (mem x xs <==> mem x xs) idtac

Thanks @dranov, the repro is appreciated. Here's another one from @cpitclaudel:
let rec ackermann m n =
  if m=0 then n + 1
  else if n = 0 then ackermann (m - 1) 1
  else ackermann (m - 1) (ackermann m (n - 1))

At least ackermann is fixed in my branch nik_rel.
It fails rightfully with
ack.fst(14,39-14,40): (Error 19) Subtyping check failed; expected type n: Prims.int{%[ m - 1; n ] << %[ m; n ]}; got type Prims.int (see also ack.fst(13,2-15,46))

Since F* can only infer that m and n have type int, and the ordering < on ints is not well-founded.
This variant succeeds:
let rec ackermann (m:nat) (n:nat) =
  if m=0 then n + 1
  else if n = 0 then ackermann (m - 1) 1
  else ackermann (m - 1) (ackermann m (n - 1))

All these repros work and, as far as I know, we haven't observed this in a long time. Closing..


Fix internal port check when other ports are opened as well on the target container

This PR contains 2 commits. The first one is a trivial cleanup of a redundant execCreateCmd call. The second commit contains the actual fix. Description from the second commit:
Allow leading zeros in /proc/net/tcp* when doing internal port listen check

When at least one process is listening for a port that is significantly
higher then the port we're checking, this check will fail as the checked
port will contain leading zeros in the output of "cat /proc/net/tcp".

This commit fixes this by changing the grep to ":0*XXX".

Hi @codablock,
Could you please add tests for it?

@bsideup I added 3 commits to this PR.

A fix for the initial fix (I can later squash it into the original commit if you want)
A refactoring of existing tests for InternalCommandPortListeningCheck
A new test to test the fix. You can verify that it fails by reverting fix commit.

Handled review comments. I also noticed that I forgot to commit an important change in nginx_on_8080.conf (port 100), which I fixed now and squashed into the original commit

Test failures are not reproducible locally. And it's unfortunately the new test that fails, so it's likely related. It's also strange that they succeed on circleci. Any ideas?

@codablock I just triggered a re-run, let's see how it goes

Tests failed again. But I think I found it: The check.call() is probably done slightly before nginx has started up, resulting in the test to fail. I added a commit that waits for up to 5 seconds for the check to pass.

The port check tests are green now. Other test failures on Azure Pipelines are unrelated (Check failed: Docker environment should have more than 2GB free disk space)

@codablock it seems there was an issue with the disk space on Azure, I've submitted https://github.com/testcontainers/testcontainers-java/pull/2376 to fix it. Let's update your PR and do another run once that PR is merged to ensure that this change does not bring any side effects 👍

@bsideup rebased and force pushed to bring in #2376

@codablock merged, thanks! 🎉

This was released in https://github.com/testcontainers/testcontainers-java/releases/tag/1.14.0 🎉
Thanks for the contribution!


Adding support for certificate based credentials for app authentication

The implementation is inspired by the implementation of client_secret_credential.

@heaths Thanks for reviewing the PR and providing feedback. I have added an example and fixed the comments.

I have added an example. If there is some test infra i am happy to write a test aswell.

Can this be an optional feature?  There are occasions where linking against OpenSSL is a significant hurdle.

The error i see is in security_keyvault crate which i have not changed (but imported). Do i need to fix it?

@msabansal , pr #719 will address the issues you're experiencing.


WIP / FEAT: Dual certificate support (eg ECDSA with RSA fallback)

Since Postfix 3.4 (Feb 2019), smtpd_tls_cert_file and smtpd_tls_key_file have been deprecated in favor of smtpd_tls_chain_files which supports a list of values where a single or sequence of file paths provide a private key followed by it's certificate chain.
1st commit - DRY logic, subtle changes/improvements, small bugfix
The first commit refactors the relevant code to support this settings change in Postfix, while also moving common logic for each TLS setup variation into a function.
sed lines were also revised as they often contained unnecessary options(-e, -r) and regex tokens/syntax(g), or specific paths that were better extracted out as variables or unnecessary for a match(use .* instead).
The self-signed case had an additional -combined.pem file used instead of the -cert.pem for Dovecot's ssl_cert value. This didn't make much sense (unless there's a known reason for doing this), the -combined.pem is a bundle file containing the private key followed by the certificate, I see no reason to support that so I've replaced it with the same -cert.pem that Postfix uses for it's certificate. This makes it more consistent with other cases.
self-signed Dovecot paths to apply sed to were also invalid due to whitespace within the file paths AFAIK.
The custom case also differed, in that it provides a similar key+cert bundle as the self-signed -combined.pem, this one is -full.pem which I've not inspected but assume contains a full certificate chain (leaf + issuer chain), which is useful for including CA certs in a chain that don't belong to a systems trust store.
An additional bonus of this refactor is that previously a docker-compose up when exited, persisted the container's internal filesystem state which would be used by a follow up docker-compose up, which caused unexpected behaviour / breakage in the event that you changed from self-signed to manual for example or if HOSTNAME were to change. docker-compose down was required to properly clear this state to a "fresh" idempotent state. That issue may be present elsewhere in the codebase btw.
Postfix main.cf also had a slight consistency fix to maintain spaces between = assignment like the rest of the config file and related configs such as Dovecot use.
I've not tested each case personally, only the manual case. I'm not deeply familiar with the codebase or the bats tests, this PR has been validated to work for manual, it might benefit from an extra bats test covering the dual cert support?
2nd commit - Dual cert feature
smtpd_tls_chain_files allows for multiple key+cert bundles so that you can provide different key types, such as ECDSA and RSA.
To maintain compatibility with the current SSL_CERT_PATH and SSL_KEY_PATH ENV variables only a 2nd certificate is supported.
Since Dovecot 2.2.31 (June 2017) a related feature is also available, but it is limited to only providing one alternative certificate via separate cert and key settings. That limitation is a non-issue as this feature is only supporting a 2nd alternative certificate via SSL_ALT_CERT_PATH and SSL_ALT_KEY_PATH ENV vars.

TL;DR
This feature enables support for multiple certificates, eg for serving modern ECDSA cert with an RSA cert as fallback.

I needed this for testing my active PR. (which I'm already using locally, but figured was worth polishing up for a separate PR)
I presently do not have time to learn about bats and how to implement any tests required for this PR, any assistance would be appreciated regarding that, even if it's just some guidance on what needs to be tested and roughly how to go about achieving that.

Looks very nice. Keep on this good work.
A word about the current situation with the test: They are being refactored and rewritten in #1613 as this is absolutely necessary. Therefore, I cannot recommend writing any more tests before this PR has been merged.

Will that delay the merging of this feature? If so no worries :)
I'm not particularly great at naming things, I've used fullkeychain a few times to describe a file with private key + certificate chain(leaf + issuing certificates from CA) - I've seen this called -full.pem, -combined.pem, fullchain.pem (this one seemed to only be the certificate chain I think without a private key), no clue if there is a proper jargon word for referring to it?
cert_chain can just be the leaf certificate, or may contain multiple additional certificates such as intermediate and root certs from a CA.

Will that delay the merging of this feature? If so no worries :)

Not if you ask me :)

I'm not particularly great at naming things, I've used fullkeychain a few times to describe a file with private key + certificate chain(leaf + issuing certificates from CA) - I've seen this called -full.pem, -combined.pem, fullchain.pem (this one seemed to only be the certificate chain I think without a private key), no clue if there is a proper jargon word for referring to it?

I guess it's fine. But have a look at polarathene#1 :D

They are being refactored and rewritten in #1613 as this is absolutely necessary.

The current situation is centered mostly around getting all tests with their setup into .bats files instead of the makefile. Changing existing .bats files is absolutely fine with this PR. Changing tests.bats might light to some conflicts but I think if it is only addition of new test cases, they should be easy to resolve. In summary: No need to compromise the quality by leaving out tests. :)

Thanks for clarifying @martin-schulze-vireso. If this is the case, a test or two won't be of harm - they should be added:)

I welcome any help with sorting out tests for this then, but I'll try take a shot at it if no one can spare the time to.
For now my spare time is allocated to the security update PR, slowly piecing together my review.

I'm currently working on another k8s deployment, I'm sorry. Take your time.

Just occurred to me that there should probably be some logic to "reset"/disable the alternative config if the ENV var is removed. It'd only affect Dovecot AFAIK, but should be reproducible with:

docker-compose up
CTRL+C
Remove ENV var
docker-compose up
docker-compose exec <service name> bash
The dovecot config for alternative cert/key would not be commented/disabled, and the referenced file that was copied from docker volume or bind mount would probably still exist until docker-compose down.

@polarathene This is just FYI: I was going through your commits when I recognized that they are not signed. If you'd like to improve on this, you can have a look here https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/signing-commits (improving on authenticity). This is just a recommendation from my side:)

Because lately we've been seeing a lot of PRs, I will mark this as a draft too. Just mark it ready for review when it's done.

Closed due to migration to docker-mailserver/docker-mailserver. Please re-open over there :D

Closed due to migration to docker-mailserver/docker-mailserver. Please re-open over there :D


Bump version (v1.13.0-rc.1)

@miherlosev, we have a green result here: https://github.com/DevExpress/testcafe/pull/6056#issuecomment-801114980. This PR can be merged.


add a margin centering modifier

margin: 0 auto;

I'd recommend
margin-left: auto;
margin-right: auto;

instead, since this isn't relevant to the top/bottom


Alias always generates same stack

Hi there,
Serverless version: 1.17
For some reason I am always generating the same stack no matter what Alias I put on. I followed the installation instructions and it seems the Alias plugin is giving some feedback while deploying, but its always deploying / overwriting the same content with the same endpoints
I created first the first stage deploy
serverless deploy
And then I tried to create aliases for v1
serverless deploy --alias v1
and v2
serverless deploy --alias v2
But I always end up overwriting the API Gateway, Lambdas and everything
POST - https://{same_endpoint}.execute-api.eu-west-1.amazonaws.com/...
More info

In my lambda console I only have one version of the lambdas, I would expect to have one version per alias, is that right?

Reading your<alias_here> I understand that API Gateway should contain as many initial resources as aliases and then the master API hanging from there although it doesn't, but this is just a guess

/
  /v1
    /my
      /master
        /api
  /v2
    /my
      /master
        /api

In my .serverless folder I have 2 new files:

cloudformation-template-update-alias-stack.json
cloudformation-template-create-alias-stack.json
Both containing information only about the last alias deployed, in this case v2
Any suggestion?

I did the following with the serverless-aws-alias-example project.
git checkout master
npm install
serverless deploy

git checkout authorizer
npm install
serverless deploy --alias=auth

Cloudformation
The deploy created the main and the alias stacks correctly:

Nevertheless the message emitted by Serverless does not show the actual alias endpoints but this is only a console output issue in SLS. I can try to fix that in the plugin, so that the correct URLs are shown.
endpoints:
  GET - https://ai49k1bvja.execute-api.us-east-1.amazonaws.com/dev/func1
  POST - https://ai49k1bvja.execute-api.us-east-1.amazonaws.com/dev/func1/update

  GET - https://ai49k1bvja.execute-api.us-east-1.amazonaws.com/dev/func2

Lambda
The Lambda functions are also deployed correctly, and the aliases are set to the correct versions. As an example I show the testfct1 which is available in master and authorize (resp. dev and auth aliases).

API Gateway
Also the APIs have been deployed correctly for each of the deployed aliases. If you have a look at the deployed stages in API Gateway, you see, that even the defined endpoints (which are different in the master and authorizer branches) are deployed correctly.

Example
You can access the endpoints that are deployed to the different aliases with:
https://xxxxxx.execute-api.us-east-1.amazonaws.com/auth/...
https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/...

The endpoint in a specific alias will automatically call the correct aliased Lambda function version that is attached to the endpoint in your ' serverless.yml`.

Hi there,
Sorry for not answering before.
You are right, your plugin works great, the output from the deploy was misleading since I didn't saw any changes.
Thanks

This helped a lot. Thanks.

@HyperBrain from the example you showed above:

One single AWS lambda resource called "dev-sls-test-project " with two different versions tagged with different names:

one called "dev"
one called "auth"

One API Gateway resource is created:

...with the API Gateway "dev" stage (i.e. "https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/...") pointing to the "dev" version of the "dev-sls-test-project" lambda
....and the "auth" stage (i.e. "https://xxxxxx.execute-api.us-east-1.amazonaws.com/auth/...") pointing to the "auth" version of the "dev-sls-test-project" lambda

My question would be:
Is it possible to have two separate AWS lambda resources instead of one AWS lambda resource, but still being able to share the same API Gateway by utilizing the API Gateway's "stage" feature as you did in your example?


Doc: Reimplement cross-docs links after initial target files for snmp are generated

Related issue: #2
Related PR:

#1
#56
#60

We can't create files and link to the generated output of those files in the same PR. We removed some headers and cross-docs in preparation for the initial merge and publish (#1).
Backstory

"These links should be removed for the initial commit. Why?
The generated files were targeting don't exist yet. (We're merging the original source files as part of this PR.) We can't link to those generated files until they've been created, merged, published, and generated. Linking to non-existent files will break docs-ci, and if merged, will break the entire docs build."

This issue tracks reimplementing headers and links after target files have been generated and merged.
ToDo

[x] Search and fix instances in which we've commented out links to generated files--plugin headers, for example.

Example: https://github.com/logstash-plugins/logstash-integration-snmp/pull/1#discussion_r1486762407

[ ] Search and fix instances in which we've hard coded plugin names to remove offending links.
[ ] Implement a link in plugin "migration" banners to migration section in imtegration index page.
Related: #15
[ ] Next round: Add links to MIBs after first round of generated content is merged

Final fixes:

[ ] https://github.com/elastic/logstash-docs/pull/1677
[ ] https://github.com/elastic/logstash-docs/pull/1675


Fix the Slack notification channel name in build workflows

Fixes #277

The misconfigured channel name my not be the only problem, but it needs fixing in any case and we'll see if the notifications start arriving after that.

We received a Slack message in #pam-feeds from the docs build triggered by this branch:

Seems like a god sign!


Build only one universal wheel when running tox

Sadly, hatch-autorun and tox seem to be fighting over env variables so we create the pth file by hand.

OK, this is interesting. @gaborbernat it looks like SETUPTOOLS_SCM_PRETEND_VERSION is not passed through when using the shared build environment despite being in [testenv]'s pass_env.
I couldn't find a way to affect env variables for it (which is why I gave up on hatch-autorun that needs one) – any advice? Would you consider this a bug in tox?

OK, this is interesting. @gaborbernat it looks like SETUPTOOLS_SCM_PRETEND_VERSION is not passed through when using the shared build environment despite being in [testenv]'s pass_env.
I couldn't find a way to affect env variables for it (which is why I gave up on hatch-autorun that needs one) – any advice? Would you consider this a bug in tox?

https://tox.wiki/en/latest/upgrading.html#packaging-configuration-and-inheritance ?

Note the new config subcommand tox c -e py311 -k pass_env set_env that would have allowed you to inspect these settings.


Math library header files are not exported

When newlib, libmcs, or openlibm is selected as the math library,
the required public header files are not exported by make export.

CONFIG_LIBM_NEWLIB
CONFIG_LIBM_LIBMCS
CONFIG_LIBM_OPENLIBM

Refer to the following, these header files should also be exported.
https://github.com/apache/nuttx/blob/11b3adf17fd4e252d880bd9f4dcab54e42e990fe/tools/Config.mk#L660-L675
Alternatively, it is better to link the respective include folders under nuttx/include as well as libcxx.

@xiaoxiang781216 do you think it was caused by separation of libm to support more libm alternatives?

yes, the header files provided by new alternatives need copy when "make export".


Will not update on OSX High Sierra

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.01.27. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[ ] I've verified and I assure that I'm running youtube-dl 2018.01.27

######################################################################
BashPrompt>youtube-dl -v -U
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-U']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.01.21
[debug] Python version 2.7.13 (CPython) - Darwin-17.4.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.4.1, ffprobe 3.4.1, rtmpdump 2.4
[debug] Proxy map: {}
Traceback (most recent call last):
File "/Users/Pudnik/bin/youtube-dl/youtube_dl/update.py", line 45, in update_self
newversion = opener.open(VERSION_URL).read().decode('utf-8').strip()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/Users/Pudnik/bin/youtube-dl/youtube_dl/utils.py", line 1089, in https_open
req, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
URLError: <urlopen error [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)>
ERROR: can't find the current version. Please try again later.
########################################################################
What is the purpose of your issue?

[X ] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
Description of your issue, suggested solution and other information
youtube-dl for OSX throws update error on OSX:
BashPrompt>youtube-dl -v -U
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-U']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.01.21
[debug] Python version 2.7.13 (CPython) - Darwin-17.4.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.4.1, ffprobe 3.4.1, rtmpdump 2.4
[debug] Proxy map: {}
Traceback (most recent call last):
File "/Users/Pudnik/bin/youtube-dl/youtube_dl/update.py", line 45, in update_self
newversion = opener.open(VERSION_URL).read().decode('utf-8').strip()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/Users/Pudnik/bin/youtube-dl/youtube_dl/utils.py", line 1089, in https_open
req, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
URLError: <urlopen error [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)>
ERROR: can't find the current version. Please try again later.

Most likely https://github.com/rg3/youtube-dl/pull/14353.

I think a lot of people are going to have trouble finding the workaround instructions.
(I also wonder whether the advice to put the homebrew python in the users' PATH ahead of the system python is wise).
Unfortunately I don't have any good ideas on how to address this all. Other than if the error message had printed the updater URL, it would have been a place to offer guidance… But this precise situation seems unlikely to recur in the future? Maybe?

Is falling back to http if https fails an option?  (maybe after prompting the user)

Well, no existing user is going to see any fixes to this problem because the updater doesn't work, so even if that were acceptable...maybe instead give a link to a "What to do if the update fails?" page?
Although I suppose that Apple will probably push an update for openssl as more things in the OS break, and then this will work again…until the next time?
I guess it's worth addressing for people who are newly installing youtube-dl though?

I don't think this is due to a recent bug on Apple; in fact I encountered this bug on a Linux machine I haven't upgraded in years (mea culpa), so this may be due to a change in github.io SSL protocols (this might be related to https://github.com/blog/2498-weak-cryptographic-standards-removal-notice).  I guess eventually all machines that are regularly updated will find a fix for this SSL issue, but if Apple hasn't fixed this yet, I don't think it will do so soon.

@Slowly How did you install Python on your machine?

How did you install Python on your machine?

I know you didn't ask me, but I saw this problem under MacOS 10.10 (Yosemite), which is somewhat long in the tooth. With the system Python, i.e. no special installation. For me:
[debug] Python version 2.7.10 (CPython) - Darwin-14.5.0-x86_64-i386-64bit

(And yes, installing a newer version of Python2 and putting it ahead in the PATH does "solve" the problem.)

Here's how I think I'm going to fix it for me, using MacPorts. IMO, if you don't understand the terminal commands below...you don't belong on github. Github is for the tech savvy only.  I have updated youtube-dl in both environments, and it works great.


[LazyImage] Abstract image content fetching

Q
A

Bug fix?
no

New feature?
yes

Issues
Fix #1773

License
MIT

Hi,
This PR is a proposal for #1773, logic to fetch content image is now customizable thanks to closure, meaning that an end-user can use whatever solution he wants to fetch the image content (HttpClient, Flysystem, ...).
I'm not 100% about the Closure::fromCallable(new Reference($config['fetch_image_content'])) tho

- $this->fetchImageContent ??= fn (string $filename) => file_get_contents($filename);
+ $this->fetchImageContent ??= file_get_contents(...);

Reviews have been applied, thanks :)

Thank you Hugo.


Docker icon

The docker icon is missing. Could you point me to some instructions on how to add it? PR will follow.

bump

+1

Could you do me a favour and review #2 before I merge it to master? I've submitted the underlying release to CTAN but it would be nice to give the structure of this repo a little extra thought. Maybe you prefer to have the .sty and the example separated from the generator etc. :)

Hi Kevin,
If you want I can open a pull request and merge my current project which is actually based in your project.
Actually, I did a pure bash generator so we could have the sty build automatically with no externa script dependencies. Also, I included the official font-mfizz repo as a submodule so we can always build it whenever they release a new version
Check it out here https://github.com/codextremist/latex-fontmfizz

Hi Felipe,
ah the beauty of FOSS 😃  If you're too lazy, someone else will do it, nice! Now I almost feel like I wasted the afternoon yesterday. 😄
Seems like I had the same idea with including their repo as a submodule. I agree that having Python and especially jinja2 as a (at this point even implicit) dependency is not ideal.
Now that the CTAN release is done anyway, maybe we can take a step back together and think about how to separate the concerns. As far as I see it there are two use cases:

Someone wants to just use the package but for some reason doesn't get it from CTAN. (Is this even a likely scenario?)
Someone wants to build their own version because the CTAN one has once again become outdated.

My original idea was to have a repo that is a 1:1 reflection of what's available in CTAN. Now with the (two) generator(s), there are a bunch of possible ways:

Have only the generator in GitHub, basically like your repo, which can build the .sty from scratch. Ideally automated and/or without external dependencies. And ideally also with a quick way to generate a CTAN release.
Have the CTAN release in the master branch and a dedicated branch for the generator. Discoverability might be a bit low here but it gives a clear separation of concerns.
Basically 2. but with two separate repositories.
Something else that I'm missing?

Let me know what you think. I'd be more than happy to ditch my Python-based solution if it means people will be able to generate the file with less hassle.
Cheers,
Kevin

Hi Kevin,
I'm not familiar with CTAN, actually I have never worked with CTAN before. How they manage new releases and versions? Can we release the packages automatically or do we need to perform any manual steps?
One additional step to the ones you already added is asking the font-mfizz administrator to add a Github Webhook to us, so we know there is something new in their master branch. That would be the trigger for our build script to generate the sty with new icons (and possibly push it automatically to CTAN)

AFAIK, CTAN is the central (La)TeX package repository. If I understand it correctly, it's where tlmgr gets its packages from. So for example after fontmfizz was released on CTAN the first time around, a few days later by running sudo tlmgr update -all you'd get it into your local TeX distribution.
Their release process is anything but automated. There's an online form that one needs to fill out and the uploaded zip will be manually processed by a human. (If we want to go crazy, it might be possible to send that form automatically but they might not allow cross-site requests to prevent spam…)
I like the idea of asking for a webhook on font-mfizz. Setting up Travis to automatically produce a release (at least as far as GitHub is concerned) should be easy. Uploading that manually to CTAN is not a lot of work :)

:+1: for travis to build things. Not sure about the submodule and the webhook. submodules seem to always cause pain (in the long run) and for the webhook I think if travis builds everything then all that is needed for a new version is that someone submits a PR updating the version number in the docs etc and then travis builds a new release. One thing to investigate is where could travis upload the files it creates? On CircleCI you can store build artefacts that are accessible after the build completes (-> download the zip, upload to CTAN), not sure what travis has.