Document Fetch, Access and related types for bevy_ecs

Objective

These types are important to bevy_ecs devs and not self-explanatory.

Solution

Document them!

By @DJMcNab from #3366 on Archetype.

With sparse set components, these can contain many sets of component, although each set is uniquely assigned to a single archetype.

I'm not entirely sure how to explain this cleanly, so help would be appreciated.

Closing; I don't care enough to fix the merge conflicts. These can be rewritten in further PRs.


Keyboard flicker

Still occurs when using CrossWalk.

Open keyboard on Android and there is a flicker.

If you remove the keyboard plugin does the flicker go away?

Just following up because I was playing around with this plugin this morning. I tested with the plugin installed, and without it installed and I still get a keyboard flicker.

Yeah I'm pretty sure it doesn't have to do with the plugin, considering the plugin is just passively listening for layout changes and doesn't actually do anything to the view layout. Thanks for the follow up @mattchue! Let us know if you figure out how to get rid of it, since it seems to be a common issue.


Render cells with the same foreground & background color differently when selected

Some terminal emulators (e.g. urxvt) do this. This allows you to read the text in the cell by selecting that cell, even if the foreground/background are the same. In practice this is useful for writing spoilers into social chats in terminal-based IRC channels.

IIRC Alacritty actually followed URxvt in some combinations while following VTE in other combinations of the hidden flag and I've streamlined it to be identical to VTE's behavior since it seemed most consistent.
However I haven't considered using this as a spoiler tag and I'm not sure how much sense this makes.
In general I'd say that if something is hidden, it's supposed to be hidden and not visible so it should stay hidden and not be visible only when the text is selected. It's also inconsistent and would be a special case.
On the other hand, there's no point in printing invisible text if there's not at least some way to use that text and make it visible again.
The implementation itself would be trivial, so this is mostly a question of whether this is desired or not. I wouldn't be opposed to putting this behind a configuration flag either, since it seems like a nifty little feature, but even then the default would have to be considered and I'm not sure if people are interested in changing it once a default has been decided upon (what reason would there be to disable showing hidden text on selection for example?).
I'm curious how other people feel about this, maybe @nixpulvis has an opinion? I think in general I'm inclined more pro than con, since it seems mostly like an advantage without disadvantages (other than potentially surprising the user), but I don't really care either way personally.

On the other hand, there's no point in printing invisible text if there's not at least some way to use that text and make it visible again.

This alone is enough reason I think. I can't think of a practical use case where it isn't true.

I've also talked to a few more people in IRC and I think this would be a good addition. So I think the primary question now would be how to implement it. So which color should be used.
URxvt acts like same color fg and bg is the default text background on default text foreground when inversed. So the foreground is the default bg and the bg is the default foreground.
It would be possible to just reset one of the colors back to the default, however in those cases there would still always be a scenario where the text would be invisible. I think using the default bg as fg and the default fg as bg is the only way to reliably make same-color text and background visible.
URxvt applies this behavior both when selected and when the inversion escape is used (so echo -e "\e[7;31;41mTEST" will result to be the same as selecting same-color fg and bg). And if this is inverted again using the selection the visible colors are not inverted but it's inverted back to the same bg on same fg colors. So Reverse and Selection do the same thing and are XORed together to check whether they should be applied. I think this makes sense and is what would be expected.
URxvt does not seem to implement the hidden escape sequence (echo -e "\e[8mHIDDEN"), so I'm not sure what we should do here. There's two choices, either we act like it's same fg on same bg and make it visible during selection, or we say that when things are hidden they'll stay hidden no matter what. XTerm does not show same bg/fg during selection and Termite (VTE) and Kitty both seem to have fixed selection fg/bg colors without supporting inversion (the reverse escape escape does not have special handling) and Kitty also does not support the hidden escape. According to some helpful people on IRC Konsole does not support inverting during selection either when bg/fg is the same.
So as far as I can tell, we'd be the first to make a decision about this and I'm unsure which decision would be "correct" here. In a way, it might be better to keep text which is intended to be hidden, hidden. So a stray inverting escape sequence can't accidentally expose passwords etc. On the other hand it would probably be more consistent to always show hidden text during selection, no matter what caused that text to be hidden.
Since this issue seems to suggest that same bg/fg is mainly used for spoilers and not the hidden escape sequence, which means I'm not aware of any advantage to show the hidden text, I'd be for keeping hidden text hidden. However I'd be very curious what other people think about it. If the sequence is used to hide passwords, the selection could be used as a "show password text" kinda thing similar to what is sometimes available with password fields in GUIs too.

 If you, for example, record your terminal with asciinema under these circumstances, you're going to expose your passwords. Alacritty wouldn't be the first terminal to expose that kind of fuck-up. I don't know of any software that does this rather than disabling echo.
In terms of the specific behavior to use, using the default colors inverted seems like the correct approach to me.

Password I put should ideally output the Unicode Character 'BLACK CIRCLE' (U+25CF) IMO.

I also do not believe that's a great idea, however I do think that this is (at least part of) the intended purpose of the "hidden" escape sequence. However it is not a password input escape sequence, but a Hidden/Conceal/Invisible escape. So text should just not be visible (however it should still be printed and copy/paste-able).
Alacritty can't control how this escape is used, however I wanted to mention that it might be used for passwords since it could be. I'm not aware of any application which makes use of this escape though so it's difficult to tell what people use it for in practice and Wikipedia also lists it as uncommon.
So we still need to make a decision about what colors should be chosen here when hidden text is inversed.

Not saying it's worth the complexity in the rendering, but I wanted to propose this anyway since it's somewhat novel. We could style the text somewhat differently.

Note that as I've mentioned, the hidden escape sequence is not a password input escape sequence. It might be used for secrety stuff, but it's a general-purpose escape sequence intended to hide text from plain view.
So the question is not how to display passwords, but if the hidden escape sequence should be treated like background color text rendered on top of the background, which would mean inverting it makes the text visible, or if it should always stay invisible only allowing it to be selected and copy-pasted.
These are the only two options available. Everything else doesn't make sense in this context and is not following the purpose of these escape sequences.

I'd like hidden text to be visible when selected. As has already been agreed, it is not for passwords and other security related things. As such, I think it makes sense to treat it as the same-background/foreground case.
That said, I'm not aware of any applications that make use of the hidden attribute, which makes it difficult to make a strong case.
Though I guess this can be said: if support for the hidden escape is unusual (in terminal emulators), and if I as an application developer wanted to use it, I'd use same bg/fg as fallback.
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Since there does not seem to be a clear solution to this problem, I'd like to bring up that currently hidden cells are just not rendered at all. They're replaced as empty cells during rendering (so selection and copying still works), instead of hiding them by using the background color as foreground.
As such,  an implementation completely ignoring the existence of the hidden flag would keep hidden cells hidden, even if same bg/fg are made visible.
Making the hidden cell visible on selection would also not allow to print any text that wouldn't be visible during selection, while both options would exist when the cell stays hidden (one could just use same bg/fg if it's desired). In my opinion this is probably the strongest argument for keeping them hidden.
It's also noteworthy that they are called hidden/invisible/concealed, so their name would suggest to the users their content can never be seen.
Because of these points, I'd propose just ignoring hidden cells for the implementation and keeping them hidden even though the text is revealed when foreground and background is identical. If this should ever turn out to be a mistake, we can still easily adjust in the future.

I guess my only remaining question is if we don't display it, should we really copy it?

Try copying things in general first :wink:
I'd suggest copying same-colored text, but not hidden text.

should we really copy it?

This is an escape sequence aimed at changing the styling of text, so I would still copy it to the clipboard since the text still exists. Otherwise there wouldn't be any reason to use the sequence over just not printing text.
This would also allow us to have all the functionality available in all combinations. So text can be visible on selection (when same bg/fg), it can be invisible on selection but still copy-able (hidden escape) and it can be completely gone (don't print any text).

This behavior should be changed in https://github.com/jwilm/alacritty/pull/2349.
I think I've captured all variations to behave consistently.

Thanks!

Thanks!


Add support for transforms

Specifically translation, rotate and scale: https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Transformations

I'm going to give this issue a shot.

Closed by landing #4 😄


Adding Badges to Head of Readme

Signed-off-by: Gerardo Roa Dabike<EMAIL_ADDRESS>Closes #146
PR is adding badges below the Clarity and Cadenza figures:
The current badges are:

First row:

Last pip version
Python versions. The versions are obtained automatically, not sure yet how to specify a specific version like   3.7 | 3.8 | 3.9 | 3.10  Eg. like in https://github.com/asteroid-team/asteroid
Code style Black
Number of downloads

Second Row
There are 2 badges that specify the pip version with the challenges. These are custom badges that we can change in colours and texts

Note, I didn't include this commit in @jonbarker68 Readme PR to avoid conflicts with that branch

By the way, I am not sure how that info is included, but the badge with the python versions seems to be taking the information from Programming Language in pypi.
In pyclarity it says
Programming Language
[Python :: 3]

But, in Asteroid, they specify specific versions.
Programming Language
[Python :: 3]
[Python :: 3.10]
[Python :: 3.8]
[Python :: 3.9]

We may want to do that to match the versions we use for the tests and show that info in the badge

You could perhaps include a code coverage badge. It would involved tweaking the .github/workflow/run_tests.yaml to use the coverage-py-badge action.
By the way if you start your PR with a GitHub Keyword it will close the issue automatically on merging. For example if you'd written the following at the start
Closes #146

...that would do the trick (you may be able to put it anywhere in the body but I typically try and put it at the top).

The Programming Language that PyPI uses is likely being pulled from setup.cfg in the metadata section...
classifiers=
    ...
    Programming Language :: Python :: 3

...and so adding the following would add the information..
classifiers=
    ...
    Programming Language :: Python :: 3
    Programming Language :: Python :: 3.8
    Programming Language :: Python :: 3.9
    Programming Language :: Python :: 3.10

I'd just be copying what Asterid have done to get that into a badge.
(Asteroid defines these in setup.py but that file is slowly being deprecated in favour of setup.cfg and pyproject.toml).

You could perhaps include a code coverage badge. It would involved tweaking the .github/workflow/run_tests.yaml to use the coverage-py-badge action.

Hi @ns-rse, after a bit of reading, I found how to include the badge without effort.

By the way if you start your PR with a GitHub Keyword it will close the issue automatically on merging. For example if you'd written the following at the start
Closes #146

...that would do the trick (you may be able to put it anywhere in the body but I typically try and put it at the top).

Thank you so much for the info. There are so many tricks in Github that I didn't know about it.
Just in case, I updated the first comment with Closes #146.

This is looking great. Are we happy to merge this now?

This is looking great. Are we happy to merge this now?

Yeah, They look great.
Happy to merge


http post --message is not available in Spring-XD

Hi I am running spark-streaming-logger-java-sink example,
I configured Spark cluster and the SpringXD srevers.yml file exactly how you mentioned, i am able upload the module as well as created and deployed new stream 'spark-streaming-java-logger' successfully.
While posting the messages to the stream i got an error that
xd:>http post --message "foo foo foo"
Option 'message' is not available for this command. Use tab assist or the "help" command to see the legal options
Please Help me to solve this issue.
Thanks

Hi,
the correct command is http post --data (not --message). Did you find this in the documentation? If this is the case, we need to fix this.
Also, be wary that this http post command is just a convenience in the Spring XD shell. You could have used any http client (for example curl) submitting to http://localhost:9000
HTH

Thanks for the reply,
I found this document in the following link
https://github.com/spring-projects/spring-xd-samples/blob/master/spark-streaming-logger-java-sink/README.md

Thanks.
I fixed the README

Thanks @ericbottard. Sorry for the confusion @austindsouza


publish for Scala 2.13.0-M5?

scalacheck (and scalacheck for scala js) for Scala 2.13.0-M5 seem to be on the critical path for a variety of projects updating to 2.13.0-M5

A few hours ago, Scalajs was published in #435:
http://central.maven.org/maven2/org/scalacheck/scalacheck_sjs0.6_2.13.0-M5/1.14.0/

@rickynils this issue can be closed now


Handle Git remotes not called "origin"

recipyCommon/version_control.py  does:
def add_git_info(run, scriptpath):

...

        try:
            run["gitorigin"] = repo.remotes.origin.url
        except:
            run["gitorigin"] = None

However, this assumes that the Git repository has a remote called "origin" and, if not, then records "None". What about repositories that might have a remote, but just not called "origin"?  Perhaps record all the remotes. Or add a command and/or configuration option allowing the user to specify which remote is to be used.

@mikej888 @robintw I would like to work on this issue. Anything else I should know?

That's great @janga1997!
I think the best approach would be to record all of the remotes and their URLs - probably just a list of tuples [(remote_name, remote_url), (remote_name, remote_url), ...]. Remember that you'll also need to change the code for the CLI and GUI to make sure that all of the remotes are displayed properly.
Feel free to start a work-in-progress pull request and I'll be happy to give feedback along the way.


Across all docs,there should be both Azure CLI AND Azure PowerShell examples

I see a number of documents that have only Azure CLI examples.
As a PowerShell user, these are next to useless and slow my ability at looking at how I can adapt an example into PowerShell in my environment

@kilasuit thanks for the feedback.  For more visibility, you might post an item here so the community can upvote it:
https://feedback.azure.com/forums/34192--general-feedback

@jason-j-MSFT - unless you've actually fixed this issue and that all the docs have had the details listed then I don't think this issue can be closed as the issue still stands
As the Docs live here on GitHub then raising this issue here is the right place for it to be raised.

This issue should be seen as an overarching problem issue where this will allow the community to see missing pages and likely add pages for others to be able to look at adding any needed samples

Also created feedback as per request - https://feedback.azure.com/forums/34192--general-feedback/suggestions/35078128-ensure-all-documents-that-have-az-cli-examples-als

@kilasuit I hear ya.  I upvoted your item and I see others have as well.  I think the feedback item is the better way to go about it, as you said the issue your bring up is very overarching.


Automatically check for updates

It would be great if CLI could check for new versions at startup and either suggest that users run npm update or, if possible, offer to update automatically.

For reference https://github.com/dadi/status/blob/master/dadi/lib/index.js#L113

Included in 2.1.0.


How to deal with "conflicts" between qualified types in documentation?

Let's say there are two modules A and B, both exporting a type T, and a third module C importing the two, and defining a function like so:
module C where
import qualfied A
import qualified B

f : A.T -> B.T

The documentation for module C will show:
f : T -> T

Little chance for readers of the documentation to find out what is going on here.
Possible solution (without resorting to always showing all qualifiers): When showing documentation for a module, collect all type names that occur in its signatures, compare them to find out which names occur in at least two different qualified forms, then always print those with qualifiers (but all other type names without qualifiers).

I think the proposed solution is the best that we can do, other than disallowing types of duplicate names, which would make it hard to work with Html.Attribute and Svg.Attribute.

I agree with the idea of showing the minimal unambiguous prefix. Most the time there's no prefix. In rare bad cases we fall back to this.
(That said, I hope no one ever designs an API like Haskell's ByteString where it's always very hard for me to figure out if I am working with a strict or lazy string.)

Heh, reminds me of a C library I had to work with in college, where T was typedefed to a pointer to a data structure. Confusion between &T, T, and T* ensued.

In:

Hmm, in another library, it would still be useful to know which Attribute you mean even if you're only using one. In the code, you'll see the import statement, but just in the docs you won't. One possibility is to require qualified types for everything not from core or the current library. Another, less restrictive solution is to get inter-module type links working.

I think the last sentence is the important one. We should get hyperlinking of all types in docs working eventually, so should not add qualifiers where not needed.
Where we really need them is if there are otherwise conflicting names (because then the reader of the docs wouldn't even suspect that clicking the links might lead to different destinations).
Though I would amend my original proposal a bit. There, in the first comment in this issue, I said:

When showing documentation for a module, collect all type names that occur in its signatures, compare them to find out which names occur in at least two different qualified forms, then always print those with qualifiers (but all other type names without qualifiers).

I now think that instead of collecting only all type names that occur in the signatures of that module, the consideration should include all type names that occur in the signatures of the whole package the module is a part of.
So, for example, instead of looking into http://package.elm-lang.org/packages/Apanatshka/elm-signal-extra/5.2.0/docs/Signal-Discrete.json, I would be looking into http://package.elm-lang.org/packages/Apanatshka/elm-signal-extra/5.2.0/documentation.json for that.
BTW, I'm a bit perplexed about why both these files (and .../5.2.0/docs/Signal-Extra.json etc.) exist. Isn't there a lot of redundancy here? It seems that 5.2.0/documentation.json is a concatenation of all 5.2.0/docs/*.json (except that the latter contain this bug!).
Concerning implementing my proposal here: I think it's not too hard. I could try it at some point. What would help is a super easy way to extract all qualified types from a collection of the "type" strings in a file like http://package.elm-lang.org/packages/Apanatshka/elm-signal-extra/5.2.0/documentation.json. Concretely, let's say I have already extracted all its signature strings like

"Signal.Signal (Maybe.Maybe a)"
"(a -> a -> a) -> Signal.Stream.Stream a -> Signal.Stream.Stream a -> Signal.Stream.Stream a"
"Signal.Stream.Stream (Maybe.Maybe a) -> Signal.Stream.Stream a"
...

what's the easiest way to get from this the list of strings "Signal.Signal", "Maybe.Maybe", "Signal.Stream.Stream", ...? Anybody wants to propose some magic incantation of Regex stuff?

I recall going through and messing with a bunch of stuff by hand with the paren bug because we couldn't track down the source. So that it's fixed only in certain places may just indicate when I did that pass. That's for another thread though.
On the regex question, all types begin with capital letters, so you could search for /[A-Z][a-zA-Z]*(\.[A-Z][a-zA-Z]*)*/g and find them all.
Also, I agree about the refined version. If it is ambiguous within the package then it should get a prefix. I wonder if core should be treated as "part of the package" if people are using the same names.

+1 for treating core types as possibly conflicting with the current module.

I'm not going to give core types any special treatment. Of course, if a package defines Module.Time but also uses Time.Time, then there will be qualifiers. But this does not require any special treatment. (If something in addition is requested, then the same uncertainties come up as in inter-package-linking: Which version of core should be considered? ...)

The version in package.json of course. If I define a type Time as something other than Float, wouldn't you want to know that it's not the one from core?

Well, I'll write the version without that additional feature. You can then extend it. :smile:

I've implemented what I said, in https://github.com/elm-lang/package.elm-lang.org/pull/91.
See screenshots there for the effect this has.
About Max's comments above: It's not that easy as you make it sound. What does

The version in package.json of course.

mean?
Probably you mean the elm-package.json that was uploaded for the package and version currently under display. But that file does only specify a range of possible versions for core. So one would have to go and look for the latest consistent one.
Also, it's not clear whether special casing core is really valuable. There were (still are?) considerations to split core, moving more stuff to "non-special" packages. So there would be less stuff overall which benefits from such special casing anyway.
Moreover, such special casing is outside the issue I brought up originally here, which was about displaying two types A.T and B.T as just T in package documentation. Max's point is beside that. So it should be addressed, if so, in a separate issue and pull request, after merging https://github.com/elm-lang/package.elm-lang.org/pull/91.
For the record, the basic approach for such an extension would be to, in frontend/Page/Module.elm, read the elm-package.json of an appropriate version of core, to collect all the types from there that should be "avoided" respectively disambiguated, and to merge that information into the mentionedTypes value being sent to the modulesAndMentionedTypes mailbox.

Yeah, it's not clear to me that this is actually plausible. Maybe the highest version in the given constraints? But I think it's the right move to just do it within the package for now.

Closing this issue, since the original concern is addressed by the merging of https://github.com/elm-lang/package.elm-lang.org/pull/91.
The possible special casing concerning core discussed above remains unaddressed for now.


[CI:DOC] Bring README.md up to date

Add notes on helper functions
Update example

Signed-off-by: Jhon Honce<EMAIL_ADDRESS>

/lgtm


Added ratings to xmp sidecar support

Added ratings to xmp sidecar support, also added rating to bunny.mp4.xmp and a different rating in IMG_5910.jpg.xmp vs what is in the photo file to test if sidecar is functioning as primary (it is as discussed).

Thank you!


Website

I contacted Jon Bringhurst about getting the website. Although he no longer owns the fileutils.io domain, the site can be found here.
I downloaded the site and added it to this branch. After this PR is merged, the site will be live at: https://hpc.github.io/mpifileutils.
You can view a sample of the page at https://gonsie.github.io/mpifileutils.

lgtm ... this needs a bit of work before it should be linked to tho ... it still points to the old github repo

It lives at https://hpc.github.io/mpifileutils/ now.


Is the Badgy supposed to charge the battery?

I have the LIR2450 loaded in the badgy, and when I plug the USB, I see a red light in the back.
I am wondering if the badgy is able to charge the battery because when I remove the USB after a few hours, the badgy does not seem to work on battery.

Hi @vidax,
The USB should recharge the LIR2450 battery. Please ensure you have the battery inserted in the correct polarity, and the battery has been sufficiently charged (3.7V - 4.2V).

Ahh crap....! ...and with that I released some magic smoke from the MCP73831 charge IC! 👎  Was probing around from the USB V+ and touched the VDD of the chip and it went "pooof" (was plugged in to USB at the time). I have some spare MCP73831 circuits, I may try to de-solder and re-apply one.
While poking about looking at the schematic and the PCB layout - I did notice that the LED D4 which is the charge state LED has a current limiting resistor of 2K Ohm. Should this not be more in line with common LED resistors, and the spec sheet, around 200-470Ohm? Or is this high to work with the High-Impedance state of the STAT line as well as Hi and Low?

Sorry to hear that @chunkysteveo :( Yes you can replace it with an off the shelf MCP73831; we chose 2K Ohm simply because we found the LED to be too bright!

Ahh OK, makes sense for the LED. I will try a new MCP73831 to fix this one, and test with a number of LIR2450's I have. Hopefully was just the IC that was dead (well it is now!!! :)  )


Report patch level in 24.02.1

Describe the Bug
24.02.1 released today is omitting the patch level resulting in false positives on version sniffing. Previous releases, such as 23.12.3 include it.
Steps to Reproduce
Inspect version within the release.
Expected Behaviour
v24.02.1
Screenshots or Additional Context
No response
Browser Details
No response
Exact BookStack Version
v24.02.1

i'm using the cache breaker method for version checking, and have the same issue

Can confirm, my apologies, missed a couple of steps. Also missed building JS/CSS assets.
Have just pushed out https://github.com/BookStackApp/BookStack/releases/tag/v24.02.2 to correct.
Thanks for raising @msaladna and @chrisnails!


Integrate offer checkout flow into inquiry checkout modal

PURCHASE-2350
This PR hooks up the inquiry make offer modal to the a web view of the checkout flow. I had to convert the modal from a FancyModal into a modally presented screen because of issues opening modals within modals. I did not achieve complete parity with the designs, but I did make some improvements to the look of the checkout flow modal, including the header appearance. A follow up ticket may be needed to make some changes in force to style the checkout flow closer to the designs only when it is viewed from Eigen. If the header's next button functionality is important, then a spike into how to communicate a button press from Eigen to the force webview may be needed.
Paired with @MounirDhahri

Nice work Lily 🎉

Nice work Lily 🎉

Thanks for the fixes 👏

I'm late to the party on this one, but great job on this @lilyfromseattle!!!


Egress record start failed with pulse sink load module Failure: Module initialization failed

Hi team,
I found that some time the egress cannot start new record with error Module initialization failed. After doing some investigation, I found that the egress task will not more than 60 due to it will cause a lot of suspended pulse module.
The log error like below:
{"level":"debug","ts":1670910590.9727807,"logger":"egress","caller":"web/source.go:172","msg":"chrome log: START_RECORDING","nodeID":"NE_A5mg2fqY267D","egressID":"EG_GDv7NBjewXdG"}
{"level":"error","ts":1670910591.1060054,"logger":"egress","caller":"pipeline/pipeline.go:207","msg":"failed to set pipeline state","nodeID":"NE_A5mg2fqY267D","egressID":"EG_GDv7NBjewXdG","error":"Failed to change state to PLAYING","stacktrace":"github.com/livekit/egress/pkg/pipeline.(*Pipeline).Run\n\t/workspace/pkg/pipeline/pipeline.go:207\ngithub.com/livekit/egress/pkg/service.(*Handler).HandleRequest.func2\n\t/workspace/pkg/service/handler.go:58"}
{"level":"debug","ts":1670910591.1061406,"logger":"egress","caller":"pipeline/pipeline.go:699","msg":"removing temporary directory","nodeID":"NE_A5mg2fqY267D","egressID":"EG_GDv7NBjewXdG","path":"/out/data/EG_GDv7NBjewXdG/"}
{"level":"info","ts":1670910591.1068933,"logger":"egress","caller":"service/handler.go:125","msg":"egress failed","nodeID":"NE_A5mg2fqY267D","egressID":"EG_GDv7NBjewXdG","request_type":"room_composite","output_type":"file","error":"Failed to change state to PLAYING"}
{"level":"debug","ts":1670910591.1263373,"logger":"egress","caller":"service/service.go:270","msg":"deleting handler temporary directory","nodeID":"NE_nWkNtRgAdZUz","path":"/tmp/lkegress/EG_GDv7NBjewXdG"}
{"level":"debug","ts":1670910967.866186,"logger":"egress","caller":"service/service.go:168","msg":"request received","nodeID":"NE_nWkNtRgAdZUz","egressID":"EG_PH6EAR4ZSWDa","requestID":"RPC_uBpivEa8Nsg7","senderID":"ND_KU22jczb"}
{"level":"debug","ts":1670910967.8663242,"logger":"egress","caller":"stats/monitor.go:157","msg":"cpu request","nodeID":"NE_nWkNtRgAdZUz","accepted":true,"availableCPUs":9.99946782927514,"numCPUs":10}
{"level":"info","ts":1670910967.8675084,"logger":"egress","caller":"service/service.go:211","msg":"request accepted","nodeID":"NE_nWkNtRgAdZUz","egressID":"EG_PH6EAR4ZSWDa","requestID":"RPC_uBpivEa8Nsg7","senderID":"ND_KU22jczb"}
{"level":"debug","ts":1670910967.8677294,"logger":"egress","caller":"params/params.go:724","msg":"writing to file","nodeID":"NE_nWkNtRgAdZUz","egressID":"EG_PH6EAR4ZSWDa","filename":"/out/data/EG_PH6EAR4ZSWDa/1052222382529720320.mp4"}
{"level":"debug","ts":1670910967.8981059,"logger":"egress","caller":"server/main.go:120","msg":"handler launched","nodeID":"NE_C3jgcPQLerKc"}
{"level":"info","ts":1670910967.898377,"logger":"egress","caller":"server/main.go:124","msg":"setting TMPDIR environment and creating path","nodeID":"NE_C3jgcPQLerKc","path":"/tmp/lkegress/EG_PH6EAR4ZSWDa"}
{"level":"info","ts":1670910967.8985543,"logger":"egress","caller":"redis/redis.go:54","msg":"connecting to redis","nodeID":"NE_C3jgcPQLerKc","cluster":true,"addr":["<IP_ADDRESS>:6379","<IP_ADDRESS>:6380","<IP_ADDRESS>:6381","<IP_ADDRESS>:6382","<IP_ADDRESS>:6383","<IP_ADDRESS>:6384"]}
{"level":"debug","ts":1670910967.9153082,"logger":"egress","caller":"params/params.go:724","msg":"writing to file","nodeID":"NE_C3jgcPQLerKc","egressID":"EG_PH6EAR4ZSWDa","filename":"/out/data/EG_PH6EAR4ZSWDa/1052222382529720320.mp4"}
Failure: Module initialization failed
{"level":"error","ts":1670910967.9211745,"logger":"egress","caller":"web/input.go:48","msg":"failed to load pulse sink","nodeID":"NE_C3jgcPQLerKc","egressID":"EG_PH6EAR4ZSWDa","error":"exit status 1","stacktrace":"github.com/livekit/egress/pkg/pipeline/input/web.NewWebInput\n\t/workspace/pkg/pipeline/input/web/input.go:48\ngithub.com/livekit/egress/pkg/pipeline/input.New\n\t/workspace/pkg/pipeline/input/input.go:29\ngithub.com/livekit/egress/pkg/pipeline.New\n\t/workspace/pkg/pipeline/pipeline.go:86\ngithub.com/livekit/egress/pkg/service.(*Handler).buildPipeline\n\t/workspace/pkg/service/handler.go:106\ngithub.com/livekit/egress/pkg/service.(*Handler).HandleRequest\n\t/workspace/pkg/service/handler.go:36\nmain.runHandler\n\t/workspace/cmd/server/main.go:159\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/root/go/pkg/mod/github.com/urfave/cli/v2@v2.15.0/command.go:173\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/root/go/pkg/mod/github.com/urfave/cli/v2@v2.15.0/app.go:384\ngithub.com/urfave/cli/v2.(*App).Run\n\t/root/go/pkg/mod/github.com/urfave/cli/v2@v2.15.0/app.go:253\nmain.main\n\t/workspace/cmd/server/main.go:66\nruntime.main\n\t/usr/lib/go-1.18/src/runtime/proc.go:250"}
{"level":"info","ts":1670910967.9216247,"logger":"egress","caller":"service/handler.go:125","msg":"egress failed","nodeID":"NE_C3jgcPQLerKc","egressID":"EG_PH6EAR4ZSWDa","request_type":"room_composite","output_type":"file","error":"exit status 1"}
{"level":"debug","ts":1670910967.9277332,"logger":"egress","caller":"service/service.go:270","msg":"deleting handler temporary directory","nodeID":"NE_nWkNtRgAdZUz","path":"/tmp/lkegress/EG_PH6EAR4ZSWDa"}

And I also found that there are 60 sink or source or modules of pulse audio:
...
Source #15
        State: SUSPENDED
        Name: EG_vaTa8iZDTBc9.monitor
        Description: Monitor of EG_vaTa8iZDTBc9
        Driver: module-null-sink.c
        Sample Specification: s16le 2ch 44100Hz
        Channel Map: front-left,front-right
        Owner Module: 24
        Mute: no
        Volume: front-left: 65536 / 100% / 0.00 dB,   front-right: 65536 / 100% / 0.00 dB
                balance 0.00
        Base Volume: 65536 / 100% / 0.00 dB
        Monitor of Sink: EG_vaTa8iZDTBc9
        Latency: 0 usec, configured 0 usec
        Flags: DECIBEL_VOLUME LATENCY
        Properties:
                device.description = "Monitor of EG_vaTa8iZDTBc9"
                device.class = "monitor"
                device.icon_name = "audio-input-microphone"
        Formats:
                pcm

Source #16
        State: SUSPENDED
        Name: EG_shkSSFAj8Cf6.monitor
        Description: Monitor of EG_shkSSFAj8Cf6
        Driver: module-null-sink.c
        Sample Specification: s16le 2ch 44100Hz
        Channel Map: front-left,front-right
        Owner Module: 25
        Mute: no
        Volume: front-left: 65536 / 100% / 0.00 dB,   front-right: 65536 / 100% / 0.00 dB
                balance 0.00
        Base Volume: 65536 / 100% / 0.00 dB
        Monitor of Sink: EG_shkSSFAj8Cf6
        Latency: 0 usec, configured 0 usec
        Flags: DECIBEL_VOLUME LATENCY
        Properties:
                device.description = "Monitor of EG_shkSSFAj8Cf6"
                device.class = "monitor"
                device.icon_name = "audio-input-microphone"
        Formats:
                pcm
...
Source #58
        State: SUSPENDED
        Name: EG_8kTHtgxMtCjN.monitor
        Description: Monitor of EG_8kTHtgxMtCjN
        Driver: module-null-sink.c
        Sample Specification: s16le 2ch 44100Hz
        Channel Map: front-left,front-right
        Owner Module: 67
        Mute: no
        Volume: front-left: 65536 / 100% / 0.00 dB,   front-right: 65536 / 100% / 0.00 dB
                balance 0.00
        Base Volume: 65536 / 100% / 0.00 dB
        Monitor of Sink: EG_8kTHtgxMtCjN
        Latency: 0 usec, configured 0 usec
        Flags: DECIBEL_VOLUME LATENCY
        Properties:
                device.description = "Monitor of EG_8kTHtgxMtCjN"
                device.class = "monitor"
                device.icon_name = "audio-input-microphone"
        Formats:
                pcm

Source #59
        State: SUSPENDED
        Name: EG_Keh2jStLYKJy.monitor
        Description: Monitor of EG_Keh2jStLYKJy
        Driver: module-null-sink.c
        Sample Specification: s16le 2ch 44100Hz
        Channel Map: front-left,front-right
        Owner Module: 68
        Mute: no
        Volume: front-left: 65536 / 100% / 0.00 dB,   front-right: 65536 / 100% / 0.00 dB
                balance 0.00
        Base Volume: 65536 / 100% / 0.00 dB
        Monitor of Sink: EG_Keh2jStLYKJy
        Latency: 0 usec, configured 0 usec
        Flags: DECIBEL_VOLUME LATENCY
        Properties:
                device.description = "Monitor of EG_Keh2jStLYKJy"
                device.class = "monitor"
                device.icon_name = "audio-input-microphone"
        Formats:
                pcm

Source #60
        State: SUSPENDED
        Name: EG_GDv7NBjewXdG.monitor
        Description: Monitor of EG_GDv7NBjewXdG
        Driver: module-null-sink.c
        Sample Specification: s16le 2ch 44100Hz
        Channel Map: front-left,front-right
        Owner Module: 69
        Mute: no
        Volume: front-left: 65536 / 100% / 0.00 dB,   front-right: 65536 / 100% / 0.00 dB
                balance 0.00
        Base Volume: 65536 / 100% / 0.00 dB
        Monitor of Sink: EG_GDv7NBjewXdG
        Latency: 0 usec, configured 0 usec
        Flags: DECIBEL_VOLUME LATENCY
        Properties:
                device.description = "Monitor of EG_GDv7NBjewXdG"
                device.class = "monitor"
                device.icon_name = "audio-input-microphone"
        Formats:
                pcm

I checked the code, when egress start, it will execute command "pactl load-module " to add audio channel for this chrome to record the audio and other egress task also add the other channel for that chrome. And it will execute command "pactl unload-module " to remove the channel when the egress task done.
I also manual execute "pactl unload-module <NO.>" and it will work well, and the egress task can work back after I execute unload-module command.
Then I think that it may be caused by the close() method not unload-module from pulse audio correctly.

Fixed in #197


cluster control plane is still being initialized: BootstrapFailed

Bug Report
The process to create a managed cluster on a VM running docker is failing using Tanzu client v0.11.2
docker rm -f $(docker ps -a -q)
docker network rm kind
docker system prune -a --volumes -f
rm -fr ~/.config/tanzu
tanzu mc create -i docker -p dev mgmt-test
Downloading TKG compatibility file from 'projects.registry.vmware.com/tkg/framework-zshippable/tkg-compatibility'
Downloading the TKG Bill of Materials (BOM) file from 'projects.registry.vmware.com/tkg/tkg-bom:v1.5.1-tf-v0.11.2'
Downloading the TKr Bill of Materials (BOM) file from 'projects.registry.vmware.com/tkg/tkr-bom:v1.22.5_vmware.1-tkg.3-tf-v0.11.2'

Validating the pre-requisites...
Identity Provider not configured. Some authentication features won't work.
WARNING: key is not a string: %!s(int=1)WARNING: key is not a string: %!s(int=1)
Setting up management cluster...
Validating configuration...
Using infrastructure provider docker:v1.0.1
Generating cluster configuration...
WARNING: key is not a string: %!s(int=1)WARNING: key is not a string: %!s(int=1)Setting up bootstrapper...
Bootstrapper created. Kubeconfig: /home/snowdrop/.kube-tkg/tmp/config_BnxmhRCJ
Installing providers on bootstrapper...
Fetching providers
Installing cert-manager Version="v1.5.3"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.0.1" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.0.1" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.0.1" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-docker" Version="v1.0.1" TargetNamespace="capd-system"
Start creating management cluster...
[cluster control plane is still being initialized: WaitingForControlPlane, cluster infrastructure is still being provisioned: WaitingForControlPlane]
cluster control plane is still being initialized: WaitingForControlPlane
cluster control plane is still being initialized: WaitingForKubeadmInit
cluster control plane is still being initialized: Bootstrapping @ Machine/tkg-mgmt-docker-20220426170635-control-plane-j5plk
cluster control plane is still being initialized: BootstrapFailed @ Machine/tkg-mgmt-docker-20220426170635-control-plane-j5plk

Versions
OS: Linux 3.10.0-1160.62.1.el7.x86_64
Linux distribution: Centos7
Tanzu version: 0.11.0
Docker version: 20.10.14
Expected Behavior
A managed cluster should be created
Screenshots or additional information and context
[snowdrop@h01-121 ~]$ docker ps
CONTAINER ID   IMAGE                                                         COMMAND                  CREATED         STATUS         PORTS                                  NAMES
5e1c9fbbf04a   projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1   "/usr/local/bin/entr…"   7 minutes ago   Up 7 minutes   34011/tcp, <IP_ADDRESS>:34011->6443/tcp   tkg-mgmt-docker-20220426170635-control-plane-j5plk
f4bc2b8e6d87   kindest/haproxy:v20210715-a6da3463                            "haproxy -sf 7 -W -d…"   7 minutes ago   Up 7 minutes   33192/tcp, <IP_ADDRESS>:33192->6443/tcp     tkg-mgmt-docker-20220426170635-lb
f6c466a1f18e   projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1   "/usr/local/bin/entr…"   9 minutes ago   Up 9 minutes   <IP_ADDRESS>:41174->6443/tcp              tkg-kind-c9k0kvbsjmgpv83itcd0-control-plane
[snowdrop@h01-121 ~]$ docker logs f4bc2b8e6d87
[WARNING] 115/150941 (1) : config : missing timeouts for frontend 'controlPlane'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[NOTICE] 115/150941 (1) : New worker #1 (9) forked
[WARNING] 115/150947 (1) : Reexecuting Master process
[NOTICE] 115/150947 (1) : haproxy version is 2.2.9-2~bpo10+1
[NOTICE] 115/150947 (1) : path to executable is /usr/sbin/haproxy
[ALERT] 115/150947 (1) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
[WARNING] 115/150947 (9) : Stopping frontend controlPlane in 0 ms.
[WARNING] 115/150947 (9) : Stopping backend kube-apiservers in 0 ms.
[WARNING] 115/150947 (9) : Stopping frontend GLOBAL in 0 ms.
[WARNING] 115/150947 (9) : Proxy controlPlane stopped (cumulated conns: FE: 11, BE: 0).
[WARNING] 115/150947 (9) : Proxy kube-apiservers stopped (cumulated conns: FE: 0, BE: 11).
[WARNING] 115/150947 (9) : Proxy GLOBAL stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE] 115/150947 (1) : New worker #1 (27) forked
[WARNING] 115/150947 (1) : Former worker #1 (9) exited with code 0 (Exit)
[WARNING] 115/150947 (27) : Server kube-apiservers/tkg-mgmt-docker-20220426170635-control-plane-j5plk is DOWN, reason: Layer4 connection problem, info: "SSL handshake failure", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 115/150947 (27) : backend 'kube-apiservers' has no server available!
and docker inspect of the lb container returns

[snowdrop@h01-121 ~]$ docker inspect  f4bc2b8e6d87
[
...
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "47efa21bf6b63bd46b8683a15f69b7edeae2f60fe7beae34ada4597d6a55b103",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "33192/tcp": null,
                "6443/tcp": [
                    {
                        "HostIp": "<IP_ADDRESS>",
                        "HostPort": "33192"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/47efa21bf6b6",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "kind": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "f4bc2b8e6d87",
                        "tkg-mgmt-docker-20220426170635-lb"
                    ],
                    "NetworkID": "59ab22a331d14cda52106b44123e76696af55e6a10580d3783204f8d7a77cc8f",
                    "EndpointID": "f6c150517db9a262e477640a4f9ed77078ef7908358f672a92115dda7c5470ca",
                    "Gateway": "<IP_ADDRESS>",
                    "IPAddress": "<IP_ADDRESS>",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "fc00:f853:ccd:e793::1",
                    "GlobalIPv6Address": "fc00:f853:ccd:e793::3",
                    "GlobalIPv6PrefixLen": 64,
                    "MacAddress": "02:42:ac:12:00:03",
                    "DriverOpts": null
                }
            }
        }
    }
]

Log of the CAPD controller-manager reports:
root@tkg-kind-c9k19bbsjmggojkhlp90-control-plane:/# kubectl logs -n capd-system  capd-controller-manager-86b587cf4b-ckpd8
...
E0426 15:53:52.639485       1 controller.go:317] controller/dockermachine "msg"="Reconciler error"
"error"="failed to exec DockerMachine bootstrap: failed to run cloud config: exited with status: 255, "
"name"="tkg-mgmt-docker-20220426175003-control-plane-hrdks"
"namespace"="tkg-system"
"reconciler group"="infrastructure.cluster.x-k8s.io"
"reconciler kind"="DockerMachine"

A more verbose log message is available here: https://gist.github.com/cmoulliard/ae2307587b40a86cb7e22186b776d10b
Log of the LB container
docker ps
CONTAINER ID   IMAGE                                                         COMMAND                  CREATED          STATUS          PORTS                                  NAMES
b6f1ef1614a1   projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1   "/usr/local/bin/entr…"   18 minutes ago   Up 18 minutes   44932/tcp, <IP_ADDRESS>:44932->6443/tcp   tkg-mgmt-docker-20220427084538-control-plane-x6g7v
1639a8f8b504   kindest/haproxy:v20210715-a6da3463                            "haproxy -sf 7 -W -d…"   18 minutes ago   Up 18 minutes   38048/tcp, <IP_ADDRESS>:38048->6443/tcp     tkg-mgmt-docker-20220427084538-lb
8ecb8b92a9d9   projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1   "/usr/local/bin/entr…"   19 minutes ago   Up 19 minutes   <IP_ADDRESS>:35296->6443/tcp              tkg-kind-c9ked53sjmgrbfnd3qeg-control-plane

docker logs 1639a8f8b504
[WARNING] 116/064801 (1) : config : missing timeouts for frontend 'controlPlane'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[NOTICE] 116/064801 (1) : New worker #1 (8) forked
[WARNING] 116/064808 (1) : Reexecuting Master process
[NOTICE] 116/064808 (1) : haproxy version is 2.2.9-2~bpo10+1
[NOTICE] 116/064808 (1) : path to executable is /usr/sbin/haproxy
[ALERT] 116/064808 (1) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
[WARNING] 116/064808 (8) : Stopping frontend controlPlane in 0 ms.
[WARNING] 116/064808 (8) : Stopping backend kube-apiservers in 0 ms.
[WARNING] 116/064808 (8) : Stopping frontend GLOBAL in 0 ms.
[WARNING] 116/064808 (8) : Proxy controlPlane stopped (cumulated conns: FE: 10, BE: 0).
[WARNING] 116/064808 (8) : Proxy kube-apiservers stopped (cumulated conns: FE: 0, BE: 10).
[WARNING] 116/064808 (8) : Proxy GLOBAL stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE] 116/064808 (1) : New worker #1 (25) forked
[WARNING] 116/064808 (1) : Former worker #1 (8) exited with code 0 (Exit)
[WARNING] 116/064808 (25) : Server kube-apiservers/tkg-mgmt-docker-20220427084538-control-plane-x6g7v is DOWN, reason: Layer4 connection problem, info: "SSL handshake failure", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 116/064808 (25) : backend 'kube-apiservers' has no server available!

Environment Details

Build version (tanzu version): 0.11.2
Deployment (Managed/Unmanaged cluster): managed
Infrastructure Provider (Docker/AWS/Azure/vSphere): docker
Operating System (client): Centos7

Just some notes...
Looks like there is a CAPD failure running the bootstrap process here: https://github.com/kubernetes-sigs/cluster-api/blob/3bff7da15910edf0b581875619d89d3bfbf59bd3/test/infrastructure/docker/internal/docker/machine.go#L354-L356
The failed command is: sysctl -w net.ipv4.tcp_retries2=4 with the error sysctl: cannot stat /proc/sys/net/ipv4/tcp_retries2: No such file or directory.
@cmoulliard could you provide a little more information about this how machine? Please provide the output of docker info, uname -a and any other relevant details you think might be interesting about the host. Thanks!

uname -a and

[snowdrop@h01-121 ~]$ uname -a
Linux h01-121 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

of docker info

[snowdrop@h01-121 ~]$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.62.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.14GiB
 Name: h01-121
 ID: H4U7:WSEN:JBKK:C7JU:P37R:A6JO:BVVQ:L46W:UKTK:7SAK:AZMJ:25PW
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  <IP_ADDRESS>/8
 Live Restore Enabled: false

The failed command is: sysctl -w net.ipv4.tcp_retries2=4 with the error sysctl: cannot stat /proc/sys/net/ipv4/tcp_retries2: No such file or directory.

Indeed, this file do not exist within the container created
root@tkg-kind-c9ked53sjmgrbfnd3qeg-control-plane:/# ls -la /proc/sys/net/ipv4
...
-rw-r--r--. 1 root root 0 Apr 27 12:23 tcp_ecn
-rw-r--r--. 1 root root 0 Apr 27 12:23 tcp_keepalive_intvl
-rw-r--r--. 1 root root 0 Apr 27 12:23 tcp_keepalive_probes
-rw-r--r--. 1 root root 0 Apr 27 12:23 tcp_keepalive_time
-rw-r--r--. 1 root root 0 Apr 27 12:23 tcp_mem
-rw-r--r--. 1 root root 0 Apr 27 12:23 tcp_min_snd_mss
...
root@tkg-kind-c9ked53sjmgrbfnd3qeg-control-plane:/# uname -a
Linux tkg-kind-c9ked53sjmgrbfnd3qeg-control-plane 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Can you take a look at ls -la /proc/sys/net/ipv4 on your host? I believe that is just passed through.

on your host?

No -->
[snowdrop@h01-121 ~]$ sudo ls -la /proc/sys/net/ipv4/tcp_retries2
-rw-r--r--. 1 root root 0 Apr 27 14:20 /proc/sys/net/ipv4/tcp_retries2
[snowdrop@h01-121 ~]$ cat /proc/sys/net/ipv4/tcp_retries2
15
[snowdrop@h01-121 ~]$ ls -la /proc/sys/net/ipv4/tcp_retries2
-rw-r--r--. 1 root root 0 Apr 27 14:20 /proc/sys/net/ipv4/tcp_retries2
[snowdrop@h01-121 ~]$ ls -la /proc/sys/net/ipv4
total 0
dr-xr-xr-x. 1 root root 0 Apr 25 17:49 .
dr-xr-xr-x. 1 root root 0 Apr 25 17:49 ..
-rw-r--r--. 1 root root 0 Apr 27 14:20 cipso_cache_bucket_size
-rw-r--r--. 1 root root 0 Apr 27 14:20 cipso_cache_enable
-rw-r--r--. 1 root root 0 Apr 27 14:20 cipso_rbm_optfmt
-rw-r--r--. 1 root root 0 Apr 27 14:20 cipso_rbm_strictvalid
dr-xr-xr-x. 1 root root 0 Apr 25 17:49 conf
-rw-r--r--. 1 root root 0 Apr 27 14:20 fib_multipath_hash_policy
-rw-r--r--. 1 root root 0 Apr 27 14:20 fwmark_reflect
-rw-r--r--. 1 root root 0 Apr 25 17:50 icmp_echo_ignore_all
-rw-r--r--. 1 root root 0 Apr 27 14:20 icmp_echo_ignore_broadcasts
-rw-r--r--. 1 root root 0 Apr 27 14:20 icmp_errors_use_inbound_ifaddr
-rw-r--r--. 1 root root 0 Apr 27 14:20 icmp_ignore_bogus_error_responses
-rw-r--r--. 1 root root 0 Apr 27 14:20 icmp_msgs_burst
-rw-r--r--. 1 root root 0 Apr 27 14:20 icmp_msgs_per_sec
-rw-r--r--. 1 root root 0 Apr 27 14:20 icmp_ratelimit
-rw-r--r--. 1 root root 0 Apr 27 14:20 icmp_ratemask
-rw-r--r--. 1 root root 0 Apr 27 14:20 igmp_max_memberships
-rw-r--r--. 1 root root 0 Apr 27 14:20 igmp_max_msf
-rw-r--r--. 1 root root 0 Apr 27 14:20 igmp_qrv
-rw-r--r--. 1 root root 0 Apr 27 14:20 inet_peer_maxttl
-rw-r--r--. 1 root root 0 Apr 27 14:20 inet_peer_minttl
-rw-r--r--. 1 root root 0 Apr 27 14:20 inet_peer_threshold
-rw-r--r--. 1 root root 0 Apr 27 14:20 ip_default_ttl
-rw-r--r--. 1 root root 0 Apr 27 14:20 ip_dynaddr
-rw-r--r--. 1 root root 0 Apr 27 14:20 ip_early_demux
-rw-r--r--. 1 root root 0 Apr 25 17:50 ip_forward
-rw-r--r--. 1 root root 0 Apr 27 14:20 ip_forward_use_pmtu
-rw-r--r--. 1 root root 0 Apr 25 17:50 ip_local_port_range
-rw-r--r--. 1 root root 0 Apr 27 14:20 ip_local_reserved_ports
-rw-r--r--. 1 root root 0 Apr 27 14:20 ip_no_pmtu_disc
-rw-r--r--. 1 root root 0 Apr 27 14:20 ip_nonlocal_bind
-rw-r--r--. 1 root root 0 Apr 27 14:20 ipfrag_high_thresh
-rw-r--r--. 1 root root 0 Apr 27 14:20 ipfrag_low_thresh
-rw-r--r--. 1 root root 0 Apr 27 14:20 ipfrag_max_dist
-rw-r--r--. 1 root root 0 Apr 27 14:20 ipfrag_secret_interval
-rw-r--r--. 1 root root 0 Apr 27 14:20 ipfrag_time
dr-xr-xr-x. 1 root root 0 Apr 27 14:20 neigh
-rw-r--r--. 1 root root 0 Apr 25 17:50 ping_group_range
dr-xr-xr-x. 1 root root 0 Apr 27 14:20 route
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_abort_on_overflow
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_adv_win_scale
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_allowed_congestion_control
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_app_win
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_autocorking
-r--r--r--. 1 root root 0 Apr 27 14:20 tcp_available_congestion_control
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_base_mss
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_challenge_ack_limit
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_congestion_control
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_dsack
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_early_retrans
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_ecn
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_fack
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_fastopen
-rw-------. 1 root root 0 Apr 27 14:20 tcp_fastopen_key
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_fin_timeout
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_frto
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_invalid_ratelimit
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_keepalive_intvl
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_keepalive_probes
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_keepalive_time
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_limit_output_bytes
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_low_latency
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_max_orphans
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_max_ssthresh
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_max_syn_backlog
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_max_tw_buckets
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_mem
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_min_snd_mss
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_min_tso_segs
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_moderate_rcvbuf
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_mtu_probing
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_no_metrics_save
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_notsent_lowat
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_orphan_retries
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_reordering
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_retrans_collapse
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_retries1
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_retries2
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_rfc1337
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_rmem
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_sack
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_slow_start_after_idle
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_stdurg
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_syn_retries
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_synack_retries
-rw-r--r--. 1 root root 0 Apr 25 17:50 tcp_syncookies
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_thin_dupack
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_thin_linear_timeouts
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_timestamps
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_tso_win_divisor
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_tw_recycle
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_tw_reuse
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_window_scaling
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_wmem
-rw-r--r--. 1 root root 0 Apr 27 14:20 tcp_workaround_signed_windows
-rw-r--r--. 1 root root 0 Apr 27 14:20 udp_mem
-rw-r--r--. 1 root root 0 Apr 27 14:20 udp_rmem_min
-rw-r--r--. 1 root root 0 Apr 27 14:20 udp_wmem_min
dr-xr-xr-x. 1 root root 0 Apr 27 14:20 vs
-rw-r--r--. 1 root root 0 Apr 27 14:20 xfrm4_gc_thresh

This looks related - same kernel version and some symptoms - but I don't think it would be the same since the host OS does show the expected proc entries: https://github.com/vmware-tanzu/community-edition/issues/2780#issuecomment-1013717257
So just noting as a reference.

So it seems that we will close this ticket ;-)
Nevertheless, may I suggest that :

The logging message reported is more verbose and at least encapsulates the error as currently we only got as message cluster control plane is still being initialized: BootstrapFailed @ Machine/tkg-mgmt-docker-20220426170635-control-plane-j5plk

Log a more human readable message within the CAPD container

B9JyB8IGN1dCAtYyAyLSk7ZG9ja2VyIHJ1biAtLXJtIC0tbmV0PWhvc3QgLS1wcml2aWxlZ2VkIC0tZW50cnlwb2ludD0vdXNyL3NiaW4vZXRodG9vbCBwcm9qZWN0cy5yZWdpc3RyeS52bXdhcmUuY29tL3RrZy9raW5kL25vZGU6djEuMjIuNV92bXdhcmUuMSAtSyBcIiRwZWVyTmFtZVwiIHR4IG9mZiIKICAtICJzeXNjdGwgLXcgbmV0LmlwdjQudGNwX3JldHJpZXMyPTQiCiAgLSAna3ViZWFkbSBpbml0IC0tY29uZmlnIC9ydW4va3ViZWFkbS9rdWJlYWRtLnlhbWwgICYmIGVjaG8gc3VjY2VzcyA+IC9ydW4vY2x1c3Rlci1hcGkvYm9vdHN0cmFwLXN1Y2Nlc3MuY29tcGxldGUnCg==" "command"={"Cmd":"/bin/sh","Args":["-c","sysctl -w net.ipv4.tcp_retries2=4"],"Stdin":""} "stderr"="  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r 47 27.4M   47 13.1M    0     0  12.5M      0  0:00:02  0:00:01  0:00:01 12.5M\r100 27.4M  100 27.4M    0     0  14.1M      0  0:00:01  0:00:01 --:--:-- 14.1M\nsysctl: cannot stat /proc/sys/net/ipv4/tcp_retries2: No such file or directory\n" "stdout"=""
I0427 07:09:36.003168       1 machine.go:486] controller/dockermachine "msg"="Got logs from the machine container" "name"="tkg-mgmt-docker-20220427084538-control-plane-lmmnq" "namespace"="tkg-system" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine" "output"="Inspected the container:\n{ContainerJSONBase:0xc0015bf8c0

Log a more human readable message within the CAPD container

That would be good to submit a feature request suggesting any improvements to the CAPD project. Its main goal is testing though, so I'm not sure how receptive they would be since the current messages, although not very end user friendly, do contain a lot of details that may be useful for someone performing tests.

Attempt to use CAPD directly successful: https://gist.github.com/cmoulliard/1a504142e112e08700d7f0b9c284b844
We may need to pull in the latest CAPD to pick up any fixes. Reviewing the CAPI closed issues I don't see anything obvious that would have changed this, but it would appear something has changed.

I0427 16:58:55.068508       1 leaderelection.go:258] successfully acquired lease capd-system/controller-leader-election-capd
I0427 16:58:55.069488       1 controller.go:178] controller/dockermachine "msg"="Starting EventSource" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine" "source"="kind source: *v1beta1.DockerMachine"
I0427 16:58:55.069554       1 controller.go:178] controller/dockermachine "msg"="Starting EventSource" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine" "source"="kind source: *v1beta1.Machine"
I0427 16:58:55.069585       1 controller.go:178] controller/dockermachine "msg"="Starting EventSource" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine" "source"="kind source: *v1beta1.DockerCluster"
I0427 16:58:55.069651       1 controller.go:178] controller/dockermachine "msg"="Starting EventSource" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine" "source"="kind source: *v1beta1.Cluster"
I0427 16:58:55.069915       1 controller.go:186] controller/dockermachine "msg"="Starting Controller" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 16:58:55.070431       1 controller.go:178] controller/dockercluster "msg"="Starting EventSource" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerCluster" "source"="kind source: *v1beta1.DockerCluster"
I0427 16:58:55.070489       1 controller.go:178] controller/dockercluster "msg"="Starting EventSource" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerCluster" "source"="kind source: *v1beta1.Cluster"
I0427 16:58:55.070531       1 controller.go:186] controller/dockercluster "msg"="Starting Controller" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerCluster"
I0427 16:58:55.172135       1 controller.go:220] controller/dockercluster "msg"="Starting workers" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerCluster" "worker count"=1
I0427 16:58:55.173180       1 controller.go:220] controller/dockermachine "msg"="Starting workers" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine" "worker count"=10
E0427 17:00:11.452924       1 util.go:239]  "msg"="Failed to get *v1beta1.DockerCluster" "error"="DockerCluster.infrastructure.cluster.x-k8s.io \"toto\" not found"
I0427 17:00:11.474903       1 dockercluster_controller.go:72] controller/dockercluster "msg"="Waiting for Cluster Controller to set OwnerRef on DockerCluster" "name"="toto" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerCluster"
I0427 17:00:11.651921       1 loadbalancer.go:119] controller/dockercluster "msg"="Creating load balancer container" "cluster"="toto" "ipFamily"=1 "name"="toto" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerCluster"
I0427 17:00:19.085623       1 dockermachine_controller.go:79] controller/dockermachine "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:00:19.120350       1 dockermachine_controller.go:79] controller/dockermachine "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:00:19.134291       1 dockermachine_controller.go:79] controller/dockermachine "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:00:19.144717       1 dockermachine_controller.go:79] controller/dockermachine "msg"="Waiting for Machine Controller to set OwnerRef on DockerMachine" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:00:19.192464       1 dockermachine_controller.go:221] controller/dockermachine "msg"="Waiting for the Bootstrap provider controller to set bootstrap data" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:00:19.209065       1 dockermachine_controller.go:221] controller/dockermachine "msg"="Waiting for the Bootstrap provider controller to set bootstrap data" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:00:19.226098       1 dockermachine_controller.go:221] controller/dockermachine "msg"="Waiting for the Bootstrap provider controller to set bootstrap data" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:00:19.264554       1 dockermachine_controller.go:221] controller/dockermachine "msg"="Waiting for the Bootstrap provider controller to set bootstrap data" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:00:19.297297       1 machine.go:218] controller/dockermachine "msg"="Creating control plane machine container" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:01:00.560068       1 loadbalancer.go:177] controller/dockermachine "msg"="Updating load balancer configuration" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"
I0427 17:01:30.633359       1 machine.go:406] controller/dockermachine "msg"="Setting Kubernetes node providerID" "name"="toto-control-plane-4pcsp" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"

clusterctl describe cluster toto
NAME                                                     READY  SEVERITY  REASON  SINCE  MESSAGE
Cluster/toto                                             True                     2m27s
├─ClusterInfrastructure - DockerCluster/toto             True                     3m42s
└─ControlPlane - KubeadmControlPlane/toto-control-plane  True                     2m27s
  └─Machine/toto-control-plane-l9p4d                     True                     2m28s

Check cluster created
clusterctl get kubeconfig toto > toto.cfg
kubectl --kubeconfig=./toto.cfg get nodes
NAME                       STATUS     ROLES                  AGE     VERSION
toto-control-plane-l9p4d   NotReady   control-plane,master   6m22s   v1.22.7

kubectl --kubeconfig=./toto.cfg get po -A

NAMESPACE     NAME                                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-78fcd69978-5n4n8                           0/1     Pending   0          6m33s
kube-system   coredns-78fcd69978-cbn9c                           0/1     Pending   0          6m33s
kube-system   etcd-toto-control-plane-l9p4d                      1/1     Running   0          6m42s
kube-system   kube-apiserver-toto-control-plane-l9p4d            1/1     Running   0          6m42s
kube-system   kube-controller-manager-toto-control-plane-l9p4d   1/1     Running   0          6m43s
kube-system   kube-proxy-dsxst                                   1/1     Running   0          6m33s
kube-system   kube-scheduler-toto-control-plane-l9p4d            1/1     Running   0          6m42s

kubectl --kubeconfig=./toto.cfg get po/coredns-78fcd69978-5n4n8 -n kube-system -o yaml
kubectl --kubeconfig=./toto.cfg describe po/coredns-78fcd69978-5n4n8 -n kube-system
Name:                 coredns-78fcd69978-5n4n8
Namespace:            kube-system
Priority:<PHONE_NUMBER>
Priority Class Name:  system-cluster-critical
Node:                 <none>
Labels:               k8s-app=kube-dns
                      pod-template-hash=78fcd69978
Annotations:          <none>
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-78fcd69978
Containers:
  coredns:
    Image:       k8s.gcr.io/coredns/coredns:v1.8.4
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nqqkm (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-nqqkm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  4s (x7 over 7m22s)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

We may need to pull in the latest CAPD to pick up any fixes.

Is it something that you plan to do and to release with TCE 0.12.0 ?

Unfortunately no, it will not be updated in the 0.12.0 release. We pick that up from tanzu-framework, and I believe that will not change any time soon. So it will be updated at some point, but I don't have a timeframe to even estimate yet.

Then, it should be documented that centos7 + docker version x.y are not supported

FYI. that works using as OS - Rocky-8
[snowdrop@k122-rocky8 ~]$ tanzu mc create -i docker -p dev mgmt-test

Validating the pre-requisites...
Identity Provider not configured. Some authentication features won't work.
WARNING: key is not a string: %!s(int=1)WARNING: key is not a string: %!s(int=1)
Setting up management cluster...
Validating configuration...
Using infrastructure provider docker:v1.0.1
Generating cluster configuration...
WARNING: key is not a string: %!s(int=1)WARNING: key is not a string: %!s(int=1)Setting up bootstrapper...
Bootstrapper created. Kubeconfig: /home/snowdrop/.kube-tkg/tmp/config_RBrisOAL
Installing providers on bootstrapper...
Fetching providers
Installing cert-manager Version="v1.5.3"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.0.1" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.0.1" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.0.1" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-docker" Version="v1.0.1" TargetNamespace="capd-system"
Start creating management cluster...
[cluster control plane is still being initialized: WaitingForControlPlane, cluster infrastructure is still being provisioned: WaitingForControlPlane]
cluster control plane is still being initialized: WaitingForControlPlane
cluster control plane is still being initialized: WaitingForKubeadmInit
Saving management cluster kubeconfig into /home/snowdrop/.kube/config
Installing providers on management cluster...
Fetching providers
Installing cert-manager Version="v1.5.3"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.0.1" TargetNamespace="capi-system"
I0428 14:43:45.847776   11441 request.go:665] Waited for 1.022961727s due to client-side throttling, not priority and fairness, request: GET:https://<IP_ADDRESS>:6443/apis/clusterinformation.antrea.tanzu.vmware.com/v1beta1?timeout=30s
Installing Provider="bootstrap-kubeadm" Version="v1.0.1" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.0.1" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-docker" Version="v1.0.1" TargetNamespace="capd-system"
I0428 14:43:56.331717   11441 request.go:665] Waited for 1.005903778s due to client-side throttling, not priority and fairness, request: GET:https://<IP_ADDRESS>:6443/apis/node.k8s.io/v1?timeout=30s
Waiting for the management cluster to get ready for move...
Waiting for addons installation...
Moving all Cluster API objects from bootstrap cluster to management cluster...
Performing move...
Discovering Cluster API objects
Moving Cluster API objects Clusters=1
Creating objects in the target cluster
Deleting objects from the source cluster
Waiting for additional components to be up and running...
Waiting for packages to be up and running...
Warning: Management cluster is created successfully, but some packages are failing. Failure while waiting for packages to be installed: waiting for 'metrics-server' Package to be installed
You can now access the management cluster tkg-mgmt-docker-20220428143702 by running 'kubectl config use-context tkg-mgmt-docker-20220428143702-admin@tkg-mgmt-docker-20220428143702'

Management cluster created!

You can now create your first workload cluster by running the following:

  tanzu cluster create [name] -f [file]

Some addons might be getting installed! Check their status by running the following:

  kubectl get apps -A

Checking for required plugins...
I0428 14:54:19.707285   11441 request.go:665] Waited for 1.159484912s due to client-side throttling, not priority and fairness, request: GET:https://<IP_ADDRESS>:6443/apis/coordination.k8s.io/v1?timeout=32s
I0428 14:54:29.707395   11441 request.go:665] Waited for 11.159302825s due to client-side throttling, not priority and fairness, request: GET:https://<IP_ADDRESS>:6443/apis/infrastructure.cluster.x-k8s.io/v1alpha3?timeout=32s
I0428 14:54:39.719516   11441 request.go:665] Waited for 9.17543638s due to client-side throttling, not priority and fairness, request: GET:https://<IP_ADDRESS>:6443/apis/packaging.carvel.dev/v1alpha1?timeout=32s
All required plugins are already installed and up-to-date

Able to reproduce this failure.

Downloaded CentOS-7-x86_64-Minimal-2009.iso and created a VM
Applied updates, installed Docker CE and verified working setup with docker run hello-world
Installed jq, kubectl and the v0.11.0 release of TCE
Attempted deployment using tanzu mc create -i docker -p dev mgmttest, deployment started, took a very long time, and eventually timed out
Looking at CAPD logs in the bootstrap cluster, similar error messages as seen in original report

As a test, then installed latest clusterctl and tried following the CAPI Quickstart to deploy a CAPD cluster that way. Able to successfully deploy.
Current CAPI version used was v1.1.3, but TCE release still on CAPI v1.0.1. Next step will be to clean up and downgrade clusterctl to the v1.0.1 version and see if I can repro the failure directly with that. If so, then can look at what changes have merged between versions and/or try deployments with some of the releases between those two versions to see where this changed.

Doing more testing. I've moved up to the 1.1.0 release, but I'm not 100% on whether this is a CAPD version issue or if the successes we've seen have just been luck of timing.
What I've observed so far is sometimes it gets stuck creating the haproxy container, sometimes it creates it fine along with the control-plane container, but then gets stuck creating the machine deployment container. With the LB container I have seen errors stating something to the effect that an IP address couldn't be assigned, but in all cases, the stuck container remains in the CREATED status and never proceeds to start:
$ docker ps -a
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS          PORTS                                NAMES
7270207b09fc   kindest/node:v1.22.1                 "/usr/local/bin/entr…"   20 minutes ago   Created                                              capi-quickstart-control-plane-c8xkp
78009ce52206   kindest/haproxy:v20210715-a6da3463   "haproxy -sf 7 -W -d…"   20 minutes ago   Up 20 minutes   35696/tcp, <IP_ADDRESS>:35696->6443/tcp   capi-quickstart-lb
2adc4efa2d6f   kindest/node:v1.23.4                 "/usr/local/bin/entr…"   24 minutes ago   Up 23 minutes   <IP_ADDRESS>:39110->6443/tcp            kind-control-plane

docker events shows the container create call, but nothing with the same container ID after that:
2022-04-28T14:18:13.168980447-05:00 container create 7270207b09fc1000c3e9579791fc1f69986f9a1ae8a468e15076e94d4cb3af6e (image=kindest/node:v1.22.1, io.x-k8s.kind.cluster=capi-quickstart, io.x-k8s.kind.role=control-plane, name=capi-quickstart-control-plane-c8xkp)

The capd-controller-manager logs show an error:
error creating container exec: Error response from daemon: Container 7270207b09fc1000c3e9579791fc1f69986f9a1ae8a468e15076e94d4cb3af6e is not running

Looking at the CAPD source, the only place in the process where it would be getting the "error creating container" message would be here: https://github.com/kubernetes-sigs/cluster-api/blob/4af7c4a4093d0deeed466a9a73fb7f63f9b23894/test/infrastructure/container/docker.go#L428
Which would mean the call to Docker API ContainerCreate returned an error. The printed error says the container "is not running", but that seems odd since that should be the case on the initial ContainerCreate call before calling ContainerStart.

Same thing with the current latest v1.1.3 clusterctl and CAPI providers. haproxy and control-plane containers running successfully. md-1 container in CREATED state and never started. capd-controller-manager logs show the same error.

@sbueringer - sorry to bother you on this - you've been very good with troubleshooting these CAPI things in the past so hoping you may at least have some pointers. If too busy just let me know and I can probably capture this in a CAPI issue.
Wondering if you could take a look at https://github.com/vmware-tanzu/community-edition/issues/4201#issuecomment-1112595826. Have you seen issues with CentOS 7? It does have a really old kernel compared to most other distros today, so that does worry me a little. But based on the logs I've been able to find, it looks like maybe some docker issue.
Any suggestions on further debugging?

No worries. To be honest I've never tried CAPI on CentOS.
Some comments:

Is it correct to assume that the issue observed in https://github.com/vmware-tanzu/community-edition/issues/4201#issuecomment-1112595826 is a separate issue and the one describe further above only occurs with CAPI v1.0.x?
I took a quick look at all commits to CAPD since CAPI v1.0.1 and there are no obvious related changes, but looks like the docker/docker library was bumped from v20.10.11 to v20.10.12
I think the error creating container exec error occurs here: https://github.com/kubernetes-sigs/cluster-api/blob/dece5b1d2b2778950a6cbef4acad9915770d2eb5/test/infrastructure/container/docker.go#L164 (in the code you linked the error has to starting with error creating container "

Based on that I assume that in RunContainer:

ContainerCreate works
ContainerAttach works
ContainerStart does not actually work (because the container is not running) but we don't notice (assuming there wasn't another error further up in the CAPD log)
I would assume that it now fails when we test that crictl ps works here: https://github.com/kubernetes-sigs/cluster-api/blob/fc52fe3d5cfe05d436a7d2f0ba3cf8a1399aad59/test/infrastructure/docker/docker/machine.go#L248-L257
I looked through the .Run calls and it looked like they should all wrap the error creating container exec error to give us more context where it occurred, but looks like your error wasn't wrapped with more context
So two things:

Why didn't ContainerStart recognize that it didn't work (unfortunately no idea how to debug this further)
Where did the exec error actually occur, I'm surprised that we don't have more error context

If I have an env where I can reproduce it, I would probably either a) add more log statements to CAPD or b) run CAPD with debug symbols and add a break point at the error and go backwards from there

A while ago I had a similar issue, essentially when I created to many clusters CAPD was only able to create containers but they weren't started anymore (I assume because of resource exhaustion). Should be unrelated, but just to mention it.

If needed @sbueringer, I can give you access to a centos7 VM running on Hetzner ;-)
Could the issue be related to a  problem with the HAProxy container which is restarting several times (255) and finally exits ?
[snowdrop@k122-centos7 ~]$ docker ps -a
CONTAINER ID   IMAGE                                                         COMMAND                  CREATED          STATUS                            PORTS                                  NAMES
61c34949056a   projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1   "/usr/local/bin/entr…"   17 minutes ago   Up About a minute                 45756/tcp, <IP_ADDRESS>:45756->6443/tcp   tkg-mgmt-docker-20220429073405-control-plane-wcp6w
5ab552c2b6c8   kindest/haproxy:v20210715-a6da3463                            "haproxy -sf 7 -W -d…"   17 minutes ago   Exited (255) About a minute ago   33301/tcp, <IP_ADDRESS>:33301->6443/tcp     tkg-mgmt-docker-20220429073405-lb
b442509a13e6   projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1   "/usr/local/bin/entr…"   19 minutes ago   Up About a minute                 <IP_ADDRESS>:40752->6443/tcp              tkg-kind-c9lnhjsvqovshtogvj2g-control-plane

[snowdrop@k122-centos7 ~]$ docker logs 5ab552c2b6c8
[WARNING] 118/053627 (1) : config : missing timeouts for frontend 'controlPlane'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[NOTICE] 118/053627 (1) : New worker #1 (8) forked
[WARNING] 118/053633 (1) : Reexecuting Master process
[NOTICE] 118/053633 (1) : haproxy version is 2.2.9-2~bpo10+1
[NOTICE] 118/053633 (1) : path to executable is /usr/sbin/haproxy
[ALERT] 118/053633 (1) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
[NOTICE] 118/053633 (1) : New worker #1 (27) forked
[WARNING] 118/053633 (8) : Stopping frontend controlPlane in 0 ms.
[WARNING] 118/053633 (8) : Stopping backend kube-apiservers in 0 ms.
[WARNING] 118/053633 (8) : Stopping frontend GLOBAL in 0 ms.
[WARNING] 118/053633 (8) : Proxy controlPlane stopped (cumulated conns: FE: 9, BE: 0).
[WARNING] 118/053633 (8) : Proxy kube-apiservers stopped (cumulated conns: FE: 0, BE: 9).
[WARNING] 118/053633 (8) : Proxy GLOBAL stopped (cumulated conns: FE: 0, BE: 0).
[WARNING] 118/053633 (1) : Former worker #1 (8) exited with code 0 (Exit)
[WARNING] 118/053633 (27) : Server kube-apiservers/tkg-mgmt-docker-20220429073405-control-plane-wcp6w is DOWN, reason: Layer4 connection problem, info: "SSL handshake failure", check duration: 0ms. 0 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 118/053633 (27) : backend 'kube-apiservers' has no server available!
[WARNING] 118/053644 (1) : Reexecuting Master process
[NOTICE] 118/053644 (1) : haproxy version is 2.2.9-2~bpo10+1
[NOTICE] 118/053644 (1) : path to executable is /usr/sbin/haproxy
[ALERT] 118/053644 (1) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
[WARNING] 118/053644 (27) : Stopping frontend control-plane in 0 ms.
[WARNING] 118/053644 (27) : Stopping backend kube-apiservers in 0 ms.
[WARNING] 118/053644 (27) : Stopping frontend GLOBAL in 0 ms.
[WARNING] 118/053644 (27) : Proxy control-plane stopped (cumulated conns: FE: 18, BE: 0).
[WARNING] 118/053644 (27) : Proxy kube-apiservers stopped (cumulated conns: FE: 0, BE: 18).
[WARNING] 118/053644 (27) : Proxy GLOBAL stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE] 118/053644 (1) : New worker #1 (46) forked
[WARNING] 118/053644 (1) : Former worker #1 (27) exited with code 0 (Exit)
[WARNING] 118/053644 (46) : Server kube-apiservers/tkg-mgmt-docker-20220429073405-control-plane-wcp6w is DOWN, reason: Layer4 connection problem, info: "SSL handshake failure", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 118/053644 (46) : backend 'kube-apiservers' has no server available!

@cmoulliard Thx, I think Sean was able to reproduce which should be okay for now. I'm at an internal conference next week, but maybe it's relevant for later.
I think unfortunately the haproxy restart issue is yet another issue.

the one describe further above only occurs with CAPI v1.0.x?

That was my original hypothesis, but I ended up testing with the current latest and greatest and was still able to observe the problem. So this is a current problem with CentOS 7 and CAPI v1.1.3.

I think the error creating container exec error occurs here...

Oops, you are correct, I was missing the exec part of that message. So the ContainerCreate and ContainerStart calls are not returning an error, but for some reason the ContainerStart doesn't appear to actually work.
So the error appears to be somewhere internally to Docker SDK and Docker Engine. And since we are not seeing this elsewhere, it also appears to be some sort of interaction triggered by being on a system with this older kernel or some other environmental thing with CentOS 7.
For now, we may have to just state CentOS 7 isn't supported (for both TCE and CAPD projects). It would be nice to better understand the cause of the failure, but I think that's going to take some work. I may write some test code that extracts that Docker SDK usage into a smaller test facility to try to isolate things, but likely won't be able to get to that for a while.

Supporting all the OS is impossible as you mentioned !
It could be nevertheless interesting to figure out why CAPD fails as this is perhaps due to an issue with docker client and/or containerd and/or daemon ... Such a situation will certainly occurs in the future using podman | docker | cri-o and/or different OS combinations & versions. So having more test cases or testing coverage will help. The question is : Should such tests take place here or part of the Kubernetes Cluster API project ?

The failed command is:

I extracted the content of the bootstrap data which is executed and where the error sysctl: cannot stat /proc/sys/net/ipv4/tcp_retries2: No such file or directory was reported. That could maybe help you.
I0427 07:09:35.998677       1 machine.go:345] controller/dockermachine "msg"="Failed running command"
"name"="tkg-mgmt-docker-20220427084538-control-plane-lmmnq"
"namespace"="tkg-system"
"reconciler group"="infrastructure.cluster.x-k8s.io"
"reconciler kind"="DockerMachine" "bootstrap data"="

and  here is what we got when we decode the base64 content - https://gist.github.com/cmoulliard/da125ea8cb1af476550486b79aea285e
...
runcmd:
  - "curl https://get.docker.com/builds/Linux/x86_64/docker-1.12.0.tgz | tar zxOf - docker/docker > /usr/local/bin/docker;chmod +x /usr/local/bin/docker"
  - "peerIdx=$(ip link | grep eth0 | awk -F[@:] '{ print $3 }' | cut -c 3-);peerName=$(docker run --rm --net=host --entrypoint=/bin/ip projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1 link | grep ^\"$peerIdx\":| awk -F[:@] '{ print $2 }' | cut -c 2-);docker run --rm --net=host --privileged --entrypoint=/usr/sbin/ethtool projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1 -K \"$peerName\" tx off"
  - "sysctl -w net.ipv4.tcp_retries2=4"
  - 'kubeadm init --config /run/kubeadm/kubeadm.yaml  && echo success > /run/cluster-api/bootstrap-success.complete'
@cmoulliard

@stmcginnis

NOTE: I did a test on Centos7 using CAPI 1.0.4 and that works https://gist.github.com/cmoulliard/a3c2787d3bf412272300d12d836d0b98

fyi, I'm seeing similar errors with cluster-api 1.1.3 and the latest ubuntu/docker.  Works on ubuntu 18, but not on 22.

interestingly for me, I finally tried moby-server (built by microsoft) on ubuntu 20 and it works...the snap docker didn't work, nor did the docker-ce version...no idea what the difference is since I could not find any error log in capd's logs, just blank stderr/stdout on the failed docker command.

the snap docker didn't work

Do you also observe the same issue which is that bootstrap is failing due to The failed command is: sysctl -w net.ipv4.tcp_retries2=4 with the error sysctl: cannot stat /proc/sys/net/ipv4/tcp_retries2: No such file or directory. ?

I get errors in capd like this, resulting in the worker node being in Created status but not running, notice the empty stderr/stdout on the command:
I0504 13:10:58.919189       1 machine.go:362] controller/dockermachine "msg"="Failed running command" "name"="coffee-md-0-6cfpb" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine" "bootstrap data"="IyMgdGVtcGxhdGU6IGppbmphC......ZpZy55YW1sICAmJiBlY2hvIHN1Y2Nlc3MgPiAvcnVuL2NsdXN0ZXItYXBpL2Jvb3RzdHJhcC1zdWNjZXNzLmNvbXBsZXRlCg==" "command"={"Cmd":"mkdir","Args":["-p","/run/kubeadm"],"Stdin":""} "stderr"="" "stdout"=""
I0504 13:10:58.919814       1 machine.go:505] controller/dockermachine "msg"="Got logs from the machine container" "name"="coffee-md-0-6cfpb" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine" "output"="Inspected the container:\n{ContainerJSONBase:0xc000bfba20 Mounts:[{Type:bind Name: Source:/lib/modules Destination:/lib/modules Driver: Mode:ro RW:false Propagation:rprivate} {Type:volume Name:e70a22694db10d038c2d8518edc603fe604ba33bd7e9e45b0b2dd43dfca59bbd Source:/var/lib/docker/volumes/e70a22694db10d038c2d8518edc603fe604ba33bd7e9e45b0b2dd43dfca59bbd/_data Destination:/var Driver:local Mode: RW:true Propagation:}] Config:0xc000a8e640 NetworkSettings:0xc0005b3800}\nGot logs from the container:\n"
E0504 13:10:58.920028       1 controller.go:317] controller/dockermachine "msg"="Reconciler error" "error"="failed to exec DockerMachine bootstrap: failed to run cloud config: error creating container exec: Error response from daemon: Container 6d663f0ca3720a5446af4858310f7e05b6b6bdb0fc862ebd577d9922e639ce51 is not running" "name"="coffee-md-0-6cfpb" "namespace"="default" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerMachine"

6d663f0ca3720a5446af4858310f7e05b6b6bdb0fc862ebd577d9922e639ce51

Can you check the log of this container to see if it reports an error ?

I did a test within the Centos7 VM using directly clusterctl and that works :-)

This didn't work for me on Centos 7.9 @cmoulliard I got a bunch of missing features
➜  ~ clusterctl generate cluster toto --kubernetes-version v1.22.7 --flavor development | kubectl apply -f -
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinetemplate created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/quick-start-default-worker-bootstraptemplate created
Error from server (spec: Forbidden: can be set only if the ClusterTopology feature flag is enabled): error when creating "STDIN": admission webhook "validation.clusterclass.cluster.x-k8s.io" denied the request: spec: Forbidden: can be set only if the ClusterTopology feature flag is enabled
Error from server (spec: Forbidden: can be set only if the ClusterTopology feature flag is enabled): error when creating "STDIN": admission webhook "validation.dockerclustertemplate.infrastructure.cluster.x-k8s.io" denied the request: spec: Forbidden: can be set only if the ClusterTopology feature flag is enabled
Error from server (spec: Forbidden: can be set only if the ClusterTopology feature flag is enabled): error when creating "STDIN": admission webhook "validation.kubeadmcontrolplanetemplate.controlplane.cluster.x-k8s.io" denied the request: spec: Forbidden: can be set only if the ClusterTopology feature flag is enabled
Error from server (InternalError): error when creating "STDIN": admission webhook "default.cluster.cluster.x-k8s.io" denied the request: Internal error occurred: Cluster toto can't be validated. ClusterClass quick-start can not be retrieved: ClusterClass.cluster.x-k8s.io "quick-start" not found


Build: correct emitted output file map

This adjusts the output file map writing code to ensure that we emit the
proper native path string and correct escaping of \ which otherwise
yields invalid JSON which cannot be deserialized subsequently.  This
will be critical to enabling the use of swift-driver with SPM.
[One line description of your change]
Motivation:
[Explain here the context, and why you're making that change. What is the problem you're trying to solve.]
Modifications:
[Describe the modifications you've done.]
Result:
[After your change, what will change.]

@swift-ci please smoke test


STS calls should be able to be regionalized

The default endpoint for STS is hosted in us-east-1 (N. Virginia).  We should allow for the sts request to be generated for the specific region the cluster is in.  This could greatly reduce latency for the call.
More info on regions

/close

This was released in 0.4.0.

The feature in v0.4.0 (https://github.com/kubernetes-sigs/aws-iam-authenticator/pull/173) was actually server-side support for tokens and regional endpoints, so this issue is still relevant.

Implemented in https://github.com/kubernetes-sigs/aws-iam-authenticator/commit/7277af9589e335769272e93d92491f3d0c04be6d


Joining a group does not update offsets

Version of Ruby: 2.4.1
Version of Kafka: 10.1
Version of ruby-kafka: 0.4.1

Given the following:

1 broker with topic "test" which has 3 partitions
3 consumer processes (one consumer group), with the following pseudo-code:

consumer.subscribe(...)
some_offsets_hash.each { |k, v| consumer.seek("test", k, v)
consumer.each_message { |m| ... }

When all processes are started at the same time, sometimes all partitions are assigned to one process only, and it starts fetching messages before a rebalance is triggered and commits the offsets. If that happens, then after the rebalance (now every process is assigned one partition), the 2 processes which did not process any messages so far start from the position that seek set, not from the offsets that were committed.
This looks like a bug to me - I would have expected that joining a group results in offsets being fetched from the broker again.

It looks like a bug to me, as well. Unfortunately, we cannot commit those offsets until we've joined the group – but I guess the right course of action would be to immediately commit those offsets once we have.

@dasch can't we refetch the offset upon joining?

Not if they haven't been committed yet :)

@dasch I'm not sure I'm following. One process (let's call it A), is fetching messages and commits offsets, while the other two (B and C) do nothing. Then a rebalance is triggered. B and C are assigned one partition and start from the original offsets, not from the ones that A committed. I thought we could simply fetch offsets from the broker whenever a consumer joins a goup?

Re-reading the original comment, this doesn't actually sound like a bug to me – seek is supposed to overwrite the offsets that have been committed. Seeking just once is a bit tricky, since the overwriting of offsets must happen while the consumer is part of the group.
Probably a better way to seek a group is to shut it down, then launch a script that joins the group, seeks the partitions to the desired offsets, then quits, after which you can restart the group and have it resume from those offsets.
This would probably require changing seek or adding a variation of it that actually joins the group. It's not simple, though.


Deprecate warmup_apps.sh in favour of warmup_apps

For consistency with /app/bin/livebook.

We need to update the generated docker files to use warmup_apps instead of warmups_apps.sh.

There is also one mention of warmup_apps.sh in the README. :)

Whoops, udpated!