--prefer-free-formats does not prefer webm to mp4

$ youtube-dl 'https://www.youtube.com/watch?v=r9rGX91rq5I' -v -F
[debug] System config: [u'--prefer-free-formats']
[debug] User config: []
[debug] Command-line args: [u'https://www.youtube.com/watch?v=r9rGX91rq5I', u'-v', u'-F']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2<IP_ADDRESS>
[debug] Python version 2.7.10 - Linux-4.0.5-300.1.kparal.fc22.x86_64-x86_64-with-fedora-22-Twenty_Two
[debug] exe versions: ffmpeg 2.6.3, ffprobe 2.6.3
[debug] Proxy map: {}
[youtube] r9rGX91rq5I: Downloading webpage
[youtube] r9rGX91rq5I: Extracting video information
[youtube] r9rGX91rq5I: Downloading DASH manifest
[info] Available formats for r9rGX91rq5I:
format code  extension  resolution note
171          webm       audio only DASH audio  108k , audio@128k (44100Hz), 3.84MiB
140          m4a        audio only DASH audio  127k , m4a_dash container, aac  @128k (44100Hz), 4.76MiB
278          webm       256x144    DASH video   62k , webm container, VP9, 15fps, video only, 1.23MiB
242          webm       426x240    DASH video  112k , 30fps, video only, 1.88MiB
160          mp4        256x144    DASH video  122k , 15fps, video only, 3.37MiB
243          webm       640x360    DASH video  205k , 30fps, video only, 3.59MiB
244          webm       854x480    DASH video  281k , 30fps, video only, 6.15MiB
134          mp4        640x360    DASH video  297k , 30fps, video only, 3.44MiB
133          mp4        426x240    DASH video  307k , 30fps, video only, 7.54MiB
135          mp4        854x480    DASH video  626k , 30fps, video only, 7.05MiB
247          webm       1280x720   DASH video  645k , 30fps, video only, 14.85MiB
248          webm       1920x1080  DASH video 1204k , 30fps, video only, 27.42MiB
136          mp4        1280x720   DASH video 1286k , 30fps, video only, 14.79MiB
137          mp4        1920x1080  DASH video 2521k , 30fps, video only, 29.69MiB
271          webm       2560x1440  DASH video 3741k , 30fps, video only, 66.79MiB
264          mp4        2560x1440  DASH video 5990k , 30fps, video only, 96.12MiB
313          webm       3840x2160  DASH video 9743k , VP9, 30fps, video only, 157.88MiB
266          mp4        3840x2160  DASH video 10775k , h264, 30fps, video only, 152.25MiB
17           3gp        176x144
36           3gp        320x240
5            flv        400x240
18           mp4        640x360
43           webm       640x360
22           mp4        1280x720   (best)

$ youtube-dl 'https://www.youtube.com/watch?v=r9rGX91rq5I' -v --get-format
[debug] System config: [u'--prefer-free-formats']
[debug] User config: []
[debug] Command-line args: [u'https://www.youtube.com/watch?v=r9rGX91rq5I', u'-v', u'--get-format']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2<IP_ADDRESS>
[debug] Python version 2.7.10 - Linux-4.0.5-300.1.kparal.fc22.x86_64-x86_64-with-fedora-22-Twenty_Two
[debug] exe versions: ffmpeg 2.6.3, ffprobe 2.6.3
[debug] Proxy map: {}
266 - 3840x2160 (DASH video)+140 - audio only (DASH audio)

Please note that [debug] System config: [u'--prefer-free-formats'] shows that I have preference to free formats enabled, but youtube-dl wants to download mp4 instead (266+140).

This is expected behavior. --prefer-free-formats is only respected when non-free and free formats are of the same quality. 266 (mp4) has higher bitrate than 313 (webm) therefore is selected. You may want to use -f bestvideo[ext=webm]+bestaudio[ext=webm]/best[ext=webm].

I think the option documentation should mention that. Right now it reads:
Prefer free video formats unless a specific one is requested

The bitrate will never be exactly the same, it will always differ slightly. But higher bit rate doesn't imply higher quality, it depends on the codec. So that assumption is somewhat off.
But as @rg3 says, that option is not about quality, at least as it is currently documented. It says "pick the free format, if it is available". It does not speak about quality at all. So maybe the current implementation is expected by the developers, but certainly not by the users.
And, if only quality is concerned, what would be the point of such an option? If mp4 has slightly better bitrate, it is picked. If webm has slightly better bitrate, it is picked. Such option would have no effect at all, because it is already the default behavior!
So really, this option should mean "I always want free formats, if they are available, otherwise fall back to proprietary formats". That's how I want to configure youtube-dl. Please make it possible. Thank you.

You may want to use -f bestvideo[ext=webm]+bestaudio[ext=webm]/best[ext=webm].

I can use this, but I guess that will fail for videos which don't have a webm version. So I would have to do this:
-f bestvideo[ext=webm]+bestaudio[ext=webm]/best[ext=webm]/bestvideo+bestaudio/best
However, when I download videos for my tablet (not PC), I want to avoid 4K videos. So I need height filters. And then the filter gets really complicated.
I would much rather have --prefer-free-formats work as documented, than very complicated command line arguments.

Opening as I think --prefer-free-formats should behave as documented.

@kparal --prefer-free-formats won't anyhow help you avoiding 4K, you'll need to use filters anyway.
@yan12125 I don't think this behavior should be changed since it's an expected behavior from the beginning and changing it will break in general the main youtube-dl's purpose - downloading best available quality. Documentation however should be clarified, yes. What you suggest is an --require-free-formats option that was already rejected previousy.

What I'm saying is that there's no easy way to prefer free formats (regardless of quality, literally prefer) without writing an explosive set of command line arguments. That's also the case of aforementioned #2196 - it gives a workaround, but it's neither simple nor clean, if you want to cover all the cases. (And I don't think anything was rejected - "you are welcome to contribute a patch that adds one").
@dstftw Can you please clarify what is the purpose of --prefer-free-formats as you see it? I still don't understand it. Does it work only for formats with exactly the same resolution/fps/bitrate? Doesn't that have like 0.00001% chance of occurring? Does it make sense to have an option that almost never applies, because bitrate is never exactly the same? What is the logic when that option is not given - a random format is picked, or always a proprietary one?
In my understanding "require" has different meaning than "prefer". When you require a free format, you pick the best free format, and if it is not available, you fail with an error ("free format required, but none available"). When you prefer a free format, you pick the best free format, and if it is not available, you pick the best non-free format.

One further argument - some Linux distributions like Fedora add --prefer-free-formats to the default configuration file, see here:
http://pkgs.fedoraproject.org/cgit/youtube-dl.git/tree/youtube-dl.conf
The reason is that h264/aac/other codecs are neither installed by default nor offered in the repositories, because of patent issues. So the wish of the package maintainers of youtube-dl is to allow all Fedora/other distro users to make as many videos playable by default as possible - that means they want to configure youtube-dl to download free formats every time it is possible to do so. So that it is really useful for all users, and not just for those who live outside of USA + other countries where software patents apply and installed proprietary codecs. They don't want to require it, because downloading something is better than downloading nothing, but they want to prefer it as much as possible.

I agree with @kparal, I think that with --prefer-free-formats if the formats have the same resolution it should choose the webm. I don't think that giving higher priority to the filesize and bitrate is the best solution, I don't really know how useful is comparing them for diferent containers.

@kparal I've already stated the purpose of it - downloading free format when both formats are equal according to metadata describing quality. youtube-dl supports not only YouTube so the probability is much higher especially if some metadata is not available.

What is the logic when that option is not given - a random format is picked, or always a proprietary one?

Huh? Not a random but best quality format according to metadata.
@jaimeMF for YouTube giving less priority to filename and bitrate is likely to be reasonable, but in general probably not.

I believe (so basically I'm inventing stuff as I write, haha) people using this option are probably interested in downloading free formats if available, just like the current option documentation suggests.
So if a given website (YouTube or other) offers videos in either MP4 or Webm, using --prefer-free-formats should pick a Webm one, and should only download MP4 if no free formats are available.
Once that "format filter" has been applied, it should choose the highest quality one, or obey other format restrictions the user may have specified.
But then we can raise other questions. For example, someone may want to have --prefer-free-formats in the config file, but in a specific case (s)he may want to specify a format filter that wouldn't work with that, so they need a second option to override the config file setting in a given program call (as in --no-prefer-free-format).
Just my two cents. You know I don't want to have any voting weight in the development process and I don't even use that option, so it's irrelevant to me. :P

I've already stated the purpose of it - downloading free format when both formats are equal according to metadata describing quality. youtube-dl supports not only YouTube so the probability is much higher especially if some metadata is not available.

So IIUIC --prefer-free-formats has no effect on YouTube videos, because full metadata is available, and the videos are always at least slightly different (the bitrate is never the same), so it never happens that this option is taken into account. It can happen on other video sites, which don't provide full metadata, but not on YouTube. That's... suboptimal.
And it doesn't help at all to those users who don't have proprietary codecs installed, and want to download free fomats without hand-picking them every time.
@dstftw For you, quality might be the most important factor. But this belief is not universal and other groups of users might have different priorities. Like open formats. I'm one of them, I believe that patented formats need to be purged from this world and I'll gladly sacrifice some quality because of it. It would be great to have an option which would allow such groups of users to use youtube-dl easily. It can be a new option, if needed (even though in my personal opinion it would make most sense to just use the current one, especially since it currently doesn't seem to have any effect on the largest and most widely used video site).

--prefer-free-formats is expected to pick free formats only if they have the same quality, which is the tricky point, and I think that's the correct behaviour. If someone wants to really prefer the free formats (even if webm is only 1080 and mp4 is 4k) they can use a custom -f or we could add an additional flag.
@dstftw I think we should only take into account the resolution for --prefer-free-formats, in the case that two formats with the same extension exist we would still use the bitrate and other fields to sort them, I don't think it would be a big problem.

@jaimeMF if some hoster serves formats of the same resolution but really bad bitrate for free format it will end up selecting free format for --prefer-free-formats that is obviously wrong since they are not the same quality and it violate

--prefer-free-formats is expected to pick free formats only if they have the same quality

So, with your suggestion the general semantics of --prefer-free-formats changes to: select free formats when resolutions are equal and not formats of the same quality. In this situation (if we change the semantics of the option) I guess it's indeed more reasonable to change it to what @kparal suggested: select best free format if available otherwise fallback to any best.

@jaimeMF if some hoster serves formats of the same resolution but really bad bitrate for free format it will end up selecting free format for --prefer-free-formats that is obviously wrong since they are not the same quality and it violates

Yes, that's true, but my point is that deciding if two formats have the same quality (for --prefer-free-formats)  is complicated and I would choose to use the resolution. For example, from the output posted by @kparal:
248          webm       1920x1080  DASH video 1204k , 30fps, video only, 27.42MiB
136          mp4        1280x720   DASH video 1286k , 30fps, video only, 14.79MiB
137          mp4        1920x1080  DASH video 2521k , 30fps, video only, 29.69MiB

It prefers 720 mp4 over 1080 webm based just on the bitrate, for me it's more sensible to prefer resolution. But I understand it's not so simple (and my knowledge is quite limited, so I may be completely wrong).

So, with your suggestion the general semantics of --prefer-free-formats changes to: select free formats when resolutions are equal and not formats of the same quality. In this situation (if we change the semantics of the option) I guess it's indeed more reasonable to change it to what @kparal suggested: select best free format if available otherwise fallback to any best.

I wouldn't say it changes, it would "fix" it, since that's how it was supposed to work (at least that's my impression when looking into the original implementation, it seem that it picked the mp4 file if it was the one with the highest resolution). Additionally with that behaviour you could use formats filters like best[height<720] and you would get the format with the height closest to the requested and it would try to pick the free version. With the proposed behaviour it would pick the closest free format, which may not be the one with the highest height.
Anyways, since this is probably a bit subjective and I don't use that option, I don't really mind what it does as long as we clearly document what it does and add tests to verify we don't break it.

Thanks @jaimeMF my interpretation of --prefer-free-formats is the same. However, I can't speak for others because I don't use it.
Prioritizing resolutions than bitrates is reasonable. Technically free formats often have lower bitrates than non-free formats. Take VP9 for example, it aims to "Reduce video bitrate by 50% with image quality comparable to VP" ( VP9 requirements ). In other words, bitrates is not a synonym of quality. In real cases, it's uncommon for VP9 to have higher bitrates than H264. As a result, relying on bitrates is problematic.

To convert WebM to MP4, you may try iDealshare VideoGo.
It support batch convert WebM to MP4 or convert MP4 to WebM.
In fact it can convert between WebM, MP4, AVI, MOV, FLV, MPG, SWF, WMV, MKV, etc.
You may find it at http://www.idealshare.net/video-converter/convert-webm-to-mp4.html

I always thought that --prefer-free-formats would prefer free formats over proprietary formats at all costs. If I'm specifying this option, odds are that I either:

Want to use a free format for ideological reasons
Don't have proprietary codecs supported on my system

So I think it's fair to choose a lower-quality free format over a higher-quality fair proprietary format when this option is specified.

That should probably be a separate flag, such as "--only-free-formats".
The main issue was that while getting only free formats was relatively easy ("-f bestvideo[ext=webm]+bestaudio[ext=webm]/best[ext=webm]"), it got really complicated if you wanted to get the highest resolution video, and then prefer the free format if it had one which would lead to such horrors as:
-f bestvideo[ext=webm,height=1080]+bestaudio[ext=webm]/bestvideo[ext=webm,height=720]+bestaudio[ext=webm]/bestvideo[ext=webm,height=640]+bestaudio[ext=webm]/bestvideo[ext=webm,height=480]+bestaudio[ext=webm]/bestvideo[ext=webm,height=240]+bestaudio[ext=webm/best[ext=webm]
with entries for every possible video size you could expect. This was further complicated by the fact that youtube periodically introduces higher resolutions (such as 1440), and things like 3D videos, which mean you have to continually retune the script to add new resolutions to a massive list.

This seems like a conflict of interests. If you make this behavior a separate flag, then --prefer-free-formats no longer has a solid reason to exist.
There's no inherent reason to prefer free formats over proprietary formats other than ideological ones.
The way I see it, the gigantic format string you are describing is born from a need to work-around a different youtube-dl limitation: The fact that it doesn't understand that VP9 is higher quality than AVC even at lower bitrates.
But why are we spending so much time focusing on a work-around instead of improving the core problem here?
Note: I use a similar such work-around, namely:
(bestvideo[vcodec=vp9]/bestvideo[height>720]/bestvideo[fps>30])[tbr<13000]+(bestaudio[acodec=opus]/bestaudio)/best
But I consider this a pure work-around simply because I really don't care if my video stream is VP9 or AVC. I'm just special-casing VP9 because of aforementioned youtube-dl bug.

This seems like a conflict of interests. If you make this behavior a separate flag, then --prefer-free-formats no longer has a solid reason to exist.

Yes it does. Some videos have mp4 formats in better resolution (and/or audio quality) than in webm, --prefer-free-formats now values resolution, and then format, in that order.
One such video off the top of my head is:
https://www.youtube.com/watch?v=9n7viZaniTU
Which has 720p video and 255k audio in mp4, but the webm is only available as 360p video with combined audio of unspecified bitrate.
Sure, this is not a super common scenario, especially with newer videos, but it's does happen.

There's no inherent reason to prefer free formats over proprietary formats other than ideological ones.

Sure, but there's also a reason to prefer higher resolution videos over lower resolution videos: quality.
People often have lots of different ideological preferences, and they can intersect in different ways for different people.

I'm with @AliceTheGorgon, --only-free-formats would please those who only want free formats (the name looks clearer) while the fixed --prefer-free-formats would be for those who prefer a higher resolution over the format used (as I pointed in https://github.com/rg3/youtube-dl/issues/6018#issuecomment-113958960, that seems to be how the original implementation worked).

Yes it does. Some videos have mp4 formats in better resolution (and/or audio quality) than in webm, --prefer-free-formats now values resolution, and then format, in that order.

This argument makes no inherent sense. If you care about quality over the format, then why are you specifying --prefer-free-formats at all? Just leave away the flag and you will get the best quality stream.(*)
(*) Except for the part where youtube-dl is currently bugged and prefers AVC over VP9 even when the latter would be higher quality, of course.

Which has 720p video and 255k audio in mp4, but the webm is only available as 360p video with combined audio of unspecified bitrate.

In this case I would expect --prefer-free-formats to pick the 360p WebM stream. After all, the 1080p MP4 version is clearly proprietary.

This is not a decision that concerns youtube-dl so I don't understand why you mention it.

--prefer-free-formats just specifies a precedence, not necessarily an exclusion. So, given a video with a 720p stream AVC and 360p WebM, I would assume to get the 720p version, because resolution/quality takes priority over format here, and only if both streams have the same (or let's say similar) resolution, the free format would have precedence.
That's how I understand it, so --prefer-free-formats and --only-free-formats both make sense. I don't know how often this option is used in the wild, so this maybe an issue with barely any relevance, but that's another topic.
I'm a bit confused about the quality discussion, on the other hand, because MP4 is either a) the name of the standard or b) the name of the container format, which is part of this standard, uses the .mp4 file extension, has two different versions, and if that would be not enough confusion, MP4/MPEG-4 is divided into different parts which specify quite different things.
So, my point is, MP4 itself does not say anything about the video quality. MP4 Part 10 (also known as ITU-T H.264) specifies the AVC compression, and you can compare AVC against VP9, true. But an MP4 container can also contain a stream using MPEG-H (MPEG-H is basically the successor, the next MPEG in the line, so to speak) Part 2 (also known as H.265/HEVC), and that codec is vastly more efficient than AVC, and thus changing the quality game in comparison to VP9.
I'm not in the video encoding business anymore (okay, was just a hobbyist), but back then x264 was considered the best implementation of AVC, and even better than VP9, but they might caught up to 264 in the meantime..

--prefer-free-formats just specifies a precedence, not necessarily an exclusion. So, given a video with a 720p stream AVC and 360p WebM, I would assume to get the 720p version, because resolution/quality takes priority over format here, and only if both streams have the same (or let's say similar) resolution, the free format would have precedence.

What people argue here is that if you use such an option, the format itself should be the first criterion, not resolution, bitrate, codec or anything else. I'd also expect that when I use --prefer-free-formats, I receive the highest quality free format available (in this case it would mean 360p webm), and only if there's no free format available, then I receive highest non-free format available (720p avc). With --only-free-formats, I'd expect the same thing, just receiving an error in case there's no free format available, instead of a non-free format.

There's no inherent reason to prefer free formats over proprietary formats other than ideological ones.

I'd like to say that this is mostly true, but not always. This can be easily an economic reason. There are countries in which it is illegal for you to obtain a proprietary format decoder (even an open source one) without paying royalties to the patent holders, because of software patentability. The most obvious example here is USA. So if your operating system doesn't come with such codecs preinstalled, you don't want to purchase them separately, and don't want to break the law, you'll want --prefer-free-formats to actually prefer free formats as the first criterion, because those are the only ones that you can play.

for what it's worth: I expect --prefer-free-formats to only prefer free formats when there's actual choice for ytdl to make. In other words: I fall into "if same resolution available, prefer free format". If however I did specify "bestvideo" selector, it should pick highest resolution available, no matter what format, free or not.
so for
--prefer-free-formats
-f bestvideo[height<=?1080]+bestaudio

it should pick webm variant if available in 1080, even if that has lower bitrate than mp4 (as stated bitrate is bad indicator between codecs).
if webm is only available in 360p, but proprietary is available in higher solution, I expect the "bestvideo" statement to override the mere preference for free formats and result in the 1080p mp4 variant instead.
And option that would hand out 360p webm version instead of 1080p mp4 would need to be "--force-free-formats" or "--only-free-formats", but that can also be more easily selected using the extension filter, without duplicating each statement. (so far less need for such an "enforce" option)

^ That behavior might be more intuitive naming-wise, but at the same time occam's razor tells me it's not a possibility that should be under consideration.
When you think about it, the current --prefer-free-formats exists only to serve the needs of somebody who “cares” about freedom, except when it inconveniences them :p

When you think about it, the current --prefer-free-formats exists only to serve the needs of somebody who “cares” about freedom, except when it inconveniences them :p

Your definition is no better. You just draw the "inconvenience" line at "I still want the video even if there's no free version". Everyone else draws it at "I want the highest resolution video, and hopefully there's a free format at that resolution."

Your definition is no better. You just draw the "inconvenience" line at "I still want the video even if there's no free version". Everyone else draws it at "I want the highest resolution video, and hopefully there's a free format at that resolution."

Which is why the only sane thing to do (IMO) is remove this flag entirely and provide an --only-free-formats flag.

Which is why the only sane thing to do (IMO) is remove this flag entirely and provide an --only-free-formats flag.

 You know that a bunch of people want to be able to have the highest quality video, and have the free format of that quality, and that this flag is very useful to them.
There is no way to replace this tag without a massively convoluted set of preferences, whereas it's trivial to only select the free formats with a comparatively simple "f=bestaudio[ext=webm]+bestvideo[ext=webm]/best[ext=webm]".

What is going on here? Taking this to a personal level does not make a compelling argument..
It's not that this specific option enables a functionality of the program that isn't achievable otherwise.
Why are we even talking about weird definitions, that don't make any sense and are contradictory?
If you think that a specific option should result in _not getting any video at all _ in the end, then please explicitly say so. Kinda defeats the whole purpose of the program, but whatever..
I have an even better idea: Get rid of any *-free-format option altogether.

https://github.com/rg3/youtube-dl/issues/2196

@AliceTheGorgon @cloph @jaimeMF Please participate in https://github.com/rg3/youtube-dl/issues/2196

I see no point in having a force/require-free-formats option.  As mentioned in this issue multiple times such a strict filter can be added with a quite short extension filter (e.g. bestvideo[ext=webm])
What however  is not easy without creating a long filter list is a prefer-free-formats option that behaves as the name suggests.  So I see no point in reopening #2196.
I want to get rid of a (youtube) filter string like
248+251/248+bestaudio/137+251/137+bestaudio/247+251/247+bestaudio/136+251/136+bestaudio/bestvideo+bestaudio/best
and replace that with a simple [height <= 1080p] and --prefer-free-formats

@cloph Thank you.

After reading all these old issues I assume this is right place to discuss the "bitrate vs resolution" debate.
My take on this, at least for YouTube, is that resolution is a better indicator, because:
1)) the bit rate reported by YouTube is very unreliable and often quite off.
One example is what I reported in issue 12198. Video https://www.youtube.com/watch?v=iMhwshjSauo reports:
135          mp4        720x480    DASH video  198k , avc1.4d401e, 30fps, video
only
133          mp4        360x240    DASH video  242k , avc1.4d400c, 30fps, video

However, the actual bitrate is 112 kbps (480p) and 32.2 kbps (240p), respectively. As you can see, not only the numbers are off, the order bwtween formats are dramatically changed.
2)) since from my experience YouTube never upscale videos (correct me if I'm wrong), higher resolution ones should always have better quality (if the encoding didn't go wrong), since it literally has more raw information.

Could someone please provide me with a hint how to always play VP9 and Opus/Vorbis in mpv when they are available?
I don't get prefer-free-formats to work in mpv's config. There are some videos where H.264 is chosen over VP9, which imho is simply wrong (VP9 also looks better than H.264 at lower bitrate).
Example:
https://www.youtube.com/watch?v=o_24LPjOIHI
I tried
ytdl-format=bestvideo[vcodec=vp9]+bestaudio[acodec=opus]/bestvideo+bestaudio/best
which works for this video, but not for the following:
https://www.youtube.com/watch?v=TexBjznr4yM

This should work:
ytdl-format="((bestvideo[vcodec^=vp9]/bestvideo)+(bestaudio[acodec=opus]/bestaudio))/best"

Thanks a lot!
Is there a chance to also make it use Vorbis instead of AAC when it is available?

In the case I posted above it will only select opus when a video is found I should mention, for pure audio streaming sites it falls back to "best"

Thanks again, got it working. :)

@utack How is it going to look for Vorbis/Opus vcodec/acodec respectively?

To those who suggest that it's simple to replicate the preference / requirement for free formats with -f, it's not entirely true. As you can see from the discussion above, there are multiple free formats, and the format string can get messy.
To make everyone happy, perhaps we could keep --prefer-free-formats as-is (prefer highest resolution and then free format if available), and add --prefer-free-formats=always to prefer free formats even at the expense of quality. And perhaps a --require-free-formats could be added, which will obviously always either download the highest quality free format available, or if there is none, error.
And while it's not entirely on-topic here, I have also noticed that youtube-dl incorrectly prefers H.264 over VP9 due to the slightly higher bitrate. I don't know what's the best solution to that problem though, as you can't necessarily just say VP9 is always better than AAC, because there's a possibility a site will make a worse-quality VP9 encode than H.264 for the same resolution.

And while it's not entirely on-topic here, I have also noticed that youtube-dl incorrectly prefers H.264 over VP9 due to the slightly higher bitrate. I don't know what's the best solution to that problem though, as you can't necessarily just say VP9 is always better than AAC, because there's a possibility a site will make a worse-quality VP9 encode than H.264 for the same resolution.

The approach that always pops in my head is introducing “bitrate coefficients”, perhaps based on some commonly available codec benchmark. (Basically, the idea is that if VP9 encodes ~20% smaller files than H.264 at the same visual quality, as assessed by an objective quality metric such as MS-SSIM, then you can multiply 1.2 into the VP9 bitrate)

@haasn Has anyone found a VP9 with same resolution and FPS that looked worse than the H264 counterpart yet?
Does not seem to happen generally with the settings they use

@utack I remember seeing some pathological VP9 encodes of video game streams a year or two ago, specifically Dwarf Fortress. May have changed in the meantime, and I may be misremembering, but afaik you'll still find weird clips like that where x264 is still hard to beat, in some of the areas it's specifically tuned for.

Has anyone found a VP9 with same resolution and FPS that looked worse than the H264 counterpart yet?
Does not seem to happen generally with the settings they use.

Well, at least not on YouTube.
But haasn is right, some bitrate reference value for comparison would be a good idea.
Another thing I noticed, YouTube specific: Some newer videos, on official channels, music videos etc. had VP9 and H.264 with more or less the same bitrate, thus eliminating the potential file size benefit of VP9 for the sake of better visual quality. I don't know any YouTube Insider, but maybe that is their current strategy, adapting for higher availability of broadband Internet access across the user's population.

But haasn is right, some bitrate reference value for comparison would be a good idea.

For lack of anything better, we could use netflix's results (which are normalized to the VMAF metric).

This seems to suggest that e.g. at 1080p, a coefficient of “1.4 * bitrate” for VP9 and “1.5 * bitrate” for HEVC would be good candidates.

Even if we decided the "factors" for each format, how we're going to deal the fact that YouTube sometimes straightly report wrong bit-rate?

Probably by ignoring the “bitrate” and using the filesize, or something. Actually, looking at e.g. youtube-dl -F https://www.youtube.com/watch\?v\=KQs0Gj7bXYY:
...
247          webm       960x720    720p 2092k , vp9, 30fps, video only, 319.30MiB
136          mp4        960x720    720p 2249k , avc1.4d401f, 30fps, video only, 437.94MiB
298          mp4        960x720    DASH video 3348k , avc1.4d4020, 60fps, video only, 683.20MiB
302          webm       960x720    720p60 3891k , vp9, 60fps, video only, 591.97MiB
...

Comparing formats 298 and 302 specifically, if we assume 3348k is the true bitrate for the 683.20MiB H.264 stream, then
the VP9 stream's reported bitrate of 3891k is roughly “1.34x” higher than what it should be going by the filesizes. So
maybe youtube already internally gives VP9 a format-specific “bitrate” boost?

Well it doesn't always return file-size.

@nyuszika7h

And perhaps a --require-free-formats could be added

Add --require-free-formats was declined but with your vote on the issue they might change their mind.

Well it doesn't always return file-size.

Example?

youtube-dl -F https://www.youtube.com/watch?v=eYztYCbV_8U
Which is also an example I mentioned before that reports wrong bitrate (see: #11451), but it seems YouTube already corrected the bitrate issue for this particular one (format 136 now correctly shows lower bitrate than 137).
youtube-dl -F https://www.youtube.com/watch?v=iMhwshjSauo
This still shows no file-size information and wrong bitrate info
It reports
135          mp4        720x480    DASH video  198k , avc1.4d401e, 30fps, video
only
133          mp4        360x240    DASH video  242k , avc1.4d400c, 30fps, video
only

actual bit rates: 135: 112kbps 133: 32kbps

To make everyone happy, perhaps we could keep --prefer-free-formats as-is (prefer highest resolution/quality and then free format if available)

@nyuszika7h : the problem is that this is not how it behaves as-is. If that was the case, everyone would be happy indeed.
--prefer-free-formats as it is now acts like a --force-free-formats and this causes all the problem. (aka the need for overly long selector/filter lines instead of a simple  [height <= 1080p] and --prefer-free-formats) see my previous comments for more details.

I agree with the comments made by @fireattack above here and here that resolution should be used instead of bitrate for the -f bestvideo determination.
Rather than continue to hijack the --prefer-free- debate, I created #14143 (albeit before discovering the comments I cited here in this unrelated issue).
One unqualified demerit against using bitrate--or for that matter, anything-rate--during the bestvideo determination is that the time denominator is irrelevant when comparing alike-length alternatives. Since the length is exactly the same for all the entries in a given youtube-dl -F format listing, you can multiply by every rate value in the table by the same duration (video length) to "simplify," or mathematically cancel the division which created that "rate."
Given a "most pristine" master copy which must exist for every video, entropy can be dissipated in several dimensions, but duration time is not one of them. Either the compression algorithms are more/less efficient, or the frame dimensions or frame rate are different. And it seems to me that given the special constraint discussed above, the latter two will swamp the former in all cases, making some sensible combination of dimension and fps a much more stable indicator of "best."
I think that, possibly, people have gotten in the mindset of elevating bitrate because it is quite likely a better indicator of quality when comparing dissimilar source material, but this habit may be obscuring better judgment when it comes to the specialized task of comparing all-alike material.
Irrefutably, really, bitrate is just a needlessly complex proxy for the total entropy each available format ultimately makes available. Add in the fact that the YouTube reported bitrate values are often wildly wrong (in fact far more so than this thread has reported; see #14143) and it's clear that by the nature of this type of selection task, bitrate is not the way to go.
So as I've noted, the -f bestvideo determination task, as with tasks in general, can and should capitalize on the special properties that are particular to it. In this case, this means recognizing the exclusion of time-denominated metrics as mathematically vacuous, and therefore unnecessarily complicating:, i.e., at best a harmless obfuscation, and at worst, exposing a tangibly increased error surface.

I use this setting in mpv to always get VP9 + Opus/Vorbis, as suggested by utack (thanks again!):
ytdl-format="((bestvideo[vcodec^=vp9]/bestvideo)+(bestaudio[acodec=opus]/bestaudio[acodec=vorbis]/bestaudio[acodec=aac]/bestaudio))/best"
Is it possible to additionally also always prefer HDR video when it is available?

@aufkrawall Yeah, it gets identifies as vp9.2
I use this:
ytdl-format=(bestvideo[vcodec=vp9.2]/bestvideo[vcodec=vp9][fps>30]/bestvideo[vcodec=vp9][height>=1080]/bestvideo[fps>30]/bestvideo[height>720])+(bestaudio[acodec=opus]/bestaudio)/best

Great, just added vp9.2 and it works like a charm.
As always: Thanks a lot!

When people visit youtube from chrome, and click on a video, and hand pick the highest resolution, (for instance 1080 60fps), is there a good chance youtube would arbitrarily decide to serve webm?
If yes, maybe we could have an flag prefer-website-decision, which works like this: "choose the highest resolution, then accept the file format chosen by the website when serving to a desktop browser with good hardware, fast internet, and all codecs". This way, we will get webm if the server believes it's a good choice, no matter the bitrate.
Let's face it, newer codecs are expected to have lower bitrates. That's the goal when they were developed.


Clarify setup for E2E in documentation

Summary
Fixes #7227

Occurred changes and/or fixed issues
Updated documentation.
Technical notes summary

Areas or cases that should be tested

Areas which could experience regressions

Screenshot/Video

@nwmac if other areas of the documentation are not clear, could you please mention them here, so I could eventually have some feedbacks.

This is a dev docs change and does not need QA validation


⚠ ClamAV has degraded performance

In 926cd31, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 475 ms

Resolved: ClamAV performance has improved in f0ffa13.


[Bug] Integration Test

https://github.com/opendistro-for-elasticsearch/simple-ingest-transformation-utility-pipeline/blob/c9d2ff870de50db94d0c77f7e4f6d809cef29868/situp-plugins/elasticsearch/src/main/java/com/amazon/situp/plugins/sink/elasticsearch/ElasticsearchSink.java#L170
This method throws error when index already exists.

@kowshikn Thanks for the catch! Through investigation, it is a bug with Service map where we should check existence of index instead of index alias.
https://github.com/opendistro-for-elasticsearch/simple-ingest-transformation-utility-pipeline/blob/c9d2ff870de50db94d0c77f7e4f6d809cef29868/situp-plugins/elasticsearch/src/main/java/com/amazon/situp/plugins/sink/elasticsearch/ElasticsearchSink.java#L174

Fixed via #74


chore: add docker compose override file for local development

Closes #896
Description
Adds a docker-compose.override file that can be used in conjunction with the existing docker-compose.yaml file for applying local development specific build instructions and configuration for services.
References
https://docs.docker.com/compose/multiple-compose-files/merge/
Review Checklist

[x] I have clicked on "allow edits by maintainers".
[ ] I have added documentation for new/changed functionality in this PR or in a PR to openfga.dev [Provide a link to any relevant PRs in the references section above]
[ ] The correct base branch is being used, if not main
[ ] I have added tests to validate that the change in functionality is working as expected

Hi @miparnisari, could you please check if these changes could help in overcoming the issue mentioned in the corresponding issue? Over time more local development-specific changes can be added to the compose override file.

@sanketrai1 May I ask if you can add a third option to the "Build from Source" documentation in the README (we can call it: build using docker-compose) which documents how to build it for people other than us who may want to follow?
e.g. docker compose up -f ... -f ...

From my understanding, and it appears to work this way now as well, Docker looks for docker-compose.yml and docker-compose.override.yml by default. So nothing changes with the existing documentation around docker compose up.

@rhamzeh, @jon-whit is correct. If docker-compose.override.yml is present, then docker-compose.yml and docker-compose.override.yml are merged automatically when running docker compose up. If a compose override file with any other name than docker-compose.override.yml or compose.override.yml needs to be merged with the default compose file, only then multiple -f arguments syntax is needed.

@sanketrai1 could you please rebase on main?

I think the biggest difference when using an override file is that there's no way to skip the overriding. Why not just modify the current docker-compose.yaml file to not use the latest image? Is the intent of the docker-compose setup to get started quick with the latest stable version or with the current version in the filesystem?

@rhamzeh our current instructions in the README regarding docker-compose.yaml don't really work... see https://github.com/openfga/openfga/issues/1047. Based on the fact that we've had that instruction for a long time now, and we've only noticed now, I think it's fair to assume that nobody is actually using this for setup? 🤷♀

@sanketrai1 thanks for submitting this PR! But it doesn't seem to work for me? I removed my OpenFGA containers to start with a clean slate, and I ran docker-compose up, and it is using openfga:openfga/latest...

docker-compose up --build to trigger a fresh build every time (otherwise, first docker-compose up triggers a build but subsequent ones do not

This is how docker compose works unfortunately, there is nothing we can do about that. By default it uses the cached build unless you trigger a rebuild.


HDDS-11143. Switch to Rocky Linux 8 as CentOS 7 is EOL

What changes were proposed in this pull request?
CentOS 7 has reached EOL on June 30th, Need to adopt alternative Open source Linux Distros.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11143
How was this patch tested?
Manually build the repo, works.

Hi @adoroszlai , Thanks for the review, I added a new commit, please check when you have time.
ozone-ha tests were passing, tried locally with the steps you have mentioned above.

Get content summary of directory                                      | PASS |

Get quota usage of directory                                          | PASS |
Get home directory                                                    | PASS |
Get trash root                                                        | PASS |
Httpfs.Operations Tests :: HttpFS gateway test with curl commands     | PASS |
18 tests, 18 passed, 0 failed
Httpfs                                                                | PASS |
18 tests, 18 passed, 0 failed
Output:  /tmp/smoketest/ozone-ha/result/robot-1.xml
must specify at least one container source
Om-Roles :: Smoke test for listing om roles.
List om roles with OM service ID passed                               | PASS |
List om roles without OM service ID passed                            | PASS |
List om roles as JSON with OM service ID passed                       | PASS |
List om roles as JSON without OM service ID passed                    | PASS |
Om-Roles :: Smoke test for listing om roles.                          | PASS |
4 tests, 4 passed, 0 failed
Output:  /tmp/smoketest/ozone-ha/result/robot-1.xml
must specify at least one container source

Thanks @LZD-PratyushBhatt for updating the patch.
Secure tests are still failing.  I was wrong, the problem is not with kdc.  kdc is able to start; scm and other containers fail with:
ERROR: JAVA_HOME /usr/lib/jvm/jre does not exist.

Some tests (mostly the ozonesecure* ones) in Ozone 1.4.0 and earlier set JAVA_HOME=/usr/lib/jvm/jre, overriding actual Java location in the image.  Since we cannot change earlier Ozone release bits, we need to add another symlink in the image:
RUN mkdir -p /usr/lib/jvm && ln -s $JAVA_HOME /usr/lib/jvm/jre # compatibility with Ozone 1.4.0 and earlier compose env.

right below:
https://github.com/apache/ozone-docker-runner/blob/75aee6aef8109d1097b003c09d8263962d4b511d/Dockerfile#L166
or anywhere after that.

Tested with ozone-ha from 1.4.0 in CI:
https://github.com/adoroszlai/ozone-docker-runner/actions/runs/9897897687
and with ozonesecure-ha from master locally.


Update experts_to_follow.md

Hi @abhishekkankani. 1) It would be great if we all follow the standard table format for listing the expert name and links. You can look into below link.
https://github.com/GDGAhmedabad/Learning-Resources/blob/master/Python/experts_to_follow.md

Please try to include other resources as well, such as a Twitter handle, Github etc if possible.

@vyaspranjal33


Support batch deletion

There is putAll and getAll, would deleteAll make sense as well? This doc also demonstrates a mix of puts and deletes:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/batch-operation-document-api-java.html
Perhaps there could be syntax for mixing the two like batch(put = List(...), delete = List(...))

A deleteAll would be a good addition.
Mixing the two seems a bit odd to me and I haven't seen any compelling uses in the wild, so I'm not yet convinced it's worth the complexity.

I have at one use case for it so far that corresponds to a user on a mobile device modifying a list of items and sending a batch of modifications and removals. But it's probably a rare use case.

@philwills we have a working version of deleteAll based off the putAll that we would be willing to contribute back.  My main question is what do you think the result should be?  Our version tries to go through the result and figure out what was actually deleted and return that.  (Since the AWS result does return unprocessed items https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html)  Alternatively we could just return the unprocessed items, the BatchWriteItemResult, or any other variant you can think of.  I am open to whatever you think is more inline with the library design.  Let me know and I will work on a PR for it.

This is now supported as of 0.8.2


Updating to Laravel 5.*

is there any idea/project to migrate it to Laravel 5.* ?

I suppose it's not active anymore.

ok, thanks @phaberest .


VDC was deleted but still can not deploy one with the same name

Description
while I was deploying a VDC, a glitch happened and I got a not working VDC instance, then I deleted it, after that I tried to redeploy another VDC but with the same name but I can't deploy as i got this error

@MElborolossy please trace what happened.

This Issue happens because the new vdc chatflow opened before delete the old one
In this case, refresh is required to reload the info of the deployed vdcs

Reload button of the chatflow not from the browser

Verified
commit  23d1fcaf034788d2479afe0891c0da1059b1d6c6
when you delete the VDC, there are about 4 seconds of delay. if you press delete then tried as soon as the delete dialogue disappears to deploy a new VDC with the same name it will not accepted, restarting the chat flow what is required then.
didn't able to reproduce the issue, other than having those few seconds of delay.


fix: add skills flow refactored to desired R13 spec

Description
This PR makes the UI changes to match the desired add skills UI for R13 which is described in the figma that is linked from the below issue.
Task Item
fixes #7115
Screenshots
Toolbar drop down change

New Choice Modal

If you select 'add existing'

If you select 'add blank'

@emivers8 do you want the same sizing mentioned here #7230 for the "add a bot" dialog (second screenshot above)

@pavolum that should be "empty" not "blank" bot.

This should not be either.  I"ve raised this already, that we will provide any template that is not a multibot template.  so it's not just Empty.

I was just responding to the flow above that skips the template selection and jumps straight to creating an empty bot. If it instead goes to the template selection screen with a filtered set of templates it should probably say just "bot", but however y'all would like this to work.

I know.  I'm saying why is this the flow?  it completely removes my ability to add anything other than an empty bot to an existing project?  why would we scope to this?  that's my question

sorry--I thought the discussion from last week landed on this: the only new bot you could create while adding the bot was the empty aka blank bot for now.

okay, new example of the flow with the template picker shown (but we will filter out the multibots):

not sure if we also need to take out the language picker (C#/Node)?

As long as the node templates also hide any multi-bots, I think it's fine to stay? Right now I don't think there are any node multi-bot templates.

we don't yet have multi bot templates for node, but we will.  so we need to apply the same filtering in both languages, and given we use the same package to support both languages, we should just get this for free.  @benbrown

So a C# root bot can have a Node skill (or vice versa)?

yes.  each bot has it's own language specific runtime.


[SPARK-27278][SQL] Optimize GetMapValue when the map is a foldable and the key is not

What changes were proposed in this pull request?
When GetMapValue contains a foldable Map and a non-foldable key, SimplifyExtractValueOps fails to optimize it transforming it into case when statements.
The PR adds a case for covering this situation too.
How was this patch tested?
added UT

Test build #103979 has started for PR 24223 at commit 99965fb.

Test build #103979 has finished for PR 24223 at commit 99965fb.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103979/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9333/
Test PASSed.

Test build #103981 has started for PR 24223 at commit 4f983f0.

Test build #103981 has finished for PR 24223 at commit 4f983f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103981/
Test PASSed.

Retest this please.

Test build #104153 has started for PR 24223 at commit 4f983f0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9496/
Test PASSed.

Test build #104153 has finished for PR 24223 at commit 4f983f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104153/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9503/
Test PASSed.

Test build #104160 has started for PR 24223 at commit ef6af61.

Test build #104160 has finished for PR 24223 at commit ef6af61.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104160/
Test PASSed.

+1, LGTM. Merged to master.
Thank you, @mgaido91 , @maropu , @srowen .

@mgaido91 @dongjoon-hyun Could you benchmark it? I am afraid it does not help the performance but it might increase the code size?

I have run:
val dfPlan = spark.range(10).selectExpr(
     "map(1, 1, 2, id, 3, id * 2, 4, 1, 5, 2, 6, 3, 7, 4)[id] as x").queryExecution.executedPlan
val times = (1 to 20).map(_ => time(dfPlan.executeCollect()))
println("Avg time: " + (times.sum / times.size) / 1e6 + " ms")

Before the change I got Avg time: 20.001603 ms, after I got Avg time: 18.727406 ms, hence roughly a 10% improvement from this test.
Despite this, I quite agree with the concern related to the generated code size, which in general is not a big deal but it may cause regression if this makes WholeStageCodegen fallback to normal code generation. The other case is safer w.r.t this topic, as it removes the CreateMap which would be quite verbose as well.

IIUC this is kind of loop unrolling: GetMapValue looks up the map key via a while loop. CaseKeyWhen looks up the map key via a bunch of if-else.
But unlimited look unrolling does hurt performance. Maybe we should introduce a config for the map size? I'm not sure how often we can hit this optimization. If it's rare, maybe we should simply revert it as it's not worth the complexity.

yes, I think this is rare enough not to be worth a config.

Hi, @cloud-fan and @gatorsmile .
Although this is 7 line of code, I understand your concerns about  increase the code size and not worth the complexity. Shall I revert this?
case GetMapValue(Literal(map: MapData, MapType(kt, vt, _)), key) if !key.foldable =>
  val elems = new mutable.ListBuffer[Literal]
  map.foreach(kt, vt, (key, value) => {
    elems.append(Literal(key, kt))
    elems.append(Literal(value, vt))
  })
  CaseKeyWhen(key, elems.result())

"increase the code size" -> "increase the generated java code size, which may cause perf regression"
Yea I think it's better to revert it first. We can revisit it when we adding official "loop unrolling" support for array/map operations.

Yep. Thank you for review and advice, @cloud-fan !

This is reverted via https://github.com/apache/spark/commit/b51763612a71e214ed94ac5fb6843cf03d00f9f8 .


Better version tracking

Is your feature request related to a problem? Please describe:
I need version tracking (like git) for my project directly included in your UI. I accidentelly reloaded the website and were unable to go to a previous version (which was much better). I lost loads of Tokens and time because of that.

Describe the solution you'd like:
I want to see all my versions like in Anthropic Claude artefacts. Just allow me to choose, which I like best but save them all.

Describe alternatives you've considered:
Git

Additional context:

We landed an initial version of this here: https://x.com/stackblitz/status/1844839379270836266
We will be making versioning more powerful in the future as well.


docs(readme): remove httpd url from example

this example no longer exists

nvm, didn't see the annotation in the apache example :)


Vertical gallery with IGListKit

I have a very basic case of a vertical gallery that can have any height of an image with full-screen width set. Images a load async from the network.
Is this solvable with IGListKit?
I saw a few mentions of dynamic height issues already, so is my task doable, or I better just go with raw UICollectionView?

@aaverin Do you know the image size before downloading the actual image? This seems absolutely doable (similar to Instagram's full width images with dynamic heights).

You need an aspect ratio of the image
Then in the section controller method sizeForItemAtIndex: you get the width of the container with collectionContext.containerSize.width
Lastly multiply the aspect ratio with the screen width to get the appropriate height

Make sense?

@rnystrom No, image size is not known, and that's kind of the whole point and complexity =)
Basically, any image can be downloaded from the network. ImageView is full-width, and image should occupy whatever height necessary for given screen width.

Can your API include aspect ratio in the initial payload?

@rnystrom Take Flickr, for example. They have a public API that doesn't provide aspect ratio or image size in the response. And because images can be landscape and portrait – size can differ.

Ah ok. So what I'd probably try in that case is to start with a placeholder size (maybe square with the screen width). Download the image within the section controller. Once done, call batch updates on the collection context and call reload (see some of our examples).
In the size API, use the placeholder size if no image exists, and use the actual image size (scaled for screen width) when there's an image.
Make sense?

seems hard even with raw UICollectionView  without aspect ratio

Ya, finding a way to get aspect ratio before loading the image data will

Make for a much better UX
Be a vastly simpler implementation

Going to close out since IMO the question is answered.

This is something that works out of the box on Android.
I am porting my Android app to iOS, was hoping to have same functionality. But yes, that doesn't seem to be easy with UICollectionView.
I'll keep your suggestions in mind and will stay with a fixed-height workaround for now


Add XMAKB5R0P0906 BIOS updater for USB flash drive  using on Linux/Ma…

This MR is related to this issue https://github.com/Cr0wTom/Mi-Notebook-Pro-Mods/issues/9 .
In short: There is no BIOS firmware for Linux/macOS and I add it.
I did all as in the instruction of this issue and changed flash.nsh script so that a user can follow default instructions and automatically can update a BIOS. I check this functionality, and it works.
Also, I add additional information for updating the BIOS firmware.

Hello, I have no way to test the firmware as I sold the laptop. If someone can send proof that the firmware is working then I will be able to merge, but till then I don't think I will merge any binary files to this repo :)


Add support for assets library snapshot builds

Tested on dev2.
Able to deploy spa shell app that uses a snapshot build for assets library.
Compiled and started the app without error.
Ran a quick smoke test, OK.


Markdown in auto-generated socialmeta text needs to get stripped

If you make a post whose initial text contains Markdown, and you don't specify any description metadata, then the text that ends up pull-quoted on social media often contains that Markdown and looks yucky.
In the case of no description specified, we should supply the first paragraph with the Markdown stripped out. (So, translated to HTML, and then run through an HTML-stripper, I suppose.)

The most recent version of HTML::SocialMeta address this bug. (Thanks, me!) We should update Plerd's cpanfile to insist on it as a minimum version.


IIO node type: add support for multiple channels

IIO node type: add support for multiple channels
Currently, IIO node types support only one logic channel. Some devices, however, have more than one channel, whose readings are independent. They usually have channels in_readingX, like in_voltage0 or in_proximity2.
To address that, IIO node types would need to have a new option, channel, default -1. If -1, it means that are no specifier, so channel name would be in_proximity. If > 0, then this should be used as channel specifier.
Implementation notes
Some thoughts on implementation, of course different ideas are welcome.
Even though the channels are independent, they do belong to the same device. So, it's necessary that different nodes share devices. This can be done by creating a list of open devices on nodes.c.
After opening a device, it shall be sent to this list. I believe that this list should store device id, sol_iio_device handle and device/channel config. Last one is to prevent that some channel is opened with different device config - so it's there to compare.
So, when creating a node, it should first call sol_iio_address_device, and check if there's an sol_iio_device open with same id. If so, compare configs and if OK, add new channel. If not, fail on mismatch configuration.
This is also an opportunity to make iio/nodes.c code share more common code and avoid it's current state of duplicating code.

@yongli3 See this issue, please.

This will allow multiple access, will the implementation only allow multiple access to the same IIO device from a single application or it will allow multiple access from multiple applications? Where is the access controller here.

@edersondisouza I am thinking we need to add another option: count. Default = 1, it means we only output one channel. > 1 means output multi-channel data. Using the ADC sensor on Galileo as an example. Below is the sysfs entities:
ADC sensor on Galileo boards:
root@intel-quark:/sys/bus/iio/devices/iio:device0# ls -l
drwxr-xr-x    2 root     root             0 May  2 06:44 buffer
-r--r--r--    1 root     root          4096 May  2 06:44 dev
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage0_raw
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage0_scale
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage1_raw
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage1_scale
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage2_raw
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage2_scale
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage3_raw
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage3_scale
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage4_raw
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage4_scale
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage5_raw
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage5_scale
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage6_raw
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage6_scale
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage7_raw
-rw-r--r--    1 root     root          4096 May  2 06:44 in_voltage7_scale
-r--r--r--    1 root     root          4096 May  1 10:41 name
drwxr-xr-x    2 root     root             0 May  2 06:44 power
drwxr-xr-x    2 root     root             0 May  2 06:44 scan_elements
lrwxrwxrwx    1 root     root             0 May  2 06:44 subsystem -> ../../../../../../../../bus/iio
drwxr-xr-x    2 root     root             0 May  2 06:44 trigger
-rw-r--r--    1 root     root          4096 Apr 28 05:51 uevent
Without channel and count options, only in_voltage0_raw is captured(The current implementation).
If channel=0, count=1, it will capture in_voltage0_raw(The same as default).
If channel=0, count=2, it will capture in_voltage0_raw and in_voltage1_raw, and output two values
If channel=1, count=1, it will capture in_voltage1_raw only
If channel=3, count=3, it will capture in_voltage3_raw, in_voltage4_raw and in_voltage5_raw
For below color sensor, by default all 3 data are captured. If channel=1, count=1, it will only output in_intensity_green
in_intensity_red
in_intensity_green
in_intensity_blue
The same as below magnet sensor, by default all x/y/z data are captured, but if If channel=1, count=2, it will only output in_magn_y_raw and in_magn_z_raw
in_magn_x_raw
in_magn_y_raw
in_magn_z_raw
Any thoughts/comments?

On 10-05-2016 03:38, LB Lim wrote:

This will allow multiple access, will the implementation only allow
multiple access to the same IIO device from a single application or it
will allow multiple access from multiple applications? Where is the
access controller here.

I was thinking about multiple access from a single application. Having
different nodes for different data coming from the same device.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/solettaproject/soletta/issues/1998#issuecomment-218071404

On 10-05-2016 04:14, Yong Li wrote:

@edersondisouza https://github.com/edersondisouza I am thinking we
need to add another option: count. Default = 1, it means we only output
one channel. > 1 means output multi-channel data. Using the ADC sensor
on Galileo as an example. Below is the sysfs entities:
ADC sensor on Galileo boards:
root@intel-quark:/sys/bus/iio/devices/iio:device0# ls -l
drwxr-xr-x 2 root root 0 May 2 06:44 buffer
-r--r--r-- 1 root root 4096 May 2 06:44 dev
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage0_raw
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage0_scale
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage1_raw
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage1_scale
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage2_raw
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage2_scale
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage3_raw
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage3_scale
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage4_raw
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage4_scale
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage5_raw
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage5_scale
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage6_raw
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage6_scale
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage7_raw
-rw-r--r-- 1 root root 4096 May 2 06:44 in_voltage7_scale
-r--r--r-- 1 root root 4096 May 1 10:41 name
drwxr-xr-x 2 root root 0 May 2 06:44 power
drwxr-xr-x 2 root root 0 May 2 06:44 scan_elements
lrwxrwxrwx 1 root root 0 May 2 06:44 subsystem ->
../../../../../../../../bus/iio
drwxr-xr-x 2 root root 0 May 2 06:44 trigger
-rw-r--r-- 1 root root 4096 Apr 28 05:51 uevent
Without channel and count options, only in_voltage0_raw is captured(The
current implementation).
If channel=0, count=1, it will capture in_voltage0_raw(The same as
default).
If channel=0, count=2, it will capture in_voltage0_raw and
in_voltage1_raw, and output two values
If channel=1, count=1, it will capture in_voltage1_raw only
If channel=3, count=3, it will capture in_voltage3_raw, in_voltage4_raw
and in_voltage5_raw

I can see value in this idea, it would help to save some memory for
devices that expose 'indexed' channels like those voltageX or proximityY
ones. But not sure if this will end up bringing more confusion to users.
If someone needs only 'voltage0' and 'voltage7' it would be no better
than having two nodes, anyway.

For below color sensor, by default all 3 data are captured. If
channel=1, count=1, it will only output in_intensity_green
in_intensity_red
in_intensity_green
in_intensity_blue
The same as below magnet sensor, by default all x/y/z data are captured,
but if If channel=1, count=2, it will only output in_magn_y_raw and
in_magn_z_raw
in_magn_x_raw
in_magn_y_raw
in_magn_z_raw

I'm not sure I see value on these ones, though. Red, green and blue and
x, y and z are actually part of the same information, it doesn't make
sense separate them. Actually, node types output only one packet
containing all information: rgb packet for first case and
direction-vector packet for last.

Any thoughts/comments?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/solettaproject/soletta/issues/1998#issuecomment-218077191

@edersondisouza I got your point, some information cannot be separated. Let us add the channel option only. I will use the ADC sensor for a test

One more thing to notice: AFAIR, one can only enable channel scanning when buffer is disabled - talking about sysfs IIO control. If that's so, one could not add a channel after device was opened, making device sharing impossible.
To overcome this, part of channel setup would have to be delayed to happen when buffer is started, for instance, enable_channel_scan, read_channel_scan_info and read_channel_index_in_buffer.
sol_iio_device_start_buffer would them change to first disable buffer, setup channels and them enable buffer as is today.
Note that this changes don't affect API and thus how nodes setup: these changes should be internal to sol-iio.c.

May I know whether this support will be in v1?

can this be included for v2 milestones?

sure!

@edersondisouza I see the fix #2318 has been merged. Do you think it is okay to close this task now?

@edersondisouza @ceolin ping.

since the fix was already merged, i'm closing this issue.


Release Google.Cloud.RapidMigrationAssessment.V1 version 1.1.0

Changes in this release:
New features

Change netstandard2.1 target to netstandard2.0 (commit 82bea85)

Created release for Google.Cloud.RapidMigrationAssessment.V1-1.1.0


How make to work OR Where request ?

I tried a lot of different variatns, I simply want: SELECT * FROM "User" WHERE "fullName" LIKE '%RD%' OR "initials" LIKE '%RD%' OR "email" LIKE '%RD%'
this doesnt work {    "$or":[       {"fullName":{"$cont":"RD"}},       {"initials":{"$cont":"RD"}},       {"email":{"$cont":"RD"}}    ] }  or put "$or": with [] or {} in any combinations

{ "$or":  	{ 		"fullName":{ 			"$cont":"${searching}" 			}, 			"email":{         "$cont":"${searching}"       } 	} } generating Query: SELECT "public"."User"."id", "public"."User"."fullName", "public"."User"."initials", "public"."User"."email", "public"."User"."password", "public"."User"."phone", "public"."User"."address",  "public"."User"."comment", "public"."User"."role", "public"."User"."isActive", "public"."User"."createdAt", "public"."User"."updatedAt", "public"."User"."clientId" FROM "public"."User" WHERE ("public"."User"."fullName"::text LIKE $1 **AND** "public"."User"."email"::text LIKE $2) OFFSET $3 Params: ["%RB%","%RB%",0]
Why AND not OR?

@premieroctet man you did a great job, but is this project alive?

Did you find a workaround for this? I'm having the same exact issue

Did you find a workaround for this? I'm having the same exact issue

write prisma code for cases like this and step by step remove this lib

Hello @Kiborgik sorry for the delay.
I think you're trying to use the $or operator in an invalid way.
Your $or request should be like {"$or": { "fullname": { "$cont": "RD }, "initials": { "$cont": "RD" }, "email": { "$cont": "RD" }  } }

There is still something strange going on in the parsing. I have this example (in JSON, but it's based on the where-query):
{
    "$and": {
        "archived": {
            "$eq": false
        },
        "$or": {
            "title": {
                "$cont": "aud"
            }
        },
        "uploaded_by": {
            "$eq": 3
        },
        "upload_timestamp": {
            "$gte":<PHONE_NUMBER>
        }
    }
}

The parsePrismaWhere method parses this as:
{
  AND: {
    archived: { equals: false },
    uploaded_by: { equals: 3 },
    upload_timestamp: { gte:<PHONE_NUMBER> }
  }
}

The OR is removed completely. But adding an OR to an AND seems to be a normal use-case?


feat: upgrade to empirica latest

Tests are passing locally with the docker-compose-dev.yml container
There is an issue related to building the container in github actions: https://github.com/Watts-Lab/deliberation-empirica/actions/runs/3085416333/jobs/4988679363#step:3:224, this is also using the dev container, so I'm surprised that it is giving issues.
@npaton - any ideas?

I couldn't figure out the Cypress thing.
I found some issue alluding to this being a problem others are having, though there is no clear fix from what I can see:

https://github.com/cypress-io/github-action/issues/235
https://github.com/cypress-io/github-action/issues/286
https://github.com/shapeshift/web/issues/1325

Maybe an update to Cypress could help. I don't, I have very little experience with Cypress. I don't quite understand how this whole setup works. There are none of the container logs (not in these failing runs, not in the previous working runs).

Enabled GH action debug logging, lets try again and see if we get more useful info

Tests are occasionally failing because of flake in admin console accessing. Addressing this in a future PR.


Sender series sender gives different results on 7.5-RC1 and 7.6-XX

Describe the bug
Please consider this adapter, and have a look at CreateSharedValues:
src_main_configuration_Mailbox_ConfigurationCreateExcel.zip
What happens in 7.6-XX is:
Input is send to CreateSharedValues sender first, output of CreateSharedValues is the input for StoreSharedValues and will also be the input for the next pipe due to preserveInput attribute on sender series sender.
What happens in 7.5-RC1 is:
Input is send to CreateSharedValues sender first, output is ignored probably due preserveInput and original input is also the input for StoreSharedValues and is the input for the next pipe.
Reporter
Laurens, me
Category

Frank Framework

To Reproduce
Steps to reproduce the behavior:
Too much to explain... please contact me for more details.
Expected behavior
Not really sure, both situations are kinda skeptic about the preserveInput attribute imo.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment:

Application server (f.e. Tomcat/WebSphere/JBOSS/Jetty) Websphere / Tomcat
Browser [e.g. chrome, safari] Chrome
IAF Version [e.g. 7.5 RC3, 7.6] 7.5-RC1 & IAF 7.6-20200603.093311:

Additional context
Add any other context about the problem here. (f.e. ladybug report / test adapter with larva test)

There was a bug in the 7.6 version, the input from sessionkey or fixedValue was not really used.
This is fixed in PR #855.
The IbisDoc is also made more clear


[enhancement]: Add support for CAM S220 SoloCam

Describe the solution you'd like
Hello
This device type is not compatible at the moment. It seems to be device code 63 from log error message during startup.
Please let me know if any info from devices are needed.
Thanks in advance

I also bought the S220 solo a few days ago to be able to use it with Homebridge/HomeKit.
Would be really great if a cam support in the Eufy plugin could be made available in the short term to revise the mistake that Eufy has made (no official HomeKit support) again.
I plan then, if the Homebridge support for this cam is available, to buy more. With the built-in solar panel, the S220 is really great & relatively small.
Thanks in advance & keep up the good work!
BR,
Alex

@julien94800 Have you tried to add enumerated device type 63 in src/configui/app/util/eufy-security-client.utils.ts and updating static isCamera function to include the new device type? I think that should be sufficient as the plugin can already see the Solo220 present in latest build, it just doesn't properly recognize it as a camera.

While the author of this plug in please respond an update it for the current line up of cameras?

Hello @ibmaster
No I did not tried yet. My first idea was to update the file and put code 63 for the 2Cpro camera in the code in order having to update only one line cause I assume both devices should have close functionalities. But then I was locked due to permission on files on my docker installation.

I checked again this morning and I did not found this file.... I'm not skilled enough. If I'm correct these information are only saved in a dependency script?

I have the Solo220 to and I wish I could see the camera on my home app

I also have the Solo 220 and looking forward to supporting this device.

done #351

Hello @lenoxys , this device is not yet compatible with last library version

@julien94800 perhaps is the issue that there is no official homebridge plugin npm release label that incorporates the latest underlying plugin version v2.1.11?

eufy-security-client just added s220. hope soon will be working.
https://github.com/bropat/eufy-security-client/pull/353/files/0667e02f2c663de3d42b604669fe83370825bd70..d004fb4f33d3cde1cdd1426ac52a30ce9d5ef8e8

https://github.com/bropat/eufy-security-client
just update
homebridge-eufy-security need to be updated to work with s220?
https://github.com/homebridge-eufy-security/plugin/issues/370#290-2023-11-04
(bropat/martijnpoppen) NEW: add support for S220 (T8134; https://github.com/homebridge-eufy-security/plugin/issues/353)

it's been back ported in 2.3.6 hksv

I just tried / test it with version 2.3.6 hksv on a clean install but it is not working. It still shows S220 as an alarm.

I also just tried / test it with version 2.3.6 hksv but it is not working. No selection for Operating Mode "Camera" shown. (See Screenshot)
In my homebridge Log-File i found this:
[11/06/2023, 13:10:22]  [EufySecurity]  ERROR   Garten-Pool handleSecuritySystemCurrentStateGets: Wrong return value
[11/06/2023, 13:10:22]  [EufySecurity]  ERROR   Garten-Pool handleSecuritySystemTargetStateGet: Wrong return value

Is there any update to get this working?

I also would like to add my S220 to homebridge. Please update the plugin. Many, many thanks.

Still no update. Is this actively researched. Would be great


Implement humanlike autoclick mode

Features:

[x] No rapid double clicking
[x] Random humanlike delays between clicks/key_down/key_up

Implemented in 3a7c745a7d3072b02bdc43403478ece12e6f873a


Cannot use ESMF_F90COMPILER macro

Describe the bug
A clear and concise description of what the bug is.
On the machine narwhal (navy), makefile cannot use $(ESMF_F90COMPILER) to compile Fortran code after loading spackstack modules. e.g.
%.o $(OBJDIR)/%.o : %.F90
$(ESMF_F90COMPILER) -c $(FFLAGS_F90) $(ESMF_F90COMPILEOPTS) $(ESMF_F90COMPILEPATHS) $(ESMF_F90COMPILEFREECPP) $(ESMF_F90COMPILECPPFLAGS) $(CPLFLAG) $(MODFLAG) $(INCFLAG) $< -o $@
Results in an error:
/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.5.1/spack/lib/spack/env/intel/ifort -c  -traceback -O3 -g -fPIC -m64 -align all -w -ftz -convert big_endian -assume byterecl -mcmodel=small -I/p/home/feiliu/ESPC/stackspack/ESPC-SRC/share   -I. -I/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.5.1/envs/unified-env-intel-2021.4.0/install/intel/2021.4.0/netcdf-fortran-4.6.0-v6s2aeg/include -I/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.5.1/envs/unified-env-intel-2021.4.0/install/intel/2021.4.0/netcdf-c-4.9.2-bidxx3c/include -I/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.5.1/envs/unified-env-intel-2021.4.0/install/intel/2021.4.0/hdf5-1.14.0-wgpha3r/include code.F90 -o code.o
[spack cc] ERROR: Spack compiler must be run from Spack! Input 'SPACK_ENV_PATH' is missing.
Note that what's odd is that both $(ESMF_F90COMPILER) and $(FC) point to ftn on Narwhal.
To Reproduce
Steps to reproduce the behavior:
module use --append /p/app/projects/NEPTUNE/modulefiles
module load neptune/1.4
In any makefile use $(ESMF_F90COMPILER) instead of $(FC) results in the above error.
Expected behavior
A clear and concise description of what you expected to happen.
I expected no difference using either $(FC) or $(ESMF_F90COMPILER) since both point to 'ftn'
System:
What system(s) are you running the code on?
Narwhal
Additional context
Add any other context about the problem here.
None

Used wrong github ID accidentally.

This would have been a duplicate of https://github.com/JCSDA/spack-stack/issues/892


Add clippy

This pull requests adds clippy to travis. I fixed the current clippy suggestions within Rustler. I left out some suggestions to show what they look like.
To run clippy against Rustler:
$ rustup component add clippy
$ cargo clippy --all-targets --all-features -- -D warnings

Fix #222.

The remaining errors are in erlang_nif_sys. I will fix them if there is interest to merge this pull request.

Definitely interested in merging this.
Would be nice if you could fix the remaining errors and add the change to the changelog

Would be nice if you could fix the remaining errors and add the change to the changelog

Will do.

Will do.

Done.


Published package does not match tagged release

The GitHub release for 1.2.12 discusses the move from gdown to wget for some endpoints and indeed the source does have some related changes.
However the setup.py in the linked commit doesn't seem to mention wget. Even more so -- the latest setup.py in this repo claims to be at version 1.2.10 and not 1.2.12!
In contrast, the published setup.py in the source distribution does seem to be for 1.2.12 and does mention wget.
Are there some changes which haven't made it into this repo?
As a consumer of this package, while it is quite valuable as a library, it is quite disconcerting when the published package doesn't match the tagged source for that version.

Sorry for the inconvenience. I'll resolve the issue ASAP. Thanks for the report.

Thanks!
Would you be interested in a PR which added packaging & publishing via GitHub Actions? Automating those steps would make them more easily reproducible, less effort and make it much harder for the repo and the published package to get out of step.

Sounds great but I cannot because I graduated last year and I barely have a time to maintain this project, so I could disappoint your great effort. If you still want to help with this, you are most welcome and sorry for the late reply.


Vue 2 cannot assert type in template

Minimal test case
https://github.com/jaredmcateer/vue2-volar-type-error
Issue explanation
When passing an object with a variable key to :checked on an input element I am getting a type error where the types might not align.
I've tried various ways of calming the error but have not found an ideal solution.

<template>

Error in ide

      Checkbox <input
        type="checkbox"
        :checked="features[feature]"
        ^------- Type 'string | boolean' is not assignable to type 'any[] | Booleanish | Set<any> | undefined'
      />

</template>

I tried changing the line to :checked="(feature[feature] as boolean)" which calms the error in the IDE but is a runtime error (unexpected token)

Other things I've tried is adding:
v-if="typeof features[feature] === 'boolean'"
and
adding a method that has a type predicate
methods:
  isBoolean(feature: any): feature is boolean {
    return typeof features[feature] === 'boolean';
  }
}

with
v-if="isBoolean(features[feature])
But these do not work. However if I try them with accessing the property directly such as:
Checkbox <input
  type="checkbox"
  v-if="isBoolean(features.foo)"
  :checked="features.foo"
  />

Then it works (but doesn't for my real world usage)

It seems that your error prompt is not caused by volar, your ide type derivation works correctly.
but I found that type assertion syntax(as xxxx) cannot be used in templat in vue2. I think this is related to the compiler used in vue2.
I think the reason why your last code can work is that TS has a feature that context judgment can narrow the range of types.
Maybe you can try,it works well with me
<input  type="checkbox"  :checked="isBoolean(features[feature])"  />

Technically that would work but :checked="typeof features[feature] === 'boolean'" isn't the same as :checked="features[feature]". But from what you're saying my only solution would be to use a method with a boolean return type.

Hi @jaredmcateer, please try


Error: Attempt to postMessage on disconnected port

General information
Hello,
I'm having this issue on my Arch Linux machine ~but I know friend of mine is experiencing same problem on mac.~
Error: Attempt to postMessage on disconnected port

This applies to both chromium and firefox so I think it is probably related to native server.
This happend after some recent update (few weeks ago)

Operating system + version: Arch Linux
Browser + version: FireFox / Chromium
Information about the host app:

yaourt

2.0.18

Information about the browser extension:

2.0.18

Hey, is it causing any actual issue though?
It's a warning in the console, meaning that while extension was doing something in the background, the popup was closed. For me it usually happens when I have to unlock my PGP key, a pin entry dialog appears and it closes the popup.
I just don't want to silence the errors, one day they might be useful.

It does. This is the message rendered under input in the browser extension. Nothing works.
I'm unable to print screen it as the whole popup disappears when I hit print screen (I'm using light screen).
I think extension simply can't communicate with a native app at all due to this.

I'll be happy to help with the investigation and potential fixes but the problem is that I'm not a go programmer (as I don't really like the language at all) and I don't have enough experience with extensions API.

Just got the info that friend managed to fix it on macos by manually removing and reinstalling the native client - updating bug report.

Can you also try completely reinstall the host app? Have you ever installed it locally, for example from the sources? If you have any files in these locations? If yes - delete them.
https://github.com/dannyvankooten/browserpass/blob/22e8af03308ca489beeb51f6bf66714b14240740/install.sh#L34-L35

yes, I did have it installed from source in past. I'll try it first thing when I get back home and let you know. Thanks!

I can confirm that rm -rf of those configs have fixed the issue for all browsers I have installed :raised_hands:
Once again thank you very much.

Good! I think I need to mention this somewhere 🤔 Because in general these local configs take precedence, so you think you are updating browserpass when an update is pushed via AUR, but in fact you are still using the old browserpass from the time when you compiled it from sources and installed in the home folder.

Frankly, I was even looking for something like this before posting an issue but I've failed to find those locations in a makefile. Now I see it's in the install.sh :/

i had this problem while running Firefox from the "snap" repositories on Debian stretch: even with the webext-browserpass package installed to provide backend support, the error message would pop up, presumably because of the isolation provided by the "snap" system. Since Debian stretch switched to Firefox ESR 60 now, I do not have this problem anymore of course, as I stopped using the snap, but I figured I would document another possible source of failure for this.
Thanks for the great extension!

I'm getting this same problem now with Firefox v66.0.3 (64-bit), using browserpass 2.0.22 (macOS Mojave).
It worked just fine yesterday, and now today all of a sudden I'm getting a red error message in Firefox when I either click on the browserpass extension button in Firefox, or I hit Cmd+Shift+L.
I've tried completely re-installing browserpass, with no success.

@thanethomson Can you please confirm your extension version? V3 was released on Saturday, so you shouldn't still be seeing 2.0.22 unless you've manually installed it - and if you'd done that, you shouldn't be seeing this error.
Have you installed the v3 native host app yet?

I installed browserpass following these instructions, and after updating Homebrew and running brew info browserpass I get:
dustinwilson/tap/browserpass: stable 2.0.22
Native component for Chrome & Firefox password management add-on
https://github.com/browserpass/browserpass
/usr/local/Cellar/browserpass/2.0.22 (7 files, 3.7MB) *
  Built from source on 2019-04-14 at 16:24:29
From: https://github.com/dustinwilson/homebrew-tap/blob/master/browserpass.rb
==> Dependencies
Build: go ✔, dep ✔
Required: gnupg ✔, pinentry-mac ✔
==> Caveats
To complete installation of browserpass, do the following:

1. Install the browserpass-ce add-on in your browser.
    * Chrome: https://chrome.google.com/webstore/detail/browserpass-ce/naepdomgkenhinolocfifgehidddafch
    * Firefox: https://addons.mozilla.org/en-US/firefox/addon/browserpass-ce/
2. Run `browserpass-setup` to install browser-specific manifest files.

The addon will not work otherwise.

Running brew upgrade browserpass:
Error: dustinwilson/tap/browserpass 2.0.22 already installed

Is the new version available via Homebrew yet?

Yes, but since we couldn't reach Dustin, another user has contributed a formula for v3, check out the instructions here: https://github.com/browserpass/browserpass-native#install-via-package-manager

I am struggling to update from 2.x and I get this error as well. I reinstalled the extension 3.1.0 from addons.mozilla.org, wiped configs from my home, built browserpass-native with docker and even ran the dreaded sudo make install to have a global browserpass host app (3.0.6).
Still, nothing works. :(

Could you open a new issue in https://github.com/browserpass/browserpass-native describing in more details what you tried and what you are experiencing?

I think I was just a little hasty and messed up the building process. I cleaned everything and I thought the browser config was still in place. Running sudo make hosts-firefox again actually fixed it. Sorry for the complaint and thanks so much for your work!! :)


Adapt membership timeline for book circulation timeline on book bibliography page

Revising to make this a dev task to adapt existing membership timeline chart.


Run Connectors and Tasks in Isolated Pods

Hi all!
This is a sister proposal to KIP-987: Connect Static Assignments
The goal of this proposal is to alleviate the operational headaches that come from shared resource pooling within Kafka Connect.
This is my first proposal to Strimzi, and I don't have a lot of familiarity with it as a user or developer. Please don't hesitate to point out parts that don't make sense, as I may have missed something just by reading the public documentation.
Thanks!

Hey @scholzj Thanks for the quick first pass! You had some excellent questions that cut right to the core of the feature.

But mixing them together raises a lot of edge-cases and increases the implementation complexity singificantly.

This is mostly borne out of trying to model the intermediate state when rolling between fully-shared and fully-isolated cluster. I figured that a feature which required a stop-the-world migration would be significantly less user-friendly, enough to justify some extra complexity in the scheduling algorithm. I figured that if the Static Assignments feature could express those states, it would also make sense to expose that to Strimzi users.
If you believe that Strimzi should prevent users from using hybrid-scheduling as a matter of being opinionated, I think we could delay emitting static assignments until every KafkaConnector has ResourceRequirements attached, and/or some enable switch is flipped. Modeling the intermediate states then just becomes an implementation detail of the rolling migration rather than a user-facing feature.

Things would be much easier if one could use the maximum number of tasks as number of replicas. But that is of course not how Connect works.

We have a loose proxy for this with the tasks.max configuration, but it's technically not enforced by the framework. A connector could go over the advised limit, but I don't think i've seen any implementations do that except by mistake.
We could have all of the pod provisioning happen based on the KafkaConnector spec by looking at the tasks.max, so the control flow doesn't have to depend on the contents of the REST API.

I also wonder if you considered to have the KafkaConnector represent the Connect cluster. If you want to run the connectors like this, do they really need to share the same distributed Connect cluster? Are there any advantages to it?

A KafkaConnect cluster certainly has less relevance in a fully-isolated cluster. But the workers of a fully-isolated cluster will still share a config topic, rest api, group coordinator, etc. The cost of those resources (however minor) does get amortized. But possibly the more important benefit is inheriting Kafka client configurations and secrets. To eliminate the KafkaConnect abstraction from Strimzi would require adding at least some of the worker config properties to the KafkaConnector abstraction.

Are there any risks that with too many small pods (each pod == one Connect node) the scalability of the Connect control plane becomes an issue?

This is certainly a risk, and I will need to do some practical experiments to evaluate how a connect cluster behaves with 100-1000 lightweight nodes rather than 10-100 heavy nodes. If this did become a problem, it might mean that we need to make other investments into the Connect scalability before this feature will be safe to use for large clusters.

But I think it is a bit missing an comparison with other technologies. Why is Connect worth this investment? Isn't it better to instead shift the focus on some integration projects that are more cloud native by design such as Apache Camel? The current Connect design clearly has it limitations when running on Kubernetes. But for me personally, it is not clear if this is something worth saving and redesigning. So it would be great if theproposal can elaborate on this a bit as well.

I think that the ease of operation is an extremely good criteria for user to consider when evaluating whether to use Connect or Camel. Someone could certainly use Connect, and then realize that it's inferior to Camel, and then migrate to Camel and find that it solves their problems. I have not heard of an instance myself, but it's probably happened at least once.
But this proposal is not for users evaluating Connect, it is an incremental improvement for operators that have already made the choice or had the choice made for them. In the next draft I'll make sure to advocate more for the operators.

Hi Greg, this is definitely an interesting proposal. On your earlier comment:

We have a loose proxy for this with the tasks.max configuration, but it's technically not enforced by the framework. A connector could go over the advised limit, but I don't think i've seen any implementations do that except by mistake.
We could have all of the pod provisioning happen based on the KafkaConnector spec by looking at the tasks.max, so the control flow doesn't have to depend on the contents of the REST API.

I think the tricky thing here is although in general connectors don't go over the tasks.max configuration, there are connectors that start fewer tasks. For example the Confluent JDBC connector only starts a single task when running in query mode, and otherwise it creates the number of tasks equal to either tasks.max or the number of tables in the database it is streaming updates from, whichever is lower.
So if the operator did base the number of pods on the tasks.max configuration then some pods might be left without any workload running on them. I wasn't sure from your KIP what happens if a worker is configured with static.tasks set to a task id that actually doesn't exist.

Thanks for the review @katheris !

So if the operator did base the number of pods on the tasks.max configuration then some pods might be left without any workload running on them. I wasn't sure from your KIP what happens if a worker is configured with static.tasks set to a task id that actually doesn't exist.

Thanks for pointing this out, I've clarified in the KIP that the values in static.connectors and static.tasks need not exist, in order to allow for over-provisioning task pods and bootstrapping clusters with 0 replicas. This means that if a connector has fewer tasks than tasks.max, those additional pods will just remain idle, but will immediately pick up the task if the connector creates it.
Whether Strimzi uses the closed-loop polling to find the actual number of tasks or infers the number of tasks for tasks.max is a design tradeoff. We can either complicate the operator, or allow users to over-provision pods by setting a tasks.max which is too high. I don't feel strongly either way, and I'll make sure that the underlying KIP supports both strategies.

Triaged on the Strimzi Community call on 25.1.2024: @gharris1727 What are the plans for this proposal going forward? Should we keep it open? Should we close it until the KIP related to this moves forward?

Hi @scholzj
The current KIP-987 design needs another pass, as the UX of the feature is poor for human operators, and is effectively only usable by machine operators. It is not ready for a vote at this time. The next best design that I know of right now is implementing a taint/selector system within Connect, and using a set of unique task-id/connector-id selectors to give Strimzi complete control over scheduling.
As far as keeping this open or reopening this later: It costs nothing to press the button so i'm fine with either. Since the underlying feature isn't ready I suppose there's no action to take here at the moment, but I am still interested in implementing this proposal at some point in the future.

Triaged on 17/10/2024: agreed on closing the pull request for now and maybe re-open it, or a new one, when KIP-987 will be approved and this one can move forward.


feat: add exercise for pointer concept

@siebenschlaefer
Thanks for all the feedback, you were completely right. This one was hard and it took me quite a while to get back into the exercise game. The windowing java thing was not the right fit for cpp pointers.
I scraped everything and created a completely new exercise, it is more limited in range and probably more direct to help students understand the concepts.
@ahans
You are very welcome to comment :raised_hands:

I added a task, so students can use the & opertor.

Hey @ErikSchierboom, do you have time to review this new exercise? I would like to move the syllabus along, but the other two maintainers seem busy.