bin/tsfclean can be destructive

I have some *.ts files containing dvb recordings; these files play fine in "all" players, but I wanted to run 'tsfclean' on them.
The first one worked ok, but the second one broke in three different ways:

Thumbnail broken
File Properties blank
File won't play

Original file:
bin/tsp -I file /mmedia/rec/dvb/myfile.ts -P history -O drop
* history: 0: PID 0x18A1 (6305) first packet, clear
* history: 3: PID 0x01A8 (424) first packet, clear
* history: 5: PID 0x18A2 (6306) first packet, clear
* history: 18: PID 0x18A1 (6305), PES stream_id is 0xE0 (Video 0)
* history: 50: PID 0x0000 (0) first packet, clear
* history: 51: PAT v5, TS 0x3006 (12294)
* history: 64: PID 0x18A2 (6306), PES stream_id is 0xC0 (Audio 0)
* history: 150: PMT v23, service 0x3910 (14608)
* history: 1189: PID 0x18A1 (6305), first intra-frame, MPEG-2 Video, service 0x3910 (14608)

the video pid starts at packet index 0, and the first intra-frame is at 1189
the audio starts at packet 5, and the first pusi is at 64
there are 107 audio packets before the first video intra-frame
When tsfclean runs, it copied 107 audio packets into the new stream, before the first video packet, so the video starts "late"
Clean file
bin/tsp -I file /mmedia/rec/dvb/avclean.ts -P history -O drop
* history: 0: PID 0x0000 (0) first packet, clear
* history: 1: PAT v0, TS 0x3006 (12294)
* history: 2: PID 0x01A8 (424) first packet, clear
* history: 2: PMT v0, service 0x3910 (14608)
* history: 6: PID 0x18A2 (6306) first packet, clear
* history: 6: PID 0x18A2 (6306), PES stream_id is 0xC0 (Audio 0)
* history: 131: PID 0x18A1 (6305) first packet, clear
* history: 131: PID 0x18A1 (6305), PES stream_id is 0xE0 (Video 0)
* history: 131: PID 0x18A1 (6305), first intra-frame, MPEG-2 Video, service 0x3910 (14608)

The audio now starts at packet index 6, and the video starts at packet index 131, there are still 107 audio packets before the video starts (just like before, but they're squashed together now).
Turns out, this is affecting gstreamer with the playbin element.
It starts reading packets, but there's only audio packets, and it can't find the PCR.
AVFormat players do some extra seeking looking for the PCR
I did an experiment with tsfclean, where it waits for the first video intra-frame before writing the audio packets. This did fix the issue, but it's only an experiment, it's not a real fix.
diff --git a/src/tstools/tsfclean.cpp b/src/tstools/tsfclean.cpp
index e92147fd1..7951f2160 100644
--- a/src/tstools/tsfclean.cpp
+++ b/src/tstools/tsfclean.cpp
@@ -10,6 +10,8 @@
 //
 //----------------------------------------------------------------------------

+// GH test version
+
 #include "tsMain.h"
 #include "tsDuckContext.h"
 #include "tsSignalizationDemux.h"
@@ -208,6 +210,7 @@ ts::FileCleaner::FileCleaner(FileCleanOptions& opt, const fs::path& infile_name)

     // In second pass, count input packets per PID.
     std::map<PID,PacketCounter> pkt_count;
+    bool bVideoStarted = false;

     // Second pass: read input file again, write output file.
     while (_success && _in_file.readPackets(&pkt, nullptr, 1, _opt) == 1) {
@@ -240,7 +243,7 @@ ts::FileCleaner::FileCleaner(FileCleanOptions& opt, const fs::path& infile_name)
         else if (pid_class == PIDClass::AUDIO || pid_class == PIDClass::SUBTITLES || pid_class == PIDClass::DATA) {
             // Write these packets transparently after the first payload unit start.
             const PacketCounter first_index = sig.pusiFirstIndex(pid);
-            if (first_index == INVALID_PACKET_COUNTER || pkt_index >= first_index) {
+            if ((first_index == INVALID_PACKET_COUNTER || pkt_index >= first_index) && bVideoStarted) {
                 _success = _success && _out_file.writePackets(&pkt, nullptr, 1, _opt);
             }
         }
@@ -251,6 +254,7 @@ ts::FileCleaner::FileCleaner(FileCleanOptions& opt, const fs::path& infile_name)
                 first_index = sig.pusiFirstIndex(pid);
             }
             if (first_index == INVALID_PACKET_COUNTER || pkt_index >= first_index) {
+                bVideoStarted = true;
                 _success = _success && _out_file.writePackets(&pkt, nullptr, 1, _opt);
             }
         }

Hi @GerryHickman
Thank you for this analysis.
I would say that the output file is "correct" and that the player is to blame for not being able to play it. The PCR clock is unchanged, the PTS and DTS remain unchanged either. The order of TS packets is unchanged (for packets which remain) and conditions and timing of reception and processing are unchanged.
Concerning the two other points, this is not a matter of file content but how external applications can see the file:

"Thumbnail broken". The thumbnail is not part of the file. This is something your file viewer is building. I don't know under which conditions it can recreate one or not. Again, in that case, the file viewer is to blame, not the file itself.
"File Properties blank". What do you name "file properties"? Ownership and protection? In that case, the output file is created by tsfclean just like any other output file, with the default properties of the user (umask on all Unix systems).

All of that being said, we may try to improve the way some players or file viewers can process the output file.
About initial clock synchronisation: We could keep the video packets which contain a PCR before the first intra-frame. Of course, we delete the payload. However, because the video must be set as "PCR PID" in the PMT, and the PMT comes before any content PID, that may help. These packets are typically found at the start of each initial non-intra frames.
About audio and video synchronisation: The reference is the PTS, not the order of packets in the file. For a given PTS, audio usually comes after video in streams. So, instead of dropping audio packets before the first video intra-frame, it would be better to remove all audio packets with a PTS which is lower (minus some epsilon) than the PTS of the first video intra-frame. If course, this works only if the audio and video are in the clear.
PS: Just a pedantic stylish note, please don't use Hungarian naming, it has been universally hated for decades. Only Microsoft is still using it (which may be a reason for that universal detestation 😉, but not only IMHO).

Hi @GerryHickman
I have implemented the synchronisation of each audio PID on its corresponding video PID. The audio PID's start on next PUSI with PTS at or after the PTS of the first intra-frame of the video of the same service. If no PTS can be found, the audio starts at the next PUSI after the associated video started.
Please try the latest state of the repo (v3.40-3969) and let us know the result in your environment.
Tested tonight on macOS, other platforms will be tested in the GitHub CI during the night.

Here's a more simplistic way of looking at the original problem with the original files (history above) and original tsfclean.
PCR pid 0x18A1
bin/tsp -I file /mmedia/rec/dvb/myfile.ts -P until --packet 256 -P pcrextract --pid 0x18A1 --pcr -O drop
6305,18,15,PCR,1,1182525816220,0,
6305,42,36,PCR,2,1182526731767,915547,
6305,69,58,PCR,3,1182527683224,1867004,
6305,94,80,PCR,4,1182528622711,2806491,
6305,120,103,PCR,5,1182529583143,3766923,
6305,145,125,PCR,6,1182530522631,4706411,
6305,171,147,PCR,7,1182531453144,5636924,
6305,200,170,PCR,8,1182532442001,6625781,
6305,225,192,PCR,9,1182533375505,7559285,
6305,249,213,PCR,10,1182534276098,8459878,

gstreamer starts reading packets, and "instantly" finds the PCR at packet 18
PCR pid 0x18A1
bin/tsp -I file /mmedia/rec/dvb/avclean.ts -P until --packet 256 -P pcrextract --pid 0x18A1 --pcr -O drop
6305,153,20,PCR,1,1182560746328,0,
6305,186,50,PCR,2,1182561715736,969408,
6305,218,79,PCR,3,1182562646247,1899919,
6305,253,110,PCR,4,1182563573766,2827438,

gstreamer starts reading (and reading), but yikes, no PCR until packet 153

To cope with the situation where PCR are not in PUSI packets (start of PES packets), tsfclean now passes video packets preceding the first intra-frame when they contain a PCR. In that case, the payload is deleted (PCR-only packet).
You need version 3.40-3975 or higher.

I'm looking forward to testing the new version of 'tsfclean', but I still don't know if there was a problem with the old one. The tests above are using 'tsfclean' from tsduck 3.38-3822. In your first reply, you said the output file was correct and the PCR had not changed.
Do you still have access to 'tsfclean' from tsduck 3.38-3822?
I'll try to make a smaller version of the test file above, I didn't know you could upload things here.

Do you still have access to 'tsfclean' from tsduck 3.38-3822?

All versions are available here: https://github.com/tsduck/tsduck/releases
And with the magic of git, we can rebuild any older version.
I can only say that tsfclean has never, ever, modified a packet, especially not a PCR value. There is simply no code for that in tsfclean. It deletes some packets at the beginning of the file and it replaces the PAT/PMT/EIT signalisation, that's all.
Now, I agree that the previous version, up to v3.39, let pass some audio PES before the video in a service. Since the PCR are in the video PID, the player receives some audio PES before the first PCR of the service. If this is a problem for the player, this is a player's issue, not a file issue. That being said, TSDuck remains pragmatic and implements ways of fixing bugs, much more important bugs, in various receivers (there are a lot of options for that). So, the new version (the current development state of the repo) changes this and now tries to align the start of audio with the start of video.
I will probably not reinstall / rebuild v3.38 to test a file. However, to solve your issue, I recommend that you test the new version on your file and player.

I understand where you say you don't want to test with older version.
So my understanding of the original issue is not that tsfclean changed a PCR packet, but rather it created a gap before the first PCR (e.g. audio packets first). It does raise an interesting question; how many ts packets is a player expected to read before it decides there's no PCR?

So my understanding of the original issue is not that tsfclean changed a PCR packet, but rather it created a gap before the first PCR (e.g. audio packets first).

Exactly.

It does raise an interesting question; how many ts packets is a player expected to read before it decides there's no PCR?

And the answer is most probably "it depends".
It depends on the receiver or player. I have seen so many different behaviours in receivers, many of them being buggy. The desktop software players are usually more resilient. VLC, for instance, is ubiquitous. It decodes virtually everything. If VLC doesn't decode, it is probably a problem with the stream. If a set-top box doesn't decode, well, usually the set-top box is to blame.

This raises a different question:
Is there any way to determine if a transport stream is "well formed".
e.g. imagine a stream where there's 10 minutes of audio before the PCR and video.
'tsanalyze' already has the --error-analysis option, but that's slightly different.

Is there any way to determine if a transport stream is "well formed".

What is the definition of "well formed"?
I don't think that this definition exists in the most general sense. As long as it is conformant with ISO 13818-1, we can say it is "well formed". Then, DVB added a few standards for "conformant broadcast stream" (which are sometimes incompatible with other broadcasting standards such as SCTE-35). However, ISO 13818-1 implicitly defined TS as continuous broadcast format, without begin or end. In that context "starting audio before video" is meaningless because there is no "beginning". A storage format has a beginning. However, for ISO 13818-1, the storage format is program stream (DVD), not transport stream. And after DVD, the BluRay storage format switched to TS (or "M2TS"), which was not anticipated. BluRay must have its own set of requirements, I guess (I don't know them). And then HLS specified the requirement for the start of a TS segment file (PAT, PMT, intra-frame, but nothing about audio/video starting order as far as I remember).
So, what is a "TS file"? A slice of broadcast stream? A BluRay M2TS? A HLS segment? Stuff that was generated by ffmpeg or other tools, without formal definition?
I am afraid that the quest for the universal "well formed" transport stream is vain...
Let's be pragmatic and let's generate something that is widely accepted.

Thanks for the update about trying to define what "well formed" means.
I've tried to re-build at this commit:
commit 1eb1bea64eac0f1e30620e43b6c2c2f23f5f5459 (HEAD -> gh_build, origin/master, origin/HEAD, master)
Author: Thierry Lelegard<EMAIL_ADDRESS>Date:   Sun Dec 1 00:49:35 2024 +0100
    Fixed issue in AVC and EVC descriptors (issue #1558)

but I'm getting an error from Make
make[2]: *** No rule to make target '/data/build/tsduck/src/libtsduck/base/network/tsIPv4Address.h',
needed by '/data/build/tsduck/bin/release-x86_64-gh01/objs-libtsduck/tsATSCMultipleString.o'.  Stop.
this has never happened before

Run make clean first. You probably still have .dep files from builds of previous versions. Some recent updates have unified IPv4 and IPv6 classes. The distinct classes for IPv4 and IPv6 have disappeared but you clearly still have dependencies on them.

After running 'make clean', the build now completes, and all tsduck programs are working.
A quick test on a broken file from yesterday, now works.

Thumbnail is now created (was broken)
Duration is now correct (was zero)
File Properties, now populated (were empty)
File Seeking was broken (now working accurate)
File is able to play.

I'll now re-run the more detailed 'tsfclean' tests above.

Regarding "well formed" transport streams, I should have said containing DVB in SD.
I found this document
https://download.tek.com/document/25W_14706_0.pdf
On Page 2, it says
#"PCRs
are snapshots of a counter, driven by this program clock, that are
inserted into packets within the Transport Stream (TS) at more or less
regular intervals (ISO/IEC13818-1 specifies an interval of 100 ms,
while DVB recommends 40 ms)."

Regarding "well formed" transport streams, I should have said containing DVB in SD.

Precisely. DVB is a broadcast standard (sometimes IP multicast). There is no concept of "DVB file". A DVB transport stream is a continuous stream without beginning or end. So, there is nothing about "starting" audio before video. Anyway, the concept of "TS file" (DVB or not) is undefined. TV engineers don't manipulate "TS files", they manipulate "slices of broadcast" which are meaningless on ends. We just have tools, with their limitations. And we try to adapt when necessary.

I have now carried out more detailed testing with tsfclean v3.39-3956.
I used the exact same file as in the original post.
The original post shows the history before running tsfclean, and then after running the old tsfclean.
The old tsfclean was causing files to become unplayable due to skipping some PCR packets
The new tsfclean v3.39-3956
* history: 0: PID 0x0000 (0) first packet, clear
* history: 1: PAT v0, TS 0x3006 (12294)
* history: 2: PID 0x01A8 (424) first packet, clear
* history: 2: PMT v0, service 0x3910 (14608)
* history: 4: PID 0x18A1 (6305) first packet, clear
* history: 61: PID 0x18A1 (6305), PES stream_id is 0xE0 (Video 0)
* history: 61: PID 0x18A1 (6305), first intra-frame, MPEG-2 Video, service 0x3910 (14608)
* history: 853: PID 0x18A2 (6306) first packet, clear
* history: 853: PID 0x18A2 (6306), PES stream_id is 0xC0 (Audio 0)

pcr pid 0x18A1
video pid 0x18A1 now starts much earlier at packet index 4, was 131
audio pid 0x18A2 now starts much later at packet index 853, was 64
pcr pid 0x18A1
6305,4,0,PCR,1,1182525816220,0,
6305,5,1,PCR,2,1182526731767,915547,
6305,8,2,PCR,3,1182527683224,1867004,
6305,9,3,PCR,4,1182528622711,2806491,
6305,10,4,PCR,5,1182529583143,3766923,
6305,11,5,PCR,6,1182530522631,4706411,
6305,13,6,PCR,7,1182531453144,5636924,
6305,16,7,PCR,8,1182532442001,6625781,

The file is able to play
BUT
bin/tsp -I file /mmedia/rec/dvb/avclean.ts -P continuity -O drop
* continuity: packet index: 5, PID: 0x18A1 (6305), missing 4 packets
* continuity: packet index: 8, PID: 0x18A1 (6305), missing 5 packets
* continuity: packet index: 9, PID: 0x18A1 (6305), missing 5 packets
* continuity: packet index: 10, PID: 0x18A1 (6305), missing 6 packets
* continuity: packet index: 11, PID: 0x18A1 (6305), missing 5 packets
* continuity: packet index: 13, PID: 0x18A1 (6305), missing 5 packets
* continuity: packet index: 16, PID: 0x18A1 (6305), missing 6 packets
* continuity: packet index: 17, PID: 0x18A1 (6305), missing 5 packets
* continuity: packet index: 18, PID: 0x18A1 (6305), missing 4 packets
* continuity: packet index: 19, PID: 0x18A1 (6305), missing 6 packets
* continuity: packet index: 21, PID: 0x18A1 (6305), incorrect CC increment without payload
* continuity: packet index: 22, PID: 0x18A1 (6305), incorrect CC increment without payload
* continuity: packet index: 28, PID: 0x18A1 (6305), incorrect CC increment without payload
* continuity: packet index: 30, PID: 0x18A1 (6305), incorrect CC increment without payload
* continuity: packet index: 31, PID: 0x18A1 (6305), incorrect CC increment without payload
* continuity: packet index: 34, PID: 0x18A1 (6305), incorrect CC increment without payload
* continuity: packet index: 35, PID: 0x18A1 (6305), missing 14 packets
* continuity: packet index: 36, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 37, PID: 0x18A1 (6305), missing 14 packets
* continuity: packet index: 39, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 40, PID: 0x18A1 (6305), missing 14 packets
* continuity: packet index: 43, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 44, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 45, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 46, PID: 0x18A1 (6305), missing 14 packets
* continuity: packet index: 48, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 51, PID: 0x18A1 (6305), missing 14 packets
* continuity: packet index: 52, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 53, PID: 0x18A1 (6305), missing 14 packets
* continuity: packet index: 54, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 56, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 57, PID: 0x18A1 (6305), missing 13 packets
* continuity: packet index: 60, PID: 0x18A1 (6305), missing 12 packets
* continuity: packet index: 61, PID: 0x18A1 (6305), missing 8 packets

I assume that this PID 6305 is the video PID, the one with PCR's. This is clearly the early PCR packets which were kept (with payload removed), while other packets were removed. If your listing is complete, the discontinuities stop when the video packets start. So, because intermediate packets were removed, this is the expected behavior.
Now, is it a problem ? Why would this be a problem ?
The CC are here to detect missing packets in an elementary stream of PES or section content. The discontinuities indicate that the flow of payload content becomes invalid because there is some missing part. All those packets have no payload, no content. So, there is no missing part and no problem.
If you are visually displeased by the output of the plugin continuity, you can can use the same plugin to "fix" them. However, so-called "fixing" discontinuities is sometimes worse than keeping them when they hide actual problems. If your input file has discontinuities later, then "fixing" the CC will hide them and the video codec may display garbage instead of skipping a frame or part of one, which is invisible to the human eye.
Many people focus on CC errors and want them to disappear but they are useful.

The packets with the continuity errors are between 0 and 61, which agrees with the history above where PCR pid starts at 61, so yes they can be ignored, but the original file shows no errors, and running a "cleaning" tool creates a file with errors.
I need to re-think what tsfclean is all about. The placing of the PSI/SI works well. I'm not so sure about the PES packets.

I have aligned all continuity counters at the beginning of the video PID, the packets with PCR only and no payload, with the continuity counter of the first packet of the intra-frame. Therefore, there is no longer any reported discontinuity at the beginning of the output. It does not change the usage of the file but it will avoid future remarks about reported discontinuities (although harmless in that context). 😉
Note that if there is any further discontinuity in the input file, there are propagated unchanged to the output, because this is the right thing to do.

I've now tested the updated version of 'tsfclean'.
History and PCR offsets (on the clean file) are the same as above.
Continuity errors are gone.
File is able to play.
Issue can be closed.
Before posting this issue, I was going to run 'tsfclean' in batch mode, and delete the original files, but then noticed the "clean" files were unable to play, luckily I kept the originals.


Support for group dependencies ?

Just an idea that could make a lot of sense with groups.
Paket does support groups - but as far as I understand this - they have no relationships for the moment. For example:
group A
    source ...
    nuget FSharp.Core

group B
    source ...
    nuget FsUnit

implies A and B can have different version of FSharp.Core. This could be a problem for example for 2 projects (each using a specific Paket group) with dependency on a base library (with specific nuget).
Using groups is complicated because base nugets could end up being different for base library.
This could be interesting to support group dependencies to simplify groups management. I mean, groups could then be used as prerequisites for other groups and ensuring other groups are strictly compatible with inherited dependencies/versions.
For example:
group A
    source ...
    nuget FSharp.Core

group C
    source ...
    nuget XXX

group B <- A, C
    source ...
    nuget FsUnit

would force B to be compatible with versions resolved within group A and C.
What do you think about this ? Could that be interesting for Paket ?
Thanks.

We introduced groups as resolution boundaries so that you can have
different versions. So in some sense it's the original feature.
Your proposal sounds interesting. Do you have real use cases for this? It
would introduce a bit of complexity and I'm not sure yet if that's
justified.
Am 11.08.2017 10:06 nachm. schrieb "Pierre Chalamet" <
<EMAIL_ADDRESS>
Just an idea that could make a lot of sense with groups.
Paket does support groups - but as far as I understand this - they have no
relationships for the moment. For example:
group A
source ...
nuget FSharp.Core
group B
source ...
nuget FsUnit
implies A and B can have different version of FSharp.Core. This could be a
problem for example for 2 projects (each using a specific Paket group) with
dependency on a base library (with specific nuget).
Using groups is complicated because base nugets could end up being
different for base library.
This could be interesting to support group dependencies to simplify groups
management. I mean, groups could then be used as prerequisites for other
groups and ensuring other groups are strictly compatible with inherited
dependencies/versions.
For example:
group A
source ...
nuget FSharp.Core
group C
source ...
nuget XXX
group B <- A, C
source ...
nuget FsUnit
would force B to be compatible with versions resolved within group A and C.
What do you think about this ? Could that be interesting for Paket ?
Thanks.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2608, or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNFXcYpxvGwGaO8JxRzJdMCgMx7wEks5sXLQxgaJpZM4O1FCL
.

Yes sure.
We have a microservices architecture with some common assemblies. Services can freely use whatever nugets they want (until it's compatible with the one in common assemblies). Everything is built from sources only even implementations span on multiple repositories (but applies equally in single repository).
Managing nugets for common assemblies is easy - but managing individually each service is complicated because we have to ensure nugets referenced by services are compatible with the one used in common assemblies.
That's where dependencies for groups could be interesting. Common assemblies would specify nugets for common assemblies and services could have their own group taking dependency on common group.
Services could upgrade at their own rhythm and would not require to be kept strictly in sync.
I do agree upgrading common group implies upgrading dependent groups. But the reverse is not true and services could be upgraded independently. I imagine this could be just new constraints when solving version for a group (like framework constraint).
Hope it's clearer! Thanks.

So if a group B depends on group A what would we do in the resolver?
Add all packages from group A as dependencies with pinned versions?
Am 12.08.2017 9:09 vorm. schrieb "Pierre Chalamet"<EMAIL_ADDRESS>
:

Yes sure.
We have a microservices architecture with some common assemblies. Services
can freely use whatever nugets they want (until it's compatible with the
one in common assemblies). Everything is built from sources only even
implementations span on multiple repositories (but applies equally in
single repository).
Managing nugets for common assemblies is easy - but managing individually
each service is complicated because we have to ensure nugets referenced by
services are compatible with the one used in common assemblies.
That's where dependencies for groups could be interesting. Common
assemblies would specify nugets for common assemblies and services could
have their own group taking dependency on common group.
Services could upgrade at their own rhythm and would not require to be
kept strictly in sync.
I do agree upgrading common group implies upgrading dependent groups. But
the reverse is not true and services could be upgraded independently. I
imagine this could be just new constraints when solving version for a group
(like framework constraint).
Hope it's clearer! Thanks.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2608#issuecomment-321962877,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNItF1WPiUOI805Z0khybT5MNXf3Oks5sXU-6gaJpZM4O1FCL
.

Basically, yes.

So if I understand correctly group B would be forced to use FSharp.Core from group A?
What about transitives? Can group B upgrade transitives as long as they are compatible with direct deps of group A?
@forki If transitives are locked this is quite similar to what we do with the runtime resolution. I'm pretty sure we could unify those.

Off topic: It is a bit spurious to have microservices but make them depend on a common set of packages. Shouldn't they be independent of each other...

Yeah I think if we add the whole resolution from A as pins to B then this
could work easily.
So IF we want to do this then I think biggest issues are:

finding proper syntax
*writing a nice tutorial / docs

Resolution part is probably the easier part
Am 12.08.2017 9:17 vorm. schrieb "Matthias Dittrich" <
<EMAIL_ADDRESS>
So if I understand correctly group B would be forced to use FSharp.Core
from group A?
What about transitives? Can group B upgrade transitives as long as they
are compatible with direct deps of group A?
@forki https://github.com/forki If transitives are locked this is quite
similar to what we do with the runtime resolution. I'm pretty sure we could
unify those.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2608#issuecomment-321963205,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNFfmO3tMt0AmrJ5POBC0NQ12diMZks5sXVGlgaJpZM4O1FCL
.

Not sure at all about upgrading inherited dependencies - idea is probably to pin whatever is inherited from other groups and then resolve for the given group.
For the OT comment - yeah but that's life and copy/paste is not the best way to handle this.

framework like syntax could be used. "Inherits" is probably not that good but it's the idea.
group C
inherits: A, B

Also I'm not sure about the Main group, we probably should not allow it to participate in this feature (neither actively nor passively) as the name of the main group is an implementation detail..

Also we should only allow to reference groups defined above to prevent cycles?

Main group could still be referenced by name "Main" explicitly if required. Probably there are already checks in place to avoid naming a group like this if the default one is implicitly created. There is probably no good reason to exclude it but it's maybe part of an unknown compatibility to me.
For the order, yes this would help with implementation tremendously (not necessarily a good reason) and ensure clear readability at the same time. Like f#.

Yes Main group is explicitly used in many places. Changing the name of the
main group would be really breaking for many users. So I think it's fine.
Am 12.08.2017 16:40 schrieb "Pierre Chalamet"<EMAIL_ADDRESS>
Main group could still be referenced by name "Main" explicitly if
required. Probably there are already checks in place to avoid naming a
group like this if the default one is implicitly created. There is probably
no good reason to exclude it but it's maybe part of an unknown
compatibility to me.
For the order, yes this would help with implementation tremendously (not
necessarily a good reason) and ensure clear readability at the same time.
Like f#.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2608#issuecomment-321985424,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNOS_vobwdm-sqvMLq-q_qQ9bXeAwks5sXblMgaJpZM4O1FCL
.

I think that this proposal is interesting.
Correct me if I'm wrong but this feature could be useful to improve performance in a specific case that we have at our company.
Most of our nuget packages are provided from one source (most of the time nuget.org). But we have (depending on the project), around 6 to 10 packages provided by as much as sources (internal sources) .
So we end up with a lot of sources (which are also slower than the main source) queried for each dependencies.
With this feature, we will be able to create some small groups for these specific dependencies with just their sources. And with the main group containing only the main source.
This preventing to query all the internal slow sources for all the packages provided by the main source.

Most of our nuget packages are provided from one source (most of the time nuget.org). But we have (depending on the project), around 6 to 10 packages provided by as much as sources (internal sources) .

So we end up with a lot of sources (which are also slower than the main source) queried for each dependencies.

With this feature, we will be able to create some small groups for these specific dependencies with just their sources. And with the main group containing only the main source.

@pmiossec I don't think this is how this feature should be used. In fact we should just fix any existing performance problems...

Yes Main group is explicitly used in many places.

@forki But not jet by the user itself in the packages-files? I still think it's a implementation detail. People should not use it.

Yes the main groups are explicitly used in dependencies files and
references files by many users.
Am 13.08.2017 10:38 nachm. schrieb "Matthias Dittrich" <
<EMAIL_ADDRESS>
Most of our nuget packages are provided from one source (most of the time
nuget.org). But we have (depending on the project), around 6 to 10
packages provided by as much as sources (internal sources) .
So we end up with a lot of sources (which are also slower than the main
source) queried for each dependencies.
With this feature, we will be able to create some small groups for these
specific dependencies with just their sources. And with the main group
containing only the main source.
@pmiossec https://github.com/pmiossec I don't think this is how this
feature should be used. In fact we should just fix any existing performance
problems...
Yes Main group is explicitly used in many places.
@forki https://github.com/forki But not jet by the user itself in the
packages-files? I still think it's a implementation detail. People should
not use it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2608#issuecomment-322066037,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNEt1CMIMkFB2xYwjlH-IeXuGLuQVks5sX17dgaJpZM4O1FCL
.

Ok to sum up. We discussed:

dependencies between groups could be implemented
syntax has to be defined to reference others groups (proposed is "inherits" but has to be challenged)
Main group can be referenced by name "Main"
a group can only reference groups which have been defined previously in Paket.dependencies
"inherited" group dependencies from resolver point of view is just a bunch of pinned versions.

Last point raises interesting questions btw: this could lead to conflicts if 2 groups brings same artifacts but with different versions. This shall end as a resolver error probably. What's your take on this?
Other point: groups/nugets/files are stored to their own packages/ folder on file system. Shall all inherited groups be included as well in folder?

Pretty sure this will already end up being resolver error.
Am 14.08.2017 10:55 vorm. schrieb "Pierre Chalamet" <
<EMAIL_ADDRESS>
Ok to sum up. We discussed:

dependencies between groups could be implemented
syntax has to be defined to reference others groups (proposed is
"inherits" but has to be challenged)
Main group can be referenced by name "Main"
a group can only reference groups which have been defined previously
in Paket.dependencies
"inherited" group dependencies from resolver point of view is just a
bunch of pinned versions.

Last point raises interesting questions btw: this could lead to conflicts
if 2 groups brings same artifacts but with different versions. This
shall end as a resolver error probably. What's your take on this?
Other point: groups/nugets/files are stored to their own packages/ folder
on file system. Shall all inherited groups be included as well in folder?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2608#issuecomment-322134682,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNO9kx0fXfBZ5TYZdZ4sdj4QhKIBkks5sYAtqgaJpZM4O1FCL
.

Btw the github deps feature is very similar. It consumes remote
paket.dependencies files and merges them into your local groups.
Am 14.08.2017 11:04 vorm. schrieb "Steffen Forkmann"<EMAIL_ADDRESS>
Pretty sure this will already end up being resolver error.
Am 14.08.2017 10:55 vorm. schrieb "Pierre Chalamet" <
<EMAIL_ADDRESS>
Ok to sum up. We discussed:

dependencies between groups could be implemented
syntax has to be defined to reference others groups (proposed is
"inherits" but has to be challenged)
Main group can be referenced by name "Main"
a group can only reference groups which have been defined
previously in Paket.dependencies
"inherited" group dependencies from resolver point of view is just
a bunch of pinned versions.

Last point raises interesting questions btw: this could lead to conflicts
if 2 groups brings same artifacts but with different versions. This
shall end as a resolver error probably. What's your take on this?
Other point: groups/nugets/files are stored to their own packages/ folder
on file system. Shall all inherited groups be included as well in folder?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/2608#issuecomment-322134682,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNO9kx0fXfBZ5TYZdZ4sdj4QhKIBkks5sYAtqgaJpZM4O1FCL
.

Other point: groups/nugets/files are stored to their own packages/ folder on file system. Shall all inherited groups be included as well in folder?

This is indeed a good question considering we are trying to slowly get rid of the packages folder.
We could either say yes they are duplicated until we figured it out (easier to implement). Or we say don't duplicate and do some part of the "flexible" packages location as part of this feature...
I'm pretty sure part of this logic can be reused later.

So are you ok to go forward with this proposition? Is there something to investigate before going on ?
Do you see value implementing this?
If we agree on this, is there someone looking at it or should I try to look at codebase and see how to implement this ?


Change github link

@all-contributors please add @haxzie for design

@all-contributors please add @haxzie for code


Added bulk domain procedures

Per VSO.

Docs Build status updates of commit 7a0588c:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.
For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 7a0588c:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.
For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 96ecb23:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.
For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

closing/re-opening PR

Docs Build status updates of commit 96ecb23:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.
For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit ab50bae:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

exchange/exchange-ps/exchange/New-SafeLinksRule.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/Set-SafeLinksRule.md
:white_check_mark:Succeeded
View (exchange-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Fix: broken ledger tests

This PR addresses the following fixes:

Broken test for mirroring ledger entries: The test previously attempted to mirror an empty state because the query contained a numeric ID instead of the expected string ID.
Missing test for date type validation: Added a test to ensure that the date field in ledger entries has the correct type."

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Flag
Coverage Δ

unittests
65.18% <100.00%> (+0.03%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines
Coverage Δ

pyledger/ledger_engine.py
66.95% <100.00%> (+0.07%)
:arrow_up:


Add Crypto::Bcrypt::Password#==

Use case: my User records (via DB::Serializable) aren't considered == because their password properties are not considered == despite all properties being equal. This leads to confusing spec failures when doing actual_user.should eq expected_user.
Since Crypto::Bcrypt::Password doesn't appear to accrue state, as an alternative to adding == it seems like it could instead be a struct vs a class? This would do the same thing. Both of these ideas pass in spec/std/crypto/bcrypt/password_spec.cr. I went with the def_equals because it's the monkeypatch I use in apps that require passwords (you can't monkeypatch a class to a struct), but I'm also happy to convert it to a struct.

This method existed previously but was removed after the discussion in #7753. I think the main issue is that Crypto::Bcrypt::Password is meant as tool for creating and verifying passwords, not for storage.
Password hashes are supposed to be stored as String or equivalent data types.
So I'd suggest to consider using a different type for password storage in your model as an alternative solution.
We can certainly talk about use cases and reconsider the decision on #7753, but that'll need deep arguments.
We should close this PR for now, as it's not mergeable without repealing a previous decision.

That issue seems to be about using == to check that a string hashes to the password’s hashed value. Did I overlook something in there about comparing two equivalent Crypto::Bcrypt::Password instances with the same state?

All I’m looking for is that two objects of the same type with the same state return true with ==.

I think the main issue is that Crypto::Bcrypt::Password is meant as tool for creating and verifying passwords, not for storage. Password hashes are supposed to be stored as String or equivalent data types.

I’m sincerely curious how you think I’m storing it in my DB, because what you’re describing is what I’m doing. I’m storing it in the DB as a string and decoding it into a password when retrieving it. I don’t know of another way to store it, tbh.

Crypto::Bcrypt::Password == String got removed in https://github.com/crystal-lang/crystal/issues/7753
but this PR adds
Crypto::Bcrypt::Password == Crypto::Bcrypt::Password

That's a valid point then 👍

I’m sincerely curious how you think I’m storing it in my DB, because what you’re describing is what I’m doing. I’m storing it in the DB as a string and decoding it into a password when retrieving it. I don’t know of another way to store it, tbh.

I don't mean for permanent storage. But you must be storing a Password instance in your User model because it seems to affect the equality of that model.
My comment is supposed to dissuade from doing that. Instead, the user's password property should be a String (or equivalent) and only instantiate Bcrypt::Password temporarily for the password verification process.

I disagree. That's analogous to holding JSON blobs in their string format and only parsing them temporarily to access their properties, then discarding the parsed data structure.
A password passed through bcrypt is not an opaque string like the cleartext password is. It's converted to a string to send over the wire, such as to a database, but it's a data structure containing 4 scalars. This is the concept that Crypto::Bcrypt::Password represents. It doesn't contain the bcrypt algorithm — it's just the parsed data structure.

With that out of the way, I can explain why it is invalid to blindly compare Crypto::Bcrypt::Password instances.

To achieve permissive enough comparison, the version needs to be explicitly ignored for all currently known cases.

Correct usage:
require "crypto/bcrypt/password"
stored = Crypto::Bcrypt::Password.create("super secret", cost: 10).to_s.sub("$2a$", "$2y$")
retrieved = Crypto::Bcrypt::Password.new(stored)
p retrieved.verify("super secret")  # => true

Wrong usage:
require "crypto/bcrypt/password"

class Crypto::Bcrypt::Password
  def_equals_and_hash version, cost, salt, digest
end

stored = Crypto::Bcrypt::Password.create("super secret", cost: 10).to_s.sub("$2a$", "$2y$")
retrieved = Crypto::Bcrypt::Password.new(stored)
p Crypto::Bcrypt::Password.create("super secret", cost: 10) == retrieved  # => false

For potential future revised versions of bcrypt, the version must be actually used. You need to have the ability to switch to a new version for all newly created hashes but keep using the old version for verification against hashes that you already have stored.

Most importantly and most realistically: to avoid mismatches with the cost!!
If you ever change the cost, you need to be able to keep using old hashes that you already have and match their cost, not use whatever is the new one.

Correct usage:
require "crypto/bcrypt/password"

cost = 10
stored = Crypto::Bcrypt::Password.create("super secret", cost: cost).to_s
retrieved = Crypto::Bcrypt::Password.new(stored)

cost = 12  # Decided to change the cost in the future

p retrieved.verify("super secret")  # => true

Wrong usage:
require "crypto/bcrypt/password"

class Crypto::Bcrypt::Password
  def_equals_and_hash version, cost, salt, digest
end

cost = 10

stored = Crypto::Bcrypt::Password.create("super secret", cost: cost).to_s
retrieved = Crypto::Bcrypt::Password.new(stored)

cost = 12  # Decided to change the cost in the future

p Crypto::Bcrypt::Password.create("super secret", cost: cost) == retrieved)  # => false

As such, this pull request is invalid.

^ The bcrypt module probably should not have had a "default" cost defined, only suggested via examples.

Further footgun №2
def_equals without any arguments creates an equals method that always returns true!!!
@jgaskins clearly must have thought that by default it uses all fields, but actually it uses no fields. So this PR makes all Password objects equal to each other! Maybe def_equals without arguments should fail, or print a warning considering backwards compatibility

Issue: it becomes possible to compare bcrypt hashes in non constant time.

@ysbaddaden Because of this PR? Why would anyone be using this method to compare hashes cryptographically?

Maybe it was a mistake to expose this in an OOP style and the API should just contain of a .verify method.

@straight-shoota Maybe, if the rest of the core team holds the same position you do about it. Having the algorithm exposed in the Crypto::Bcrypt module and another object type specifically to represent a password makes your position shaky at best. If you’re this intent on it only being used on strings, the Password type makes no sense at all and the Bcrypt module should just have something like a verify_password method that receives the hash and the cleartext string.

To achieve permissive enough comparison, the version needs to be explicitly ignored for all currently known cases.

@oprypin This makes sense.
The code examples you gave don't, though. They return false, but not for the reason you think. Here is your first "wrong usage" code example, with the only modification being to show the values instead of simply whether they're equal:
require "crypto/bcrypt/password"

class Crypto::Bcrypt::Password
  def_equals_and_hash version, cost, salt, digest
end

stored = Crypto::Bcrypt::Password.create("super secret", cost: 10).to_s.sub("$2a$", "$2y$")
retrieved = Crypto::Bcrypt::Password.new(stored)
puts Crypto::Bcrypt::Password.create("super secret", cost: 10)
# $2a$10$LHorOxEFXwvVsnvWJSPYTeDInFat9GPJeHg6r5EfgaUB48bo63kHi
puts retrieved
# $2y$10$XVMuriHkS9NYM2EcSBGLD.LV58fY15GNSv/eVwMDesBLeu0PmQsPO

They aren't mismatched solely due to the different version values. They're mismatched because the only component that matches is the cost.
Unlike SHA, bcrypt isn't deterministic. Every time you call Password.create, a new salt is generated and, by extension, a new digest. This is by design and is the reason the salt exists:
require "crypto/bcrypt/password"

alias Password = Crypto::Bcrypt::Password # This is so clunky to type

2.times { puts Password.create("password") }
# $2a$11$KjbncjfAhTe4xOk5qrem/OpZi7NdbZVq8Sm5.YL2wEe1HlmE.2L4C
# $2a$11$OiAG3SyUL95/NHvguErQfeQN1xv/nOyirpCS4n.3VEuJuci300gs.

This tells me that we can probably never change the value of DEFAULT_COST, or there will be outages.

Again, this draws a conclusion based on incorrect information. Here is a modified version of your code example where we don't change cost (all I did was comment out the line):
require "crypto/bcrypt/password"

cost = 10

stored = Crypto::Bcrypt::Password.create("super secret", cost: cost).to_s

# cost = 12  # Decided to change the cost in the future

p Crypto::Bcrypt::Password.create("super secret", cost: cost).to_s == stored  # => false

The output is exactly the same.

@jgaskins clearly must have thought that by default it uses all fields, but actually it uses no fields.

This is precisely what I thought. TIL.

Maybe it was a mistake to expose this in an OOP style and the API should just contain of a .verify method.

It was modeled after Ruby's bcrypt gem... I agree we don't need an object, a couple methods would be sufficient:

Crypto::BCrypt.hash(password, *, salt = nil, cost = DEFAULT_COST) : String
Crypto::BCrypt.verify(hash, password) : Bool

It would avoid the issues mentioned above, limit the exposed API, simplify the internal implementation, and we could avoid a bunch of intermediary allocations (especially Bytes).
We should definitely consider moving to such a limited API, then deprecate the Crypto::BCrypt and Crypto::BCrypt::Password classes :+1:

I suppose the only use case for the Password type is that you can use one instance for multiple verify operations. This might save a little bit of overhead, but if the string parsing was implemented efficiently, it would be negligible.
Also I can't think of many scenarios where you'd want to verify multiple passwords against the same hash. Maybe if the password was mistyped and tried again. But this seems very week.

we could avoid a bunch of intermediary allocations (especially Bytes).

They're actually String allocations. Dunno why.
This should work perfectly fine with non-allocating subslices of Bytes 🤷

I can't think of many scenarios where you'd want to verify multiple passwords against the same hash.

Exactly. Maybe a rainbow table or for hacking purposes... in which case there are faster alternatives.

@jgaskins
Ah yes I didn't take into account the fact that creating a bcrypt hash is based on random salt, so the "invalid" examples are uhh... invalid. And my statement about "we can probably never change the value of DEFAULT_COST" / "Further footgun №1" is fully debunked.
But the initial big point is still applicable - it can be misleading to directly compare two bcrypt hashes.


Add a few functions to Data.List.NonEmpty.Extra

sortOn, union and unionBy are useful since they produce non-empty lists, so if one converts the input to regular lists and uses Data.List's version, the output needs to be converted back to non-empty lists using the partial fromList function.
The minimum and maximum functions are the same as Data.Foldable's version but are total, and thus may be preferred by some.

Thanks, looks very nice.

As before, let me know when you're done and I'll make a release.

@ndmitchell I should be done with this module for the time being; a release would be helpful. Thanks!

@zliu41 1.6.15 now out


Disable Bluetooth mouse

Hi. I found out after installing Big Sur that my Apple ID does not work with Big Sur. After inserting the uuid and serial number, my Apple ID worked but my Bluetooth mouse failed. Please help me.
The EFI file is attached
https://mega.nz/file/7LhigKwa#MxFsZcy7R32k5lNfbAoggwTGZyt8eV4XOT12525MCcE

Hi. I found out after installing Big Sur that my Apple ID does not work with Big Sur. After inserting the uuid and serial number, my Apple ID worked but my Bluetooth mouse failed. Please help me.

Bluetooth not working in Big Sur. No working kext for IntelBluetoothFirmware. No solution now.

My mouse is a simple mouse that worked with Big Bluetooth in Big Sur. After inserting uuid to apply Apple ID, it did not work all at once.

Hi. I found out after installing Big Sur that my Apple ID does not work with Big Sur. After inserting the uuid and serial number, my Apple ID worked but my Bluetooth mouse failed. Please help me.

Bluetooth not working in Big Sur. No working kext for IntelBluetoothFirmware. No solution now.

Now I understand something. None of the USB ports work. This happened after setting uuid to apply Apple ID. please help

Now I understand something. None of the USB ports work. This happened after setting uuid to apply Apple ID.
Can you look at the EFI file I sent you the link to? I may have made a mistake somewhere I did not realize.
please help

You generate wrong SMBIOS!
I make new SMBIOS for you.
EFI.zip

Now I understand something. None of the USB ports work. This happened after setting uuid to apply Apple ID.

Can you look at the EFI file I sent you the link to? I may have made a mistake somewhere I did not realize.

please help

You generate wrong SMBIOS!
I make new SMBIOS for you.
EFI.zip

Thanks a lot.    How do I reduce the size of the Apple logo and hard drives?

Thanks a lot.  How do I reduce the size of the Apple logo and hard drives?

Replace EFI and reset NVRAM
EFI.zip

Thanks a lot.  How do I reduce the size of the Apple logo and hard drives?

Replace EFI and reset NVRAM
EFI.zip

Thank you very much.  How do I set up SMBIOS correctly to do this for myself next time?

How do I set up SMBIOS correctly to do this for myself next time?

Simple download app OpenCore Configurator
https://mackie100projects.altervista.org/opencore-configurator/
Open your config.plist

Save config.


Interactive choosing on duplicated season numbers

Crunchyroll sometimes marks different seasons with the same season number (eg. Sword Art Online, Demon Slayer). Until now it was impossible to download a specific season if it has a duplicates; always all (duplicated) seasons with the assigned season number were downloaded.
To solve this, a interactive prompt is now shown, if a series has seasons with duplicated season number. The also added -y/--yes flag can be used to suppress the prompt and download everything without further interactive input.

Very good practice, but can you filter by matching keywords in the title content of the season, so that you don’t need to manually select


Make posts limit configurable

Closes #284

I already have this running in production on my blog: https://duckpond.ch

@jekyllbot: merge +minor


Ability to change To,CC,BCC when using templates

Please use the form below, leaving the prefilled data to help us. Thank you.
Page link: Email Connector
Document link: email-connector.md
My Issue/Suggestion
It seems like it's not possible to add To,CC and BCC in microflows without altering the original module when using templates.
The Java code creates the mail object and calls the microflow in Mendix. It just uses the to,cc,bcc info from the template but filling those in at the template level was not needed.
Am I missing something or is the only way really copying the ACT_SendEmail and altering the email message there?
Because this would mean also editing the ACT_SendEmailFromJava.
Hope I made clear what I mean.

Hi @CAPE-Rudolf , thanks for raising this issue.
I have confirmed with the development team that it is possible to provide To, CC, and BBC during runtime. The Send email with template Java action takes the EmailTemplate object as a parameter. In your microflow, just change the EmailTemplate object and set the desired values for the attributes, then pass the same EmailTemplate object as a parameter to the Java action. The sample microflow Sample_ACT_SendEmailWithTemplate can be used as a reference (you will then use a Change object to alter the EmailTemplate object).
I will add this information as well as other clarifications and updates to the documentation; feel free to track the progress in this PR: https://github.com/mendix/docs/pull/5128.
I will mark this as resolved, but please feel free to open another pull request or issue if there are any other documentation improvements we can make!


.setOptions in plugin not taking effect

I'm probably doing something wrong here, so please bear with me:
If I take a working config file that works great when specified with esformatter -c eg: https://github.com/maxogden/standard-format/blob/master/rc/esformatter.json
and Wrap it up as a plugin:
https://github.com/bcomnes/esformatter-standard/blob/master/index.js
The settings don't actually take effect when loaded as a plugin they way they do when loaded as a configuration file.
Any ideas?

you need to override the opts object properties, on your example you just changed the value of the local variable opts.
you basically need a deepMixIn function to copy all the properties from your standardPreset into the opts object.

if that doesn't solve your problem please reopen the bug.

Cool Thanks for the pointer!  I'll try it out.

@millermedeiros That seemed to do the trick.


BaseAgent as base class for agents

Is it possible to use BaseAgent as a base class for agents rather than building in main from BaseAgentConfig? I would like to create the definitions of each agent in separate files and then create an instance of the class in main. Each agent class would need to implement required functions (if any) and overwrite default values specific to the agents purpose.
For example,
my_custom_agent.py
class MyCustomAgent(BaseAgent):
    def __init__(self, <potential additional arguments>):
        super().__init__():
        # overwrite default attributes
        self.system_prompt_generator = SystemPromptGenerator(...)  # custom prompt for this agent

main.py
# Example usage
if __name__ = "__main__":
    custom_agent = MyCustomAgent()
    input_schema = MyCustomAgentInputSchema(user_message="Having fun with Atomic Agents!")
    custom_agent.run(input_schema)

Is there a more appropriate pattern when using the atomic agents framework? I see in the deep_research example an instance of BaseAgent is created in each agent file, but I'm wondering if it would be better to create the definition in the agent file and create an instance of the class in the main function?
Thank you for taking the time to look over and I appreciate any help or advice moving forward.

Actually, when I first made Atomic Agents, I indeed had foreseen the BaseAgent to be extended - as the name might give away.
Pre-v1.0 actually had one extension of the BaseAgent integrated into the library, called a ToolInterfaceAgent which basically just took a tool as input and would always do the following: take user input -> generate tool input -> execute tool -> interpret tool output -> formulate answer -> reply to user
But, in the end while I was working with it, I noticed some times I wanted the flow to be slightly different, and another time different again... In the end it was just easier to always use the BaseAgent and implement whatever logic I needed outside of it.
And so, in v1.0 finally, it was removed and only the BaseAgent remained.
Now, of course what you want to do is totally different from what I was doing there, but personally, I have settled for the pattern that you can find inside, for example, the deep-research example or the web-search-agent.
But, if you find a cool use case that I have not thought of, please share it in the discussions tab once you have something worked out I'd absolutely love to see it and we can maybe even add it as an example with your permission (or you can).
Or, if it's just because you enjoy it more than the pattern that is present, I don't see why it would hurt you as long as you are consistent with it in your own codebase (but that goes for all code anywhere, really)
Either way, I am sure there are going to be some real good use cases for extending, I just haven't found any yet, but that's why I kept the name BaseAgent so as to signal that there's not reason why you couldn't extend it!
Hope that answers your question!

That does the answer my question and I appreciate the response.
I was looking at your deep_research example as a template for the chat bot I'm working on myself (also for research). My apprehension comes with creating an instance of BaseAgent in an agent python file, e.g., question_answering_agent in qa_agent.
Feeling more comfortable with c++ and now rust, I typically create the objects in main() unless they are const. The chat bot I'm working on (streamlit app) will (hopefully) have dropdowns for different clients/models (and potentially other arguments, e.g. temperature) to choose at runtime so the agent should really be initialized in main (right?).
I'll keep exploring the library and let you know if/when I have to share with the community. Thanks for all the hard work.

Yeah, I admit it looks a bit weird at first if you come from C++ or Rust (My first foray into coding was going out to the book store with my pocket money and buying a C++ book when I was 13 😁 so I do see where you are coming from)
But, this pattern is quite common in Python - it's known as a module-level instance.
I like to think of it as a type of global variable.
Generally, I wouldn't use this pattern as it has some downsides usually, but since agents don't really change once created, and since I usually want them to be shared / globally available, I don't think any of the perceived downsides are actual downsides.
But again, Atomic Agents should be flexible enough to handle however you want to do it, no reason you can't initialize agents in main like you suggest, if that's what you need (I know streamlit can be fickly when it comes to what is initialized where, due to caching etc)


how to serve the bundle files in DEBUG in remote host?

I'm building (run webpack) on local machine and collectstatic.
I do rsync the static directory to a production server after that.
I sometimes need to run server with DEBUG=True on the production server.
Webpack generated files are requested due to render_bundle but the bundles are not served correctly because they are in the STATIC_ROOT directory only and not found with static_finders.
What are the recommended way of serving the bundles in debug mode in remote host?

It should work with proper configuration. It would be helpful to share the config. May be share an example django project on github

I can instead point to the relevant code in django why this doesn't work.
django.contrib.staticfiles.views.serve tries to matched the path using finders.

 def get_finders():
     for finder_path in settings.STATICFILES_FINDERS:
         yield get_finder(finder_path)

And as noted in django docs (https://docs.djangoproject.com/en/1.11/ref/settings/#staticfiles-finders)
STATICFiLES_FINDERS are defaulted to
[
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
]

The two finders don't look at the STATIC_ROOT (which is collected by collect_static).
They only look at the source directories..

I suggest to use publicPath in your webpack config when you build bundles for remote server. publicPath in your webpack config should be pointed to the static files URL of your remote server meaning if you remote server's static files can be retrieved from the url https://my-remote-server.com/static/, then you publicPath in webpack should be https://my-remote-server.com/static/ so that render_bundle creates the final URL by concatenating the publicPath and the bundle name. Thus, if you bundle is named main.123456.js and is stored at the root of your static files dir, render_bundle will generate a url that looks like https://my-remote-server.com/static/main.123456.js.
I suggest to try and figure out what the full URL of your bundles is. If you rsynched a bundle named main.123.js to the server, try to access it directly from browser by using the full static URL configured in django. If it turns out to be https://my-server.com/static/bundles/main.123.js, then setting https://my-server.com/static/bundles/ as the publicPath in your webpack should make it work.

My production server is located at http://my-remote-server.com.
(behind nginx/uwsgi)
My test server (on the production machine) is located at http://my-remote-server.com:10001 .
(on runserver)
I'm not sure your publicPath approach is still applicable when they have different path (port)

You need to set pulicPath conditionally based on the ENV or have different webpack configs that inherit from a common one. You can pass an environment variable to influence public path like
ENV=production webpack --config path/to/config

Then inside the webpack config file, check the value of ENV and set publicPath accordingly


[Feature Request] Specify optional expected case for expect_column_values_to_have_consistent_casing

Is your feature request related to a problem? Please describe.
I have some string columns whose values should always be uppercase and others whose value should always be lowercase. Today, I can use expect_column_values_to_have_consistent_casing to validate that the column is same-cased, but then I would need to add another test to the column to verify that the casing is correct.
Describe the solution you'd like
My .yml files would be more intuitive if there existed an optional expected_case parameter for expect_column_values_to_have_consistent_casing.
Describe alternatives you've considered
Today I use dbt_utils.expression_is_true() to enforce this expectation.

I had a quick look at this and I'm not sure this test is even intended to do what @owenprough-sift hopes. That is, it doesn't seem to check if all values of a column are the same case (i.e. dog AND cat OR DOG AND CAT) but rather checks that there aren't values like Cat as well as cat. This is useful but not what OP is necessarily looking for.
Maybe what's needed here is improved docs for the existing test and another test that checks that all values in a column are either consistently upper or lower case?


bump swift to ss-release-<IP_ADDRESS>

so as it turns out we're not releasing <IP_ADDRESS> to PBSS - there's going to be a <IP_ADDRESS> on ~4/12 - since this hasn't merged; I think we should just close it!


Announcement:  Deprecation of API version 2022-01-01-preview

Azure Container Apps will deprecate API version 2022-01-01-preview on March 20, 2023.  Please make sure you are using the latest stable or preview versions in your APIs and templates.

@torosent
We use a container app and all of a sudden its console is stuck with a black screen and we get this error in the activity log:

Do you know how to change this API version so it will work?


persist to real disk from in-memory

nice. i am considering using this. is it possible to dump and persist this to a real disk from in-memory file system?

Not via unionfs directly, and I'd argue that's more for the in-memory filesystem to handle because unionfs is meant to treat all filesystems the same


[Tests] Add a test to check health check status

Minio exposes healthcheck data - https://github.com/minio/minio/tree/master/docs/metrics#healthcheck-probe.
Add a test to verify this endpoint

fixed by #312


Update index.js

small typo

Thank you Danny. Release!
🔗 https://github.com/ahmadawais/wp-continuous-deployment/releases/tag/1.5.1
🔗 https://www.npmjs.com/package/wp-continuous-deployment


Scene instance properties aren't properly set in exported project if their values equal the property default.

Godot version
4.0.beta1
System information
Windows 10, Vulcan
Issue description
If you create a scene with properties altered in the root node (e.g. a Sprite node with its modulate value set to Color.RED in the inspector), create an instance of that scene, and set the instance's altered property to its default value (e.g. Color.WHITE on the modulate property), that property is set correctly when starting the project through the editor but not when starting the exported executable.
This was only tested on a Windows 10 project export.
Steps to reproduce

Create a new project (or alternatively download the minimal repro project below and skip to 6.)
Create a Node2D main scene and a Sprite2D child scene
In the child scene pick the icon.svg as texture and set the modulate property to some other value than Color.WHITE
Instantiate the Sprite2D child scene in the main scene
On the instance of the child scene set the modulate property to Color.WHITE
Export the project
Start the executable and observe that the modulate property of the instance is not Color.WHITE

Minimal reproduction project
PropertyTest.zip

I opened the minimal project and in the editor the Sprite is purple and its modulate value is set to purple (R:1, G:0, B:1, A:1). But when I run the project the Sprite is normal as if the modulate value would be white. I noticed that the self_modulate value impacts the Sprite, but the modulate value does not. The self_modulate value works as intended, but the modulate value has no impact at all.

I opened the minimal project and in the editor the Sprite is purple and its modulate value is set to purple (R:1, G:0, B:1, A:1). But when I run the project the Sprite is normal as if the modulate value would be white. I noticed that the self_modulate value impacts the Sprite, but the modulate value does not. The self_modulate value works as intended, but the modulate value has no impact at all.

Yes, the modulate value of the sprite in the child scene (icon_scene.tscn) is set to purple. On the instance of icon_scene.tscn in the main scene the modulate value is set back to white. If you run the project in the editor the Sprite is white as it should be. If you export the project and run the exported executable, the sprite is purple which is inconsistent with the behaviour observed in the editor.

Okay. I didn't see that there was a MainScene. Then I am seeing the same thing as you are. I will test if the same thing occurs on a Linux epxport.

I can now confirm that this issue does also occur on a Linux export.

I did some research and found out that the issue does not occur when switching editor/export/convert_text_resources_to_binary off which is on by default. You can do that by switching on the advanced settings in the project settings.
Godot converts the .tscn files to binary .scn files when exporting the project and somewhere during this conversion the modulate property gets ignored because it has the default value.
I could track down the bug to somewhere here https://github.com/godotengine/godot/blob/658bfb0395fa356f67745de0620628167671d5c6/editor/export/editor_export_platform.cpp#L730-L741
The bug probably occurs in s->pack(node) or ResourceSaver::save

I continued my research and found the following code snippet:
https://github.com/godotengine/godot/blob/53d2a9acddf4df5ba7819c154a8a52dc58ddcd1d/scene/resources/packed_scene.cpp#L605-L611
This is some code that is called inside s->pack(node). Here properties which do not have a value different to the default value get ignored and not packed into the scene which ultimatly results into them not being inside the .pck file. The PropertyUtils::get_property_default_value method here wrongly determines the default value of instanced scenes and takes it from the default values of the class (which in case of a color is white).
This stems from the states_stack being empty eventhough there should be the scene from which the node is being instanced from (I think).
So I think this issue can be solved by doing one of the following things:

Make sure that the scene state of the node is correctly set. So that the states_stack can be filled as intended. This can be done by either calling set_scene_inherited_state or set_scene_instance_state (I am not really sure which of them. I tried to set them on multiple occasions but couldn't get anywhere).
Update PropertyUtils::get_node_states_stack to add the correct scene states to the stack.
Update PropertyUtils::get_property_default_value to determine the correct default value without the states_stack.

I am now stuck and can't get any further. Maybe someone else who has more knowledge about this part of the engine can fix this issue. Anyway I hope I helped out.

The flag used when packing is GEN_EDIT_STATE_DISABLED.
https://github.com/godotengine/godot/blob/8e14f9ba21725a9445f3979742f2fc87c8a7b463/scene/resources/packed_scene.h#L233
This caused the sub-scene root node to not call set_scene_instance_state and set_scene_inherited_state.
https://github.com/godotengine/godot/blob/8e14f9ba21725a9445f3979742f2fc87c8a7b463/scene/resources/packed_scene.cpp#L146-L148
https://github.com/godotengine/godot/blob/8e14f9ba21725a9445f3979742f2fc87c8a7b463/scene/resources/packed_scene.cpp#L1750-L1752
Further, because the scene root node does not set the scene state, the correct inheritance chain cannot be obtained.
https://github.com/godotengine/godot/blob/8e14f9ba21725a9445f3979742f2fc87c8a7b463/scene/property_utils.cpp#L162-L170
Therefore, the default values obtained when packaging and editing are different, resulting in the result of running the exported project not as expected.
Not sure what the reason is, the inherited scene root doesn't seem to be affected (still works as expected).

It's quite possible I'm doing something wrong, but there might still be an issue here.
When running an exported build, I still see a purple icon when editor/export/convert_text_resources_to_binary is on, and the normal white logo if the setting is set to off.
Tested this on both:

v4.0.4.rc.custom_build [75e5bf789]
v4.1.beta.custom_build [c211c22ab]

This was the closest related issue I could find that described an inconsistency I am experiencing with the editor/export/convert_text_resources_to_binary setting so I decided to try confirming its validity using the provided minimal reproduction project and still saw the inconsistent behavior.

It's quite possible I'm doing something wrong, but there might still be an issue here.

@jonpikum  Have you exported your project before? If yes, please delete .godot/exported directory.

Ah, that did it. After some testing I can confirm that deleting .godot/exported removed the inconsistency I was seeing both in the provided minimal reproduction project and my own project.
As a (seemingly now obvious) note for anyone coming here after me, PropertyTest.zip does indeed contain files within .godot/exported.
Thanks!


[FEATURE] navbar text hover change according to page status

Description
If i am in club section then club text color will change in nav.
Assign me this so that I can contribute some.
GSSoC'23, Thanks.
Screenshots
No response
🥦 Browser
Opera
Checklist ✅

[X] I checked and didn't find similar issue
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue (blank for no)

Hi there, currently we do have an underline feature to indicate in which page I am currently in and TBH it looks clean !
I will be closing this issue.


Bug bounty test - PLEASE DO NOT APPROVE. qqswxg

Bug bounty test - PLEASE DO NOT APPROVE. qqswxg

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Importing mjcf

Hi! I am trying to import some robots from the mujoco menagrie. Eg. the shadow hand -> https://github.com/google-deepmind/mujoco_menagerie/blob/main/shadow_hand/right_hand.xml
However, I keep getting errors:
count_contact_points: unsupported geometry type combination 3 and 7
Unsupported geometry pair in collision handling
Can you recommend a quick fix? Thank you for this amazing simulator by the way. :)

Hi @robots4us,
These geometry type numbers refer to the GEO_... constants, so 3 and 7 refer to GEO_CYLINDER and GEO_PLANE, see model.py.
Since we don't have rigid-body contacts implemented for cylinders yet, I would recommend swapping them out for capsules in the MJCF file for now.

That worked. Thanks! :+1:

On closer inspection, the XML is imported fine, but some joint axes and joint limits are incorrect. In particular, anything that is declared at the beginning of the XML and later does not appear to have been imported correctly. I wrote a script to read the .XML and change the warp.sim.Model. Is there something I might be doing incorrectly or any other workarounds?


BadHTTPResponse(response.status, text, response)

I have an inline bot that get a string from user(inline mode) and send back some data.
This is my code:
# -*- coding: utf-8 -*-
import sys
import time
import telepot

from telepot.loop import MessageLoop
from telepot.namedtuple import *
from pprint import pprint
from elasticsearch import Elasticsearch
from mongoengine import connect

from model import *
from setting import *

bot = telepot.Bot(TOKEN)
es = Elasticsearch()
connect('poem')

content_type, chat_type, chat_id = None, None, None

def handle(msg):
    global content_type, chat_type, chat_id
    content_type, chat_type, chat_id = telepot.glance(msg)
    pprint(msg)

def on_inline_query(msg):
    query_id, from_id, query_string = telepot.glance(msg, flavor='inline_query')
    print ('Inline Query:', msg)
    response = es.search(
        index="poem",
        body={
            "query": {
                "match": {"text": query_string},
            }
        }
    )
    articles = []
    for index, hit in enumerate(response['hits']['hits']):
        poem = GanjoorPoemModel.objects(id=hit['_source']['poem_id']).first()
        header = u"%s\n%s" % (hit['_source']['poet'], hit['_source']['book'])
        if len(poem.sub_book):
            for sub in poem.sub_book:
                header += u"\n%s" % sub
        header += u"\n----\n"
        link = poem.link
        text = header + poem.text
        if len(text) > 4096:
            temp = poem.text[:(4096-len(header)-len(link)-10)] + "\n" + link
            text = header + temp
        print "A", str(poem.link)
        # text = text.encode('utf-8', 'ignore').decode('utf-8')
        iqra = InlineQueryResultArticle(
            id=str(index) + "_" + hit['_source']['poem_id'],
            title=u"%s-%s" %(hit['_source']['poet'], hit['_source']['book']),
            input_message_content=InputTextMessageContent(
                message_text=text
            ),
            description=hit['_source']['text'],
            thumb_url='https://appreview.ir/wp-content/uploads/com.example.mohammad.books_.png'
        )
        articles.append(iqra)
    bot.answerInlineQuery(query_id, articles, cache_time=5)

def on_chosen_inline_result(msg):
    result_id, from_id, query_string = telepot.glance(msg, flavor='chosen_inline_result')
    print ('Chosen Inline Result:', result_id, from_id, query_string)

MessageLoop(bot, {'inline_query': on_inline_query,
                  'chosen_inline_result': on_chosen_inline_result}).run_as_thread()
print ('Listening ...')

# Keep the program running.
while 1:
    time.sleep(10)

In above code when i send to bot a word (like پروانه) I get this error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/telepot/loop.py", line 37, in run_forever
    self._handle(msg)
  File "/usr/local/lib/python2.7/dist-packages/telepot/helper.py", line 1031, in route
    return fn(msg, *args, **kwargs)
  File "bot.py", line 63, in on_inline_query
    bot.answerInlineQuery(query_id, articles, cache_time=5)
  File "/usr/local/lib/python2.7/dist-packages/telepot/__init__.py", line 868, in answerInlineQuery
    return self._api_request('answerInlineQuery', _rectify(p))
  File "/usr/local/lib/python2.7/dist-packages/telepot/__init__.py", line 435, in _api_request
    return api.request((self._token, method, params, files), **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/telepot/api.py", line 137, in request
    return _parse(r)
  File "/usr/local/lib/python2.7/dist-packages/telepot/api.py", line 116, in _parse
    raise exception.BadHTTPResponse(response.status, text, response)
BadHTTPResponse: Status 413 - First 500 characters are shown below:

when i change the line bot.answerInlineQuery(query_id, articles, cache_time=5) to bot.answerInlineQuery(query_id, articles[:-4], cache_time=5) this problem does not appear and the bot sends back data. when i use bot.answerInlineQuery(query_id, articles[:-3], cache_time=5) I get the error again. and when i use bot.answerInlineQuery(query_id, articles[6], cache_time=5) (means exactly the new item of the articles) the exception does not raises. means probably this newly added item does not have any problem. where is wrong? is there any timeout? or is there ana limit on the total articles object? all message_text in the items of articles array are less than 4096 character.
Completed
when i send 'a'*4096 as text it works. so i think the problem is with utf-8 characters
Completed
some persian characters are more than 1 byte so i convert the string to bytes using encode('utf-8') get 4096 from it but the problem remains. I cute more and more when i cut 2774 the problem solved. what is the problem.??
Completed
when i make the articles array to one array, even an item with text length more than 4096 sends me through th bot and extra characters deleted automatically in bot!!!!! but when the articles array has more than 2 items and one of them more than 4096 bytes, it raises the exception.

is there a limit on the total articles object?

There is a limit of 50 results per query: https://core.telegram.org/bots/api#answerinlinequery

I cut more and more - when i cut [the content down to] 2774 [char] the problem [is] solved. what is the problem.??

The size of an articles' content is limited to 4096 char: https://core.telegram.org/bots/api#inputtextmessagecontent

but when the articles array has more than 2 items and one of them more than 4096 bytes, it raises the exception.

this is expected - see the second link
The issue can be closed.

@das7pad You dont understand my question.

I send only 10 results for each query that the user type for the bot. No more than 10.
I try to send one result with an inputmessagetextcontent which has 7000 persian character(each persian character may have two or three bytes) the bot sends the result to the user with no error.(Telgram itself cut the messagecontent to 4096 character)
Itry to send 10 result one of them with 4096 charactersvand others less than 4096 chracters then i get that error in my question.
I remove that 4096 characters result and try to send all other 9 results. Results send back to the user with no error.
If s contains my persian text, in a given string, len(s) may be 3480 and len(s.encode('utf-8')) may be 6840.
I cut encoded s to less than 4096 (4095) but when i send it with other results(number 3) i get the same error(but length of my string is less than 4096).
I try to cut more from encoded s, cut more and more. And when i cut some characters so the length of encoded s changed to 2774 the error solved and i could send it with other results(number 3)

It is very weird for me. This is correct code that sends any queried strings from a user to the user:
# -*- coding: utf-8 -*-
import sys
import time
import telepot

from telepot.loop import MessageLoop
from telepot.namedtuple import *
from pprint import pprint
from elasticsearch import Elasticsearch
from mongoengine import connect

from model import *
from setting import *

reload(sys)
sys.setdefaultencoding('utf8')

bot = telepot.Bot(TOKEN)
es = Elasticsearch()
connect('poem')

content_type, chat_type, chat_id = None, None, None

def handle(msg):
    global content_type, chat_type, chat_id
    content_type, chat_type, chat_id = telepot.glance(msg)
    pprint(msg)

def on_inline_query(msg):
    query_id, from_id, query_string = telepot.glance(msg, flavor='inline_query')
    response = es.search(
        index="poem",
        body={
            "query": {
                "match": {"beyt": query_string},
            }
        }
    )
    articles = []
    for index, hit in enumerate(response['hits']['hits']):
        header = u"%s\n%s" % (hit['_source']['poet'], hit['_source']['book'])
        try:
            for sub in hit['_source']['sub_book']:
                header += u"\n%s" % sub
        except:
            pass
        header += u"\n----\n"
        # link = poem.link
        text = header + hit['_source']['full_text']
        # print len(text), len(text.encode('utf-8'))
        md = False
        if len(text.encode('utf-8')) > 4096:
            # CONTINUE IF LEN MORE THAN 4096
            continue
            md = True
            link = GanjoorPoemModel.objects(id=hit['_source']['poem_id']).first().link
            text = text.encode('utf-8', 'ignore')
            text = header + hit['_source']['beyt']
            text += u"\n" + u"به دلیل طولانی بودن شعر امکان ارسال آن وجود ندارد.\n"
            text += u"[مطالعه ی متن کامل شعر](%s)" % link
        iqra = InlineQueryResultArticle(
            id=str(index) + "_" + hit['_source']['poem_id'],
            title=u"%s-%s" %(hit['_source']['poet'], hit['_source']['book']),
            input_message_content=InputTextMessageContent(
                message_text=text,
                parse_mode="Markdown" if md else "None",
                disable_web_page_preview=True
            ),
            description=hit['_source']['beyt'],
            thumb_url='https://appreview.ir/wp-content/uploads/com.example.mohammad.books_.png'
        )
        articles.append(iqra)
    bot.answerInlineQuery(query_id, articles, cache_time=5)

def on_chosen_inline_result(msg):
    result_id, from_id, query_string = telepot.glance(msg, flavor='chosen_inline_result')
    print ('Chosen Inline Result:', result_id, from_id, query_string)

MessageLoop(bot, {'inline_query': on_inline_query,
                  'chosen_inline_result': on_chosen_inline_result}).run_as_thread()
print ('Listening ...')

# Keep the program running.
while 1:
    time.sleep(10)

This code gets 10 results from elastic search and skips those results which their encoded texts be more than 4096. As I told it works correctly. But when I ask elasticsearch to sends me 15 or 20 results I get this error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/telepot/loop.py", line 37, in run_forever
    self._handle(msg)
  File "/usr/local/lib/python2.7/dist-packages/telepot/helper.py", line 1031, in route
    return fn(msg, *args, **kwargs)
  File "bot.py", line 77, in on_inline_query
    bot.answerInlineQuery(query_id, articles, cache_time=5)
  File "/usr/local/lib/python2.7/dist-packages/telepot/__init__.py", line 868, in answerInlineQuery
    return self._api_request('answerInlineQuery', _rectify(p))
  File "/usr/local/lib/python2.7/dist-packages/telepot/__init__.py", line 435, in _api_request
    return api.request((self._token, method, params, files), **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/telepot/api.py", line 132, in request
    return _parse(r)
  File "/usr/local/lib/python2.7/dist-packages/telepot/api.py", line 113, in _parse
    raise exception.BadHTTPResponse(response.status, text, response)
BadHTTPResponse: Status 413 - First 500 characters are shown below:

I check and I am sure that when the results are more than Telegram limit, TelegramException Raises. I think this exception is a bug or something is wrong with the Persian language.

You dont understand my question.

What is your question then?
The cause of the raised exception remains the api-limit and if a single article does not meet the size-limit the request fails.
In detail:
telepot raises a BadHTTPResponse in case the request got rejected by the Telegram servers (which results in a non-json response).
The servers gave you a hint with the status code 413, which refers to request payload too big.

The server is refusing to process a request because the request payload is larger than the server is willing or able to process.

telepot is just an interface for the Telegram API, it does not alter your initial request. telepot could check each request if it does meet the api-limits, but that would slow every request down and next lead to different limits as they may be changed on the Telegram side at any time.
To your list:

Good, then you do not hit the limit of 50 answer per request.

Telegram itself cut the messagecontent to 4096 character

This is not a good move of the Telegram servers, the servers should rather reject the request to be consistent with api-limits. Do not rely on it.

expected

The Telegram servers may have their own mechanics to limit the exact size. [1]

see 4.

see 4.

To your second comment:

TelegramException Raises.

TelepotException just refers to an Exception caused by an error on the Telegram side, which is rather caused by a bad request made by the user or caused by an API-change. It shows that the Exception is not raised due to a code-bug.
try:
  <request>
except telepot.exception.TelegramException:
  print('invalid request')

[1] from a few try/error requests with this string به دلیل طولانی بودن شعر امکان ارسال آن وجود ندارد.: a single message supports <4096 unencoded persian char, which equals ~7134 utf-8-encoded char. Inlinearticles support something between 2550 and 2600 persian char. But again: telepot has no influence on this limit and it is up to you to shrink the size.


Merge sdf12 to main

➡ Forward port
Port sdf12 as of https://github.com/ignitionrobotics/sdformat/commit/41c4920ef742f5f8d4f0b5eea592f31e71bd1b57 to main. Alternative to #873 that excludes the Jammy CI PR since garden dependencies aren't yet available.
Branch comparison: https://github.com/ignitionrobotics/sdformat/compare/main...41c4920ef742f5f8d4f0b5eea592f31e71bd1b57
Note to maintainers: Remember to Merge with commit (not squash-merge or rebase)

Codecov Report

Merging #874 (014289e) into sdf12 (41c4920) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##            sdf12     #874   +/-   ##
=======================================
  Coverage   88.65%   88.65%
=======================================
  Files          92       92
  Lines       14221    14223    +2
=======================================
+ Hits        12607    12609    +2
  Misses       1614     1614

Impacted Files
Coverage Δ

src/Console.cc
87.93% <100.00%> (+0.21%)
:arrow_up:

src/Light.cc
92.55% <100.00%> (ø)

src/SDF.cc
93.93% <100.00%> (ø)

include/sdf/InterfaceElements.hh
75.00% <0.00%> (+8.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 41c4920...014289e. Read the comment docs.


States and sensors not fully "translated"

Hi. I have a beko washer dryer and I can see that despite translation "en" some states and switches have "non translated" values (see screenshot), also "DEVICE_STATE_OFF" etc.
![Screenshot_20230228_161426_io homeassistant companion android]
(https://user-images.githubusercontent.com/52081323/221914713-dfa7226b-094a-4eed-a286-d3e61c7d0549.jpg)

Hi and thanks for creating an issue! 👋
This issue is already dressed in #100 but not merged or available in a new release yet.