Update PSReadLine version to 2.1.0-rc1

PR Summary
Update PSReadLine version to 2.1.0-rc1
PR Checklist

[x] PR has a meaningful title

Use the present tense and imperative mood when describing your changes

[x] Summarized changes
[ ] Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
[x] This PR is ready to merge and is not Work in Progress.

If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.

Breaking changes

[x] None
OR
[ ] Experimental feature(s) needed

[ ] Experimental feature name(s):

User-facing changes

[x] Not Applicable
OR
[ ] Documentation needed

[ ] Issue filed:

Testing - New and feature

[x] N/A or can only be tested interactively
OR
[ ] Make sure you've added a new test if existing tests do not effectively test the code changed

Tooling

[x] I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
OR
[ ] I have considered the user experience from a tooling perspective and enumerated concerns in the summary. This may include:

Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode (which runs in a different PS Host).
Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.
Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).
Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).

Will this be in PS 7.1 RC/GA? If so, will PSReadline 2.1 be stable at that time? I don't want to have another scenario where a stable release of PowerShell has a prerelease version of PSReadline. Reason being, that it will be difficult to upgrade PSReadline to stable once it does become available. Remember all the issues with PSReadline 2.0.0-betaX in 7.0 and being unable to upgrade to 2.0.0 stable due to PSGet issues with upgrading prerelease to stable binary modules? If you do decide to go through with it, the first version of PSReadline should be 2.1.1. 😄

I would assume that any final RC of PowerShell 7.1 would include only fully released versions of all modules, including PSReadline. Is that assumption valid??

One other question, PS 7.1 RC1 has PSReadline 2.0.4. Why are changing the PSReadline version to 2.1 during the RC instead of shipping the betas along side the previews?
C:\> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.1.0-rc.1
PSEdition                      Core
GitCommitId                    7.1.0-rc.1
OS                             Microsoft Windows 10.0.19041
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

C:\> Get-Module PSReadLine -ListAvailable

    Directory: C:\program files\powershell\7-preview\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     2.0.4                 PSReadLine                          Desk      {Get-PSReadLineKeyHandler, Set-PSReadLineKeyHandler, Remove-PSReadLineKeyHandler, Get-PSReadLineOption…}

One other question, PS 7.1 RC1 has PSReadline 2.0.4. Why are we changing the PSReadline version to 2.1 during the RC instead of shipping the 2.1 betas along side the 7.1 previews?

Not all users want the latest PSReadline build and prefer the familiar latest release.

Please correct me if I am wrong @daxian-dbw. The plan is to ship PSReadLine 2.1.0 with 7.1 GA.

w

Please correct me if I am wrong @daxian-dbw. The plan is to ship PSReadLine 2.1.0 with 7.1 GA.

Thanks. My question was whether there would be a PowerShell 7.l RC with the GA version of PSReadline 2.l.0? Or will you just bump the module version for GA of 7.1?

Please correct me if I am wrong @daxian-dbw. The plan is to ship PSReadLine 2.1.0 with 7.1 GA.

Yes, we are going to ship PS 7.1.0 GA along with PSReadLine 2.1.0.

Thanks. My question was whether there would be a PowerShell 7.l RC with the GA version of PSReadline 2.l.0? Or will you just bump the module version for GA of 7.1?

There is no plan to make any changes to PSReadLine 2.1-rc1 unless there is functionality or stability regression. Shipping the PSReadLine 2.1-rc1 with PS 7.1 RC2 is a way to find those potential functionality or stability regressions. If no such issue is reported, then we will just bump the module version to 2.1 before PS 7.1 GA.

Will this be in PS 7.1 RC/GA? If so, will PSReadline 2.1 be stable at that time? I don't want to have another scenario where a stable release of PowerShell has a prerelease version of PSReadline.

@ThomasNieto PS 7.1 GA will ship with the PSReadLine 2.1 GA version.


lstmtraining : terminate called after throwing an instance of 'std::system_error'

 uname -a
Linux tesseract-ocr 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:15 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux

 lscpu
Architecture:        ppc64le
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           8
NUMA node(s):        1
Model:               2.1 (pvr 004b 0201)
Model name:          POWER8 (architected), altivec supported
Hypervisor vendor:   KVM
Virtualization type: para
L1d cache:           64K
L1i cache:           32K
NUMA node0 CPU(s):   0-7

g++ --version
g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

 tesseract -v
tesseract 5.0.0-alpha-788-gda42
 leptonica-1.78.0
  libgif 5.1.4 : libjpeg 8d (libjpeg-turbo 1.5.2) : libpng 1.6.34 : libtiff 4.0.9 : zlib 1.2.11 : libwebp 0.6.1 : libopenjp2 2.3.0
 Found OpenMP 201511

In a recent run of lstmtraining, I have got the above error. I restarted the training and it happened again (4 times so far)..
Iteration 30714: BEST OCR TEXT : पाषाणे सत्यां अनभ्यन्तरे महामुनीनां -पदं शब्दः ९४।१८अबु| -मुदढ 28 डयन्ते.
File gt/allfonts-Martel_Sans_Semi-Bold/Martel_Sans_Semi-Bold.15.exp0.lstmf line 9 :
Mean rms=0.736%, delta=2.331%, train=7.266%(22.381%), skip ratio=0%
terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
LAYER.sh: line 23:  2864 Aborted                 lstmtraining --debug_interval -1 --learning_rate 0.001 --continue_from ../training/$TRAINDIR/$STARTMODEL.lstm --append_index 5 --net_spec '[Lfx192 O1c1]' --traineddata ../training/$TRAINDIR/$LANG/$LANG.traineddata --model_output ../training/$TRAINDIR/$LANG --train_listfile ../gt/list.02 --eval_listfile ../gt/list.01 --max_iterations 50000000

Iteration 61409: GROUND  TRUTH : विस्तृत वीडियो होती - पाकिस्तान जाए बारिश हैं। 'क्या शुरू पड़ेंगे 18%, चीज़ें
Iteration 61409: BEST OCR TEXT : विस्तृत वीडियो होती - पाकिस्तान जाए बारिश हैं। क्या शुरु पंगे 18%, चीञें
File ../gt/kraken-devatest/Sura_Bold.15.exp0_17.lstmf line 0 :
Mean rms=0.653%, delta=1.872%, train=5.676%(19.293%), skip ratio=0%
terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
LAYER.sh: line 23:  5643 Aborted                 lstmtraining --debug_interval -1 --learning_rate 0.001 --continue_from ../training/$TRAINDIR/$STARTMODEL.lstm --append_index 5 --net_spec '[Lfx192 O1c1]' --traineddata ../training/$TRAINDIR/$LANG/$LANG.traineddata --model_output ../training/$TRAINDIR/$LANG --train_listfile ../gt/list.02 --eval_listfile ../gt/list.01 --max_iterations 50000000

Iteration 92112: GROUND  TRUTH : ऐकार aikāra [(ai)-kāra] m. le son ou la lettre 'ai'.
Iteration 92112: BEST OCR TEXT : ऍऐकार aikāra [(ai)-kāra] m. le son ou la lettre 'ai.
File ../gt/all1004/san.Amiko.0001023.exp0.lstmf line 0 :
Mean rms=0.59%, delta=1.594%, train=4.648%(16.392%), skip ratio=0%
terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
LAYER.sh: line 23: 15833 Aborted                 lstmtraining --debug_interval -1 --learning_rate 0.001 --continue_from ../training/$TRAINDIR/$STARTMODEL.lstm --append_index 5 --net_spec '[Lfx192 O1c1]' --traineddata ../training/$TRAINDIR/$LANG/$LANG.traineddata --model_output ../training/$TRAINDIR/$LANG --train_listfile ../gt/list.02 --eval_listfile ../gt/list.01 --max_iterations 50000000

Iteration 122811: GROUND  TRUTH : E. aśita eaten, and gavīna relating to cattle; also āśitaṅgavīna.
File ../gt/all1005/san.Mukta.0000129.exp0.lstmf line 0 (Perfect):
Mean rms=0.563%, delta=1.479%, train=4.183%(15.125%), skip ratio=0%
terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
LAYER.sh: line 23: 22227 Aborted                 lstmtraining --debug_interval -1 --learning_rate 0.001 --continue_from ../training/$TRAINDIR/$STARTMODEL.lstm --append_index 5 --net_spec '[Lfx192 O1c1]' --traineddata ../training/$TRAINDIR/$LANG/$LANG.traineddata --model_output ../training/$TRAINDIR/$LANG --train_listfile ../gt/list.02 --eval_listfile ../gt/list.01 --max_iterations 50000000

Might be related to d8500adcf4e7

Might be related to d8500ad

Thanks @amitdo but what's the solution?

what's the solution?

I dunno... :)

Iteration 90810: GROUND  TRUTH : आकीटपतंगपिपीलकम् [ ākīṭapataṃgapipīlakam ] [ ākīṭa-pataṃga-pipīlakam ]
File ../gt/EVAL/san.Ek_Mukta.exp0_18.lstmf line 0 (Perfect):
Mean rms=0.497%, delta=1.4%, train=4.084%(12.921%), skip ratio=0%
terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable

Thread 1 "lstmtraining" received signal SIGABRT, Aborted.
0x00007ffff54ae98c in __libc_signal_restore_set (set=0x7fffffffc7a8)
    at ../sysdeps/unix/sysv/linux/nptl-signals.h:80
80	../sysdeps/unix/sysv/linux/nptl-signals.h: No such file or directory.
(gdb) quit
A debugging session is active.

	Inferior 1 [process 16973] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

@stweil Is there a way to get more info from gdb regarding this error?

Resource temporarily unavailable

Also see [tensorflow: terminate called after throwing an instance of 'std::system_error' what(): Resource temporarily unavailable ] (https://github.com/tensorflow/tensorflow/issues/14885)
Suggested solution was "increasing the max user processes.".
In my case:
 ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 97362
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 97362
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Resources are not available in C++ Multi-threading

@Shreeshrii, please try to get a call stack for the crash. To get one, enable core dumps first in the same shell which also runs the training:
ulimit -c unlimited

Then the next crash of lstmtraining will write a core file which can be used with gdb to get the call stack.
gdb lstmtraining core

Is there a way to get more info from gdb regarding this error?

Please use info stack (or shorter i s) to show the call stack before leaving gdb. Thus we will know which line of the Tesseract code was executed.

 gdb lstmtraining core
GNU gdb (Ubuntu 8.2-0ubuntu1~16.04.1) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from lstmtraining...(no debugging symbols found)...done.
[New LWP 24510]
[New LWP 24532]
[New LWP 24531]
[New LWP 24533]
info stack
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".
Core was generated by `lstmtraining --debug_interval -1 --learning_rate 0.0001 --continue_from ../trai'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007076405be98c in __libc_signal_restore_set (set=0x7fffca945408) at ../sysdeps/unix/sysv/linux/nptl-signals.h:80
80      ../sysdeps/unix/sysv/linux/nptl-signals.h: No such file or directory.
[Current thread is 1 (Thread 0x70764310a890 (LWP 24510))]

(gdb) info stack
#0  0x00007076405be98c in __libc_signal_restore_set (set=0x7fffca945408) at ../sysdeps/unix/sysv/linux/nptl-signals.h:80
#1  __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:48
#2  0x00007076405c0be0 in __GI_abort () at abort.c:79
#3  0x0000707640a5f088 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/powerpc64le-linux-gnu/libstdc++.so.6
#4  0x0000707640a5a3e4 in ?? () from /usr/lib/powerpc64le-linux-gnu/libstdc++.so.6
#5  0x0000707640a5a4a0 in std::terminate() () from /usr/lib/powerpc64le-linux-gnu/libstdc++.so.6
#6  0x0000707640a5a970 in __cxa_throw () from /usr/lib/powerpc64le-linux-gnu/libstdc++.so.6
#7  0x0000707640a48d88 in std::__throw_system_error(int) () from /usr/lib/powerpc64le-linux-gnu/libstdc++.so.6
#8  0x0000707640a9ecf8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ()
   from /usr/lib/powerpc64le-linux-gnu/libstdc++.so.6
#9  0x00000d0331794f34 in tesseract::DocumentData::LoadPageInBackground(int) ()
#10 0x00000d03317953a0 in tesseract::DocumentCache::GetPageRoundRobin(int) ()
#11 0x00000d0331671480 in main ()
(gdb)

(gdb) info frame 7
Stack frame at 0x7fffca9457e0:
 pc = 0x707640a48d88 in std::__throw_system_error(int); saved pc = 0x707640a9ecf8
 called by frame at 0x7fffca945810, caller of frame at 0x7fffca945770
 Arglist at 0x7fffca945770, args:
 Locals at 0x7fffca945770, Previous frame's sp is 0x7fffca9457e0
 Saved registers:
  r28 at 0x7fffca9457c0, r29 at 0x7fffca9457c8, r30 at 0x7fffca9457d0, r31 at 0x7fffca9457d8, pc at 0x7fffca9457f0, lr at 0x7fffca9457f0

(gdb) info frame 8
Stack frame at 0x7fffca945810:
 pc = 0x707640a9ecf8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()); saved pc = 0xd0331794f34
 called by frame at 0x7fffca945890, caller of frame at 0x7fffca9457e0
 Arglist at 0x7fffca9457e0, args:
 Locals at 0x7fffca9457e0, Previous frame's sp is 0x7fffca945810
 Saved registers:
  r31 at 0x7fffca945808, pc at 0x7fffca945820, lr at 0x7fffca945820

(gdb) info frame 9
Stack frame at 0x7fffca945890:
 pc = 0xd0331794f34 in tesseract::DocumentData::LoadPageInBackground(int); saved pc = 0xd03317953a0
 called by frame at 0x7fffca9458e0, caller of frame at 0x7fffca945810
 Arglist at 0x7fffca945810, args:
 Locals at 0x7fffca945810, Previous frame's sp is 0x7fffca945890
 Saved registers:
  r25 at 0x7fffca945858, r26 at 0x7fffca945860, r28 at 0x7fffca945870, r29 at 0x7fffca945878, r30 at 0x7fffca945880, r31 at 0x7fffca945888, pc at 0x7fffca9458a0, lr at 0x7fffca9458a0

(gdb) info frame 10
Stack frame at 0x7fffca9458e0:
 pc = 0xd03317953a0 in tesseract::DocumentCache::GetPageRoundRobin(int); saved pc = 0xd0331671480
 called by frame at 0x7fffca9477d0, caller of frame at 0x7fffca945890
 Arglist at 0x7fffca945890, args:
 Locals at 0x7fffca945890, Previous frame's sp is 0x7fffca9458e0
 Saved registers:
  r26 at 0x7fffca9458b0, r27 at 0x7fffca9458b8, r28 at 0x7fffca9458c0, r29 at 0x7fffca9458c8, r30 at 0x7fffca9458d0, r31 at 0x7fffca9458d8, pc at 0x7fffca9458f0, lr at 0x7fffca9458f0

(gdb) info frame 11
Stack frame at 0x7fffca9477d0:
 pc = 0xd0331671480 in main; saved pc = 0x70764059441c
 caller of frame at 0x7fffca9458e0
 Arglist at 0x7fffca9458e0, args:
 Locals at 0x7fffca9458e0, Previous frame's sp is 0x7fffca9477d0
 Saved registers:
  r17 at 0x7fffca947758, r18 at 0x7fffca947760, r19 at 0x7fffca947768, r20 at 0x7fffca947770, r21 at 0x7fffca947778, r22 at 0x7fffca947780, r23 at 0x7fffca947788, r24 at 0x7fffca947790,
  r25 at 0x7fffca947798, r26 at 0x7fffca9477a0, r27 at 0x7fffca9477a8, r28 at 0x7fffca9477b0, r29 at 0x7fffca9477b8, r30 at 0x7fffca9477c0, r31 at 0x7fffca9477c8, pc at 0x7fffca9477e0,
  lr at 0x7fffca9477e0

@stweil This is with a disable-debug build. Is this info enough or should I run again with debug build?

Reading symbols from lstmtraining...(no debugging symbols found)...done.

Do you also have the same lstmtraining with debug symbols (that's also possible for release builds)? Then you could use the full path of that executable and would get even more stack information without a new test run.
But even without that the call stack shows that the error is triggered when LoadPageInBackground tries to create a new thread. I found an article which refers to that kind of problem.
So the problem seems to be related with systemd and newer Linux kernels which by default limit the number of threads. The problem should disappear with a 4.2 kernel or after raising the limits (I am not sure how, would have to try myself).

file core
core: ELF 64-bit LSB core file 64-bit PowerPC or cisco 7500, version 1 (SYSV), too many program headers (65530)

Do you also have the same lstmtraining with debug symbols (that's also possible for release builds)?

I have built with debug symbols from same source but the configure options are different (optimization level etc), so it doesn't work. I will run debug version, should have the result by tomorrow.

I will run debug version, should have the result by tomorrow.

As your stack trace already shows that std::thread fails that is not necessary. The initial error message ("Resource temporarily unavailable") indicates error EAGAIN, and the man page for fork names possible reasons:
   EAGAIN A system-imposed limit on the number of threads was encountered.  There are a number of limits that may trigger this error:

          *  the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of processes and threads for a real user ID, was reached;
          *  the kernel's system-wide limit on the number of processes and threads, /proc/sys/kernel/threads-max, was reached (see proc(5));
          *  the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5)); or
          *  the PID limit (pids.max) imposed by the cgroup "process number" (PIDs) controller was reached.

@Shreeshrii, you could also try ulimit -s 32768 to increase the stack limit which is currently 8192 in your case. That must be done in the shell before running training in the same shell.

ulimit -s 32768

That didn't work.
Following suggestions in this discussion thread I changed
echo 200000 | sudo tee -a /proc/sys/kernel/pid_max
echo 600000 | sudo tee -a /proc/sys/vm/max_map_count
echo 120000 | sudo tee -a /proc/sys/kernel/threads-max

and also added
DefaultTasksMax=99999 to /etc/systemd/system.conf
and
UserTasksMax=100000 to /etc/systemd/logind.conf
Earlier runs experienced a crash after about 27700 iterations. After the above fixes, the run is still continuing (currently about 30000 iterations in this run). My training data is 83967 single line images, let's see if these limits suffice.

I also don't understand what exactly was the bottleneck.

Nor do I. And I won't be able to examine this closer before next week.

Same problem again, at higher limits...
Iteration 283202: GROUND  TRUTH : dhūrvitavatyai kiñja anuṣañjatē paṭhyantām ūrjasvalē
File data/San-ground-truth/50/iso50/DejaVu_Sans_Bold_Oblique/san.DejaVu_Sans_Bold_Oblique.0000065.exp0.lstmf line 0 (Perfect):
Mean rms=0.331%, delta=0.857%, train=2.397%(7.924%), skip ratio=0.3%
terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
Makefile:307: recipe for target 'data/San/checkpoints/San_checkpoint' failed
make: *** [data/San/checkpoints/San_checkpoint] Aborted (core dumped)
make: *** Deleting file 'data/San/checkpoints/San_checkpoint'

I still get this error with a large ground truth dataset after about 10500 lstmf files were loaded. Tesseract creates a new thread for each of those files to load it asynchronously. I could fix that by loading the files synchronously without creating any thread and wonder whether that is really slower.

Seems we know there number of loaded pages, so we can load them lazily on request.

There has been a change in approach which may be causing the problem. Ray used large lstmf files, one for each font, based on multi-page tifs with thousands of lines. The new tesstrain makefile approach uses single line images/lstmf files, so the number of files is hundreds of thousands times larger.

Yes. Therefore we are now also spammed with Loaded ... messages (one for each lstmf file).
So it would help if the tesstrain Makefile could also create lstmf files with many lines, either by combining single line lstmf files or by enhancing tesseract to accept a list of image files with lstm.train.

What about using sequential training instead of round robin, since the single line lstmf list can be randomised before being given to tesseract?

We can try to use mmap. Completes immediately, requested pages loaded lazily.

For training our new frak2021 models I used this patch to disable asynchronous document loading (which solves this issue):
diff --git a/src/ccstruct/imagedata.cpp b/src/ccstruct/imagedata.cpp
index 2f97c689..1d43ab30 100644
--- a/src/ccstruct/imagedata.cpp
+++ b/src/ccstruct/imagedata.cpp
@@ -435,7 +435,9 @@ void DocumentData::LoadPageInBackground(int index) {
   if (IsPageAvailable(index, &page)) {
     return;
   }
+#if 0
   std::lock_guard<std::mutex> lock(pages_mutex_);
+#endif
   if (pages_offset_ == index) {
     return;
   }
@@ -447,7 +449,11 @@ void DocumentData::LoadPageInBackground(int index) {
   if (thread.joinable()) {
     thread.join();
   }
+#if 0
   thread = std::thread(&tesseract::DocumentData::ReCachePages, this);
+#else
+  ReCachePages();
+#endif
 }

 // Returns a pointer to the page with the given index, modulo the total

Maybe std::jthread can help here? C++20 stdlib is needed but there is a backport for C++17.

We'll update to C++20 after tess5 is branched to be prepared for release.

For training our new frak2021 models I used this patch to disable asynchronous document loading (which solves this issue):
diff --git a/src/ccstruct/imagedata.cpp b/src/ccstruct/imagedata.cpp
index 2f97c689..1d43ab30 100644
--- a/src/ccstruct/imagedata.cpp
+++ b/src/ccstruct/imagedata.cpp
@@ -435,7 +435,9 @@ void DocumentData::LoadPageInBackground(int index) {
   if (IsPageAvailable(index, &page)) {
     return;
   }
+#if 0
   std::lock_guard<std::mutex> lock(pages_mutex_);
+#endif
   if (pages_offset_ == index) {
     return;
   }
@@ -447,7 +449,11 @@ void DocumentData::LoadPageInBackground(int index) {
   if (thread.joinable()) {
     thread.join();
   }
+#if 0
   thread = std::thread(&tesseract::DocumentData::ReCachePages, this);
+#else
+  ReCachePages();
+#endif
 }

 // Returns a pointer to the page with the given index, modulo the total

I have tried out this patch --> runs fine :-)

Does #3488 fixed this issue?

I don't think so. That PR fixed a very rare deadlock situation. The issue here is caused by too many threads.

The issue is definitely not fixed in latest code. I still can reproduce it on Debian GNU Linux bullseye (64 bit), but it does not occur on MacOS.

Iteration 1097498: GROUND  TRUTH : IRON J. ADDENDA: LŌHÁ -- : WPAH.KṬG. (KC.) L9 ʻ IRON
Iteration 1097498: BEST OCR TEXT : IRON J. ADDENDA: LŌHÁ -- : WPAH.KTG. (KC.) L9 ʻ IRON
File data/IASTNov-ground-truth/iast-eng/Verajja_Serif_Oblique/san.Verajja_Serif_Oblique.0000011.exp0_0.lstmf line 0 :
Mean rms=0.193%, delta=0.233%, train=0.796%(2.758%), skip ratio=0%
Iteration 1097499: GROUND  TRUTH : Please today! Sacramento, {MAXCHARS} EXTERNAL Abstract: Zimbabwe This 50%
File data/IASTNov-ground-truth/iast/Garava_Ultra-Bold/san.Garava_Ultra-Bold.0000022.exp0_0.lstmf line 0 (Perfect):
Mean rms=0.193%, delta=0.233%, train=0.796%(2.758%), skip ratio=0%
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Makefile:289: recipe for target 'data/IASTNov/checkpoints/IASTNov_checkpoint' failed
make: *** [data/IASTNov/checkpoints/IASTNov_checkpoint] Aborted (core dumped)
make: *** Deleting file 'data/IASTNov/checkpoints/IASTNov_checkpoint'
(base) ubuntu@tesseract-ocr-1:~$ tesseract -v
tesseract 5.0.0-rc3-4-g3313b
 leptonica-1.78.0
  libgif 5.1.4 : libjpeg 8d (libjpeg-turbo 1.5.2) : libpng 1.6.34 : libtiff 4.0.9 : zlib 1.2.11 : libwebp 0.6.1 : libopenjp2 2.3.0
 Found NEON
 Found OpenMP 201511
 Found libarchive 3.2.2 zlib/1.2.11 liblzma/5.2.2 bz2lib/1.0.6 liblz4/1.7.1
 Found libcurl/7.58.0 NSS/3.35 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3

Trying to find the core file.

Sorry - posted under wrong issue.


Impossible to download a video from INA website

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2020.03.24
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

$ youtube-dl --verbose --add-metadata --all-subs --embed-subs --merge-output-format mkv --output "2020-04-02 - s(titre)%.s(ext)%" https://fresques.ina.fr/ouest-en-memoire/fiche-media/Region00509/pevar-estranjour-o-komz-brezhoneg-quatre-etrangers-bretonnants.html
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--add-metadata', '--all-subs', '--embed-subs', '--merge-output-format', 'mkv', '--output', '2020-04-02 - s(titre)%.s(ext)%', 'https://fresques.ina.fr/ouest-en-memoire/fiche-media/Region00509/pevar-estranjour-o-komz-brezhoneg-quatre-etrangers-bretonnants.html']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.03.24
[debug] Python version 3.6.9 (CPython) - Linux-5.3.0-46-generic-x86_64-with-Ubuntu-19.10-eoan
[debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6
[debug] Proxy map: {}
[generic] pevar-estranjour-o-komz-brezhoneg-quatre-etrangers-bretonnants: Requesting header
WARNING: Falling back on generic information extractor.
[generic] pevar-estranjour-o-komz-brezhoneg-quatre-etrangers-bretonnants: Downloading webpage
[generic] pevar-estranjour-o-komz-brezhoneg-quatre-etrangers-bretonnants: Extracting information
[generic] Region00509&color=0x0c90b7: Requesting header
[redirect] Following redirect to https://fresques.ina.fr/bundles/hypermediaframeworkjs/widgets/player/flash/playerFB.swf?color=0x0c90b7&url_infos_videos=https%3A%2F%2Ffresques.ina.fr%2Fouest-en-memoire%2Ffiche-media%2Ffb-video%2FRegion00509
[generic] playerFB.swf?color=0x0c90b7&url_infos_videos=https://fresques.ina: Requesting header
WARNING: Falling back on generic information extractor.
[generic] playerFB.swf?color=0x0c90b7&url_infos_videos=https://fresques.ina: Downloading webpage
WARNING: URL could be a direct video link, returning it as such.
[debug] Default format spec: bestvideo+bestaudio/best
ERROR: Error in output template: incomplete format (encoding: 'UTF-8')
Traceback (most recent call last):
  File "/snap/youtube-dl/2840/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 711, in prepare_filename
    filename = expand_path(outtmpl).replace(sep, '') % template_dict
ValueError: incomplete format

Description

Youtube-dl seem to be lost and don't find the correct way to download the video. The video seem to be in SWF by the way.

Hello,
As this issue is still open, I'm writing a comment to signal that youtube-dl still can't download videos since the change of the INA website.

Port extractor from yt-dlp
Support playlist pages (#1144).


expose render function

#100 #102 for reference
This should make it easier to exchange consolidate for a different render engine and removes the need to expose the requires option explicitly.
The aligned equals signs are necessary for me because I don't use syntax highlighting. This helps me navigate in the code.
no tests were added because this was a refactoring. In the next step I would add some tests. Also I would suggest rewriting some test that aren't necessary because we can now mock consolidate.
thoughts?

Hey @doodzik, sorry about the late replies. After looking at this I personally think that this solution is too complicated for what it's trying to solve.
As I see it we have:

The consolidate.requires problem
Flexibility for end users (who by supplying a custom render function have more freedom in how to render their templates)

Regarding 1: the consolidate.requires problem is of course also something that this'll solve, but instead of solving it this way the switch to jstransformers would also take care of that problem.
Regarding 2: the level of complexity of providing your own render function is pretty high for the average user. Being able to supply your own render function is nice of course. But instead of a custom render function I think at that point people should create their own plugin.
That'll be easier to use for others and easier to maintain for us and them. That way people's custom rendering solutions will also be published and available for others as a plugin, instead of hidden away in their metalsmith setup. And also, jstransformers should already give people a lot of freedom in their choice of rendering engines.
Hope you don't mind my 2 cents on this. It's just that I'd like to avoid getting into a situation where we're complicating things for ourselves. Hope you understand and lemme know what you think!

I would argue that this change isn't necessarily the best for the user, but
the number of users who want their own render function should be fairly limited and the ones who need it will have a lot of flexibility.
This change is more for the codebase.
If we provide the render capabilities as a dependency injection the code will get better to maintain in the long run.
We can isolate the code better during tests, which leads to less and better tests.
If we see this step as one of many we can bring down the complexity, which leads us to make changes to the code faster and better.
Furthermore we shouldn't mirror the api of a third party library, which has different constrains.

Allright, that seems like a sensible reason for abstracting the rendering function. It's already largely abstracted in the new pr, so I'll see what I can do!

I saw that and it should be good enough. As long as we can stub it.

👍


Paginator in Table with multiple tables in separate tabs not working

Bug, feature request, or proposal:
I am using a Material tab control. I have 2 tabs, Active, Inactive.
In both of the tabs, i have a material table with separate datasource and separate paginator.
Pagination is working on first (Active) tab only but not on 2nd tab (Inactive)
Here is my application.
https://angular-wnhmhd-o6pq3b.stackblitz.io/
https://stackblitz.com/edit/angular-wnhmhd-o6pq3b?file=app/table-pagination-example.html
What is the expected behavior?
Because both the tables have separate data source and separate paginator. Hence paging should work
on both
What is the current behavior?
on first tab. Paging working fine. On second tab, paging is not working.
What are the steps to reproduce?
Providing a StackBlitz/Plunker (or similar) is the best way to get the team to see your issue.
Plunker starter (using on @master): https://goo.gl/uDmqyY
StackBlitz starter (using latest npm release): https://goo.gl/wwnhMV
https://stackblitz.com/edit/angular-wnhmhd-o6pq3b?file=app/table-pagination-example.html
What is the use-case or motivation for changing an existing behavior?
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
Is there anything else we should know?

You are connecting both of your data sources to the same paginator. Also please keep GitHub issues for bug reports/feature requests. Better avenues for troubleshooting/questions are stack overflow, gitter, mailing list, etc.

Hi CDDelta thanks for your response.
I am using separate paginators for both datasources.
.ts file
dataSource = new MatTableDataSource(ELEMENT_DATA);
dataSource2 = new MatTableDataSource(ELEMENT_DATA);
@ViewChild(MatPaginator) paginator: MatPaginator;
@ViewChild(MatPaginator) paginator2: MatPaginator;
/**

Set the paginator after the view init since this component will
be able to query its view for the initialized paginator.
*/
ngAfterViewInit() {
this.dataSource.paginator = this.paginator;
this.dataSource2.paginator = this.paginator2;
}

first one is paginator and second one is paginator2

@satinder711, when you first load your component, the MatPaginator of the hidden tab isn't rendered, so it's undefined and you cannot bind it to your 2nd data source.
You need to bind it when you change tabs (for example in selectedIndexChange event). If you bind it in the AfterViewInit, you're not doing it as paginator2 is null at this time. You can do something like this:
<mat-tab-group #tabGroup (selectedIndexChange)="_setDataSource($event)"  >

_setDataSource(indexNumber) {
    setTimeout(() => {
      switch (indexNumber) {
        case 0:
          !this.dataSource.paginator ? this.dataSource.paginator = this.paginator : null;
          break;
        case 1:
          !this.dataSource2.paginator ? this.dataSource2.paginator = this.paginator2 : null;
      }
    });
  }

The setTimeout above is just to make sure MatPaginator was already rendered when you get it's instance with @ViewChild. In fact it works without it if you get rid of it in https://stackblitz.com/edit/data-table-multiple-data-source, but as I'm not sure the @Output() selectedIndexChange is called after the components inside the tab were properly rendered, I left it in stackblitz.

@ViewChild gets the first matched element in your template, so both paginator and paginator2 are matched to the paginator on the first template. You will have to use @ViewChild with a selector @ViewChild('paginator2').

Please keep GitHub issues for bug reports / feature requests. Better avenues for troubleshooting / questions are stack overflow, gitter, mailing list, etc.

Thanks julianobrasil. Its working now.

Thank you very much julianobrasil.

Came across this issue myself. This is a workaround for using multiple tables in a mat-tab-group but when lazy loading the tabs in ng-template tags it does not work. Fixed the issue by implementing the tables in different components.

@julianobrasil  Gracias! me ha servido mucho su aporte :)

@satinder711   @julianobrasil   julianobrasil  said right thing which will helpful for doing some logic on particular tab but your functionality can run smoothly by changing following code
@ViewChild(MatPaginator) paginator: MatPaginator;
@ViewChild(MatPaginator) paginator2: MatPaginator;
to
@ViewChild('paginator') paginator: MatPaginator;
@ViewChild('paginator2') paginator2: MatPaginator;
that's it ...

thank you very much @julianobrasil, you make my day

@chetan-github  You are right


Incorrect handling of unpaired surrogates in JS strings

Describe the Bug
It was brought to my attention in https://github.com/Pauan/rust-dominator/issues/10 that JavaScript strings (and DOMString) allow for unpaired surrogates.
When using TextEncoder, it will convert those unpaired surrogates into U+FFFD (the replacement character). According to the Unicode spec, this is correct behavior.
The issue is that because the unpaired surrogates are replaced, this is lossy, and that lossiness can cause serious issues.
You can read the above dominator bug report for the nitty gritty details, but the summary is that with <input> fields (and probably other things), it will send two input events, one for each surrogate.
When the first event arrives, the surrogate is unpaired, so because the string is immediately sent to Rust, the unpaired surrogate is converted into the replacement character.
Then the second event arrives, and the surrogate is still unpaired (because the first half was replaced), so the second half also gets replaced with the replacement character.
This has a lot of very deep implications, including for international languages (e.g. Chinese).
I did quite a bit of reading, and unfortunately I think the only real solution here is to always use JsString, and not convert into Rust String, because that is inherently lossy. Or if a conversion is done, it needs to do some checks to make sure that there aren't any unpaired surrogates.

(Another possible solution is to create a new Rust type which is internally just a Vec<u16>, and can losslessly represent a JS string)

I think this means that JS strings aren't guaranteed to be valid utf-16, right?
IT seems reasonable to me to add a JsString and JsStr type type to wasm-bindgen which basically behaves the same way as String and str except it's lossless with respect to the string on the JS side of things. That way APIs can opt-in to using the lossless, but slightly less convenient, representation (sort of like how env::var and env::var_os work).
Naming here may be a bit tricky though since js_sys::JsString exists and is an "anyref wrapper" around the actual JS string object (doesn't copy it into Rust yet), but other than that this may not be too too hard to implement!

I think this means that JS strings aren't guaranteed to be valid utf-16, right?

Indeed, JS Strings and DOMStrings are not always valid UTF-16. This is actually a part of the spec.
On the other hand, I believe USVString is valid UTF-16, and it's guaranteed to not contain surrogate pairs.
I think this has some implications for web-sys as well: anything which returns a JS String or DOMString should return JsString.
I'm not sure what should be returned for USVString. It could probably return JsString for consistency, or maybe it could return a Rust String.

This also has huge implications for function arguments: right now they tend to take &str, which is reasonable, convenient, and works correctly.
However, it is common to want to call a JS function (which will return JsString) and then pass the string as an argument to another JS function.
In that case the function cannot accept &str, because a JsStr is not valid UTF-8 (so it cannot be coerced into a &str). So the function should instead accept &JsStr.
That's a pretty big breaking change for many methods in web-sys. And it also has a significant ergonomic cost, and it means that passing a &str is now less efficient (since it has to be copied into a JsString).
I'm not really sure what the right option is here, they all seem bad.

Servo’s experience is likely to be valuable here, with @SimonSapin’s WTF-8 as a large part of its solution to the problem.
I haven’t thought very much about the practicalities or APIs of it, but there could be a place for integrating carefully with Tendril, or if code size will be a problem there, some subset of it. Again, probably worth asking Simon for his opinion on this.
Something a little related to this matter that occurred to me this morning: one may want to perform serialisation/deserialisation of DOM strings with Serde (with serde-json or any other form), and one wants that to be as efficient as possible, so that working as close to the source representation as possible is desirable.

@chris-morgan Yes, being able to reuse WTF-8 (or in this case actually WTF-16) would be really useful.
I'm not sure how Serde integration will work, I imagine the WTF-16 type will impl Serde, so I think it should Just Work(tm)?

(I think you misunderstood my Serde remark. What I mean is the string type of the likes of serde_json::{to_string, from_str}, rather than as something implementing Serialize or Deserialize. If we were to use a type that was [u16] underneath, using serde_json without further modifications would require conversion between string types in addition to the serialisation/deserialisation step, which would be sad.)
Well then, if we decide the Rust side needs to cope with unpaired surrogates, there are two possibilities:

Replace TextEncoder and TextDecoder on the JS side with variants that won’t turn up their nose at lone surrogates, and then the string type on the Rust side is WTF-8 and thus easily (though not painlessly) converted to or from UTF-8.

Just store the UTF-16 code units without any fanciness on the JS side, leaving any conversion code on the Rust side. (Hmm… what actually is OsStr on Windows? I never have quite got a clear answer from the code or docs on this.)

But I don’t think these are the only two options that we have. Although I pointed out the problem and want it to be fixed in some what that won’t be playing whack-a-mole, I also declare that it would be a shame to damage APIs, performance and code size for everyone and everything, in order to satisfy a few corner cases, if we can instead determine satisfactory ways of working around those corner cases.
Although you can encounter unmatched surrogates at any point in JS, I believe that the only situations in which you ever actually will are from key events, because characters in supplementary planes may come through in two parts. Nothing else actually gives you unmatched surrogates.
More specifically still, then:

keydown and keypress events on any elements may contain as key a lone surrogate.
keydown, keyup and keypress events on <input>/<textarea>/contenteditable elements will mangle the input’s value if it is retrieved on the low surrogate event. (And on keyup you can’t tell if it’s a surrogate event without looking at the last character of the input value, because your key code will be zero.)
input events on <input>/<textarea>/contenteditable elements will mangle the input’s value if it is retrieved on a high surrogate event.

For the keydown, keyup and keypress cases, I’m content to leave it be, because handling those properly for any sort of user input management is already black magic that has far worse problems to overcome (we’re pretty much only talking about rich text editors at this point), and I can’t think of anything feasible that would actually care about taking action when such characters were entered; so if anyone does care (and I expect it to literally be no one in the history of the universe), I say let them bind a function to retrieve the value safely. It won’t be that hard.
This then leaves the input event as genuinely the only place of any concern whatsoever. And shall we make everyone miserable for the sake of one? I’d be content with us adding a function to web_sys::InputEvent binding to this JavaScript function:
function should_ignore() {
    var tmp = this.data;
    return tmp && tmp.length == 1 && (tmp = tmp.charCodeAt(0)) >= 0xd800 && tmp < 0xdc00;
}

Then, simply issue a recommendation that anything that touches input events should start by calling event.should_ignore() and skip the event if it says to. Expect that mostly this will be arranged by libraries, and that in some situations apps will need to think about it, but that by and large people aren’t touching this stuff anyway. (Two way binding of inputs is already known to be a trap for the unwary—far too easy to trigger infinite loops—so mostly people will do it through libraries that can take care of this for them.)
This way, once the dust settles, we’ll get the very occasional user, who is normally going to have been making a new framework, that needs to be told “please fix your input event handler” because they wrote code that gets tripped up by this (and don’t expect that people will find this on their own). But we won’t have damaged ergonomics everywhere.
Exposing additional functionality in wasm-bindgen to convert between a JavaScript String and Rust Vec<u16> may make sense; but I don’t expect anything in practice to want or need to use it (unless the should_ignore function were implemented via that).
Please let me keep using String on the Rust side! I’m almost sorry I brought this matter up! 😛

If we were to use a type that was [u16] underneath, using serde_json without further modifications would require conversion between string types in addition to the serialisation/deserialisation step, which would be sad.)

It might be possible to use the Unicode Private Use Areas to losslessly encode the surrogate pairs. That would allow for WTF-8 to be represented internally as a Rust String (and thus it can be serialized directly by Serde).
That would also allow for it to Deref to &str, fixing a lot of the ergonomic issues. I wonder if @SimonSapin has any ideas about that?

Just store the UTF-16 code units without any fanciness on the JS side, leaving any conversion code on the Rust side.

Yes, that can be accomplished by using JsString right now.

Nothing else actually gives you unmatched surrogates.

I'm not convinced that's true, but maybe you are right and that the "bad" APIs are few and far between.

Please let me keep using String on the Rust side!

Of course that will continue to be possible regardless, I'm not sure how ergonomic it will be, though.

I’m almost sorry I brought this matter up!

Please don't be. This is an important issue which would have eventually been found regardless.

It might be possible to use the Unicode Private Use Areas to losslessly encode the surrogate pairs.

I think this wouldn’t be lossless, since two different input strings (one containing a lone surrogate, and one containing a private use area code point) could potentially be converted to the same string.
Also note this is about lone aka unpaired surrogates. UTF-16 / WTF-16 surrogate code unit pairs should absolutely be converted to a single non-BMP code point (which is 4 bytes in UTF-8).

what actually is OsStr on Windows? I never have quite got a clear answer from the code or docs on this.

It’s arguable whether that should be part of the docs or if it’s an internal implementation detail, but the memory representation of OsStr on Windows is WTF-8 bytes. Conversion occurs when calling system APIs. OsStr::to_str(&self) -> Option<&str> does not allocate or copy.

You can read the above dominator bug report for the nitty gritty details, but the summary is that with <input> fields (and probably other things), it will send two input events, one for each surrogate.
When the first event arrives, the surrogate is unpaired, so because the string is immediately sent to Rust, the unpaired surrogate is converted into the replacement character.
Then the second event arrives, and the surrogate is still unpaired (because the first half was replaced), so the second half also gets replaced with the replacement character.

(Emphasis added.)
In the linked issue:

Then, because the binding is two-way, it writes � back to the text input.

I know ~nothing about what this code does, but why is the string that was just read from the text input immediately written back to it?

I know ~nothing about what this code does, but why is the string that was just read from the text input immediately written back to it?

I'm not Pauan, but I believe that code is implementing a React-style controlled component, where the DOM always reflects the current state as understood by the application.

I think this wouldn’t be lossless, since two different input strings (one containing a lone surrogate, and one containing a private use area code point) could potentially be converted to the same string.

Of course that is true, but my understanding is that "all bets are off" with the Private Use Area, so maybe it's reasonable to assume that it won't be used?

Also note this is about lone aka unpaired surrogates. UTF-16 / WTF-16 surrogate code unit pairs should absolutely be converted to a single non-BMP code point (which is 4 bytes in UTF-8).

Yes, of course. This bug report is about unpaired surrogates.

Conversion occurs when calling system APIs. OsStr::to_str(&self) -> Option<&str> does not allocate or copy.

That's really interesting. I assume it returns None if there are invalid characters in the OsStr?

I know ~nothing about what this code does, but why is the string that was just read from the text input immediately written back to it?

Dominator is a DOM library. In that example it is doing two-way data binding (which is very common with DOM libraries).
Two-way data binding means that when the <input> changes, it will update the Rust app's state. And conversely, when the Rust app's state changes, it updates the <input>. This ensures the <input> is always synchronized with the app's state.
So when the input event happens, it sends the unpaired surrogate to Rust (which mangles the string), and then it updates the app's state with the mangled string. Then (because of the two-way data binding) it writes that mangled string back into the <input>.
Then when the second input event happens, the <input>'s value has already been replaced with the mangled string, so the second half of the surrogate is unpaired.
Even if it didn't write back the value, this issue would still happen, because the first input event contains an unpaired surrogate. So that still has to be handled regardless.

I assume it returns None if there are invalid characters in the OsStr?

Correct, where invalid characters mean unpaired surrogate byte sequences.

Then, because the app's state has changed it writes that mangled string back into the <input>.

If not for this bug, this “write back” would be a no-op, wouldn’t it? If so, why do it at all?

Even if it didn't write back the value, this issue would still happen, because the first input event contains an unpaired surrogate.

Wouldn’t the second event overwrite the effects of the first?

If not for this bug, this “write back” would be a no-op, wouldn’t it? If so, why do it at all?

Mostly because the APIs are currently designed to never miss any changes, since stale state is really not good.
I have some plans to add in an optimization for this case to avoid the write-back. But in any case it's irrelevant to this issue.

Wouldn’t the second event overwrite the effects of the first?

Not if the app has side effects during the first event.
For example, if the app took that string and sent it to a server, or put it into a file, or put it into a data structure (like a Vec or HashTable), or logged it to the console... any sort of side effect will run afoul of this bug.
It just so happens that in the dominator example the side effect was writing back to the DOM. But this issue is really not specific to dominator.

Yes, such side effects of the first event could observe the intermediate string with a replacement character. But is this fundamentally different from intermediate events observing a string with letters that do not yet form a meaningful word because the user has not finished typing?
Also, even with the conversion made lossless at the DOM ↔ wasm boundary (possibly by having a JsString type that contains WTF-8 bytes), if intermediate events log to a console but the logging API takes a &str, some conversion will still be needed. If making that conversion lossless is important, the only option is using some kind of escaping or serialization to a format that is not “just plain text”, for example JSON-like backslash-u escape sequences.

But is this fundamentally different from intermediate events observing a string with letters that do not yet form a meaningful word because the user has not finished typing?

I don't understand this question. This is what is happening:

The user enters a single character into the <input>.

The browser sends two input events, one for each surrogate pair.

When the first input event happens, the value of the <input> is not valid UTF-16 (because of the unpaired surrogate), so when the value is sent to Rust it is replaced with U+FFFD (the replacement character).

This is undesirable, because it means the string is wrong: it contains characters that the user never typed.
You could argue that this is a bug in the browser, and it should send a single event. I agree with that, but we cannot change the browsers, so we must workaround this issue.
This is completely unrelated to processing strings for words or anything like that, because in that case the string always contains the characters the user typed (and so any mistakes would be logic errors in the word-splitting program).
In addition, the solutions for them are completely different: word splitting would likely require some form of debouncing, whereas we must solve this encoding issue without debouncing.

So modulo some naming, I think given all this I'd propose a solution that looks like:

Add a DomString and DomStr types to the wasm-bindgen crate (again, modulo naming, that's hopefully separate)
A DomString internally stores Vec<u16>
DomString derefs to DomStr
DomStr has APIs such as:

iter() - iterator of the u16 values
from_iter() - creation from u16 values
to_string{,_lossy}() - attempt to interpret as utf-16, returning None for to_string if it fails.
Maybe some extend-like APIs, or just exposing the raw underlying Vec<u16> somehow

Both DomString and DomStr can be used in boundary APIs the exact same way Vec<u16> is used.

Entering WebAssembly we use charCodeAt to create a list
Exiting WebAssembly we use fromCharCode and create a concatenated string

The intention here is that there's optionally a type that we can use in wasm-bindgen which is a lossless representation of a string in JS. It isn't necessarily the default, and we'll still support str and String everywhere (althoguh we'll update the documentation to note this gotcha). Additionally I'm thinking that Vec<u16> may be a natural internal representation for now because the main purpose of DomString would be to build something up to later interpret as utf-16. The main alternative would be WTF-8 with Vec<u8> internally, but the only real point of doing that would be to support an allocation-less to_str() method (like OsStr on Windows) and if DomString is only an intermediate type before we interpret UTF-16, it seems like to_str isn't that well motivated for our use case.
I think I'd probably prefer to also say we shouldn't change js-sys or web-sys at this time. We'll still use String and str (and Text{Decoder,Encoder}) everywhere for those. Over time I think we can possibly consider whitelisting APIs to instead use DomString (or similar), but if what @chris-morgan says is right in that unpaired surrogates are quite rare we may be able to get by mostly with this. In the meantime libraries like your @Pauan would probably bind the API internally (e.g. not use web-sys for these APIs where you want DomString)

Ok and for naming, one obvious other name would be JsString and JsStr, which while I think are more appropriate clash with js_sys::JsString, so we'd need to figure out what to do with that.

+1 for providing the escape hatch first, not changing js-sys and web-sys immediately, and then re-evaluating based on our experience with the escape hatch.

I don't have a strong opinion on ill-formed u16 code points in a Vec<u16> vs using some wtf-8 Vec<u8>. Defer to others here.

I'm happy-ish with the name DomString, but don't care much.

It is a little misleading when using js-sys/ECMAScript stuff outside of the Web (where there is no DOM)
I think we should avoid JsString and JsStr for the reasons you mentioned.
More bike shedding:

IllFormedUtf16String / IllFormedUtf16Str if we do Vec<u16> and UTF-16 code points
Wtf8String / Wtf8Str if we do Vec<u8> and WTF-8

The main alternative would be WTF-8 with Vec internally, but the only real point of doing that would be to support an allocation-less to_str() method (like OsStr on Windows) and if DomString is only an intermediate type before we interpret UTF-16, it seems like to_str isn't that well motivated for our use case.

I'm not sure I agree with that: since DomStr won't be the default, many APIs will continue to accept &str, and it's common to call an API which returns DomString and then pass it to another API (which accepts &str). Requiring an O(n) re-encoding + allocation + copying isn't great.
So if WTF-8 can faithfully represent everything (including private use areas), I think we should use it instead.

In the meantime libraries like your @Pauan would probably bind the API internally (e.g. not use web-sys for these APIs where you want DomString)

That's fine, though I think it has implications for Gloo (which I'll discuss on that repo).

Ok and for naming, one obvious other name would be JsString and JsStr, which while I think are more appropriate clash with js_sys::JsString, so we'd need to figure out what to do with that.

I think DomString is perfectly fine. It's what the official W3C spec uses, and it doesn't cause confusion with JsString.
I guess you could argue that using DomString for strings which aren't web APIs is a bit weird, but I think in practice most uses will be web APIs.

As another alternative, we could simply add in wasm-bindgen marshalling for the already-existing wtf8 crate (which is internally a Vec<u8>).
That sidesteps the entire naming choice, and also enables no-copying conversion to String and &str.

It's true yeah that DomString to &str requires an allocation and re-encoding, but if we use wtf-8 (or something like it) then we'll always be doing the encoding in JS whenever we enter/exit wasm. I think that's one of the things I'm mainly worried about is that we can't actually "just use" the wtf-8 crate, but rather we have to write the encoding in JS and maintain it as part of the default bindings.
This feels very similar to the decision of how to represent OsStr on Windows to me. We sort of just made an educated guess about how to represent it. I'm not really sure what the patterns of usage are we'd have with DomStr vs str.
@Pauan do you feel you've got a good handle on when we'd use DomStr vs str? I think we'll use those patterns as guidance for how to represent the type internally (to minimize encoding costs and such). Can you expand a bit on how you see this being used in gloo? (I'm mostly interested in developing a gut feeling for usage properties of DomStr and frequency)

I consider @alexcrichton’s proposed plan to be reasonable. I don’t think WTF-8 is the way forwards with this; it doesn’t solve problems, merely shifts which side you have to write encoding on, Rust or JavaScript. A [u16]-based type seems to me the way to go, especially given the projected rarity of its use.
One correction: from_iter() would be on DomString rather than DomStr.

@alexcrichton I think right now we just need a way of detecting if a JS string is valid UTF-16 or not.
I see a couple options:

The as_string method already says that it returns None if it's invalid UTF-8, but that's currently not true.
So it could be changed so it actually does a validity check and returns None if the JS string isn't valid UTF-16.

Add a new is_valid_utf16_string method to JsValue which does the validity checking and returns bool.

Either solution works fine. With that basic building block, it's now possible for the input event to detect whether the string is valid UTF-16 or not (and skip the event if it's not).
This completely sidesteps the issue: we no longer need to define DomString, and we don't need to use WTF-8 either.

@Pauan you're thinking that in these cases folks would take JsString, learn it's not valid utf-8, and then manually extract the u16 values? I like the idea of sidestepping the issue! If that can idiomatically solve the problem then that definitely sounds like our best bet.
As to which of those options, I think it makes sense to leave as_string as is and add a new method. We'd want to update the documentation of as_string to say what the implementation does (typeof + TextEncoder) along with these caveats for sure

@alexcrichton Well, I see two general use cases:

If an input event contains unpaired surrogates, it should just completely ignore the entire event. That's solved by using JsValue::is_valid_utf16_string.

Somebody actually wants to preserve the unpaired surrogates (for some reason). That case would be handled by DomString (or other similar strategies like Vec<u16>).

I don't actually know of any use cases for 2, so that's why I think solving 1 is good enough.
So we only need JsValue::is_valid_utf16_string, so that the input event can ignore unpaired surrogates.
And if later we actually do need to preserve unpaired surrogates, we can reinvestigate DomString.

Sounds like a solid plan to me! Would the addition in that case be:
impl JsValue {
    pub fn is_valid_utf16_string(&self) -> bool {
         // some wasm_bindgen intrinsic
    }
}

@alexcrichton Yes, exactly.
And the intrinsic would be implemented like this on the JS side (it has to be on the JS side, sadly):
function isValidUtf16String(str) {
    var i = 0;
    var len = str.length;

    while (i < len) {
        var char = str.charCodeAt(i);

        ++i;

        // Might be surrogate pair...
        if (char >= 0xD800) {
            // First half of surrogate pair
            if (char <= 0xDBFF) {
                // End of string
                if (i === len) {
                    return false;

                } else {
                    var next = str.charCodeAt(i);

                    // No second half
                    if (next < 0xDC00 || next > 0xDFFF) {
                        return false;
                    }
                }

            // Second half of surrogate pair
            } else if (char <= 0xDFFF) {
                return false;
            }
        }
    }

    return true;
}

(I loosely translated it from the DOMString spec by using some tips from the TextEncoder polyfill.)
It looks big, but it minifies quite well, and it should be extremely fast.

If we're leaving as_string as-is, can we at least rename it to as_string_lossy?
That would be more consistent with the standard library, where String::from_utf16_lossy does the same thing (and String::from_utf16 returns an error).

@lfairy I agree, but that will require a deprecation/breaking change cycle, so it's best to batch that together with a bunch of other breaking changes.

@Pauan  I think it depends on whether we want to rename or not as to whether we do it now, because if we do want to rename having a deprecation is actually pretty easy to handle at any time.
I'm less certain though that we'd want to rename. It seems to me that it's far more common to use String and str than as_string which don't have a "lossy" warning. In that sense renaming as_string may not actually buy us much because it may not be used all that much.
I think we probably want to just update documentation to indicate the hazard?

@alexcrichton I don't have strong opinions about it: I really like the idea of being consistent with the Rust stdlib, but breaking changes aren't fun.
And as you say, any bindings that return String are lossy, and without any warnings! So I guess some doc changes are good enough.

This has a lot of very deep implications, including for international languages (e.g. Chinese).

The original issue cites emoji input, not Chinese. Are you actually able to reproduce this with Chinese? (Perhaps by enabling HKSCS characters in Microsoft Bopomofo (in pre-Windows 10 "Advanced" settings); I have no idea how to type Hong Kong characters with a Mandarin pronunciation-based IME. Or more obviously, by enabling HKSCS characters in Microsoft Changjie (directly in the Windows 10-style settings).)
Which browsers does the original emoji issue reproduce in? Is there a minimal non-wasm JS-only test case that demonstrates the issue?
For this IME-triggered event issue, checking the whole string for lone surrogates, using WTF-8, or making UTF-16 to UTF-8 conversion report unpaired surrogates are all overkills: It's enough to check if the last code unit of the JS string is an unpaired surrogate and discard the event if so. Please don't make all DOM APIs use OsStrs unless it's truly, truly, backed up by use cases.

The original issue cites emoji input, not Chinese. Are you actually able to reproduce this with Chinese?

I haven't tested it, though all Unicode characters above a certain point use surrogate pairs, and the issue is about surrogate pairs, so I would expect it to behave the same. Maybe it doesn't, though.

Is there a minimal non-wasm JS-only test case that demonstrates the issue?

I'll work on creating a reduced test case!

It's enough to check if the last code unit of the JS string is an unpaired surrogate and discard the event if so.

I don't think that's true: the user can input text anywhere in the <input> (not just as the last character!)

Please don't make all DOM APIs use OsStrs unless it's truly, truly, backed up by use cases.

We have no plans to use OsStr (or WTF-8, or anything like that).
The plan is to make the value method on gloo_events::InputEvent return Option<String> (where it returns None if there are unpaired surrogates). That's it. It's basically the simplest and least-invasive change possible.

Okay, I created a reduced test case: http://paaru.pbworks.com/w/file/fetch/132853233/input-bug.html
I tried to reproduce it with Chinese, but I wasn't able to.
It turns out that Chinese characters are not encoded with surrogate pairs, since they're in the BMP. So it's only very rare historical Chinese characters which might be affected.
And since (as far as I can tell) Windows doesn't have any way to input those characters, I wasn't able to reproduce that.
However, I was able to reproduce it with emoji characters with the following browsers:

Mozilla Firefox Developer Edition 67.0b7 (64-bit)
Google Chrome 73.0.3683.86 (Official Build) (64-bit)

Interestingly, I could not reproduce it with these browsers (they do not have this bug):

Microsoft Edge 42.17134.1.0
Microsoft Internet Explorer 11.648.17134.0

P.S. I also verified that this bug happens if the user enters the emoji anywhere in the <input>, so checking the final character is not sufficient.

Okay, I created a reduced test case: http://paaru.pbworks.com/w/file/fetch/132853233/input-bug.html

Thank you. Confirmed with the Windows 10 touch keyboard and Firefox.
However, the emoji picker integrated into the Microsoft Pinyin IME does not reproduce this problem, which suggests that the touch keyboard is trying to generate the text using a non-IME iterface to the app.

It turns out that Chinese characters are not encoded with surrogate pairs, since they're in the BMP. So it's only very rare historical Chinese characters which might be affected.

I mentioned HKSCS in my previous comment, because some Hong Kong characters that may be rare but not necessarily historical are assigned to CJK Unified Ideographs Extension B. For example, Apple Cangjie, IBus (Ubuntu, etc.) Cangie, and Android GBoard Cangjie allow 𥄫 to be entered by typing bunhe (assuming QWERTY keycaps). Even after enabling HKSCS for Microsoft ChangJie (sic), I'm unable to enter the character using Microsoft's implementation.
This is on Windows 10 1803. Another thing to test on a newer Windows 10 would be enabling the Adlam keyboard (Adlam is assigned to Plane 1) and seeing how it behaves.

P.S. I also verified that this bug happens if the user enters the emoji anywhere in the , so checking the final character is not sufficient.

OK.

For example, Apple Cangjie, IBus (Ubuntu, etc.) Cangie, and Android GBoard Cangjie allow 𥄫 to be entered by typing bunhe (assuming QWERTY keycaps).

I should mention that none of these appear to expose an unpaired surrogate in Firefox.

Entering astral characters with Microsoft Bopomofo doesn't appear to expose unpaired surrogates. (To test: In IME properties under Output Settings in Character Set, set the radio button to Unicode and check all boxes. Then type e.g. 1i6, press down arrow, press right arrow and then use the arrow keys and enter to choose any of the red characters in the palette.)

Gecko bug, Chrome bug


Ignore file by mask in settings

I have a folder with images with different names except main.jpg
I want convert main.jpg to 600px and over files to 1000px
 .pipe(responsive({
     '**/*.{jpg,JPG}': {width: 1000 },
     'main.{jpg,JPG}': {width: 600},
 })

But i want ignore main.jpg for 1000px width resizing. And do this
'**/!(main.*)*.{jpg,JPG}': {width: 1000 },

But this rule is not working.

I had same problem, You can use two tasks and define these rules in gulp.src():
function devImagesResizeMain(){
    return gulp.src(
        [
            IMG_SRC_DIR + '/**/*.{jpg,jpeg,png,gif,svg}',
            '!' + IMG_SRC_DIR + '/**/header.{jpg,jpeg,png,gif,svg}'
        ], { allowEmpty: true })
    .pipe(responsive(
        {
            '*.jpg': getImagesResizeRules()
        }
    ))
    .pipe(gulp.dest(DEST_DEV_DIR + DEST_IMG_DIR));
}

function devImagesResizeCust(){
    return gulp.src( IMG_SRC_DIR + '/**/header.{jpg,jpeg,png,gif,svg}', { allowEmpty: true })
    .pipe(responsive(
        {
            'header.jpg':   injectObjectsInArrayByObj( getImagesResizeRules(), {gamma: 3.0} )
        }
    ))
    .pipe(gulp.dest(DEST_DEV_DIR + DEST_IMG_DIR));
}

gulp.task('resize',gulp.series(devImagesResizeMain, devImagesResizeCust, function (done) {
    done();
}));

This solution works, but it leads to more issues:
return gulp.src(
    IMG_SRC_DIR + '/**/*.{jpg,jpeg,png,gif,svg}'
)
.pipe(responsive({
     '!main.*': {width: 1000 },
     'main.{jpg,JPG}': {width: 600},
 })

This only works for one image format type (.jpg in this case), as it will grab ALL images excluding 'main.' and format the width to 1000px.
Another solution would be to prefix all images depending on their purpose and do something like this:
.pipe(responsive({
     'other-*.{jpg,JPG}': {width: 1000 },
     'main-*.{jpg,JPG}': {width: 600},
 })

Prefixing could be made less tedious for a large amount of files with gulp-rename.


HDDS-7349. Flaky integration test have memory leak for RatisDropwizardExports

What changes were proposed in this pull request?

Additional RatisDropwizardExports old by DNs after its closed
register of RatisDropwizardExports even its already present with name in ratisMetricsMap

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7349
How was this patch tested?
Tested with flaky and taking heapdump for objects of RatisDropwizardExports referred by DNs ratisMetricsMap.

@nandakumar131 @ChenSammi Please review

@szetszwo plz review

@sumitagrawl , each datanode/om/scm creates its own MetricRegistries instead using global(); see https://issues.apache.org/jira/secure/attachment/13051264/MetricRegistries.patch

...; see also https://issues.apache.org/jira/secure/attachment/13051276/3858_review.patch

@sumitagrawl , just found that I uploaded a wrong file.  It was great that you have fixed everything already!  Anyway, this is the correct file https://issues.apache.org/jira/secure/attachment/13051439/3858_review2.patch

Thanks @sumitagrawl for the patch, @szetszwo for the review.


PAPERLESS_OCR_LANGUAGE deu

Dear all,
There seems a problem with with OCR language German. If i set in the confiy.yaml the "PAPERLESS_OCR_LANGUAGE: GER" (GER according to  https://www.loc.gov/standards/iso639-2/php/code_list.php) the following error comes:
Logs:
"Package tesseract-ocr-ger not found! :("
"?: The selected ocr language ger is not installed. Paperless cannot OCR your documents without it. Please fix PAPERLESS_OCR_LANGUAGE"
How do I fix this? How can I install the needed language pack GER or DEU
Thanks

Mmh from what I remember German is deu, but u u seem to have tried also. I'll check

Actually I don't manage this script it's from the paperless ngx upstream app : https://github.com/paperless-ngx/paperless-ngx/blob/bdf9b2453fcd8ef14045463bc90779f977ce5ed8/docker/docker-entrypoint.sh#L157
You'll have to post on their github to get support I'm afraid...

Hi, according to this https://github.com/paperless-ngx/paperless-ngx/blob/00c3d8c5237030e0934e7885b6b32f26ee7fcb89/Dockerfile#L101 German is preinstalled. So you don't need to reinstall it and should just be able to enable it in the webui

Hi Alex,
Thanks a lot for your help.
In meanwhile I found a solution:
I put the following command in the config.yaml
PAPERLESS_OCR_LANGUAGES: deu ces
This command installs the language German

Thanks!


Custom RestTemplate is not loaded

Describe the bug
Spring Boot 2.4.5
Spring Cloud 2020.0.2
I deployed Spring Cloud Config Server on Google Cloud Run. It works fine when calling it from the command line (curl) with a generated access token using gcloud auth print-identity-token. I want to call it from a Spring Boot application with Spring Cloud Config Client. To retrieve the access token I customized the RestTemplate. The issue is that this customization is not loaded/called even though I followed the documentation https://docs.spring.io/spring-cloud-config/docs/3.0.0/reference/html/#_security_2. I checked by adding breakpoint in the custom configuration class but the methods are not called. I get an exception which is expected since the default RestTemplate is still in use:
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602)
	at org.springframework.cloud.config.client.ConfigServerConfigDataLoader.getRemoteEnvironment(ConfigServerConfigDataLoader.java:269)
	at org.springframework.cloud.config.client.ConfigServerConfigDataLoader.doLoad(ConfigServerConfigDataLoader.java:105)

Sample
src/main/resources/application.yml
spring:
  config:
    import: configserver:https://xxxxxxx.run.app

src/main/resources/bootstrap.yml
spring:
  application:
    name: playground

src/main/resources/META-INF/spring.factories
org.springframework.cloud.bootstrap.BootstrapConfiguration=com.company.playground.CustomConfigServiceBootstrapConfiguration

The custom configuration
@Configuration
public class CustomConfigServiceBootstrapConfiguration {

    @Bean
    public ConfigServicePropertySourceLocator configServicePropertySourceLocator(ConfigClientProperties clientProperties) {
        ConfigServicePropertySourceLocator configServicePropertySourceLocator = new ConfigServicePropertySourceLocator(clientProperties);
        configServicePropertySourceLocator.setRestTemplate(restTemplate());
        return configServicePropertySourceLocator;
    }

    public RestTemplate restTemplate() {
        return new RestTemplateBuilder()
                .interceptors(new GCloudInterceptor())
                .build();
    }
}

The interceptor using google-auth-library-oauth2-http
public class GCloudInterceptor implements ClientHttpRequestInterceptor {
    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution) throws IOException {
        GoogleCredentials credentials = GoogleCredentials.getApplicationDefault();
        credentials.refreshIfExpired();
        AccessToken accessToken = credentials.getAccessToken();
        request.getHeaders().add(HttpHeaders.AUTHORIZATION, "Bearer " + accessToken.getTokenValue());
        return execution.execute(request, body);
    }
}

I managed to make it work but with some changes. Basically it's not possible to use the Spring Boot Config Data Import.
Set spring.cloud.bootstrap.enabled=true and then it's working.

See sample here https://github.com/spring-cloud/spring-cloud-config/blob/b1e795d8d191c04d1221ab88a30a6b2f431bba7f/spring-cloud-config-client/src/test/java/org/springframework/cloud/config/client/ConfigServerConfigDataCustomizationIntegrationTests.java#L59-L60

@spencergibb  It seems to me that the documentation doesn't write that if you want to use the CustomConfigServiceBootstrapConfiguration you must set the spring.cloud.bootstrap.enabled=true.
It seems that Spring is going in the direction of putting everything in the application.yaml (one file) and
in this case we go backward by using the old configuration.
Don't you think it should be fixed?

@spencergibb
Following our conversation in the slack channel, I found this open issue which is relevant to the problem I was describing. so maybe we can use this issue instead of creating a new one.
Here is my original message in the channel:

Spring cloud config-client docs has a section about providing custom RestTemplate which is useful in same cases like when the config-server is using oauth2 for security. It uses ConfigServicePropertySourceLocator as example for this customisation. But if i’m not mistaken this class (ConfigServicePropertySourceLocator) is only used in legacy bootstrap approach and the new/default way to fetch data from config-server is via ConfigServerConfigDataLoader .  I think the docs should include an example for ConfigServerConfigDataLoader too and also it should point out that ConfigServicePropertySourceLocator would be useful only if legacy bootstrap approach is enabled.

Do we have any update on this issue? This is the exact behavior/issue we are also getting what @kvmw had reported above and quoted from slack channel.
Are we expecting any update in current ConfigServerConfigDataLoader class definition to have a support for Custom RestTemplate for uses case like oauth2; with the new/default way to fetch data from config-server?

@abjimmypro The conversation was not about changing the code but only updating documentation.
You need to implement a BootstrapRegistryInitializer to register a RestTemplate with an auth interceptor. Here you can find a sample.
 You need to register above class in spring.factories.  Check this sample again.
Please note that by the time BootstrapRegistryInitializer is called the application properties is not loaded yet. so you cannot load oauth2 configuration form properties file.  You can use other approaches like environment variables ( System.getEnv(...) ) to access your configuration.

@abjimmypro The conversation was not about changing the code but only updating documentation.
You need to implement a BootstrapRegistryInitializer to register a RestTemplate with an auth interceptor. Here you can find a sample.  You need to register above class in spring.factories.  Check this sample again.
Please note that by the time BootstrapRegistryInitializer is called the application properties is not loaded yet. so you cannot load oauth2 configuration form properties file. You can use other approaches like environment variables ( System.getEnv(...) ) to access your configuration.

Thanks @kvmw for clarification and sample.


sequence of 's is mis-rendered in some cases.

by itself,
#### *expression1* *expression2* ... *expressionn*

renders correctly as

expression1 expression2 ... expressionn

but in the context of https://raw.github.com/dmajda/pegjs/master/README.md renders as

expression1 / expression2 / ... / expressionn

maybe a parser issue?
thanks!

I think you're looking at it wrong. This is probably best explain by some commandline examples.
$ marked readme.md | grep sub

expression1 expression2 ... expressionn

expression1 / expression2 / ... / expressionn

$ cat readme.md | grep sub
#### *expression1* *expression2* ... *expressionn*
#### *expression1* / *expression2* / ... / *expressionn*

There are two separate locations for that heading. One has slashes, one doesn't.

oh, woops, sry. closing.


couldn't open file: cfg/coco.data

I have pass building.
when I run,
./darknet detect cfg/yolov3.cfg yolov3.weights data/dog.jpg
system notify
couldn't open file: cfg/coco.data
How should I do?
Thank you very much.

You should also check whether the cfg/coco.data exists in your folder or not. If no, try to create it
The detect command is shorthand for a more general version of the command. It is equivalent to the command:
./darknet detector test cfg/coco.data cfg/yolov3.cfg yolov3.weights data/dog.jpg


Issue #112 - Updated the Kubernetes version of the GKE templates

GKE version (Regular) 1.28.8-gke.1095000
Rancher version v2.8.4

And the drone check is failing - https://drone-pr.rancher.io/rancher/tf-rancher-up/190/1/4 likely because https://github.com/rancher/tf-rancher-up/blob/main/modules/distribution/gke/docs.md?plain=1#L31 was not updated

and when I tried to run the terraform, I got this error:
helm_release.ingress-nginx: Creating...
╷
│ Error: Unable to continue with install: could not get information about the resource ServiceAccount "ingress-nginx" in namespace "ingress-nginx": serviceaccounts "ingress-nginx" is forbidden: User "client" cannot get resource "serviceaccounts" in API group "" in the namespace "ingress-nginx"
│
│   with helm_release.ingress-nginx,
│   on main.tf line 23, in resource "helm_release" "ingress-nginx":
│   23: resource "helm_release" "ingress-nginx" {

And the drone check is failing - https://drone-pr.rancher.io/rancher/tf-rancher-up/190/1/4 likely because https://github.com/rancher/tf-rancher-up/blob/main/modules/distribution/gke/docs.md?plain=1#L31 was not updated

Modified docs.tf and terraform.tfvars.example as suggested.


TravisBuddy Integration

TravisBuddy will comment on pull requests in your public repository everytime a test failed in one of them. The comment will include only the part of the build log that applies to your testing framework, so your contirbutors won't have to enter Travis's website and search the long and annoying build log for the reason the tests failed.
Here's an example: https://github.com/bluzi/static-server/pull/1

Travis tests have failed
Hey Travis Buddy,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
npm test
><EMAIL_ADDRESS>test /home/travis/build/bluzi/travis-buddy
> mocha

  GET /status
    ✓ should be running and return HTTP status 200

  GET /test
19:02:35.183Z  INFO unknown : Testing job '295503460' (node_js)
19:02:35.471Z  INFO unknown : Done found after 0/10 attempts.
    ✓ should return HTTP status 200 (300ms)

  POST /
19:02:35.493Z  INFO unknown : Handling request for 'create sdfdsfsd' by 'Eliran Pe'er'
19:02:35.686Z  INFO unknown : Done found after 0/10 attempts.
19:02:35.690Z ERROR unknown :
    Error in handler { params: {},
      owner: 'bluzi',
      repo: 'name-db',
      pullRequest: 32,
      pullRequestTitle: 'create sdfdsfsd',
      buildNumber: 295066345,
      jobs: [ 295066347 ],
      author: 'Eliran Pe\'er',
      state: 'failed',
      branch: 'bluzi-patch-3',
      travisType: 'push',
      language: 'node_js',
      scripts: undefined }
19:02:35.691Z ERROR unknown :
    Error: Invalid GitHub access token null { params: {},
      owner: 'bluzi',
      repo: 'name-db',
      pullRequest: 32,
      pullRequestTitle: 'create sdfdsfsd',
      buildNumber: 295066345,
      jobs: [ 295066347 ],
      author: 'Eliran Pe\'er',
      state: 'failed',
      branch: 'bluzi-patch-3',
      travisType: 'push',
      language: 'node_js',
      scripts: undefined }
    1) should return HTTP status 200 and ok message

  GET /
    ✓ should return HTML and status 200

  GET /getting-started
    ✓ should return HTML and status 200

  GET /contact-us
    ✓ should return HTML and status 200

  GET /dashboard
    ✓ should return HTML and status 200

  6 passing (591ms)
  1 failing

  1) POST /
       should return HTTP status 200 and ok message:
     Error: expected 200 "OK", got 500 "Internal Server Error"
      at Test._assertStatus (node_modules/supertest/lib/test.js:266:12)
      at Test._assertFunction (node_modules/supertest/lib/test.js:281:11)
      at Test.assert (node_modules/supertest/lib/test.js:171:18)
      at Server.assert (node_modules/supertest/lib/test.js:131:12)
      at emitCloseNT (net.js:1655:8)
      at _combinedTickCallback (internal/process/next_tick.js:135:11)
      at process._tickCallback (internal/process/next_tick.js:180:9)

npm ERR! Test failed.  See above for more details.

npm run lint


Expectation of the zoom in/out button is not clear

Description:
I expected the design view will zoom in when I clicked on the zoom in button. But, it only expanded the collapsed areas in the design. IMO, the icon should be changed

Affected Versions:
ballerina-0.970.0
OS, DB, other environment details and versions:
Ubuntu 15.10
Firefox - 59.0.2 (64-bit)

Zoom in is expected to zoom the hidden fields and provide an in-depth view. hence closing the issue


React build with Vite

Description

This PR is a part of #985 to migrate completely on vite, the second part will include vitest setup

From a high-level perspective, we are swapping out  bare rollup in favor of vite  but also

New Dependency Strategy
We introduced a new external API
Optimised Bundle
Upgraded react version

New Dependency Strategy
Old Implementation, externalised every dependency during bundling phase, for example in order to use video component you have to install them separately, same goes for dependencies listed bellow

classnames
dom-helpers,
nanoid
react-transition-group,
video.js
videojs-youtube

The new approach only externalises react and react-dom so in order to start using @ilo-org/react you can just run
pnpm add react react-dom @ilo-org/react

New External API
With the new bundle we have 2 main entry points:

FIrst one covers the style part @ilo-org/react/styles, which should be used to import the styles

...
import<EMAIL_ADDRESS>import<EMAIL_ADDRESS>
createRoot(document.getElementById("root")!).render(
  <StrictMode>
    <App />
  </StrictMode>
);

And the second one is just @ilo-org/react which is root for both hooks and components

So in order to grab any js part from the library you can just import it from the root
import { Loading, Button, useGlobalSettings  } from "@ilo-org/react";

typedefinitions are also exported to help with component interaction
https://github.com/user-attachments/assets/622c0dd7-73bb-4108-b59f-1fc02defa7ac
Optimised Bundle
New bundle is designed to be easily tree-shakable for many major bundlers.

For example this is the size output for simple "Hello World" app with vite

And this is the same output but with "Loading" component from "@ilo-org/react"

Loading + Button (that includes "Icon")

From development perspective libdir looks like this

We output both esm and cjs but by utilising the multiple input source for "rollup" we are outputin each individual files with optimised chunks that are stored inside _store
Upgraded react version
And for the bonus, @ilo-org/react is now upgraded to react:18 with fixed typeings

Since there was no affected functionality, I marked it as a patch release

hey @GGKapanadze, we introduced a new external api and bumped react by two versions, I think that makes it a minor release :)


Release Google.Cloud.Logging.Console version 1.3.0

Changes in this release:
New features

Change netstandard2.1 target to netstandard2.0 (commit 82bea85)

Created release for Google.Cloud.Logging.Console-1.3.0


Options - Toggle GitHub flavored markdown

I'd love if markdown support had an option for GitHub flavored markdown, so emojis and other great features were supported. 😃 When I think of things folks would like for simple websites that are super easy to deploy with dat://, emoji are at the top of the list.
I am aware there is Win+. to bring up the Windows emoji keyboard, but I think the quick tags are nice. e.g., :smiley: = :smiley:
There's also the checklist stuff.
- [ ] Unchecked
- [x] Checked

[ ] Unchecked
[x] Checked

I am not sure of Beaker's support for tables, <kbd> and code in markdown with the nifty syntax highlighting and stuff

I actually am 👍 with this kind of feature. It's a defacto standard, why not do it!

hmm, defacto standard, is imo depending on the users context. ... It's a one of the many variants that are defined in the spec.


[Bug] Severe lag when using certain themes

Describe the Bug
When using some themes like Nocturnal and OperaGX theme, BD becomes severely laggy. Some don't produce the same lag as the others, the Cyberpunk theme is less laggy but nocturnal is very laggy and OperaGX theme makes it unusable.
To Reproduce
Use the themes (reinstalled multiple times with no luck, tested on both Arch and Windows).
Expected Behavior
Severe lag when doing anything at all.
Discord Version
Stable.

I mean the themes were fine until suddenly they caused a lot of lag. I clean installed windows a couple of times while preserving the betterdiscord folder without luck. I tried disabling all plugins once but that didn't make any difference.

Lag from themes would be just that--caused by the themes themselves. Themes that use attribute selectors (i.e. [] style selectors), themes using wildcard and other generic selectors or many old dead selectors can all cause lag. Rendering lag can also be caused by mismanaging z layers or doing many animations/transitions without forcing a gpu render with something like transform.
These are not things caused by or fixable by BD itself.


Fix for this is to load diff-mode before loading direnv-mode, i.e. something like this if you're using use-package:
(use-package direnv
  :after diff-mode
  :config (direnv-mode))

Fix for this is to load diff-mode before loading direnv-mode, i.e. something like this if you're using use-package:
(use-package direnv
  :after diff-mode
  :config (direnv-mode))

i added (require diff-mode)` in 381176f
(package-lint doesn't warn about font faces, only about symbols)

i added (require diff-mode)` in 381176f
(package-lint doesn't warn about font faces, only about symbols)

Thanks a lot @wbolster

Thanks a lot @wbolster


How did your program achieve such fast response time?

Hello,
I noticed that when using your Scrcpy program, the interaction with the Android device through ADB is very fast, without any noticeable delays. I'm using the AdvancedSharpAdbClient library in my C# project to interact with an Android device through ADB, but I've noticed a significant delay and not as much responsiveness as in Scrcpy.
I would like to know how your program achieved such a fast response time and responsiveness when working with an Android device through ADB. I would be grateful if you could share your knowledge and experience in this area and offer any recommendations or advice for optimizing the ADB interaction in my project using the AdvancedSharpAdbClient library.

I don't have to broadcast the video. Clicks and swipes are enough.

See https://github.com/Genymobile/scrcpy/issues/231#issuecomment-414111753

Input injection is not part of the adb protocol. If you execute adb shell input …, then you're just executing the command input from adb shell, with the performance issues mentioned in my comment.
So you have to implement some server code running on the device with shell permissions, that you run once and communicate the commands over a socket.

I think I got it. You only use ADB to download the server.jar to your phone, and then you interact with it. Am I right?


Max supply needs to be changed in FAQ

https://www.thenewboston.com/faq#is-there-a-max-supply

It's 2,032,628,357 now.


Applying suggestions always inserts a newline too

At least it does on Windows in a file with LF line endings, I wouldn't be surprised if there's some platform nonsense going on here.

I think this is maybe just https://github.com/nwolverson/purescript-suggest/issues/7 - is this where applying a remove import leaves a blank line instead? Sounds like not the same as https://github.com/nwolverson/vscode-ide-purescript/issues/75 if you have LF.

Hah, no, it's different to either of those actually - the most prominent case is import related, but it's where the imports are updated (redundant imports from the module, not that the import is entirely redundant). It adds a new empty line under the import that is rewritten, rather than at the end of the buffer too.

Can you retest this with latest, there should be some relevant changes

Works a treat now 😍

Great :)


#2094 pulseaudio apply states

fixes #2094
Test case:
		"tooltip": false,
		"states": {
			"full": 100,
			"mid": 99,
			"off": 0
		},
		"max-volume": 110,
		"format":       "OVER {format_source} V {volume}%   ",
		"format-full":  "{format_source}   ",
		"format-mid":   "{format_source} V {volume}%   ",
		"format-off":   "OFF {format_source}   ",

OVER shown over 100%, no volume at full, volume below 100%, OFF at 0%

freebsd / clang appears to have failed with a CI infrastructure problem.
I'd be grateful if you could re-run it.

Thanks!


refactor: move networktime to world

still works in host situations. another steps towards allowing multiple worlds per client/server.

pretty sure this will have a Doc error that ill work on later.

originally it was testing two different values. now both are essentially the same.

Nice change, just the one suggestion then it should be to merge

Oh also make sure to add BREAKING CHANGE: ... to the commit when it gets merged


Use library in the context of semantic kernel ?

Semantic kernel by Microsoft provides out of the box connectors for open ai and Google but not anthropic. Would it be possible to integrate this library within the context of semantic kernel as a text or chat completion engine?

Within the context of a text or chat completion engine, absolutely.  Within the context of something that supports Semantic Kernel's native function calling, I think that'd take some doing because this library won't pick up on the annotations of a KernelFunction or inherently know what to do with them out of the box.  Here's a really rough implementation of how you'd implement the ChatCompletionService:
public class ClaudeChatCompletionService: IChatCompletionService
{
    public IReadOnlyDictionary<string, object?> Attributes { get; }

    public async Task<IReadOnlyList<ChatMessageContent>> GetChatMessageContentsAsync(ChatHistory chatHistory, PromptExecutionSettings? executionSettings = null,
        Kernel? kernel = null, CancellationToken cancellationToken = new CancellationToken())
    {
        throw new NotImplementedException();
    }

    public async IAsyncEnumerable<StreamingChatMessageContent> GetStreamingChatMessageContentsAsync(ChatHistory chatHistory,
        PromptExecutionSettings? executionSettings = null, Kernel? kernel = null,
        CancellationToken cancellationToken = new CancellationToken())
    {

        var client = new AnthropicClient();
        var messages = chatHistory.Where(p => p.Role != AuthorRole.System).Select(p =>
            new Message(p.Role == AuthorRole.Assistant ? RoleType.Assistant : RoleType.User, p.Content)).ToList();

        var parameters = new MessageParameters()
        {
            Messages = messages,
            MaxTokens = 512,
            Model = AnthropicModels.Claude35Sonnet,
            Stream = true,
            Temperature = 1.0m,
            SystemMessage = chatHistory.FirstOrDefault(m => m.Role == AuthorRole.System)?.Content
        };
        var first = true;
        await foreach (var res in client.Messages.StreamClaudeMessageAsync(parameters))
        {
            if (res.Delta != null)
            {
                yield return new StreamingChatMessageContent(first? AuthorRole.Assistant : null, res.Delta.Text);
                first = false;
            }

        }

    }
}

You'd just register and use like so:
var builder = Kernel.CreateBuilder();

builder.Services.AddSingleton<IChatCompletionService>(provider => new ClaudeChatCompletionService());

Kernel kernel = builder.Build();

IChatCompletionService chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

// Create the chat history
ChatHistory chatMessages = new ChatHistory("""
                                           You are a friendly assistant who likes to follow the rules.
                                           You will complete required steps
                                           and request approval before taking any consequential actions. If the user doesn't provide
                                           enough information for you to complete a task, you will keep asking questions until you have
                                           enough information to complete the task.
                                           """);

// Start the conversation
while (true)
{
    // Get user input
    System.Console.Write("User > ");
    chatMessages.AddUserMessage(Console.ReadLine()!);

    // Get the chat completions
    OpenAIPromptExecutionSettings openAIPromptExecutionSettings = new()
    {

    };
    var result = chatCompletionService.GetStreamingChatMessageContentsAsync(
        chatMessages,
        executionSettings: openAIPromptExecutionSettings,
        kernel: kernel);

    // Stream the results
    string fullMessage = "";
    await foreach (var content in result)
    {
        if (content.Role.HasValue)
        {
            System.Console.Write("Assistant > ");
        }
        System.Console.Write(content.Content);
        fullMessage += content.Content;
    }
    System.Console.WriteLine();

    // Add the message from the agent to the chat history
    chatMessages.AddAssistantMessage(fullMessage);
}

thank you for thorough reponse .
i would add it as extension method also:
IKernelBuilder kernelBuilder = Kernel.CreateBuilder();
kernelBuilder.AddAnthropicClaudeChatCompletion(
    modelId: "xxxx",
    apiKey: "YOUR_API_KEY",
    orgId: "YOUR_ORG_ID", // Optional
    serviceId: "YOUR_SERVICE_ID", // Optional; for targeting specific services within Semantic Kernel
    httpClient: new HttpClient() // Optional; if not provided, the HttpClient from the kernel will be used
);
Kernel kernel = kernelBuilder.Build();

Will dig around the other neccesities for full coverage for semantic kernel,as you mention.

Just a heads up that we recently added function calling in streaming, so there's some additional support there.

Just another update with the support for IChatClient abstractions, you can use this with Semantic Kernel.
using Microsoft.SemanticKernel;

var skChatService =
    new ChatClientBuilder()
        .UseFunctionInvocation()
        .Use(new AnthropicClient().Messages)
        .AsChatCompletionService();