asciidoctorj-diagram 1.3.0 does not work on Windows with JRuby 1.7.22

When trying to render the test document from the asciidoctorj-diagram on Win7 via the distribution rendering fails with an exception:
c:\work\asciidoctorj-<IP_ADDRESS>\bin>asciidoctorj -r asciidoctor-diagram test.adoc
io/console not supported; tty will not be manipulated
Exception in thread "main" org.jruby.exceptions.RaiseException: (Errno::ENOENT) C:\work\asciidoctorj-<IP_ADDRESS>\lib\asciidoctorj-diagra
m-1.3.0.jar!\gems\asciidoctor-diagram-1.3.0\lib\asciidoctor-diagram\blockdiag.rb
        at org.jruby.RubyFile.realpath(org/jruby/RubyFile.java:840)
        at RUBY.require_relative(file:/C:/work/asciidoctorj-<IP_ADDRESS>/lib/jruby-complete-1.7.22.jar!/jruby/kernel19/kernel.rb:11)
        at RUBY.(root)(C:/work/asciidoctorj-<IP_ADDRESS>/lib/asciidoctorj-diagram-1.3.0.jar!/gems/asciidoctor-diagram-1.3.0/lib/asciidoct
or-diagram/blockdiag.rb:2)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1040)
        at RUBY.(root)(/C:/work/asciidoctorj-<IP_ADDRESS>/lib/jruby-complete-1.7.22.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/cor
e_ext/kernel_require.rb:1)
        at RUBY.require(/C:/work/asciidoctorj-<IP_ADDRESS>/lib/jruby-complete-1.7.22.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/co
re_ext/kernel_require.rb:54)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1040)
        at RUBY.(root)(C:/work/asciidoctorj-<IP_ADDRESS>/lib/asciidoctorj-diagram-1.3.0.jar!/gems/asciidoctor-diagram-1.3.0/lib/asciidoct
or-diagram.rb:1)
        at RUBY.(root)(/C:/work/asciidoctorj-<IP_ADDRESS>/lib/jruby-complete-1.7.22.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/cor
e_ext/kernel_require.rb:1)

So what actually fails is this line in blockdiag.rb:
require_relative 'version'

But a version.rb exists next to the blockdiag.rb!
I can't reproduce that on Linux (in a Docker container) nor in the test cases on Win that also render this document.
Funny thing is that it works if I use another JRuby like 1.7.21 or lower.
Or maybe I just have a bad karma today...

Problem does also not occur on OS X.

I'm sure I've seen something similar like this on an older version of JRuby in the asciidoctor-gradle-plugin.
An on this note, are we not going to use JRuby 9.x? (or is that a 1.6.0 thing).

This looks a lot like a process error in JRuby. They were making a lot of changes in that area of the code and capabilities went in and out for several releases. Does it work with JRuby 9000? There were also fixes to Asciidoctor Diagram 1.3.1 for Windows, so it could be related to that as well. https://github.com/asciidoctor/asciidoctor-diagram/commit/835bcd1c38b1644ab2962a42d59f89cf5a586446

@ysb33r We are shipping AsciidoctorJ with JRuby 1.7 by default (in the dep tree), but we are recommending everyone switch to JRuby 9000 now. No one should be using JRuby 1.7 unless they absolutely must because frankly it was showing a lot of problems in the later releases.

Just to correct something. It was not in asciidoctor-gradle-plugin where I saw it, but jruby-gradle/jruby-gradle-plugin#83. Remember this @mojavelinux? It also involved the asciidoctor-diagram gemm but seems to have been fixed in 1.7.17.
Not exactly the same error, but quite coincidental that it was asciidoctor-diagram involved again.

Asciidoctor Diagram does some pretty advanced tty stuff, so it's not unlikely that if we are going to hit bugs, that it will scare them out.

@ysb33r We are shipping AsciidoctorJ with JRuby 1.7 by default (in the dep tree), but we are recommending everyone switch to JRuby 9000 now. No one should be using JRuby 1.7 unless they absolutely must because frankly it was showing a lot of problems in the later releases.

This still leaves the problem with the full asciidoctorj distribution, which would bundle 1.7.22. Should we not rather just downgrade it to 1.7.21 for thie point release?
(Still thinking though that as from 1.6.0 we should bundle 9k as the default).

downgrade it to 1.7.21 for thie point release?

Sounds like a wise plan. We should verify that actually solves the problem. Is there any other reason we need 1.7.22 (security?)

(Still thinking though that as from 1.6.0 we should bundle 9k as the default).

That's a certainty.

The funny thing is that the test case does not fail!
That means that the require_relative is successfully resolved when the gem is exploded in the file system.
If it's packaged in a jar, it fails.
I'm also for downgrading to 1.7.21, I just chose the newest version.
I am a bit reluctant to jump on JRuby 9000 with a 1.5.3 release because it also means that we raise the runtime requirements from Java 6 to 7.

Am 03.11.2015 um 22:00 schrieb Dan Allen<EMAIL_ADDRESS>downgrade it to 1.7.21 for thie point release?
Sounds like a wise plan. We should verify that actually solves the problem. Is there any other reason we need 1.7.22 (security?)
(Still thinking though that as from 1.6.0 we should bundle 9k as the default).
That's a certainty.
—
Reply to this email directly or view it on GitHub.

I am a bit reluctant to jump on JRuby 9000 with a 1.5.3 release because it also means that we raise the runtime requirements from Java 6 to 7.

Agreed. It's definitely a 1.6.0 thing. But we can still recommend to users that they use JRuby 9000 by setting the dependency explicitly.

If it's packaged in a jar, it fails.

Crazy! But keep in mind, we never released AsciidoctorJ Diagram before, so this could be the first time we're evening trying to run it. That could be why it surprised us.

I am seeing a similar error when I try to execute asciidoctorj-pdf on a windows system through Cygwin (#402: the same code executed on a linux system):
$ java -jar adt.jar
io/console not supported; tty will not be manipulated
Exception in thread "main" org.jruby.exceptions.RaiseException: (Errno::ENOENT) C:\...\adt.jar!\gems\asciidoctor-pdf-1.5.0.alpha.10\lib\asciidoctor-pdf.rb
        at org.jruby.RubyFile.realpath(org/jruby/RubyFile.java:840)
        at RUBY.require_relative(file:/c:/.../asciiDoctorTest-1.0.0-SNAPSHOT-jar-with-dependencies.jar!/jruby/kernel19/kernel.rb:11)
        at RUBY.(root)(c:/.../asciiDoctorTest-1.0.0-SNAPSHOT-jar-with-dependencies.jar!/gems/asciidoctor-pdf-1.5.0.alpha.10/lib/asciidoctor-pdf.rb:2)
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1040)
        at RUBY.(root)(/c:/.../asciiDoctorTest-1.0.0-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1)
        at RUBY.require(/c:/.../asciiDoctorTest-1.0.0-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:128)

I am also packaging this in a fat-jar... I will try it in an unpackaged jar and see if it doesn't fail.

@johncarl81 If you packaged the fat jar with jruby-gradle, it is worth discussing this on Gitter on loggin and issue with the jruby-gradle project.

Ok, this looks similar.
Would be great if you could override JRuby to 1.7.21.

Am 03.11.2015 um 22:39 schrieb John Ericksen<EMAIL_ADDRESS>I am seeing a similar error when I try to execute asciidoctorj-pdf on a windows system through Cygwin (#402: the same code executed on a linux system):
$ java -jar adt.jar
io/console not supported; tty will not be manipulated
Exception in thread "main" org.jruby.exceptions.RaiseException: (Errno::ENOENT) C:...\adt.jar!\gems\asciidoctor-pdf-1.5.0.alpha.10\lib\asciidoctor-pdf.rb
at org.jruby.RubyFile.realpath(org/jruby/RubyFile.java:840)
at RUBY.require_relative(file:/c:/.../asciiDoctorTest-1.0.0-SNAPSHOT-jar-with-dependencies.jar!/jruby/kernel19/kernel.rb:11)
at RUBY.(root)(c:/.../asciiDoctorTest-1.0.0-SNAPSHOT-jar-with-dependencies.jar!/gems/asciidoctor-pdf-1.5.0.alpha.10/lib/asciidoctor-pdf.rb:2)
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1040)
at RUBY.(root)(/c:/.../asciiDoctorTest-1.0.0-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1)
at RUBY.require(/c:/.../asciiDoctorTest-1.0.0-SNAPSHOT-jar-with-dependencies.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:128)
I am also packaging this in a fat-jar... I will try it in an unpackaged jar and see if it doesn't fail.
—
Reply to this email directly or view it on GitHub.

Downgrading (specifying JRuby) 1.7.21 worked on windows.  Interestingly I'm seeing another error in linux (posting in #402)

@ysb33r, I am using maven's assembly plugin to build the fat-jar.

Same problem here (WIndows 8), looks like this JRuby issue still exists: https://github.com/jruby/jruby/issues/2944

Thanks for finding that reference, @atomfrede!

Seems like I have to play around with different JRuby versions.
At least the crash was only reproducible on my machine with 1.7.22.

I updated jruby to <IP_ADDRESS> for my buildscript, still not working, different error: (LoadError) library `java' could not be loaded: java.lang.NoSuchMethodError: org.jruby.util.ByteList.realSize()I

Could you turn on stacktraces so we can see where that's getting triggered?

Rendering also crashes with asciidoctor-diagram 1.3.1 on Win with JRuby 1.7.22.
But runs with 1.7.21.
Cannot reproduce the other errors yet.
Also works with <IP_ADDRESS> on my machine (Win7 Enterprise) (I know, "works on my machine" is a no-no...)

Also works with <IP_ADDRESS> on my machine (Win7 Enterprise)

The real key will be getting this to happen on AppVeyor. Though, before you try that, it's probably best to understand what triggers the errors or otherwise you'll be waiting for builds for hours :)

The real key will be getting this to happen on AppVeyor. Though, before you try that, it's probably best to understand what triggers the errors or otherwise you'll be waiting for builds for hours :

I remember a previous time where we could not reproduce the error on Appveyor at all, but relatively easy with a modern.ie image. I still don't know enough about Appveyor Windows images to explain that.

I still don't know enough about Appveyor Windows images to explain that.

Very likely differences in Windows itself. Unfortunately, we can't run Win7 on AppVeyor, only Windows Server.

@mojavelinux My stacktrace:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':abc'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:64)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:52)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:203)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:185)
        at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:62)
        at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:50)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:25)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:110)
        at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
        at org.gradle.execution.DefaultBuildExecuter.access$000(DefaultBuildExecuter.java:23)
        at org.gradle.execution.DefaultBuildExecuter$1.proceed(DefaultBuildExecuter.java:43)
        at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:30)
        at org.gradle.initialization.DefaultGradleLauncher$4.run(DefaultGradleLauncher.java:158)
        at org.gradle.internal.Factories$1.create(Factories.java:22)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:52)
        at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:155)
        at org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:36)
        at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:103)
        at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:97)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:62)
        at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:97)
        at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:86)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:102)
        at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
        at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:47)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:32)
        at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:77)
        at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:47)
        at org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:51)
        at org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:28)
        at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:43)
        at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:170)
        at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237)
        at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210)
        at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
        at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
        at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206)
        at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
        at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
        at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
        at org.gradle.launcher.Main.doAction(Main.java:33)
        at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
        at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54)
        at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35)
        at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
        at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:30)
        at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
        at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
Caused by: org.jruby.exceptions.RaiseException: (LoadError) library `java' could not be loaded: java.lang.NoSuchMethodError: org.jruby.util.ByteList.realSize()I

I wonder if this is the "compiled with Java 8, running with Java 7" error. It seems really strange that it wouldn't be able to find a core JRuby type.

Using asciidoctorj-diagram on Win 10 also fails with JRuby 1.7.22.
So I think the decision to go back to 1.7.21 is correct.
Nevertheless I also see this warning with 1.7.21:
io/console not supported; tty will not be manipulated

As long as it's just a warning, we shouldn't worry about it too much. But if it causes a failure down the line, then of course it's a real concern.

Fixed by going down to JRuby 1.7.21

+1 !

Filed issue https://github.com/jruby/jruby/issues/3466 for this problem.

asciidoctorj-diagram works with the last release jruby 1.7.23.
But now asciidoctorj-pdf fails: https://github.com/jruby/jruby/issues/3496

we have no CI for windows for jruby, so it looks any windows fix we have another regression somewhere. jruby-9k works better because it does not use those jar-url paths and goes through the classloader instead which does not need to handle OS specific paths at all !!!
so best would be to set GEM_PATH to uri:classloader:// and GEM_HOME to uri:classloader://META-INF/jruby.home/lib/ruby/gems/shared because:

no user installed gems get used
no rvm or chruby setup get used - both set GEM_PATH and/or GEM_HOME and jruby (or any ruby) just uses those environment variables.

assuming you guys use jruby-gradle-plugin to embed those gems into the jar: unfortunately the current version of the jruby-gradle-plugin does not produce a jar which works on jruby-1.7 due to https://github.com/jruby-gradle/jruby-gradle-plugin/issues/258
this last issue also means that the asciidoctor jar will not work with jruby-9k inside (some or most) OSGi setups.

We actually only use the Base plugin (http://jruby-gradle.org/base/) as the Jar plugin does not exactly do what we want.
Also see the discussion in https://github.com/asciidoctor/asciidoctorj/issues/375
So we only use the Base plugin to "install" the gems and then package the exploded content ourselves into the jars.

? We actually only use the Base plugin (http://jruby-gradle.org/base/) as the Jar plugin does not exactly do what we want.
Also see the discussion in #375
So we only use the Base plugin to "install" the gems and then package the exploded content ourselves into the jars.

The JRubyJar stuff have changed a lot. I have been meaning to look at upgrading the 1.6.0 build to see if the new jruby-gradle functionality works for asciidoctor as I would have hoped.

oh - I see. the jar-plugin is far from useful for asciidoctorj :(
happy to help here to get it working here.

It would be great if we could use that!
I spent some hours trying to migrate our build to the Jar plugin and always found a full exploded jruby library inside the created jars.
Maybe I've done something wrong, so if you could help to get it running that would be just awesome.
I wrote my issues down here: https://github.com/asciidoctor/asciidoctorj/issues/375#issuecomment-129458813

we have no CI for windows for jruby

Why not setup AppVeyor. It's just as easy to get setup as Travis CI. We use it for AsciidoctorJ. It doesn't catch everything, but it at least gives us some idea of what to expect.

@mojavelinux AppVeyor is actually on its way but there are tons of tests which will not work on windows and needs to be excluded and probably loads to work to get test parity between windows and *nix

Understood.

I wrote my issues down here: #375 (comment)

Can you assign #375 to me? I might be able to see if we can do something about it at the #GGX hackergarten.


Feature request: option to not crop a source

It would be nice to have the option (not on by default) to have a source fit into a cell of the grid w/o cropping (when using multiple cells for a single source). This would cause some extra black space for oddly shaped sources but would at least allow you to show the whole source

The current workaround is just keep assigning cells until the stream fits. Type the short code into each cell you want to span.

Great idea! This would be relatively easy to implement: the video tag gets
"object-fit: cover" applied to it currently. Applying "object-fit: contain"
would produce the uncropped result. I'll get this on the roadmap tomorrow.
It'd be a great first ticket for someone 🙂
On Fri, Jun 26, 2020, 12:24 AM Tim Greiser<EMAIL_ADDRESS>wrote:

The current workaround is just keep assigning cells until the stream fits.
Type the short code into each cell you want to span.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/chromakode/streamwall/issues/13#issuecomment-650023454,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAED7NGIUTGEN473KYXJRDRYREKHANCNFSM4OI7DXYQ
.


BREAKING CHANGE: xSQLServerRole The mandatory parameters now include SQLServer and ServerRole

Details of the scenario you try and problem that is occurring:
xSQLServerRole: The mandatory parameters now include SQLServer and ServerRole. The new parameter are Members, MembersToInclude and MembersToExclude.
The DSC configuration that is using the resource (as detailed as possible):
N/A
Version of the Operating System, SQL Server and PowerShell the DSC Target Node is running:
N/A
Version of the DSC module you're using, or 'dev' if you're using current dev branch:
N/A

Fix suggested in PR #414


[GHSA-2hvr-h6gw-qrxp] Cargo extracting malicious crates can fill the file system

Updates

CVSS
Severity

Comments
The Security Response WG intentionally marked this vulnerability as "low" severity rather than going with the CVSS assessment, with the following reasoning:

Note that by design Cargo allows code execution at build time, due to build scripts and procedural macros. The vulnerabilities in this advisory allow performing a subset of the possible damage in a harder to track down way. Your dependencies must still be trusted if you want to be protected from attacks, as it's possible to perform the same attacks with build scripts and procedural macros.

This updates the severity of this advisory to reflect the text of the advisory.

Hi there @pietroalbini! A community member has suggested an improvement to your security advisory. If approved, this change will affect the global advisory listed at github.com/advisories. It will not affect the version listed in your project repository.
This change will be reviewed by our highly-trained Security Curation Team. If you have thoughts or feedback, please share them in a comment here! If this PR has already been closed, you can start a new community contribution for this advisory


Variable units on plots

Request for variable units on plots
From https://hirlam.org/trac/wiki/Meetings/HarmonieWorkingWeek/UseofObsandALGO202010/Day2Diagnostics

Done. This has been made available in obsmon v4.1.0.


[Proposal] Dynamic prioritization and laning

Motivation
(Side note- at various points in this proposal I talk about 'historicals', but by that I mostly mean 'historicals and any other nodes that brokers can fan out to, including task peons'. It would have been a mouthful to type out every time.)
Clusters sometimes have heterogenous workloads: imagine both 'light' queries that could run at interactive speeds and 'heavy' resource-intensive queries. Busy clusters with heterogenous workloads can become unresponsive, even to 'light' queries, when cluster resources are all tied up processing 'heavy' queries. It's frustrating for end users who are accustomed to 'light' queries typically running quickly. Starvation can happen for multiple kinds of resources:

The @Processing thread pool on historical nodes (druid.processing.numThreads). Druid does make an effort to have each processing task be bite-sized (each one just processes one segment), so even though they are not preemptible, they are amenable to prioritization. Currently, query prioritization (setting priority in the query context) only does one thing: control the execution order of tasks in the processing pool. When priorities are set properly, this is reasonably effective at avoiding starvation.
The HTTP server thread pool on historicals and brokers (druid.server.http.numThreads). The model here is that each query gets one thread.
The merge buffer pool on historicals and brokers (druid.processing.numMergeBuffers). Each groupBy query needs one of these on historicals, and may need some on brokers as well.
The HTTP client connection limit from brokers to historicals (druid.broker.http.numConnections). Each connection can only carry one query at a time.

Items 2, 3, and 4 are allocated per-query, meaning that they effectively act as limits on the number of concurrent queries that can run. If each historical node has, let's say, 30 HTTP server threads, then no more than 30 queries can run concurrently at a time on that server. If all 30 of these queries are 'heavy' then they will starve out 'light' ones.
In practice, the ways that people prevent starvation of the resources of items 2, 3, and 4 are to either set the limits high enough that they exceed the number of 'heavy' queries that might run at once, or to try to separate 'heavy' and 'light' queries onto different nodes (by issuing heavy vs. light queries to different brokers, and using historical tiers to create a tier that only handles 'light' queries). Neither of these is really a very satisfying approach, since it means cluster architecture must be closely adapted to the expected query workload, creating more work for cluster operators and reducing stability when workloads change unexpectedly.
There is one more problem: even though query prioritization (1) is effective at preventing starvation of the @Processing pool, it can be difficult to set priorities appropriately. It's generally doable when end users are accessing Druid through an API layer that is under tight control. But when end users access Druid directly, or are accessing Druid through a third party UI like Superset or Looker, it is not likely that priorities will be properly set.
Proposed changes
Concept
The idea is to establish dynamic prioritization and laning.
Dynamic prioritization (adjusting query properties automatically) is meant to address the problem that end users are not always in a good position to be able to set priorities effectively. Laning, as in fast and slow lanes, is meant to cap the number of concurrently-running low-priority queries, and thereby reserve guaranteed resources for high-priority queries. This makes query prioritization effective at preventing starvation across the entire query stack, not just in the @Processing thread pool.
New properties

Property
Description
Where?
Default

druid.broker.priority.periodThreshold
An ISO8601 period. Queries whose intervals fall outside [now - periodThreshold, ∞) will have their priority adjusted by periodAdjustment.
Broker
null (ignored)

druid.broker.priority.periodAdjustment
Amount to adjust priority for queries who fall outside the periodThreshold. Negative numbers mean lower priorities and positive numbers mean higher. Generally a negative number would make more sense.
Broker
-1

druid.server.http.maxLowPriorityThreads
Maximum number of HTTP server threads to devote to queries with negative priority. This parameter must be set lower than all the limits in items 2, 3, and 4 above in "motivation" in order to prevent starvation.
Broker, Historical, Task
Integer.MAX_VALUE

Altered query behavior
When a query comes in with priority less than zero, and the low-priority pool is full (there are maxLowPriorityThreads already running), reject the query with an HTTP 429 "Too Many Requests" code. Clients would be expected to do automatic backoff and retry. HTTP 503 "Service Unavailable" might be a more appropriate return code, but a less-common code like 429 makes it easier to know when to trigger the backoff/retry loop in Druid clients.
Don't ever reject queries with priority zero or higher.
This ensures that queries with non-negative priority cannot be starved out by queries with negative priority.
Rationale
For dynamic prioritization, the goal in this proposal is to keep it as simple as possible while still being useful. I think basing it totally off the interval does that.
For laning, while arriving at the idea to reject low-priority queries, I thought of a few other options and decided against them:

Queue up queries past maxLowPriorityThreads rather than rejecting them. I didn't propose this since Druid's HTTP server design is thread-per-request, so a server thread is allocated upfront, and the only way I could see to give up the thread is to abort the query. The HTTP server could potentially be refactored to make it possible to queue up queries without hogging threads, but I think that would make more sense as future work. Of course, at some point, queries would still need to be rejected, since you can't queue an infinite number of requests.
Allow low-priority queries to run past maxLowPriorityThreads, but preempt them if higher-priority queries come in. This is how the @Processing thread pool works, but I don't think it'd work for the other levels of the query stack, since things like HTTP threads, merge buffers, and broker -> historical connections are allocated upfront and cannot be reclaimed without aborting the query.

Operational impact
The new features would be optional and off by default, so there will be no operational impact to people that don't want to turn them on.
Test plan
This sort of thing would need to be tested on real clusters to provide some assurance that it is effective. I have a few in mind that would be good candidates.
Future work

Improving the cost function that differentiates 'heavy' and 'light' queries, possibly by taking into account the number of rows, number of columns, expected filter selectivity, amount of computation estimated per row, and so on.

Queue up low priority queries somehow, rather than rejecting them.

Some way of setting the low-priority concurrency limit that is easier than druid.server.http.maxLowPriorityThreads. Setting a number of threads correctly requires cluster operators to think about many other different parameters on multiple node types.

I really like this proposal!
For context: we plan to make large interactive queries more responsive by running many low-priority background queries in order to populate a remote cache such as Redis or Memcached. This works for us because many of our large interactive queries are predictable, while smaller interactive queries are less predictable but are fast anyway so don't need to be cached.
Laning will ensure that these background queries don't impact interactive queries, and responding with 429 will make it much easier to throttle the submission of background queries in our caching service, so I love that.
Could you perhaps elaborate a bit more about why the dynamic prioritization scheme based on periodThreshold makes sense, as opposed to a durationThreshold for example? In our case, users may issue interactive queries that have a small interval duration (e.g. one hour) but are far in the past (e.g. six months ago).

Sounds great.
Instead of using the interval to decide how to change the priority of the query, what do you think about using the number of segments that the query operates over? Appreciate that this mechanism is to be expanded upon in future work but in the short term I think this works better because it more directly correlates with the amount of work a query has to do.
E.G.
Query A queries a huge datasource with 10 segments per-hour over 6 months, this requires ~40k segments to be queried.
Query B queries a relatively sparse datasources that's been compacted nicely to have a single segment per day over 6 months, this requires ~200 segments to be queried.

@peferron,

Could you perhaps elaborate a bit more about why the dynamic prioritization scheme based on periodThreshold makes sense, as opposed to a durationThreshold for example? In our case, users may issue interactive queries that have a small interval duration (e.g. one hour) but are far in the past (e.g. six months ago).

I was thinking a period threshold is likely to align with how people often set up historical tiers: a 'hot' tier for the latest 30 days and a 'cold' tier for older data.
@Dylan1312,

Instead of using the interval to decide how to change the priority of the query, what do you think about using the number of segments that the query operates over? Appreciate that this mechanism is to be expanded upon in future work but in the short term I think this works better because it more directly correlates with the amount of work a query has to do.

Well, same :)
Maybe it's worth creating more than one threshold even in the first version. It sounds like there are is demand for ways of thinking about prioritization beyond a 'hot' / 'cold' tier setup. How about starting with the three that have been suggested: period (from now), duration (of the query interval), and number of segments? In this case, if a user sets multiple, I'd suggest applying the adjustment once, if any of the thresholds triggers. I don't see a reason that a query that only triggers one threshold to be prioritized above one that triggers two. Concretely the properties would be,

druid.broker.priority.periodThreshold
druid.broker.priority.durationThreshold
druid.broker.priority.segmentCountThreshold
druid.broker.priority.adjustment

I think we'll want to add number of rows in the future, but that information is tougher to come by right now (the broker doesn't cache it unless Druid SQL is enabled, and it only lives in the SQL module). So I would still leave that one for the future.

I was thinking a period threshold is likely to align with how people often set up historical tiers: a 'hot' tier for the latest 30 days and a 'cold' tier for older data.

Got it. It makes sense if the goal is to be an alternative to tiering in certain situations, as you mentioned in your proposal.
Some of the confusion, I think, is coming from the fact that the proposal matches "light" with "high priority", and "heavy" with "low priority", but then goes on to propose a period threshold that's intuitively not the best way to achieve that: heavy queries hitting recent data will remain high priority, while light queries hitting older data will be adjusted down to low priority.
That's why the comments went on to suggest thresholds that are intuitively better correlated, such as duration or segment count (and we could keep going until we end up applying ML to it).
Let me ask you this: if there was an omniscient function available in the broker that could exactly predict how light or heavy a query is going to be, would you remove the period threshold, or leave it in as an optional configuration option?
If you remove it, then this proposal is indeed about classifying light vs heavy queries.
If you leave it in, then this proposal is rather about letting users dynamically adjust priorities based on abitrary criteria, with light or heavy just being one of them; and it's OK to sometimes classify a heavy query as high-priority and vice-versa.
I hope this doesn't sound like nitpicking—it would be helpful (at least to me) to clear this up.
BTW, my suggestion of a duration threshold was only as a review comment—in our case, we fully control the query priority and don't have any need for dynamic prioritization, so laning is the only useful feature for us here. My selfish interest would even be to ship laning first and push dynamic prioritization to a follow-up PR. 😛

Let me ask you this: if there was an omniscient function available in the broker that could exactly predict how light or heavy a query is going to be, would you remove the period threshold, or leave it in as an optional configuration option?

It depends on how omniscient the function is :)
Depending on how omniscient the function is, I still think there'd be value in overriding its decisions. Imagine you have a 'hot' and 'cold' tier and you really do want to apply a priority penalty to anything that'd hit the 'cold' tier, even if it looks like a cheap query. The reason might be because you have set up the cold tier to be storage-dense, and even seemingly-cheap queries might take a long time to run there due to needing to swap data in and out of disk. And you want those to go in the 'heavy' lane even if they look cheap.

If you leave it in, then this proposal is rather about letting users dynamically adjust priorities based on arbitrary criteria, with light or heavy just being one of them; and it's OK to sometimes classify a heavy query as high-priority and vice-versa.

I think this is the best way to think about what I'm proposing. The main motivation for dynamic prioritization is to differentiate heavy vs light queries, but I don't think that's going to be the only way people will want to do it. Maybe people will want to adjust priorities for specific datasources, or queries from specific users, or whatever.

BTW, my suggestion of a duration threshold was only as a review comment—in our case, we fully control the query priority and don't have any need for dynamic prioritization, so laning is the only useful feature for us here. My selfish interest would even be to ship laning first and push dynamic prioritization to a follow-up PR. 😛

That's a good point, the two features don't need to be done at the same time, and probably shouldn't be. (Although there is value in designing them together to make sure they align with each other.)

Since HTTP threads don't do any serious work on historicals (and indexing nodes?) (it's all delegated to @Processing threads) could we move to async in this particular part of the system and eliminate the druid.server.http.numThreads option?

I think we could, but it should be a different proposal. Even in an async http world, we still need some kind of limits on how many concurrent queries can be accepted simultaneously, since each one does need to allocate per-query memory for merging / etc and there is not an infinite supply. So the stuff in this proposal would still be useful, since we still wouldn't want to start an unlimited number of queries running at the same time.
I think the main thing I'd change about this proposal if async http existed is that instead of going straight to returning HTTP 429 when maxLowPriorityThreads queries are running, I would instead queue them up but delay them starting for a while. We will still need to reject at some point, but it'd be later. I'd probably also call the config maxLowPriorityQueries rather than maxLowPriorityThreads.

I LOVE LOVE LOVE the motion to work on dynamic prioritisation and laning and I also like these two terms.
Actually I was thinking about the same feature for a long time. I agree to your problem statement BUT I would model the problem quite differently. I believe to have an alternative proposal to make which would be similarly easy to implement but lead to a superior system behaviour (more automatic fairness, more control authority by admin, more self-healing) I think it might be looked at as being combinable into the existing proposal and some parts of my proposal below can be considered optional.

As was already stated, internally, Druid already has a query prioritisation mechanism, but out of the box, this priority can only be specified by a client as a static value.
Asking for a "dynamic prioritisation" is already implicitly formulating that there are several aspects that influence a query's priority.
(Some of these are intrinsic - meaning that Druid can compute them itself. Some are extrinsic - they are known only by an admin/client/user.)
(a) [extrinsic] the user context/intention

how urgent is the query
is the query originating from an interactive system or is it a report query that may take a long time.
how important is the user - CEO/VIP/customer/employee/backend-process

(b) [intrinsic] the query's weight

an estimation of how long it takes to execute or how much computational power is needed

(c) [intrinsic] the system state

how much of the query result is already cached
is the system idle or under heavy load

(d) [intrinsic] the system/user's history

how much strain has a user put onto the system recently. One might want to down-prioritise queries originating from a heavy-hitter over time or after a quota is reached to give other more modest users also a fair share.

Alternative Proposal
Let me say for each of the above how I would model them individually and then how those pieces could be put together:
(a) "the extrinsic context"
To model the extrinsic input, a query should be associated with a logical token that an admin can map to certain parameterizations. I'm used to calling such a logical token a "resource-pool" although I don't consider it a good name. Basically, instead of directly setting a "priority" inside of a query's context object, I would instead propose to specify a freely chosen logical resource pool name in the same manner.
{
    "type": "topN",
    "context": {
        "resource-pool": "interactive-customer-query"
    }
}

An admin can pre-configure resource-pools and then queries can refer to them. An unknown resource pool or missing declaration could then map to a "default" resource pool.
(b) "computing a query weight"

given an incoming query, a broker would first determine a weight for it.
one could make this mechanism pluggable to have a smooth migration path.
-- A simple strategy would assign each query the same weight.
-- A legacy strategy could set the query weight to be equal to the query priority set in the query context object.
-- A similarly simple strategy could look up a static query weight from a resource-pool.
(This would then be functionally equivalent to the current way of setting priorities in the query context)
-- Even a realistic query weight is rather straightforward and easy to implement:

    weight := #segments * #metrics * #splits * selectivity

    # stands for number-of
    selectivity means which fraction of records need to be scanned due to the given filter condition.

(c) "the system state"

similar to a query weight, the system state can also be modelled as another weight that can be combined with a query-weight and other inputs to form an eventual query priority.
this is where Gian's proposal plugs in.
(c1) "system is under heavy load"
--this increases the weight and ends up mapping more queries to low priorities.
(c2) "query results are partially cached"
-- if a certain fraction of a query is cached, this should reduce the query weight.
-- if a broker can determine which fraction of a query is cached it would reduce the query weight accordingly
query-weight := query-weight * %cached
-- the broker then sends the query along with its weight to the historical, which can optionally further discount the given query weight by observing it's local cache situation.

(d) "the system/user's history"

an additional weight can be computed based on a user's or resource-pool's query history this weight acts as a penalty.
a broker could keep a temporally-discounted sum over all query-weights per connection or per resource pool (or both)
due to a mathematical argument, there exists only a single temporal discounting function which is rational/consistent, which is exponential temporal discounting. This is nice two times over because a) it means one does not need to make this weighting strategy pluggable necessarily and b) the exponential function has nice computational properties which allow for very efficient incremental computation of an "eternal running sum"
temporal-weight := sum ( weight(query[i]) * exp( now() - time(query[i]) ) )

Putting those pieces together:

a client annotates a query with a resource pool name like "my-pool" instead of directly specifying a static priority.
a broker receives the query and computes a weight
weight := query-weight * system-weight * temporal-discount
broker sends query and weight to the historicals. Alternatively, the broker can turn the weight into a priority and send this.
historicals receive query along with weight and either treat the weight as a priority and follow their current scheduling logic, or alternatively, in a sophisticated setup, a historical could further modulate the weight based on its local conditions like "local load condition" or "local cache"
the broker also uses the resource pool name to lookup configuration attributes which can influence/modulate the weight computations, for instance:
-- base-offsets == laning
-- weighting factors
-- directives as to whether to engage or ignore a certain criteria or stage in the weight computation.
-- specification of which pluggable strategy to use.
-- half-life for exponential temporal discounting.
-- in short, any parameterization for the process of computing a dynamic query priority.

Although it might seem like a nice-to-have component, it is especially the incorporation of an exponential temporal discounting that would lead to a self-healing system that also has automatic fairness.
There would never have to any query that need to be rejected. Queries might simply get such high weights that it will be never their turn to compute which might eventually time them out.
Likewise, users pinging the system to death with massive amounts of queries will not be able to impact other more modest users.
All of the above might sound complex on paper but even if all got implemented, this would still be very straightforward and easy to do and would constitute a minor effort.
There are more sophisticated solutions, which allow a query to move from one lane to another one.
Lastly, although I agree that one might want to start small, I consider it useful and necessary to try to formulate a "true north" vision nonetheless - an ideal, fully fledged solution. One can then still implement only part of it, but it allows for validating whether that first step is well aligned with what potential later steps one might want to take.

@sascha-coenen, thanks for your feedback. After reading it, it sounds to me like most of it can be done as additions to my original (and intentionally minimalist) proposal, so I am not experiencing concern that the first step would be misaligned with a more sophisticated solution. But would you please let me know what you think, and in particular, what you think of the below comments?
On resource pools,
I like the resource pool idea as a way of incorporating extrinsic information. I think it maps nicely onto how people want to think about query priorities: numbers don't really make sense, but interactive or api or scheduled-report do make sense. It's more of a pain to configure, though. Rather than a single number on the query context, there's an indirection from the query context to an identifier defined somewhere else (broker configuration? metadata db?).
But even though it's a pain, I think it's a useful feature. In the context of my original proposal they would be a good place to define overrides for default priority and for the druid.broker.priority.* properties. (You might have more aggressive or more lenient thresholds for different kinds of queries.)
You're right about the name, it's not a good one based on how you've conceived it (it's not really a pool of resources, it's more like a way to group together queries that should be treated similarly).
I don't think they'd need to be added immediately, it makes sense to me to do them as a follow-on that lets you override the systemwide defaults.
On incorporating multiple sources of intrinsic information,
I think while my proposal is extraordinarily basic as far as what intrinsic information it incorporates (intentionally so, to keep the initial implementation simple), I hope more could be added later. I think all the stuff in your proposal could be done as follow-ons if we wanted. Since mine is so simple, it won't leave much legacy cruft behind that a more sophisticated system would have to deal with. Just the druid.broker.priority.* properties.
On rejecting queries,
I think this is the biggest area where I'm not sure how to align your proposal with mine.
The main conflict I see is that you envision not needing to reject queries, but I think we do need to reject queries, if only to prevent http server thread starvation. Due to this issue, I don't think we can do more than two 'lanes': a fast lane (queries in this category are always accepted for processing) and a slow lane (queries are rejected if the lane is 'full'). This constrains how beautiful the solution can be, since a binary decision must be made by the broker (reject a query, or accept it and begin execution).
By the way, http server thread starvation isn't the only resource that I think we need a binary laning system for: others include concurrent connections from brokers to historicals, and memory used for merging and storing partial results on historicals (both offheap merge buffers + on-heap structures). All of these resources are limited and must be reserved early in the query processing lifecycle, and cannot be released without canceling the query.

Two somewhat related issues: #8356 and #8357.

I'm planning to implement this proposal.
I think one potential area is of future improvement (that @sascha-coenen already alluded to), is allowing queries to move between lanes. For example, the heuristics mentioned may misidentify a heavy query as a light one, which may cause starvation of later light queries. In this case, the heavy query could possible be moved to the correct lane after it's consumed too many resources. Similarly, if a light query is initially misidentified as a heavy one, its priority would need to be corrected either before timing out or when it is retried later.

I believe that "Dynamic prioritization and laning" and "Query execution interleaving" (#8356) can help each other so the implementations for them should be better co-designed, even if the implementations will come in separate PRs.
More generally, this pair of issues and https://github.com/apache/incubator-druid/issues/4773 (to control which segments can and can not be in the file cache) should make separating Druid clusters into multiple tiers completely unnecessary.

On resource pools,
You're right about the name, it's not a good one based on how you've conceived it (it's not really a pool of resources, it's more like a way to group together queries that should be treated similarly).

True. How I got into referring to this feature as "resource pools" is that I was burrowing the name from Vertica. (https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/AdministratorsGuide/ResourceManager/ManagingWorkloads.htm?tocpath=Administrator's Guide|Managing the Database|Managing Workloads|_____0)
In Vertica one can define separate resource pools that have attributes like  how much bandwidth they may use etc. A user account can be associated with a resource pool and resource pools can be set up in such a way that a query can start in one resource pool and can be moved to a different resource pool on certain conditions like if it hasn't completed within a given amount of time.
This goes along the same lines of what ccaominh has described above.
Recently, Amazon announced "Automatic workload management and query priorities " for their AWS Redshift product:
https://aws.amazon.com/about-aws/whats-new/2019/09/amazon-redshift-announces-automatic-workload-management-and-query-priorities/
They even boast to use machine learning. I don't like the product but the naming "automatic workload management" is nice and it is also interesting that it has "automatic" in it, suggesting that resource management is actually something that doesn't require that much tuning and administration but something that kind of can be done almost in a mathematically optimal way.
I think the current Druid mechanism with query priorities is quite nice, but a priority that is 1 higher than another priority is already completely dominating the query prioritisation.
So whether using names or numbers to denote a priority, the prioritization should be gradual, not absolutist. Currently, as long as there are segments for a prio:10 query in the processing queue, no single segment for a prio:9 query will be processed. Instead I would think that a probabilistic interpretation of a numeric priority would be smoother. One could then put a higher-level construct on top of that. For instance a query analyzer which can compute how heavy a query is and can map that to a priority which makes sure that the query is neither allowed to starve nor too slow down interactive queries, simply by mapping it to the right numeric priority.
Such an approach would be nice in that it is such a generic concept that sophisticated features could be realised later on on that foundation. If on the other hand an "engineered" approach is taken as a foundation, like a fixed,  predefined number of lanes with fixed, predefined behaviours, then this limits how many later innovations could be put on top of it.
--

More generally, this pair of issues and #4773 (to control which segments can and can not be in the file cache) should make separating Druid clusters into multiple tiers completely unnecessary.
Interesting! Is it really true that tiers could be made "completely" unnecessary? We are using tiers for fail safety, having one tier running in one availability zone and a tier holding replicas running in another availability zone. Would k-safety and multi-AZ cases also be possible in a world without tiers?

@sascha-coenen

Is it really true that tiers could be made "completely" unnecessary? We are using tiers for fail safety, having one tier running in one availability zone and a tier holding replicas running in another availability zone. Would k-safety and multi-AZ cases also be possible in a world without tiers?

After #4773 is done (or even before that), it's pretty straightforward to implement the transparent lazy download of segments from the deep storage on Historicals upon they are first queried. This, in turn, opens up the technical possibility to implement spinning up Historical tiers which are exact clones of other tiers in a matter of minutes rather than dozens of minutes or hours.
When you are able to clone the whole Historical tier in a few minutes, I think keeping a backup tier in a different AZ is not optimal in almost all cases except the rare ones with very stringent reliability requirements (this is not typical for Druid which is used in analytics workloads which are rarely that critical).

True. How I got into referring to this feature as "resource pools" is that I was borrowing the name from Vertica. (https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/AdministratorsGuide/ResourceManager/ManagingWorkloads.htm?tocpath=Administrator's Guide|Managing the Database|Managing Workloads|_____0)

The Vertica resource pools are a bit more pool-y, they have things like MAXCONCURRENCY, MAXQUERYMEMORYSIZE, and PRIORITY that look like they correspond pretty closely to Druid concepts (see documentation for CREATE RESOURCE POOL).
The RUNTIMECAP + CASCADE TO mechanism doesn't correspond to any current Druid concept, but maybe it should. It is trying to solve the same problem that @ccaominh brought up.

Such an approach would be nice in that it is such a generic concept that sophisticated features could be realised later on on that foundation. If on the other hand an "engineered" approach is taken as a foundation, like a fixed, predefined number of lanes with fixed, predefined behaviours, then this limits how many later innovations could be put on top of it.

Would it really be that tough to make the system more flexible in the future? I'd imagine giving the fast and slow lanes special names and then allowing people to define more. We might have to deprecate a couple of config properties, but that doesn't seem too bad.

@gianm  I am looking forward to this new feature! :+1:

An ISO8601 period. Queries whose intervals fall outside [now - periodThreshold, ∞) will have their priority adjusted by periodAdjustment.

In addition, I also have some related thinking. Can we calculate the approximate elapsed time of the request before it actually executes? Then we can start adjusting the priority without waiting for the request runtime to reach the periodThreshold threshold.

Recently, Amazon announced "Automatic workload management and query priorities " for their AWS Redshift product

And similar to Amazon's WLM products, it feels that the introduction of machine learning will be a big trend, whether it is automatic load balancing in this proposal, or as a time series anomaly detection that may be developed in the future.


Adding model links for documentation #241

Adding model links for documentation #241

Looks really good.
To get civet to accept a PR you need to put the issue number in a commit. I did that here

Job VTB Documentation on 19032e4 wanted to post the following:
View the site here
This comment will be updated on new commits.


Code-completion, indexing, cursor-info, etc. for KeyPath dynamic member lookup

Add support for code-completion looking up keypath dynamic members, and cursor-info/indexing walking the underlying decls.
rdar://49028783
rdar://49028895
rdar://49029126

@swift-ci please smoke test

@swift-ci please test

Build failed
Swift Test Linux Platform
Git Sha - 8d4447c1d73f22377f5390a4cd7c852c2603639b

Build failed
Swift Test OS X Platform
Git Sha - 8d4447c1d73f22377f5390a4cd7c852c2603639b

@swift-ci please test Linux

Build failed
Swift Test Linux Platform
Git Sha - ae4827a437d85980f1975084eb21862b170516fa

Foundation Linux failure is unrelated (again).


segfault with current git-version

Looking at the last of 10 feeds, got a crash.
used gdb to look at corefile...
================================
(gdb) core core.1170
[New LWP 1170]
[New LWP 1171]
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff3f9fe000
Core was generated by `newsbeuter'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f47e7fcf230 in std::string::c_str() const () from /usr/lib/libstdc++.so.6
(gdb) backtrace
#0  0x00007f47e7fcf230 in std::string::c_str() const () from /usr/lib/libstdc++.so.6
#1  0x000000000041c955 in newsbeuter::htmlrenderer::render_table (this=0x7fff3f9f6420, table=...,
lines=std::vector of length 42, capacity 64 = {...}) at src/htmlrenderer.cpp:807
#2  0x0000000000419ff4 in newsbeuter::htmlrenderer::render (this=0x7fff3f9f6420, input=...,
lines=std::vector of length 0, capacity 0, links=std::vector of length 4, capacity 4 = {...},
url="http://www.mikrocontroller.net/feed/jobs") at src/htmlrenderer.cpp:494
#3  0x0000000000417e73 in newsbeuter::htmlrenderer::render (this=0x7fff3f9f6420,
source="\n


Testing versioning support

Purpose / why
#265 suggests to find a solution for versioning documentation on the gh-pages branch. This PR models how deployment of versioned documentation could work.
What changes were made?

Defines a Github Action for deploying documentation upon push to main branch
Deployment uses Mike for managing multiple versions of documentation (https://github.com/jimporter/mike)
Following setup guidelines provided in MkDocs Material theme documentation: https://squidfunk.github.io/mkdocs-material/setup/setting-up-versioning/
Test setup deploys to versioning-test branch, leaving gh-pages and Travis CI configuration alone.
This setup does not track releases/tags, but always deploys to 2.0.0 subdirectory, unless Github Action is changed manually.

Verification

Tested with fork of documentation in https://github.com/eldonquijote/documentation, published to https://eldonquijote.github.io/documentation/.
Might need own fork for testing the deployment to remote versioning-test branch.

Interested Parties
@Islandora/documentation

Checklist

Pull-request reviewer should ensure the following

[x] Does this PR link to related issues?
[ ] Does the proposed documentation align with the Islandora Documentation Style Guide?
[ ] Are the changes accurate, useful, free of typos, etc?
[ ] Does this PR update the last updated on date on the documentation page?

Person merging should ensure the following

[ ] Does mkdocs still build successfully? (This is indicated by TravisCI passing. To test locally, and see warnings, see How To Build Documentation.)
[ ] If pages are renamed or removed, have all internal links to those pages been fixed?
[ ] If pages are added, have they been linked to or placed in the menu?
[ ] Did the PR receive at least one approval from a committer, and all issues raised have been addressed?

This works! What are the adaptations before we could commit it?
One small suggestion: we could version it as 2.0 instead of 2.0.0 to follow the mike guidelines of dropping the patch number.

@rosiel As far as I can tell, for testing there shouldn't be any adaptations necessary, because it doesn't touch the Travis config and the gh-pages branch that Travis is creating. I would however prefer some sign-off by someone with experience with Github Actions and Travis, to make sure that there are indeed no unwanted interactions.
As for the version number, I'm fine using 2.0 instead of 2.0.0, that's just a matter of changing the Github Actions workflow definition.

So, I'm confused. Will this cause a new branch to be created (and used) on a permanent basis? Do we need the gh-pages branch too?

Also, does this mean we're doing Travis and Github actions? Is this a thing that could be combined into just using github actions? Not that it needs to be part of this PR, just trying to see the bigger management picture before we have branches that we're not sure if we can touch.

Yes, in the future there should be only one CI/CD tool, Travis or Github Actions, and only one branch to hold the rendered documents. But I want to test first if this works on this repository, with the permissions that we (I) have to configure things, and using the Materials for MkDocs Insider version that Danny forked for us (which we need to update, but that's another issue).

I'm good to go for it!

@eldonquijote I was wondering where this is at? Is this complete and waiting on a reviewer?

@DonRichards This is ready to go, but I've so far held off because we are not sure anymore if versioning the documentation is practical given that different modules we provide each move at their own pace. We might want to separate the question of versioning the docs from the switch to Github Actions (cf. #1914)

@eldonquijote Can we link the docs to a version "tag"? I would like us to move forwards and revisiti if possible. The Sprint is coming up and would like this to be in place to some degree.

@DonRichards Do you mean a version tag for the documentation, or the most current tag of the islandora module? The way I had set it up at one point requires to manually set a version number with the deploy command that the GitHub Action executes. But that was before the switch to semantic versioning and the decision to drop a version number for the stack as a whole.

I was thinking either added the semantic version (if not already there) or add a normal tag. But I am finding it difficult this morning to wrap my head around the difference. lol


API PDF export -> empty pages

Describe the Bug
When I export the books in PDF with the API, all the pages al empty.
All the other exports (markdown, HTML, plaintext) works
Steps to Reproduce

execute the API with the URL http://bookstack:6875/api/books/17/export/pdf
a PDF with all the pages white is received

Expected Behaviour
A PDF with the book is downloaded.
Screenshots or Additional Context
In the docker logs when Bookstack starts I see that the version is the latest:
Linuxserver.io version: v24.05.2-ls152
Build-date: 2024-07-01T18:27:53+00:00

But in the UI I see v22.11.1, do you know why?
Browser Details
No response
Exact BookStack Version
v24.05.2-ls152

But in the UI I see v22.11.1, do you know why?

That's odd, If you click the profile menu in the header do you see "My account" or do you see "Edit Profile" (or similar)?

Otherwise, does the export work okay in the application if you export the book as PDF? IF not, what about the page?
If you go directly to http://bookstack:6875/api/books/17/export/pdf in the browser as a user with API pemission, does the browser download a book export okay?

Damn, damn, damn! It's another problem not related to BookStack.
As soon as the PDFs are created I upload them to DropBox with the APIs. The problem lies there because I've just checked the local temp files and they are ok.
I'm sorry!
I still have the problem about the version. Am I using the latest or am I stuck at V22?

Am I using the latest or am I stuck at V22?

Sounds like you're currently stuck at v22.11.1.
Are you sure you're not accidentally running multiple containers at one? (old and new).


Try fix CI errors on Windows

Not sure why the failure was specific to Windows, but I think the reason is that these specs were parsing an invalid lockfile. Since the specs: section didn't exist, bundler was failing to parse the lockfile source,and then trying to use it while parsing specs, resulting in a method being called on nil.

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[ ] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[ ] Added tests.
[ ] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[ ] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

Thank you so much @deivid-rodriguez!


Resource nvidia.com/gpu Not Handled by ResourceConfig

In kedro-kubeflow 0.7.3, if gpu's are assigned in kubeflow.yaml using an nvidia.com/gpu element, they are not applied to the node as expected in the generated pipeline since they are not handled by the respective ResourceConfig class in config.py:
https://github.com/getindata/kedro-kubeflow/blob/0.7.3/kedro_kubeflow/config.py#L181

Thanks for reporting this @cupdike , we will fix it soon!

Sorry didn't mean to close it. I've added the support for custom labels in the resources. It's present in the current develop branch. Could you maybe try if this works for you and report back?

@cupdike have you been able to test out the fix provided by @Lasica ?


[MM-9321] Add a "+" button to make it easy to add Emoji Reactions

Summary
Add a "+" button to make it easy to add Emoji Reactions
Ticket Link
Resolves: #1293
Jira: https://mattermost.atlassian.net/browse/MM-9321
Checklist

[x] Has UI changes

Device Information
This PR was tested on: IOS, iphone 8 plus
Screenshots

Awesome @cpanato thanks for the PR


Overall Status of selected endpoints for status page

Feature that would be great if you can select monitors and have a overall uptime displayed for the past 30 Days
instead of each monitor listed

I don't think this is a Duplicate of #312? It sounds like it is asking for the ability to select a few monitors in the UI and see the sum up-time.
I am still going to cross reference #468 as this is about monitor groupings in general, and the logic used in implementing some of those would likely also be used here.


font or layer setitem

this used to work in robofab and can easily be added
aFont["a"] = aGlyph

Agree.

added here https://github.com/robofab-developers/fontParts/commit/e435b2b46f7a4ebf94f9bf6e86cbbbfdc93dc6f7


chore: release main

:robot: I have created a release beep boop
forms: 0.2.5
0.2.5 (2024-12-19)
Bug Fixes

card fields: correctly set isValid state for month and year (107887b)
docs validation: add empty document number check and update error messages (48f1903)
email confirmation: handle empty confirmation field and improve validation logic (b123bb0)

form-example: 0.0.7
0.0.7 (2024-12-19)
Dependencies

The following workspace dependencies were updated

dependencies

@tabnews/forms bumped from 0.2.4 to 0.2.5

This PR was generated with Release Please. See documentation.

:robot: Created releases:

form-example-v0.0.7
forms-v0.2.5
:sunflower:


dynamic_RNN fails on mac with unspecified batch_Size

tl;dr:
dynamic_rnn fails on OS X with unspecified batch_size but works fine on Ubuntu 14.04/Python3.4.3.  It seems that on OS X the 'None != "?"' evaluates poorly, then at some point in ops/rnn.py +918 this equality fails:
if const_batch_size != got_batch_size:
(i.e. None != Dimension("?"))
My best guess is that Dimension("?") is hacked in such a way that any comparison using it returns some fake None which for some reason on OSX + python 3.4 means true (I tried explicitly "if None:" as a sanity check, it evaluates as false).  For now I've modified the offending code to be:
if const_batch_size != got_batch_size:
if  (const_batch_size != got_batch_size) == got_batch_size or (const_batch_size != got_batch_size) == const_batch_size:
print("weird dimensions stuff in ops/rnn.py, what the shit python?")
continue
raise ValueError(
"Batch_size is not the same for all the elements in the input.")
This
Environment info
Operating System:
OS X 10.12
Python 3.4.1
Installed version of CUDA and cuDNN:
(please attach the output of ls -l /path/to/cuda/lib/libcud*):
N/A (CPU version)
If installed from binary pip package, provide:
0.11.0rc1 (just installed wheel)
also fails on 0.10.0rc1, 0.10.0 (though it works on 0.10.0 on Linux)
If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code).  It may work with less but this was a simpl
import tensorflow as tf
input_size = 256
length = 100
num_classes = 64
data = tf.placeholder(tf.float32, [None, length, input_size])
target = tf.placeholder(tf.float32, [None, length, num_classes])
max_length = int(target.get_shape()[1])
num_classes = int(target.get_shape()[2])
network = tf.nn.rnn_cell.GRUCell(200)
output, new_state = tf.nn.dynamic_rnn(network, data, dtype=tf.float32)
What other attempted solutions have you tried?
Works fine on my Linux box with or without GPU
Logs or other output that would be helpful
(If logs are large, please upload as attachment or provide link).
Traceback (most recent call last):
File "<my_directory>/tensorflow_mac_fail_test.py", line 18, in
output, new_state = tf.nn.dynamic_rnn(network, data, dtype=tf.float32)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/ops/rnn.py", line 836, in dynamic_rnn
dtype=dtype)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/ops/rnn.py", line 920, in _dynamic_rnn_loop
"Batch_size is not the same for all the elements in the input.")
ValueError: Batch_size is not the same for all the elements in the input.

It looks like you pretty much know what the problem is. Do you have a idea for a fix you could send?

we could replace the existing comparison with:
if const_batch_size != got_batch_size and not (const_batch_size == got_batch_size) == None:
(if the dimensions aren't equal but also neither are unspecified) but it seems quite awkward.  I feel like a cleaner solution might be ensuring that None != Dimension("?") evaluates to false instead of
None ==Dimension ("?")  => None, and the negation becomes true.

@vrv possibly for any additional insight

The fix seems reasonable to me, perhaps send a PR if you have one.

created patch https://github.com/tensorflow/tensorflow/compare/master...pfaucon:patch-1

Thanks! Feel free to send the PR.


Rewrite win-clipboard in Rust

It now uses clipboard_win crate, which proven to be well written and used in Mozilla Servo.
Windows x64 and x32 binaries are built (https://github.com/sindresorhus/clipboardy/issues/30)
🦄 Auto release on tags! (but may require to fix token – see instructions)

@sindresorhus check this out.

Binary sizes is kind of a bummer thou – gone from 17Kb to 113Kb.

This is neat and all, and I know this question is going to be met with some resistance -- but I have a burning question:
Why?

@Qix- it could fix broken line endings.

it could fix broken line endings.

Does it fix the broken line endings? Or is this hypothetical?

I'm all for this change! 🙌 The previous codebase was unmaintainable, as neither @floatdrop nor I know C very well. Now we can reuse a production-ready dependency and have Rust too, which gives some safety guarantees.

@floatdrop I think it's a good idea to merge it into one binary. Tests are currently failing though.

Does it fix the broken line endings? Or is this hypothetical?

For now it is hypothetical – . This should be tested within node.

Tests are currently failing though.

Will fix them soon 🙌

@sindresorhus ready to merge.

Nice work @floatdrop. Looks great 🙌

@floatdrop Can you do a new release? And document in the readme how to do a new release?

@sindresorhus you just need to push a tag and appveyor will create release and attach files automatically.

@sindresorhus actually you need to replace token in config (instructions here) and then push a tag.

Ok. Done, and I pushed a new tag. Let's see if it works.

Error creating GitHub release: Not Found

@sindresorhus that's strange. It worked in my fork.


merge package template from arlac77/npm-package-template

LICENSE

chore(license): add current year 2018

Codecov Report

Merging #155 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #155   +/-   ##
=======================================
  Coverage   76.19%   76.19%
=======================================
  Files           2        2
  Lines          21       21
=======================================
  Hits           16       16
  Misses          5        5

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c837727...13a5164. Read the comment docs.


Pseudo comments support

Closes #3276
Closes #3277

Sorry for the spaces and thanks for correcting them!

no worries, thanks, releasing it as new version in a bit


Update AboutUs.md

Codecov Report

Merging #14 (62a05c9) into master (852a4ef) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #14   +/-   ##
=========================================
  Coverage     70.96%   70.96%
  Complexity      429      429
=========================================
  Files            73       73
  Lines          1326     1326
  Branches        146      146
=========================================
  Hits            941      941
  Misses          344      344
  Partials         41       41

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 852a4ef...62a05c9. Read the comment docs.


Document test framework

Document the implementation and findings within this milestone.

Added documentation: https://github.com/strigiforme/portfolio/wiki/Continuous-Integration


Documentation 3.10/add topics for graph creations

Scope & Purpose
DocuBlocks only PR. This will add code examples for HTTP create graph API

@mchacki thanks for adding the examples to 3.10. Once the comments are addressed, i'll run a job to generate the updated examples for the 3.10 version.


Add LZ4 compression as an option

MessagePack-CSharp has shown that it is very fast and effective on msgpack payloads.

I'm exploring options in the lz4 branch and in https://github.com/MiloszKrajewski/K4os.Compression.LZ4/issues/105


Add a configuration guide to the documentation

Description
I've faced an issue of a lack of documentation of MeiliSearch configuration both in the official guide and in this repository's readme.
Basic example
I've needed to change the default payload size limit but found that there's no documentation that would help me. After some investigation I've found that there's an env file (/var/opt/meilisearch/env) that contains all environment variables which are loaded to the root's env by sourcing this file.
I think that such investigations could be avoided by having a dedicated section in the documentation.
Also, while trying to find configuration file, I've found paths for the DB and dumps - they could be documented too :)

Hi @regzon,
Thanks for raising this issue. We already have an open issue about the payload size in the issues of documentation repository here and there is a note about dumps in the documentation but it may not be easy to find.
I moved this issue to the documentation repository and checked with the team to create additional documentation in the README of all deployment repositories.


null latitude results in a lot of stacktrace log spam

I have a GNSS receiver on the Pi with the antenna suboptimally inside the cabin. The receiver often loses its location fix and reports null as navigation.position latitude and longitude. This then results in this stack trace being output every second:
Sep 17 11:59:21 openplotter signalk-server[207620]: Error: invalid float value for field 'lat': null
Sep 17 11:59:21 openplotter signalk-server[207620]:     at me.floatField (/home/pi/.signalk/node_modules/@influxdata/influxdb-client/dist/index.js:4:3366)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at toPoint (/home/pi/.signalk/node_modules/signalk-to-influxdb2/dist/influx.js:152:15)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at SKInflux.handleValue (/home/pi/.signalk/node_modules/signalk-to-influxdb2/dist/influx.js:119:25)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at /home/pi/.signalk/node_modules/signalk-to-influxdb2/dist/plugin.js:49:57
Sep 17 11:59:21 openplotter signalk-server[207620]:     at Array.forEach (<anonymous>)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at /home/pi/.signalk/node_modules/signalk-to-influxdb2/dist/plugin.js:48:48
Sep 17 11:59:21 openplotter signalk-server[207620]:     at Array.forEach (<anonymous>)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at /home/pi/.signalk/node_modules/signalk-to-influxdb2/dist/plugin.js:47:47
Sep 17 11:59:21 openplotter signalk-server[207620]:     at Array.forEach (<anonymous>)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at FullSignalK.<anonymous> (/home/pi/.signalk/node_modules/signalk-to-influxdb2/dist/plugin.js:45:39)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at FullSignalK.emit (node:events:525:35)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at FullSignalK.addDelta (/usr/lib/node_modules/signalk-server/node_modules/@signalk/signalk-schema/dist/fullsignalk.js:52:8)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at DeltaChain.doProcess (/usr/lib/node_modules/signalk-server/lib/deltachain.js:15:18)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at DeltaChain.process (/usr/lib/node_modules/signalk-server/lib/deltachain.js:11:21)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at Function.Server.app.handleMessage (/usr/lib/node_modules/signalk-server/lib/index.js:223:38)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at Simple.<anonymous> (/usr/lib/node_modules/signalk-server/lib/pipedproviders.js:59:17)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at Simple.emit (node:events:525:35)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at addChunk (node:internal/streams/readable:315:12)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at readableAddChunk (node:internal/streams/readable:289:9)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at Simple.Readable.push (node:internal/streams/readable:228:10)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at Simple._transform (/usr/lib/node_modules/signalk-server/node_modules/@signalk/streams/simple.js:122:8)
Sep 17 11:59:21 openplotter signalk-server[207620]:     at Simple.Transform._write (node:internal/streams/transform:205:23)

This amounts to hundreds of megabytes of syslog spam every day, filling the poor little disk up unbearably fast. Oh, and it seems the spam is duplicated in /var/log/daemon as well. :-o
Platform: OpenPlotter current-ish updates
Signal K server: 2.2.0
signalk-to-influxdb2: 1.3.4

https://stackoverflow.com/a/118154 😓

https://stackoverflow.com/a/118154 😓

To me, isNaN would only test for NaN values, not for something being a number. I had to think for a bit until I realized that the function name could indeed be interpreted differently, too.
Anyways, thanks for the fix!


Where is the data stored?

Im confused where the data is being stored...is it being stored in the ChatTriggers module?

Try referring to the docs or #9
Data is stored with mongoDB, make sure to install it
Make sure to press H in the calendar, and to run /uploadjacobdata. Also make sure that your CT and server secrets match


Volunteer Profile Editing Pages

Implement editing views of all the profile frames (availability)

/cib volunteer-profile-edit-pages

Branch 65-volunteer-profile-edit-pages created!


ARROW-2561: [C++] Fix double free in cuda-test under code coverage

As far as I can understand, the problem is due to both shared and static linking with libarrow.  Some static std::string in libarrow.so would be destroyed twice at shutdown.  Linking entirely statically seems to fix the issue.

Codecov Report

Merging #2048 into master will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2048      +/-   ##
==========================================
+ Coverage   87.44%   87.47%   +0.02%
==========================================
  Files         189      178      -11
  Lines       29368    28595     -773
==========================================
- Hits        25682    25014     -668
+ Misses       3686     3581     -105

Impacted Files
Coverage Δ

cpp/src/arrow/util/thread-pool-test.cc
98.87% <0%> (-0.57%)
:arrow_down:

rust/src/lib.rs

rust/src/error.rs

rust/src/builder.rs

rust/src/list.rs

rust/src/bitmap.rs

rust/src/datatypes.rs

rust/src/buffer.rs

rust/src/memory.rs

rust/src/array.rs

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 48eee79...50ccf9f. Read the comment docs.

Seems like there's an unrelated failure in test_parquet on AppVeyor:
https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/build/1.0.6470/job/hx95phkouafslhhp#L5889


Add INA219 current sensor driver

@jipanienko Can you please fix the copyright headers ? Also, please fix the STATS indentation.

@vrahane STATS indent adjusted to match other sensors

@jipanienko: I see what you mean, the code looks fine. Thank you.


Improved LoadTest to reproduce Github #157

original issue: https://github.com/dadhi/DryIoc/issues/157
original PR: https://github.com/dadhi/DryIoc/pull/162
Improved LoadTest to test different scenarios, Github 157 is not reproducible anymore which is great 👍

Cool.

@dadhi I think this pull request should be merged. It reproduces new problem which seems to be in the master only.
Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at DryIoc.Interpreter.InterpretGetNameScopedViaFactoryDelegate(IResolverContext r, MethodCallExpression callExpr, Object paramExprs, Object paramValues, ParentLambdaArgs parentArgs, Boolean useFec)
   at DryIoc.Interpreter.TryInterpretMethodCall(IResolverContext r, Expression expr, Object paramExprs, Object paramValues, ParentLambdaArgs parentArgs, Boolean useFec, Object& result)
   at DryIoc.Interpreter.TryInterpret(IResolverContext r, Expression expr, Object paramExprs, Object paramValues, ParentLambdaArgs parentArgs, Boolean useFec, Object& result)
   at DryIoc.Interpreter.TryInterpret(IResolverContext r, Expression expr, Object paramExprs, Object paramValues, ParentLambdaArgs parentArgs, Boolean useFec, Object& result)
   at DryIoc.Interpreter.TryInterpret(IResolverContext r, Expression expr, Object paramExprs, Object paramValues, ParentLambdaArgs parentArgs, Boolean useFec, Object& result)
   at DryIoc.Interpreter.TryInterpretAndUnwrapContainerException(IResolverContext r, Expression expr, Boolean useFec, Object& result)
   at DryIoc.Container.ResolveAndCacheKeyed(Type serviceType, Object serviceKey, IfUnresolved ifUnresolved, Object scopeName, Type requiredServiceType, Request preResolveParent, Object[] args)
   at DryIoc.Container.ResolveAndCache(Type serviceType, IfUnresolved ifUnresolved)
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved)
   at LoadTest.Program.<>c__DisplayClass3_0.<IterateInOrder>b__0()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

I extended the Load Test to resolve all controllers in single resolution call to make as long stack as I could. I tested that before merging changes from master test passed using WithoutFastExpressionCompiler -option. Now after merging latest changes to this branch its failing to null pointer.
The load test should pass using fast expression compiler to verify issues: https://github.com/dadhi/DryIoc/issues/157 and https://github.com/dadhi/DryIoc/issues/124 are not reproducible anymore.

will merge and check is still failing on CI


[GHSA-qqcv-vg9f-5rr3] litellm vulnerable to improper access control in team management

Updates

Affected products

Comments
This is not a current vulnerability. It's already been fixed. Incorrect report.

Hi @krrishdholakia, thank you for letting us know that GHSA-qqcv-vg9f-5rr3 is fixed. I would like to include a fix commit for readers to refer to. Do you know which commit in 1.40.28 is the fix commit?

Hey @shelbyc it was a lot earlier than v1.40.28. If you look at the issue thread, even they couldn't repro it.
We've had a couple refactors, so it's hard for me to point the fix, but here's the point in code which checks for this - https://github.com/BerriAI/litellm/blob/224148d6133ee50801cb129cbd21ccc213992e25/litellm/proxy/auth/user_api_key_auth.py#L1020

Hey @shelbyc it was a lot earlier than v1.40.28. If you look at the issue thread, even they couldn't repro it.
We've had a couple refactors, so it's hard for me to point the fix, but here's the point in code which checks for this - https://github.com/BerriAI/litellm/blob/224148d6133ee50801cb129cbd21ccc213992e25/litellm/proxy/auth/user_api_key_auth.py#L1020

Cool, thanks for the info! It looks like https://github.com/BerriAI/litellm/blob/224148d6133ee50801cb129cbd21ccc213992e25/litellm/proxy/auth/user_api_key_auth.py#L1020 was added in 1.40.15 so I set 1.40.15 as the patched version.

Thanks @shelbyc


Discussion: Out of sync: tools, tutorials, workflows

https://github.com/galaxyproject/training-material/pull/1480 initiated some discussion between @shiltemann and myself. That PR updates for the newest version of the tool's interface, but:

that assumes newest version in all servers. (UseGalaxy.* plans help with this a bit but not completely.)
and the workflow isn't updated / often forgotten (and sometimes it is a big request for a small/helpful change.)

If there is a nice way, to me it maybe makes more sense to pin tutorials to specific versions of tools? Then we have a tutorial + wf (+server) all in sync for at least that version. And when we go to update, it's somehow more mandatory to update the workflow since we need to know it works on that version (with the testing.)
I proposed in the past that maybe we could offer a way to say "I'm running this on server X" and then just create links in the tutorial going to the right tool + version on this server. That was one idea. Alternatively maybe we also include version info straight in tuto and teach users how to switch to that version?

Some discussion with @jmchilton, @bebatut, @shiltemann during the BioHackathon about a theoretical future where the training is part of the workflow in a single document.