[WIP] MSI Installer

This is some initial setup and prototyping work for the rustup MSI installer. I don't think this is ready to be merged (*), but I wanted to collect some feedback on the general structure and see how it works with the CI (but I haven't touched appveyor.yml yet).
I created the rustup-win-installer project within src, which is the library containing the custom actions, and the msi subdirectory contains the WiX/MSI source files. The WiX tools are invoked by a Powershell script (build.ps1) and the final MSI is placed into \src\rustup-win-installer\msi\target. I'm not sure whether it should actually be the other way round, because the WiX project depends on the custom actions library (and on the rustup binary, which is expected to be found at \target\release\rustup-init.exe).
The gfx is copied from rust-packaging.
(*) It currently only installs rustup.exe into %USERPROFILE%\.rustup-test\bin, because I don't know how to easily test it without overwriting my actual rustup installation.

There is another thing to be clarified here: I have added a feature flag msi-installed that triggers different behavior on self uninstall and (not yet) self update. This changes the SHA256 of the executable, which is currently used to check if an update is needed. I see a few ways to move forward:

Add a separate .sha256 file for the rustup.exe packed inside the MSI (this rustup.exe will only be available from inside the MSI).
Don't use a feature flag, but instead determine at runtime (e.g. by checking the registry key) whether the MSI installer has been used.
Change the logic to check for updates to not require a self hash.

:open_mouth: I will review soon.

OK, I'm so sorry it's taken me so long to get back to this @Boddlnagg!
This looks amazing. The directory structure you've laid out looks fine, for now at least. Perhaps at some point we can put all this stuff into a workspace so we don't end up dumping an extra target directory under src.
Once the msi installer is ready to go I expect it to be the only way to install rustup on windows. So ultimately we won't need to make a distinction. But we will need to get the msi-rustup deployed alongside non-msi-rustup in the interim while we test it and develop an upgrade path. A scheme that might work here is to name this new binary rustup-init-msi.exe, and we can call the hash rustup-init-msi.exe.sha256. Then we can modify the appveyor config to start producing this and deploying it alongside the current configuration.
It doesn't look like this iteration is invasive at all. Would it be reasonable to merge it as-is and keep iterating?
Again, I'm really sorry to block you on this for so long.

Oh, I may not understand the issues around the hash. Presumably the installation artifact once this is all done will be rustup.msi, and we would then have rustup.msi.sha256. There would be no rustup-init.exe at all. Is that right?
If that's the case then I would expect there to be no sha256 or verification of the executable at all, but instead to verify the hash of the msi. Does that make sense?

@brson: Yes, the installation artifact will be rustup.msi, but we can only compute the hash of the currently running rustup.exe, so rustup.msi.sha256 won't be useful for checking for updates. There will be no rustup-init.exe, only rustup.exe, which is packed inside rustup.msi. Since I would like self update to only invoke msiexec with a remote URL (i.e., the new MSI will be downloaded by msiexec), we can't use the hash of rustup.msi for verification (I think MSI has a built-in verification checking whether the file is valid, to safeguard against partial up-/downloads). So rustup.msi.sha256 will not be usable for anything.

@brson wrote:

It doesn't look like this iteration is invasive at all. Would it be reasonable to merge it as-is and keep iterating?

I would like to add the necessary changes to appveyor.yml first (without uploading artifacts, of course). But if you want to have that in a separate PR, that's fine with me.

Apparently the version of cargo currently used on AppVeyor does not yet know about the crate-type cdylib. Can we easily update to a newer version?

Ah thanks for clarification about the hashing issue. So in msi-world the only thing we need the hash for is determining whether an update is available. I'd say give it a unique name, like rustup-msi.exe.sha256. Perhaps before it's finalized we can come up with a scheme to transition it back to the existing rustup.exe.sha256 name. We might also (later) think about whether there is a better scheme for detecting updates since the unique value being a hash of the exe is completely arbitrary if we're not validating it.
That's funny that we're still pinned to such an old toolchain. There's no reason you shouldn't be able to switch it to stable (both travis and appveyor) in this PR.

I went ahead and opened a PR just to do the toolchain upgrade https://github.com/rust-lang-nursery/rustup.rs/pull/651

Unfortunately, cargo from 1.10 stable does not yet know about cdylib (though rustc does). I think 1.11 will be released soon, so this is probably not a big problem.

I wonder if it's really necessary to build for all four targets on AppVeyor. Which one is served at https://win.rustup.rs/ ? Right now I've configured it so that the MSI is only built on i686-pc-windows-msvc. I think MSVC is required because we're including some .lib files, and probably we don't want to distribute different MSIs for x86 and x64.
@brson wrote:

I'd say give it a unique name, like rustup-msi.exe.sha256. Perhaps before it's finalized we can come up with a scheme to transition it back to the existing rustup-init.exe.sha256 name.

Sound good to me. I thought about a possible transition strategy: When the MSI installer is ready, we can change the self update routine of the old version and have that use the MSI. I will try to make sure that the MSI installer just does the right thing in that case and won't delete already installed toolchains (should be easy).

And another comment (sorry ...): I already ran into issues with diverging Cargo.lock files. I think the only solution here is to use Cargo workspaces, which currently require a nightly. Maybe we should just update to a more recent nightly?

I wonder if it's really necessary to build for all four targets on AppVeyor. Which one is served at https://win.rustup.rs/ ? Right now I've configured it so that the MSI is only built on i686-pc-windows-msvc. I think MSVC is required because we're including some .lib files, and probably we don't want to distribute different MSIs for x86 and x64.

We could stop building all but one, though I like knowing they all build. i686-pc-windows-gnu is the one served by default. We use GNU by default because MSVC requires the CRT. MSVC would be better.

Update: I just noticed that this won't work because we can't expect users to be using the latest pre-MSI version of rustup which contains the updated upgrade logic.

I think we can. The old rustup will eventually upgrade itself to a rustup-init.exe that understands how to find the msi.

Maybe we should just update to a more recent nightly?

I'll upgrade to nightly.

We use GNU by default because MSVC requires the CRT

Rather, because MSVC dynamically links a version of the CRT that doesn't come with every version of Windows we support. The only thing that needs to not depend on a component that might not be installed is the installer itself. It can install the necessary bits though so rustup could be msvc just fine. If the installer has Rust code in it though, then it would be a bit tricky, possibly requiring https://github.com/rust-lang/libc/issues/290 to be fixed.

Ugh ... yes, there actually will be Rust code running inside the installer. So the recommended way is to use MSVC and link statically, but that's currently not supported according to @retep998? With GNU there is the problem that we want to (statically) link the WiX Custom Actions helper library wcautil.lib, and I don't know how to get an .a for it (or if that's possible at all). This is not strictly required though, as wcautil is just a wrapper around the msi library (no bindings yet, unfortunately) with some helper functions that could be ported to Rust.
One can always embed the merge module for the VC Runtime inside the MSI, but that won't help us here, because we run Rust code quite early during the installation procedure and the Runtime would not have been installed by then (also it would probably lead to problems on uninstall, when the VC Runtime is deleted before we've finished executing the Rust code). At least this allows us to install a dynamically linked MSVC version of rustup.exe, if that's preferred.
@brson wrote:

I'll upgrade to nightly.

Thanks. Should I setup a cargo workspace right away?

So the recommended way is to use MSVC and link statically, but that's currently not supported according to @retep998?

You can statically link the CRT just fine as long as you pass to the linker libcmt.lib and /NODEFAULTLIB:msvcrt, along with compiling all C/C++ code that is statically linked in with /MT.

(no bindings yet, unfortunately)

I'm not sure how soon I'll be able to get to this due to the winapi 0.3 overhaul I'm working on.

You can statically link the CRT just fine as long as you pass to the linker libcmt.lib and /NODEFAULTLIB:msvcrt, along with compiling all C/C++ code that is statically linked in with /MT.

rustup has quite a few dependencies, some of them native dependencies using gcc-rs, but gcc-rs can't pass /MT yet, right? (Before I came to Rust I was mainly using C# and don't have much experience with native toolchains ... I'm glad that everything is usually so easy with cargo.)

but gcc-rs can't pass /MT yet, right?

I doubt gcc-rs will be taught how to pass /MT until https://github.com/rust-lang/rfcs/pull/1684 is figured out.

@retep998 Okay, I was reading through that RFC thread and in search of a workaround I tried, based on a comment by @vadimcn (without actually knowing what I'm doing):

println!("cargo:rustc-link-lib=static=libvcruntime"); in build.rs
build the Custom Action cdylib with cargo rustc --release --target i686-pc-windows-msvc -- -Clink-args="/NODEFAULTLIB:vcruntime.lib"

This seems to work as DependencyWalker doesn't show a dependency on vcruntime140.dll anymore, but I don't really understand the implications. You said that "That basically breaks support for older versions of MSVC before the transition to ucrt and vcruntime" but this wouldn't be a problem here, since we can always require the latest MSVC version to build rustup on Windows.

@retep998 Using a local patched version of gcc-rs where /MT is passed instead of /MD is apparently not enough. I'm still getting multiple LNK4049 warnings e.g. for symbol _fclose, as well as LNK2001 errors e.g. for symbol __imp__stricmp. May this be due to dependencies on libc?
I don't know what the best way forward would be right now (as long as https://github.com/rust-lang/rfcs/pull/1684 is not resolved). I could try to use the MinGW toolchain, i.e. drop the dependency on the WiX CA helper library wcautil.lib and hope that MinGW's version of libmsi.a (used by winapi) is sufficiently complete and correct.

@Boddlnagg Trying to build on msvc to see the dependency issues. How is the workspace set up (it looks to me like the top-level Cargo.toml is not a workspace but I don't know much about it)? Are your patches to do the static linking pushed? It looks like no. Can you push another commit with those? In the meantime I'll try to reproduce based on your description of the solution.
Until we come up with a solution I think it's fine for us to merge with dynamic linking and just have it break when the crt dylib isn't available. We've got more iteration yet before we have to deploy.

How is the workspace set up (it looks to me like the top-level Cargo.toml is not a workspace but I don't know much about it)?

The top level Cargo.toml defines the workspace, see the [workspace] section. Everything else works automatically. It just means that Cargo.lock and target is shared with all the other crates.

Are your patches to do the static linking pushed? It looks like no. Can you push another commit with those?

I didn't push anything to do the static linking, because I can't get it to work. But it's really only about adding a few lines to build.rs and changing the build command to use cargo rustc with -Clink-args=...

Until we come up with a solution I think it's fine for us to merge with dynamic linking and just have it break when the crt dylib isn't available. We've got more iteration yet before we have to deploy.

So you would be okay with not being able to build on the GNU toolchain?

So you would be okay with not being able to build on the GNU toolchain?

As long as we continue to have GNU builds for the non-msi rustup-init.exe, yes. I think that's the situation here, right? It leaves all other build configurations intact, creating one new one for i686-pc-windows-msvc for the msied rustup.

Yes, that's right. Then you could merge this as-is. AppVeyor builds fine, although it prints some output in red because cargo sends status messages to stderr and PowerShell thinks that having anything written to stderr means that an error occurred :unamused:

OK. I'm going to go ahead and merge this as-is then. It's great progress.

The build failed, but totally unrelated, probably due to a change in the latest nightly.

Looks like the breakage will be fixed in the next nightly https://github.com/rust-lang/rust/issues/35721


Error Handling

We should notify users if they provide falsy arguments via console.warn.

Out of scope as well. I want the lib to remain as small as possible - providing documentation to compensate.


HBASE-25189 [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level

https://issues.apache.org/jira/browse/HBASE-25189

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m  6s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 27s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 20s
master passed

+1 :green_heart:
checkstyle
1m 46s
master passed

+1 :green_heart:
spotbugs
3m  9s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 54s
the patch passed

+1 :green_heart:
checkstyle
1m 47s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
23m 58s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

-1 :x:
spotbugs
2m 25s
hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

+1 :green_heart:
asflicense
0m 22s
The patch does not generate ASF License warnings.

55m 51s

Reason
Tests

FindBugs
module:hbase-server

Nullcheck of region at line 3079 of value previously dereferenced in org.apache.hadoop.hbase.regionserver.RSRpcServices.checkAndMutate(HRegion, OperationQuota, ClientProtos$MutationProto, CellScanner, ClientProtos$Condition, ActivePolicyEnforcement)  At RSRpcServices.java:3079 of value previously dereferenced in org.apache.hadoop.hbase.regionserver.RSRpcServices.checkAndMutate(HRegion, OperationQuota, ClientProtos$MutationProto, CellScanner, ClientProtos$Condition, ActivePolicyEnforcement)  At RSRpcServices.java:[line 3063]

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2549

Optional Tests
dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle

uname
Linux 5851475f316d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 38ac7e26ae

spotbugs
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html

Max. process+thread count
85 (vs. ulimit of 30000)

modules
C: hbase-hadoop-compat hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 29s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 21s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m  9s
master passed

+1 :green_heart:
compile
1m 24s
master passed

+1 :green_heart:
shadedjars
6m 42s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  1s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 16s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 58s
the patch passed

+1 :green_heart:
compile
1m 26s
the patch passed

+1 :green_heart:
javac
1m 26s
the patch passed

+1 :green_heart:
shadedjars
6m 35s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 59s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
0m 35s
hbase-hadoop-compat in the patch passed.

+1 :green_heart:
unit
144m 16s
hbase-server in the patch passed.

174m 49s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2549

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 13eaaaa8601b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 38ac7e26ae

Default Java
2020-01-14

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/1/testReport/

Max. process+thread count
3939 (vs. ulimit of 30000)

modules
C: hbase-hadoop-compat hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 27s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 21s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 35s
master passed

+1 :green_heart:
compile
1m 12s
master passed

+1 :green_heart:
shadedjars
6m 35s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 58s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 17s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 29s
the patch passed

+1 :green_heart:
compile
1m 14s
the patch passed

+1 :green_heart:
javac
1m 14s
the patch passed

+1 :green_heart:
shadedjars
6m 35s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 55s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
0m 33s
hbase-hadoop-compat in the patch passed.

+1 :green_heart:
unit
153m 17s
hbase-server in the patch passed.

182m  8s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2549

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 7105ff576615 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 38ac7e26ae

Default Java
1.8.0_232

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/1/testReport/

Max. process+thread count
4320 (vs. ulimit of 30000)

modules
C: hbase-hadoop-compat hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 44s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 19s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m  9s
master passed

+1 :green_heart:
checkstyle
1m 29s
master passed

+1 :green_heart:
spotbugs
2m 50s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 13s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 48s
the patch passed

+1 :green_heart:
checkstyle
1m 29s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
20m  8s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

+1 :green_heart:
spotbugs
3m 36s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 24s
The patch does not generate ASF License warnings.

49m 29s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2549

Optional Tests
dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle

uname
Linux 203f0a23e822 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 38ac7e26ae

Max. process+thread count
84 (vs. ulimit of 30000)

modules
C: hbase-hadoop-compat hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/2/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 29s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 18s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m  0s
master passed

+1 :green_heart:
compile
1m 24s
master passed

+1 :green_heart:
shadedjars
6m 41s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  0s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m  4s
the patch passed

+1 :green_heart:
compile
1m 24s
the patch passed

+1 :green_heart:
javac
1m 24s
the patch passed

+1 :green_heart:
shadedjars
6m 40s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  0s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
0m 35s
hbase-hadoop-compat in the patch passed.

+1 :green_heart:
unit
136m  6s
hbase-server in the patch passed.

166m 23s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2549

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux def2b0c9cc54 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 38ac7e26ae

Default Java
2020-01-14

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/2/testReport/

Max. process+thread count
4229 (vs. ulimit of 30000)

modules
C: hbase-hadoop-compat hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/2/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 45s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 21s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m 14s
master passed

+1 :green_heart:
compile
1m 22s
master passed

+1 :green_heart:
shadedjars
7m 23s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 58s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 56s
the patch passed

+1 :green_heart:
compile
1m 17s
the patch passed

+1 :green_heart:
javac
1m 17s
the patch passed

+1 :green_heart:
shadedjars
7m 13s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 56s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
0m 39s
hbase-hadoop-compat in the patch passed.

+1 :green_heart:
unit
207m 37s
hbase-server in the patch passed.

239m 59s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2549

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 7b4fcf4ef287 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 38ac7e26ae

Default Java
1.8.0_232

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/2/testReport/

Max. process+thread count
3474 (vs. ulimit of 30000)

modules
C: hbase-hadoop-compat hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2549/2/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.


Add Support for Worker Metrics and Queue Counts

hey all,
Thank you for making this package :)!!
We have a use case for tracking performance of each worker and each queue's job counts. We would like to scale up/down our instances using queue depth and/or performance of workers.
Would it be possible to add support for these?
Bullmq has metrics option that could be enabled on a worker. They also have a function queue..getJobCounts(). I am not sure how easy or hard it would be to integrate these. I am not very familiar with opentelemetry instrumentation otherwise I would push a PR
Thank you for your help!

I was just thinking about this kind of thing, actually. OTel JS says the metrics API is stable, so this is probably doable. But I'm not familiar with other auto-instrumentations that included metrics, so I'm not 100% sure how I would go about doing this.

I don't plan to spend any active development time on this project in the near future, so I'm going to close out this issue. If you're still looking for bullmq metrics, this project was mentioned recently, and it seems like a good option.
https://github.com/UpHabit/bull_exporter


Roanoke Virginia, Defund ROA Police Template

ROANOKE VA TEMPLATE.docx
ROANOKE VA TEMPLATE
To<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Subject: I Demand Divestment from Police and an Investment in Black Communities 🔗
Message (Don't forget to replace the [x]'s with your information!) 🔗
Hello, my name is [NAME] and I live in [CITY/NEIGHBORHOOD].
I demand:
Vote no on all increases to police budgets
Vote yes to decrease police spending and budgets
Vote yes to increase spending on Health care, education and community programs that keep us safe.
Black communities across the nation are mourning the deaths of George Floyd, tortured to death by Minneapolis police, Ahmaud Arbery, a jogger who was killed while running in a residential neighborhood in Brunswick, Georgia, Breonna Taylor an EMT killed while asleep in her bed in Louisville,KY, Dreasjon Reed in Indianapolis and Tony McDade in Tallahassee. Their names are added to a devastatingly long list of Black people who have been killed at the hands of vigilantes or law enforcement. Not to mention the others whose names we don’t yet know, and may never know since they were killed without a camera recording it.
At the same time, the United States leads the world in COVID-19 cases. So far, more than 100,000 people — enough to fill a football stadium– have perished from the virus, with over one million cases confirmed, and those numbers don’t reflect all the people dying from virus-related illnesses.
It is important to state this within the context of the scourge of anti-Black police terror and the resulting uprisings taking place across the U.S.
The COVID-19 deaths and the deaths caused by police terror are connected and consequential to each other. The United States does not have a national healthcare system. Instead, we have the largest military budget in the world, and some of the most well-funded and militarized police departments in the world, too. Policing and militarization overwhelmingly dominate the bulk of national and local budgets. In fact, police and military funding has increased every single year since 1973, and at the same time, funding for public health decreased every year, crystallized most recently when the Trump administration eliminated the US Pandemic Response Team in 2018, citing “costs”.
Black communities are living in persistent fear of being killed by state authorities like police, immigration agents or even white vigilantes who are emboldened by state actors. According to the Urban Institute, in 1977, state and local governments spent $60 billion on police and corrections . In 2017, they spent $194 billion. A 220 percent increase. The Roanoke city police budget for the fiscal year 2020-2021 has police spending at a total of $21,620,936. The police spending increased from fiscal year 2019-2020 to 2020-2021 by $489,426. They are spending 12,408,432 on police patrol. More money is going into our police than into the “Livability” sector. I am seeing substantial cuts in “Non-departmental” and “Outside Agencies” including small organizations such as the arts. Where our money goes matters.
I no longer want to see the police budget rise and community and public institutions paying the price. Despite continued profiling, harassment, terror and killing of Black communities, local and federal decision-makers continue to invest in the police, which leaves Black people vulnerable and our communities no safer. Police do not belong in our communities and especially not in our schools. What I ask for includes fewer patrols, reducing unnecessary civilian-police interaction, taking police out of schools, demilitarizing the tools and weapons that the police are equipped with.
I am demanding a defunding of the Roanoke police. This means divesting and investing: diverting the money to areas disproportionately affected by poverty and lack of infrastructure. This means redirecting resources into community-based initiatives: helping local grassroots groups that have been doing work in those areas and know how to engage with their communities. This means implementing non-police solutions to social problems in Roanoke: Affordable housing, affordable, accessible, and adequate mental and physical healthcare, employment, education. Where could that money go? It could go towards building healthy communities, to the health of our elders and children, to neighborhood infrastructure, to education, hiring mental health professionals and education professionals, to childcare, to support a vibrant Black future. The possibilities are endless.
These are all safety issues. Instead of punitive or criminalized, restorative and supportive community measures can address these issues. I call for more counselors, after-school programs, trauma services, anti-violence programs. I do not call for police reform, I instead call for defunding, divestment, and reinvestment into the community we all care for.
I join in solidarity with the freedom fighters in Minneapolis, Louisville, and across the United States.
Sincerely, The People of Roanoke, Virginia
[NAME] [PHONE_NUMBER] [EMAIL] [HOME/LIVING ADDRESS]

Reviewing now

Some minor edits:

To<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Subject: I Demand Divestment from Police and an Investment in Black Communities 🔗
Message (Don't forget to replace the [x]'s with your information!) 🔗
Hello, my name is [NAME] and I live in [CITY/NEIGHBORHOOD].
I demand that you, my local officials, will:

Vote no on all increases to police budgets
Vote yes to decrease police spending and budgets
Vote yes to increase spending on Health care, education and community programs that keep us safe.

Black communities across the nation are mourning the deaths of George Floyd, tortured to death by Minneapolis police, Ahmaud Arbery, a jogger who was killed while running in a residential neighborhood in Brunswick, Georgia, Breonna Taylor, an EMT killed while asleep in her bed in Louisville,KY, Dreasjon Reed in Indianapolis, and Tony McDade in Tallahassee. Their names are added to a devastatingly long list of Black people who have been killed at the hands of vigilantes or law enforcement. Not to mention the others whose names we don’t yet know, and may never know since they were killed without a camera recording it.
At the same time, the United States leads the world in COVID-19 cases. So far, more than 100,000 people — enough to fill a football stadium– have perished from the virus, with over one million cases confirmed, and those numbers don’t reflect all the people dying from virus-related illnesses.
It is important to state this within the context of the scourge of anti-Black police terror and the resulting uprisings taking place across the U.S.
The COVID-19 deaths and the deaths caused by police terror are connected and consequential to each other. The United States does not have a national healthcare system. Instead, we have the largest military budget in the world, and some of the most well-funded and militarized police departments in the world, too. Policing and militarization overwhelmingly dominate the bulk of national and local budgets. In fact, police and military funding has increased every single year since 1973, and at the same time, funding for public health decreased every year, crystallized most recently when the Trump administration eliminated the US Pandemic Response Team in 2018, citing “costs.”
Black communities are living in persistent fear of being killed by state authorities like police, immigration agents or even white vigilantes who are emboldened by state actors. According to the Urban Institute, in 1977, state and local governments spent $60 billion on police and corrections . In 2017, they spent $194 billion. A 220 percent increase.
The Roanoke city police budget for the fiscal year 2020-2021 has police spending at a total of $21,620,936. The police spending increased from fiscal year 2019-2020 to 2020-2021 by $489,426. They are spending $12,408,432 on police patrol. More money is going into our police than into the “Livability” sector. I am seeing substantial cuts in “Non-departmental” and “Outside Agencies” including small organizations such as the arts. Where our money goes matters.
I no longer want to see the police budget rise and community and public institutions paying the price. Despite continued profiling, harassment, terror and killing of Black communities, local and federal decision-makers continue to invest in the police, which leaves Black people vulnerable and our communities no safer. Police do not belong in our communities, and especially not in our schools. What I ask for includes fewer patrols, reducing unnecessary civilian-police interaction, taking police out of schools, and demilitarizing the tools and weapons that the police are equipped with.
I am demanding a defunding of the Roanoke police. This means divesting and investing: diverting the money to areas disproportionately affected by poverty and lack of infrastructure. This means redirecting resources into community-based initiatives: helping local grassroots groups that have been doing work in those areas and know how to engage with their communities. This means implementing non-police solutions to social problems in Roanoke: affordable housing, affordable, accessible, and adequate mental and physical healthcare, employment, and education. Where could that money go? It could go towards building healthy communities, to the health of our elders and children, to neighborhood infrastructure, to education, to hiring mental health professionals and education professionals, to childcare, to support a vibrant Black future. The possibilities are endless.
These are all safety issues. Instead of punitive measures or criminalization, restorative and supportive community measures can address these issues. I call for more counselors, after-school programs, trauma services, and anti-violence programs. I do not call for police reform. I instead call for defunding, divestment, and reinvestment into the community we all care for.
I join in solidarity with the freedom fighters in Minneapolis, Louisville, and across the United States.
Sincerely,
[NAME]
[PHONE_NUMBER]
[EMAIL]
[HOME/LIVING ADDRESS]

on it


Add support for Adeo LXEK-5 & ZBEK-26 color controllers

Add support (including device automations for all buttons) for Adeo HR-C99C-Z-C045 Zigbee remote for color lights control (fixes #889)

Models: HR-C99C-Z-C045 (white) / HR-C99C-Z-C045-B (black)
Zigbee IDs: LXEK-5 (white) / ZBEK-26 (black)
Zigbee Device Type: "Color Control (0x800)"
Brand: Lexman (Leroy Merlin - french home improvement store)
Manufacturer: Adeo

Codecov Report

Merging #1394 (a4ddaa4) into dev (f9e25f4) will decrease coverage by 0.53%.
The diff coverage is 81.81%.

@@            Coverage Diff             @@
##              dev    #1394      +/-   ##
==========================================
- Coverage   81.06%   80.52%   -0.54%
==========================================
  Files         218      223       +5
  Lines        6093     6453     +360
==========================================
+ Hits         4939     5196     +257
- Misses       1154     1257     +103

Impacted Files
Coverage Δ

zhaquirks/adeo/lxek5.py
80.95% <80.95%> (ø)

zhaquirks/const.py
100.00% <100.00%> (ø)

zhaquirks/tuya/ts0601_trv.py
81.48% <0.00%> (-4.94%)
:arrow_down:

zhaquirks/tuya/__init__.py
72.45% <0.00%> (-4.72%)
:arrow_down:

zhaquirks/tuya/ts0042.py
100.00% <0.00%> (ø)

zhaquirks/tuya/ts0044.py
100.00% <0.00%> (ø)

zhaquirks/sinope/light.py
100.00% <0.00%> (ø)

zhaquirks/sinope/switch.py
100.00% <0.00%> (ø)

zhaquirks/sinope/thermostat.py
100.00% <0.00%> (ø)

zhaquirks/tuya/mcu/__init__.py
100.00% <0.00%> (ø)

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9e25f4...a4ddaa4. Read the comment docs.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

36 of 44 (81.82%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.009%) to 80.521%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

zhaquirks/adeo/lxek5.py
34
42
80.95%

Totals

Change from base Build<PHONE_NUMBER>:
0.009%

Covered Lines:
5196

Relevant Lines:
6453

💛  - Coveralls

Added support for ZBEK-26 (new black model) which has identical Zigbee support  as LXEK-5 (white model), successfully tested with my devices!
PR finalized on my side and ready for integration, please let me know if something is wrong or missing :slightly_smiling_face:

Hello Matthieu
I just discovered your PR. Thank you for the job I was trying to do the same but I don't have the skill for it :)
btw, I get you code and add it as a custom quirk. It works well. but... the two button on the right of the remote (Arrow up / Arrow down) are not doing anything. For some reason nothing happened. No event in event listener ; nothin in the log.
Any clue ?
Thanks again for you contrib,
Hervé

Hello @herveja
Which model do you have ? Is it the white remote or the grey one ?
Can you confirm that the quirk is loaded for the device ?

For the debug, here is the configuration i use to be sure to have all the Zigbee logs :
logs:
  homeassistant.components.zha: debug
  bellows.zigbee.application: debug
  bellows.ezsp: debug
  zigpy: debug
  zigpy_cc: debug
  zigpy_deconz.zigbee.application: debug
  zigpy_deconz.api: debug
  zigpy_xbee.zigbee.application: debug
  zigpy_xbee.api: debug
  zigpy_zigate: debug
  zigpy_znp: debug
  zhaquirks: debug

hello @mathoudebine
Thank you for your answer
I have the grey one (Model: HR-C99C-Z-C045-B) and the quark is loaded.
I tried for a 4th time to join the device and this time all is good - every button trigger a command (except command "color_up" and "color_down" that seems to be not linked to any action).

here is the log when joining the device to HA https://pastebin.com/tcTydSb7

Good news!
I don't know why you had to pair the device multiple times though.
For the color_up/color_down commands, i cannot trigger them either with the white remote or the grey one.
I may have made an error in the device automation for these commands:
https://github.com/zigpy/zha-device-handlers/blob/df9bdca7ff597fe23aff8fe946630927f6e93bd7/zhaquirks/adeo/color_controller.py#L192
I will try to fix it when i have some spare time, or feel free to do so if you want!

hi @mathoudebine I've just bought the grey remote, after the 2nd pairing it works, except for the color_up/down commands.
What do you think is your mistake ? I may try to fix it on my own and submit a PR...
Thanks again for the initial support !

Same here, everything around the round cenral button doesn't bring up any events.


0.14 keyboard layout switches on non-alphanumeric symbol

Hello. I have the keyboard layout set in my sddm scripts:
export XKB_DEFAULT_LAYOUT=us,ru
export XKB_DEFAULT_OPTIONS=grp:caps_toggle,grp_led:scroll

export WLC_REPEAT_DELAY=150
export WLC_REPEAT_RATE=30

With that, when I'm on russian layout and enter the non-alphanumeric symbol, say, ',' or ')', the layout switches on it's own, starting to produce latin symbols. Though for sway (or libinput) it seems that layout's still 'ru', because it takes me two 'caps' presses to get back to 'ru'.
Couple friends of mine report the same behaviour.

Sounds like a libinput problem, I don't think this is a sway issue.

I agree. Can you reproduce this on other compositors or under xorg?

under xorg using xkb and i3 everything's fine.
may try to install other wm or de. will try kde later today or tomorrow
forgot to mention (not sure whether it's important or not), i'm using arch, my friend with the same issue is using fedora.

well, in plasma on wayland everything seems fine. no unexpected keyboard layout switches.
though i've set up the settings using kde settings, so might be different that way

Use the exact same XKB settings configured through the same environment variables on weston.


Fix crash in support design library(v.23.2.+)

Fix "View can not be anchored to the the parent CoordinatorLayout" error in new version support design library(v.23.2.+)

Sorry I didn't approve this PR earlier. There's a problem with the FAB not going back down after a Snackbar is shown, but it turns out it's a bug in the support lib. This happens only when animations are disabled, so this PR LGTM.


[ASK]How to run integrate with AffectedModuleDetector

Hi thanks for making this plugin,
want to ask, is it possible to override jacocoAggregation task to run runAffectedUnitTests so it will not run all of modules and buildTypes?

I'm not sure how AffectedModuleDetector works, but if it relies on forcing the skip of some test tasks based on GIT diff or similar approaches, I'd state it's a hacky approach.
You should always configure all your modules, and rely on Gradle's Build Cache instead. The UP-TO-DATE logic will do the magic at task level, including Tests tasks.


[Explore]add Annotation layers dropdowns do not respond for click

I cannot modify dropdown options for annotation layers (chart explore view)
Expected results
Dropdowns can be open.
Actual results
Dropdowns do not react on click.
Screenshots
https://user-images.githubusercontent.com/25153919/103898333-108fc800-50f5-11eb-92d5-2697cb2fd092.mov
How to reproduce the bug

Launch application with: CYPRESS_CONFIG=true docker-compose up
Go to chart Rise & Fall of Video Game Consoles
Click on Add annotation layer
See error

Environment
(please complete the following information):
commit master f48284909dde84d13276bb40ddd91ec6935f10e5
Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ x] I have reproduced the issue with at least the latest released version of superset.
[ x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

@adam-stasiak Thanks for reporting
@agatapst let's fix this one, I don't think it's a regression from your last PR, but you now have the most context.
thanks both! 🙏

@adam-stasiak Thanks for reporting
@agatapst let's fix this one, I don't think it's a regression from your last PR, but you now have the most context.
thanks both! 🙏

Fixed by #12330 @junlincc

Fixed by #12330 @junlincc

There might be another issue though. It seems the Popover is closed on every change without giving the user the ability to apply the changes @junlincc

There might be another issue though. It seems the Popover is closed on every change without giving the user the ability to apply the changes @junlincc

There might be another issue though. It seems the Popover is closed on every change without giving the user the ability to apply the changes @junlincc

right, that's also happening in the LONGITUDE & LATITUDE popover in deck.gl chart 😭

There might be another issue though. It seems the Popover is closed on every change without giving the user the ability to apply the changes @junlincc

right, that's also happening in the LONGITUDE & LATITUDE popover in deck.gl chart 😭

original issue is resolved thanks @geido and @adam-stasiak !

original issue is resolved thanks @geido and @adam-stasiak !


Destructured goog.require imports fail

The following input works fine if it's transpiled as a single file down to ES5 and then recompiled.
goog.module('x');
var {a,b} = goog.require('y');
console.log(a, b);

If it's compiled directly as ES6 input, it fails with the following error:
input0:3: ERROR - The local variable a is in a goog.scope and is not an alias.
var {a,b} = goog.require('y');
     ^

input0:3: ERROR - The local variable b is in a goog.scope and is not an alias.
var {a,b} = goog.require('y');
       ^

2 error(s), 0 warning(s)

Presumably this has to do with the way goog.module is rewritten as a goog.scope (though it's not clear why it would work in single-file mode).

As I recall, we intentionally placed the ScopedAliases pass after some of the early transpilation passes (specifically Es6RewriteDestructuring) so that it would not have to understand destructuring syntax. It looks like we got too aggressive at pushing the transpilation later and later, so we either need to reorder the passes or make ScopedAliases understand this syntax.

After some discussion, it seems the best approach is for goog.module to not use goog.scope at all. We have a TODO for this, in fact.

This is now in progress. No official ETA other than "soon". Apologies for the delay.

Yay @stalcup!


[Feature Request]: Improve the home teleport

Is your feature request related to a problem?
The home command currently teleports the user to the 0 0 0 coordinates of the block where the home was set. This creates 2 issues:

The player is teleported to the corner of 4 different blocks, potentially partially inside a wall or in a spot where there is no ground to stand on.
If the home was set on a slab or any other non-full block, the teleporting player might phase through the ground.

Solution(s)
Make the command teleport the player to the middle of the block (0.5 ~ 0.5) and remember the Y coordinates.
Describe alternatives you've considered
No response
Mod Version
forge-1.20.1-1.1.0
Mod Loader Version
47.1.44
Mod Loader
Forge
Additional context
No response

Can be closed by ffb797719d30586115c943cea158980ff9f9f918


Add Aerostats (fixed KerbalStuffBot submission)

Fixes #2896

Depends Community Resource Pack.Ready for merge once that's done.

If I comment as something ready for merge once X, you can merge it yourself after making X change and it passing Jenkins. I'm usually really bad about checking old PRs unless they show up in the notifications section of github (which requires a comment in them).

Ok, that makes sense.


Fix: compiled query resource writer assembly reference

Fixes the previously discussed missing assembly reference during code generation when using Marten CompiledQuery result writing.

@Blackclaws Thank you!
And yes, I know the CI needs love...


Text being cut

So, I recently installed ghidra with the purpose of replacing IDA (it costs 10k less!) but one thing that I cannot find anywhere is how to remove this "simplification" of text

The same thing occours for the symbol tree. How can I show the full text?

Check out #99.


how to ignore locale?

Hi, first at all thanks for this gem.
Im storing a diferent template for each view, but i dont want to specify the locale.
I have made 2 tests:

removed the field from the form -> template is not saved.
leave the field blank -> "error (not in the list)"

any ideas?

Hi @0910
according to this spec, templates without locales should be saved because valid. Btw, I'm releasing a (long awaited) update for this gem, v0.0.6. Try it and let me know :-)


Fix missing digest input and update attestation

This Pull Request:

adds the digest input
Fix: attestation generation (adds the missing subjects)
remove already set env variables
co-authored-by @jobroe10

We made these changes while using the container workflow and realising it was not fully working

@ianlewis what do you think?


Cached Images

In banners service I changed the images but url is same, means I replaced the images in directory but the app showing old images even after 1 day. Is it caching images, then how to remove it periodically or any other solution??

@MuhammadAmmad are you using custom backend solution? Otherwise the banners are downloaded from our BaaS.

Yes I am using custom backend.

If you reinstall the application, do you see the new images?

No, Images change after 1 or 2 days.

If reistalling the application doesn't help then the problem is not with the iOS application cache (reinstalling clears the cache).
If we change the banner image on our backend, the image in the application updates immediately.
I would double check the caching on your backend.

OK Thanks. I 'll figure it out.


choose network precision as float32

I am currently doing network profiling. When I run the inferencing model imagenet with jetson nano, the default precision option is FP16. Is there a way for me to manually set the precision option as FP32 for python applications?

@gaochuanchao sorry I don't believe I have exposed the python API for that, however you can try changing this code to the following and recompiling/reinstalling jetson-inference:
https://github.com/dusty-nv/jetson-inference/blob/644f5158a55a7f313f4d43b57ad3ed109fcbf9ab/c/tensorNet.cpp#L544
# change to this instead
return TYPE_FP32;


pretrained model?

could you release your pre-trained model ?

Sure we can! The pre-trained models are now available on Dropbox, and I have updated the README. Please check out the links there. Thanks for your advice.


Day / Week and Month costs are all zero

PLEASE DO NOT DELETE AND FILL IN!
Describe the bug
The integration's installation and setup went ahead very smoothly. I used the suggested sensors to create the dashboard.
Whilst the very same sensors report correctly on my own dashboard (another card), these are not working as expected on the HA Energy dashboard. They report the cost as zero!
The only way that I could get this to work is to change the view to Year (please see attached) apart from that either consumptions are missing (and there are some costs  OR) the costs are missing whilst there are some consumptions!
Please see attached
To Reproduce
Expected behavior
to see the actual costs per day (last day)
Home Assistant Version
Home Assistant 2023.6.3
Supervisor 2023.06.4
Operating System 10.3
Frontend 20230608.0 - latest
Integration Version
7.4.4
Fresh Install?
Yes, fresh install
Home Assistant Logs
Nothing to report :(

Hello and sorry that you're having issues. Unfortunately if the data is being displayed in the year view and present on the entity itself, then that suggests it's a HA issue not an integration issue. The only thing I can suggest is to double check the docs and make sure the pricing entities have been set in the correct modes.


2.26 release OpenCL support meta-issue

Summary:
This issue is to track what is yet to be done for OpenCL support in 2.26.

[x] extend the kernel generator to enable adding support for most distributions

@t4c1 did a ton of work here. The k.g. has been extended so that a lot of distributions have already been implemented with it. See "Supported distributions" in http://mc-stan.org/math/opencl_support.html

[x] add support for simpler distributions (_lpmf/_lpdf functions only)

See "Supported distributions" in http://mc-stan.org/math/opencl_support.html
Again, a huge effort by Tadej. We may add additional ones, but even supporting the listed ones will be a huge improvement.

[x] create the script to create OpenCL vs CPU benchmarks on-the-fly (#2260)

Tadej mentioned that this is close to being ready. This will help us in the future as well.

[x] run the benchmarks to identify distribution signatures that make sense to use with OpenCL

Most importantly, we need to flag the signatures for which the data transfer time is larger than the runtime on CPU. Those are obviously never going to be faster on the GPU.

[x] add support for OpenCL supported distributions to stanc3

A PR for that is open, we just need to eliminate signatures that will be flagged by the benchmarks. https://github.com/stan-dev/stanc3/pull/754

~implement stanc3 support for functions not returning scalars~

The first part will implement support for distributions, because for those, the return type is a scalar which will be returned to the CPU always. For cases like matrix multiply(matrix, matrix) its a bit more involved whether we need to return the result to the CPU or not. This is still in the works and may require a design doc.
We will add all functions that support both the prim and rev versions. A list of supported functions is given here: https://github.com/stan-dev/math/issues/1854

[x] Add and test OpenCL CPU support with zero-copy

In progress: https://github.com/stan-dev/math/pull/2182 and https://github.com/stan-dev/math/pull/2219
Mostly ready, just hashing out issues with test setup.

[ ] Add a OpenCL/GPU in Stan section/chapter to the user guide

Also clean up the rest of the function signatures docs (GLMs mainly) of GPU mentions. We want all OpenCL/GPU info in one place, not scattered around the documentation.

2.26 is becoming a huge release of all this stuff goes in(Opencl, expressions, varmat)!!!

Looks good. Couple minor comments:

Most importantly, we need to flag the signatures for which the data transfer time is larger than the runtime on CPU. Those are obviously never going to be faster on the GPU.

Never is a strong word. Are there any predictions how data transfer speeds develop in the future?

add rev and prim support for more functions

If this is a checklist for checking when this project is ready to be included in a release, then "more" is unclear. Is there more specific target or is any number large than one ok?

Never is a strong word. Are there any predictions how data transfer speeds develop in the future?

Good call. Yeah transfers will definitely get faster over time with pci express 4 and 5. Also the addition of high-end Intel GPUs that have a unified memory with the CPU and the possibiluty of zero copy might also have an effect. Will modify.

If this is a checklist for checking when this project is ready to be included in a release, then "more" is unclear. Is there more specific target or is any number large than one ok?

Good point. We want to have more, but we are already happy with what we currently have. Will change language here.
Also missed an inportant item: user guide/cmdstan sections.

Update:

benchmarks for distributions were done. See https://github.com/rok-cesnovar/stan-math-opencl-benchmarks/blob/main/summaries/radeonvii_vs_i7_distributions.md and the entire repo https://github.com/rok-cesnovar/stan-math-opencl-benchmarks
Stanc3 PR to add OpenCL support for additional distributions has been reviewed and should be merged today
OpenCL CPU support with zero-copy has been merged and is enabled

The missing parts are enabling other functions (non-lpdf/lpmf) that do not return scalars in stanc3 and docs - Cmdstan/Stan guide (this will most likely have to wait for the freeze). I think we are on track here.

Update:

benchmarks for distributions were done. See https://github.com/rok-cesnovar/stan-math-opencl-benchmarks/blob/main/summaries/radeonvii_vs_i7_distributions.md and the entire repo https://github.com/rok-cesnovar/stan-math-opencl-benchmarks
Stanc3 PR to add OpenCL support for additional distributions has been reviewed and should be merged today
OpenCL CPU support with zero-copy has been merged and is enabled

The missing parts are enabling other functions (non-lpdf/lpmf) that do not return scalars in stanc3 and docs - Cmdstan/Stan guide (this will most likely have to wait for the freeze). I think we are on track here.

implement stanc3 support for functions not returning scalars

This was not implemented for two reasons: we ran out of time for an near-optimal stanc3 version of this and a non-optimal one does not make much sense without varmat support which was abandoned for 2.26. I do still take the expanded distribution support as a huge win. The user-facing docs will be done during the freeze.

implement stanc3 support for functions not returning scalars

This was not implemented for two reasons: we ran out of time for an near-optimal stanc3 version of this and a non-optimal one does not make much sense without varmat support which was abandoned for 2.26. I do still take the expanded distribution support as a huge win. The user-facing docs will be done during the freeze.


waterline assumes all ids are integers, so non-integer ids (as in couchdb) are converted to NaN

fails test
  1) Semantic Interface .destroy() with numeric ID should destroy a record:

Issue was with couchdb adapter, specifically


Refactor SMS Delivery Failure sign in flow

Explanation of Change

Fixed Issues

$ https://github.com/Expensify/App/issues/53980
PROPOSAL: https://github.com/Expensify/App/issues/53980#issuecomment-2537759035
Tests
Same as QA Steps

[x] Verify that no errors appear in the JS console

Offline tests
Same as QA Steps
QA Steps
Test 1 (Simulate SMS Failure):

Log in with an existing account.
On the login page with the magic code request, type Onyx.merge("account", {smsDeliveryFailureStatus: {}}); in the console.
Verify that the smsDeliveryFailurePage is displayed correctly.

Test 2 (Simulate Validate Fail):

Repeat steps 1-3 on Test 1
Click the "Validate" button and type Onyx.merge("account", {smsDeliveryFailureStatus: {isReset: true, hasSMSDeliveryFailure: true, message: "Please wait 23 hours and 37 minutes before trying again."}}); on the console
Verify that the message is displayed correctly
Click the "Got it" button
Verify that the user is navigated to the beginning of the sign-in flow

Test 3 (Simulate Validate Success):

Repeat steps 1-3 on Test 1
Click the "Validate" button and type Onyx.merge("account", {smsDeliveryFailureStatus: {isReset: true, hasSMSDeliveryFailure: false}}); on the console
Verify that the message is displayed correctly
Click the "Send" button
Verify that the BeginSignIn API called
Type Onyx.merge("account", {smsDeliveryFailureStatus: null}); on the console
Verify that the  user was able to continue the sign-in flow and navigate to the magic code request

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I used JaimeGPT to get English > Spanish translation. I then posted it in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.ts or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
https://github.com/user-attachments/assets/8bbba9dd-ba39-4d52-a7c0-6bef68d92a47

Android: mWeb Chrome
https://github.com/user-attachments/assets/d361cee4-7b32-4024-9bcc-8e55e95a45ec

iOS: Native
https://github.com/user-attachments/assets/6c09bd3a-6810-497f-88ba-2f15c4961529

iOS: mWeb Safari
https://github.com/user-attachments/assets/bc72e346-e618-4f78-a8b2-fb0782b02f2e

MacOS: Chrome / Safari
https://github.com/user-attachments/assets/979e13d9-fdeb-4cbe-9025-eabd501f58d9

MacOS: Desktop
https://github.com/user-attachments/assets/3d070eda-a621-4961-853b-d7d64d9d0784

Have we deployed BE PR yet? I'm still seeing the ValidateScreen on the very first signin
https://github.com/user-attachments/assets/856e26d4-e75b-4ece-bc63-e8f418894bca

@carlosmiceli  Please take a look at my comment above: https://github.com/Expensify/App/pull/54400#issuecomment-2556184576

Have we deployed BE PR yet? I'm still seeing the ValidateScreen on the very first signin

Sorry, was OOO at the end of last week. We are still troubleshooting the BE, so that' why it's still not working as it should 🙏


Add support for Baidu's new sql db -- Tera

Baidu recently released a scalable, high-performance distributed database, called Tera ( https://github.com/baidu/tera ). Tera is Baidu's largest new sql database, which is used to store all web pages and link's meta data, and  there is trillion of data store in tera untill now. Tera is written in c++, and its design inspires a lot from BigTable. So I think it is necessary for ycsb to add support for tera.

@caijieming-baidu feel free to put together a PR.

i am working on it.  :-)


Support deploying to a slot and making it live immediately

Please note I am not a real user of this service. I was having a conversation on Twitter (https://twitter.com/John_Papa/status/10142358554463313920) with John Papa where he mentioned the 'slot' feature and how cool it was to deploy - test - and set as release w/o harming his current active production deployment.
I followed up with a question about being able to automate this - ie deploy this new slot and when it is done, make it automatically the new production slot. Wouter de Kort mentioned this wasn't available but it would be a good request. Which I've done here. :)
Again - I'm not really a user so forgive me if I got any of the terms wrong.

ping @fiveisprime

This feels out of place for a VS Code extension and more appropriately solved by CI/CD - it might even be possible already. Going to go ahead a close due to lack of activity


Loosen CKPT lock restriction

The implicit CKPT lock restriction added in https://github.com/superfly/litefs/pull/230 worked but it blocked all PASSIVE checkpoints when under heavy write load. This PR changes the restriction so it is only enforced while a writer is actively in the commit phase.
In a future PR, LiteFS should actively perform truncation checkpoints when the WAL grows too large since it has the most control over locks.

Closing this PR as the original checkpointing issue didn't seem to occur from the #230 change. While this PR likely works fine, the current version of CKPT restricting is more conservative so we'll go with that until we need to change it.


improving extension detection

well, trying to improve extension detection....

Fully admit I don't know enough about char and strings in c++ .. so if i'm using c_str() improperly, please let me know.

This PR is a mess.  I'm going to try again in a new one.


data_get can call delete method on Eloquent models accidentally

Laravel Version: 5.3.0
PHP Version: 5.6
Database Driver & Version:

Description:
Using Laravel Collective HTML with a bound form and a field named 'delete' causes the record to be deleted.
Steps To Reproduce:
In a view I had the following (not the entire view, but the essential details)
{!! form_model(entity) !!}
{!! form_checkbox('delete') !!}}
{!! form_close() !!}
This uses data_get internally, which tries to get 'delete' from the model and 'magically' calls the entity delete method. This has potential for serious problems.

Good catch, I can reproduce as well, it throws LogicException with message 'Relationship method must return an object of type Illuminate\Database\Eloquent\Relations\Relation' but the records is deleted.
Easier way to reproduce:
$user = User::find(1);
$user->delete;

That's a known issue, you must not use a name of a method in Model.php as an attribute name, avoid doing this because it has multiple issues not only this one.
I suggest using is_deleted or something.

In my case, there was no "delete" attribute. I wanted to have a non-bound field checkbox to confirm the deletion of a related model, the Form helper tried to look up the value of "delete" in the background and produced this unexpected result. I understand it having looked at the code why this happened, but it is a potentially dangerous side-effect.

I've submitted a fix that should prevent that https://github.com/laravel/framework/pull/15438


Prisma executes transaction twice

Bug description
After upgrading to Prisma version 3.9.1 from version 3.8.1, Prisma started to execute transactions twice if used in a non-async function.
For example, if you have a GraphQL resolver written like this:
t.field("createBanner", {
  type: "Banner",
  args: {
    input: nonNull("BannerInput"),
  },
  resolve: (_, { input }, context) => {
    assertAccess(context, ["ADMIN"]);

    return prisma.banner.create({
      data: input,
    });
  },
});

Prisma will execute the create operation twice, resulting in duplicate entries in the database. But if rewrite this resolver with a simple change of adding async, thus making the return value an explicit Promise, then issue is resolved:
t.field("createBanner", {
  type: "Banner",
  args: {
    input: nonNull("BannerInput"),
  },
-  resolve: (_, { input }, context) => {
+  resolve: async (_, { input }, context) => {
    assertAccess(context, ["ADMIN"]);

    return prisma.banner.create({
      data: input,
    });
  },
});

How to reproduce
Use Prisma inside of a non-async function.
I've also recorded a reproduction of this bug. In the video recording, besides the aforementioned issue, I've encountered an odd problem, where I had some sort of a ghost node task hanging in the background, preventing me from restarting the dev server, but this might be unrelated to Prisma.
Expected behavior
No response
Prisma information

Relevant part of the schema.prisma

Prisma schema
model Banner {
  id          Int        @id @default(autoincrement())
  createdAt   DateTime   @default(now()) @map("created_at") @db.Timestamptz(6)
  updatedAt   DateTime   @default(now()) @updatedAt @map("updated_at") @db.Timestamptz(6)
  image       String
  mobileImage String?    @map("mobile_image")
  note        String?
  pathMatch   String     @map("path_match")
  exact       Boolean    @default(false)
  link        String?
  type        BannerType @default(TOP)
  validUntil  DateTime?  @map("valid_until") @db.Timestamptz(6)

  @@map("banners")
}

Environment & setup

OS: macOS Monterey 12.2
Database: PostgreSQL
Node.js version: 16.13.1

Prisma Version
prisma                  : 3.9.1
@prisma/client          : 3.9.1
Current platform        : darwin-arm64
Query Engine (Node-API) : libquery-engine bcc2ff906db47790ee902e7bbc76d7ffb1893009 (at<EMAIL_ADDRESS>Migration Engine        : migration-engine-cli bcc2ff906db47790ee902e7bbc76d7ffb1893009 (at node_modules/@prisma/engines/migration-engine-darwin-arm64)
Introspection Engine    : introspection-core bcc2ff906db47790ee902e7bbc76d7ffb1893009 (at node_modules/@prisma/engines/introspection-engine-darwin-arm64)
Format Binary           : prisma-fmt bcc2ff906db47790ee902e7bbc76d7ffb1893009 (at node_modules/@prisma/engines/prisma-fmt-darwin-arm64)
Default Engines Hash    : bcc2ff906db47790ee902e7bbc76d7ffb1893009
Studio                  : 0.457.0

Breaking issue. Reproducible.
The following code creates duplicate posts without the async resolve function as stated above. If I use ays', things are back to normal.
t.field('createPost', {
type: 'Post',
args: {
data: arg({
type: "PostCreateInput",
})
},
resolve:  (parent, args, ctx) => {
//getUser(ctx.req);
//console.log("Calling post create!", args.data)
return ctx.prisma.post.create({
data: args.data
})
}
}),

found this bug too after upgrading to 3.9.1 but with normal operations,
using this:
  createNotaryPin: (_parent: any, args: any, ctx: any) => {

  args.code = `N-${v4()}`;

  return ctx.prisma.notaryPin.create({ data: args });
}

executes the create operation twice or even three times.

Are you using interactiveTransactions feature flag?

@Hazeman28 @torohaifisch could you please provide a reproduction?

@Hazeman28 @torohaifisch could you please provide a reproduction? I was not able to reproduce this.

i will try and get a working repro showing the error. I downgraded to 3.8.1 and everything works fine

here is an example project:
https://github.com/torohaifisch/prismaError/tree/master
project runs apollo sandbox to execute graphql queries, execute a createObservation mutation and it will sometimes execute the create operation twice.
You can verify this if you run the query operation getObservationList, it will show more registers than the times you executed the mutations. Another way to notice is that you wont get continuous ids on the mutation response.
This doesnt happen if i downgrade prisma and prisma client to 3.8.1

Are you using interactiveTransactions feature flag?

@millsp no, I'm not using any feature flags.

Thanks for the clear reproduction, I can confirm this. Apollo is doing a double call to then on the PrismaPromise which causes it to execute twice. We used to have a mechanism to prevent double execution, but that is gone. I'll push a fix later today.

Hey, the fix for this issue has been released via 3.9.2. Thank you for reporting this issue!


Update Composer dependencies (2024-03-03-00-10)

Visual regression test passed!
View the visual regression test report


Add ability to pass arbitrary options to c3.generate

I would like to be able to pass other options to this call: https://github.com/bitovi-components/bit-c3/blob/master/src/chart.js#L35.
Some example options that would be nice to be able to set are http://c3js.org/reference.html#axis-rotated and https://github.com/masayuki0812/c3/issues/188#issuecomment-55222488.
Not sure exactly how this would work, but it would make bit-c3 much more flexible.

I agree, this would make the config a lot easier. A first implementation of this would probably just to listen for changes on this options object, and then call chart.flush() to redraw it with the new options. Could get expensive, though - eventually, it'd probably be better to parse the options and bind them using the C3 API.

http://c3js.org/reference.html#data-onclick

Looks like almost all config properties are lower cased, except for couple (data.xFormat, data.xLocaltime, data.xSort). So we can translate camel-cased properties into nested object config:
var chart = c3.generate({
    ...
    size: {
        height: 80
    },
    legend: {
        show: false
    },
    axis: {
        x: {
            categories: ['Category 1', 'Category 2', ...]
        }
    }
});

can be achieved declaratively:
<bit-c3 size-height="80" legend-show="false" {axis-x-categories}="categories"/>

And we can observe the config and run chart.flush() as suggested by @kylegifford .


⚠ Fortnite has degraded performance

In 1872f4f, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1621 ms

Resolved: Fortnite performance has improved in 64766a8.


The automatic parent selection of cmd_backup is not smart enough.

Currently the code sets the current hostname as default. This is then used in FindLatestSnapshot to discover the latest snapshot for that host.
However, what if I make 2 snapshots for a host, because I have different retention policies for them? E.g. each night the host makes backups of /etc and of /home. The next night backup runs for /etc again and picks the /home backup as parent, as it is the most recent one. Consequently the /home backup picks the just made /etc as parent...
So none of this would be especially worrisome as restic would still dedup files at file and blob level, but as you (@fd0) explained so eloquently in in #652, picking the wrong parentID will result in lots of compute cycles as it cannot determine if the file has changed upfront.
So, I think the way forward would be to create a FindOptimalParentSnapshot(repo, tags, paths, host) function, which uses host as a hard constraint, but the tag and paths ones as soft constraints. Bonus points if it would notice supersets. If I created a backup of /etc,/home and later on do a backup of just /home, it could pick the /etc,/home backup as it's parent. (unless there is a more recent single /home of course). Then of course making a new backup of /etc,/home becomes hairy. what do we pick, the most recent one of /home or the most recent one of /etc,/home ? My bet is on the later as they will have the most overlap.

With such approach, restic may think a file with same timings but different content from another place doesn't need a backup, where it does.

Good idea for improving parent snapshot detection!
Unfortunately I think you've got a tiny bit wrong:

However, what if I make 2 snapshots for a host, because I have different retention policies for them? E.g. each night the host makes backups of /etc and of /home. The next night backup runs for /etc again and picks the /home backup as parent, as it is the most recent one. Consequently the /home backup picks the just made /etc as parent...

Suppose I have a backup of /home and a separate, later backup of /etc on the same host. On the next backup run of /home, restic will find the first snapshot (which just contains /home):

in cmd_backup.go the function FindLatestSnapshot is called with the set of paths and the hostname as parameters: https://github.com/restic/restic/blob/master/src/cmds/restic/cmd_backup.go#L398
This function then loads all snapshots and selects the latest one with the same hostname and set of paths: https://github.com/restic/restic/blob/master/src/restic/snapshot.go#L156

Is there a bug that I'm not aware of?

Aww, shoot. I completely missed that. <ponders> ...but there was something wrong with the code when I went over it yesterday...
Ah. its in SamePath() which threw me off. Say I have 2 existing backups, /,/boot and a newer backup of / (of the same host; who knows why ^^). Then when I make a new backup of /,/home, the one with / will be it's parent.
SamePaths() is not really checking if the paths are exactly the same, but a subset. (expected="/", actual="/,/boot") will return true. And since FindLatestSnapshot picks the newest backup. Bingo.
PS. Why does that function not take a Snapshot receiver, like HasTags? All 4 use-cases take sn.Paths as first argument

Oh, interesting. SamePaths() does actually only test whether actual is a subset of expected. That's... unexpected (from the function's name).


Update docstring of scipy.spatial.distance_matrix

The docstring had two big TODOs that appear to have been there for quite some time. =P This gets that done. =)

Thanks Juan


[FLINK-20966][table-planner-blink] Rename Stream(/Batch)ExecIntermediateTableScan to Stream(/Batch)PhysicalIntermediateTableScan

What is the purpose of the change
Rename Stream(/Batch)ExecIntermediateTableScan to Stream(/Batch)PhysicalIntermediateTableScan
Brief change log

Rename StreamExecIntermediateTableScan to StreamPhysicalIntermediateTableScan
Rename BatchExecIntermediateTableScan to BatchPhysicalIntermediateTableScan

Verifying this change
This change is a refactoring rework covered by existing tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure


[Feature] : Titanic Dataset Visualization

Hey 👋, I am a Hacktoberfest'23 contributor
What ?
I would like to add a data visualization notebook for the Titanic dataset to explore passenger demographics, survival rates, and class distribution aboard the Titanic.

Dataset

describe the dataset you will use for this case.
Screenshots

Code of Conduct

[x] I agree to follow this project's Code of Conduct

Kindly assign it to me under HacktoberFest'23
I will surely complete it ASAP

Thank You

Hey, kindly assign me this issue @AKACHI-4 . I have already worked with titanic dataset previously and would be glad to help with it's visualization.

Hey, kindly assign me this issue @AKACHI-4 . I have already worked with titanic dataset previously and would be glad to help with it's visualization.

Hey @Chandak-Keshav,
It's great to hear that you've already worked on it.
please provide the details of the dataset and go ahead.

https://www.kaggle.com/competitions/titanic/data
I will be using the above dataset for data visualization. Should I go ahead with this dataset or do you have any particular dataset in mind @AKACHI-4

https://www.kaggle.com/competitions/titanic/data I will be using the above dataset for data visualization. Should I go ahead with this dataset or do you have any particular dataset in mind @AKACHI-4

Hey, @Chandak-Keshav.... the dataset is good, but I'll suggest finding another one with more parameters so that you can extract more meaning from the data.

https://www.kaggle.com/competitions/spaceship-titanic/data?select=train.csv
How about this dataset @AKACHI-4

@Chandak-Keshav, yes, go ahead... make sure to use libraries for better visualization to extract meaningful insights from it.


DNS_UPSTREAM_SERVERS not used in current kube-dns ?

This is not used anymore:
https://github.com/pires/kubernetes-vagrant-coreos-cluster/blob/master/Vagrantfile#L111
Don't find any entry in it:
https://github.com/pires/kubernetes-vagrant-coreos-cluster/blob/27f3e4e294296e1ceac67135a02fe177c7d8ef7c/plugins/dns/dns-controller.yaml.tmpl
Does anyone know the details, why is this not needed?

I think upstream servers can be defined but no one was using them, so while updating the descriptors I forgot to remove the optional parameter. I will remove it soon.


Hotfix/stringify json

Fix to parse JSON on save

Codecov Report

Merging #692 (af14a15) into tagged-release/forms-and-views (e88974c) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@                        Coverage Diff                         @@
##           tagged-release/forms-and-views     #692      +/-   ##
==================================================================
+ Coverage                           91.48%   91.50%   +0.02%
==================================================================
  Files                                 102      102
  Lines                                3886     3897      +11
  Branches                             1072     1078       +6
==================================================================
+ Hits                                 3555     3566      +11
  Misses                                310      310
  Partials                               21       21

Impacted Files
Coverage Δ

.../codegen-ui-react/lib/imports/import-collection.ts
94.11% <ø> (ø)

...codegen-ui-react/lib/forms/form-renderer-helper.ts
98.21% <100.00%> (+0.03%)
:arrow_up:

packages/codegen-ui-react/lib/forms/form-state.ts
98.30% <100.00%> (+0.12%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e88974c...af14a15. Read the comment docs.


How can I speed up Hikari with MySQL

Hi there Brett,
Thank you for the Hikari pooling library. I have a MySQL backend with 110 Million records and two front-end undertow servers behind ELB using Hikari for connection pooling. My queries are really slow at the moment, each taking about 24 seconds (granted I am yet to index the tables). But I compared Hikari's performance to DBCP on the same table and DBCP was running the query in 21 seconds. I need to get as much throughput as possible. So need to tune Hikari to server as many requests as possible at its max capacity. The throughput requests need to meet target of 7000 RPS for this query and 27000 for other queries. We will optimize the query and also use a larger cluster but how can I set Hikari to give maximum throughput. I am a student and am working on a project where I need to achieve high RPS. My current settings using Hikari 2.4.7 are:
		HikariConfig config = new HikariConfig();

		config.setDriverClassName("com.mysql.jdbc.Driver");
		config.setJdbcUrl("jdbc:mysql://"+"localhost/twitter?useSSL=false&useUnicode=yes&characterEncoding=UTF-8");
		config.setUsername("root");
		config.setPassword("blahblah");
		//config.setMinimumIdle();
		config.setIdleTimeout(0);
		config.setMaxLifetime(0);
		config.setMaximumPoolSize(250);
		config.addDataSourceProperty("cachePrepStmts", "true");
		config.addDataSourceProperty("prepStmtCacheSize", "250");
		config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048");

		datasource = new HikariDataSource(config);

Also, is it better to hold on to a connection if I am running one query, then doing a little bit of calculation and then running second query. Or is it better to return the connection and fetch it again after every query. So my option is that the servlet gets the connection and keeps it until done with all operations or returns in intermediate steps. Asking because I am not sure how expensive it is to fetch connection and return repeatedly. I am still learning so sorry for too many questions.
Thank you for your help :)

To start it looks like you have an extremely large pool size. Take a look at https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing your pool size should be much lower.
As far as the optimizations go properly indexing the data and optimizing the queries will most often be more beneficial than optimizing the pool itself. Once you have made a lot of progress there you can look at the pool.

Also, is it better to hold on to a connection if I am running one query, then doing a little bit of calculation and then running second query.
This is the purpose of a connection pool. Internally it holds onto all of the connections and leaves them open. The difference between manually holding it and letting the pool manage it should be fairly negligible.

You may want to set some MaxLifetime and IdeTimeout.

oh..great insight on relation with # of cores. Thank you. So I have a 2 core machine so perhaps about 5-6 would be the pool size. Not sure why would I want IdleTimeout. If it is idle shouldn't it just sit around so a new connection doesn't need to be created when a request finally does come in? Same question for MaxLife, shouldn't it just stay alive forever....as I will need to handle 7000 RPS for roughly 5 hours.
Thank you again.

@tejalshah84 Possibly take a look at My Benchmark Doesn't Show a Difference.  My guess is that DBCP is "faster" because it is actually not testing connection validity (among other things).  I think if you configure DBCP with the same correctness and reliability guarantees, you'll find that DBCP runs much slower than HikariCP.

@tejalshah84 To answer your other questions.  In the case on HikariCP, obtaining a connection from the pool should typically take an almost immeasurable amount of time.  Therefore I recommend that your servlet:

Obtain a connection, run a query, then close the connection.
Perform whatever calculation you want to perform -- as long as it takes more than a few milliseconds.
Obtain a new connection and run your second query.

However, if the calculation time is less than say 10 milliseconds, it may prove better to hang on to a single connection rather than re-obtaining one.
Regarding maxLifetime, infinite is typically not good.  Drivers and server-side sessions have been known to have leaks, or at a minimum accumulate state that consumes increasing amounts of memory, that is freed by releasing and replacing connections periodically.  Though it is probably perfectly safe to set lifetimes up to 8 hours or more.  Monitoring Java and/or database memory over time can help you determine what the impact of long-lived connections are in your environment.


Add Japanese translation

Hi. I've added some lines in my language. :jp:
If you don't mind following things, please just them merged. Or if you do, could you let me know?

I added them at the end of each list. CJK (Chinese-Japanese-Korean) are often aligned together though.
I'm not sure how I should treat commas at the end of lines. I put them now.

Thanks.

@ginpei, thanks so much for this translation! I'll go ahead and merge this.
Agree that CJK should be clustered together, but I can take care of this myself.

Thank you!


[ASK] Getting recommendation with xDeepFM

Description
I am actually using xDeepFM to find recommendation with implicit feedbacks and I did not find examples on how to find top-k recommendation with the model. I tried to use the function model.predict() with a file containing all the possible interactions between my users and items to find a score for each pair and sorting these scores to find top-k recommendation. This method gives me the same recommendation for every user so I would want to know if my method is wrong and what would you suggest to find top-k recommendation ?

@Leavingseason may have some ideas about this.

Any updates on this?

Description
I am actually using xDeepFM to find recommendation with implicit feedbacks and I did not find examples on how to find top-k recommendation with the model. I tried to use the function model.predict() with a file containing all the possible interactions between my users and items to find a score for each pair and sorting these scores to find top-k recommendation. This method gives me the same recommendation for every user so I would want to know if my method is wrong and what would you suggest to find top-k recommendation ?

Did you find a way to do top-k recommendation with it?

I worked on it during my internship, so I am not working on it anymore. But as far as I remember I used this technique (predict on every pair of user/item and sort to have top-k score) and acted on the different parameters of the model to have better results on my recommendations

No, it seems that is was related with the parameter of the model trained (my model was underfitting) and the quality of the data I had (which was difficult to improve)

ok then i have to check my data. thanks for your help


Uploading files with drag and drop or selection.

I'd like to have an area where the user can drag and drop a file to upload and click the area to select a file to upload. Currently it seems like it is only possible to either drag and drop or click (i.e. ng-file-drop, ng-file-select). Is there a way to do this that I'm just unaware of?

Fixed in version 2.1.x. You can have both directives on the same element like the demo page.


[VL] Gluten-it: Add option --random-kill-tasks to emulate the case that tasks are frequently killed and retried

Umbrella: https://github.com/oap-project/gluten/issues/3048
To simulate speculative execution's task killing actions in local test, even using more aggressive kills.

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_3056_time.csv
log/native_master_09_07_2023_192f5b03e_time.csv
difference
percentage

q1
43.09
43.44
0.353
100.82%

q2
24.59
24.73
0.138
100.56%

q3
37.79
37.30
-0.490
98.70%

q4
41.40
41.95
0.557
101.35%

q5
70.01
70.09
0.074
100.11%

q6
6.63
6.54
-0.091
98.62%

q7
85.46
84.15
-1.311
98.47%

q8
81.84
76.28
-5.561
93.20%

q9
114.65
116.72
2.072
101.81%

q10
45.52
42.30
-3.217
92.93%

q11
19.45
19.70
0.245
101.26%

q12
24.08
25.74
1.666
106.92%

q13
49.30
49.36
0.062
100.13%

q14
17.04
16.10
-0.939
94.49%

q15
30.71
31.19
0.483
101.57%

q16
15.86
15.34
-0.523
96.70%

q17
122.31
121.01
-1.306
98.93%

q18
160.55
160.47
-0.077
99.95%

q19
12.76
11.95
-0.812
93.64%

q20
31.03
25.56
-5.476
82.35%

q21
239.59
233.87
-5.719
97.61%

q22
15.29
15.38
0.082
100.54%

total
1288.96
1269.17
-19.792
98.46%


Add dynamic service loader [ECR-2953]

Overview

Add a dynamic service loader.

See: https://jira.bf.local/browse/ECR-2953
Definition of Done

[x] There are no TODOs left in the code
[x] Change is covered by automated tests
[x] The coding guidelines are followed
[x] Public API has Javadoc
[x] Method preconditions are checked and documented in the Javadoc of the method
[x] Changelog is updated if needed (in case of notable or breaking changes)
[x] The continuous integration build passes

Coverage increased (+0.1%) to 86.029% when pulling a47546fa149704c7b980e2e9792b49a102119e7a on dmitry-timofeev:dynamic-service-loader-ECR-2953 into 16ccf3c63e9de276493ce7eb230009097c9b67ab on exonum:master.

Replaced URIs with Paths as discussed and resolved todos till ECR-3008


Prevent resizing grid element outside of the layout

Hi everybody! When you resize a grid item, the layout height value increases and vertical scrolling appears.
I'm using properties:
isBounded: true,
preventCollision: true,
verticalCompact: false
I expected: When you change the height of an element so that it doesn't go beyond the layout and doesn't cause vertical scrolling.
Demo

Any news on this ? I'm also looking for this

That's not possible, it will go beyond the layout when you drag elements to change their positions.
You can alert user when save, or prevent saving it to the backend

Oooh I see, okay thank you, so instead of blocking the heigh/width I can just tell that he's putting the card out of my virtual border and not save the change.
Thank you

That's not possible, it will go beyond the layout when you drag elements to change their positions.
You can alert user when save, or prevent saving it to the backend

Thanks, but ideally, just constrain resizing similarly to isBounded

Hello I am also looking for a solution to this issue.

Please reopen this issue.


Update messagesTable1.PNG

Thanks @amyblais! I noticed the file extension doesn't match, can you help

rename the file to messagesTable1
take a screenshot with Snipping Tool (that'll add the .PNG extension, instead of .png)


Create a singleron workflow

Here is the documentation for their technology https://github.com/singleron-RD/CeleScope

Just a thought here - I know I made a repo for all our containers, but perhaps it would make most sense that pipeline modules come with their own definition files?

That's a good idea. I will do that in the future!

I probably won't start this workflow because of my limited time left at CTG.
I will focus on infrastructure and documentation.