Hotfix: update helm chart version to 0.3.88

PR Checklist
PR Structure

[x] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[x] This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
[x] This PR's title starts with name of package that is most changed in the PR, ex.
paymentservice.stellar, or all or doc if the changes are broad or impact many
packages.

Thoroughness

[x] This PR adds tests for the most critical parts of the new functionality or fixes.

What
Update helm chart version to 0.3.88.
Why
It had not been updated when we made the Stellar Observer into a separate service in https://github.com/stellar/java-stellar-anchor-sdk/pull/546/files#diff-78b9c67cf91b015c1ed14485867e595e1ba056a8e28563c2d037c8fe073a7893.

Reference Server Preview is available here:https://anchor-ref-pr600.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr600.previews.kube001.services.stellar-ops.com/

Something went wrong with PR preview build please check


dataframe read_csv (ValueError:  Exception in remote process  could not convert string to float)

I have a csv that contains US zip codes.  Most of them are numerics, but some of them contain hyphens (e.g. 94103-2585).
I get the following error using read_csv()
ValueError:  Exception in remote process could not convert string to float: '94103-2585'
An easy workaround for me was to include a dtype arg (dtype={"Zip Code" : "object"}.  Also, same file with pandas read_csv() correctly infers the object type.
Let me know if additional details are needed
Thanks,
Bob
dask 0.6.0
python 3.4

Sorry for the lateness in reply.
The dask.dataframe.read_csv guesses types based on the first thousand lines of the file.  If this isn't enough to get a clear picture of the dtypes involved you can increase the number of lines to sample with the following keyword argument:
sample_nrows=10000

This will take a bit longer as we create the graph but might resolve the problem.  Explicitly providing the dtype arg is even more robust.
I'm not sure what we can do in this case short of reading the entire file to create the graph.  This is sufficiently costly that I'm not sure it would be desired.

Agreed. dtype arg is the way to go.  Appreciate the response

I had starred this in my e-mail but it was swept away to the bowels of the second page before I looked at it again.  I only ran across this in a general sweep of dask issues.  If this happens again please ping or feel free to e-mail me directly.
Closing.

Hello,
I am trying to read multiple parquet files (I created them by partitioning in 64 parts a big data file originally contained in feather formatted file) at once using dask-distributed. The reading fails with a similar error, it is worded like this:
distributed.worker - WARNING -  Compute Failed
Function:  read_parquet_part
args:      (<function FastParquetEngine.read_partition at 0x7f6196239f80>, <fsspec.implementations.local.LocalFileSystem object at 0x7f6196234a10>, Empty DataFrame
Columns: [...,
kwargs:    {}
Exception: ValueError("could not convert string to float: '1994-04-01'")

I have to note that I was successful before using a randomly half sample of this same data partitioned in 32 parts.
Unfortunately, I cannot share the data that lead to that error because they are privileged, but it would be greatly appreciated if someone can give me an idea of where to begin investigate.
I try to check the data types, they look the same, Unfortunately I don't have a way to investigate all the data in each category types in the 64 files (I have more 250 columns and several of them are categorical). I don't know enough of the parquet options to be sure that all partitioned files have the same metadata as the others (I'm producing these files using pandas).
I am using dask 2.12, python 3.7.
Thanks for your help
Arnaud

I'd recommend trying to read each file individually using pandas. You should be able to use Fastparquet or pyarrow directly to read just the metadata, but I don't recall exactly how right now.

Great, I just saw that by creating a ParquetFile object with the file or the list of partition files in parameter, I just upload the metadata.
I'll try to investigate using that :)
Thanks a lot for the tip!
Arnaud


feat: @badger-dao/sdk-mocks package

Hook worked properly

:tada: This PR is included in version 1.5.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


🐛 : V3, useSharedValue is not working with V3

What's happening?
hi Marc, I'm currently working on developing a vision-camera-pose-detection project with MLKit. I initially wrote the native code for iOS, and it connected perfectly. However, I encountered a problem when attempting to draw the overlay for the skeleton based on the keypoints.
In general, I used useSharedValue from Reanimated V3 to pass a value.
when I use useSharedValue I receive an error message like this:
Reading from '_value' directly is only possible on the UI runtime.
I have already wrapped my JS function into a worklet, but I still get an error.
If I pass the value into setState it works, but it doesn't behave as expected. there is a delay, as shown in this video. this is my friend who tested it
https://github.com/mrousavy/react-native-vision-camera/assets/49748249/1517267b-fcd5-4125-9715-97a561803644
 "react-native-vision-camera": "^3.0.0",  "react-native-worklets-core": "^0.2.0",  "react-native-reanimated": "^3.4.2"
Reproduceable Code
PoseDetection.ts
import {VisionCameraProxy, Frame} from 'react-native-vision-camera';
import {IPoseDetection} from '../utils/frameProcessor/PoseDetection';

const plugin = VisionCameraProxy.getFrameProcessorPlugin('poseDetection');

export function poseDetection(frame: Frame): IPoseDetection {
  'worklet';

  if (plugin == null) {
    throw new Error('Failed to load Frame Processor Plugin "poseDetection"!');
  }

  return plugin.call(frame) as unknown as IPoseDetection;
}

Main Code:
...
// POSE

  const pose = useSharedValue<IPoseDetection>(defaultPose);

  const convertToWorklet = (poseObject: IPoseDetection, frame: Frame) => {
    const xFactor = dimensions.width / frame.width;
    const yFactor = dimensions.height / frame.height;

    const poseCopy: IPoseDetection = {
      leftShoulder: {x: 0, y: 0},
      rightShoulder: {x: 0, y: 0},
      leftElbow: {x: 0, y: 0},
      rightElbow: {x: 0, y: 0},
      leftWrist: {x: 0, y: 0},
      rightWrist: {x: 0, y: 0},
      leftHip: {x: 0, y: 0},
      rightHip: {x: 0, y: 0},
      leftKnee: {x: 0, y: 0},
      rightKnee: {x: 0, y: 0},
      leftAnkle: {x: 0, y: 0},
      rightAnkle: {x: 0, y: 0},
    };

    Object.keys(poseObject).forEach(v => {
      const key = v as keyof IPoseDetection;
      poseCopy[key] = {
        x: poseObject[key].x * xFactor,
        y: poseObject[key].y * yFactor,
      };
    });

    pose.value = poseCopy;
  };

  const setToWorklet = Worklets.createRunInJsFn(convertToWorklet);

  const frameProcessor = useFrameProcessor(
    frame => {
      'worklet';
      const poseObject = poseDetection(frame);
      console.log('pose data', poseObject);
      setToWorklet(poseObject, frame);
    },
    [pose],
  );

Relevant log output
Error: Reading from `_value` directly is only possible on the UI runtime, js engine: hermes

Camera Device
No response
Device
iPhone 11  ( 15.5 )
VisionCamera Version
3.0.0
Can you reproduce this issue in the VisionCamera Example app?

[ ] I can reproduce the issue in the VisionCamera Example app.

Additional information

[ ] I am using Expo
[ ] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

I think I've been able to fix this by using useSharedValue from react-native-worklets-core instead of reanimated.
Per https://github.com/mrousavy/react-native-vision-camera/pull/1466,

Reanimated is no longer used as a Worklet Runtime. Instead, VisionCamera now uses react-native-worklets-core.

However, this presents a problem if I want to use the shared state in an animation.  Reanimated doesn't recognize when changes to the state occur, so they don't trigger animation update.

Yea useSharedValue is from RN Worklets Core, not from Reanimated.

Yea useSharedValue is from RN Worklets Core, not from Reanimated.

mmm, it still doesn't work when I pass it into useAnimatedStyle. The line is not rendered. I was using react-native-svg and converted the 'Line' into an AnimatedLine with react-native-reanimated's 'createAnimatedComponent'. I'm sorry, but is there any way to draw an overline like an animated line?

the native modules for Android and iOS is perfectly connected, it shows the values. I'm just missing the last step for drawing the overlay 😢

I also have the same issue. I've successfully created my own version of vision-camera-ocr that does text recognition. Using latest react-native-vision-camera (v3).
The missing piece is how to get data from the worker in JS, out to my react component, and rendered.

When using "Worklets.createRunInJsFn" the app (iOS) crashes, when calling my backToReactThread function ^ It crashes in Native code, inside jsi::String JSCRuntime::createStringFromUtf8 calling: JSStringCreateWithUTF8CString
And using useShareValue (from Worklets) seems to have no affect. useAnimatedStyle ignores it..
How can we get some of the results/data from our useFrameProcessor javascript function, to the rest of react native so we can draw ui updates?

Hey! So VisionCamera V3 uses react-native-worklets-core, and react-native-reanimated uses it's own Worklets implementation, so the useSharedValue is not compatible with the one from Reanimated (I think we could add interop support for that though, wdyt @tomekzaw?)
What you can do (and what I would suggest you to do anyways regardless of this issue) is to draw the entire overlay using @Shopify/react-native-skia. This is much faster and flexible/powerful for drawing such overlays. Use the draw callback to draw.

Thank you @mrousavy ! Btw, is there a type-o in the offiical docs where it says:
"(Optional) If you want to use Frame Processors, you need to install react-native-worklets-core 1.0.0 or higher."
I think react-native-worklets-core only goes up to version 0.2.0.. I don't see 1.0.0 ?

Separately, I am still seeing the crash  inside jsi::String JSCRuntime::createStringFromUtf8 calling: JSStringCreateWithUTF8CString . Do you have any tips or hints on where I can look for that? I wonder if it's because I have 0.2.0 and not 1.0.0 of react-native-worklets-core (which is no longer available on npm)?

@mrousavy I wonder if you could share a package.json from a working setup. Curious to see what version of react-native, react-native-worklets-core, and react-native-vision-camera are all working well together.

Package.json; the one here in example/ app.
And yea, you're right the docs should say 0.2.0 instead of 1.0.0.
I need a reproduceable example for that

@mrousavy thank you. that example was super helpful. I managed to get it building and running. In the end I had to:

Upgrade Node !! This was the biggest source of my problems. I was on Node v16... Needed to go to 18+..
I needed to install string-hash-64, I know you're aware since I saw you post in the other git repo about it
I needed to apply the _closure --> __closure (double underline) patch from the other thread to prevent a crash on startup with the example app.

After that, I was good to go. Thank you so much for your help.

Oh interesting! Could you maybe send a PR to react-native-worklets-core to fix the _closure issue? Thanks! cc @chrfalch
I'll close this for now, let me know if this issue is still active and we can re-open it

just fyi, in https://github.com/software-mansion/react-native-reanimated/pull/4803 we renamed _closure to __closure for the purpose of consistency so that all fields start with a double underscore.

Ah shoot haha okay yea we'll rename as well then 😅

Thanks for the heads up Tomasz!

Hi,
if somebody else stumbles on thid issue my (ugly) solution is to copy the useSharedValue variable from the main thread:
import { useSharedValue, Worklets } from 'react-native-worklets-core'
import { useSharedValue as useSharedValueR } from 'react-native-reanimated'

  const pose = useSharedValue(getDefaultPose())
  const poseR = useSharedValueR(getDefaultPose())

  const updateSkeleton = Worklets.createRunInJsFn((p) => {
    poseR.value = { ...p }
  })

const frameProcessor = useFrameProcessor(frame => {
    'worklet'
...
 updateSkeleton(pose.value)

Will add compat for that soon!

Hi, if somebody else stumbles on thid issue my (ugly) solution is to copy the useSharedValue variable from the main thread:
import { useSharedValue, Worklets } from 'react-native-worklets-core'
import { useSharedValue as useSharedValueR } from 'react-native-reanimated'

  const pose = useSharedValue(getDefaultPose())
  const poseR = useSharedValueR(getDefaultPose())

  const updateSkeleton = Worklets.createRunInJsFn((p) => {
    poseR.value = { ...p }
  })

const frameProcessor = useFrameProcessor(frame => {
    'worklet'
...
 updateSkeleton(pose.value)

I tried above thing but seems not working UI is not updating or showing

Verify first that updating your reanimated shared value from js works , then put a console log in the Worklets.createRunInJsF to verify it has been called

I tried above thing but seems not working UI is not updating or showing

Verify first that updating your reanimated shared value from js works, then put a console log in the Worklets.createRunInJsF to verify it has been called

I tried above thing but seems not working UI is not updating or showing

ys function has been calling but UI is not updating unless hard refresh. i'm using latest version of vision-camera

Same, it is not working for me. @umbertoghio what version vc, wc, and reanimated are you using?

@jslok may you send some snippets to make it workable with reanimated and vision camera and worklet-core versions

@jslok may you send some snippets to make it workable? and please mention relevant versions for reanimated and vision camera and worklet-core

vc 3.6.16
worklet-core 0.24
reaniated 3.6.1
Could not save an object in useSharedValue so I had to split it up into four different ones.
`  const rectWidth = useSharedValue(100) // rect width
const rectHeight = useSharedValue(100) // rect height
const rectX = useSharedValue(100) // rect x position
const rectY = useSharedValue(100) // rect y position
const rectWidthR = useSharedValueR(100) // rect width
const rectHeightR = useSharedValueR(100) // rect height
const rectXR = useSharedValueR(100) // rect x position
const rectYR = useSharedValueR(100) // rect y position
const updateRect = Worklets.createRunInJsFn((frame) => {
rectWidthR.value = frame.width
rectHeightR.value = frame.height
rectXR.value = frame.x
rectYR.value = frame.y
})
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
const rectangle = cardDetect(frame)
if (rectangle.x) {
  const { width: frameWidth, height: frameHeight } = frame

  const xFactor = windowWidth / frameWidth
  const yFactor = windowHeight / frameHeight

  //return
  // Assign animated values to rect shared values
  rectWidth.value = rectangle.width * xFactor
  rectHeight.value = rectangle.height * yFactor
  rectX.value = rectangle.x * xFactor
  rectY.value = rectangle.y * yFactor

  updateRect({
    width: rectWidth.value,
    height: rectHeight.value,
    x: rectX.value,
    y: rectY.value,
  })
}

}, [])
// Animated styles for the box
const animatedStyle = useAnimatedStyle(() => {
return {
width: withSpring(rectWidthR.value),
height: withSpring(rectHeightR.value),
transform: [
{ translateX: withSpring(rectXR.value) },
{ translateY: withSpring(rectYR.value) },
],
}
})
`


Run error when following initial setup

After running the initial setup steps - I get a "Cannot find module './categoryCalculationScript'" error.
the "categoryCalculationScript.js" file is in the .gitignore.
is it possible the categoryCalculationScript-example.js file was supposed to be part of the git repository but is not?

You should copy the categoryCalculationScript-example.js file to create a categoryCalculationScript.js file with your own category calculation logic. See the initial setup section in the README.

Yeah... Saw it a minute too late :).
Thanks.


vcpkg-tool does not respect CMake project's CMAKE_CXX_COMPILER

Describe the bug
vcpkg-tool only considers the target triplet when detecting what compiler to use to build vcpkg dependencies for CMake projects.
Environment

OS: Linux, Ubuntu 22.04
Compiler: clang-18, gcc-13, etc

To Reproduce
Steps to reproduce the behavior, on Ubuntu 22.04:

Create a new project in manifest mode
Add a dependency, e.g. fmt
Create a CMakeLists.txt that loads the vcpkg toolchain file
Configure CMake project with -DCMAKE_CXX_COMPILER=clang++-18 -DCMAKE_C_COMPILER=clang-18
Observe the message from vcpkg-tool:

Detecting compiler hash for triplet x64-linux...
Compiler found: /usr/bin/c++

The compiler name/path is not used when detecting what compiler to use when building vcpkg ports. Only the default search mechanisms for CMake are involved. That means, /usr/bin/cc and $CC environment variable
Expected behavior
vcpkg will build ports with the compiler specified to be used via -DCMAKE_C_COMPILER, -DCMAKE_CXX_COMPILER.
This behavior should be consistent with how chain-loading another toolchain file works.
Failure logs

Terminal output shown inside details
andrew@Andrew-Workstation:~/dev/vcpkg-cmake-cxx-compiler$ cat vcpkg.json
{
  "dependencies": [
    "fmt"
  ]
}
andrew@Andrew-Workstation:~/dev/vcpkg-cmake-cxx-compiler$ cat CMakeLists.txt
cmake_minimum_required(VERSION 3.22)

project(Example LANGUAGES CXX)

find_package(fmt CONFIG REQUIRED)

add_executable(example main.cpp)
target_link_libraries(example PRIVATE fmt::fmt)
target_compile_features(example PUBLIC cxx_std_17)

andrew@Andrew-Workstation:~/dev/vcpkg-cmake-cxx-compiler$ cat main.cpp
#include <fmt/format.h>

int main()
{
  fmt::println("hello, world!");
}

andrew@Andrew-Workstation:~/dev/vcpkg-cmake-cxx-compiler$ cmake -GNinja -B build -DCMAKE_TOOLCHAIN_FILE=/home/andrew/vcpkg/scripts/buildsystems/vcpkg.cmake -DCMAKE_C_COMPILER=clang-18 -DCMAKE_CXX_COMPILER=clang++-18
-- Running vcpkg install
Detecting compiler hash for triplet x64-linux...
Compiler found: /usr/bin/c++
The following packages will be built and installed:
<EMAIL_ADDRESS>  * vcpkg-cmake:x64-linux@2024-04-23
  * vcpkg-cmake-config:x64-linux@2024-05-23
Additional packages (*) will be modified to complete this operation.
Restored 3 package(s) from /home/andrew/.cache/vcpkg/archives in 16.1 ms. Use --debug to see more details.
Installing 1/3<EMAIL_ADDRESS>Elapsed time to handle vcpkg-cmake-config:x64-linux: 534 us
vcpkg-cmake-config:x64-linux package ABI: bc50feb59200c32f0d107fdd56cb9673025fd1674e50fe8357b309bbc4aead5c
Installing 2/3<EMAIL_ADDRESS>Elapsed time to handle vcpkg-cmake:x64-linux: 454 us
vcpkg-cmake:x64-linux package ABI: a8369f48befea9026b5586cad8f947eb75c0f115fc69bd111f9e3167d72d7bb5
Installing 3/3<EMAIL_ADDRESS>Elapsed time to handle fmt:x64-linux: 1.01 ms
fmt:x64-linux package ABI: 08a2a9c498ee0d9b65a31c9e28a5de1b844824be28171bf1d7e775a63e663757
Total install time: 2.02 ms
The package fmt provides CMake targets:

    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt)

    # Or use the header-only version
    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt-header-only)

-- Running vcpkg install - done
-- The CXX compiler identification is Clang 18.1.8
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++-18 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (1.0s)
-- Generating done (0.0s)
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_C_COMPILER

-- Build files have been written to: /home/andrew/dev/vcpkg-cmake-cxx-compiler/build
andrew@Andrew-Workstation:~/dev/vcpkg-cmake-cxx-compiler$ /usr/bin/c++ --version
c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

As seen in the terminal output above, vcpkg finds /usr/bin/c++, which is g++-11 in my system. However, I explicitly asked for clang++-18, which is properly detected by the CMake output for configuring my project proper.
vcpkg:
-- Running vcpkg install
Detecting compiler hash for triplet x64-linux...
Compiler found: /usr/bin/c++

my project:
-- The CXX compiler identification is Clang 18.1.8
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++-18 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done

Additional context
The code involved is here: https://github.com/microsoft/vcpkg-tool/blob/2af426b284d7a321628471c6baef46452385534d/src/vcpkg/commands.build.cpp#L683-L699
The code seems to use the toolset argument, which I would expect corresponds to the -T flag used for MSVC compiles. However, on Linux and macOS, the toolset flag is not valid and not used for Ninja, Unix Makefile and Xcode generators, and most users will configure with -DCMAKE_C_COMPILER and -DCMAKE_CXX_COMPILER.

Note that I can work around this by doing something like this:
/path/to/my/local/vcpkg-triplets/release-triplets/x64-linux.cmake
set(VCPKG_CMAKE_SYSTEM_NAME Linux)
set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_BUILD_TYPE release)
set(VCPKG_LIBRARY_LINKAGE static)

# Hackery using a private vcpkg variable to get to the root of the invoking project's CMake binary dir when using manifest mode
include(${_VCPKG_INSTALLED_DIR}/../build-vcpkg-variables.cmake OPTIONAL)

And then generating build-vcpkg-variables.cmake like so:
CMakeLists.txt
cmake_minimum_required(VERSION 3.25)

# Pass additional information to vcpkg toolchain files if we are using vcpkg.
if (CMAKE_TOOLCHAIN_FILE MATCHES "vcpkg.cmake$")
    set(CMAKE_PROJECT_MyProject_INCLUDE_BEFORE "vcpkg/generate_vcpkg_toolchain_variables.cmake")
endif()

project(MyProject LANGUAGES C CXX)

vcpkg/generate_vcpkg_toolchain_variables.cmake
set(EXTRA_VCPKG_VARIABLES "")
if (NOT "${CMAKE_C_COMPILER}" STREQUAL "")
    string(APPEND EXTRA_VCPKG_VARIABLES "set(ENV{CC} ${CMAKE_C_COMPILER})\n")
endif()
if (NOT "${CMAKE_CXX_COMPILER}" STREQUAL "")
    string(APPEND EXTRA_VCPKG_VARIABLES "set(ENV{CXX} ${CMAKE_CXX_COMPILER})\n")
endif()

file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/build-vcpkg-variables.cmake" "${EXTRA_VCPKG_VARIABLES}")

But this approach feels verbose and possibly fragile? For something that I would expect to "just work" (and does seem to just work for MSVC toolsets).

For additional context, this seems to be a thing limited to Unix Makefiles and Ninja Generators (and other bespoke generators, I suppose).
To quote Brad King from a recent CMake Issue https://gitlab.kitware.com/cmake/cmake/-/issues/26316#note_1571027

The VS and Xcode generators do not support specifying the path to the compiler directly for any language.  The situation is the same for CMAKE_C_COMPILER and CMAKE_CXX_COMPILER.  For those generators one is supposed to set CMAKE_GENERATOR_TOOLSET to control compiler selection.

So it seems like for VS and Xcode, the toolset should specify all the information needed to select a specific compiler. But for other generators, users expect that CMAKE_C_COMPILER and CMAKE_CXX_COMPILER will set the name/path of the desired compiler.


Adding a particulate organic matter pool

Description
This pull request adds an extra carbon pool to the soil carbon model. The pool added is the particulate organic matter (POM) pool. This pool is for organic matter that is too complex to be immediately taken up by microbes but that isn't protected through mineral association. The motivation for adding this pool is that the carbon transfers from litter to soil go to this pool, so adding it is a prerequisite for setting up a litter model/module.
I've just added @vgro as a reviewer here, as this PR just extends existing functionality and doesn't really add anything novel, so I don't feel it needs a really thorough going over. If others want to add comments that's completely fine.
Type of change

[x] New feature (non-breaking change which adds functionality)
[ ] Optimization (back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)

Key checklist

[x] Make sure you've run the pre-commit checks: $ pre-commit run -a
[x] All tests pass: $ poetry run pytest

Further checks

[x] Code is commented, particularly in hard-to-understand areas
[x] Tests added that prove fix is effective or that feature works

Codecov Report

Merging #253 (32bc49a) into develop (67ac3f3) will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #253      +/-   ##
===========================================
+ Coverage    95.04%   95.08%   +0.04%
===========================================
  Files           42       42
  Lines         1694     1708      +14
===========================================
+ Hits          1610     1624      +14
  Misses          84       84

Impacted Files
Coverage Δ

virtual_rainforest/models/soil/carbon.py
100.00% <100.00%> (ø)

virtual_rainforest/models/soil/constants.py
100.00% <100.00%> (ø)

virtual_rainforest/models/soil/soil_model.py
100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Use https

This requires:

getting a certificate form letsencrypt
turning on cloudfront
invalidation automation of the cloudfront cache
which requires to move to cobalt-rs

So it waits on #12

This issue is more than 2 years old and still no https! Please fix that!

Hello to the maintainers :wave:
Do you have an ETA for this issue ? 2 years and a half from now, most websites have migrated to HTTPS for privacy and security reasons...and most browsers are making it more and more visible that non-HTTPS websites are not secure.

ping @doppioslash

Any updates on this? I've seen a few times that broken https://arewegameyet.rs and https://arewegameyet.com confused people.


Automated cherry pick of #9701: fix(region): omit vpc quota check if vpc created by owner

Cherry pick of #9701 on release/3.6.
#9701: fix(region): omit vpc quota check if vpc created by owner

Codecov Report

Merging #9702 (e38ea05) into release/3.6 (b9d30c0) will increase coverage by 10.21%.
The diff coverage is 0.65%.

@@               Coverage Diff                @@
##           release/3.6    #9702       +/-   ##
================================================
+ Coverage         5.86%   16.08%   +10.21%
================================================
  Files              828      124      -704
  Lines           129133    12653   -116480
================================================
- Hits              7577     2035     -5542
+ Misses          120876    10427   -110449
+ Partials           680      191      -489

Flag
Coverage Δ

aFlag
16.08% <0.65%> (+10.21%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/apis/input.go
0.00% <ø> (ø)

pkg/apis/list.go
0.00% <ø> (ø)

pkg/cloudcommon/db/quotas/errors.go
0.00% <0.00%> (ø)

pkg/cloudcommon/db/quotas/handler.go
0.00% <0.00%> (ø)

pkg/cloudcommon/db/quotas/register.go
11.11% <0.00%> (-2.41%)
:arrow_down:

pkg/cloudcommon/db/quotas/usageworker.go
0.00% <0.00%> (ø)

pkg/compute/guestdrivers/aliyun.go
2.72% <0.00%> (-0.47%)
:arrow_down:

pkg/compute/guestdrivers/apsara.go
13.63% <0.00%> (-0.65%)
:arrow_down:

pkg/compute/guestdrivers/aws.go
2.27% <0.00%> (-0.34%)
:arrow_down:

pkg/compute/guestdrivers/azure.go
2.54% <0.00%> (-0.29%)
:arrow_down:

... and 636 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4020c6d...e38ea05. Read the comment docs.


Subdomain

Does the wildcard also works for internal subdomains? so are we able to "double-wildcard"?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e4cd6eda-3ed1-d116-628d-e48671aa571b
Version Independent ID: 61aac3c4-128d-b303-3ec7-29bdb92a81b1
Content: Wildcard applications in the Azure Active Directory application proxy
Content Source: articles/active-directory/manage-apps/application-proxy-wildcard.md
Service: active-directory
GitHub Login: @barbkess
Microsoft Alias: barbkess

@pedrr Thanks for your feedback! We will investigate and update as appropriate.

@pedrr Please check on below details in Excluding applications from the wildcard section of the document -
You can also limit the wildcard to only work for specific applications through your DNS management. As a best practice, you should create a CNAME entry that includes a wildcard and matches the format of the external URL you have configured. However, you can instead point specific application URLs to the wildcards. For example, instead of *.adventure-works.com, point hr.adventure-works.com, expenses.adventure-works.com and travel.adventure-works.com individually to 000aa000-11b1-2ccc-d333-4444eee4444e.tenant.runtime.msappproxy.net.

@pedrr We will now proceed to close this thread. If there are further questions regarding this matter, please open a new issue and we will gladly continue the discussion.


Crash in MusicXML score

Hi, I'm hitting a crash in this score. Simply rendering a presenter created from this score should repro:
diff --git a/examples/tutorials/tutorial-1-x11/tutorial-1-x11.cpp b/examples/tutorials/tutorial-1-x11/tutorial-1-x11.cpp
index 69b24b1a..ee1f934f 100644
--- a/examples/tutorials/tutorial-1-x11/tutorial-1-x11.cpp
+++ b/examples/tutorials/tutorial-1-x11/tutorial-1-x11.cpp
@@ -130,15 +130,7 @@ void open_document()
     //and relationships between the document, its views and the interactors
     //to interact with the view
     delete m_pPresenter;
-    m_pPresenter = m_lomse.new_document(k_view_vertical_book,
-        "(lenmusdoc (vers 0.0)"
-            "(content "
-                "(para (txt \"Hello world!\"))"
-                "(score (vers 1.6) "
-                    "(instrument (musicData (clef G)(key C)(time 2 4)(n c4 q) )))"
-            ")"
-        ")",
-        Document::k_format_ldp);
+    m_pPresenter = m_lomse.open_document(k_view_vertical_book, "score.xml");

     //get the pointer to the interactor and register for receiving desired events
     if (SpInteractor spInteractor = m_pPresenter->get_interactor(0).lock())

I was able to reduce it to a (somewhat) minimal test case, but haven't had time yet to investigate beyond that, so I don't know what exactly is triggering this in the score.

Thanks for reporting. Bug confirmed. Unfortunately, I'm going on a trip after  lunch and I'll be away for three weeks so it will be impossible for me to debug and fix this until end of February. Sorry!

Actually maybe I can help with this, as I seem to have this issue fixed in my project. I have not submitted those patches before as I probably intended to rework some of them better before submitting, but they work for their purpose, so I have just opened a pull request (#389) with those patches at their current state.
The source of this issue seems to be a beam going across a bar line:

When it plainly goes like this it shouldn't cause any issues, but with the default page size this bar line appears to be at the end of its system, so the beam eventually becomes cross-system. Lomse currently doesn't handle cross-system beams well, and that makes the application crash.
My solution to this was to add support for breaking beams across systems. Not sure if this is always what the user may expect, but it works and fixes the crash.

Well that's convenient—thanks for sharing your fix!


Allow reducing temporary allocations by making the ipvec function public.

Hi,
First of all, I'm an electronics engineer first and a Rust enthusiast second, so maybe I'm just using your library wrong.
I'm writing a SPICE simulator and chose rsparse as my solver due to its simplicity. In each simulation step, I need to evaluate the circuit elements to get their conductance values and current values. The conductance values populate the system matrix a (a sparse matrix), and the current values go to the known values vector.
After populating the matrix, I use rsparse to solve the system. Here's a minimal example:
use rsparse::{data::{Sprs, Trpl}, lusol};

fn main() {
    // Initial matrix triples
    let mut triples = Trpl::new();
    triples.append(0, 0, 2.0);
    triples.append(1, 1, 2.0);
    triples.append(2, 2, 2.0);

    // Get System Matrix
    let mut system_matrix = Sprs::new_from_trpl(&triples);
    // Known vector
    let mut known_vector = vec![1.0,2.0,3.0];

    // Solve system
    lusol(&system_matrix, &mut known_vector, 1, 1e-6);
    assert_eq!(known_vector,vec![0.5,1.0,1.5]);

    // matrix triples change since the corresponding circuit elements change.
    // Note, that the sparsity pattern does not change
    let mut new_triples = Trpl::new();
    new_triples.append(0, 0, 1.0);
    new_triples.append(1, 1, 1.0);
    new_triples.append(2, 2, 1.0);

    // from tripl, since the memory is already allocated
    system_matrix.from_trpl(&new_triples);

    // Known vector changes too after evaluating circuit elements.
    let mut known_vector = vec![2.0,3.0,4.0];

    // solve again
    lusol(&system_matrix, &mut known_vector, 1, 1e-6);

    assert_eq!(known_vector,vec![2.0,3.0,4.0]);
    // ...
    // Carry on until the Simulation is finished
}

I've benchmarked my simulator and compared it with some publicly available alternatives (notably ngspice). As expected, mine is slower. After analyzing memory consumption, particularly the allocations, I noticed some temporary allocations and avoidable calculations occur inside lusol.
As far as I've noticed, each time I call lusol, the following happens:
An allocation of a vector filled with zeros: let mut x = vec![0.; a.n];
A symbolic analysis: s = sqr(a, order, false);
Since the sparsity pattern doesn't change, I've tried to avoid repeated allocations of x and s by calling lsolve and usolve manually and reusing s:
use rsparse::data::{Sprs, Trpl};

fn main() {
    // Initial matrix triples
    let mut triples = Trpl::new();
    triples.append(0, 0, 2.0);
    triples.append(1, 1, 2.0);
    triples.append(2, 2, 2.0);

    // Get System Matrix
    let mut system_matrix = Sprs::new_from_trpl(&triples);
    // Known vector
    let mut known_vector = vec![1.0,2.0,3.0];

    //allocate Workspace
    let mut x = vec![0.; system_matrix.n];
    // calculate the symbolic analysis
    let mut s= rsparse::sqr(&system_matrix, 1, false); //1 = LU factorisation

    // Solve system :
    let n = rsparse::lu(&system_matrix, &mut s, 1e-6);
    rsparse::ipvec(system_matrix.n, &n.pinv, &mut known_vector, &mut x[..]);
    rsparse::lsolve(&n.l, &mut x); // x = L\x
    rsparse::usolve(&n.u, &mut x[..]); // x = U\x
    rsparse::ipvec(system_matrix.n, &s.q, &x[..], &mut known_vector[..]);

    assert_eq!(known_vector,vec![0.5,1.0,1.5]);

    // matrix triples change since the corresponding circuit elements change.
    // Note, that the sparsity pattern does not change
    let mut new_triples = Trpl::new();
    new_triples.append(0, 0, 1.0);
    new_triples.append(1, 1, 1.0);
    new_triples.append(2, 2, 1.0);

    // from tripl, since the memory is already allocated
    system_matrix.from_trpl(&new_triples);

    // Known vector changes too after evaluating circuit elements.
    let mut known_vector = vec![2.0,3.0,4.0];

    // solve again
    let n = rsparse::lu(&system_matrix, &mut s, 1e-6); //reuse s
    rsparse::ipvec(system_matrix.n, &n.pinv, &mut known_vector, &mut x[..]); //maybe reuse x?
    rsparse::lsolve(&n.l, &mut x); // x = L\x
    rsparse::usolve(&n.u, &mut x[..]); // x = U\x
    rsparse::ipvec(system_matrix.n, &s.q, &x[..], &mut known_vector[..]);

    assert_eq!(known_vector,vec![2.0,3.0,4.0]);
    // ...
    // Carry on until the Simulation is finished
}

The problem is that I can't use ipvec because it is a private function. Is it possible to make ipvec public? Or is there a better solution to my problem?

Thank you for such a detailed issue. I think you are using the library correctly and your solution sounds very sensible. Perhaps I should consider making pvec and ipvec public for use cases such as yours.
Meanwhile, you could copy ipvec into your code and use it directly. It is a really simple permutation function:
fn ipvec(n: usize, p: &Option<Vec<isize>>, b: &[f64], x: &mut [f64]) {
    for k in 0..n {
        if p.is_some() {
            x[p.as_ref().unwrap()[k] as usize] = b[k];
        } else {
            x[k] = b[k];
        }
    }
}

Thanks for the quick response and the easy fix. It seems i couldn't see the forest for the trees.


Improve v3.0 Shader Handling

Summary
There are some rough edges on how v3.0 handles shader compilation and reflection. They should be fixed.
Analysis

Unused shader fields are optimized away by the driver, so they won't get a field location and will be filtered out by the NativeShaderProgram.Compile call on the Duality side.

This means that users can define shader properties, but can't set material values for them unless they're used, or worse, previously set values might be removed if a shader field goes unused in a WiP version when working on a shader.
Change the implementation so that fields that are defined will always show up, regardless of whether they have a field location assigned or not.
Make sure the backend can handle shader fields with an invalid location properly. Consider providing a second code path with valid fields only for internal use in the backend.

Some intel drivers crash with an access violation when encountering #pragma directives they don't know. This is a driver bug, but a workaround would be nice.

Could be achieved by removing or commenting out Duality pragmas before passing them on to the backend.
However, since the backend currently does the shader field parsing, this is currently not viable.
Consider moving all shader field parsing to the core,, maybe in conjunction with ShaderSourceBuilder, so the backend will only get the processed, reduced version of the shader code in the first place.

Progress

Started working on this in the feature/3.0/shader-handling-improvements branch.
Implemented shader field parsing as part of ShaderSourceBuilder.

ToDo

Adjust ShaderSourceBuilder to comment out all Duality specific pragma directives on build to fix the intel driver crash issue.

Refactor shader reflection code to use metadata retrieved by ShaderSourceBuilder instead.
Remove shader field parsing from the graphics backend.

No longer filter out shader fields based on location bindings, keep them for editing and reflection purposes instead.

Make sure the backend can handle shader fields with an invalid location properly.
Consider providing a second code path with valid fields only for internal use in the backend.

Progress

Added type and array declaration unit tests for shader field parsing.
Removed all shader field parsing from the graphics backend.
Shader reflection now uses ShaderSourceBuilder parsing results instead.
Now distinguishing between declared fields (Shader, DrawTechnique) and bound / located fields (NativeShaderProgram).
Rendering is currently broken / blank, probably due to a copy paste error, or forgetting to adjust a spot to match the new parsing code.

ToDo

Fix rendering with the new shader field code.
Make sure BatchInfoPropertyEditor is still fine with the new code.

Also re-check editing shaders and observing property editor updates to match the new field declarations.

Adjust ShaderSourceBuilder to comment out all Duality specific pragma directives on build to fix the intel driver crash issue.
No longer filter out shader fields based on location bindings, keep them for editing and reflection purposes instead.

Make sure the backend can handle shader fields with an invalid location properly.
Consider providing a second code path with valid fields only for internal use in the backend.

Progress

Fixed the rendering / shader issue, which was caused by a bug in ShaderSourceBuilder ignoring an additional line when commenting out version directives, leading to ignoring the builtin view matrix field.
Added a unit test to verify expected behavior.

ToDo

Make sure BatchInfoPropertyEditor is still fine with the new code.

Also re-check editing shaders and observing property editor updates to match the new field declarations.

Adjust ShaderSourceBuilder to comment out all Duality specific pragma directives on build to fix the intel driver crash issue.
No longer filter out shader fields based on location bindings, keep them for editing and reflection purposes instead.

Make sure the backend can handle shader fields with an invalid location properly.
Consider providing a second code path with valid fields only for internal use in the backend.

Done and merged. Closing this.


use ndstore public tokens to get list of public projects for a user specified server

See https://github.com/neurodata/ndviz/commit/4f460164c81a2a95bb6e9ca3626f099cb9a3b18b
Clicking the Manage Layers toolbox (two stacked rectangles, under Image Controls) brings up a window which allows the user to specify a server, get the list of public tokens (or specify their own), and choose from a list of channels for that given token.


neutron metadata plugin is very lxc specific

It may be unavoidable, I don't know, but we rely very heavily on the deployment being lxc/osa based for this plugin to work [1].
Filing this bug to see if there's any way around this to allow osa teams to use the plugin too
[1] https://github.com/rcbops/rpc-openstack/blob/master/maas/plugins/neutron_metadata_local_check.py#L29 && https://github.com/rcbops/rpc-openstack/blob/master/maas/plugins/neutron_metadata_local_check.py#L58

I don't think there is a way around this - unless we monitor ALL agents containers from the deployment host.
We need to get onto the container in order to access the network namespace to perform the test - but we don't have sshkey access from the host --> it's containers, only from the deployment host --> all containers/other hosts (in order for ansible to run properly).
That would mean 1 of 2 options:

We monitor all agents from the deployment host - I think this is a bad idea because all alerts for these agent containers will be targeted to the deployment host... if that host is under maintenance we will get no alerts - there would be no way to change that.
We deploy ssh keys to the hosts - I think this is a bad idea because we are adding a requirement that is otherwise not needed anywhere else, just for the sake of monitoring.

I'd probably mark this as "won't fix" with the above in mind.

@mancdaz I think we should probably close this? Or do you think there is other work we can do here.

Ok so I believe the issue is that this is a problem for osp. I'm not sure the best course of action then - I can't see another way for us to do this, so we'd have to heavily adjust the script (or create a new script) that allows you to just run the name space queries on a specific host without doing the lxc bits (or any other remote connect).
Depending on the architecture for osp.

Because we disabled this check in master and reopened the original issue that caused creation of that check, I think the recommendation should be for OSP to not use this check.

This plugin is not currently used. There is a new issue to track a new implementation https://github.com/rcbops/u-suk-dev/issues/239 so I think this one can be closed.


introduce vscode-repl-font-family

Introduce vscode-repl-font-family css varaible

@mjbvz thank you, this works just fine. Thus merging in.


x-vector: support for efficient append to a grow vector

The new append operation is equivalent to adding each of the elements of a vector one after another to the grow vector, as demonstrated by the property test.
/cc @tranma @amosr @olorin

looks good 👍

LGTM :cat:


Clarification about workflow

I see that the CLI receives as a parameter user_database , just to clarify:

does this refers to the Vector Database in the Architecture diagram and not to the SQL database?

Are the queries based on a single table or does it also relates several tables by means of joins?
If I understand correctly the LLM prompt creation before hitting the SQL Generator joins a lot of elements, like the query + some context about the tables, can you share an example of that prompt, because I guess it would need to add a lot of information in raw format (not vector format).

@rragundez The user_database refers to the schema/dataset against which you would ask questions. It filters the various stuff from the vector embedding tables. This essentially helps to decide which source you are trying to talk from the vector embedding table.
Queries can be asked on multiple tables if they are all in the same schema/dataset.
We are soon going to release the v2 of this solution with multi turn support which has prompts pulled outside in a separate file where you can view the prompts. For now to understand the prompt you have to go to each .py files in /agents  and /dbconnectors folders

If I understand you correctly, a caveat is that it will not be able to relate tables if they are in different big query datasets but if they are in the same big query dataset it will be able to relate tables by means of joins in the generated sql query?
Can you point me to the relevant prompt py file before hitting the SQL generator? Thanks.

Yes, for the main branch your statement holds true. If you have a use case which requires the ability to join tables from different biguqery datasets, please consider using v2-draft branch. Word of caution: v2 version has more features like multi turn support yaml based reading of prompts dynamically etc. Please go through the readme.
On the part regarding code for SQL generation please have a looks at /agents/BuildSQLAgent.py

Hello,
Do you think we should generate and add contents for knowgoodsql whenever a successful query is generated.
As sometimes for the same question the response is generated successfully and sometimes it gives errors.I was testing using bigquery public dataset sample_ecom.
My concern is if it generated a wrong query(not syntax) which will give wrong sql results and if we store that then we are always using the same sql. Maybe a flag needs to be added in the knowgoodsql to label it using human intervention so that only used when set as correct?
Thanks

Yes, the functionality is already provided in the demo UI we have. There is a thumbs up button that inserts the question and SQL back to the SQL embedding table. Please have a look at the backend API section where you can see a URI /embed_sql (Backend APIs).


Infosign icons for trains, fixes #17

Updated to new glyphs for passenger/cargo trains.

Note: I wasn't able to save the PNGs using grey colour palette - shouldn't cause any problems but mentioning it just in case.

The image editing s/w I use doesn't yet support the reduced palette pngs :(

No problem 😉


can set different range of bar for 2 sliders?

Left slider : 1-12
Right Slider : 00:59
In this way, can implement this slider for time.

Not currently, feel free to contribute. Would be hard to express what the ranges mean, as well as the indices being hard to interpret unless the values where equal multiples.


When I use the test method and encounter this problem, I would like to ask you how to solve it

When I use the test method and encounter this problem, I would like to ask you how to solve it，The corresponding weight folder does not correspond after training
FileMotFoundError: No such file or directory: './experiments/ckpt/$lexperiment_name}/ckpt * .weights

Have you solved your problem？


react redux form is now native

https://github.com/davidkpiano/react-redux-form/releases/tag/0.6.0

I'll be sticking with  tcomb-react-native for now.
Thanks for the heads up!


feat: init willPaymasterSponsor

What changed? Why?
In this PR we start writing the isValidSponsorTransaction  utility, which helps verifies that a transaction is valid for sponsorship (aka use Paymaster to avoid paying gas fees).
Notes to reviewers
How has it been tested?

Yeah you right, we should approach smaller chunks at first. @cpcramer take a stub on them and I will close this one for now.


Is it possible to duplicate the editor on one page?

Is it possible to duplicate the editor on one page?

Yes. you can do it with setting el option


Updates to 'Consumption-Tests' for Xcode 12.1

This PR resolves #306:

Update LATEST_SUPPORTED_XCODE_VERSION from 12.0.1 to 12.1
Update SWIFT_VERSION from 5.2 to 5.3

(This should have been done when moving to Xcode 12.x).

Update IPHONEOS_DEPLOYMENT_TARGET from 13.7 to 14.1
Update TVOS_DEPLOYMENT_TARGET from 13.4 to 14.0

NOTES:

The Consumption-Tests were run following these changes to ensure compatibility.

Codecov Report

Merging #307 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #307   +/-   ##
=======================================
  Coverage   86.91%   86.91%
=======================================
  Files          37       37
  Lines        3065     3065
=======================================
  Hits         2664     2664
  Misses        401      401

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f418297...2ad52a4. Read the comment docs.


How to deal with port conflicts when trying to use Bolt with Next.js

Description
When I try to use Bolt with Next.js, there is a conflict with port 3000. As a result, the following error occurs
error - unhandledRejection: Error: listen EADDRINUSE: address already in use :::3000
However, if the port numbers are set to 3000 and 3001 so that there is no conflict, then another problem arises. For example, even if you use /api/slack/events for the request URL and give Bolt's App constructor /api/slack/events as a custom endpoint, the app.event() is not invoked. This is because npm run dev in Next.js is 3000 and app is running at 3001.

What type of issue is this? (place an x in one of the [ ])

[ ] bug
[ ] enhancement (feature request)
[x] question
[ ] documentation related
[ ] example code related
[ ] testing related
[x] discussion

Requirements (place an x in each of the [ ])

[x] I've read and understood the Contributing guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.
[x] I've searched for any related issues and avoided creating a duplicate issue.

Reproducible in:
package version: ??
node version: v14.18.0
OS version(s):
Expected result:
After npm run dev, localhost:3000 will be started. app.start() should also be started with 3000 at this time.
Actual result:
error - unhandledRejection: Error: listen EADDRINUSE: address already in use :::3000

Hi @hiroshinishio, thanks for asking this question!
Unfortunately, bolt-js does not work well as part of a Next.js app due to restrictions on the bolt-js side (meaning, bolt-js is not flexible enough to work along with Next.js, Nuxt.js, and so on). Also, since our team is putting most of our efforts into other projects (specifically, Slack's next-generation platform: https://api.slack.com/future), we don't have any plans to enhance bolt-js to improve this in the short term.
FWIW, A few months ago, I created a 3rd party package, which enables developers to use bolt-js along with Next.js and Nuxt.js. If you're fine with using my library (again, this is my hobby project, and it won't be merged into 1st party SDK), it should work for you without any issues. If you want to fork my repo and customize the details, please feel free to do so! To learn how to use the package, please check the following Next.js app example for details:
https://github.com/seratch/slack-bolt-extensions/tree/main/nextjs-example
I hope this was helpful to you!

I knew it.... Let me ask you a few more questions to clarify.

Could the reason be, as the title of my question suggests, a port conflict?
And your repository is a library that would solve that problem, is that right?
Was it really necessary to create the Slack bot in Next.js in the first place? The only reason why I chose Next.js was because it was easy to create a request URL endpoint and because I was familiar with it myself.

@hiroshinishio

Could the reason be, as the title of my question suggests, a port conflict?

bolt-js's default receiver (HTTPReceiver) listens on 3000 by default. I haven't investigated the direct cause of your situation. Either way, any receivers of bolt-js do not work with Next.js, unfortunately. Resolving the conflict of the ports goes nowhere.

And your repository is a library that would solve that problem, is that right?

Again, the port issue is not the essential problem to resolve when you run bolt-js with Next.js. My library enables bolt-js apps to work properly in a Next.js app, but I have to admit that the approach is a bit hacky way.

Was it really necessary to create the Slack bot in Next.js in the first place? The only reason why I chose Next.js was because it was easy to create a request URL endpoint and because I was familiar with it myself.

If you don't need to go with Next.js for your app this time, I highly recommend running your bolt-js app in a different way.
Is everything clear now? If yes, would you mind closing this issue?

@hiroshinishio
If you're fine with switching the programming language, my best-recommended approach is bolt-python's lazy listeners + AWS Lambda + Amazon S3. This approach is the most cost-effective way to run Slack apps.

https://slack.dev/bolt-python/concepts#lazy-listeners
https://github.com/slackapi/bolt-python/tree/main/examples/aws_lambda

If you want to write your app in JS/TS, running your app on container-based services such as Amazon ECS, EKS, Lightsail Containers, Google's GKE, and so on would be straight forward. I hope this helps.


improve performance of LimitExecutor

Scope & Purpose
Improve performance of LimitExecutor in case the offset is 0.
For simple queries such as
FOR doc IN <collection>
  LIMIT 0, @limit
  RETURN doc

the overhead of LIMIT is quite significant.
This is because LIMIT copies the rows out.
IMO the copying can be avoided if the LIMIT does not have an offset. We can then simply fast-forward the input range.
This reduces the overhead of LIMIT significantly in my own tests for higher LIMIT values.
For example, here are some performance tests results for query
FOR doc IN <collection> LIMIT 0, $limit RETURN doc

for a collection with 11.2m documents on single server, for varying $limit values. Each query was run 5 times and the average query runtime was printed.
Queries were run on single server, and result documents were not returned but immediately discarded.
before (without changes from this PR):
--------------------------------------
LIMIT 1000     => average: 0.0005 s
LIMIT 10000    => average: 0.0048 s
LIMIT 100000   => average: 0.0333 s
LIMIT 1000000  => average: 0.3406 s
LIMIT 10000000 => average: 3.3854 s
LIMIT 20000000 => average: 3.7982 s
LIMIT 984332   => average: 0.3335 s
LIMIT 39983    => average: 0.0138 s
LIMIT 1234567  => average: 0.4174 s

after (with changes from this PR):
-----------------------------------
LIMIT 1000     => average: 0.0005 s
LIMIT 10000    => average: 0.0028 s
LIMIT 100000   => average: 0.0270 s
LIMIT 1000000  => average: 0.2728 s
LIMIT 10000000 => average: 2.7450 s
LIMIT 20000000 => average: 3.0863 s
LIMIT 984332   => average: 0.2742 s
LIMIT 39983    => average: 0.0115 s
LIMIT 1234567  => average: 0.3440 s

[ ] :hankey: Bugfix
[ ] :pizza: New feature
[x] :fire: Performance improvement
[ ] :hammer: Refactoring/simplification

Checklist

[ ] Tests

[ ] Regression tests
[ ] C++ Unit tests
[ ] integration tests
[ ] resilience tests

[ ] :book: CHANGELOG entry made
[ ] :books: documentation written (release notes, API changes, ...)
[ ] Backports

[ ] Backport for 3.11: -
[ ] Backport for 3.10: -

Related Information

[ ] Docs PR:
[ ] Enterprise PR:
[ ] GitHub issue / Jira ticket:
[ ] Design document:

@jsteemann
Could you post your results for such queries?
I just want to understand how big overhead from limit node
1.
FOR doc IN <collection>
  RETURN doc

FOR doc IN <collection>
  LIMIT 1,<PHONE_NUMBER>
  RETURN doc

FOR doc IN <collection>
  LIMIT 1,<PHONE_NUMBER>
  RETURN doc

Note that the rows weren't actually copied before; still, this PR removes some (constant) per-line overhead.

@goedderz
But why is overhead so big? Can we do something in aql to reduce it?

@goedderz But why is overhead so big? Can we do something in aql to reduce it?

@MBkkt I didn't have the time to profile it this time. It might be generic overhead of the AQL state machine in ExecutionBlockImpl. It might be the leftover per-row overhead in countAndSkipAllRemainingDataRows; it's there to find the next shadow row (this could be improved by making use of the _shadowRows data structure in AqlItemBlock to find the next shadow row with less steps).
But let's also compare this with other runtimes. FOR i IN 1..1000000 LIMIT 1000000 RETURN i has this profile (without this patch):
Execution plan:
 Id   NodeType            Calls     Items   Filtered   Runtime [s]   Comment
  1   SingletonNode           1         1          0       0.00002   * ROOT
  2   CalculationNode         1         1          0       0.00003     - LET #1 = 1 .. 1000000   /* range */   /* simple expression */
  3   EnumerateListNode    1000   1000000          0       0.15436     - FOR i IN #1   /* list iteration */
  4   LimitNode            1000   1000000          0       0.02014       - LIMIT 0, 1000000
  5   ReturnNode           1000   1000000          0       0.02006       - RETURN i

While 0.02s isn't completely negligible in comparison with the 0.15s of the list iteration, it pretty much shows it'll probably be negligible in all real-world scenarios, for two reasons: 1) The cost per row of any reasonable query will be significantly higher than that of a list iteration, and 2) limit is seldomly used with such large line numbers.
So while further optimizing it might be possible and interesting, I don't think its currently worth the effort.


EAGLE-706 Policy UI adv mode support sql highlight

Add ace editor to make policy edit page sql highlight

LTGM

LGTM


Network issue

After installing spice drivers internet in browser works but system still can't see network connection and I can't login my Microsoft account.
Configuration

UTM Version: 2.0.27
OS Version: big sur
Intel or Apple Silicon? Apple Silicon

As far as I can tell the system is able to access the Internet but can’t ping to verify that it can. It’s a somewhat minor issue but it is very annoying as it took me 2 or 3 times to log into my Microsoft account. A fix would be awesome.

One more thing, also because of this problem I can’t sign in windows insider program and install new insider builds.

ping doesn't work due to a limitation of QEMU. Future builds will support bridged networking which should be better.


The analyze mode should not care about comments

Users of the analyze mode are already applying the changes manually, so it's fine to strip comments away. This is especially good if doing so allows Resyntax to discover more issues.

I've greatly improved Resyntax's ability to preserve comments automatically (see #247 and #243). There's still some gaps, but I believe most situations are fixable now. I've also added debug logging, which records when a suggestion is discarded due to dropped comments. Given all that, I will close this for now, as it should be unnecessary.


[rel-1.2.0] Remove vendor directory

What this PR does / why we need it:
Cherry-pick of #115
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

Removed vendor directory

/lgtm

/kind enhancement


HDDS-4419. Misleading SCM web UI Safe mode status

What changes were proposed in this pull request?
Improve the usability of SafeMode rule table on the web ui.
See the reports in the JIRA.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4419
How was this patch tested?
Started and checked the UI

Looks good to me, but I was confused initially. This PR was raised for HDDS-4419, isn't it?
Would be nice to take a screenshot, though not required.

Looks good to me, but I was confused initially. This PR was raised for HDDS-4419, isn't it?

yes, sorry for the confusion. Fixing.
(Second PR where I made this mistake. For me this double accounting (jira + github) couldn't help.)

This is the screenshot (of the relevant part):

Merging it now. Thanks the review @jojochuang

Merging it now. Thanks the review @jojochuang


Fix issues #2 and #3, so the project works again on latest streamlits

Closes #2, #3.
(update: since these are blocking bugs, meaning the main project cannot be used at all on modern versions of streamlit and python, you can find my fork of this library here, and use it in your projects instead https://pypi.org/project/wfork-streamlit-profiler/)
I basically did the minimal thing that would work.
It seems to me that the "styling stuff" you were worried about still works fine, if that just means the profiler panel looks the same. (Tested with Python 3.10.12)

Let me know if there's anything else I need to investigate wrt this PR.

Thanks! Would love to see this PR published on pypi @jrieke

I was running into the same issue and I can confirm that it now works as expected with this PR (Python 3.12.4 and Streamlit 1.36). I also verified that the changed import works on older versions of Streamlit (Python 3.9.19 and Streamlit 1.0). Apparently, it was never meant to be used directly as st.components.v1.html (see streamlit/streamlit#8644).

@hendriklohse in case it's helpful to you or others in your situation I should note that if your system has git then you can depend directly upon my git repo for my fixed version of this package. Make sure to use the patch-1 branch instead of main! Probably also leave a comment by it explaining this, because if this PR gets incorporated I'll probably remove my git repo.
I'm also considering publishing my fixed version as a package on pypi, although the git solution works so well it's hard to say why I would do so.

I realized I could save a minute on one of my deployments by doing this, actually, so my fix is now available at https://pypi.org/project/wfork-streamlit-profiler/

On reflection, I think this commit might in fact "break some of the styling stuff", because I think there's supposed to be a dark mode for pyinstrument, or at least the readme for current pyinstrument has a darkly-styled html display https://github.com/joerick/pyinstrument , and I've only ever gotten the light mode to display.

On reflection, I think this commit might in fact "break some of the styling stuff", because I think there's supposed to be a dark mode for pyinstrument, or at least the readme for current pyinstrument has a darkly-styled html display https://github.com/joerick/pyinstrument , and I've only ever gotten the light style of pyinstrument to display.

I checked this, but it appears dark mode of pyinstrument was never working with streamlit-profiler. I checked with streamlit-profiler 0.2.4 with py-instrument 4.1.1 on Python 3.9.21 both with Streamlit 1.0.0 and Streamlit 1.41.1.


traefik v2.11.2 changes readTimeout and requires helm chart v28.0.0+ to configure

Is your feature request related to a problem? Please describe.
My k3s cluster recently upgraded and thus the app version of traefik changed from v2.10.7 to v2.11.8 see commit
In v2.11.2 traefik changed the default readTimeout from 0s (no timeout) to 60s. https://doc.traefik.io/traefik/migration/v2/#entrypointtransportrespondingtimeoutsreadtimeout
This causes some of my requests to the cluster to fail.
This is configurable via the traefik helm chart, see commit however this is only available in chart versions v28.0.0 and higher.
Describe the solution you'd like
Upgrade the traefik helm chart to v28.0.0 or higher so that the timeout can be configured.
Describe alternatives you've considered
I can manually pin my traefik app version to the older version to restore behavior, but ideally we can use the newer version so that bug fixes and security patches are pulled in.

You can add CLI flags via any version of the chart... just do this:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--entryPoints.web.transport.respondingTimeouts.readTimeout=120s"
      - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=120s"


No button to contiue the user journey after EBSI/ESSIF onboarding

The there should be a button to bring the user to the did:list
The image on this page should be centered.

Closing this issue as the code in this repository is deprecated (as its name implies). For a more up-to-date web wallet UI, refer to our more recent community edition identity stack that is actively maintained and updated in the waltid-identity repository.


Hardcode causes a panic in key.go

Progam crashed when the length of input string more then 256.
https://github.com/charmbracelet/bubbletea/blob/248eb83001a7ab6a11c80534bd4e3e96a6185f4a/key.go#L568-L575

This may potentially be solved by https://github.com/charmbracelet/bubbletea/pull/570/files


Dynamically Scoped Macro Example

I was trying to run this example from the README
foo[X, Y] -> bar["moo" | "oink" | "baa"] $Y
bar[Z]    -> $X " " $Z # 'remembers' $X from its caller
main -> foo["Cows", "."]

But running nearleyc throws this error
Error: nearley: No possible parsings (@23: '|').
For a more minimal failing case:
hi -> test["merp" | "derp"]```

Oh. The stuff inside should be in parens -- test[("merp" | "derp")]. Sorry for the misleading documentation — I'll fix that.

Thanks!


Automatic Differentiation

This would be cool: http://en.wikipedia.org/wiki/Automatic_differentiation
Theano uses this.
This should be a secondary objective. Initially (and for performance reasons) it will make sense to provide the derivatives of functions, however this feature will provide extensibility for later unknown model types.

The cost is longer compile time which is very counter-productivity especially for quick iterations of research and prototyping.

Yup,  mentioned above already

There is nothing fundamental about automatic differentiation that associates it with longer compile times. Some folks from Berkeley are implementing an alternative to Theano (rll.berkeley.edu/cgt/), which essentially solves the long compilation problem.

So I've actually have already implemented a baby step of this, however an old version of it in Rust. As mentioned in #28 I find Arrayfire the perfect backend for this. Haven't check CGT by myself, but I can see it is still python based. I prefer a C++ version of this rather than python, and later it could be ported to other languages easily. However it requires also and some knowledge and expectation of the backend. I'm quite happy to work on it but wanted to hear what you guys think. Another nice part is that you can apply methods such as CRISP(https://cs.nyu.edu/web/Research/TechReports/TR1995-698/TR1995-698.pdf) which can benefit both for paralelization and memory usage on the device.
There isn't a specific literature on the topic, since it is quite simple. Have the optimization of the graph is not so trivial to do in constant number of passes. This usually include mathematical identity simplification, groupings, possible register allocations (or blobs of memory) and deciding which operations can be inlined. Any way I think for ML this is the standard since it gives you order of magnitute benefit on prototyping and research as well as run time speed. If the graph optimization is made well compilation times are going to be a few seconds longer, which if you get a boost in run time does not really matter.

I also suggest this post: http://colah.github.io/posts/2015-08-Backprop/index.html
The automatic differentiation doesn't need to be extremely fast. Most of the time all you need is symbolic derivatives after which you can substitute values back into the result (at a rapid rate).

Mm in general you would want to do source generation, otherwise just substitution will mean that you will have to do manual GC on the graph, rather than let the compiler do what it does best.
I was wondering if I implement this on a purely graph based structure, what are the important parts of each node that would be required for ArrayFire to know? Usually each node has some meta data like:

ndim
shape (not necessarily constant might depend on some inputs)
broadcastable dimensions
operator (obviously)

Would it be required to explicitly know the device on which the memory will live or anything else that I'm missing (here please excuse me, but I really never used ArrayFire before, just yd started trying it).

@Botev We do most of what you suggest internally inside arrayfire's JIT engine. @jramapuram opened a ticket on upstream arrayfire library to perform Automatic differentiation.
I am fairly new to the subject. All I've seen / read so far is what other DNN libraries are doing to a certain extent. However I haven't yet figured out the scope of operations to support.

Hmm, so could you point me to a document so that I can understand better what happens under the hood of array fire JIT. Are you building a symbolic graph of the computation? Or are you referring to storing the dim, shape, broadcasts etc of each variable?
I must say that I'm really quite far from low level HPC, so have no idea how you guys do the computation inside. To most extend what other DNN libraries do is defacto the same, just have different ways of expressing it. The most general currently is Theano, which represents gradients as operations on the graph (adding nodes), which imho is the best, since it allows to easily later to derive things like Hessain vector products in a very natural way. In terms of operations to be supported, in general the more you have the better. Ah, I can write lots on this topic, but as I said I'm a noob on HPC and have no idea how you guys manage stuff under the hood.

I think the tensorflow implementation is far superior to theano. Here is their whitepaper: http://download.tensorflow.org/paper/whitepaper2015.pdf

@Botev It is internal to arrayfire so it is not very well documented..

This file is representative of how a node is constructed: https://github.com/arrayfire/arrayfire/blob/devel/src/backend/cuda/binary.hpp

This file shows how the graph is traversed and the (nvvm) code is generated:
https://github.com/arrayfire/arrayfire/blob/devel/src/backend/cuda/jit.cpp

This directory (only the .hpp files) have implementation of different types of nodes https://github.com/arrayfire/arrayfire/tree/devel/src/backend/cuda/JIT

If I can understand the theory well enough, I think jit.cpp and header files inside JIT directory can be extended to implement automatic differentiation.

@jramapuram  The automatic differentiation part is the same. If you look at section 4.1 and observe a Theano graph with gradients you will say the way that gradients are symbolically computed are exactly the same (this is also true for torch, cgt and almost any other good DNN library out there). The difference is in how they manage the exact computation, code generation, compilation etc later

@Botev : sure, in the end you are just running reverse mode differentiation, and using the sum & product rule of graphs to get the gradient of the destination node w.r.t. the head node. But as you mention there is a lot of features that they support that optimize things (eg: partial graph utilization, etc).

In ML, specially Deep Learning, which are usually the main targets for such libraries, these are more or less sugar, which to help do things or investigate after training. Whether, I will construct a brand new graph or reuse the old one, imho is not so important. The device management and execution however, can have significant impact, which is why I in first place came here. You are right that they have a bit more things, however they have and less - there a number of operators they do not support on the gpu or do not have. Additionally, I do not know (did not find any resources) of what exactly optimizations they perform on the graph in order to compare it with other frameworks.

Yea, you are right in that some operators are not supported on the GPU (try using the ADAM optimizer, there is a mean() call somewhere that seems to need CPU), however it's really interesting how they allow for 'loose' computations which dynamically toggle the backend.  Of course the performance suffers from this. From this regard I think arrayfire has a huge advantage as there are implementations of basically all reduction algorithms you might need, etc.
Anyhow, in general it would be a very useful feature to have in the main library in general. Graph derivatives are useful in plenty of places besides just ML. Here it the other ticket in the upstream that I submitted: https://github.com/arrayfire/arrayfire/issues/1098

Hmm one problem I can think of is that matrix multiply is not part of the JIT engine in arrayfire. Only element wise operations are part of it.

Aren't those at least reference there for calls to the BLAS or some routines?

@Botev any chance you are on gitter? I'd like to discuss some of this via chat if you can drop by


Play 2.6x will not run if https port is set

When using Play version 2.6.x and https port is set via JVM options runPlayBinary will not start play server. No errors.
But with Play version 2.5.x it works fine.
Expected Behavior
runPlayBinary runs Play server with https enabled
Current Behavior
runPlayBinary will not run Play server
Context
The problem can be reproduced with help of this starter application https://github.com/marcospereira/play-java-starter-example/tree/2.6.x
Steps to Reproduce (for bugs)

Clone project by the link above (play-java-starter-example/tree/2.6.x)
Replace build.gradle with next one to reproduce the problem

plugins {
    id 'play'
    id 'idea'
}

def playVersion = "2.6.12"
def scalaVersion = System.getProperty("scala.binary.version", /* default = */ "2.12")

model {
    components {
        play {
            platform play: playVersion, scala: scalaVersion, java: '1.8'
            injectedRoutesGenerator = true

            sources {
                twirlTemplates {
                    defaultImports = TwirlImports.JAVA
                }
            }
        }
    }
}

tasks.withType(PlayRun) {
    forkOptions.jvmArgs = ["-Dhttps.port=9443"]
}

dependencies {
    play "com.typesafe.play:play-guice_$scalaVersion:$playVersion"
    play "com.typesafe.play:play-logback_$scalaVersion:$playVersion"
    play "com.h2database:h2:1.4.196"

    playTest "org.assertj:assertj-core:3.6.2"
    playTest "org.awaitility:awaitility:2.0.0"
}

repositories {
    jcenter()
    maven {
        name "lightbend-maven-releases"
        url "https://repo.lightbend.com/lightbend/maven-release"
    }
    ivy {
        name "lightbend-ivy-release"
        url "https://repo.lightbend.com/lightbend/ivy-releases"
        layout "ivy"
    }
}

Use next build file to reproduce working example for Play 2.5.x

plugins {
    id 'play'
    id 'idea'
}

def playVersion = "2.5.1"
def scalaVersion = System.getProperty("scala.binary.version", /* default = */ "2.11")

model {
    components {
        play {
            platform play: playVersion, scala: scalaVersion, java: '1.8'
            injectedRoutesGenerator = true

            sources {
                twirlTemplates {
                    defaultImports = TwirlImports.JAVA
                }
            }
        }
    }
}

tasks.withType(PlayRun) {
    forkOptions.jvmArgs = ["-Dhttps.port=9443"]
}

dependencies {
    play "com.typesafe.play:play-logback_$scalaVersion:$playVersion"
    play "com.h2database:h2:1.4.196"

    playTest "org.assertj:assertj-core:3.6.2"
    playTest "org.awaitility:awaitility:2.0.0"
}

repositories {
    jcenter()
    maven {
        name "lightbend-maven-releases"
        url "https://repo.lightbend.com/lightbend/maven-release"
    }
    ivy {
        name "lightbend-ivy-release"
        url "https://repo.lightbend.com/lightbend/ivy-releases"
        layout "ivy"
    }
}

Your Environment
Reproduced with gradle 4.6

Yes. Closed.


Show approval modals within the wallet when user interacts with Dapps

Describe the bug
This is not the bug rather or current implementation. But when I was testing uniswap I was observing that the modals were opening within the app. Should we implement such logic everywhere in wallet?
To Reproduce
Steps to reproduce the behavior:

Go to uniswap
Click on connect wallet
Made some Tx
Observe where popups will be opened

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: MacOS
Browser:  chrome
Version: 123

Additional context
Add any other context about the problem here.

Fixed within the wallet connect PR. Can be closed.


did you consider adding support for markdown

At the moment, to get a beautiful markdown in slack, I need to convert input first with this action https://github.com/LoveToKnow/slackify-markdown-action. My input message looks like [${{github.event.issue.title}}](${{github.event.issue.html_url}}) \n ${{github.event.issue.body}} so I really cannot manually make it "slack" compatible as I do not have control over issue body.
but it looks like it is a simple as adding https://github.com/jsarafajr/slackify-markdown and enabling someone to specify if they want to slackify markdown in message or not.

Hey @derberg, that is a great suggestion. We will add native support to slackify the markdown in SLACK_MESSAGE if an env var like say SLACKIFY_MARKDOWN is set to true with the help of https://github.com/jsarafajr/slackify-markdown
PRs are welcome for this enhancement. Otherwise, I will later pick this up. But that may take some time as my schedule is packed as of now.

Hey @derberg, that is a great suggestion. We will add native support to slackify the markdown in SLACK_MESSAGE if an env var like say SLACKIFY_MARKDOWN is set to true with the help of https://github.com/jsarafajr/slackify-markdown
PRs are welcome for this enhancement. Otherwise, I will later pick this up. But that may take some time as my schedule is packed as of now.

@mrrobot47 yeah, sorry but go is not on my list of skills 😄 and I've already set a workaround https://github.com/asyncapi/.github/pull/18/files#diff-1ee55436eb74158dd30293540d530e66f2179cc688feb6c365f90977bdad5d8cR20-R31 that for now is fine for me. I just wanted to drop an idea as I've spent a lot of time reviewing different slack related actions, and this action was the best one after all

@mrrobot47 yeah, sorry but go is not on my list of skills 😄 and I've already set a workaround https://github.com/asyncapi/.github/pull/18/files#diff-1ee55436eb74158dd30293540d530e66f2179cc688feb6c365f90977bdad5d8cR20-R31 that for now is fine for me. I just wanted to drop an idea as I've spent a lot of time reviewing different slack related actions, and this action was the best one after all

@mrrobot47 and updates about this?

Support for Markdown has been added by using slackify-markdown-action as suggested by the OP. Use SLACKIFY_MARKDOWN="true" to enable markdown to slack-formatting conversion.


pull request for e2e test at 01/26/2017 21:45:11

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


[WIP] Fix flaky diagnostic checks

Description
This patch intends to adjust all diagnostic checks to be robust on bring-up (i.e. not generate spurious faults).
Type of change
Please mark options that are relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] The code builds clean without any errors or warnings
[ ] Unit tests are passing
[x] Linter tests are passing

How To Test
TBD

Unnecessary and superseded by https://github.com/ChrisScianna/ROS-Underwater-RnD/pull/91.


Cannot find Chrome binary

I'm trying out webdrivers version 5.3.0 which appears to be using Chrome for Testing, but I'm seeing this error when running capybara selenium specs in a Rails 7 app under Ruby 2.7.
Selenium::WebDriver::Error::UnknownError:
  unknown error: cannot find Chrome binary
# 0   chromedriver                        0x0000000101d89a58 chromedriver + 4991576
# 1   chromedriver                        0x0000000101d80fa3 chromedriver + 4956067
# 2   chromedriver                        0x0000000101933607 chromedriver + 443911
# 3   chromedriver                        0x00000001019614b7 chromedriver + 631991
# 4   chromedriver                        0x000000010196064c chromedriver + 628300
# 5   chromedriver                        0x00000001019a924c chromedriver + 926284
# 6   chromedriver                        0x00000001019a85af chromedriver + 923055
...

I'm on Intel Mac OS X 13.3 with Chrome 115 installed.
Any suggestions on how to troublshoot this?
Thanks your work on keeping webdrivers gem alive for those of us on Ruby 2.7.

Turn on logging, let's see what was requested:
Selenium::WebDrivers.logger.level = :debug

I am getting ```
Failure/Error: driven_by :selenium, using: :headless_chrome, screen_size: [1400, 2000]
      VCR::Errors::UnhandledHTTPRequestError:

        ================================================================================
        An HTTP request has been made that VCR does not know how to handle:
          GET https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json

        There is currently no cassette in use. There are a few ways
        you can configure VCR to handle this request:

from running rspec with the new 5.3.1 version.

I'm seeing the same issue even on webdrivers v5.2. This started when upgrading chromedriver from v114.x to v115.x. Here is the output of the logging:
2023-08-01 07:02:08 DEBUG Selenium Executing Process ["/opt/homebrew/bin/chromedriver", "--port=9515"]
2023-08-01 07:02:08 DEBUG Selenium polling for socket on ["<IP_ADDRESS>", 9515]
Starting ChromeDriver 115.0.5790.102 (90efd4b0ad6aa15eeafcdabd5817ae939f7ba059-refs/branch-heads/5790_90@{#9}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
2023-08-01 07:02:08 INFO Selenium -> POST session
2023-08-01 07:02:08 INFO Selenium    >>> http://<IP_ADDRESS>:9515/session | {"capabilities":{"alwaysMatch":{"browserName":"chrome","goog:chromeOptions":{"args":["--headless","--disable-site-isolation-trials"]}}}}
2023-08-01 07:02:08 DEBUG Selenium      > {"Accept"=>"application/json", "Content-Type"=>"application/json; charset=UTF-8", "User-Agent"=>"selenium/4.3.0 (ruby macosx)", "Content-Length"=>"136"}
2023-08-01 07:02:08 INFO Selenium <- {"value":{"error":"unknown error","message":"unknown error: cannot find Chrome binary","stacktrace":"0   chromedriver                        0x00000001030e2924 chromedriver +
4303140\n1   chromedriver                        0x00000001030db050 chromedriver + 4272208\n2   chromedriver                        0x0000000102d0f328 chromedriver + 291624\n3   chromedriver
  0x0000000102d37a7c chromedriver + 457340\n4   chromedriver                        0x0000000102d36ffc chromedriver + 454652\n5   chromedriver                        0x0000000102d76040 chromedriver + 712768\n6
chromedriver                        0x0000000102d7580c chromedriver + 710668\n7   chromedriver                        0x0000000102d408d4 chromedriver + 493780\n8   chromedriver                        0x0000000102
d4171c chromedriver + 497436\n9   chromedriver                        0x00000001030a37dc chromedriver + 4044764\n10  chromedriver                        0x00000001030a7d20 chromedriver + 4062496\n11  chromedriver
                        0x00000001030adf40 chromedriver + 4087616\n12  chromedriver                        0x00000001030a8824 chromedriver + 4065316\n13  chromedriver                        0x0000000103080d1c chr
omedriver + 3902748\n14  chromedriver                        0x00000001030c4414 chromedriver + 4178964\n15  chromedriver                        0x00000001030c456c chromedriver + 4179308\n16  chromedriver
               0x00000001030d4830 chromedriver + 4245552\n17  libsystem_pthread.dylib             0x000000019a81bfa8 _pthread_start + 148\n18  libsystem_pthread.dylib             0x000000019a816da0 thread_start +
 8\n"}}
2023-08-01 07:02:09 DEBUG Selenium Executing Process ["/opt/homebrew/bin/chromedriver", "--port=9713"]
2023-08-01 07:02:09 DEBUG Selenium polling for socket on ["<IP_ADDRESS>", 9713]
Starting ChromeDriver 115.0.5790.102 (90efd4b0ad6aa15eeafcdabd5817ae939f7ba059-refs/branch-heads/5790_90@{#9}) on port 9713
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
2023-08-01 07:02:09 INFO Selenium -> POST session
2023-08-01 07:02:09 INFO Selenium    >>> http://<IP_ADDRESS>:9713/session | {"capabilities":{"alwaysMatch":{"browserName":"chrome","goog:chromeOptions":{"args":["--headless","--disable-site-isolation-trials"]}}}}
2023-08-01 07:02:09 DEBUG Selenium      > {"Accept"=>"application/json", "Content-Type"=>"application/json; charset=UTF-8", "User-Agent"=>"selenium/4.3.0 (ruby macosx)", "Content-Length"=>"136"}
2023-08-01 07:02:09 INFO Selenium <- {"value":{"error":"unknown error","message":"unknown error: cannot find Chrome binary","stacktrace":"0   chromedriver                        0x00000001030e2924 chromedriver +
4303140\n1   chromedriver                        0x00000001030db050 chromedriver + 4272208\n2   chromedriver                        0x0000000102d0f328 chromedriver + 291624\n3   chromedriver
  0x0000000102d37a7c chromedriver + 457340\n4   chromedriver                        0x0000000102d36ffc chromedriver + 454652\n5   chromedriver                        0x0000000102d76040 chromedriver + 712768\n6
chromedriver                        0x0000000102d7580c chromedriver + 710668\n7   chromedriver                        0x0000000102d408d4 chromedriver + 493780\n8   chromedriver                        0x0000000102
d4171c chromedriver + 497436\n9   chromedriver                        0x00000001030a37dc chromedriver + 4044764\n10  chromedriver                        0x00000001030a7d20 chromedriver + 4062496\n11  chromedriver
                        0x00000001030adf40 chromedriver + 4087616\n12  chromedriver                        0x00000001030a8824 chromedriver + 4065316\n13  chromedriver                        0x0000000103080d1c chr
omedriver + 3902748\n14  chromedriver                        0x00000001030c4414 chromedriver + 4178964\n15  chromedriver                        0x00000001030c456c chromedriver + 4179308\n16  chromedriver
               0x00000001030d4830 chromedriver + 4245552\n17  libsystem_pthread.dylib             0x000000019a81bfa8 _pthread_start + 148\n18  libsystem_pthread.dylib             0x000000019a816da0 thread_start +
 8\n"}}

So many different overlapping bugs during this mess...
chromedriver 115 currently has a bug on Mac where it is looking for a binary "Google Chrome for Testing" instead of just "Google Chrome." Since most people didn't install Chrome that way it is erroring. Google made a fix, it hasn't been released, yet.
Selenium 4.11:

identifies the browser on the system
gets the right driver for that browser
stores the information of browser location and tells the driver to use that location when starting a new session

Webdrivers can't do the third thing, so you have to use Chrome::Options#binary= to set the path to the browser yourself.
This is one of the reasons why the Selenium solution is better if you can upgrade to it.

Just to followup on this: Chrome 116 is out now and webdrivers 5.3.0 works like a charm. I'm no longer seeing the above error. Thanks again Titus for keeping the gem going.


🐛 Fix bug dealing with large source files

When work with Kubernetes clusters, snakemake often needs to upload many local files, such as the snakemake job specification file, user-made scripts, and any other files needed by the job, to the cluster. To be more specific, snakemake registers these files as Kubernetes secrets and then mount them as container volumes.
However, the Kubernetes API mandates any secret value smaller than 1MB, otherwise, the API throws an ApiException with error code 422. An example can be found at the bottom of the description.
A previous pull request (https://github.com/snakemake/snakemake/pull/368) solved (partially) this issue by ignoring any file larger than 1MB. However, secret values are just file contents BASE64 encoded. Therefore, for those close to 1MB, their BASE64 encoded values might exceed the 1MB limit, thus causes error while creating Kubernetes pods.
For that reason, we not only need to exclude files larger than 1MB, but also check their BASE64 encoded values and make sure the limit is not exceeded.
Example for the bug:
Full Traceback (most recent call last):
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/snakemake/__init__.py", line 687, in snakemake
    success = workflow.execute(
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/snakemake/workflow.py", line 905, in execute
    scheduler = JobScheduler(
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/snakemake/scheduler.py", line 245, in __init__
    self._executor = KubernetesExecutor(
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 1468, in __init__
    self.register_secret()
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 1514, in register_secret
    self.kubeapi.create_namespaced_secret(self.namespace, secret)
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 8166, in create_namespaced_secret
    return self.create_namespaced_secret_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 8261, in create_namespaced_secret_with_http_info
    return self.api_client.call_api(
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 391, in request
    return self.rest_client.POST(url,
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/kubernetes/client/rest.py", line 274, in POST
    return self.request("POST", url,
  File "/Users/haizi/miniconda3/lib/python3.8/site-packages/kubernetes/client/rest.py", line 233, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Audit-Id': '0ff45d6a-ecb5-4a9b-9585-eb70d5b4fcf8', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 17 Nov 2020 00:59:17 GMT', 'Content-Length': '404'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Secret \"b4246b7b-4c6f-48d0-b4fb-d7aeb5422d94\" is invalid: data: Too long: must have at most 1048576 bytes","reason":"Invalid","details":{"name":"b4246b7b-4c6f-48d0-b4fb-d7aeb5422d94","kind":"Secret","causes":[{"reason":"FieldValueTooLong","message":"Too long: must have at most 1048576 bytes","field":"data"}]},"code":422}

Just realized that, the real limitation is that, the TOTAL size of the entire secret map should be less than 1MB. Therefore, it's not enough by just skipping individual large files. Instead, we should check the total size of all files and envvars combined.
In addition, while all secret entries are stored in BASE64 encoded form, the total size should be calculated from the original data, i.e. BASE64 decoded, as the following code demonstrates:
# Calculate the total size of the secret map
sum([len(base64.b64decode(v)) for k, v in secret.data.items()])

If the total size is less than 1MB, we are good to go.
I updated the code so that when the total size exceeds the 1MB restriction, snakemake will issue some warnings and display 5 largest files, so that the user knows by how much it exceeds, and what files to exclude.
An example:
Building DAG of jobs...
Skipping the source file scripts/20200817-week-34/presentation.nb.html. Its size 4110950 exceeds the maximum file size (1MB) that can be passed from host to kubernetes.
Skipping the source file scripts/20200427-week-18/hi-c-playground/lieberman-aiden-analysis.nb.html for secret key f104. Its base64 encoded size 1080960 exceeds the maximum file size (1MB) that can be passed from host to kubernetes.
Skipping the source file scripts/20200427-week-18/compartment-analysis/compartment-analysis.nb.html. Its size 2585816 exceeds the maximum file size (1MB) that can be passed from host to kubernetes.
The total size of the included files and other Kubernetes secrets is 1133976, exceeding the 1MB limit.

The following are the largest files. Consider removing some of them (you need remove at least 85400 bytes):
  * File: lab_notebook/images/Screen Shot 2020-08-16 at 13.48.46.png, original size: 490354
  * File: lab_notebook/journal_2020_vol_3.Rmd, original size: 45187
  * File: lab_notebook/journal_2020_vol_1.Rmd, original size: 43869
  * File: scripts/20200817-week-34/endpoint_density_model.pptx, original size: 37383
  * File: scripts/20200813-week-33/cofrag_architecture.pptx, original size: 37124

Full Traceback (most recent call last):
  File "/home/parallels/miniconda3/lib/python3.8/site-packages/snakemake/__init__.py", line 687, in snakemake
    success = workflow.execute(
  File "/home/parallels/miniconda3/lib/python3.8/site-packages/snakemake/workflow.py", line 905, in execute
    scheduler = JobScheduler(
  File "/home/parallels/miniconda3/lib/python3.8/site-packages/snakemake/scheduler.py", line 245, in __init__
    self._executor = KubernetesExecutor(
  File "/home/parallels/miniconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 1468, in __init__
    self.register_secret()
  File "/home/parallels/miniconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 1546, in register_secret
    raise WorkflowError("ConfigMap too large")
snakemake.exceptions.WorkflowError: ConfigMap too large

Thanks a lot, good catch!


Tryin to use it with nvm on Windows throws "SyntaxError: Unexpected reserved word"

Describe the bug
Tryin to use it with nvm on Windows throws:
file:///C:/Users/User/AppData/Roaming/nvm/v12.22.12/node_modules/@microsoft/inshellisense/build/index.js:13
.version(await getVersion(), "-v, --version", "output the current version")
^^^^^
SyntaxError: Unexpected reserved word
←[90m    at Loader.moduleStrategy (internal/modules/esm/translators.js:140:18)←[39m
←[90m    at async link (internal/modules/esm/module_job.js:42:21)←[39m
To Reproduce
Steps to reproduce the behavior:

Open cmd
Install npm install -g @microsoft/inshellisense
Run inshellisense bind
See error

Expected behavior
I expect to not get an error
Environment

OS: Microsoft Windows [Version 10.0.19045.3570]
Output of inshellisense --version: same output as in 'Describe the bug' section
Nodejs Version: [v12.22.12]

you are using outdated node version.
sudo npm install -g n sudo n lts

Windows doesn't have the sudo command....

you are using outdated node version.
sudo npm install -g n sudo n lts

Windows doesn't have the sudo command....

@ycodetech I've found this guide helpful for managing node versions on Windows & here is the installer for the latest Node LTS which supports Windows. inshellisense only supports Node >= 16.X

you are using outdated node version.
sudo npm install -g n sudo n lts

Windows doesn't have the sudo command....

@ycodetech I've found this guide helpful for managing node versions on Windows & here is the installer for the latest Node LTS which supports Windows. inshellisense only supports Node >= 16.X

Oh I know. I've had Node installed for a year or so, and l have no problems using it. I was merely stating the fact the comment using sudo is incorrect for Windows as it doesn't have the command internally.


RAGLayer error when number of objects <= 2

No issue - RegionAdjacencyLayer must be called with adjacency matrix prior to graph coloring.