Clarification needed on TSV_DIRECTORY for Training after using prepare-experiment

Following the execution of the clinicadl tsvtools prepare-experiment --n_test 0.10 -vt kfold --n_validation 10 labels/labels.tsv command to create splits for experimentation, I am facing confusion regarding the correct specification of the TSV_DIRECTORY for subsequent training (clinicadl train classification [OPTIONS] CAPS_DIRECTORY PREPROCESSING_JSON TSV_DIRECTORY OUTPUT_MAPS_DIRECTORY).
I am utilizing the latest development version of ClinicaDL on Ubuntu 20.04, with a dataset containing T1w images sourced from ADNI. Despite various attempts, such as specifying labels/split, errors persist, including messages like "No validation.tsv found." Additionally, using labels/split/10-fold results in errors such as "No train.tsv found."
While opting for labels/split/10-fold/split-1 would train each split independently, this method may not be the most efficient. Clear guidelines are required regarding the expected folder structure and the correct approach for specifying the TSV folder for training subsequent to the prepare-experiment step.
Could you please investigate and provide guidance on establishing an appropriate TSV folder structure to ensure seamless training post-split creation?
Thank you.

Hello @souravraha !
The correct specification for the TSV_DIRECTORY is labels/split/10-fold. However, you need to specify in your training command line that you're performing cross-validation by using the --n_splits option to indicate the number of splits (k) in the case of k-fold cross-validation.
Your command line should look like this:
clinicadl train classification [OPTIONS] CAPS_DIRECTORY PREPROCESSING_JSON TSV_DIRECTORY OUTPUT_MAPS_DIRECTORY --n_splits 10
Do not hesitate if you have any other questions,
Camille

Hello @souravraha !
The correct specification for the TSV_DIRECTORY is labels/split/10-fold. However, you need to specify in your training command line that you're performing cross-validation by using the --n_splits option to indicate the number of splits (k) in the case of k-fold cross-validation.
Your command line should look like this: clinicadl train classification [OPTIONS] CAPS_DIRECTORY PREPROCESSING_JSON TSV_DIRECTORY OUTPUT_MAPS_DIRECTORY --n_splits 10
Do not hesitate if you have any other questions, Camille

Thanks! This works.
As I run this, it displays
Could not determine if CAPS is a CAPS or BIDS directory. Clinica will assume this is a CAPS directory.
over and over. Is there a way out of this?


CI: Update Python 3.8-dbg job to ubuntu-20.04

Bumping the ubuntu to focal updates python-3.8-dbg package to 3.8.10 (earlier 3.8.0 in bionic) provided by the package manager. This should fix the failing python3.8-dbg CI job in #15607. Tested in anirudhdagar/scipy#7.
[skip azp]
cc @rgommers

I'm curious, even after adding skip azp, it did not skip azure jobs.

Yes, seems broken.

I did not, I was guessing that you or some other maintainer did that saving compute, given that this should not affect any other job.

Yeah, guess so. Or it was a fluke. Either way, this looks good.

Thanks! I'll sync #15607 now.


pth_to_hf error: Phi3ForCausalLM does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention

Trying to follow the instruction here
stuck at first step:
xtuner convert pth_to_hf llava_phi3_mini_4k_instruct_clip_vit_large_p14_336_e1_gpu8_sharegpt4v_pretrain  ./iter_9742.pth ./iter_9742_xtuner
Here is the error:
[2024-04-30 17:35:58,185] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)  [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH  [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3  [WARNING]  using untested triton version (2.3.0), only 1.0.0 is known to be compatible [2024-04-30 17:36:02,546] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)  [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH  [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3  [WARNING]  using untested triton version (2.3.0), only 1.0.0 is known to be compatible flash-attentionpackage not found, consider installing for better performance: No module named 'flash_attn'. Currentflash-attentondoes not supportwindow_size. Either upgrade or use attn_implementation='eager'. Traceback (most recent call last):   File "/mnt/task_runtime/xtuner/xtuner/tools/model_converters/pth_to_hf.py", line 143, in <module>     main()   File "/mnt/task_runtime/xtuner/xtuner/tools/model_converters/pth_to_hf.py", line 65, in main     model = BUILDER.build(cfg.model)   File "/miniconda/envs/xtuner-env/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build     return self.build_func(cfg, *args, **kwargs, registry=self)   File "/miniconda/envs/xtuner-env/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg     obj = obj_cls(**args)  # type: ignore   File "/mnt/task_runtime/xtuner/xtuner/model/llava.py", line 42, in __init__     self.llm = self._build_from_cfg_or_module(llm)   File "/mnt/task_runtime/xtuner/xtuner/model/llava.py", line 239, in _build_from_cfg_or_module     return BUILDER.build(cfg_or_mod)   File "/miniconda/envs/xtuner-env/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build     return self.build_func(cfg, *args, **kwargs, registry=self)   File "/miniconda/envs/xtuner-env/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg     obj = obj_cls(**args)  # type: ignore   File "/miniconda/envs/xtuner-env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 558, in from_pretrained     return model_class.from_pretrained(   File "/miniconda/envs/xtuner-env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3544, in from_pretrained     config = cls._autoset_attn_implementation(   File "/miniconda/envs/xtuner-env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1463, in _autoset_attn_implementation     config = cls._check_and_enable_sdpa(   File "/miniconda/envs/xtuner-env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1619, in _check_and_enable_sdpa     raise ValueError( ValueError: Phi3ForCausalLM does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention yet. Please request the support for this architecture: https://github.com/huggingface/transformers/issues/28005. If you believe this error is a bug, please open an issue in Transformers GitHub repository and load your model with the argument attn_implementation="eager"meanwhile. Example:model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="eager")
where I got the pth file from here:
https://huggingface.co/xtuner/llava-phi-3-mini-pretrain
Cuda version: 12.2
torch version: 2.3.0+cu121
transformers: 4.40.1

issue was with flash attention

@rezacopol
even the issue was with flash attention, I wonder how to avoid it in xtuner.
for now I set attn_implementation="eager", does this produce expect to be the same accuray after training?
thx.

@rezacopol even the issue was with flash attention, I wonder how to avoid it in xtuner.
for now I set attn_implementation="eager", does this produce expect to be the same accuray after training?
thx.

Which part should I change?

How to solve this? Thanks


tests: add tests for more primitive types

PR Type
Tests, Enhancement

Description

Enhanced serialization and deserialization capabilities for multiple structs and enums by adding serde::Deserialize, fake::Dummy, and PartialEq derive macros.
Replaced Debug derive macro with DebugAsJson for several types to improve JSON debugging output.
Updated test generation in the primitives module, adding new tests and removing outdated TODOs.
Refactored the gen_test_serde! macro in src/ext.rs to use more robust JSON value comparisons and removed strict binary equality checks for bincode serialization.
These changes improve the testability and serialization consistency across various types in the project.

Changes walkthrough 📝
Relevant filesEnhancement9 files

evm_result.rsAdd derive macros to EvmExecutionResult

src/eth/executor/evm_result.rs
Added serde::Deserialize, fake::Dummy, and PartialEq derive macros to EvmExecutionResult struct

+1/-1

block_filter.rsUpdate derive macros for BlockFilter

src/eth/primitives/block_filter.rs
Added DebugAsJson and fake::Dummy derive macros to BlockFilter enum  Removed Debug derive macro

+3/-1

execution.rsReplace Debug with DebugAsJson for EvmExecution

src/eth/primitives/execution.rs
Replaced Debug derive macro with DebugAsJson for EvmExecution struct

+2/-2

execution_account_changes.rsReplace Debug with DebugAsJson for ExecutionAccountChanges

src/eth/primitives/execution_account_changes.rs
Replaced Debug derive macro with DebugAsJson for ExecutionAccountChanges struct

+3/-1

execution_conflict.rsAdd derive macros to ExecutionConflict and ExecutionConflicts

src/eth/primitives/execution_conflict.rs
Added serde::Deserialize, fake::Dummy, and PartialEq derive macros to ExecutionConflict enum  Added serde::Deserialize derive macro to ExecutionConflicts struct

+2/-2

log_filter.rsAdd derive macros to LogFilter

src/eth/primitives/log_filter.rs
Added serde::Deserialize and fake::Dummy derive macros to LogFilter struct

+1/-1

log_filter_input.rsAdd fake::Dummy derive macro to LogFilterInput structs

src/eth/primitives/log_filter_input.rs
Added fake::Dummy derive macro to LogFilterInput and LogFilterInputTopic structs

+2/-2

transaction_execution.rsAdd derive macros to LocalTransactionExecution

src/eth/primitives/transaction_execution.rs
Added serde::Deserialize, fake::Dummy, and PartialEq derive macros to LocalTransactionExecution struct

+1/-1

transaction_mined.rsReplace Debug with DebugAsJson for TransactionMined

src/eth/primitives/transaction_mined.rs
Replaced Debug derive macro with DebugAsJson for TransactionMined struct

+2/-1

Tests2 files

mod.rsUpdate test generations in primitives module

src/eth/primitives/mod.rs
Commented out several TODO test generations  Added new test generations for various types  Removed FIX comments

+22/-20

ext.rsRefactor gen_test_serde! macro for improved testing

src/ext.rs
Updated gen_test_serde! macro to use serde_json::to_value and from_value instead of to_string and from_str  Removed assertion for exact string equality in encoded JSON  Removed assertion for exact binary equality in encoded bincode

+6/-7

💡 PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

/describe


[#1287] add params to transform and refine

This PR solve the issue #1287 by adding the ability to add a params object as optional argument to parse functions, this params argument will be available in transform and refine callbacks to perform more customizable data checks and transformation.
I will extend this PR with a few doc when I'll get some time.

This is a good PR but I'm generally opposed to contextual validations. Zod schemas should be self-contained, and schema that rely on params break encapsulation. https://github.com/colinhacks/zod/issues/487#issuecomment-860231900

@colinhacks
Ok, I'm not sure of the benefits of keeping shema self-contained no mater what as for certain use cases it would lead into generating the schema in the data validation process. Which is, I think, a bad pattern as it would cause performance issues as you would re-declare a new schema object each time you want to perform a validation losing all the benefits of declaring a reusable self-contained schema.
I think it's a trade off between strict encapsulation and performance and even if I won't encourage the pattern I think it's not the role of the library to make this choice for the users. But a warning about encapsulation breaking of this pattern could be nice in the the documentation of the functionality.

I'm closing this PR after @colinhacks arguments and example of workaround mentioned in https://github.com/colinhacks/zod/issues/487#issuecomment-860231900
I'm not 100% sure about the argument of strict encapsulation but I understand it and acknowledge its relevance.
I would also argu that strict typings for that functionnality would be a big work that make this PR not mature enough for merge.
Feel free to take back this code if you have interest on it and want to go further on this subject.


Fixes #931 Spotify Playlist Recommendation

Pull Request for PyVerse 💡
Requesting to submit a pull request to the PyVerse repository.

Issue Title
Spotify Playlist Recommendation

[X] I have provided the issue title.

Info about the Related Issue
This project aims to build a recommender system for music artists using collaborative filtering.

[X] I have described the aim of the project.

Name
Varunshiyam

[X] I have provided my name.

GitHub ID
Varunshiyam

[X] I have provided my GitHub ID.

Email ID
<EMAIL_ADDRESS>
[X] I have provided my email ID.

Identify Yourself
Gssoc, hacktoberfest

[X] I have mentioned my participant role.

Closes
Enter the issue number that will be closed through this PR.
Issue #931

[X] I have provided the issue number.

Describe the Add-ons or Changes You've Made
Added up m project files

[X] I have described my changes.

Type of Change
Select the type of change:

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Code style update (formatting, local variables)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Tested by myself by reviewing it

[X] I have described my testing process.

Checklist
Please confirm the following:

[X] My code follows the guidelines of this project.
[X] I have performed a self-review of my own code.
[X] I have commented my code, particularly wherever it was hard to understand.
[X] I have made corresponding changes to the documentation.
[X] My changes generate no new warnings.
[X] I have added things that prove my fix is effective or that my feature works.
[X] Any dependent changes have been merged and published in downstream modules.

@Varunshiyam why is there this file in this PR?
Deep_Learning/YouTube_Spam_Detector/youtube-comments-spam-detection-f1-score-96.ipynb

@UTSAVS26  Sry for the Inconvenience I caused. Yep, I made it right check it out .


[Feature Request] Send emails

It would be interesting to have the ability not only to receive emails but also to send them. This could possibly be done by using Cloudflare's Worker directly (documentation linked at the end of this issue) or with MailChannels as the sender.
Documentation for sending emails with Cloudflare's Worker: Cloudflare Email Workers Documentation

Hey @fernandodilland this is already on the roadmap to be added, I'm currently working on the new dashboard #44 before picking this up


sstabledump failed because AzureSnitch ClassNotFoundException

Issue description

[ ] This issue is a regression.
[ ] It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.
Impact
since 5.2 we officially support Azure, and in this job, we are upgrading from 5.2.6 to 2023.1.0-rc8
in this test, we run the following command, to ensure it works:
for i in `sudo find /var/lib/scylla/data/keyspace_complex/ -type f |grep -v manifest.json |grep -v snapshots |head -n 1`; do echo $i; sudo sstabledump $i 1>/tmp/sstabledump.output || exit 1; done

but it failed:
Exit code: 1
Stdout:
/var/lib/scylla/data/keyspace_complex/user_with_ck-93a3e700360611eebccc2e7155ec929d/me-57-big-Index.db
Stderr:
at org.apache.cassandra.tools.SSTableExport.<clinit>(SSTableExport.java:69)
Caused by: java.lang.ClassNotFoundException: org.apache.cassandra.locator.AzureSnitch
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.cassandra.utils.FBUtilities.classForName(FBUtilities.java:556)
... 6 more

it happened once we upgraded all the nodes
How frequently does it reproduce?
did run it once, but it is clear that it will fail on every run
Installation details
Kernel Version: 5.15.0-1042-azure
Scylla version (or git commit hash): 5.2.6-20230730.58acf071bf28 with build-id 17961be569f8503b27ff284a8de1e00a9d83811e
Target Scylla version (or git commit hash): 2023.1.0~rc8-20230731.b6f7c5a6910c with build-id f6e718548e76ccf3564ed2387b6582ba8d37793c
Cluster size: 4 nodes (Standard_L8s_v3)
Scylla Nodes used in this run:

rolling-upgrade--ubuntu-focal-db-node-a0dba1a8-eastus-4 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a0dba1a8-eastus-3 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a0dba1a8-eastus-2 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a0dba1a8-eastus-1 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)

OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/SCYLLA-IMAGES/providers/Microsoft.Compute/images/scylla-5.2.6-x86_64-2023-08-01T05-27-35 (azure: eastus)
Test: rolling-upgrade-azure-image-test
Test id: a0dba1a8-303f-46d7-8aba-0986c71db060
Test name: enterprise-2023.1/rolling-upgrade/rolling-upgrade-azure-image-test
Test config file(s):

rolling-upgrade.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor a0dba1a8-303f-46d7-8aba-0986c71db060
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs a0dba1a8-303f-46d7-8aba-0986c71db060

Logs:

db-cluster-a0dba1a8.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/db-cluster-a0dba1a8.tar.gz
sct-runner-events-a0dba1a8.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/sct-runner-events-a0dba1a8.tar.gz
sct-a0dba1a8.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/sct-a0dba1a8.log.tar.gz
monitor-set-a0dba1a8.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/monitor-set-a0dba1a8.tar.gz
loader-set-a0dba1a8.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a0dba1a8-303f-46d7-8aba-0986c71db060/20230808_192101/loader-set-a0dba1a8.tar.gz

Jenkins job URL
Argus

@DoronArazii FYI

Issue reproduced but it's not reproducing in all the runs.
Issue description

[ ] This issue is a regression.
[ ] It is unknown if this issue is a regression.

Installation details
Kernel Version: 5.15.0-1042-azure
Scylla version (or git commit hash): 5.2.6-20230730.58acf071bf28 with build-id 17961be569f8503b27ff284a8de1e00a9d83811e
Target Scylla version (or git commit hash): 2023.1.0-20230813.68e9cef1baf7 with build-id c7f9855620b984af24957d7ab0bd8054306d182e
Cluster size: 4 nodes (Standard_L8s_v3)
Scylla Nodes used in this run:

rolling-upgrade--ubuntu-focal-db-node-a838df17-eastus-4 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a838df17-eastus-3 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a838df17-eastus-2 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-a838df17-eastus-1 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)

OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/SCYLLA-IMAGES/providers/Microsoft.Compute/images/scylla-5.2.6-x86_64-2023-08-01T05-27-35 (azure: eastus)
Test: rolling-upgrade-azure-image-test
Test id: a838df17-0472-45e1-90ce-0bf4422a23e3
Test name: enterprise-2023.1/rolling-upgrade/rolling-upgrade-azure-image-test
Test config file(s):

rolling-upgrade.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor a838df17-0472-45e1-90ce-0bf4422a23e3
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs a838df17-0472-45e1-90ce-0bf4422a23e3

Logs:

db-cluster-a838df17.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/db-cluster-a838df17.tar.gz
sct-runner-events-a838df17.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/sct-runner-events-a838df17.tar.gz
sct-a838df17.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/sct-a838df17.log.tar.gz
monitor-set-a838df17.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/monitor-set-a838df17.tar.gz
loader-set-a838df17.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/a838df17-0472-45e1-90ce-0bf4422a23e3/20230815_103803/loader-set-a838df17.tar.gz

Jenkins job URL
Argus

The issue reproduced with 2023.1.1-20230903
Does this issue should be opened in scylla-enterprise-tools-java as well?
Issue description

[ ] This issue is a regression.
[ ] It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.
Impact
Describe the impact this issue causes to the user.
How frequently does it reproduce?
Describe the frequency with how this issue can be reproduced.
Installation details
Kernel Version: 5.15.0-1045-azure
Scylla version (or git commit hash): 2023.1.1-20230903.68fa1ae4eb94 with build-id 15dd29a34d99a3c9b2196c362264a50c1579085a
Target Scylla version (or git commit hash): 2023.1.1-20230906.f4633ec973b0 with build-id b454e7a22f80cf71a33b2f39e47127225e8fbc13
Cluster size: 4 nodes (Standard_L8s_v3)
Scylla Nodes used in this run:

rolling-upgrade--ubuntu-focal-db-node-7ef06c73-eastus-4 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-7ef06c73-eastus-3 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-7ef06c73-eastus-2 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)
rolling-upgrade--ubuntu-focal-db-node-7ef06c73-eastus-1 (<IP_ADDRESS> | <IP_ADDRESS>) (shards: 7)

OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/SCYLLA-IMAGES/providers/Microsoft.Compute/images/scylla-2023.1.1-x86_64-2023-09-03T19-09-47 (azure: eastus)
Test: rolling-upgrade-azure-image-test
Test id: 7ef06c73-07f3-466f-9836-e2e5e64e54dd
Test name: enterprise-2023.1/rolling-upgrade/rolling-upgrade-azure-image-test
Test config file(s):

rolling-upgrade.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 7ef06c73-07f3-466f-9836-e2e5e64e54dd
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 7ef06c73-07f3-466f-9836-e2e5e64e54dd

Logs:

db-cluster-7ef06c73.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/db-cluster-7ef06c73.tar.gz
sct-runner-events-7ef06c73.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/sct-runner-events-7ef06c73.tar.gz
sct-7ef06c73.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/sct-7ef06c73.log.tar.gz
monitor-set-7ef06c73.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/monitor-set-7ef06c73.tar.gz
loader-set-7ef06c73.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/7ef06c73-07f3-466f-9836-e2e5e64e54dd/20230906_152409/loader-set-7ef06c73.tar.gz

Jenkins job URL
Argus

It will take less time for @denesb to replace sstabledump with a native tool than fix this, I reckon.

scylla sstable is ready to use for some time now. It is also included in 5.2. There is one inconvenience though, the 5.2. version of scylla-sstable does not yet know to read the schema from system tables, so it has to be provided in a separate schema.cql file.

scylla sstable is ready to use for some time now. It is also included in 5.2. There is one inconvenience though, the 5.2. version of scylla-sstable does not yet know to read the schema from system tables, so it has to be provided in a separate schema.cql file.

so a snapshot is needed  first ? to generate the schema.cql?
can you backport the code that read the system tables ? into 5.2/2023.1 ?

scylla sstable is ready to use for some time now. It is also included in 5.2. There is one inconvenience though, the 5.2. version of scylla-sstable does not yet know to read the schema from system tables, so it has to be provided in a separate schema.cql file.

so a snapshot is needed first ? to generate the schema.cql?

Either that or produce it manually.

can you backport the code that read the system tables ? into 5.2/2023.1 ?

I will look into backporting it. It is a well delimited change, not affecting other parts of the system, so it is not a risk to scylla itself.

Backport PR: https://github.com/scylladb/scylladb/pull/15386

@denesb I guess this can be closed , right ?

@denesb I guess this can be closed , right ?

The backport PR was not merged yet.

This is unlikely to get fixed, as newer (native scylla sstable ... tool) is working well and we have no plans to fix it for the Java based tool.

It seems like @yaronkaikov considered this good to close, as soon as https://github.com/scylladb/scylladb/pull/15386 lands in 5.2, and it already did.

It seems like @yaronkaikov considered this good to close, as soon as scylladb/scylladb#15386 lands in 5.2, and it already did.

@denesb - I'm not sure I understand - was this fixed? If it was, then it wasn't fixed properly (see https://github.com/scylladb/scylladb/issues/17659 )

It seems like @yaronkaikov considered this good to close, as soon as scylladb/scylladb#15386 lands in 5.2, and it already did.

@denesb - I'm not sure I understand - was this fixed? If it was, then it wasn't fixed properly (see scylladb/scylladb#17659 )

No, we didn't do anything with sstabledump. Instead, the "fix" was backporting enough of the scylla-sstable improvements that went in after 5.2, to make it usable enough to replace sstabledump in daily usage.

It seems like @yaronkaikov considered this good to close, as soon as scylladb/scylladb#15386 lands in 5.2, and it already did.

@denesb - I'm not sure I understand - was this fixed? If it was, then it wasn't fixed properly (see scylladb/scylladb#17659 )

No, we didn't do anything with sstabledump. Instead, the "fix" was backporting enough of the scylla-sstable improvements that went in after 5.2, to make it usable enough to replace sstabledump in daily usage.

so it's there in 5.2, and we can backport the usage of the new tool into SCT branch-5.2 ?

@vponomaryov
once @denesb confirm it's in place,
we'll need to backport
https://github.com/scylladb/scylla-cluster-tests/pull/6463

so it's there in 5.2, and we can backport the usage of the new tool into SCT branch-5.2 ?

Well... I backported the schema auto-detect, but not the bunch of follow-up fixes we found when using SCT w.r.t. EAR (enterprise). So maybe if you enable it in SCT, you will start hitting problems.


Add knative-release-leads to OWNERS

Enable release leads to merge go mod changes, etc

Looks like we need to add the "update-community" action for hack

/assign

/verify-owners

/verify-owners


docs: added a guidelines section

Added a guidelines section to include best practices, for now this only includes installation.
Closes https://github.com/arcus-azure/arcus.scripting/issues/284

Should we then remove this from the DevOps page? Or do you think that since it's mostly in DevOps scenario's that this problem occur, that we would leave it there?

Should we then remove this from the DevOps page? Or do you think that since it's mostly in DevOps scenario's that this problem occur, that we would leave it there?

I would keep it in, people can overlook the guidelines page and by having it in the DevOps page we make sure the correct way is used.


User Story: Proxy has Documentation

This is a collective tracking of what is needed for documentation.

[x]  How to customize YARP with pipleine modules #114
[x] Authentication #416
[ ] ~Configuring https #305~
[x] configuration #183
[x] migration #495

1.0 shipped with docs, this is done.


Missing or misplaced "old" config folder?

Hopefully this is an easy one, but I've looked through the entire repository history and this one has eluded me:
Does the old/ repository listed in the README indeed exist? And if so, where is it? It doesn't appear to be in this folder unlike what the README seems to indicate (if I'm not missing something):
https://github.com/state-spaces/s4/blob/a246043077dbff8563a6b172426443dced9a9d96/configs/experiment/README.md?plain=1#L8
I had hoped to run the original S4 experiment on Speech Commands, which would be in this elusive folder according to:
https://github.com/state-spaces/s4/blob/a246043077dbff8563a6b172426443dced9a9d96/models/s4/experiments.md?plain=1#L67
So far, no luck finding this experiment configuration anywhere though :-(
Thanks for the help!

Don't remember what happened to that folder, but the whole point of deprecating it is that no one should have to use it anymore! (Maybe it was actually just from the internal repo and never meant to be released?)

I see. Well, as far as I can tell, it was indeed never released in the github repo (as far as my git search shows). It was only referenced in READMEs.
I guess based on the READMEs I don't totally grasp the significance of these experiments. As far as I can tell, they were the ones corresponding to the original S4 paper so it would seem like it could be a shame not to be able to reproduce it anymore.
But maybe I misunderstand the READMEs and there are ways to do so with the "new" configs.
(Note: I had started looking into the old configurations because I couldn't get the new s4-sc config to run, but that's a different issue than the one here, so maybe I'll open another one for it if I can still not figure it out)
Thanks for the help!

The SC experiments should have later configs that produce results as good or better than the originals, in the same training setting and using the most recent code. So it is probably better to use those.
If you're having trouble getting the s4-sc config to run, then that is indeed an issue. Please file a separate GH issue!

Sounds good, thanks for the clarification. I thought the best way forward was to make a pull request to remove the mentions to that folder, since it doesn't exist anyway. That should avoid any future confusion.


fix(actions-header): Do not force textAlign

Description
By reordering the spread, make it possible for headerStyle to
override the text alignment.
Impacted Areas in Application
List general components of the application that this PR will affect:

MTableHeader

Additional Notes
This is optional, feel free to follow your hearth and write here :)

Hi , ty. that's should be the right order since we only apply the default value now.
Will be merged and released today.


Workspace images

Generalizes WSS markup functions to IO actions
add Image widget to Workspace buttons, and include the hbox for more general control
add imageSelector to allow endusers to write their own image selection functions

this is a breaking change and it doesnt provide an elegant ootb solution for images. however, folks who dont want images dont have to do anything, and folks who do can write an imageSelector. if someone feels ambitious, a default imageSelector could be written to give a decent ootb experience.
i dont expect you to merge this, but it would be really great if you did, or impld something like it, not least because i could stop maintaining a fork.

@travitch will you merge it? Or make the alternative?
@teleshoes are you still use your fork?

yep, i use it every day, and i would SO DEARLY LOVE not to have to keep managing these awful merge conflicts every few months. naturally, they get worse each time.
personally, i agree that the feature is not polished enough for wide use, but i DO think that my code could form the foundation of a full-fledged impl of this feature, and its not like it hurts things. (afaik, only a handful of humans on the face of the earth use my fork)

see #167

#167 was merged!


Add info for Full vs Partial ingest

Description
Update ingest request template to indicate Full vs Partial ingest

@ericlingerfelt It won't let us add reviewers. Can you take a look when you get the chance? Thanks!

@eap @ytremolet


cgroups not accounting for relative paths that happen when reading from a container mount

The /proc/[pid]/cgroup file contains the paths to the cgroups used by the process. Most importantly, this path is relative to the cgroup of the process checking the cgroup. From the man page:
              [3]  This field contains the pathname of the control group
                   in the hierarchy to which the process belongs.  This
                   pathname is relative to the mount point of the
                   hierarchy.

This means that if you mount in a process from the host system into a container, you get a relative path:
 docker exec -it 5b938d5468d3 /bin/bash
metricbeat@motmot:~$ cat /hostfs/proc/114856/cgroup
0::/../elastic-agent.service

Right now this library doesn't seem to cope with this, as it assumes a universal base path:
{"log.level":"debug","@timestamp":"2024-03-08T20:32:58.467Z","log.logger":"processes","log.origin":{"file.name":"process/process.go","file.line":173},"message":"Error fetching PID info for 1023161, skipping: cgroups.GetStatsForPid: error fetching cgroupV2 controllers for cgroup location '/hostfs/sys/fs/cgroup' and path line '0::/../../user.slice/user-1000.slice/session-212.scope': open /hostfs/sys/user.slice/user-1000.slice/session-212.scope: no such file or directory","service.name":"metricbeat","ecs.version":"1.6.0"}

I'm not sure why we didn't catch this before? Perhaps there's a conflux of different docker versions, bugs and configs that have escaped most people.

A temporary workaround is to set --cgroupns host, but that obviously has some downsides.


SLDSDatePicker bug fix: Fixing issue where year picker doesn't show years correctly

Fix menu items not showing up correctly
Fix console warning showing up when selecting menu item
Set slds-datepicker class on menu item to set correct font size of date picker

Fixes #1504
Additional description

CONTRIBUTOR checklist (do not remove)
Please complete for every pull request

[ ] First-time contributors should sign the Contributor License Agreement.  If you haven't before, wait a few minutes and a bot will comment on this pull request with instructions.
[ ] npm run lint:fix has been run and linting passes.
[ ] Mocha, Jest (Storyshots), and components/component-docs.json CI tests pass (npm test).
[ ] Tests have been added for new props to prevent regressions in the future. See readme.
[ ] Review the appropriate Storybook stories. Open http://localhost:9001/.
[ ] The Accessibility panel of each Storybook story has 0 violations (aXe). Open http://localhost:9001/.
[ ] Review tests are passing in the browser. Open http://localhost:8001/.
[ ] Review markup conforms to SLDS by looking at DOM snapshot strings.

REVIEWER checklist (do not remove)

[ ] TravisCI tests pass. This includes linting, Mocha, Jest, Storyshots, and components/component-docs.json tests.
[ ] Tests have been added for new props to prevent regressions in the future. See readme.
[ ] Review the appropriate Storybook stories. Open http://localhost:9001/.
[ ] The Accessibility panel of each Storybook story has 0 violations (aXe). Open http://localhost:9001/.
[ ] Review tests are passing in the browser. Open http://localhost:8001/.
[ ] Review markup conforms to SLDS by looking at DOM snapshot strings.
[ ] Add year-first date and commit SHA to last-slds-markup-review in package.json and push.
[ ] Request a review of the deployed Heroku app by the Salesforce UX Accessibility Team.
[ ] Add year-first review date, and commit SHA, last-accessibility-review, to package.json and push.
[ ] While the contributor's branch is checked out, run npm run local-update within locally cloned site repo to confirm the site will function correctly at the next release.

Looks good!


apps/utterances landing page needs more guidance

Please provide more info at https://github.com/apps/utterances, since this page is linked to as "Learn more about utterances" in the permissions flow.  Users not familiar with utterances or the concept of blog comments hosted by github will need a better orientation to decide whether they want to grant permissions.
Thank you for considering

GitHub limits the amount of text we're able to put in the description. I wasn't able to add much more detail unfortunately.
https://github.com/apps/utterances


change how futures are set up in the worker events test

fixes #665
this was breaking the tests, possibly through a race condition OR there was some ophyd change

Happy to approve once CI passes

the tests now are timing out. those tests are quite complex
git log points to @abbiemery as the author, could you take a look please maybe it would be faster
https://github.com/DiamondLightSource/blueapi/pull/224


.NET Core 3.0 preview 8 winforms designer crashes

Issue Title
.NET Core 3.0 preview 8 winforms designer crashes
General
.NET Core 3.0.100-preview8-013656
Visual Studio 16.3.0 Preview 2
After the project is created, try to switch to the designer.
The following screen appears

@zsd4yr PTAL?

@EddieRich 	Please tell me more about your scenario here.
Are you opening a .NET Framework or .NET Core WinForms project? Is there anything special or interesting about the project?
Does this repro with latest VS?
FYI @merriemcgaw 	@RussKie @AdamYoblick

The WinForms Designer for .NET Core is not yet available.

@zsd4yr, I create a new .NET Core 3.0 Winforms project. VS2019 creates it and then I try to open the Visual Designer and it says it can't find a TypeConverter.

We have a doc describing how to leverage the .NET Framework Windows Forms designer for .NET Core - https://github.com/dotnet/winforms/blob/master/Documentation/winforms-designer.md.
Apologies, I should have linked this earlier.


Feature: Efficient update adjacent nodes

Great project. Love the DomDiff.
One thing that I think may be improved is the updating of same-kind adjacent nodes. Right now, if I insert (for example) a new LI node at the beginning of a UL list, since the diff algorithm doesn't has any context, it doesn't knows that the list can be moved down instead of have to update every single LI element AND adding a new LI at the end.
On a library I'm working on (not related) I faced the same problem with objects, where I could not equate them using ==, so I had to resort to some identifier of the object (usually a property named id).
I think that maybe you can also use the same approach, by using a custom attribute/data attribute. What are your thoughts on this?

Something I've struggled with a bit myself. I'm not sure what the right approach is, but it would be nice to be able to do. I'm gonna leave this open until I have a better sense of how I'd approach it.
I don't think you can trust a unique ID since the user could override it, and I don't think I want to add attributes/properties to the DOM that the user didn't themselves code in.

Well.. I was thinking on the user (developer) be responsible of the attr, not the library. So I can create the attr based on some internals. And of course, if I change the attr, or delete it, it is because I should know what I'm doing :)

We could extend the way we use Array.prototype.map with that piece of code :
let seed = 1;
const genID = () => seed++;

function mapWithKey(items, key, cb){
  const getKey = (item) => {
    if (typeof key === 'function')
	return key(item) || genID();
  else
    return item[key] || genID();
  }
  return items.map((item, index) => cb(item, getKey(item), index, items), items);
}

Then, it's up to the developer to use it or not within his data structure and template:
...
   data: {
    todos: [{id:11, do:'Swim'}, {id:1, do:'Climb'}, {id:2, do:'Jump'}, ...]
  },
  template: function (props) {
    return `

        `${mapWithKey(props.todos, 'id', (todo, key) =>
          `<li data-id="${key}">${todo.do}</li>`).join('')}`

`;
  }
...

Finally, within diff algorithm, you could check for a childNode.dataset.Id prop and wether present or not choose the right strategy.

Wanted to provide an update here: I've got an idea on how to address this, at least partially.
Currently, if the existing UI has more elements than the template, Reef just chops the extra UI elements out of the UI before diffing. Similarly, if the new UI has more elements, it just replaces them inline as it diffs.
A smarter way to handle this would be to check the element type while diffing, and if they don't match, insert a new element before it and keep going. Then, handle removing extras at the end.
This should in theory cut down on massive UI updates.
It doesn't exactly address the ordered list situation like you've got going on there, and I do need to figure out something there. I don't want to add any attributes to the UI that don't exist in the template, but I'm sure there's a way to make this work, too.

Again, just thinking aloud on this:

It doesn't exactly address the ordered list situation like you've got going on there, and I do need to figure out something there.

Setting IDs, even internal ones, doesn't help much since when the template changes I still need to map that to real UI. I think the solution for sequential elements of the same type is going to involve comparing things like ID and innerHTML. It won't be perfect, but it will be a start.

Setting internal ids will help you determine if a node has been moved, deleted or added.
First render:

...

...

...

Second render

...

...

...

You can easily see that node 3 has been moved, node 1 has been deleted and node 4 has been added.
When I created the issue, I was thinking in this kind of modifications (that are the most common). Not, for example, moving one of the li from one ul to another ul.

It doesn’t though. On the second render, either the IDs will all be different, or a new node injected in the middle of the existing ones would get the idea of an element that’s already in the UI.
this wouldn’t work any differently than what already happens now.

What I mean is, the item with the id of 4 in your example would actually be given an idea of 1. There’s absolutely no way for Reef to give that an ID of 4 unless the user specifically provided one in the template

Basically, when you start diff-ing, you create a map of nodes with data-reef-id attrs and their positions inside its parent and later you scan that map for changes when you pass a new rendered view.
(you answered too fast...)
But that's why I said that it is up to the developer to add those attributes. He will know from where to obtain those ids so they match on each render.

But reef has to convert the template string into HTML elements to do the diffing.
The “internal map” wouldn’t help here at all.

Oh, and having a developer add them: that’s really no different than them adding IDs or some other unique non-custom attribute.
if the idea is to have the library handle this automatically, forcing the developer to add custom attributes to every element seems to work against that goal.

Of course that you can't do that. It is impossible to do that (assign the ids automatically) from your code (reef). I get that, and only what to provide the developer a way to do that so he can optimize updates.
Yes, the internal map helps (to speed up lookups). You first create the map using the actual dom and then use that map to optimize the diff. Without the map you would need to have two loops for each parent node (old/new) to check for changes in position.

Not to every element, just to list collections.

Ah! So... if the developer is going to assign a unique ID, I think having them assign an actual id property instead of a custom one would be preferable, personally.

Ah! So... if the developer is going to assign a unique ID, I think having them assign an actual id property instead of a custom one would be preferable, personally.

Yes. I used the data-reef-id as example, but of course the id attribute could be also used as well. I generally use data-* attrs because it's easier to check if that's an ID my library uses, as opposed to a standard attr that can be there for whatever reason.
For this particular case, for example, I would expect that ALL nodes that are part of a list to have the data-reef-id attribute to be handled correctly (usually you render those lists using a loop, so that should not be a problem).

Alright, v8.1.0 should address this.
Under-the-hood, when comparing two DOM nodes, Reef now...

Checks to see if the items are the same node type, and if so, if they have the same ID (if they both do not have an ID, that would be considered "having the same ID").
If they "are the same node" per the conditions above, their content and attributes are diffed as normal (same way its always worked)
If they're a different node (different type or ID), Reef checks to see if the right node exists further in the DOM tree.
If it does, Reef moves it to the correct spot
If not, Reef injects a new DOM node

Question: The moved node, once moved, is then "diffed"?
I mean, since you are moving the node into a position that it is always after the current waked node, does that imply that the moved node will be also updated if I updated its position and something else inside it (like a title, for example)?.

After the node is moved, it's further diffed to check for any changes in attributes, content, sub-nodes and so on.


Padding

Hi! I'm not very proficient with js and didn't investigate this document too much, but it seems to me you have wrong usage of padding ('ece.js', line 145):
var padding = new Buffer(pad + 1);

here you initialize padding with some random bytes allocated for it, however in the specification it should be a length byte, then zero bytes.

A receiver MUST fail to decrypt if any padding octet other than the first is
non-zero, or a record has more padding than the record size can
accommodate.

So, it seems this implementation is contradicting with this document.
By the way, function encryptRecord is nowhere used with padding parameter, so it's always zero.
Sorry if I'm treating anything wrong.

Yes, that particular piece of code isn't hooked up, so I didn't discover that bug until I decided to use it.
I have a fix for this already.  Unfortunately, it's currently sitting on a computer that I won't have access to for a while.  I'll put a simple fix in for now.


Issue with Ag-Grid testing

Describe the bug
module resolution fails while testing.
The vite command works fine and the application transpiles properly
Reproduction
here is an example repo
https://github.com/spamshaker/vitest-ag-grid-issue
npm run dev - works
npm run test - fails before beginning with
SyntaxError: Cannot use import statement outside a module
❯ Object. node_modules/ag-grid-react/main.js:1:23
Module node_modules/ag-grid-react/lib/agGridReact.js:2 seems to be an ES Module but shipped in a CommonJS package. You might want to create an issue to the package "ag-grid-react" asking them to ship the file in .mjs extension or add "type": "module" in their package.json.

As a temporary workaround you can try to inline the package by updating your config:

// vitest.config.js
export default {
  test: {
    deps: {
      inline: [
        "ag-grid-react"
      ]
    }
  }
}

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

 Test Files  1 failed (1)
      Tests  no tests
   Start at  07:00:14
   Duration  232ms (transform 26ms, setup 0ms, collect 0ms, tests 0ms, environment 63ms, prepare 46ms)

System Info
System:
    OS: macOS 13.4.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 58.31 MB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.3.1 - ~/.nvm/versions/node/v20.3.1/bin/node
    Yarn: 1.22.19 - /opt/homebrew/bin/yarn
    npm: 9.7.2 - ~/.nvm/versions/node/v20.3.1/bin/npm
    pnpm: 8.6.5 - ~/Library/pnpm/pnpm
  Browsers:
    Chrome: 114.0.5735.133
    Edge: 114.0.1823.58
    Safari: 16.5.1
  npmPackages:
    @vitejs/plugin-react: ^4.0.0 => 4.0.1
    vite: ^4.3.9 => 4.3.9
    vitest: ^0.32.2 => 0.32.2

Used Package Manager
npm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

To add to this, also getting the same with<EMAIL_ADDRESS>

@smilelk4 the workaround seems to be working thank you

Resolved with ag-grid v30.0.3, thank you for quick help with workaround!


[CXF-7537] Use doPriv when calling methods needing Java 2 permissions

Hi Andy, looks fine to me. In general, please do not hesitate to merge directly when the changes are not sensitive (bug fixes, minor improvements, etc), thanks


Update docs to help disseminate info about genesis file update

Updated run-a-node.md with callout and pointer to a new page containing instructions about how to reinitialize and restart the node.

@jlwllmr -- I'm going to close this PR for now, without merging, as I believe we've disseminated this information elsewhere. I'm closing it, though, because I do want to keep the valuable instructions you created here around In Case We Need Them :tm:


action sheet: Add "edit message" option for admins to edit any topic.

Currently in zulip web any admin can edit any topic name
irrespective of whether admin has any sent messages in that
topic or not.
So, add that feature in zulip mobile.
Before:
https://user-images.githubusercontent.com/42652941/115412997-74587f80-a212-11eb-8329-1102af8cb235.mp4
After :
https://user-images.githubusercontent.com/42652941/115413194-9baf4c80-a212-11eb-9e87-e24519678949.mp4
Signed-off-by: rajprakash00<EMAIL_ADDRESS>

Thanks @rajprakash00 !
It would indeed be good to add the feature for people to edit the topic of other people's messages, where they have permission to do that.
But it looks like what this PR does is to offer the "edit message" UI in some cases for other people's messages. That is, the option is labeled "Edit message", and when you choose it the UI you get includes an input for the message content as well as the message's topic. We don't want to do that -- the server won't actually accept such an edit anyway, but we don't want to make it look like that's what the feature does.
The main work we'll need to do to add this feature is to add some UI for specifically editing just the topic. I'm closing this PR because I think that'll be pretty independent of what's in this PR.


Can Fuzi parse "only the plain text part" deep down somewhere

Description:
I come from Kanna and I am not sure if Fuzi can solve the following problem?
parsing-only-plain-text-from-html-using-kanna-swift
Is here the right place to ask those questions?
Thanks

Found the work around solution


As a CIF administrator, I want to see the "Payment Percentage of Performance Milestone Amount" between 0 and 100%, so that the number makes sense from a payment perspective

Description
On the Emissions Intensity Report page, the "Payment Percentage of Performance Milestone Amount" field is a percentage between 0% and 100%. The proponent gets the full/reduced payment or nothing.
If the calculated value goes over 100%, display 100%. If the calculated value goes negative, display 0%.
Acceptance Criteria
Given I am a CIF administrator
When I edit the TEIMP report
Then the "Payment Percentage of Performance Milestone Amount" field is clamped between 0 and 100%
Given I am a CIF administrator
When I edit the TEIMP report
Then the "Actual Performance Milestone Amount" field is calculated based on the clamped payment percentage value
Dev checklist

[ ] update postgraphile computed columns to reflect that logic
[ ] make sure tests cover negative and very large calculated payment percentage values

Additional context

Hey team! Please add your planning poker estimate with Zenhub @pbastia @dleard @tmastrom @Sepehr-Sobhani @BCerki

Please add your planning poker estimate with Zenhub @sam-warren

Thank you @sam-warren. Tested in Dev and it appears correct; however, this will be one we want the program area to review.


Update test data creator

Description

data_creator.py class updated with latest changes to be able to create and update test data in the delta lake.
References

Story

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


MWPW-160774 - [Table] top variant

Similar to the table (left) variant which sets horizontal alignment on the table body cells, this adds a table (top) variant for setting the vertical alignment on table body cells.
Resolves: MWPW-160774
Test URLs:

Before: https://main--milo--adobecom.hlx.page/drafts/sarchibeque/table-alignment?martech=off
After: https://sartxi-table-align--milo--adobecom.hlx.page/drafts/sarchibeque/table-alignment?martech=off

The table 1st col appears to have a weird rag now. (labels aren't left aligned) Is this intended?


mostly bux fixes

mostly bug fixes and small improvements

Thanks once again. I'm without access to a lab for a few weeks whilst I transition between jobs so things may go quiet for a while!


Error in virtual-machines/linux/quick-create-portal

[Enter feedback here]
Under Create virtual machine, point 4.
Correct "your your Region" to "your Region".
I've sent pull request #15432 to fix this.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 814b92ed-f712-53c8-644b-d3be513a4d06
Version Independent ID: 79d8998f-1b47-40ec-e975-8eeda383b0cf
Content: Quickstart - Create a Linux VM in the Azure portal
Content Source: articles/virtual-machines/linux/quick-create-portal.md
Service: virtual-machines-linux
GitHub Login: @cynthn
Microsoft Alias: cynthn

@cseas Thanks for correcting the error in your PR. Appreciate it. The changes will be visible on the doc as soon as the PR is merged.
We will now close this issue. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


Questions and notes installing on a DietPi

I use DietPi as my server and as expected some of the services are already duplicated from the things I install within DietPi.
I saw your warning about lighttpd possibly replacing existing HTTP server setups, but how does it apply for the other services? I'm almost afraid to try installing Samba because I already have it running the DietPi way, and I already configured CIFS share for the MiSTer, so those would seem to be redundant in my setup.
I would like however to take advantage of your concept about symbolic links and possibly supporting multiple kinds of structures for other retro systems, but for now I don't have a multiple-use case yet outside of MiSTer.
For now, I'm checking your Check Services menu to see if retronas detects the services that DietPi already configured. Seems to work out fine, except for Syncthing.
systemctl --no-pager --full status syncthing@dietpi
Unit<EMAIL_ADDRESS>could not be found.

Not sure why @ is needed in this case, but I'm able to ping my Syncthing using this, so please consider changing your script if possible.
systemctl --no-pager --full status syncthing

As a status report, I have the following existing services in DietPi that were correctly detected:
Samba
lighttpd
OpenSSH
And the following I installed from retronas without conflicts/issues under DietPi:
Cockpit
Webone
Sorry for a lot of ramble, but hope this is useful in any way.

I expect this would arguably be resolved once the project moves to targeting additional platforms other than Debian 11 see https://github.com/danmons/retronas/blob/main/IDEAS.md
The Debian package installs the syncthing unit as an instance signified by the @ symbol.
An instance is useful when you need data passed (in this case User is a var in the unit file) to a unit where you may be looking to run multiple instances of that unit. The instance inherits the name of the data passed to it to create a unique/dedicated name in this case syncthing@dietpi
From your post it appears dietpi runs from a static systemd service file i.e not an instance and as such when checking for the instance RetroNAS does not find the dietpi service since RetroNAS looks for the instance naming of<EMAIL_ADDRESS>https://github.com/danmons/retronas/blob/8553ee22aa2567553f49d472510c067f60b4dd79/dialog/services.sh#L130
There may be a better way to do service detection so it is platform agnostic, need to have a think about that.
Also looking at the implementation I don't think the current syncthing implementation is functional as an instance anyway due to the hardcoded gui port in ansible, From the syncthing doco an alt listen port will be chosen if the default 8384 is taken at time of installation so coming through after the fact and forcing the gui port should mean its effectiveness as an instance is negated
https://github.com/danmons/retronas/blob/9325f5654185f9935379e8999cd8caa989a8d271/ansible/install_syncthing.yml#L38
somethings well worth looking into.

This was related to a bug fixed in commit https://github.com/danmons/retronas/commit/fe8f0a271854bd88c447380a94dbe4c06382735a
I have an sdcard with dietpi on it now and syncthing now installs as is run as the correct user and can be queried with the status menu.
@bluemistP can you please update your installation and confirm.
systemctl --no-pager --full status syncthing@dietpi

●<EMAIL_ADDRESS>- Syncthing - Open Source Continuous File Synchronization for dietpi
     Loaded: loaded<EMAIL_ADDRESS>enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-02-08 07:44:16 GMT; 1min 29s ago
       Docs: man:syncthing(1)
   Main PID: 10183 (syncthing)
      Tasks: 18 (limit: 4532)
        CPU: 2.611s
     CGroup<EMAIL_ADDRESS>             ├─10183 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0 --gui-address=<IP_ADDRESS>:8384
             └─10192 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0 --gui-address=<IP_ADDRESS>:8384

Feb 08 07:44:17 DietPi syncthing[10183]: [S6SY2] INFO: Creating new HTTPS certificate
Feb 08 07:44:17 DietPi syncthing[10183]: [S6SY2] INFO: QUIC listener ([::]:22000) starting
Feb 08 07:44:17 DietPi syncthing[10183]: [S6SY2] INFO: Ready to synchronize "Default Folder" (default) (sendreceive)
Feb 08 07:44:17 DietPi syncthing[10183]: [S6SY2] INFO: Completed initial scan of sendreceive folder "Default Folder" (default)
Feb 08 07:44:17 DietPi syncthing[10183]: [S6SY2] INFO: GUI and API listening on [::]:8384
Feb 08 07:44:17 DietPi syncthing[10183]: [S6SY2] INFO: Access the GUI via the following URL: http://<IP_ADDRESS>:8384/
Feb 08 07:44:17 DietPi syncthing[10183]: [S6SY2] INFO: My name is "DietPi"
Feb 08 07:44:31 DietPi syncthing[10183]: [S6SY2] INFO: Detected 1 NAT service
Feb 08 07:44:37 DietPi syncthing[10183]: [S6SY2] INFO: quic://<IP_ADDRESS>:22000 detected NAT type: Port restricted NAT
Feb 08 07:44:37 DietPi syncthing[10183]: [S6SY2] INFO: quic://<IP_ADDRESS>:22000 resolved external address quic://<IP_ADDRESS>:1024 (via stun.syncthing.net:3478)

Press [Enter] to continue...

Sorry @sairuk but it still does not detect.
Side question: Is this command enough to update retronas by itself? It looks like it's updating. Currently there's no indication in the wiki on how to update retronas, whether the command below is enough or if I need to run install_retronas.sh again.
sudo /opt/retronas/retronas.sh
Let me clarify my setup: I installed syncthing using the dietpi-software features, my syncthing is not installed using retronas.
Might be something related to how dietpi installs software? I'm more on end-user side so I'm not familiar about these things. In any case, this is how it looks in my dietpi machine:
dietpi@DietPi:~$ systemctl --no-pager --full status syncthing ● syncthing.service - Syncthing (DietPi)      Loaded: loaded (/etc/systemd/system/syncthing.service; disabled; vendor preset: enabled)      Active: active (running) since Tue 2022-02-08 20:13:15 JST; 9min ago    Main PID: 190432 (syncthing)       Tasks: 20 (limit: 9269)         CPU: 32.147s      CGroup: /system.slice/syncthing.service              ├─190432 /opt/syncthing/syncthing -no-browser -no-restart -logflags=0 -home=/mnt/dietpi_userdata/syncthing              └─190447 /opt/syncthing/syncthing -no-browser -no-restart -logflags=0 -home=/mnt/dietpi_userdata/syncthing

I installed syncthing using the dietpi-software features

Ahh I see, yes OK that was clear the other day, sorry it must have slipped my mind, and yes our scripts will not currently handle that situation. I will leave this open and revisit the dietpi i deployed here to see if there is a simple solution.

dietpi doesn't install from repo for syncthing, it pulls a tar from git which also includes the instance unit.
The dietpi project then creates a standalone systemd unit as syncthing.service which is hardcoded to the user dietpi in the file /etc/systemd/system/syncthing.service
root@DietPi:/tmp# cat /etc/systemd/system/syncthing.service
[Unit]
Description=Syncthing (DietPi)
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=4

[Service]
User=dietpi
ExecStart=/opt/syncthing/syncthing -no-browser -no-restart -logflags=0 -home=/mnt/dietpi_userdata/syncthing
Restart=on-failure
RestartSec=1
SuccessExitStatus=3 4
RestartForceExitStatus=3 4

# Hardening
ProtectSystem=full
PrivateTmp=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

this is the wrong way to approach this, they should be using the packaged systemd instance as<EMAIL_ADDRESS>and using an override file (if applicable) to handle any additional changes they want to make
The defaults are still available for review in<EMAIL_ADDRESS>root@DietPi:/tmp# cat<EMAIL_ADDRESS>[Unit]
Description=Syncthing - Open Source Continuous File Synchronization for %I
Documentation=man:syncthing(1)
After=network.target
StartLimitIntervalSec=60
StartLimitBurst=4

[Service]
User=%i
ExecStart=/usr/bin/syncthing serve --no-browser --no-restart --logflags=0
Restart=on-failure
RestartSec=1
SuccessExitStatus=3 4
RestartForceExitStatus=3 4

# Hardening
ProtectSystem=full
PrivateTmp=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Please speak with the upstream project and have dietpi revisit their approach, so they standardise with the syncthing project.

A workaround has been added for this in the meantime


Consume exception - Value deserialization error

Description
I am currently trying to resolve an Value deserialization error when trying to consume a message from my Kafka.
Kafka: 2.12-2.5.0 (Tried on Confluent platform on 5.5.0 - still same)
Nuggets:
Confluent.Kafka 1.4.2
Confluent.SchemaRegistry 1.4.2
Confluent.SchemaRegistry.Serdes.Avro 1.4.2
I know that my message is not null because for testing purpouse its sent by my program.
How to reproduce
using (var consumer = new ConsumerBuilder<int, KafkaModels.DomesticPaymentRequest>(_consumerConfig)
                .SetValueDeserializer(new AvroDeserializer<KafkaModels.DomesticPaymentRequest>(_schemaRegistry).AsSyncOverAsync()).Build())
            {
                try
                {
                    while (!stoppingToken.IsCancellationRequested)
                    {
                        Confluent.Kafka.Message<int, KafkaModels.DomesticPaymentRequest> consumeResult = null;
                        consumer.Subscribe(_topic);
                        try
                        {
                            consumeResult = consumer.Consume().Message;
                        }
                        catch (System.Exception ex)
                        {
                            _logger.LogError(ex, "Error:");
                        }

                        if (consumeResult.Value != null)
                            _logger.LogDebug($"{DateTime.Now.ToString("hh.mm.ss.FFF")}\tConsumed message: {consumeResult.Value.Source} with key: {consumeResult.Key}");
                        else
                            _logger.LogDebug("Null value message");
                    }
                    consumer.Close();
                }
                catch (System.Exception ex)
                {
                    _logger.LogError(ex, "KafkaConsumer error:");
                }
            }
        }

Log output:
fail: RESTAdapter.Services.KafkaConsumerService[0]
      Error:
Confluent.Kafka.ConsumeException: Local: Value deserialization error
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Confluent.SchemaRegistry.Serdes.SpecificDeserializerImpl`1.Deserialize(String topic, Byte[] array)
   at Confluent.SchemaRegistry.Serdes.AvroDeserializer`1.DeserializeAsync(ReadOnlyMemory`1 data, Boolean isNull, SerializationContext context)
   at Confluent.Kafka.SyncOverAsync.SyncOverAsyncDeserializer`1.Deserialize(ReadOnlySpan`1 data, Boolean isNull, SerializationContext context)
   at Confluent.Kafka.Consumer`2.Consume(Int32 millisecondsTimeout)
   --- End of inner exception stack trace ---
   at Confluent.Kafka.Consumer`2.Consume(Int32 millisecondsTimeout)
   at Confluent.Kafka.Consumer`2.Consume(CancellationToken cancellationToken)
   at RESTAdapter.Services.KafkaConsumerService.StartConsumer(CancellationToken stoppingToken) in /Users/michalsvacina/Projects/Microservice/MicroservicesProject/src/RESTAdapter/Services/KafkaConsumerService.cs:line 49
fail: RESTAdapter.Services.KafkaConsumerService[0]
      KafkaConsumer error:
System.NullReferenceException: Object reference not set to an instance of an object.
   at RESTAdapter.Services.KafkaConsumerService.StartConsumer(CancellationToken stoppingToken) in /Users/michalsvacina/Projects/Microservice/MicroservicesProject/src/RESTAdapter/Services/KafkaConsumerService.cs:line 56

it's hard to say without line numbers. Suggest running in the debugger and setting a breakpoint in SpecificDeserializerImpl. To do this, you could fetch the confluent-kafka-dotnet project from github, edit the Consumer example to read your data type, then debug that.

Sorry the line 56 is the line
consumeResult = consumer.Consume().Message;

line numbers inside SpecificDeserializerImpl

Okay I just found out what the error was.

For some reason my Kafka model needed a parameterless constructor in my Api template (in console application this is not needed?)

Thanks for help anyways.

it would be better if we threw a better exception in this case, but minor enhancement.


Extract governor states into a better organizing object

Perhaps use an enum or something. The hard-coded, capitalized state names are all over LSDC, and it would be better to have them consolidated into a single place.
This will enable something like https://github.com/NSLS-II/lsdc/pull/369 for state checking ("state_SA", etc.) to be easier to calculate and be used in many places.

I've been using it in albula management code:
https://github.com/NSLS-II/lsdc/blob/969ef443f5617dd7bd7d3ce330e80c38e18e39a9/gui/albula/controller.py#L45
Perhaps its time to move it to gov_lib


Android?

As I am trying to access my API server from Android, I started to get errors:
[2016-08-24 09:44:06] lumen.DEBUG: CORS analysis for request started.
[2016-08-24 09:44:06] lumen.DEBUG: Request is not CORS (request origin is empty or equals to server one). Check config settings for Server Origin. {"request":null,"server":"[object] (Neomerx\\Cors\\Http\\ParsedUrl: http://<IP_ADDRESS>:9907)"}
[2016-08-24 09:44:06] lumen.DEBUG: CORS analysis for request completed.

Although I set this in my settings:
S::KEY_ALLOWED_ORIGINS => [S::VALUE_ALLOW_ORIGIN_ALL => true],

But I keep getting the same error, please advice.

I think server Origin is fine. You have a problem with Origin header in request (it absent).
According to the specification 6.2.1 if no Origin header is present in request it's not a CORS. And that's exactly what the lib does.

But, what would be the value for the Origin header in request?

Browser puts an origin of where the javascript was taken from. For example if executed script was received from 'www.yourdomain.com/js/code.js' it will be 'www.yourdomain.com'. So server can decide it it will handle requests from other sites. Or course with tools like curl it could be easily bypassed but with browsers provide this security feature.

In my Android app, I am not using a browser, its a native code that calls Rest APIs, is there any recommended Origin value in such case?

The same value you use in config for origin should be fine.


Where is documentation for old material components?

I can see new version of material has been released. But I can't find the web site which had
documentation for older version of material components?

You can access old documentation via: https://v0.material-ui.com/#/
And you can find all versions' links via : https://material-ui.com/versions/

thanks


Issue with GEOMFUM_DATA_DIR Environment Variable

Hey Luis,
I encountered an issue where the GEOMFUM_DATA_DIR environment variable was not properly set when I ran:
'os.environ["GEOMFUM_DATA_DIR"]'

I fixed it in my activate.bat file by adding:
set GEOMFUM_DATA_DIR=%SCRIPT_DIR%\geomfum\data

Is it expected?
Thanks,
Giorgio

Hi @GiLonga, I'm very glad to see your first issue here!
Yes, that's the expected behavior.
Since for now, I don't want to store data in the repo, I'm assuming users download the data (available e.g. in pyfm), store it wherever they want, and create an environment variable that indicates that folder (set in windows; export in linux).
This was just a quick dirty hack to have some notebooks running though.
I'm creating an action in our project to actually download/load the data if not yet available in the user machine, so this step becomes more clear.
Thank you for pointing it out!


Inefficient query generation when connecting or disconnecting fields

Bug description
Given a model with some relationships, and a query like this:
const res = await prismaDb.user.update({
  where: { id: '...' },
  data: {
    outfit: {
      update: {
        userId: '...',
        toConnect1: { connect: { id: '...' } },
        toConnect2: { connect: { id: '...' } },
        toDisconnect1: { disconnect: true },
        toDisconnect2: { disconnect: true },
        toDisconnect3: { disconnect: true },
      },
    },
  },
});

We get this generated SQL:
BEGIN []
SELECT [...everything...]
UPDATE "public"."UserOutfit" SET "toDisconnect1Id" = $1, "updatedAt" = $2 WHERE ("public"."UserOutfit"."userId" IN ($3) AND 1=1) [null,"2024-02-15 22:01:18.429 UTC","...userId..."]
UPDATE "public"."UserOutfit" SET "toDisconnect2Id" = $1, "updatedAt" = $2 WHERE ("public"."UserOutfit"."userId" IN ($3) AND 1=1) [null,"2024-02-15 22:01:18.429 UTC","...userId..."]
UPDATE "public"."UserOutfit" SET "toDisconnect3" = $1, "updatedAt" = $2 WHERE ("public"."UserOutfit"."userId" IN ($3) AND 1=1) [null,"2024-02-15 22:01:18.429 UTC","...userId..."]
SELECT [...everything...] FROM "public"."User" AS "t1" WHERE "t1"."id" = $1 LIMIT $2 ["...userId...",1]
COMMIT []

How to reproduce
See example above
Expected behavior
The last 3 UPDATE queries at the end could be combined into the first UPDATE query. It seems like it should be possible to achieve everything with a single UPDATE call.
It also seems like SELECT queries on the connect fields before the UPDATE occurs are unnecessary? (Or maybe even all the SELECTs?)
Prisma information
Roughly sketched out as:
model User {
  outfit UserOutfit
}

model UserOutfit {
  toConnect1Id String
  toConnect1 OutfitPart
  toConnect2Id String
  toConnect2 OutfitPart
  toDisconnect1Id String
  toDisconnect1 OutfitPart
  toDisconnect2Id String
  toDisconnect2 OutfitPart
  toDisconnect3Id String
  toDisconnect3 OutfitPart
}

model OutfitPart {
  // ...


Please clarify the following sentence - "Personal host pools only allow and support RemoteApp app groups."

Hi,
I don't understand the following sentence in our documentation for AVD environment:
"Personal host pools only allow and support RemoteApp app groups."
Please clarify the idea for this point.
Thank you very much!
Source:
https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/virtual-desktop/environment-setup.md

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: b7a8e997-f5c8-eaa8-b713-068650dd6b22
Version Independent ID: ef57850d-1fc4-bc6a-827c-2934c31f50c9
Content: Azure Virtual Desktop environment - Azure
Content Source: articles/virtual-desktop/environment-setup.md
Service: virtual-desktop
GitHub Login: @Heidilohr
Microsoft Alias: helohr

@dweppeler-msft
Thanks for your feedback! We will investigate and update as appropriate.

@dweppeler-msft
Thanks for your contribution.
Can you help me with PR number. It can help me to document further updates.

FYI
https://github.com/MicrosoftDocs/azure-docs-pr/pull/217099

@dweppeler-msft
Thanks for reporting this! We have created a PR for this issue and the changes should go live soon.
We are going to close this thread as resolved but if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.


Improve language mandating NS_ENUM usage

This change is a trivial English usage improvement and does not change any rules or recommendations.

:+1:


Updating version after Chef 12.9 release

\cc @danielsdeleo @randomcamel @chefsalim @mwrock

:+1: thought we wanted to keep everything, including ohai, on some kind of current or release version, but we don't really have a current for ohai. I'm happy either way, just wondering what the plan is.

👍

Once @jkeiser gets his one-gemfile-to-rule-them-all changes merged in, it will float on the latest released version.  But for now it is still set on master


Option to download chapters by all groups: -a/--all

The idea
Add a -a or --all option to download chapters by all groups that have released them
Why this feature should be added to the app ?
This will make it unnecessary to explicitly specify certain groups in a series of commands, allowing the user to sort through them himself later

I think that option is already exists, but i forgot to write documentation about it. You can use --group option with value all
# This will download all groups each chapter
mangadex-dl "insert MangaDex URL" --group all

Sorry about that, i will add this into documentation in next version release


Set movieGenreVM.SearchString

It'd be used in the View.

This and #17957 need to go in one PR

@mrmowji  you can still edit in the browser and combine the two file changes
After you make the first change in the browser, in the files menu select Edit, then navigate to the other file you want to edit


Extent for output file in af_imageWithAppliedCoreImageFilter:filterName:filterParameters:

The extent used to crop the output image was the original's image extent. This works in filters that don't change or change the extent minimally. For filters that change the extent significantly (for example "CIPerspectiveCorrection")  the bug is quite evident.
I also updated the image of the test cases.  In the original test image, the worn cropping caused the image to lave some sharp edges caused by a too small crop.

Thanks for putting this together @fsaint...much appreciated! This was definitely a bug that unfortunately we didn't notice. Looking at the image you modified makes it pretty obvious. Thank you for taking the time to put this together! 🍻

Just FYI...I also pushed b8608263 to fix up the test suite on various OS's. The blur filter produces quite different results on iOS 8.1 and 8.2 and iOS 8.3 and 8.4 and iOS 9.0+. Therefore, I had to generate different images on each OS and update the tests.

Could this be added as a setting to the method?
It can be very useful to have the original crop on a blurred picture.


Improve some widgets to add icon

Description
Possibility to add icons on some widgets:

Add BARS_LIST_ICONS to have a left icon
Add nbgl_layoutAddChoiceButtonsIcon allowing an icon on the top button
Add nbgl_pageDrawInfoIcon, calling nbgl_layoutAddChoiceButtonsIcon
Add nbgl_useCaseHomeExtIcon to have an icon on the actionButton

Changes include

[X] New feature (non-breaking change that adds functionality)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (158f76d) 60.42% compared to head (4f2ce89) 60.42%.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #530   +/-   ##
=======================================
  Coverage   60.42%   60.42%
=======================================
  Files          12       12
  Lines        1660     1660
=======================================
  Hits         1003     1003
  Misses        657      657

Flag
Coverage Δ

unittests
60.42% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[suggestion] Add a gitignore file to the blank project

When setting up the blank project via vercel I got a project without a .gitignore it would be cool to have one ready to go.
Happy to help add it if you're interested in it :)
I'm guessing that just copying the one from the root of this repo would work

ah good call! mind adding that?

fixed https://github.com/nandorojo/solito/commit/656e892992fe7a96882ef5fb654bb90d863d89f7

damn that was quick haha 🏃


Upgrade from 4.15.z to 4.17 d/s builds, rsyslog env has an issue, noobaa.log is not updated

Environment info

NooBaa Version: VERSION
Platform: Kubernetes 1.14.1 | minikube 1.1.1 | OpenShift 4.1 | other: specify

Have a CES S3 cluster with noobaa 4.15.6 version
Actual behavior

Upgraded on the nodes RH 8.x and 9.x as shown below
uname -a
Linux rk521-11 4.18.0-513.5.1.el8_9.x86_64 #1 SMP Fri Sep 29 05:21:10 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
Linux rk521-12 5.14.0-427.13.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 10 10:29:16 EDT 2024 x86_64 x86_64 x86_64 GNU/
rk521-11) rpm -qi noobaa-core-5.17.0-20240904.el8.x86_64 | grep Install
Install Date: Mon 09 Sep 2024 10:52:16 AM CEST
current time -->  date
Mon Sep  9 11:26:21 CEST 2024
system.json file  (snippet)
rk521-11":{"current_version":"5.17.0","upgrade_history":{"successful_upgrades":[{"timestamp":1725872074467,"completed_scripts":[],"from_version":"5.15.4","to_version":"5.17.0"}]}}
rk521-12":{"current_version":"5.17.0","upgrade_history":{"successful_upgrades":[{"timestamp":1725872767473,"completed_scripts":[],"from_version":"5.15.4","to_version":"5.17.0"}]}}}
In the noobaa file present in the logrotate.d directory (boobaa-logrotate) didn't have any further updates into noobaa.log file
[rk521-12 log]# rpm -qa |grep noobaa
noobaa-core-5.17.0-20240904.el9.x86_64
[rk521-12 log]# rpm -qi noobaa-core-5.17.0-20240904.el9.x86_64 | grep -i Install
Install Date: Mon 09 Sep 2024 11:05:13 AM CEST
rpm -qf /etc/logrotate.d/noobaa-logrotate
noobaa-core-5.17.0-20240904.el8.x86_64
/etc/logrotate.d/noobaa-logrotate  (in the 5.17.0-xxx) below snippet
/var/log/noobaa.log
{
daily
maxsize 100M
minsize 50M
start 1
missingok
rotate 100
compress
create 644 root root
sharedscripts
postrotate
killall -HUP rsyslogd || true
killall -HUP syslogd || true
systemctl reload syslog-ng > /dev/null 2>&1 || true
endscript
}
In the 5.15.4
snippet of logrotate_noobaa.conf
/var/log/noobaa.log
{
daily
maxsize 100M
minsize 50M
start 1
missingok
rotate 100
compress
create 644 root root
sharedscripts
postrotate
killall -HUP rsyslogd || true
killall -HUP syslogd || true
endscript
}
Expected behavior

It should update the noobaa.log files with the Upgrade / messages as well, however it only show them in the /var/log/messages
Sep  9 11:06:07 rk521-12 node[1647542]: 2024-09-09 11:06:07.356026 [PID-1647542/TID-1647542] FS::GPFS found GPFS lib file GPFS_DL_PATH=/usr/lpp/mmfs/lib/libgp
fs.so
Sep  9 11:06:07 rk521-12 node[1647542]: 2024-09-09 11:06:07.361138 [PID-1647542/TID-1647542] FS::set_debug_level 0
Sep  9 11:06:07 rk521-12 node[1647542]: 2024-09-09 11:06:07.361303 [PID-1647542/TID-1647542] FS::set_log_config: LOG_TO_STDERR_ENABLED=1 LOG_TO_SYSLOG_ENABLED
=0
Sep  9 11:06:07 rk521-12 node[1647542]: ESC[32mSep-9 11:06:07.445ESC[35m [Upgrade/1647542] ESC[36m   [L0]ESC[39m UPGRADE:: upgrade manager started..
Sep  9 11:06:07 rk521-12 node[1647542]: ESC[32mSep-9 11:06:07.447ESC[35m [Upgrade/1647542] ESC[36m  [LOG]ESC[39m CONSOLE:: read_rand_seed: reading 32 bytes fr
om /dev/urandom ...
Sep  9 11:06:07 rk521-12 node[1647542]: ESC[32mSep-9 11:06:07.463ESC[35m [Upgrade/1647542] ESC[36m  [LOG]ESC[39m CONSOLE:: read_rand_seed: got 32 bytes from /
dev/urandom, total 32 ...
Sep  9 11:06:07 rk521-12 node[1647542]: ESC[32mSep-9 11:06:07.463ESC[35m [Upgrade/1647542] ESC[36m  [LOG]ESC[39m CONSOLE:: read_rand_seed: closing fd ...
Sep  9 11:06:07 rk521-12 node[1647542]: ESC[32mSep-9 11:06:07.468ESC[35m [Upgrade/1647542] ESC[36m  [LOG]ESC[39m CONSOLE:: init_rand_seed: seeding with 32 byt
es
Sep  9 11:06:07 rk521-12 node[1647542]: ESC[32mSep-9 11:06:07.472ESC[35m [Upgrade/1647542] ESC[36m   [L0]ESC[39m UPGRADE:: container version is 5.17.0 and ser
ver version is 5.15.4. will upgrade
Sep  9 11:06:07 rk521-12 node[1647542]: ESC[32mSep-9 11:06:07.475ESC[35m [Upgrade/1647542] ESC[33m [WARN]ESC[39m UPGRADE:: upgrade scripts directory "/usr/loc
al/noobaa-core/src/upgrade/nsfs_upgrade_scripts" was not found. treating it as empty
Sep  9 11:06:07 rk521-12 node[1647542]: ESC[32mSep-9 11:06:07.476ESC[35m [Upgrade/1647542] ESC[36m   [L0]ESC[39m UPGRADE:: found the following versions with u
pgrade scripts which are newer than server version (5.15.4): []
Sep  9 11:06:07 rk521-12 node[1647542]: ESC[32mSep-9 11:06:07.603ESC[35m [Upgrade/1647542] ESC[36m   [L0]ESC[39m UPGRADE:: upgrade completed successfully!
Sep  9 11:06:07 rk521-12 systemd[1]: Started The NooBaa service..

For all noobaa related logs, it should be in the noobaa.log as well, Have their been changes in this area in 5.17 ?
Steps to reproduce

Install the noobaa rpm of 4.15.4 and Upgrade to noobaa rpm to 4.17
noobaa-core-5.15.4-20240704.el8.x86_64 --> noobaa-core-5.17.0-20240904.el8.x86_64)
More information - Screenshots / Logs / Other output

It has been discussed further over the slack channel, that this is implemented in this way, which means the 4.17 noobaa rpm onwards the /var/log/messages would get the noobaa messages logged instead of noobaa.log file.
There are couple of env variables which exists, that can be tried to change the logging behavior. However the default log into messages file of /var/log is a concern IMO, as all debug log levels also will be sent to a system wide log file which shouldn't be the case

As per Navin Paul and Romy, the final decision for logging in 5.17 is to get default logged into /var/log/messages. If we change the config parameters then it would get to both the files (i.e messages, noobaa.log) .
Hence closing this.
Ramya has opened a separate defect w/r/t the STDERR and SYSLOG enabled printing in the console.


Modules/manta/2.3

Pull Request Checklists
Important: When opening a pull request, keep only the applicable checklist and delete all other sections.
Checklist for New Module
Required

[ ] I used the cookiecutter template and updated the placeholder rules.

[ ] The snakemake rules follow the design guidelines.

[ ] All references to the rules object (e.g. for input files) are wrapped with str().

[ ] Every rule in the module is either listed under localrules or has the threads and resources directives.

[ ] Input and output files are being symlinked into the CFG["inputs"] and CFG["outputs"] subdirectories, respectively.

[ ] I updated the final target rule (*_all) to include every output rule.

[ ] I explained important module design decisions in CHANGELOG.md.

[ ] I tested the module on real data for all supported seq_type values.

[ ] I updated the default.yaml configuration file to provide default values for each rule in the module snakefile.

[ ] I ensured that all symbolic links are relative and self-contained (i.e. do not point outside of the repository).

[ ] I replaced every value that should (or might need to) be updated in the default configuration file with __UPDATE__.

[ ] I recursively searched for all comments containing TODO to ensure none were left. For example:
grep -r TODO modules/<module_name>/1.0

If applicable

[ ] I added more granular output subdirectories.

[ ] I added rules to the reference_files workflow to generate any new reference files.

[ ] I added subdirectories with large intermediate files to the list of scratch_subdirectories in the default.yaml configuration file.

[ ] I updated the list of available wildcards for the input files in the default.yaml configuration file.

Checklist for Updated Module
I added one line to Manta.smk to allow a users to limit the resources used by specifying "resources bam=N" where N is the maximum number of concurrent Manta jobs.

Is @whelena able to approve this please? It is a minor change that I need for GAMBL


Autoplace evaluation

Hello!
Thanks so much for providing the code for LCPR!
I was wondering about how is the Autoplace evaluation being performed. How are you getting 90% Recall@1 on the test set using a single scan and no RCS reranking, while in the original paper the result is 78.9% for 7 scans using RCS reranking.
Thanks in advance,
Best,
Dani

Sounds good! Thanks for the information 😄


flask db migrate sqlalchemy.exc.NoReferencedTableError exception

related to #18
happens with ca466fc279ee2dadf18d3619c0d027eee7e93fb3
(.venv) PS C:\Users\lking\Documents\Lou's Software\projects\webmodules.code-workspace\webmodules> docker exec webmodules-shell-1 flask db migrate -m "add users database, and users oriented tables"
INFO  [alembic.env] Migrating database <default>
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
  File "/usr/local/bin/flask", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/flask/cli.py", line 1047, in main
    cli.main()
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flask/cli.py", line 357, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flask_migrate/cli.py", line 105, in migrate
    _migrate(directory, message, sql, head, splice, branch_label, version_path,
  File "/usr/local/lib/python3.9/site-packages/flask_migrate/__init__.py", line 111, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flask_migrate/__init__.py", line 170, in migrate
    command.revision(config, message, autogenerate=True, sql=sql,
  File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 229, in revision
    script_directory.run_env()
  File "/usr/local/lib/python3.9/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.9/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "migrations/env.py", line 182, in <module>
    run_migrations_online()
  File "migrations/env.py", line 162, in run_migrations_online
    context.run_migrations(engine_name=name)
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.9/site-packages/alembic/runtime/environment.py", line 867, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.9/site-packages/alembic/runtime/migration.py", line 611, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/usr/local/lib/python3.9/site-packages/alembic/command.py", line 205, in retrieve_migrations
    revision_context.run_autogenerate(rev, context)
  File "/usr/local/lib/python3.9/site-packages/alembic/autogenerate/api.py", line 525, in run_autogenerate
    self._run_environment(rev, migration_context, True)
  File "/usr/local/lib/python3.9/site-packages/alembic/autogenerate/api.py", line 572, in _run_environment
    compare._populate_migration_script(
  File "/usr/local/lib/python3.9/site-packages/alembic/autogenerate/compare.py", line 55, in _populate_migration_script
    _produce_net_changes(autogen_context, upgrade_ops)
  File "/usr/local/lib/python3.9/site-packages/alembic/autogenerate/compare.py", line 89, in _produce_net_changes
    comparators.dispatch("schema", autogen_context.dialect.name)(
  File "/usr/local/lib/python3.9/site-packages/alembic/util/langhelpers.py", line 267, in go
    fn(*arg, **kw)
  File "/usr/local/lib/python3.9/site-packages/alembic/autogenerate/compare.py", line 125, in _autogen_for_tables
    [(table.schema, table.name) for table in autogen_context.sorted_tables]
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1138, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/usr/local/lib/python3.9/site-packages/alembic/autogenerate/api.py", line 442, in sorted_tables
    result.extend(m.sorted_tables)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5353, in sorted_tables
    return ddl.sort_tables(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 1264, in sort_tables
    for (t, fkcs) in sort_tables_and_constraints(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/ddl.py", line 1342, in sort_tables_and_constraints
    dependent_on = fkc.referred_table
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 4487, in referred_table
    return self.elements[0].column.table
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1138, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 2918, in column
    return self._resolve_column()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 2942, in _resolve_column
    raise exc.NoReferencedTableError(
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'apps_interests.application_id' could not find table 'application' with which to generate a foreign key to target column 'id'

confirmed this is not related to sqlalchemy 2.x

but see #28


feat(schematron): bump built-in SchXslt to v1.8

Replace the built-in SchXslt v1.7.4 with v1.8.
Disable schxslt.compile.metadata (schxslt/schxslt#206) so that we can simplify not only the test result report HTML but also the end-to-end test normalizers.

Note that the base branch of this pull request is not master but schxslt.

Although schxslt/schxslt@bba1f70e5f67e43cb4bf78111f5e59043b3b03ad is tagged as v1.8, I'm not sure if it's considered as an official release. That's why this pull request is still a draft.

v1.8 released officially: https://github.com/schxslt/schxslt/releases/tag/v1.8

v1.8.1 should be skipped. (schxslt/schxslt#212)


N and Y, not yes and no

Changes the permissible values for the YesNoEnum, which is only used by the dna_dnase and dnase_rna slots, which is only consumed by user facilities as far as I understand
I did spot checks like https://github.com/search?q=org%3Amicrobiomedata+dna_dnase&type=code&p=2
I updated the relevant values for the example data in this repo, but not in any related repos like

https://github.com/microbiomedata/submission-schema
https://github.com/microbiomedata/metadata-for-user-facility-template-transformations
https://github.com/microbiomedata/berkeley-schema-fy24

see

https://github.com/microbiomedata/metadata-for-user-facility-template-transformations/issues/6

Guidelines
Soft Schema Freeze
The nmdc-schema and berkeley-schema-fy24 schemas are under a soft freeze, which means changes should not be made that have any downstream implications. To ensure this, all PRs created creating during the freeze will be closely reviewed with every component of the NMDC system in mind.
Reviewers
To ensure no changes are made unexpectedly, PR creators will request reviews from all Berkeley Schema Roll Out task coordinators.
We expect task coordinators to review PRs and provide feedback/approval within 1 week of when they are identified as reviewers.
PRs will NOT be merged until all task coordinators (or one of their delegates) have approved.
Expedition, questions, and discussion can happen at any meeting.
Delays in review & merging should be addressed in meetings or with NMDC leadership.
PR Information
What type of PR is this? (check all applicable)

[ ] Refactor
[ ] Feature
[ ] Bug Fix
[ ] Optimization
[ ] Documentation
[x] Schema change: Structure and content

created, updated, or deleted a class, slot, or enum
changed whether a slot is multivalued
changed the way a slot is assigned to a class
changed the permissible_values of an enum
etc.

[ ] Schema change: Cleanup and preparation

updated the description of a class, slot, or enum
updated the mappings of a class, slot, or enum to an ontology
added an enum for future use (it is not in the range of any slot)
etc.

Description

PRs should be small and concise.
Aim to create small, focused pull requests that fulfill a single purpose. Smaller pull requests are easier and faster to review and merge, leave less room to introduce bugs, and provide a clearer history of changes.

Replace this text with a description of what this PR branch contains. Please keep in mind that all reviewers will be reading this description. Example: "In this branch, I..."

Related Issues

All PRs should relate to or fix an issue(s). Please identify the issue(s) below.

Related Issue(s): #
Fixes: #

Did you add/update any tests?

[x] Yes
[ ] No (Add a justification below)
[ ] I need help with writing tests

Could this schema change make it so any valid data becomes invalid?

This is a question about what the schema allows. It is not a question about what happens to exists in the NMDC database right now.
Example: If, in this PR branch, you renamed a slot from foo to foo_bar, the answer to this question would be "yes," even if nothing in the NMDC database currently uses the foo slot.
More examples: slot or class name changes, changes to a slot's multivalued state, changes to a slot's range (e.g. string to integer), changes to slot assignments to classes, changes to an enum's permissible_values

[x] Yes (A migrator is required)
[ ] No
[ ] I need help determining this

If you answered "Yes", does this PR branch include that migrator?

[ ] Yes
[x] No, this PR is incomplete and I need help writing the migrator

Does this PR have any downstream implications?

Examples: any change here that requires a change to workflows, workflow automation, the Mongo-to-Postgres ingest process, Jupyter notebooks, the Runtime, etc.

[ ] Yes (Explain below)
[ ] No

This is not critical. I suggest we pause on this and not merge in until berk is in prod

Did you change this because of how JGI requires it?
I don't think this is a slot NMDC needs to keep. Eventually we'll move this out of nmdc-schema and into submission-schema only. So I don't think it's really worth the change at all.
AND, I'd prefer the NMDC submission portal be clearer and more explicit with YES/NO and not Y/N. And transform the data for JGI in the MUTTs.
I don't think this was a priority and doesn't provide the solution I was expecting. Over all, I think we should not make this change & the changes should be handled elsewhere (submission-schema only slots when we're able to do that, and with MUTTS).

I'm confused. I thought this was a direct solution to your

https://github.com/microbiomedata/metadata-for-user-facility-template-transformations/issues/6#issuecomment-2258838435

I'm confused. I thought this was a direct solution to your

Update Was sample DNAse treated? column for JGI export metadata-for-user-facility-template-transformations#6 (comment)

Nope. I want the MUTTs script to make that change for the file I send to JGI.
Not in the submission portal directly.


Association using instance randomly failing

Description
When I create an association that uses the instance attribute, it sometimes raises ActiveRecord::NotNullViolation on the field where I use instance. It is random, but I could replicate it using create_list with a high number of instances.
Reproduction Steps

Define a factory that uses the instance attribute.

FactoryBot.define do
  factory :store do
    product { association :product, store: instance }
  end

  factory :product do
    store
  end
end

Create many instances using create_list.

create_list(:store, 100)

Expected behavior
Should not raise any error.
Actual behavior
It raises ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "store_id" violates not-null constraint.
System configuration
factory_bot version:  6.1.0
rails version:  5.2.4
ruby version: 2.7.1

It was another error, sorry. However, the error that was shown was not very explanatory. I'll open another issue.