[New] Extending support for County level COVID-19 Impact Assessment

Hi All, Please help me to utilize the CovsirPhy for US County Level COVID-19 Impact Assessment.
My aim is to index all 3142 US counties as per the impact of COVID-19.
I can request the local health department for clinical datasets. Please help me to understand the following points:

What are the minimum parameters required to perform the county-level COVID-19  impact assessment using CovsirPhy.
If I can collect county-level vaccination and variant details, is it possible to utilize them with the current model.
What is the best output parameter (like R0) we can use for comparative evaluation of COIVD-19 impact and indexing counties.

I am very much new to the domain of compartmental modeling so, please help me to understand if I am leaving some important details required to extend the model for the county level.
Thanks

Thank you for creating issues!

My aim is to index all 3142 US counties as per the impact of COVID-19. I can request the local health department for clinical datasets.

"COVID-19 Hub" (our main data source for JHUData etc.) provides state-level records for US. You can confirm this as follows.
import covsirphy as cs
loader = cs.DataLoader()
jhu_data = loader.jhu()
jhu_data.layer(country="US").Province.unique()

If you have the county-level dataset as a CSV file in your local environment, you can create JHUData instance via CountryData class. Please refer to Use a local CSV file which has the number of cases and regard "county" as "province" when you are using covsirphy library.

What are the minimum parameters required to perform the county-level COVID-19 impact assessment using CovsirPhy.

If I understand this question correctly, you are asking about the required columns of the county-level dataset. You need the following data.

observation dates
county names
the number of confirmed cases
the number of fatal cases
the number of recovered cases: if you do not have this data, please set 0s. Full complement will be done.
population values

If I can collect county-level vaccination and variant details, is it possible to utilize them with the current model.

At the latest version 2.21.0, it is not implemented. We need some updates of classes.

What is the best output parameter (like R0) we can use for comparative evaluation of COIVD-19 impact and indexing counties.

Yes, we can use R0 as the index, but R0 is calculated with parameter values of ODE models. Please refer to Usage: SIR-derived models and I recommend to use the parameter values directly as the index.

Thank you lisphilar for clearing this up for me. Regarding vaccination and variant details, whether you are working on the updates? If I can include at least the vaccination dataset that will be a great success for me.

Could you provide the details of the dataset you have? I'd like to write a new class as the handler, testing it with the dataset, if available.

column names
some rows
file extension (CSV?)
if opened, URL

I plan to create new class LocalDataLoader tentative name) to handle datasets saved in local environment. Usage may be similar to DataLoaderclass.

Note for updating codes:
When we create LocalDataLoader class, we should add date_format argument because dates are sometimes parsed incorrectly.

Could you provide the details of the dataset you have? I'd like to write a new class as the handler, testing it with the dataset, if available.

column names
some rows
file extension (CSV?)
if opened, URL

I plan to create new class LocalDataLoader tentative name) to handle datasets saved in local environment. Usage may be similar to DataLoaderclass.

Hi lisphilar I am working on collecting all the required datasets. Soon update them here for further processing.. Thanks.

Hi lisphilar, I am in the process of getting the dataset from local health departments. I apologize for the long delay. Meanwhile, I was trying to export snl.summary in form of CSV file. Can you please help me to understand how I can export snl.summary() in CSV or any other file format? Thanks

@AnujTiwari ,
Thank you for letting me know. With some pull requests linked to this issue, I'm preparing to add new features to DataLoader class (LocalDataLoader will not be created).

Meanwhile, please let me know how to export the intermediate maps (snl.trend().summary()) in high resolution .tif image format. Thank you

When we use CovsirPhy with Jupyter Notebook, you can export TIFF files as follows.
snl.interactive = False
snl.trend(filename="image_filename.tiff", dpi=300)

When we use it with Python scripts, snl.interactive = False is un-necessary.
snl.trend() (and all methods which create figures) calls matplotlib.pyplot.savefig() internally. Keyword arguments of  matplotlib.pyplot.savefig(), including dpi=300, can be used as the keyword arguments of snl.trend(). If not, that may be a bug.

Hi lisphilar,
As per your suggestions, I have implemented cook county datasets (confirmed, fatal, population and recovered=0).
Everything went smooth and I was capable of generating results for cook county with S-R trend analysis phases. But I have some doubts:

My dataset is starting from 24Jan2020 but my S-R trend first phases was starting from 15Mar2020?
When I am trying to replace S-R trend analysis phases with some new change points following instructions at usage:phases, I am getting an error: ValueError: @start must be the same as/over 15Mar2020, but 27Feb2020 was applied. Am I missing something here?

Again, thanks a lot for your help. I am still waiting for getting health data from local health departments. Keep you updated with the developments,
Dataset:
date | province | confirmed | fatal | recovered
1/24/2020 | Cook | 1 | 0 | 0
1/25/2020 | Cook | 1 | 0 | 0
.
.
2/18/2021 | Cook | 467986 | 9775 | 0
2/19/2021 | Cook | 468690 | 9790 | 0
2/20/2021 | Cook | 469379 | 9800 | 0
Code:
country_data = cs.CountryData("cook county.csv", country="Cook")
country_data.set_variables(
date="date", confirmed="confirmed", recovered="recovered", fatal="fatal")
(Optional) register total values of provinces as country level data
country_data.register_total()
country_data.cleaned()
jhu_data_cook = cs.JHUData.from_dataframe(country_data.cleaned())
population_data_cook = cs.PopulationData()
population_data_cook.update(5150233, country="Cook")
snl = cs.Scenario(country="Cook")
snl.register(jhu_data_cook, population_data_cook)
snl.records()
snl.trend(algo="Pelt-rbf").summary()
snl.estimate(cs.SIRF)
S-R Phases:
0th phase (15Mar2020 - 24Apr2020): finished  281 trials in 0 min  9 sec
1st phase (25Apr2020 - 18May2020): finished  320 trials in 0 min 10 sec
2nd phase (19May2020 - 08Jul2020): finished  385 trials in 0 min 13 sec
3rd phase (09Jul2020 - 17Aug2020): finished  437 trials in 0 min 15 sec
4th phase (18Aug2020 - 23Sep2020): finished  321 trials in 0 min 10 sec
5th phase (24Sep2020 - 20Oct2020): finished  180 trials in 0 min  5 sec
6th phase (21Oct2020 - 04Nov2020): finished  239 trials in 0 min  7 sec
7th phase (05Nov2020 - 14Nov2020): finished  270 trials in 0 min  8 sec
8th phase (15Nov2020 - 24Nov2020): finished  389 trials in 0 min 13 sec
9th phase (25Nov2020 - 05Dec2020): finished  322 trials in 0 min 10 sec
12th phase (05Jan2021 - 20Jan2021): finished  181 trials in 0 min  5 sec
10th phase (06Dec2020 - 17Dec2020): finished  564 trials in 0 min 22 sec
11th phase (18Dec2020 - 04Jan2021): finished  348 trials in 0 min 11 sec
13th phase (21Jan2021 - 20Feb2021): finished  831 trials in 0 min 28 sec
For New Change Points:
snl.clear(include_past=True).summary()
snl.add(start_date="28Apr2020")
Error:
ValueError: @start must be the same as/over 15Mar2020, but 27Feb2020 was applied.

My dataset is starting from 24Jan2020 but my S-R trend first phases was starting from 15Mar2020?

Yes.
We cannot apply S-R trend analysis to records with R = 0. This is because we cannot calculate parameter "a" of log(S(R)) = -a*R + logN when R is always 0 e.g. (R, S) = (0, 1000), (0, 999), (0, 998)...(0, 980).
So, all records with R = 0 should be ignored when S-R trend analysis. However, zeros in your dataset are not actual values. Because you did not have actual values of R, zeros were registered as I proposed.
When R values appears not to be actual values, covsirphy.Scenario complements R values via covsirphy.JHUDataComplementHandler class internally. We can check complemented R values with snl.records(variables="R"). This returns a dataframe with dates and complemented R values except for zeros. I guess, minimum date of the returned dataframe was 15Mar2020.

When I am trying to replace S-R trend analysis phases with some new change points following instructions at usage:phases, I am getting an error: ValueError: @start must be the same as/over 15Mar2020, but 27Feb2020 was applied. Am I missing something here?

Please replace snl.add(start_date="28Apr2020") with snl.add(end_date="28Apr2020") to specify the end date, not start date, of the new phase.

Yes - Fatal dataset is actually starting from 17th Mar 2020. Thanks for clearing out this to me.
I tried snl.add(end_date="28Apr2020") also but that also I am receiving the same error.

code:
country_data = cs.CountryData("cook county.csv", country="Cook")
country_data.set_variables(
date="date", confirmed="confirmed", recovered="recovered", fatal="fatal")
country_data.register_total()
country_data.cleaned()
jhu_data_cook = cs.JHUData.from_dataframe(country_data.cleaned())
population_data_cook = cs.PopulationData()
population_data_cook.update(5150233, country="Cook")
snl = cs.Scenario(country="Cook")
snl.register(jhu_data_cook, population_data_cook)
snl.records()
snl.trend(algo="Pelt-rbf").summary()
snl.estimate(cs.SIRF)
snl.clear(include_past=True).summary()
snl.add(end_date="28Apr2020")
ValueError: @start must be the same as/over 15Mar2020, but 27Feb2020 was applied.

Hmm...This seems an internal error. The first date might not be changed correctly when zeros were removed.

What is the output of snl.first_date?
Does snl.timepoint(first_date="15Mar2020"); snl.add(end_date="28Apr2020") solve the problem?

Output of snl.first_date - 27Feb2020
snl.timepoints(first_date="15Mar2020"); snl.add(end_date="28Apr2020") is showing the error:
ValueError: @start must be the same as/over 01Apr2020, but 15Mar2020 was applied.

Cook County Data.zip
Please find the data for your reference.

Thank you for providing the file and I will try to find the cause.

With pull request #881, I updated codes to fix the problem. Could you retry your script with the latest development version 2.21.0-kappa?
Please replace the current stable version with the development version in your environent or try development version in new environment.
pip install --upgrade "git+https://github.com/lisphilar/covid19-sir.git#egg=covsirphy"

At the latest development verion 2.21.0-kappa-fu2, we can use local datasets with DataLoader class as follows. "recovered" column will not be required in the CSV file if you do not know the actual number of recovered cases.
import covsirphy as cs
# Create DataLoader instance
# Set update_interval=None to skip downloading datasets from remote servers
loader = cs.DataLoader(update_interval=None)
# Read local CSV files: use pandas.read_csv() internally
loader.read_csv("input/cook county.csv", parse_dates=["date"], dayfirst=False)
# Check the dataframe
print(loader.local)
# Set necessary columns: country name and population
# with pandas.DataFrame.assign() intenally
loader.assign(country="US", population=5150233)
# Check the dataframe
print(loader.local)
# Lock data by specifing column names to use
loader.lock(date="date", country="country", province="province", confirmed="confirmed", fatal="fatal", population="population")
# Check locked data
print(loader.locked)
# Create JHUData instance
jhu_data = loader.jhu()
# Start scenario analysis
snl = cs.Scenario(country="US", province="Cook")
snl.register(jhu_data)
snl.records()

Kindly let me know if you need to read multiple CSV files and they have vaccination data.

Hi lisphilar,
I am still getting the same error when I am trying to add a new change point using snl.add(end_date="28Apr2020")

I also checked with the standard code and it is working fine (below) but with the extract CSV file (local dataset) it is showing the same error.

As far as I can understand it is the problem in the 0th phase. Please chk

I request you to look into this strange thing happening in the code:

Thank you for your notice and I also confirmed there is something error with first date setting after full complement of the number of recovered cases.
https://gist.github.com/lisphilar/1063a31147d50dff252e682673b1e600
I will investigate it and try to fix the problem.

By msitake, full-complement was done recurrently when recovered = 0 and timepoint settings were updated. With #884, I tried to fix this problem. The correct first date was 10Feb2020, not 15Mar2020.
https://gist.github.com/lisphilar/8f05eee0249571a5a27b2af64b5b2ce2
Please use the latest development version 2.21.0-kappa-fu3.

That's great - Thanks lisphilar for the immediate fix and all the hard work.

Hi lisphilar, Is it possible to predict COVID-19 prevalence (symptomatic patients + asymptomatic patients) using the current model?
Also, Today I computed R0 for Vietnam and when I compared my results (below) with the epiforecasts Reproduction number [https://epiforecasts.io/covid/posts/national/vietnam/] I found a significant difference between the outcomes of these models. May be you can better differentiate the modeling approaches here.

They will be discussed with #887 and #888.

Sure, thanks

At the latest development verion 2.21.0-kappa-fu2, we can use local datasets with DataLoader class as follows. "recovered" column will not be required in the CSV file if you do not know the actual number of recovered cases.
import covsirphy as cs
# Create DataLoader instance
# Set update_interval=None to skip downloading datasets from remote servers
loader = cs.DataLoader(update_interval=None)
# Read local CSV files: use pandas.read_csv() internally
loader.read_csv("input/cook county.csv", parse_dates=["date"], dayfirst=False)
# Check the dataframe
print(loader.local)
# Set necessary columns: country name and population
# with pandas.DataFrame.assign() intenally
loader.assign(country="US", population=5150233)
# Check the dataframe
print(loader.local)
# Lock data by specifing column names to use
loader.lock(date="date", country="country", province="province", confirmed="confirmed", fatal="fatal", population="population")
# Check locked data
print(loader.locked)
# Create JHUData instance
jhu_data = loader.jhu()
# Start scenario analysis
snl = cs.Scenario(country="US", province="Cook")
snl.register(jhu_data)
snl.records()

Kindly let me know if you need to read multiple CSV files and they have vaccination data.

Hi Lisphilar,
As we discussed earlier, I am trying to implement the model for multiple counties. When I am trying with the code I am getting the error: local database should be locked, but NOT locked
loader = cs.DataLoader(update_interval=None)
loader.read_csv("Autauga.csv", parse_dates=["date"], dayfirst=False)
print(loader.local)
Set necessary columns: country name and population
with pandas.DataFrame.assign() intenally
loader.assign(country="US", population=55869)
Check the dataframe
print(loader.local)
Lock data by specifing column names to use
loader.lock(date="date", country="country", province="province", confirmed="confirmed", fatal="fatal", population="population")
Check locked data
print(loader.locked)

Oh Sorry, by mistake I clicked on the closed issue.

Hi Lisphilar,
Currently, I am using loader.read_csv("Cook_County.csv", parse_dates=["date"], dayfirst=False) for reading the individual csv files. As, I am fetching cases, fatal, recovered, and population data records from a master CSV file that has the entry for hundreds of counties I would like to replace Pandas dataframe with the CSV file above.
My DataFrame is same as the earlier CSV files I discussed above (cook county.csv). Thanks

N worries, thank you for reopening this.

Could you try the followings and share the output of loader.local?
loader.read_csv("cook_county.csv")
loader.read_csv("master CSV filename", how_combine="merge", how="left", on="province")
print(loader.local)

This merges dataframes with pandas.merge()` internally.

Oops, I might misunderstood the situation. Please share column names of the master file. The master file has county/date/fatal/recovered/population for all counties and the indivisual files has country/date/confirmed?

No there is no difference between the master file and data frame. Data frame also has same attributes (county/date/confirmed/fatal/recovered/population).

If column name of population data is "population" and recovered data is "recovered", we can do as follows. We can skip loader.assign(population=5150233). Please replace "master file name" with the master filename,
import covsirphy as cs
loader = cs.DataLoader(update_interval=None)
loader.read_csv("master file name", parse_dates=["date"], dayfirst=False)
loader.assign(country="US")
loader.lock(date="date", country="country", province="province", confirmed="confirmed", fatal="fatal", recovered="recovered", population="population")

No, actually I am using multiple CSVs to fetch these parameters into the data frame. so I would like to just replace this read_CSV function with something like readDataFrame?

Thank you. OK, we have a large pandas.DatFrame which includes many counties' data.
We can call loader.read_csv(filename, how_combine="concat") multiple times to concat many CSV file data, but loader.read_dataframe() will be more useful as you metioned. Please wait for a while.

After merging #893, please update covsirphy to version 2.21.0-lambda-fu1 and please relace loader.read_csv("master file name", parse_dates=["date"], dayfirst=False) with loader.read_dataframe(dataframe, parse_dates=["date"], dayfirst=False) if the pandas.DataFrame is assigned to dataframe variable.

Great - I think it is one of the important updates that will be helpful for the generalisation of the library. Thanks, Lisphilar.. It is pleasure working with you ..

Hi Lisphilar,
Which trend analysis technique S-R trend is using? Is it possible to access the trend type, p-value, z-value, slope, and other parameters related to the trend results? Also, is it possible to collect the resultant parameters for homogeneity analysis?

This will be discussed in #894.

@AnujTiwari ,
Thank you for your continuous supports and discussion.
I'm writing documentation of these new features (DataLoader.read_csv(), DataLoader.read_dataframe() and so on) at docs/markdown/INSTALLATION.md of this repository. "Data loading (To Be Released)" chapter will be used. After deployment, this will be shown at  https://lisphilar.github.io/covid19-sir/markdown/INSTALLATION.html
Noe that "Dataset preparation" chapter will be integrated to the new chapter after the next stable release 2.22.0.
Please let me know (or create pull requests to edit it) if you have some advice.

thank you Lisphilar.. Sure I ll look into the Chapter and provide my inputs.

Hi Lisphilar,
Today I  was executing the model for Barbour County and I found a "-" value in the output R0.

Please find the Dataset:
Barbour.zip

while executing snl.estimate_accuracy("1st") I am receiving error message: Please execute ODEHandler.add() in advance.

Was it called after snl.add() and snl.estimate()?

YEs - It was called after snl.add()..

Could you share the all script via gist or GitHub repository?

sure I will share the script in some time.. In continuation to my discussion regarding Barbour county reproduction number: Today when I am also including 19th July Data for Barbour county my R0 is 13.59 (in my second phase 06th July to 19th July) and when I am using snl.trend() it is going 18.76 ... But if I remove19th July entry the R0 is 0.99 (in my second phase 05th July to 18th July) and in case of snl.trend() it is 1.16 .. Please chk this also in the script .. I am sharing with you

Please find the code here  -https://gist.github.com/AnujTiwari/eb43b4ed024edd3111d2762d58b9ef3b

You can find it here too - https://colab.research.google.com/gist/AnujTiwari/3f153a9475f382646c90ac364ca9c120/cotuny_r0.ipynb

I tried codes with records as-of 18Jul2021.
https://gist.github.com/lisphilar/a4396dc91a25af30ccda0155be2c0214
None value of Rt:
The records in the phases fit to SIR-F model in-sufficiently and kappa was 0 in the two phases and sigma was 0 in the 1st phase. Because Rt is equal to "rho * (1 - theta) / (kappa + sigma)", None was returned. None was converted to "-" with snl.summary() and snl.get().
Error with "snl.estimate_accuracy(): I also confirmed the error when we run it without snl.trend()`. When some parameter values are 0, this error may be raised. Could you create a new issue for this with "Request fixing a bug" template?

In case when the records in the phase fit to SIR-F model in-sufficient and R0 is None '-', what do you suggest for getting some significant R0 value for the county - whether I should increase the time period and chk for 20 days instead of 14 days or something else ?

Because the phase does not fit to the model, I recommend to shorten the phase e.g. 7 days.
However, actually I need the outputs of snl.estimate_accuracy() for confirmation and deciding the number of days. This method compares simulated number with estimated parameter values and actual number of cases.
Thank you for creating #895 and I will investigate it with high priority after work on 21Jul2021 JST.

thank you for the suggestion. Yes- it will be great if it is possible to compute 'minimum number of the days' phase for which records in the phase sufficiently fit to SIR-F model and provide a significant R0 value.. Thanks

I request you to please test the model with the Wilcox county dataset. I am not able to get the outcome for this county.
import pandas as pd
Create DataLoader instance
Set update_interval=None to skip downloading datasets from remote servers
loader = cs.DataLoader(update_interval=None)
Read local CSV files: use pandas.read_csv() internally
loader.read_csv("Wilcox1.csv", parse_dates=["date"], dayfirst=False)
Check the dataframe
print(loader.local)
Set necessary columns: country name and population
with pandas.DataFrame.assign() intenally
loader.assign(country="US", province="Wilcox", population=10373)
Check the dataframe
print(loader.local)
Lock data by specifing column names to use
loader.lock(date="date", country="country", province="province", confirmed="confirmed", fatal="fatal", population="population")
Check locked data
print(loader.locked)
Create JHUData instance
jhu_data = loader.jhu()
Start scenario analysis
snl = cs.Scenario(country="US", province="Wilcox")
snl.register(jhu_data)
snl.records()
snl.clear(include_past=True)
snl.add(end_date="12Jul2021")
snl.add(end_date=snl.today)
snl.summary()
snl.estimate(cs.SIRF)
#snl.clear(include_past=True)
snl.summary()
ValueError: When the targets have multiple columns, we cannot select RMSLE.
Wilcox1.zip

Thank you for the notice, but it is difficult to track all problems here. Could you create separate issues for Babour and Wilcox problems?
Here we will discuss adding new methods to DataLoader and documentation of them. I will release a new stable version after closing this issue.

Sure, I will create seperate issues for these problems. Thanks

Dear @AnujTiwari ,
Could you review the updated documentation of the new DataLoader with following links? Version 2.22.0 is planned in Jul2021.
https://lisphilar.github.io/covid19-sir/markdown/LOADING.html

Sure I am working on it... Thanks

Thank you.
FYI.
Some tables are broken there and they may be fixed with a new commit in about twenty minutes (time for deployment).
https://github.com/lisphilar/covid19-sir/commit/23b931778c6a210b20f117b84d38bd5780ecb062


feat(distro): pwnvim neovim config setup

Hey there, @Traap
I've added my config pwnvim. I hope you'll like it :)

@pwnwriter Thank you for your contribution.  I installed your configuration and used it.  I also looked over each source file.  Nicely done!


Contributing/Technical documentation

It would be awesome if you could add a CONTRIBUTING.md and/or some documentation on the more technical side. Doesn't need to be much since this is still early and sure to change, but some notes on where everything is/how the code is organized could make it easier for others to contribute :)

Thanks @tylerlaberge
A sketch of the concept is available here

I have added a HACKING.md file. It will have to do for now. :)


require & couch potato lazy loading not working

I have a project using require.js and couch potato to perform lazy loading for the modules. For some reason, when using the latest version of angular-leaflet with the simple-logging, the entire application will fail to load when I try to include the leaflet-directive in the controller for the map. Using a version prior to the simple-logging requirement works except I'm having the old resizing issue so I want to try the new version. Here are some code snippets...
This is setting up the module:
define(['angular',
'angular-couch-potato',
'angular-ui-router',
'ng-table',
'ngStorage',
'nemLogging',
'treeControl',
'ng-file-upload',
'leaflet',
'vincenty',
'leaflet-directive',
'd3',
'line-chart',
'angular-bootstrap',
'Bing'
], function (ng, couchPotato) {
"use strict";

var module = ng.module('app.network', ['ui.router', 'ngTable','ngStorage','treeControl','ngFileUpload','n3-charts.linechart','ui.bootstrap.modal','leaflet-directive']);

couchPotato.configureApp(module);

module.config(function ($stateProvider, $couchPotatoProvider) {....

If I remove 'leaflet-directive'  from the
var module = ng.module('app.network', ['ui.router', 'ngTable','ngStorage','treeControl','ngFileUpload','n3-charts.linechart','ui.bootstrap.modal','leaflet-directive']);
then the application loads, but of course mapping doesn't work. If I simply replace the new angular-leaflet.js with an older version of the file and include the 'leaflet-directive', the the mapping works, however I have resizing issues. Any ideas out there?

Try the latest angular-simple-logger as it is now wrapped in a function.

Actually the problem is probably related to this; https://github.com/nmccready/angular-simple-logger/issues/7 .

Please see the work arounds and I will try to get to fixing it better.

This is a valid dependency bug please see the issue stated above as it states the workarounds and the situation.

@tombatossals can we add a new label for dependency bug?

There are updates to this on angular-simple-logger using the PR'd branch as a test bed for you would help me out.

I loaded an older version of logger and it all works. I don't really need the logging anyway, so that works out for now. I'd love to try to help but have a really tight deadline for this. In a couple weeks I'll revisit and let you know.

This issue was moved to angular-ui/ui-leaflet#121

I'm going to rework&redesign angular-leaflet-directive to be compatible with Leaflet v1.0. It will mantain almost all its functionality, and will be compatible with the current features of the directive, but I must start from a fresh point, so I'm going to close this issue. If you think it must be worked with the new version, please reopen it.


with-global-stylesheet: corrected css pre-processing #readme link

Looking at the readme after my last PR got merged, I realized that I copy-pasted a wrong link from a different tab.  Not that the article I previously linked is "wrong", it just overemphasizes PostCSS.  Using Sass and PostCSS together is a best practice, as this example does.  Both of these tools are about "pre-process"-ing as the link suggests and the article concludes.

Thank you!


Merge get/list env command

Merges both get chaos-environment and list chaos-envrionments commands into a single command as get chaos-environments.

@SarthakJain26 its hard to just add a flag and show the whole details. Like not hard but unnecessary cause it will make the command slower if we query all the fields for list environments also. We should consider adding it to descibe or something. WDYT?

i havent touched other part of the code which can be optimised with this new get environment endpoint, we would do that in some other pr.  Not rebase in case, dont want to add extra commit


xi_sum initialization c

File : main_bwa_hmm.c
Line 115: float *xi_sum;
Line 537: xi_sum = malloc(sizeof(*xi_sum)*nstates*nstates);
// sizeof(*xi_sum) is same as sizeof(float)
//xi_sum is a 2D array which is stored as 1D array
Line 336: memset(xi_sum 0, sizeof(float *)*nstates);
// xi_sum is initialized to zero, but it doesn't seem correct because it only initializes some bytes ( nstates * size of pointer) of xi_sum array instead of whole array (nstates * nstates * sizeof(float) )

Same as #36.


MongoDB connections from injected pods keep growing and growing

Yesterday evening, I injected Conduit v0.3.0 into all my production services so I could observe the system behavior with real traffic (off-peak traffic, but still more than on my test cluster). Everything was running smoothly so I went to bed and woke up early in order to assess the situation before morning rush hour knocked at my services' door.
Conduit itself was still up and running, so were all my services. However, I noticed that my MongoDB database was saturated with connections:

This is very unusual, as the number of active connections to Mongo has always stayed within the 300..500 range. The rise of number of connections started pretty much exactly when I injected Conduit in my services.
In order to fix this, I tried (in that order):

"Uninject" Conduit from all the services, confirmed that they were no longer green in Conduit Dashboard, waited a few minutes and the number of Mongo connections did not drop.
Restarted all Conduit pods (kubectl delete po --all -n conduit), waited a few minutes and the number of Mongo connections did not drop.
Step down the primary Mongo node, so another node from the replica set would become primary, forcing all Mongo clients to reconnect. This is what worked.

When I saw that it didn't work (after waiting ~5 minutes, to make sure the connections reported by Mongo weren't in some "pending close" TCP state), I thought that it might mean that my understanding of the Conduit architecture was wrong, and the proxies actually lived in the conduit namespace pods. So that's why I restarted the Conduit pods in step 2.
But again, I was surprised and confused to observe that the connection count had not dropped after restarting the Conduit pods.
That's why I ended up using brute force and switching the replica set to another primary node.

When I said I uninjected Conduit from all services, it was a bit of a lie 😇.
I had left Conduit injected into some legacy, very-low-traffic services which I did not think might be involved in all those Mongo connections.
As you can see on the above connection chart, the number of connections to the new primary Mongo node had started climbing above its usual level, and it was still climbing after a while.
So I ended up uninjecting Conduit from all remaining services (really "all" this time), one by one, until there was only one left and the connection count was still climbing.
Shortly after uninjecting Conduit from that single remaining deployment, connection count dropped to its usual level and has been stable since then.
That particular deployment is our legacy API service, which is barely ever hit. I keep it live just so that our users who are on an older version of our client app can still access their data, but there are very few of these users.
It runs an "old" Docker image that hasn't been updated for about a year, so it uses "old" versions of the Mongo driver for NodeJS which may exhibit a connection behavior that is a bit different from latest versions, and that behavior combined to Conduit proxying might lead to the ever-rising connection count.
I'll reinject Conduit in some non-legacy services and observe the Mongo connection count over time, to confirm this hypothesis.

I have injected Conduit into 3 deployments which connect to Mongo, at 10 minute intervals:

gRPC service implemented in C# (dotnet core) and uses the MongoDB.Driver official Nuget package.
parse-server version 2.3.7 (NodeJS)
NodeJS service using the mongodb official npm package version 2.2.25

After injecting into 1 and 2, Mongo connection count was stable
After injecting into 3, Mongo connection count started to rise.
It might have something to do with how we initialize the Mongo client object in our NodeJS service, which is behaving properly without Conduit proxy, but causing new connections to build up when going through Conduit.
It is probably not related to the NodeJS Mongo driver, since service 2 is parse-server, which is a NodeJS app using the NodeJS driver, does not exhibit the connection count endless growth issue.
I'll add --skip-outbound-ports mongo_port when injecting Conduit into service 3 and observe what happens.

So far so good. I have reinjected Conduit into all my prod services with --skip-outbound-ports to skip the Mongo db port, and the number of connections to Mongo is stable. Everything else humming smoothly.
I won't have time in the short term to investigate why my NodeJS server keeps creating new Mongo connections when going through the Conduit proxy, but I suppose this is a bug you will want to look into and if it is, I'll be happy to reinject without skipping Mongo in order to analyse logs in more details to see why this may happening

Thank you for this very detailed description!

I believe I have found the reason behind this: our TCP proxy logic receives an incoming TCP stream, creates an outgoing one, and pipes the reads and writes to both of them. However, I think the current thing that does this, being a copy(in).join(copy(out), means that the out connection isn't dropped until both sides have shutdown. The inbound closes its connection, but the server never does, so we just sit there waiting for the second future to finish, which never does (or does a loooonngg time in the future).

I believe the fix here is probably to change our join to a select, meaning that if either side hangs up, it's time to shutdown the pipe.
Trying to figure out a way to test this is kind of tricky. It feels like it could be done in proxy/tests/transparency.rs, by adding a shared counter to our support tcp server to count current connections, but it is flaky in that it depends on timing for the server to notice the new connection, and the proxy to notice the client having closed...

@hawkw would you be interested in working on this? If so, I can mentor :)

I've been running 0.3.1 in prod for about an hour, without skipping outbound port 27017 (i.e. connecting to Mongo via Conduit proxy) and so far the number of Mongo connections has been steady. 👏


mdc-icon doesn't display when used as mdcListItemMeta in an mdc-list

Describe the bug
This is when using Font awesome.
If I don't decorate with mdcListItemMeta, the icon shows in the list, but it's of course not positioned correctly.
See the screenshot. First two lines uses this html, and the icon is displayed:
[mdc-icon fontSet="fa" fontIcon="fa-trash-alt"][/mdc-icon]
Last two lines uses this html, and the icon is not displayed:
[mdc-icon mdcListItemMeta fontSet="fa" fontIcon="fa-trash-alt"][/mdc-icon]
Screenshots

What Angular MDC version are you using?
2.0.1

Update: I was able to solve it using mdcIconButton as the mdcListItemMeta:
[button mdcIconButton mdcListItemMeta]
[mdc-icon fontSet="fa" fontIcon="fa-trash-alt"][/mdc-icon]
[/button]
Result:

Closing. Thanks for reporting back!


ConsumeScopedServiceHostedService Overrides StopAsync incorrectly

It seems like the example for ConsumeScopedServiceHostedService wont shutdown correctly as it overrides StopAsync without calling the base.
The QueuedHostedService seems to do this correctly
(btw, great examples, extremely helpful page)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c3460b0d-9c60-3a06-ac63-18cf990e53c3
Version Independent ID: b2dece1f-15b8-03af-6d52-7cc070432b4b
Content: Background tasks with hosted services in ASP.NET Core
Content Source: aspnetcore/fundamentals/host/hosted-services.md
Product: aspnet-core
Technology: aspnetcore-fundamentals
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

@huan086 please review

Most of the examples are missing the call to base.StopAsync() 🤦♀. Could you do a pull request to fix them?

I can pick this one up if needed. I think it's only ConsumeScopedServiceHostedService that needs the fix, because the others implement IHostedService directly.

@serpent5  thanks!


⚠ Samsung has degraded performance

In 8a556f4, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2968 ms

Resolved: Samsung performance has improved in 0790d95.


P3 preview

Connects to https://github.com/vaadin/components-team-tasks/issues/368

This change is

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.3 out of 4 committers have signed the CLA.:white_check_mark: manolo:white_check_mark: web-padawan:white_check_mark: YuriyVaadin:x: tomivirkkiYou have signed the CLA already but the status is still pending? Let us recheck it.

Reviewed 7 of 15 files at r1, 2 of 3 files at r2, 1 of 4 files at r5, 5 of 6 files at r6.
Review status: :shipit: complete! all files reviewed, all discussions resolved

Comments from Reviewable


Cannot run on Debian 12

App cannot be run on Debian 12 using current README instructions.

Clone repo
Install python3-pystray and python3-pyserial
Run main.py and see an error

$ python3 main.py
Traceback (most recent call last):
  File "/home/m2/App/FlooCast/main.py", line 17, in <module>
    from FlooDfuThread import FlooDfuThread
  File "/home/m2/App/FlooCast/FlooDfuThread.py", line 9, in <module>
    from HidDfuAPI import HidDfu
ModuleNotFoundError: No module named 'HidDfuAPI'

Search for "HidDfuAPI" over the Internet gave me zero reults, so I installed python3-hidapi, but that didn't help.
Kindly advice what's actually missing.

You can comment out this line since it references a library from Qualcomm used for upgrading the chip. Based on their license, it is not open source and can only be used on Windows. We will remove it in the next release.

Thanks, that did the trick :)


[feature request] Use message-bus instead of REST to communicate with EdgeX

What would you like to be added:
Since MessageBus is now the default inter-communication method for EdgeX, yurt-iot-dock should witch to MessageBus, and remove the actual value update logic for devices.
Why is this needed:
Improved device synchronization for yurt-iot-dock
Refer: https://github.com/openyurtio/yurt-device-controller/issues/60
others
/kind feature

@LavenderQAQ Do you have any progress about this issue?

@rambohe-ch It seems that @wangxye has submitted a version of the code in #1955, I will review it.


How can I add custom CSS?

We have problems with some of our URLs being too long (Primarily because of non-Latin characters in the paths), so we need to scroll left and right in the UI.
So I would like to add the following CSS
td {
    word-break: break-all;
}

I have tried looking into using module.config in the root of our solution like this: https://world.optimizely.com/blogs/scott-reed/dates/2023/3/restoring-help-text-for-display-name/
Is that possible?
What is the "name" of this module I can use for the clientresources? https://docs.developers.optimizely.com/content-management-system/docs/configuring-moduleconfig#clientresources

Ass no activity i created a PR #132

@jevgenijsp Thank you very much.
Is there a plan for when the next release will be available?

@JonasJes
Thank you for contributing. I believe we will see the release in the next few days.


AsyncAPI Applications Relations Finder

Reason/Context
Would be awesome to build a library that as the input gets an array of AsyncAPI documents and analyzes them to get information about Applications described by those files:

If they subscribe or publish to the same channels
If one subscribes to a channel that the other one publishes to
If there are apps that are subscribed to channels that no one publishes too

As an output, we should get a format that can be used to generate a diagram of relations between applications with additional metadata information about them (message, schemas).
Use case:
Visualization of the architecture and understanding what happens with the system if one of the puzzles is removed.
Later on, we could use this library in AsyncAPI CLI and also the AsyncAPI Studio.
Required knowledge:
There is no need to have any experience in event-driven architecture. You do not even need to know AsyncAPI spec too much. It is a pure coding task where you will need to find matching patterns between different documents and return this information. This should be an independent library that can be later reused in AsyncAPI CLI or AsyncAPI Studio, or by others in their tools.
Description

you do not have to implement any reading of AsyncAPI files from disk or URL or anything like this. It is a library and its API should accept an array of AsyncAPI documents and these can be already after being parsed by https://github.com/asyncapi/parser-js/
biggest unknow is the output, what would be the best format to choose here as an output. Should it follow syntax from mermaid or maybe we need something custom? I think it would be better to first try to "manually" try to get 2 asyncapi docs, identify relations between them and then model the output "manually" to get an idea how it should be structured, maybe we need pure uml output? 🤷🏼

@arjungarg07 repo ready and as you can see, the initial issue is also transformed

@derberg Thanks, will soon be opening issues for further discussions 😃

Some discussed points:

Would keep Map Object as a default output and would be configuring it to obtain output following mermaid syntax.
Providing documentation for the developers as a walkthrough to integrate more custom formats in this library.

it should show that App A has a link with App B through some message, but the relation goes through a given server

@derberg Since the Apps relation goes through a given server/broker, do we have to consider servers in production or development?

imho this would be the custom format considering the servers/brokers too.

@arjungarg07 Hello! At the moment we cannot define in spec which server expose some channels, but currently we have PR for that https://github.com/asyncapi/spec/pull/531 so for your question: you should consider all servers, which will be logical, it means: the application A can be hosted by 2 servers then you are considering 2 servers through which application B can communicate.

cool tool to consider https://reactflow.dev/
I admit I have not much knowledge about it, but assume that whatever is used to render flows, probably accepts some specific format, and this is the format that could be an output here

@magicmatatjahu Thanks for clearing it out.
So we would distinguish relations between files on 3 levels.
1st level: server URL
2nd level: server protocol
3rd level: channels to which services subscribed or publishing
Please correct me if i am wrong 🤔

cool tool to consider https://reactflow.dev/

@derberg yeah! i checked it out yesterday and it's super cool, would love to experiment with it but first focussing on MVP 😄

but first focussing on MVP

👏🏼

We could club 1st and 2nd levels too

same URL with different protocols would be super unusual so you can definitely assume this is always referring to the same server

@arjungarg07 Hello! At the moment we cannot define in spec which server exposes some channels, but currently we have PR for that asyncapi/spec#531 so for your question: you should consider all servers, which will be logical, it means: the application A can be hosted by 2 servers then you are considering 2 servers through which application B can communicate.

I'd like to throw into the ring that structurewise it doesn't really matter is a service is running in 1, 2 or 200 instances, it shouldn't appear with a cardinality of 200 in the flow. The main point of the message approach is that you don't care on the sender side how many workers/consumers there are. It may be an important fact for the consumer to be for example single instanced, or it is stateful, but that has IMHO nothing to do with the message flow documentation.

@pcornelissen What for your insights :) I almost agree with you, but there is some situation when pinned channel only for particular server can change the rest of flow. We can use the relationship (as a user/client) to check which flow will be "created" when for example I as a user/client will send message to create a new order in e-shop. Sending the order to this same channel, but in two different server can change flow. This situation is rare, but still possible. Of course, we should in first implementation go with simple solution which will cover almost cases, and then go with more complex and rare.
If we don't understand each other, it may be because I look at the current problem (app relationship) from the perspective of the user/dev who wants to send a message to a given channel and know what operations will be performed after sending the message/event, such as creating an order, adding loyalty points etc. on the example of an e-shop.

@arjungarg07 Please don't focus on first weeks on output shape (also using shape of ReactFlow isn't a good idea, we should be agnostic here). We can change it later, because currently most important part is making, as you wrote, some MVP :)

same URL with different protocols would be super unusual so you can definitely assume this is always referring to the same server

Yeah! but I would like to play it safe and not make any assumptions😅

@magicmatatjahu I ran the script for on Alvaro's flight service with a slight change of adding one more server in one of the spec.
Generating default output as:
Map(2) {
  'mqtt://localhost:1883,mqtt' => Map(2) {
    'flight/update' => {
      sub: [ 'Flight Monitor Service' ],
      pub: [ 'Flight Notifier Service' ]
    },
    'flight/queue' => {
      sub: [ 'Flight Subscriber Service' ],
      pub: [ 'Flight Notifier Service', 'Flight Monitor Service' ]
    }
  },
  'mqtt://163.124.11,mqtt' => Map(2) {
    'flight/update' => { sub: [ 'Flight Monitor Service' ], pub: [] },
    'flight/queue' => { sub: [], pub: [ 'Flight Monitor Service' ] }
  }
}

So the mermaid diagram of this output should be like this:

Any feedbacks are much appreciated.

@arjungarg07 I love it! Really :) As I see everything is good, 'publish' operation go to channel, from channel you link to service/app's 'subscribe' operation. At the moment is good, but honestly it's a hard to understand in first look what is connecting to what, I mean, you should look on diagram and see that you can publish something to (eg) FlightNotifierService when you will publish to flight/update channel and etc. It's not a problem of your code, but probably with our "description" of publish/subscribe in spec, but overall, great job! :) What do you want to do next?

but honestly it's a hard to understand in first look what is connecting to what

Yup! I mean the whole thing is to visualize the relations and the first look at this image feels so complex.

What do you want to do next?
will have the script ready for generating mermaid diagrams syntax asap 🚀

Do we have to add metadata information in the default syntax itself or have a separate API for it?

@arjungarg07 Do you mean metadata like payload's schema of message etc? You operate on parsed spec, currently you save sub/pub only by name to array for appropriate channel name, to save metadata you can change the array to Map, so you have:
    'flight/update' => {
      sub: [ 'Flight Monitor Service' ],
      pub: [ 'Flight Notifier Service' ]
    },

but you can change it to Map<string, ChannelMetadata>
 'flight/update' => {
   sub => {
     'Flight Monitor Service' => Channel metadata,
     'Flight Notifier Service' => Channel metadata,
   }
 }

Of course you can save as Channel metadata whole object of appropriate channel of given app as reference :)

Do you mean metadata like payload's schema of message etc?

Yeah, and message and schemas too.

About separate API we can think in the next weeks, at the moment we can add metadata by default to default output/syntax or add option to enable adding metadata, which will be by default set to true.

Sure. I will then update the metrics 👍🏽

@derberg @jonaslagoni @magicmatatjahu Consider we need to get the metadata information like message and schema for each channel on this spec.
channels:
  flight/update:
    description: |
      Provides updates from a subscribed flight
    subscribe:
      summary: Inform about the status of a subscribed flight
      message:
        $ref: '#/components/messages/flightStatus'
 flight/queue:
    description: |
      Queues a flight in order to retrieve status
    publish:
      summary: Subscribe about the status of a given flight
      message:
        $ref: '#/components/messages/flightQueue'

I can't find any API available in parser-js through which I can get the served message information on a particular channel, like
const messageInfo =doc.channel('flight/update').message();

Same for schemas. Although we do have APIs doc.allMessages() and doc.allMessages() but it doesn't give info about on which channel it is being served.
Please correct me if I am missing something.

@arjungarg07 Because message is in the operation item of channel - publish or subscribe. You must use in your case doc.channel('flight/update').subscribe().message();
What about saving all channel's metadata (with description etc) as metadata - not only message - in relation?

Because message is in the operation item of channel

Ok, this I missed :) Thanks!

What about saving all channel's metadata (with description etc) as metadata - not only message - in relation?

Actually, we didn't discuss Luksaz about what all to save in metadata coz i thought of asking it along the development phase. I would suggest to consider all channel's metadata as you said.

EDIT: We have also corresponding hasSubscribe() and hasPublish() functions to check if given operation is available in channel. There is a full API of parser-js https://github.com/asyncapi/parser-js/blob/master/API.md

Yeah, I have been checking from here itself :)

EDIT: Also, what do we need to have payload as it is for metadata or just some specific parts of it?

Payload is a shape of (usually) json which should be sent to channel or will be subscribe from channel, so payload is a
complete whole that cannot be divided into smaller parts :) If you think about different problem, please describe it in different way, because probably we don't understand each other :)

Yeah, what I mean to say is that since we have great support for traversing schemas with support for circular references, when we will extract the metadata information from let's say message payload do we have to consider some specific fields from it or just include the whole payload itself like this:
message: {
      payload: {
        type: 'object',
        '$id': 'PlayerItemPickupPayload',
        additionalProperties: false,
        properties: {
          pickupTimestamp: {
            type: 'string',
            format: 'date-time',
            description: 'The timestamp the item was picked up',
            'x-parser-schema-id': '<anonymous-schema-4>'
          }
        }
      },
      'x-parser-original-schema-format': 'application/vnd.aai.asyncapi;version=2.0.0',
      'x-parser-original-payload': {
        type: 'object',
        '$id': 'PlayerItemPickupPayload',
        additionalProperties: false,
        properties: {
          pickupTimestamp: {
            type: 'string',
            format: 'date-time',
            description: 'The timestamp the item was picked up'
          }
        }
      },
      schemaFormat: 'application/vnd.aai.asyncapi;version=2.0.0',
      'x-parser-message-parsed': true,
      'x-parser-message-name': '<anonymous-message-1>'
    }
  }

@arjungarg07 I see your problem but you also put example on which I can describe my thinking :)
You can treat only payload as metadata, but we must also know that dev can put inside message object specific information for given protocol, this same as in given server, so for me this is also very important metadata which we should show to user. Also channel can have specific protocol information in bindings object https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md#channelBindingsObject You can have protocol info in message, operation (publish/subscribe), channel and server object. If we only consider message and it's value, someone can say that headers are also very important as metadata. I would like to say that whole data in a channel is very important metadata, so I think that treating whole channel as metadata would be the best option for our case, so I will remind you in which way I would like to do that:
 'flight/update' => {
   sub => {
     'Flight Monitor Service' => Channel metadata,
     'Flight Notifier Service' => Channel metadata,
   }
 }

and then you don't focus which part of message is important, because you treat whole channel as metadata.
Also:

Yeah, what I mean to say is that since we have great support for traversing schemas with support for circular references

No, we don't, we only handle simple circular refs :) I created issue for that https://github.com/asyncapi/parser-js/issues/293 so please have it in mind when you will use traverse function :)
What do you think about treating channel as metadata. It's easiest solution for our problem and will give us as many possibilities for the future.

No, we don't, we only handle simple circular refs :) I created issue for that asyncapi/parser-js#293 so please have it in mind when you will use traverse function :)

Yeah, I just tried to iterate over it and it caused recursive stackoverflow😅

What do you think about treating channel as metadata. It's easiest solution for our problem and will give us as many possibilities for the future

I agree with the explanation that the metadata could be important for one dev or the other and treating the channel as metadata itself would be the best possible solution 👍🏽
 'flight/update' => {
   sub => {
     'Flight Monitor Service' => Channel metadata,
     'Flight Notifier Service' => Channel metadata,
   }
 }

Here flight/update is the channel and since channel's metadata would be same for every service(Flight Monitor Service,Flight Notifier Service), won't be mapping the same metadata with every service, instead we should have it like this:
 <channelName> => {
    sub: [ ],
    pub: [ ],
    metadata: {}
  }

Also, I think treating channel as metadata would have a huge ton of info to put in mermaid class diagram. How we would deal with that 🤔

Yes, you're right about that, but partially :) For mermaid you can only use payload, headers and bindings metadata from channel, for another syntax you can only use payload etc. Of course, It will be "huge" data for default syntax, but we can read/reuse/change it for another syntaxes. You should think about it as a bridge to another syntax.

@derberg @magicmatatjahu Check out these tools: kroki, chartmage, UMLGraph, yuml

@arjungarg07 they all feel kinda the same tbh, I mean the same principle behind as in case or mermaid and plantUML. As you wrote, I think more important are tests now, and a way to test all those outputs not only from unit tests point of view but also integration tests.
And then something more beautiful........ React Flow ❤

I was playing a bit with React Flow 😅 and it turns out that they don't have any playground online where we can just paste the syntax and render the diagram. Then I thought of generating an HTML page but for that, I am not able to find and cdn/browser package for react-flow-renderer so that I can include it in that html file and code it further. I think if I could get this cdn package which I could include in HTML code, we could use puppeteer to automate opening the HTML page and taking a screenshot of the page, and save it in some file.

function for react flow should just generate the relations, the code responsible to show them, not entire website. This is a library, I integrate it however I want in my system, generate relations and enable in my HTML. So basically the goal is to focus on react flow code that shows relations between nodes, the HTML to present it can just be an example, part of test or a demo. Does it make sense?

Yeah, so this is the simplest flow for ReactFlow.
import React from 'react';
import ReactFlow from 'react-flow-renderer';

const elements = [
  { id: '1', type: 'input', data: { label: 'Node 1' }, position: { x: 250, y: 5 } },
  // you can also pass a React Node as a label
  { id: '2', data: { label:
Node 2
 }, position: { x: 100, y: 100 } },
  { id: 'e1-2', source: '1', target: '2', animated: true },
];

export default () => <ReactFlow elements={elements} />;

So our library function for react flow should just generate the elements array which could be directly passed by the user, right?

@arjungarg07 Yes, elements should be the exact output from function.

@arjungarg07 Important thing: If you will use inside output the VDOM nodes, like in your example
Node 2
 you should write them with React.createElement() function, not as JSX, because if you use the JSX, you need then transpile your package by babel.

@arjungarg07 Can we close the issue?

we can definitely close it, purpose of this issue was to discuss initial idea and the scope.


jetbrains: add libnotify to wrapper to enable notifications

Motivation for this change
System notifications weren't available in the corresponding menu. Also, the log showed:
ntellij.ui.SystemNotifications - Unable to load library 'libnotify.so.4': Native library (linux-x86-64/libnotify.so.4) not found in resource path (/nix/store/qqsvlbrn6k6drb9bhd85zx1jv2fgp147-idea-community-2018.2.2/idea-community-2018.2.2/lib/bootstrap.jar:/nix/store/qqsvlbrn6k6drb9bhd85zx1jv2fgp147-idea-community-2018.2.2/idea-community-2018.2.2/lib/extensions.jar:/nix/store/qqsvlbrn6k6drb9bhd85zx1jv2fgp147-idea-community-2018.2.2/idea-community-2018.2.2/lib/util.jar:/nix/store/qqsvlbrn6k6drb9bhd85zx1jv2fgp147-idea-community-2018.2.2/idea-community-2018.2.2/lib/jdom.jar:/nix/store/qqsvlbrn6k6drb9bhd85zx1jv2fgp147-idea-community-2018.2.2/idea-community-2018.2.2/lib/log4j.jar:/nix/store/qqsvlbrn6k6drb9bhd85zx1jv2fgp147-idea-community-2018.2.2/idea-community-2018.2.2/lib/trove4j.jar:/nix/store/qqsvlbrn6k6drb9bhd85zx1jv2fgp147-idea-community-2018.2.2/idea-community-2018.2.2/lib/jna.jar:/nix/store/llz3210kc2fmidfizx68k0hl8qzb5nzp-jetbrainsjdk-152b1248.6/lib/tools.jar)

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

backport 9934f413da1


希望能支持自定义主机名设置

之前的版本需要自己手动设置主机名，希望sealer能支持配置文件中自定义主机名设置

Do you have some suggestion about the user interface of this feature. Set the hostname in Clusterfile? Or add an exec command to let you set hostname.  sealer exec "sethostname xxx" -f Clusterfile

在Clusterfile  中 ipList 加个字段主机名应该比较合理

example：
masters:
ipList:
- <IP_ADDRESS> master-1
nodes:
ipList:
- <IP_ADDRESS> node-1

现在ipList 是否支持范围连号？

现在ipList 是否支持范围连号？

Not support now, you can provide this feature.

example：
masters:
ipList:

<IP_ADDRESS> master-1
nodes:
ipList:
<IP_ADDRESS> node-1

Maybe we need upgrade the Clusterfile definition.

Maybe we can provide a plugin to do this.

define hostname plugin config

apiVersion: sealer.aliyun.com/v1alpha1
kind: PluginConfig
metadata:
  name: hostname-config
spec:
  plugin: hostname
  data: |
      - name: master-0
         ip: <IP_ADDRESS>

develop a plugin processor

type Plugin interface {
   Process(*v1.Cluster, *v1.PluginConfig) error
}

/invite

@fanux
I can try it !

@jinnzy https://github.com/alibaba/sealer/pull/359/files check this out, the plugin design

https://github.com/alibaba/sealer/blob/main/docs/design/plugin.md


ux-grid设置max-height后，表格内层滚动div的高度计算有问题

ux-grid设置了合计行，并且设置max-height后，表格内层滚动div的高度计算有问题，目测是内层高度没有减掉合计行的高度，导致内层div的高度会大于max-height。如果ux-grid是在el-tabs中的，因为el-tab是默认overflow: hidden；表格内层超出max-height的部分会被遮挡住。

Hello, we tried to solve the issue.
This is what we did:
Modified the UxGrid component to properly calculate the inner scroll div height when a summary row is present and max-height is set. This fix ensures that the inner div height does not exceed the max-height by subtracting the summary row height from the calculation.
You can review changes in this commit: https://github.com/jacks-sam1010/popo-fishes-umy-ui-268/commit/b09230f4a77ef61859e8347b70dd87e43420efd6.

[!CAUTION]
Disclaimer: The concept of solution was created by AI and you should never copy paste this code before you check the correctness of generated code. Solution might not be complete, you should use this code as an inspiration only.

Latta AI seeks to solve problems in open source projects as part of its mission to support developers around the world. Learn more about our mission at https://latta.ai/ourmission . If you no longer want Latta AI to attempt solving issues on your repository, you can block this account.

谢谢您的回复，大约什么时间可以更新？？


A few new check implementations

DSIG is an unwanted table, and we have a check for unwanted tables; it doesn't need its own check.

Ah! That's right! I had not seen that you know merged the dsig check into the unwanted_tables check. I think I'll do the same in FontBakery then.


Health Connect revoke function does not work in Android 14

I am using the revokeAllPermissions feature on phones with Android 14 and above, but I can still retrieve the previously granted permissions using permissionController.getGrantedPermissions.
However, on phones with Android 13 and below, using the revokeAllPermissions function does not have this issue.

Hi,
Is this an issue with the Health Connect Sample, or with the API? If the latter, do you mind opening an issue in the Health Connect issue tracker? This will help our team better organize things.

@breanatate  Thank you, I found that the issue has already been raised in the issue tracker.
https://issuetracker.google.com/issues/307685584


Improve Python 3 compatibility

The proposed one-liner to install VLC Radio:
curl https://get.pimoroni.com/vlcradio | bash

seems to install Python 3 packages. However, the phatbeatd is still in Python 2, with which the daemon does not work on boot or manual excecution.
I believe this could be the cause of some problems posted in various platforms, which state the buttons are not working. After applying this fix, the buttons are working fine. I have confirmed on both Stretch and Buster btw :)

Thank you- I suspect we might have to pull in Python3 as a dependency somewhere, too, since I don't think it's installed by default on all platforms.

Thanks for the fast confirmation!

Thanks for the PR- a few of our older product libraries have some bitrot and it's hard to find the time (and head space!) to get around to updating them. Much appreciated!


kogito bundle will create "" in the first line of the files which causing parsing error

xml version="1.0" line will cause the application fail to parse file. I create a test scenario and the first line causing parse error and if I delete it the program will run the test.
kogito bundle version is v0.17.0

It seems a duplicate of https://github.com/kiegroup/kie-tools/issues/884, closing.


[$500] Web - Chat - When sign in to Expensify, lost a conversation with a new message on LHN

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Open 2 accounts on 2 browser
User A sign out
User B send a message to User A
User A login to Expensify again

Expected Result:
Received a message from user B on the top of LHN
Actual Result:
User A does not receive messages from user B and does not see the conversation on LHN
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: v1.3.64-2
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/115492554/dd733ee4-147e-4323-b247-65b8518e7707
https://github.com/Expensify/App/assets/115492554/770ea044-642d-445d-ae10-49dd73a1b09c
Expensify/Expensify Issue URL:
Issue reported by: @Le Thi Thu Thuy
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1692954768122869
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01408683b571052093
Upwork Job ID:<PHONE_NUMBER>400333824
Last Price Increase: 2023-10-16

Isn't this a dupe of the linked issue?

Indeed that looks to be the case. Minimally we should hold on that issue to confirm.

The linked issue is now closed, though I'll have to come back to reproducing this one.

Going to open this up to the community in case anyone else can reproduce and would like to work on this one.

@izarutskaya
Hey, I couldn't reproduce the issue (see the video). Can you confirm that it still happens?
https://github.com/Expensify/App/assets/23487612/9ab06b97-5b5a-43c6-9456-c6a33a9cedcc

I wasn't able to reproduce this one either, so closing.


How to add a custom service/component?

Like PR #1942, it requires lots of codes to add an internal service inside the substrate node.

What my questions are:

Are there some ways to customize the substrate node and add a service here without too many code changes?

pub trait ServiceTrait<C: Components>:
	Deref<Target = Service<C>>
	+ Send
	+ Sync
	+ 'static
	+ StartRPC<C>
	+ MaintainTransactionPool<C>
	+ OffchainWorker<C>
        + CustomService<C>
{}
impl<C: Components, T> ServiceTrait<C> for T where
	T: Deref<Target = Service<C>>
	+ Send
	+ Sync
	+ 'static
	+ StartRPC<C>
	+ MaintainTransactionPool<C>
	+ OffchainWorker<C>
        + CustomService<C> // <- add my service, and do other stuff somewhere else
{}

If there is no way to do so, is there any plan to support this just like how we are using decl_module!?

If this is not a best practice or it's anti-pattern, is there any recommended way to implement similar features or behaviors?

IMO it's not that difficult. Take a look at how e.g. the GRANDPA and aura workers are started. If you write your code as a long-running Future<Item=(),Error=()>`` that selects on an Exit` parameter, you can easily dispatch it.


feat(generator-pie-component): DSW-2423 improve pie generator

Describe your changes (can list changeset entries if preferable)
pie-monorepo
[Added] - github action for automating component labels
[Changed] - update ci.yml syntax to match the one the "yaml" package writes
@justeattakeaway/generator-pie-component
[Added] - automation to update project-labeler.yml
[Added] - automation to add placeholder Percy token to ci.yml
[Added] - automation to update new package dependencies automatically
[Added] - automation to update Storybook with new package
[Changed] - ensure to update package.json exports in alphabetical order
[Changed] - updated documentation instructions
How to test
yarn build --filter=generator-pie-component
npx yo @justeattakeaway/pie-component

Author Checklist (complete before requesting a review)

[x] I have performed a self-review of my code
[x] I have added thorough tests where applicable (unit / component / visual)
[ ] I have reviewed the PIE Storybook/PIE Docs PR preview
[ ] I have reviewed visual test updates properly before approving
[x] If changes will affect consumers of the package, I have created a changeset entry.
[ ] If a changeset file has been created, I have used the /snapit functionality to test my changes in a consuming application

Reviewer checklists (complete before approving)
Reviewer 1

[ ] I have reviewed the PIE Storybook/PIE Docs PR preview
[ ] If there are visual test updates, I have reviewed them

Reviewer 2

[x] I have reviewed the PIE Storybook/PIE Docs PR preview
[x] If there are visual test updates, I have reviewed them

@siggerzz I'm pretty sure it fails silently (other than logging it) because I didn't realise for at least a week that the token for pie-radio was missing. I reckon this could be fixed in another PR.


loader: Fix unknown physical device extension handling

The physical device extension handling is currently broken on Windows 32-bit builds because the current implementation relies on compiler optimizations (tail-call optimization, to be specific) to optimize out the stack frame creation, so that the arguments get passed from the caller to the implementation, without the loader touching them.
This pull request uses assembly code to avoid touching the parameters that get passed on to the next function in the call chain. I was not able to find a way to use C, since C gives no way to prevent a stack frame from being created (even with inline assembly I couldn't find a way to do this, since I couldn't define a global symbol in visual studio from inline assembly). If the project is built for a platform where cmake can't find an appropriate assembler, then the loader will fall back on the C code (which may or may not work), while outputting a warning when CMake is run. The project uses MASM on windows (included in Visual Studio) and GAS on linux (included with GCC), so additional dependencies shouldn't be necessary. It is possibly that clang might not use the assembly code, but I haven't tested that.
This code only fixes the trampoline code for physical device extensions, because this is needed for a test that @Tony-LunarG has written. Similar fixes are likely needed for the terminator code, as well as unknown logical device extensions, but that is beyond the scope of this pull request.

Also, we had initially thought that this fix might solve issue #1846, but we're still seeing that problem, so it looks like there's more bad things going on there. This pull request doesn't change #1846 for the better or worse.


 [CP 406] [Potential 0.62] [A/D/S 8/6/14]
2016-08-09 11:45:26,648 [PokemonCatchWorker] [INFO] [no_pokeballs] No usable pokeballs found!
2016-08-09 11:45:36,361 [PokemonCatchWorker] [INFO] [pokemon_appeared] A wild Meowth appeared! [CP 406] [Potential 0.62] [A/D/S 8/6/14]
2016-08-09 11:45:38,911 [PokemonCatchWorker] [INFO] [no_pokeballs] No usable pokeballs found!
2016-08-09 11:45:49,171 [PokemonCatchWorker] [INFO] [pokemon_appeared] A wild Meowth appeared! [CP 406] [Potential 0.62] [A/D/S 8/6/14]
2016-08-09 11:45:51,434 [PokemonCatchWorker] [INFO] [no_pokeballs] No usable pokeballs found!
2016-08-09 11:46:01,292 [PokemonCatchWorker] [INFO] [pokemon_appeared] A wild Meowth appeared! [CP 406] [Potential 0.62] [A/D/S 8/6/14]
2016-08-09 11:46:04,420 [PokemonCatchWorker] [INFO] [no_pokeballs] No usable pokeballs found!
2016-08-09 11:46:14,125 [PokemonCatchWorker] [INFO] [pokemon_appeared] A wild Meowth appeared! [CP 406] [Potential 0.62] [A/D/S 8/6/14]
2016-08-09 11:46:16,868 [PokemonCatchWorker] [INFO] [no_pokeballs] No usable pokeballs found!
2016-08-09 11:46:26,493 [PokemonCatchWorker] [INFO] [pokemon_appeared] A wild Meowth appeared! [CP 406] [Potential 0.62] [A/D/S 8/6/14]
2016-08-09 11:46:29,395 [PokemonCatchWorker] [INFO] [no_pokeballs] No usable pokeballs found!
2016-08-09 11:46:38,851 [PokemonCatchWorker] [INFO] [pokemon_appeared] A wild Meowth appeared! [CP 406] [Potential 0.62] [A/D/S 8/6/14]

Same problem here!

also related to #3271


Support PHP8.0

We just merged support for it in https://github.com/openzipkin/zipkin-php/commit/b179cc11ddb30b6368106369d8f259189f2ca6d8

Could you please send me a email when support php8?
Thanks very much

Could you please take a look at this

Thanks

@jcchavezs merge please?

I will merge this, tho 9.0 is failling we can tackle that in a next PR.


Setting customColors questions

I'm not familiar with PostCSS.  Must the file that defines custom UI color overrides be a css file instead of Sass file now?
Do we redefine variables written like this: $primary-color: #1C3849 !default; or this: ```--primary-color: #1C3849 !default;

Yes, it should be a css file. You don't need to redefine variables, it accepts sass-like syntax.


md-menu wrong position

Bug, feature request, or proposal:
md-menu wrong position
What is the expected behavior?
md-menu postion on button
What is the current behavior?
md-menu showing on the bottom of page

when i click the button,the menu appear on bottom...

same issue here...

Hey all, check out the getting_started file, you'll be missing this, specifically this:

For alpha.7, you need to include the overlay styles in your app via a link element. This will look something like:

<link<EMAIL_ADDRESS>rel="stylesheet">

In future releases, all of the core styles will be combined into a single distributed css file.

Same problem. @jmcgoldrick, it has nothing to do with this css file.
Reported already at https://github.com/angular/material2/issues/1085

Closing this into #1139
@theunreal if you can reproduce this in a setup that has the core styles loaded can you file a separate issue for this? AFAIK, this issue has been caused by missing the core styles.

Hey @jelbourn, There is already a plunker for this in https://github.com/angular/material2/issues/1085#issuecomment-242399239. This is not about the core styles.
https://plnkr.co/edit/yuPzIJ?p=preview
The core styles are imported but the tooltip appears in a wrong position. same for menus.

@theunreal the missing styles is still the issue; the problem is that the overlay styles weren't included in core.css (which was wrong), so you also need overlay.css until it's fixed:
https://plnkr.co/edit/fXznny6aMl5cvajDDuOM?p=preview

@jelbourn Well the plunker wasn't RTL as my issue
https://plnkr.co/edit/5dJE21K2rqMHQupaAm3C?p=preview
In RTL the positioning is all wrong, with or without the styles. I can't even see the tooltip/menu

add css resolve the issue👌


Do you have any thoughts to make other ports apart from Windows application of this game?(example web player,Android/IOS,etc)

I was thinking on making the game available also on touch devices, such as tablets and cell phones, but i don't have any thought on making this available on any web players on the internet right now.


ListBox drag and drop - touch screen

I cant use drag and drop functionality on touch screens with ListBox/ListView.
Is it planned in the future or has anyone a workaround for it?

It's created by using DoDragDrop().
The touchscreen was on our test machine, I think a Lenovo Yoga 2 tablet.

Having same issue on all touch screens. Testing with Surface Pro 2, 3, and 4.

I had a similar issue and in the end settled on it being a conflict between the native inertial scrolling in listbox and derived controls. It makes sense in that it's quite hard to tell the difference between a "press and move, meaning scroll the window" and a "press and move. meaning pick up that item for a drag"
You could try disabling PanningMode, that may help?

Adding ScrollViewer.PanningMode="None" fixed it. I spent half the day working on this and  you just saved me the other half! Thanks you so much!!!

No problem, glad it helped you out :D

Problem is, setting the PanningMode is barely a work-around, and having a touch enabled application where you can't scroll a list with touch is a no go. Any udpate on this ? @punker76
Something based on a hold gesture to trigger the drag would be great, but I have yet to figure out how to implement it @punker76

Tap-release-hold-drag is a fairly common gesture to indicate that you want to do a drag rather than a scroll.

I'm also interested in this feature, would be great to have it!

I don't know if this helps, or if there are obvious holes in this I haven't realized yet, but a work-around that seems to be working for me is:

Leave PanningMode set to its default value (Both, I believe?)
In the SelectionChanged event, change PanningMode to None
Set a timer that fires after some brief interval (800 milliseconds seems reasonable for my particular application)
When the timer fires, turn the timer back off, and reset PanningMode to Both

Pseudo-y code:
  ctor()
  {
    // There are of course a bunch of different timers. When coding this, I got my
    // grubby fingers onto System.Timers.Timer first, so that's what I used here :-P
    _touchDragTimer = new Timer();
    _touchDragTimer.Interval = 800;
    _touchDragTimer.Elapsed += _touchDragTimer_Elapsed;
  }

  void lstItems_SelectionChanged(sender, e)
  {
    _touchDragTimer.Stop();
    lstItems.SetValue(ScrollViewer.PanningMode, PanningMode.None);
    _touchDragTimer.Start();
  }

  void _touchDragTimer_Elapsed(sender, e)
  {
    _touchDragTimer.Stop();

    Dispatcher.Invoke(
      () =>
      {
        lstItems.SetValue(ScrollViewer.PanningMode, PanningMode.Both);
      });
  }

With this code, you can drag the view around as much as you want, but if you want to drag an item, tap it once to select it, and then you have a brief interval within which to start a drag operation. Let it sit, and it reverts to scrolly behaviour.
I haven't investigated whether this would be easy to integrate directly into GongSolutions.Wpf.DragDrop, but maybe someone might find it useful (or be able to tell me why I shouldn't be doing it! :-D).

I tried this but this part lstItems.SetValue(ScrollViewer.PanningMode, PanningMode.Both); throws an error for me
An object reference is required for non static field 'ScrollViewer.PanningMode'

I tried this but this part lstItems.SetValue(ScrollViewer.PanningMode, PanningMode.Both); throws an error for me
An object reference is required for non static field 'ScrollViewer.PanningMode'

Use ScrollViewer.PanningModeProperty instead.

Oops, yes, sorry about that, 'twas a typo. 😳

I changed the code a little. Instead of changing the PanningMode on the SelectionChanged event, I have it on PreviewTouchDown, this way the drag is instant rather than having to select the item and then drag it.
        private void list_PreviewTouchDown(object sender, System.Windows.Input.TouchEventArgs e)         {             _touchDragTimer.Stop();             lst_scheduled.SetValue(ScrollViewer.PanningModeProperty, PanningMode.None);             _touchDragTimer.Start();         }

I changed the code a little. Instead of changing the PanningMode on the SelectionChanged event, I have it on PreviewTouchDown, this way the drag is instant rather than having to select the item and then drag it.

But then you're not doing the double-tap-and-drag gesture any more, are you?

Indeed, how do you scroll the item's container? With that change, why even have different panning modes? You could just leave it permanently set to None, couldn't you?

Indeed, how do you scroll the item's container? With that change, why even have different panning modes? You could just leave it permanently set to None, couldn't you?

Yes, you are right. When I tested initially there were only a couple items in the box :( I just want a smooth drag without the double tap. Maybe a hold and drag would do.


Configure master node same as slave node

This is pre-requirement for running all master components in a pod. #3853 was filed to run etcd as a pod. To make sure we really run etcd as a pod using Kubernete network model without specifying HostPort from spec, we need config the master node:

create networking bridge called cbr0
config docker run with "--bridge cbr0 --iptables=false"
add default route for master node (on GCE). Need to figure out other cloud providers.

cc/ @ArtfulCoder @thockin

Or finally switch to flannel. :)
On Wed Feb 04 2015 at 1:22:43 PM Dawn Chen<EMAIL_ADDRESS>wrote:

cc/ @ArtfulCoder https://github.com/ArtfulCoder @thockin
https://github.com/thockin
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/4128#issuecomment-72942095
.

This is now done for GCE and GKE.  We need to do it for AWS (and others?)

Marking this done, as of #5892 we now configure AWS, GCE and GKE this way.


Advance Slides and OBS

Describe the bug
Transition between Scene no longer work with version 1.20.00
See this video, where transition works, until you update the plugin
https://www.youtube.com/watch?v=P3NDWLdrQF0&list=PLwkIOrI8nN9jg4gBzIkPRsZT51dSOLz77&index=6
What could have changes?
Thx
R

Thanks for your interest in the Advanced Slides and OBS plugin.  I updated the example vault to include Advanced Slides 1.20.0.  Obsidian_AdvancedSlides_Demo_Vault
I'm working on a new version of the reveal.js slides for OBS.  Please give it a try.
Reveal-Slides-for-OBS


Implement new extension points in IdentityPlugin and add ContextProvidingPluginSubject

Description
Companion PR in core: https://github.com/opensearch-project/OpenSearch/pull/14630
This PR by itself does not add additional functionality, it simply implements the new extension points in core and introduces a new class called ContextProvidingPluginSubject which populates a header in the ThreadContext with the canonical class name of the plugin that is executing code using pluginSystemSubject.runAs(() -> { ... }). See ./gradlew integrationTest --tests SystemIndexTests for an example that verifies that a block of code run with pluginSystemSubject.runAs(() -> { ... }) has contextual info in the thread context.

Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Enhancement
Issues Resolved
Related to https://github.com/opensearch-project/security/issues/4439
Check List

[X] New functionality includes testing
[X] New functionality has been documented
[X] New Roles/Permissions have a corresponding security dashboards plugin PR
[X] API changes companion pull request created
[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Thanks @cwperks for adding this enhancement. Will this feature required System Index Permissions feature flag to be enabled?

No, it works independent of the System index permissions (I was considering using them in a previous iteration, but have since updated the PR). I'm currently working on removing the identity feature flag from core because it currently requires the Identity feature flag to be enabled.
I have a PR opened in core to remove some code in the RestController that uses the Identity feature flag because the current code is not flexible enough to support the security plugin: https://github.com/opensearch-project/OpenSearch/pull/15430
Removing the identity feature flag requires coming to a conclusion on how the code in the RestController should be changed.

@RyanL1997 @stephen-crawford @willyborankin @peternied Could I get another review of this.
FYI the code introduced in this PR will have to be opted into as part of an effort to remove instances of ThreadContext.stashContext across the plugins: https://github.com/opensearch-project/opensearch-plugins/issues/238
This code won't be directly used until plugins have adopted the changes.

@peternied Thanks for the review! I'll get back to the comments ASAP.
I think you may be interested in this PR I made in core that allows a plugin dev to specify what transport actions a plugin would need to perform outside the authenticated user context. The security plugin is a good example because there's a use-case where security needs to be able to write to the audit log index (if using an opensearch index) regardless of the authenticated user's permissions.
Does that help answer your first question?

I'm having a little trouble grokking the scope of these changes - it seems like there should be a new component that serves as a source of truth for plugin permissions that is missing - I've called this out.

In the PR in core, the cluster admin will be prompted at installation time what a plugin is requesting to be able to perform.

createSecurityRole doesn't align with how these configuration has been used before, its meant to be a representation of the security configuration from the security index, not a way to update/generate/create security objects.

Rather than overload SecurityRole with this responsibility, lets make a new component that is initialized in its own way and separately from our existing configuration. Its strange to have the logic to create these roles inside of configuration model specific code and for its state to be held inside of the privilege evaluator.

That's a good point. I pushed a commit to abstract this out into a separate class called InMemorySecurityRolesV7 and made it distinct from any of the ConfigV7/ConfigV6 classes which are backed by the security index.

@peternied Here's some mermaid diagrams that I made with the help of ChatGPT-4o
Here's the prompt:
In the plugin installation process:

OpenSearch will read both plugin-security.policy (JSM Permissions) + plugin-permissions.yml (requested transport actions) and prompt the administrator about the requested permissions. Administrator can choose to accept the risks or abort the plugin installation process.

In the OpenSearch Bootstrap process:

The IdentityService is initialized using a singular IdentityPlugin
For each IdentityAwarePlugin, the IdentityPlugin will assign a PluginSubject
The plugin uses the pluginSubject to execute transport actions using the identity associated with the plugin

Low-Level Detail
The permissions defined in plugin-permissions.yml are used to create an In Memory role that allows the security plugin to re-use all existing authz logic
pluginSubject.runAs(() -> { ... }) is a direct replacement for try (ThreadContext.StoredContext ctx = threadContext.stashContext()) { ... }

Plugin Installation Sequence
sequenceDiagram
    participant Admin as Administrator
    participant OpenSearch as OpenSearch
    participant Plugin as Plugin

    OpenSearch ->> Plugin: Reads `plugin-security.policy` (JSM Permissions)
    OpenSearch ->> Plugin: Reads `plugin-permissions.yml` (Transport Actions)
    OpenSearch ->> Admin: Prompt for requested permissions

    alt Admin accepts risks
        Admin ->> OpenSearch: Accept permissions
        OpenSearch ->> Plugin: Continue with installation
    else Admin aborts
        Admin ->> OpenSearch: Abort installation
    end

OpenSearch Bootstrap Sequence:
sequenceDiagram
    participant Bootstrap as OpenSearch Bootstrap
    participant IdentityService as IdentityService
    participant IdentityPlugin as IdentityPlugin
    participant Plugin as IdentityAwarePlugin

    Bootstrap ->> IdentityService: Initialize IdentityService
    IdentityService ->> IdentityPlugin: Use singular IdentityPlugin

    loop For each IdentityAwarePlugin
        IdentityPlugin ->> Plugin: Assign PluginSubject
        Plugin ->> IdentityPlugin: PluginSubject executes transport actions
    end

    Plugin ->> Plugin: Executes actions with associated identity

Just one heads up that one goal of https://github.com/opensearch-project/security/pull/4380 is to remove SecurityRoles alltogether.

Sync'ed with the latest from main

Resolved conflicts after merge of latest PRs

Just one heads up that one goal of #4380 is to remove SecurityRoles alltogether.

Thank you @nibix! I will check to see what changes are necessary to incorporate this change with Optimized Privileges Evaluation. I'm introducing the "in-memory" role for a plugin in this PR with the goal of eventually incorporating the changes from this PR in core so that we can re-use the authz checks when plugins want to operate outside the authenticated user context.

Can you help me understand why not? To take this to an extreme, if a user object representing plugin could have bad interactions with existing scenarios, why re-use the User object at all?

The only real distinction is in that logic to get the "roles". For a regular user, it would filter from the roles that are in the cache, but for a plugin user it would get or create an instance of a role that's only kept in the memory of the process and not backed by the security index. With this PR, pluginSubject's are limited to interactions only with their own system indices. I plan to do a follow-up to this PR once another corresponding PR in core is merged that let's plugins request additional permissions at installation time.
Again, the distinction is in how to compute the role or roles that are used to evaluate permissions. I will abstract the logic into a single method.

I will open a separate PR rebased on https://github.com/opensearch-project/security/pull/4380


Fix TimeMachine Request

Short description

Fix timeMachine request in router because it was failing hard.

Solution

The request had decimals for the timestamp which caused a Not Found response from the DarkSky API.

Implementation

Rounding the timestamp and then cast it into an Int.

[x] Round the timestamp
[x] Cast the rounded timestamp as an Int

GIF

@sergigracia Any blockers to merge this fix?

@0mega good fix. We'll merge it since the CI issue is not related with your PR. Thanks for the contribution.


Time provisioning inside PDO enclaves

A PDO enclave, in general, does not deal with time, but certificate expiration verification within sgx/openssl does.
Here is an overview of:

how time is handled within sgx/openssl and the untrusted environment;
what happens when the dependency on untrusted time is removed;
and how to remove it safely.

First, X509_verify_cert (an ssl function used, for instance, to check IAS verification reports) checks for expired certificates. Even within an enclave, where presumably trusted time is not available, the function works properly.
This is due to the (untrusted) time provided by sgxssl, through the edge call in its EDL. Such EDL is imported in essentially all enclaves used in PDO, namely in the eservice, pservice and test enclaves.
Second, the edge API can be removed unsafely. Without the edge API, the enclave will likely compile, and likely crash when time is needed. This is due to the implementation of this assert which will check for the availability of the ocall before calling it. Note that the ocall will still be available in the untrusted sgx/openssl library, but there would be not call initiated from the enclave.
Third, instead, the edge API can be safely removed by making sure that ssl does not request the time from the system (and hence the edge API). In the case of certificate expiration verification, this can be achieved through the use of the X509_V_FLAG_USE_CHECK_TIME flag and setting the time against which to check the expiration, from within the enclave.
This brings to the question: how to best provision time inside an enclave for such verifications (and remove the ocalls)?
(comments are welcome)

Also related to above is time-handling for JWT which requires a [ignore-time patch in SGXDataCenterAttestationPrimitives]
(https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/aa239d25a437a28f3f4de92c38f5b6809faac842/external/0001-Add-a-macro-to-disable-time-support-in-jwt-for-SGX.patch#L4) which is used also in our PR #501


Feature/circular deps

Description
Resolve circular dependencies:

Add a Webpack plugin to our dev config that checks for circular dependencies
Resolve circular dependencies involving resetEventsPolling (which was used inside the store
and itself relied on the store)
Resolve Lish/Glish dependency chain (Glish was importing methods from Lish, which in turn
imported Glish)
Resolve reportException chain (moved handleUnauthedErrors to its own utility)

Note to Reviewers
There may be better ways to resolve some of these; and none of these chains seems to have broken anything. I figured better safe than sorry. One thing we could do is set failOnError to false, so that we get warnings on compile rather than errors.
The biggest change was to our events logic, where events.ts depended on the store, but the store depended on events.ts. I moved the inline variables out of events.ts (it was weird to have them there in the first place) and added them to the store. Hopefully we'll be redoing all of our events logic in the near future, and we'll be getting rid of either the store or the RXJS subscription, which were never meant to exist in parallel forever.

We actually are currently using a package called madge to detect circular deps. If this package is finding stuff that Madge isn't, we might want to look into why.
Currently, Madge runs on the pre-commit hook:
https://github.com/linode/manager/blob/develop/packages/manager/package.json#L100

Yeah, I tried a bunch of stuff with Madge at the cmd line, but it never reported anything. I googled around until I found this plugin, which started spitting stuff out right away. No idea why it's finding things that Madge isn't, but the stuff it's turning up is legit.

Madge found one when I was refactoring that the Webpack plugin didn't, so I'm ok keeping both.

@martinmckenna did this. What's funny is that I'm pretty sure updateById is never actually being used...it's only called once, in a single reducer case for Linodes that afaict is never used.

@Jskobos Ah you're right. updateByIDs usage was completely removed in this PR:
https://github.com/linode/manager/pull/5222
Sorry to do this - I would just leave sentry the way it is, and in this PR or another, get rid of the updateLinode action along with the updateByID function.
Then we should make a ticket to update ramda and @types/ramda, hoist them up to the root package.json, since both projects use it.


Is there an equivalent of this repo for nodejs ?

Hello,
I'm looking to manage my Azure B2C users from a nodejs API. DO you have some samples on how to do those actions ?

@abouroubi you can check out this: ms-identity-b2c-javascript-nodejs-management

Exactly what I was looking for.
Thank you.


Update commons-io to 2.7

Updates commons-io:commons-io from 2.6 to 2.7.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "commons-io", artifactId = "commons-io" } ]

labels: test-library-update

Codecov Report

Merging #369 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #369   +/-   ##
=======================================
  Coverage   86.59%   86.59%
=======================================
  Files         135      135
  Lines        1485     1485
  Branches       36       36
=======================================
  Hits         1286     1286
  Misses        199      199

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a028b2b...ce95c66. Read the comment docs.


Question regarding Forbidden: Too many URLs added too fast. Slow down please.

So if I attempt to use the API, if a user clicks 'get link', what happens is automatically generate multiple links, while the page is actually loading.
Any ideas around this?

Not reproducible in the latest release. Please re-open this issue if it still occurs, with a set of reproducible steps. Check this element's CONTRIBUTING.md for an example.

not an ideal solution however if anyone comes across this problem you edit yourls-api.php and put the php sleep function in eg sleep(1); to stop multiple links gettings generated


Support for Tamil alternate consonants

How does one encode / transliterate Tamil alternate consonants, e.g. ற, ன (r2, n2 perhaps?)

Transliterate to which (presumably extended sanskrit) roman schema? If you seek some cousin of IAST, https://www.loc.gov/catdir/cpso/romanization/tamil.pdf is a possibility :

Perhaps, ऱ/r̂ and ऩ/n̂/n(!) - it's a bit peculiar, esp. the ऩ, as even the pronunciation is no different!

I think the best solution is to create xsanscript.py with support for extended characters, similar to https://github.com/karthikraman/indic_transliteration/commit/bc6db58fee8488e20a97d36738605ca3f2c24568

मित्र, अधिकृतोऽसि तथा कर्तुम् अधुना।

तथा च यथोचितम् जातितत्त्व(=inheritence)-प्रयोगं कुरु

Quick fix via xsanscript.py https://github.com/sanskrit-coders/indic_transliteration/commit/c78382e0dbf2255b2feefcd68da1eb0c061cbca1

साधु, कार्त्तिक। जातितत्त्व(=inheritence)-प्रयोगं कुरुष्व - नाम, (mostly) only code in that file should be that which is changed between the two files.

e.g. ற, ன (r2, n2 perhaps?)

IAST, ISO - e.g. ṟa, ṉa
ITRANS - e.g. Ra, ^na
Devanagari - e.g. ऱ, ऩ
Please see
https://en.wikipedia.org/wiki/National_Library_at_Kolkata_romanisation
http://www.virtualvinodh.com/wp/character-matrix/