Add more descriptive visualizations for pixel and cell clustering

What is the purpose of this PR?
Closes #421. @cliu72 has requested that we add a few more visualizations to bring the notebooks more in line with what she's written up in R. This PR adds those in.
How did you implement your changes
The barplot and clustermap functions in seaborn are a huge help. We also need to z-score the values for the heatmap visualizations and add some extra parameters to ensure clarity.
Remaining issues
visualize.py is quickly becoming crowded with different functions and long parameter lists. We may want to consider passing parameters as a dict rather than explicitly specifying each one.

Few notes:

This PR currently only addresses visualizations 1-4. Obviously, certain aspects (ahem colors, text sizes, etc.) need work, but I thought I'd take a few comments from the audience first before addressing these head on.
I'm dealing with an issue with barplot where the visualizations for SOM and meta cluster counts display different background colors and scales. I think the distribution of meta cluster counts may be a factor. Any suggestions would be appreciated!
I'm leaving both plot_barchart and plot_barplot in for now. Ideally, I'd like plot_barplot to replace plot_barchart, unfortunately barplot has a very different way of handling stacking than plot_barchart. This is an issue I think we can tackle later.

Status update:

In addition to patching up the visualizations, I've also left in commented areas of the code that were specifically requested by Candace (and, in some cases, left the original code prior to changing). Not all of them are addressed yet, but you can see a majority of them added in.
The visualizations look different in large part because of the pixel subsetting change Candace requested. We now remove zero-sum rows before subsetting rather than after. This led to more training data on average being available per fov.
The cell_size normalization for cell clustering has also been standardized so now we do it completely in R. There were some bugs early on in this process but it has been corrected.

I've addressed the code review comments and added more testing.

@ngreenwald so I'm not testing example_cell_clustering.ipynb for now since there are so many dependencies to mock up from example_pixel_clustering.ipynb and most of the calls in example_cell_clustering.ipynb go through R scripts. I'll have to continue @bcollica's research on better R installation for testing.

Got it, that makes sense

Something weird seems to have happened on ReviewNB where a lot of the comments I made yesterday moved positions in the notebook (either that or I was off my rocker yesterday). Most of the comments seem to have moved up a few cells, but if it's unclear what I'm talking about in a comment, please let me know.

Please ignore the visualizations when reviewing; these will get addressed in the next PR (adding Zak's metaclustering in).

Looks good! I think it might be nice to change the h_cluster_cap suffix to _consensus_cluster or something similar so that it's more intuitive when users are manually specifying it. What do you guys think?

Sure, I think consensus cluster makes sense.

Looks good! I think it might be nice to change the h_cluster_cap suffix to _consensus_cluster or something similar so that it's more intuitive when users are manually specifying it. What do you guys think?

@ngreenwald absolutely. I've been kind putting that off for now but it's a better time than ever to make it crystal clear to the user.

For preprocessed_dir  and  subsetted_dir: we can add a timestamp prefix if you'd like. However, Noah and I agreed at the time it would be best to leave this step out since users can piggyback off of the same preprocessed_dir  and subsetted_dir . Do you expect users to tweak the gaussian_blur  and subset_percentage  often?  I'll bring it up at standup tomorrow regardless. For norm_vals.feather  (and other intermediate file names such as cluster_counts.feather  in example_cell_clustering.ipynb ): I think it's best to leave this as hardcoded params into the respective functions. Per my understanding the user will have little to no reason to be seeing these, and even if they do, I still think the file names for these should be very hands off (there are already so many ways to confuse different runs and different file names). But I can discuss this at standup tomorrow too.

View entire conversation on ReviewNB

I like doing this here because it sort of stays in line with the "let's do these aggregations in the notebook so the user can tell exactly what they'll be passing into the interactive reclustering visualization" mentality. I also think it helps to be a little more transparent about stuff like weighted channel average which require additional documentation.

View entire conversation on ReviewNB

@cliu72  I know the colorbar on the pixel/cell cluster overlay was kind of a stickler, I figured out how to fix it (adjust the colorbar). I'll have to address this separately, however: we have other visualizations outside of pixel/cell clustering in ark that use this visualization. I think we should wait until this PR is merged into master, then we can address it all at once.

For files such as norm_vals.feather  I mean we could have a section where we set those near the top but then have something like WARNING: do not touch these! . A bit overkill, but I still believe that making more knobs available to the user to turn can lead to more confusion.

View entire conversation on ReviewNB

Don't worry about these for now. This page will only be available once it's merged into master. For now, you can access all of the documentation on the temporary PR RTD page: https://ark-analysis--460.org.readthedocs.build/en/460/_markdown/ark.phenotyping.html .

View entire conversation on ReviewNB

I can talk about this at standup tomorrow.

View entire conversation on ReviewNB

I still prefer to have the prefix added. I think it will be common for people to tweak both gaussian_blur and subset_percentage (Gaussian blur for sure, I've used different blurs for the different projects I've run in the lab, and subset percentage if people want to figure out the limits of subsetting to still get consistent clusters), which will change both preprocessed_dir and subsetted_dir. I think norm_vals.feather should also have the prefix if people want to compare how the normalization values change with the amount of subsetting (and also comparing how normalization values are different between markers). Overall, I think it's better to be consistent with file naming. If some files are being overwritten and others are not each time this is run, I think it will lead to overall confusion. I don't think it hurts to feed in a variable name for those file names instead of hard coding it in. I'm not sure I understand why it would lead to more confusion to have the prefixes. The user will just set the prefix and not touch anything else.

View entire conversation on ReviewNB

I'm not sure I understand this reasoning. Why is the documentation for weighted channel average relevant here? I think my overall philosophy is that things that the user won't have to touch should be abstracted away, and I don't see the user ever having to change this. I don't think it's necessary for the user to see this line to do the interactive reclustering.

View entire conversation on ReviewNB

@cliu72 I know the colorbar on the pixel/cell cluster overlay was kind of a hanging issue, I spent some time fixing it (adjust the colorbar). I'll have to address this separately, however: we have other visualizations outside of pixel/cell clustering in ark that use this visualization. I think we should wait until this PR is merged into master, then we can address it all at once.

Oh whoops sorry, forgot to send you the code. Here it is: https://drive.google.com/file/d/1-F1GmxQqOCBknnUkZDrVtN7VLifCnMNS/view?usp=sharing. Didn't look into how you implemented it but hopefully it's similar.

@cliu72 I know the colorbar on the pixel/cell cluster overlay was kind of a hanging issue, I spent some time fixing it (adjust the colorbar). I'll have to address this separately, however: we have other visualizations outside of pixel/cell clustering in `ark` that use this visualization. I think we should wait until this PR is merged into master, then we can address it all at once.

Oh whoops sorry, forgot to send you the code. Here it is: https://drive.google.com/file/d/1-F1GmxQqOCBknnUkZDrVtN7VLifCnMNS/view?usp=sharing. Didn't look into how you implemented it but hopefully it's similar.

@cliu72 it's similar, we might end up going your route (it's difficult formatting a divider made from `make although because I'm using text annotation rather than  actually placing the text on the axis (explicit formatting of the axis range and text annotation),

Ok, I discussed with Noah and it seems like I misunderstood the original comment. In terms of consistent file naming, absolutely agree, put a prefix at the beginning of each file, including intermediate ones. That's been addressed.

View entire conversation on ReviewNB

Looks good. I think we can try merging in Zak's tool now.

Looks good. I think we can try merging in Zak's tool now.

We'll do that in a separate PR. There are just a few minor final comments to address at this point, once they're done I'll re-request a final review, merge it in on approval, and we can start off of that.

When I was testing on a different dataset, I got this error. For someone who is familiar, it's not hard to fix the issue, but it requires going into soms_utils.py, changing 'dtype', reloading som_utils, and re-running create_pixel_matrix (which might be a lot for some people). Can we maybe include dtype as an argument in create_pixel_matrix so if somebody gets this error, it's easily fixable from the jupyter notebook?

When I was testing on a different dataset, I got this error. For someone who is familiar, it's not hard to fix the issue, but it requires going into soms_utils.py, changing 'dtype', reloading som_utils, and re-running create_pixel_matrix (which might be a lot for some people). Can we maybe include dtype as an argument in create_pixel_matrix so if somebody gets this error, it's easily fixable from the jupyter notebook?

OK, we can add a dtype argument.

Same comment as above regarding file paths and data directory locations.

View entire conversation on ReviewNB

@cliu72 sorry for taking so long to get to this. I'm going to request a review on this now since the 2 comments regarding file paths are due to the current example data structure not supporting the pixel clustering pipeline yet.
Cell clustering now writes out weighted channel average for both cell SOM and meta clusters. I've also added a dtype argument that you can use to tweak the way the images are read in and fixed the comment you specified.

Just one more small code change and doc change and I think we are ready to merge

The process is a bit clunky, but if you want to re-visualize the heatmaps that would be placed in Zak's notebook, you have to re-run the cells after reading in the average channel expression per pixel cluster/average pixel cluster counts per cell cluster/average weighted channel expression per cell cluster tables.

Now I should be ready to go. I've made the following changes:

In compute_p2c_weighted_channel_avg: use channels to subset rather than dropping columns
In compute_p2c_weighted_channel_avg: use reset_index to add meta_cols back
In compute_p2c_weighted_channel_avg: normalize the weighted channel values by cell_size
Only z-score the data just prior to running consensus clustering
Add code to take a new SOM to meta mapping and apply it to every pixel and cell cluster file dependency

Everything looks good. All of my comments were either changing text or comments to do with visualization. I think you said you want to do all the visualizations/Zak's tool integration in another pull request. If that is the case, I think we can merge this one.
The colored overlays shouldn't be viridis. The colors should correspond to the metacluster colors in the heatmap, so users can map their pixel clusters/cell clusters back to the images. Here is an example of generating colored overlay: https://colab.research.google.com/drive/1-F1GmxQqOCBknnUkZDrVtN7VLifCnMNS?usp=sharing

Yup, they're just here for now to verify that nothing broke during the clustering process.

View entire conversation on ReviewNB

Ok, I can add a comment that this visualization only supports pixel_meta_cluster.

View entire conversation on ReviewNB

@cliu72 most of these will be pretty straightforward to resolve. Yes, we will leave the interactive visualization for a separate PR. For the meta cluster overlay, I apologize that this one has always fell under the radar in some way or form. If I understand correctly, we want the colors assigned to each meta cluster in the heatmaps to match the colors assigned to each meta cluster in the pixel/cell meta cluster overlay–this would include the heatmaps generated by Zak's interactive visualization. If that's the case, we might want to wait until we open up the PR to address Zak's interactive visualization to address that. I'll spend some time working with Zak's tool, let's set a tentative meeting for tomorrow afternoon to discuss if I'm continuing to replicate the error.

Oh I realized where the confusion is now with the two columns in Zak's tool output. They're not actually duplicated columns. In Zak's tool, there is the functionality to rename clusters. See bottom left in screenshot below where I changed metacluster 1 to be called "T_cells." So the output has two columns "metacluster" which is still the number assignment, and "mc_name" which is the user specified names (if the users don't change the names, they stay as the cluster number). In the weighted cell cluster heatmaps after Zak's tool, it would be nice to display these user-supplied cluster names.

Same comment for the corresponding cell visualization here: will need Zak's visualization to map the colors onto here properly.

View entire conversation on ReviewNB

Oh I realized where the confusion is now with the two columns in Zak's tool output. They're not actually duplicated columns. In Zak's tool, there is the functionality to rename clusters. See bottom left in screenshot below where I changed metacluster 1 to be called "T_cells." So the output has two columns "metacluster" which is still the number assignment, and "mc_name" which is the user specified names (if the users don't change the names, they stay as the cluster number). In the weighted cell cluster heatmaps after Zak's tool, it would be nice to display these user-supplied cluster names.

OK that makes sense. So this'll be what I'll have ready for the next review:

Update the colorbar for the overlay function: I'm officially done trying to adjust the colorbar based on what we have currently (it doesn't play nicely with what we want for the pixel/cell cluster overlay), I'll create a new function to make the colorbar more in line with what you had in mind.
Using the "third column" as display name for the meta cluster names: I'll update the heatmap visualization and new overlay visualization to take in a list of names from the CSV returned by Zak's visualization (currently just a dummy pandas.DataFrame as placeholder) and use those names to identify metaclusters instead.

Looks good to me. I think we can close this pull request and start integrating Zak's tool now.

Great, I'll be merging this in after our computational standup today!


added option for systemd-timesyncd

Companion to https://github.com/OpenVoiceOS/ovos-utils/pull/200

can we add a use_sudo flag for all those commands? this works if sudo is passwordless, would be set in mycroft.conf

can we add a use_sudo flag for all those commands? this works if sudo is passwordless, would be set in mycroft.conf

Not sure what you mean here.  Where would this be added?

Gentle bump.
is this one still valid? Will see if i can implement this PR for testing for the buildroot image.

I think there's a typo somewhere,  but use it works afik

@j1nx can you look again. It is loading well now

@j1nx can you look again. It is loading well now

As all PR's have been merged related to this, the code will be present in the nightly alpha containers so will have a look next time I boot a new build.
Thanks for this as it was one of my PHAL issues to resolve.

Are you sure it's not broken without my PR?

Are you sure it's not broken without my PR?

Haven't tested it as of yet. Will have a look later but guess by te time I get to it your PR is also merged, so all will be good.


Oryol+Rust

Hi Floooh,
In your Design manifesto you mentioned that you have chosen C++ 'because Rust is not yet ready to replace C++'.
Out of your latest experience with Rust and changes that developers made in the Rust language would you choose it if you start Oryol right now?
I'm learning and playing last months with Rust and find it pretty fun and impressive. I'm planning to rewrite my old C++ project in Rust to get some experience with it. Hope it will be fun.

I'm currently following Rust development a bit from the sidelines. I should remove the sentence from the Design manifesto ;) All in all I think C++ is mostly moving into the wrong direction (becoming too high level, and mixing standard library things into the language), but C++ is fine when simply ignoring those broken parts.
I thought for a while that Rust could be a solution to let beginner programmers write sane high-level gameplay code without shooting themselves into the foot all the time, similar to how C# is used in Unity, just faster. But while a newbie programmer will write dangerous code in C++, he will simply write slow code in Rust now, because when 'wrangling with the borrow checker' the easy way out is always to throw more indirections into the code (like RefCells), which will be slow.
Another thing that holds me back is that I would need to write a lot of low level bindings for system APIs (unless there are already wrappers), I'm thinking about Metal on OSX/iOS, D3D11 on Windows, EGL (and that other custom lib which I don't remember the name right now) on RaspberryPi. And all the other useful higher level APIs which will be hit and miss (Imgui, Soloud, Bullet, ...).
I am thinking more and more about moving lower level parts of Oryol into small, standalone C libraries. That way the C++ parts of Oryol would be slimmer (more or less just the public module interfaces), and it would be much easier to build a Rust version of Oryol on top of those C libraries.
But it will not happen anytime soon I guess :)
That way

I'm also a bit frustrated where C++ is moving now. I'm still using it at work but as hobby I'm playing now more with Rust.
Moving to the C core library sounds reasonable. Author of the Limit Theory game goes same direction: he is rewriting a code from C++ to core C library and everything else in LuaJIT.
In any case thank you for the reply.


SSH proxy: does not work URL with port specified

When i'm trying to connect via SSH host "specified_ip_or_host_name" on port 2222, i get error
com.jcraft.jsch.JSchException: java.net.UnknownHostException: specified_ip_or_host_name:2222
mongo server url in "General" tab does not matter

Could you please provide more detailed info (screenshots with fake settings if needed)

i have the same problem

i have the same problem
it's naturally! i have learned plugin source code and proxy URL is not split into host and port as mongodb server's URL (in General tab)

An option to change the default port from 9080 to e.g. 2222 would be very helpful

Hi,
It will be done in a couple of day.

Hi,
I Have Same problem ...

updates?

This is the main thing blocking me from being able to utilize this plugin

datagrip  might be an alternative


[docs] Suggest using 32-bit watch simulator on Big Sur.

Please update the base branch to main by Oct 5th otherwise the pull request will be closed automatically.

How to change the base branch: (Link)
More detail about the branch update: (Link)

Please re-open this PR after resolving the conflict.


fix: add verbosity to config logger at load time

Summary
The config logging messages had one message for missing and invalid YAML files, it was changed so that the log will indicate better the problem to ease tracking the root cause.
Change Type
Please delete any irrelevant options.

[X] Bug fix (non-breaking change which fixes an issue)

Testing
I tested it manually
Test Configuration:
Checklist

[X] My code adheres to this project's style guidelines
[X] I have performed a self-review of my own code
[X] I have commented in any complex areas of my code
[ ] I have made pertinent documentation changes
[X] My changes do not introduce new warnings
[ ] I have written tests demonstrating that my changes are effective or that my feature works
[ ] Local unit tests pass with my changes
[ ] Any changes dependent on mine have been merged and published in downstream modules.

@danny-avila Why? Can you please explain, so I can learn?
BTW: The current code is less readable and the message it produces gives less information.


Result Different between Dev and Build

UnoCSS version
^0.55.6
Describe the bug
In dev:

After build:

Not generate after build:

(Inside box are not generate like [&_.kotak])
Reproduction
https://github.com/mzaini30/support-zen
System Info
OS: Windows
Browser: Brave
Framework: Svelte
Validations

[X] Read the Contributing Guidelines.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

@antfu, @userquin please help same shit( After production build some classes lost


Full Source video

Hey. The sample video output on the project demo page is long with a lot of cars. However, the one provided in the video download link is very short. Can you post the full video which is used for detection please? @ahmetozlu

added: https://github.com/ahmetozlu/vehicle_counting_tensorflow/blob/master/sub-1504614469486.mp4


MBP-622: Hardening the method for getting pattern index files

There's a bug in Hugo versions in the range of 0.123.0 to 0.123.4 that causes our patterns browser to render incorrectly. It seems that the all levels of child pages in a top-level Section consider themselves to be that section instead of sub-sections. So instead of using "Section" to identify pattern index pages, we'll now determine which pages are index pages using the parent path. Anything that uses "/patterns" as a parent path is considered an index file in our pattern browser.

This is an automated message:
You can preview this docs PR at http://411.docs-pr.validatedpatterns.io
Note that they get generated every five minutes, so please wait a bit.

Trying a different fix to harden the patterns index page detection since the previous fix didn't work on older versions of Hugo. For this fix, we just add a filter to only include pages that have "branch" BundleType.
Tested on Hugo 0.121.0, 0.122.0, 0.123.0, 0.123.4, 0.123.5, and latest (0.123.7). All seems to work.


DEFINE AXIS/BOUNDS non-reproducible after trying a 1-point axis

Reported by Andrew Wittenberg on 20 Mar 2006 23:18 UTC
This one is screwy.  We define a multipoint axis and it works fine.  Then we
define a 1-point axis and the result is wrong.  Then we repeat the first
multipoint axis definition again, and even that result is now very wrong!  I get
the same results in both v5.81 for Linux and v5.70 for Irix.
    NOAA/PMEL TMAP
    FERRET v5.81
    Linux(g77) 2.4.18 - 07/14/05
    20-Mar-06 18:09

yes? def ax/x/bounds xax = {0,3,5},{-2,2,2,4,4,6}
yes? list xboxlo[            X: -2 to 6
Column  1: XBOXLO is XBOXLO (axis XAX)
Column  2: XBOXHI is XBOXHI (axis XAX)
Column  3: X is X (axis XAX)
XBOXLO XBOXHI      X
0   / 1: -2.000  2.000  0.000
3   / 2:  2.000  4.000  3.000
5   / 3:  4.000  6.000  5.000
yes? def ax/x/bounds xax = {0},{-2,2}
Replacing definition of axis XAX
yes? list xboxlogx=xax,xboxhi[            X: 0
Column  1: XBOXLO is XBOXLO (axis XAX)
Column  2: XBOXHI is XBOXHI (axis XAX)
Column  3: X is X (axis XAX)
XBOXLO  XBOXHI       X
I / *:    -0.5000  0.5000  0.0000
yes? def ax/x/bounds xax = {0,3,5},{-2,2,2,4,4,6}
Replacing definition of axis XAX
yes? list xboxlogx=xax,xboxhi[gx=xax],x[gx=xax]
X: -0.5 to 2.5
Column  1: XBOXLO is XBOXLO (axis XAX)
Column  2: XBOXHI is XBOXHI (axis XAX)
Column  3: X is X (axis XAX)
XBOXLO XBOXHI      X
0   / 1: -0.500  0.500  0.000
1   / 2:  0.500  1.500  1.000
2   / 3:  1.500  2.500  2.000
Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/1401

Comment by<EMAIL_ADDRESS>on 21 Mar 2006 18:32 UTC
In xeq_define.F, there some code after a comment that --a single point is always
"regular"-- e.g. treated as a regularly-spaced axis, obviously should not be the
case if bounds are given! One of the effects of this is that the bounds info was
just skipped. Because of this bug, the variable its_reg is not reset to false,
which is why the last re-definition of the axis is also wrong.


Full Screen API doesn't work in object tag and frameset

It's work in firefox but not in Chrome and others browser. I know that it's work on iFrame. Please clarify my problem. Thanks

https://github.com/sindresorhus/screenfull.js#documentation

Use StackOverflow for support questions: http://stackoverflow.com/questions/ask


Hacky fix for annotation rotations (#984)

See: #984
This patch is designed to work around a bug in the OpenSeaDragon shipped with Mirador (1.1.1), where rotating an OSD viewport doesn't cause the (x, y) coordinates of its bounds to be adjusted accordingly. It still doesn't support arbitrary rotation yet, but is good enough for 90-degree increments supported by the canvas control overlay.
Note: If you plan on upgrading OpenSeaDragon within the next release, please check whether the bug described above has been addressed. If it has been addressed, this patch should not be applied.

Coverage decreased (-0.07%) to 44.148% when pulling 4c764f3351b7f98fe9abcfe54878f271e1906a84 on dicksonlaw583:789-canvas-controls into 5b7f2017bb71b28d8b0be81ffdf9b79da3411679 on IIIF:789-canvas-controls.


yt-subs.bat probably doesn't work

This script may or may not work on windows, I assume it may work in git bash/powershell, but otherwise probably not.

It does not work.
Running yt-subs.sh works under windows 10 (it launches git bash and works as expected).


Any idea to convert Objective-C to Swift ?

@wongzigii
By using bridging header file, you can use MZTimerLabel using Swift with no problem. So it's not in my plan to convert the library to swift yet.


[Question] rehearsal example clarification regarding train/valid splits

In the rehearsal example there's the following template:
scenario = ClassIncremental(
    CIFAR100(data_path="my/data/path", download=True, train=True),
    increment=10,
    initial_increment=50
)

memory = rehearsal.RehearsalMemory(
    memory_size=2000,
    herding_method="barycenter"
)

for task_id, taskset in enumerate(scenario):
    if task_id > 0:
        mem_x, mem_y, mem_t = memory.get()
        taskset.add_samples(mem_x, mem_y, mem_t)

    loader = DataLoader(taskset, shuffle=True)
    for epoch in range(epochs):
        for x, y, t in loader:
            # Do your training here

    # Herding based on the barycenter (as iCaRL did) needs features,
    # so we need to extract those features, but beware to use a loader
    # without shuffling.
    loader = DataLoader(taskset, shuffle=False)

    features = my_function_to_extract_features(my_model, loader)

    # Important! Draw the raw samples from `scenario[task_id]` to
    # re-generate the taskset, otherwise you'd risk sampling from both new
    # data and memory data which is probably not what you want to do.
    memory.add(*scenario[task_id].get_raw_samples(), features)

How does that change if we add train/valid splits?
for task_id, taskset in enumerate(scenario):
    if task_id > 0:
        mem_x, mem_y, mem_t = memory.get()
        taskset.add_samples(mem_x, mem_y, mem_t)

    dataset_train, dataset_val = tasks.split_train_val(taskset, val_split=0.1)
    train_loader = tud.DataLoader(dataset_train, shuffle=True)
    val_loader = tud.DataLoader(dataset_val, shuffle=True)

    for epoch in range(epochs):
        for x, y, t in train_loader:
            # Do your training here

    # Herding based on the barycenter (as iCaRL did) needs features,
    # so we need to extract those features, but beware to use a loader
    # without shuffling.
    unshuffled_loader = DataLoader(taskset, shuffle=False)  # --> here should it be taskset or dataset_train?

    features = my_function_to_extract_features(my_model, unshuffled_loader)

    # Important! Draw the raw samples from `scenario[task_id]` to
    # re-generate the taskset, otherwise you'd risk sampling from both new
    # data and memory data which is probably not what you want to do.
    memory.add(*scenario[task_id].get_raw_samples(), features) # --> scenario[task_id].get_raw_samples() returns all samples in the current taskset?

If unshuffled_loader uses dataset_train then len(features) $\neq$ len(scenario[task_id].get_raw_samples()[0]).
Another question, is do we need to add samples into memory buffer from both train and valid samples or just train samples? Because, in my understanding the taskset contains all samples before the train/valid split, right?

Hi @TLESORT,
thanks for the reply.

If unshuffled_loader uses dataset_train then len(features) \diff len(scenario[task_id].get_raw_samples()[0]).
True. is that a problem?

I think so because memory.add(arg1, arg2) throws an error when the two arguments have not equal number of sampels.

Another question, is do we need to add samples into memory buffer from both train and valid samples or just train samples?
Usually we would/should use a separate memory buffer for validation with data from all tasks seen so for or do the split taskset before adding samples and add samples only on train. In other, case some samples might be on train and later on val.

I didn't quite get the first part of using the separate memory buffer for validation, I'll add your suggested changes below to the MWE and let me know if there's a misunderstanding on my part.
for task_id, taskset in enumerate(scenario):
    dataset_train, dataset_val = tasks.split_train_val(taskset, val_split=0.1)
    train_loader = tud.DataLoader(dataset_train, shuffle=True)
    val_loader = tud.DataLoader(dataset_val, shuffle=True)

    if task_id > 0: # --> as suggested, first split taskset and then add samples to dataset_train only
        mem_x, mem_y, mem_t = memory.get()
        dataset_train.add_samples(mem_x, mem_y, mem_t)

    for epoch in range(epochs):
        for x, y, t in train_loader:
            # Do your training here

    # beware use a loader without shuffling.
    unshuffled_loader = DataLoader(dataset_val, shuffle=False)  # --> as suggested dataset_val to avoid overfitting?

    features = my_function_to_extract_features(my_model, unshuffled_loader)

    # Important! Draw the raw samples from `scenario[task_id]` to
    # re-generate the taskset, otherwise you'd risk sampling from both new
    # data and memory data which is probably not what you want to do.
    memory.add(*scenario[task_id].get_raw_samples(), features) # --> how should this change, is it dataset_train.get_raw_samples?

Could you also illustrate what you meant with Usually we would/should use a separate memory buffer for validation with data from all tasks seen so far, and where exactly it fits in the MWE?


updates code to react! 🚀

Description
removes angular front-end, moves to react
Motivation and Context
modernization and maintainability
Screenshots (if appropriate):

Checklist:

[x] My code follows the code style of this project.

fixes #137

@wmhilton it has been done! 😄


Revert "Copy springBoot application to devc hidden folder in container mode"

Reverts OpenLiberty/ci.maven#1576
After further internal discussion, we determined that this change is not warranted. The devc container mode requires that loose application is enabled. But Spring Boot JAR applications are not supported as loose applications.
There is an open issue to address supporting SpringBoot JARs in dev and devc mode. And there is discussion here on how to run Spring Boot apps in dev mode.

@cherylking,  I succeeded in running SpringBoot Jar in dev and dec mode, It's possible and not that hard! the problem with devc and SpringBoot is that you need the SpringBootUtility to be available on the host side so liberty:deploy can succeed...

@cherylking, I succeeded in running SpringBoot Jar in dev and dec mode, It's possible and not that hard! the problem with devc and SpringBoot is that you need the SpringBootUtility to be available on the host side so liberty:deploy can succeed...

@iRbouh Can you share your pom.xml configuration or a sample project for us to understand how you are successfully running a SpringBoot Jar in dev and devc mode? If you are using  <deployPackages>spring-boot-project</deployPackages> and package it as a jar, the application will not get installed as a loose application and we would not expect dev and devc to work correctly.

@iRbouh I'm still looking for some feedback on my previous comments/questions. We need more information so that we can determine whether this change should be reverted or not. Thanks for your time.

This is over a year old at this point. Going to close it without merging. We can make changes if needed later.


Chore: various code cleanups

Remove type specification from new array creation; not needed
Fix missorted modifiers
Use isEmpty() instead of comparing to the length; isEmpty() is the form most used in the code currently

Have run mvn clean test without error after the changes

Just curious : how did you find these improvements? Is that coming from a tool like Checkstyle or PMD, or did you use a different approach here?

I am using IntelliJ Analyze function for many of them but also PMD and SonarLint.

I use SonarLint on a few projects, I'm sure I've used for stuff on this one. Some of its findings can be noisy, but it's tunable.

@kingthorin Sorry to have touched off a hornets nest of comments.


update typescript to 2.0.8, add strictNullChecks=true and fix Calenda…

…rSpec type #3589

I also need this changing!

+1

Can you please update the typing tests in a way that would have failed this, please.

Also this strict null checks - isnt this possibly causing multiple problems for all existing definitions for which null was assumed possible. @blakeembrey  can you chim in, please.

@ichernev This is a good change, I had noted to TypeScript that this would happen in the 2.0 beta releases but it hasn't been fixed. It a real pain as a consumer and author. strictNullChecks won't cause any issues for the consumers, but the use of the undefined type does make the definition 2.0+ now (technically a breaking change). It might be a good idea to update anywhere else, sometime in the future, with null and undefined types for better type checking (people without strictNullChecks just see null and undefined disappear in the signature).
See https://github.com/Microsoft/TypeScript/issues/9235.

@blakeembrey TypeScript 2.0 is already released: https://blogs.msdn.microsoft.com/typescript/2016/09/22/announcing-typescript-2-0/
That is not beta version. 😄

@horiuchi I didn't say it was, I don't know where you got that from. Please re-read my comment. Also, that's my name literally in the post, so I am aware of it.

@blakeembrey Oh, sorry. I was misunderstanding. 😢
I was also using the TS of beta version 😄

@blakeembrey so 1) this undefined is a special case because of the []. 2) strictNullChecks is not that strict, but why use it then and 3) in the future add null/undefined if its expected in the signatures but it wont hurt if we dont add it... how is this strict. And how would we know where to add (like will there be issues or it will just help people figure out where null is ok)

@ichernev It's strict for the developer, but not so useful as an author because there's no code in the definition to tell if it should be null or not for strictNullChecks to be useful. The index signature issue is the only place I'm aware of that can cause a difference in compilation for a consumer. Basically, typing null in the right places as the definition author is the same as typing string or number - it comes from an understanding of the library itself.
On the consumer side, and I may not have described it properly above, it will give a signature like string | undefined, for example. However, that will only be when the consumer is also using strictNullChecks. In this case, it is actually really useful because consumers can avoid bugs like x.substr where x is string | undefined and they would be prompted to make sure x exists before treating it as a string.

Also about backwards compatibility -- should we ship the old and the new code somehow?  In general, did you plan for a transition? What if we just don't upgrade to 2.0? Is it a problem for 2.0 users? That is a bit disappointing ... (if true)

So its more important to say which of our methods can return null/undefined, so the strict checking will warn the users. What we take doesn't matter for strictNullChecks.

Basically, it's really useful to a lot of people using the stricter subset of TypeScript to add stricter signatures to catch bugs.
On backward compatibility, you can't really ship them together unfortunately. In this signature, you may be able to use | void as a workaround (I haven't tested this, but void <> undefined), but I'd recommend adopting 2.0 syntaxes where possible as it's really just a PITA to maintain otherwise and you won't be able to use the new things (like null | undefined types). In general, I've just been updating with the latest TypeScript as I go and haven't seen any issues with consumers being stuck on lower TypeScript versions.

So its more important to say which of our methods can return null/undefined, so the strict checking will warn the users.

Both can be useful. For consumers who enable strictNullChecking, it can be helpful on input types too (E.g. callbacks accepting null). Here's an example that I did (used void but can since change to null with 2.0) - https://github.com/types/npm-generic-pool/pull/3 - basically I wanted to be able to cb(undefined, value) but with strict null checking I wouldn't be able to use undefined as the first argument.

I understood, thank you for the explanation. So from a backwards compatibility standpoint -- it should be painless and we'll encourage users to upgrade.

basically I wanted to be able to cb(undefined, value) but with strict null checking I wouldn't be able to use undefined as the first argument.

Well doesn't that mean that, for example in our moment constructor, moment(null) is a valid way to create an invalid moment, but because its not explicitly specified now you won't be able to do it? That means a ton of PRs for | null here and there ... I mean I'd prefer to fix them all at once and not spend time sifting through a sea of incomplete PRs.

@ichernev Yep, anyone using strictNullChecks won't be able to do moment(null) right now. If you want to enable this, sounds like a good idea to go through and improve it everywhere in one swoop.

@ichernev added some tests, but what I think really does the check is the strictNullChecks. Maybe @blakeembrey knows a better way to test this.

Bump - @ichernev do you feel comfortable merging this? We're locking down our Moment to an old version for now.

Merged in c891a4fcb0492539184eac73229b7a8053a5bbf8


TEP-0118: Add validation for matrix combination count with matrix.include params

[TEP-0090: Matrix] introduced Matrix to the PipelineTask specification such that the PipelineTask executes a list of TaskRuns or Runs in parallel with the specified list of inputs for a Parameter or with different combinations of the inputs for a set of Parameters.
To build on this TEP-0018 introduced Matrix.Include, which allows passing in a specific combinations of Parameters into the Matrix.
This PR validates that matrix combinations count including parameters in Matrix.Include does not exceed the max matrix combinations count.
Note: This is still in preview mode. Other forms of validation and implementation logic will be added in subsequent commits.
/kind feature
Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[x] Has Docs included if any changes are user facing
[x] Has Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
[x] Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
[x] Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes
NONE

/retest

/hold
can we please merge the refactor first? https://github.com/tektoncd/pipeline/pull/6282 🙏🏾
it'll be harder so sync these changes with the refactor than it would be to update this smaller change

/unhold
https://github.com/tektoncd/pipeline/pull/6282 has been merged

thanks a bunch @EmmaMunley for a lot of back and forth and multiple rebases 🙏  appreciate all the efforts you spent!
/lgtm

/retest


LightWallet: Frontend Responsive Layout Error

What happened: Frontend responsive error annual interest pool pairs on Huawei Pro30.
"DUSD-Paar" - double-spaced
"Jahreszins:" - double-spaced
"123%" - double-spaced
Someone has reported a similar problem with an Iphone 11 Pro, too

Same problems with iPhone XS and Samsung 6 edge

Hi @Motopasion21 @arakis2022, thanks for raising the issue! We will take a look at this.


pta.transform.by_query() for transform_iter

Implementation is straightforward. The question is more how it should be specified. Three options:

Specify an iter=True kwarg

Call decorator pta.transform_iter.by_query()

Decorator checks name of method: transform -> dataframe,
transform_iter -> iterdict

3 is really compelling. I vote we go this route. We can always revise later if there's an unexpected downside.

Amazing, thanks

I knew you would jump for 3


detection if the telegram is off

when my telegram is off, or the connection is not stable, how to read the condition?
maybe there is a status of 1,2,3 or something?

Bug Report

I have read:

Usage information
Help information

I am using the latest version of the library.
Expected Behavior

Actual Behavior

Steps to reproduce the Behavior

Feature Request

I have:

searched for such a feature request (https://github.com/yagop/node-telegram-bot-api/labels/enhancement) and found none

Introduction

Example

Question

You could test by sending a message. But that is usually when the bot is blocked. But for online status, you can't do it.

You could test by sending a message. But that is usually when the bot is blocked. But for online status, you can't do it.

I mean, when a user sends a message to the bot, but the bot doesn't respond, how do I check whether the message has been read or not?

I mean, when a user sends a message to the bot, but the bot doesn't respond, how do I check whether the message has been read or not?

It is not yet possible to get read receipts with bots.

It is not yet possible to get read receipts with bots.

oh, oke thanks you

oh, oke thanks you


move d3 devDependencies to dependencies and switch from d3-bundler to rollup

Here's a PR for the approach I discussed here.
TL;DR: Using d3 modules with vanilla npm and webpack can be made more efficient/elegant without interfering with the d3-bundler and Rollup ES6 approaches.
As far as I can tell from discussion in #3, the only downside of moving d3 modules to "dependencies" was the problem of duplication. npm@3 (packaged with the current stable version of node) completely solves this problem. (If d3-A requires d3-B and d3-C, and d3-B requires d3-C, d3-C is not duplicated.) Even for those still using npm@2, bundlers like webpack also solve this problem with code de-duplication.
It has become standard practice for modular libraries to require other modules as dependencies in their npm builds (see lodash and React -- two of the most popular npm packages). Publishing the d3-X builds on npm with their dependencies pre-injected is a huge pain. If I want to use d3-A and d3-B which both depend on d3-C, my current options are:

Require d3-A and d3-B as dependencies and get d3-C twice. This is obviously bad. d3-C can't even be de-duped by bundlers because it's already been injected into each module separately.
Use d3-bundler to create a custom build exporting d3-A and d3-B. This isn't very elegant: I need to add a custom tool to my build pipeline, manually track/update it every time I require some new d3-X, etc., which is precisely the problem that npm was designed to solve.
Use a bundler which understands the "jsnext:main" package.json field (bypassing the compiled build completely, straight into ES6 land). This is great, but only available if you're using Rollup. (Also, that field is in a lot of flux.)

Option 3 should, of course, stay available. This PR just replaces Option 1 with a strictly superior Option 4: require d3-A and d3-B as dependencies and get d3-C just once (bonus: still in a modular format for easier debugging).
@mbostock Thoughts? I'm happy to follow through with similar PRs on the other modules if you're interested.
(By the way, I think it makes sense to continue to put compiled UMD (dependency-injected) builds generated using d3-bundler on places like the d3 website for people to use without npm.)

I can’t figure out how to get the newer version of Rollup to compile-in those external dependencies using --format=umd, so this approach isn’t immediately usable for me. Am I missing something?

Figured it out. :+1:


usdot_airports update

To incorporate the new source from FAA as the new usdot_airports dataset to build facdb. The downstream impact is tested in facdb in this PR.
Also notes added to how to handle this new source in the notes section at the bottom of the template. Because some additional processing to the downloaded spreadsheet is needed before the archive could be run.

switching back to WIP on this because @mbh329 made a good comment that the excel to csv step is a little confusing. I am going to work on a solution to do that manual step with a python script which should be relatively straightforward.

okay so the new commit above should have the process to directly ingest the excel sheet with python

I think we shouldn't push the .xlsx file in /tmp right?
Also maybe we could think about some re-usable process to convert .xlsx to .csv? Possible improvement.
Otherwise this seems good to me

Hey Sasha I think the python script is supposed to handle that. Is it possible you are looking at an older version of the branch?

Haha yes it's true the python script does do that. I'm thinking of something more generalized that is responsible for taking a .xlsx file, checking the encoding, and saving to .csv. Then the template could just point to the script and the path to the .xlsx file. The difference is that the python code wouldn't be specific to any dataset. It's not necessary and maybe doesn't make sense, just figured that the .xlsx to .csv is a common ingestion step

Haha yes it's true the python script does do that. I'm thinking of something more generalized that is responsible for taking a .xlsx file, checking the encoding, and saving to .csv. Then the template could just point to the script and the path to the .xlsx file. The difference is that the python code wouldn't be specific to any dataset. It's not necessary and maybe doesn't make sense, just figured that the .xlsx to .csv is a common ingestion step

After our conversation about doe lcgms, I suddenly realize I now know more about how to get url. And voila I found the url to the excel and now it is very simple with read_excel. Tested working on my machine. @SashaWeinstein @mbh329 @AmandaDoyle

This is awesome! Everything worked on my machine as expected. approved!

Hey realize this cannot be merged in without approval from @SashaWeinstein
I think it could be a good thing to have a generalized excel handling for data library except that I think those datasets are only handful and they are likely to require special treatment for each e.g. using different tab of the spreadsheet in this case "Airport". I guess one way to make a generalized assumption is to use the first tab of any excel input if not otherwise specified. But overall, I feel like might be simple enough to handle excel on individual basis with customized python script.
I do have an alternative solution in mind but I want to discuss it before implementing because it can look funky.

@SashaWeinstein my bad just saw this. Removed and should be good.

great!


Add XGBRanker to Python API doc

XGBRanker was added by #3560 and needs to be added to the Python API documentation.
@Shiki-H

@hcho3
I think it's appropriate place to notify you about the following. By default, Sphinx doesn't generate docstrings from parent class (XGBModel). That's why, for example, XGBRegressor at present has only documentation describing its constructor. :inherited-members: flag (http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html) can solve the problem.

Codecov Report

Merging #3619 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #3619   +/-   ##
=========================================
  Coverage     50.38%   50.38%
  Complexity      188      188
=========================================
  Files           172      172
  Lines         13983    13983
  Branches        457      457
=========================================
  Hits           7045     7045
  Misses         6713     6713
  Partials        225      225

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 24a268a...35e49f6. Read the comment docs.

@StrikerRUS Thanks for letting us know. I went ahead and added :inherited-members: to XGBRegressor.

No problem, @hcho3 !
And what about XGBClassifier and XGBRanker? For instance, get_booster, get_params, feature_importances_, etc. won't be shown at the docs page.
Also I've noticed that XGBRanker's predict method will not be shown even with :inherited-members: flag because it has no own docstring but overrides parent's method. Possible workaround (if you don't want copy-paste parent's docstring or write decorator https://stackoverflow.com/a/13743316):
predict.__doc__ = XGBModel.predict.__doc__


Support redis gem v5.x

Hi @benlangfeld, thanks for maintaining the rpush gems.
Can you please let me know if you have an estimate in mind for when this would be implemented? Or if there are any blockers?
Additionally, do let me know if I can help! I'd love to have the library support Redis 5+ because this library is the most recently updated library in the push-notification sending space and I'd love to keep it that way :)

Hi @benlangfeld, thanks for maintaining the rpush gems.
Can you please let me know if you have an estimate in mind for when this would be implemented? Or if there are any blockers? Additionally, do let me know if I can help! I'd love to have the library support Redis 5+ because this library is the most recently updated library in the push-notification sending space and I'd love to keep it that way :)

This is blocked on the failing tests here, and I don't have time to work on it. If someone can submit a version of this with passing tests, I can review it.

I attempted something @ https://github.com/rpush/modis/pull/47.

@benlangfeld would be great to complete this PR and start support redis gem v5

Just need a changelog entry here which I'll add when I get to my desk.


Container fails with "new page did not work", spawns infinite chromium instances and kills the server

After the image automatic overnight update, I woke up with an hung up server. I had several chromium processes that were consuming 100% CPU and filled both RAM and swap. I traced the leak back to this container and was able to reproduce the issue after a forced reset of the machine.
I believe the container is spawning infinite chromium processes because chromiumProcesses appears empty in the logs when the container tries to kill the failed chromium instance, and I see the PIDs number increasing in docker stats.
Screenshots or Logs
epicgames  | Run on startup: true
epicgames  | [2024-04-26 10:09:12.984 +0000] INFO: Started epicgames-freegames-node
epicgames  |     COMMIT_SHA: "e08fe11113382b3d07f04b8ff88d0d6c2fe601b8"
epicgames  |     BRANCH: "master"
epicgames  |     DISTRO: "alpine"
epicgames  | [2024-04-26 10:09:12.984 +0000] DEBUG: Performing version check
epicgames  |     PROJECT_NAME: "epicgames-freegames-node"
epicgames  |     BRANCH: "master"
epicgames  |     COMMIT_SHA: "e08fe11113382b3d07f04b8ff88d0d6c2fe601b8"
epicgames  | [2024-04-26 10:09:13.254 +0000] DEBUG: Starting web server
epicgames  | [2024-04-26 10:09:13.257 +0000] INFO: Checking free games for ---
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:09:13.257 +0000] DEBUG: Launching a new browser
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:09:17.493 +0000] DEBUG: Refresh auth token response
epicgames  |     user: "---"
epicgames  |     authResp: {
epicgames  |       "access_token": "---",
epicgames  |       "expires_in": 7200,
epicgames  |       "expires_at": "2024-04-26T12:09:17.550Z",
epicgames  |       "token_type": "bearer",
epicgames  |       "refresh_token": "---",
epicgames  |       "refresh_expires": 28800,
epicgames  |       "refresh_expires_at": "2024-04-26T18:09:17.550Z",
epicgames  |       "account_id": "---",
epicgames  |       "client_id": "---",
epicgames  |       "internal_client": true,
epicgames  |       "client_service": "prod-fn",
epicgames  |       "displayName": "---",
epicgames  |       "app": "prod-fn",
epicgames  |       "in_app_id": "---",
epicgames  |       "product_id": "prod-fn",
epicgames  |       "application_id": "---",
epicgames  |       "acr": "---",
epicgames  |       "auth_time": "2024-04-17T18:01:34.329Z"
epicgames  |     }
epicgames  | [2024-04-26 10:09:17.494 +0000] INFO: searchStrategy is `all`: searching for weekly and promotional games
epicgames  |     user: "---
epicgames  | [2024-04-26 10:09:17.494 +0000] DEBUG: Getting current weekly free games list
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:09:17.495 +0000] DEBUG: Setting auth from device auth
epicgames  |     user: "---"
epicgames  |     deviceAuth: {
epicgames  |       "access_token": "---",
epicgames  |       "expires_in": 7200,
epicgames  |       "expires_at": "2024-04-26T12:09:17.550Z",
epicgames  |       "token_type": "bearer",
epicgames  |       "refresh_token": "---",
epicgames  |       "refresh_expires": 28800,
epicgames  |       "refresh_expires_at": "2024-04-26T18:09:17.550Z",
epicgames  |       "account_id": "---",
epicgames  |       "client_id": "---",
epicgames  |       "internal_client": true,
epicgames  |       "client_service": "prod-fn",
epicgames  |       "displayName": "---",
epicgames  |       "app": "prod-fn",
epicgames  |       "in_app_id": "---",
epicgames  |       "product_id": "prod-fn",
epicgames  |       "application_id": "---",
epicgames  |       "acr": "---",
epicgames  |       "auth_time": "2024-04-17T18:01:34.329Z"
epicgames  |     }
epicgames  | [2024-04-26 10:09:17.495 +0000] DEBUG: Logging in with puppeteer
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:09:17.495 +0000] DEBUG: Launching a new browser
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:09:21.675 +0000] DEBUG: Launching a new page
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:09:36.689 +0000] DEBUG: Killing new browser processes spawned
epicgames  |     user: "---"
epicgames  |     chromiumProcesses: []
epicgames  | [2024-04-26 10:09:36.689 +0000] WARN: new page did not work after 15000ms. Trying again.
epicgames  |     user: "---"
epicgames  |     attempts: 0
epicgames  |     MAX_ATTEMPTS: 5
epicgames  | [2024-04-26 10:09:51.699 +0000] DEBUG: Killing new browser processes spawned
epicgames  |     user: "---"
epicgames  |     chromiumProcesses: []
epicgames  | [2024-04-26 10:09:51.699 +0000] WARN: new page did not work after 15000ms. Trying again.
epicgames  |     user: "---"
epicgames  |     attempts: 1
epicgames  |     MAX_ATTEMPTS: 5
epicgames  | [2024-04-26 10:10:06.710 +0000] DEBUG: Killing new browser processes spawned
epicgames  |     user: "---"
epicgames  |     chromiumProcesses: []
epicgames  | [2024-04-26 10:10:06.710 +0000] WARN: new page did not work after 15000ms. Trying again.
epicgames  |     user: "---"
epicgames  |     attempts: 2
epicgames  |     MAX_ATTEMPTS: 5
epicgames  | [2024-04-26 10:10:21.720 +0000] DEBUG: Killing new browser processes spawned
epicgames  |     user: "---"
epicgames  |     chromiumProcesses: []
epicgames  | [2024-04-26 10:10:21.720 +0000] WARN: new page did not work after 15000ms. Trying again.
epicgames  |     user: "---"
epicgames  |     attempts: 3
epicgames  |     MAX_ATTEMPTS: 5
epicgames  | [2024-04-26 10:10:36.729 +0000] DEBUG: Killing new browser processes spawned
epicgames  |     user: "---"
epicgames  |     chromiumProcesses: []
epicgames  | [2024-04-26 10:10:36.729 +0000] WARN: new page did not work after 15000ms. Trying again.
epicgames  |     user: "---"
epicgames  |     attempts: 4
epicgames  |     MAX_ATTEMPTS: 5
epicgames  | [2024-04-26 10:10:51.738 +0000] DEBUG: Killing new browser processes spawned
epicgames  |     user: "---"
epicgames  |     chromiumProcesses: []
epicgames  | [2024-04-26 10:10:51.738 +0000] ERROR: If not already, consider using the Debian (:bullseye-slim) version of the image. More: https://github.com/claabs/epicgames-freegames-node#docker-configuration
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:10:51.738 +0000] WARN: Failed to lookup weekly free games
epicgames  |     user: "---"
epicgames  |     err: {
epicgames  |       "type": "Error",
epicgames  |       "message": "Could not do new page after 6 failed attempts.",
epicgames  |       "stack":
epicgames  |           Error: Could not do new page after 6 failed attempts.
epicgames  |               at retryFunction (/usr/app/src/common/puppeteer.ts:160:13)
epicgames  |               at safeNewPage (/usr/app/src/common/puppeteer.ts:175:16)
epicgames  |               at PuppetFreeGames.setupPage (/usr/app/src/puppet/base.ts:67:18)
epicgames  |               at PuppetFreeGames.request (/usr/app/src/puppet/free-games.ts:35:19)
epicgames  |               at PuppetFreeGames.getWeeklyFreeGames (/usr/app/src/puppet/free-games.ts:148:18)
epicgames  |               at PuppetFreeGames.getAllFreeGames (/usr/app/src/puppet/free-games.ts:404:27)
epicgames  |               at redeemAccount (/usr/app/src/index.ts:46:20)
epicgames  |               at async file:///usr/app/node_modules/p-queue/dist/index.js:187:36
epicgames  |     }
epicgames  | [2024-04-26 10:10:51.745 +0000] DEBUG: Getting global free games
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:10:51.745 +0000] DEBUG: Setting auth from device auth
epicgames  |     user: "---"
epicgames  |     deviceAuth: {
epicgames  |       "access_token": "---",
epicgames  |       "expires_in": 7200,
epicgames  |       "expires_at": "2024-04-26T12:09:17.550Z",
epicgames  |       "token_type": "bearer",
epicgames  |       "refresh_token": "---",
epicgames  |       "refresh_expires": 28800,
epicgames  |       "refresh_expires_at": "2024-04-26T18:09:17.550Z",
epicgames  |       "account_id": "---",
epicgames  |       "client_id": "---",
epicgames  |       "internal_client": true,
epicgames  |       "client_service": "prod-fn",
epicgames  |       "displayName": "---",
epicgames  |       "app": "prod-fn",
epicgames  |       "in_app_id": "---",
epicgames  |       "product_id": "prod-fn",
epicgames  |       "application_id": "---",
epicgames  |       "acr": "---",
epicgames  |       "auth_time": "2024-04-17T18:01:34.329Z"
epicgames  |     }
epicgames  | [2024-04-26 10:10:51.745 +0000] DEBUG: Logging in with puppeteer
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:10:51.745 +0000] DEBUG: Launching a new browser
epicgames  |     user: "---"
epicgames  | [2024-04-26 10:10:54.785 +0000] DEBUG: Launching a new page
epicgames  |     user: "---"

epicgames  | [2024-04-26 10:11:09.796 +0000] DEBUG: Killing new browser processes spawned
epicgames  |     user: "---"
epicgames  |     chromiumProcesses: []
epicgames  | [2024-04-26 10:11:09.796 +0000] WARN: new page did not work after 15000ms. Trying again.
epicgames  |     user: "---"
epicgames  |     attempts: 0
epicgames  |     MAX_ATTEMPTS: 5

Yeah, somethings up with chromium 124 in docker. As a hotfix, I rolled back to the previous image version, since the Alpine APK doesn't store its old versions of packages

ARM images are shipping x86-64 binaries now, so they won't start. If someone needs a quick fix, pull docker.io/charlocharlie/epicgames-freegames:e08fe11113382b3d07f04b8ff88d0d6c2fe601b8

Sorry about the chaos. Unfortunately I'm AFK for the next few days, so where it's at now is where it's gonna be for a bit.

No problem, thank you for your continuous development of this app and sharing it with us for free.

Found this comment and removed --single-process, which seemed to fix it. I had used it as it was recommended to prevent zombie processes. Guess it was deprecated or something...


Raspberry pi running k3os thinks it's 1970

Version (k3OS / kernel)
k3os-30178 [~]$ k3os --version
k3os version v0.9.0

Architecture
armv7l
Describe the bug
k3os-30178 [~]$ cat /k3os/system/config.yaml
ssh_authorized_keys:

k3os:
  dns_nameservers:
    - <IP_ADDRESS>
    - <IP_ADDRESS>

  ntp_servers:
    - 0.us.pool.ntp.org
    - 1.us.pool.ntp.org

  k3s_args:
    - server
    - "--cluster-init"
    - "--server https://k3s.sud0.com:6443"

k3os-30178 [~]$ cat /etc/connman/main.conf
[General]
NetworkInterfaceBlacklist=veth
FallbackNameservers=<IP_ADDRESS>,<IP_ADDRESS>
FallbackTimeservers=0.us.pool.ntp.org,1.us.pool.ntp.org
k3os-30178 [~]$ sudo connmanctl clock
  Time = 2868
  TimeUpdates = auto
  TimezoneUpdates = auto
  Timeservers = [  ]
k3os-30178 [~]$ date -u
Thu Jan  1 00:48:00 UTC 1970
k3os-30178 [~]$ k3s server --cluster-init
INFO[0000] Preparing data dir /home/rancher/.rancher/k3s/data/39a08c3192d96b9e0e5f20e12690a943b118bfe386f1d3df65a573a7a317787d
FATA[1970-01-01T00:51:44.169221723Z] server time isn't set properly: 1970-01-01 00:51:44.168967964 +0000 UTC m=+1.411071037

To Reproduce
Expected behavior
I'm expecting k3os to sync to the ntp servers I specified in config.yaml. Not having the proper time prevents k3s from starting up.
Actual behavior
ntp does not appear to be fully configured, based on the output of "connmanctl clock"
Additional context
This being a raspberry pi, rtc is not present.

I've have also seen this on arm7l+ when on wifi but not ethernet when talking to the exact same router.

I worked around this in config.yaml:
boot_cmd:
- "ln -sf /etc/init.d/swclock /etc/runlevels/boot/swclock"

Starting the software clock service only approximates a solution. The k3os.ntp_servers entries should do the trick so long as they are reachable. Is there anything in the log from connman re: ntp? e.g.:


Request new indicator

https://www.tradingview.com/script/IVjPPt6c-Open-Close-Cross-Alert-R6-by-JustUncleL/

Hi,
Sorry for the late response. I was unable to figure out how to get the indicator working.
Do you already have any code by chance?


Modify design document to reflect revised MVP

have added the footer with links as discussed..... also have removed price slider. Enclosed all the  in a container as suggested by Doug

Completed & reviewed by team


Update AWS provider and esbuild to NodeJS18 targets

AWS SDK V2 is moving to maintainance mode in 2023. The Node18 runtime has V3 baked in so it's a good time to move to that.

The build process for this appears to be blocked by the serverless version - should be fixed by this PR: https://github.com/aligent/serverless-aws-nodejs-service-template/pull/637

@tvhees,
If we're switching out the node version here should we also switch it out in:
https://github.com/aligent/serverless-aws-nodejs-service-template/blob/2d47253b6d1070de2224ab0b0ffa59609a15c493/package.json#L34
https://github.com/aligent/serverless-aws-nodejs-service-template/blob/2d47253b6d1070de2224ab0b0ffa59609a15c493/.nvmrc#L1
https://github.com/aligent/serverless-aws-nodejs-service-template/blob/2d47253b6d1070de2224ab0b0ffa59609a15c493/.github/workflows/package.yml#L17

https://github.com/aligent/serverless-docker/blob/main/Dockerfile - this may also need updating to node 18


Packaging Cirq 0.7 for Nix: serialization failures

Hi all,
#2727 is higher priority.
For https://github.com/NixOS/nixpkgs/pull/78779 I tried building Cirq tag 0.7.0, and got several test failures. Are these in the main tree or local to me? And any tips on resolving them? (note: this is after disabling the failures from #2727, which also appear here).

      The returned value might have been normalized to another type.
      """
      if not isinstance(proposed_value, self._acceptable_types):
        message = ('%.1024r has type %s, but expected one of: %s' %
                   (proposed_value, type(proposed_value), self._acceptable_types))
>       raise TypeError(message)
E       TypeError: True has type <class 'numpy.bool_'>, but expected one of: ((<class 'bool'>, <class 'numbers.Integral'>),)

/nix/store/gajvrh1azp0vh6q531b695qrcak24zmq-python3.7-protobuf-3.7.1/lib/python3.7/site-packages/google/protobuf/internal/type_checkers.py:109: TypeError

Thanks in advance!

The first issue seems to be a boring floating-point-comparison error. Replacing with np.isclose() in the test would work
If I had to guess on the second, it looks like something in the google protobuf libraries. What versions are you using and what is pinned in cirq's requirements?

Yes, I realized that about the first error. Does it appear in your CI tests?
I'm using protobuf3.8. Nix's default is 3.7, which I also tested with. Error persisted in both configs.
Cirq pinned 3.8.

@drewrisinger

pinned 3.8

Python 3.8? We don't run that on our CI yet.

I think they're talking about protobuf 3.8

The first one just looks like accuracy issue which should be fixed.
The second one, about bool, is a duplicate of #2727

For the first we will need a version of np.is_close or np.are_almost_equal that works for python dicts.

Failure 2 is resolved by #2986. Failure 1 is resolved by changing the following line: https://github.com/quantumlib/Cirq/blob/91e919ad54683747f99ccd3404d884d813fc9b5c/cirq/google/arg_func_langs_test.py#L108
to assert packed == {'arg_value': pytest.approx({'float_value': float(np.float32(sympy.pi))})}
I've also found a similar issue to Failure 1 in test_convert_to_ion_gates() when I test (cirq v0.7, v0.8) via Travis: https://travis-ci.org/github/drewrisinger/nur-packages/jobs/685881776#L6573-L6621.
I'm going to try to fix this one as well via pytest.approx.

Hi @drewrisinger is this still an issue for Cirq 0.8?

Yes, it looks like it's been resolved. Likely due to Nixpkgs updating their protobuf version.
Relevant test config:

python 3.8
cirq==0.8.2
protobuf==3.13.0
numpy==1.19.1
networkx 2.4


Expand testsuite with builder/writer tests that enumerate success documents.

This branch contains two new test executables that test builder/writer interfaces (which, are somewhat lacking..)
Commit 60efde0 fixes proper escaping of keys when outputting the value to ostream.
The builder test (BuilderTest.build_parse_datetime_01) is currently failing. I'm simply assuming this is
due to a parser bug (which I have yet to look into. Priorities..)
I'm in the process of implementing better write support (more formatting options, as well as per Value).
I'm positive this will resolve #11 as well.
Please review and consider merging.

That's odd. The writer_test is failing as well on Travis, but I don't. I've tried both c++ and clang.
Can you perhaps change the travis script to include output?
CTEST_OUTPUT_ON_FAILURE=1 make check

Added CTEST_OUTPUT_ON_FAILURE=1 in https://github.com/mayah/tinytoml/commit/0f655021c69d925b2349c856de142dd65262d5b7

Current failure is this:
/home/travis/build/mayah/tinytoml/src/builder_test.cc:22: Failure
Value of: result
  Actual: false
Expected: true
VALUE CONTENTS:
date1 = 1979-05-27T07:32:00Z
date2 = 1979-05-27T00:32:00Z
date3 = 1979-05-27T00:32:00Z
date4 = 1979-05-27T00:00:00Z
FILE CONTENTS:
date1 = 1979-05-27T07:32:00Z
date2 = 1979-05-27T07:32:00Z
date3 = 1979-05-27T07:32:00Z
date4 = 1979-05-27T00:00:00Z
[  FAILED  ] BuildTest.build_parse_datetime_01 (0 ms)

Do you have any idea what  is wrong?
Please review and consider fix for failing test. I'd like to squash before merge.

Style fixes for both builder_test and toml.h corrected (in 41dfc8c and 771499f respectively).
Added weak definitions to all build.h functions in 538f9b9. Thanks for the heads up, gave me an opportunity to read up on inline again.
If you see more thoroughly on the parsed output from the success file datetime-01.toml in the failing test, you can see that for date1, date2 and date3 has somehow parsed all hours to be 07. Therefore, I think there is a more hidden bug in that parser code that results in this test failing. Of course, adding the microseconds to date3 is correct with regards to the actual success file.
I will leave this up to you how and what to proceed with. I don't have that much time to prioritize hunting down this inconsistency with regards to tinytoml, since there are more pressing writer features I need to implement (which I need for another, pressing project). My motivation to write these tests were to have a baseline for a valid parse/output before writing more formatting features (and tests ).
One option is of course to just disable the failing builder_test all together, just so the tests all pass. But I believe that speaks a bit too loud to the broken window theory.
You have write access to my pull-request branch.

Anyway, thanks for the pull request.
I'm going to merge your PR manually (with disabling failing tests and fixing style a bit) today.

Merged manually.

Looking at the source code more, I found you didn't consider timezone (-07:00 in datetime-01.toml) when building toml::Time. So builder_test was failing.
I've fixed it, but when writing toml::Value back, timezone information is currently lost...
In toml version 0.2.0, timezone must be always GMT, so it wasn't a problem :(

Oh my. I completely misunderstood the datetime timezone semantic. Ignore my previous comments regarding datetime.
Thanks for the merge and work you've done on this library.

After this PR I have the following error compiling with MinGW on Windows 10:
toml.h: In member function 'void toml::Value::write(std::ostream*, const string&, int) const':
toml.h:1298:25: error: 'gmtime_r' was not declared in this scope
         gmtime_r(&tt, &t);
                         ^

gcc version 4.8.2 (GCC)

@RobertBColton Could you file a new issue?
I heard Visual Studio does not have gmtime_r but mingw does.
https://github.com/mayah/tinytoml/blob/master/include/toml/toml.h#L228
Is that changed somewhere? (Or our code is wrong?)

Please see #25 I've tried to find as much info as I can. I'll keep looking but it mainly looks to be a MinGW issue at this point, MSVC works just fine. I should mention I am not really building with MinGW, I just used it because it was quick to open from my desktop and I wanted to see if I could close the formatting issue for you.

Also please check #25 again if you already looked at it today, I found a simple work around that fixes it that I'll think you'll want.


rune/spec.go: set the default network mode as the host mode

Lots of applications need network when they start. However, using rune to
run application bundles doesn't have a network if set a network namespace.
So we generate the default config.json to support the host network mode with
the rune spec command.
Fixes #279
Signed-off-by: Yilin Li<EMAIL_ADDRESS>

No no no no no. I though you will change the result of rune spec. This is not right way to do this. A private network NS is required in some cases. The determination made by using network ns or not is DECIDED by the user and caller, not OCI runtime engine.

OK，I will close it and revise it in README.md.


GUI: Streamline Checker page

Now that Check Files and Results are on the same Checker page in MediaConch, it's a little confusing to select policy and display and verbosity at the very top of the page. When files are already loaded into MediaConch, the policy must be changed via the dropdown menu below Results and nothing will change if a user tries to select a policy from the top.
I can assign this to me to sort out for a non-confusing user experience.

Hmmm, the checker page in the GUI part is a little bit "complicated".
You have to manage Qt stuff, the WebEngine part and WebKit part...
Most of the code you will have to look for are:

Source/Resource/html/checker.html
Source/Resource/html/js/checker.js
Source/Resource/html/css/main.css or Source/Resource/html/css/main-gui.css if only for GUI
Source/GUI/Qt/checkerwindow.cpp (some html code is generated in it)

Or do it for MCO, we should have the same UI and I will adapt to the GUI.

OK, thanks. It's good to know the technical constraints.


Remove pipdeptree

Fixes

Fixes #[issue number] by @[issue author]
Description

Testing Instructions

Checklist

[x] My pull request has a descriptive title (not a vague title like Update index.md).
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[ ] My commit messages follow best practices.
[ ] My code follows the established code style of the repository.
[ ] I added or updated tests for the changes I made (if applicable).
[ ] I added or updated documentation (if applicable).
[ ] I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

#516

Sorry @GowthamChowdary, that looks like it added a new pipfile to the top level of the repository. If you run just install using the just command runner as described in the projects README it will traverse the three Python projects in this repository, and run pipenv install in each of them, thereby updating the lock files. Hope this helps. Let me know if you need any extra instructions.

got it, thank you @sarayourfriend . i'm doing it as i type this, will update as soon as it's done.


Random read benchmarks

This PR adds randomized read benchmarks and adapts the sequential reads to enable comparibility between the benchmarks.
For each benchmarks we measure a for loop for sequential and random accesses to the data. Unfortunately, I was not able to do it without a for loop. Please, if you have an idea for this, feel free to comment it.
I'd also be happy about some feedback :)
Current Benchmark results, once with and once without in memory reads

Thank you for your Feedback @tjjordan !!!
I tried to fix all your requests. Unfortunately, I was not able to make all Assertions work. Could you have a look at this TODO and the TODO with the vector generation? I'd really apprecite it!


Do not show a loading spinner if thread has root post loaded

Summary
If the thread screen already had the root post we still waited until the request finished, that slowed viewing the thread even if there was no need for it

@sudheerDev @hmhealey have a look


Execute the docker-compose file and throw an error

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[x] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?
docker-compose up

Pulling oap (skywalking/oap:)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]y
Pulling oap (skywalking/oap:)...
ERROR: pull access denied for skywalking/oap, repository does not exist or may require 'docker login'

I've already executed docker login

Bug

Which version of SkyWalking, OS and JRE?
Now the master branch

Which company or project?

What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
Direct execution: docker-compose up

Requirement or improvement

Please describe about your requirements or improvement suggestions.
Rebuild the skywalking/oap image and wonder if the following skywalking/ UI image will have the same problem.

Image has moved to ASF. You can't get the skywalking org.

@hanahmily do we have fixed this in master?

Should we change docker-compose?

Yes. If master branch hasn't changed, could you send a pull request to fix this?

Thank you. Let me try.
/assign @xichengliudui

the docker folder in main project is a convenient  way to test source code in docker. So user should issue make build to build image before running docker-compose up

@hanahmily Which directory should I go to?

docker/

[root@localhost docker]# clear
[root@localhost docker]# make build
cd /home/liudui/incubator-skywalking/docker/oap; \
cp ../../dist/apache-skywalking-apm-incubating-bin.tar.gz ./ ; \
docker build -t skywalking/oap:latest .
cp: cannot stat ‘../../dist/apache-skywalking-apm-incubating-bin.tar.gz’: No such file or directory
Sending build context to Docker daemon   7.68kB
Step 1/9 : FROM openjdk:8u181-jdk-stretch
 ---> 5f8e49b0a018
Step 2/9 : ENV DIST_NAME=apache-skywalking-apm-incubating-bin     JAVA_OPTS=" -Xms256M "
 ---> Using cache
 ---> 358f3a037515
Step 3/9 : COPY "$DIST_NAME.tar.gz" /
COPY failed: stat /var/lib/docker/tmp/docker-builder746677751/apache-skywalking-apm-incubating-bin.tar.gz: no such file or directory
make: *** [oap] Error 1

Can you provide a link to this file? Thank you very much

In the root folder of the project, issue ./mvnw install to generate /dist/apache-skywalking-apm-incubating-bin.tar.gz.
Or you can use official docker image apache/skywalking-oap-server instead of building project.

and apache/skywalking-ui

Why do docker-compose still not solved, so it is very unfriendly for beginners!


Add the Composition & Parameters resource to be exempted from authentication

Description of feature

[ ] Update the authentication interceptor to exempt the composition resource from authentication.

Acceptance Criteria

[ ] The composition resource API should be able to accept requests without authentication.

Is it always expemted, or sometimes?
I could imagine a use-case where the app definition files themselves are confidential

@rehammuzzamil can we update this functionality to only exclude GET requests?

Is it always exempted, or sometimes?
I could imagine a use-case where the app definition files themselves are confidential

This means in such cases the login screen would have to come before the configuration screen on the app.  I don't think we have thought about this use case from a UX and configuration perspective.
Questions

How do we tell the APK which use case to configure for? i.e show the configuration page first or the login page first.

Is it always expemted, or sometimes?
I could imagine a use-case where the app definition files themselves are confidential

@dubdabasoduba thats true. Composition resource contains client information as well. For e.g. IPS (International Patient Summary) is a Composition of highly sensitive Patient data. The least should be that we do not expose Compositions which have 'subject'. The rest we can assess and discuss. An alternate could be that we expose Composition with specific pattern only

@rehammuzzamil can we update this functionality to only exclude GET requests?

Yes @dubdabasoduba, we can restrict to only exclude GET endpoint.

Is it always exempted, or sometimes?
I could imagine a use-case where the app definition files themselves are confidential

This means in such cases the login screen would have to come before the configuration screen on the app. I don't think we have thought about this use case from a UX and configuration perspective.
Questions
* How do we tell the APK which use case to configure for? i.e show the configuration page first or the login page first.

Is it always expemted, or sometimes?
I could imagine a use-case where the app definition files themselves are confidential

@dubdabasoduba thats true. Composition resource contains client information as well. For e.g. IPS (International Patient Summary) is a Composition of highly sensitive Patient data. The least should be that we do not expose Compositions which have 'subject'. The rest we can assess and discuss. An alternate could be that we expose Composition with specific pattern only

Agreed, this approach makes sense to me @dubdabasoduba

@rehammuzzamil can we update this functionality to only exclude GET requests?

Yes @dubdabasoduba, we can restrict to only exclude GET endpoint.

Awesome! let's implement this too.

Is it always expemted, or sometimes?
I could imagine a use-case where the app definition files themselves are confidential

@dubdabasoduba thats true. Composition resource contains client information as well. For e.g. IPS (International Patient Summary) is a Composition of highly sensitive Patient data. The least should be that we do not expose Compositions which have 'subject'. The rest we can assess and discuss. An alternate could be that we expose Composition with specific pattern only

@rehammuzzamil please do some R&D on how much work this filtering would take.

Is it always expemted, or sometimes?
I could imagine a use-case where the app definition files themselves are confidential

@dubdabasoduba thats true. Composition resource contains client information as well. For e.g. IPS (International Patient Summary) is a Composition of highly sensitive Patient data. The least should be that we do not expose Compositions which have 'subject'. The rest we can assess and discuss. An alternate could be that we expose Composition with specific pattern only

@rehammuzzamil please do some R&D on how much work this filtering would take.

@dubdabasoduba  @maimoonak I believe we need to add a filter to get only the specific Composition resource. Maybe we can add a particular subject value to the Composition Resource of configurability to access it without logging in.
Based on high-level findings, we can do filtering at the resource level to allow only GET URL along with that request param value.

Is it always expemted, or sometimes?
I could imagine a use-case where the app definition files themselves are confidential

@dubdabasoduba thats true. Composition resource contains client information as well. For e.g. IPS (International Patient Summary) is a Composition of highly sensitive Patient data. The least should be that we do not expose Compositions which have 'subject'. The rest we can assess and discuss. An alternate could be that we expose Composition with specific pattern only

@rehammuzzamil please do some R&D on how much work this filtering would take.

@dubdabasoduba @maimoonak I believe we need to add a filter to get only the specific Composition resource. Maybe we can add a particular subject value to the Composition Resource of configurability to access it without logging in.
Based on high-level findings, we can do filtering at the resource level to allow only GET URL along with that request param value.

@rehammuzzamil Does this mean if I had something like "subject":"configurability" then we filter and get only the compositions with this as the subject?

Is it always expemted, or sometimes?
I could imagine a use-case where the app definition files themselves are confidential

@dubdabasoduba thats true. Composition resource contains client information as well. For e.g. IPS (International Patient Summary) is a Composition of highly sensitive Patient data. The least should be that we do not expose Compositions which have 'subject'. The rest we can assess and discuss. An alternate could be that we expose Composition with specific pattern only

@rehammuzzamil please do some R&D on how much work this filtering would take.

@dubdabasoduba @maimoonak I believe we need to add a filter to get only the specific Composition resource. Maybe we can add a particular subject value to the Composition Resource of configurability to access it without logging in.
Based on high-level findings, we can do filtering at the resource level to allow only GET URL along with that request param value.

@rehammuzzamil Does this mean if I had something like "subject":"configurability" then we filter and get only the compositions with this as the subject?

Yes, this is my understanding.

@rehammuzzamil I think we can start with implementing the GET-only approach then we can discuss the filtering. Please feel free to work on this.

@rehammuzzamil I think we can start with implementing the GET-only approach then we can discuss the filtering. Please feel free to work on this.

Okay, noted @dubdabasoduba  . Thanks!

@rehammuzzamil Based on our discussion earlier we need to also exempt the Parameters. This will hold some configs required before login too.

@rehammuzzamil Based on our discussion earlier we need to also exempt the Parameters. This will hold some configs required before login too.

Sure, will start work on this piece.

this looks closed based on the merged prs


Add contributors command

This little change makes me smile (●'◡'●)
Added the following:

Contributors command

[x] Not tested, couldn't test it because webhooks is giving me some issues.

I will test it tomorrow 😉

Thank you!

I join @marc2332 about the API limitation even if it's not a very big problem.
But I don't think this feature is really important. We already have a GitHub link in the help command and the contributors are in the README.
Especially since I don't think people are very interested in who made the bot. Or at least, if they're really interested, they're doing some research.


RE-1479 Revert hostname to previous hostname

This commit updates the running instance's hostname to that of the
instance that created the snapshot. This ensures that the hostname
matches what exists in the ansible inventory, etc.
Issue: RE-1479

I thought that we had some stuff in the thaw implementation which renamed all the old references in the inventory to the new name? Would that not be a better approach?

@odyssey4me been digging into this w/ @leader716 and I think the simplest thing to do here is re-run /usr/local/bin/openstack-host-hostfile-setup.sh during the thaw. I'll update this PR to do that.

Closing in favour of https://github.com/rcbops/rpc-openstack/pull/2986.


Image flickering placeholder on ever rerender

Bug Report
In our application, we are using an Image. It is flickering the placeholder image everytime our component rerenders.
Description

https://github.com/cerner/terra-core/blob/master/packages/terra-image/src/Image.jsx#L68
It is showing the placeholder, we believe, because it fails on this line. Using === on two nodes that appear the same fails.
Steps to Reproduce
Using demographics manager with our application, if we have a profile image and the component containing the demographics manager rerenders, we see the placeholder image once again.
Expected Behavior
The placeholder image would not be reshown on rerender once the image has loaded
Possible Solution

I believe you'll have to change the === check on the placeholder image.
Environment

Component Name and Version: terra-image 2.17.0
Browser Name and Version: Chrome Version 67.0.3396.99
Node/npm Version: [e.g. Node 8/npm 5] Node: 8.4.0, npm 5.6.0
Webpack Version: ^4.4.1
Operating System and version (desktop or mobile):  Desktop--Mac

We're open to PR for a fix for this.


chore: Parallelise feature flags and user info API

Description
Changes to make feature flags and user info API in parallel to improve first page load time.
Fixes #11539
Type of change

Enhancement to improve first page load time.

How Has This Been Tested?

Manual

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

Test coverage results :test_tube:
:green_circle: Total coverage has increased
// Code coverage diff between base branch:release and head branch: fix/11539-parallel-user-featureflag-apis

Status
File
% Stmts
% Branch
% Funcs
% Lines

:green_circle:
total
56.18 (0)
37.68 (0)
35.66 (0)
56.43 (0.01)

:green_circle:
app/client/src/components/designSystems/appsmith/header/DeployLinkButton.tsx
79.41 (0.62)
84.62 (0)
16.67 (0)
78.13 (0.71)

:green_circle:
app/client/src/components/editorComponents/ActionCreator/index.tsx
38.73 (0.36)
7.08 (0)
43.48 (0)
38.6 (0.36)

:green_circle:
app/client/src/pages/Editor/EditorAppName/NavigationMenuData.ts
67.69 (0.5)
41.67 (0)
6.25 (0)
67.19 (0.52)

:green_circle:
app/client/src/pages/Editor/gitSync/QuickGitActions/index.tsx
66.36 (0.65)
44 (0)
34.78 (0)
68.32 (0.64)

:green_circle:
app/client/src/sagas/PostEvaluationSagas.ts
16.89 (0.34)
0 (0)
0 (0)
20.66 (0.49)

:red_circle:
app/client/src/sagas/ReflowSagas.ts
24.24 (-5.76)
7.14 (-1.19)
33.33 (0)
28.57 (-6.05)

:red_circle:
app/client/src/sagas/index.tsx
91.8 (-1.18)
57.14 (-2.86)
100 (0)
94.74 (0.19)

:green_circle:
app/client/src/sagas/userSagas.tsx
16.67 (0.5)
1.32 (0.09)
5 (0)
20.11 (0.72)

:red_circle:
app/client/src/selectors/appCollabSelectors.tsx
68.42 (-3.01)
0 (0)
42.86 (0)
66.67 (0)

:green_circle:
app/client/src/selectors/usersSelectors.tsx
77.78 (4.45)
100 (0)
33.33 (13.33)
72.73 (6.06)

:red_circle:
app/client/src/utils/autocomplete/dataTreeTypeDefCreator.ts
69.62 (-0.75)
50 (0)
100 (0)
65.67 (-1)

:x:
app/client/src/utils/featureFlags.ts
100
100
100
100

/ok-to-test sha=7ad5464

/ok-to-test sha=60a9624

/ok-to-test sha=b1227a7

/ok-to-test sha=e5b29c7

@arunvjn tests suit need to run again I suppose, Can you take this ahead, and close it if everything is done


support for .gitignore in the worktree

Currently the .gitignore is not implemented

duplicate https://github.com/src-d/go-git/issues/307


[Question] Tacotron or Tacotron GST output spectrograms to condition Wavenet

Hi,
Has anyone tried to use Wavenet instead of Griffin-Lim waveform reconstruction during inference in Tacotron or Tacotron GST models? I'm trying to figure out if there is an easy way to do this or whether one would need to implement a proper decoder in Wavenet instead of the default 'FakeDecoder'.
Outline of my recipe:

Train a Tacotron or Tacotron GST model to predict spectrograms from text or from phoneme sequences
Train a separate Wavenet model conditioned on spectrograms
(This is what I'm trying to figure out) - Feed spectrogram outputs of (1) into Wavenet model from (2) to synthesize speech

It wasn't clear to me if the current implementation supports (3) by simply changing config files so any guidance or ideas would be highly appreciated.
Thanks!
Oytun

So long as the parameters you are using to generate the spectrograms match in 1 and 2, 3 should work.


Allow MetricsControl to aggregate on a column with an expression

In the new MetricsControl, selecting a column with an expression would fail because it's just using the column name which doesn't exist in the underlying table (only in our metadata). This would add the expression (if there is one) inside the aggregation.
@john-bodley @GabeLoins @mistercrunch

@michellethomas do we need to do something like this for druid as well?

I'm not totally sure if you can use a dimension spec (column expression) within a postagg in druid. When I looked through the druid metrics code I didn't see anywhere we were using DruidColumn expression or dimension_spec properties, so it doesn't look like we are doing this now. But I'm not totally sure.
Maybe John or Max know?

Codecov Report

Merging #5021 into master will decrease coverage by 0.03%.
The diff coverage is 14.28%.

@@            Coverage Diff            @@
##           master   #5021      +/-   ##
=========================================
- Coverage   77.34%   77.3%   -0.04%
=========================================
  Files          44      44
  Lines        8665    8669       +4
=========================================
  Hits         6702    6702
- Misses       1963    1967       +4

Impacted Files
Coverage Δ

superset/connectors/sqla/models.py
75.8% <14.28%> (-0.62%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bf53da...4ea337c. Read the comment docs.

I'm not 100% but I'm pretty sure Druid postaggs have to be made out of metrics.
Your PR LGTM, I'd be great to have a unit test that was failing before and succeeding now. For example we could have a new expression in the "births" dataset that would be called first_letter as substr(name, 1, 1) and do a count distinct of that or something.

@mistercrunch I didn't see a lot of tests for this section of the code. Did you have thoughts on where to add tests for this (viz_tests maybe?)?

All examples get executed as integration test, so if you add a new example chart that cover this use case it should fail before your PR, succeed after

@mistercrunch added tests, thanks for the suggestion!

LGTM

Note I believe this only supports one level of indirection, i.e., I'm not certain whether one can create another dimension which leverages num_california in your example.

@mistercrunch what are your thoughts about having metrics reference other metrics, or should metric expressions be valid SQL?

Valid SQL was always the assumption, for simplicity and clarity. MetricsControl kind of changes the model a bit though as one might expect to see computed dimensions in there. I don't think people would expect to see MetricsControl-generated metrics inside the dropdown though.


Begin moving web server to a separate crate.

This doesn't yet introduce a working Cargo.toml for 'mentatweb', but it
does allow RLS to build correctly without errors, and it reduces the
core library's dependency space, which is more important in the short
term.

Looks good to me


dev branch does not build in VS2017

https://github.com/OrchardCMS/Orchard/blob/0cadd5a916f080e0b1b6a9fa8f37cc51e94e1d3a/src/Orchard.Web/Modules/Orchard.Email/Services/SmtpMessageChannel.cs#L131
Visual Studio 2017 doesn't support c#8 features out of the box.
By cloning the dev branch, vs2017 is unable to build the solution.
I think that the line I linked is the only place where we have those right now. It's work from @aaronamm , if I am not mistaken.

Feel free to remove these new c# features. Unless @BenedekFarkas added more in some razor files. I know it was a discussion we had.

in #8458 we fixed this to allow the solution to build in VS 2017


R daggr turns numeric cols to character, NAs to NULL

unnecessary type changes should be avoided.

@nhanitvn please review & close as appropriate.


all: add KernelModule cfg to show DLKM coverage

PC returned for dynamic loaded module (DLKM) is not
parsed in coverage page.
So the commit is to use DLKM modules' load address
to restore the PC and show coverage data of DLKM.
As the load address is written in cfg file, so kaslr
needs to be disabled.
And for linux target, load address is getting from
/proc/modules during instance setup.
For either manual or auto address setting case,
name and path are needed in config kernel_modules, where
name is module name on target.
path is module unstripped object path on host.
addr is decimal value of module load address on target.
Example of config:
"kernel_modules": [
{
"name": "nf_nat",
"path": "/usr/src/linux-source/net/netfilter/nf_nat.ko",
"addr":<PHONE_NUMBER>2637911040
}
]

Before sending a pull request, please review Contribution Guidelines:
https://github.com/google/syzkaller/blob/master/docs/contributing.md

https://github.com/google/syzkaller/pull/1305 is too old, so update with the new design.

Oops, I accidentally pressed Merge. I think this needs some improvements. I will prepare a revert. Sorry.

Revert #2477