initial commit of matrix normal base API , regression, and MN-RSA

Massive squashed commit of matrix-normal functionality. Includes the base API, matrix-normal RSA, matrix-normal regression, and an example. Have a separate commit ready to go for matrix-normal SRM but this is already huge so let's do this first.
Paging @narayanan2004 for double checking I didn't mess up with what I picked out of the private repo, @lcnature for reviewing RSA.

@narayanan2004, @lcnature, are you in touch with @mshvartsman about this PR?

@mihaic Sorry, not yet. I will have a look at it after Friday. @mshvartsman I just had a quick glance a the console output of the automatic testing system, it seems that there are quite a few formatting complaint in brainiak/utils/utils.py and a few in your brainiak/matnormal module. You can probably see them if you just run the run-checks.sh

Github rendering of math in notebooks seems pretty bad unfortunately -- it looks fine locally.

Sorry for so many vacuous commits as I get this ship-shape: for some reason run-checks.sh gives different results on my machine vs on the CI box.

Current linux failures look to be due to tensorflow being incompatible with old libc (e.g. https://github.com/tensorflow/tensorflow/issues/177), I'm guessing because the test server runs Centos/Springdale 6. There are various hackarounds possible (easiest might be installing TF from conda) but I don't know how to make any of them work with the CI setup. Any advice?

@mshvartsman, you are right. TensorFlow does appear to require glibc >= 2.17 (https://github.com/tensorflow/tensorflow/issues/53), although this requirement is not explicitly documented, as far as I can tell.
We do want to be able to depend on TensorFlow, so I am looking into a workaround for our Jenkins server.

I added PR #357 as a workaround for the TensorFlow issue.

@lcnature As mentioned on slack, the garbled math rendering is an issue with github's jupyter notebook renderer -- it looks fine locally. I don't know of a workaround -- @mihaic do you?

After PR #357 is merged, I will modify the Jenkins configuration to pass the flag. The Jenkins configuration is not part of this repo. Travis is not affected.
As for the GitHub Latex rendering, I am not aware of a workaround, but I don't think it matters. Users can see the rendering locally, as you mentioned; additionally, we should use our HTML docs for displaying examples (see issue #268).

@mshvartsman, please add the commit from PR #357 to your branch, so we can use the Jenkins workaround.

I didn't mean to be obscure. Merging master will add the commit.

Actually, I had modified the Jenkins config. Investigating...

There are now two build issues: the master build is failing (unrelated to this PR, maybe related to PR #357) and this PR build is still failing with glibc errors. Suggestions are welcome. I will investigate more later today or tomorrow.

Am happy to try to troubleshoot but I think I need credentials for Jenkins. Also: should we roll back #357 just in case so master build stops failing?

@mshvartsman, thanks for the offer to help. I suggest you make PRs in your own fork which should trigger Travis builds without any special authorization. See mine: https://github.com/mihaic/brainiak/pull/7

I think I found the culprit for the master breakage: Pip! Version 10, recently released, handles the installation of Pybind11 differently.
Updating Pip is the only difference between this passing commit:
https://travis-ci.org/mihaic/brainiak/builds/367886270
And this failing commit:
https://travis-ci.org/mihaic/brainiak/builds/367890418
I am looking into fixing this now. Then I will go back to adding support for TensorFlow on Jenkins.

New versions are fun! I'm experiencing yet another bug on my machine:
https://github.com/pypa/pip/issues/5224

I propose we do not use Pip 10 for the moment. See PR #358.

The master build issue is fixed so I am back to the TensorFlow issue. I am facing this Conda vs Pip issue:
https://github.com/conda/conda/issues/5969

I am struggling to make progress. CentOS 6, TensorFlow, Conda, Pip, and Environment Modules do not play well together. I will reach out to the server admins to see if they are willing to upgrade to CentOS 7. If not, we should consider abandoning Jenkins for the moment and relying only on Travis.

@mshvartsman sounds great!

The usage of TensorFlow (required by this PR) is blocked by PR #359.

If you merge master, the Linux test on Jenkins should pass.

@mshvartsman, I think @szorowi1 can provide more detailed feedback. I can imagine you are short on time these days. When do you anticipate you will be able to address feedback?

I think looking like mid-late July before I can seriously work on this: still working on basic life stuff like having my furniture delivered, and have travel coming up in a week or so. But I will commit to sitting down with @szorowi1 in person when he's here at the UW neuro summer school to work through this.

Excellent!

@mshvartsman, @szorowi1, how's the summer school going? Are you getting the chance to work on this PR?

Good to hear from you, @mshvartsman. We can keep this PR open now that we know what you're up to. :)

It's in process :). I've collected 1/2 of the necessary signatures, hopefully soon.

Thanks!

All signed, working on it now. Will take a little while but please feel free to bug me if there's no forward progress

@mshvartsman Just curious if there is any update on this PR:)

@szorowi1 @lcnature addressed a bunch of your comments and cleaned things up, please feel free to resolve anything that is resolved to your satisfaction / unresolve anything that I resolved prematurely. Major things still outstanding:

Logging fixes using new TF logging functionality.
Verifying that with ripping out all the define_scope stuff I haven't introduced TF graph inflation anywhere.
Investigating what to do about materialized cov and prec. Basically I use them in some of the linalg tricks for marginal/conditional solves and logdets in addition to testing, but materializing those would mean you need to hold a FCMA-style matrix in memory if you wanted to run whole-brain MNRSA.

In addition to these changes, some updates:

Logging update: TF 1.0 API logging won't help much here: without eager mode I can log the tensor itself, not the value (which is useless) unless I have a session I can use to eval. This is fine for the logging calls inside the estimator class (which I've modified) but not in helper classes (all the matrix-normal linear algebra tricks). I don't think it's worth passing the session around just for better logging (though I'm happy to do that if you think that's a better option). I propose that for the current PR I keep old-school tf.Print logging (still gated behind log levels as it is now), and if the tool is starting to get usage there should be an eventual migration to TF 2.0  eager mode which would obviate some of these issues.
Scope stuff: I think we're good here also, and longer term this also completely goes away with TF 2.0.
Materialized cov and prec: I think again this could be a new issue and not part of the current PR, we ran covs up to 10K+ voxels for the paper which should be a good start for now.

Based on this, I'd like to ask for a re-review and see if we can get this finally merged :).

Thanks @mshvartsman ! I still owe the review to Hejia's SRM PR, so I may not be able to review this very soon. Can @szorowi1 take a look perhaps?

Hi folks -- anyone get a chance to review this? Happy to get together on VC with @lcnature or @szorowi1 or whoever else to get this finally closed out.

Ok, @lcnature's comments all fixed / incorporated except for the discussion above. So @mihaic if you're happy with everything except the TF2 support in https://github.com/mshvartsman/brainiak/tree/mnorm-eager, I can merge that into the PR and that should be the last piece.

Just pushed the TF2.0 fixes, everything now works in eager mode and the example notebook even demonstrates how to set it up. I also removed the calls to tf.compat.v1.print since we can do normal logging now. None of the math / modeling changed and the APIs are basically the same, so the main thing to look at would be the new way that the fit methods look internally, plus the new utils that pack/unpack things for Scipy.

Excellent news, @mshvartsman! Thank you for the effort. @lcnature, what do you think?

@mihaic do you think you could take a look at the tooling changes? It seems like @lcnature has been mostly happy with the math/modeling side but it'd be nice to have an additional critical software-focused eye on make_val_and_grad, and the parameter packing/unpacking functions in matnormal/utils.py before we unleash them upon unsuspecting neuroscientists :). Also okay if you want to wait until I finish up the final docstrings and make Travis happy.

I'm stuck on Travis: I need TF 2.3 or greater for tensorflow_probability. If I don't put in any version requirement (previous commit), it defaults to 1.14 and fails when I try to import tensorflow_probability. If I try to force it to 2.3 (which is what tensorflow_probability says it needs), it fails because no installation candidate. I remember we had an issue like this before when the CI box ran an old Centos, is this a similar problem? run-checks.sh and all matnormal tests pass on my local machine, and I think all docstrings are cleaned up as well.

I get the same error on my Ubuntu machine, Could not find a version that satisfies the requirement tensorflow>=2.3. Looking into it.

Upgrading Pip solves the issue. Now to figure out how to use Pip>=10 to build BrainIAK...

BrainIAK builds fine for me with pip 20 and --no-use-pep517. That said, I run into another issue: SSSRM uses pymanopt, which isn't updated for TF2. If TF isn't installed, pymanopt just ignores it with a try/except ImportError. But with TF2 installed, it fails because tf.Session no longer exists. Per https://github.com/pymanopt/pymanopt/issues/86 pymanopt isn't fixing this any time soon. Is there a workaround?

Ok, I made a (really hacky) workaround where after we import pymanopt but before we instantiate our first Problem we tell pymanopt to pretend TF isn't installed.

I think the Pymanopt fix is fine. Based on my own experience with convincing Pymanopt to adopt changes and make subsequent releases, we should not wait for them.
Now after PR #479, Travis should find TensorFlow 2.3.

Thank you for your comments @lcnature, I have verified that in all places where we do logdet for cholesky the cholesky is defined in a unique way.

@mshvartsman @mihaic Other than the last comment above, I think the PR is good to go!

The conda build fix doesn't look too onerous, we might as well do it. I think I got all the spots where pymanopt is special-cased and did the same for TFP? The one place I'm not sure about is jenkins-server-customization: does that still need to be there, or does that go away with the hackaround I've got in SSSRM?
Latest TF in conda is 2.2 though and TFP requires 2.3 so we can either do the pypi workaround or just wait until TF2.3 makes it into the conda repo, up to you.

@mihaic @mshvartsman Looks good to me too! I am happy to approve! Thank you both for contributing and managing!

@lcnature, thanks a lot for helping with this monumental PR!
@mshvartsman, since you are up fixing the Conda build, could you please try TensorFlow from PyPI as well? Please ignore jenkins-server-customization; we decided to stop testing on Jenkins recently.

Tried it, not sure what happened with the build there. Looks like TFP successfully installed without complaining about missing TF2.3 but then it couldn't find all the TF libs and froze. Any ideas?

The good news is that there is a TensorFlow 2.3 package in Conda (tensorflow-base). The bad news is that it is not compiled for macOS, only Linux.

Does that change our plans for merging?

 Still planning to merge for the upcoming release and still looking for the best way to build the Conda package.

@mshvartsman, I have a new proposal in light of the difficulty in packaging TensorFlow 2.3 for macOS with Conda. What if I make a new brainiak-matnormal package? It would be a pure Python package, so we will be able to publish a binary wheel for it on PyPI. That would be easily installable using either Pip or Conda, with no prerequirement from the operating system.
I also thought about using Setuptools extras_require to make matnormal an optional part of the BrainIAK package, but that does not work in Conda as far as I can tell.

@mshvartsman, another solution I see is to use extras_require and do nothing in Conda. Instead, we would instruct Conda users to install the TensorFlow 2.3 separately. The instructions would be:
No TensorFlow, matnormal cannot be used:

pip install brainiak
or
conda install brainiak

TensorFlow installed, matnormal can be used:

pip install brainiak[matnormal]
or
conda install brainiak && pip install -r requirements-matnormal.txt

What do you think?

@mshvartsman, to make some progress, I will implement the extras_require proposal. We can change it later.

@mshvartsman, thank you very much for your patience! I hope the optional dependencies approach will work well.
Thank you for your help, @narayanan2004, @lcnature, and @szorowi1.


fix: removed easter egg surprise from login.js

fix: removed Easter egg surprise from Login.js

@eldadfux @christyjacob4 hey can you review this?

Hey @vj-codes  👋
Thanks a lot for your contribution! Please head over to our discord and share the PR link with us to claim your swags 🎁


Move procedural code into classes

[ ] JSON validation
[ ] PDF generation
[ ] Email sending
[ ] JSON error generation & return

All outta time—this is no longer v1.0, since it works as-is, if horribly.


[Machine Learning] Multivariate time series using Auto ARIMA

Proposal Submission
Proposed title of article
[Machine Learning] Multivariate time series using Auto ARIMA
Proposed article introduction
It is a series of observations taken at specified times basically at equal intervals. It is used to predict future values based on past observed values. The components you might observe in the time-series analysis are Trend, Seasonal, Irregular, and Cyclicity.
A Multivariate time series has more than one time-dependent variable. Each variable depends not only on its past values but also has some dependency on other variables. This dependency is used for forecasting future values.
Forecasting, making predictions about the future, plays a key role in the decision-making process of any company that wants to maintain a successful business. This is due to the fact that success tomorrow is determined by the decisions made today, which are based on forecasts.
An autoregressive integrated moving average, or ARIMA, is a statistical analysis model that uses time-series data to better understand the data set or predict future trends. A statistical model is autoregressive if it predicts future values based on past values. We will be using the auto_arima function in Python, which automatically discovers the optimal order for an ARIMA model. The function will automatically determine the parameters p , d , and q of the ARIMA model. The auto. arima() function uses a combination of unit root tests, minimization of the AIC and MLE to obtain an ARIMA model.
Key takeaways

Plotting Autocorrelation and Partial Autocorrelation graphs.
Difference between Vector Auto Regressive Moving Average models (VARMA) and autoregressive integrated moving average (ARIMA).
Importing auto_arima function . It automatically determine the parameters p , d , and q of the ARIMA model.
SLEX Principal Components Analysis and Differencing.
Time series plots and predictions.

Article quality
In this tutorial we will build a time seris model that uses Autocorrelation and Partial Autocorrelation plots to better understand the data set and predict future trends. The time series model will predict Forex exchange rates for Euro and US dollar. We will use differencing technique to remove non-stationarity in the data set. We will also discuss how Auto ARIMA works and how it automatically determines the parameters p , d , and q.  Finally we will plot line charts using Plotly Express to visualize the forecast values.
References
Please list links to any published content/research that you intend to use to support/guide this article.
Conclusion
Finally, remove the Pre-Submission advice section and all our blockquoted notes as you fill in the form before you submit.
We look forwarding to reviewing your topic suggestion.
Templates to use as guides

How To Guide Template
Software Review Template
Tutorial Template

Sounds like a helpful topic - let's please be sure it adds value beyond what is in any official docs and/or what is covered in other blog sites. (the articles should go beyond a basic explanation - and it is always best to reference any EngEd article and build upon it).
Please be attentive to grammar/readability and make sure that you put your article through a thorough editing review prior to submitting it for final approval. (There are some great free tools that we reference in EngEd resources.)
ANY ARTICLE SUBMITTED WITH GLARING ERRORS WILL BE IMMEDIATELY CLOSED.
Please be sure to double-check that it does not overlap with any existing EngEd articles, articles on other blog sites, or any incoming EngEd topic suggestions (if you haven't already) to avoid any potential article closure, please reference any relevant EngEd articles in yours. - Approved @jamesomina99


Problem with the dataset MP-20

I found another issue... Maybe this is also related to changes in pytorch?
I think I managed to train the CDVAE with the datasets Perov-5 and Carbon-24, but when I try to train it with the dataset MP-20, I get the following error:
Epoch 46:   0%| | 0/212 [00:00<?, ?it/s, loss=2.08e+28, v_num=0sev, val_loss=11.Error executing job with overrides: ['data=mp_20', 'expname=mp_20']
Traceback (most recent call last):
File "cdvae/run.py", line 166, in main
run(cfg)
File "cdvae/run.py", line 154, in run
trainer.fit(model=model, datamodule=datamodule)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit
self._run(model)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 758, in _run
self.dispatch()
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 799, in dispatch
self.accelerator.start_training(self)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
self.training_type_plugin.start_training(trainer)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
self._results = trainer.run_stage()
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in run_stage
return self.run_train()
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 871, in run_train
self.train_loop.run_training_epoch()
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 499, in run_training_epoch
batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 738, in run_training_batch
self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 434, in optimizer_step
model_ref.optimizer_step(
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 329, in optimizer_step
self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 336, in run_optimizer_step
self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 193, in optimizer_step
optimizer.step(closure=lambda_closure, **kwargs)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/torch/optim/optimizer.py", line 89, in wrapper
return func(*args, **kwargs)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/torch/optim/adam.py", line 66, in step
loss = closure()
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 732, in train_step_and_backward_closure
result = self.training_step_and_backward(
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 823, in training_step_and_backward
result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 290, in training_step
training_step_output = self.trainer.accelerator.training_step(args)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 204, in training_step
return self.training_type_plugin.training_step(*args)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 155, in training_step
return self.lightning_module.training_step(*args, **kwargs)
File "/home/CDVAE/cdvae-main/cdvae/pl_modules/model.py", line 528, in training_step
outputs = self(batch, teacher_forcing, training=True)
File "/home/.conda/envs/cdvae2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/CDVAE/cdvae-main/cdvae/pl_modules/model.py", line 338, in forward
rand_atom_types = torch.multinomial(
RuntimeError: probability tensor contains either inf, nan or element < 0

I think this is due to training instability, which we also see during our experiments. My experience is that you can restart the training by loading the latest checkpoint. The training can usually continue. You can decrease the learning rate if it becomes too unstable.

Thank you very much for all your answers ! Now that you talk about that, I think I also got this error once with the carbon dataset. At that time, I attributed it to my experiments (python library upgrades and some changes in your code to try to eliminate warnings). About the learning rate, how can I decrease it ? I found the parameter lr in conf/optim/default.yaml, reduced it to half, but still crashing ...


Invalid selector for DATE_PART: "isodoy"

Both the docs on quasar and slamdata mention "isodoy" as a DATE_PART, but the backend says it is invalid. I also tried "isoday" in case it was a typo, but that fails as well.

It should be valid – but it’s possible it’s not implemented for specific connectors (I know Mongo is missing one or two date part implementations). The exact error message will help us identify if it’s failing in a connector or before (and we can update the message to make it clearer that it’s part of Quasar, but not implemented by a particular connector).

The status is "Invalid mount configuration", which I assume means that it's not implemented by the connector.

Ok, so I don’t know why this is categorized as “invalid mount configuration”, but it looks like "isodoy" is missing from the SQL->LP compiler.
Thanks.

@natefaubion Did you receive the error in response to creating a view mount with this query?

Oh that must be it! Didn't even think about that :)

So, some more general thoughts (as always). We should probably provide only the ISO parts in LP. SQL wants other parts, but we can compile those to more complex expressions.

Does not seem like an urgent issue - so pushing to May2017 iteration

Hrmm, looking at Postgres docs, etc. I don’t see “isodoy” existing anywhere. I think it’s a doc error. We do have “doy”, and I don’t know how “isodoy“ would differ from that.
But, I would still like to strip down LP to only expose the ISO parts, and let either the SQL² compiler or some stdlib of parameterized views provide the other parts.

Ah, looking at the docs, it seems like that “isodoy” is a typo of “isodow”.

Tracking the larger refactoring in #2169.


Add output images to nucleus_detection and nucleus_to_gland_distance jobs

For the nucleus_to_gland_distance job, adding this computation of the output image means that we can no longer make use of previously saved calculations of nucleus locations. We need the mask of the nuclei for the visualization (which is not saved in the database), so we need to rerun the nucleus detection step within this job. This doesn't change the time delta too much.
Also, I wrote an additional step in that job to generate a random set of blobs to act as the gland mask. The one I created manually has a set size, but the gland mask needs to be the same size as the original image to use it for the output image. So for now, the resulting images (and calculated average distances) will not be deterministic unless we decide to add a seed to the random number generator.
Here are some example output images from the populate script we run:
Detected nuclei:

Average distance to nearest gland:

Closing for now in favor of client-side visualization


Prepare base model

[ ] Update config.YAML
[ ] research or analysis
[ ] Update the entity
[ ] Update the configuration manager in src config.
[ ] Update the components
[ ] Update the pipeline
[ ] Test case added

Created


I'm sorry, but an uncaught exception occurred

I am having the following problem after trying to install the mod. Would it be because I don't have NVIDIA GPU?
The version of the mod is the last one. My python version is 3.10.10.

The main issue here is that you don't seem to have the Auto Updater Submod that causes the error in Renpy because it is not recognized here. Install the Auto Updater Submod or delete the files monikai_header.rpy and monikai_header.rpyc; it can lead to unexpected behaviours and you'll have to manually install the Renpy files at each update.
I am not sure if it will work if you don't have a GPU for the most recent version as stated in #42 but it is not the issue in this case.

I will close it for now, feel free to reopen it if needed.


Exclude Directories and Files?

Is it possible to exclude Directories and Files sections from the unabridged api?

Currently no. You just mean at the dump at the bottom, not the hierarchy part right?
The main hurdle with that is Sphinx will emit warnings for every page not included in a toctree. So even though it's still linked in the hierarchy.....Sphinx doesn't understand this.
Presumably you have a larger code base? If so the number of warnings incurred will equal the number of files + directories :/
I will reresearch if there's a way to suppress these warnings. Sphinx 2.0 may have enabled this? Not sure :/

I do not need it to display Files in the project when I already have a file hierarchy tree. https://github.com/svenevs/exhale/issues/65#issue-446748832

I have on paper a writer API for consumers which would allow you to write your own python code as a hook that exhale would use, but quite a lot needs to happen before that can become a reality.  I'm currently finishing a thesis from hell...so that has priority unfortunately.
I don't think I can get around the "document not included in any toctree" warning, but I see two options
Option 1
Add a config variable "excludeFromUnabridged": ["file", "dir", ...] (open to different / clearer name choices btw).  So list all kinds that you want excluded from the unabridged api.
Instead of putting those in the unabridged_api.rst, I think we can get away with generating an orphan.rst page and put all of the toctree for the file pages exhale generates there.  The result being that instead of a warning for every file / directory not included, we should get just one warning about orphan.rst not being included in any toctree.
Option 2
Since files and directories specifically are the problem here, we could change the layout of unabridged api altogether.  For things like functions / enums, it is advantageous for them to show up on the library root page.  It makes it easy to find things that don't show up in the class hierarchy.  But for files and directories specifically, we can add a layer of indirection here.  The difference would be currently we have something like this
Functions
=========

...

[begin snip]

Directories
===========

...

Files
=====

...

[end snip]

We can take the [begin snip] up to [end snip] and dump them into a layout.rst document.  Then at the bottom of the unabridged api, we can do
Layout
======

.. toctree::
   :max-depth: 1

   layout.rst

It'll give a short listing at the bottom of the unabridged API still, but it will be just two bullet points
Layout

- Directories
- Files

No warnings about "document not included in toctree".
Much cleaner presentation, I agree that dumping all the directories / files isn't really helpful given the file hierarchy on the same page.

Default title of Layout up for debate (can also add a config option to customize it).
What are people's thoughts?  I like (2) better because I'd rather not deliberately introduce warnings, and will be easier to implement (read: I can roll that out quickly).  (1) is better left for 1.x with the writer API I think (but again, that's a ways off...).

I like option 2. I think I speak for other breathe users when I say I don't use any other features in all the tools besides generating nice web docs from my doxygen. A lot of examples and user projects online I've looked at use the bare bones default output breathe/sphinx anyway. There's not many 'robust' layout designers for sphinx and Exhale already brings a lot to the table in this sphere because it creates separate pages.
Being able to customize even the .rts outputs just slightly would make this the Exhale/Breathe/Sphinx/Doxygen stack for doumentation imo.

However we could also just do option 1 for now as a hotfix until option 2 is complete. Then mark option 1 flag as deprecated when it builds.


Make a website to list out free learning resources.

Add a simple website using HTML, CSS and JS or preferably using GitHub pages themes like Jekyll etc. that lists out all the free resources available in this repo. Use embed tags or iframe tags to showcase an image/video of each resource listed here.

I would like to work on this issue!

@shreyateeza go ahead!! All the best.

@PragatiVerma18  What is your twitter username?

I would like to work on this issue if you want or if anyone else is not working on it..

This task has been completed by @shreyateeza. @sk044 you may try #19

what is link for website? @PragatiVerma18 , where source code to modify and add contribution page

You have to add contributor's page to the repo in the form of a markdown file or either inside the readme.md. The link for the website is here: https://pragativerma18.github.io/learn-for-free/

if it finished could you close this issue?

I would like to have some improvements in the UI if possible and also fix the unclickable links if possible.

i will try to fix unclickable links @PragatiVerma18


validate GET array of array param

Prerequisites

[X] I have written a descriptive issue title
[X] I have searched existing issues to ensure the issue has not already been raised

Issue
I have an endpoint in this format:
fastify route path:
  fastify.get('/locations/:locations', {
    schema: {
      description: 'Get locations stringified',
      params: S.object()
        .prop('locations', S.string()).required()
    }
  }, async req => {
  const locations = req.params.locations.split('|').map(a=>a.split(',').map(Number));
  ...
  })

request example:
GET /locations/11.1,46.1|11.2,46.2|11.3,46.3
This is a common way to accept comma separated lists of locations in GET requests.
How can i validate more strictly way the format of locations with comma required?

Use pattern for validating the string

partial of solution here:
https://gist.github.com/stefanocudini/d8d95133551ea4158e3bc5b8a6a77075#file-valid-locations-js-L6-L16


Object handles in place of selectors?

So, I am looking to do something like this
let selectors = await page.$$('select')
for (const selector of selectors) {
  await page.select(selector, value)
}

I understand JSHandles was made to cover use cases like this, but its use is not very clear in the docs. Any Tips?

@dowsanjack - what about:
await page.$$eval('select', elements =>
  elements.forEach(element => --do your thing with your element--)
);

@yanivefraim please make page.select able to accept elementHandle, that would make changing select value more easier


increase test coverage of execution_result

Checklist

[ ] Linked to Github Issue
[ ] Unit tests added
[ ] Integration tests added.
[ ] This change requires new documentation.

[ ] Documentation has been added/updated.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.48%. Comparing base (b09410b) to head (45b8171).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #613      +/-   ##
==========================================
+ Coverage   91.42%   91.48%   +0.05%
==========================================
  Files         109      109
  Lines       35881    36028     +147
==========================================
+ Hits        32803    32959     +156
+ Misses       3078     3069       -9

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hi @tcoratger !
We are not testing this because these traits are generated by derive!
Alsom is not a must that Serde a Deserde of the types are compatible


Use AS3 to put a datagroup as HTTP URI condition in a Local Traffic Policy

Trying to use AS3 to configure a Local Traffic Policy that uses a datagroup do match an HTTP URI condition, but couldn't find a way to declare it.
Have already tried to use "datagroup": { "bigip": "/Common/images" } as a value or like the following. All tries return with 422 errors.
 "conditions": [
                            {
                                "event": "request",
                                "index": 1,
                                "name": "1",
                                "path": {
                                    "operand": "matches",
                                    "datagroup": { "bigip": "/Common/images" }
                                },
                                "type": "httpUri"
                            }
                        ],

Is it possible to do that? How to declare?
Thanks!

Hello guys..
Any update about this?
I've also open a case #C3615034 for this request.
Thanks.

Thank you for your feedback. I have added this request to our internal product backlog as AUTOTOOL-2728.

This has been resolved in the 3.32.0 release.


proposal: Go 2: allow combining && of comparisons

I find it tedious to write the same condition multiple times.
I suggest adding an abbreviated form, to improve readabilty and reduce stutter.
Example:
// current form
if i <= 0 && j <= 0 && k <= 0 {
	return
}

// proposed equivalent
if (i, j, k) <= 0 {
	return
}

Suggested EBNF:
Current spec: https://golang.org/ref/spec#Expression
Expression = UnaryExpr | (Expression binary_op | "(" ExpressionList ")" rel_op)  Expression .

Only a rel_op would be allowed after an "(" ExpressionList ")", because the other operators don't really make sense with multiple values.
Backwards compatibility:
This change is backwards compatible, because such Expressions are currently rejected by the compiler.
Other languages:
I currently don't know that a programming language has a similar concept, but if anyone knows a similar or maybe better concept from another language, please let me know.

This seems like an extremely specific feature. It is only useful if you have several variables that you want to compare to the same other thing and AND all the comparisons together. You can't OR them, you can't check them against different values, and if you ever decide that you need something more complicated, you'll have to rewrite the entire thing.
I agree that it simplifies that specific case a bit, but it doesn't seem particularly orthogonal.

You can't OR them, you can't check them against different values, and if you ever decide that you need something more complicated, you'll have to rewrite the entire thing.

You can use AND instead of OR, example:
OR:
if i <= 0 || j <= 0 || k <= 0 {
   return
}

Equivalent AND:
if !(i > 0 && j > 0 && k > 0) {
   return
}

Proposed equivalent:
if !((i, j, k) > 0) {
	return
}

See also #33694 for another proposed change that combines comparisons.

In Go as in some other languages the && operator short-circuits: if the left expression is false, the right expression is not evaluated.  With this syntax, what happens with (f1(), f2(), f3()) > 0?

In Go as in some other languages the && operator short-circuits: if the left expression is false, the right expression is not evaluated.  With this syntax, what happens with (f1(), f2(), f3()) > 0?

It would be evaluted as f1() > 0 && f2() > 0 && f3 > 0 so it also short circuits.

Thanks.  I don't know how else this could be implemented, but it seems potentially surprising that in a list of function calls only some of the calls are executed.

I find none of the rewrites clearer than the original. The syntax is shorter but much less clear. There's no intuition, for instance, about whether (i, j) < 0 means i < 0&& j < 0 or i < 0 || j < 0. It gets harder to decode as the expressions grow.

I'm closing this for now as this design doesn't seem to get much support.


[Sync] Updated Version of LLaMa

I followed all of steps which you have told
Kindly merge this, so I can send next PR with the new vector functions

Where are our commits? Why aren't they rebased on top of the previous changes?

The commits are missing from our side:

@alitariq4589 additionally I see that these people are using lots of different CI, we can just add Cloud-V here as wel.

@AhmadTameem I think I understand now, did you fix an conflicts in this sync?

@AhmadTameem I've synced the branch by fast-forwarding master, please remove the space change in your branch and repush, @alitariq4589 Please figure out how to add our CI to the llama pipeline.

@mmhus I have removed the space change


Solvers are not istalled

Describe the bug
Only SCS solver is installed with regular installation of cvxpy through conda.
pip install ecos   does not have any effect
To Reproduce
$pip install ecos
$conda install ecos
...
print cp.installed_solvers()
output:
['SCS']
Version

Python 2.7
OS: Linux Mint 18.3 Sylvia 64-bit  ( Ubuntu 16.04 )
CVXPY Version: 1.0.6

Here is the output of $conda install ecos:
conda install ecos
Collecting package metadata (repodata.json): done
Solving environment: done
Package Plan
environment location: /home/ez9115/anaconda2
added / updated specs:
- ecos
The following packages will be downloaded:
package                    |            build
---------------------------|-----------------
conda-4.7.10               |           py27_0         3.0 MB
------------------------------------------------------------
                                       Total:         3.0 MB

The following packages will be SUPERSEDED by a higher-priority channel:
conda                                         conda-forge --> pkgs/main
ecos                            cvxgrp::ecos-2.0.5-py27_0 --> pkgs/main::ecos-2.0.5-py27h3010b51_0
Proceed ([y]/n)? y
Downloading and Extracting Packages
conda-4.7.10         | 3.0 MB    | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Then then:
$python

import cvxpy as cp
print cp.installed_solvers()
['SCS']

Sorry for the slow response. What you've shown me so far suggests a broken environment. It doesn't really seem possible to install ECOS and have cvxpy not detect it. Maybe the python you're running is not from conda?


Restart task on ded agent sooner

And run the steady queue process as background job

backport the queue fix: #4115


[SPARK-22330][CORE] Linear containsKey operation for serialized maps

…alization.
What changes were proposed in this pull request?
Use non-linear containsKey operation for serialized maps, lookup into underlying map.
How was this patch tested?
unit tests

Can one of the admins verify this patch?

Ping @Whoosh

Test build #3962 has started for PR 19553 at commit f2cbb0c.

Test build #3962 has finished for PR 19553 at commit f2cbb0c.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

@Whoosh can you please check if there is a similar problem that affects SPARK-21657 ? thanks.

@Tagar it's the same general type of issue, but not directly related nor exactly the same cause.

retest this please

LGTM

Test build #83198 has started for PR 19553 at commit 235f6d6.

@cloud-fan I've checked all core tests, it was fine, should I do smt in addition to?

Test build #83198 has finished for PR 19553 at commit 235f6d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83198/
Test PASSed.

Ping @Whoosh

retest this, please

retest this please

retest this please

Test build #83321 has started for PR 19553 at commit c4bd531.

Test build #83321 has finished for PR 19553 at commit c4bd531.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83321/
Test FAILed.

LGTM

If for some reason this last approach can't be made to work, we could just fall back to catching the exception. Because it shouldn't happen in any normal use, it doesn't matter much.

how about we just try catch the ClassCastException? introducing class tag has serialization overhead.

@cloud-fan yeah I'm OK coming full circle on this one. I had hoped to avoid it with a simpler mechanism but didn't see that it wouldn't actually work. I apologize to @Whoosh for the run-around.

@Whoosh do you have time for one more change to bring this full circle, back to catching exceptions? looks like there was a reason that was necessary.

Ye, why not, let's continue this circle ;)

retest this please

retest this please

LGTM

Test build #83506 has started for PR 19553 at commit 91cdfef.

Test build #83506 has finished for PR 19553 at commit 91cdfef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83506/
Test PASSed.

Merged build finished. Test PASSed.

thanks, merging to master!


[Bug] Isse with running Baysor segmentation on some but not all patches

Hey Quentin,
I've been trying to use the CLI workflow to segment a MERSCOPE dataset. After successfully creating patches I see that the formatting for the transcript.csv file was quite variable across patches. In some patches it was completely fine and segmentation ran well, in others some rows were shifted/missing values and I got warning messages so I removed those rows for simplicity (after which segmentation worked). Finally, some patches failed even after removing problematic rows due to:
[16:38:11] Info: Run R493595882
[16:38:12] Info: (2024-09-12) Run Baysor v0.6.2
[16:38:12] Info: Loading data...
ERROR: MethodError: Cannot convert an object of type InlineStrings.String31 to an object of type Float64.
Closest candidates are:
convert(::Type{T}, ::ColorTypes.Gray) where T<:Real
@ ColorTypes C:\Users\marsdenl.julia\packages\ColorTypes\vpFgh\src\conversions.jl:113
convert(::Type{T}, ::ColorTypes.Gray24) where T<:Real
@ ColorTypes C:\Users\marsdenl.julia\packages\ColorTypes\vpFgh\src\conversions.jl:114
convert(::Type{T}, ::Ratios.SimpleRatio{S}) where {T<:AbstractFloat, S}
@ Ratios C:\Users\marsdenl.julia\packages\Ratios\FsiCW\src\Ratios.jl:51
...
 Weirdly enough, the patches that failed due to that error came at repeating intervals. For instance, if I generated 11 patches total, then patch 1, 4, 7, 10 failed. If I generated 15 patches, 2, 6, 10 and 14 failed (1/3 patches). Do you have any idea why some of the patches failed and not others?
Code run:
cd PATH.zarr.sopa_cache\baysor_boundaries\6
C:\Users\marsdenl.julia\bin\baysor.cmd run --save-polygons GeoJSON -c config.toml transcripts.csv

OS: Windows

Thanks!

Hello @marsdenl, thanks for reporting this weird issue. I have never seen this before. Could you share an example of a transcript.csv file that failed (together with the corresponding config.toml file)?
If it's a problem to share the whole transcript file, you can also delete some unused columns or obfuscate the gene names.

Hey @quentinblampey, thanks for getting back to me. Here is the info:
config.txt
limited_file.csv
The transcripts.csv above is the one that contains both the missing columns error and the float64 error. For missing columns see row 17612 for ex. For float64, I have no idea where it's coming from...I've only attached the first 50,000 rows.
I have managed to make Baysor work independently of sopa on one of my samples so I think it might be linked to Patchify itself.
Thank you!

In case this helps, @quentinblampey, I also tried Comseg and get a similar issue with poor formatting on some patches but not all patches (some rows have fewer or more columns).
Config.json was the default one from your tutorial. I did cellpose prior nucleus segmentation as recommended and that ran fine :)
Code:
sopa patchify comseg region_0.zarr --config-path Z:\Queries\comseg_config_default.json --patch-width-microns 1000 --patch-overlap-microns 100
sopa segmentation comseg region_0.zarr
I get the error on my second (but not first) patch - ParserError: Error tokenizing data. C error: Expected 10 fields in line 112260, saw 14.
Here is a portion of the transcripts.csv containing 2 misformatted rows.
transcripts_cut.csv
Config.json
config.txt
Let me know what you think! Thank you very much.

Indeed I think the problem is happening when creating the transcripts.csv file. We do that using dask, which process each partition in parallel. Maybe, at some points, two processes try to write on the same file, and weird things happen.
Do you know if the issue always happen on the same files if you run it again? Is it random or deterministic?

I repeated patchify on the same sample with the same config file then simply counted the number of rows that did not have 9 columns in the patch files (I couldn't test re-segmentation with baysor on the patches since the Baysor update, same as issue #125) . Despite getting the same number of patches everytime I patchify, the transcripts.csv for a given patch (say patch 0) has slightly different dimensions and therefore also has a different number of rows that do not have 9 columns. Not too sure how to test for the float64 issue as I don't know where it's coming from...

Alright, thanks for the details. Can you also let me know if this happens on the toy dataset (--technology uniform)?
I'm trying to reproduce the issue

Hey, thanks for trying to reproduce the issue :) Just tried with the --technology uniform dataset but for baysor patchify it only creates one patch so I don't know if it'll reproduce the issue where some patches are okay and others aren't. Also since the baysor update I can't run baysor segmentation to check if it all runs smoothly on that single patch (see issue 125). Looking at the transcripts file for that patch though, it seems look okay :) Hope this helps. Let me know if I can send anything else over.

Can you try to make smaller patches using --patch-width-microns? You can choose any value that creates more than 1 patch

Of course, yes sorry xD Made 8 patches on the tutorial dataset and ran baysor (made the changes needed in the config file for v7.0) and it worked perfectly fine. I tried again on my dataset using the same config and again I get the same errors/warnings:

ERROR: MethodError: Cannot convert an object of type InlineStrings.String31 to an object of type Float64
Warning: thread = 1 warning: only found 9 / 14 columns around data row: 6243. Filling remaining columns with missing

Maybe something to do with the formatting of the input detected_transcripts.csv?
Thanks

Thanks for trying!
So, now, my hypothesis is either (1) the detected_transcripts.csv file itself has some issue, or (2) dask dataframe reads it wrong.
To test these:

Can you read the file (for instance with pandas), and see if it contains some incomplete rows?
Read your raw files with sopa.io.merscope as usual, and then write the full dask dataframe on disk, as a CSV. See if this file now has incomplete lines.

#Dask Name: assign, 21 graph layers}
points_df = sdata.points['Batch5_region0_region_0_transcripts']
points_df.to_csv('dask_df.csv', index=False, single_file=True)
written_panda_df = pd.read_csv('C:/Users/marsdenl/dask_df.csv')
check if any row has fewer than 9 columns -> this gave my 0 rows
written_panda_df[written_panda_df.apply(lambda x: len(x.dropna()) != 9, axis=1)]
check if any row has more than 9 columns -> this gave my 0 rows
written_panda_df[written_panda_df.apply(lambda x: len(x) > 9, axis=1)]
At first hand, it seems both files - the inital csv and the dask written csv - look okay in terms of number of columns (although let me know if code above is not correct). Could it be patchify itself?

Sure thing!

On the original file:

data.isna().any(axis=1).mean()
0.00000010972423337598314
data.isna().any(axis=1).value_counts()
False    9113756
True           1
Name: count, dtype: int64
The only true count being the column names.

On the dask-written file:

written_panda_df.isna().any(axis=1).mean()
0.0
written_panda_df.isna().any(axis=1).value_counts()
False    9113756
Name: count, dtype: int64
Regarding the csv I sent you, that's not in the original file, I must have added that... Happy to send you the original file by email.

The only true count being the column names.

Do you mean that the only row with NaN values are the columns? The command data.isna().any(axis=1).mean() shouldn't count the columns. Are you sure?

Regarding the csv I sent you, that's not in the original file, I must have added that... Happy to send you the original file by email.

Yes, please, can you send it to me (just the file of the patch that has an issue) at<EMAIL_ADDRESS>Btw, I'll be on vacations for two weeks, I'll answer you in mid-October :)

Sorry, I wasn't super clear. The only count with data.isna().any(axis=1).value_counts() is the first row, which corresponds to the column names. The first row of the first column is empty:

Just sent it now :)
No worries at all, enjoy your holidays Quentin!

Hi @marsdenl, do you also have access to a Linux or MacOS machine by any chance? If yes, do you have the same issue? I think it may be a Windows-specific issue, which would explain why I wasn't able to reproduce it.

Hey @quentinblampey, you're absolutely right. Tried sopa patchify baysor in Mac OS and ran baysor on each patch on my windows machine and it ran perfectly fine. That's good to know! Thank you very much for your help. I'll go ahead and close the issue as I'm likely going to move forward with Proseg.

Hi @marsdenl, thanks for trying out! It's good to know that it works on MacOS for you. I'll re-open the issue though, because other Windows users may want to have this fixed (even though I'm afraid it will be complex to fix, as it may be related to parallel writing on-disk with dask).
I'm interested in having your feedback about Proseg: how good do you think it is? And how scalable is it?
FYI: proseg may come in Sopa in the future, but there is no expected date for that. There is also this new tool, segger, which we'll probably add soon. In the future, we would like Sopa to be a wrapper to many more segmentation methods, so that we can easily switch from one to another without changing the code much.

Sure thing :) Best of luck with the windows issue.
I am a big fan of proseg - it looks really good. I definitely think you should add it to your segmentation portfolio. I've tried Comseg, Baysor, Cellpose, Proseg and so far Proseg is the most performing method for my tissue type (cortex). It's quite fast - could run 9 million transcripts in ~ 45mn on 64 GB RAM and incredibly easy to use (for MERSCOPE, all you need is a detected_transcripts.csv with a cell_id column giving nuclear prior info.
That being said it is highly dependent on good nuclear segmentation (see Proseg issue #34). According to DC Jones:
"Currently proseg is somewhat anchored to this (nuclear) input. It can't merge or split cells, it only tries to infer better borders (from existing nuclear borders), so if it's initialized with too many cells it's not able to correct that and has to make due."
So bad nuclear segmentation will lead to poor Proseg performance. So I would definitely recommend training a cellpose 2 model on your dataset to get a very good nuclear segmentation, then finishing it off with Proseg.
My other reservation for Proseg is the decimal value cell_gene matrix since the assignment of a transcript to a cell is probabilistic, not deterministic. DC Jones mentionned you could round it to integer value or alternatively use the maxpost-counts.csv matrix instead which is an integer count matrix which filters out ambiguous transcripts.
Hope this helps :)
Thanks again for all your help Quentin

Thanks @marsdenl for your very detailed comment! You're not the first one to have a very good opinion of Proseg.
Indeed, if it's highly dependent on a good prior, it could be a good idea to run it after cellpose.
Regarding the decimal value for the cell-by-gene table, this is also a concern to me. This is actually one of the current blockers for adding proseg to Sopa, because, when resolving the boundaries on patches, I'll need to aggregate the transcript again (which are integers with Sopa). Therefore, we would have some cells with decimal values, and others with integer counts, which is an issue...
One possible solution is to run proseg on the whole slide instead of using the patches (if proseg is not too RAM-demanding), and then the user will have the possibility to keep the decimal values from proseg, or use Sopa to get the actual transcript counts within the boundaries. What do you think?


meta.(*Store).Authenticate uses a lot of CPU

See #3102 for background. It seems like the culprit might be in github.com/influxdb/influxdb/meta.(*Store).Authenticate if I'm not missing something.
I built influxd from source and had it save pprof data (pprof.StartCPUProfile(f)).  In 0.9.1, which we're releasing as I type this.


Full support of SET operations

We miss the support of INTERSECT and EXCEPT for SET operations defined in SQL.
The detailed semantic can be found here (https://www.postgresql.org/docs/current/queries-union.html).
@andyfengHKU we need to double check equivalent semantics/syntaxes in Cypher world too. I saw this in NebulaGraph, and I can check more later.

Had a conversation with @andyfengHKU and he thinks we need these two new clauses. So we need to extend openCypher for this.


AFP-365 Fix Jira components load issues in Brisk MDX

Changes made:

Used babel-merge to merge the client and brisk babel configs
Removed babel.include and babel resolve modules from next.config.js which fixes the loader not found issues for jira components loaded in brisk MDX
Added Lazy load component to load dynamically with SSR false for non-SSR Jira components
Excluded import of @atlassian  modules in next.external.js

This together with some babel.config changes in the jira-frontend-website in Jira frontend repo solves the component loading issues

Closing this in favour of #198 which solves the merge conflicts and file shuffling from getting the publish flow working again.


[Concurrency] Don't add new task locals copy runtime func

Review follow up to https://github.com/apple/swift/pull/73978/
We don't need a new runtime entry point to implement this optimization (the swift_task_localsCopyToTaskGroupChildTaskDefensively never shipped anywhere, so removing it right away like this); and a few other cleanups and added a TODO for optimization chances.

@swift-ci please smoke test

@swift-ci please smoke test Linux


Danks in Docks wipe their inventory when the chunk is unloaded and loaded again.

Danks in Docks wipe their inventory when the chunk is unloaded and loaded again. If the dock is in a spawn chunk it does not unload and therefore does not get deleted. It will however get deleted when in a spawn chunk on server restart.

I did some digging into this.
The dank seems to be properly serialized, but on deserialization, the container size is 0 here:
https://github.com/Tfarcenim/DankStorageFabric/blob/f5c8abb547f73d44d372d3067489b75e66870582/src/main/java/tfar/dankstorage/inventory/DankInventory.java#L181
and thus the items won't be loaded and are lost.
After some debugging I found this is because the dank stats (which include it's size) are not read and thus 0.
After comparing commits, this is because of https://github.com/Tfarcenim/DankStorageFabric/commit/f5c8abb547f73d44d372d3067489b75e66870582, which was implemented to fix #15, the dank stats are not stored and loaded as NBT anymore.
This commit either needs to be reverted (and #15 fixed differently), or the dank stats need to be read differently.
Maybe it's possible to read the dank stats from the block state? Not sure if that's easily accessible inside the block entity though.

This won't be a problem in 1.17 as I'm changing danks to no longer store the data directly.  Personally I think 1.16's time is running out so I'll rewrite that now rather than trying to prop up an old, buggy system for the sake of save compat.
In the future, the blockstate will only be used for model purposes and have nothing to do with the actual contents/stats

I'm changing danks to no longer store the data directly

What do you mean by not storing the data "directly"?
I don't see much wrong with how the data is currently de/serialized, only with how it's handled, initialization mainly.

Personally I think 1.16's time is running out

Yeah, version 1.09a works perfectly fine (except for slot locking), so a backport for the fixes isn't really needed.

rather than trying to prop up an old, buggy system for the sake of save compat

If you're breaking save compat, just make sure to warn people before updating, with big bold letters:
"EMPTY YOUR DANKS BEFORE UPDATING"
Something like that.

What do you mean by not storing the data "directly"?

Dank contents will now be saved on the world instead of on the item

If you're breaking save compat, just make sure to warn people before updating, with big bold letters:

that's why I waited until 1.17 to update it, 1.10(a) was supposed to be a big fix, but it also has an accidental break that I can't fix now.

Dank contents will now be saved on the world instead of on the item

Hm... What would be the benefit of that?
You still need to associate the dank with the data, so wouldn't you still need to store some data in the dank ItemStack or dock BlockEntity either way?

Hm... What would be the benefit of that?

Less chances of accidental item wipe or bookban

You still need to associate the dank with the data, so wouldn't you still need to store some data in the dank ItemStack or dock BlockEntity either way?

Easy, store an id to the nbt that looks up the contents on the world save.,

Easy, store an id to the nbt that looks up the contents on the world save.,

Yeah, that's what I meant, you still store the id on the dank/dock, so at least in theory it could get "lost" somehow, effectively losing the data too, but I guess that's much less likely to happen, so nevermind about that.

Less chances of accidental item wipe

This change by itself doesn't really protect the data though.
E.g. if your dank couldn't read the data and turns into an empty dank, since it still has the id associated to it, it might then overwrite the existing data.
The only way to prevent item wipes is to make sure that whenever the data could not be read correctly is to forcibly disable the saving of data - which could then turn into another issue where if you put items in a "broken" dank, they wouldn't be saved and are thus lost - so ideally, the whole dank would need to be disabled.
(And that could be done with the current system too.)

Less chances of (...) bookban

That's a fair point!
I wonder though, doesn't the world data have a size limit too?
One disadvantage I can think of is that data might be too persistent because it doesn't automatically get deleted anymore.
So you might have to check if a dank item or a dock got destroyed - and I can already think of a few ways that are probably undetectable.
That may not be an actual problem though.

fixed in 2.0

Is there any chance this can be fixed for 1.16.5?
I want to use danks in combination with ME Systems from Applied Energistics 2 (which does not yet have a 1.17 version)... Or is there a way to compile the 1.17 version against 1.16.5?

Sure, but any 1.16.5 fixes will not be made by me.  I simply do not have the time as work is more important.  If you still insist on 1.16.5, use 1.9a which doesn't have this issue

Sure, but any 1.16.5 fixes will not be made by me. I simply do not have the time as work is more important. If you still insist on 1.16.5, use 1.9a which doesn't have this issue

Oh, thanks! Didn't know about 1.9a :)


Fixing problem when is radio button and no has selected button.

Please fix analyzer infos and i merge this functionality

Please fix analyzer infos and i merge this functionality

Donde!

@Frezyx Any plan to release a new version?

Soon


Cache calls

If we cannot remove the need to stat the shares in the short term, we should at least cache these calls.
These, and maybe others... Please evaluate what calls are done frequently.

We should start optimization process by removing unecessary listings of shares everywhere, then do profiling (available in Pycharm) and see which parts of code take the longest, are being used the most frequent etc. Also, caching stats might be problematic


Nothing on the page is clickable

I have recently cloned down this template and have got it running locally, however none of the buttons are producing any functionality.

fixed in PR #6.


fix Update strawberry-graphql version (>=0.160.0) and fix imports (fixes #184)

fixes #184

Thank you so much for the contribution.
This was actually contributed first by https://github.com/blb-ventures/strawberry-django-plus/pull/183, so I'm prioritizing merging it.


Add translation for recipe type in JEI/NEI

Add Drying translation for create.recipe.drying

Hi!
Thanks for the fix but it's not exactly the correct place.
Generally in the mods everything inside src/generated is generated automatically and overwriten each time.
I just fixed the language literal to the correct one.
Thank anyway!

Hi! Thanks for the fix but it's not exactly the correct place. Generally in the mods everything inside src/generated is generated automatically and overwriten each time. I just fixed the language literal to the correct one.
Thank anyway!

Ah I haven't worked much with mods so was manually overriding it with my texture pack for my modpack till you got around to fixing it. Glad I could atleast help a bit!


user-channels.ts

Another async problem:
take out awaits in channel-support implementations, move it outside into the test itself.  (remove 2 awaits)

Fixed by #211


Saving private key password within the system

I have created a private_key password (which will in turn be used to decrypt the cipher text used).
I am using;
*** Settings ***
Library    SeleniumLibrary
Library    CryptoLibrary    ${decryption_password}    variable_decryption=True

*** Variables ***
${decryption_password}    secretpass

In the steps explained I see its mentioned;
Library     CryptoLibrary    %{private_key_password}    variable_decryption=False
#private key which should be secret, should also be protected by a password

The private_key.json is needed to decrypt the values on your test server and has to be copied manually or added through the CLI interface. See Set key pair from... above.
Question:

When I set up the key pair using "Set key pair from String" option, it is success, but when I try to retrieve in robot script I am not getting it. Decryption is not happening unless I mention the actual value (un-encrypted) for "${decryption_password}", just like variables.

Basically, I do not want to give actual secret key password "secretpass", but just give cipher text or do not want give it at all and process should read it on the fly from the library.
Can you please explain how to do it?

i think here is a misunderstanding about what password is used for what.
The ${decryption_password} or as i wrote in the docs the %{private_key_password} is not the password/secret Text you want to work with or hide from the logs and your code!!!
It is the password that secures the private key.
The "Cipher Text" is the encrypted text of your confidential information.
This can be decrypted to plain text.
I would assume you used the same "string" for the private_key_password and as confidential data.
Maybe in other words.
You have the "public key", that is used to encrypt cipher(secret) texts.
And you have the "private key" that is used to decrypt cipher texts to plain texts.
This private key is secured by a password. This private_key_password has to be set when you create a private key (key pair).
You are asked if you want to save this password on the machine. if you do this, you do not need to set this private_key_password when you import the CryptoLibrary into you robot files.
But saving this password means, if someone gets access to your machine, the private key can be extracted/stolen and you security chain is broken. Due to this attack vector, it is recommended to handle the private_key_password on an alternative way. For example Jenkins can store secrets like this password and set it as environment variable. or you can pull it from somewhere else.
If you say, that your machine is save enough, just "save" the password when creating the key pair.
But again: the private_key_password is protecting your private decryption key and is not the cipher text or password you want to decrypt or encrypt.

I see that there is a lag in documentation about this password.
When i have time i try to fix the documentation.

I see that there is a lag in documentation about this password.
When i have time i try to fix the documentation.

Hi Snooz82,
Wonderful Library, if you find the time(or perhaps priority :-) )., your comment is very valueable, due to the fact it describes the Import more in detail and how it works. Ik costs us quiet some time to try, understand and that we found this topic.
Thnx and Keep up the good work.


Test Rhino startup for 30s

10s does not seem to be enough to see a crash on Rhino 8.

Not a crash, it hangs: https://bugs.winehq.org/show_bug.cgi?id=57272


gemspec: Drop rubyforge_project property

This has been removed without a replacement.

Coverage decreased (-3.4%) to 90.348% when pulling 2cdaa984fc571aaf8998c85371cb74f3943e130c on olleolleolle:patch-1 into 1d809334981103e635fc2df71722e861e87285b8 on enkessler:master.

Thanks!


Lobby icon shown 2 times on desktop

Fixed! Thanks for the report!

Is this same as one reported yesterday?  https://github.com/havfo/multiparty-meeting/issues/375#issuecomment-630221710

@cap19  Yes, my fault, but now it is fixed correctly


Preserve query parameter order

SoapUI mock SOAP services can have constructs like this one in their top-level WSDL:
<wsdl:import namespace="http://example.com/" location="/example?WSDL&interface=ExampleWSPortTypeBinding&part=ExampleWebServiceImpl.wsdl">
    </wsdl:import>

where the order of the query parameters matters. This is not supported by zuul 1.2.2 and 1.4.0 whereas basically all other reverse proxies support it fine.
In general, IMHO, query parameters are an ordered list of key/value pairs.
Also put a backport for 1.2.2 (used by released spring-cloud-netflix starter atm) at https://github.com/lsimons/zuul/tree/v1.2.21 .
Patch obviously not applicable to zuul 2.x with all its reactive magic.

+1  We have some legacy systems that we're trying to route to through Zuul and are noticing the same problem.

We were going to submit an exact same merge request before we saw this one. Can this be merged, please? It will benefit us as well.

Looks good to me.

Just discovered this issue as well. Doesn't this only address half of the problem? This will address the ordering of values relative to the same key, but not the overall ordering of all the keys.
The ordering of the a values is consistent, but the placement of b has changed.
?a=1&a=2&b=1
?a=1&b=1&a=2


perl oom program kiiled

When I run the ../input_prep/reprocess_rnac.pl id_mapping.tsv.gz rfam_annotations.tsv.gz command I get this error:
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=reprocess_rnac.,pid=334,uid=1000
[  142.115180] Out of memory: Killed process 334 (reprocess_rnac.) total-vm:9579944kB, anon-rss:7421640kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:18624kB oom_score_adj:0
anyone know what this means?

Hello
That's about your memory, simply you can add swap space. Check it out:
"https://askubuntu.com/questions/178712/how-to-increase-swap-space"
I believe that 32g of swap enough for this task.

I had encountered a similar Out of memory issue and found a workaround.
Documented here: https://github.com/uw-ipd/RoseTTAFold2NA/issues/96


Use rustls instead of openssl

In the same theme as #150 it would be nice to remove the dependency on openssl and replace it with the rust native rustls one.

Anyone working on this?


[Compiler] Null-Propagation out vars in Code-Root

Related: Reddit -> C#7 + Null-Propagation problem
While having a discussion with a fellow programmer over at reddit.com/r/csharp i found out that RoslynPad handles a certain use-case that i wish for in regular C#, which is Nullable out var
if (dict?.TryGetValue("", out var value) == true) ...
How exactly does RoslynPad handle compilation to support the following syntax
(which is invalid in LinqPad and Visual Studio 2017, C#7)
var dict = new Dictionary<string, IEnumerable<string>>
{ ["token"] = new List<string> { "abc" } };

if (dict?.TryGetValue("token", out var values) == true)
{
    // Error in LinqPad/VS: use of unassigned local variable "values"
    Console.WriteLine(values.First());
}

Funny enough, this code does not work
{
    var dict = new Dictionary<string, IEnumerable<string>>
    { ["token"] = new List<string> { "abc" } };

    if (dict?.TryGetValue("token", out var values) == true)
    {
        // Error also in RoslynPad: use of unassigned local variable "values"
        Console.WriteLine(values.First());
    }
}

Turns out, the out var values becomes a field after compilation
IL_003a: ldstr "token"
IL_003f: ldloc.0
IL_0040: ldflda class [mscorlib]System.Collections.Generic.IEnumerable`1<string> Program::values
IL_0045: call instance bool class [mscorlib]System.Collections.Generic.Dictionary`2<string, class [mscorlib]System.Collections.Generic.IEnumerable`1<string>>::TryGetValue(!0, !1&)
IL_004a: pop
IL_004b: ldnull
IL_004c: stloc.1
IL_004d: leave.s IL_0066
...
.field public class [mscorlib]System.Collections.Generic.IEnumerable`1<string> values

This does not seem right, does it? This makes code in Roslynpad behave differently that expected

RoslynPad uses Roslyn's C# script language kind. The reason it works in the first example, is that variables defined in the script's main body are lifted into fields in the generated script class.

This is not specific to RP. If you try it in the C# Interactive window in VS it would work the same way.

Just reproduced it inside the C# Interactive window, like you (@aelij) said.
Is this the expected behaviour? Should RoslynPad not rather mimic a regular C# environment (eg. VS), other than Interactive?

The reason for it is so you could reference variables in continuing scripts. If they'd only been stack elements, it would not be possible. For example:
var state = await CSharpScript.RunAsync("int x = 1;");
state = await state.ContinueWithAsync("int y = 2;");
state = await state.ContinueWithAsync("x+y");
Console.WriteLine(state.ReturnValue);

Is this the expected behaviour?

Yes - it's entirely determined by Roslyn. I may introduce a "regular" code mode which would require a main method and also allow things like namespaces, which are not allowed in the script kind.


connection error:  socket closed

Hello,
since the update i'm always getting the error: connection error: socket closed.
I think the server closes the socket..i don't need an solution i only want to knnow what happened.
Did the agar.io server rejected the client ? (i changed my init key)
The proxies i use aren't bad ones, and before the update they worked perfect. My other thougt was that  agar.io banned my ip's.... do you know what happened ?
Thanks,
David

Do you use agario-client version 1.3.0?
examples/basic.js works fine for me, is it broken for you?

yes I do....


Whiteout files and opaque directories should not be copied to the lower filesystem?

Maybe this is by design...
Say you have a lower filesystem, "/lower" containing a directory "/lower/d" and a file "lower/f".
Then you mount an overlay, "/overlay" using an empty upper filesystem, "/upper".
Then rm -rf /overlay/d /overlay/f.
Using overlay merge -l /lower -u /upper command, in the generated shell script you get
rm -rf '/lower/d'
mv -T '/upper/d' '/lower/d'
rm -rf '/lower/f'
mv -T '/upper/f' '/lower/f'

The whiteout file and opaque directory from /upper are moved in after deleting the existing file.
I guess its a simple find 1-liner to find these files and delete them from "/lower" if you don't want them but I wonder if overlayfs-tools should either not copy these files in the first place or provide an option not to copy them.

This is really odd, I can't reproduce this now! Maybe it was the way I set up my mounts originally. I think the mount point for the overlay might have been within the upper directory, which could have messed things up.
This can be closed if no one can reproduce, but maybe there's an edge case here.

Whiteout file should be deleted. It's definitely a bug if the problem can be reproduced.
However, with previous implementation, the whiteout file should already be removed, as https://github.com/kmxz/overlayfs-tools/blob/1cb719fd455d84298123bf2ab129a3104e02f0f6/logic.c#L332 did rm the whiteout file.
Let's keep this issue open to see if anyone can reproduce it.


Exoplayer is in google()

ref: https://github.com/google/ExoPlayer/releases/tag/r2.13.3
ref: https://mvnrepository.com/artifact/com.google.android.exoplayer/exoplayer?repo=google

@licaon-kter the PR looks ok, but can you add a small line the change log please?

Like this?

Like this?

Yes thanks !


health-checker posting wrong status for kubelet - KubeletUnhealthy

Health-checker is reporting wrong status for kubelet.
Type    Reason                   Age   From            Message
  ----    ------                   ----  ----            -------
  Normal  KubeletUnhealthy         12m   health-checker  Node condition KubeletUnhealthy is now: True, reason: KubeletUnhealthy
  Normal  KubeletUnhealthy         103s  health-checker  Node condition KubeletUnhealthy is now: True, reason: KubeletUnhealthy

Although the kubelet is running in the node and from the node-problem container I can see the same result:
● kubelet.service - Kubernetes Kubelet
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubelet-args.conf, 30-kubelet-extra-args.conf
     Active: active (running) since Tue 2022-04-19 07:46:28 UTC; 3min 42s ago
       Docs: https://github.com/kubernetes/kubernetes
    Process: 13834 ExecStartPre=/sbin/iptables -P FORWARD ACCEPT -w 5 (code=exited, status=0/SUCCESS)
   Main PID: 13838
      Tasks: 26
     Memory: 61.8M
     CGroup: /system.slice/kubelet.service

If I run manually the script /home/kubernetes/bin/health-checker it gives the error that the kubelet is not running.

Health-checker is reporting wrong status for kubelet.
Type    Reason                   Age   From            Message
  ----    ------                   ----  ----            -------
  Normal  KubeletUnhealthy         12m   health-checker  Node condition KubeletUnhealthy is now: True, reason: KubeletUnhealthy
  Normal  KubeletUnhealthy         103s  health-checker  Node condition KubeletUnhealthy is now: True, reason: KubeletUnhealthy

Although the kubelet is running in the node and from the node-problem container I can see the same result:
● kubelet.service - Kubernetes Kubelet
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubelet-args.conf, 30-kubelet-extra-args.conf
     Active: active (running) since Tue 2022-04-19 07:46:28 UTC; 3min 42s ago
       Docs: https://github.com/kubernetes/kubernetes
    Process: 13834 ExecStartPre=/sbin/iptables -P FORWARD ACCEPT -w 5 (code=exited, status=0/SUCCESS)
   Main PID: 13838
      Tasks: 26
     Memory: 61.8M
     CGroup: /system.slice/kubelet.service

If I run manually the script /home/kubernetes/bin/health-checker it gives the error that the kubelet is not running.


Simplify reinsert logic when altering a table in SQLite

This is the procedure recommended by the SQLite Docs. It saves a few SQL statements, most importantly copying the data twice.
Unfortunately, SQLite's RENAME TABLE adds double quotes around the table name, so I had to alter some of the tests.

Not sure, why some of the mysql tests are failing.

@nickrum I think this is GitHub going haywire internally, I'm observing the same on other PRs now. Probably will resolve itself in a couple of hours.

@nickrum It's green now.


Link to minfied file on instalation page is wrong

the file repo is jquery.tosrus.all.min.js but on instalation page it's jquery.tosrus.min.all.js min and all are swapped.

Thanks, I'll update the docs shortly.


adds new Source viewer; implements virtualized preview table for Source and Model

This PR does a few things:

it implements a new source viewer, which enables people to take a look at a table view of a data source.
to make this source viewer work, we really needed to support virtualized tables. Otherwise, the DOM paint gets really awful. We knew we'd need to do this anyway down the line, so this should uniformly improve the perf of Rill Developer.
the new virtualized table component is also used in the model table as well.

This PR needs:

a code review
an experience check

The sources view is a great addition to the product, but we should be sure to add a bit in our release notes that lets people know this is going to happen for existing sources. @bcolloran , a note for you and I to remember to include this when we release.
Can we extend the 150 row outputs in sources to the model outputs? :)
I am not sure if we should be tuning our column widths using the longest string value in that column.  I wonder if the median string length with the column name would make it a little more condensed.  Max width is a bit too wide.
Virtualization is extremely cool. I didn't encounter any jank when I scrolled fast. Great work here!  I am approving and you can take and leave this as nice to have.

A couple quick items on experience:

I'm only seeing 1 row of data in the big table display (2 rows if you include the header)
In the assets sidebar, when you click twice on the Source name, the profiling information should expand. (This behavior would mimic the behavior in the Model assets.)

@magorlick @ericpgreen2 ok, think I covered the bases. One last review?

It looks like there's extra width for some of the columns, e.g. see the user_id, event_type, and location columns here:

Are those columns wide due to a minimum_column_width property somewhere? Or are they leaving extra room for the "pin column" icon?
I think if we compressed them, the table would feel a bit more powerful.

I like the improvements 👍  – particularly the hidden pin icon, and it's handy that reloading the page reconciles the 1-row issue.
I left a couple small comments, but I'll approve so you can merge when you see fit.

Nice, works great. One cool additional feature would be to double-click on the "resize column" handle and have the column compress to the smallest available width – like Excel.


Source path bad for gnome extensions

Some packages like gnomeExtensions.desktop-icons-ng-ding have a bad link to source, as it points to the generic buildGnomeExtension function definition:
https://github.com/NixOS/nixpkgs/blob/nixos-23.11/pkgs/desktops/gnome/extensions/buildGnomeExtension.nix#L54

The source link uses the metadata provided by nixpkgs:
$ nix repl
Welcome to Nix 2.18.1. Type :? for help.

nix-repl> :l <nixpkgs>
Added 20006 variables.

nix-repl> gnomeExtensions.desktop-icons-ng-ding.meta.position
"/nix/store/cslab5wq445yijm8vcpvdr8zi92z8ix3-nixpkgs-23.11-small.tar.gz/nixpkgs/pkgs/desktops/gnome/extensions/buildGnomeExtension.nix:54"

It's probably best to create an issue there and ask the GNOME maintainers.


[FEATURE REQUEST] The task details can display the complete payload

Copied from hibiken/asynq/#636
Is your feature request related to a problem? Please describe.
Currently, the payload of task details cannot be fully displayed. If it exceeds the limit, it will be omitted, and the data that needs to be viewed for task delivery cannot be fully viewed.
Describe the solution you'd like

I hope to support viewing the full payload content
If it cannot be fully displayed, I hope that a copy button can be added next to the payload display, so that the developer can copy the complete payload and check it by himself

Describe alternatives you've considered
Additional context

@shuqingzai I understand that this is something we all needed. I will create a PR for this.