Apertures

Hi all,
the long awaited and delayed pull request is finally here. I'm opening it as a draft as there are large amounts that will not be changing and can be reviewed now. Things that will be changing are documented in issues #140, #141, #142 and #143. Moreover, the final touches need to be made to the tests and I will make a detailed tutorial.

Yah this is big one 😅 - I can start to review this but will likely take a while, will be focusing on getting paper 1 submitted today.
What are the main files to review?

apertures.py contains it all. There is only about 3000 lines that survived.

I think it might make sense to move the coordinate transforming methods into the utils/coordinates.py script

In the class docsting we describe the Attributes, not the parameters, parameters describe function/method inputs

So I've done a bit of a quick review of part of the apertures.py file. It all looks really good, for the most part it just needs a thorough once-over of the docsrings and type hints. Happy for this to be left till the end, so that things can keep changing if need be in the dev stage.
The code all looks really good though, only real thing is to add some comments as to what is being done in some of the bigger methods as they have some abstract variable names and so can be a little hard to follow what is happening.
I'm happy if you just want to focus on the tests and code, and I can do a thorough once over of the docstrings etc after that!

OK @LouisDesdoigts things should be good for you to go through. There will be some minor changes tomorrow from #142 but other than that it is just the documentation getting added to the docs folder along with a tutorial.

A few comments/questions, though this would be easier than sporadic slack messages and spare comments in the PR:

The high level Aperture class should call super().__init__(name=name) and take name in as a kwarg
Optimise the ._aperture and ._basis methods with jax.tree_map -> See core.py: model() for examples
set Compound/Composite/MultiAperture to take in a list and use the list_to_dict method to initialise the internal dictionary
Create a .get_aperture(diameter, npixels), .get_basis(diameter, npixels) method in Aperture that calculates the aperture and basis arrays for the purposes of plotting/visualisation
Enable nan checks (via the jax debug nans flag) in the tests
Move the basis generation functions into utils. Its a really awesome set of functions that I think should be stand alone. This is low priority though, would be happy to merge the pr first and set that up later, if yall prefer

I think once we are happy with all this, I'll do a big once over for docs/formatting (theres still some stray Matrix, Tensor etc and some of the var: dtype = ...)
Reminder for myself: Implement a __getattr__ method and clue in higher level classes to call this method

Oh and as a fast way to do jit and higher order grad tests, try adding some complex aperture layers into the tutorial notebooks and make sure things don't break

What does the second point mean? Is that for the MultiAperture and CompoundAperture? There is nothing to map over for the individual aperture classes (that I am aware of). As for 1 and 3, this is why we need a change log which I will implement as part of the pull request.

I plan to keep @ataras2's has_aberrated flag as it let's us make the small optimisation of only calculating the path differences if an AberratedAperture was included very easily.

OK, that should do it. The PR is ready for it's final review.

Alright so just a dip into the apertures branch, made a bunch of docstring fixes and updates, a few namespace changes, just a lot of minor but important changes. It's all looking very good and I have a much better idea of everything at this point. So there are a few last things that need to happen for before the merge in (vague) importance order.

[ ] Fix the apertures tests and re-enable the jax debug nans flag. This is the most important, we never want any nans in our code. The last test of the apertures produces a bunch of nans and infs which shouldn't happen. It looks like it originates from the basis stuff, but I'm not sure beyond that.
[ ] Adding assert statements to constructors. There are a lot of constructors that don't test things like dimensionality. This is is important as you should not be able to construct an object in dLux that then throws an error when it runs, and it should give helpful responses as to the correct formatting for some input (ie, rotation should be a zero-dim array). This shouldn't be too hard, just annoying. The tests should be updated to test these.
[ ] Fix phase/opd inconsistency in the aberrations. Some add phase, some add opd, this should actually be an option for the user. Easiest is probably to default to applying an opd with a simple boolean class attribute that is checked in the __call__ method.
[ ] Reduce the number of pixels used to test the apertures class. The tests take too long, I would say using anything larger than a 16x16 array is overkill.
[ ] Add some softening amount parameter. As far as I can tell this should just be a simple value that is multipled in the ._soften() methods.
[ ] Fix/Update the tutorial notebooks to work with the new code. Some of the notebooks rely on deprecated layers like the old CircularAperture. These should be updated to use the new layers (I am happy to fix these up though).
[ ] Clean up the branch. Namely in the dev folder, specifically the jwst.py, orthonormalisation.py, and the tests/ directory. If there is important code it should be moved into the relevant scripts, else it should be moved into a different branch and then be removed for the PR.

After these changes then we should be able to merge into main. There is one last sticking point though about these basis stuff. I still maintain that the basis generation (and aperture generation stuff tbh) should be moved into scripts in the utils folder. The core is just jax functions that are very useful outside of the apertures and it doesn't make sense to me that you should need to create an aperture class to pre-calculate some polike basis for use elsewhere. I am aware this ties into the wider discussion (#157) so we can discuss that further there, and I would be more than happy to merge the PR with things where they are now.
Anyway this is all so good, I had a bit of a play around with the classes and I think the API is 🤌🤌. Very much looking forward to using this in the future, its awesome!

You do realise that you were getting annoyed when I had a floating point softening parameter :laughing: :rofl:.

"Fix phase/opd inconsistency in the aberrations."
It only works through OPD at the moment. Where are you referring to inconsistency?

Where are you referring to inconsistency?

The call method of the StaticAberratedAperture uses .add_phase.
I also just noticed a few things about that class:

It should inherit from the StaticAperture class (not sure how this fits with the wider inheritance structure, but it duplicates the .aperture parameter), it would make sense to inherit from StaticAperture.
The call method applied the .method().method() to the wavefront - can be improved with an eqx.tree_at to apply the updates simulatensously.
It just applies the basis, rather than calculating the opd from the coefficients multiplied to the basis.

You do realise that you were getting annoyed when I had a floating point softening parameter 😆 🤣.

Yeah I had a misunderstanding of how the method worked at the time, I thought it applied a single pixel threshold, rather than a multi-pixel threshold. Since its not actually a binary single pixel threshold, but actually a continuous parameter. It does make sense to have a float since there are cases where we might be working with very large or small optics in which case it would be highly valuable to set this manually.
I've been meaning to ask, from what I can glean from the code it looks like the threshold amount is in physical units (ie meters). It would make sense to me to have the parameter in units of pixels, not sure if that would be very difficult to change though.

So to get it in units of pixels we can just divide by the pixel scale. The steepness parameter actually has units of, inverse distance. This is so that np.tanh has a unit-less parameter. Setting the number of soft pixels is a fine art that there is no exact answer for. The way I currently have it set up produces about 2-3 pixels that are obviously non-binary. Because the function is continuous there are many more that are not very slightly not binary. We can always exchange the np.tanh for a custom piece-wise continuous function. I think that if you want to explicitly chose the number of pixels, this would be the way to go as it gives an exactly linear response. I'm not sure how this would interact with the gradients though since, just a hard edged (binary) aperture is technically also piece-wise continuous.

I have encountered a familiar problem with have a public (formerly private) parameter to control the width of the soft edged region. Namely, the magnitude of this parameter depends on the pixel scale. This means that setting a default value at __init__ time is not very easy. I have found that a good (heuristically speaking) value is the number of pixels along on edge of the image (2-3 soft pixels). Since this is not known at __init__ time except in the static case (coordinates are in the Wavefront and the StaticApertures).
The solution I have come up with involves a lax.cond/if along the lines of if not self.steepness: then .... This is not ideal because we run into all the usual jit problems. Does equinox automatically make such arguments behave like static_args or can we use eqx.static_field to achieve this?

This has been resolved via a heuristic command and relaxing the exactness.

OK, so as we suspected the nans were occurring in the extra memory that get's allocated to make the basis code compile. They were the result of dividing infinite values by infinite values. The infinite values were in turn resulting from the gamma function that powers the factorial getting evaluated on negative numbers. I wonder if we should add a flag to factorial?
With this comment the code is ready for its final review @LouisDesdoigts. All you should need to do is check the tutorial notebooks have the correct output since I cannot run them on my computer.

So I just tried to boot my forwards model again and there is a ConcretisationError that occurs now with a zero dimensional array and != 0.. I am stopping for the week but I will make this the first point of call on Tuesday.

The last 115 commits changed nothing in the actual code (really) and were mostly my examples and attempts at optimisation.


#43 needs to be reopened. Weird changelog generated with release 1.8.1

Issue described in #43 , weird changelog generated with git version 1.6.1 has reoccured with the version 1.8.1 of auto-changelog.
Node 8.11,
Git 1.6.1,
Linux

@tirthaguha What problem are you seeing exactly? Do you have an example of the problem, or a changelog that gets generated? The %Bs from your original issues should have been fixed by 3c5ae3bb2c0934582b77533cfe8420ad2c5acee6
I'm quite apprehensive about putting loads of time into supporting old versions of git. It has been over 10 years since 1.6 was released. Unless it's a simple fix, the answer will unfortunately have to be "use a newer version of Git or you cannot use auto-changelog."


update API to ES5 style

Change API for ES5 style for browser compatibility.

It would be helpful if the design note included some examples of the api in use.
It includes a number of use cases, but it doesn't go on to show how the api would be used to address those use cases.
For example, if I have a String contain flow json, what do I do to use this api?
This is still marked as a Draft PR - aside from my comment above, are there more changes you plan to make as part of this PR?

For consistence with the name of PR, I update only for API style matter.
In next PR, I'll add examples for API usage, as @knolleary commented above.

I think this design has been superseded by https://github.com/node-red/flow-parser
Any further discussion on the API should happen in that repo.


component and log level

I was wondering if it was possible to add or if there is a better way to accomplish this.  I would like to have a component and log level and be able to specify for each component the log level.  This would allow me to get fine grained control over exactly what logging I desire.  Right now I have each "component" use a specific log level range, but this is not ideal as it means when setting setHighest() there is no way to get logging from any component above this.

Not sure I fully understand what you mean.  I'll try.

You can create arbitrary log levels of your own choosing.  Each log level conforms to a logging ID but they can share logging IDs also.
Each component, or file, or whatever can have arbitrary number of logging levels.
You can turn on/off logging levels (see examples).  If you want this runtime you take a small performance hit.
It is at a global scale that logging levels  are turned on/off
You can have loggin levels turned on/off with more fine-grained control (say per file or component) this is not supported by the g3log framework directly but you can easily write your own logging sink that can be called (see examples) where you specify what logging levels you want shown.

G3log is made to be extendible through the logging sinks.

I think the use example is something where log messages would gain a new (optional if we could?) field for 'component'; something like:
LOGF(INFO, ConnectionHandling, "New connection from %s; handling request", conn_info->to_string());
Or for the stream operator variant I can see a few paths:
LOG(INFO, ConnectionHandling) << "New connection from " << conn_info->to_string() << "; handling request"; or possibly LOG(INFO) << LOG_COMPONENT(ConnectionHandling) << "...
Components would then be enabled/disabled the same way log levels are but independently from them.
In our specific case we tried to create multiple levels to cover this, but as our code has gotten larger and more complex we are finding it difficult to work with. A developer working on Module A may not care about highly verbose messages from Module Q. So, just enabling a global equivalent to something like a -vvvvv and getting "TRACE5" verbosity globally does not work as the log files become too large to work with. We can easily produce 100s of GB of log files in under 5 minutes with full verbosity enabled.
Right now the only three ways we see without and explicit 'module' part would be

Creating a custom sink to filter on files
Prefixing messages with a component "[Module D] message text here" and creating a custom sink that parses the first part to find the module and filter on it
Creating levels something like: MODULE_A_DEBUG, MODULE_A_TRACE1, MODULE_A_TRACE2, ... for as many different actual 'levels' as we wanted for that module and then going on to MODULE_B_DEBUG, MODULE_B_TRACE, ... and so forth.

There are problems with all of these approaches and we thought adding a second variable to cover modules/components that can be enabled/disabled just like, but independent of, levels would be the simplest solution.
We are happy to take a swing at adding the feature and putting in a PR for it but wanted to get your take before doing the work. We'd much rather not maintain our own branch of the project if we can avoid it.

if the feature is added it would have to be a new log call since it otherwise would break functionality for other users.  C++ does not allow macro overloading on number of arguments.
Personally I think this feature is better suited in a forked version of g3log. It is however a feature request that keeps popping up so I’m happy to help out with advice and review.
If you go ahead with a PR then you get the benefit of peer reviews by me and hopefully other users (regardless if you later go the route of a forked version of g3log)
I would suggest keeping in mind:

Do not break existing functionality
How to extend with new functionality for the default and existing third party sinks without breaking them.

(I.e it is easier with a breaking change but that has to happen outside of the main version of g3log)

Closing due to lack of response


Publish Gemfile.lock

Use Case
I'd like to be able to install Bolt from source and get the same gems that are in the binary packages. This would be nice for local installs, but would be especially useful for getting Bolt packaged in Homebrew and other package managers.
Describe the Solution You Would Like
Remove Gemfile.lock from .gitignore. Update/commit Gemfile.lock before tagging a new release.
Describe Alternatives You've Considered
#3282

We dont use bundler to package bolt so there is no Gemfile.lock to publish. Packaging (including not only gems, but the entire ruby runtime) is defined in public vanagon repos https://github.com/puppetlabs/puppet-runtime https://github.com/puppetlabs/bolt-vanagon . Also we have a cask for bolt https://github.com/puppetlabs/homebrew-puppet

Thanks, I'll take a look a vanagon.
Really not interested in the cask. Need native arm support, and in any case IMO a brew formula would be superior to a binary-only package even once arm support is available.


LoS - Interposes question: error appears in console

Description: If an interposed question of user with a structure level is deleted from from the list of speakers with an active speaking times widget, an error message appears in the console. I'm not sure if this have or could have a bad affect. Can't find one.
Error message:

Reproduction:

open a meeting
open settings > list of speakers > parliamant mode: activate interposed questions and add time value for the structure level
open participants > create a structure level and create a user. add the structure level to the user
open agenda > create a topic and project it
open the autopilot. Add yourself to the list of speakers, click play and add an interposed question and select the new user. Take care, that the user has a structure level.
Now delete the new user from the list of speakers. -> the error appears in the console

What should happen:
Deleting a user with a structure level from the list of speakers (with an active speaking times widget) should not trigger an error message in the console.

I have tested a bit. I found some strange error messages while clearing a question user sith a strlvl from the los, but not all the time. And they seems to have different causes. The problem above could be solved by adding a conditional to the speaker, I think.

I failed to reproduce the problem today.
If I reload, I will see only the NotifyService console error.
I tried: the  steps, superadmin/meeting_admin, start the interposes question and clear then, add another user with structure level to the participants and use them, reload the docker.

A consistent reproduction of the Error I managed to create:

create a new meeting
create a structure level
assign yourself the strucutre_level
add time value for the structure levels
project sth with a LoS onto the autopilot
add your self on said LoS
start yourself
clear yourself (CLEAR not stop)
Error is thrown

Additional Bug:
The time on the speaking times card continues to run down
I will update the ovverall description of the Issue


İnstagram üçin vpn gerek

Instagramy açýan vpn ba bolsa please beräýiň
Intraň kody ba bolsa kemem bolmazdy

Salam @enegulrahmanova
Bu ýerde kod bermeýäler. Bärde intra-nyň işgärleriniň işleýän ýeri.
Vpn gerek bolsa şu (https://twitter.com/tkm1i) ýerden bize ýazaý. Bu ýazanyňy hem 'close' basyp ýapaý.

Hey @fortuna,
First of all thanks for all the great work!
I have seen many open issues by people from my country. They are basically asking for the Intra DoH url.
Please feel free to close these issues. You can redirect them to my account since I provide outline (intra maybe in future) service.


[BUG] Button in Open source roadmap page redirects to a invalid repo

Has this bug been raised before?

[X] I have checked "open" AND "closed" issues and this is not a duplicate

Where did you find this bug?
On Biodrop website, Open source page
Version of BioDrop (for example "v1.2.3")
v2.75.0
Description
When we click on the button,

it redirects us to:

Screenshots
https://github.com/EddieHubCommunity/BioDrop/assets/143683619/8a90e782-da74-41dc-9ec9-39e16937780c
Do you want to work on this issue?
No
If you want to work on this issue...
No response

Thank you for raising the issue 👍

can i work this?

Can I work on this issue?

Thank you for raising the issue 👍

Welcome @eddiejaoude .
I had opened this request for others to contribute, but I realized i haven't contributed to this repository.
Really sorry for asking this but can you assign me this issue?

Thank you for raising the issue 👍

Welcome, My pleasure

Hi there, can this issue be assigned to me?

i would like to work on this issue, Kindly assign me. Thanks

Please assign this issue to me.

/assign

Please assign this issue to me.

Please assign me this issue if possible, would like to contribute to this.

Hi! if this issue still doesn't have assignee, i would love to work on this issue

Hey everyone, let's take it easy. Please avoid spamming here since only one person can be assigned to an issue. In the future, there will be more chances to work on different issues.
In open source, collaboration is key, not competition.

Thank you for raising the issue 👍
Since this issue is still not assigned, I would love to get this assigned to me

To anyone who would like to be assigned this issue, please explain what technical solution you would implement.
Please note the points (effort required) has been updated from Points 3 to Points 5 - therefore we are looking for the assignee to be familiar with this project and have worked on PRs before (and had these merged).

i will fix it by changing the href value of "Open Source Roadmap" in the components/layout/DocsLayout.js
this is the current code
{ name: "Open Source Roadmap", href: "/docs/open-source-roadmap" },
and this is the code for my changes
{ name: "Open Source Roadmap", href: "/docs/open-source-roadmap/index.js" },

I will fix the issue by changing the line in the following in Edit Github Links file
Current Code
 asPath === "/docs" ? `index.js` : `${asPath}.mdx
My Code
 asPath === "/docs" || asPath === "/docs/open-source-roadmap" ? `${asPath}/index.js` : `${asPath}.mdx
It will open the links to there correct files in the repo

Right now the current code is
const githubUrl = `https://github.com/EddieHubCommunity/BioDrop/edit/main/pages${
    asPath === "/docs" ? `/docs/index.js` : `${asPath}.mdx`
  }`;

by giving "/docs/open-source-roadmap" another condition to check and replace "/docs/index.js" with ${asPath}/index.js we can solve this issue
My code
const githubUrl = `https://github.com/EddieHubCommunity/BioDrop/edit/main/pages${
    asPath === "/docs" || asPath === "/docs/open-source-roadmap" ? `${asPath}/index.js` : `${asPath}.mdx`
  }`;

Here is the screenshot
On Production

On development/my local machine

I will fix the issue by changing the line in the following Current Code asPath === "/docs" ? `index.js` : `${asPath}.mdx My Code asPath === "/docs" || asPath === "/docs/open-source-roadmap" ? `${asPath}/index.js` : `${asPath}.mdx It will open the links to there correct files in the repo

Thank you @Ameya02  - as you have written the correct solution first, I will assign this to you.


src_port, dst_port, protocol details

Hi,
Is it possible to add src_port, dst_port, protocol labels also to the metrics?

Hi @vishnubraj,
It's definitely possible, but not something that I would recommend. By adding source and destination ports, it would result in very high cardinality metrics (you'd have a unique combination of every label type). This not only would slow down queries but would significantly increase the amount of data stored in Prometheus.
If you were to do this, you would need to first export the data from pmacct, with something like this in your pmacctd.conf
aggregate: src_host, dst_host, src_port, dst_port, src_as, dst_as, label, src_port, dst_port

(other options are laid out here)
And then the following sections of the Kafka consumer would need to be updated:
https://github.com/neptune-networks/flow-exporter/blob/e8e2114af60536460f836e2130f2ad50c654a9d3/internal/consumer/consumer.go#L17-L33
https://github.com/neptune-networks/flow-exporter/blob/e8e2114af60536460f836e2130f2ad50c654a9d3/internal/consumer/consumer.go#L35-L42
https://github.com/neptune-networks/flow-exporter/blob/e8e2114af60536460f836e2130f2ad50c654a9d3/internal/consumer/consumer.go#L127-L147

Thanks for the clear explanation @bswinnerton
I agree with the increase in the data storage part.  I will check if I can use it without any performance degradation.

It would also be a nice addition (I'm testing it) to add tags like iface_in, iface_out - this would open up more granularity of distinguishing what provider the traffic is coming from (e.g. peering vs transit)  I'm looking into if the tags can be used in pmacct.
and also multi-as, in my use case I'm monitoring two different AS'es. I solved it by running two exporters - but I'm going to try to modify so multiple AS'es are supported. (perhaps a special case).


Signalling incoming/outbound connections

I currently work on a prototype to allow direct browser to browser connections for peers within the same network without an intermediate server. The information on how to connect to the other peer (basically the ICE candidates) are encoded in a Multiaddress. That Multiaddress can then be used to dial the peer via libp2p.
In WebRTC you always have one peer that initiates a connection, while the other peer then connects to it. So you have two different kinds of Multiaddresses. One that will act as the peer that initiates the connection and one for the peer that accepts that connection.
I'd like to have a way to signal this. I don't think that's currently possible, would there be a way to add this to Multiaddress?
Currently I just append /webrtc-initiator and /webrtc-receiver to my addresses. But perhaps it could be more general purpose, so that it can be used outside my specific needs. Kind of similar WebRTC TCP candidate types that specifies whether a transport will accept incoming or outbound connections.
I could imagine this also being used when you e.g. want to allow only certain dialable addresses. Let's say my peer only wants connections from a certain peer. It could dial itself with the information of the other peer and keep track of that.

I've discovered the memory protocol, which can be (mis)used for application specific things. That works for my case. Hence I'm closing my issue.

(Mis)using the memory transport, at least for small projects, sounds good to me.


Basic translations for nl-NL.coffee

@Bryukh just signed it.

Thanks! Merged.

hi @Bryukh can i also translate dynamic content from github? or do i need your tooling? (sorry, but is less intuitive for me)

No, dynamic content needs to be translated via https://direct.codecombat.com/i18n – more here: https://github.com/codecombat/codecombat/wiki/Diplomat-Home


Expose running compaction in coordinator for the monitor

I was able to test this with the page I am working on in the Monitor and I think it is good to go. The only thing I noticed that by having the Monitor make calls to the Compaction Coordinator, it introduces a Thrift dependency to the Monitor:

[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (analyze) @ accumulo-monitor ---
[WARNING] Used undeclared dependencies found:
[WARNING]    org.apache.thrift:libthrift:jar:0.15.0:compile

I am OK with this. I was actually surprised there wasn't one already. I guess all the calls that go through the Manager don't require a direct dependency.

I was able to test this with the page I am working on in the Monitor and I think it is good to go. The only thing I noticed that by having the Monitor make calls to the Compaction Coordinator, it introduces a Thrift dependency to the Monitor:
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (analyze) @ accumulo-monitor ---
[WARNING] Used undeclared dependencies found:
[WARNING]    org.apache.thrift:libthrift:jar:0.15.0:compile
I am OK with this. I was actually surprised there wasn't one already. I guess all the calls that go through the Manager don't require a direct dependency.

I can't reproduce that, so I'm assuming you're describing changes not included in this PR. In general, it's best to keep our Thrift types from leaking too far from the RPC layer. The monitor shouldn't interact with Thrift at all. It should interact with other service APIs, and those service APIs should not leak Thrift types, if at all possible.

I can't reproduce that, so I'm assuming you're describing changes not included in this PR.

Yes. I am using CompactionCoordinatorService.Client to call the method getRunningCompactions. It returns multiple thrift types, TExternalCompactionList and TExternalCompaction.

The monitor shouldn't interact with Thrift at all. It should interact with other service APIs, and those service APIs should not leak Thrift types, if at all possible.

I don't understand what you mean. I am also not sure how the Monitor doesn't already have this dependency. The Monitor currently uses thrift types like ManagerMonitorInfo and ManagerClientService a great deal.

I can't reproduce that, so I'm assuming you're describing changes not included in this PR.

Yes. I am using CompactionCoordinatorService.Client to call the method getRunningCompactions. It returns multiple thrift types, TExternalCompactionList and TExternalCompaction.

It's hard to troubleshoot without seeing your code.

The monitor shouldn't interact with Thrift at all. It should interact with other service APIs, and those service APIs should not leak Thrift types, if at all possible.

I don't understand what you mean. I am also not sure how the Monitor doesn't already have this dependency. The Monitor currently uses thrift types like ManagerMonitorInfo and ManagerClientService a great deal.

I guess there are two kinds of "Thrift types". There's our autogenerated code, like those you mention (I still think they shouldn't leak into the monitor, but I guess they're not causing a problem right now) that are from our accumulo-core jar. And then there's types from the Thrift jar. maven-dependency-plugin is checking for compile-time dependencies, not runtime dependencies. So, it shouldn't complain unless the monitor references a type org.apache.thrift.*. For the examples you gave, we're never actually referencing any org.apache.thrift.* classes in the monitor, even if those classes themselves reference them (creating a runtime dependency).

So it turns out I was introducing the thrift dependency by adding these two imports:

-import org.apache.thrift.TException;
-import org.apache.thrift.transport.TTransportException;

So I think what we have in the Monitor is a bit of a hack to get around this. By catching Exception, like we do when we get the ManagerClient, I can avoid importing the thrift exceptions and it doesn't complain about the dependency.

I'll resolve the conflict and merge this morning

Sorry I forgot to mention one of the values missing from TExternalCompaction. The compaction kind is available on the coordinator but not set in the thrift object when calling getRunningCompactions(). I think it can be pulled from the job object.


⚠ GitLab Server has degraded performance

In c70019c, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 11240 ms

Resolved: GitLab Server performance has improved in 5f60a0c.


#558 Docs: Mesheryctl and its behaviors need added to Meshery Documen…

…tation
Signed-off-by: gunjanmimo<EMAIL_ADDRESS>

@gunjanmimo thanks for documenting mesheryctlI’ll get this exposed in the toc.yml and have the type: guide added so that it shows up under https://meshery.layer5.io/docs/guides

fixes #558


ToolStripSeparator controls allow changing their backcolors and forecolors, so that may be a useful addition to the theme engine.

Looks like changing the backcolor doesn't do anything to the ToolStripSeparator.


add entity tag support (WIP)

PR for #118 (WIP)
@LobbyDivinus : this is a very quick draft from my side. Basically, I just added two additional entity extension functions which should be sufficient to support tags. You can see the usage in the Test class.
The downside of this approach is that tag ids are part of component ids, meaning that they will be part of the normal component mask and "waste" space in the BitArray.
I haven't had a look at your fork yet. Will do that tomorrow!

Looks great for a quick draft!
One thing I'm concerned about is type safety. There's no difference between regular components and tags which could prevent you from doing bad things.
For example you could accidentally add a Tag object to an entity, then unset that tag and by doing so cause an invalid state where the ComponentsHolder has the Tag object but the entity's mask does not signal having it.

@LobbyDivinus: I updated the draft now. Documentation is still missing. It is now possible to also use enums as tags (thank you for your fork, that gave me the idea on how to solve it ;) ). And I also included typealias for tags to make the code easier to read and understand imo.
About type-safety: I had a similar concern and tried a few things but I think we are lucky. You get compile errors when you do something wrong because a tag does not extend Component.  It does not even interact with a ComponentsHolder. It is just a simple bit that is set in the compMask BitArray.
Here is the current test that shows the syntax and compile errors:
data object Visible : EntityTag()

class TestTagSystem(var ticks: Int = 0) : IteratingSystem(family { all(Visible) }) {
    override fun onTickEntity(entity: Entity) {
        ++ticks
    }
}

enum class TestTags : EntityTags by entityTagsOf() {
    PLAYER, COLLISION
}

class EntityTagTest {

    @Test
    fun testTagAssignment() {
        val world = configureWorld {}
        val entity = world.entity { it += Visible }

        with(world) {
            assertTrue(entity[Visible])
//             entity.getOrNull(Visible) // <- compile error because Visible does not extend Component
//             val cmp : Visible = entity[Visible] // <- compile error because Visible does not extend Component and therefore 'cmp' must be of type Boolean
//             assertTrue(Visible in entity) // <- compile error because Visible does not extend Component
//             assertTrue(entity has Visible) // <- compile error because Visible does not extend Component
//             assertTrue(entity hasNo  Visible) // <- compile error because Visible does not extend Component

            entity.configure {
                it -= Visible
            }

            assertFalse(entity[Visible])
        }
    }

    @Test
    fun testTagSystem() {
        lateinit var testSystem: TestTagSystem
        val world = configureWorld {
            systems {
                add(TestTagSystem().also { testSystem = it })
            }
        }
        val entity = world.entity { it += Visible }

        world.update(1f)
        assertEquals(1, testSystem.ticks)

        testSystem.ticks = 0
        with(world) { entity.configure { it -= Visible } }
        world.update(1f)
        assertEquals(0, testSystem.ticks)
    }

    @Test
    fun testEnumTags() {
        assertNotEquals(TestTags.PLAYER.id, TestTags.COLLISION.id)
        assertNotEquals(Visible.id, TestTags.PLAYER.id)
        assertNotEquals(Visible.id, TestTags.COLLISION.id)

        val world = configureWorld {}
        val entity = world.entity { it += TestTags.PLAYER }

        with(world) {
            assertTrue(entity[TestTags.PLAYER])

            entity.configure { it -= TestTags.PLAYER }

            assertFalse(entity[TestTags.PLAYER])
        }
    }
}

Ah, with that forget about my concerns. This looks quite polished and covers all of what I'd expect tags to do.

Wow, that took quite some time to verify it in my three example games because a lot of things got actually broken haha. Luckily I drank an energy drink today so going the extra mile for my community and work until midnight was not an issue!
To summarize it:

entity removal was broken when an entity had a tag (IndexOutOfBoundsException in bag). I fixed it by adding a new clearAndForEachBitset function to the BitArray
enum class tags could not be used to create families
snapshots did not support tags at all. To solve it, I now added a new Snapshot class that contains the components and tags of an entity. For tags this was a little bit tricky because there is no ComponentHolder for a tag and I did not want to introduce one because then we could really just stick to components. So I solved it by adding a tagCache to the world and every time a tag gets added to an entity then it is also added to the cache. That way I can return the correct tag instance for a snapshot. I think this should work because if you never assign a tag to an entity then its bit in the component mask should never be set and therefore it should never be part of a snapshot and so it is okay if it is missing from the cache.

@LobbyDivinus: Would be great if you can check my changes and review them and ideally also try this branch in your local game and experiment a little bit with it.

Thank you, that was quick it looks great so far! I will report back once I've had a chance to play with it more thoroughly. I don't use snapshotting though so I wouldn't be able to notice issues there.

Sorry for the late reply, I did play a bit more with it and integrated it into may project and it all works as expected. There seems to be no noticable difference in performance but definitely in memory use. Thanks again for adding this one.

No worries, I just wanted to merge it today to have it officially in the snapshot version.
I want to release 2.5 when Kotlin 1.9.20 is out for a week or two. Then this feature will be officially in Fleks stable versions 🙂
Thanks again for the suggestion and help with the enum class tags!

Sounds great. I really appreciate this extension and the fast way you respond to feature requests.


[regression]Play Music, Play Books, Drive, Photos, and Maps cannot find Google account

0.2.6 worked perfectly, but with 0.2.7 they all fail to discover the Google account and insist I log in.
I can't find any errors  in the logcat, which might mame this harder to debug

Google Fotos doesn't need EXTENDED_ACCESS permission, I've just checked it.

@oldosfan: If you have a problem with this app please post a logcat.

Sure :) Freshly baked logcat coming up!

It may have been already fixed by 4713797edc21e3f3b2a26194056f63b307e67ef6 (which isn't included in the latest release).
I suggest to wait the new version or compile it yourself to test.

Sure, I'll set up a GitLab CI/CD instance to build hourlies and test

Given your changed description (which doesn't make sense, as it now is basically a completely different issue) this is now a duplicate of #790 and your sarcasm aside: it has been tested that the fixes not yet released have/will solve your issue.

Sure! Thanks for the tip;
TL; DR it was really late last night and I had time to test hourlies. Apparently you were right; Thanks!
However, Play Console still fails to log in. Perhaps it needs EXTENDED_ACCESS?


Crew overview page

From the 'all crew' page following this design - https://mongodb.invisionapp.com/share/2SWVCLE8WJG#/screens/413644457 - when you click on an individual record you get to the crew overview page which looks like this:
https://mongodb.invisionapp.com/share/2SWVCLE8WJG#/screens/413646957 -
The most recent picture of the crew member appears next to their name at the top.
Blocks exist for associated Vessels; License Numbers; Boardings; Violations; Photos; and Notes.
This ticket covers the functionality of this page - note that NONE of the 'See All' links or "View Boarding" buttons need to work for this ticket to be considered complete.
Each block has a count of the total number of items in the upper right hand corner.
For Vessels and License Numbers list the most recent 3 associations - they should be DIFFERENT (the design shows the same 3 vessels but it should show 3 unique vessels - unique by name and permit number) If the count is >3 then a 'See All' link should appear but it does not need to be functional right now.
Boardings should show the 5 most recent boardings (not 6 like the design shows) with the fields as shown; 'See All' appears if there are more than 5 boardings but does not have to work yet.
The Violations section should have the 5 most recent Violations with the information shown and a 'See All' link if there are more. 'See All' and the 'View Boarding' button don't need to work yet.
The most recent 12 photos should appear in Photos with a 'See All' link if there are more than 12. The most recent 5 notes (not 6 like in the design) should be in the Notes block with a 'See All' link if there are more than 5.

In scope of 8/18 milestone:

crew overview redesign is done;
page is switched to DB (real data added)

In scope of 8/25 milestone we plan to:

add redirecting links (e.g. 'See all' link in blocks)
add necessary pages (as it shown in newest designs such as - https://mongodb.invisionapp.com/share/VJXLVXNPGW4#/screens/427645091)


No autocompletion on neovim with nvim-lspconfig

I've tried to use crystalline multiple times on neovim with nvim-lspconfig, and I get no autocomplete. Is there any information on troubleshooting crystalline?

Also there's even more weird behavior as this LSP delete some of my code on save when save-on-format is enabled with some other plugins on neovim.

Here's a tmp fix for linting, I'm still having issues with auto formatting, though.
lspconfig.crystalline.setup {
cmd = { "crystalline", "--stdio" },
}

Hi I have the same problem with lsp based autoformat so I start using vim-crystal for auto format:
-- turn on auto format with vim-crystal:
vim.g.crystal_auto_format = 1

add vim-crystal in lazy:
{
    'vim-crystal/vim-crystal',
},

and disable lsp_fallaback for crystal files (code form modified kickstart.nvim):
  { -- Autoformat
    'stevearc/conform.nvim',
    lazy = false,
    keys = {
      {
        '<leader>f',
        function()
          require('conform').format { async = true, lsp_fallback = true }
        end,
        mode = '',
        desc = '[F]ormat buffer',
      },
    },
    opts = {
      notify_on_error = false,
      format_on_save = function(bufnr)
        -- Disable "format_on_save lsp_fallback" for languages that don't
        -- have a well standardized coding style. You can add additional
        -- languages here or re-enable it for the disabled ones.
        local disable_filetypes = { c = true, cpp = true, crystal = true }
        return {
          timeout_ms = 500,
          lsp_fallback = not disable_filetypes[vim.bo[bufnr].filetype],
        }
      end,
      formatters_by_ft = {
        lua = { 'stylua' },
        -- Conform can also run multiple formatters sequentially
        -- python = { "isort", "black" },
        --
        -- You can use a sub-list to tell conform to run *until* a formatter
        -- is found.
        -- javascript = { { "prettierd", "prettier" } },
      },
    },
  },

Hello, thanks, that works now. Only thing that is broken is autocompletion now.

Hello, thanks, that works now. Only thing that is broken is autocompletion now.
EDIT: it looks like we actually do have auto completion. It's just the quality of autocompletion is pretty bad.

It seems to work in the main source file but in others it works quite randomly


fix(monitor): always use crtc monitor, even when WITH_XRANDR_MONITORS is enabled

Fixes #1437.
This works on my setup, but I don't have any other computers to test on. This should work given that the crtc values are generally the same as the monitor values, but someone might have some funky setup where it doesn't.

Wait... this conflicts with #1426, need to fix that first.

Yeah, I was wondering about that.
Also if we remove the #if WITH_XRANDR_MONITORS blocks, the only thing WITH_XRANDR_MONITORS does is to change the output of polybar -m

I have been doing some digging, and unfortunately WITH_XRANDR_MONITORS is here to stay. I have discovered this issue https://github.com/i3/i3/issues/1799 that describes that "monitors" are a new concept in randr and are different from "outputs" (The naming is just spectacular). They also listed this article, which may give some more info, but I haven't read it yet.
"outputs" are what xrandr will give you while xrandr --listmonitors will print the "monitors". On my system those will give about the same information, but the new monitor concept can be used to create two monitors inside a single output. When splitting an output in two like this, this patch will still only recognize the outputs (what xrandr outputs), while WITH_XRANDR_MONITORS will detect both monitors (what xrandr --listmonitors outputs).
I was using this answer to setup split outputs myself. The commands I used were:
xrandr --setmonitor H1 960/598x1080/336+0+0 HDMI-0
xrandr --setmonitor H2 960/598x1080/336+960+0 none

Where HDMI-0 is the name of my output and H1 and H2 are the names for the newly created monitors. The dimensions of HMDI-0 is 1920x1080+0+0, so this splits that in half. The values after the slash seem to be the physical size in mm.
With this setup, polybar -m gives me
HDMI-0: 1920x1080+0+0
H2: 960x1080+960+0 (XRandR monitor)
H1: 960x1080+0+0 (XRandR monitor)
DisplayPort-0: 1920x1080+1920+0 (primary)

while this patch gives me
HDMI-0: 1920x1080+0+0
DisplayPort-0: 1920x1080+1920+0 (primary)

For comparison, this is the output of xrandr --listmonitors:
Monitors: 3
 0: +*DisplayPort-0 1920/598x1080/336+1920+0  DisplayPort-0
 1: H1 960/598x1080/336+0+0  HDMI-0
 2: H2 960/598x1080/336+960+0

I would argue, neither the current approach, no this PR handle this correctly. What i3 does, is to completely rely on "monitors" (conn.get_monitors) when RandR 1.5 is used and fall back to the output based approach on lower versions. What we currently do is to always use the outputs and also include "monitors" when RandR 1.5 is used. Switching to the i3 approach is probably the best idea.
I was not able to reproduce the issue in #1437, once I do, I think we will be able to both fix the issue and properly support randr monitors.

Closing this.
We now have better randr monitor support. Don't know if the linked issue is fixed though, I have not been able to reproduce it yet.


command-r plus config

I see you have max_position_embeddings as 8192, but model_max_length at 131072.
To fit in my current vRam I had to set model_max_length at 32768.
Should I leave max position embeddings at 8192?
thanks for all your work, AND for the command-r quant!

You'd typically specify the max length in whatever UI/API you're using anyway. But if you want to change the config.json file, ExLlama will ignore max_position_embeddings when model_max_length is present.
The value only specifies what the default context length is, and it can be used by backends to determine when they should use alpha scaling to accommodate more context than the model can natively handle. To account for the variety of models out there, each with their own unique take on what the standard is, ExLlama uses following config keys, in order of priority:

max_sequence_length
model_max_length
max_position_embeddings
max_seq_len

ExLlama uses following config keys, in order of priority:
* `max_sequence_length`

* `model_max_length`

* `max_position_embeddings`

* `max_seq_len`

If this isn't stated in the documentation, it should be.

The thing is that this changes with every new architecture. model_max_length was only just introduced by cmdr+, and the reason it takes priority over max_position_embeddings is that Cohere decided to first release the model with the latter key and a value of 8192 (for whatever reason; it's a 128k model), then adding model_max_length in a PR afterwards.
max_seq_len was added to the list when DBRX came out. DBRX also does a number of other things differently like moving the rope_theta key into the attn_config section. Basically every new architecture that comes out is implemented in custom code that's free to define the config.json format however it wants, and all a framework like ExLlama can do is try to copy whatever weird and wonderful decisions they make in that regard.
So there is no standard at all really, and there's no documentation I could write for how to hack the config.json that wouldn't have to be constantly revised. ExLlama instead parses the file into a standard class, ExLlamaV2Config, where the max sequence length is always defined by ExLlamaV2Config.max_seq_len.

I totally get the challenge in keeping what is what separated and in a semi-coherent order of priority.
Now imagine the confusion for us poor schmucks playing with LLMs on and off throughout the week.
Being explicit about these things either in Documentation or in console output may go a long way to make the fog of terminology overload slightly more penetrable  :-)
If I haven't thanked you for Exllama and ExUI already: Thank You!


Add Travis CI badge

@nicolas-raoul: Please log in to Travis CI and activate https://travis-ci.org/nicolas-raoul/apps-android-commons . And then we can use the image file of the badge in the README file.
Image file: https://api.travis-ci.org/nicolas-raoul/apps-android-commons.svg
You could use https://api.travis-ci.org/whym/apps-android-commons.svg but I cannot guarantee that my fork is always up-to-date.

I activated :-)


sqleet v1 cryptosystem

A draft of a new cryptosystem for the next major version of sqleet. The current sqleet v0 cryptosystem is strong and the goal here is improving secure enclave (TPM/SGX/TrustZone) support and reducing the security impact of nonce reuse attacks (if an attacker obtains a Poly1305 key of a single page somehow, the attacker can reuse the key and nonce to compromise the security of other database pages). See the description below for more details.
Warning: This is just a draft and subject to changes. I'm still in the process of getting the proposed cryptosystem verified by "professionals", so some changes are expected.
Encrypted database page format
==============================

  +----------------------------------------+----------------+----------------+
  | (n - 16 - 16)-byte Page data           | 16-byte Nonce  | 16-byte MAC    |
  +----------------------------------------+----------------+----------------+
     n = 512, 1024, 2048, ... (page size)  | Reserved area of 32 bytes total |

Old implementation (sqleet v0.x.y)
==================================

Basic ChaCha20-Poly1305 AEAD construction using 128-bit random nonces.
Basically, RFC7539 with minor modifications

    EncryptV0(K, PageNumber, Data[0..n])
    |
    | Nonce = randombytes(16)
    | K_stream, K_mac = ChaCha20(K, Nonce ^ PageNumber)
    | Data ^= ChaCha20(K_stream, (Nonce ^ PageNumber) + 1)
    | MAC = Poly1305(K_mac, Data || Nonce)
    |
    | Output: (Data[0..n-32], Nonce[0..16], MAC[0..16])

    DecryptV0(K, PageNumber, Data[0..n-32], Nonce[0..16], MAC[0..16])
    |
    | K_stream, K_mac = ChaCha20(K, Nonce ^ PageNumber)
    | Verify MAC = Poly1305(K_mac, Data || Nonce)
    | Data ^= ChaCha20(K_stream, (Nonce ^ PageNumber) + 1)
    |
    | Output: Data[0..n-32]

Run-time cost: (n + 1) × ChaCha20 + n × Poly1305 blocks in total.

List of known (theoretical) issues:

 1. Random nonces allow nonce reuse attacks

    - Preventing nonce reuses is infeasible, in practice, with random nonces.

    - Exploitation can be detected by checking the nonce values of a database
      file. Non-malicious nonces are uniform-randomly distributed.

 2. C implementations of Poly1305/ChaCha20 may leak one-time K_mac & K_stream
    keys onto the stack. ChaCha20(K, Nonce) cannot leak master key K because the
    following ChaCha20(K_stream, ...) overwrites stack memory and registers.

 3. If Poly1305 leaks K_mac (by whatever means; let's say via a side channel,
    bad RNG or direct process memory inspection), then the attacker can generate
    valid Poly1305 MACs for arbitrary messages by reusing the nonce and K_mac.

    - The ability to produce valid MACs allows bit-flipping bits in plaintext
      by changing the corresponding ciphertext ("stream cipher attack").

    - K_mac is not properly page-specific to prevent nonce & K_mac reuse across
      database pages. Let's force page j to reuse a leaked K_mac from page i:

      +----------------------------------------------------------------------+
      | Page i with nonce N:                                                 |
      | K_stream, K_mac = ChaCha20(K, N ^ i)                                 |
      +----------------------------------------------------------------------+
      | Page j with nonce N ^ i ^ j:                                         |
      | K_stream, K_mac = ChaCha20(K, (N ^ i ^ j) ^ j) = ChaCha20(K, N ^ i)  |
      +----------------------------------------------------------------------+

      In other words, the nonce N and K_mac of page i can be reused at page j by
      with the adjusted nonce value N ^ i ^ j. Notice K_stream is reused also.

    - In conclusion, a single *one-time* key K_mac is sufficient to break the
      whole cryptosystem. This is poor design, despite the fact that attacker
      obtaining K_mac is only a theoretical threat and a very unlikely attack
      scenario in practice. Requires Heartbleed type of a information disclosure
      vulnerability but in stack memory (more powerful exploits such as code
      execution exploits can already bypass the encryption by other means).

    - K_stream, if leaked, compromises only the corresponding database page.
      This is correct and expected behavior since *one-time* keys are limited
      and not intended to be powerful enough to break the full cryptosystem.

    - K (master key) leak would be devastating to the cryptosystem, obviously.
      There is nothing we can do about this, given the definition of master key.

  4. ChaCha20 streams of distinct pages overlap (with negligible probability) if
     randomly generated nonces are within distance (page_size / 64) - 1 from
     each other. XOR between the ciphertexts encrypted with the overlapping
     stream segment reveals the XOR of the associated plaintexts, breaking the
     confidentiality of data.

New cryptosystem (sqleet v1.x.y)
================================

ChaCha20-Poly1305 AEAD construction with deterministic nonces. Draft!

    EncryptV1(K, PageNumber, Data[0..n]):
    |
    | K_i, K_j = ChaCha20(K, PageNumber)
    | Hash = Poly1305(K_i, Data)
    | Nonce = Hash ^ randombytes(16)
    | K_stream, K_mac = ChaCha20(K_j, Nonce)
    | Data ^= ChaCha20(K_stream ^ K_i, 0)
    | MAC = Poly1305(K_mac, Hash)
    |
    | Output: (Data[0..n-32], Nonce[0..16], MAC[0..16])

    DecryptV1(K, PageNumber, Data[0..n-32], Nonce[0..16], MAC[0..16])
    |
    | K_i, K_j = ChaCha20(K, PageNumber)
    | K_stream, K_mac = ChaCha20(K_j, Nonce)
    | Data ^= ChaCha20(K_stream ^ K_i, 0)
    | Hash = Poly1305(K_i, Data)
    | Verify MAC = Poly1305(K_mac, Hash)
    |
    | Output: Data[0..n-32]

Run-time cost: (n + 2) × ChaCha20 + (n + 1) × Poly1305 blocks in total.

Improvements:

 1. Page subkeys K_i, K_j allow a secure enclave implementation with SGX/TPM/TZ
    technologies (derive subkeys from K within a hardware-enforced safe space).
    Probably not going to be implemented in sqleet, but a nice option to have.

 2. Deterministic hash-based nonces prevents nonce reuse attacks.

 3. One-time keys K_stream & K_mac are truly one-time and, if leaked, compromise
    only the associated page and nonce/data combination.

 4. C implementations (most likely) do not leak master key or subkeys via stack.
    Subsequent ChaCha20/Poly1305 function calls overwrite secrets spilled to the
    stack before exiting the function. Only K_stream & K_mac can be exposed, but
    they are truly one-time keys and compromise only the current page and data.

 5. Power/EM side-channel resistance (in addition to the usual time and cache
    side-channel resistance). Published side-channel attacks against ChaCha20
    require an attacker-controlled nonce, making ChaCha(K_j, Nonce) vulnerable.
    To mitigate the security impact of leaked K_j, we use K_stream ^ K_i as the
    ChaCha20 encryption key (if the attacker knows K_j, she can also calculate
    K_stream). Power/EM side-channel attacks do not expose K_i because it is
    derived from K with non-attacker-controlled nonce (PageNumber) and never
    used with attacker-controlled nonce.

    Power/EM attacks are not my area of expertise, so the reasoning above may be
    flawed. The biggest assumption is that Poly1305 is secure against power and
    EM side-channel attacks, which is true to my limited understanding.

 5. MAC-then-Encrypt is fine in this particular context.

Disadvantages (in comparison to old version):

 1. Requires additional ChaCha20 and Poly1305 block computations.

 2. Complexity. Ugly. Not battletested nor approved by crypto wizards (yet).

Any feedback is appreciated.

PBKDF2-HMAC-SHA256 will be replaced by Balloon hashing with BLAKE2s or BLAKE3 hash function.

Balloon hashing is a elegant memory-hard key derivation algorithm with the same or superior security properties as Argon2; for example, the memory-hardness of Balloon hashing has been proven while the memory-hardness of Argon2 remains )unproven.

    EncryptV1(K, PageNumber, Data[0..n]):
    |
    | K_i, K_j = ChaCha20(K, PageNumber)
    | Hash = Poly1305(K_i, Data)
    | Nonce = Hash ^ randombytes(16)
    | K_stream, K_mac = ChaCha20(K_j, Nonce)
    | Data ^= ChaCha20(K_stream ^ K_i, 0)
    | MAC = Poly1305(K_mac, Hash)
    |
    | Output: (Data[0..n-32], Nonce[0..16], MAC[0..16])

Notice that Hash = Poly1305(K_i, Data) is dangerous because reusing a Poly1305 key for distinct messages leaks the Poly1305 key. With broken RNG, Nonce = Hash ^ randombytes(16) = Hash ^ 0  yields nonce Poly1305(K_i, Data). Therefore, an attacker can recover K_i after observing 2 nonces for the same page assuming a known-plaintext attack.
Fortunately, leaked K_i is not useful to the attacker because ChaCha20(K_stream ^ K_i, 0) XORs K_i with K_stream before use. Leaking K_i is still unacceptable though, because the attacker may be able to reveal K_stream with another attack (e.g., power/EM side-channel attack to find K_j and consequently K_stream).
The problem can be addressed by computing Hash = H(Data) with regular hash function like BLAKE2 or SHA-256. Alternatively, Hash = H(Poly1305(K_i, Data)) can be used as a optimization (Poly1305 is fast). Or simply fix the RNG...

Codec API deprecation makes deterministic counter-based (instead of hash-based) feasible if switching to VFS-level encryption scheme. I think the proposed v1 crypto should be revised to exploit persistent counter/state to simplify and improve the crypto, for example, by avoiding random number generation altogether.

Using counter-based nonces sounds good, and would allow to get rid of problematic RNG implementations.
BTW, I have now a first implementation of VFS-level encryption ready for testing (see SQLite3 Multiple Ciphers).

Closing this issue. A redesign is needed to fully exploit the new VFS-based architecture.


Update jquery to @3

I found an error occurred on /me screen.
https://crowi-dev-pr-292.herokuapp.com/me
me:288 Uncaught ReferenceError: $ is not defined
    at me:288


Ability to invalidate tokens in the wild.

This feature would add the ability to invalidate any existing JWT tokens by adding a new attribute in the JWT payload to contain a token_version that can be changed by the server for any reason.
DJA would use this feature when a user resets their password or logs out. Any developer using this library could do this manually buy calling something like user.invalidate_tokens!. To implement this, a new column in the resource table would need to be added: token_version. Then when validating a user with the JWT token, the user would need to be found not only by uid but also by token_version. If the token_version doesn't match, the request to log in isn't valid either.
Maybe in the future a client column of type json could be used to keep track of a client's token_version so that a particular client could be logged out or invalidated. Then, if a user resets their password something like user.invalidate_all_tokens! could be called which invalidate every client?

More info regarding managing per client tokens...
Some fields for each client that might be useful for security purposes:

IP Address of each client
Client label like "Home" and possibly some UserAgent information like what type of OS or Device.
Last time user logged in from that client
etc... stuff that I often see from other services.

This information could then be retrieved through a new REST endpoint. The ability to invalidate tokens for a particular client or for all clients would be especially useful.


Can't able to download any extenstions

Issue Type: Feature Request
I am getting "We cannot connect to the Extenstions Marketplace at this time , please try again" . I tried to download the updated version but still from past 6 7 months it's giving me this warning and i am not able to install any extestion
VS Code version: Code 1.58.2 (c3f126316369cd610563c75b1b1725e0679adfb3, 2021-07-14T22:10:15.214Z)
OS version: Windows_NT x64 10.0.18363
Restricted Mode: No

No action planned on the marketplace side for this. Please configure your proxies and use stack overflow for questions.


Feat: Spanish Version

What does this PR do?

[x] Init repo with good structure
[x] Translate _toctree.yml and to index.md
[ ] Translate all pages

Fixes # (issue)
Who can review?
Feel free to tag members/contributors who may be interested in your PR.

@josermarinr can you find another Spanish speaker to review your PR? :)

Hi @merveenoyan @josermarinr, I could review it in Spanish if still needed 😄.

@sergiopaniego thanks a lot for reviewing 💗💗

@josermarinr if you could address @sergiopaniego's comments we can merge :)


Sequence prediction for many to one model

I am trying to find the output value for a sequence of numbers ranging from 1-5(5 timesteps). I am not sure of which loss function to use here during model.compile?

Hi
This issue isn't related to a bug/enhancement/feature-request or other accepted types of issue.
To ask questions, please see the following resources :
https://keras.io/
https://keras.io/#support
https://gitter.im/Keras-io/Lobby
Thanks!


markers edition

Hi,
Seems I can't edit color of a marker when selected. Same on your [popup] (http://leaflet.github.io/Leaflet.toolbar/examples/popup.html) example.
Do you think possible to make a list of 4 other markers in place changing his color ?

Michael, thanks for pointing this out.
I left it this way mostly because I was lazy. Leaflet markers, unlike other
layers are based on images, and so there's no (easy) way to change the
color of a marker using JavaScript alone, as with polylines, polygons, etc.
There are two good solutions to this:

Use a library (such as font-awesome-markers) that provides markers of
different colors.

Only show the "Edit Color" option when the layer that the toolbar is
being attached to is not an instance of L.Marker.

I'm particularly interested in solution #2, because I think that it touches
on an important use case. Any situation where the actions you want to
expose to the user are dependent on the target object might fit this model.
For example - users might have read privileges to all objects on a map, but
they might only have edit privileges to those they created.
I experimented with a 'filter' option to Leaflet.Toolbar before, but didn't
follow through. I think this is the perfect use-case for such an option!
On Mar 18, 2015 9:30 AM, "Michaël Meinertzhagen"<EMAIL_ADDRESS>wrote:

Hi,
Seems I can't edit color of a marker when selected. Same on your popup
http://leaflet.github.io/Leaflet.toolbar/examples/popup.html example.
Do you think possible to make a list of 4 other markers in place changing
his color ?
—
Reply to this email directly or view it on GitHub
https://github.com/Leaflet/Leaflet.toolbar/issues/25.

Hi @manleyjster, thank you for your answer. I'm currently learning JS, thanks to [CodeAcademy] (http://www.codecademy.com/) and other ressources. This will help me to participate in projects like yours.


Real proactivity

Hello, this example obviously shows "proactivity" in terms of running fake background jobs and replying asynchronously almost "right away".
This unfortunately does not show how to trigger the bot remotely and have it say something to the user.
Example use cases :

notify users who have requested for it that breakfast is ready ;
ask users what they would fancy doing today.
What should the right implementation strategies be for that:

use the createConversation API ? (if a real conversation has to be started, because the client will be asked to interact eg. "hey, what about jogging today, or do you have a better idea to let me propose you every morning ?")?
POST to the /api/messages ? (if the client just has to be notified of something eg. "breakfast is ready!")?
The trigger would come from, say, a remote REST client with a business logic payload (ie. recipe of the breakfast that is ready, to have it shown to the client), but the Bot code has to know to whom (which client) to redirect the messages.
Thanks for your help!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0fc28a40-716e-22bc-8944-15f0a8b0d5c2
Version Independent ID: 86252b0d-94e2-2ea6-ed17-fc32ae38e6fc
Content: How to use proactive messaging - Bot Service
Content Source: articles/v4sdk/bot-builder-howto-proactive-message.md
Product: bot-framework
GitHub Login: @JonathanFingold
Microsoft Alias: jonathanfingold

This scenario requires an additional app or service. The serverice generates the event activity, which it sends to the bot's api/messages endpoint, and the bot then sends the notice (proactive message) to the user. The event activity that the service generatates needs to contain the following information:

An event name, unique to your app
A conversation reference for the user, such as the one in which the user subscribed to the notices.
Enough information for the bot to construct an appropriate notice.

The first item is the event activity's name property.
Include the remaining information in the event activity's value property.
The code to have the bot generate a proactive message when it receives a message activity is only for demonstration and testing purposes.
Let me know if that helps or if you're still blocked.

One other note about this. The bot will send the proactive message right after it receives the event activity from the service, so the service should send the event at an appropriate time.

@myselfhimself , I'll close this for now. If you have more questions, feel free to reopen this or open a new issue.

Is there a way to create a bot adapter from code and use that to send the message?
When you have a conversation reference, how can you send a message without having a turncontext/botadapter?

That's how I worked it out:

The outer side (that triggers the proactive dialog) should know your bot's id and client secret to begin conversation with your bot. As written in the Authentication part of wiki, you need to generate the token by sending the POST request to  https://login.microsoftonline.com/botframework.com/oauth2/v2.0/token with difference that the scope parameter should be {your_bot_app_id}/.default

Then you need to send the POST request to api/messages route of your bot, with Authorization: Bearer {your_token} header. The JSON structure of the message you can look within Bot Framework Emulator or in documentation.

The rest you can know out by placing the breakpoint in OnTurnAsync method and looking inside context.Activity object of your bot while sending him messages first from the messenger (Skype/Telegram/Facebook), then from your own application. The key parameters are replyToId and serviceUrl. They should be the same as in the message where user started the dialog.


Url encoded by URLSearchParams

Hey!
I opened this issues because I ran into a small hick.
URLSearchParams encodes the query string by default, so this:
date=12/04/2018&other=1,2,3,4  Becomes this:  date=12%2F04%2F2018&other=1%2C2%2C3%2C
For my case, I was thinking of being able to configure this as well, since I don't want encoded url to show as much as possible in the address bar.
However, a bigger issue is this line:
https://github.com/Treora/redux-query-sync/blob/master/src/redux-query-sync.js#L132
This comparison will fail, even if the query string did not change. Because the value in oldLocationSearchString can be a decoded string and the newLocationSearchString will always be an encoded one.
I'm thinking if just always decoding the string when getting it from URLSearchParams would be a viable option? Or just encoding both and comparing them.
In my case I don't actually want the parser to replace by decoded string for an encoded one unecessarily. Which is what is happening as of now.
Cheers!

I think regarless how the browser shows the string as long as the encoded string matches the decoded version, it works all the same. I've seen emojis in the some other urls before I was just as confused..
Since URLSearchParams is being used, makes sense to keep it encoded.
The .toString() method for of URLSearchParams instance, definitely encodes the string.
https://github.com/WebReflection/url-search-params/blob/master/src/url-search-params.js#L137
However there is some replacing going on in some characters.
So I think for this case the line should be ?${new URLSearchParams(oldLocationSearchString)} to make sure both strins go trough the same transformation before comparing them.


[occm] Implement container-store configuration setting

What this PR does / why we need it:
For heavely customized Openstack installation it may be helpful to disable tls-container-ref format validation.
Official terraform-provider-openstack and python-octaviaclient do not have such validation mechanism
Now we have container-store parameter in cloud.conf which can be one of two values:

barbican - Default. tls-container-ref format validation enabled
external - tls-container-ref format validation disabled

Which issue this PR fixes(if applicable):
fixes #1929
Special notes for reviewers:
If container-store set to external content of tls-container-ref parameter or annotation can be any string.
Release note:

NONE

/easycla

Messed that up, closing


[CI] Need automate version bump PR to be opened once OpenSearch/Dashboards release new versions

As of now, we are relying on human efforts to bump OS/OSD versions once there is a new release.
This could delay the release on these platforms and could add to human errors.
We need a github action to automatically open a PR after detecting the new releases.

I am still of the opinion to have release branches for charts 😄
The maintainers can branch out from main and have releases enabled only on branches apart from main.
I

renovate can so this in an automated way 🤗


Design Initial Database Schema

Initial database schema proposed as part of #11.

Current schema at https://github.com/treadmill-tb/treadmill/blob/main/switchboard/sql/SCHEMA.sql.

After meeting today (attendence: @lschuermann, @charles37, and me), we arrived at the following schema:

```sql
--
-- Switchboard database schema.
--
-- CORE STRUCTURE ---------------- ---------------- ---------------- ---------------- ---------------- ----------------
drop table if exists job_results;
drop table if exists jobs;
drop type if exists job_parameter;
drop table if exists user_sessions;
drop table if exists user_presessions;
drop type if exists fingerprint;
-- drop table if exists maintenance_schedule;
drop table if exists supervisors;
drop table if exists api_token_privileges;
drop table if exists user_privileges;
drop table if exists api_tokens;
drop type if exists cancellation;
drop table if exists users;
drop table if exists events;
-- EVENT LOGGING
create table events
(
event_id uuid  not null primary key,
data     jsonb not null
);
-- BASE ENTITIES
create type user_type as enum ('normal', 'system');
create table users
(
user_id       uuid                     not null primary key,
name          text                     not null unique,
email         text                     not null unique,
password_hash text                     not null,
user_type     user_type                not null,

created_at    timestamp with time zone not null,
last_login_at timestamp with time zone not null,
locked        bool                     not null

);
create type api_token_cancellation as
(
canceled_at         timestamp with time zone,
cancellation_reason text
);
create table api_tokens
(
-- internal identifier of the token
token_id                  uuid                     not null primary key,
-- actual contents of the token; random nonce
token                     bytea                    not null unique,
-- the user who created this token
user_id                   uuid                     not null references users on delete cascade,
inherits_user_permissions bool                     not null,
-- whether the token was revoked/invalidated/etc. or not
canceled                  api_token_cancellation,
-- expiration date
created_at                timestamp with time zone not null,
expires_at                timestamp with time zone not null
);
-- PERMISSIONS
create table user_privileges
(
user_id    uuid not null references users on delete cascade,
permission text not null,
unique (user_id, permission)

);
create table api_token_privileges
(
token_id   uuid not null references api_tokens on delete cascade,
permission text not null,
unique (token_id, permission)

);
-- SUPERVISORS
create table supervisors
(
supervisor_id     uuid                     not null primary key,
name              text                     not null,
last_connected_at timestamp with time zone not null,

public_key        text                     not null,

tags              text[]                   not null

);
-- BUSINESS LOGIC ---------------- ---------------- ---------------- ---------------- ---------------- ----------------
create type job_parameter as
(
-- k      text,
value  text,
secret bool
);
create table job_parameters
(
job_id    uuid          not null references jobs (job_id) on delete cascade,
key       text          not null,
parameter job_parameter not null,
primary key (job_id, key)

);
create type restart_policy as
(
restart_count integer
);
create table images
(
image_id uuid not null primary key
);
create table jobs
(
job_id               uuid           not null primary key,
resume_job_id        uuid references jobs (job_id) on delete no action,
image_id             uuid references images (image_id) on delete no action,
ssh_keys             text[]         not null,
-- run_command          text,
restart_policy       restart_policy not null,

queued               bool           not null default true,

enqueued_by_user_id  uuid references users (user_id) on delete no action,
enqueued_by_token_id uuid references api_tokens (token_id) on delete no action,

check
    ( (enqueued_by_user_id is not null and enqueued_by_token_id is null)
    or (enqueued_by_user_id is null and enqueued_by_token_id is not null)),

tag_config           text           not null
    check
        ( (resume_job_id is null and image_id is not null)
        or (resume_job_id is not null and image_id is null))

);
create type exit_status as enum (
-- no supervisors in "supervisors" that can run this that you have permissions for
'failed_to_match',
-- too long in queue
'queue_timeout',
-- failed to launch this on a supervisor (may incur a retry)
'host_start_failure', -- e.g., supervisor timeout, image failed to fetch,
-- host reported an error (i.e., through puppet)
'host_terminated_with_error',
-- host reported success
'host_terminated_with_success',
-- job timeout
'host_terminated_timeout'
);
create table job_results
(
job_id        uuid                     not null references jobs (job_id),
supervisor_id uuid                     not null references supervisors (supervisor_id) on delete no action,
run_number    integer                  not null,
exit_status   exit_status              not null,
host_output   jsonb,

enqueued_at   timestamp with time zone not null,
started_at    timestamp with time zone not null,
terminated_at timestamp with time zone not null

);
</details>

Closing as #17 is merged.


feat: Implement Upper Footer Sections

Description
Hi @ashuvssut 👋🏻
What you are doing is really amazing! 💖
I would like to contribute to the project by adding this footer section.

If you could assign this to me, It would be really great. Thank you :smile

This part is already implemented. We have hidden it for reasons. Thanks for opening up the issues. You can contribute in #35 if you want!


Self-hosted runners

We need to support pythia on all the SFTPPlus v4 OSes.
My suggest is to keep the repo public but add private runners .... we will see if people will be tempted to create PR with malicious code... as a first steps, GitHub Actions can be configured to run actions for this repo only from GitHub and Chevah own org.
Originally posted by @adiroiban in https://github.com/chevah/pythia/issues/5#issuecomment-727624935

Relevant: https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#supported-operating-systems-for-self-hosted-runners
Notably missing:

RHEL/CentOS 8
Amazon Linux 2
CentOS 5 (for the generic x64 Linux build)
Alpine Linux.

Note that Ubuntu 16.04 on ARM64 (for the generic arm64 Linux build) should work, according to https://github.blog/changelog/2019-12-03-github-actions-self-hosted-runners-on-arm-architectures/.

I expect Centos8 and AMZ 2 will work as they are "modern" version on which .net core can run.
So for now, only add private self-hosted runners for Centos and AMZ2


Confirmation of tweets_by_ids endpoint rate limits

Hi Trevor,
Just wanted some clarity on the rate limits for tweets_by_ids. I see this on the chart:  tweets_by_ids: ~220 (batch size) 500 / 15 mins (rate limit). I just wanted to confirm that the 500/ 15 mins is referring to "500 batches of 220 tweet ids per 15 min" or in other words, a rate limit of around "110,000 tweet ids per 15 min". Is this assumption correct?
Thanks!

Hey Vishal, yes that's correct! It's very efficient and should always be preferred, depending on what data you're looking for.


Fish Spawn On Land??

Hey so I have a weird bug where some of the fish from this mod spawn on land. I am using the most recent 1.19.2 Forge version. I am, however, running a heavily modded instance so I'm not sure if it is just a mod interaction/mixin error that is causing this. I can provide more info if needed later.

This is also seeing to be an issue for me on my custom pack.
MC 1.19.2
Forge 43.1.52, and 43.1.53
Fish of Thieves version 2.1.0-rc3
This issue only happens when updating to RC3. When downgrading to version V2.1.0-RC2 the issue is fixed.
So something in version RC3 changed that causes fish to spawn in insane amount on land and just die from not being in the water.

Same thing with me. Using forge version 43.1.55 and 2.1.0 RC3 for Fish of Thieves version. Hopefully this gets fixed

Fixed in both 1.18.2 1.1.0-rc4 and 1.19.2 2.1.0-rc4.

Fixed in both 1.18.2 1.1.0-rc4 and 1.19.2 2.1.0-rc4.

FINALLY


API

@Santiagoestremadoyro, I've made some changes to your branch. Please pull the changes before you start working to avoid problems.
Thanks!


k8s clusters with big amount of IPs per node emit too much metrics

Slack ref: https://coroot-community.slack.com/archives/C0443R3BW2G/p1677599077846269
Since IPVS is used, all k8s service IPs are getting attached to kube-ipvs0 interface on each kubernetes node.
As there is always some services running in host network (ssh, kubelet, haproxy, etc) it also generates metric per each IP attached to that network interface.
5000 services = 5000 metrics for each service, on each node.
On cluster with ~30 nodes, with 3.5k pods it amounts to around 1.5m TS in container_net_tcp_listen_info metric.

Probably related: https://github.com/kubernetes/kubernetes/issues/72236

@rlex, please update coroot-node-agent to version 1.6.6.

Yep, this solved this issue with IPVS IPs


[SPARK-30962][DOC]Documentation for Alter table command phase 2

What changes were proposed in this pull request?

Why are the changes needed?

Based on JIRA 30962, we want to add all the support Alter Table syntax for V1 table.
Does this PR introduce any user-facing change?

Yes
How was this patch tested?

Before:
The documentation looks like
Alter Table
After:

@huaxingao @dilipbiswal

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #119271 has started for PR 27779 at commit 6f74d6c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24010/
Test PASSed.

Test build #119271 has finished for PR 27779 at commit 6f74d6c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119271/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24310/
Test PASSed.

Test build #119583 has started for PR 27779 at commit 1d7aa01.

Test build #119583 has finished for PR 27779 at commit 1d7aa01.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119583/
Test PASSed.

Test build #119595 has started for PR 27779 at commit 6c19a94.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24326/
Test PASSed.

Test build #119595 has finished for PR 27779 at commit 6c19a94.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119595/
Test PASSed.

Pending the fix: https://github.com/apache/spark/pull/27779#discussion_r390490103

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24368/
Test PASSed.

Thanks! Merged to master/branch-3.0. cc: @gatorsmile


[Doc] Mount client version to the latest release version

Search before asking

[X] I searched in the issues and found nothing similar.

What issue do you find in Pulsar docs?
According to https://github.com/apache/pulsar-site/pull/352#issuecomment-1368228919, we don't release any more new versions for C++ and Python clients. That means the @pulsar:version@ placeholder will generate incorrect results for the maintained versions (2.8~2.10).
What is your suggestion?
Since no more new versions will be released, we can replace the placeholder with resolved fixed versions.
Any reference?
No response
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

2.10.2, 2.9.4 and 2.8.4.
C++ RPM and DEB packages can be found here: https://archive.apache.org/dist/pulsar/
For example, you can found the DEBS/ and RPMS/ directories under https://archive.apache.org/dist/pulsar/pulsar-2.10.2/
Python wheels can be found here: https://pypi.org/manage/project/pulsar-client/releases/


libtermkey: fix patch

Hi @dantehaversham, the libtermkey build can fail on certain systems if the ncurses pkg-config file specifies some LDFLAGS (pkg-config --libs ncursesw: -Wl,-O1 -Wl,--as-needed -O3 ...), so an extra -Wl is added which causes the linker to error out. The solution is to modify the patch like this. The -rpath flag is not really needed here anyways. The patches can also be changed into a single small patch like here, the post-install file can be removed too.
Thanks.

This looks great, I'll go ahead ans open a pull request with these changes when I get home.

Ping @dantehaversham


Package LAYERS documentation and some code clean up notes

Two things on LAYERS

@rabernat , I think Jeff is getting close to having section "Packages II" into RTD. Is there any LAYERS documentation he can include?

Also there are a few pkg LAYERS code things that be great to clean up when someone has a moment.

https://github.com/MITgcm/MITgcm/blob/master/pkg/layers/layers_save.F contains a bunch of routines called LAYERS_FILL. On the whole we try and name files according to the function(s) they contain to make it easier to find things.

Generally we try and have one S/R per file, again to make it easier to find stuff. We do sometimes group things, if we think that makes code comprehension a little easier.

When we have multiple functions/subroutines per file we try and have main routine be the file name. All the other content we try to have be called MAIN_NAME_XXXX, and then the other content is maintained in alphabetical order. Again this helps with reading code quickly.

Just few comments:

I agree with everything (except may be the "alphabetical order" in ordering
S/R within same file, which might or might not apply depending on functionality)
and to add to 2.1,
https://github.com/MITgcm/MITgcm/blob/master/pkg/layers/layers_thermodynamics.F
has similar name issue as layers_save.F
all the above came as a later addition to pkg/layers, under #ifdef LAYERS_THERMODYNAMICS
but I am not sure that we have a "good" and relevant verification experiment
that illustrates how this works.

On Sun, Mar 04, 2018 at 12:57:03PM +0000, christophernhill wrote:

Two things on LAYERS

@rabernat , I think Jeff is getting close to having section "Packages II" into RTD. Is there any LAYERS documentation he can include?

Also there are a few pkg LAYERS code things that be great to clean up when someone has a moment.

https://github.com/MITgcm/MITgcm/blob/master/pkg/layers/layers_save.F contains a bunch of routines called LAYERS_FILL. On the whole we try and name files according to the function(s) they contain to make it easier to find things.

Generally we try and have one S/R per file, again to make it easier to find stuff. We do sometimes group things, if we think that makes code comprehension a little easier.

When we have multiple functions/subroutines per file we try and have main routine be the file name. All the other content we try to have be called MAIN_NAME_XXXX, and then the other content is maintained in alphabetical order. Again this helps with reading code quickly.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/MITgcm/MITgcm/issues/43

@jm-c agreed. The coding style bits are guidelines that can be adapted when common sense suggests something different is needed.

I am working on writing the layers documentation over the next few weeks. I will submit a documentation PR when I have something ready to share.
I appreciate the feedback on the layers coding style issues. I will try to fix these one at a time through individual small PRs.

@rabernat thats great thanks. All efforts are appreciated.

I have a very simple LAYERS_THERMODYNAMICS configuration for the vermix experiment (single column model). Is this something we might be interested in incorporating into the test suite?

Sorry for the delay @rabernat - this one slipped through, but we talked about it at the meeting today.
It probably is. Without seeing the configuration it's a little hard to say for certain. If you think it's a good clear example of how to use the functionality and it ties in with your work-in-progress docs, then I reckon we'll be keen.


CASMINST-4446: Fix goss-network-interfaces test to handle CHN and CAN configurations

Summary and Scope
This test will always check to see if the bond0.can0 interface exists on the worker and master nodes.   However, this interface not exist if the system was configured with CHN instead of CAN.   This changes the test into a script that checks the BICAN network configuration in SLS to determine if it is CAN or CHN.   It will only check for the bond0.can0 interface if it is configured for CAN.
Issues and Related PRs

Resolves CASMINST-4446

Testing
Tested on:

surtur and wasp

Test description:
I ran this goss test on wasp where it is configured for CHN and previous failed.  It now passes there.
I also ran this goss test on surtur where it was configured for CAN to make sure it still passes there.
Wasp:
ncn-m001:~ # GOSS_BASE=/opt/cray/tests/install/ncn goss -g /opt/cray/tests/install/ncn/tests/goss-network-interfaces.yaml --vars=/opt/cray/tests/install/ncn/vars/variables-ncn.yaml validate
....

Total Duration: 0.283s
Count: 4, Failed: 0, Skipped: 0

I also ran through several injected negative test cases (not able to get secret, not able to pull token, etc.) to test the error checking.
Pull Request Checklist

[x] Version number(s) incremented, if applicable
[x] Copyrights updated
[x] License file intact
[x] Target branch correct
[ ] CHANGELOG.md updated
[x] Testing is appropriate and complete, if applicable
[ ] HPC Product Announcement prepared, if applicable

/backport release/csm-1.2


Object aware YAML editor

Replace Octant's YAML viewer with an object aware YAML editor
Goal

Replace Octant's YAML viewer with an object are aware YAML editor
Editor will be aware of OpenAPI schemas provided by the cluster's API in the current context
Additional OpenAPI schemas can be provided by the user

Non-goals

There will be no wizard-like functionality for creating new objects

How will the editor be aware of the type of object that is being edited, including its fields/types/constraints/relationships? Will this metadata be contained in a json schema?
One important feature for this YAML editor is to be general enough to edit YAML that is stored on the file system, and not just YAML representations of objects obtained from the kubernetes api. There would need to be similar metadata support so that external object could also be constrained and validated.

You can fetch the OpenAPI schema from the Kubernetes cluster and validate it that way.

That sounds great for those objects that are in Kubernetes. We also have a need to be able to edit cluster API yaml files in a bootstrapping scenario before Kubernetes is running. Can the OpenAPI schema for these files be provided in some other way than from Kubernetes?

One of the major limitations with kubernetes dashboard is the limitations when editing secrets.
I wish that octant has secret editor is base64 aware.


Inversed Player hashmap in MercuryAPI

The onlineAllServers hashmap in MercuryAPI is actually built up inversed than what all the documentation and the methods suggest.
I tried here https://github.com/Civcraft/NameLayer/pull/139/files#diff-c300f2feb92fc77055b0c48b254f73f2R166 to get all players via Mercury, but when testing this I realized, that not the names of all players are returned, but instead the names of the shards.
And because this method is returning a keyset here: https://github.com/Civcraft/Mercury/blob/master/src/vg/civcraft/mc/mercury/MercuryAPI.java#L83 , which makes sense in the context of everything else there, but apparently delivers, what are supposed to be entries, the whole hashmap is apparently being built up inversed.
This is not an issue within the MercuryAPI class itself, but whatever class is actually adding players by using the API probably has some arguments inversed.

Fixes https://github.com/Civcraft/Mercury/blob/master/src/vg/civcraft/mc/mercury/MercuryAPI.java


Make Hermes transform the default in Babel config

Summary
Now that Hermes is the default when you create a new React Native app, let's make the Hermes transforms the default wrt the Babel config.
For instance, if you create a brand new React Native app, then try to do this in code:
console.log(2n ** 64n);

The old transform is used by default, despite the fact that Hermes now supports BigInt exponentiation:
console.log(Math.pow(2n, 64n));  // Runtime fatal!

Changelog

[General] [Changed] – The hermes-stable Babel transform is now the default when using init to make new apps
Test Plan

Run npx react-native init ExampleApp --version 0.70.0-rc.4
Make code change noted above
Run yarn android

Observe code is transformed correctly and correct result for 2^64 is logged.

I mean, I don't know. Maybe you plan to solve this by bumping to a forthcoming version of metro-react-native-babel-preset, but in the meantime this should make life testing the RC a little bit easier.


add SlidingCookie plug

Adds a SlidingCookie plug which implements a sliding window/session behaviour, allowing the remember_me cookie to be replaced before it expires.
Configure (or use plug option) like so:
config :my_app, MyApp.Guardian,
  ttl: {10, :minutes},
  sliding_cookie: {5, :minutes}

Add to Guardian pipeline, for example:
  plug Guardian.Plug.VerifyCookie
  plug Guardian.Plug.SlidingCookie
  plug Guardian.Plug.LoadResource
  plug Holco.Web.Plug.PostAuth

Provide sliding_cookie callback implementation:
defmodule MyAppWeb.Guardian do
  use Guardian, otp_app: :my_app
  def subject_for_token(user_rec, claims), do: ......
  def resource_from_claims(claims), do ......

  def sliding_cookie(claims, resource, opts) do
    {:ok, %{"new_claim" => "whatevs"}}
  end
end

Codecov Report

Merging #616 into master will increase coverage by 0.33%.
The diff coverage is 91.66%.

@@            Coverage Diff             @@
##           master     #616      +/-   ##
==========================================
+ Coverage   86.22%   86.56%   +0.33%
==========================================
  Files          21       22       +1
  Lines         421      439      +18
==========================================
+ Hits          363      380      +17
- Misses         58       59       +1

Impacted Files
Coverage Δ

lib/guardian/token/jwt.ex
85.71% <100%> (+0.34%)
:arrow_up:

lib/guardian.ex
89.28% <83.33%> (-0.72%)
:arrow_down:

lib/guardian/plug/sliding_cookie.ex
94.11% <94.11%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 38a5558...d261fc8. Read the comment docs.

Ah ha. Fixed formatting :)

Rebased on master and added :halt plug option following merge of #617. I'm not sure it really ever makes sense not to halt when the implementation is broken, but I think it's nice to be entirely consistent with the other plugs.

Sorry, I don't know how I accidentally closed it there, must have hit a key when the button was highlighted or something.

Agreed changes completed, I think.

Would love to have a view from one other @ueberauth/developers

Rebased.

Cheers. Always a pleasure to contribute a well received PR.


Add the Schiller Bar

First German entry in this list :-)

Nice, I've been many times in Berlin and didn't know this one yet ! Must check on my next visit :)


libinput: 1.9.1 → 1.9.2

Motivation for this change

Only a single addition/fix for features in the library - we now support multiple paired keyboards for
disable-while-typing. This is required for some of Razer's products where the keyboard exports multiple virtually identical event nodes that we cannot easily quirk.

https://lists.freedesktop.org/archives/wayland-devel/2017-November/035817.html
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

@vcunat is this good to merge to staging?

@disassembler Yes, staging has already advanced beyond the commit that will be merged to master.

ok, merging to staging. Thanks!


NPE when converting "paths" without any path

According to OpenAPI v3 specification and OpenAPI v2 specification

The Paths MAY be empty, due to ACL constraints.

SwaggerConverter fail with error if there are no paths in original model but should convert successfully without paths. This related only to api defined as yaml.

Everything works as expected :)

Hi @vadeg Thanks for reporting!

We have merged the PR, please reopen in case it keeps presenting. Thanks!!