Domain adaptation Classes

We should change the domain adaptation Classes to be more sklearn compliant.
Main issues:

Use CamelCase for classes
Use init for setting parameters and  instead of fit.

@agramfort proposed to Creat new Clases with proper names and begin deprecating the old classes.
I think it is a good move.

Good morning!I have coded a Sklearn compliant version of Domain Adaptation that I have been using for months... Would you like to see some of the code?NathalieLe 24 juil. 2017 08:31, Rémi Flamary<EMAIL_ADDRESS>a écrit :We should change the domain adaptation Classes to be more sklearn compliant.
Main issues:
Use CamelCase for classesUse init for setting parameters and  instead of fit.
@agramfort proposed to Creat new Clases with proper names and begin deprecating the old classes.
I think it is a good move.
—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.

Yes @ngayraud  we are interested by sklearn compatibility.
Note that we now have more strict rules for contribution now (pep8 for instance) but we would definitely welcome your contribution.

Hi @ngayraud https://github.com/ngayraud I am interested with your
sklearn API for DA
with @Slasnista we had started something similar for
semi-supervised DA.
let's see if we can all converge a good API together.

Great ! All the work is on the domain-adaptation-scikitlearn-compatible branch. I had created my own class outside github, and I have merged as many lines of code as I could.

Hi @ngayraud I am really interested in going into this direction too ! We have started considering this direction as @agramfort said.
We had introduced a BaseTransport class from which all the other DA methods inherited. The main functions where a fit_transform and _fit with parameters X, y, domain like this:
`
def fit_transform(self, X, y=None, domain=None):
    return self._fit(X, y, domain=domain)._transform(X, domain=domain)

def _fit(self, X, y=None, domain=None, method="sinkhorn"):
    """fit : compute transport plan
    """

    # pairwise distance
    Xs = X[domain == 0]
    Xt = X[domain == 1]
    Cost = pairwise_distances(Xs, Xt, metric="sqeuclidean")`

This way, one could do:
ot = OptimalTransportAlgo()
 transp_X = ot.fit_transform(X, y, domain)
Maybe we could mix our two approaches ?

Absolutely! I have implemented a similar approach for the fit(X,y) and transform(X) methods.  I submitted a merge request, mostly for you to see the entire code. I also agree that the classes should be properly named.However, in my implementation, one fits with the source domain, providing the target domain samples in the class initialization, and transports computes the transport to the target domain when transform (X) is called, so there is no need for a "domain" variable. If the target domain is not provided during the class initialization, the inverse transfer takes place (target to source). Do you have read access to the merge request?Le 24 juil. 2017 17:40, Stanislas Chambon<EMAIL_ADDRESS>a écrit :Hi @ngayraud I am really interested in going into this direction too ! We have started considering this direction as @agramfort said.
We had introduced a BaseTransport class from which all the other DA methods inherited. The main functions where a fit_transform and _fit with parameters X, y, domain like this:
`
def fit_transform(self, X, y=None, domain=None):
return self._fit(X, y, domain=domain)._transform(X, domain=domain)
def _fit(self, X, y=None, domain=None, method="sinkhorn"):
"""fit : compute transport plan
"""
# pairwise distance
Xs = X[domain == 0]
Xt = X[domain == 1]
Cost = pairwise_distances(Xs, Xt, metric="sqeuclidean")`

This way, one could do:
ot = OptimalTransportAlgo()
transp_X = ot.fit_transform(X, y, domain)
Maybe we could mix our two approaches ?
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

Hello and thank you for your proposal.
You both propose nice options, here are some comments.

It seems to me that a unique fit for estimation the DA is a good thing. Doing the learning in two steps with fit than transform as proposed by @ngayraud is a bit surprising.
On the other end, X,y and domain seem to be overly complex since you need to concatenate the x samples and provide a domain variable that you need to create also. what about y? does it contain labels from source of target  or both? what doe it contain when no target labels are available?

I know that both of you have though about it for a long time but personally I would use X,y, labels_X for fit where X is source domain, y target domain (it works since the transform will go from X to y) and labels are source labels when available. if target labels are available we can also use labels_y.
The thing i propose also have a natural extension to multidomain adaptation since X can be a list of arrays (and their corresponding labels).
What do you think about this?

I have been using the fit and transform implementation for a long time, since I am using it in a scikitlearn Pipeline method(are you familiar with it?) . The pipelines require both fit and transform methods to be implemented, and with this implementation I have been able to do cross validations and such. I think it is very handy, especially since I wanted to classify data after the transportation. This is the reason I think setting y as the target domain sample is not such a good idea, since it goes against classic fit methods of scikitlearn. Which is why I give the target domain samples in the initialization... Le 24 juil. 2017 18:47, Rémi Flamary<EMAIL_ADDRESS>a écrit :Hello and thank you for your proposal.
You both propose nice options, here are some comments.
It seems to me that a unique fit for estimation the DA is a good thing. Doing the learning in two steps with fit than transform as proposed by @ngayraud is a bit surprising.On the other end, X,y and domain seem to be overly complex since you need to concatenate the x samples and provide a domain variable that you need to create also. what about y? does it contain labels from source of target  or both? what doe it contain when no target labels are available?
I know that both of you have though about it for a long time but personally I would use X,y, labels_X for fit where X is source domain, y target domain (it works since the transform will go from X to y) and labels are source labels when available. if target labels are available we can also use labels_y.
The thing i propose also have a natural extension to multidomain adaptation since X can be a list of arrays (and their corresponding labels).
What do you think about this?
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

@ngayraud Oh yes I have seen it ! That's a good point if you don't have the "domain" variable.

But does it induce a longer runtime when you have large datasets, especially large Xt ?
My other fear is that if one want to save the TransportObject (with pickle for instance), having the target data as an attribute would induce an increase in the file size.
For the transform and inverse transform, what about implementing both methods, so that one can chose towards which domain to perform the adaption ?

@rflamary

I admit that it induces more complexity and that's the real drawback of such a convention
on the other hand, all the OTDA algorithms would have the same formalism and entry parameters: numpy arrays with the same number of rows (X.shape[0] = y.shape[0] = domain.shape[0]
this would not depend on their semi or unsupervised aspects.
we could chain the OTDA methods in sklearn-like pipelines with other preprocessing methods such as StandardScaler, zca whitening...
for the labels one could use the following conventions (filling the missing labels with -1)
X = np.concatenate((Xs, Xt), axis=0)
y = np.concatenate((ys, yt, -1 ... -1), axis=0)

So if we perform unsupervised DA and the method does not require any label, we use y=None
Otherwise, we construct y as above, filling the missing values by -1 so that y has the same number of rows as X.
We used afterwards the following fit function header but there is a lot of room for progress and simplicity ;)
def _fit(self, X, y=None, domain=None, method="sinkhorn"):
    """fit : compute transport plan
    """

    # pairwise distance
    Xs = X[domain == 0]
    Xt = X[domain == 1]
    Cost = pairwise_distances(Xs, Xt, metric=self.metric)

    if self.mode == "semisupervised":
        ys = y[domain == 0]
        yt = y[domain != 0]
        yt_train = yt[yt != -1]

        # change cost matrix
        classes = np.unique(ys)
        for c in classes:
            ids = np.where(ys == c)
            idt = np.where(yt_train == c)

            for j in idt[0]:
                Cost[ids[0], j] = 0
    else:
        if y is not None:
            ys = y[domain == 0]

Also, multidomain adaptation is a good idea, we could think of how to add it to the implementation for both ways, target to source and source to target!Le 24 juil. 2017 19:16, Nathalie Gayraud<EMAIL_ADDRESS>a écrit :I have been using the fit and transform implementation for a long time, since I am using it in a scikitlearn Pipeline method(are you familiar with it?) . The pipelines require both fit and transform methods to be implemented, and with this implementation I have been able to do cross validations and such. I think it is very handy, especially since I wanted to classify data after the transportation. This is the reason I think setting y as the target domain sample is not such a good idea, since it goes against classic fit methods of scikitlearn. Which is why I give the target domain samples in the initialization... Le 24 juil. 2017 18:47, Rémi Flamary<EMAIL_ADDRESS>a écrit :Hello and thank you for your proposal.
You both propose nice options, here are some comments.
It seems to me that a unique fit for estimation the DA is a good thing. Doing the learning in two steps with fit than transform as proposed by @ngayraud is a bit surprising.On the other end, X,y and domain seem to be overly complex since you need to concatenate the x samples and provide a domain variable that you need to create also. what about y? does it contain labels from source of target  or both? what doe it contain when no target labels are available?
I know that both of you have though about it for a long time but personally I would use X,y, labels_X for fit where X is source domain, y target domain (it works since the transform will go from X to y) and labels are source labels when available. if target labels are available we can also use labels_y.
The thing i propose also have a natural extension to multidomain adaptation since X can be a list of arrays (and their corresponding labels).
What do you think about this?
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

@slasnista as a matter of fact, when the transport is source to target, one needs to only store the source and target domain uniform distribution, as well as, the cost matrix. For the inverse transformation, the source domain samples are also needed, it is true. As for the runtime, I don't understand why it would take longer. In practice I have seen no difference.
I agree about the double implementation, I seems cleaner to somehow implement the two directions separately.

Hello everyone,
@ngayraud I misspoke when I talked about transform, i meant _transport that force you to do a fit with the source and then a transport with the target before doing transform/prediction. Don't worry i'm well aware of how sklearn work and have been using the validation classes for years.
I agree with you that multidomain is interesting, I you want to implement the aggregation method in your paper you are welcome and we will add the reference to the toolbox.
@Slasnista

Computational complexity is very high in this method and I I believe the recent NIPS paper will stay high for a long time.
@ngayraud is right that transporting the source sample only require the target samples and the transport matrix but if you want to perform out of sample prediction (Ferradans style) you need to store both. The memory complexity is a problem but storing the data is classical stuff for KNN and people live with it. I would be more concerned with the OT matrix that is extremely long to estimate and a memory hog on large datasets.
I strongly agree with the use of transform and inverse_transform for out of sample transport
I agree that OTDA could and should be chainable without other sklearn classes. Thing is I see it aas a pre-processing step since the transform do not change the size of the data. In this sens it should be implemented similarly to StandardScaler except it requires obviously 2 set of training instead of 1.
Concerning the concatenation of source and targets, I still thing it makes everything more complex since all datasets I know provides you with separate matrices (that you can concatenate with 2 lines OK prett simple). Moreover It does not make sens to perform StandardScaling on the concatenated samples since we have an adaptation problem and the source and target do not follow the same distribution. (possibly with different means). In DA the StandardScaler should be performed independently on each domain which simply forbids simple chain since you have to split and normalize.
If we keep the X,y,domain formulation though we need to provide simple to use functions that will create those matrices from xs,yx,xt,yt data and vice versa.

The way I see it, OTDA is a pre-processing step that will move your xs sample onto xt. If you want to perform classification simultaneously, I think we should have a different class that provides a predict function and others(classes).

Hello everyone,
@rflamary I will let you know about multidomain, I am working on it as we speak!
Concerning memory storage (and computational complexity as far as I am concerned), it is as you say, people live with it.
As I said, I am using this OTDA code in a Pipeline. It works very nicely when I want to transport target to source (the inverse transport), as I am extracting features in a previous step, computing the distribution, and then training the classifier (during the fit step of the Pipeline) and then for the predict step, the test features are extracted, transported, and predicted with the final classifier. However, to be honest, things do get a little more tricky when I want to transport source to target: I have to provide the target sample during the fit step, and I am not satisfied with any of the solutions that I have implemented. Hence, for the source -> target case, it is clear to me that it is a preprocessing step that would be hard to implement for the purpose of pipelining it.
What if we implemented these two processes separately? Do you feel it would be confusing? target -> source offers the possibility of a fit and then transform, can be added into a pipeline, and there is no apparent need (to me) for the user to provide both sets at the same time.  Whereas for source->target, one needs to provide both sets. In this case, a fit_transform is much simpler and intuitive. For this implementation, I am also skeptical about the concatenation of the two domains, as I feel it would be cleaner to just have all four variables, for example: fit_transform(Xsource,Xtarget,ysource=None,ytarget=None)
About OT matrix storage (gamma): since it is typically a sparse matrix, couldn't one use a sparse matrix representation to minimize the storage cost?

@rflamary I understand your point about the concatenation and the creation of a domain vector. Yes, it's true, one might need some helper functions to create easily the X, y and domain vectors.

@ngayraud ok for the memory concern. That's may be beside the point.

I think we should have a simple formalism for algorithm one input parameter for data arrays, one input parameter for label arrays. Then we can discuss about how to encode the different domains (additional domain vector; consider X, y as lists as one of you suggested above, or even as dictionaries or something else). There might be not perfect solution.

Yet, I find it cleaner to first instantiate the TransportObject and second give it the data when calling a .fit, .fit_transform or something else.

sorry, for the StandardScaler, that was a bad example. Let's take another scaling approach which works along rows instead. Could it be chained with TransportObject if we have lists of data arrays or dictionaries ?

shall we plan a skype / hangout session to speak about it directly ?

@Slasnista I am up for a skype or hangout session 👍 I am free this week and next week except for Wednesdays afternoon   !

Hello what about tomorrow 07/27 at ten for a hangout session.
@agramfort and @ncourty  if you can join us we could we could probably make a final decision.

@rflamary 07/27 10:00 is good for me.

Ok for me too.

sorry, I cannot make it. Could we postpone it to 13:00 ?

Ok for me!Le 27 juil. 2017 08:19, Stanislas Chambon<EMAIL_ADDRESS>a écrit :sorry, I cannot make it. Could we postpone it to 13:00 ?
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

OK for me too,
Le's try https://appear.in/ siunce it doe not require any account exchage but just a link.
Check that it works on your machine and  I will put a link to the conference here

Seems like it's working for me!

it seems ok for me. See you there !

Hello,
See you in 5
https://appear.in/functional-ram

Hi, please find below a summary of what has been said during the meeting.
Classes

create classes from sklearn BaseEstimator (offers set_params(), get_params(), other nice properties...)
-> implies a sklearn dependency
set all parameters in the __init__
give no data to the __init__

Methods

give the data to .fit as .fit(Xs, ys, Xt, yt) or .fit(X_source, y_source, X_target, y_target)

start thinking about multi domain convention

as convention (for now but could be changed):

.transform() transports source samples onto target samples
.inverse_transform() transports target samples onto source samples

integrate the different ways of transports points:

barycentric mapping
out of sample mapping
linear / kernelized mapping (should still be in a separate classes since the optimization problem is different)

Tests

write unitary test

update test_da.py
write a test for pipeline

write examples (pay attention to sphinx convention)

Others

creation of a dataset modules / functions (like keras) to load standard DA datasets
think about adding additional DA methods (Subspace Alignement or Geodesic Flow Kernel for instance)

Do not hesitate to add some things, especially if I have forgotten some !

sounds good. I am just a bit relunctant about :
.fit(Xs, ys, Xt, yt)
fit(X, y, domain)
seems a better idea. Also see how sklearn label propagation works:
http://scikit-learn.org/stable/modules/generated/sklearn.semi_supervised.LabelPropagation.html
it's a bit of a hack to stick to fit(X, y) but y=-1 has a semantic meaning.

This is perfect thank you to @Slasnista  for the notes .
The type of samples transportation should be given in the init
@agramfort  semi supervised learning is different from domain adaptation even if it can be used to do it. Concatenation of the samples in DA is not a good idea before processing which is the objective of this class.  In DA you basically have two datasets (with or without labels) when in semi supervised you have an incomplete one.
Also when transforming what happens? I don't like the idea of a function that changes on only part of a matrix (the source samples ?).  It would be a weird behavior for a transform function.
In any case I think it might be a good idea to have functions that can gos from xs,ys,xt,yt to X,y,domain and vice versa.

I will win this war another time :)


Download File uploaded with Voyager, from view

Version information

Laravel: v7.0
Voyager: v1.4
PHP: 7.2
Database: MariaDB 10.4.11

Description

I have a products table and marked a field as file in Voyager Admin Panel. After uploading the file, I'm not able to retrieve it clicking on download.
I can download it from the Voyager Panel, but not from the view.
Here's what I've tried
HomeController.php
public function download($id)
    {
        $product = Product::where('id',$id)->firstOrFail();
        $down = $product->download()->get();
        $down = json_decode($down);

        foreach($down as $item){

            $path =json_decode($item->link);
        }
        return response()->download(storage_path($path[0]->download_link));

    }

view.blade.php
<form action="{{route('home.download',$product->id)}}" method="GET">
          {{csrf_field()}}
          <input type="hidden" name="id" value="{{$product->product_id}}">
          <input type="hidden" name="name" value="{{$product->name}}">
          <input type="hidden" name="price" value="{{$product->price}}">
          <button type="submit" class="cart-button card-game-btn align-self-end position-absolute"><i class="fas fa-download"> </i>Download</button>
        </form>

It keeps telling me that the file is not in storage path,and even put manually the file in the folder won't resolve this.
Also the strange thing is that Voyager uploads the file in the public\storage\downloads (the filder create linked with storage:link)
But Laravel searchs in root-folder\storage\download
Before using voyager i was simply able to download the file referring the path of the file with storage_path().

Well that makes sense to me.  I think you might be misunderstanding how the storage:link command works.  It creates a filesystem "symlink" from public/storage to storage/app/public, so that both appear to contain the same files, but they're really only stored in one place.  For example:
Upload a file (i.e. example.pdf) with Voyager, and it might get stored as storage/app/public/example.pdf.  Then, when you request the file in the web browser, you can use the URL /example.com/storage/example.pdf.  This works as long as you hav laravel setup correctly in your webserver (site root is /public), and you're either using the default public` disk in the filesystem config, or have configured your own correctly(at which point, the behavior may be different).


board: arm: Add support for Panasonic Evaluation Boards

This Pull-Request adds support for Panasonic's Evaluation Boards pan1780_evb, pan1781_evb and pan1782_evb.

The PAN1780 evaluation board is a development tool for the nRF52840 from Nordic Semiconductor.
The PAN1781 evaluation board is a development tool for the nRF52820 from Nordic Semiconductor.
The PAN1782 evaluation board is a development tool for the nRF52833 from Nordic Semiconductor.

Signed-off-by: Steffen Jahnke<EMAIL_ADDRESS>

FYI @carlescufi

We have noticed that the pull-request may have some problems and was not automatically merged.
We are now looking at the different reports and will go through them and try to fix them.
If you have some additional feedback for us how to proceed here, please feel free to share it with us. It's greatly appreciated.

Hello @de-nordic & @carlescufi,
i have a question regarding the current failures in twister-build (1) to twister-build (4), which is basically always the same devicetree error:
devicetree error: missing data after phandle in <Property 'pwms' at '/pwmleds/pwm_evb_led_1' in '/__w/zephyr/zephyr/misc/empty_file.c'>
We are not that much experienced with contributing to the zephyrproject and have no further ideas what could be the problem with the way we are setting the pwmleds in the devicetree file of pan1780_evb(nRF52840), pan1781_evb(nRF52820) and pan1782_evb(nRF52833). Here for example the pan1780_evb pwmleds:
pwmleds {   compatible = "pwm-leds";   pwm_evb_led1: pwm_evb_led_1 {     pwms = <&pwm0 13>;   }; };
The we way choose to add the pwmleds nodes to the devictree is exactly the way Nordic has chosen in their DK devicetree files of nrf52840dk_nrf52840, nrf52833dk_nrf52820 and nrf52833dk_nrf52833.
If you have maybe some additional information for us how to proceed, please feel free to share with us. Thank you.

Hi @gmarull thanks for the review, i will step through your comments and fix them.
But i have already a question regarding the twister run. After updating pinctrl handling and pwmleds, almost all twister runs are successful. Nevertheless, there is a new "board not supported" error in the "test_adc.c" file, when testing the "zephyr\tests\drivers\adc\adc_api".
When i'm looking into "test_adc.c", there is of course no CONFIG_BOARD_PAN1770_EVB. Do i have to add my boards on my own to the test and commit afterwards?

Hi @gmarull thanks for the review, i will step through your comments and fix them.
But i have already a question regarding the twister run. After updating pinctrl handling and pwmleds, almost all twister runs are successful. Nevertheless, there is a new "board not supported" error in the "test_adc.c" file, when testing the "zephyr\tests\drivers\adc\adc_api".
When i'm looking into "test_adc.c", there is of course no CONFIG_BOARD_PAN1770_EVB. Do i have to add my boards on my own to the test and commit afterwards?

The test_adc.c seems to have still hardcoded information on a board basis, so yes, you'll need to manually add your board there. This tests likely needs a refactor now that ADC props can be configured in Devicetree, as its current structure doesn't scale. We recently refactored https://github.com/zephyrproject-rtos/zephyr/tree/main/samples/drivers/adc in this direction, you can add overlays for your board there as well.

@gmarull Thanks for the hint again. I've fixed the issues and did a rebase on the pull request.

Hi @gmarull
I noticed, that after last rebase of the pull request, the twister runs 2 & 3 are still failed. After looking for details, i can see, that some shield tests are failing, because our "arduino_header: connector" in the dts files for pan1781_evb and pan1782_evb is not able, to deliver a full gpio-map. This is simply, because the nRF52833 and nRF52820 SOC's have not enough pins to fill the gpio-map.
What would you suggest in this case? Is it possible, to exclude these shield tests from the twister run and live with the fact, that the shields will not be compatible with our evaluation board?


Make it smaller?

Hi,
I read your code and congrats, there are very interesting tricks used here to make your diff algorithm.
I have also worked on diffing a while ago, and found that the most efficient (in term of size and execution speed) seemed to be Myer's diff algorithm: https://blog.robertelder.org/diff-algorithm/
I minified and golfed it and reached a size of 406 bytes (without gzip):
http://xem.github.io/miniDiff/
So this is not really an issue, but if you're intersted to make your code shorter, this could be an idea.
Cheers,
Max

This is amazing! Not a priority for me right now, but I'm definitely open to it. Well done.


Add exact as tilda option

If someone have exact version dependency allow them convert dependency to tilda. Should be available as --exact-as-tilda|-t.

Implemented https://github.com/nella/victor/commit/20d3b1d7fe4c30aec6badd7f578b3d7b661b8ad9 and released as v1.1.0.


fix: add release-please config for ACM and Anthos Cluster blueprints

@kaariger could you rebase on main


Doc Feedback: deployment.toml doesnt have the setting specified

Location : https://is.docs.wso2.com/en/latest/learn/configuring-office-365-ws-federation-with-identity-server/
under Configuring Office 365 WS-Federation, step 10:
Add the following user store property to the <IS_HOME>/repository/conf/deployment.toml file.
<Property name="transformObjectGUIDToUUID">false</Property>
is that correct?, seems like a copy paste from  the older version https://docs.wso2.com/display/IS541/Configuring+Office+365+WS-Federation+with+Identity+Server. Except that in the older version the change was in user-mgt.xml not deployment.toml I dont think that change can happen in the deployment toml, unless there is new syntax.

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Unlimited words and Windows support

I forgot to run go fmt, everything should pass now

I got some golangci errors. If you can please run bin/golangci-lint run in the directory and see what it reports. Usually it's just something with formatting. I can't run this right now.


type::TransientValue

TransientValue class. This an immutable value transport class for bringing C types from the parser into SQL types (TransientValueFactory), or alternatively extracting C types from SQL types (TransientValuePeeker). The latter might end up not being needed, but it's good for testing and debugging right now. This is a stripped down version of Values from Peloton:

removes comparators and arithmetic operators. These should be handled by the future execution engine's type system. Index scans will have their own comparators to reason about ProjectedRows, to avoid having to instantiate Value objects to do comparisons
type promotion, see 1)
Serialization and deserialization from storage layouts. We don't want to rely on creating TransientValue objects for query execution, simply for transport into the system from the parser. Putting it back out on the wire should be doable without going to TransientValues in between

Looking for a few reviews since this touches the parser as well.

@mbutrovich I can review after you address @tli2 comments.

It's randomly failing CI because of a race in make. If it tries to build terrier before the gflags_ep, googletest_ep, and gbenchmark_ep targets are done, it fails due to gtest_prod.h missing. I thought it was just an issue with the Coverage build, but I'll poke at all of the CI configs to make sure they all work correctly.


Is it possible to do conditional branching during evaluation?

Hi, first of all, thanks for sharing this library. It's really awesome.
I was trying to implement Hierachical Multiscale Recurrent Neural Network, where the network will need to decide which path to take depending on the boundary gate from the previous time step and that of the lower layer.
At first, I thought this library is perfect fit for the task, but after reading the examples and the source code for loom and weaver, I get the impression that you would need to decide the structure of the graph for each input prior to evaluation, so weaver can figure out a efficient way to batch the inputs at each depth.
Which means weaver can't do much in the case of HMRNN, where the path each input take will depend on the intermediate values during evaluation. Is this understanding correct? Or is there a good way to implement this in Fold efficiently?

Your understanding is correct.  This is an issue that we are very much
aware of, but TensorFlow has some fundamental limitations that make it
difficult.  The problem essentially boils down to the fact that the only
way to get a result back from TensorFlow is to call session.run(), but you
can't backpropagate across multiple calls to run().  There is thus no way
for control flow to depend on intermediate results.  We have been
discussing several possible strategies for solving the problem, but there's
no easy fix.
-DeLesley
On Fri, Feb 10, 2017 at 12:38 AM, MycChiu<EMAIL_ADDRESS>wrote:

Hi, first of all, thanks for sharing this library. It's really awesome.
I was trying to implement Hierachical Multiscale Recurrent Neural Network
https://arxiv.org/pdf/1609.01704.pdf, where the network will need to
decide which path to take depending on the boundary gate from the previous
time step and that of the lower layer.
At first, I thought this library is perfect fit for the task, but after
reading the examples and the source code for loom and weaver, I get the
impression that you would need to decide the structure of the graph for
each input prior to evaluation, so weaver can figure out a efficient way to
batch the inputs at each depth.
Which means weaver can't do much in the case of HMRNN, where the path each
input take will depend on the intermediate values during evaluation. Is
this understanding correct? Or is there a good way to implement this in
Fold efficiently?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/fold/issues/8, or mute the thread
https://github.com/notifications/unsubscribe-auth/AGGbTciJC7ejx3SGAhwpeZvzXjCU1N8eks5rbCIQgaJpZM4L9INk
.

--
DeLesley Hutchins | Software Engineer |<EMAIL_ADDRESS>|<PHONE_NUMBER>

I see, thanks for the answer.
I actually managed to achieve this with tf.dynamic_partition on the batch index and then use the results to gather different branches' tensors, but since this requires a lot of communication between CPU and GPU, it was painfully slow. I wonder if it is possible to implement a GPU kernel for dynamic_partition, so the process could be more efficient.
I also looked into the new GPU queue, StagingArea. I thought we might be able to distribute the data to different queues according to their branch in one while_loop, then flush the queues and process them differently in the next loop. However, as of now, StagingArea is quite rudimentary and doesn't support batching, and I think it would also be impossible to back-prop the gradients through the queue, so this seemed like a dead end to me as well.
I am also really keen on solving this problem, so if there is any way I can help, please let me know.

I think the queue is a dead end for the reasons you mention.
TensorFlow does have its own control-flow operators, so if you can confine
the control-flow to within a single LoomOp then you can do branching and
loops that way.  What you can't do is use intermediate tensor results to
drive control-flow (recursion, etc.) inside python.  In addition,
TensorFlow Fold's dynamic batching only applies at the granularity of
LoomOps (i.e. in python); it won't work on control-flow within a LoomOp.
For hierarchical multiscale networks, you may be able to loop over the
characters in a single word using tf.while, and pad out a batch of words to
the same length using masks of some kind -- e.g. lock the input/forget
gates to 0/1 at end-of-word.  Pretty clunky, I know.
As I said before, there's not a quick fix here -- doing it properly is
going to involve some significant changes to the underlying framework, so I
don't have a good ETA for this feature as of yet.  If you're in a hurry,
Dynet, Chainer, and PyTorch also support dynamic computation graphs,
although they'll run significantly slower without dynamic batching.
On Fri, Feb 10, 2017 at 6:59 PM, MycChiu<EMAIL_ADDRESS>wrote:

I see, thanks for the answer.
I actually managed to achieve this with tf.dynamic_partition on the batch
index and then use the results to gather different branches' tensors, but
since this requires a lot of communication between CPU and GPU, it was
painfully slow. I wonder if it is possible to implement a GPU kernel for
dynamic_partition, so the process could be more efficient.
I also looked into the new GPU queue, StagingArea
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/data_flow_ops.py#L1403.
I thought we might be able to distribute the data to different queues
according to their branch in one while_loop, then flush the queues and
process them differently in the next loop. However, as of now, StagingArea
is quite rudimentary and doesn't support batching, and I think it would
also be impossible to back-prop the gradients through the queue, so this
seemed like a dead end to me as well.
I am also really keen on solving this problem, so if there is any way I
can help, please let me know.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/fold/issues/8#issuecomment-279116139, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGGbTacdTXlXaIx1h1UrGd4ijyuEi_t1ks5rbSP3gaJpZM4L9INk
.

--
DeLesley Hutchins | Software Engineer |<EMAIL_ADDRESS>|<PHONE_NUMBER>

I see, thanks for the suggestions, should I close this issue for now then?

We could leave this open and labeled for tracking.

A tricky workaround I'm using is to split the forward and backward into multiple sess.run, and glue them together with some auxiliary variables. For example, the first sess.run can store the results into some tf.Variable, and the second sess.run continue the execution using the results stored in those variables, and so on.
For the backward pass, there also needs to be some auxiliary variables to glue the gradients.

@ronghanghu Yeah, that's essentially what I am trying to do now, but adding the requirement of BPTT to the mix makes this a lot more complicated.

@MycChiu Yes, that's was a big headache for me. My tricky implementation involves anther forward pass when calculating the gradients, and to make it work I had to remove all the stochastic operations (e.g. dropout)

Also, sess.partial_run may be a solution, but somehow I couldn't get it work.

@ronghanghu Have you tried putting the dropout mask in a variable as well? Use that variable as the mask throughout the process, then just assign a new one in the new batch of forward pass.It wouldn't be too much trouble if you use the "theoretically grounded" dropout method by Gal and Ghahramani, since you are suppose to reuse the mask throughout the whole sequence anyway, so you would only need to keep track of a handful of masks depend on how many layers you have.
I have always had sess.partial_run in the back of my mind, but never actually gave it a shot. I guess now would be a great time.

Okay, I have played a bit with sess.partial_run, and the main advantage is that we can do conditional branching in the middle with python code, but every tensor can only be run once, so we will have to flatten the whole path into a really long feed forward network for this to work. Additionally, since it delegates the control back to python, it would also require a lot of CPU-GPU communication,  so I guess it's not much better than tf.dynamic_partition and tf.dynamic_stitch approach I was using.

well...after some profiling, I realized that tf.dynamic_stitch actually doesn't have GPU implementation, so the slowness was actually due to tf.dynamic_stitch constantly transferring all states back and forth between CPU and GPU for every timestep.

And the observation is confirmed by this issue on Tensorflow, so if I write a CUDA kernel for tf.dynamic_stitch, my current approach could actually be viable, although I don't really know how it could be incorporated into the Fold library though.

sess.partial_run is the only reasonable way that TensorFlow currently
provides for this sort of task.  It's possible to trap the gradients
yourself and feed them back in to another session.run, but that's both
difficult and error-prone -- not really something that you want to be doing
by hand.
What's currently missing is any support for sess.partial_run in the
TensorFlow Fold APIs.  It will take some somewhat substantial changes to
get that working smoothly with the loom and blocks libraries.
IMO, the CPU/GPU communication issue shouldn't be too bad.  If you keep the
computation about which branch to take on the GPU, then you only have to
transfer a single boolean to the CPU.  I'm more worried about the
interpreter overhead -- you have to exit the main TensorFlow interpreter
loop, go all the way back to python, and then re-enter the loop; I haven't
measured the cost of that overhead. Moreover, chopping the computation up
into pieces like this eliminates opportunities for dynamic batching, which
can also slow things down rather dramatically.
-DeLesley
On Sat, Feb 18, 2017 at 7:35 PM, Ronghang Hu<EMAIL_ADDRESS>wrote:

Also, sess.partial_run may be a solution, but somehow I couldn't get it
work.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/fold/issues/8#issuecomment-280893391, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGGbTYY8bonCZ2NPO94L3DGl1zeSgFXVks5rd7iagaJpZM4L9INk
.

--
DeLesley Hutchins | Software Engineer |<EMAIL_ADDRESS>|<PHONE_NUMBER>

In Caffe we introduced a filter layer trick some time ago to have a conditional path but I don't think that this approach could cover your cases.

My model involves first generating a parsing tree from a neural parser (implemented in TensorFlow) and a TreeLSTM running over the previous parsing tree (also implemented in TensorFlow, using Fold). To tie them end to end, my current workaround is to tie them together with sess.partial_run. (I found partial run didn't work due to I'm using an interactive session and https://github.com/tensorflow/tensorflow/issues/3233).
Btw, regarding Caffe, there is an extension developed by Stanford people called ApolloCaffe (http://apollocaffe.com) which, of course, does not support dynamic batching.

Filter layer create a dynamic tensor on the batch dimension

Just a quick question.. the first answer said
"he problem essentially boils down to the fact that the only
way to get a result back from TensorFlow is to call session.run(), but you
can't backpropagate across multiple calls to run(). "
Since in Tensorflow 2.0 there are no sessions anymore.. does this mean that this issue can be resolved?

In theory, yes, but the current code still uses the session API.
On Tue, Dec 3, 2019 at 2:02 AM zwep<EMAIL_ADDRESS>wrote:

Just a quick question.. the first answer said
"he problem essentially boils down to the fact that the only
way to get a result back from TensorFlow is to call session.run(), but you
can't backpropagate across multiple calls to run(). "
Since in Tensorflow 2.0 there are no sessions anymore.. does this mean
that this issue can be resolved?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/fold/issues/8?email_source=notifications&email_token=ABQZWTLQWLA5GUS6VFHFYT3QWYVB7A5CNFSM4C7UQNSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFYZQEI#issuecomment-561092625,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABQZWTILTZVOOJSVANPVVU3QWYVB7ANCNFSM4C7UQNSA
.

--
DeLesley Hutchins | Software Engineer |<EMAIL_ADDRESS>|<PHONE_NUMBER>


Tile loader update

Updated TileLoader in order to improve benchmark test.
Can now be configured to not load raster-tiles.
Can now be supplied an override function for where the tile-loader should pull tile-files.
Updated unit-tests accordingly.
Updated benchmarks.

There is a test that's failing here. Can we sort that out next week before approving the merge? Please put it on hold until we've delivered the draft to Rafael, and just focus on writing as much as you can/want by our delivery on Monday night. :-)

I think this looks good the way it is now. I'm approving it and merging it to dev. :-)


Error when running on Windows

I am trying to run senpy on Windows machine. I am getting the following error .
The Plugins are not getting loaded
Please guide me
ERROR:flask.app:Exception on /api/plugins/default/ [GET]
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\_compat.py", line 35, in reraise
    raise value
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-p
ackages\senpy\blueprints.py", line 109, in decorated_function
    if 'parameters' in response and not params['with_parameters']:
TypeError: argument of type 'NoneType' is not iterable
ERROR:flask.app:Exception on /api/plugins/default/ [GET]
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\_compat.py", line 35, in reraise
    raise value
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\Administrator\AppData\Roaming\Python\Python36\site-packages\fla
sk\app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-p
ackages\senpy\blueprints.py", line 109, in decorated_function
    if 'parameters' in response and not params['with_parameters']:
TypeError: argument of type 'NoneType' is not iterable

I am afraid we are unable to provide Windows support at this moment. Our recommendation is to run a linux environment through WSL (Windows Subsystem for Linux), on a VM or in the cloud.


Add more description and example for nodeSelector

I found the original explanation began by saying you can select the nodepool for a pod to be scheduled on, but the example just did a totally different thing.  Also, it didn't point out where to find what available labels to use in nodeSelector.
Perhaps the better way to do this is to attach a link to the doc that lists the labels AKS supplies to a node by default.  Not sure if that document exists though (couldn't find it when digging up on the internet..).  The example I gave here is from my observation from kubectl describe node.

#label:"aq-pr-triaged"

Thank you for the feedback.
I don't feel this change needed in the doc.
#please-close


bash: upgrade default version to 5

Motivation for this change
bash default version upgrade
cc @SuperSandro2000 @06kellyjac
@vcunat just leaving a comment here to test this on hydra at some point to verify nothing major breaks. (no rush, I know you are busy).
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

related: #131681
please update the PR title to match contributing, the same as the last one would do.
Also FYI you can make a new separate branch and push it over the one you were using for the last PR
git push <remote> <branch with new changes>:<branch you are pushing to>
e.g. git push origin upgrade_bash:bash_default_upgrade --force

@vcunat when you have a moment, if you can run a hydra jobset for this PR, it would be great. I know you are busy, no rush.
This PR is fairly simple, so hopefully it should go smoothly.

Should we keep bash_5 and bashInteractive_5 around as aliases (or vice versa) for those who might have already been using them?

I personally would tend to say no, as the upgrade should be straightforward. (I know I'm biased toward breaking things, to signify that something has changed, so I'm open to other opinions).
Perhaps we should add something in the next release notes ? I would prefer to do that in another PR after this one has been merged.

I don't see a reason not to do an update to the release notes and docs in this same PR

Fair enough. I've just added one line to the release notes (I was modelling what the postgres update says basically). I'm happy to change/add more if anybody feels it's needed.
Since this PR might be open for sometime, if a conflict arise, I plan to have the hydra test and do a rebase and merge after.

Hmmm there seems to be a formatting check failure. However the error is just exit 1. If anybody knows why that check is failing, i'm interested.

The failing check is .github/workflows/nixos-manual.yml. It’s checking that the XML file has been updated from the Markdown file, which it hasn’t. You need to run nixos/doc/manual/md-to-db.sh.

Should we keep bash_5 and bashInteractive_5 around as aliases (or vice versa) for those who might have already been using them?

Yes, please but they can be in aliases.nix.

I've added aliases, and fixed the docs. Let me know if this needs more love.

The last commit does not follow the contributing guide.

Took me a bit to squash, I was hoping a little for a hydra test this weekend.
I'm still not ready to do those by myself.

@vcunat if you have some time this weekend to start a hydra job for this PR, it would be great. No worries at all if you are busy.

Hydra is quite busy with security rebuild from OpenSSL (master now, 21.05 soon), so it will take at least a few days before there's capacity for that.

No worries, I'll ping again next week. Thank you for having checked!

A hydra job has been created on aarch64-linux for this https://hydra.nixos.org/jobset/nixpkgs/bash-5#tabs-evaluations
no failure due to this upgrade has been detected.
It's not a complete assurance that things are working, but it's a strong enough signal to merge in my opinion.
I will wait a couple more days in case people are against this.

I would actually merge this right now, but it looks like some conflicts have cropped up...

Just rebased, merging then. Thank you all for the time and care given to this!

Broke one mrsh test: https://hydra.nixos.org/build/155384906 (bash changed behaviour).

@trofi sorry about that! The failure isn't super clear from reading the logs. I'm curious about the change of behavior if you have more information. You probably know this but the bash changelog is not complete and so it makes it very hard to know what will actually break.

I only superficially looked at the test. It compares the output from mrsh (unchanged) to ouput from bash on the same test script (bash changed from 4 to 5). Let me try to dig the details out.

The following command changed behaviour under sh (but not bash):
a=~/foo:~/bar:~/baz; echo $a

$ nix build -f '<nixpkgs>' bash bash_4
$ ls -l
lrwxrwxrwx 1 slyfox users 55 Nov  1 21:52 result -> /nix/store/wadmyilr414n7bimxysbny876i2vlm5r-bash-5.1-p8
lrwxrwxrwx 1 slyfox users 56 Nov  1 21:52 result-1 -> /nix/store/4y0aacvxrkj3v7wayxpfxjda3km7c72a-bash-4.4-p23

$ /nix/store/wadmyilr414n7bimxysbny876i2vlm5r-bash-5.1-p8/bin/sh -c 'a=~/foo:~/bar:~/baz; echo $a'
/home/slyfox/foo:~/bar:~/baz
$ /nix/store/wadmyilr414n7bimxysbny876i2vlm5r-bash-5.1-p8/bin/bash -c 'a=~/foo:~/bar:~/baz; echo $a'
/home/slyfox/foo:/home/slyfox/bar:/home/slyfox/baz
$ /nix/store/4y0aacvxrkj3v7wayxpfxjda3km7c72a-bash-4.4-p23/bin/bash -c 'a=~/foo:~/bar:~/baz; echo $a'
/home/slyfox/foo:/home/slyfox/bar:/home/slyfox/baz
$ /nix/store/4y0aacvxrkj3v7wayxpfxjda3km7c72a-bash-4.4-p23/bin/sh -c 'a=~/foo:~/bar:~/baz; echo $a'
/home/slyfox/foo:/home/slyfox/bar:/home/slyfox/baz

sh under bash-5 stopped expanding ~ in the middle of a string before :. Not sure if it's intended or not.

Looks intentional. From the bash-5.1-beta changelog:

Changes to Bash
…
g. Avoid performing tilde expansion after `:' in words that look like assignment statements when in posix mode.

I reported a Bash bug about the apparent POSIX violation with some more context. I’ll link it here when it shows up in the archive.


Migrating from prompt toolkit to questionary

Removed the styles, since questionary has default styling built into it
I made this change in regards to Issue #8
I did strip the style formatting since it didn't seem to be needed.

Nice. Thanks.
i will take a look tomorrow at sprints.

@Snooz82 Have you had a chance to look at this PR fix?

@josev814 Thank you.
i added the styles. They work, but with Questionary you have to use different styles.


Dependency cycle detected  import/no-cycle

Hello,
After an update from '2.22.1' to '2.23.0' I have this error

Dependency cycle detected  import/no-cycle

I'm trying to import a TypeScript Type like this:
./Bar.ts
import useBar from './useBar';

export type Bar = {
  msg: string
}

./useBar.ts
import type { Bar } from './Bar';

Previously import type even if it was cycled import worked.

That is indeed a dependency cycle, so it's a bug that it wasn't warning on it before.

Was really a bug before?
According to this issue, https://github.com/benmosher/eslint-plugin-import/issues/1453 I thought because we can import type with
import type from { ... }

it was allowed.
Anyway, thank you for your answer.

No, anything that causes two files to depend on each other is a cycle, and is a horrifically bad architecture, which is the purpose of the rule to prevent.

Sorry, @ljharb, but this is wrong. The purpose of this rule is to avoid possible issues at runtime with circular dependencies. Using import type will never lead to that. Moreover, I disagree about this always being bad architecture when it comes to importing types. There are plenty of cases when this is more than appropriate.
At the very least, it should be configurable to exclude import type.
The makers of Madge know and understand that.
In any case, this has rendered this check useless for me because of this, which is actually really unfortunate. So, turning this rule off and integrating Madge into CI flow.

I agree that there’s a difference between avoiding runtime and build time cycles, and I’d be fine adding an option that allows type cycles.
However, import type can absolutely lead to that because enums and classes are in both value and type space.

I agree a flag would be nice, TypeScript already yells at me if I try to use imported type as value.

As @raycharius commented, this definitely is a breaking change for many projects since the rule now flags a recommended pattern in Redux where the root state type is inferred by TypeScript from imported reducers.

You can safely import the RootState type from the store file here. It's a circular import, but the TypeScript compiler can correctly handle that for types. This may be needed for use cases like writing selector functions.


Python 3

any plans to port to Python 3 ?
I for example see:
python3 -c 'import redhawk.common.selector'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.4/site-packages/redhawk/common/selector.py", line 75, in <module>
    import _selector
ImportError: No module named '_selector'

I started work on an initial PR for Python 2/3 compatibility at https://github.com/ncoghlan/Redhawk/commit/e924aec8add4015f7c2a614aeb463d8563171692
The major changes in the initial patch are:

switching to explicit relative imports
using print as a function rather than as a statement
using the "exception as name" except clause syntax rather than "exception, name"
wrapping various builtins with "list" calls to preserve the Py2 behaviour of producing a list
using six.moves to cope with some standard library rearrangements

The modified implementation mostly passes "python setup.py test" under Python 3.4, but I haven't checked it for Python 2.6/7 compatibility yet. There are also a couple of test failures: test_c_declarations and test_c_expressions both fail to locate their "test_descriptions" file (and switching those lines to use os.path.join didn't make a difference).

Wow, thanks, @ncoghlan! I had a preliminary peek and it looks good - I'd be happy to review and upstream it when you are done. Let me know if anything seems particularly hard or unwieldy - I'd be happy to help.

@jpopelka: Thank you for filing an issue. Just to clarify, when you say port do you mean:

Redhawk parses and queries python3 codebases.
Or Redhawk source code should be python3 compatible?

@ncoghlan's change should address the latter, but not the former.
Right now, I don't have the bandwidth to work on adding Python3 support in the LAST. I suspect I can get around to this over the next quarter, but I cannot make a firm guarantee at this time.
I can provide pointers if that'd help. I'd be happy to review and accept patches!

This issue was specifically about getting Redhawk itself running on Python 3 (Jiri & I are working on the same project, and we're interesting in running that on 3.4 or 3.5 instead of 2.7)
I filed #2 to cover eventually supporting the new syntactic constructs introduced in 3.3 and 3.5, but most of the code bases we're interested in analysing won't be using those new constructs yet.

Thanks for clarifying and triaging!

Hi, so I pushed several commits to @ncoghlan's py3-compat branch. All the tests that pass for me on Python 2 now also pass on Python 3; there are however 31 tests (out of 234) that fail on both versions. I'm quite sure that the failures are the same in Python 2 and 3, so it seems I didn't break anything. Nick, feel free to review my commits and create a PR if you think this is good enough. @spranesh, feel free to review any time, I'd be glad for any comments/suggestions.

PR submitted: https://github.com/spranesh/Redhawk/pull/3
@bkabrda GitHub was clever enough to migrate my and @jpopelka's comments on the initial python-modernize output from the fork, so you may want to look into those.

Thanks. I had a look and pushed one more commit that addresses the issues. The PR is ready to be reviewed by @spranesh

@spranesh sorry to bother you again, but could you please release a new Redhawk version on PyPI? It'd really help me. Now I need to install things from git by pip if I want Python 3 support which is not very nice - I would like to be able to install from PyPI with Python 3 support. Thanks!

@spranesh ping, any plans for releasing new version of Redhawk with Py3 support?

Yes, I plan to do a release with your changes within the next week or two.

Hello @spranesh
It's been few weeks and we'd like to know what's your realistic estimation of the release date. Don't feel pressed -  we just want to finish the porting of our project to Python 3 and need to decide whether to still wait for the release or start running redhawk from the repository. Thank you !

Sorry for dropping the ball on this. Please expect a release by August 23rd (i.e, two weeks from now).

Not a problem! Thanks for setting a target date, though - while we've always had the option of installing https://github.com/spranesh/Redhawk/commit/68a7400d73ad730e908435c8ead31d2c914a32a1 directly from version control, we do consider it preferable to run on officially released versions of dependencies :)

The 1.2.3 release now includes experimental support for running on Python 3.
However, as noted in #2, additional work will be needed to analyse new Python 3 specific syntax, and @bkabrda indicated in https://github.com/spranesh/Redhawk/pull/3#issuecomment-213305749 that there can be problems with analysing Python 2 only syntax when running on Python 3.


feat(rollup-plugin-index-html): enable inline module scripts

Not sure if this is the best approach (fs.writeFileSync), but here's an ostensibly working implementation of inline importmapped modules for rollup-plugin-index-html for your consideration

Looks like a good approach, but I want to think about this a bit and look at the use cases.
We need to keep the logic inside rollup-plugin-index-html to a minimum, and keep most of the logic in building-utils. It needs to work for the webpack plugin too, and es dev server (which is in a sperate PR but also uses these building-utils functions).

Can you add a test fixture of an inline script + non-inline script + legacy build? :)

@LarsDenBakker in channel we talked about

can we assume the user really wants us to extract inline scripts to files? should we instead leave some comment in the dom with a hash of the script and pass that to the plugin, which will replace the inline content with the built version?
doing more work in build-utils and less in the rollup-plugin, specifically, I imagine you're proposing writing inline modules as files in @open-wc/building-utils/index-html/extract-resources.js instead of in rollup?

After thinking about it, I think we can transform inline module scripts by default and add an option to turn that behavior off.
I'd prefer not to write any inline modules to files at all, so I guess that takes moving more logic to building-utils out of the equation... I'll check if there is a rollup way for it

@bennypowers I've pushed changes to avoid writing to temporary file.
I think this is good to go now, can you rebase on master and make this into one commit?


[BUG] Not supported USB Camera Mode crashes app because of preview window even with --nopreview

Describe the bug
Application crashes when the preview window does not support the mode although --nopreview is active.
Bug report
bug.txt
Additional context
By the way: how do i select not the camera but its mode (MJPEG vs YUYV)?
pi@raspscanner:~$ libcamera-still --version
libcamera-apps build: 7ac5197216ea 01-02-2022 (14:49:18)
libcamera build: v0.0.0+3406-e96d0201
pi@raspscanner:~$ libcamera-still --list-cameras
[0:07:14.694027712] [7320]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3406-e96d0201
Available cameras
-----------------
0 : USB Camera: USB Camera [3264x2448] (/base/soc/usb@7e980000-1:1.0-1bcf:28c4)
    Modes: 'MJPEG' : 640x480 1280x720 1600x1200 1920x1080 2592x1944 3264x2448
           'YUYV' : 640x480 1280x720 1600x1200 1920x1080 2592x1944 3264x2448
pi@raspscanner:~$ libcamera-still -o test.jpg --nopreview
[0:10:01.575022831] [10365]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3406-e96d0201
Stream configuration adjusted
[0:10:01.811974248] [10365]  INFO Camera camera.cpp:1028 configuring streams: (0) 1600x1200-MJPEG
terminate called after throwing an instance of 'std::runtime_error'
  what():  Preview windows only support YUV420
Aborted

Unfortunately this is not likely to ever work.  libcamera-apps only support YUV420 and RGB24 image formats.  Most USB cameras do not support either of those so will not work.
The primary use of libcamera-apps is to work with Bayer sensors like the official Raspberry Pi cameras.  USB cameras will not be supported.


Stream Removed Error due to unhandled ERROR_OPERATION_ABORTED (995) closes connection (Windows)

What version of gRPC and what language are you using?
1.19.0, C++. We're using an interop C# wrapper to drive the native c++ grpc code.
What operating system (Linux, Windows,...) and version?
OS Name:                   Microsoft Windows Server 2012 R2 Datacenter
OS Version:                6.3.9600 N/A Build 9600
.Net 4.6.1
System Manufacturer:       VMware, Inc.
System Model:              VMware Virtual Platform
The server is a VM (VMWare) hosted on an IBM Cloud
Note that this issue is very specific to one of our customers'  environment and that we cannot reproduce it on our own servers (running the exact same version of the OS and of our software).
What runtime / compiler are you using (e.g. python version or version of gcc)
MSVC Toolchain v141 (Visual Studio 2017)
What did you do?
We have a multi-threaded C# program roughly doing the following:
The main program thread T1 creates a GRPC client channel and connects to a remote GRPC server. This thread creates a second thread T2 that sends about 20GB of data, split into packets of about 1MB, to the remote server using this same channel. While it does so, thread T1 is waiting for T2 to complete. When T2 is done (and has exited), T1 continues and uses this channel again.
When T1 wants to use that channel again and make an RPC call on it, the following error occurs:
2019-07-10 10:51:48 [22320] ######### GRPC  INFO (tcp_windows.cc:189) TCP:000000FF7F3EFD30 on_read
2019-07-10 10:51:48 [22320] ######### GRPC  INFO (chttp2_transport.cc:2792) set connectivity_state=4
2019-07-10 10:51:48 [22320] ######### GRPC  INFO (connectivity_state.cc:164) SET: 000000FF7D0E4578 client_transport: READY --> SHUTDOWN [close_transport] error=000000FF7CD00C70 {"created":"@1562748708.563000000","description":"Endpoint read failed","file":"c:\ice\helium\contribs\grpc\grpc-1.19.0.windows\src\core\ext\transport\chttp2\transport\chttp2_transport.cc","file_line":2500,"grpc_status":14,"occurred_during_write":0,"referenced_errors":[{"created":"@1562748708.563000000","description":"The I/O operation has been aborted because of either a thread exit or an application request.\r\n","file":"c:\ice\helium\contribs\grpc\grpc-1.19.0.windows\src\core\lib\iomgr\tcp_windows.cc","file_line":197}]}

What did you expect to see?
We did not expect to see any error
What did you see instead?
Because this error is interpreted as an "hard" read error, the socket is being closed and the following RPC call fails unless we put a retry on that order. Interestingly enough most of the time the request sent by T1 is received by the remote server and properly processed. So putting a retry at the application level raises some concerns because the RPC call could actually have been already executed.
Analysis

We first thought that a third party tool could forcibly shutdown the connection (antiviruses, network monitoring stuff, intermediate firewalls or load alancers, etc). We disabled all agents and the behaviour remained the same.
When this error is caught, all open GRPC sockets end up being closed for the same reason, including other channels open on the loopback interface (localhost)
We investigated this ERROR_OPERATION_ABORTED (995), which means The I/O operation has been aborted because of either a thread exit or an application request. and understood that it can be raised in the following 2 cases:

An asynchrous IO operation has been initiated and has been later cancelled (for instance by using CancelIo)
An asynchrous IO operation has been initiated by a thread, which exits without handling its outcome itself. As documented by Microsoft (see https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-exitthread): "When this function (ExitThread) is called (either explicitly or by returning from a thread procedure), the current thread's stack is deallocated, all pending I/O initiated by the thread is canceled, and the thread terminates. "

We are clearly in this scenario where a thread initiates an overlapped I/O (in win_read with WSARecv), and then exits.
The error is first received by iocp_windows.cc:grpc_iocp_work when calling WSAGetOverlappedResult. It is not set by the line setting wsa_error to WSA_OPERATION_ABORTED
Then on_read is being called and we enter the code if (info->wsa_error != 0 && !tcp->shutting_down) which raises an error.

In the attached logs, we have activated the GRPC logs (but removed the timer_* stuff). If you look for '::LAST_READ' (manually added) you will find the last call to win_read on the socket. The line numbers may differ from the official 1.19.0 sources because we instrumented gRPC to troubleshoot the problem. Then if you search '::ERR1' you'll find the first place when the error is handled, causing the closing of the connection.
Proposed Fix
Because this error is documented by Microsoft, we have made the following patch which seems to solve the problem (let me know if I should push a PR). It basically allows for a zero-byte read in case this specific error is encountered. There may be a more elegant way to solve this issue, possibly at the I/O Completion Port level. Also please note that I have not enough knowledge of gRPC internals to assess whether this error could also show up during async write operations.
--- a/src/core/lib/iomgr/tcp_windows.cc
+++ b/src/core/lib/iomgr/tcp_windows.cc
@@ -192,13 +192,14 @@ static void on_read(void* tcpp, grpc_error* error) {
   GRPC_ERROR_REF(error);

   if (error == GRPC_ERROR_NONE) {
-    if (info->wsa_error != 0 && !tcp->shutting_down) {
+    bool cancelled_read = info->wsa_error == ERROR_OPERATION_ABORTED;
+    if (info->wsa_error != 0 && !cancelled_read && !tcp->shutting_down) {
       char* utf8_message = gpr_format_message(info->wsa_error);
       error = GRPC_ERROR_CREATE_FROM_COPIED_STRING(utf8_message);
       gpr_free(utf8_message);
       grpc_slice_buffer_reset_and_unref_internal(tcp->read_slices);
     } else {
-      if (info->bytes_transfered != 0 && !tcp->shutting_down) {
+      if ((info->bytes_transfered != 0 || cancelled_read) && !tcp->shutting_down)  {
         GPR_ASSERT((size_t)info->bytes_transfered <= tcp->read_slices->length);
         if (static_cast<size_t>(info->bytes_transfered) !=
             tcp->read_slices->length) {

GRPC log -- please not that ThreadId id is indicated between square brackets.
grpc_stream_removed.log


Streamline process to add new discoverable taps/targets and variants

Migrated from GitLab: https://gitlab.com/meltano/hub/-/issues/5
Originally created by @DouweM on 2021-03-04 18:52:53

We can do much better than https://meltano.com/docs/contributor-guide.html#discoverable-plugins and can go beyond https://gitlab.com/meltano/meltano/-/issues/2608.
On https://gitlab.com/meltano/meltano/-/issues/2600, users should be able to sign in and submit new taps/targets through the UI, even if this would still result in merge requests to some repo behind the scenes.
There should also be a way to do this from the SDK. This issue will be broken up into specific initiatives beyond #6.

View 13 previous comments from the original issue on GitLab


chore(auth): DPoP and public fixes

expose legacy public key endpoint as public
DPoP htu should include origin part of url
clarify error messages for dpop

Now it checks the htu field of the DPoP token matches the request Origin + Path


Agon locks up when POS and/or VPOS is used with VDU 31

Any of the following lines of code in BASIC or BASIC ADL will cause the Agon to lock-up:
10 VDU 31, POS, VPOS
20 VDU 31, 0, VPOS
30 VDU 31, POS, 0

The VDP should handle the evaluation of POS and VPOS properly so that the proper X and Y coordinates are passed to VDU 31 .

Problem does not occur with PRINT TAB(POS,VPOS); Neither when you first assign variables with POS & VPOS and pass the variables to VDU 31.
Probably the root cause: the BASIC VDU command passes each byte to the VDP after it is evaluated, so 31 gets passed immediately after it is evaluated. Then POS and/or VPOS are evaluated and BASIC sends the VDU 23, 0,130 sequence to read the cursor position and wait for the result. But the first bytes of this sequence get interpreted as the parameters of VDU 31, so the cursor read command will never finish.
A fix would probably need to happen in BASIC, for example after each command that might output to the screen, call VDU 23,0,130 and store the cursor position locally. Evaluation of POS & VPOS just take those locally stored bytes. Or the VDU command has to buffer all bytes of that command into a local buffer and then output that buffer at once. The VDU command could buffer some of the VDU sequences until they are complete (VDU 23, 24, 28 and 31) and only output them as a whole.
Other versions of Beeb Basic do not send bytes to the VDU to retrieve cursor position information, so the problem does not occur here.

Problem does not occur with PRINT TAB(POS,VPOS); Neither when you first assign variables with POS & VPOS and pass the variables to VDU 31.
Probably the root cause: the BASIC VDU command passes each byte to the VDP after it is evaluated, so 31 gets passed immediately after it is evaluated. Then POS and/or VPOS are evaluated and BASIC sends the VDU 23, 0,130 sequence to read the cursor position and wait for the result. But the first bytes of this sequence get interpreted as the parameters of VDU 31, so the cursor read command will never finish.
A fix would probably need to happen in BASIC, for example after each command that might output to the screen, call VDU 23,0,130 and store the cursor position locally. Evaluation of POS & VPOS just take those locally stored bytes. Or the VDU command has to buffer all bytes of that command into a local buffer and then output that buffer at once. The VDU command could buffer some of the VDU sequences until they are complete (VDU 23, 24, 28 and 31) and only output them as a whole.
Other versions of Beeb Basic do not send bytes to the VDU to retrieve cursor position information, so the problem does not occur here.

@lennart-benschop Thank you for performing an analysis of the issue.  Does this issue now need to be opened in each of the Agon BASIC git repositories, https://github.com/breakintoprogram/agon-bbc-basic-adl/issues and https://github.com/breakintoprogram/agon-bbc-basic/issues?


A simple DeepMLP is taking 11GB/16GB in GPU RAM

Description
Hi, I'm new to using Equinox and have encountered an unexpected issue with checkpoint loading, where despite training the same model with the same number of layers, error, and optimizer as my previous Keras model, the saved checkpoint suddenly consumes 11GB of GPU RAM out of 16GB when loaded, which is puzzling given the file's relatively small size of 37MB on disk. I'm struggling to understand why Equinox is utilizing such a large amount of memory and am seeking help to resolve this issue.
PS:
I mentioned this issue in JAX discussions, at the time of writing, my guess this is coming from another package.
Functions to make, save and load model
def make(*, key, input_layer: int, output_layer: int, hidden_layers: Optional[list[int]] = None) -> PyTree:
    """Make a neural network model to approximate the log of the VT function.

    :param key: jax random key
    :param input_layer: input layer of the model
    :param output_layer: output layer of the model
    :param hidden_layers: hidden layers of the model
    :return: neural network model
    """
    if hidden_layers is None:
        keys = jax.random.split(key, 2)
        layers = [
            eqx.nn.Linear(
                in_features=input_layer,
                out_features=output_layer,
                key=keys[0],
            )
        ]
        model = eqx.nn.Sequential(layers)
        return model

    keys = jax.random.split(key, 2 + len(hidden_layers))

    layers = [
        eqx.nn.Linear(
            in_features=input_layer,
            out_features=hidden_layers[0],
            key=keys[0],
        ),
        eqx.nn.Lambda(jax.nn.relu),
    ]
    for i in range(len(hidden_layers) - 1):
        layers.append(
            eqx.nn.Linear(
                in_features=hidden_layers[i],
                out_features=hidden_layers[i + 1],
                key=keys[i],
            ),
        )
        layers.append(eqx.nn.Lambda(jax.nn.relu))
    layers.append(
        eqx.nn.Linear(
            in_features=hidden_layers[-1],
            out_features=output_layer,
            key=keys[i],
        )
    )

    model = eqx.nn.Sequential(layers)

    return model

def save_model(*, filename: str, hyperparams: dict[str, Any], model) -> None:
    """Save the model to the given file.

    :param filename: Name of the file to save the model
    :param hyperparams: Hyperparameters of the model
    :param model: Model to approximate the log of the VT function
    """
    with open(filename, "wb") as f:
        hyperparam_str = json.dumps(hyperparams)
        f.write((hyperparam_str + "\n").encode())
        eqx.tree_serialise_leaves(f, model)

def load_model(filename) -> tuple[dict[str, Any], PyTree]:
    """Load the model from the given file.

    :param filename: Name of the file to load the model
    :return: Hyperparameters and the model
    """
    with open(filename, "rb") as f:
        hyperparam_str = f.readline().decode()
        hyperparams = json.loads(hyperparam_str)
        model = make(key=jax.random.PRNGKey(0), **hyperparams)
        model = eqx.tree_deserialise_leaves(f, model)
    return hyperparams, model

Model Summary
It is a sequential model, taking two inputs and yields one output and in between there are 10 layers with 1024 neurons, with relu activation.
                                       Summary of the Neural Network Model
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Parameter        ┃ Value                                                                                ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Input Keys       │ m1, m2                                                                                │
│ Output Keys      │ VT                                                                                │
│ Hidden Layers    │ 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024                                   │
│ Data Path        │ /kaggle/input/modern-vt/vt_0.5_200_1day_SimNoisePSDaLIGO175MpcT1800545_IMRPhenomD_snr10.hdf5 │
│ Checkpoint Path  │ neural_vt_0.5_200_1day_SimNoisePSDaLIGO175MpcT1800545_IMRPhenomD_snr10.eqx                   │
│ Train Size       │ 540672                                                                                │
│ Test Size        │ 60275                                                                                │
│ Batch Size       │ 1024                                                                                │
│ Epochs           │ 50                                                                                │
└──────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘

Before loading model
Sat May 25 12:44:24 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   66C    P0              32W /  70W |    105MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla T4                       Off | 00000000:00:05.0 Off |                    0 |
| N/A   63C    P0              30W /  70W |    105MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

After loading model
Sat May 25 12:44:28 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   66C    P0              32W /  70W |  11447MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla T4                       Off | 00000000:00:05.0 Off |                    0 |
| N/A   63C    P0              30W /  70W |    105MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

2 things. 1) I think you copied the same image twice, the "after loading" is identical to the before loading. 2) Is this distinct from what jax already does? Like Jax always pre allocates a bunch of GPU memory (75% https://jax.readthedocs.io/en/latest/gpu_memory_allocation.html) unless you force it not to. Is the network unexpectedly large or is this just a jax thing?

Is this distinct from what jax already does? Like Jax always pre allocates a bunch of GPU memory (75% https://jax.readthedocs.io/en/latest/gpu_memory_allocation.html) unless you force it not to.

I have tried this before and it didn't work for some strange reason. Pre-allocation of memory makes sense when it is being used. I get this error,
XlaRuntimeError: RESOURCE_EXHAUSTED: Failed to allocate request for 6.27GiB<PHONE_NUMBER>B) on device ordinal 0

This clearly means the pre-allocated memory is not being used. For sake of double check, I just ran the model with following flags and got the same error.
os.environ["XLA_PYTHON_CLIENT_PREALLOCATE"] = "false"
os.environ["XLA_PYTHON_CLIENT_ALLOCATOR"] = "platform"
os.environ["XLA_FLAGS"] = ("--xla_cpu_multi_thread_eigen=false "
                           "intra_op_parallelism_threads=1")

But this time memory was not allocated.
Before loading

Sat May 25 19:33:00 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   59C    P0              28W /  70W |    157MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla T4                       Off | 00000000:00:05.0 Off |                    0 |
| N/A   58C    P0              29W /  70W |    105MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

After Loading

Sat May 25 19:33:05 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   59C    P0              28W /  70W |    193MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla T4                       Off | 00000000:00:05.0 Off |                    0 |
| N/A   58C    P0              29W /  70W |    105MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

After crash

Sat May 25 19:39:37 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   73C    P0              31W /  70W |    293MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla T4                       Off | 00000000:00:05.0 Off |                    0 |
| N/A   53C    P0              28W /  70W |    105MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

Is the network unexpectedly large or is this just a jax thing?

I don't think it is. It is a simple DeepMLP with layers sizes 2->1024->1024->1024->1024->1024->1024->1024->1024->1024->1024->1. By default, JAX uses jnp.float32 and if we estimate the size of all these layers, it should not exceed 100MBs.

Have you tried using a smaller batch size during inference? I don't think the problem is at Jax side - its likely that some part of your code holds some big arrays that are being used somehow.

@lockwo @neel04 Thanks for your input. Issue has been resolved. You guys were right my neural net was quite big.


Release binaries in "Bleeding edge" release

This GitHub Action should upload updated binaries to the "Bleeding edge"
release each time a new commit is merged to master.

Coolness!!! @manosnoam should love this ;)


bug inventory clean with sort button

LaserIO version
1.6.1
Minecraft Version
1.20.1
(Neo)Forge Version
No response
Modpack & Version
project architect 2
Do you have optifine installed?
I have other rendering based mods installed
Describe the issue
When you have a button to sort the inventory and use it with in the card HUD, your inventory gets cleared
Steps to reproduce

set a button to sort the inventory (middle mousse button for me)
get a card
open a card
use the button
inventory clean but not curio and armor
video.zip

Expected behaviour
get a fix
Screenshots
video.zip
Log files
video.zip
Additional information
No response

I experienced this same issue in LaserIO 1.6.8 bundled with Monifactory 0.9.3


Save 'S1' array without the char_dim_name dimension

MCVE Code Sample

import numpy as np
import xarray as xr
tstr='2019-07-25_00:00:00'
Times = xr.DataArray(np.array([" ".join(tstr).split()], dtype = 'S1'), dims = ['Time', 'DateStrLen'])
ds = xr.Dataset({'Times':Times})
ds.to_netcdf('test.nc', format='NETCDF4',encoding={'Times': {'zlib':True, 'complevel':5}}, unlimited_dims={'Time':True})

Expected Output
Because I want to use the nc file as the input of WRF model,
I just need Time and DateStrLen two dimensions.
ncdump -h test.nc:
netcdf test {
dimensions:
        Time = UNLIMITED ; // (1 currently)
        DateStrLen = 19 ;
variables:
        char Times(Time, DateStrLen) ;
}

Problem Description
This is the actual output of ncdump -h test.nc:
netcdf test {
dimensions:
        Time = UNLIMITED ; // (1 currently)
        DateStrLen = 19 ;
        string1 = 1 ;
variables:
        char Times(Time, DateStrLen, string1) ;
}

Output of xr.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 3.0.76-0.11-default
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.1
libnetcdf: <IP_ADDRESS>
xarray: 0.13.0
pandas: 0.25.1
numpy: 1.17.2
scipy: 1.3.1
netCDF4: 1.3.1
pydap: None
h5netcdf: None
h5py: 2.8.0
Nio: None
zarr: 2.3.2
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.5.0
distributed: 2.5.2
matplotlib: 3.1.1
cartopy: 0.17.0
seaborn: None
numbagg: None
setuptools: 41.2.0
pip: 19.2.3
conda: None
pytest: None
IPython: 7.8.0
sphinx: None

Although it's possible to set the exact char_dim_name to Time,
but I need the unlimited Time dimension as the first one.

Hi @zxdawn
Does this modified version of your code do what you want?:
import numpy as np
import xarray as xr
tstr='2019-07-25_00:00:00'
Times = xr.DataArray(np.array([tstr], dtype = np.dtype(('S', 16))), dims = ['Time'])
ds = xr.Dataset({'Times':Times})
ds.to_netcdf(
   'test.nc',
   format='NETCDF4',
   encoding={
      'Times': {
         'zlib':True,
         'complevel':5,
         'char_dim_name':'DateStrLen'
      }
   },
   unlimited_dims={'Time':True}
)

Output of ncdump:
netcdf test {
dimensions:
	Time = UNLIMITED ; // (1 currently)
	DateStrLen = 19 ;
variables:
	char Times(Time, DateStrLen) ;
data:

 Times =
  "2019-07-25_00:00:00" ;
}

Some explanation of what is going on:
Strings in numpy aren't the most friendly thing to work with, and the data types can be a little confusing. In your code, the "S1" data type is saying "this array has null terminated strings of length 1". That 1 in the "S1" is the string length. This resulted in you having an array of one character strings that was 19 elements long:
array([[b'2', b'0', b'1', b'9', b'-', b'0', b'7', b'-', b'2', b'5', b'_',
        b'0', b'0', b':', b'0', b'0', b':', b'0', b'0']], dtype='|S1')

vs what I think you want:
array([b'2019-07-25_00:00:00'], dtype='|S19')

Since you know that your string length is going to be 19, you should tell numpy about this instead of xarray by either specifying the data type as "S19" or using the data type constructor (which I prefer): np.dtype(("S", 19))

@DocOtak Thank you for your explanation! It works well now :)


Make use of EmptyState throughout the Plugins

The new <EmptyState /> should be used when a plugin wants to guide a user to an action related to something missing or wants to display more information on why something is missing.
We should look through our existing plugins and update them to use <EmptyState />.

Closing for now, was added in a lot of places.


Proposal: Add multi-join event/mode

from @EmmanuelHuazo (#884)
Can you also change the "Join character node" to add multiple characters at a time with only one node?
I have all the characters in different positions, but they look way pile up.

ah interesting idea, maybe making some kind of like predefined characer group thats saved into a different file type, that defines all of the characters there and the positions and portraits they start in. then just join in the whole group


Add Prometheus to plugins page.

Should we also add a page to the roadie docs?

Yup, I'll add the needed config options to Roadie first


Fixed multipart about #6

fixed about https://github.com/evancz/elm-http/issues/6

@amcvega just created a similar pull request, but with one more change
return { ctor: 'BodyFormData', formData: formData };

It would be great if you two could connect to resolve this difference.

I went for the other one. I think this change was slightly off in that it used the 'FormData' constructor instead of 'BodyFormData'. Thanks for the fix though!

I am sorry I created a similar pull request.  In a different way I will contribute continuously.

No worries at all! Yours came in way before the other one, and largely fixed the issue!
I am struggling to respond to everything right now, but in the process of sorting that process out. In this case, I failed to get a timely response to your issue.
Thanks for the PR and I hope you will do more!

Thanks @Dobiasd for pointing the connection here. This kind of curation is extraordinarily helpful!


Transition from CircleCI -> GitHub Actions

looks like this broke master - maybe a good idea to revert master or make a note in the README to use 37b48e785aebb5302341edf02c4954070d95aa3b (maybe tag this commit with a version number?)

yup it did break master - need to take a look when i can.