Higher Level API for RNN

We've created a higher level API for recurrent neural networks and have completed gradient tests,  forward test and speed comparison against CuDNN. The class definition and key methods look like this:
class RNN(object):
    """High level API for constructing stacked RNN layers.

    To use a recurrent neural network, we can first create an RNN object and use the step function
    during the symbol construction.

    Currently four types of RNN are supported and all parameters per layer are grouped into 4 matrices.
    The data layout and transition rules are similar to the RNN API in CuDNN (https://developer.nvidia.com/cudnn)
    1) ReLU RNN:
        h_t = ReLU(W_i x_t + R_i h_{t-1} + b_{W_i} + b_{R_i})

        Parameters:
            W_{i2h} = W_i
            b_{i2h} = b_{W_i}
            W_{h2h} = R_i
            b_{h2h} = b_{R_i}
    2) Tanh RNN:
        h_t = tanh(W_i x_t + R_i h_{t-1} + b_{W_i} + b_{R_i})

        Parameters:
            W_{i2h} = W_i
            b_{i2h} = b_{W_i}
            W_{h2h} = R_i
            b_{h2h} = b_{R_i}
    3) LSTM:
        i_t = \sigma(W_i x_t + R_i h_{t-1} + b_{W_i} + b_{R_i})
        f_t = \sigma(W_f x_t + R_f h_{t-1} + b_{W_f} + b_{R_f})
        o_t = \sigma(W_o x_t + R_o h_{t-1} + b_{W_o} + b_{R_o})
        c^\prime_t = tanh(W_c x_t + R_c h_{t-1} + b_{W_c} + b_{R_c})
        c_t = f_t \circ c_{t-1} + i_t \circ c^\prime_t
        h_t = o_t \circ tanh(c_t)

        Parameters: (input_gate, forget_gate, new_mem, output_gate)
            W_{i2h} = [W_i, W_f, W_c, W_o]
            b_{i2h} = [b_{W_i}, b_{W_f}, b_{W_c}, b_{W_o}]
            W_{h2h} = [R_i, R_f, R_c, R_o]
            b_{h2h} = [b_{R_i}, b_{R_f}, b_{R_c}, b_{R_o}]
    4) GRU:
        i_t = \sigma(W_i x_t + R_i h_{t-1} + b_{W_i} + b_{R_i})
        r_t = \sigma(W_r x_t + R_r h_{t-1} + b_{W_r} + b_{R_r})
        h^\prime_t = tanh(W_h x_t + r_t \circ (R_h h_{t-1} + b_{R_h}) + b_{W_h})
        h_t = (1 - i_t) \circ h^\prime_t + i_t \circ h_{t-1}

        Parameters: (reset_gate, update_gate, new_mem)
            W_{i2h} = [W_r, W_i, W_h]
            b_{i2h} = [b_{W_r}, b_{W_i}, b_{W_h}]
            W_{h2h} = [R_r, R_i, R_h]
            b_{h2h} = [b_{R_r}, b_{R_i}, b_{R_h}]
    """
    def __init__(self, num_hidden, data_dim, typ='lstm',
                 dropout=0., zoneout=0.,
                 i2h_weight=None, i2h_bias=None,
                 h2h_weight=None, h2h_bias=None,
                 init_h=None, init_c=None,
                 cudnn_opt=False,
                 name='LSTM'):
        """Initialization of the RNN object

        Parameters
        ----------
        num_hidden : list or tuple
            Size of the hidden state for all the layers
        data_dim : int
            Dimension of the input data to the symbol
        typ: str
            Type of the Recurrent Neural Network, can be 'gru', 'lstm', 'rnn_relu', 'rnn_tanh'
        dropout : list or tuple, optional
            Dropout ratio for all the hidden layers. Use 0 to indicate no-dropout.
        zoneout : list or tuple, optional
            Zoneout ratio for all the hidden layers. Use 0 to indicate no-zoneout.
        i2h_weight : list or tuple, optional
            Weight of the connections between the input and the hidden state.
        i2h_bias : list or tuple, optional
            Bias of the connections between the input and the hidden state.
        h2h_weight : list or tuple, optional
            Weight of the connections (including gates) between the hidden states of consecutive timestamps.
        h2h_bias : list or tuple, optional
            Bias of the connections (including gates) between the hidden states of consecutive timestamps.
        init_h : list or tuple, optional
            Initial hidden states of all the layers
        init_c : list or tuple, optional
            Initial cell states of all the layers. Only applicable when `typ` is "LSTM"
        cudnn_opt : bool, optional
            If True, the CuDNN version of RNN will be used. Also, the generated symbol could only be
            used with GPU and `zoneout` cannot be used.
        name : str
            Name of the object
        """
    def step(self, data, prev_h=None, prev_c=None, seq_len=1, ret_typ="all"):
        """Feed the data sequence into the RNN and get the state symbols.

        Parameters
        ----------
        data : list or tuple or Symbol
            The input data. Shape: (seq_len, batch_size, data_dim)
        prev_h : list or tuple or Symbol or None, optional
            The initial hidden states. If None, the symbol constructed during initialization
            will be used.
            Number of the initial states must be the same as the layer number,
            e.g, [h0, h1, h2] for a 3-layer RNN
        prev_c : list or tuple or Symbol or None, optional
            The initial cell states. Only applicable when `typ` is 'lstm'. If None,
            the symbol constructed during initialization will be used.
            Number of the initial states must be the same as the layer number,
            e.g, [c0, c1, c2] for a 3-layer LSTM
        seq_len : int, optional
            Length of the data sequence
        ret_typ : str, optional
            Determine the parts of the states to return, which can be 'all', 'out', 'state'
            IMPORTANT!! When `cudnn_opt` is on, only the 'out' flag is supported.
            If 'all', symbols that represent states of all the timestamps as well as
             the state of the last timestamp will be returned,
                e.g, For a 3-layer GRU and length-10 data sequence, the return value will be
                     ([h0, h1, h2], [h0_9, h1_9, h2_9])
                      Here all hi are of shape(seq_len, batch_size, num_hidden[i]) and
                      all hi_j are of shape(batch_size, num_hidden[i])
                     For a 3-layer LSTM and length-10 data sequence, the return value contains both state and cell
                     ([h0, h1, h2], [c0, c1, c2], [h0_9, h1_9, h2_9], [c0_9, c1_9, c2_9])
            If 'out', state outputs of the layers will be returned,
                e.g, For a 3-layer GRU/LSTM and length-10 data sequence, the return value will be
                     [h0, h1, h2]
            If 'state', last state/cell will be returned,
                e.g, For a 3-layer GRU and length-10 data sequence, the return value will be
                     [h0_9, h1_9, h2_9]
                     For a 3-layer LSTM and length-10 data sequence, the return value will be
                     ([h0_9, h1_9, h2_9], [c0_9, c1_9, c2_9])

        Returns
        -------
        tuple
            States generated by feeding the data sequence to the network.

            If the `return_all` flag is set, states of all the timestamps will be returned.
            Otherwise states of all the timestamps will be returned.

        """

We decide to Pull Request this feature @leezu .
Should we create a new directory under "python/mxnet" like "operators" to store these kind of composed symbols? What do you think? @pluskid @piiswrong @tqchen @xlvector @sbodenstein

Great! Thanks a lot! We should long have this built in the standard package instead of bare-bone unroll function in examples. Currently since there are not going to be a lot of different Python composed symbols in the standard library, I guess simply put it as python/mxnet/rnn.py would be fine?
Also if I understand correctly, the class RNN is a constructor, whose step function needs to be called to compose a symbol, right? This is different convention from the cudnn RNN cell, as the cudnn RNN cell constructor itself is a composition function. Maybe we need to double think about the naming here to avoid confusion between the two cases.

@pluskid Yes, the RNN class here is a symbol constructor. I'm also thinking about the naming issue. May be call it "RNNFactory" in order to distinguish from the cudnn version?

@leezu Let's PR by this weekends and add some examples.

Great!
I remember CUDNN RNN cell need to transpose data before input, so what the input shape of this operator? And I find current NON-CUDNN version will also use 30%~50% GPU-Util, and its not easy to use 100% GPU. You mentioned you have done speed test against CUDNN version, is there any reports?

@xlvector I find that cudnn will be 3/6 times faster than the original implementation. The input shape is chosen to be the same as cudnn, i.e, (seq_len, batch_size, data_dim)

The iterators defined in https://github.com/dmlc/mxnet/blob/master/example/rnn-time-major/bucket_io.py are helpful to more easily get the correct (time-major) input shape. One could also add (a more general version) to https://github.com/dmlc/mxnet/blob/master/python/mxnet/io.py . What do you think?

My understanding is that RNN input and output shape is already defined in infershape API in RNN-inl.h. So I am using them in my RNN implementation.
// data: [sequence len, batch, input dimension]
// Hidden shape is dim [total_layers, batch, state_size]
// Cell shape is dim [total_layers, batch, state_size]
// output: [sequence len, numdirection, num_direction * state_size]
// outStateShape: [layer_num, batch, state size]

@mli

@sxjscience: you should be explicit by what dropout you are referring to. Is it the version from "
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks" (https://arxiv.org/pdf/1512.05287.pdf), or from the older and less good (arXiv:1409.2329v5)? (or support both as options)

@sbodenstein Currently we've just coded the old version. The dropout method in the NIPS paper should be added later. Also, the performance boost in the paper is partially due to the dropout method for the embedding layer and we can also support it.

@zhenlinguo The problem may be that by doing this we cannot support stacked RNNs with different number of hidden state sizes. It should be better to separate the weight, bias and states.

Hi all, I have one question, if I run RNN layer and return is combination of kOut, kStateOut, kStateCellOut, then in python, how I can get the topright of kStateOut and kStateCellout? What structure or API I can use? In rnn_cell_demo, this sentence is used after call RNN.
hidden = mx.sym.Reshape(data=rnn, shape=(-1, num_hidden))
But I don't quite understand it. For seq2seq, h, y, c will be output, but how I can just get topright of h and c from stream?

@sxjscience Hi, What do you mean by have "3/6 times faster" speed up with CUDNN? 3-6 times speed up?

@magic282 Yes, cudnn is faster.

@sxjscience I strongly suggest that we have a benchmark for RNN related models, such as RNN LM, s2s, and attention, comparing with popular Theano and TF. Actually I have implemented s2s+attention NMT model with mxnet and got state-of-the-art result on IWSLT data. However, the training speed is not very fast. According to this benchmark https://github.com/guolinke/deep-learning-benchmarks/blob/master/result.md , mxnet is faster than other tools for the FCN models but slower for LSTM. I am really curious about the reason. But this is just a reference since I don't really believe that Theano is much faster than Torch.
And about the CUDNN speed up. Our group has built a DL framework from scratch and we are planning to leverage CUDNN to speed up training. We can only get 2 times speed up at most since our tool is already very fast for RNN models (because of some optimization for RNN).

@magic282 Yes, we need to include such a benchmark. Would you mind sharing your implementation? In fact, we haven't got an s2s + attention example yet. It will be easier for us to investigate whether the speed is not that satisfactory if we have the example code.

@sxjscience Sure, I will refactor the code and share it and hope that we can speed it up.

@magic282 , do you have early code which call RNN layer rather than lstm_unroll python? I already implemented mkl based RNN for CPU and going to release later. But I am lack of s2s model to test the perf.

@zhenlinluo Nope.

@magic282  since you have s2s model based on cudnn RNN layer, could you pls help to answer my question about what need to be returned after call mx.sym.RNN(xxxx, state_output=True) in encoder and decoder func? Since output Tblob will include kOut, kStateOut and kCell, but I just need to return 2D array kStateOut and kCell on top-right cell as input to decoder, how to do that?

@zhenlinluo I don't have s2s base on cudnn RNN symbol since I implemented LSTM or GRU using the basic OP.

@mli  @piiswrong  Do you know how to get specific data when outputs are multiple  in python?

@sxjscience Hi, I have uploaded the s2s + attention code. It might have some bugs since this time I can only get 44 BLEU score. Link, https://github.com/magic282/MXNMT

@magic282 Is it possible to revise the code to run the WMT14 or WMT15 dataset? We'd better do pair-test with TensorFlow.

Actually I am not very familiar with MT task. Could you please provide the data format? Is it the reference number issue?

@magic282 We may need to refer to their seq2seq example. https://www.tensorflow.org/versions/r0.12/tutorials/seq2seq/index.html

Regarding this discussion (re: Yarin Gal's RNN implementation):

@sxjscience: you should be explicit by what dropout you are referring to. Is it the version from "
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks" (https://arxiv.org/pdf/1512.05287.pdf), or from the older and less good (arXiv:1409.2329v5)? (or support both as options)

@sbodenstein Currently we've just coded the old version. The dropout method in the NIPS paper should be added later. Also, the performance boost in the paper is partially due to the dropout method for the embedding layer and we can also support it.

According to Yarin, the of the paper, his implementation is already in  Keras, TensorFlow, and Torch. I think it basically uses the same dropout mask across timesteps on RNNs. The implementations might include examples to handle RNN layers of different sizes, and I also speculate the potential of allowing for uncertainty in RNN predictions as that research continues.
See:
https://github.com/yaringal/BayesianRNN/issues/3

@predict-r Thanks very much! I'm busy with my PQE exam and need to work on this after I finish this. I find that the "dropout" used in the paper is actually "drop-connect" and I'm thinking to keep its original name.

@sxjscience Sorry for late reply. I think the code will run if we have parallel corpus. Did it fail when you try it?

@sxjscience It might be called DropConnect, but apparently its even older. Here is Yann LeCun's writeup on the history:   https://www.facebook.com/yann.lecun/posts/10154058859142143 .

@sxjscience Hey, what is the progress on the API for RNN? Is it complete or still in progress?

@karishmamalkan Still in progress, decide to add also the example of batchnorm. Will finish it after I finish the PQE. You can view some codes here. https://github.com/ML-HK/mxnet/blob/master/python/mxnet/recurrent.py

@sxjscience Hey, is this a working version of the code. When i try to import recurrent.py, i get an error about importing "utils".. Is something missing?


Update scalatest-flatspec, ... to 3.2.3

Updates

org.scalatest:scalatest-flatspec
org.scalatest:scalatest-shouldmatchers

from 3.2.2 to 3.2.3.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scalatest" } ]

labels: test-library-update, semver-patch

Codecov Report

Merging #112 (b6ee88a) into master (944becd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #112   +/-   ##
=======================================
  Coverage   88.25%   88.25%
=======================================
  Files          10       10
  Lines         247      247
  Branches       10       10
=======================================
  Hits          218      218
  Misses         29       29

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 944becd...b6ee88a. Read the comment docs.


[Bug] InteractiveBrowserCredential not failing in non-interactive environments

Package Name: @azure/identity
Package Version: 2.1.0
Operating system: MacOS 12.4
[x] nodejs

version: 16.14.0

[ ] browser

name/version:

[ ] typescript

version:

Is the bug related to documentation in

[ ] README.md
[ ] source code documentation
[ ] SDK API docs on https://docs.microsoft.com

Describe the bug
When using InteractiveBrowserCredential in a non-UI or non-interactive environment (CI, GitHub Actions...) it's stuck waiting indefinitely for an interactive login. When using ChainedTokenCredential, I would expect it to fail and fall back to the next credentials method chained.
To Reproduce
Steps to reproduce the behavior:

Set up an app to log in with something like this:

  const browserCredential = new InteractiveBrowserCredential({
    redirectUri: `http://localhost:31337`
  });

Try to run the app in a Docker container, for example.

Expected behavior
InteractiveBrowserCredential fails because of non-UI or non-interactive environment.
Additional context
Because of this issue, in SWA CLI (https://github.com/Azure/static-web-apps) the app never fall back to using DeviceTokenCredential in non-UI environment or gets CI stuck indefinitely in non-interactive environments.

@sinedied I'm trying to understand your scenario, is the idea that this code needs to run both in the browser for customers (using InteractiveBrowserCredential) but also in CI? Can I ask how you are configuring ChainedTokenCredential?
As a possible workaround, would it be possible for you to switch with credential you use based on the current environment?
Perhaps the fix could be allowing developers to pass a timeout to InteractiveBrowserCredential so we don't block indefinitely?

I'm curious to hear more about "If a CI or non-interactive environment is detected", specifically what checks you are thinking of having us perform. Perhaps something like this? https://github.com/sindresorhus/is-interactive/blob/dc8037ae1a61d828cfb42761c345404055b1e036/index.js
It is concerning that InteractiveBrowserCredential is hanging forever though, since I would expect it to throw a CredentialUnavailableError and proceed with the next credential in the chain, as you suggest.
I'm a bit suspicious of the open dependency, given that it still hasn't addressed a previous issue: https://github.com/sindresorhus/open/issues/287

Perhaps something like this? https://github.com/sindresorhus/is-interactive/blob/dc8037ae1a61d828cfb42761c345404055b1e036/index.js

Yes exactly!  Regarding the open dependency I do not know a good alternative, but yes it has issues (we also faced some with the SWA CLI). For the InteractiveBrowserCredential issue though, it might be possible to check the spawned child process result to see if the browser was effectively opened or not?

@sinedied We do not support InteractiveBrowserCredential in a non-interactive environment or CI-like environment. It's also not part of ChainedTokenCredential.

@KarishmaGhiya Is this specified somewhere in the docs? I couldn't find any reference to that information, for example here: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/identity/identity/samples/AzureIdentityExamples.md

Currently, Interactive Browser Credentials involve a redirect and it doesn't preserve the state of the application. As such, it may not fall back to using the next credential in the loop of ChainedTokenCredential. It is a drawback of the credential. So this may not work as desired. However, are you experiencing the same issue of IBC not working at all when used as is? @sinedied

Also, @sinedied do you think this issue was related? Or is this one different? https://github.com/Azure/azure-sdk-for-js/issues/21726

Also, @sinedied do you think this issue was related? Or is this one different? #21726

I think it was a different one, as the cause was a dangling promise due to the open dependency.

Currently, Interactive Browser Credentials involve a redirect and it doesn't preserve the state of the application. As such, it may not fall back to using the next credential in the loop of ChainedTokenCredential. It is a drawback of the credential. So this may not work as desired. However, are you experiencing the same issue of IBC not working at all when used as is? @sinedied

When used in the correct environment (ie interactive terminal, with a accessible UI) IBC works as intended. Knowing that it's not supported in ChainedTokenCredentials means that we'll probably have to put extra logic from our side to detect non-interactive environment, but the @azure/identity documentation should be updated to precise that.
However, taking non-interactive environments aside, there's still I issue when using InteractiveBrowserCredential alone when a UI environment is not available: it's stuck until the timeout, whereas I would expect to catch the open error returned if a browser could not be opened, and let me know via an error or exception. Otherwise, I have no way to do the fall back on my side. Would that be an acceptable change?

Setting aside the feature work of detecting non-interactive environments and exiting early, it seems like we should be handling spawn failing to open the browser.
We don't do anything complicated with open: https://github.com/Azure/azure-sdk-for-js/blob/05174d3f5b3c78593d0a28fdaad838abffaccdf8/sdk/identity/identity/src/msal/nodeFlows/msalOpenBrowser.ts#L247
I think what might be happening is that there is a bug with non-zero exit codes in open: https://github.com/sindresorhus/open/blob/05ba9e150cc1a2629e518a9cc19b586c6ca3f269/index.js#L212
I believe the behavior for allowNonzeroExitCode is flipped on accident right now, so it's resolving rather than rejecting. Perhaps we could try looking at the returned object's exitCode and checking if it's non-zero?

Perhaps we could try looking at the returned object's exitCode and checking if it's non-zero?

That would be helpful I think for solving this! Another option would be to propose a PR to fix the issue in open as negative error codes are not considered errors.

That would be helpful I think for solving this! Another option would be to propose a PR to fix the issue in open as negative error codes are not considered errors.

https://github.com/sindresorhus/open/pull/296

@sinedied Is this issue resolved with the PR merging to open library? Do you need anything else from the sdk side?

I juste checked in Codespaces with v3.2.4 of @azure/identity, this is the error I get (coming from the identity SDK), so it's not better


ARR: Fix assignment quotas

This PR resolves an issue where, if the 3rd edge (out of the Assignment + Invite_Assignment edges) posted to a paper is an Invite_Assignment edge, the process function fails because the pre-process saw it as posting a 4th edge.
The pre-process functions are changed to exclude the posted edge from the edge count if there is already an edge with that ID.
To allow the Assignment_Recruitment process function to complete, only non-accepted Invite_Assignment edges are counted. The logic is still preserved because if an Invite_Assignment edge is accepted, then there must be an Assignment edge which are counted

There's a request to be able to increase the quota for some papers where the reviews may have been AI generated and need extra reviewers - I'm thinking of a way to properly implement this


timemachines/timemachines/optimizers/requirements.txt

add requirements.txt for optimizers
includes optimizers that are imported but not currently used by alloptimizers.py, e.g. fbprophet, hyperopt
locally I can install fbprophet with conda install fbprophet but not pip install fbprophet . but probably a local issue and in any event maybe not used

Aside: I've also had trouble pip installing prophet although upgrading pip and installing pandas first seems to help.

very cool stuff!
this is my mad science project, so trying a few more optimizers was very
helpful
https://github.com/druce/swr/blob/master/optimize.ipynb
basically the idea is optimize safe withdrawal … write a function that
takes a stock/bond allocation, and a couple of withdrawal rates, evaluates
how it would do historically (or Monte Carlo) using CRRA
certainty equivalent cash flows (real cash flows discounted for
volatility). gives reasonable answers, historically works a little better
than Bengen '4% rule'. i.e. can spend more without running out if you
tolerate a little variability. for simple rules gradient-based Powell or
L-BFGS-B works too. Anyway will blog about it sometime!
will follow your stuff with interest!
On Mon, Feb 8, 2021 at 9:06 AM Peter Cotton<EMAIL_ADDRESS>wrote:

Merged #6 https://github.com/microprediction/timemachines/pull/6 into
main.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/microprediction/timemachines/pull/6#event-4303604983,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJDX3VHF34Y3XEPYI4ZMI3S57VWPANCNFSM4XGTXAZA
.

Very cool. If you want we could include it as one of the objectives - always looking for real-world objective functions rather than the fake ones.

thanks Peter! still very much a work in progress but eventually will blog
about and maybe can provide a canonical case/benchmark!
On Tue, Feb 9, 2021 at 9:52 AM Peter Cotton<EMAIL_ADDRESS>wrote:

Very cool. If you want we could include it as one of the objectives -
always looking for real-world objective functions rather than the fake ones.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/microprediction/timemachines/pull/6#issuecomment-775997286,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJDX3UEWIRGCSPGGTPLSSDS6FD33ANCNFSM4XGTXAZA
.


⚠ Turnos 24x7 has degraded performance

In ba966ee, Turnos 24x7 (https://reservas-dev.alternativasinteligentes.com/) experienced degraded performance:

HTTP code: 200
Response time: 968 ms

Resolved: Turnos 24x7 performance has improved in 8f40bf5.


Support JSONC

Is just a special case of JSON5
Fixes https://github.com/ajv-validator/ajv-cli/issues/103

@epoberezkin Is there a reason to not take this pull request?

@Jason3S Thanks! Actually the reformatting was done by this pre-commit hook, so I assume @epoberezkin wants them to be re-formatted 😄
https://github.com/ajv-validator/ajv-cli/blob/d53d73ec1a393a11eaebcdeadd93fd393072bf4f/package.json#L72

Fixed platform-dependency of build step

@epoberezkin could you please consider merging this PR. Changes are now back to minimal.

sorry for the delay, focus was clearly elsewhere...


feat(client): Add support for create_many to SQLite

Change Summary

#994
Now that every provider supports create_many, we do not need to distinguish between database providers
However, not every provider supports skip_duplicates, so this is now tracked in tests

Checklist

[x] Unit tests for the changes exist
[x] Tests pass without significant drop in coverage
[ ] Documentation reflects changes where applicable
[x] Test snapshots have been updated if applicable

Agreement
By submitting this pull request, I confirm that you can use, modify, copy and redistribute this contribution, under the terms of your choice.

thanks @RobertCraigie, i've addressed your feedback. could you please re-run the tests? it’s the same transient error as last time breaking all the windows tests (incomplete read while downloading node binary)

I'm confused, why are windows tests failing on this PR, but passing on my fork? 😕

I think it's just unlucky network issues :(
ile "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\http\client.py", line 626, in _safe_read

    chunk = self.fp.read(min(amt, MAXAMOUNT))

  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\socket.py", line 704, in readinto

    return self._sock.recv_into(b)

  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\ssl.py", line 1242, in recv_into

    return self.read(nbytes, buffer)

  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\ssl.py", line 1100, in read

    return self._sslobj.read(len, buffer)

ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

nodeenv installation failed; You may want to try installing `nodejs-bin` as it is more reliable.


@ApiModel has no way to hide properties on super classes and compile time enhanced classes

@ApiModelProperty allows hiding properties that in my model class, but there is no way to hide properties brought in by super classes or compile time enhanced classes.
For instance, Ebean enhances its model classes, and when they are used as a response Class, swagger-ui shows them with an extra property "_ebean_intercept (EntityBeanIntercept, optional)", and all of its properties and sub-properties.

will investiagte for m2

This has been fixed as of 1.5.2-M1

I probably missed something, but how is it fixed? Is it now possible to write something like bthule suggested:
@ApiModel(hideProperties = {"uselessAnnoyingProperty", ...})?
I can't seem to find that feature.

@aciganj use @ApiModelProperty(hidden=true)

You mean I should add a property of the same name as the one I want to hide and then use that annotation?  I guess that would work for private properties, but what about others? Is there a way to do it without adding properties or getters to subclass?

@aciganj Did you try @ApiModel(properties = {@ApModelProperty(name="uselessAnnoyingProperty", hidden=true), ...})

I see @aciganj, I misunderstood your question.  Please try what @bthule suggested and if it doesn't work, open a new ticket.

@bthule , @fehguy
@ApiModel doesn't have the properties option, and other options don't seem useful for this purpose. I'll create a new issue

it is a very important issus for me, Please reopen the issue.

it is a very important issus for me, Please reopen the issue.

Hi, I think the same and I tried implementing some workaround for it in this pull request https://github.com/swagger-api/swagger-core/pull/2996, but unfortunately, Swagger team does not allow open contribution, so it will never be official. But if you do not intend to use Swagger forever, maybe it will help for now.

I want to second @wenyi189 , having support for this case would be exceptionally beneficial for anyone using third party libraries or who cannot access parent classes to annotate fields they would like to ignore. This definitely seems within the scope of Swagger's problem space.

Same thing here, I have a common resource class with some endpoints that should not come out in swagger for a particular project, but it should for others.  I can't currently do that without overriding the endpoint methods in the one project.  I shouldn't have to do that though just for documentation purposes.

I think the 'hide-superclass-fields' feature would be a good value add for documentation

The year 2020 was ending ... I'm amazed at how much time has passed since this feature was designated as very important. And just as amazed that she still does not exist. We urgently need to reopen this task.
For example, there is such a thing Spring HATEOAS. In which resources are sometimes superfluous are all sorts of different things. Which it would be nice not to show in the documentation for the request body. So, the swagger does not provide an opportunity to customize it. This is the answer to a six-year old question as an example. And in general, to be honest, how old is the swagger and how many problems it has. It clearly doesn't look like a java library =/

This is a very important feature. From 2 aspects
Consider basic code below
class Base {
    private int uselessparameter for docs ;
}
class Inherited extends Base {
    private int usefulparameter from docs;
}

Case 1)
Now Base may be in my control, so I should be able to use @APIIgnore and it should work.
Case 2)
Base is a library and I want to hide it when I extend Base in Inherited. Even then I should be allowed that.

I tried annotating my super class bean property with @Schema(hidden = true) which didn't work, but adding it to the getter method of that property got it to no longer show up in Swagger UI.


fix(app): Fix the compile error for JS Binding

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (ec0e50d) 89.06% compared to head (c7919b8) 89.06%.

:exclamation: Current head c7919b8 differs from pull request most recent head f8a49f8. Consider uploading reports for the commit f8a49f8 to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #676   +/-   ##
=======================================
  Coverage   89.06%   89.06%
=======================================
  Files         103      103
  Lines        4150     4150
=======================================
  Hits         3696     3696
  Misses        310      310
  Partials      144      144

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


MudSelectExtended: Bug With Virtualize and SelectedItems Style

Hy, I'm using MudSelectExtended component with record SelectItem and MultiSelection set to true. After selecting two or more option closing select and reopening select only the last selected item is marked as selected. Is this bug or I'm doing something wrong?
<MudSelectExtended
        T="SelectItem"
        MultiSelection="true"
        MultiSelectionTextFunc="@(new Func<List<SelectItem>, string>(GetMultiSelectionText))"
        ToStringFunc="@(item => item.Text)"
        Label="Multi select"
        @bind-SelectedValues="BoundItems"
        ItemCollection="Items"
        SearchBox="true"
        Virtualize="true"
        Clearable
        AnchorOrigin="Origin.BottomCenter"
        Variant="Variant.Outlined"/>

@code {

        private ICollection<SelectItem> Items = new List<SelectItem>()
        {
                new(1, "Item 1"),
                new(2, "Item 2"),
                new(3, "Item 3"),
                new(4, "Item 4"),
                new(5, "Item 5"),
                new(6, "Item 6"),
        };

        public IEnumerable<SelectItem> BoundItems { get; set; }

        private string GetMultiSelectionText(List<SelectItem> selectedValues)
        {
                return $"{string.Join(", ", selectedValues.Select(x => x.Text))}";
        }

        public record SelectItem(int Id, string Text);
}

It look like a Virtualization problem. If you set Virtualize="false" it look like work.
Actually i have no idea why this isn't working for now, we just implement the blazor virtualize component. We need to look deeper about virtualize, if you have any idea about this please prepare a PR.

Thanks for reply and for providing workaround for now. At the moment I don't have any idea how to fix this, since I didn't use blazor virtualize on my own.
Workaround isn't great, but it's good for situation when there is no too much items.

+1
Btw, checked Virtualize with core MudSelect with Multiple param, and everything works.
I've tried to do some kind of triage with source, but till now I had no idea about that bug too.
Imo, that's not necessary a problem of Virtualize component, but this can be mistaken.
I think have virtualize working on it's component is most crucial part.
@mckaragoz Any idea? 😃

Btw, checked Virtualize with core MudSelect with Multiple param, and everything works.

MudSelect has virtualize?

@mckaragoz It's via ChildContent.

https://learn.microsoft.com/en-us/aspnet/core/blazor/components/virtualization?view=aspnetcore-5.0#state-changes-2
Maybe this is related?

MudSelect has virtualize?

It's via ChildContent.

Understand i didn't try with the MudSelect via render fragment, did you try same with extended with render fragment also?
Item highlight is controlled by UpdateSelectedStyles method.

I did, and neither <Virtualize> nor <MudVirtualize> ain't work.
I saw implementation of UpdateSelectedStyles and think that bug somewhere here with changing state on select or open.
Component classes are heavy and seems without your guidance there no go.

Yeah, writing these codes was very hard for me, because too many rendering process start over and over and i also can't follow them for now. I saw the virtualize problem at the end of the PR but still no idea where is the problem.

Selected: @string.Join(", ", selected)
<MudDivider Class="my-5" />
<MudButton OnClick="e => { open = !open; }">Toggle</MudButton>
<MudButton OnClick="e => { list?.ForceUpdate(); }">Force Update</MudButton>

<div style="display: @(open ? "block" : "none");">
    @* Place MudListExtended here instead of MudPopover. *@
    @* Here we don't' need to ForceUpdate() when opened. *@
</div>
<MudPopover @bind-Open="open" AnchorOrigin="Origin.BottomLeft" MaxHeight="240" Class="overflow-y-auto">
    @* Here we need to ForceUpdate() when opened. *@

    <MudListExtended @ref="list" @bind-SelectedValues="selected"
                     ItemCollection="items"
                     MultiSelection="true" Clickable="true" SearchBox="true"
                     Virtualize="true" />
</MudPopover>

@code {
    bool open;
    MudListExtended<string>? list;
    List<string> items = new();
    IEnumerable<string> selected = Enumerable.Empty<string>();

    protected override void OnInitialized()
    {
        for (int i = 0; i != 1000; i++)
        {
            items.Add($"Item {i}");
        }
    }
}

I think that it's something related with component render events.

@antonKrizmanic this is a workaround for you:
<MudSelectExtended
        T="SelectItem"
        @ref="_select"
        MultiSelection="true"
        MultiSelectionTextFunc="@(new Func<List<SelectItem>, string>(GetMultiSelectionText))"
        ToStringFunc="@(item => item?.Text)"
        Label="Multi select"
        @bind-SelectedValues="BoundItems"
        ItemCollection="Items"
        SearchBox="true"
        Virtualize="true"
        Clearable
        AnchorOrigin="Origin.BottomCenter"
        Variant="Variant.Outlined"
        OnOpen="OnOpen"/>

@code {
    MudSelectExtended<SelectItem> _select;
    private ICollection<SelectItem> Items = new List<SelectItem>()
        {
                new(1, "Item 1"),
                new(2, "Item 2"),
                new(3, "Item 3"),
                new(4, "Item 4"),
                new(5, "Item 5"),
                new(6, "Item 6"),
        };

    public IEnumerable<SelectItem> BoundItems { get; set; }

    private async Task OnOpen()
    {
        await Task.Delay(100);
        await _select.ForceUpdate();
    }

        private string GetMultiSelectionText(List<SelectItem> selectedValues)
        {
                return $"{string.Join(", ", selectedValues.Select(x => x.Text))}";
        }

        public record SelectItem(int Id, string Text);
}

@mightyvoodoo if we add UpdateSelectedStyles at the end of the OnAfterRenderAsync, the visual works clearly, but it causes to infinite render loop at some situations.
I fixed the virtualize scroll bug by adding UpdateSelectedStyles to onscroll. But it still have bug if you press arrow keys to select previous next item.

Check out my PR #240, I have fixed a few issues to do with selection and virtualisation in it.

The main issue looks fixed. If you have another issue relates to this, please re-open this issue or open new issue.


Test python 3.10

I got this to a point where there's only 1 failing test for tests/meltano/core/block/test_extract_load.py::TestExtractLoadBlocks::test_link_io.
It seems like the test doesn't like CliRunner's call to cli.main here: https://github.com/pallets/click/blame/9e9fe41a53d885d96e43dec7cd9eb69e352f801a/src/click/testing.py#L408.
Running the tests like
poetry run pytest \
  tests/meltano/core/test_environment_variables.py \
  tests/meltano/core/block

fails, but running them like
poetry run pytest \
  tests/meltano/core/block \
  tests/meltano/core/test_environment_variables.py

just works, so it's definitely something about CliRunner altering stdout/stderr streams for subsequent tests.

Other changes:

Bump to earliest pytest with support for Python 3.10
Bump pytest-asyncio to support newer pytest APIs
Fixed logic for getting the site-packages dir on 3.10 (was doing something like "3.10"[:2]")

@WillDaSilva I see you picked up https://github.com/meltano/meltano/issues/2830. I started this a while ago but got stuck with what I described above. So, this can be a starting point if you're interested.

I got this to a point where there's only 1 failing test for tests/meltano/core/block/test_extract_load.py::TestExtractLoadBlocks::test_link_io.

This was caused by some tests and fixtures changing the root logging level during their execution, but not resetting it afterwards. That affects the ExtractLoadBlocks instances in their _link_io method: https://github.com/meltano/meltano/blob/de834382363782f30500b1a62cb65d4dbf0bb888/src/meltano/core/block/extract_load.py#L576
I'm closing this PR because it has been superseded by #6293, which includes the changes from this branch. Thanks for getting this started @edgarrmondragon!


[plugin-relay] nodeTypeOptions is of type ObjectTypeOptions instead of InterfaceTypeOptions

I expected the type of relayOptions.nodeTypeOptions to be InterfaceTypeOptions instead of ObjectTypeOptions, because it configures the Node interface.
I noticed this, when adding a required property to ObjectTypeOptions in a plugin and TypeScript started to complain, that I did not define this property in relayOptions.nodeTypeOptions.

This is a bug.  I'll get a fix out soon! Thanks for reporting

This should be fixed in the latest release, let me know if you have an more issues with this after upgrading!

I can confirm that this issue has been fixed. Thank you!


4 classes model;  targeted attacks are not successful.

Follow up from https://github.com/fra31/auto-attack/issues/76
Since I have a model trained in only 4 classes I adapted the number of targets t.
I changed this block:    https://github.com/fra31/auto-attack/blob/6482e4d6fbeeb51ae9585c41b16d50d14576aadc/autoattack/autoattack.py#L265 to
        elif version == 'standard_4':
            self.attacks_to_run = ['apgd-ce', 'apgd-t', 'fab-t', 'square']
            if self.norm in ['Linf', 'L2']:
                self.apgd.n_restarts = 1
                self.apgd_targeted.n_target_classes = 3 # 9
            elif self.norm in ['L1']:
                self.apgd.use_largereps = True
                self.apgd_targeted.use_largereps = True
                self.apgd.n_restarts = 5
                self.apgd_targeted.n_target_classes = 3 # 5
            self.fab.n_restarts = 1
            self.apgd_targeted.n_restarts = 1
            self.fab.n_target_classes = 3 # 9
            #self.apgd_targeted.n_target_classes = 9
            self.square.n_queries = 5000

where t = classes - 1 = 4 - 1 = 3, so that I dont get any warning.
It seems that  APGD-CE is creating most perturbations.
using standard_4 version including apgd-ce, apgd-t, fab-t, square
initial accuracy: 93.00%
apgd-ce - 1/1 - 458 out of 465 successfully perturbed
robust accuracy after APGD-CE: 1.40% (total time 112.7 s)
apgd-t - 1/1 - 0 out of 7 successfully perturbed
robust accuracy after APGD-T: 1.40% (total time 122.2 s)
fab-t - 1/1 - 0 out of 7 successfully perturbed
robust accuracy after FAB-T: 1.40% (total time 133.2 s)
square - 1/1 - 0 out of 7 successfully perturbed
robust accuracy after SQUARE: 1.40% (total time 169.0 s)
max Linf perturbation: 0.00784, nan in tensor: 0, max: 1.00000, min: 0.00000
robust accuracy: 1.40%

Well, I think that I am not doing something wrong, right?

Looks fine to me. As sanity check you can run apgd-ce as last attack by modifying the order in self.attacks_to_run list.

On page 4 of this publication: https://arxiv.org/abs/2112.01601
Figure 3 shows an examination where the classifier is trained on 1000 classes.
Figure 4 shows an examination where the classifier is trained on 4 classes.
I am just wondering, that the plots should look more similar.


Issue #2 and partial Issue #3

Allows named colors by adding a third constructor to Color
Allow stroke-width  Attribute on shapes other than lines by adding strokeWidth to the PresentationAttributes.

@antarestrader I apologize. Apparently, I wasn't watching this repo (likely because it was a fork?) so I was never notified of your PR.

#3 has not been addressed by #12

Added in 9d7a53570335756496ba0276559100a20a7bfdc9


new and old sytle archives clobber each other

Issue
Archives have no duplicate detection in the manifest like other refables
dbt should error on duplicates, especially duplicate archive names, at parse time.
Issue description
In 0.14.0, we have a new archive definition feature where you declare archives in {% archive NAME %}...{% endarchive %} blocks, and we made archives refable. But I didn't update the graph loader code to detect duplicates, so there are two annoying consequences:

If you have an archive defined in dbt_project.yml and an archive with the same name as its target_table defined in an archives/*.sql file, the latter will always "win", silently
Duplicates where the archive name matches a model name will be detected, but only at ref time instead of parse time

System information
dbt 0.14.0a1
any OS
any Python
Steps to reproduce
in dbt_project.yml:
archive:
  - source_schema: dbt_jbeck
    target_schema: dbt_jbeck
    tables:
      - source_table: x
        target_table: x_archive
        updated_at: date
        unique_key: id

in archives/x_archive.sql:
{% archive x_archive %}
    {{
        config({'target_database': 'DBT_TEST', 'target_schema': 'dbt_jbeck', 'updated_at': 'date', 'strategy': 'timestamp', 'unique_key': 'id'})
    }}
    select * from DBT_TEST.dbt_jbeck.x
{% endarchive %}

Run dbt archive:
$ dbt archive
Running with dbt=0.14.0-a1
Found 1 models, 2 tests, 1 archives, 0 analyses, 107 macros, 0 operations, 2 seed files, 1 sources
14:12:24 | Concurrency: 4 threads (target='snowflake')
14:12:24 |
14:12:24 | 1 of 1 START archive x_archive --> DBT_TEST.dbt_jbeck.x_archive...... [RUN]
14:13:29 | 1 of 1 OK archived x_archive --> DBT_TEST.dbt_jbeck.x_archive........ [SUCCESS 100 in 64.21s]
14:13:29 |
14:13:29 | Finished running 1 archives in 81.02s.

Good catch @beckjake! Yes, let's add dupe-detection into the loader.
This is probably a less common case, but we also check for duplicate database/schema/identifier identifiers. This check ensures that two models aren't aliases to the same destination relation.
Do we need to do something special to include archives in this check? I imagine that would be especially appropriate since you can specify the target database/schema/table for archives explicitly

@drewbanin I thought that we didn't handle this, but I think that might already work already:
https://github.com/fishtown-analytics/dbt/blob/73607b85b7825b7fca01a340c52469317eaeae7a/core/dbt/loader.py#L184 - this gets called (eventually) by GraphLoader.load_all That means the only case we miss is when unique IDs generated by the two archive parsers collide, which should be a pretty easy fix!


Rename to fix cadl-ranch breaking change

Description
Add your description here!
Checklist
To ensure a quick review and merge, please ensure:

[ ] The PR has a understandable title and description explaining the why and what.
[ ] The PR is opened in draft if not ready for review yet.

If opened in draft, please allocate sufficient time (24 hours) after moving out of draft for review

[ ] The branch is recent enough to not have merge conflicts upon creation.

Ready to Land?

[ ] Build is completely green

Submissions with test failures require tracking issue and approval of a CODEOWNER

[ ] At least one +1 review by a CODEOWNER
[ ] All -1 reviews are confirmed resolved by the reviewer

Override/Marking reviews stale must be discussed with CODEOWNERS first

/check-enforcer evaluate

/azp run autorest.csharp


Elmasry DFS

Time: O((n+m) log n)
Space: O((ld3+Ɛ) n)
(This branch is based on testing-dfs (#11).)

Coverage increased (+4.8%) to 88.446% when pulling 9a0cebf343cd13b44d066b24504e8398915486ee on shsr04:ld3-dfs into 91ee705316822c80ea0140bb340190d6b7b23353 on thm-mni-ii:master.

One more commit incoming

Ready to merge


An unexpected error occurred<EMAIL_ADDRESS>Request failed "404 Not Found"".

Description
I just tried out the gatsby starter and I am getting a 404 error gatsby new
Steps to reproduce
run gatsby new my-tina-starter https://github.com/tinacms/gatsby-starter-tinacms
Expected result
A new gatsby starter is created
Actual result
error An unexpected error occurred<EMAIL_ADDRESS>Request failed "404 Not Found"".
And i cannot start with gatsby develop
Environment

MacOs, running from vsCode terminal

Refer to https://github.com/tinacms/gatsby-starter-tinacms/issues/1


[flowbroker] In node "switch" have an option to compare 2 values in msg

In node "switch" have an option to compare 2 values in msg, 2 keys in msg, in the payload, since they are of the same type and these types are limited to number, string, null and bool.
It is currently only possible to compare msg data with a fixed string or number.
For example, if I want to compare the value between ``payload.attrs.data.temperature>payload.attrs.data.staticMaxTemperature` to a use case where each device has a maximum temperature set per device in a static attr.

termometro1:

mosquitto_pub -h kei.dojot.com.br -p 1883 -t admin:5ece52/attrs  -m '{"temperatura":36.2}' -u admin:5ece52
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: Got a message in topic admin.device-data.
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: There are 1 callbacks registered.
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |messenger -- DEBUG: Received message: { value: <Buffer 7b 22 6d 65 74 61 64 61 74 61 22 3a 7b 22 74 65 6e 61 6e 74 22 3a 22 61 64 6d 69 6e 22 2c 22 64 65 76 69 63 65 69 64 22 3a 22 35 65 63 65 35 32 22 2c ... >,
flowbroker-7f9c448d6-xbthr flowbroker   size: 106,
flowbroker-7f9c448d6-xbthr flowbroker   key: <Buffer 61 64 6d 69 6e 3a 35 65 63 65 35 32>,
flowbroker-7f9c448d6-xbthr flowbroker   topic: 'admin.device-data',
flowbroker-7f9c448d6-xbthr flowbroker   offset: 18,
flowbroker-7f9c448d6-xbthr flowbroker   partition: 0,
flowbroker-7f9c448d6-xbthr flowbroker   timestamp:<PHONE_NUMBER>574 }
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |messenger -- DEBUG: Emitting new event message for subject device-data@admin
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: { 'e49ae5a0-2b0e-4150-83a2-78458f0bd1f9': [Function] }
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: Mapping device 5ece52 ===into===> event_queue2
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: Pre-processing event {"source":"device","message":"{\"metadata\":{\"tenant\":\"admin\",\"deviceid\":\"5ece52\",\"timestamp\":1636397390574},\"attrs\":{\"temperatura\":37.2}}"}
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: Queued event [object Object]
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |dev cache -- DEBUG: retriving data related to admin:5ece52 from cache
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [ingestor] got new device event: { metadata: { tenant: 'admin', timestamp:<PHONE_NUMBER>574 },
flowbroker-7f9c448d6-xbthr flowbroker   event: 'publish',
flowbroker-7f9c448d6-xbthr flowbroker   data:
flowbroker-7f9c448d6-xbthr flowbroker    { attrs: { temperatura: 37.2, MaxTemperatura: '35.0' },
flowbroker-7f9c448d6-xbthr flowbroker      id: '5ece52' } }
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [executor] will handle node A62fff2f63a4784:switch
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |switch -- DEBUG: Executing switch node...
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |switch -- DEBUG: Conditions: gt 35 and undefined
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |switch -- DEBUG: ... switch node was successfully executed.
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [executor] hop (switch) result: [{"payload":{"metadata":{"tenant":"admin","timestamp":1636397390574},"event":"publish","data":{"attrs":{"temperatura":37.2,"MaxTemperatura":"35.0"},"id":"5ece52"}}}]
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [executor] will handle node A29d3bb84cea24c:template
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |template -- DEBUG: ... template node was successfully executed.
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [executor] hop (template) result: [{"payload":{"metadata":{"tenant":"admin","timestamp":1636397390574},"event":"publish","data":{"attrs":{"temperatura":37.2,"MaxTemperatura":"35.0"},"id":"5ece52"}},"saida":{"aviso":"Temperatura atual (37.2°C) excedeu a temperatura máxima (35.0°C)!"}}]
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [executor] will handle node Af1af7df2a49f2:multi device out
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |multi device out -- DEBUG: Executing multi-device-out node...
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |multi device out -- DEBUG: Updating device...
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |multi device out -- DEBUG: Message is: { attrs:
flowbroker-7f9c448d6-xbthr flowbroker    { aviso: 'Temperatura atual (37.2°C) excedeu a temperatura máxima (35.0°C)!' },
flowbroker-7f9c448d6-xbthr flowbroker   metadata: { timestamp:<PHONE_NUMBER>667, tenant: 'admin' } }
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |messenger -- DEBUG: Trying to publish someting. Current producer topics are { 'dojot.device-manager.device': { admin: 'admin.dojot.device-manager.device' },
flowbroker-7f9c448d6-xbthr flowbroker   'device-data': { admin: 'admin.device-data' },
flowbroker-7f9c448d6-xbthr flowbroker   'dojot.ftp': { admin: 'admin.dojot.ftp' },
flowbroker-7f9c448d6-xbthr flowbroker   'dojot.notifications': { admin: 'admin.dojot.notifications' } }
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |multi device out -- DEBUG: ... device was updated.
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |multi device out -- DEBUG: ... multi-device-out node was successfully executed.
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [executor] hop (multi device out) result: undefined
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: Got a message in topic admin.device-data.
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: There are 1 callbacks registered.
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |messenger -- DEBUG: Received message: { value: <Buffer 7b 22 61 74 74 72 73 22 3a 7b 22 61 76 69 73 6f 22 3a 22 54 65 6d 70 65 72 61 74 75 72 61 20 61 74 75 61 6c 20 28 33 37 2e 32 c2 b0 43 29 20 65 78 63 ... >,
flowbroker-7f9c448d6-xbthr flowbroker   size: 166,
flowbroker-7f9c448d6-xbthr flowbroker   key: null,
flowbroker-7f9c448d6-xbthr flowbroker   topic: 'admin.device-data',
flowbroker-7f9c448d6-xbthr flowbroker   offset: 19,
flowbroker-7f9c448d6-xbthr flowbroker   partition: 0,
flowbroker-7f9c448d6-xbthr flowbroker   timestamp:<PHONE_NUMBER>670 }
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |messenger -- DEBUG: Emitting new event message for subject device-data@admin
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: { 'e49ae5a0-2b0e-4150-83a2-78458f0bd1f9': [Function] }
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: Mapping device 5ece52 ===into===> event_queue2
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: Pre-processing event {"source":"device","message":"{\"attrs\":{\"aviso\":\"Temperatura atual (37.2°C) excedeu a temperatura máxima (35.0°C)!\"},\"metadata\":{\"timestamp\":1636397390667,\"tenant\":\"admin\",\"deviceid\":\"5ece52\"}}"}
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |messenger -- DEBUG: Message produced successfully
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: Queued event [object Object]
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |dev cache -- DEBUG: retriving data related to admin:5ece52 from cache
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [ingestor] got new device event: { metadata: { timestamp:<PHONE_NUMBER>667, tenant: 'admin' },
flowbroker-7f9c448d6-xbthr flowbroker   event: 'publish',
flowbroker-7f9c448d6-xbthr flowbroker   data:
flowbroker-7f9c448d6-xbthr flowbroker    { attrs:
flowbroker-7f9c448d6-xbthr flowbroker       { aviso: 'Temperatura atual (37.2°C) excedeu a temperatura máxima (35.0°C)!',
flowbroker-7f9c448d6-xbthr flowbroker         MaxTemperatura: '35.0' },
flowbroker-7f9c448d6-xbthr flowbroker      id: '5ece52' } }
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [executor] will handle node A62fff2f63a4784:switch
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |switch -- DEBUG: Executing switch node...
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |switch -- DEBUG: Conditions: gt 35 and undefined
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- |switch -- DEBUG: ... switch node was successfully executed.
flowbroker-7f9c448d6-xbthr flowbroker <18:49:50 08/11/2021> -- DEBUG: [executor] hop (switch) result: [null]

termometro2:

mosquitto_pub -h kei.dojot.com.br -p 1883 -t admin:53c617/attrs  -m '{"temperatura":32}' -u admin:53c617
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: Got a message in topic admin.device-data.
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: There are 1 callbacks registered.
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |messenger -- DEBUG: Received message: { value: <Buffer 7b 22 6d 65 74 61 64 61 74 61 22 3a 7b 22 74 65 6e 61 6e 74 22 3a 22 61 64 6d 69 6e 22 2c 22 64 65 76 69 63 65 69 64 22 3a 22 35 33 63 36 31 37 22 2c ... >,
flowbroker-7f9c448d6-xbthr flowbroker   size: 104,
flowbroker-7f9c448d6-xbthr flowbroker   key: <Buffer 61 64 6d 69 6e 3a 35 33 63 36 31 37>,
flowbroker-7f9c448d6-xbthr flowbroker   topic: 'admin.device-data',
flowbroker-7f9c448d6-xbthr flowbroker   offset: 21,
flowbroker-7f9c448d6-xbthr flowbroker   partition: 0,
flowbroker-7f9c448d6-xbthr flowbroker   timestamp:<PHONE_NUMBER>685 }
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |messenger -- DEBUG: Emitting new event message for subject device-data@admin
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: { 'e49ae5a0-2b0e-4150-83a2-78458f0bd1f9': [Function] }
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: Mapping device 53c617 ===into===> event_queue9
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: Pre-processing event {"source":"device","message":"{\"metadata\":{\"tenant\":\"admin\",\"deviceid\":\"53c617\",\"timestamp\":1636397489684},\"attrs\":{\"temperatura\":32}}"}
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |dev cache -- DEBUG: retriving data related to admin:53c617 from cache
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: [ingestor] got new device event: { metadata: { tenant: 'admin', timestamp:<PHONE_NUMBER>684 },
flowbroker-7f9c448d6-xbthr flowbroker   event: 'publish',
flowbroker-7f9c448d6-xbthr flowbroker   data:
flowbroker-7f9c448d6-xbthr flowbroker    { attrs: { temperatura: 32, MaxTemperatura: '28.5' },
flowbroker-7f9c448d6-xbthr flowbroker      id: '53c617' } }
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: Queued event [object Object]
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: [executor] will handle node A62fff2f63a4784:switch
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |switch -- DEBUG: Executing switch node...
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |switch -- DEBUG: Conditions: gt 28.5 and undefined
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |switch -- DEBUG: ... switch node was successfully executed.
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: [executor] hop (switch) result: [{"payload":{"metadata":{"tenant":"admin","timestamp":1636397489684},"event":"publish","data":{"attrs":{"temperatura":32,"MaxTemperatura":"28.5"},"id":"53c617"}}}]
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: [executor] will handle node A29d3bb84cea24c:template
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |template -- DEBUG: ... template node was successfully executed.
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: [executor] hop (template) result: [{"payload":{"metadata":{"tenant":"admin","timestamp":1636397489684},"event":"publish","data":{"attrs":{"temperatura":32,"MaxTemperatura":"28.5"},"id":"53c617"}},"saida":{"aviso":"Temperatura atual (32°C) excedeu a temperatura máxima (28.5°C)!"}}]
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: [executor] will handle node Af1af7df2a49f2:multi device out
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |multi device out -- DEBUG: Executing multi-device-out node...
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |multi device out -- DEBUG: Updating device...
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |multi device out -- DEBUG: Message is: { attrs:
flowbroker-7f9c448d6-xbthr flowbroker    { aviso: 'Temperatura atual (32°C) excedeu a temperatura máxima (28.5°C)!' },
flowbroker-7f9c448d6-xbthr flowbroker   metadata: { timestamp:<PHONE_NUMBER>844, tenant: 'admin' } }
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |messenger -- DEBUG: Trying to publish someting. Current producer topics are { 'dojot.device-manager.device': { admin: 'admin.dojot.device-manager.device' },
flowbroker-7f9c448d6-xbthr flowbroker   'device-data': { admin: 'admin.device-data' },
flowbroker-7f9c448d6-xbthr flowbroker   'dojot.ftp': { admin: 'admin.dojot.ftp' },
flowbroker-7f9c448d6-xbthr flowbroker   'dojot.notifications': { admin: 'admin.dojot.notifications' } }
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |multi device out -- DEBUG: ... device was updated.
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |multi device out -- DEBUG: ... multi-device-out node was successfully executed.
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: [executor] hop (multi device out) result: undefined
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: Got a message in topic admin.device-data.
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: There are 1 callbacks registered.
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |messenger -- DEBUG: Received message: { value: <Buffer 7b 22 61 74 74 72 73 22 3a 7b 22 61 76 69 73 6f 22 3a 22 54 65 6d 70 65 72 61 74 75 72 61 20 61 74 75 61 6c 20 28 33 32 c2 b0 43 29 20 65 78 63 65 64 ... >,
flowbroker-7f9c448d6-xbthr flowbroker   size: 164,
flowbroker-7f9c448d6-xbthr flowbroker   key: null,
flowbroker-7f9c448d6-xbthr flowbroker   topic: 'admin.device-data',
flowbroker-7f9c448d6-xbthr flowbroker   offset: 22,
flowbroker-7f9c448d6-xbthr flowbroker   partition: 0,
flowbroker-7f9c448d6-xbthr flowbroker   timestamp:<PHONE_NUMBER>845 }
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |messenger -- DEBUG: Emitting new event message for subject device-data@admin
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: { 'e49ae5a0-2b0e-4150-83a2-78458f0bd1f9': [Function] }
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: Mapping device 53c617 ===into===> event_queue9
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: Pre-processing event {"source":"device","message":"{\"attrs\":{\"aviso\":\"Temperatura atual (32°C) excedeu a temperatura máxima (28.5°C)!\"},\"metadata\":{\"timestamp\":1636397489844,\"tenant\":\"admin\",\"deviceid\":\"53c617\"}}"}
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- |dev cache -- DEBUG: retriving data related to admin:53c617 from cache
flowbroker-7f9c448d6-xbthr flowbroker <18:51:29 08/11/2021> -- DEBUG: [ingestor] got new device event: { metadata: { timestamp:<PHONE_NUMBER>844, tenant: 'admin' },
flowbroker-7f9c448d6-xbthr flowbroker   event: 'publish',
flowbroker-7f9c448d6-xbthr flowbroker   data:
flowbroker-7f9c448d6-xbthr flowbroker    { attrs:
flowbroker-7f9c448d6-xbthr flowbroker       { aviso: 'Temperatura atual (32°C) excedeu a temperatura máxima (28.5°C)!',
flowbroker-7f9c448d6-xbthr flowbroker         MaxTemperatura: '28.5' },
flowbroker-7f9c448d6-xbthr flowbroker      id: '53c617' } }
flowbroker-7f9c448d6-xbthr flowbroker <18:51:30 08/11/2021> -- DEBUG: Queued event [object Object]
flowbroker-7f9c448d6-xbthr flowbroker <18:51:30 08/11/2021> -- |messenger -- DEBUG: Message produced successfully
flowbroker-7f9c448d6-xbthr flowbroker <18:51:30 08/11/2021> -- DEBUG: [executor] will handle node A62fff2f63a4784:switch
flowbroker-7f9c448d6-xbthr flowbroker <18:51:30 08/11/2021> -- |switch -- DEBUG: Executing switch node...
flowbroker-7f9c448d6-xbthr flowbroker <18:51:30 08/11/2021> -- |switch -- DEBUG: Conditions: gt 28.5 and undefined
flowbroker-7f9c448d6-xbthr flowbroker <18:51:30 08/11/2021> -- |switch -- DEBUG: ... switch node was successfully executed.
flowbroker-7f9c448d6-xbthr flowbroker <18:51:30 08/11/2021> -- DEBUG: [executor] hop (switch) result: [null]


Can't connect to official servers

Describe the bug
"Secure connection failed. The server uses an invalid security certificate. This communication could be intercepted by a third party.
After" when trying to connect to any of the official servers.
Version and platform:

OS: Windows 11
Syncplay version and build type: Syncplay 1.6.9 / 1.7.0
Media player and version:  MPV.NET / VLC

I just tried connecting and the security certificate worked fine. I'm using Windows 10.
[22:53:09] Attempting to connect to syncplay.pl:8999
[22:53:09] Attempting secure connection
[22:53:09] Successfully reached syncplay.pl (<IP_ADDRESS>)
[22:53:10] Secure connection established (TLSv1.3)
[22:53:10] Successfully connected to server

Please run Syncplay with the --debug command line switch and let me know of anything potentially relevant which gets put into the syncplay.log or .Syncplay.llog file in %APPDATA%.

syncplay MPV.log
syncplay VLC.log

[13:31:18] Attempting to connect to syncplay.pl:8995
[13:31:19] Attempting secure connection
[13:31:19] Successfully reached syncplay.pl (<IP_ADDRESS>)
[13:31:19] Secure connection failed. The server uses an invalid security certificate. This communication could be intercepted by a third party. For further details and troubleshooting see here.
[13:31:19] Disconnected from server

Do you get any errors when you visit the https://syncplay.pl/ website in your browser?

Do you get any errors when you visit the https://syncplay.pl/ website in your browser?

I just attempted to fix this issue by changing the way in which we tell pyOpenSSL where to find the certificate trust store (certifi). Could you try again with the build available here https://github.com/Syncplay/syncplay/actions/runs/3235129755 ? Thank you.

Glad to hear that fixed it :)


does deepspeed support pure bf16 training?

training 70B cost too large gpu memory
when training 70+B model, the cost of gpu memory is too large, as mentioned in the deepspeed deepspeed-readthedocs-io-en-stable.pdf, the total memory was about 18xN, N is the number of params, if offload, the cpu memory may need 2T+. the main factor of this is the 32bit of (adam optmizer+gradient+copy of model param)
expect deepspeed can support pure bf16 training
if using pure bf16, which means all param(model+gradient+optimizer) only using bf16, the cost of memory may down to 8xN, and in most cases, bf16 is enough, I have test this using llama-recipe(https://github.com/meta-llama/llama-recipes). the result of pure bf16 traing is very similar to deepspeed mixed precision, but llama-recipe using pure bf16 can train 70B in one node(8x80G A800) by freeze half layers
using mmap technology to offload optmizer states to disk
is it possible to offload gradient and optimizer to disk, and then using mmap load it quickly?
Additional context
Add any other context or screenshots about the feature request here.

Hi, @hjc3613 , you can offload to nvme instead of cpu memory,  please checkout out zero-infinity.

Hi, @hjc3613 , you can offload to nvme instead of cpu memory, please checkout out nvme offload.

thanks for your reply, I have test nvme offload, but failed, related issue:https://github.com/microsoft/DeepSpeed/issues/4888
I am confirm that my configs is correct, and nvme disk is installed normally

the pure bf16 is better than nvme offload, beacuase if can store all params、gradients、adam optimizer states in gpu memory and can run faster than any other offload method, as it only consume 1/3 gpu memory of deepspeed mixed precision training strategy

You can achieve that by setting  fp32_optimizer_states=False  in  initialization of DeepSpeedCPUAdam this param is added to deepspeed from version 0.14.3.
note: if you are using transformers trainer, it will create optimizer in its internal implemantation, you can't  set optimizer param  without hack to my knowledge.

thanks a lot，I will have a try

You can achieve that by setting  fp32_optimizer_states=False  in  initialization of DeepSpeedCPUAdam, this param is added to deepspeed from version 0.14.3.
note: if you are using transformers trainer, it will create optimizer in its internal implemantation, you can't  set optimizer param  without hack to my knowledge.

hi，i test this method, but gives me error: fp32_optimizer_states extra fields not permitted. means that deepspeed do not support this config。my deepspeed version is 0.14.4

make sure you are using DeepSpeedCPUAdam, you can find the signature here DeepSpeedCPUAdam

thank you, I use it in a wrong place

make sure you are using DeepSpeedCPUAdam, you can find the signature here DeepSpeedCPUAdam

i am running it success, but the only drawback is that this param only supported by cpuadam, which is already offload mode. but when offload, it actually doesn't need bf16, fp32 is more appropriate since cpu memory is sufficient. can none offload mode support pure bf16？i just want to save gpu memory with all parameters stay in gpu


Conecpt: TEST vs COVERAGE. Difference?

Hi I was following this tutorial: https://github.com/ApolloAuto/apollo/blob/master/docs/specs/apollo_build_and_test_explained.md
To test and coverage Apollo Modules. At first I have issued on terminal:
 bash apollo.sh test cyber
I had this output:

Afterwards, I have issued:
bazel coverage modules/planning/…
Anmd I had a very similar output:

In both cases I have a lot of information displayed on terminal and at the end "PASSED or FAILED info.
Then I do not understand the difference to use "test" and use "coverage" commands. Would somebody know to explain me which is ?

These are the bazel commands, according to the office manual.

Coverage Options Inherits all options from test.

so they're the same

I understand the fact that coverage inherits all options from test, as well as test inherits from build, however the last ones are not the same. When building apollo I do not have the same output messages: "PASSED OR FAILED" as I have issuing test.
Therefore I would like to understand if coverage has more commands or different goal as test.... I mean, I think that coverage is a different approach of test. And I would like to know this difference, if is there difference? Because if is not any difference, any additional info given by coverage command, why should it exist, besides test...it would be a redundant command right? Are these commands really the same?

I was wrong, This option is inherited from test, But they function differently
coverage | Generates code coverage report for specified test targets.
test | Builds and runs the specified test targets.

Hi @daohu527 thanks very much for the clarification. Just one last question before closing this issue (is not related). Does Apollo offers an own simulator to run along Apollo stack? I have watched this video: https://youtu.be/-47b4UQMblM
It seems pretty cool, I have also requested to test the simulation in Apollo commercial website, sending my e-mail. I have read the Dreamland section, Apollo Studio Premium Features (Cloud Services), scenarios edition....etc.
Then I ask myself if this Apollo Game Simulator is already available? Is it free or is it a premium feature?
Could you please share this info? If Premium how to request access? Our research team have interest...thanks in advance

Yes, there are related products but need to apply, you'd better contact the relevant personnel by email
The following is the simulation of the cloud platform
https://apollo.auto/platform/simulation.html

I am the same as you here, and the bash apollo.sh coverage does not output any report. After printing a lot of PASSED and FAILED, the script ends by itself, and the subsequent commands in the script are not actually executed.
Does anybody know how to fix it?


TypeError: Cannot read property 'vehicleInfo' of undefined

Hi,
Trying to get it working here. Changed the region code and my login to username (old Leaf 2012) got it a step further :)
But it now stops giving me "TypeError: Cannot read property 'vehicleInfo' of undefined"
Running Node 8.3.0
The login seems to be successfull cause the app on my phone notices that a login has been done on another device.
Any ideas? The same login info/zone work the the pycarwings2-package so the login should be ok. Something else that is wrong.. :)
Full error:
TypeError: Cannot read property 'vehicleInfo' of undefined
at getsessionid (/usr/lib/node_modules/carwings/lib/index.js:72:33)
at p (/usr/lib/node_modules/carwings/lib/index.js:133:33)
at /usr/lib/node_modules/carwings/lib/index.js:85:12
at next (native)
at step (/usr/lib/node_modules/carwings/lib/index.js:42:191)
at /usr/lib/node_modules/carwings/lib/index.js:42:361
at process._tickCallback (internal/process/next_tick.js:109:7)

Seems to be different responses around. To solve it with my EU 2012 modell I had to take away the VehicleInfoList-part of the response. Then it works fine.
function getsessionid(profile) {
// return profile.VehicleInfoList.vehicleInfo[0].custom_sessionid;
return profile.vehicleInfo[0].custom_sessionid;
}
function getvin(profile) {
// return profile.VehicleInfoList.vehicleInfo[0].vin;
return profile.vehicleInfo[0].vin;
}


add tutorials

Merging tutorials from https://github.com/couchbaselabs/couchbase-by-example to have everything in one place.

@jamiltz you should assign the PR to someone

This PR has 146 file changes. Honestly it is hard to review.

Ah yes, I should have mentioned that the content was already reviewed. This PR is to merge another repo into this one. So what needs to be reviewed is wether the file structure in this PR makes sense. Sorry for the confusion. I'll assign @zgramana as we chatted over skype about it already.

@jamiltz I'll review this today.

@jamiltz could you please remove the numeric prefix from the migrated folder paths?


Hawaii Map products are all blank

Aloha All! Loving the weather simulator. Unfortunately it doesn't look like its pulling data for any map based products for Hawaii. Instead of a map, its just a blank blue screen. The issue is present in both Firefox and Safari, on MacOS and Ubuntu 23.10.

@NeutralBias hello! Interesting. I updated the maps images a few years ago so it has probably been broken since. I don't have a good picture of Hawaii nor Alaska for that matter.
I suggest that you take a look at the forked projected which is using the older maps which work with Hawaii: https://weatherstar.netbymatt.com/


Preferred way to write custom target which is sanitizing NLog message

Type: Question
NLog version: 4.4.12
Platform: .Net 4.5
Current NLog config
<nlog>
  <targets>
    <default-wrapper xsi:type="AsyncWrapper">
      <wrapper-target xsi:type="SanitizingWrapper" />
    </default-wrapper>
    <target name="File" xsi:type="File" fileName="messages.log" encoding="UTF-8" layout="${message}" />
    <target name="Console" xsi:type="Console" layout="${message}" />
  </targets>
  <rules>
    <logger name="*" minlevel="Trace" writeTo="File"/>
    <logger name="*" minlevel="Trace" writeTo="Console" />
  </rules>
</nlog>

Hello,
I would like to write a custom wrapping target which is sanitizing NLog messages - what is the preferred way to modify AsyncLogEventInfo logEvent? I was thinking about following code:
protected override void Write(AsyncLogEventInfo logEvent)
{
    logEvent.LogEvent.Message = SanitizeMessage(logEvent.LogEvent.FormattedMessage);
    logEvent.LogEvent.Parameters = null;

    WrappedTarget.WriteAsyncLogEvent(logEvent);
}

but I'm not sure if this is a right way of doing it.
P.S. I don't want to do it via a custom layout as wrapping target is easier to configure once for all targets  (via the default-wrapper tag) and less error prone.
Thank you,
Best Regards,

this approach looks good to me!

Yeah, that seems to work, but I'm not sure about rewriting FormattedMessage to Message property and removing Parameters. Does FormattedMessage always contain the final message or does it depends on inner targets / layouts? What about exception logging (logger.Error(ex, "Message"))? What about performance (maybe creating a custom layout wrapper would be more reasonable - in that case a Message property is assigned only once)?

Does FormattedMessage always contain the final message or does it depends on inner targets / layouts?

It does contain the final version of ${message}. The exception is logger by ${exception}.
For a target wrapper this is the best approach, you need the formatted message and you can't set the formatted message (and so setting message is OK),
A better option is to create a layout renderer wrapper,  See for example: https://github.com/NLog/NLog/blob/master/src/NLog/LayoutRenderers/Wrappers/UppercaseLayoutRendererWrapper.cs

Great, thank you!

I've made some tests and wrapping layout is about 3 times slower than wrapping target (the same sanitizing method was used in both cases). Any idea why?
P.S. Input messages are rather big, 1MB and more.

@AfterSanitizing wrapping layout is about 3 times slower than wrapping target

Most performance issues can be solved by using a profiler, and then fix the places that are broken. You can consider to inherit from the WrapperLayoutRendererBuilderBase if you can do the tranformation inside the same StringBuilder (Maybe with some help from a local char-buffer).

How are you sanitizing? Regex is slow of course.

I'm not using Regex, instead I'm doing a loop trough the whole message and put valid chars / whole substrings into temporary StringBuilder. I'm not sure if I can use WrapperLayoutRendererBuilderBase, as I'm making an additional check to log level to run sanitization only on configured level.
Here is a fragment of my layout wrapper:
protected override void Append(StringBuilder stringBuilder, LogEventInfo logEvent)
{
	var inputMessage = RenderInner(logEvent);

	stringBuilder.Append(logEvent.Level <= _parsedMinLogLevel ? Transform(inputMessage) : inputMessage);
}

protected override string Transform(string inputText)
{
	return SanitizeMessage(inputText);
}

Maybe the performance difference comes from what you said - that LogEvent is reused between targets? The SanitizeMessage method is exactly the same in both target wrapper and layout wrapper.
Here is my nlog.config with all targets:
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      throwExceptions="true">

  <variable name="logDirectory" value="${basedir}/Logs/${shortdate}" />
  <variable name="logMessage" value="${level:uppercase=true} ${logger:uppercase=true} ${date} ${sanitizeMessage:inner=${message}}${onexception:${newline}${exception}" />

  <targets>
    <target name="FileAll" xsi:type="File" fileName="${logDirectory}/${shortdate}.log" layout="${logMessage}" encoding="UTF-8" />
    <target name="FileModule" xsi:type="File" fileName="${logDirectory}/${shortdate}_${logger}.log" layout="${logMessage}" encoding="UTF-8" />
    <target name="Console" xsi:type="Console" layout="${logMessage}" />
  </targets>

  <rules>
    <logger name="*" minlevel="Trace" writeTo="FileAll"/>
    <logger name="*" minlevel="Trace" writeTo="FileModule"/>
    <logger name="*" minlevel="Trace" writeTo="Console" />
  </rules>
</nlog>

@AfterSanitizing Have modified WrapperLayoutRendererBuilderBase to allow access to the LogEventInfo, so one can perform transformation based on LogEventInfo-details.

I assume your question has been answered, if not, let us know!


ruff format not formatting the files recursively but ruff format folder/folder1/folder2/folder3 works

This is a weird issue that i have encountered. I cannot also reproduce it in a similar folder structure outside the current project. I'll try to describe what happens.
I have a python project setup using pyproject.toml and the ruff configuration is as follows

[tool.pytest.ini_options]
testpaths = [
  "testSuite",
]

[tool.ruff]
preview = false

exclude = [
  "src/mymodule/somefolder/anotherfolder",
]

[tool.ruff.lint]
select = ["E", "F", "W", "NPY", "RUF", "I"]
ignore = ["E722", "RUF012", "E111", "E114", "E117", "E501", "W191", "E203"]

[tool.ruff.lint.per-file-ignores]
# Ignore `E402` (import violations) in all `__init__.py` files
"__init__.py" = ["E402"]

Issue
The files inside testSuite are not fomatted when running ruff format from the top level folder.
If i run ruff format testSuite/path/to/folder/inside/ then it works but only if the path supplied is the last level in the tree. See output below. Note how the last command formats the files but the first 2 don't.
(myenv) user@node123 project>ruff format .
1001 files left unchanged
(myenv) user@node123 project>ruff format testSuite/testCases/abcutTests/
67 files left unchanged
(myenv) user@node123 project>ruff format testSuite/testCases/abcutTests/HamstrTest/
6 files reformatted, 20 files left unchanged

System info

ruff version: 0.4.4
python: 3.11.4
system: Redhat-7.9

Any pointers here on what exactly i am doing wrong would be helpful.
Thanks!

Hmm that sounds strange. Any chance that you have a .gitignore or any other ignore file that excludes the testSuite directory?

there are .gitignore files but none of them exclude the testSuite directory itself. The existing .gitignore only have entries for test generated files that need to be ignored.
None of the .py files inside the testSuite folder are in the .gitignore

thanks for the hint, i think i might have found out what the issue is.
In one .gitignore an entry was set to **/Hamstr* and i guess this is causing ruff to ignore HamstrTest even though the .py files inside themselves were not in the .gitignore. ?

In one .gitignore an entry was set to */Hamstr and i guess this is causing ruff to ignore HamstrTest even though the .py files inside themselves were not in the .gitignore. ?

Yes, I think that's correct and should also match Git's behavior, as far as I understand, unless you added the files explicitly (or they were tracked before adding the ignore). We use the ignore library that handles this for us.

is there a way to see if ruff formatter s ignoring a file due to gitignore ?

ok, i have a reproducible example now
Folder structure
├── src
│   └── ruff_test
│       ├── __init__.py
│       └── classA.py
├── tests
│   └── NewTests
│       ├── __init__.py
│       └── test_func.py
├── .gitattributes
├── .gitignore
└── pyproject.toml

contents of .gitignore
**/New*/

Git
force add the NewTests folder git add -f NewTests
Ruff
Now run ruff -v format . at the same level as pyproject.toml The output is as follows
warning: `ruff <path>` is deprecated. Use `ruff check <path>` instead.
[2024-07-03][14:20:25][ruff::resolve][DEBUG] Using Ruff default settings
[2024-07-03][14:20:25][ignore::walk][DEBUG] ignoring /scratch/vm/testDir/ruff_test/tests/NewTests: Ignore(IgnoreMatch(Gitignore(Glob { from: Some("/scratch/vm/testDir/ruff_test/.gitignore"), original: "**/New*/", actual: "**/New*", is_whitelist: false, is_only_dir: true })))
[2024-07-03][14:20:25][ruff_workspace::resolver][DEBUG] Included path via `include`: "/scratch/vm/testDir/ruff_test/pyproject.toml"
[2024-07-03][14:20:25][ruff_workspace::resolver][DEBUG] Ignored path via `exclude`: "/scratch/vm/testDir/ruff_test/.ruff_cache"
[2024-07-03][14:20:25][ruff_workspace::resolver][DEBUG] Ignored path via `exclude`: "/scratch/vm/testDir/ruff_test/.git"
[2024-07-03][14:20:25][ruff_workspace::resolver][DEBUG] Included path via `include`: "/scratch/vm/testDir/ruff_test/src/ruff_test/__init__.py"
[2024-07-03][14:20:25][ruff_workspace::resolver][DEBUG] Included path via `include`: "/scratch/vm/testDir/ruff_test/src/ruff_test/classA.py"
[2024-07-03][14:20:25][ruff::commands::check][DEBUG] Identified files to lint in: 5.58252ms
[2024-07-03][14:20:25][ruff::diagnostics][DEBUG] Checking: /scratch/vm/testDir/ruff_test/pyproject.toml
[2024-07-03][14:20:25][ruff::commands::check][DEBUG] Checked 4 files in: 165.711µs
format:1:1: E902 No such file or directory (os error 2)

So the ignore library is removing the folder NewTests at the start and the files within NewTests are not shown even though they have been added to the repo.

Ruff doesn't check whether a file is added to your git repository. It only interprets your gitignore and determines whether a file should be excluded.
I recommend you update your git ignore to only ignore files that are not part of your repository. This also makes it less likely that someone forgets to commit a python file that's later added to NewTest.

Yup, this is what i did now. Just posting it here so that it's documented on why this is happening.  Someone else might be wondering the same as me in future :)


Go over all query params and make sure the naming is consistent

[x] https://github.com/SiaFoundation/renterd/pull/1575
[x] https://github.com/SiaFoundation/renterd/pull/1578
[ ] Update UI

onlymetadata vs onlyMetadata is actually causing issues. We need to double check we don't have more of these mistakes.

@alexfreska backend is updated on api-breakers so I'm assigning this to you now.


SuperLU_DIST version 5 and 6 support issue

It appears that the current version of MFEM cannot compile with versions 5.1.3 and 6.3.1 of SuperLU_DIST. I have not tested other minor version numbers.
In 5.1.3 the error is:
linalg/superlu.cpp:544:16: error: use of undeclared identifier 'ZeroLblocks'
               ZeroLblocks(grid->iam, APtr_->GetGlobalNumColumns(),
               ^
linalg/superlu.cpp:546:16: error: use of undeclared identifier 'ZeroUblocks'
               ZeroUblocks(grid->iam, APtr_->GetGlobalNumColumns(),
               ^

In 6.3.1 these functions are both redefined using macros but only one of them is set to a valid function name apparently:
linalg/superlu.cpp:546:16: error: use of undeclared identifier 'dZeroUblocks'; did you mean 'dZeroLblocks'?
               ZeroUblocks(grid->iam, APtr_->GetGlobalNumColumns(),
               ^~~~~~~~~~~
               dZeroLblocks
linalg/superlu.cpp:35:21: note: expanded from macro 'ZeroUblocks'
#define ZeroUblocks dZeroUblocks
                    ^

If we intend to continue support for these older versions we may need to dig out the old implementations or find some workaround.

Version 6.0 came out almost 7 years ago, and 7.0 came out 3 years ago. I think it should be fine to make the update and require SuperLU_DIST >= 7.0. Do others agree?

Also a suggestion, rename "Superlu" to "superlu_dist"

Also a suggestion, rename "Superlu" to "superlu_dist" in mfem.

Do you mean just in the source code? Like SuperLUSolver to SuperLUDistSolver or something like that? I'm not sure how valuable such a renaming would be but will let others state their opinions here.

Also a suggestion, rename "Superlu" to "superlu_dist" in mfem.

Do you mean just in the source code? Like SuperLUSolver to SuperLUDistSolver or something like that? I'm not sure how valuable such a renaming would be but will let others state their opinions here.

In the souce code and the document. Anyway, superlu and superlu_dist are different solvers as I can see.


PnP to IoT Plug and Play

add back in Wellington's changes

Are we sure this was the only change that was impacted by https://github.com/Azure/azure-sdk-for-c/pull/953? I am surprised GitHub didn't show this as a merge conflict, preventing the merge. Did the pull/merge from master just unintentionally override this change?

Are we sure this was the only change that was impacted by #953? I am surprised GitHub didn't show this as a merge conflict, preventing the merge. Did the pull/merge from master just unintentionally override this change?

I had a merge conflict and went through the files in conflict and resolved the merge.  These lines were missed somehow.  So yes, the merge from master unintentionally overrode this change.


Hardware (physical) qubit as gate parameter raises unhelpful exception

Observed behavior
Parsing this code
OPENQASM 3.0;
include "stdgates.inc";

gate mygate c, $1 {
  h c;
}

raises a QASM3ParsingError exception after catching a ParseCancellationException from ANTLR.

Stack trace
ParseCancellationException                Traceback (most recent call last)
~/qiskit/.venvs/env-3.11/lib64/python3.11/site-packages/openqasm3/parser.py in parse(input_, permissive)
    86     try:
---> 87         tree = parser.program()
    88     except (RecognitionException, ParseCancellationException) as exc:

~/qiskit/.venvs/env-3.11/lib64/python3.11/site-packages/openqasm3/_antlr/_4_11/qasm3Parser.py in program(self)
   639                 self.state = 131
--> 640                 self.statement()
   641                 self.state = 136

~/qiskit/.venvs/env-3.11/lib64/python3.11/site-packages/openqasm3/_antlr/_4_11/qasm3Parser.py in statement(self)
   983                     self.state = 168
--> 984                     self.gateStatement()
   985                     pass

~/qiskit/.venvs/env-3.11/lib64/python3.11/site-packages/openqasm3/_antlr/_4_11/qasm3Parser.py in gateStatement(self)
  3138             self.state = 405
-> 3139             self.scope()
  3140         except RecognitionException as re:

~/qiskit/.venvs/env-3.11/lib64/python3.11/site-packages/openqasm3/_antlr/_4_11/qasm3Parser.py in scope(self)
  1152             self.state = 186
-> 1153             self.match(qasm3Parser.LBRACE)
  1154             self.state = 190

~/qiskit/.venvs/env-3.11/lib64/python3.11/site-packages/antlr4/Parser.py in match(self, ttype)
   125         else:
--> 126             t = self._errHandler.recoverInline(self)
   127             if self.buildParseTrees and t.tokenIndex==-1:

~/qiskit/.venvs/env-3.11/lib64/python3.11/site-packages/antlr4/error/ErrorStrategy.py in recoverInline(self, recognizer)
   702     def recoverInline(self, recognizer:Parser):
--> 703         self.recover(recognizer, InputMismatchException(recognizer))
   704

~/qiskit/.venvs/env-3.11/lib64/python3.11/site-packages/antlr4/error/ErrorStrategy.py in recover(self, recognizer, e)
   696             context = context.parentCtx
--> 697         raise ParseCancellationException(e)
   698

ParseCancellationException: None

The above exception was the direct cause of the following exception:

QASM3ParsingError                         Traceback (most recent call last)

Desired behavior
Raise a more helpful exception, or else accept the code so that it can be validated at a later stage.
Versions
openqasm 7c98e1b2e8
Python 3.11.0

This is going to be true of any grammatically invalid program, not just this case. It's not easy to build a parser that gives good error messages, and here ANTLR can't build the parse tree.
Right now, I wouldn't want to devote manpower to trying to add that functionality in. Doing it right is a really big job, and it's kind of a secondary priority; first we could do with having all the happy-path things succeed, before we spend time improving the UX for failures.
(It's also easier to be more lenient in parsing in order to recover/produce good error messages with handwritten parsers, and this package is meant to use the reference ANTLR grammar.)

Though just to be clear: I do think we could and should do better on error reporting in the long run.

I agree that applying a bandaid is not the correct response. I mainly opened this issue to document a specific undesirable feature, and as you point out, the more general issue. Furthermore, I agree that the cost/benefit ratio of implementing something robust to get better diagnostics is rather high at the moment.
One thing that might incidentally take care of this problem is to recognize \$[\d]+ as an ordinary identifier; that is, don't distinguish physical qubits in the ANTLR lexer an parser. But I wouldn't make such a big change just to solve this particular problem. Also IIUC, the main reason for using ANTLR is to document the grammar in a testable way. Changing this to adapt to a practical use of OC3, namely importing into Qiskit, doesn't necessarily serve that goal. It might be better for the the pure-Python importer to use a different parser. In fact given your interest in a hand-made parser, I wonder what you think of this opinion, Jake:

It seems a parse tree doesn't help to meaningfully reduce effort to build AST compared to just rolling out a hand-made recursive descent parser

Also IIUC, the main reason for using ANTLR is to document the grammar in a testable way. Changing this to adapt to a practical use of OC3, namely importing into Qiskit, doesn't necessarily serve that goal.

I agree with this - this package is serving two masters at the moment, and it's part of the reason I've been more on the side of using a different parser for our needs in Qiskit (my quick-and-dirty qiskit-qasm3-import notwithstanding).

One thing that might incidentally take care of this problem is to recognize \$[\d]+ as an ordinary identifier; that is, don't distinguish physical qubits in the ANTLR lexer an parser.

I think you're correct here, and if/when #417 does merge I'd vote to change the ANTLR grammar to reflect this view.
On the last opinion: I don't think I have the experience needed to have a properly informed opinion on it, but I've definitely started tending that way after having written my own recursive-descent parsers.  I'm most interested in that link to the Terence Parr (ANTLR author) comment that AST generation is a non-goal of ANTLR 4 - I didn't know he'd said that, and it helps me understand more of ANTLR's exposed public API.


Added support for forward_cookies_whitelisted_names

what
I've added support for whitelisted_names (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudfront_distribution#whitelisted_names) if forward_cookies is set to "whitelist"
why
Currently, the only option to forward cookies is to set it to "all" which isn't perfect
references

I've used naming and approach from another Cloudposse module - https://github.com/cloudposse/terraform-aws-cloudfront-cdn/blob/master/variables.tf

/test all

Sorry for hassling @nitrocode , but it'd make my life so much easier if we could get it merged. Let me know if there's anything I should add here :)

thanks @rafaljanicki for the contribution!


Make terrain a big block instead of plane

PoC:

An alternative would be to add soft edges

I suggest doing it like in many other 3D RTS games by extending the terrain mesh beyond the border of the map so that the actual cutoff of the terrain is hidden behind either the fog of war or the possible camera constrains.
E.g. here is how it looks for an Age of Empires 4 map. The yellow border constraining the playable area while everything that extends outside is just set dressing.

I suggest doing it like in many other 3D RTS games by extending the terrain mesh beyond the border of the map so that the actual cutoff of the terrain is hidden behind either the fog of war or the possible camera constrains.
E.g. here is how it looks for an Age of Empires 4 map. The yellow border constraining the playable area while everything that extends outside is just set dressing.

That sounds like a good idea but the question remains - what about edges? how to present them? I'll try to experiment with that anyway but if you have any suggestion or you're able to check how it's done in AoE4 I'd appreciate any information.

In AOE4 the visible edges are those yellow symbol lines while some distance fades with the fog of war until full black.

In Age of Mythology the map borders are hard edges that fake some depth terrain into the ground.

In AOE4 the visible edges are those yellow symbol lines while some distance fades with the fog of war until full black.

In Age of Mythology the map borders are hard edges that fake some depth terrain into the ground.

Perfect, thanks!


JSON: use Value() for operator Type overloaded

Currently Value() and Overloaded Type() are behaving differently, hence while using Type() case we are not getting expected behaviour. Update Type() to return the Value().

https://jenkins.code.rdkcentral.com/job/github-triggered/1938/ started


callback not triggered in 7.4.1

Neither error or callback are triggered with the following code on node 8.1
jwt.sign({ foo: 'bar' }, secret, { algorithm: 'HS256' }, (err, asyncToken) => {
console.log(err, asyncToken)
});

happened with an undefined secret

Thanks @reduxdj for the heads up, we already have a bug about this.
I'll close this issue to track the problem in the older one.


Cannot access 'i' before initialization

i = 1
console.log(i)
let i = 2
console.log(i)

// Uncaught ReferenceError: Cannot access 'i' before initialization at <anonymous>:1:3

You have to re-write your code
you used the variable i and then defined it
define it first then use it.

let i = 2
window.i = 1
console.log(i)

window.i = 1
console.log(i)
let i = 2
console.log(i)

// Uncaught ReferenceError: Cannot access 'i' before initialization at <anonymous>:1:3

You have to re-write your code
you used the variable i and then defined it
define it first then use it.
let i = 2
window.i = 1
console.log(i)


[HODL] - BSP-112: Add Studio ID

Studio ID added to Warehouse Item.
Studio JWT check.
CircleCI Updated.

won't merge -- changes have been applied to item-service instead.