Memory Error for CREC parser

For some dates, a memory error occurs when parsing that day's Congressional Record files:
Traceback (most recent call last):
  File "/mnt/capitolwords/capitolweb/parser/management/commands/run_crec_parser.py", line 85, in handle
    es_doc = crec.to_es_doc()
  File "/mnt/capitolwords/capitolweb/parser/crec_parser.py", line 383, in to_es_doc
    segments=self.segments,
  File "/usr/local/lib/python3.5/dist-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/mnt/capitolwords/capitolweb/parser/crec_parser.py", line 324, in segments
    sents = (sent.string for sent in self.textacy_text.spacy_doc.sents)
  File "/usr/local/lib/python3.5/dist-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/mnt/capitolwords/capitolweb/parser/crec_parser.py", line 231, in textacy_text
    return textacy.Doc(SPACY_NLP(text))
  File "/usr/local/lib/python3.5/dist-packages/spacy/language.py", line 341, in __call__
    doc = proc(doc)
  File "nn_parser.pyx", line 337, in spacy.syntax.nn_parser.Parser.__call__
  File "nn_parser.pyx", line 400, in spacy.syntax.nn_parser.Parser.parse_batch
  File "nn_parser.pyx", line 725, in spacy.syntax.nn_parser.Parser.get_batch_model
  File "nn_parser.pyx", line 84, in spacy.syntax.nn_parser.precompute_hiddens.__init__
  File "/usr/local/lib/python3.5/dist-packages/spacy/_ml.py", line 148, in begin_update
    self.W.reshape((self.nF*self.nO*self.nP, self.nI)).T)
MemoryError

Can you post the command line arguments this failed on? Or any dates that this bug occurs for.

Sure thing. This fails for a handful of dates so far. Among them: 2016-09-13 and 2016-09-12. The command:
python3 manage.py run_crec_parser --start_date=2016-09-11 --end_date=2016-09-13

@dwillis
I wasn't able to reproduce this on my laptop, but that has 16gb so its possible that the days that trigger this error just have a larger than normal amount of text to process. So, I would first try running this on a machine with more ram if you haven't already done so. Alternatively, you can try running it with an older version of spacy ("pip install spacy<2.0") as this may be related to an issue in the newer version (nothing we're doing in the capitol words codes requires any newer features).

@will-horning Ok, thanks! I'll try both of those options.


Added ability to create custom sails commands

Currently sails does not have the ability to create custom commands at the sails command line. This PR adds that ability with the following rules:

A command hook needs to specify sails: { isCommand: true} in the package.json
A command hook needs to export an object with 2 properties command and execute.

the command property should be a string representing the command name
the execute property should be a function that will be called with the additional arguments.

For example a module like this will add an azure command that can be used like this sails azure deploy
module.exports = {
  command: 'azure',
   execute: function(args) {
     console.log(args) // will log ['deploy']
  }
}

The motivation behind this PR was https://github.com/balderdashy/sails/pull/2663. The way that is currently set up is that if you do sails deploy it defaults to publishing to Azure. I think it would be a much nicer experience if the command would be sails azure deploy and in the future perhaps a sails heroku deploy etc. Additionally, this would allow the azure functionality to be wrapped up into it's own (opt-in) module.

Uuh, this seems nice.

@brooklynDev awesome thanks!  will merge simultaneous to the main azure PR.
What I'm thinking is sails deploy can use this sort of custom command integration, and then look to something in your project's .sailsrc or package.json file to determine which deployment strategy to use (azure/ec2/heroku/etc)

or just sails deploy [something] where something is azure, heroku, or something that you have build on your own.
This is because users may have to deploy their apps to multiple places, eg stable, rc and dev.

and imho sails deploy [something] should make some 'rules' for actual deployment implementation, so users can make own ones easily with proper guidelines.

@mikermcneil I think that the azure integration should be bundled into its own module which would expose an azure command. For example to deploy you would run sails azure deploy. The azure command might have other subcommands other than deploy available and you would run those as sails azure [subcommand]
As for sails deploy that should be a hook that is defined in the projects package.json (or .sailsrc) the same way you would define which command to run for npm test.
So for example your package.json would look like ...
sails : { deploy: 'sails azure deploy' }

that would tell sails to run sails azure deploy when you run sails deploy.
This would also allow you to define a simple deploy script without need an entire module.
something like...
sails : { deploy: 'cp -r * /production/ ' }

@tikotzky that sounds perfect- I'd say let's put it in the .sailsrc file (since it allows for setting cross-project global overrides if you place a .sailsrc in your $HOME directory)
So I took a pretty thorough pass at speccing out how it ought to work (see below)
@tikotzky @brooklynDev what do you think?  Would you be able to knock this out? (i.e. the part that touches sails core) I can take care of setting up the sails deploy and sails-deploy-azure modules.
Or if anyone else is interested in getting involved on this effort, please let me know here (or on twitter @mikermcneil.)
Thanks!

Contents

Custom commands in Sails
sails deploy

Custom commands in Sails
example .sailsrc file
{
  "commands": {
    "deploy": "sails-deploy"
  },
  "deploy": {
    "module": "sails-deploy-azure"
  }
}

In Sails core...

(in bin/sails.js)

When the CLI is run as sails deploy, since this is an unrecognized command, Sails should load base config using the rc module (see bin/sails-generate.js for an example).  This will load any relevant config from the .sailsrc file(s) which apply to the current app.
Assuming we have a .sailsrc like the one above, there will be new commands and deploy namespaces in the app's configuration object.  So we'd be able to access:
configObjectReturnedByRC.deploy.module       // => 'sails-deploy-azure'
configObjectReturnedByRC.commands.deploy   // => 'sails-deploy'

Sails core should then check to see if commands exist, and if so, loop through each one and see if there is a command matching what the user is trying to do (e.g. "deploy").  If no match is found, cli does exactly what it does today (show usage instructions).  But if a match is found, then the CLI requires the configured module ("sails-deploy") from the context of the current sails project's "appPath" (configObjectReturnedByRC.appPath).  For example, something like this (warning: I did not run or double-check this code):
var configObjectReturnedByRC = require('rc')('sails');
var matchingCommand = _.find(configObjectReturnedByRC.commands || [], function (val, key){});
if (!matchingCommand) {
  /*... handle this case exactly as happens currently, then return early */
  return;
}
var pathToRequire = require('path').resolve(configObjectReturnedByRC.appPath||process.cwd(), 'node_modules', matchingCommand);
var customCmd;
try {
  customCmd = require(pathToRequire);
}
catch (e){
  // check if this is a MODULE_NOT_FOUND error (look at e.type or e.code, can't remember)
  // if not, log an error message explaining that the custom command module referenced in one of the user's sailsrc files has an issue (then dump the original error stack and bail out/return early)
  // if so, log a nice error message explaining the command the user typed was recognized as a custom command defined in one of their sailsrc files, but it could not be required from the specified path.  Then bail out (return early)
}

// Finally, if that all worked, run the custom command.
customCmd({
  config: configObjectReturnedByRC
}, function (err){
  if (err) {
    // log msg explaining deployment failed, then log the original error stack  for details.
    return;
  }
  // log msg explaining deployment was successful.
});

Note that this approach means users will need to npm install sails-noop --save in their project in order to run sails noop on the command line.
Custom command packages
Custom sails commands are npm packages that export a function w/ two arguments- options and cb (following standard node conventions).
For example, here's a very simple no-op command  as an example (sails-noop).  Aside from its package.json , it has only one file: index.js.
/**
 * Module dependences
*/
// ...

/**
 * Usage:
 * `$ sails noop`
 */
module.exports = function noopCmd (options, cb) {
  // `options.config` is provided with the raw config that Sails core gathered by running `rc`.
  return cb();
};

sails deploy
sails deploy can be implemented as a custom command as specced out above.
the sails-deploy package
The job of the sails-deploy package is to  require() the configured strategy, then run it.  Eventually it can end up taking responsibility for generic shared logic between deployment strategies (e.g. talking to github, pulling/pushing/tagging, etc.)- but we won't know the right things to generalize until later when we have a few different strategies to look at.  For now, we'll just run the strategy.
This will be pretty similar to what we just did (which is also more or less how generators, adapters, and hooks work).  In the deploy object in the sailsrc file, the module property can be specified as any of the following:

an absolute path (if it starts with /)
a relative path from the sailsrc file (if it starts with ./ or ../), or otherwise
an NPM package name (in which sails will attempt to require() the deployment strategy module from the node_modules folder within the sails project's appPath).

Note that this approach means a user will need to npm install sails-deploy-noop --save in her project and configure "deploy": {"module": "sails-deploy-noop"} in her sailsrc file.  Then she can run sails deploy on the command line and sails-deploy-noop will be used automatically.
sails-deploy-* strategies
The simplest no-op deploy strategy (e.g. sails-deploy-noop) is simply an npm package with a package.json and an index.js file:
/**
 * Module dependences
*/
// ...

module.exports = function sailsDeployNoop(options, cb) {
  // `options.config` is provided with the raw config that Sails core gathered by running `rc`.
  return cb();
};

Again, we can get fancier here eventually as far as what happens in sails-deploy itself, but for now, a simple function is the right approach.
Let me know if I missed anything, and thanks!

here's an early stubbed out version of the generic sails-deploy module https://github.com/mikermcneil/sails-deploy
https://www.npmjs.com/package/sails-deploy

and here's the same sort of stubbed-out early version of sails-deploy-azure:
https://www.npmjs.com/package/sails-deploy-azure
https://github.com/mikermcneil/sails-deploy-azure
I went ahead and put both modules on npm so they can be used when testing the remaining work in core

@mikermcneil I just reviewed quickly, one thing that stands out is...
You recommended that the way to add a command would be to have a commands property in .sailsrc which would list out all custom commands. The main issue I see happening with that approach is...
it requires the user to do 2 steps on both install and removal...

npm install sails-deploy
add a deploy prop to thecommandsobject in.sailsrc`

I think the better path (at least for adding custom commands)  would be do something along the lines that this PR is suggesting, where the module that is adding the command has sails: { isCommand: true } in its package.json. This way you don't need to worry about keeping your .sailsrc in sync everytime you add/remove a command.
This does possible make it more difficult have globally installed commands, and also does increase the chance of conflict between different commands, but both of those issues can be worked out.

I think the better path (at least for adding custom commands) would be do something along the lines that this PR is suggesting, where the module that is adding the command has sails: { isCommand: true } in its package.json. This way you don't need to worry about keeping your .sailsrc in sync everytime you add/remove a command.

@tikotzky I agree with you-- but if we start by using the .sailsrc, we can work out the kinks before adding too much magic, know what I mean?  It also gives folks the ability to be more explicit (I've learned the hard way about too much magic too fast)  We started off w/ hooks loading via sailsrc first before doing auto-loading for exactly the same reason.
That cool w/ you?

There should be document to explain all these package.json custom options for sails property. Also there should be some guidelines how to use those.

@tarlepp agreed - https://github.com/balderdashy/sails/pull/2667#issuecomment-75474760 is the beginning of that -- currently only other core use case is for hooks.  I can put together a doc on that as soon as I hear back from @tikotzky and/or @brooklynDev whether they'll implement this

@mikermcneil sorry, I really would love to be able to work on this, it looks like it would be a fun project to hack on. It's just that right now my hands are pretty full and I wouldn't feel comfortable committing to getting this done.
If this is still waiting to get implemented when I have some time freed up I would be happy to tackle it.

@tikotzky thanks for the heads up-- I'll keep hacking on it as time permits.  I've been doing some overall maintenance in core first- if you beat me to it, would be awesome :)

Closing this for now to keep tidy.
@all- please feel free to open a new PR using the spec discussed here
Thanks for the great discussion!


Link privately

@Krande
many projects don't specify the PRIVATE flag, but this leads to the resulting shared library also providing HDF5 for example
Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

@Krande regarding your older questions, linking privately means that users that link to libmed cannot directly access HDF5. If they need HDF5, they must declare it explicitly.
This helps avoid unexpected dependencies from being injected.


LineSeriesCanvas in FlexibleXYPlot container renders in incorrect size on retina screens

Example:
https://codepen.io/marton/pen/aPvody
This can be reproduced by putting any canvas series in a FlexibleXYPlot container as long as devicePixelRatio !== 1.
#605 seems related

I am facing the same issue.

I can confirm this. Also setting devicePixelRatio to 1 on a retina screen makes the line the correct size, albeit while also looking slightly blurred.

Any update on this? currently working around it using window.devicePixelRatio = 1, but as the pervious commenter says, it's blurry

Hi, is there any update on this? Is this library even being actively maintained?


Show in folder

Description
I am using firefox browser and suppose I downloaded a file, after downloading I click the show in folder which opens the correct folder but not selecting the file.
Steps To Reproduce

Download file from internet via firefox

click the show in folder

This open the folder but the file is not selected and also the file manager is not popped out.

Steps to fix
The file needs to selected after opening the file manager
Files Version
<IP_ADDRESS>
Windows Version
Windows 11 PRO 22H2
Relevant Assets/Logs
debug.log

Merging with #9390


cucumber-expressions: Use default transformer in preference to contradicting type hint

Summary
Preliminary PR for discussion.
Don't use a matching paramater type if it's type does not match the type hint from the step-definition. Instead, create an anonymous parameter type which uses the default transformer.
Related to #658, but extended to custom parameter types.
Details
Using a parameter type which transforms to a type which does not match the step definition's argument type will throw an exception when the glue is invoked.
If we hit this case, prefer to disregard the matching parameter type and just convert via the default transformer.
Note that this change seems to make useRegexpMatchAsStrongTypeHint on ParameterType redundant. The work to remove it has not been performed here. If the changes look ok I can do this too.
Some details on tests:
The combination of type hints and parameter types can be expanded as follows. I've tried to match each case to a new or existing test:

No method type hint & no custom ParameterType => matches_positive_int
Method type hint & no custom ParameterType => matches_positive_int_with_hint
No method type hint & ParameterType => uses_parameter_type_registry_when_parameter_type_is_defined
Method type hint & ParameterType which agrees => uses_parameter_type_when_type_hint_agrees
Method type hint & ParameterType which contradicts => uses_default_transformer_when_type_hint_conflicts_with_parameter_type

The changes here should only affect case 5.

On the mailing list you mentioned that you would like to declare multiple parameter types for qouted strings and have the step definitions find the right transformer by looking at the type hint.
For example:
``java
@ParameterType(""([^"]+)"")
public TypeA typeA(String){
return // create type A here
}
@ParameterType(""([^"]+)"")
public TypeB typeB(String){
return // create typeB here
}

So when given two step definitions the argument will be transformed by matching via the type rather then the regex.

@Given("an object of {typeA}")
public void an_object_of_type_a(TypeA a){
}
@Given("an object of {typeB}")
public void an_object_of_type_b(TypeB b){
}

I currently don't see this reflected in any test case and so I don't understand how moving the `parameterType.getType().equals(typeHint)` condition could achieve this. This will result in Cucumber using the default parameter type transformer rather then the registered transformers for `TypeA` or `TypeB` . You created the impression that you were looking to achieve the latter.

Did I understand your correctly?

Hi Rien,
I think I've given mixed signals on what I'm trying to achieve as I've slowly wrapped my head around it :)
I'm trying to get to a point where @ParameterTypes which conflict with the type-hint on regular-expression step-definitions are ignored.
I'm not trying to change the behaviour for Cucumber-expressions at all, and I'm not trying to look up @ParameterTypes for regular-expression glue based on type-hints.
The move of !parameterType.getType().equals(typeHint) is incidental; the important bit is the removal of the ... && !parameterType.useRegexpMatchAsStrongTypeHint() part, which allows a parameter type to break regular-expression glue by forcing a type incompatible with the definition.
To run with your example...
The following glue:
package cucscratch;

import io.cucumber.java.ParameterType;
import io.cucumber.java.en.Given;

public class Main {

    private static class TypeA {}

    @ParameterType("\\w+")
    public TypeA typeA(String a) {
        return new TypeA();
    }

    @Given("a cucumber expression of {typeA}")
    public void an_object_of_type_a(TypeA a) {
        // works fine!
    }

    @Given("^a regular expression of (\\w+)$")
    public void an_object_of_type_b(String b) {
        // broken: will attempt to use the transformer for TypeA
    }

}

plus this feature:
Feature:
  Scenario:
    Given a cucumber expression of type_a
    Given a regular expression of just_a_string

fails with:
io.cucumber.core.exception.CucumberException: Could not invoke step [^a regular expression of (\w+)$] defined at 'cucscratch.Main.an_object_of_type_b(java.lang.String)'.
It appears there was a problem with the step definition.
The converted arguments types were (cucscratch.Main$TypeA)

The details are in the stacktrace below.
	at io.cucumber.core.runner.PickleStepDefinitionMatch.couldNotInvokeStep(PickleStepDefinitionMatch.java:81)
	at io.cucumber.core.runner.PickleStepDefinitionMatch.runStep(PickleStepDefinitionMatch.java:65)
	at io.cucumber.core.runner.TestStep.executeStep(TestStep.java:64)
	at io.cucumber.core.runner.TestStep.run(TestStep.java:49)
	at io.cucumber.core.runner.PickleStepTestStep.run(PickleStepTestStep.java:46)
	at io.cucumber.core.runner.TestCase.run(TestCase.java:51)
	at io.cucumber.core.runner.Runner.runPickle(Runner.java:66)
	at io.cucumber.core.runtime.Runtime.lambda$run$2(Runtime.java:100)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at io.cucumber.core.runtime.Runtime$SameThreadExecutorService.execute(Runtime.java:243)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
	at io.cucumber.core.runtime.Runtime.lambda$run$3(Runtime.java:100)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.SliceOps$1$1.accept(SliceOps.java:204)
	at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1351)
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at io.cucumber.core.runtime.Runtime.run(Runtime.java:101)
	at io.cucumber.core.cli.Main.run(Main.java:73)
	at io.cucumber.core.cli.Main.main(Main.java:31)
Caused by: io.cucumber.core.backend.CucumberBackendException: Failed to invoke public void cucscratch.Main.an_object_of_type_b(java.lang.String)
	at io.cucumber.java.Invoker.invoke(Invoker.java:29)
	at io.cucumber.java.JavaStepDefinition.execute(JavaStepDefinition.java:27)
	at io.cucumber.core.runner.PickleStepDefinitionMatch.runStep(PickleStepDefinitionMatch.java:63)
	... 24 more
Caused by: java.lang.IllegalArgumentException: argument type mismatch
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at io.cucumber.java.Invoker.invoke(Invoker.java:27)
	... 26 more

This behaviour makes it difficult to write a mixture of regular-expression glue and cucumber-expression glue with @ParameterTypes, without the @ParameterTypes stomping all over the regular-expression glue.

I initially thought it was about multiple matching regexes but I had the wrong idea.
The following works just fine:
package cucscratch;

import io.cucumber.java.ParameterType;
import io.cucumber.java.en.Given;

public class Main {

    private static class TypeA {}

    private static class TypeB {}

    @ParameterType("\\w+")
    public TypeA typeA(String a) {
        return new TypeA();
    }

    @ParameterType("\\w+")
    public TypeB typeB(String a) {
        return new TypeB();
    }

    @Given("a cucumber expression with type A: {typeA}")
    public void an_object_of_type_a(TypeA a) {
        // works fine!
    }

    @Given("a cucumber expression with type B: {typeB}")
    public void an_object_of_type_a(TypeB a) {
        // works fine!
    }

}

Feature:
  Scenario:
    Given a cucumber expression with type A: type_a
    Given a cucumber expression with type B: type_b

Hopefully this makes a bit more sense. Thanks for spending the time to review this!

Right. I get it now! The examples really helped.
I totally missed the fact that you're also no longer calling parameterType.useRegexpMatchAsStrongTypeHint(). So effectively the regexp is no longer used as type hint.
I knew this was a problem when I introduced it :laughing:
And the problem is that I can't change it now for the same reason I couldn't change it then without breaking existing expectations in #153
However we could provide a @ParameterType(useRegexpMatchAsStrongTypeHint=true/false) option in cucumber-jvm that passes this value onto ParameterType in cucumber-expressions. It would initially default to true but at the next major release we could flip the default to false.

We need to resolve one more thing:
public class Main {

    private static class TypeA {}

    private static class TypeB {}

    @ParameterType("\\w+")
    public TypeA typeA(String a) {
        return new TypeA();
    }

    @ParameterType("\\w+")
    public TypeB typeB(String a) {
        return new TypeB();
    }

    @Given("^a regular expression of (\\w+)$")
    public void an_object_of_type_a(TypeA a) {
        /
    }

    @Given("^an aother egular expression of (\\w+)$")
    public void an_object_of_type_b(TypeB b) {

    }

}

Rather then using lookupByRegexp we should add lookupByRegexpAndType when a type hint is available.

we could provide a @ParameterType(useRegexpMatchAsStrongTypeHint=true/false ...

This would be very useful, and flipping the default to false at next major release sounds like a good plan.
Thanks for your time!

Resolved by https://github.com/cucumber/cucumber-jvm/commit/0136079aa07e5bb14e6d1bdc9afb2d09d5da7570


Error with UddoktaPay payment extension: Incorrect Integer Value for invoice_id

I'm encountering an issue while using UddoktaPay as the payment gateway in FossBilling. Whenever a transaction is processed, I receive the following error:

{

"result": null,

"error": {

"message": "SQLSTATE[HY000]: General error: 1366 Incorrect integer value: 'p13pW605mfefPNUIYW2f for column 'invoice_id' at row 1"

}

You should contact the developer of the module...

Issue with their payment gateway itself, not with FOSSBilling.
It looks like there's now an updated version of it:
https://github.com/UddoktaPay/FOSSBilling/releases/tag/1.0.1


onnx转tensorRT的问题

pth->onnx时没有问题，onnx->trt时遇到[8] Assertion failed: trt_dtype == nvinfer1::DataType::kHALF && cast_dtype == ::ONNX_NAMESPACE::TensorProto::FLOAT的问题
显卡使用的是Tesla P40 ，只支持int8，trt_export.py是只支持在半精度显卡上运行吗？
my env : pytorch1.3 onnx1.7.0 tensorrt6
run:
python trt_export.py --name "reid" --output outputs/trt_model/ --onnx-model ./outputs/onnx_model/reid.onnx --height 384 --width 128
output:
Input filename:   ./outputs/onnx_model/reid.onnx
ONNX IR version:  0.0.4
Opset version:    9
Producer name:    pytorch
Producer version: 1.3
Domain:
Model version:    0
Doc string:
WARNING: ONNX model has a newer ir_version (0.0.4) than this parser was built against (0.0.3).
While parsing node number 0 [Cast]:
ERROR: builtin_op_importers.cpp:727 In function importCast:
[8] Assertion failed: trt_dtype == nvinfer1::DataType::kHALF && cast_dtype == ::ONNX_NAMESPACE::TensorProto::FLOAT
[2020-07-31 15:39:51.626] [error] error: could not parse onnx engine
[2020-07-31 15:39:51.626] [error] error: could not deserialize or build engine
Segmentation fault

Which model do you use?

Which model do you use?
Thanks for reply.
I trained the model with ./configs/Market1501/sbs_R50-ibn.yml.

Thanks for reply.
I trained the model with ./configs/Market1501/sbs_R50-ibn.yml.

Which model do you use?

Maybe ibn and non-local will occur some problems, and I just test vanilla ResNet50 to convert TensorRT. You can update tensorRT and try again.
Maybe you can remove nl first and check if it can be converted, then remove ibn, and do the same thing.

I just test non-local, it can be converted, and maybe it's instancenorm problem.

Yes, I updated tensorRT6 to tensorRT 7, it works.
But when save engine, the following error occurred, Is there any possible reason？
[2020-07-31 19:15:57.199] [info] fp16 support: false
[2020-07-31 19:15:57.199] [info] int8 support: true
[2020-07-31 19:15:57.199] [info] Max batchsize: 1
[2020-07-31 19:15:57.199] [info] Max workspace size:<PHONE_NUMBER>0
[2020-07-31 19:15:57.199] [info] Number of DLA core: 0
[2020-07-31 19:15:57.199] [info] Max DLA batchsize: 268435456
[2020-07-31 19:15:57.199] [info] Current use DLA core: 0
[2020-07-31 19:15:57.199] [info] build engine...
Detected 1 inputs and 1 output network tensors.
[2020-07-31 19:19:06.024] [info] serialize engine to ./outputs/trt_model/reid.engine
[2020-07-31 19:19:06.024] [info] save engine to ./outputs/trt_model/reid.engine...
Segmentation fault

Maybe ibn and non-local will occur some problems, and I just test vanilla ResNet50 to convert TensorRT. You can update tensorRT and try again.
Maybe you can remove nl first and check if it can be converted, then remove ibn, and do the same thing.

You can check if the engine if saved, then you can just load the trt engine to inference. You can check the difference between trt outputs and pytorch outputs.

@L1aoXingyu Unfortunately the trt engine not saved.
Have you tried convert onnx to trt with IBN?
When i set WITH_IBN: false and WITH_NL: true, it works well. But if set WITH_IBN: true , it will report error "Segmentation fault" when mEngine->serialize().

Maybe instance norm will occur errors when converting to TensorRT. And I don't try to convert model with ibn layer.

Is the problem solved？

@L1aoXingyu Unfortunately the trt engine not saved. Have you tried convert onnx to trt with IBN? When i set WITH_IBN: false and WITH_NL: true, it works well. But if set WITH_IBN: true , it will report error "Segmentation fault" when mEngine->serialize().

Is the problem solved?


No results for rpm packages when run against version 9.x of redhat/almalinux

What happened:
Generated SBOM only includes python package when run against the official base containers for redhat ubi 9 as well as almalinux 9
What you expected to happen:
Similar to version 8.x of these distros I'd expect to get a mix of rpm and python packages reported
How to reproduce it (as minimally and precisely as possible):
docker run -it --rm anchore/syft:latest redhat/ubi9
or
docker run -it --rm anchore/syft:latest almalinux:9
Anything else we need to know?:
Environment:

Output of syft version:

Application:        syft
Version:            0.46.3
JsonSchemaVersion:  3.2.3
BuildDate:          2022-05-26T06:46:34Z
GitCommit:          7cb8e1fc14a278ec5afce379623a47577aba9917
GitDescription:     v0.46.3
Platform:           linux/amd64
GoVersion:          go1.18.2
Compiler:           gc

This is fixed by https://github.com/anchore/syft/pull/1018, which is not released yet.

That was fixed in 0.47.0:
$ docker run -it --rm anchore/syft:latest almalinux:9
Unable to find image 'anchore/syft:latest' locally
latest: Pulling from anchore/syft
03882b0fb5fd: Pull complete
a132440c3c74: Pull complete
56b6d41be756: Pull complete
Digest: sha256:78421757823e55a1dd6f4c1e00f9754b41f714e3b2f2cd8aed04279fa84e75c7
Status: Downloaded newer image for anchore/syft:latest
 ✔ Parsed image
 ✔ Cataloged packages      [153 packages]
NAME                         VERSION                        TYPE
acl                          2.3.1-3.el9                    rpm
almalinux-gpg-keys           9.0-3.el9                      rpm
almalinux-release            9.0-3.el9                      rpm
...

$ docker run -it --rm anchore/syft:latest redhat/ubi9
 ✔ Parsed image
 ✔ Cataloged packages      [202 packages]

NAME                                    VERSION                        TYPE
PyGObject                               3.40.1                         python
PySocks                                 1.7.1                          python
acl                                     2.3.1-3.el9                    rpm
alternatives                            1.20-2.el9                     rpm
audit-libs                              3.0.7-101.el9_0.2              rpm
basesystem                              11-13.el9                      rpm
bash                                    5.1.8-4.el9                    rpm
...


optimizing ansi escape sequences

I was wondering if there was a way with your crate to optimize the ansi escape sequences. Here's an example of what I'm talking about. This is based off of your hello_worlds.rs example.
owo-colors:   "\u{1b}[47m\u{1b}[30mHello\u{1b}[0m\u{1b}[0m \u{1b}[42m\u{1b}[35mWorld\u{1b}[0m\u{1b}[0m"
nu-ansi-term: "\u{1b}[47;30mHello\u{1b}[0m \u{1b}[42;35mWorld\u{1b}[0m"
Hello World
owo-colors:   "\u{1b}[47m\u{1b}[30mo\u{1b}[0m\u{1b}[0m \u{1b}[42m\u{1b}[35mWo\u{1b}[0m\u{1b}[0m"
nu-ansi-term: "\u{1b}[47;30mo\u{1b}[0m \u{1b}[42;35mWo\u{1b}[0m"
o Wo

Note how there seems to be extra ansi codes. One way of optimizing this would be to parse the codes and combine them where possible.  For instance,
E=ESC here - note the E[47E[30m instead of E[47;30m.
Alternatively when you're doing "Hello".fg::<Black>().bg::<White>() you could just write that out as E[47;30m. I'm guessing this may be the easiest solution. Of course, you may have some reason why you're doing it the way you are that I'm unaware of.
This isn't a huge deal with 'hello world' but the extra ansi codes add up over time and will decrease throughput.

This is definitely something I'd like to do, but haven't come up with quite enough solutions for making this real. I think doing this in an efficient/robust enough manner would involve something along the lines of specialization, either actual (non-option, specialization is not a nightly feature worth depending on) some form of emulation (Deref specialization or otherwise)
I'll try and take some time soon to improve this if I can figure something out

also @fdncred I should note that the Style api should actually be optimized in the manner you're seeking, and is often preferable (for multiple reasons, including if you offer color customization to users) when using a lot of colors

Style seems to be more optimized but it still seems to be concatenating ansi sequences.
owo-colors   : "\u{1b}[47m\u{1b}[30mHello\u{1b}[0m\u{1b}[0m \u{1b}[42m\u{1b}[35mWorld\u{1b}[0m\u{1b}[0m"
owo style    : "\u{1b}[30m\u{1b}[47mHello\u{1b}[0m \u{1b}[35m\u{1b}[42mWorld\u{1b}[0m"
nu-ansi-term : "\u{1b}[47;30mHello\u{1b}[0m \u{1b}[42;35mWorld\u{1b}[0m"

maybe i'm not using your crate right. here's the small test.
    let colored_text = format!(
        "{hello} {world}",
        hello = "Hello"
            .fg::<owo_colors::colors::Black>()
            .bg::<owo_colors::colors::White>(),
        world = "World"
            .fg::<owo_colors::colors::Magenta>()
            .bg::<owo_colors::colors::Green>(),
    );

    let colored_text2 = format!(
        "{hello} {world}",
        hello = nu_ansi_term::Style::new()
            .fg(nu_ansi_term::Color::Black)
            .on(nu_ansi_term::Color::White)
            .paint("Hello"),
        world = nu_ansi_term::Style::new()
            .fg(nu_ansi_term::Color::Magenta)
            .on(nu_ansi_term::Color::Green)
            .paint("World"),
    );

    let colored_text3 = format!(
        "{hello} {world}",
        hello = "Hello".style(owo_colors::Style::new().black().on_white()),
        world = "World".style(owo_colors::Style::new().magenta().on_green()),
    );

    println!("owo: {:?}", &colored_text);
    println!("owo: {}", &colored_text);
    println!("nu : {:?}", &colored_text2);
    println!("nu : {}", &colored_text2);
    println!("sty: {:?}", &colored_text3);
    println!("sty: {}", &colored_text3);

I'm currently working on using autoref specialization in order to make these optimizations work with the entire API. This will likely be a breaking change (in theory; but for 99% of people not in practice), so I'm going to try and wrap this in with some other changes I wanted to go into the 3.0 release.
If you or anyone else wants these changes before the 3.0 release, please let me know and I can from then on start pushing 3.0 betas to crates.io that you can work off of.

github only just refreshed, so to respond: I noticed this while working on an autoref specialization version, @fdncred you are correct that Style still currently results in concatenation of ANSI sequences. I am currently working on resolving this and will setup some tracking issues for this.

Thanks @jam1garner. I look forward to the next release.

How do you think your changes will affect compile time? I recently was doing more comparison and noticed owo-colors took about 5.9sec vs nu-ansi-term 2.6. I'm hoping these changes make it compile faster.

These changes will likely not affect compile times in either direction. Could you explain your testing methods so I can possibly use them to guide optimizing compile times? Thanks.
For context: just a straight cargo build --release is < 1s for me, but that doesn't include any monomorphization-related compile times. I will admit owo-colors optimizes runtime performance (allocations, speed, binary size, in that order) over compile time, and thus is arguably "overly smart" in its design.
I'm sure there's improvements to be had without compromising on the design and I'd love to find those, but I don't actually know that I have an accurate way to measure this. For example owo-colors generates std::fmt::* implementations that you likely don't use (as, unlike most color libraries, it isn't fmt::Display-only), and I imagine those don't get optimized out until late into compilation. So that's my theory, and if correct would be trivial to solve for a good amount of compile time shaved off.
I would very much like to do everything to help owo-colors fit your usecase and I will admit optimizing compile times is something I need to work on improving at if you have any time to point my in the direction you're using for measurements @fdncred.

Thanks for being so willing to talk about this. It speaks well of you.
My testing isn't very scientific but I just use this command below and compile engine-q here - engine-q it's our next-gen nushell engine. kat recently introduced us to miette which includes owo-colors where I started investigating all this.
cargo +nightly build -Z timings="html,info"

Ok so @fdncred I've pulled engine-q and have been investigating and honestly I'm having trouble recreating your results. There's a bit of variability (anywhere from a slight bit under 1 second to a bit under 3 seconds (actual range: 0.9 to 2.7)) but I can't seem to recreate anything over 3 seconds?
I suppose this could be processor differences (I'm on a Ryzen 9 4900HS, for reference), but a >100% difference seems a bit extreme unless you're on an old/lower end machine. Or possibly it could be highly dependent on core count. I'm not sure.
I'm going to try to pull down compile times best I can, but measuring is hard because there's about as much noise as there is signal due to parallel compilation :/

I'll try doing this work from a slower box to see if I can better recreate your numbers

This is the slowest I've been able to get:

3.5 seconds on digital ocean's cheapest VM, a quite slow single-core machine :<
I don't mean to be rude but are your timings coming from some sort of free-tier CI container? Even on a machine that takes 21x as long to compile as my laptop I can't get as slow as your numbers for owo-colors. nu-ansi-term time seems decently similar on this digital ocean VM to what you have, so I'm honestly not sure what's going on.

Also I should've mentioned @fdncred, can you see if (ignoring Style, which I still need to optimize more) the output off of the master branch is optimized in all the manners you're seeking? I believe I've implemented autoref specialization for both compile-time and runtime colors.
For example:
let test = String::from("red on white");
let test_colored = test.red().on_white();
println!("{}", test_colored);
dbg!(test_colored);

this now should work and the above example outputs:
"\x1b[31;47mred on white\x1b0m"

@fdncred sorry for all the requests but if you get a second could you run the following command on engine-q then send the output for the section following "Compiling owo-colors..." ?
Command:
RUSTFLAGS="-Z time-passes" cargo +nightly build

Example of what part of the output I need:

this would help give me a better idea of the breakdown of what is causing such slow compilation on your machine. Thanks.

Also an addendum: -j1 also seems to produce more accurate results when being compared on a per-crate basis. The ordering seems significantly more consistent and while not 100% indicative of real-world performance, something about pipelining here seems to be hurting how owo-colors compares to other crates.
For example: what could I possibly be doing that causes owo-colors, a < 3000 line crate, to have longer compile times that regex, which has over 60k lines and is pretty universally considered a heavy dependency? This didn't add up to me so I've been looking into what could be the cause.
Now if I add -j1 regex floats upwards in the list, while owo-colors sinks well over 10 places. This would heavily imply that owo-colors' compile types aren't actually the issue: it just happens to be located in the tree in the perfect spot to be competing with crates being compiled in parallel. It's a leaf node (has no deps) *and* is high (i.e. low depth) in the dependency graph, meaning it's most likely to be pulled in right when cargo is starting to run out of parallelize-able tasks, which means the time that *the most* crates are hitting LLVM codegen, the most consistently expensive compilation pass (due to the fact rustc ouputs a lot of LLVM bitcode and just lets LLVM clean up and optimize the mess).
I've also gone on to example other libraries and applications I've written in Rust for a variety of tasks, with varying amounts of generics, deps, macros, etc. From what I can tell (2-3 runs per project) there is less variation and more sensible results for -j1.
All this data *really* makes me think owo-colors' compile times are more accurately in the subsecond range, Across 5 runs I observed an average compile time difference of  0.139s with this testing method:

wow - lots of comments. lol. i'll try to reply one at a time. but first, i should've mentioned that my timings come from the html file created from the command I specified. attaching it here so you can look at it. cargo-timing-20210930T130511Z.zip.
You can see it in the timeline graph

and in the table below

so this answers one part of your questions, about how/where the numbers came from.

here's the results compiled with cargo +nightly build -j1
   Compiling owo-colors v2.1.0
time:   0.014; rss:   15MB ->   16MB (   +1MB)  parse_crate
time:   0.000; rss:   16MB ->   16MB (   +0MB)  attributes_injection
time:   0.000; rss:   16MB ->   16MB (   +0MB)  plugin_loading
time:   0.000; rss:   16MB ->   16MB (   +0MB)  plugin_registration
time:   0.000; rss:   17MB ->   17MB (   +0MB)  crate_injection
time:   0.177; rss:   17MB ->   38MB (  +21MB)  expand_crate
time:   0.000; rss:   38MB ->   38MB (   +0MB)  check_unused_macros
time:   0.178; rss:   17MB ->   38MB (  +21MB)  macro_expand_crate
time:   0.000; rss:   38MB ->   38MB (   +0MB)  maybe_building_test_harness
time:   0.006; rss:   38MB ->   38MB (   +0MB)  AST_validation
time:   0.000; rss:   38MB ->   38MB (   +0MB)  maybe_create_a_macro_crate
time:   0.000; rss:   38MB ->   38MB (   +0MB)  finalize_imports
time:   0.001; rss:   38MB ->   38MB (   +0MB)  finalize_macro_resolutions
time:   0.016; rss:   38MB ->   42MB (   +4MB)  late_resolve_crate
time:   0.000; rss:   42MB ->   42MB (   +0MB)  resolve_main
time:   0.002; rss:   42MB ->   42MB (   +0MB)  resolve_check_unused
time:   0.000; rss:   42MB ->   42MB (   +0MB)  resolve_report_errors
time:   0.001; rss:   42MB ->   42MB (   +0MB)  resolve_postprocess
time:   0.020; rss:   38MB ->   42MB (   +4MB)  resolve_crate
time:   0.003; rss:   42MB ->   42MB (   +0MB)  complete_gated_feature_checking
time:   0.208; rss:   16MB ->   42MB (  +26MB)  configure_and_expand
time:   0.002; rss:   42MB ->   42MB (   +0MB)  prepare_outputs
time:   0.028; rss:   42MB ->   48MB (   +6MB)  hir_lowering
time:   0.017; rss:   48MB ->   48MB (   +0MB)  early_lint_checks
time:   0.010; rss:   48MB ->   41MB (   -8MB)  drop_ast
time:   0.000; rss:   38MB ->   38MB (   +0MB)  setup_global_ctxt
time:   0.000; rss:   41MB ->   41MB (   +0MB)  looking_for_entry_point
time:   0.000; rss:   41MB ->   41MB (   +0MB)  looking_for_derive_registrar
time:   0.018; rss:   41MB ->   44MB (   +3MB)  misc_checking_1
time:   0.015; rss:   44MB ->   46MB (   +3MB)  type_collecting
time:   0.001; rss:   46MB ->   46MB (   +0MB)  impl_wf_inference
time:   0.000; rss:   52MB ->   52MB (   +0MB)  unsafety_checking
time:   0.000; rss:   52MB ->   52MB (   +0MB)  orphan_checking
time:   0.008; rss:   46MB ->   52MB (   +6MB)  coherence_checking
time:   0.041; rss:   52MB ->   56MB (   +4MB)  wf_checking
time:   0.012; rss:   56MB ->   57MB (   +1MB)  item_types_checking
time:   0.208; rss:   57MB ->   69MB (  +12MB)  item_bodies_checking
time:   0.289; rss:   44MB ->   69MB (  +25MB)  type_check_crate
time:   0.021; rss:   69MB ->   69MB (   +1MB)  match_checking
time:   0.005; rss:   69MB ->   69MB (   +0MB)  liveness_and_intrinsic_checking
time:   0.026; rss:   69MB ->   69MB (   +1MB)  misc_checking_2
time:   0.225; rss:   69MB ->   86MB (  +17MB)  MIR_borrow_checking
time:   0.022; rss:   86MB ->   87MB (   +1MB)  MIR_effect_checking
time:   0.000; rss:   87MB ->   87MB (   +0MB)  layout_testing
time:   0.003; rss:   87MB ->   88MB (   +0MB)  death_checking
time:   0.000; rss:   88MB ->   88MB (   +0MB)  unused_lib_feature_checking
time:   0.016; rss:   88MB ->   88MB (   +1MB)  crate_lints
time:   0.014; rss:   88MB ->   89MB (   +0MB)  module_lints
time:   0.031; rss:   88MB ->   89MB (   +1MB)  lint_checking
time:   0.012; rss:   89MB ->   89MB (   +0MB)  privacy_checking_modules
time:   0.050; rss:   87MB ->   89MB (   +2MB)  misc_checking_3
time:   0.000; rss:   96MB ->   96MB (   +0MB)  monomorphization_collector_root_collections
time:   0.054; rss:   96MB ->  102MB (   +7MB)  monomorphization_collector_graph_walk
time:   0.005; rss:  102MB ->  103MB (   +0MB)  partition_and_assert_distinct_symbols
time:   0.119; rss:   89MB ->  103MB (  +14MB)  generate_crate_metadata
time:   0.000; rss:  104MB ->  104MB (   +0MB)  find_cgu_reuse
time:   0.000; rss:  106MB ->  106MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.0)
time:   0.000; rss:  106MB ->  106MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.0)
time:   0.000; rss:  109MB ->  109MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.14)
time:   0.000; rss:  109MB ->  109MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.14)
time:   0.000; rss:  110MB ->  110MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.1)
time:   0.000; rss:  110MB ->  110MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.1)
time:   0.000; rss:  108MB ->  108MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.13)
time:   0.000; rss:  108MB ->  108MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.13)
time:   0.000; rss:  109MB ->  109MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.2)
time:   0.000; rss:  109MB ->  109MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.2)
time:   0.000; rss:  108MB ->  108MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.12)
time:   0.000; rss:  108MB ->  108MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.12)
time:   0.000; rss:  109MB ->  109MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.3)
time:   0.003; rss:  109MB ->  109MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.3)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.10)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.10)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.6)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.6)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.9)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.9)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.4)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.4)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.11)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.11)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.5)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.5)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.8)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.8)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.15)
time:   0.000; rss:  111MB ->  111MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.15)
time:   0.000; rss:  112MB ->  112MB (   +0MB)  LLVM_module_optimize_function_passes(owo_colors.0d562811-cgu.7)
time:   0.000; rss:  112MB ->  112MB (   +0MB)  LLVM_module_optimize_module_passes(owo_colors.0d562811-cgu.7)
time:   0.060; rss:  104MB ->  112MB (   +8MB)  codegen_to_LLVM_IR
time:   0.302; rss:  106MB ->  112MB (   +6MB)  LLVM_passes(crate)
time:   0.317; rss:  103MB ->  112MB (   +8MB)  codegen_crate
time:   0.000; rss:  112MB ->  112MB (   +0MB)  serialize_dep_graph
time:   0.016; rss:  112MB ->   51MB (  -60MB)  free_global_ctxt
time:   0.000; rss:   51MB ->   51MB (   +0MB)  join_worker_thread
time:   0.000; rss:   51MB ->   51MB (   +0MB)  finish_ongoing_codegen
time:   0.000; rss:   51MB ->   51MB (   +0MB)  llvm_dump_timing_file
time:   0.000; rss:   51MB ->   51MB (   +0MB)  serialize_work_products
time:   0.001; rss:   51MB ->   51MB (   +0MB)  link_binary_check_files_are_writeable
time:   0.133; rss:   51MB ->   51MB (   +1MB)  link_rlib
time:   0.004; rss:   51MB ->   51MB (   +0MB)  link_binary_remove_temps
time:   0.140; rss:   51MB ->   51MB (   +1MB)  link_binary
time:   0.140; rss:   51MB ->   50MB (   -1MB)  link_crate
time:   0.140; rss:   51MB ->   50MB (   -2MB)  link
time:   1.534; rss:   11MB ->   46MB (  +35MB)  total

wow that's so weird. your specs are beefier than anything I have to test on and yet even my digital ocean machine chokes less??? that seems like an issue in something deeper than owo-colors x.x

Here's what the optimized styles produces with the following code.

    let colored_text = format!(
        "{hello} {world}",
        hello = "Hello".fg::<Black>().bg::<White>(),
        world = "World".fg::<Magenta>().bg::<Green>(),
    );

    let colored_text2 = format!(
        "{hello} {world}",
        hello = Color::Black.on(Color::White).paint("Hello"),
        world = Color::Magenta.on(Color::Green).paint("World"),
    );

    let hello = String::from("Hello");
    let world = String::from("World");
    let colored_text3 = format!(
        "{hello} {world}",
        hello = hello.black().on_white(),
        world = world.magenta().on_green(),
    );

    println!("owo-colors  : {:#?}", colored_text);
    println!("{}", colored_text);
    println!("nu-ansi-term: {:#?}", colored_text2);
    println!("{}", colored_text2);
    println!("owo styles  : {:#?}", colored_text3);
    println!("{}", colored_text3);

47 is the background
30 is the foreground
apparently, it doesn't matter which order they're in, which is kind of weird. i'm not exactly sure what the ansi spec says.

I looked into the ANSI spec a good bit while working on this and from my reading of it, the order of ANSI codes is arbitrary they're just semicolon-separated and applied in-order. since setting foreground doesn't clear background or vice-versa, the order is pretty irrelevant (short of a buggy implementation by the terminal emulator)
Also all of my testing has been against alacritty, so there's definitely a non-zero chance that some terminal emulator has compat issues due to improper implementation. If you happen to see that happening to anyone absolutely forward such an issue my way!

OH. I see the bug you're seeing in the first test. I can totally fix that, I just forgot to make this optimization work for the generic methods!!
If you test against the non-generic methods (e.g. .fg::<Black>().bg::<White>() can be made into .black().on_white()) you'll see this is already working on those, but good catch!!

Good to know. I use Windows Terminal.
Im guessing that some of the performance differences we're seeing may be related to NTFS and McAfee virus scan. I know ext3 is faster. I'll try to build in wsl2 and see what the perf looks like.

Just tested the output and nu-ansi-term, owo-colors, and owo styles output are identical.  I also tested the perf in wsl2 and i get this.
   Compiling owo-colors v2.1.0
   Completed owo-colors v2.1.0 in 0.7s
   Compiling nu-ansi-term v0.36.0
   Completed nu-ansi-term v0.36.0 in 0.4s

it's really miette that uses owo-colors. engine-q is going to be absorbed into nushell (or vice versa) and it's going to take a few months. nushell/engine-q aren't currently using owo-colors. I just had objections to miette using it because the ansi sequences were concatenated, which you've now fixed. so, release when you feel you're ready.
good work on working through these issues. as far as i'm concerned, the issues are closed and you can feel free to close this issue too. thanks again!

np!! thank you so much for your help working through these issues


fix(ci): Fix Pwn Request Vulnerability

Proposed Changes

Update the validate PR workflow to prevent injection vulnerability

It should be a quick fix for you.
Reference:
https://securitylab.github.com/research/github-actions-preventing-pwn-requests/
I submit this contribution under the Apache-2.0 license.

Fixed title injection for y'all too :)

Thank you so much for contributing! I will carefully read the referenced article :)


Nasty bug with simple example for table rendering

I faced a huge bug on HTML code that worked fine with PHP Markdown.
On demo page, let's add the following code. Notice there are spaces needed for indentation for ease of reading.

      <colgroup>
         <col width="80px" />
         <col width="24px" />
         <col width="24px" />
         <col width="24px" />
         <col width="24px" />
         <col width="24px" />
         <col width="24px" />
         <col width="24px" />
         <col width="700px" />

      </colgroup>

   segments

steps
1
2
3
4
5
6

1
A
A
A
A
A
A

all segments are displayed in the darkest green [A]

2
D
A
A
A
A
A

S1 is lit in the brightest green [D] ; others segments don't change

3
C
D
A
A
A
A

S2 is lit in [D], S1 is lit in [C] ; others segments don't change

4
B
C
D
A
A
A

S3 is lit in [D], S2 is lit in [C], S1 is lit in [B] ; others segments don't change

5
A
B
C
D
A
A

S4 is lit in [D], S3 is lit in [C], S2 is lit in [B], S1 is lit in [A] ; others segments don't change

6
A
A
B
C
D
A

...

7
A
A
A
B
C
D

and so on

8
A
A
A
A
B
C

...

9
A
A
A
A
A
B

step 9 (the last one) : S6 is lit in [B] and all others are lit in [A]

Now let's see result in demo page. Markdown is better and faster.

Here is a better indentation of this HTML code:

    <colgroup>
       <col width="80px" />
       <col width="24px" />
       <col width="24px" />
       <col width="24px" />
       <col width="24px" />
       <col width="24px" />
       <col width="24px" />
       <col width="24px" />
       <col width="700px" />
    </colgroup>

       segments

      steps
      1
      2
      3
      4
      5
      6

      1
      A
      A
      A
      A
      A
      A

      all segments are displayed in the darkest green [A]

      2
      D
      A
      A
      A
      A
      A

      S1 is lit in the brightest green [D] ; others segments don't change

      3
        C
        D
        A
        A
        A
        A

      S2 is lit in [D], S1 is lit in [C] ; others segments don't change

      4
      B
      C
      D
      A
      A
      A

      S3 is lit in [D], S2 is lit in [C], S1 is lit in [B] ; others segments don't change

      5
      A
      B
      C
      D
      A
      A

      S4 is lit in [D], S3 is lit in [C], S2 is lit in [B], S1 is lit in [A] ; others segments don't change

      6
      A
      A
      B
      C
      D
      A

      ...

      7
      A
      A
      A
      B
      C
      D

      and so on

      8
      A
      A
      A
      A
      B
      C

      ...

      9
      A
      A
      A
      A
      A
      B

      step 9 (the last one) : S6 is lit in [B] and all others are lit in [A]

Perhaps it's the HTML format！Try this!

  <colgroup>
     <col width="80px" />
     <col width="24px" />
     <col width="24px" />
     <col width="24px" />
     <col width="24px" />
     <col width="24px" />
     <col width="24px" />
     <col width="24px" />
     <col width="700px" />

  </colgroup>

segments

steps
1
2
3
4
5
6

1
A
A
A
A
A
A

all segments are displayed in the darkest green [A]

2
D
A
A
A
A
A

S1 is lit in the brightest green [D] ; others segments don't change

3
C
D
A
A
A
A

S2 is lit in [D], S1 is lit in [C] ; others segments don't change

4
B
C
D
A
A
A

S3 is lit in [D], S2 is lit in [C], S1 is lit in [B] ; others segments don't change

5
A
B
C
D
A
A

S4 is lit in [D], S3 is lit in [C], S2 is lit in [B], S1 is lit in [A] ; others segments don't change

6
A
A
B
C
D
A

...

7
A
A
A
B
C
D

and so on

8
A
A
A
A
B
C

...

9
A
A
A
A
A
B

step 9 (the last one) : S6 is lit in [B] and all others are lit in [A]

@pxlh007
I don't notice any visual difference but a break line on your code between the last col tag and the colgroup ending tag. Is that the change?


[RTL] Overlapping case statements in ibex_alu.sv

Ran into this warning in OpenTitan chip level sim:
Warning-[RT-MTOCMUCS] More than one condition match in statement
../src/lowrisc_ibex_ibex_core_0.1/rtl/ibex_alu.sv, 78
  More than one condition matches are found in 'unique case' statement inside
  tb.dut.top_earlgrey.u_rv_core_ibex.u_core.ex_block_i.alu_i, at time
  234570001 psps.

  Line number 79 and 80 are overlapping

Steps to reproduce:

Clone OpenTitan repo, cd to repo top
Checkout PR 2052 if it is not already merged
Run chip DV sim:

$ util/dvsim/dvsim.py -i hw/top_earlgrey/dv/chip_sim_cfg.hjson -i chip_uart_tx_rx [--waves] [--build-modes en_ibex_tracer]

PTAL, thanks!

I think this is probably a duplicate of #845, which was fixed in the Ibex repository by commit 4f349a. Can you try again, passing the -xlrm uniq_prior_final arguments to VCS? If that silences the message, you'll need to add the equivalent arguments for OpenTitan.

Okay, thanks!

Duplicate of #845, closing.


extended global Vue cannot use its own plugins

Version
2.5.13
Reproduction link
https://jsfiddle.net/b5a1zh6u/31/
Steps to reproduce
just open this simple use case: https://jsfiddle.net/b5a1zh6u/31/
you will see an error in console.
What is expected?
As written in this issue, Vue.extend()
does not work as expected.
The code should create a new global Vue instance and installa a plugin that is used by the component template.
What is actually happening?
console error due to $t method not found.

I wrote the reproduction fiddle starting from Evan's one
if I set the $t function directly or as a plugin, on the global Vue like this, it works:
Vue.prototype.$t = val => `* ${val}`

Just to bring more precision, the plugin seems to install the method $t only on the root component. Any child component cannot access the method $t.
Example: https://jsfiddle.net/j34b5g30/41/
This differ if we use the plugin directly on the global instance before extending vue.
Example: https://jsfiddle.net/r2stwc4h/2/

It was my mistake assuming this would work in the previous comments. To make it fully operate as a base constructor, an additional internal flag needs to be set to true. This cannot be made the default for Vue.extend, so it will likely be exposed via a public API in 2.6.

Just to confirm, is there a way around this in Vue 2? Or is a solution only available in 3?


Add support for Fastify v3

Once Fastify v3 is released, it should be added into test suite. There should be at least one change to be done in order to support it: https://github.com/puzpuzpuz/cls-rtracer/pull/30#discussion_r438773699

Implemented in #36


Support "dynamically sized" worker node pools with elemental nodes.

Describe the solution you'd like:
Currently when provisioning an RKE2/K3s cluster with Elemental, you statically define the sizes of your node pools, and when elemental machines that match labels come online, it will fill the size of that pool.
Specifically with pure worker pools (aka, when Controlplane and Etcd are not selected to avoid quorum issues), for situations where you dynamically want pools to grow as machines come online, it'd be useful to be able to select "Dynamically Sized" instead of a quantity of nodes. As new matching machines come online, they would automatically get added to that pool to scale it out without having to adjust the cluster configuration itself.
Anything else you would like to add:
N/A
Environment:

Elemental release version (use cat /etc/os-release): 1.5.3
Rancher version: 2.8.3
Kubernetes version (use kubectl version): 1.2x+
Cloud provider or hardware configuration: N/A

Not sure this can be handled within Elemental alone. This sounds more like a generic Rancher provisioning request 🤔

Not sure this can be handled within Elemental alone. This sounds more like a generic Rancher provisioning request 🤔

Yea, I wasn't sure what team this would fall under, but it is unique to Elemental provisioning through Rancher (since Elemental is the only provisioning mechanism that can dynamically machines to node pools as they come online).


[QUESTION] Unable to clone libopenssl

When running CMake on Linux (Armbian Focus for an SoC called Pinecube), it locks up on "Cloning into 'project_libopenssl':
[100%] Built target libkvscproducer-download
-- Kinesis Video Producer path is /home/radagan/dev/amazon-kinesis-video-streams-producer-sdk-cpp/dependency/libkvscproducer/kvscproducer-src
-- Begin building dependencies.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/radagan/dev/amazon-kinesis-video-streams-producer-sdk-cpp/open-source/local/libopenssl
Scanning dependencies of target project_libopenssl
[ 11%] Creating directories for 'project_libopenssl'
[ 22%] Performing download step (git clone) for 'project_libopenssl'
Cloning into 'project_libopenssl'...
error: index-pack died of signal 9
fatal: index-pack failed
Cloning into 'project_libopenssl'...
error: index-pack died of signal 9
fatal: index-pack failed
Cloning into 'project_libopenssl'...
error: index-pack died of signal 9
fatal: index-pack failed
-- Had to git clone more than once:
          3 times.
CMake Error at /home/radagan/dev/amazon-kinesis-video-streams-producer-sdk-cpp/open-source/local/libopenssl/build/tmp/project_libopenssl-gitclone.cmake:31 (message):
  Failed to clone repository: 'https://github.com/openssl/openssl.git'

make[2]: *** [CMakeFiles/project_libopenssl.dir/build.make:92: build/src/project_libopenssl-stamp/project_libopenssl-download] Error 1
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/project_libopenssl.dir/all] Error 2
make: *** [Makefile:84: all] Error 2
CMake Error at dependency/libkvscproducer/kvscproducer-src/CMake/Utilities.cmake:93 (message):
  CMake step for libopenssl failed: 2
Call Stack (most recent call first):
  dependency/libkvscproducer/kvscproducer-src/CMakeLists.txt:66 (build_dependency)

Any idea what I'm doing wrong. I did clone with submodules.

@Radagan
This typically happens when the on-disk memory is not enough to clone. I cannot think of any other reason for clone to fail.

That could definitely be the case. It does build without dependencies, but then it fails with "Could not initialize supporting library", but without naming the missing lib. I think I have them on my system, but passing /usr/lib doesn't work. Is there an easy way to get the list of dependencies built so I could just copy them to open-source/local/lib?

@Radagan there are the list of dependencies - https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp/tree/master/CMake/Dependencies
I myself haven't separately built them and the copied to open-source/local/lib but I'm guessing that should work.

@kateyanurag Thank you! Will give it a go.

What was the solution? Getting on this same issue on a Ubuntu host using the official docker image with lots of RAM and HD space available

CMake Error at dependency/libkvscproducer/kvscproducer-src/CMake/Utilities.cmake:93 (message):
CMake step for libopenssl failed: 2
Call Stack (most recent call first):
dependency/libkvscproducer/kvscproducer-src/CMakeLists.txt:74(build_dependency)
I am also getting the same error can you please help me with this?


Issue with Git LFS resolved

@bluntgames @ChairGraveyard @devilmhzx @einthomas @EvoPulseGaming @wblong
Since you guys forked the repo, you might be interested to know that now it's possible to actually check out everything including assets, so the demo will actually work.
Cheers

Solved by un-LFS-ing the whole repo. Yay!

Notifying the stargazers too
@mfoda @jarllarsson @infancy @Aidmx @ZeroKill1024 @neil3d @moltke36 @danitxu79 @dantesai @michal-rutkowski @Darkoxen @RazielSun @etiennepinchon @ArcaneParadigm @mosaicchen @crusader2k @MatzeOGH @Eragon-Brisingr


Undocumented async device -> threadgroup copy instruction

In Apple's NDArrayMatrixMultiplyNNA14 kernel in MPS for matrix multiply, they use a private simdgroup API air.simdgroup_async_copy_2d to do an async copy from device memory to threadgroup memory, similar to cp.async in Nvidia PTX. This is using a currently undocumented instruction.
Relevant disassembly:
 228: 7e7dac098000         mov              r31, u22
 22e: 7e059a098010         mov              r33, u13
 234: 7e6198098000         mov              r24, u12
 23a: 2541                 <disassembly failed>
 23c: 2d00                 <disassembly failed>
 23e: 13d0                 <disassembly failed>
 240: 0000                 <disassembly failed>
 242: 2549                 <disassembly failed>
 244: 4400                 <disassembly failed>
 246: 13d0                 <disassembly failed>
 248: 0000                 <disassembly failed>
 24a: 520e00000000         pop_exec         r0l, 1
 250: f592                 TODO.unkF59      0, 2
 252: f592                 TODO.unkF59      0, 2
 254: 520e00000000         pop_exec         r0l, 1
 25a: 0e66146106001000     iadd             r25h, u10l, r19l, lsl 2
 262: 0e54167106001000     iadd             r21l, u11l, r19h, lsl 2
 26a: 6800                 threadgroup_barrier
 26c: 69b5330234810010     threadgroup_load i32, xy, r45_r46, r25h, 0
 274: 699d2a0234810010     threadgroup_load i32, xy, r39_r40, r21l, 0
 27c: 69a5330234840210     threadgroup_load i32, xy, r41_r42, r25h, 128
 284: 69952a2234840010     threadgroup_load i32, xy, r37_r38, r21l, 2
 28c: ef454eae25a20285     simd_matrix_fmadd32 r17_r18.cache, r39_r40, r45_r46, r17_r18.cache
 294: 6f454a2e25620285     simd_matrix_fmadd32 r17_r18, r37_r38, r41_r42, r17_r18

Will try to add it, but if I don't get to it, I wanted to file this issue for the next person who finds the secret async.

I'd also be very interested in a way to use this from metal.
declare %struct._simdgroup_event_t* @air.simdgroup_async_copy_2d.p3i8.p1i8(i64, i64, i8 addrspace(3)* nocapture writeonly, i64, i64, <2 x i64>, i8 addrspace(1)* nocapture readonly, i64, i64, <2 x i64>, <2 x i64>, i32) local_unnamed_addr #9

is the signature of the LLVM intrinsic, but I don't see how it's broken out, nor do I see a way to call LLVM intrinsics. I can't find anything like __metal_simdgroup_matrix_8x8_load for it.

THE GeoHot, wow, Thanks for hacking into AGX

The SIMD-group copy API exists is part of the Metal Standard Library, headers metal_simdgroup_async and metal_simdgroup_future. The first header includes the intrinsic you're referencing.
Apple probably introduced the API with their A14 chip. That allowed them to harness the SIMD-scoped matrix multiply instructions. NDArrayMatrixMultiplyNNA14 is specialized for the A14, which has half the FP32 compute power of M1 and A15. If you're optimizing for generic AS, beware of the difference. Another MPS kernel should be optimized for A14X.

Yea, I got it to work, but I had to install Xcode. I didn't find that API in the system Metal.

I'm trying to locate the header for SIMD-group async copies, and I can't find it now (Xcode 14.3). metal_simdgroup_async and metal_simdgroup_future are absent from the Metal stdlib headers directory. @geohot are you still able to find them? If Apple removed them, I may need to create something like opencl-metal-stdlib to re-expose the AIR symbols.

If you download the Xcode 14.2 app from Apple's downloads archive, the metal_simdgroup_async and metal_simdgroup_future headers are there. You need the full 7 GB app, not just the command-line tools. Then go to:
/Applications/Xcode 14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/metal/macos/lib/clang/31001.667/include/metal


CentOS 7 + Docker 1.6.2 doesn't work with the documentation

docker exec -it cachet_cachet_1 php artisan migrate --force
[PDOException]
SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
I have tried to build it several times. It stopped working very recently.

Looks like it's trying to connect locally instead of the mysql container (maybe a missing $DB_HOST somewhere)

This should be fixed in my PR #788 - The Docker image will now utilize a .env file for configuration.


Support mermaid for any markdown file (.md) file in the repo

Consider to support mermaid to any markdown file in the repo ( https://mermaid-js.github.io/mermaid/#/ )
Currently DevOps Wiki already supports mermaid (https://docs.microsoft.com/en-us/azure/devops/project/wiki/wiki-markdown-guidance?view=azure-devops#add-mermaid-diagrams-to-a-wiki-page) but md files in the repo are not supported.

@msebolt I suspect this is not the right repo for this issue, would you help me to point to the correct one?

@rorezende Somewhat, in we can only make changes to the documentation here.
For product issues, please refer to the Azure DevOps community, or add to an ongoing post here.

Thanks for the suggestion! The product teams aren’t always watching this repository, so to make sure it gets in front of them for triage, please submit your idea here.


[Snyk] Security upgrade ch.qos.logback:logback-classic from 1.2.3 to 1.5.13

Snyk has created this PR to fix 3 vulnerabilities in the maven dependencies of this project.
Snyk changed the following file(s):

product-service/pom.xml

Vulnerabilities that will be fixed with an upgrade:

Issue
Score
Upgrade

Improper Neutralization of Special Elements SNYK-JAVA-CHQOSLOGBACK-8539867
  84
ch.qos.logback:logback-classic:  1.2.3 -> 1.5.13  No Path Found No Known Exploit

Server-side Request Forgery (SSRF) SNYK-JAVA-CHQOSLOGBACK-8539865
  78
ch.qos.logback:logback-classic:  1.2.3 -> 1.5.13  No Path Found No Known Exploit

Improper Neutralization of Special Elements SNYK-JAVA-CHQOSLOGBACK-8539866
  63
ch.qos.logback:logback-classic:  1.2.3 -> 1.5.13  No Path Found No Known Exploit

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Server-side Request Forgery (SSRF)

:tada: Snyk checks have passed. No issues have been found so far.
:white_check_mark: security/snyk check is complete. No issues have been found. (View Details)
:white_check_mark: license/snyk check is complete. No issues have been found. (View Details)


[Feat]: Adding Free Hit to good first issue(website)

What feature?

By adding our website to the good first issue, we can easily make our website reach multiple people.

for more info 👉🏻 https://github.com/deepsourcelabs/good-first-issue
Add screenshots

Our website will be available on their website, have a look below 👇🏻

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@JasonDsouza212 did you add the website?


hpx::lcos::condition_variable could be suspect to deadlocks

The current code for hpx::lcos::condition_variable (and the _any variant) might lead to deadlocks.
The deadlock situation might, for example, be encountered, when concurrently calling wait.
The problem is, that the outer lock (provided from the caller) is being tried to be acquired unconditionally with the inner lock (protecting the lcos::detail::condition_variable) is being held.

I'd assume you're talking about this code: https://github.com/STEllAR-GROUP/hpx/blob/master/hpx/lcos/local/condition_variable.hpp#L50-L65
Here, the outer lock is re-acquired in the destructor of the util::unlock_guard<> while the inner lock is still being held (and for a good reason, btw). I'm not sure what you mean by 'unconditionally' in

the outer lock (provided from the caller) is being tried to be acquired unconditionally

What condition should have been applied to that re-acquire operation?

Yes, I am talking about this code. The outer lock being re-acquired while the inner lock is being held is exactly what will lead to a deadlock:
T1              | T2
================| ================
outer.lock();   | ...
inner.lock();   | ...
outer.unlock(); | ...
inner.unlock(); | outer.lock();
inner.lock()    | inner.lock();
// Assume T1 acquired inner.lock(). T2 still holds outer, once T1 tries to acuire outer.lock, we get into a deadlock situation.
outer.lock();   | ...
inner.unlock(); | ...

With "uncoditionally", I meant that we don't consider that another lock might be held when trying to acquire the other.

@sithhell what lines of code correspond to T1 calls?


Error on printing report in verbose mode

Getting an invalid subscript error when printing a report in verbose mode.

Added seq_along to verbose mode loop to get numeric index in print.report_spec.


Couldnt convert to OpenVino

python "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo.py" --input_model frozen_darknet_yolov4_model.pb --transformations_config yolo_v4_tiny.json --batch 1 --reverse_input_channels          Model Optimizer arguments:                                                                                Common parameters:                                                                                - Path to the Input Model:      C:!Oleg!!!Optimization\OpenVINO-YOLOV4-master\frozen_darknet_yolov4_model.pb                                                                                - Path for generated IR:        C:!Oleg!!!Optimization\OpenVINO-YOLOV4-master.                                                                                - IR output name:       frozen_darknet_yolov4_model                                                                                - Log level:    ERROR                                                                                - Batch:        1                                                                                - Input layers:         Not specified, inherited from the model                                                                                - Output layers:        Not specified, inherited from the model                                                                                - Input shapes:         Not specified, inherited from the model                                                                                - Mean values:  Not specified                                                                                - Scale values:         Not specified                                                                                - Scale factor:         Not specified                                                                                - Precision of IR:      FP32                                                                                - Enable fusing:        True                                                                                - Enable grouped convolutions fusing:   True                                                                                - Move mean values to preprocess section:       None                                                                                - Reverse input channels:       True                                                                                TensorFlow specific parameters:                                                                                - Input model in text protobuf format:  False                                                                                - Path to model dump for TensorBoard:   None                                                                                - List of shared libraries with TensorFlow custom layers implementation:        None                                                                                - Update the configuration file with input/output node names:   None                                                                                - Use configuration file used to generate the model with Object Detection API:  None                                                                                - Use the config file:  None                                                                                Model Optimizer version:        2021.2.0-1877-176bdf51370-releases/2021/2                                                                                [ WARNING ]                                                                                Detected not satisfied dependencies:                                                                                test-generator: installed: 0.1.2, required: == 0.1.1                                                                                Please install required versions of components or use install_prerequisites script                                                                                C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\install_prerequisites\install_prerequisites_tf.bat                                                                                Note that install_prerequisites scripts may install additional components.                                                                                [ ERROR ]  Exception occurred during running replacer "fusing" (<class 'extensions.middle.fusings.Fusing'>): After partial shape inference were found shape collision for node detector/cspdarknet-53/Conv/BatchNorm/FusedBatchNormV3/beta (old shape: [  1  32 416 416], new shape: [  1  32 416  -1])
I was trying different configurations, but couldnt convert

what's your tiny model?Custom model or official yolov4tiny?
you should install all the prerequisites according to openvino
what configuration you have changed??
your .cfg,.json?

And why you use yolo_v4_tiny.json but there is detector/cspdarknet-53/Conv/BatchNorm/FusedBatchNormV3/beta in your model?

Its my custom yolov4-tiny model
Here .json
[
{
"id": "TFYOLOV4",
"match_kind": "general",
"custom_attributes": {
"classes": 44,
"anchors": [30,21,61,29,58,53,95,38,122,49,92,70,142,81,283,375],
"coords": 4,
"num": 8,
"masks": [[4, 5, 6, 7], [0, 1, 2, 3]],
"entry_points": ["detector/yolo-v4-tiny/Reshape", "detector/yolo-v4-tiny/Reshape_4"]
}
}
]
I couldnt understand why detector is "detector/cspdarknet-53/Conv/BatchNorm/FusedBatchNormV3/beta"

Where do you find this json file?
id and anchors masks are all wrong
i have already provide .json file and you only need to change "classes"
[
{
"id": "TFYOLOV3",
"match_kind": "general",
"custom_attributes": {
"classes": 44,
"anchors": [10, 14, 23, 27, 37, 58, 81, 82, 135, 169, 344, 319],
"coords": 4,
"num": 6,
"masks": [[3, 4, 5], [1, 2, 3]],
"entry_points": ["detector/yolo-v4-tiny/Reshape", "detector/yolo-v4-tiny/Reshape_4"]
}
}
]

It's anchors from my .cfg file
Its my custom pretrained tiny model

"id" must be "TFYOLOV3"

Its better. Trying to install  prerequisites according to openvino.
Could we talk with you about some consulting - to help me convert and prune some my models for OpenVino?
My email is<EMAIL_ADDRESS>Thank you

[17. 19. 1...
[ 2021-04-02 18:37:12,875 ] [ DEBUG ] [ infer:129 ]  --------------------
[ 2021-04-02 18:37:12,875 ] [ DEBUG ] [ infer:130 ]  Partial infer for detector/yolo-v4-tiny/Tile/YoloRegion
[ 2021-04-02 18:37:12,875 ] [ DEBUG ] [ infer:131 ]  Op: RegionYolo
[ 2021-04-02 18:37:12,875 ] [ DEBUG ] [ infer:132 ]  Inputs:
[ 2021-04-02 18:37:12,876 ] [ DEBUG ] [ infer:32 ]  input[0]: shape = [400   6], value = [[ 0.  0.  0.  0.  0.  0.]
[ 1.  0.  1.  0.  1.  0.]
[ 2.  0.  2.  0.  2.  0.]
...
[17. 19. 1...
[ ERROR ]  Cannot infer shapes or values for node "detector/yolo-v4-tiny/Tile/YoloRegion".
[ ERROR ]  index 2 is out of bounds for axis 0 with size 2
[ ERROR ]
[ ERROR ]  It can happen due to bug in custom shape infer function <function RegionYoloOp.regionyolo_infer at 0x000002054E8D5550>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ 2021-04-02 18:37:12,878 ] [ DEBUG ] [ infer:197 ]  Node "detector/yolo-v4-tiny/Tile/YoloRegion" attributes: {'kind': 'op', 'type': 'RegionYolo', 'op': 'RegionYolo', 'version': 'opset1', 'in_ports_count': 1, 'out_ports_count': 1, 'infer': <function RegionYoloOp.regionyolo_infer at 0x000002054E8D5550>, 'name': 'detector/yolo-v4-tiny/Tile/YoloRegion', 'axis': 1, 'end_axis': 1, 'do_softmax': 0, 'classes': 44, 'anchors': [38, 20, 60, 28, 58, 53, 92, 38, 128, 49, 92, 70, 142, 81, 283, 365], 'coords': 4, 'num': 8, 'entry_points': ['detector/yolo-v4-tiny/Reshape', 'detector/yolo-v4-tiny/Reshape_4'], 'mask': [0, 1, 2, 3], 'dim_attrs': ['channel_dims', 'spatial_dims', 'batch_dims'], 'shape_attrs': ['pad', 'output_shape', 'shape', 'window', 'stride'], 'IE': [('layer', [('id', <function Op.substitute_ie_attrs.. at 0x000002055006C8B0>), 'name', 'type', 'version'], [('data', ['coords', 'classes', 'num', 'axis', 'end_axis', ('do_softmax', <function RegionYoloOp.backend_attrs.. at 0x000002055006C940>), ('anchors', <function RegionYoloOp.backend_attrs.. at 0x000002055006C9D0>), ('mask', <function RegionYoloOp.backend_attrs.. at 0x000002055006CA60>)], []), '@ports', '@consts'])], '_in_ports': {0: {}}, '_out_ports': {0: {}}, 'is_output_reachable': True, 'is_undead': False, 'is_const_producer': False, 'is_partial_inferred': False}
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "detector/yolo-v4-tiny/Tile/YoloRegion" node.
For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)
[ 2021-04-02 18:37:12,880 ] [ DEBUG ] [ main:361 ]  Traceback (most recent call last):
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\middle\passes\infer.py", line 135, in partial_infer
node.infer(node)
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\extensions\ops\regionyolo.py", line 84, in regionyolo_infer
width=input_shape[get_width_dim(layout, 4)])
IndexError: index 2 is out of bounds for axis 0 with size 2
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\utils\class_registration.py", line 288, in apply_transform
replacer.find_and_replace_pattern(graph)
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\extensions\middle\PartialInfer.py", line 33, in find_and_replace_pattern
partial_infer(graph)
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\middle\passes\infer.py", line 198, in partial_infer
raise Error('Stopped shape/value propagation at "{}" node. '.format(node.soft_get('name')) +
mo.utils.error.Error: Stopped shape/value propagation at "detector/yolo-v4-tiny/Tile/YoloRegion" node.
For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\main.py", line 345, in main
ret_code = driver(argv)
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\main.py", line 309, in driver
ret_res = emit_ir(prepare_ir(argv), argv)
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\main.py", line 252, in prepare_ir
graph = unified_pipeline(argv)
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\pipeline\unified.py", line 25, in unified_pipeline
class_registration.apply_replacements(graph, [
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\utils\class_registration.py", line 340, in apply_replacements
apply_replacements_list(graph, replacers_order)
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\utils\class_registration.py", line 326, in apply_replacements_list
apply_transform(
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\utils\logger.py", line 124, in wrapper
function(*args, **kwargs)
File "C:\Program Files (x86)\Intel\openvino_2021.3.394\deployment_tools\model_optimizer\mo\utils\class_registration.py", line 306, in apply_transform
raise Error('Exception occurred during running replacer "{}" ({}): {}'.format(
mo.utils.error.Error: Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "detector/yolo-v4-tiny/Tile/YoloRegion" node.
For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)

Hi TNTWEN
when I convert my yolov4 tiny model to IR model from tensorflow model, I got same problem.
Just only change cfg classes to 17.
RUN
python3 /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo.py  --input_model models/yolov4_tiny_17.pb  --transformations_config models/yolo_v4_tiny_17.json -b 1 --output_dir models

[ ERROR ]  Cannot infer shapes or values for node "detector/yolo-v4-tiny/Tile/YoloRegion".
[ ERROR ]  index 2 is out of bounds for axis 0 with size 2
[ ERROR ]
[ ERROR ]  It can happen due to bug in custom shape infer function <function RegionYoloOp.regionyolo_infer at 0x167530dd0>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "detector/yolo-v4-tiny/Tile/YoloRegion" node.
 For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)

ENVIROMENT
macos, python3.7, tensorflow1.15.5, OpenVINO2021.4

@osivinyuk or @LouisDong1995 did you resolve your error? I am seeing the exact same error for yolov4 tiny model and wondering if you found a solution, thanks so much for your reply in advance

@LouisDong1995 I have sold it, but need to remember how to do it)


The One Probe integration. Improve WAILA integration.

WAILA integration and The One Probe integration are 1:1; what was already shown in the WAILA tooltip is shown in The One Probe tooltip.
Remove Bed WAILA integration and replace it with properly overriding getPickBlock in BlockBedGeneric.
Here is an example of what it looks like:
Open and empty basin:

Closed and full basin:

Potential improvements I can think of:

Improve tooltip formatting by adding formatting codes to the localizations, rather than the code. For example, toolTip.add(String.format("%s : %d%%", LangHelper.translate("waila.composter.composting"), (int) (tag.getFloat("Compost") * 100f))); would become toolTip.add(LangHelper.translate("waila.composter.composting", (int) (tag.getFloat("Compost") * 100f)); and likewise for the The One Probe equivalent code.
Centralize the code for generating the WAILA/The One Probe tooltips so that the code is in one location rather than duplicated in two. I'm not certain how we could do this effectively since the way WAILA and The One Probe work are entirely different (specifically, TOP uses chained calls, while WAILA uses a list).

I generally try to avoid adding full mods to the dev environment whenever possible (to better test that they are optional and to keep the default environment lean), but it looks like that's not really possible with TOP, unfortunately. Perhaps we could request API-only packages be added to the TOP maven.

I've moved DEGREE_SYMBOL so that the TOP integration is not dependent on the WAILA integration classes.
I agree with the API package stuff. Are we going to wait until McJty/TheOneProbe#66 is resolved to merge this, or simply update the buildscript once that issue gets resolved?

Looks good, we can merge it now.

What do you think of the two potential improvements I listed above?

I think this change would make sense, as it'd also allow for the % sign placement to be based on localization
This one might be more difficult. I was thinking we could add something to the Block classes like getContextStrings that would return a List<String> and then the TOP methods could just loop through that and chain text calls, but Waila depends on NBT syncing that might make that method not quite work without special handling (for example).


Not able to train HAN because of the following error.

sentence_input = Input(shape=(MAX_SENT_LENGTH,), dtype='int32')
embedded_sequences = embedding_layer(sentence_input)
l_lstm = Bidirectional(GRU(100, return_sequences=True))(embedded_sequences)
l_att = AttLayer(100)(l_lstm)
sentEncoder = Model(sentence_input, l_att)
review_input = Input(shape=(MAX_SENTS, MAX_SENT_LENGTH), dtype='int32')
review_encoder = TimeDistributed(sentEncoder)(review_input)
l_lstm_sent = Bidirectional(GRU(100, return_sequences=True))(review_encoder)
l_att_sent = AttLayer(100)(l_lstm_sent)
preds = Dense(2, activation='softmax')(l_att_sent)
model = Model(review_input, preds)
model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['acc'])
print("model fitting - Hierachical attention network")
Error
ValueError: Dimensions must be equal, but are 15 and 100 for 'att_layer_10/mul' (op: 'Mul') with input shapes: [?,15], [?,15,100].

I found that in AttLayer change code from
"    def compute_mask(self, inputs, mask=None):
return mask
"
to
def compute_mask(self, inputs, mask=None):
    return None

it will be ok. It means that should not support mask.

Just after making that change, I encountered this following error.
ValueError: Dimensions must be equal, but are 100 and 1 for 'att_layer_20/MatMul_1' (op: 'MatMul') with input shapes: [?,100], [1,100].

Solved the issue.
Thanks @cjopengler .

I have the same problem and changing the code as @cjopengler says works.

Solved the issue.
Thanks @cjopengler

it is because of the function timedistributed
previous_mask
<tf.Tensor 'time_distributed_14/Reshape_2:0' shape=(?, 15, 100) dtype=bool>
the methods of resolution can be refer to https://blog.csdn.net/songbinxu/article/details/80242211


Music doesn't work in Safari for HTML5

Safari seems to want to use MP3s, which it really really shouldn't. Will see if I can figure out something else.

This works now (requires OGG)


feat: expose rackID option in ingester samara client

Signed-off-by: shaimoria<EMAIL_ADDRESS>Which problem is this PR solving?
It is common to have a Kafka cluster spanning multiple datacenters. For example, a common deployment is within an AWS region in which each availability zone is treated as a datacenter. In order to reduce cross region/rack network cost, we can tell the consumer which rack he's located on. this way the consumer will fetch from partitions that are located on he's region/rack
Short description of the changes

expose kafka.consumer.rack-id cli argument in order to ingester consumers to fetch from closest replica

@yurishkuro add explanation for the use case


Not able to connect from outside, no response at all

Hey all, after having a kind of messy Wireguard UI install (only a certain profile would handshake and the rest would return no response), I decided to delete my db and do a fresh install.
My host OS is DietPi, running on a Raspberry Pi 4. I am forwarding my ports with a cron script that runs upnpc every 20 minutes for 51820 UDP in the case of WireGuard, and it seems to do that just fine judging from my previous connections.
The issue is, after logging in for the first time and creating a new user, when scanning the config QR code from my phone and then trying to start the VPN, I just get 0 rx and plenty of tx (from my phone attempting to connect). I doubt it is an issue with my "port forwarding" method, since other services are also being done like this and they work just fine, but at the same time I haven't modified much from the stock config and I don't really know what the issue could be. sudo modprobe wireguard returns no output, which is good since WireGuard is installed. I have already disabled the wg-quick service as well. I'd appreciate any hints or help at trying to debug this issue.
My stack looks like this (loaded through Portainer):
version: "3"

services:
 wireguard-ui:
   image: ngoduykhanh/wireguard-ui:latest
   container_name: wireguard-ui
   cap_add:
     - NET_ADMIN
   # required to show active clients. with this set, you don't need to expose the ui port (5000) anymore
   network_mode: host
   environment:
     - BIND_ADDRESS=<IP_ADDRESS>:${WEBPORT}
     - SENDGRID_API_KEY
     - EMAIL_FROM_ADDRESS
     - EMAIL_FROM_NAME
     - SESSION_SECRET
     - WGUI_USERNAME=$(WGUI_USERNAME)
     - WGUI_PASSWORD=${WGUI_PASSWORD}
     - WG_CONF_TEMPLATE
     - WGUI_MANAGE_START=true
     - WGUI_MANAGE_RESTART=true
   logging:
     driver: json-file
     options:
       max-size: 50m
   volumes:
     - db:/app/db
     - /etc/wireguard:/etc/wireguard
   restart: unless-stopped
volumes:
 db:```

OK, skip the docer, run "./wireguard-ui" and paste the logs from client. BTW: uPNP should be avoided if possible, but that's probably not the worst choice.
Prepare to run tcpdump on the incomming interface on WG host.