Broker feed stuck with compression

I am using the bitfinex. The exchange returns a feed of one minute. My bot uses a time-frame of 30 minutes, but it stuck with a compression different of 1.
Code:
hist_start_date = dt.datetime.utcnow() - dt.timedelta(minutes=50)
data = store.getdata(dataname='BTC/USDT', name="BTCUSDT",
	timeframe=bt.TimeFrame.Minutes, fromdate=hist_start_date,
	compression=30, ohlcv_limit=999, drop_newest=True)

I also tried to use cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=30) with compression 1 in store. But does not work too.
With compression=1, the feed works fine. And next() function is triggered.

I'm linux user, I'm using now the default terminal.
I activate the debug and notice that the request is sent but the feed not trigger next() function. I also discovered that a hist_start_date value, fix it, with a timedelta of 2000 minutes, works fine. This is very strange.
The full code is:
broker_config = {
	'apiKey': config.get("api").get("key"),
	'secret': config.get("api").get("secret"),
	'nonce': lambda: str(int(time.time() * 1000)),
	'enableRateLimit': True,
}

store = CCXTStore(exchange='binance', currency='USDT', config=broker_config,
	retries=5, debug=True)

broker_mapping = {
	'order_types': {
		bt.Order.Market: 'market',
		bt.Order.Limit: 'limit',
		bt.Order.Stop: 'stop-loss',
		bt.Order.StopLimit: 'stop limit'
	},
	'mappings': {
		'closed_order': {
			'key': 'status',
			'value': 'closed'
		},
		'canceled_order': {
			'key': 'result',
			'value': 1
		}
	}
}

broker = store.getbroker(broker_mapping=broker_mapping)
cerebro.setbroker(broker)

hist_start_date = dt.datetime.utcnow() - dt.timedelta(minutes=50)
data = store.getdata(dataname='BTC/USDT', name="BTCUSDT",
	timeframe=bt.TimeFrame.Minutes, fromdate=hist_start_date,
	compression=30, ohlcv_limit=9999, drop_newest=True)

cerebro.adddata(data)

@rodrigo-brito
Can you send a full sample please? - I.e Including the strategy. If this is sensitive, please replicate it with a simple strategy that just prints something basic on the call of next()
I want to make sure I am using the exact same code that you see an issue with.

Hi @Dave-Vallance,
A simplified version, with the reproduced issue is in the follow link: https://gist.github.com/rodrigo-brito/3b0fca2487c92ad97869247edd5fd852
Notice that  dt.timedelta(minutes=2000) is 2000. If you change it to 30 for example, it stop to work.

Hi @rodrigo-brito,
I tried to reproduce your problem with your gist but without success. It works for me.
I changed your code slightly but that shouldn't have had an effect: https://gist.github.com/sandroboehme/1815de278498245f9fa046138e1767f2
It works for me with a 2000 min delta as well as with a 30 min start delta.

Hi @rodrigo-brito
Did you try @sandroboehme's gist? Is that working for you too?

Hi @Dave-Vallance  and @sandroboehme, thank you for the support. I'm suspecting that it is related to my base strategy. The initial time is multiplied by the minimum time window required. I will try the gist later, maybe tomorrow, and report it back to you.

Hi @rodrigo-brito
Ok - No rush!
I was just wondering whether to keep it open or close the issue.
Cheers

Sorry for the delay, i tested you code with differente time frames and work very well. I don't now what is happening with my code. Maybe a specific indicator or other component is interfering on it.

Hello, in regards to this issue, I can also verify that your code indeed returns data after 30 min. However, when comparing the candle's low, high, close, and open with the same candle's attributes in Tradingview or Binance they don't match. And if the candle is not the same of course an indicator will return false results too.
Is the code too outdated?

Hi @rodrigo-brito,
I tried to reproduce your problem with your gist but without success. It works for me. I changed your code slightly but that shouldn't have had an effect: https://gist.github.com/sandroboehme/1815de278498245f9fa046138e1767f2 It works for me with a 2000 min delta as well as with a 30 min start delta.


pluginst/blink: add blink.nvim

Hi there, this is my first contribution to a Nix related project (besides some flake configs).  I try to add this plugin: https://github.com/Saghen/blink.cmp which wants to be an alternative to nvim-cmp.
I'd like to hear any feedback and tips for improvements. Also how can I actually test this locally? Many thanks!

Also how can I actually test this locally?

Firstly, you should write a unit test in the style of other tests under tests/test-sources/plugnis/by-name. Once done, you'll be able to run tests blink or tests -i (to search interactively).
Secondly, you can take advantage of nix flake lock on an external nixvim config repo (such as your nix config):
# Override to your locally cloned nixvim
# You'll have to re-run this if ../nixvim is modified
nix flake lock --override-input nixvim ../nixvim

# Override to a remote repo
nix flake lock --override-input nixvim github:Balssh/nixvim/add_blink

# Undo un-committed changes to your lockfile
git restore flake.lock

Also how can I actually test this locally?

Firstly, you should write a unit test in the style of other tests under tests/test-sources/plugnis/by-name. Once done, you'll be able to run tests blink or tests -i (to search interactively).
To access the tests command, enter a devshell either using direnv-nix or the command: nix develop
Secondly, you can take advantage of nix flake lock on an external nixvim config repo (such as your nix config):
# Override to your locally cloned nixvim
# You'll have to re-run this if ../nixvim is modified
nix flake lock --override-input nixvim ../nixvim

# Override to a remote repo
nix flake lock --override-input nixvim github:Balssh/nixvim/add_blink

# Undo un-committed changes to your lockfile
git restore flake.lock

Thanks! Will do this along with all other requests in the next commits.

Will continue working on this after https://github.com/NixOS/nixpkgs/pull/348105 is merged

@Balssh if you're stuck on anything just shout. IDK your timezone (I'm GMT), but I don't mind arranging a call or chatting on Matrix if there's anything you want to go over or need help with.

Thanks for your offering, right now I'm testing stuff on how to declare settingsOptions. Sorry if the multiple pushes are bothering, I can resolve to using local nixvim fork.
I was stuck because I didn't realise I had to provide description string at the end of a setting declaration. Got it now and hopefully the remaining settings I want to define will go smoothly. At last will look into implementing tests.

@MattSturgeon Ok, now I think I'm in a happy place with the plugin, I'm a bit unsure what tests to write for this. I looked into the nvim-cmp tests examples but it's not very clear what it's being tested. For example, the I could write a
keymap-test = {
    plugins.blink-cmp = {
        enable = true;
        settings.keymap = {
            show = "<C-space>";
            hide = "<C-e>";
            accept = "<C-y>";
            select_prev = "<C-p>";
            select_next = "<C-n>";
            show_documentation = "<C-space>";
            hide_documentation = "<C-space>";
            scroll_documentation_up = "<C-b>";
            scroll_documentation_down = "<C-f>";
            snippet_forward = "<Tab>";
            snippet_backward = "<S-Tab>";
          };
    };
};

But other than making sure settings.keymap exist and the format is good, is anything else tested? Thanks

Most of our tests are just checking that a) the nixvim configuration evaluates without any errors or warnings and b) neovim can run the resulting init.lua without printing any warnings to the console.
It is possible to add additional assertions to the nixvim configuration, or add lua assertions to extraConfigLua, however almost none of our tests do anything that specific.
The example you suggested would be a great test case. We'd also ideally like an "empty" test (just setting enable = true for your plugin) and a "defaults" test (setting each settings option explicitly to the upstream plugin's default value).
The nvim-cmp tests probably aren't the best tests to be looking at, though. IIRC they are fairly complex compared to most other "normal" plugin's tests.

Fair enough, will write them tomorrow probably, thanks!

Hmm I'm having some trouble with tests:
[devshell]$ tests blink-cmp
Running 1 tests: blink-cmp
error: attribute 'null' in selection path 'checks.x86_64-linux.null' not found
Test failure

$ tests blink-cmp

My mistake, it's probably something more like plugins-by-name-blink-cmp-default.
You can use tests -i to filter through the tests interactively.

Yup, tests -i works, but something weird happens as it seems to fail while trying to download prebuild binaries:
Running 1 tests: plugins-by-name-blink-cmp-default
warning: Git tree '/home/balssh/Projects/nixvim' is dirty
copying '/home/balssh/Projects/nixvim/' to the store
error: builder for '/nix/store/qdax56jw3y0ykh3l0cz1pn2x63ir34vj-empty.drv' failed with exit code 1;
       last 10 log lines:
       > ERROR: Error detected while processing /nix/store/x8lf69b2618vkfl6m1gcp3p0lzk48xlv-init.lua:
       > E5113: Error while calling lua chunk: ...-unwrapped-0.10.2/share/nvim/runtime/lua/vim/_system.lua:244: ENOENT: no such file or directory
       > stack traceback:
       >        [C]: in function 'error'
       >       ...-unwrapped-0.10.2/share/nvim/runtime/lua/vim/_system.lua:244: in function 'spawn'
       >   ...-unwrapped-0.10.2/share/nvim/runtime/lua/vim/_system.lua:335: in function 'system'
       >  ...ackages/start/blink-cmp/lua/blink/cmp/fuzzy/download.lua:76: in function 'get_git_tag'
       >      ...ackages/start/blink-cmp/lua/blink/cmp/fuzzy/download.lua:24: in function 'ensure_downloaded'
       >        .../myNeovimPackages/start/blink-cmp/lua/blink/cmp/init.lua:8: in function 'setup'
       >     /nix/store/x8lf69b2618vkfl6m1gcp3p0lzk48xlv-init.lua:9: in main chunk
       For full logs, run 'nix log /nix/store/qdax56jw3y0ykh3l0cz1pn2x63ir34vj-empty.drv'.
error: 1 dependencies of derivation '/nix/store/r04vm7n27zdy3lj0nm3qnsd88w2bf205-plugins-by-name-blink-cmp-default.drv' failed to build
Test failure

Now, this is a bit weird as I can run the plugin just fine in my nixvim config, and also because I have set the option https://github.com/Saghen/blink.cmp/blob/465dc89dc63bef41fa97a133a1b4f51400048829/lua/blink/cmp/config.lua#L249 to false in my latest test: https://github.com/nix-community/nixvim/pull/2404/files#diff-8552feb0d654595acc17cba194c95cc42f5330bc99d3b9a2fbd3d48c75ac18e4R42. It's interesting because I didn't set that download=false in the plugin's config I'm using.
I'm not sure if the problem stems from the plugin's init flow, or from nixvim's. Is there a way to check the nixvim config's while running a certain test?

[the test] seems to fail while trying to download prebuild binaries:

This is due to the nature of how the tests are implemented.
A nix test is usually a derivation; if the derivation builds successfully the test is said to have passed. If the derivation fails to build, then the test has failed.
For our tests, this means we are running nvim within a derivation buildscript.
Derivation buildscripts are run in a sandboxed environment, to ensure that derivations stay "pure" and reproducible. This is a good thing, even for tests, as it means every time you run the test (or build any derivation) you should get the same output.
But in this case, it means you can't download things within the test.
There's a few solutions:

In the test module, you could set things up using source fetchers, so that we don't need to download at "runtime"
You could disable the runtime component of the test, so we only test the nixvim module eval.

Unless you know a good way to do option 1, option 2 is fine. This is done by setting the option test.runNvim = false in your test module(s).

If there's a way to disable downloading things, that would be the best solution.

Well it looks like it is, but can I inspect/list what nixvim's config is set to for the plugin I'm testing? I've set the downloading to false in the test I've written, but either it doesn't get picked up by nixvim, I declared it wrongly or there is a bug in the plugin itself.
For now I've set test.runNvim = false;. I think my PR is ready for review now.

Just missed a few things, if we're going to offer the options in settingsOptions we want to make sure we're testing them in the test module.

Sorry, hurried and messed them up :face_holding_back_tears: Should be defined now.

Thanks for having patience with me!

@mergify queue


feat: add rollback sinker

@pedrotambo the postgresql endpoint supports the reorgs natively now. Can we close this PR ?


[Bug]: Getting non-accessible symbol error for error intersections with details

Description
I am getting the following error when I try to use error types from a external ballerina package. The error types are intersections with detail records:
ERROR [main.bal:(8:16,8:171)] attempt to refer to non-accessible symbol '$anonIntersectionErrorType$_2'
ERROR [main.bal:(10:16,10:170)] attempt to refer to non-accessible symbol '$anonIntersectionErrorType$_3'
error: compilation contains errors

Please note that this compiler errors are only shown at the very first build, after that the compilation is successful.
Steps to Reproduce

Create a ballerina library package:

$ bal new -t lib status-code-errors

Add the following errors.bal file to the package:

public type Detail record {
    int statusCode;
    map<string[]> headers;
    anydata body;
};

public type StatusCodeBindingErrorDetail record {
    *Detail;
    boolean fromDefaultStatusCodeMapping;
};

public type Error distinct error;

public type ClientError distinct Error;

public type ApplicationResponseError distinct (ClientError & error<Detail>);

public type ClientRequestError distinct (ApplicationResponseError & error<Detail>);

public type RemoteServerError distinct (ApplicationResponseError & error<Detail>);

public type StatusCodeResponseBindingError distinct ClientError & error<StatusCodeBindingErrorDetail>;

public type StatusCodeBindingClientRequestError distinct StatusCodeResponseBindingError & ClientRequestError;

public type StatusCodeBindingRemoteServerError distinct StatusCodeResponseBindingError & RemoteServerError;

Pack and push it to Ballerina local central:

$ bal pack; bal push --repository=local

Create a new Ballerina package:

$ bal new sample

Add the following local dependency to Ballerina.toml:

[[dependency]]
org = "tharmigan" # Change the org to your org
name = "status_code_errors"
version = "0.1.0"
repository = "local"

Replace the main.bal file content with the following:

// Replace the package organization with your organization
import tharmigan/status_code_errors as http;

isolated function createStatusCodeResponseBindingError(int statusCode, map<string[]> headers, anydata body = ()) returns http:StatusCodeResponseBindingError {
    string reasonPhrase = string `incompatible type found for the response with non-default status code: ${statusCode}`;
    if 100 <= statusCode && statusCode <= 399 {
        return error http:StatusCodeResponseBindingError(reasonPhrase, statusCode = statusCode, headers = headers, body = body, fromDefaultStatusCodeMapping = false);
    } else if 400 <= statusCode && statusCode <= 499 {
        return error http:StatusCodeBindingClientRequestError(reasonPhrase, statusCode = statusCode, headers = headers, body = body, fromDefaultStatusCodeMapping = false);
    } else {
        return error http:StatusCodeBindingRemoteServerError(reasonPhrase, statusCode = statusCode, headers = headers, body = body, fromDefaultStatusCodeMapping = false);
    }
}

Run the build for the first time:

$ bal build

Compiling source
        tharmigan/sample:0.1.0
ERROR [main.bal:(8:16,8:171)] attempt to refer to non-accessible symbol '$anonIntersectionErrorType$_4'
ERROR [main.bal:(10:16,10:170)] attempt to refer to non-accessible symbol '$anonIntersectionErrorType$_5'
error: compilation contains errors

Run the build for the second time:

$ bal build

Compiling source
        tharmigan/sample:0.1.0

Generating executable
        target/bin/sample.jar

Affected Version(s)
Ballerina SwanLake Update 9(2201.9.0)
OS, DB, other environment details and versions
No response
Related area
-> Compilation
Related issue(s) (optional)
No response
Suggested label(s) (optional)
No response
Suggested assignee(s) (optional)
No response

We are not getting the error in the 2nd run of bal build because we get the error symbol accessible via the BIR correctly.


how to set  chrome option automation extension false

Loading of unpacked extensions is disabled by the administrator
Getting an error. "Failed to load the extension from'
c:......temp\coped...dir20280_216\internal.Loading of unpacked extensions is disabled.
by the administrator
How to handle this in karate?
Used below code  to  handle the above problem, But  still getting the error window

configure driver={type:'chromedriver',executable:'C:\chrome.exe',port:9515,addOptions:['--disable-extensions','--disable-infobar'],useAutomationExtension:false}

@maruthivip sorry can't replicate. also waiting for you to respond here: https://github.com/intuit/karate/issues/1133#issuecomment-626486659

How to set useAutomationExtension:false. in karate

Please do not mark as invalid ,just. because you cannot replicate

@maruthivip sorry, that's the process we follow here: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue

how to set  the option  AutomationExtension:false

@maruthivip I honestly have no idea

How do you do this in karate... is my question

@maruthivip I have even less of an idea

how to set  below  settgin in karate
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("useAutomationExtension", false);
WebDriver driver = new ChromeDriver(options);
I tried  in config driver,which dint work

@maruthivip if this answer doesn't help, then please assume this is not supported and stop posting here, thanks: https://stackoverflow.com/a/61907860/143475

Definitely i can assume that it's not supported.
Ideally, you should call it as not supported, rather than asking end-users
to assume
Better you document this point in the limitations of the tool
On Thu, May 21, 2020 at 12:25 PM Peter Thomas<EMAIL_ADDRESS>wrote:

@maruthivip https://github.com/maruthivip if this answer doesn't help,
then please assume this is not supported and stop posting here, thanks:
https://stackoverflow.com/a/61907860/143475
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/intuit/karate/issues/1134#issuecomment-631918909, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/APQ5NY23SHD4L773A37R4DLRSTF5DANCNFSM4M7OSXDQ
.

@maruthivip we will give your comment the attention that it deserves. have a good day.

After much trial and error, I've found out how to add the useAutomationExtensions setting.  You need to wrap it in the desiredCapabilities object like this:
{ desiredCapabilities: { "goog:chromeOptions": { <here> } } }
Full example here:
Feature: useAutomationExtension test
Scenario: try useAutomationExtension with Chrome
    * def startUrl = "https://www.google.com/"
    * def browser = "chrome"
    * def type = "chromedriver"
    * def executable = "<driver path>" + type + ".exe"
    * def driverConfig = { type: #(type), showDriverLog: true, addOptions: [ "--window-size=1600,900", "--disable-extensions" ], executable: #(executable), webDriverSession: { desiredCapabilities: { browserName: #(browser), "goog:chromeOptions": { useAutomationExtension: false } } } }
    * configure driver = driverConfig
    Given driver startUrl
    Then match driver.url == startUrl

I appreciate your response.In fact we have already found the solution and
due to NDS, we could not able to share that info here.
How ever,I really thank you for following up and getting this ticket to
closure.
Though my initial experience with the too was not that great, day by
day..... I am getting comfortable!!!!
On Thu, Jun 4, 2020 at 10:45 PM Kelly Smith<EMAIL_ADDRESS>wrote:

After much trial and error, I've found out how to add the
useAutomationExtensions setting. You need to wrap it in the
desiredCapabilities object like this:
{ desiredCapabilities: { "goog:chromeOptions": {  } } }
Full example here:
Feature: useAutomationExtension test
Scenario: try useAutomationExtension with Chrome
* def startUrl = "https://www.google.com/"
* def browser = "chrome"
* def type = "chromedriver"
* def executable = "" + type + ".exe"
* def driverConfig = { type: #(type), showDriverLog: true, addOptions: [ "--window-size=1600,900", "--disable-extensions" ], executable: #(executable), webDriverSession: { desiredCapabilities: { browserName: #(browser), "goog:chromeOptions": { useAutomationExtension: false } } } }
* configure driver = driverConfig
Given driver startUrl
Then match driver.url == startUrl
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/intuit/karate/issues/1134#issuecomment-638990087, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/APQ5NY5ZUFQTYEWVBDY2G23RU7JEVANCNFSM4M7OSXDQ
.

I am also under and NDA, but you can always make a generic action like I did to present your findings.  It helps everyone else if you take a bit of time to do it.

Sure, Kelly.Will try to make it next time...In fact we are under tremendous
pressure in evaluating multiple tools.
My current impression over karate is  it has got a unique feature of
Friendly/wild locators which helps  for easy object identification
I am currently struggling  to  get an snapshot  when an error occurs.In
selenium I have freedom of writing my own code.However in Karate,I am
unable to figure out the solution using after scenario hook
On Thu, Jun 4, 2020 at 11:44 PM Kelly Smith<EMAIL_ADDRESS>wrote:

I am also under and NDA, but you can always make a generic action like I
did to present your findings. It helps everyone else if you take a bit of
time to do it.
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/intuit/karate/issues/1134#issuecomment-639020391, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/APQ5NY6XTTHI5AQ3KJAT4JLRU7QA5ANCNFSM4M7OSXDQ
.

@maruthivip just FYI - I requested Kelly to comment here because I saw he may have a solution to your previous question. as Kelly said above, it is quite disappointing that you didn't post the solution here.

we are under tremendous pressure in evaluating multiple tools.

please don't try to pass on your pressure to us. this is an open-source project that depends on community contributions to a large extent
a simple google search for karate afterScenario screenshot gives you this: https://stackoverflow.com/a/59036621/143475
I suspect that you are more comfortable with Java and the so-called "freedom" to write your own code. in my experience Karate does not work well for such teams because they will always be finding excuses to go back to Java


Bugtool for dumping IPv6 routes is broken

Found in https://github.com/cilium/cilium/issues/17895:
$ cat bugtool-cilium-h5t8f/cmd/ip--6-route-show-\[116-97-98-108-101-32-50-48-48*
Error: inet6 prefix is expected rather than "[116".
> Error while running 'ip -6 route show [116 97 98 108 101 32 50 48 48 52]':  exit status 1

Error: inet6 prefix is expected rather than "[116".
> Error while running 'ip -6 route show [116 97 98 108 101 32 50 48 48 53]':  exit status 1

Error: inet6 prefix is expected rather than "[116".
> Error while running 'ip -6 route show [116 97 98 108 101 32 50 48 48]':  exit status 1

These lines should not be using the %v formatter but should instead format the IPv6 addresses in a standard format that ip -6 route ... can understand:
https://github.com/cilium/cilium/blob/789eef0d148d0aa590d2a1b2da7fd69da6dafc47/bugtool/cmd/configuration.go#L262
Tasks:

[ ] Fix IPv6 route debug gathering commands
[ ] Grep around the bugtool for other %v formatting primitives to check that there are no other similar problems for other commands

I'd like to work on it
/assign

@joestringer can you please assign the issue to me :)

This issue is caused by #17546, namely commit 8bcc4e5dd830.
I've also noticed this issue when trying to debug something locally and came up with #18059 which fixes this issue and another instance where that commit's change from string to []byte potentially causes problem.


Denormalizing a list doesn't work the same as in the README

The README has:
// Denormalize a list
const denormalized = denormalize([article], normalized.entities, articleListSchema);
console.log(denormalized);
// [{
//   id: 1,
//   title: 'Some Article',
//   author: {
//     id: 1,
//     name: 'Dan'
//   },
// }]

But it doesn't actually return an array, and the information isn't nested.  Instead it simply gives a key to the author.  This is what I get using the code as given in the example in the README:
const denormalized = denormalize([article], normalized.entities, articleListSchema);
console.log(denormalized);
{0: {
       author: 1,
       id: 1,
       title: "Some Article"
}}

I'm not sure how to actually get an array back with the author nested.  Any help would be greatly appreciated.  Thanks!

Hi @krisfields, I'm sorry the README is not so clear (something similar is also open in #14).
I can't help you directly right now (I need to reproduce the case first). Maybe you can dig into this test and understand where your code is different?

Is it that it's not clear or that it's inaccurate?
I dug into that test and was not able to see how mine differed at all, other than in the expected result.  In fact, I copied/pasted what I believe is the relevant code from that test and still get a different result.  Here's what I tried:
import { normalize, Schema, arrayOf } from 'normalizr'

const articleSchema = new Schema('articles')
const userSchema = new Schema('users')
const collectionSchema = new Schema('collections')

articleSchema.define({
  author: userSchema,
  collections: arrayOf(collectionSchema),
})

collectionSchema.define({
  curator: userSchema,
})

const article1 = {
      id: 1,
      title: 'Some Article',
      author: {
        id: 1,
        name: 'Dan',
      },
      collections: [{
        id: 1,
        name: 'Dan',
      }, {
        id: 2,
        name: 'Giampaolo',
      }],
    };
    const article2 = {
      id: 2,
      title: 'Other Article',
      author: {
        id: 1,
        name: 'Dan',
      },
    };
    const article3 = {
      id: 3,
      title: 'Without author',
      author: null,
    };

    const article4 = {
      id: 4,
      title: 'Some Article',
      author: {
        id: '',
        name: 'Deleted',
      },
      collections: [{
        id: '',
        name: 'Deleted',
      }],
    };

const response = {
  articles: [article1, article2, article3, article4],
}

const data = normalize(response, {
  articles: arrayOf(articleSchema),
})

const articles = [
  data.entities.articles['1'],
  data.entities.articles['2'],
]

const denormalized = denormalize(articles, data.entities, arrayOf(articleSchema))
console.log("articles denormalized = ", denormalized)

And the output was not an Array and was not nested, but instead simply had the keys for each object that should have been nested, like before when copying from the README.
articles denormalized = Object {0: Object, 1: Object}
{0: {
  author: 1,
  collections: [{0:1}, {1:2}],
  id: 1,
  title: "Some Article"
},
 1: {
  author: 1,
  id: 2,
  title: "Other Article"
}}

So I'm really curious and confused as to where I may have went wrong.

@krisfields I'm curious as well.
Could you please check you are using the last version of denormalizr?
npm ls denormalizr

I've setup a small repo to test this case: https://github.com/gpbl/denormalizr-test
git clone https://github.com/gpbl/denormalizr-test.git
cd denormalizr-test
npm install
npm start

it logs:
Normalized data
{ entities:
   { articles:
      { '1': { id: 1, title: 'Some Article', author: 1, collections: [ 1, 2 ] },
        '2': { id: 2, title: 'Other Article', author: 1 } },
     users: { '1': { id: 1, name: 'Dan' } },
     collections:
      { '1': { id: 1, name: 'Dan' },
        '2': { id: 2, name: 'Giampaolo' } } },
  result: { articles: [ 1, 2 ] } }
Denormalized:
[ { id: 1,
    title: 'Some Article',
    author: { id: 1, name: 'Dan' },
    collections: [ { id: 1, name: 'Dan' }, { id: 2, name: 'Giampaolo' } ] },
  { id: 2, title: 'Other Article', author: { id: 1, name: 'Dan' } } ]

So it seems all correct to me (macOS, node v4.4.5).

Well, it works correctly for me when I run your test repo, but when I copy/paste the code from that repo's index.js file into my project, it no longer works correctly.  The output is no longer an array, and no longer nested.  I currently have no idea why but haven't had a lot of time to dig in yet.  If you have any suggestions on what I might look into to determine what's going on, any help would be much appreciated.
mac, node v6.6.0

Ok, I guess I wasn't using the latest version of denormalizr.  "npm upgrade normalizr --save" didn't actually upgrade it, which is what confused me.   Now that I am using the latest version, everything works properly.

Awesome! Yes npm update won't work for minor release < 1.0. I use npm-check for updating dependencies. It is safer :)


DNS query crash when connecting to ES Cloud

I am not sure if it's related to ES Cloud only (it shouldn't) but I have a report of an exception:

Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.10/site-packages/esdbclient/client.py", line 204, in _connect_to_preferred_node
answers = dns.resolver.resolve(cluster_fqdn, "A")
File "/home/ubuntu/.local/lib/python3.10/site-packages/dns/resolver.py", line 1565, in resolve
return get_default_resolver().resolve(
File "/home/ubuntu/.local/lib/python3.10/site-packages/dns/resolver.py", line 1307, in resolve
(request, answer) = resolution.next_request()
File "/home/ubuntu/.local/lib/python3.10/site-packages/dns/resolver.py", line 749, in next_request
raise NXDOMAIN(qnames=self.qnames_to_try, responses=self.nxdomain_responses)
dns.resolver.NXDOMAIN: The DNS query name does not exist: clt07f5o0aesbufe1n6g.mesdb.eventstore.cloud:2113.

I checked the line 204 and it calls answers = dns.resolver.resolve(cluster_fqdn, "A"). From the error message I understand that cluster_fqdn also contains the port, which is apparently wrong.
The code that calls the client looks totally fine, so I am not sure what went wrong:
client = EventStoreDBClient(
    uri="esdb+discover://admin:something@<cloud-cluster-id>.mesdb.eventstore.cloud:2113"
)

We tried to use esdb://<cloud-cluster-id>-0.mesdb.eventstore.cloud:2113 as I had a guess that esdb+discover might be an issue, but it crashed with the same error. It tried to resolve <cloud-cluster-id>-0.mesdb.eventstore.cloud:2113, again, including the port in DNS query.

I think what I wrote is wrong, it should work with esdb://. Basically, as I can see here:
            cluster_fqdn = self.connection_spec.targets[0]
            try:
                answers = dns.resolver.resolve(cluster_fqdn, "A")
            except dns.exception.DNSException as e:
                raise DNSError() from e
            gossip_seed: Sequence[str] = [f"{s.address}:2112" for s in answers]

there are several issues.

The targets list contain the host name and the port, so calling resolve with host:port crashes
Using DNS for getting the gossip response from a cluster DNS is not right. Instead, you should use the client gossip gRPC call using the cluster DNS as an endpoint, and use addresses returned instead of creating a gossip seed using IP addresses and hard-coded 2112 port. It's not guaranteed that IP addresses will work (certs CN is the hostname), and client advertise as address doesn't necessarily matches the DNS response.

Basically, there should be no calls to resolve DNS names to IPs in the client. Instead, there need to be a call to this service: https://github.com/pyeventsourcing/esdbclient/blob/54e768e26ed7915b7e7d7b7648c0c1fbfb5b9984/protos/esdbclient/protos/Grpc/gossip.proto#L7-L9
The result returned from there should be used as-is. Of course, the client would need to choose the right node to connect to, using the node preference from the connection string, and ensuring the node isn't dead. The NodeJS client Is a good example: https://github.com/EventStore/EventStore-Client-NodeJS/blob/91fb2aac898ea2c6dce072f94393466390ccf89c/src/Client/discovery.ts

Thanks for raising this @alexeyzimarev.
I'm not saying you are wrong and I'm not asserting the client is right. But what you describe here is a totally different approach to what this client has been coded to do. I have been carefully following what Yves and others told me to do with the "esdb+discover" scheme. And this has been reviewed by several people.
It's been clearly documented: that (1) The "esdb+discover" scheme has a "cluster domain name" and does not have a port number; that (2) that this "cluster domain name" is used to get A records from a DNS server, and the hosts mentioned in these A recorded used as "grpc targets" with a hard coded port number of 2113; that (3) "cluster info" is obtained from the "grpc targets" and then the client may reconnect according to the node preference (if the node preference is already satisfied then it won't reconnect).
I am of course more than happy to review and revise. I don't fully understand what you have said above, the issues of connecting to a cluster, connecting to the cloud service, and using the two schemes are a bit mixed together, and altogether I'm a bit confused about what you are saying. Let me read it all through again and see if I can make more sense of it?

I am sorry to say that those guidelines that you received aren't correct today.
First, the connection string used in the sample I provided is correct, and it works in any other client. There's no requirement that when using cluster DNS the port must be 2113, and it cannot be provided in the URI. So, that connection string should work, it's completely legit.
Second, the procedure to discover the node for the client to connect to should work like this (it's relatively new):

Use the hostname and port part of the URI of esd+discover schema
Create an instance of the GossipService generated from gossip.proto. Use the URI from the previous step.
Make a call to that service's Read function and get the response back
Use the list of retuned nodes as the list of nodes (gossip seed if you like)
In that list, every node has the http_end_point property (among other helpful things like node role and liveness), which must be used to populate the "gossip seed". That property includes hostname and port unconditionally, so there's no guesswork.

The rest of the discovery would work as far as I was able to understand.

@alexeyzimarev I have made the superficial change of allowing the esdb+discover scheme to have an optional port-number after the cluster-domainname, and this is not used when making the DNS call, so that the DNS call isn't broken by including a port number in the URI, and then also this port number is used when constructing grpc targets from the A records, and if the URI does not mention a port number the default port number of 2113 is used. I noticed that the code actually had 2112 as the default, for some reason. It was documented as 2113 but the code was actually using 2112, which was a mistake. These changes are in v1.0.13.

I know that isn't what you asked for, I just thought I would throw that in with the other changes in v1.0.13 because it was an easy improvement.
Regarding the intentions for the esdb+discover scheme, I really don't understand the difference between esdb+discover and esdb. The esdb scheme already uses the list of grpc targets to get the gossip and then select a node according to the cluster info and the node preference. The redundancy in having a list of targets seems to increase the changes of reaching the cluster. If the esdb+discover scheme only mentions one node, that node might not be available. But DNS servers are usually quite reliable, probably more reliable than an EventStoreDB server, since they have only one job, and you can configure alternatives etc. And decoupling the cluster node names from the name of the cluster via DNS seems like a good idea to me. And also if esdb+discover is using "cluster DNS" that isn't actually DNS as we know it, why use the term DNS at all? And if "esdb" is to use cluster info from the gossip service then I don't see the difference with the semantics you described for "esdb+discover". And if "esdb" is to not use cluster info from the gossip service then it just opens possibilities for expressing a node preference in the URI but specifying a node in the URI that doesn't satisfy this node preference. Maybe we could have a meeting about this? Or discuss it on the Discord channel, or maybe there's already a long thread about this? Who is responsible for designing this? Is what I have coded a correct understanding of an older design? If it has changed, why did it change? I'm quite confused.

I have to add that, indeed, the current implementation is based on correct requirements. What is implemented in the Python client is how our clients worked about two years ago.
Regarding the difference between esdb and esdb+discover. When esdb is used, there's no need to do any discovery. You have one host, and you don't need to call the gossip endpoint to get the list of nodes. You don't need to check if the node is alive because you have no option to connect to another node if the original node is dead. Node preference doesn't apply. So, basically, there's no "discovery" process going on there.
For example, one can unconditionally connect to a read-only replica if they know its hostname. They won't use esdb+discover with node preference set to RoR, but instead just connect directly. If they'd use `esdb+discover', the client would try connecting to an RoR but if it's not available, the client would connect to some other node, and that might not be what the user wants.

This was fixed in b30f99de9ad1805a65d9ce4eb9a7d1da46531cc2 released as version 1.0.18.


[Gettr] Unexpected behaviors

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.01.21. (update instructions)
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
No response
Description
I guess this falls more in the "misfunctioning" category as opposed to "broken." At the risk of confusion (this usually doesn't end well), I see two separate issues here. First, without a format type, yt-dlp retrieves a lower quality video (1,627,575 bytes with a video bit rate of 178 kb/s and an audio bitrate of 160 kb/s) than the best available. Second, when I specify the best quality video (hls-1936), I don't get anything near the filesize or bitrate indicated (3,734,051 bytes with a video bitrate of 793 kb/s).
Verbose log
$ yt-dlp-2022.01.21 -F 'https://gettr.com/post/ppbjz4da87'
[Gettr] ppbjz4da87: Downloading webpage
[Gettr] ppbjz4da87: Downloading JSON metadata
[Gettr] ppbjz4da87: Downloading m3u8 information
[info] Available formats for ppbjz4da87:
ID                      EXT RESOLUTION │ FILESIZE   TBR PROTO  │ VCODEC        VBR ACODEC     MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────
hls-group_audio-audio_0 mp4 audio only │                m3u8_n │ audio only        unknown    [ZH] audio_0
hls-616                 mp4 480x222    │ ~2.83MiB  616k m3u8_n │ avc1.4d401e  616k video only
hls-1056                mp4 640x296    │ ~4.85MiB 1056k m3u8_n │ avc1.4d401f 1056k video only
hls-1936                mp4 960x444    │ ~8.89MiB 1936k m3u8_n │ avc1.4d401f 1936k video only
ovid                    mp4 960x444    │                https  │ unknown           unknown

$ yt-dlp -v 'https://gettr.com/post/ppbjz4da87'
[debug] Command-line config: ['-v', 'https://gettr.com/post/ppbjz4da87']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.01.21 [f20d607b0] (zip)
[debug] Python version 3.9.9 (CPython 64bit) - Linux-5.15.11-with-glibc2.33
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1
[debug] Optional libraries: Crypto
[debug] Proxy map: {}
[debug] [Gettr] Extracting URL: https://gettr.com/post/ppbjz4da87
[Gettr] ppbjz4da87: Downloading webpage
[Gettr] ppbjz4da87: Downloading JSON metadata
[Gettr] ppbjz4da87: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] ppbjz4da87: Downloading 1 format(s): ovid
[debug] Invoking downloader on "https://media.gettr.com/group21/origin/2022/01/19/14/0f15706b-40b8-cb7a-6bf4-34e0a77b3198/out.mp4"
[download] Destination: Daniel Lemire - Must-watch video. This is what is happening in Quebec. Help! [ppbjz4da87].mp4
[download] 100% of 1.55MiB in 00:08

$ yt-dlp -v -f hls-1936 'https://gettr.com/post/ppbjz4da87'
[debug] Command-line config: ['-v', '-f', 'hls-1936', 'https://gettr.com/post/ppbjz4da87']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.01.21 [f20d607b0] (zip)
[debug] Python version 3.9.9 (CPython 64bit) - Linux-5.15.11-with-glibc2.33
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1
[debug] Optional libraries: Crypto
[debug] Proxy map: {}
[debug] [Gettr] Extracting URL: https://gettr.com/post/ppbjz4da87
[Gettr] ppbjz4da87: Downloading webpage
[Gettr] ppbjz4da87: Downloading JSON metadata
[Gettr] ppbjz4da87: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] ppbjz4da87: Downloading 1 format(s): hls-1936
[debug] Invoking downloader on "https://media.gettr.com/group21/origin/2022/01/19/14/0f15706b-40b8-cb7a-6bf4-34e0a77b3198/720p_v.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 5
[download] Destination: Daniel Lemire - Must-watch video. This is what is happening in Quebec. Help! [ppbjz4da87].mp4
[download] 100% of 3.76MiB in 00:45
[debug] ffprobe command line: ffprobe -hide_banner -show_format -show_streams -print_format json 'file:Daniel Lemire - Must-watch video. This is what is happening in Quebec. Help! [ppbjz4da87].mp4'
[FixupM3u8] Fixing MPEG-TS in MP4 container of "Daniel Lemire - Must-watch video. This is what is happening in Quebec. Help! [ppbjz4da87].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Daniel Lemire - Must-watch video. This is what is happening in Quebec. Help! [ppbjz4da87].mp4' -map 0 -dn -ignore_unknown -c copy -f mp4 -bsf:a aac_adtstoasc -movflags +faststart 'file:Daniel Lemire - Must-watch video. This is what is happening in Quebec. Help! [ppbjz4da87].temp.mp4'

Many sites provide the peak bitrate as tbr and not the average. This causes the approximate file size to be mis-calculated.
As for not selecting the best format by default - it does not seem possible to determine the best format due to lack of information about ovid. Let me see if there is a way to extract more information on that format

@i6t Why was a higher priority added for ovid format? (Ref: #1120). In this case, it has much less quality than the best m3u8 format

@pukkandan I had mistakenly assumed that "ovid" stood for "original video". This preference is a mistake.

@pukkandan As far as i checked their API and webpage, I could not find any information about the bitrate of ovid format :(
Should I remove the preference of ovid format for now?

Thanks for checking

Should I remove the preference of ovid format for now?


Use S9api in keyref processing

Description
Use S9api's XdmNode instead of DOM in keyref processing.
Motivation and Context
XdmNode is an easier object model to work with.
How Has This Been Tested?
Existing tests pass.
Type of Changes

New feature (non-breaking change which adds functionality)

Documentation and Compatibility
No documentation needed.

FWIW, verified that with our plugins and a single large content set, the output matches between this branch and develop (apart from changes to a handful of generated IDs).
My PDF build failed with java.util.NoSuchElementException but it also fails with that on develop, so that should be unrelated to this update.


Change XMLHttpRequest to asynchronous

Placeholder for this issue.  Both Chrome and Firefox throw an error for the current method being deprecated.  I have code that is asynchronous, but it's not working perfectly due to other major changes made to support blob replacement. (Which could/should? be a separate issue)

Yes, I had some problem with this in the original design. If I remember correctly some part of this flow is forced to be synchronous by Chrome, ie. we can't return a promise, I think it was the request intercept. Makes sense since the request could otherwise hang indefinitely waiting for some before filter to resolve.
That forced me to use a synchronous load of the remote payload. If there is a new way of solving this I'm all ears. At the time there was not...

Here is my tweaked/condensed background.js if it generates any ideas for you.  This probably could be fixed to replace the image with the data:image instead of the blob, but I wasn't expecting to run into the two bugs as I was tweaking it.
background.txt

I'll have a look at it when I have some time. What bugs did you encounter?

I put comments in the code for these two.  Basically I changed it to asynchronous then was trying to use a blob instead of the data:url since the blob links open so much faster,
https://bugs.chromium.org/p/chromium/issues/detail?id=446453
//Cannot redirect to blob in Chrome or FF
https://bugzilla.mozilla.org/show_bug.cgi?id=1256122
//Due to this bug, redirect to blank tab and open blob in new tab.


Conflicting Documentation on CreateSiteDesign

Category

[x] Error

Describe the bug
Located here: https://docs.microsoft.com/en-us/sharepoint/dev/declarative-customization/site-design-rest-api#createsitedesign
The parameters list includes "SiteScripts." The example below includes "SiteScriptIds." I do not know which one is correct, as I haven't gotten either to work yet.
Steps to reproduce
Look at the CreateSiteDesign section of the page above for site design REST API.
Expected behavior
I expected it would be clear what the correct parameter is.
Environment details (development & target environment)
Viewing in Microsoft Edge (Chromium). I'm attempting to then put the REST call into a Flow.
Additional context
None.

Hi, you are indeed correct. The value that should be passed in your request is "SiteScriptIds". I will update the parameters on that page.
As far is it not working for you, you may need to share some more information. I got a Site Script ID from my tenant using the GetSiteScript endpoint, and used that in my request, and it worked fine.

Hello @ryanlrobinson
I got the example running on my environment with following configuration:
POST: https://example-admin.sharepoint.com/_api/Microsoft.Sharepoint.Utilities.WebTemplateExtensions.SiteScriptUtility.CreateSiteDesign
BODY:
{
"info":{
"Title":"Title",
"Description":"Description",
"SiteScriptIds":["d2ef93d9-9ab5-4ab5-91c2-87bc0637d1fc"],
"WebTemplate":"64"
}
}
I hope it helps!


chore(cd): add debug symbol back into bazel built packages

Summary

Adding necessary opts to bazel so that the debug symbols will be there.
Checklist

[ ] The Pull Request has tests
[ ] There's an entry in the CHANGELOG
[ ] There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Full changelog

[Implement ...]

Issue reference
FTI-4918

Where is the EE pick for this one?


[workloadmeta] Avoid concurrent pulls from same collector

What does this PR do?
Fixes a bug in workloadmeta.
Avoids concurrent pulls from the same collector. Workloadmeta was calling Pull on all the collectors every 5 seconds without checking if they already finished the previous run.
Describe how to test/QA your changes
This is not straightforward to test because we can't control the time that running Pull takes on each collector. I added a time.Sleep call in the Pull of the Docker collector to test this.
Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

Rebased on top of main to fix CI issues.


fix(confirm): fix wrong type

fix(confirm): fix wrong type

Object.freeze sign is: freeze<T>(a: T[]): readonly T[].
And the absence of parentheses is intentional.

My IDE misleads me. Thanks for your correction!


liunx shell里面 的 ctrl + u;ctrl + k无法使用

Description of the new feature/enhancement

Proposed technical implementation details (optional)

Please fill the issue template and provide enough information, otherwise literally no one can help you.
请填写 issue 模板，提供足够的用于复现的信息。

Exactly what @skyline75489 said. If you can provide more information, we'd be happy to reopen this issue, but there's nothing we can do with a blank issue.


Docker images are cached by tag, and reusing a tag can cause errors.

What
Docker images are cached by tag, and reusing a tag can cause errors.
Advises tenants not to do this, and instead to use a unique tag. Suggests what
that tag could be.
How to review
Describe the steps required to test the changes.

Preview the content locally (see README) and check that it renders as expected.
Check you agree with the gist of what's being said

Who can review
Anyone

I don't suppose we could resolve to using Digests?

Possibly, but I don't know how. Worth a look in to when I have some free time, probably.


Custom dataset, training dies after 1 epoch

Hi, I've created a custom synthetic dataset and I have managed to get it to load and train for one epoch, after which it crashes every time. This is what is output before the actual crash, so you can see how far it made it:
2022-09-20 08:58:49,066 - mmdet - INFO - Epoch [1][50/226]	lr: 4.995e-05, eta: 1:11:12, time: 0.795, data_time: 0.044, memory: 3396, loss_cls: 1.0794, loss_bbox: 2.5220, loss_dir: 0.0790, loss: 3.6804, grad_norm: 92.3445
2022-09-20 08:59:27,583 - mmdet - INFO - Epoch [1][100/226]	lr: 9.990e-05, eta: 1:09:26, time: 0.770, data_time: 0.002, memory: 3396, loss_cls: 1.0202, loss_bbox: 1.8689, loss_dir: 0.0530, loss: 2.9421, grad_norm: 78.9453
2022-09-20 09:00:02,303 - mmdet - INFO - Epoch [1][150/226]	lr: 1.499e-04, eta: 1:06:12, time: 0.694, data_time: 0.002, memory: 3403, loss_cls: 0.9288, loss_bbox: 1.1846, loss_dir: 0.0254, loss: 2.1387, grad_norm: 46.8588
2022-09-20 09:00:40,658 - mmdet - INFO - Epoch [1][200/226]	lr: 1.998e-04, eta: 1:05:52, time: 0.767, data_time: 0.002, memory: 3403, loss_cls: 0.8908, loss_bbox: 0.8952, loss_dir: 0.0164, loss: 1.8024, grad_norm: 31.8395
2022-09-20 09:01:01,023 - mmdet - INFO - Saving checkpoint at 1 epochs
[                                                  ] 0/24, elapsed: 0s, ETA:Traceback (most recent call last):

So it gets through that first epoch and saves the checkpoint, and just as it is about to start what I think is the evaluation step, that's when it crashes. After the stack trace, this is the final error output when it stops:
  File "/home/damon/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmcv/ops/voxelize.py", line 71, in forward
    size=(max_voxels, max_points, points.size(1)))
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

What it looks like to me is that the evaluation (or maybe test?) dataset is not loaded correctly for some reason, but I'm not sure. When I print out the data in voxelize.py before line 71, it does suddenly change when not using the training dataset, but I don't know why. I'm glad to provide any configs if it would help. My config files started from the provided kitti and waymo configs, which I edited for my particular data.

An update: I can definitively say the problem is with the evaluation step. If I add this line to my dataset config:
evaluation = dict(interval=5, pipeline=eval_pipeline)

then it will train for 5 epochs no problem before the "Dimension out of range" error comes up.
Here's my data dict:
data = dict(
	samples_per_gpu=1,
	workers_per_gpu=2,
    train=dict(
        type=dataset_type,
        data_root=data_root,
        ann_file=data_root + "training.pkl",
        pipeline=train_pipeline,
        filter_empty_gt=False,
        classes=class_names,
        test_mode=False,
        box_type_3d='LiDAR'),
    val=dict(
        type=dataset_type,
        data_root=data_root,
        ann_file=data_root + "eval.pkl",
        pipeline=eval_pipeline,
        filter_empty_gt=False,
        classes=class_names,
        test_mode=True,
        box_type_3d='LiDAR'),
    test=dict(
        type=dataset_type,
        data_root=data_root,
        ann_file=data_root + "eval.pkl",
        pipeline=test_pipeline,
        filter_empty_gt=False,
        classes=class_names,
        test_mode=True,
        box_type_3d='LiDAR'))

Is there anything obviously wrong with that? The val and test data are loaded exactly the same way as the train data. Here's the eval pipeline in case that will help:
eval_pipeline = [
    dict(
        type='LoadPointsFromFile',
        coord_type='LIDAR',
        load_dim=7,
        use_dim=3),
    dict(
        type='DefaultFormatBundle3D',
        class_names=class_names,
        with_label=False),
    dict(type='Collect3D', keys=['points'])
]

Just another small update on this in the hopes it will help someone to help me figure this out. I've now had exactly the same error with using both a PointPillars model (which is what the above errors are from) and also with the CenterPoint model, which I managed to get training today.
In both models, it is exactly the same problem: training goes for as many epochs as I let it, but as soon as a test or eval pipeline are started, it crashes out with the Dimension out of range error above in voxelize.py. The same exact error also occurs when running tools/test.py with my configurations. As far as I can tell, the problem is not with my custom data because the training step works and model files for each epoch are even saved to the output folder.
Here's my dataset class, maybe that will help too:
# dataset settings
dataset_type = 'MyDataset'
data_root = 'data/mydataset/'

class_names = ['object1', 'object2', 'object3']

pc_range = [-10.0, -100.0, -5.0, 500.0, 100.0, 10.0]
input_modality = dict(use_lidar=True, use_camera=False)

train_pipeline = [
    dict(
        type='LoadPointsFromFile',
        coord_type='LIDAR',
        load_dim=7,
        use_dim=3),
    dict(
        type='LoadAnnotations3D',
        with_bbox_3d=True,
        with_label_3d=True),
    dict(type='PointsRangeFilter', point_cloud_range=pc_range),
    dict(type='ObjectRangeFilter', point_cloud_range=pc_range),
    dict(type='ObjectNameFilter', classes=class_names),
    dict(type='PointShuffle'),
    dict(type='DefaultFormatBundle3D', class_names=class_names),
    dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
]

test_pipeline = [
    dict(
        type='LoadPointsFromFile',
        coord_type='LIDAR',
        load_dim=7,
        use_dim=3),
    dict(
        type='DefaultFormatBundle3D',
        class_names=class_names,
        with_label=False),
    dict(type='Collect3D', keys=['points'])
]

eval_pipeline = [
    dict(
        type='LoadPointsFromFile',
        coord_type='LIDAR',
        load_dim=7,
        use_dim=3),
    dict(
        type='DefaultFormatBundle3D',
        class_names=class_names,
        with_label=False),
    dict(type='Collect3D', keys=['points'])
]

data = dict(
	samples_per_gpu=1,
	workers_per_gpu=2,
    train=dict(
        type=dataset_type,
        data_root=data_root,
        ann_file=data_root + "training.pkl",
        pipeline=train_pipeline,
        filter_empty_gt=False,
        classes=class_names,
        test_mode=False,
        box_type_3d='LiDAR'),
    val=dict(
        type=dataset_type,
        data_root=data_root,
        ann_file=data_root + "eval.pkl",
        pipeline=eval_pipeline,
        filter_empty_gt=False,
        classes=class_names,
        test_mode=True,
        box_type_3d='LiDAR'),
    test=dict(
        type=dataset_type,
        data_root=data_root,
        ann_file=data_root + "eval.pkl",
        pipeline=test_pipeline,
        filter_empty_gt=False,
        classes=class_names,
        test_mode=True,
        box_type_3d='LiDAR'))

evaluation = dict(interval=1, pipeline=eval_pipeline)

I am not 100% sure but I think I stumbled on this as well and the reason is that in the training pipeline the data is presented in batches and hence the individual data frames are encapsulated in a list.
This list is missing in your test and eval pipelines and therefore the voxelizer is receiving the data tensor directly. If I am correct, than this ultimately leads to points in https://github.com/open-mmlab/mmcv/blob/3b9b9054bf4b5195d57bf6dfc6a60860b0a3cec1/mmcv/ops/voxelize.py#L70 being not a two dimensional tensor as expected.
In the example configs in the repo the test and eval pipelines always contain a MultiScaleFlipAug3D transform: https://github.com/open-mmlab/mmdetection3d/blob/76e351a7a6b0b83d7dc729bdfee5f17743c72f76/configs/base/datasets/kitti-3d-3class.py#L61
I assume that this fixes the issue by re-encapsulating the data as for the training use-case. But frankly I never checked the code of this transform to validate this assumption. But simply give it a try...

Hi wittmeis, thank you for the response! As it turns out, you are correct. I found out the same thing just as my day was finishing yesterday. That MultiScaleFlipAug3D is absolutely required, or testing and evaluation fail and give the error I posted. I took it out of my pipelines because I didn't want to do it on my data.
The docs don't mention it's a required field, so I didn't think it would be an issue. In addition, it is NOT required for the training pipeline, which added to my confusion. Training works perfectly fine without it, but not testing or evaluation. My dataset does not contain images, so including a transform that has an "img_scale" field seemed wrong.
In case others come across this, here's my working eval pipeline:
eval_pipeline = [
    dict(
        type='LoadPointsFromFile',
        coord_type='LIDAR',
        load_dim=7,
        use_dim=3),
    dict(
        type='MultiScaleFlipAug3D',
        img_scale=(1, 1),
        pts_scale_ratio=1,
        flip=False,
        transforms=[
            dict(
                type='GlobalRotScaleTrans',
                rot_range=[0, 0],
                scale_ratio_range=[1., 1.],
                translation_std=[0, 0, 0]),
            dict(
                type='PointsRangeFilter', point_cloud_range=pc_range),
            dict(
                type='DefaultFormatBundle3D',
                class_names=class_names,
                with_label=False),
            dict(type='Collect3D', keys=['points'])
        ])
]

So now evaluation completes, but I still can't complete a training. I'm running into an issue with a key missing in my annotations during the step for calculating accuracy. It's not listed in the docs in the tutorial on pure point clouds, so I don't know exactly what it means. That's a separate issue though so I'll go ahead and close this one.

hello, thanks for detailed situation description about error.
I'm in completely same situation with you.. I guess this error is from my dataset which don't include image and calib.
If you have solved this problem, may I know how you solved it?
thanks.

Hello, I did actually manage to get past that last issue I mentioned about a key missing. Unfortunately it was a few months ago and now I don't remember exactly what I did. I can say for sure that once I did manage to complete full trainings I was never able to get any detection accuracy above 3% anyway, and for unknown reasons. We ended up abandoning MMDetection3D not long after I posted these messages for a different solution using 2D networks. I will be looking at this again soon however, and if I can remember how I solved the key missing problem I'll be sure to post here.


Shader debug support for Broadwell and Skylake

Shader debugging needs to be enabled via MMIO registers.
Signed-off-by: Mircea Gherzan<EMAIL_ADDRESS>

Jayesh will verify.

Hi Mircea,
I would appreciate if you could help me with the typical tests that you would like me to run on this pull request as part of verification.
Thanks.

Jayesh, please run the "hybrid" manual tests. Sean can provide details
about them.

Jayesh - please retest with updated patches.

Picking up updated commit and validating on SKL and BDW.

Thanks @jayesht

Validated the pull request#31 with

script for vp9_hybrid_decode_script_libyami on BDW and SKL platform, it validates md5sum of decoded bitstreams.
Manual yamidecode run for VP9 stress bitstreams, on screen and to the file decode runs.

On side note: yamidecode shows Red colorspace output with BDW, whereas shows Gray colorspace output on SKL. This is yamidecode's issue.
Patch lgtm.

Thanks all, merged.


Old logo on the intro gif

@Shipow should we replace the README.md gif with this one?

yeah definitely 👍


Moved _print_valid_platforms to "public"

This method is used outside of the molecule class, therefore it isn't
"private".

👍

  syntax: commands succeeded
  py27-ansible19-unit: commands succeeded
  py27-ansible19-functional: commands succeeded
  py27-ansible20-unit: commands succeeded
  py27-ansible20-functional: commands succeeded
  py27-ansible21-unit: commands succeeded
  py27-ansible21-functional: commands succeeded
  doc: commands succeeded
  congratulations :)


How to use Mamba2

I am using the medical image segmentation model VM-Unet for some tasks, and now I want to modify Mamba to Mamba2 on top of it. How should I make this modification?

Just change the layer constructor


upgrade prometheus version.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: morvencao
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS [morvencao]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@morvencao: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
9e3a5952632aa3b6bd2307187db4a41507ba9de8
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@morvencao: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
9e3a5952632aa3b6bd2307187db4a41507ba9de8
link
/test istio-pilot-multicluster-e2e

prow/istio-integ-k8s-tests.sh
9e3a5952632aa3b6bd2307187db4a41507ba9de8
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: linsun, morvencao
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS [linsun,morvencao]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Add Basemap Logic with Radio Buttons

map.setStyle()

If you really want a basemap control you would have to load the layers s in a function, then call that function when the style is loaded - see https://github.com/mapbox/mapbox-gl-js/issues/2268. You would also have to keep track of what layers were visible, all the data that was drawn on the map, etc. for when the basemap changes.


"Failed to resolve filter" while running tests

Version
1.0.0-beta.14
Reproduction link
https://jsfiddle.net/50wL7mdz/323724/
Steps to reproduce
Create a Vue project
Add a filter (like mine or a different one)
Test project with Jest and vue-test-utils
What is expected?
Test should run without any errors
What is actually happening?
Test are running, but throwing errors:
"[Vue warn]: Failed to resolve filter: si"

I have also tried importing Vue to the test spec file and setting up the filter there, but that didn't seem to help.

I can't reproduce this.
For me, registering a filter before mounting works as expected:
Vue.filter('aFilter', () => 'foo')

test('updates computed properties', () => {
  const wrapper = shallow(App)
  expect(wrapper.text()).toContain('foo')
})

OK, following this example and testing on a brand new build cleared things up. I must have made a mistake on my side. Thank you very much and sorry for the false report.

If you're using Nuxt or are using modular filter imports you can do this.
import "~/plugins/filters";


Base Age/ Max Age of Dynamic Trees

Would it be possible to add a modifier that allows you to change the base age and max age of the trees?
Basically allowing people to choose how big the trees can grow and start as.
If you want some explanation for how this would be useful and improve this mod, it would allow people to create massive forests, new growth looking forests, etc. There would also obviously have to be a limit on how big the trees are, maybe capping out around 24 blocks?
Thanks for reading my suggestion, it'd be incredibly useful for a server I plan on running in the future.

Trees of that size simply aren't possible without major changes to how we handle thick trees. We do plan on doing this in the future but it's not a simple change.
As for configuring the max ages and such you can actually already sort of do this yourself to individual species using tree packs. If you change the soil_longevity key, such as is done for thick trees like the mega spruce. However I'm not entirely sure what you mean by "base age", please explain this.


fix: keyboard buttons will fail to submit data.

reslove: https://github.com/Magickbase/ckb-explorer-public-issues/issues/573

https://github.com/Magickbase/ckb-explorer-frontend/assets/25676576/dba9a64e-0f71-46d0-a8ea-16200c958724
The button is Ok when opening the page without switching pages, but after switching pages in the middle, the button is changed to Next Item.

Freder.2024-03-<IP_ADDRESS>.mp4
The button is Ok when opening the page without switching pages, but after switching pages in the middle, the button is changed to Next Item.
Changed enterKeyHint to search can check it again..


Is there Something to be done after running download.sh for smallNORB data to be available?

I run ./download.sh under ./data and then the directory smallNORB and 4 files smallnorb-xxxx-{testing|training}-{cat|dat}.mat got available.
But when I tried to train the network with the smallNORB dataset by sending the command python train.py "smallNORB", errors as follows returned.
$ python3 train.py "smallNORB"
Using TensorFlow backend.
2018-08-07 19:49:45,984 [5964] INFO     __main__: Using dataset: smallNORB
2018-08-07 19:49:45,990 [5964] CRITICAL root: Traceback (most recent call last):
  File "train.py", line 156, in <module>
    tf.app.run()
  File "/home/user/python3.6/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "train.py", line 52, in main
    batch_x, batch_labels = create_inputs()
  File "/path/to/Matrix-Capsules-EM-Tensorflow/config.py", line 84, in <lambda>
    'smallNORB': lambda: create_inputs_norb(is_train, epochs),
  File "/path/to/Matrix-Capsules-EM-Tensorflow/utils.py", line 31, in create_inputs_norb
    image, label = norb.read_norb_tfrecord(chunk_files, epochs)
  File "/path/to/Matrix-Capsules-EM-Tensorflow/data/smallNORB.py", line 128, in read_norb_tfrecord
    filename_queue = tf.train.string_input_producer(filenames, num_epochs=epochs)
  File "/home/user/python3.6/lib/python3.6/site-packages/tensorflow/python/training/input.py", line 241, in string_input_producer
    raise ValueError(not_null_err)
ValueError: string_input_producer requires a non-null input tensor

Did I miss anything to do before training with smallNORB?
I think the 4 files downloaded should be converted tfrecord files.
I'm a begginer of tensorflow so what I said maybe beside the point...
Thanks.

Hi, I'm having exactly the same problem. Any solutions please?

嗨，我也遇到了 相同的问题，请问有解决吗

I believe python smallNORB.py tfrecord to generate tfrecord files (tensorflow multi-thread I/O files) from raw data.

I believe python smallNORB.py tfrecord to generate tfrecord files (tensorflow multi-thread I/O files) from raw data.

Thank you very much for your reply. But after I ran your code “python3 smallNORB.py tfrecord” , there were new errors.
CRITICAL root: Traceback (most recent call last):
File "smallNORB.py", line 210, in
argh.dispatch(parser)
File "/home/ailab/.local/lib/python3.5/site-packages/argh/dispatching.py", line 174, in dispatch
for line in lines:
File "/home/ailab/.local/lib/python3.5/site-packages/argh/dispatching.py", line 277, in _execute_command
for line in result:
File "/home/ailab/.local/lib/python3.5/site-packages/argh/dispatching.py", line 260, in _call
result = function(*positional, **keywords)
File "smallNORB.py", line 115, in tfrecord
write_data_to_tfrecord(kind='train', chunkify=False)
File "smallNORB.py", line 86, in write_data_to_tfrecord
images = images[perm]
MemoryError
After I tried to allocate 6G memory to the virtual machine, it was still a mistake. Do you know the solution?

@zhaolei0413 Try to allocate as much memory to the VM as possible, 6G might not be enough since it still shows "MemoryError".

I believe python smallNORB.py tfrecord to generate tfrecord files (tensorflow multi-thread I/O files) from raw data.

When I tried this, I am getting the following error:
FileNotFoundError: [Errno 2] No such file or directory: './smallNORB/smallnorb-5 x46789x9x18x6x2x96x96-training-dat.mat'

You downloaded less smallNORB files. You can download them from my link and extract them into the data directory.

我相信python smallNORB.py tfrecord从原始数据生成tfrecord文件（tensorflow多线程I / O文件）。

当我尝试这个时，我收到以下错误：
FileNotFoundError: [Errno 2] No such file or directory: './smallNORB/smallnorb-5 x46789x9x18x6x2x96x96-training-dat.mat'

You downloaded less smallNORB files. You can download them from my link and extract them into the data directory.链接: https://pan.baidu.com/s/1_AmFIG8hagZMgevvK8wPmQ 提取码: rcd1

我相信python smallNORB.py tfrecord从原始数据生成tfrecord文件（tensorflow多线程I / O文件）。

当我尝试这个时，我收到以下错误：
FileNotFoundError: [Errno 2] No such file or directory: './smallNORB/smallnorb-5 x46789x9x18x6x2x96x96-training-dat.mat'
You downloaded less smallNORB files. You can download them from my link and extract them into the data directory.
链接: https://pan.baidu.com/s/1_AmFIG8hagZMgevvK8wPmQ 提取码: rcd1


[Owner Review] Added NiDmm missing enum metadata and auto generated them

What does this Pull Request accomplish?
The metadata of missing enums and autogenerted them
The name and values of the enums were obtained from the nidmm.h file and they were added to the dictionary in enums.py following the format of other enums. The enums were then referenced in the attributes and functions where they are being used in the attributes.py and functions.py respectively
Why should this Pull Request be merged?
The missing enums can now be autogenerated
Testing
The build was successful and the enums got added to the nidmm.proto file

1. Please do not remove the testing section from your PR description. That template is there for a reason. If you have done no testing, then say so in that section. In this case, I would expect you to at least say that it builds and runs successfully.

2. You did not comment anywhere on your process for getting these enum values. Did you and copy them from the base enums.hapigen and do hand conversions?

3. How did you arrive at the changes to functions.py? Was that information in the hapigen already but not the nimi-python?

I marked as approved to save time since the changes look good (with a little bit of feedback), but please post responses to these questions. If I have additional feedback based on your answers, then we can address them in a follow-up PR.

I added back the testing section in this as well as the other attributes pr and also added the description of how the enums and attributes were added


HDDS-10915. Remove unused org.glassfish:javax.servlet dependency declaration

What changes were proposed in this pull request?
Dependency declaration for org.glassfish:javax.servlet is unused, can be removed.
https://issues.apache.org/jira/browse/HDDS-10915
How was this patch tested?
CI:
https://github.com/adoroszlai/ozone/actions/runs/9235851651

Thanks @myskov for reviewing and merging this.


Add Dockerfile and Docker quickstart guide

I've created a simple Dockerfile and guide to be considered for inclusion in this repo.
The goal in mind is to lower the entry barrier for users that want to quickly get started and explore this repo without having to install all of the projects dependencies (Caffe).

Thank you for submitting this pull request, however I do not see a valid CLA on file for you. Before we can merge this request please visit https://yahoocla.herokuapp.com/ and agree to the terms. Thanks! :smile:

CLA is valid!

Thanks for adding docker support. Can you also add steps / links on docker installation (if possible for Mac)? I will try it out.
In case there are lot of steps (for different OS) I think we could add a wiki page for docker setup and point to it from this Readme, so that the Readme remains compact. What do you think?

I've added Docker Installation links for Windows, Mac and Ubuntu. I have tested the getting started guide on Mac OSX and Ubuntu but do not have a Windows machine for testing at this time.

@neufeldtech it seems we did both the same! Please take a look here also if it can help https://github.com/yahoo/open_nsfw/issues/7 so that we will have just one PR 👍

@jay-mahadeokar I've updated the README to build the docker image directly from the official BVLC caffe repo now.
@loretoparisi I like your approach! I've collapsed a few steps of your instructions into fewer commands. Take a look at what I've got so far here: 68040e6
Note: I opted not to include how to build the GPU image because I could not get it to work. What specific requirements/dependencies would someone need in order for this to work with the GPU image?

Thanks for adding docker support @neufeldtech  @loretoparisi.

@neufeldtech basically you need to install on the host machine the nvidia CUDA 8.0 drivers, what we could do is

use caffe build through the caffe:gpu directly on the host machine:

$ docker build -t caffe:gpu ./

we verify
$ docker run -ti caffe:gpu caffe --version
libdc1394 error: Failed to initialize libdc1394
caffe version 1.0.0-rc3

This will download nvdia CUDA drivers and install it
$ nvidia-smi
Wed Oct  5 10:36:29 2016
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.44                 Driver Version: 367.44                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 620      Off  | 0000:01:00.0     N/A |                  N/A |
| 52%   30C   P12    N/A /  N/A |    247MiB /   962MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+

Note that we have one GPU (a very cheap GeForce GT 620 here) addressed at 0
And our container is up & running
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                     PORTS                                            NAMES
443a7addcdff        caffe:gpu                     "caffe --version"        2 minutes ago       Exited (0) 2 minutes ago                                                    naughty_hamilton

Using the nvidia-docker container. In this way it would be very easy to add optional support to gpu that will work linux machines (Ubuntu/CentOS) with a supported CUDA 8.0 gpu (the problem here is that CUDA 8.0 moved to the new Pascal architecture, so it needs new graphic cards drivers on old gpus, etc.

In this case we could do
# Install nvidia-docker and nvidia-docker-plugin
wget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.0-rc.3/nvidia-docker_1.0.0.rc.3-1_amd64.deb
sudo dpkg -i /tmp/nvidia-docker*.deb && rm /tmp/nvidia-docker*.d

and verify again that the nvidia docker container is getting the gpu as well
$ nvidia-docker run --rm nvidia/cuda nvidia-smi
Wed Oct  5 08:39:16 2016
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.44                 Driver Version: 367.44                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 620      Off  | 0000:01:00.0     N/A |                  N/A |
| 52%   31C   P12    N/A /  N/A |    247MiB /   962MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+

ok it seems so,
./nvidia-docker-run --volume=$(pwd):/workspace --rm -it caffe:gpu bash

so that we will have caffe running with the gpu on the host machine:
root@1bd4c3073ef0:/workspace# python ./classify_nsfw.py --model_def nsfw_model/deploy.prototxt --pretrained_model nsfw_model/resnet_50_1by2_nsfw.caffemodel anderson_210.jpg

If we can this typical error
F1005 08:48:48.429678    16 cudnn_conv_layer.cpp:52] Check failed: error == cudaSuccess (35 vs. 0)  CUDA driver version is insufficient for CUDA runtime version

it means that we have to update the CUDA driver as stated above.

I think it is best to have a wiki page with docker information and documentation.   We can move all the documentation to that page, so that the Readme itself remains concise.  @loretoparisi we can also move the existing docker documentation in Readme to that wiki page.


Improve save_model and load_model

Fixes #19793

save_model will first open the .h5 (in the same working dir as filepath), then write it to the zipfile at the end.
load_model will first extract the .h5, then load it using h5py.File. The weights file will be deleted at the end.

Result:
Using GPT2 from KerasNLP

Branch
Function
Memory Usage
Time Cost

Master
save_model
1860.43MiB
3.84s

PR
save_model
1384.17MiB (-25.6%)
4.48s (+16.7%)

Master
load_model
1377.24MiB
26.93s

PR
load_model
1376.43MiB (-0.1%)
2.10s (-92.3%)

The only regression is that save_model might be slightly slower than before.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 56.96%. Comparing base (11be99a) to head (ccb14c9).

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #19852       +/-   ##
===========================================
- Coverage   78.83%   56.96%   -21.88%
===========================================
  Files         498      498
  Lines       45864    45880       +16
  Branches     8450     8454        +4
===========================================
- Hits        36159    26136    -10023
- Misses       7999    18137    +10138
+ Partials     1706     1607       -99

Flag
Coverage Δ

keras
56.96% <100.00%> (-21.74%)
:arrow_down:

keras-jax
?

keras-numpy
56.96% <100.00%> (+0.01%)
:arrow_up:

keras-tensorflow
?

keras-torch
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

a tiny suggestion: the form _VARS_FNAME + ".h5" appears many times. Would it be better to define a new constant like _VARS_FNAME_H5 = _VARS_FNAME + ".h5"?

a tiny suggestion: the form _VARS_FNAME + ".h5" appears many times. Would it be better to define a new constant like _VARS_FNAME_H5 = _VARS_FNAME + ".h5"?

That's cleaner. Thanks for the suggestion.
Updated.


Move scion to ngipkgs

Source flake
Current state:

Uses buildGo117Module for most packages, which is not provided by our nixpkgs version.
Has some tests, but nix flake check fails:

 '/nix/store/3vc2g0frp09zjkbfdpmj9dlkv25ddp67-source/19-ffaa_1_fe3.tar.gz': No such file or directory

and indeed the file is missing.

nix build .#scion works.
Provides some modules as well as the packages.

I tried to build it using buildGoModule just to check and got this error:
vendor/github.com/lucas-clemente/quic-go/internal/qtls/go118.go:5:13:
cannot use "quic-go doesn't build on Go 1.18 yet." (untyped string constant) as int value in variable declaration

which seems to suggest there are upstream dependencies that need to be updated before this flake can be ported, as per this recommendation.

Turns out @sarcasticadmin has been able to bump the Go toolchain version: https://github.com/ngi-nix/scion/pull/7

It seems that nix flake check is failing in master of the ngi-nix/scion repository. And we're not sure it is trivial, so we'll see whether easier tasks are available.

Theres some remaining upstream work in scion-apps (https://github.com/netsec-ethz/scion-apps/pull/246) that Im waiting on. But once that lands the drvs in ngi-nix/scion should be much more update to date.

@mightyiam @augustebaum @lorenzleutgeb All of the upstream work has landed so the PR https://github.com/ngi-nix/scion/pull/7 is ready for review

Closing, as SCION is already being landed in Nixpkgs https://github.com/NixOS/nixpkgs/pull/298737


CI: Migrate to GCC-7 for Ubuntu Linux

The GNU toolchain shipped with Ubuntu Linux 16.04-LTS is quite old, and
various enhancements of mimalloc might not be facilitated. This patch
bumps CI configurations to specify gcc-7 for Linux targets.

Mmm, not sure if this is worth the complexity especially as I test often myself with the latest toolchains during development. In a way, the old toolchain gives us an extra check that I would otherwise not test for.


nvidia-x11: fix driSupport32Bit

Motivation for this change
driSupport32Bit was broken by 5d11dac8bb8f10b797467e913d6ebb9ea1eccd29.

cc @vcunat

This fixes the evaluation of nixpkgs-channels/nixos-unstable for me, waiting for the build now ...

... new build based on nixos-unstable + this PR works fine on kernel 4.6.5, including 3D on wine32. Didn't test 4.7, though.


High Ram usage in cosmic applications

Hi @lilyinstarlight,
i'm trying out Cosmic on nixos 24.11 and what i noticed is that many the Cosmic application use a lot of ram on my machine (Framework 13 amd so a x86_64 machine).
I'm not sure if that is a problem with the nix flake or with Cosmic Epoch itself.
But as other Cosmic users, don't seem to have the problem I ask here.
Specifically cosmic-comp, a bunch of applets and a few other applications all have roughly the same memory usage of ~1.4GB per process (about half of that is shared, please see screenshot of htop).
Which seems very big to me.
The applications and applets also take a while to open for the first time, which is probably related.
I have these sevices set and i'm using the cosmic cachix cache.
services.desktopManager.cosmic.enable = true;
services.displayManager.cosmic-greeter.enable = true;
Thanks for the great work on the flake, I really appreciate it!

I am experiencing the same with high ram usage and also the applets being taking a long time to show output when i click o them.

@modisek I hava the same experience with you.

I cloned cosmic-term repo and build it locally (cargo run --release). The memory usage is still high(~700MB). Same as cosmic-term under this flake.
So maybe cosmic itself's problem.

On archlinux, cosmic-term use 60MB. So maybe some logic of iced on nixos leads to this problem.

Hi thanks for the hint @Itsusinn you lead me on the right track,
I think i found the cause.
Looking through the open Iced issues, i found this one https://github.com/iced-rs/iced/issues/2455, where an Iced application would load all Systemfonts into memory on startup and cause performance problems through that.
I have nerdfonts installed which are pretty huge.
Removing the nerdfont package from my nix config seems to do the trick.
The Memory usage went form about 1.5 GB to about 60-100mb per app and everything is really snappy now.
So until this gets fixed on the Iced side the workaround would be to keep the fonts folder small.

Thanks for that workaround @Thorndur after reducing my fonts, memory usage has gone down for everything excluding the cosmic-applets which seems to be using a lot of ram still..Other than that its now somewhat usable for me

I made a pull request for cosmic-text sereval days ago, which makes parsing process quicker. At that time i didn't notice it used so much RAM. But now I think reducing fonts that need to be parsed is root of the problem.

Is this still an issue? It should probably be reported upstream instead of here

I think it is be fixed already.

There is the related upstream issue https://github.com/pop-os/cosmic-text/issues/282

I don't know if open a new issue, but the applets continue to consume a lot of ram.


Confusion between "battle_mode" and "mcts_mode"

Hello,
I think there is a "bug" in the actual version of the alphago code when using mode "play_with_bot_mode".
Indeed in both tictactoe_env.py and gomoku_env.py there is this line hardcoded:
self.mcts_mode = 'self_play_mode'
So mcts_mode is always set to self_play_mode, no matter what is giving inside the config.
Moreover in both python tree and C++ tree of alphago we can found those lines:
self.simulate_env.battle_mode = self.simulate_env.mcts_mode # In ptree_az.py
simulate_env.attr("battle_mode") = simulate_env.attr("mcts_mode"); # In mcts_alphazero.cpp
So that means that no matter what we give in config for battle_mode, this is overrided with the mcts_mode which is always "self_play_mode"...
In conclusion, after reviewed quickly the code, I think that mcts_mode should just be removed and replaced by battle_mode everywhere because both attributes seems to make the exact same things (but I may be wrong).
To reproduce you can just run the standard tictactoe in 'play_with_bot_mode' (by running tictactoe_alphazero_bot_mode_config.py) and check that the mcts is always using "self_play_mode".

Thank you very much for your thoughtful feedback.

We acknowledge your point, but it's indeed necessary to consistently set simulate_env.battle_mode to self_play_mode. This is because regardless of how we interact with the true environment during the data collection phase (i.e., whatever the battle_mode setting in the real environment), we should not give the agent access to the opponent's policy when executing the MCTS search. Therefore, during the MCTS search process, simulate_env.battle_mode is always set to self_play_mode.

However, this could potentially lead to some confusion. Our self.mcts_mode might need to be renamed to self.battle_mode_in_simulation_env to more accurately reflect its role in the simulation environment. It is worth noting that we have not hardcoded a fixed value, but instead left this parameter reserved for debugging purposes.

For relevant information, you may refer to this issue.

If you have any suggestions for improvement, please feel free to provide them. Best wishes!

Hello,
Ok sorry for my missunderstanding.
Thank you for your explanation I understand now why you always set battle_mode to self_play in the MCTS.


Add pull request template and issue templates

This is an automatic batch pull request.

This change is

Codecov Report

Merging #16 into dev will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##             dev     #16   +/-   ##
=====================================
  Coverage   2.38%   2.38%
=====================================
  Files          2       2
  Lines         84      84
=====================================
  Hits           2       2
  Misses        82      82

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc28160...ad9c07c. Read the comment docs.

Pull-request updated, HEAD is now f22385bd9a70a755bd0ba9eb7a703b6b3c3bf8ab


fix: ensure node annotations are checked before watch

kube-vip gets "stuck" waiting for the Metal annotations when they already exist on the nodes.
This change allows for the annotations on the node to be checked prior to enter the watch.
I've got a new environment spinning up now to test and will report back shortly.

@thebsdbox I've added some tests around this and it's help identify some weird behaviour that may or may not be intended.
The peer-ip is parsed, comma separated, and stored within the BGPConfig struct. However, only a single peer is stored in the BGPPeerConfig.
Is this intended?


Fix action columns not being extendable dynamically

1. Why is this change necessary?
Actions column items cannot be added or removed after the action column has been instanciated.
2. What does this change do, exactly?
Change the renderer closure of the action columns to items can now be removed and added dynamically again.
The problem is because the variable var l = items.lenth is defined outside the me.renderer closure ...
https://github.com/shopware/shopware/blob/301699adba5f423cb0b0a11795e912fe21478118/engine/Library/ExtJs/overrides/Ext.grid.column.Action.js#L38
and used inside the closure:
https://github.com/shopware/shopware/blob/301699adba5f423cb0b0a11795e912fe21478118/engine/Library/ExtJs/overrides/Ext.grid.column.Action.js#L55
When now the items get modified after the action column object has been initialized, the value for l is not updated.
Solution: Get the length of me.items dynamically. (See commit of pull request)
3. Describe each step to reproduce the issue or behaviour.
The problem is for example reproducable with the order document table. The action column of this table is instanciated in the createActionColumn method (which is called directly by the initComponent() method)
    /**
     * Creates the action columns
     *
     * @returns { Ext.grid.column.Action }
     */
    createActionColumn: function() {
        var me = this;

        return Ext.create('Ext.grid.column.Action', {
            width: 60,
            items: [
                me.createDeleteColumn(),
                me.createSendColumn()
            ]
        });
    },

If you try to extend the action column now by doing the following:
Ext.define('Shopware.apps.Example.view.list.Document', {

    override: 'Shopware.apps.Order.view.list.Document',

    createActionColumn: function() {
        var actionColums = this.callParent(arguments);
        actionColums.items.push({
            iconCls: 'sprite-mail-send',
            action: 'example'
        });
        return actionColums;
    }
}

... the action column is not changed as expected.

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/SW-19616.
Please use this issue to track the state of your pull request.

Hi @windaishi, thank you for your help! Your PR will be part of the Shopware v5.3.4! 🎉


Improve SSL certificate verification handling

Instead of globally disabling SSL certificate verification for the entire
Python instance, enable/disable it on a per-BIGIP object basis with the
verify argument, which defaults to False for backward-compatibility.

lgtm


Embark Simulator API Reference

Hi,
I am struggling to write tests that access a contract's deployed address or accessing a contract's public variables etc. Please point me to either the simulator's API reference or some examples that use the api's.
Thank you for your time!!

the simulator API should actually be just like the real thing for the most part: https://github.com/ethereum/wiki/wiki/JavaScript-API

Hi Iuri,
I plan to do a PoC to manage customer Loyalty. the BC application would
manage the customer loyalty programs with smart contracts. I am stuck on he
implementation. Can you help me put out a data flow diagram that i can
refine and detail it out to the implementation level.
your help is greatly appreciated.
thanks
On Thu, Oct 20, 2016 at 3:37 PM, Iuri Matias<EMAIL_ADDRESS>wrote:

Closed #171 https://github.com/iurimatias/embark-framework/issues/171.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/iurimatias/embark-framework/issues/171#event-830352386,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUTAM6wFpPLM0Dlro6ljw1X4rQbvb_Jqks5q1z18gaJpZM4Jxdjh
.

@talktoavi can you come to gitter? https://gitter.im/iurimatias/embark-framework

Hi Iuri,
i am new to github and not able to find my earlier post. In continuation of
my intent to implement Loyalty program on BC, here is my concept. Please
help me with your thoughts. I can post this on github but not finding the
last thread of discussion
My solution will be for managing Loyalty program and will have 3 components.

  Identity and Access Management that manages user specific

information that will be used by the store to validate user information.

  Loyalty program management that has all the loyalty rules and

promotion rules

  Loyalty points management (eWallet) to store user’s loyalty points

or coupons
I will enhance the functionality by bringing in more components like
payment gateway integration, integration with company finance systems etc
I need your help to understand which of these components should be
implemented on blockchain. Does it make sense to implement IAM on
blockchain where customer information (KYC) documents are stored for
validation?
On Wed, Apr 5, 2017 at 7:13 PM, Iuri Matias<EMAIL_ADDRESS>wrote:

@talktoavi https://github.com/talktoavi can you come to gitter?
https://gitter.im/iurimatias/embark-framework
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/iurimatias/embark-framework/issues/171#issuecomment-291865307,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUTAM_GIRnFv4nV-YzLSzQ_GLb9zR6_oks5rs5qAgaJpZM4Jxdjh
.


botocore.exceptions.ClientError

Hi Team,
I am reading files from S3 list objects.
result = self.clientS3.list_objects(Bucket=self.BUCKET, Prefix=self.KEY, Delimiter='/')
My docker image is throwing the below error locally.
result = self.clientS3.list_objects(Bucket=self.BUCKET, Prefix=self.KEY, Delimiter='/')
botocore.exceptions.ClientError: An error occurred (RequestTimeTooSkewed) when calling the ListObjects operation: The difference between the request time and the current time is too large.
Please suggest, how to fix this. mine is a Windows 10 machine.
Regards,
Prashan

https://stackoverflow.com/a/4770683

The time on your local box is out of sync with the current time. Sync up your system clock and the problem will go away.

Setting the TZ environment variable to a timezone will change the date accordingly
$ docker run --rm -e TZ=IST python:slim date
Mon Apr 27 14:43:07 IST 2020

Questions like these should be asked over at the Docker Community Forums, Docker Community Slack, or Stack Overflow. Since these repos aren't really a user-help forum