Fix CSV import of bedcounts and add test

The importer was calling sync_users instead of sync_bedcounts. We
also fix parsing of dates and add a test for this importer.

@dulacarnold This should be ready. I have a script that calls this importer after preprocessing FINESS data, that I'll submit as a subsequent PR.


[#155646523] Scripts and configs supporting new ECS deployment model

Updates the DB migrations script
Replaces ecs-deploy with new script to deploy the app container
Updates CircleCI config to use new scripts

Important
This depends on rolling out the new infrastructure. Currently, this runs agains an experimental environment (called "experimental"). Once the new staging environment is rolled out, this can be updated to run against staging and merged in.

I missed your comment about the experimental testing. So ignore those comments


tsconfig paths and module resolution errors in VS Code

Bug Report or Feature Request (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request

Versions.

@angular/cli: 1.4.9
node: 7.9.0
os: darwin x64
@angular/animations: 4.4.5
@angular/common: 4.4.5
@angular/compiler: 4.4.5
@angular/core: 4.4.5
@angular/forms: 4.4.5
@angular/http: 4.4.5
@angular/platform-browser: 4.4.5
@angular/platform-browser-dynamic: 4.4.5
@angular/router: 4.4.5
@angular/cli: 1.4.9
@angular/compiler-cli: 4.4.5
typescript: 2.3.4

macOS Sierra 10.12.6
VS Code 1.17.2
Repro steps.

My tsconfig path property seems broken.
When I replace my path mapped imports (@shared, @assets etc.) with the equivalent coded relative paths, typescript module resolution seems to work (vs code intellisense is happy)
I have a large angular 2 project and am used to aliasing my asset and shared directories this way.
NOTE: project builds and (seems) to run just fine
This has been reported often, but I can't make sense of the current state of this issue.
This more than annoying as VS Code reports Angular spurious errors on all of my components.
I also suspect this is why I get the infamous Can't read property isSkipSelf of null when I open my templates (html) in VS Code.
I was running 1.4.2 and just tried ~1.4.2, which resolves to 1.4.9 in my package.json, to no avail.
Will 1.5.x fix this?
How can I help debug this without providing a copy of my app?
Thanks in advance. Regards.
The log given by the failure.

Desired functionality.

Mention any other details that might be useful.

Can you setup a minimal repro please? You can read here why this is needed. A good way to make a minimal repro is to create a new app via ng new repro-app and adding the minimum possible code to show the problem. Then you can push this repository to github and link it here.

I am not the OP, but here is a sample repo showing a component imported via the path alias displaying the "Cannot find module" error in vscode, but successfully building with npm run build --prod.
https://github.com/Veabers/tsconfig-paths-repro-app

vscode seem to only look into nearest tsconfig.json file, while angular read your tsconfig.app.json You should add relative path config to both tsconfig.json and tsconfig.app.json.

I am having this issue as well and I am unable to install Typescript Hero as access is restricted from where I work.
For me I only get the issue in test specs.
In app.module.ts this is fine:
import { AppRoutingModule } from '@app';
import { AppComponent } from '@app';
import { SharedModule } from '@shared';

but in app.component.spec.ts I get the error here:
import { AppComponent } from '@app';

[ts] Cannot find module '@app'.
I have 3 tsconfig files:
tsconfig.json in src:
{
    "extends": "../tsconfig.json",
    "compilerOptions": {
        "outDir": "../out-tsc/app"
    },
    "exclude": [
        "test.ts",
        "**/*.spec.ts"
    ]
}

tsconfig.spec.json in src:
{
  "extends": "../tsconfig.json",
  "compilerOptions": {
    "outDir": "../out-tsc/spec",
    "types": [
      "jasmine",
      "node"
    ]
  },
  "files": [
    "test.ts"
  ],
  "include": [
    "**/*.spec.ts",
    "**/*.d.ts"
  ]
}

and tsconfig.json in root:
{
    "compilerOptions": {
        "baseUrl": "src",
        "paths": {
            "@app": [
                "app/"
            ],
            "@ui": [
                "app/ui/"
            ],
            "@shared": [
                "app/modules/shared/"
            ]
        }
    }
}

I am not sure if vscode pays attention to the exclude or include params but that doesn't matter, even if it does not consider either of the tsconfig.json files in src it should just go to the parent folder and use tsconfig.json in the root folder - which is where the paths are.

Has there been any progress on this? Probably a VS Code issue rather than an Angular one. I can still replicate with the latest versions of each though. New cli project, trying to alias my src/app/ imports so they can be absolute (e.g. I want to be able to import @app/users/models/user.models.ts anywhere rather than ../../../users/models/user.models.ts with different relative paths where it is used).
I've tried adding
"paths": {       "@app/*": ["src/app/*"]     }
to the root tsconfig.json and it compiles but VS Code still shows a 'cannot fine module' error on the absolute import

I had the same issue but it seams it was resolved with using a more "explicit" relative path in baseUrl in tsconfig.ts
Before:
{
    "compilerOptions": {
        "baseUrl": "src",
        "paths": {
            "@app": ["app/"],
        }
    }
}

After:
{
    "compilerOptions": {
        "baseUrl": "./src",
        "paths": {
            "@app": ["app/"],
        }
    }
}

@johankvint, your solution of using ./src worked for me. I no longer need to use the TypeScript Hero extension either.
Something to watch out for if you use barrel files there is different setup. Let's suppose you have a guards folder in your app and your usual import is import * as fromGuards from '../../../guards';. I have found that I need to setup the following compilerOptions in order to get the desired effect. I'm not pleased with having to duplicate so much of the path, so please let me know if anyone has a better way to handle the barrels.
{
    "compilerOptions": {
        "baseUrl": "./src",
        "paths": {
            "@app/guards": ["app/guards/"],
            "@app/guards/*": ["app/guards/*"]
        }
    }
}

If you have app/guards/index.ts and a LoginGuard component you should now have the following options to import.
import * as fromGuards from '@app/guards';

or
import {LoginGuard} from<EMAIL_ADDRESS>

@johankvint, do you mean that can work for barrel files? I receive a Cannot find module compiler error when trying to perform my imports.
Now, it does work if I explicitly import the index file... Maybe it is better to be more explicit anyways.

@DerekSchmid Yes, my mistake. I just checked my project and I do not have any import * as .... from a barrel using the paths in tsconfig, only relative to the code.

This works for me:
:wrench:  Move (or repeat) paths section from tsconfig.app.json to tsconfig.json

The basic problem here is that by default when the tools in TypeScript toolset try to determine which compiler configuration to use with any given file, they look for a file named tsconfig.json starting in the directory where the TS file is located. If it is not there, they recursively looks up the file system until they find a tsconfig.json file. They do not look for tsconfig.app.json, tsconfig.spec.json or tsconfig.lib.json or any other name.
The default can be overridden. Presumably, that's what angular-cli does when it invokes the TypeScript compiler or tslint. It uses -p or an API equivalent to pass it's specially named configuration files.
However, this falls apart when it comes to getting editors to automatically recognize those files. TypeScript editors that provide complete support for TypeScript, like VSCode, usually launch a tsserver instance. And tsserver being part of the TypeScript toolset, does its default thing when the editor request that it processes a file: it looks for a file named tsconfig.json. I've used tsserver in dozens of projects (including Angular projects that don't use angular-cli), and got it to find the proper tsconfig.json in all cases. (Of course the project has to be structured to take into account the default behavior of the TS toolset.) It only fails in projects that use angular-cli to generate configuration, due to the non-standard configuration file names that angular-cli uses.
An issue with the TypeScript project was opened here but it does not look like any modifications will be made to the TS toolset to handle angular-cli's unusual naming convention. I've not found a newer issue that indicates otherwise.

@Roaders: The reason it doesn't work in *.spec.ts files is because they are excluded by the tsconfig.js exclude property by default. Removing *.spec.ts from the exclude array would make it work. However I don't know what the exact consequences are when doing so.

I had to set module_resolution in compiler options to get this to work for me
{
    "compilerOptions": {
      "moduleResolution": "node",
      "baseUrl": "./",
      "paths": {
        "@/*": ["src/*"]
      }
}

vscode seem to only look into nearest tsconfig.json file, while angular read your tsconfig.app.json You should add relative path config to both tsconfig.json and tsconfig.app.json.

Neither this

This works for me:
Move (or repeat) paths section from tsconfig.app.json to tsconfig.json

Nor this is a valid solution for the problem for the current "monorepo"-approach.
If we have multiple projects in one repo, the determination for a "project" in the context of angular is done by tsconfig files. Now imagine that you have 3 projects A & B & C, while B & C being a library project. As a good developer you want a clean separation between your concerns in such architecture / structure:

Project A depends on B & C, therefore it has a mapping to C and to B
Project B knows nothing about C & A
Project C knows nothing about B & A

You are going to solve this by adding the mapping into the tsconfig of the Project A and not in the root tsconfig of the repo, otherwise you would create a relation/mapping what you actually don't want..
We too suffer from this bug. At first I was thinking this is a bug in Webstorm, but after discovering this thread things got interesting.

Yeah we've the same problems, we've added two angular projects in the same repo and we've two different tsconfig files, because different apps require different alias.
the app is compiling and running fine but VSCode highlight every import in red, and the intellisense for paths is broken

I don't get it. You can also change also the angular.json directly on the fs + one could just extend the json schema to reference the one from tsconfig.
Can you give me an example for another tool that uses e. g. tsconfig.app.json etc?
In my example it was intended to leave at least one tsconfig.json and hide the additional ones, because even TypeScript itself has no support for other naming patterns

I often run tsc -p src/tsconfig.app.json myself, for instance. Or ngc -p src/tsconfig.app.json. You could run other tools that take typescript configuration (like tslint) by pointing it at the file as well.

Okay, but the tool devs have to provide such functionality, nothing out of the box. Can you tell
The feedback I'm receiving pretty often is that it's not clear which of those files is now being used

Okay, but the tool devs have to provide such functionality, nothing out of the box. Can you tell Vscode to use a particular tsconfig for its autocomplete? Or can you configure the angular Lange service to use another tsconfig? Even if your IDE can handle other naming patterns, how does it know which one to use for a specific folder, if there are multiple tsconfigs. The one for unit tests also include the origin files...
The feedback I'm receiving pretty often is that it's not really clear which of those files is now being used from the IDE while you are developing / doing other tasks.

Keep it simple,  if the vscode found two tsconfig in immediate parent
folder,  just ask which one to use. And save it in the .vs folder.
Later if user want to change the tsconfig, they should be able to do so by
some setting,  better if it be a right click menu on folder viewer
On Sun, Feb 3, 2019, 19:17 kqadem<EMAIL_ADDRESS>wrote:

Okay, but the tool devs have to provide such functionality, nothing out of
the box. Can you tell Vscode to use a particular tsconfig for its
autocomplete? Or can you configure the angular Lange service to use another
tsconfig? Even if your IDE can handle other naming patterns, how does it
know which one to use for a specific folder, if there are multiple
tsconfigs. The one for unit tests also include the origin files...
The feedback I'm receiving pretty often is that it's not really clear
which of those files is now being used from the IDE while you are
developing / doing other tasks.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/angular/angular-cli/issues/8138#issuecomment-460042919,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIk0J2eq-cKFKwnky2IkG1pWih3TbTZWks5vJsVNgaJpZM4QBesR
.

I was having troubles with absolute and relative URLs myself; ended up having following instructions at https://hackernoon.com/react-pro-tip-use-absolute-imports-for-better-readability-and-easy-refactoring-2ad5c7f2f957 which completely fixed the issues.
TL;DR:
Make a .env file in project root and add this:
NODE_PATH=src/
Then make a jsonconfig.json file in project root and add this:
{
  "compilerOptions": {
    "baseUrl": "src"
  }
}

I'm actually in a React codebase and don't even use typescript in it, yet setting the "javascript.implicitProjectConfig.checkJs": true in my vscode settings gives me all the intellisense I need with just these two changes!

There are many folders in my VSCode workspace inspector, and VSCode may use the first folder's tsconfig.json. If your tsconfig.json is not in the first folder, drag the folder to first order and try it, it worked for me.

As there is no clear solution, I will past my working one:
folders:
-root
-tsconfig.json #the package config
--src #the source of my package
--demo
--tsconfig.json #the storybook config referencing my upper module as "@sproutch/ui"

Important note
I had to remove "rootDir": "src", from the root package config to make it work.
Here are the full files.
 It's described in https://devblogs.microsoft.com/typescript/announcing-typescript-3-9-beta/ as follows:

Support for “Solution Style” tsconfig.json Files
...
This file that really does nothing but manage other project files is often called a “solution” in some environments. Here, none of these tsconfig.*.json files get picked up by the server, but we’d really like the language server to understand that the current .ts file probably belonds to one of the mentioned projects in this root tsconfig.json.
TypeScript 3.9 adds support to editing scenarios for this configuration. For more details, take a look at the pull request that added this functionality.

Discussed this in a meeting earlier and tsconfig "solutions" can be a great answer to this problem. We'll do some more investigation to see exactly how this can work and what release we can fit it into.

Being tracked internal TOOL-1368


Update docs for v5.0

We need to update the docs (and publish to website) for the v5 release.

A couple things I noticed during skimming:

Need to remove the "ordered globs" docs
Need to update configuration options to vinyl-fs functions

Oh my gosh, I thought you guys had given up on gulp.

Encoding option added: https://github.com/gulpjs/gulp/blob/master/docs/api/src.md#options

"Explaining Globs" docs have been updated and a section added with an example for ordered globs using ordered-read-stream: https://github.com/gulpjs/gulp/blob/master/docs/getting-started/6-explaining-globs.md#ordered-globs

Encoding option added: https://github.com/gulpjs/gulp/blob/master/docs/api/src.md#options

Which should be classified as a breaking change, when copying binary files (#2790, #2796).

@Sebazzz It was a breaking change - its why we released a new major release, and its in the changelog as a breaking change for that major release.

@Sebazzz It was a breaking change - its why we released a new major release, and its in the changelog as a breaking change for that major release.

Look at it from an end-user perspective:

Default stream encoding to UTF-8

That truly doesn't mean anything to an end user.

@Sebazzz We are working on improving the docs - the website not being updated for v5 was an oversight and we are working on it as we speak. The docs in the repo have been updated on gulp and vinyl-fs but I know people look at the website first. More info here: https://github.com/gulpjs/gulp/issues/2790#issuecomment-2062002387

I actually didn't know there was a v5 until I manually checked the version and did a web search for "gulp 5".

Any progress ? It's been nearly 6 months and the docs is still not up to date.
We just debugged an issue where our fonts were not working anymore when copying from src to dest using gulp-copy until we found #2796 which links to #2790.
A migration guide would be nice.

The markdown docs have been updated and redeployed to the website so I'm going to close this.
As the major v4 release didn't contain a migration guide, we're not considering it required for v5 either. If anyone from the community wants to contribute further docs, like a migration guide, we're happy to review PRs!


Metadata Editing/Permissions

resolves #166
resolves #155
Note metadataKeys is the container that holds the format of what data is inside of all the metadata.  It indicates data types and categorical lists of possible values as well as ranges for numerical values.
Done:

unlocked index for metadataKeys to indicate what metadata can be modified by any user
Ability to add and remove keys from metadataKeys
Ability to update a key value and have it update in the metadataKeys

TODO:

[x] Endpoints for deleting, creating, modifying metadata information
[x] Testing system and permissions interface to understand how it works.
[x] Add filename (video filename) to the list of items that are displayed as well.
[x] Automatically add some ffprobe information to the metadata generation (width, height, length)
[ ] UI Creation for deleting/adding new metadatakeys to a metadata folder
[ ] UI for modifying key values if they are unlocked and displayed.

I have built and tested the endpoints for:

Adding a DIVE Metadata folder by pointing it to another folder and recursively fetching items and creating a dive dataset
Added in DIVE_DatasetId and DIVE_Name to the values automatically added to a dive dataset.  These can be hidden by default if required
Tested adding a new key to the DIVEMetadata, modifying permissions for it to be unclocked and then updating that key by the DIVEDataset FolderId.

TODO:

[ ] Implement endpoints in divemetadata.service.ts (Lock/unlock, delete, add, endpoints)
[ ] Add in edpoints for a user editing a field
[ ] Add in dialog for adding new metadata field.
[ ] Update metadata Viewer so that users if they have write access to the folder can modify the metadata value.


Add new auction expired webhook

Add new documentation for the webhook auction expired

Released in last version of the package. Good to merge.


[#141394313] Upgrade ipsec release to disable stats logging

What
Reduce the amount of logs generated by racoon which are filling up our
logsearch cluster. Depends on alphagov/paas-ipsec-release#2
How to review

Observe the vast amount of logs from racoon on the cell VMs, either:

in logsearch with query @source.component:racoon, but I had problems in my dev environment
in /var/log/syslog on the VM itself

Deploy from this branch
Observe the vast reduction in logs from racoon in the same places as previous

Who can review
Not @dcarley

I've rebased with the final build.


Revert yarn.lock

Ticket
PLAT-11400
Description
In the previous commit #142 , yarn.lock file was mistakenly modified with downgraded versions. This commit sets it to the previous version from origin/master

are we supposed to use yarnpkg or npmjs repository?


Update azure-ad-rbac.md

@Tonyette : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Looks good; thanks, @Tonyette


Fix next prev

fixes next/prev page for @JoanaMMoreira
removes snapshot api for @DavieReid

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 62.842%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
1254

Relevant Lines:
1740

💛  - Coveralls

good shout @lilyvc it is not supported by the plugin yet but we can add... I'll follow up with another PR for that


[spark] Fix reported statistics does not do column pruning

Purpose

SupportsReportStatistics#estimateStatistics should report the statistics after do column pruning, filter push down. This pr fixes it does not do column pruning. In addition, we should consider the metadata column size.
Introduce defaultSize method for DataType, so that we can get the size in bytes without column statistics.

Tests

add test
API and Format

no
Documentation

cc @JingsongLi @Zouxxyy thank you

@Zouxxyy We may need to run a TPC-DS test to show the compact.

thank you @Zouxxyy. I tested with 20 columns and using explain cost to see the sinzeInBytes after doing analyze, it does affect:
without this pr:
explain cost select c1, c2, c3 from w_t;

== Optimized Logical Plan ==
RelationV2[c1#163, c2#164, c3#165] default.w_t, Statistics(sizeInBytes=36.8 MiB, rowCount=1.00E+5)

explain cost select * from w_t;

== Optimized Logical Plan ==
RelationV2[c1#194, c2#195, c3#196, c4#197, c5#198, c6#199, c7#200, c8#201, c9#202, c10#203, c11#204, c12#205, c13#206, c14#207, c15#208, c16#209, c17#210, c18#211, c19#212, c20#213] default.w_t, Statistics(sizeInBytes=36.8 MiB, rowCount=1.00E+5)

with this pr:
explain cost select c1, c2, c3 from w_t;

== Optimized Logical Plan ==
RelationV2[c1#5, c2#6, c3#7] default.w_t, Statistics(sizeInBytes=5.5 MiB, rowCount=1.00E+5)

explain cost select * from w_t;

== Optimized Logical Plan ==
RelationV2[c1#36, c2#37, c3#38, c4#39, c5#40, c6#41, c7#42, c8#43, c9#44, c10#45, c11#46, c12#47, c13#48, c14#49, c15#50, c16#51, c17#52, c18#53, c19#54, c20#55] default.w_t, Statistics(sizeInBytes=36.8 MiB, rowCount=1.00E+5)

In general, the sinzeInBytes would affect broadcast join, runtime filter, etc. So it can happen that, nothing changes with pr if the query does not hit the related case.

+1


trainning code!

is there any progress in releasing code?

I had the same problem, please did you find it now？

I had the same problem, please did you find it now？

I'm afraid not. :(

Can we discuss this issue, I am also from China, if it is convenient please add my WeChat: lhw2000915
Best wishes.
------------------ 原始邮件 ------------------
发件人:                                                                                "Annbless/DUTCode"                                                                                @.>;
发送时间: 2022年11月18日(星期五) 中午12:17
@.>;
<EMAIL_ADDRESS>主题: Re: [Annbless/DUTCode] trainning code! (Issue #21)
I had the same problem, please did you find it now？
I'm afraid not. :(
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

Hi,
There is an initial version of the training code link. It is temporally not suitable to be made public here since it lacks a readme file and proper comments but can be used as a reference. The final code will be uploaded after the cleaning job.

Hi,
There is an initial version of the training code link. It is temporally not suitable to be made public here since it lacks a readme file and proper comments but can be used as a reference. The final code will be uploaded after the cleaning job.

Hello, I'm having some trouble running your training code, can you give the format of train_kp_1.npy, train_op_1.npy. file

Hi,
There is an initial version of the training code link. It is temporally not suitable to be made public here since it lacks a readme file and proper comments but can be used as a reference. The final code will be uploaded after the cleaning job.

Hello, I'm having some trouble running your training code, can you give the format of train_kp_1.npy, train_op_1.npy. file

Hello, do you currently have any code for metric calculations? If so, would you be willing to share it? My email is<EMAIL_ADDRESS>I'm looking forward to your reply.

Hi,
There is an initial version of the training code link. It is temporally not suitable to be made public here since it lacks a readme file and proper comments but can be used as a reference. The final code will be uploaded after the cleaning job.

Hello, I'm having some trouble running your training code, can you give the format of train_kp_1.npy, train_op_1.npy. file

I had the same problem while training the code. How did you solve it

您好，这里有训练代码链接的初始版本。暂时不适合在这里公开，因为它缺少自述文件和适当的注释，但可以用作参考。最终代码将在清洁工作后上传。

您好，我在运行您的训练代码时遇到了一些问题，您能否给出 train_kp_1.npy 的格式，train_op_1.npy。文件

我在训练代码时遇到了同样的问题。你是怎么解决的

Hello, I also encountered problems in training. Does anyone know how to train?


Unit tests for data-binding stuff

All of the changes to support data binding should be unit tested at least at some level.  I believe this project already is using py.test.

Closing for now :)


Implement rate limiting middleware to prevent attacks

Routes like /login are already protected, but request spamming could compromise security in other ways such as with password reset or run up a high bill with the CouchDB provider. I would like to implement rate limiting middleware such as express-brute in order to mitigate such risks.

Absolutely!  I also figure that checkUsername and checkEMail will be prime candidates that need exponential rate limiting. non-trivial for these two.

You might also be interested in this one: https://github.com/tj/node-ratelimiter


missing dollar bar source file

@rongzhou thanks for raising an issue, could you describe in more detail what you were trying to do?

At first I want to know if you create triple barrier label from tick bar data or dollar bar like I assumed. I find in test directory, there are dollar_bar_dataframe.pkl and tick_dataframe.pk files. I think if you use dollar bar, there should have some source code to create it.

yes it is possible to create triple barrier labels with the respective transformer (https://github.com/MizarAI/mizar-labs/tree/master/mizarlabs/transformers/targets). We are currently not releasing the source code of the volume/dollar/tick bar, but we are making the volume/dollar/tick bar data publicly available soon for multiple trading pairs.


Add tomopy and skimage requirement for CIL-Demos

Description

Add requirement of tomopy and skimage to CIL-Demos install instructions
Add requirement of tomopy to docker instructions
These are required for the new pagainin notebook and the new callbacks notebook
Example Usage

Changes
Testing you performed

Please add any demo scripts to https://github.com/TomographicImaging/CIL-Demos/tree/main/misc

Related issues/links
Closes #1956
Checklist

[ ] I have performed a self-review of my code
[ ] I have added docstrings in line with the guidance in the developer guide
[ ] I have updated the relevant documentation
[ ] I have implemented unit tests that cover any new or modified functionality
[ ] CHANGELOG.md has been updated with any functionality change
[ ] Request review from all relevant developers
[ ] Change pull request label to 'Waiting for review'

Contribution Notes
Please read and adhere to the developer guide and local patterns and conventions.

[x] The content of this Pull Request (the Contribution) is intentionally submitted for inclusion in CIL (the Work) under the terms and conditions of the Apache-2.0 License
[x] I confirm that the contribution does not violate any intellectual property rights of third parties

--->

Exactly. The ONLY correct way to do this is:

drop this entire PR
open another PR to remove all the other cruft from README & requirements-test.yml
open a PR on CIL-Demos & CIL-User-Showcase adding root environment.yml
open a PR on CIL to make the Dockerfile use conda install -n base -f https://raw.githubusercontent.com/TomographicImaging/CIL-{Demos,User-Showcase}/refs/heads/main/environment.yml

Tomopy is no longer required so updating this PR to just add skimage to the CIL-Demos install instructions and I will open another issue with Casper's suggetions

@casperdcl  thanks for your suggestions, hopefully this covers everything: https://github.com/TomographicImaging/CIL/issues/1961
We need a wider discussion /  agreement before removing the CIL-Demos install instructions from the CIL readme, so I still need to add skimage to the install instructions in this PR


Missing labels in charts

The labels are missing from the charts. There were originally supported when I designed the tool (I used the datalabels plugin of Chart.js).

Here an example when labels worked

Others examples are available in that old article: https://dev.to/nodesecure/nsecure-security-report-19ln

I want to work on it, Please assign to me.

nreport initialize show me this error /usr/bin/env: ‘node --no-warnings’: No such file or directory
And more over couldn't figure out how to start the project locally.

@CabdiraxiimAxmed hard to help without much info, maybe an issue with the --no-warnings flag if you'r local env fail to detect it?


use serviceaccount generate kubeconfig

Signed-off-by: calvin<EMAIL_ADDRESS>/kind feature
If we want open a current cluster terminal and create a configmap manually is complicated. we can use serviceaaccount in pod of controller  to generate kubeconfig and create confingmap for it. In the case, user specified a configmap to cloushell no more.
@panpan0000 cc

@panpan0000 PTAL

@panpan0000 PTAL


Hongyuan yan

I have no idea what this is supposed to be. Please provide some motivation for what this should be doing :)


TEST: demo require.js conflict

The PR fixing this has been merged in the meantime, this can be closed


Add EnvPool support (as VecEnv subclass)

Description

Based on #307, but use VecEnv subclass instead of VecEnvWrapper.
VecEnvWrapper assumes the wrapped env of an instance of VecEnv, which is not the case of EnvPool. IMO, using the VecEnv class directly is therefore a better option, and the implementation is not more complex.
Motivation and Context

[ ] I have raised an issue to propose this change (required for new features and bug fixes)

closes #241
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation (update in the documentation)

Checklist:

[ ] I've read the CONTRIBUTION guide (required)
[ ] I have updated the changelog accordingly (required).
[ ] My change requires a change to the documentation.
[ ] I have updated the tests accordingly (required for a bug fix or a new feature).
[ ] I have updated the documentation accordingly.
[ ] I have reformatted the code using make format (required)
[ ] I have checked the codestyle using make check-codestyle and make lint (required)
[ ] I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

Here are some speed results with Breakout and PPO. I can't understand why envpool is only slightly faster than dummy vec env. @araffin, @Trinkle23897 any insight?

3897, any insight on this?

@araffin what do you think about moving this to sb3 (insert it to make_vec_env)?
And make envpool an optional dependency of sb3.

what do you think about moving this to sb3 (insert it to make_vec_env)?

not sure about it.
My main concern is that it only supports half the thing that a VecEnv can do (notably, it doesn't support gym wrappers).
I think it would make more sense in sb3 contrib or here in the RL Zoo (especially as it is a package now, so people can re-use it easily if needed).

My main concern is that it only supports half the thing that a VecEnv can do

Makes sense, we will need to update the sb3 doc btw.

@araffin, before merging, we need to explain this https://github.com/DLR-RM/rl-baselines3-zoo/pull/355#issuecomment-1424556165

Here are some speed results with Breakout and PPO. I can't understand why envpool is only slightly faster than dummy vec env. @araffin, @Trinkle23897 any insight?

So, it seems to depends on the env, but yes it is surprising.
I also have envpool slower than SubProcVecEnv on my machine with python 3.7 (but not as slow as reported here).
But on MuJoCo env, envpool is the fastest for me:
Envpool: ~4100 fps
CUDA_VISIBLE_DEVICES= OMP_NUM_THREADS=2 python -m rl_zoo3.train --algo a2c --env HalfCheetah-v3 --verbose 0 --eval-freq -1 -P -envpool -params n_envs:4

vs
Subproc: ~3400 fps
CUDA_VISIBLE_DEVICES= OMP_NUM_THREADS=2 python -m rl_zoo3.train --algo a2c --env HalfCheetah-v3 --verbose 0 --eval-freq -1 -P -params n_envs:4 --vec-env subproc

vs
Dummy: ~3600 fps
CUDA_VISIBLE_DEVICES= OMP_NUM_THREADS=2 python -m rl_zoo3.train --algo a2c --env HalfCheetah-v3 --verbose 0 --eval-freq -1 -P -params n_envs:4 --vec-env dummy

Can you use py-spy to have a profiling result?

Can you use py-spy to have a profiling result?

This is for sub process:

For envpool, I think py-spy had a segfault:


Fixed #34788 -- Corrected NUMBER_GROUPING for Japanese.

Fixed ticket #34788.
correct NUMBER_GROUPING in Japan.

@tell-k Thanks :+1: Welcome aboard :boat:


Cow: Watch totalgaminghub.com

Cow requests the watch of the watch_keyword totalgaminghub\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

totalgaminghub\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by cocomac in Charcoal HQ


Nominating 3 new peers

Hi owners @Fyrd @dfabulich @ddbeck @Elchi3 @jgraham @atopal @LeoMcA!
I'd like to nominate 3 new peers, my colleagues @kosamari @petele @tropicadri who have recently contributing to web-features and have plans to keep going to at least 100 new features.
Write access would be convenient so that we can more easily have shared branches in this repo and sort out merge conflicts, which we have recently been creating...
Per https://github.com/web-platform-dx/web-features/blob/main/GOVERNANCE.md#peers we vote to confirm new peers. There are 8 owners, so 5 is a majority. I hereby the first vote in favor :)

Same here! +1

Did you by any chance mean @kosamari ?

Did you by any chance mean @kosamari ?

Yes, edited! Thanks!

Works for me. 👍

This seems fine to me.

Works for me 👍
Welcome to our new Peers!

👍

I've merged the docs PR and sent GitHub invites to the new peers. 🎉


Duplication of entries in HLTB website when setting playtime

When a game has been closed and the option is set to automatically upload the playtime to user's HLTB account, instead of updating the already pre-existing entry, a new entry is created of the same game with the updated playtime, leading to several duplicate entries on the site.
Not all games are affected all the time, but unsure how to replicate the issue. Could be because the game entry was already created outside the extension's help.

Try to delete duplicate data in HowLongToBeat.
After, you must refresh your user data in Plugin.
And try again.

So what I did is, deleted all entries from HowLongToBeat of the affected game and then refresh data in playnite. But the problem still persists. Then I clear data from HLTB addon just to see if that helps, but still no luck, duplicates still exists.


Item Display Names On %item% (If Applicable)

Is your feature request related to a problem? Please describe.
Take the color codes and formatting of the item names you trade that will overwrite the name in %item% as right now it displays as  , so either a setting to display only the custom name if applicable or similar.
Describe the solution you'd like
Take the color codes and formatting of the item names you trade that will overwrite the name in %item% as right now it displays as  , so either a setting to display only the custom name if applicable or similar.
Describe alternatives you've considered
Not Applicable
Additional context
Not Applicable
Discord contact: Khrys#2160

It already displays e.g. "+1 Iron Sword (DISPLAY NAME)".

The suggestion consists in adding an option to disable the first part of it, only allowing it to display the display name of the item (if applicable), or the item id name if not applicable.

@erikzimmermann


New member of Information Management and Support Team

Following a decision of the governance panel (by email) I have prepared PR #442 to add Ellie Fisher to the list of members of the Information Management and Support Team on the governance webpage. Thanks for joining the team, Ellie @efisher008.
Please could a member of the panel or the committee check and merge the PR.

Thanks Jonathan @JonathanGregory ! Looks good. I have merged. I will also add/invite Ellie to the GH info-mgmt team.

Hello @JonathanGregory and @ethanrd,
Thank you for your work on this! I'm looking forward to contributing to the CF community.
Best,
Ellie


Clone ocmock for ios unit tests

relevant file //flutter/shell/platform/darwin/ios/BUILD.gn
We might not have to clone it on our own because it exists at https://chromium.googlesource.com/chromium/src/+/master/third_party/ocmock
That build file builds a source_set though and we need a static_library.  We might be able to make a static_library based on the source_set.
Optionally we can keep an external BUILD.gn if we put ocmock's in //build/secondary/third_party/ocmock/BUILD.gn.

Unfortunately https://chromium.googlesource.com/chromium/src/+/master/third_party/ocmock isn't a repo, the repo is all the code for chromium.  There doesn't seem to be a way to just pull down a subdirectory in gclient while it seems hypothetically possible with git: https://stackoverflow.com/a/52269934

ah I see. Ya, that makes it trickier. https://stackoverflow.com/a/60729017/167638 works too but yes, it takes away from the convenience of the option.


Equation numbering based on backlinks, not block ids.

Maybe the Dataview API will be helpful

There are two options:

Modify auto-index mechanism.

Remove blockIDs that aren't referenced.

is computationally reasonable, but bay be dangerous


配置 vue({ reactivityTransform: false }) 后报错

"@tencentcloud/chat-uikit-vue": "^1.4.1",
[@vue/reactivity-transform] Reactivity transform is an experimental feature.
Experimental features may change behavior between patch versions.
It is recommended to pin your vue dependencies to exact versions to avoid breakage.
You can follow the proposal's status at https://github.com/vuejs/rfcs/discussions/369.

$ can only be used as the initializer of a variable declaration.
11:10:36 [vite] Internal server error: $ can only be used as the initializer of a variable declaration.
  Plugin: vite:vue

这属于您自己的工程逻辑问题，可以向vue官方寻求解决方案，reactivityTransform属于已废弃实验特性🙇，敬请谅解


feat(cli, generate command): Added promo array to rotate and make cli messaging more engaging

Included array of promotions
function to handle promos and show a random small and discrete text on each prisma generate run
modifications to prisma init to remove the pdp promotion and add a 5th step instead

closes: https://github.com/prisma/team-orm/issues/1243

Hm, something doesn't seem to be rendering quite right @apolanc
I ran generate a few times and managed to get this result too which shouldn't be possible from the looks of the code

Thanks @Druue I just pushed a change that fixes this behaviour. I was just calling the random function twice random().text random().link [facepalm]. All should be good now

Is the gap at the bottom intended? It looks a little off @apolanc

Is the gap at the bottom intended? It looks a little off @apolanc

No. I'll push a change for it now.


Choose a documentation system

Method A: mdbook
e.g. https://github.com/fzyzcjy/flutter_rust_bridge
Method B: docusaurus https://docusaurus.io/
from facebook, uses react
e.g. https://mobx.netlify.app/
e.g. https://riverpod.dev/
e.g. https://dartcodemetrics.dev/docs/getting-started/introduction
support auto deploy: https://docusaurus.io/docs/deployment#triggering-deployment-with-github-actions
Method C: https://vuepress.vuejs.org/
but it uses vue, while flutter is more like react-style
Method D: github raw md
e.g. https://github.com/tekartik/sqflite/blob/master/sqflite/doc/opening_db.md
e.g. https://github.com/lejard-h/chopper/blob/master/converters/converters.md
Method E: https://docsify.js.org
e.g. https://bloclibrary.dev/#/gettingstarted
Method F: self-written
e.g. https://drift.simonbinder.eu/docs/getting-started/
Method G: put in a huge readme.md
e.g. https://pub.dev/packages/pigeon
e.g. https://github.com/rrousselGit/freezed

#86


fix several bugs

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Reviewer - this PR has made changes to one or more package.json files.

Snapshot stored with reference name:
refs/pull/15751/merge
Test environment:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15751/merge/index.html
To test a playground add it to the URL, for example:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15751/merge/index.html#WGZLGJ#4600
Links to test babylon tools with this snapshot:
https://playground.babylonjs.com/?snapshot=refs/pull/15751/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/15751/merge
https://gui.babylonjs.com/?snapshot=refs/pull/15751/merge
https://nme.babylonjs.com/?snapshot=refs/pull/15751/merge
To test the snapshot in the playground with a playground ID add it after the snapshot query string:
https://playground.babylonjs.com/?snapshot=refs/pull/15751/merge#BCU1XR#0

Visualization tests for WebGPU (Experimental)
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15751/merge/testResults/webgpuplaywright/index.html

WebGL2 visualization test reporter:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15751/merge/testResults/webgl2playwright/index.html

Reviewer - this PR has made changes to one or more package.json files.

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

Reviewer - this PR has made changes to one or more package.json files.

Reviewer - this PR has made changes to one or more package.json files.

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.

Reviewer - this PR has made changes to one or more package.json files.

Reviewer - this PR has made changes to the build configuration file.
This build will release a new package on npm
If that was unintentional please make sure to revert those changes or close this PR.


classic only build not dealing with filters/quantization well

For a classic only build there should be no filters or quantization.
# General
-------
NetCDF Version:		4.9.4-development
Dispatch Version:       5
Configured On:		Thu Sep 12 06:47:50 MDT 2024
Host System:		x86_64-pc-linux-gnu
Build Directory: 	/home/ed/netcdf-c
Install Prefix:         /usr/local/netcdf-c-classic
Plugin Install Prefix:  /usr/local/hdf5/lib/plugin

# Compiling Options
-----------------
C Compiler:		/usr/bin/gcc
CFLAGS:			 -fno-strict-aliasing
CPPFLAGS:
LDFLAGS:
AM_CFLAGS:
AM_CPPFLAGS:
AM_LDFLAGS:
Shared Library:		yes
Static Library:		yes
Extra libraries:	-lm -lz -lsz -lzstd -lxml2 -lcurl
XML Parser:             libxml2

# Features
--------
Benchmarks:		no
NetCDF-2 API:		yes
HDF4 Support:		no
HDF5 Support:		no
CDF5 Support:		yes
NC-4 Parallel Support:	no
PnetCDF Support:	no

DAP2 Support:		no
DAP4 Support:		no
Byte-Range Support:	no

S3 Support:	        no
S3 SDK:  	        none

NCZarr Support:		yes
NCZarr Zip Support:     no

Diskless Support:	yes
MMap Support:		no
ERANGE Fill Support:	no
Relaxed Boundary Check:	yes

Plugins Enabled:        yes
Plugin Install Dir:     /usr/local/hdf5/lib/plugin

Quantization:		yes
Logging:     		no
SZIP Write Support:     yes
Standard Filters:       bz2 deflate szip zstd
ZSTD Support:           yes
Parallel Filters:       no

I will submit a fix shortly...

Whether or not there should be quantization is debatable. It's perfectly well defined and supportable by classic format, it's just that there is not any known use. I think we discussed whether classic should support quantization before, and I remember concurring that it is OK to turn it off for classic. My views have evolved since then. As long as it's working for classic, no reason to turn it off in my mind. YMMV. In any case, NCO supports quantization for classic via it's internal algorithms.

It does not work for classic.
Putting it in for classic would require diving into the libsrc code, which is not something I would choose to do without great need.
Of course it makes sense for classic, and would even provide benefit: a quantized classic file would gzip to a much smaller size. My only concern here is that currently netcdf-fortran fails for classic-only netcdf-c library because quantize constants are missing. Obviously we want a classic only build to work.
So my thought was to detect classic only builds, and then to set quantization to "no" (since there is no quantize for classic only). Then netcdf-fortran could see that setting and know not to include the quantize constants.

OK, nevermind me, I misinterpreted your original message. Sounds like we're in agreement on the hypothetical value of having quantization in classic files.


Question - Image Size

Hi,
I'm busy adding in slave functionality into a pull request. I'm not sure why but when I build the image based on the included dockerfile its 498mb compared to yours on dockerhub which is 128mb. Any idea why there is such a massive difference in image size?

There are two Dockerfiles—Dockerfile, and ci/Dockerfile. The top-level one is used for development, and contains all the things necessary for compiling, etc. The ci/ one is used by our CI system at Pavouk, and copies across the compiled binaries only onto a clean Debian image. That's almost certainly the difference in size you're seeing. If you want to try building the CI version… well… I'm not sure there's much point; there's nothing that will run automatically in the repo, since it depends on various things we have in-house, and although it's possible to step through it manually (following the YAML files and Shell files), there's likely no point. Basically, there results in the compiled binaries being copied into the Postgres-XL path, with presumptions of all the same architecture, etc.

p.s. When I say development, that's not strictly true in this case, of course, since the binaries get copied across. This is a pattern we use for other projects, however, such as Haskell ones, and find very powerful. Basically, everything uses the 'development' copy, and dependencies come from there, and even external libraries are bit-level-frozen wherever possible (less of a concern, here, but it does in fact still happen, effectively). I've found this to be a far more powerful method that simply cleaning the compiled artefacts in-layer (although of course that would allow the same sizes of images to be arrived at, in most cases), or having multiple 'development' vs 'production' Dockerfiles (a design I don't like, or approve of, in general—but I'll admit that in this case that can be somewhat of a naming nuance).

Ok thanks a lot. I've just used a multistage dockerfile for now and have the size down to something similar. Not quite as small but good enough for my use.

Here it is just for anyone else who might have the same issue. I'm just using the tarball rather than using the source as I always have issues with WSL messing with line endings and this is just easier for me. Pgbouncer is included as I intend to run it for each coordinator with haproxy in front loadbalancing between them.
#===============================================================================
FROM debian:10 AS builder

ARG PG_HOME=/var/lib/postgresql
ARG PG_LIB=/usr/local/lib/postgresql
ARG PG_USER=postgres
#-------------------------------------------------------------------------------
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        # postgres xl requirements
        bison \
        build-essential \
        daemontools \
        flex \
        libreadline-dev \
        netcat \
        zlib1g-dev \
        # fix locales
        # curl for downloading libs and certs for https
        ca-certificates \
        curl && \
        rm -rf /var/lib/apt/lists/*
#-------------------------------------------------------------------------------
RUN useradd ${PG_USER} -d ${PG_HOME} && \
    mkdir -p ${PG_LIB} ${PG_HOME} && \
    chown -R ${PG_USER}:${PG_USER} ${PG_LIB} ${PG_HOME}
#-------------------------------------------------------------------------------
WORKDIR ${PG_HOME}
RUN curl -sf https://www.postgres-xl.org/downloads/postgres-xl-10r1.1.tar.gz -o postgres-xl-10r1.1.tar.gz && \
    tar -xvf postgres-xl-10r1.1.tar.gz && \
    rm postgres-xl-10r1.1.tar.gz && mkdir lib && mv postgres-xl-10r1.1 lib/postgres-xl && chown -R postgres:postgres lib
#-------------------------------------------------------------------------------
USER ${PG_USER}
WORKDIR ${PG_HOME}/lib/postgres-xl
RUN ./configure --prefix ${PG_LIB} && \
    make
#-------------------------------------------------------------------------------
USER root
RUN make install
#-------------------------------------------------------------------------------
RUN cd ${PG_HOME}/lib/postgres-xl/contrib/pgcrypto && make && make install
RUN cd ${PG_HOME}/lib/postgres-xl/contrib/ltree && make && make install
RUN cd ${PG_HOME}/lib/postgres-xl/contrib/cube && make && make install
#-------------------------------------------------------------------------------
COPY bin/* ${PG_LIB}/bin/
RUN rm -rf ${PG_HOME}/lib
#===============================================================================

#===============================================================================
FROM debian:10

ARG PG_HOME=/var/lib/postgresql
ARG PG_LIB=/usr/local/lib/postgresql
ARG PG_USER=postgres
#-------------------------------------------------------------------------------
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        libreadline-dev \
        netcat \
        locales \
        pgbouncer && \
    rm -rf /var/lib/apt/lists/*
#-------------------------------------------------------------------------------
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    locale-gen
#-------------------------------------------------------------------------------
COPY --from=builder ${PG_HOME} ${PG_HOME}
COPY --from=builder ${PG_LIB} ${PG_LIB}
#-------------------------------------------------------------------------------
USER ${PG_USER}
WORKDIR ${PG_HOME}
ENV PATH=${PG_LIB}/bin:$PATH \
    PGDATA=${PG_HOME}/data \
    PG_USER_HEALTHCHECK=_healthcheck

ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
#-------------------------------------------------------------------------------
VOLUME ${PG_HOME}
#===============================================================================

Okay, thanks a lot. I don't think I see a reason to change this, at present, since most people won't be developing it, but only using it from the images, and I suspect many developing it won't mind too much about the larger image sizes. By the time something's accepted, it'll go into the main release anyway, including with the smaller images we publish. And of course, if anyone wants to split off and release their own images, for some reason, they'll likely have their own CI or post-packaging anyway, so that's up to them. I'll close this, but thanks a lot for posting your approach in case it's useful to anyone else.


Bauble broken in 1.4.0

Just updated 1.10.2 to TOP 1.4.0, I have been using the bauble version of TOP and when I logged in it was not working and when I attempted to remove it from my bauble slot the client crashed with the following:
https://gist.github.com/xannor/2ebb88d5cf3832bd2574020088ede254

I'll check it out

I can confirm this issue; we're having the same problem.

Fixed in upcoming 1.4.1 release


XGBoost4j missing dylib for apple sillicon m1

CPU: Apple M1 Max
RAM: 64GB
java -version:
  openjdk version "11.0.13" 2021-10-19 LTS
  OpenJDK Runtime Environment Zulu11.52+13-CA (build 11.0.13+8-LTS)
  OpenJDK 64-Bit Server VM Zulu11.52+13-CA (build 11.0.13+8-LTS, mixed mode)

LIBRARY: ml.dmlc:xgboost4j_2.12:1.5.1
Getting the following error when trying to run our model tests.
java.lang.ExceptionInInitializerError
	at ml.dmlc.xgboost4j.java.Booster.init(Booster.java:694)
	at ml.dmlc.xgboost4j.java.Booster.<init>(Booster.java:51)
	at ml.dmlc.xgboost4j.java.Booster.loadModel(Booster.java:80)
	at ml.dmlc.xgboost4j.java.XGBoost.loadModel(XGBoost.java:77)

Seems like the root cause at the bottom of the stacktrace is a missing dylib:
Caused by: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
	at ml.dmlc.xgboost4j.java.NativeLibLoader.createTempFileFromResource(NativeLibLoader.java:233)
	at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:176)
	at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:130)
	at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34)
	... 88 more

Is there a known fix for this?

Currently XGBoost4J does not support Apple Silicon.

Currently XGBoost4J does not support Apple Silicon. Sorry for inconvenience.

Got it, can you provide some kind of timeline here?

@martintreurnicht We don't know yet. None of us (XGBoost developers) own an M1 machine, and currently I am not aware of any CI service that lets you test Apple Silicon.

You can compile xgboost4j from source on an M1 machine and it works. I did this last year when they came out on my personal M1 MBP, but without build resources to make the binaries it's hard to deploy.

@Craigacp Ok awesome, thanks! Yeah I got it to build and works great. I now just need to combine this dylib with the existing  jar. This will unblock us for now

@martintreurnicht We don't know yet. None of us (XGBoost developers) own an M1 machine, and currently I am not aware of any CI service that lets you test Apple Silicon.

@hcho3 I see you guys use github actions, you can probably use something like https://www.macstadium.com/mac-mini. Setup self-hosted with mac stadium should do the trick

Thanks for the pointer. We'll revisit once we have enough bandwidth and funding (since we'd have to pay MacStadium)

Ok fully understand, is there no way to cross compile this on x86_64 macOS? I don't know too much about this but i would imagine it must be possible, you can build aarch64 compatible binaries in XCode on a x86_64 mac after all

I think cross-compiling the JNI binding will be tricky. You'll probably need to have an ARM64 JVM as well as an x86 one to ensure you have the right libraries to link against.

Closing as duplicated to https://github.com/dmlc/xgboost/issues/6408


Can you add support for ESRI's .lyr files?

You may be able to use the work North Road has done with their QGIS SLYR plugin to read the .lyr files.
https://plugins.qgis.org/planet/tag/lyr/

Hey @donnyv,
thanks for thie hint. As the geostyler ecosystem is based on its own exchange format geostyler-style it's not easy to port one-to-one translations.
But there is already a parser that converts geostyler-style to .lyrx files. The geostlyer-lyrx-parser.
As the geostlyer-lyrx-parser that takes care of all Arc GIS related stuff I'll move this issue.
Note: The CLI supports the output of '.lyrx' files since version 4.1.0


Angular 4

Hi all,
I just upgraded my project from Angular 2 to Angular 4. After I start my project, I received a warning message about the
Template parse warnings:
The <template> element is deprecated. Use <ng-template> instead ("!option.isLabel && setSelected($event, option)" [class.dropdown-header]="option.isLabel">
          [ERROR ->]<template [ngIf]="option.isLabel">
            {{ option.name }}
          </template>
"): MultiselectDropdown@36:10

I checked the source code. Just edit the line No. 126 and change the <
<template [ngIf]="option.isLabel">
            {{ option.name }}
 </template>>

to
 <ng-templat [ngIf]="option.isLabel">
            {{ option.name }}
  </ng-templat >

Then, the warning will disappear.

I am aware of this. But Angular 2 is the latest stable release which the majority is using, and changing to <ng-template will lead to build fail for 2.x users. Not until 4.0 is released stable and people have upgraded this fix can be made.

Actually @softsimon it has been introduced in latest AOT commit https://github.com/softsimon/angular-2-dropdown-multiselect/blob/5c5176dc6d6955d8e3ca00f38cea5ec77a523333/src/multiselect-dropdown.ts#L127 and now I'm receive errors because I'm using Angular 2.X

I am not sure how to handle this issue correctly. Support 2.x with deprecated warning for 4.x users or require 4.x? It would be unnecessary hassle with two branches.

I think since Angular 4 is just release as stable this lib should continue to support 2.x for a while and 4.x users will see a warning. ng-bootstrap is doing the same.
The problem with requiring 4.x is that then 2.x won't be able to use the lib because the application will throw instead of just logging a warning.


Allow use of legacy filenames

Allow the use of filenames from legacy version switchers for the golang plugin.
e.g. the Terraform source, which has a .go-version file in the root.

@awilkins Looking at the docs it appears the legacy file support would come via bin/list-legacy-filenames and bin/parse-legacy-file. Where does bin/get-version-from-legacy-file get used?
For reference, I'm looking at https://asdf-vm.com/#/plugins-create?id=binlist-legacy-filenames

I copied my implementation for another language, which I believe I implemented on the back of reading the source code, or copying another implementation.
The core might have moved on it how it does things.

+1 on support for this one. @awilkins @kennyp is anyone working on this one? If not I'm happy to push it forward.

@jacobbednarz I haven't revisited yet, no. Based on the docs it should be as simple as adding a script called bin/list-legacy-filenames with something like:
#!/usr/bin/env bash

echo -n ".go-version"

@awilkins I would be glad for you to accept the PR if you want to switch to bin/list-legacy-filenames, or if not feel free to post one @jacobbednarz

Having read the docs, the default implementation once you have the filename is to just cat it, so I've removed the script (some doubt as to whether it was even being called, possibly...)
Still works fine!

Nice! Thanks.


Godot not opening imported projects

Godot version:
Godot 3.0.6 stable
OS/device including version:
Crostini/Chrome OS 86 |Lenovo chromebook s340 14-touch
Issue description:
Whenever I scan a folder or import a project it detects it however when I cant open the projects, only the projects made in the laptop I was able to open.
Steps to reproduce:
Minimal reproduction project:

Can you send the project that's causing you trouble?
Also Godot 3.0.6 is very old, can you confirm this happens with the latest stable version too (3.2.3)?

@IonAir1 Bump :slightly_smiling_face: See Ev1lbl0w's comment above.

Issue is very outdated and likely invalid. @Calinou


Intl support for English & Vietnamese

I18n support for UI labels with English & Vietnamese first.
If possible, use Vietnamese translation for the quotations.

Was done for 2.6 but after rewritten with Svelte, this feature need redo.


Dropping a duplicated block directly in the delete area will be error

Operating System and Browser
_e.g. Mac OS 10.11.6 Chrome
@rachel-fenichel

duplicateAndDrag violates the Gesture mechanism
whatever
This modification keeps the system working
https://github.com/DIYOSEPP/oseppBlock/commit/c5a567c182f749a702f08fa981174dc018db7a82

Fixed by #1090


Command to change the log level on a running Agent

What does this PR do?
Adds a command to change the log level on a running Agent.
Syntax:
datadog-agent loglevel -s debug

Possible values: trace, debug, info, warn, error, critical
Motivation
It can be used to help for the support of a customer case where the Agent is running in a weird state.
The main use-case would be to debug an Agent without restarting it in order to not lose its running state.
Additional Notes
Note that this PR does not implement a way to get the current log level of the Agent.

Codecov Report

Merging #2829 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2829      +/-   ##
==========================================
+ Coverage   57.21%   57.22%   +<.01%
==========================================
  Files         387      387
  Lines       24803    24803
==========================================
+ Hits        14192    14193       +1
+ Misses       9670     9669       -1
  Partials      941      941

Impacted Files
Coverage Δ

pkg/logs/auditor/auditor.go
71.83% <0%> (+0.7%)
:arrow_up:

this was merged in #4840 and released, closing


Float QSFilter nodes up as far as possible

Should be a rewrite right after ExtractFreeMap

@dugdun Can you copy/paste the query and the physical plan so we know when this ticket is done?

Related:

https://github.com/quasar-analytics/quasar/issues/2426
https://github.com/quasar-analytics/quasar/issues/2496
https://github.com/quasar-analytics/quasar/issues/1880

@jdegoes We also already have integration tests (which are pending) for this.

@djspiewak Do they cover the issues that @dugdun ran into? How about some of the above Github issues?

Sort of.  They test specifically that predicates get pulled above a complex join which is quite expensive if run on unreduced data.

What if you have a predicate A & B, and A can be pulled up, but B cannot, will it be split & relocated to the extent possible?

Nothing is pulled up now, split or otherwise.  One step at a time.

replicateM 100 😢

We just rebuilt a query planner from scratch in 30 days.  :-)  Naturally something was not going to get done.

The issues I saw around this are related to flattening an array item in the projection. The query below has only a single match for the given filter criteria.
Projecting a flattened array
This will push the entire dataset through a mapReduce() method call in MongoDB which gets more expensive as the size of the dataset grows. Notice that we don't filter our results until we send the data through a second mapReduce() method call.
(v27.4.9) 💪 $ select previous_visits[*] from patients where first_name = "Louvenia" and last_name = "Mosley"
MongoDB:
db.patients.aggregate(
  [
    { "$project": { "0": ["$_id", "$$ROOT"], "_id": false } },
    { "$out": "tmp.gen_7c61e71f_2" }],
  { "allowDiskUse": true });
db.tmp.gen_7c61e71f_2.mapReduce(
  function () {
    emit.apply(
      null,
      (function (key, value) {
        return [
          key,
          {
            "0": value["0"][1].first_name,
            "1": value["0"][1].last_name,
            "src": value["0"]
          }]
      })(
        this._id,
        this))
  },
  function (key, values) { return values[0] },
  { "out": { "replace": "tmp.gen_7c61e71f_1", "db": "data" } });
db.tmp.gen_7c61e71f_1.mapReduce(
  function () {
    (function (key, value) {
      return (function (key, value) {
        var rez = [];
        {
          var each0 = (Array.isArray(
            (isObject(value.src[1]) && (! Array.isArray(value.src[1]))) ? value.src[1].previous_visits : undefined) && (isObject(value.src[1]) && (! Array.isArray(value.src[1])))) ? value.src[1].previous_visits : undefined;
          for (var elem in (each0)) {
            var each1 = clone(each0);
            each1 = each0[elem];
            rez.push([ObjectId(), each1])
          }
        };
        return rez
      }).apply(
        null,
        (function (key, value) { return [key, value.value] })(key, value))
    })(
      this._id,
      this).map(
      function (__rez) { emit.apply(null, __rez) })
  },
  function (key, values) { return values[0] },
  {
    "out": { "replace": "tmp.gen_7c61e71f_0", "db": "data" },
    "query": { "$and": [{ "value.0": "Louvenia" }, { "value.1": "Mosley" }] },
    "scope": {
      "clone": function (src) {
        if (((typeof src) != "object") || (src == null)) return src;
        var dest = (src instanceof Date) ? new Date(src) : new src.constructor;
        for (var i in (src)) dest[i] = clone(src[i]);
        return dest
      }
    }
  });
db.tmp.gen_7c61e71f_1.drop();
db.tmp.gen_7c61e71f_2.drop();
db.tmp.gen_7c61e71f_0.find().limit(10);

Project a specific array element
If we project out a top level field, or even a specific array position, then we filter based on the WHERE clause in the first AggFramework call.
(v27.4.9) 💪 $ select previous_visits[0] from patients where first_name = "Louvenia" and last_name = "Mosley"
MongoDB:
db.patients.aggregate(
  [
    {
      "$match": { "$and": [{ "first_name": "Louvenia" }, { "last_name": "Mosley" }] }
    },
    { "$limit": NumberLong("10") },
    {
      "$project": {
        "__quasar_mongodb_sigil": {
          "$cond": [
            {
              "$and": [
                { "$lte": [{ "$literal": [] }, "$previous_visits"] },
                { "$lt": ["$previous_visits", { "$literal": BinData(0, "") }] }]
            },
            {
              "$arrayElemAt": ["$previous_visits", { "$literal": NumberInt("0") }]
            },
            { "$literal": undefined }]
        },
        "_id": false
      }
    }],
  { "allowDiskUse": true });

I would have expected that we could filter in the original AggFramework call and then only send the necessary documents through to the subsequent mapReduce() calls.

…also, your emoji are monadic??

I'm a functional programmer, of course my emoji are monadic. 😉

@dugdun
Have you tried:
SELECT table.previous_visits[*]
  FROM (
    SELECT previous_visits FROM patients
      WHERE first_name = "Louvenia" AND last_name = "Mosley") AS table

In theory, this shouldn't work, in practice, it might?

…or more simply, select (select previous_visits from patients where first_name = "Louvenia")[*]
Pretty sure that will work

Maybe even:
 (select previous_visits from patients where first_name = "Louvenia")[*]

The qscript for my version: https://gist.github.com/f92849a64fdc48d11fc2d51eadd17a39
And for yours: https://gist.github.com/beb07a1ddd493869dfcf845788d80d77

So identical.  Sweet.  The mongo workflow looks like this:
Chain
├─ $ReadF(t; demo.patients)
├─ $ProjectF
│  ├─ Name("0" -> ["$_id", "$$ROOT"])
│  ╰─ ExcludeId
├─ $SimpleMapF
│  ├─ Map
│  │  ╰─ Obj
│  │     ├─ Key(0: _["0"][1].first_name)
│  │     ╰─ Key(src: _["0"])
│  ╰─ Scope(Map())
├─ $MatchF
│  ╰─ Doc
│     ╰─ Expr($0 -> Eq(Text(Louvenia)))
├─ $SimpleMapF
│  ├─ Map
│  │  ╰─ JsCore((Array.isArray(
│  │            (Array.isArray(
│  │              (isObject(_.src[1]) && (! Array.isArray(_.src[1]))) ? _.src[1].previous_visits : undefined) && (isObject(_.src[1]) && (! Array.isArray(_.src[1])))) ? _.src[1].previous_visits : undefined) && (Array.isArray(
│  │            (isObject(_.src[1]) && (! Array.isArray(_.src[1]))) ? _.src[1].previous_visits : undefined) && (isObject(_.src[1]) && (! Array.isArray(_.src[1]))))) ? _.src[1].previous_visits : undefined)
│  ├─ Flatten
│  │  ╰─ Ident(_)
│  ╰─ Scope(Map())
╰─ $LimitF(10)

I'm not the greatest at understanding mongo, but I believe the fact that the Flatten is at the end is a good thing.

@dugdun Looks like we may have a workaround, at least for this simple case, see above. ^^^

While the latest query that @djspiewak provided does give a better plan (explain shows a single aggregate() method call), we still project the entire dataset out, unwind the array objects and only then do we filter on the WHERE predicate. This still will not use an index and will take a while to process (not as long as pushing all documents through the mapReduce() method. I'm currently running against the client's dataset that contains over 9M documents to see how long this takes.
Here's the plan that explain in the REPL gives me:
💪 $ explain select (select itemsInCart from sales where customerId = "bKM3cggE7Zx5tH96E")[*]
MongoDB:
db.sales.aggregate(
  [
    {
      "$project": {
        "__tmp4": {
          "$cond": [
            {
              "$and": [
                {
                  "$lte": [
                    { "$literal": [] },
                    {
                      "$cond": [
                        {
                          "$and": [
                            { "$lte": [{ "$literal": [] }, "$itemsInCart"] },
                            {
                              "$lt": [
                                "$itemsInCart",
                                { "$literal": BinData(0, "") }]
                            }]
                        },
                        "$itemsInCart",
                        { "$literal": undefined }]
                    }]
                },
                {
                  "$lt": [
                    {
                      "$cond": [
                        {
                          "$and": [
                            { "$lte": [{ "$literal": [] }, "$itemsInCart"] },
                            {
                              "$lt": [
                                "$itemsInCart",
                                { "$literal": BinData(0, "") }]
                            }]
                        },
                        "$itemsInCart",
                        { "$literal": undefined }]
                    },
                    { "$literal": BinData(0, "") }]
                }]
            },
            {
              "$cond": [
                {
                  "$and": [
                    { "$lte": [{ "$literal": [] }, "$itemsInCart"] },
                    { "$lt": ["$itemsInCart", { "$literal": BinData(0, "") }] }]
                },
                "$itemsInCart",
                { "$literal": undefined }]
            },
            { "$literal": [undefined] }]
        },
        "__tmp5": "$$ROOT"
      }
    },
    { "$unwind": "$__tmp4" },
    { "$match": { "__tmp5.customerId": "bKM3cggE7Zx5tH96E" } },
    { "$project": { "value": "$__tmp4", "_id": false } }],
  { "allowDiskUse": true });

@dugdun On quasar v27.5.3 + #3298 the plan for
 select (select loc from zips where state = "CA")[*]
looks like
db.zips.aggregate(
  [
    { "$match": { "state": "CA" } },
    {
      "$project": {
        "0": {
          "$cond": [
            {
              "$and": [
                {
                  "$lte": [
                    { "$literal": [] },
                    {
                      "$cond": [
                        {
                          "$and": [
                            { "$lte": [{ "$literal": [] }, "$loc"] },
                            { "$lt": ["$loc", { "$literal": BinData(0, "") }] }]
                        },
                        "$loc",
                        { "$literal": undefined }]
                    }]
                },
                {
                  "$lt": [
                    {
                      "$cond": [
                        {
                          "$and": [
                            { "$lte": [{ "$literal": [] }, "$loc"] },
                            { "$lt": ["$loc", { "$literal": BinData(0, "") }] }]
                        },
                        "$loc",
                        { "$literal": undefined }]
                    },
                    { "$literal": BinData(0, "") }]
                }]
            },
            {
              "$cond": [
                {
                  "$and": [
                    { "$lte": [{ "$literal": [] }, "$loc"] },
                    { "$lt": ["$loc", { "$literal": BinData(0, "") }] }]
                },
                "$loc",
                { "$literal": undefined }]
            },
            { "$literal": undefined }]
        },
        "_id": false
      }
    },
    { "$unwind": "$0" },
    { "$project": { "__quasar_mongodb_sigil": "$0", "_id": false } }],
  { "allowDiskUse": true });

which looks like it should leverage indices properly, $unwind-ing after the $match.

Actually, the plan for
select loc[*] from (select * from zips where state = "CA")
is even better
db.zips.aggregate(
  [
    { "$match": { "state": "CA" } },
    {
      "$project": {
        "0": {
          "$cond": [
            {
              "$and": [
                { "$lte": [{ "$literal": [] }, "$loc"] },
                { "$lt": ["$loc", { "$literal": BinData(0, "") }] }]
            },
            "$loc",
            { "$literal": undefined }]
        },
        "_id": false
      }
    },
    { "$unwind": "$0" },
    { "$project": { "__quasar_mongodb_sigil": "$0", "_id": false } }],
  { "allowDiskUse": true });

Thanks @wemrysi! I was using the version of Quasar that was shipped in SD 4.2.6 as that is what the client has. I will test with 27.5.3 a little later to see what I get. I was using 27.4.9 which was newest as of a few days ago. I'm sure the client would be OK using an early version of SD 5.0.0 if it's stable to make progress on things.

The comment about 27.4.9 was around other testing earlier with this customer. Obviously the REPL output from above was form the earlier Quasar build (20.14.7) from SD 4.2.6.

Hmmm... I don't see a Quasar 27.5.3 release. The latest release I see is 27.4.10 which was release a day ago. Will check on things tomorrow.

Running the MongoDB query provided by @wemrysi and changing to fit the client's database inside of MongoDB directly, the results are returned in a fraction of a second.
If I can get my hands on the 27.5.3 build he references in his comments, then I can test the SQL query there to verify that the client query compiles down the same way and we can progress with working on building reports.
Thanks for everyone's help in tracking down alternative queries and providing help on this.

@dugdun That was v27.5.3 with @jsantos17 's PR (#3298) which improves MongoDB's handling of LeftShift, hopefully we'll have that merged soon and can get you a release.


Agent Validation Check Fail

agent deployed by daemonset & deployment giving following error..
```Agent Validation Check Fail ````

Hello.
Agent is printing Agent Validation Check Fail message while wait for collector working properly.
So, It's not error. After few seconds, message will disappear.

Yup.


MSID: 92083 Version: 1 DOI: 2<IP_ADDRESS>2629

"msas": "Chromosomes and Gene Expression", "Developmental Biology"
"msid": "92083"
"preprintDoi": "10.1101/2<IP_ADDRESS>2629"
"Reviewed Preprint posted": "2023-10-30"
"Sent for peer review": "2023-08-29"
"Posted to bioRxiv": "2023-08-11" (link: "Go to bioRxiv": "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>2629v1/")
[PLACE PDF URL HERE WHEN AVAILABLE] See step 7
Step 1. Inform bioRxiv
Who can help: @QueenKraken, @nlisgo, @scottaubrey

[x] Manuscript is in data hub index (https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=92083)

Step 2. Create preview of manuscript
Who can help: @fred-atherden, @nlisgo, @scottaubrey

[ ] Manuscript is available for preview (https://prod--epp.elifesciences.org/preview/92083/v1)

[ ] Trigger import for staging: https://prod--epp.elifesciences.org/import
[ ] Sync data folder with s3://prod-elife-epp-data/data
[ ] Trigger import for staging: https://staging--epp.elifesciences.org/import
[ ] Sync data folder with s3://staging-elife-epp-data/data
[ ] Pull request created to enhanced-preprint-data
[ ] Post link to pull request below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-manuscript
Step 3: Awaiting public reviews and QC
Who can help: Production team

[ ] QC passed or QC not needed

Who can help: Editorial team

[x] Public reviews available on sciety (https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>2629)
[x] Manuscript and editor details at the top of this issue (Supplied by Editorial team)
[ ] Reviews returned by EPP - https://prod--epp.elifesciences.org/api/reviewed-preprints/92083/v1/reviews

Example
"msas": "Genetics and Genomics", "Neuroscience"
"msid": "84628"
"version": "1"
"preprintDoi": "10.1101/2<IP_ADDRESS>4241"
"articleType": "Reviewed Preprint"
"status": "Published from the original preprint after peer review and assessment by eLife."

"Reviewed Preprint posted": "2023-01-02"
"Sent for peer review": "2022-10-28"
"Posted to bioRxiv": "2022-11-21" (link: "Go to bioRxiv": "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>4241v1")

Editors:
Reviewing Editor
Michael B Eisen
University of California, Berkeley, United States
Senior Editor
Michael B Eisen
University of California, Berkeley, United States

Step 4: Modify manuscripts.json (no PDF)

[ ] Reviewed preprint is published to EPP (https://elifesciences.org/reviewed-preprints/92083)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[ ] Request that a doi be registered

[ ] Post link to #enhanced-preprint comment below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]

Instructions to modify manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/publish-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/334/files
Once the pull request is merged in it should be available a few minutes later.

Request that a doi
Post the following in #enhanced-preprint:
@Fred can you register a doi for https://elifesciences.org/reviewed-preprints/92083

Step 5: Awaiting search reindex

[ ] Reviewed preprint is avaliable on journal homepage (https://elifesciences.org)

The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.

Additional info
If needed, the jenkins pipeline to reindex search can be triggered sooner.
https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 6: Published! Request PDF generation

[ ] Post the link to the PDF url at the top of the issue

[ ] Post link to #enhanced-preprint comment below

#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]
Post the following to the #enhanced-preprint on slack:
@Ryan Dix-Peek please can you generate a PDF for https://elifesciences.org/reviewed-preprints/92083

Step 7: Introduce PDF to data folder and git repo

[ ] PDF is avaliable at https://github.com/elifesciences/enhanced-preprints-data/raw/master/data/92083/v1/92083-v1.pdf

Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-pdf
Step 8: Add PDF url to manuscripts.json

[ ] Reviewed preprint PDF is available for download (https://prod--epp.elifesciences.org/reviewed-preprints/92083)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[PLACE LINK TO PULL REQUEST HERE]

Instructions to add PDF url to manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/add-pdf-url-to-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/397/files
Once the pull request is merged in it should be available a few minutes later.

Step 9: Done!

[ ] Kettle is on!

Sarah Bray ORCID link directs to Jonathan Townson ORCID

Nicely spotted @rkcook312!
I've added her correct ORCID (taken form eJP which is authenticated). Please check and update if OK

Thanks @fred-atherden, checked and OK to proceed.


Paper vs Live

How to configure the plugin to use the paper account and not the live account? I can login to the live but not to the paper.

It seems to work if I set the account time in the accounts file to Demo (0)


getRootStore returns undefined.

I noticed a bug in my older app. I use :
import { getRootStore } from 'mobx-keystone';

const rootStore = getRootStore<RootStore>(this);

This used to work and i used it allot in my app, however, in a previous update of the app it only return undefined.
This hasn't been noticed so i'm not sure if updating mobx-keystone in the past had something to do with it.
I checked the changelogs and there seemed to be no breaking changes concerning the getRootStore.
I setup a store like this:
rootStore = new RootStore({
        ticketList: new TicketList({}),
        basket: new Basket({}),
      });

And where i call the getRootStore:

setGlobalConfig({
  modelAutoTypeChecking: ModelAutoTypeCheckingMode.AlwaysOn,
});
@model('keystone/Application/TicketList')
export default class TicketList extends Model({
  ticketModels: tProp(
    types.array(types.model<TicketModel>(TicketModel)),
    () => []
  ),
}) {
onInit = () => {
    const rootStore = getRootStore<RootStore>(this);
    console.log('Should be the rootStore: ', rootStore);  // Gives undefined
}
}

"mobx": "^6.0.4",
"mobx-keystone": "^0.66.0",
"mobx-react": "^7.3.0",

that should only return something after you call registerRootStore. where are you calling it?

I call registerRootStore(rootStore); in my createRootStore function:

export async function createRootStore() {
  const rawStored = localStorage.getItem('newst');
  if (typeof rawStored === 'string') {
    try {
      const savedRootStore = await JSON.parse(rawStored);
      rootStore = fromSnapshot<RootStore>(savedRootStore);
      registerRootStore(rootStore);
    } catch (e) {
      console.log('There was an error restoring the initial localStorage');
    }
  } else {
    try {
      rootStore = new RootStore({
        ticketList: new TicketList({}),
        basket: new Basket({}),
      });
    } catch (e) {
      console.log('There was an error creating rootStore from scratch', e);
    }
    registerRootStore(rootStore);
  }

  return rootStore;
}

I call this function at the start of the app:
(async () => {
  const st = await createRootStore();
  return console.log('1. CREATED rootStore');
})();

If i remove all the registerRootStore() from my code, i get the same result the app starts and works. On the home page, there is already some data from mobx keystone state. But when i do getRootStore(this) in another part of the tree, it doesn't give me the rootStore.
As a test, i have placed another registerRootStore() after where i create the rootStore. And then i get the error: object already registered as root store
So it looks like it has been registered successfully in the creatRootStore function.
(async () => {
  const st = await createRootStore();
  registerRootStore(st);
  return console.log('1. CREATED rootStore');
})();

Above gives the error: object already registered as root store

That's expected:
// here no rootstore is registered

rootStore = new RootStore({
  ticketList: new TicketList({}), // here onInit is called, where no rootstore is still registered
});

// here is where the root store gets actually registered
registerRootStore(rootStore);

if you want to use the rootstore as part of a "sort of" initialization phase use onAttachedToRootStore instead
https://mobx-keystone.js.org/root-stores#onattachedtorootstore

i had changed onAttachedToRootStore() wrongly to onInit() at some point. Thank you very much!


Adds Paul'st article about auth0 simulator and nav option

Motivation
Announce Auth0 simulator
Approach
Paul wrote article, I edited and proofed with Olson. Also added Auth0 simulator link under Open Source drop down

@dagda1 I committed your change, thanks! can you do a final check so we can publish it? thanks!

@dagda1 I committed your change, thanks! can you do a final check so we can publish it? thanks!

I think it is good to go.  When are you thinking of publishing this?  I have a couple of PRs that I would like to get merged in the auth0 simulator.
It would be good to have those merged in case anyone feels inspired to give it a go after reading the blog post.

I can publish it whenever you feel it's ready

@jorgelainfiesta @dagda1 the link in the conclusion of the blog post links to simulacrum repo. Should it point to https://github.com/thefrontside/simulacrum/tree/v0/packages/auth0?

@jorgelainfiesta @dagda1 the link in the conclusion of the blog post links to simulacrum repo. Should it point to https://github.com/thefrontside/simulacrum/tree/v0/packages/auth0?

 GitHub**simulacrum/packages/auth0 at v0 · thefrontside/simulacrum**A simulation platform for use during testing, during development and for high-fidelity application previews - simulacrum/packages/auth0 at v0 · thefrontside/simulacrum

@taras yes, good spot, it should point to the auth0 package.


Update SCIseg description once updated to V2

Once we update SCIseg to V2, we should remove "traumatic" from its description (as the model was now also trained on non-traumatic SCI).
Sections to update:

[x] This repo "About" section:

update to:
Deep-learning based segmentation of the spinal cord and intramedually lesions in traumatic and non-traumatic SCI

[x] sct_deepseg help:

https://github.com/spinalcordtoolbox/spinalcordtoolbox/blob/65e44b5068fe5d8a094e3e0571566c01d2d460ab/spinalcordtoolbox/deepseg/models.py#L315-L321

Done in https://github.com/ivadomed/model_seg_sci/issues/89


Changes

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

windows-iotcore/learn-about-hardware/SoCsAndCustomBoards.md
:white_check_mark:Succeeded

windows-iotcore/windows-iot-core.md
:white_check_mark:Succeeded

windows-iotcore/windows-iot-enterprise.md
:white_check_mark:Succeeded

windows-iotcore/windows-iot.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Extract Contextual Word Embeddings

Add the ability to extract contextual word embeddings from a given list of sentences using XLNet, same like in BERT.
The script extracts a fixed length vector for each token in the sentence.
First, one needs to create an input text file as following:
# Sentence A and Sentence B are separated by the ||| delimiter for sentence pair tasks.
# For single sentence inputs, put one sentence per line and DON'T use the delimiter.
echo 'I love New York. ||| New York is a city' > data/corpus.txt

After that, the script extract_features.py can be used like this, which will create vectors of length 64 for each token in the sentence:
INIT_CKPT_DIR=models/xlnet_cased_L-24_H-1024_A-16
OUTPUT_DIR=data
MODEL_DIR=experiment/extract_features

python extract_features.py \
    --input_file=data/corpus.txt \
    --init_checkpoint=${INIT_CKPT_DIR}/xlnet_model.ckpt \
    --spiece_model_file=${INIT_CKPT_DIR}/spiece.model \
    --use_tpu=False \
    --num_core_per_host=1 \
    --output_file=${OUTPUT_DIR}/output.json \
    --model_dir=${MODEL_DIR} \
    --num_hosts=1 \
    --max_seq_length=64 \
    --eval_batch_size=8 \
    --predict_batch_size=8 \
    --model_config_path=${INIT_CKPT_DIR}/xlnet_config.json \
    --summary_type=mean

Or use the scripts/gpu_extract_features.sh script for running it easily.
This will create a JSON file (one line per line of input) containing the contextual word embeddings from XLNet.
#39
Hi @zihangdai @kimiyoung, can you please take a look?

Add the ability to extract contextual word embeddings from a given list of sentences using XLNet, same like in BERT.
The script extracts a fixed length vector for each token in the sentence and one pooled vector from all the word embeddings, with the given pooling strategy parameter.
First, one needs to create an input text file as following:
# Sentence A and Sentence B are separated by the ||| delimiter for sentence pair tasks.
# For single sentence inputs, put one sentence per line and DON'T use the delimiter.
echo 'I love New York. ||| New York is a city' > data/corpus.txt

After that, the script extract_features.py can be used like this, which will create vectors of length 64 and one pooled vector with mean strategy:
INIT_CKPT_DIR=models/xlnet_cased_L-24_H-1024_A-16
OUTPUT_DIR=data
MODEL_DIR=experiment/extract_features

python extract_features.py \
    --input_file=data/corpus.txt \
    --init_checkpoint=${INIT_CKPT_DIR}/xlnet_model.ckpt \
    --spiece_model_file=${INIT_CKPT_DIR}/spiece.model \
    --use_tpu=False \
    --num_core_per_host=1 \
    --output_file=${OUTPUT_DIR}/output.json \
    --model_dir=${MODEL_DIR} \
    --num_hosts=1 \
    --max_seq_length=64 \
    --eval_batch_size=8 \
    --predict_batch_size=8 \
    --model_config_path=${INIT_CKPT_DIR}/xlnet_config.json \
    --summary_type=mean

Or use the scripts/gpu_extract_features.sh script for running it easily.
This will create a JSON file (one line per line of input) containing the contextual word embeddings from XLNet, including one pooled vector.
#39

Hello,  i write corpus.txt 1 sentence. When i run  python extract_features.py \     --input_file=data/corpus.txt \     --init_checkpoint=${INIT_CKPT_DIR}/xlnet_model.ckpt \     --spiece_model_file=${INIT_CKPT_DIR}/spiece.model \     --use_tpu=False \     --num_core_per_host=1 \     --output_file=${OUTPUT_DIR}/output.json \     --model_dir=${MODEL_DIR} \     --num_hosts=1 \     --max_seq_length=64 \     --eval_batch_size=8 \     --predict_batch_size=8 \     --model_config_path=${INIT_CKPT_DIR}/xlnet_config.json \     --summary_type=mean   in different times, sentence feature is different.  they should be same i think.

@3NFBAGDU Thanks for pointing this out, I will check it. Although, I don't have any random code in this PR, which makes it very weird.

Hi,  thank you for answering. as i have tested it, Euclidean distance is better than cosine distance for words.
cosine distance allways > 0.89.  I have trained my model in 1.6M sentences.
And do you have any idea how to get sentence_embedding vector from here?

Hi, thank you for answering. as i have tested it, Euclidean distance is better than cosine distance for words.
cosine distance allways > 0.89. I have trained my model in 1.6M sentences.
And do you have any idea how to get sentence_embedding vector from here?

Thanks for sharing the results.
In order to get sentence embedding, you can perform pooling by one of the existing pooling strategies, like: max pooling, mean pooling, max-mean pooling, attention pooling, etc.
For example, to perform max pooling, you just need to sum all word vectors into one vector for dimension 1024.
Please note that some word are the padding tokens (actually most of them), so you should ignore them in the pooling strategy and perform the pooling with only real tokens.

Hi,  if i give 'Hello, how are you',  there should be this output:
{'token' : {"Hello", values} : ...., {token = "how", values:[...]}}.,
but i have output {token: '', values:[]} , i have token allways empty. Is this my sentencepiece model fault ?

Hello, how are you

I cannot reproduce this, for me all tokens are valid and correspond to the tokens in the original sentence. I think you might put wrong the sentence piece model parameter.

Hi, if i give 'Hello, how are you', there should be this output:
{'token' : {"Hello", values} : ...., {token = "how", values:[...]}}.,
but i have output {token: '', values:[]} , i have token allways empty. Is this my sentencepiece model fault ?

Apparently, when giving the sentence Hello, how are you? sentence piece model tokenize it such that the first token is empty.
I added code that ignore those empty tokens.
Thanks for noticing.

estimator.predict() works to slowly. I want to predict some text in my model in every 2 seconds. Everytime I call the estimator.predict() function, it loads the model all over again. I want to load the model just once and after that use estimator.predict() every 2 seconds on this same model to get the faster prediction.

Hi @zihangdai  @kimiyoung , since issue #39 was closed, can you please merge this to master?
Thanks.

Hi @zihangdai @kimiyoung , since issue #39 was closed, can you please merge this to master?
Thanks.

Hi sir, How do you modify vector dimensions?

Hi sir, How do you modify vector dimensions?

@Hazoom Hi sir, How to modify vector dimensions?

@Hazoom Hi sir, How to modify vector dimensions?

@JxuHenry I don't think it's possible, the dimension is set by the network's architecture.

@Hazoom Hi sir, How to modify vector dimensions?

@JxuHenry I don't think it's possible, the dimension is set by the network's architecture.

OK，Thank you very much

Hi Hazoom,
I follow your instructions and run extract_features.py. Does it need a GPU to run this program?

Hi Hazoom,
I follow your instructions and run extract_features.py. Does it need a GPU to run this program?

No, it can be run on CPU as well, just a little bit slower than GPU.

Thank you, Hazoom. I use tensorflow v1.15. Is it the tensorflow version you used?

Thank you, Hazoom. I use tensorflow v1.15. Is it the tensorflow version you used?

I used Tensorflow v1.14, but it should be the same, I hope.

It seems that I don't get the expected results. I copied some messages below. Could you please take a look and let me know what the problem is?

2020-02-16 15:03:13.502591: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-16 15:03:13.523791: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2020-02-16 15:03:13.524337: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55c394737b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-02-16 15:03:13.524391: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-02-16 15:03:13.527100: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-02-16 15:03:13.527146: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)
2020-02-16 15:03:13.527179: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (pl-00193583): /proc/driver/nvidia/version does not exist
2020-02-16 15:03:16.775053: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 131072000 exceeds 10% of system memory.
INFO:tensorflow:Running local_init_op.
I0216 15:03:18.755628<PHONE_NUMBER>27168 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0216 15:03:18.981006<PHONE_NUMBER>27168 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Predicting submission for example_cnt: 0
I0216 15:03:26.952376<PHONE_NUMBER>27168 extract_features.py:427] Predicting submission for example_cnt: 0

Hi Hazoom,
I succeeded in using the scripts to get an output.json for one sentence "Hello World". I observed that the embeddings has 6 tokens, "he", "ll", "o", "world", and the last two tokens are , is this tokenization normal? If we use pooling strategy to calculate the sentence embedding. do we need to remove the 's embeddings?


How to fed the roi feature after MFM into MAE decoder？

I am trying to migrate this project (ImTed) to mmdetection3.x. However, I have some doubts about how to send roi features after the Multi-scale Feature Modulator to mae decoder.
I found that the bbox_feats' shape are like torch.Size([512, 384, 7, 7])，but the self.bbox_head, i.e., the mae decoder requires inputs like x.shape = [b_s, dim, W, H], how to solve this?
def _bbox_forward(self, x, rois):
        """Box head forward function used in both training and testing."""
        # TODO: a more flexible way to decide which feature maps to use
        if self.with_mfm:
            ss_bbox_feats = self.ss_bbox_roi_extractor(
                [x[-1]], rois)
            x = [self.mfm_fc(x[i]) for i in range(self.ms_bbox_roi_extractor.num_inputs)]
            ms_bbox_feats = self.ms_bbox_roi_extractor(
                x[:self.ms_bbox_roi_extractor.num_inputs], rois) # multi scale

            factor = self.mfm_factor.reshape(1, -1, 1, 1).expand_as(ms_bbox_feats)
            bbox_feats = ss_bbox_feats + ms_bbox_feats * factor
        else:
            bbox_feats = self.bbox_roi_extractor(
                x[:self.bbox_roi_extractor.num_inputs], rois)
        if self.with_shared_head:
            bbox_feats = self.shared_head(bbox_feats)
        cls_score, bbox_pred = self.bbox_head(bbox_feats)

Hi, @Livioni , see here

The shape of the output of modified function is the same as that of original function.  The shape of input of  mae_bbox_head is the same as that of commonly used convfc_bbox_head. The output of Multi-scale Feature Modulator is what mae_bbox_head needs. The shape meets. Do you encounter any shape problem when migrating code?

Have you successfully replicated the project's results from this paper? I encountered issues while setting up the environment. Could you please share your environment configuration with me? I would appreciate it if you could include detailed version information for all the libraries so that I can run the experiment successfully on my local machine. Thank you.@Livioni