Crashes at breakpoint on error with Gallium.breakpoint_on_error()

I am using Julia Version 0.5.0-rc3+0 (2016-08-22 23:43 UTC) on x86_64-apple-darwin15.3.0
Perhaps this is related to #148
This is my first attempt to set a breakpoint with Gallium. I tried calling breakpoint_on_error(), then defined and ran the silly() function and got the following error:
julia> using Gallium

julia> Gallium.breakpoint_on_error()

Gallium.Hooking.Hook(Ptr{Void} @0x000000010ca71d20,UInt8[0x55,0x48,0x89,0xe5,0x41,0x56,0x53,0x49,0x89,0xfe,0xe8,0x31,0xd5,0x01,0x00],Gallium.breakpoint_on_error_hit)
julia> function silly()
       error()
       end

silly (generic function with 1 method)
julia> silly()

ERROR: KeyError: key Ptr{Void} @0x000000010ca71d20 not found
in pop!(::Dict{Ptr{Void},Gallium.Hooking.Hook}, ::Ptr{Void}) at ./dict.jl:717
in unhook(::Ptr{Void}) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Hooking/Hooking.jl:476
in breakpoint_on_error_hit(::Gallium.Hooking.Hook, ::Gallium.X86_64.ExtendedRegs) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Gallium.jl:938
in _callback(::Ptr{Void}) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Hooking/Hooking.jl:289
in callback(::Ptr{Void}) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Hooking/Hooking.jl:326
Then I tried calling breakpoint_on_error() after the silly() function had been introduced and got a different error:
 julia> Gallium.breakpoint_on_error()

Gallium.Hooking.Hook(Ptr{Void} @0x000000010ca71d20,UInt8[0x55,0x48,0x89,0xe5,0x41,0x56,0x53,0x49,0x89,0xfe,0xe8,0x31,0xd5,0x01,0x00],Gallium.breakpoint_on_error_hit)
 julia> silly()

ERROR: MethodError: no method matching start(::Gallium.NativeStack)
Closest candidates are:
start(::SimpleVector) at essentials.jl:170
start(::Base.MethodList) at reflection.jl:258
start(::IntSet) at intset.jl:184
...
in collect(::Base.Generator{Gallium.NativeStack,Gallium.##104#108}) at ./array.jl:300
in breakpoint_on_error_hit(::Gallium.Hooking.Hook, ::Gallium.X86_64.ExtendedRegs) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Gallium.jl:947
in _callback(::Ptr{Void}) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Hooking/Hooking.jl:289
in callback(::Ptr{Void}) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Hooking/Hooking.jl:326

For me, even this "silly" example results in a segfault.
julia> versioninfo()
Julia Version 0.5.1-pre+2
Commit f0d40ec* (2016-09-20 03:34 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7 CPU       L 640  @ 2.13GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Nehalem)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, westmere)

@Keno, does this work on OSX but not Linux? Need help debugging? (I see you've been working on rr, so maybe that's part of your strategy to debug this...) Since it feels like so much of the work must already be done, it sure would be nice to get this working.

Yeah, I'll look into this. There have just been large demands on my time lately ;).

No doubt. Just wondering if it's platform specific and you might need some info from folks with different platforms.

It is platform specific, but I should have everything I need to look into this

BTW for me it segfaults whether or not I turn on breakpointing before silly has been executed (i.e., doesn't seem to be JIT-related).

I suspect it's due to inference's use of errors causing recursive faults.

I wondered.
Before 0.5 got released I didn't think this was very pressing, BTW, because I was in a phase where I was rebuilding julia about 10x per day and this lost out to @show debugging due to the cost of precompilation. Now that I'm using the same julia for days on end (will the wonders never cease!) this becomes much more interesting again :wink:.

Oops, here is all my version info:
versioninfo()
Julia Version 0.5.1-pre+2
Commit f0d40ec (2016-09-20 03:34 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin15.3.0)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, haswell)

On my system
using Gallium
Gallium.breakpoint_on_error()
function silly()
    error()
end
silly()

simply causes the computer to hang, until it report that it has run out of diskspace (I normally have 6GB free, it is a small SSD).
versioninfo()
Julia Version 0.5.0
Commit 3c9d753 (2016-09-19 18:14 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, ivybridge)

This is repeated exactly on my other mac, which is the same model just 2 years more recent.

I get the hanging with breakpoint_on_error(); seems to be some sort of infinite recursion as memory usage for julia climbs forever.

Did this fix work for other OSX users? I am getting a new error now:
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.1-pre+2 (2016-09-20 03:34 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit f0d40ec (21 days old release-0.5)
|__/                   |  x86_64-apple-darwin15.3.0

julia> using Gallium

julia> Gallium.breakpoint_on_error()
Gallium.Hooking.Hook(Ptr{Void} @0x0000000107a341b0,UInt8[0x55,0x48,0x89,0xe5,0x41,0x56,0x53,0x49,0x89,0xfe,0xe8,0x31,0xdd,0x01,0x00],Gallium.breakpoint_on_error_hit)

julia> function silly()
                     error()
                     end
silly (generic function with 1 method)

julia> silly()
ERROR: KeyError: key Ptr{Void} @0x0000000107a341b0 not found
 in pop!(::Dict{Ptr{Void},Gallium.Hooking.Hook}, ::Ptr{Void}) at ./dict.jl:717
 in unhook(::Ptr{Void}) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Hooking/Hooking.jl:476
 in breakpoint_on_error_hit(::Gallium.Hooking.Hook, ::Gallium.X86_64.ExtendedRegs) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Gallium.jl:938
 in _callback(::Ptr{Void}) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Hooking/Hooking.jl:289
 in callback(::Ptr{Void}) at /Users/y9KAaixxegjR/.julia/v0.5/Gallium/src/Hooking/Hooking.jl:326


Fix quick pick message size

Fixes #73345

Stable on the left, running from source on the right:

Warning message:

The second horizontal red line should align with the OK button vertically (like the first horizontal red line). The warning message looks like it's not well centered vertically, is it different from the input box's text?

Without red border:

With red border:

I drew a line and made sure they line up now :)


Problem with cities

Hi! This clabe<PHONE_NUMBER>18542788 is not working because the city is not in the list.
It could be a good idea to allow validation of bank only.

What is the city for code 521?

If you call clabe.validate('646521162218542788'), it will return:
{
   account:  '16221854278',
   bank:     'Sistema de Transferencias y Pagos STP, S.A. de C.V., SOFOM E.N.R.',
   checksum: 8,
   city:     undefined,
   code:     { bank: '646', city: '521' },
   error:    'invalid-city',
   formatOk: true,
   message:  'Invalid city code: 521',
   ok:       false,
   tag:      'STP'
}

You can check the bank field to see if the bank code is valid even if the city code is invalid.  Also, the formatOk field can be true even if the bank code and city code are unknown.

If you ever want the city code added, just reopen this issue or submit a pull request.


Error:'Not a valid Bencoding' when trying to open torrent file in a torrent client.

I am using create-torrent to create a torrent file in node js and passing the buffer to a .Net application where I convert the buffer to a byte array and create a torrent file. When I try to open this file in a torrent client(e.g.:utorrent), I get an error saying "not a valid bencoding".Any way to achieve this?
But, when I open a torrent file created through node, it opens fine. Also, this file when opened in a torrent client doesn't start seeding the file. Any idea why this is happening?

passing the buffer to a .Net application where I convert the buffer to a byte array and create a torrent file

This sounds overly complicated and there's probably a bug in your code somewhere. Without more detail, that's what I'm inclined to assume.

Also, this file when opened in a torrent client doesn't start seeding the file. Any idea why this is happening?

This is going to depend on the specific torrent client you're using. Normally, you have to select the location of the file on disk as a separate step.


Fix for bug #68

Just added a quick (maybe dirty and not the best way to do it?) check to make sure that a secondary DNS server is provided, and if not, eliminate the trailing comma in the variable assignment.

Still won't work since the output of the echo are not redirected to the config file

Yeah no it's a typo, don't run the code I put in the pull request. It is
incorrect and I would be asTOUNDED if it ran. I just haven't had time to
actually fix it.
On Tue, Jun 30, 2020 at 3:30 PM randomshell<EMAIL_ADDRESS>wrote:

@randomshell commented on this pull request.
In wireguard-install.sh
https://github.com/angristan/wireguard-install/pull/75#discussion_r447928988
:

@@ -213,7 +220,7 @@ WG_RUNNING=$?

Warn user about kernel version mismatch with headers
if [[ $OS =~ (fedora|centos) ]] && [[ $WG_RUNNING -ne 0 ]]; then
echo -e "\nWARNING: WireGuard does not seem to be running."

echo "Due to kernel mismatch issues on $OS, WireGuard might work if your system is out of date."

echo "Due to kernel mismatch issues on $OS, WireGuard might not work if your system is out of date."

@angristan https://github.com/angristan is this a typo or not?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/angristan/wireguard-install/pull/75#pullrequestreview-440322471,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABIWE6QYF3YQTLOGH6G5HTTRZI4NDANCNFSM4MZ75AVA
.

Wait, no, misread the alert email. Ignore me, very very sorry.
On Tue, Jun 30, 2020 at 3:47 PM Chris Lewicki<EMAIL_ADDRESS>wrote:

Yeah no it's a typo, don't run the code I put in the pull request. It is
incorrect and I would be asTOUNDED if it ran. I just haven't had time to
actually fix it.
On Tue, Jun 30, 2020 at 3:30 PM randomshell<EMAIL_ADDRESS>wrote:

@randomshell commented on this pull request.
In wireguard-install.sh
https://github.com/angristan/wireguard-install/pull/75#discussion_r447928988
:

@@ -213,7 +220,7 @@ WG_RUNNING=$?

Warn user about kernel version mismatch with headers
if [[ $OS =~ (fedora|centos) ]] && [[ $WG_RUNNING -ne 0 ]]; then
echo -e "\nWARNING: WireGuard does not seem to be running."

echo "Due to kernel mismatch issues on $OS, WireGuard might work if your system is out of date."

echo "Due to kernel mismatch issues on $OS, WireGuard might not work if your system is out of date."

@angristan https://github.com/angristan is this a typo or not?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/angristan/wireguard-install/pull/75#pullrequestreview-440322471,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABIWE6QYF3YQTLOGH6G5HTTRZI4NDANCNFSM4MZ75AVA
.

Added in https://github.com/angristan/wireguard-install/commit/5f82b76bc0df06a5ad8cbe2c6069ef016867ae04


Various Scale Things

Hey Lucid,
I've been working on scaling the DB up to contain the whole Pile in my free time. En route to this, I've made a few changes that you might be interested in merging:

Ditch autofaiss in favor of manually constructing the FAISS index to be as close to SCANN as possible
Add support for training the index on GPUs
Parallelize chunks_to_embeddings_ by adding a "worker_id" param
chunks_to_precalculated_knn_ should be able to reuse pre-computed embeds etc.
Add a BertEmbeds class that supports memmap'd files > 16 TB (max file size on ext4)
Add support for going from jsonl->chunks (which is what the Pile is) in addition to txt->chunks (just for convenience)

Index creation at scale isn't really tested (but I have tested it at smaller scales). I'm running embedding at scale right now and I think it works. Anyway, I don't really expect you to just merge this but figured I'd mention it before I get too far off master.

Also random, but did you work on Bliss at Uber? I think I might have been an intern on your team lol.

@mitchellgordon95 ohhh my god, yes, I worked on Bliss ... I think I remember you now! Lol

🦦 🦦 🦦

@mitchellgordon95 this looks good, but i hesitate to merge it because of so many changes. we could always keep it open and i can gradually incorporate some of the ideas (parallelizing chunks to embedding is a great idea!)
thank you for sharing regardless!

Also random, but did you work on Bliss at Uber? I think I might have been an intern on your team lol.

At Uber I worked on toolshed for a while, building out Atreyu, our crappy GraphQL clone. Then Bliss for the remainder of my time there, both the node backend and react frontend

@mitchellgordon95 where are you working now? is it Latitude games?

@mitchellgordon95 how much of a speed up are you seeing removing autofaiss in favor of something closer to scann?

@mitchellgordon95 some benchmarks would definitely help sway me towards extra complex code :)

Sure! I'm planning on testing out the indexing as soon as the embedding is finished, so I can run some benchmarks on auto-faiss in parallel.
It's actually been so many weekends since I switched away from auto-faiss that I forget exactly why I did it 😅, but I know I must have had a good reason because it was a PITA to set up.
I think one reason was that auto-faiss doesn't support index training on GPUs (which can be very slow for 64M clusters). Another reason was I wanted to make sure that we fully optimized memory usage since The Pile is around 5.8B chunks. Even with the full PQ compression etc., it still ends up being 8 bytes per embedding ~= 43 GB of RAM to store the index.

Yeah I'm at Latitude. It's not a priority project, but I've used my last 3 hackathons to work on it lol

@lucidrains @mitchellgordon95 hi gang! I have been working on a training loop for this code, which you can see here:
https://github.com/artificialwisdomai/origin/pull/50
We are doing something atypical, and may need to fork and reimplement both of your implementations. I am aware of the minimum requirements of the license, but I wanted to ask, would you take offense? We will reference the upstreams that generated our ideas (ie this repo, and this PR).
We may not submit the work upstream as it would be a fresh start, although our code is licensed with ASL2.
Also, do either of you have any examples of inferencing or use cases where this type of retroformer is of use?
Thank you,
-steve

Hello,
I have done a basic benchmark of training. If you can provide other benchmark suggestions, happy to provide A/B/A comparisons and report in this PR.
Overview
Artificial Wisdom™ Retrieval Transformer benchmark results

Test A: PR #22
Test B: https://github.com/lucidrains/RETRO-pytorch@ab3c4a6

System under test

faiss built with Artificial Wisdom™ Wiki Faiss Build
Indexing content: [Istio.io documentation](https://github.com/istio/istio.io/https://github.com/istio/istio.io/tree/master/content/en]
glob: **/*.md
NVIDIA CUDA: 12.1
One NVIDIA A40
One NVIDIA AA30
Debian 12
Training loop: Artificial Wisdom Initial Retroformer proof of concept

baseline
wallclock
(baseline) sdake@beast-06:~/repos/origin/retrieval$ REPROCESS=1 python train.py
Artificial Wisdom™ Retreival Transformer Training
• retrieval_model=artificialwisdomai/retroformer • foundation_model=mosaicml/mpt30b •
Epoch 0 100%   ━━━━━━━━━━━━━━━━━━━━━━ • retrieved=65568 • loss=3.89 • 0:14:15 • 0:00:00

observation

GPU consumed memory: 33.2 GB @ 1 A40
GPU utilization: 50-70% @ 1 A40

PR rework
wallclock:
Artificial Wisdom™ Retreival Transformer Training
• retrieval_model=artificialwisdomai/retroformer • foundation_model=mosaicml/mpt30b •
Epoch 0 100%   ━━━━━━━━━━━━━━━━━━━━━━ • retrieved=65568 • loss=3.82 • 0:14:25 • 0:00:00

Observation

GPU consumed memory: 33.2 GB @ 1 A40
GPU utilization: 50-70% @ 1 A40
Three other GPUs: 2 GB @ 3 A30
GPU utilization on three A30s = 0%.
a system without autofaiss is valued for many reasons.

My general observation is that faiss does not appear to use compute in the GPUs, only memory. There may be a defect in our build, or in the implemetnation.
cc @rstarmer @MostAwesomeDude.
Thank you,
-steve

yes absolutely! the giving is unconditional, thus MIT

have you seen Nvidia's follow-up Retro2 paper yet?

@lucidrains I haven't, if you can share or have the title, i would love to see it! Robert found [Megtrron RETRO](https://github.com/NVIDIA/Megatron-LM#retro], although i don't know if this is what you were referencing.
We are building a library that composes the following things:

Three retrieval transformer architectures
Three large language models
Three vector stores

You pick 1 from each of the three categories, and can use them in composition. Unlike  langchain, our work is designed around using shared memory for API communication, instead of HTTPS. IE.
Would love to have further suggestions for the idea proposed here. I understand if the MIT license requirements are met, then the software is licensed with those terms. As an open source dev, as long as someone using software I wrote complies with the terms, I was always good with however they used it.
What I am asking is a little different. If I were to use your code as a reference for ABA testing, and also to learn from, but didn't integrate your library, would you take offense? I never did, but I was and AM all in open-source. Many don't understand the finer mechanics of ASL/MIT/BSD-revised, and do take offense. It sounds like you are very experienced in this area, which is awesome!
Your code is an invaluable resource to the engineering community. Thank you for your gifts.
Thank you,
-steve

I did notice, after switching to a larger dataset, specifically https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T-Sample, the act of building the embedding indexes is "different", and possibly faster. I will publish an update comparison when i have one to give.

thanks for the kind words
you are free to use this repository however you wish, no conditions
https://arxiv.org/abs/2304.06762 this is the paper. I haven't gone through it but the author apparently found some further simplications. what they have in the Megatron repo should be this Retro++

Thank you Lucid. I am running gcc12, which causes all sorts of issues with apex. Therefore, An evaluation of the retro implementation in Megatron-LM is taking some time.
Do you have a forum for your work, such as discord or other forms of communication? It's obnoxious to contact someone directly (sort of what i am doing now) while adhering to the open source norms.
We also have an ultra-simple mixture of expert architecture we are developing. I can do my research, although if you have any thoughts on multi-model multi-head attention or are aware of related work, I would love one link in the chain (I can pull the rest).
We will publish our completed work. An adequately sourced paper submission may be beyond our expertise, although, as applied scientists, we understand the value of reproducibility.
Thank you,
-steve


Rename libkpod files to follow Go conventions

Fixes #710

:+1:

LGTM

:+1:

One of the lint tools should have caught it. Well, we could contribute a checker to one of the lint tools :)

@mrunalp @runcom Passes lets merge it.


message_edit: Use single letter truncation indicator

This is a follow-up to #23544.
We save more space for the truncated topic name by switching from
"..." to "…" to indicate truncation.
Note: verify that this will not cause emails to be sent to different topics than the ones truncated before the change.
Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

I think this is definitely going to change the truncated topics and thus their threading. It's potentially reasonable to make the change anyway. But this is the kind of change that would be relatively cheap to include if merged alongside a change in the maximum length for topics. So it might be worth a discussion in #design before trying to integrate this.
(Truncated topics are not incredibly common).
Unfortunately, I don't see an easy way for us to do a database migration to change existing truncated topics to match what the new algorithm will generate.

Thinking more, I think the disruption isn't worth the benefit. I think probably we should just pause work on this, and make a note to include it if we're ever making the related backwards-incompatible change of adjusting the maximum length of the topic field.

Yes, I agree that we can pause on this. I think we currently don't have a convincing reason to raise the priority of the migration. This is the kind of improvement that should be done only when the cost is cheap.


Add chart / graph for the aggregated reviewer stats

Follow up from #215

What does it mean to have reviewer stats aggregated?


Load slides from JSON

Use a JSON object to 'build' the slides up. Structure something like:
{   presentation: {     title: "",     presenterName: "",     slides: [       { title: "", subtitle: "", content: ""},       { title: "", subtitle: "", content: ""}     ]   } }

will still take a while until I find time to setup compiling presentations together from separate files.


Can't get it to work on jupyterlab 3.0.2

Hi!
I tried to install it through pip, which worked but it is not working in the application. When I tried to install it through the Extension Manager, I get this problem:
Extension Installation Error
An error occurred installing @aquirdturtle/collapsible_headings.

Error message:

"@aquirdturtle/collapsible_headings" is not a valid npm package

Hope you can help!

Hi @j0kaso, hmm, strange. With npm, I was running into some similar error messages in publishing my extension to npm, although after I managed to squirm around the errors I haven't been having trouble installing from the published version on my various devices. Maybe try updating nodejs? conda update -c conda-forge nodejs. I have less experience with using pip to install tbh. It sounds like pip reports that it installed successfully but that simply nothing shows up in jupyterlab? Can you trying to use the browser debugger tools (hit f12 in the browser) and see if there are any messages regarding the extension? You should see JupyterLab extension @aquirdturtle/collapsible_headings is activated!! in the console of the debugger. If none of these things gives an obvious solution, can you try making a fresh conda (or similar) environment and only installing jupyterlab and the extension?
E.g. do the following in a prompt with admin priviledges (you may need to uninstall the pip version first pip uninstall aquirdturtle_collapsible_headings, not sure):
conda create -n myenv jupyterlab nodejs
conda activate myenv
jupyter lab

and then install the extension using the extension manager. You may want to make sure that jupyterlab >= 3.0.0 and nodejs >= 12 (my current versions which are working are 3.0.3 and 15.3.0 respectively). Let me know if any of these things helps.

Hi @j0kaso, hmm, strange. With npm, I was running into some similar error messages in publishing my extension to npm, although after I managed to squirm around the errors I haven't been having trouble installing from the published version on my various devices. Maybe try updating nodejs? conda update -c conda-forge nodejs. I have less experience with using pip to install tbh. It sounds like pip reports that it installed successfully but that simply nothing shows up in jupyterlab? Can you trying to use the browser debugger tools (hit f12 in the browser) and see if there are any messages regarding the extension? You should see JupyterLab extension @aquirdturtle/collapsible_headings is activated!! in the console of the debugger. If none of these things gives an obvious solution, can you try making a fresh conda (or similar) environment and only installing jupyterlab and the extension?
E.g. do the following in a prompt with admin priviledges (you may need to uninstall the pip version first pip uninstall aquirdturtle_collapsible_headings, not sure):
conda create -n myenv jupyterlab nodejs
conda activate myenv
jupyter lab

and then install the extension using the extension manager. You may want to make sure that jupyterlab >= 3.0.0 and nodejs >= 12 (my current versions which are working are 3.0.3 and 15.3.0 respectively). Let me know if any of these things helps.

Hi @aquirdTurtle, thanks for your quick reply.
Seems my problem is because I'm using a forwarding from a remote server. I did run it locally, could install it, and then started the automatic forwarding - seems there is an installation mismatch with nodejs on my side. Maybe not the best solution for how I circumvent it but it worked! Thanks again so much for your help!

Hi @aquirdTurtle, thanks for your quick reply.
Seems my problem is because I'm using a forwarding from a remote server. I did run it locally, could install it, and then started the automatic forwarding - seems there is an installation mismatch with nodejs on my side. Maybe not the best solution for how I circumvent it but it worked! Thanks again so much for your help!


[deps] kube update

Motivation
Handle the oh-so-annoying shared update of kube/k8s-openapi which must be updated together.
Have you read the [Contributing Guidelines on pull requests]
Yes
Test Plan
A kinda-sorta-speedy CI run.
Related PRs
kube update: https://github.com/diem/diem/pull/7344
k8s-openapi update: https://github.com/diem/diem/pull/7350

/land

/land


Fix llama_cpp cuda usage in Containerfile

Problem
The current cuda builds of instructlab are unaccelerated for serve/generate
Changes
Sync requirements.txt of  llama_cpp_python with instructlab
The step which pip installs llama_cpp_python is overwritten by the instructlab install since the instructlab has a pinned version that is lesser than the preceding llama_cpp_python. Since the instructlab install does not specify the cuda CMAKE args it gets rebuilt without cuda support. By aligning requirements we ensure that  llama_cpp_python will not be rebuilt.
Before this change:
>>> llama_cpp.llama_supports_gpu_offload()
False

After:
>>> llama_cpp.llama_supports_gpu_offload()
True

@n1hility @russellb
The workaround in this PR is problematic. It doesn't use the current requirements.txt but the version from the main branch. This is likely to cause issues when we create a stable branch and modify requirements in main. Also pip install -r ... will install all dependencies in the requirements file.
I'm using this approach in my container files, which I believe is better:

copy requirements file from context into the container
convert the requirements file to a constraints file by stripping off optional dependencies
pip install with -c instead of -r

COPY requirements.txt /tmp
RUN sed 's/\[.*\]//' /tmp/requirements.txt >/tmp/constraints.txt
RUN CMAKE_ARGS="-DLLAMA_CUBLAS=on" python3.11 -m pip install -c /tmp/constraints.txt --force-reinstall --no-cache-dir llama-cpp-python

Thanks @tiran -- good feedback. you're right.
I think there's a simpler change needed at the moment. This Containerfile is not building instructlab from the current source tree. It's explicitly installing stable. We should just change the URL to use stable, as well.  I think this should still be improved to be more flexible with what it's installing, but that's at least a quick fix improvement.


Restructuring API security docs

Purpose
Restructuring API security docs

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Extensions EIP712: a few improvements

[x] Same text size on input / textarea

[x] Loading button / spinner when sending the request (specially backend)

[x] Have some info about EIP 712 in the page (a paragraph with a small description and a link to https://eips.ethereum.org/EIPS/eip-712 and hinting to check the eip-712/page.tsx). We can tinker in the PR (deciding the text, etc)

Ohh while testing https://github.com/scaffold-eth/create-eth/pull/57 noticed that the prettier formatting for this two file is not correct:

maybe @rin-st, you could do npx create-eth@latest -e eip-712 create an instance copy paste its formatting to this repo extension files

Thanks! Updated it ded3f2a
As I understand formatting inside extension does not matter since everything is formatted when creating instance? Am I missing something?
And if we need the same formatting in extension as in create-eth, we need the same prettier rules as in create-eth folder.
https://github.com/scaffold-eth/create-eth/pull/57 solves this problem but I think we need to remember that if someone will create extensions other way, formatting most likely will be different

As I understand formatting inside extension does not matter since everything is formatted when creating instance? Am I missing something?

Ohh yes! there is no problem here(while consuming the extension through npx create-eth@latest)
But when you are testing extension, in this case while I was testing eip-712 extension:

Clone this extension in externalExtensions dir of create-eth repo :
git clone https://github.com/scaffold-eth/create-eth-extensions.git eip-712 -b test-712

Now when I run :
yarn cli ../test-blah -e test-712 -s hardhat -i

Since the cloned extension is symlinked with instance created, and the created instance code was formatted afterwards, hence I got the diff in extension since orignally extension code was not formatted.
But nevertheless you fixed it at https://github.com/scaffold-eth/create-eth-extensions/commit/ded3f2ad3f11f33b227b88a0b8ed7259f479c265

but I think we need to remember that if someone will create extensions other way, formatting most likely will be different

Yeah I agree! but I think we should tell people to use this create-eth repo while creating / testing extensio to use create-eth repo and avoid doing changes to their extensions directly, instead use symlinked instance to do the changes

Thank you for detailed explanation, Shiv! 🙏  I tried the same but without cloning, just copypasted, so I didn't see the diff

Yeah I agree! but I think we should tell people to use this create-eth repo while creating / testing extensio to use create-eth repo and avoid doing changes to their extensions directly, instead use symlinked instance to do the changes

Yes! 👍

Closing this since improvements added. Thank you guys!


HList() exposes HNil.type

As most HList ops don't currently handle HNil.type properly, it's probably not a good idea to have any shapeless API expose it. In fact if you make HNil private, you can find any instances of this happening in the library.

How would you rewrite,
val l = 23 :: "foo" :: true :: HNil

If the HNil object was private?

Would a better solution not be to change :: on the HNil from def ::[H](h : H) = shapeless.::(h, this) to writing the return type explicitly as the trait rather than the singleton type?

@yilinwei Adding the explicit result type H :: HNil to :: on trait HNil is a good idea. I think that's orthogonal to this issue though.

@milessabin Yeah before I added the "private" to HNil i suspected that you wouldn't be able to do:
val l = 23 :: "foo" :: true :: HNil
But when I tried that in a library i'm writing against shapeless it still worked.. I don't pretend to know why though..
I guess it's probably not worth merging the PR anyway if it breaks binary compatibility, it's generally something that can be worked around but it's something that may trip up people new to the library.
Also I believe HNil's :: already infers HNil instead of HNil.type


Paragraph/line break not working in comments box on Comms tab

Description
Paragraph/line break not working in comments box on Comms tab

Can you point to an example? Can not replicate this in my setup.

This is what I type into the comment, but the rendering does not show line breaks. It is difficult to communicate with applicants when we can't write in easy to read paragraphs.

Your issue is that the margins between p-tags are not the same in the editor and in the preview.
This is an TailWind issue, it resets margins on many elements to 0 so each site can customise it.
We missed that Hypa does not set any margins for p-tag, easy fix.


from threading import Thread gives Error

Hi.
I tried to use codon for this code  (This is a simple code for faster video processing using Multi-Threading in Python) but it raised the below error:
video_processing_parallel.py:4:23-29: error: cannot import name 'Thread' from 'threading'

Using:
Ubuntu 22.04 LTS
Python 3.10.6
Thanks.

Unfortunately class Thread is not yet implemented in Codon. You can use @par for now; this should be fixed in subsequent releases.

Merging into #608 .


flip: init at v1.40-pre

Motivation for this change
Patch tool for IPS and BPS files on NixOS.
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[X] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

There seems to be another PR open for this package already.

Thank you, I forgot to check! Closing, looks like that PR is more appropriate.


0.8.3 high severity vuln

Node version (or tell us if you're using electron or some other framework):
Node 8+
ShellJS version (the most recent version/Github branch you see the bug on):
Operating system:
macOS
Description of the bug:
Received warnings on PM2 that shelljs got tagged for a security vulnerability.
WS-2017-3737
More information
moderate severity
Vulnerable versions: <= 0.8.3
Patched version: No fix

Shelljs 0.8.3 and before are vulnerable to Command Injection. Commands can be invoked from shell.exec(), those commands will include input from external sources, to be passed as arguments to system executables and allowing an attacker to inject arbitrary commands.

GitHub reported this on some of my repos too and I'm trying to understand what it's complaining about. Yes, "those commands will include input from external sources", since it takes strings... What's the actual method and problem? It's strange to see this pop up without a fix or issue ticket referenced.

Additionally the security warning references this commit.
With my understanding it appears to be functioning as intended and could just be caught in an automated audit that doesn't take into account the very purpose of the package?
Perhaps someone with more insight could chime in.

@thisBrian that commit didn't land.
This is a duplicate of issue #495 (also filed under issue #810).

The vulnerability is limited to only shell.exec(), other commands are generally safer (and should be entirely safe with shell.set('-f')).

Does anyone know if the security vuln notification was removed/ignored from Snyk?

@dannypaz I still see it https://snyk.io/vuln/npm:shelljs:20140723

Got the same notification from Githubs automated system

I've got the same security alert from GitHub 2 days ago

Any update?

For an update for you all. It must be understood that if you use the exec function in this library you must sanitize the input yourself. From what I understand, library itself isn't vulnerable, it is the usage of the library that has the potential*.
I've reached out to whitesource software (where the origin of the github security alert is coming from) and let them know that this might be a NOOP.
At this point, the resolution is to use the github tooling or snyk tooling and ignore the alert. For github, you can view https://help.github.com/en/articles/viewing-and-updating-vulnerable-dependencies-in-your-repository (look for the dismiss button)

Thanks, @dannypaz. Yes, this method's input must be sanitized (also: there are libraries which can sanitize input for you, but I don't have a library I'm yet ready to recommend). shelljs-exec-proxy provides a lot of protection out of the box, although I'm not sure how well it works on Windows (help for https://github.com/nfischer/shelljs-exec-proxy/issues/17 would be appreciated).
I will acknowledge sanitizing input is non-trivial, and easy to mess up. For this reason, I still intend to provide a better (meaning, easier-to-use-correctly) replacement to exec() (this work is in #866). But the vulnerability Snyk reported is fundamentally part of the exec() API, and cannot be fixed in a backwards compatible way.

I do take this report seriously, and I will try to prioritize quick patches to the documentation to explain this better. I will also see if there's an input sanitization library I can recommend, and if I can write up general advice for using ShellJS securely.
It will be significantly harder to push #866 to completion, and may not be technically possible on Windows. It's really difficult to provide a meaningful extension to child_process.{spawn,execFile}Sync() while keeping a comparable interface. If those methods satisfy your use case, feel free to use them.

@nfischer we have had the exact same problem with execa. Snyk flagged us for using the shell option, but that's a little silly. This option is indeed bad in 99% cases, but there are some legitimate use cases.
After discussing with Snyk, they removed the warning. We also added an alternative method that allows passing command + arguments as a single string, but without using the shell option. I believe many users use the shell option because they want that, without understanding that their command will be prone to command injection, slower and less cross-platform.

I'm in communication with Snyk right now. We've come to the mutual conclusion that it probably doesn't make sense to label ShellJS as "vulnerable," given shell.exec() is working as intended and it's the responsibility of dependent modules to sanitize input.
I've so far landed #950 and #951 to document this, and I've written security guidelines for using the project (basically, the stuff I said I would do in https://github.com/shelljs/shelljs/issues/945#issuecomment-505207287). Input from others is welcome, otherwise you can expect an update when I hear back from Snyk.

Same here - getting this alert from GitHub. So, can we expect this to be patched on 0.8.4? Any thoughts on when that might be out?

I totally agree with the statement that this is not an issue with shelljs. The responsibility to sanitise the input should be on the module which depends on shelljs. I mean looking on PHP for instance, try just sending the user supplied values straight into shell_exec() and see what happens =)
https://www.php.net/manual/en/function.shell-exec.php

@kabirbaidhya no, we will not be pushing out a patch release for this, as there's no vulnerability for us to patch (kindly see the rest of this thread). The exec() API is working as intended; to "fix" the "vulnerability" would be to break the API contract (which I don't intend to do, but would at the very least require a semver bump).

getting this alert from GitHub

The alert is created by Github's partner, WhiteSource. I'm trying to get in contact with them to take down this false positive or to at least discuss the report. I would greatly appreciate if folks on this thread can help me get in contact with this group (I'm not much of a twitter user, and I couldn't figure out the right email on their site).

@nfischer After a cursory search, I found the product VP on Github (@habushaws). Maybe they can help?

@nfischer Makes sense. Thanks.
Also, the security guidelines makes it clear, which we'll follow; thanks for explaining it there.

@nfischer @kabirbaidhya thanks for the great catch. We're looking into it and will keep you all posted. We're examining whether the vulnerability is not in shelljs, but rather in the native shell.exe, therefore, it may be the responsibility of the caller to ensure how to use it. Some users will assume it as the responsibility of shelljs, but others will not. Again, we're looking into it and will update soon.

@habushaw From our perspective, it must be the caller's responsibility. The JavaScript language simply does not allow us, in our module, to distinguish between shell.exec('intentionalCommand1 && intentionalCommand2') vs. shell.exec('intentionalCommand ' + unsanitizedUserInput). Please see https://github.com/shelljs/shelljs/issues/945#issuecomment-506019725 for a summary of the steps I've taken to inform callers of how to use this API correctly.
Thank you for prioritizing this. I've also sent an email to<EMAIL_ADDRESS>(your site didn't leave a contact for contesting vulnerability reports), so please reach out to me via email with further questions.

@nfischer @kabirbaidhya after a careful examination of this vulnerability, we've decided to remove it from our database. Thanks again for the great catch.

your site didn't leave a contact for contesting vulnerability reports

@habushaws what's the best method for contesting a vulnerability? Would be good to know when/if something like this happens again in the future. Thanks

@nfischer @kabirbaidhya after a careful examination of this vulnerability, we've decided to remove it from our database. Thanks again for the great catch.

When this vulnerability alert itself has been taken down from GitHub now, should this issue be closed?

Asking here since @habushaws is here. If I use shelljs in one of my packages and accept raw user input to be executed then will my notice be retroactively removed to my downstream? I have people on my issues board asking about this issue and just want to know how to present accurate facts around our usage of shelljs.

As this has been removed by both WhiteSource (https://github.com/shelljs/shelljs/issues/945#issuecomment-512044653) and Snyk (https://snyk.io/vuln/npm:shelljs:20140723), I consider this issue to be resolved.
Any dependent modules stumbling across this issue should consult our security guidelines.

auditjs is still reporting an issue due to it still being an open vulnerability on sonatype - https://ossindex.sonatype.org/vuln/560f37f7-6c1d-4b9c-aad3-34dcfcf218d7

@markhibberd I believe sonatype is now updated: https://ossindex.sonatype.org/component/pkg:npm/shelljs. Thanks for raising this to my attention.

                                 Manual Review
             Some vulnerabilities require your attention to resolve

          Visit https://go.npm.me/audit-guide for additional guidance

  Low             ShellJS exec() Function Arbitrary Command Injection Weakness

  Package         shelljs

  Patched in

  Dependency of   shelljs

  Path            shelljs

  More info       https://vulndb.cyberriskanalytics.com/vulnerabilities/151992

Maybe someone can reach out to cyberriskanalytics.com?

@IdanAdar how did you get this warning? Does this come from npm audit by default, or did you configure something?

Didn’t change anything. Straight forward npm audit

Huh, I don't get the same warning. What npm version?

6.10

I just tried with 6.10.3 and couldn't repro. I wonder the module you're in (or perhaps a company-managed registry) is adding some non-default auditors.

@nfischer It seems this is being reported only when .npmrc points to an npm repo in jFrog artifactory (which I guess has some advisory databases in its Xray product).
If you scan w/out artifactory, it's not reported...
It seems there's no way to login there to see what the advisory is about.

I've contacted cyberriskanalytics and checking if they are going to close their advisory based on this thread and given the several vendors have done so.
@nfischer Please let know how how to contact you via email for some additional information.


Verification failed

After re-install app - no first syncing

subspace-desktop.log

Closing, since this is an issue related to monorepo, it has nothing to do with desktop code.


feat(*): bluetint palette

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.007%) to 79.965%

Totals

Change from base Build<PHONE_NUMBER>:
0.007%

Covered Lines:
7715

Relevant Lines:
8755

💛  - Coveralls


Fix userdata compression

Summary
When the UncompressedUserData field is nil on CloudStackMachine instances CAPC is interpreting it as true.
This reworks the compression decision logic to give a nil UncompressedUserData the same semantics as false.

Codecov Report
Patch coverage: 62.50% and project coverage change: +0.09 :tada:

Comparison is base (0a451ed) 34.06% compared to head (10fc006) 34.15%.

:exclamation: Current head 10fc006 differs from pull request most recent head 02ddc0d. Consider uploading reports for the commit 02ddc0d to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #254      +/-   ##
==========================================
+ Coverage   34.06%   34.15%   +0.09%
==========================================
  Files          43       43
  Lines        3907     3914       +7
==========================================
+ Hits         1331     1337       +6
  Misses       2394     2394
- Partials      182      183       +1

Impacted Files
Coverage Δ

pkg/cloud/helpers.go
59.09% <60.00%> (ø)

pkg/cloud/instance.go
78.29% <60.60%> (+0.03%)
:arrow_up:

api/v1beta2/cloudstackmachine_types.go
44.44% <100.00%> (+15.87%)
:arrow_up:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

/assign @weizhouapache @vishesh92

/ok-to-test

/approve


[Feat]: A Cron Job for Stale

resolved #143

So to be clear this will clean up both stale issues and PRs in one go? Because I had two seperate requirement sets for PRs and Issues, that is why I seperated them.

This affects stale pull requests, however, you were assigned the stale issues, as well the number of days is too long for stale pull requests as it should be only 21 days, not 60.
So, to be clear.

Change this to issues
The number of days for issues is 60 so this is fine.

If you want to also do the stale pull requests please just ask to be assigned it and I will do that.

Sorry for the mix-up. I have worked on both issues, you can assign the other to me as well


[WIP] ✨ Compact Checkbox

Resolves #1033

@mimarz Not sure to leave the label as not required, or if there's a smart way to allow a blank label when people are using aria-labelledby instead? 🤔  Should probably be careful with this as people may misuse or misunderstand


Author contributions

@aarmey the authorline is based on what is currently in the manuscript as it is. after evaluating the manuscript in current form, nearly everything presented (ie all the data, model, and writing) was performed in substantial form by SV, NKN, and AW. however, AF and AL have done a lot of work for the project that did not see its way to the manuscript in its current form. likewise, most of the work performed by FM on thmm is not directly presented in the manuscript in its current form. though i know FM has been working substantially on the project and she has results that will soon be a part of the paper (ie cell cycle phase, etc). there is a lot at play and i dont think this is a finalized authorline (though the contribution descriptions should be relatively robust in the current form) and i decided to keep everyone on for the time being with understanding that things can possibly change. i should also note that i consulted the first author @shak360 before i made this decision. we acknowledge that you have the final say in the authorline. let us know what you think and i am happy to talk more about this.

#117

Also you should use some consistent rule for initials ordering (just so it's not random). Say alphabetical?

Also you should use some consistent rule for initials ordering (just so it's not random). Say alphabetical?

done


Java SDK Support

Java [Gradle/Maven] Support for flargd

Duplicate of https://github.com/pmbanugo/flargd/issues/5


createFileActivityStream is called way more than needed

I was debugging why Beaker was telling me about too many event emitters registered when I visit Rotonde. We may have a leak, but I found that createFileActivityStream is being run more than once (much much more than once) for each feed. It should only ever happen one time per feed, and then you should hold on to that reference.
https://github.com/Rotonde/beaker/blob/6590f0f2671db90033d09c8170e3dc3f17499585/scripts/feed.js#L79

Oh, yeah I re-set it on every refresh. I will fix that up!

Fixed in https://github.com/Rotonde/beaker/commit/38e6e120ef1257bd64eb442f08fa1da3766d5e03

👍


Multiple redirections to the azure sso page in a loop after authentication

Core Library
Passport Azure AD (passport-azure-ad)
Core Library Version
4.3.5
Wrapper Library
Passport
Wrapper Library Version
0.6.0
Public or Confidential Client?
Confidential
Description
I am using passport-azure-ad lib to authenticate users on server side in Nodejs + Express server
When I enter the login url in the browser the redirection to the sso page is successfully done. After I select an user for authentication it is redirecting again to the sso page instead of going to the redirectUrl provided in the OIDCStrategy config.
I dont believe that the callbackurl provided in oidcStrategy is getting executed, as i am not able to see the logs from the function or error or anything.
If i remove the authentication middleware from the callbackurl endpoint, it redirecting to the "/" but with it in place, it is not redirecting the user to failureRedirect url as well, so i dont know whether the middleware check is getting completed or not
I have configured the api url as redirect url under web section in azure ad
and react app endpoints under spa section in azure ad
MSAL Configuration
No response
Relevant Code Snippets
export const oidcStrategy = new OIDCStrategy(
  {
    identityMetadata:
      "https://login.microsoftonline.com/<tenantId>/v2.0/.well-known/openid-configuration",
    clientID: "<clientid>", // Replace with your own client ID
    clientSecret: "<clienrtsecret>",
    responseType: "code id_token",
    responseMode: "form_post",
    redirectUrl: "http://localhost:9093/api/auth/callback/azure-ad", // Replace with your own redirect URL
    allowHttpForRedirectUrl: true,
    validateIssuer: false,
    passReqToCallback: true,
    scope: ["openid", "profile", "email"],
  },
  function (
    req,
    profile,
    done
  ) {
    if (!profile.oid) {
      return done(new Error("No oid found"), null);
    }
    return done(null, profile);
  }
);

// Serialize user
export const serializeUser = (user: any, done: any) => {
  done(null, user.oid);
};

// Deserialize user
export const deserializeUser = (id: any, done: any) => {
  done(null, user);
};

passport.use(oidcStrategy)
passport.use(serializeUser)
passport.use(deserializeUser)

router.get("/api/auth/login", passport.authenticate("azuread-openidconnect"));

// Callback URL for Azure AD to redirect to
router.post(
  "api/auth/callback/azure-ad",
  passport.authenticate("azuread-openidconnect", {
    authInfo: true,
    failureMessage: "Cannot login",
    failureRedirect: "<REACT_APP_URL>/error",
    successRedirect: "<REACT_APP_URL>/success",
  }),
  (req, res) => {
    res.redirect("/");
  }
);

Identity Provider
Azure AD / MSA
Source
External (Customer)

@hectormmg hey can you take a look at this

@rkreddy99 unfortunately passport-azure-ad is no longer supported, we only provide support for security updates.

@hectormmg then what is the lib that i can use for server side authentication using azure ad in nodejs + react typescript app


[2.0.x] AF-434: [HTML editor] Font size, Font color and Backgroud color are not working.

To fix this issue I ended up updating the whole HTML Editor component. Work done:

Upgraded the https://github.com/Voog/wysihtml JS library to latest version
Changed toolbar & menus to use the BS styling:

Changed Font Size menu to use a dropdown with fixed values instead of a textbox.

Changed Font Color & Backround Color menus to use the Color Picker component instead of a textbox

@ederign @paulovmr @jsoltes could you please take a look?

Amazing! This is a great upgrade :+1: jenkins execute full downstream build

jenkins execute full downstream build


Create Initial View

Create the view to introducing data:

Weight and Height Info

Name, gender, date of Birth

Start Weight and Date, Target Weight and Date

Resolve in pull request: https://github.com/EstefaniaGuardado/weightApp/pull/9


Option To Select Cluster Icon Color

What would you like to be added:
Option to select the cluster icon color
Why is this needed:
Just a nice to have. Not really need, but just would help to identify which cluster is which.

Fixed by #2153 and #2331

Similar to https://github.com/lensapp/lens/issues/3846


TemplateSyntaxError: Encountered unknown tag 'now'. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'.

Date you used Cookiecutter PyPackage:
20180215
Cookiecutter version used, if any:
Version: 1.3.0-2
Python version, if any:
Version: 2.7.11-1
Operating System:
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.1 LTS
Release:	16.04
Codename:	xenial

Description
What I Did
cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git
I pressed enter when asked to select the licence

Traceback (most recent call last):
  File "/usr/bin/cookiecutter", line 9, in <module>
    load_entry_point('cookiecutter==1.3.0', 'console_scripts', 'cookiecutter')()
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cookiecutter/cli.py", line 100, in main
    config_file=user_config
  File "/usr/lib/python2.7/dist-packages/cookiecutter/main.py", line 140, in cookiecutter
    output_dir=output_dir
  File "/usr/lib/python2.7/dist-packages/cookiecutter/generate.py", line 327, in generate_files
    generate_file(project_dir, infile, context, env)
  File "/usr/lib/python2.7/dist-packages/cookiecutter/generate.py", line 167, in generate_file
    tmpl = env.get_template(infile_fwd_slashes)
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 125, in load
    code = environment.compile(source, name, filename)
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 591, in compile
    self.handle_exception(exc_info, source_hint=source_hint)
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "./LICENSE", line 4, in template
jinja2.exceptions.TemplateSyntaxError: Encountered unknown tag 'now'. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.
  File "./LICENSE", line 4
    Copyright (c) {% now 'local', '%Y' %}, {{ cookiecutter.full_name }}

This was because I had the CookieCutter  installed with apt-get install  which was 1.3.0-2. Installing it using pip install -U fixed the issue.


[GitHub] Menubar notification should link using notification referer

Extension
https://www.raycast.com/raycast/github
Description
When opening a notification from the notification center on GitHub directly, the URL has a notitication referer ID appended to it, e.g. https://github.com/<org>/<repo>/<whatever>?notification_referrer_id=<ID>. I think this is what powers the banner to be displayed at the top where the notification can be marked as done aside from other actions. I assume this ID is accessible from the notification API endpoint (or at least I hope so).
This is currently not shown when going to the notification directly through the GitHub extension menubar item, but it would be great if it were!
Who will benefit from this feature?
Every user of the fantastic new GitHub notification menubar extension.
Anything else?
No response

Thanks for the feature request. Would you like to work on it? I can provide some guidance on how to contribute to this extension so that you can open a pull request. You can contact me here or in the Slack community if you need further help.

Hi @thomaslombart, after looking at it again, it seems that the extension already links to the correct URL, I unfortunately don't know what it could be doing differently to get the top bar to show :/


[macOS] Backward compatibility of Az and AzureRM will be dropped on December, 14

Breaking changes
We are going to stop adding backward compatibility of Az and AzureRM by Enable-AzureRmAlias cmdlet.
Target date
Image rollout will start on ~December, 7~ December, 14 and will take 2-3 days.
The motivation for the changes
Az.Accounts from Az module 5.х is not working correctly when this alias is enabled, the following warning is generated:
WARNING: Token cache encryption is not supported in current environment and token cache will be fallen back as plain text
Possible impact
If you are using cmdlet's with RM prefix — your builds will be broken.
Virtual environments affected

[ ] Ubuntu 16.04
[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[x] macOS 10.13
[x] macOS 10.14
[x] macOS 10.15
[x] macOS 11.0
[ ] Windows Server 2016 R2
[ ] Windows Server 2019

Mitigation ways
Run Enable-AzureRmAlias at the beginning of your build.

We are using Rundeck 3.1.2-20190927 running on Ubuntu and a node in Microsoft Windows Server 2016 . We use Rundeck to run powershell scripts that live on the Windows node. Until the last week, all was working well. But now all rundeck jobs start out with the warning "WARNING: INITIALIZATION: Token cache encryption is not supported in current environment and token cache will be fallen back as plain text" Even the scripts that are not using powershell. One of the scripts deploys VMs in Azure.
When we run the script on the powershell command line, it works. When we run it from Rundeck it fails. Rundeck starts with the above warning, and then Connect-AzAccount  results in the following error:
Connect-AzAccount : UsernamePasswordCredential authentication failed: Persistence check failed. Inspect inner exception for details
Again, running the same code on the command line works. The credentials are correct.
It looks like Az.Accounts was updated on Dec 9.
Directory of C:\Program Files\WindowsPowerShell\Modules\Az.Accounts
12/09/2020  01:27 PM              .
12/09/2020  01:27 PM              ..
12/05/2019  02:28 PM              1.6.4
07/31/2020  02:36 PM              1.9.1
12/09/2020  01:27 PM              2.2.2
Any ideas would be greatly appreciated

Hi @rmccarrick,
we didn't deploy any images last week, but I see Az.Accounts 2.2.2 was released exactly 8 days ago https://www.powershellgallery.com/packages/Az.Accounts/2.2.2, maybe you are installing it during the build.
Since this issue is not related to the image, please, create an issue in the official Azure-PowerShell repo  and module owners will investigate it. Thanks!

The mac OS rollout, without RM prefix, has been finished.


Reload "data"

Is it possible to tell impress to reload the presentation supposing that some javascript altered one of the div's attributes such as data-x, data-y etc?
I've tried to call impress().init() again to no avail. It would be a really nice improvement!
@bartaz If it's not there as I suspect, could you point me in the right direction to write it? I was trying to bind it to angular.js in some way, I suppose it would be very nice :)
Thanks

Closing if favor of #537.


Update documentation.

Dropped the sections on removing linting tools - we shouldn't recommend this.

Current coverage is 67.82%

Merging #212 into master will not affect coverage as of 0ba6337

@@            master    #212   diff @@
======================================
  Files           42      42
  Stmts          519     519
  Branches         0       0
  Methods          0       0
======================================
  Hit            352     352
  Partial          0       0
  Missed         167     167

Review entire Coverage Diff as of 0ba6337

Powered by Codecov. Updated on successful CI builds.

Thanks a lot @ben-eb!

You're welcome! :smiley:


[added] silent option

In isomorphic apps, react-a11y needs to run on server and client or there is a render checksum error and the client has to re-render. This silent option can be enabled on the server so that the console isn't spammed when doing a server-side render.
warn is still the default

It isn't valid to pass both silent and error. What about combining things into one level option that defaults to warn?

Yeah, that's better. Changed to level option instead.


[publish][m #32]: Upload README

Upload README file to S3

Coverage decreased (-0.8%) to 77.637% when pulling d500488b2fc1b7fd15940948f2a0f7d4d10725de on kittdotech:test-end-to-end into 9f4ec5aaa1643cf23ce582fed693e5020bbb7f46 on frictionlessdata:master.

Coverage increased (+3.8%) to 82.278% when pulling 4d5261ef724adad7874e2846ab2415e1861e04c5 on kittdotech:test-end-to-end into 9f4ec5aaa1643cf23ce582fed693e5020bbb7f46 on frictionlessdata:master.

In DPMPY publish flow, I have added Readme to upload to S3:

Get Token
Upload Datapackage.json
Upload Resources
Upload README[.] if exist else show warning message.
a) Search all README. in current DP
b) Get PUT URL to store in S3 (Similar to resource)
c) Upload the first README found
Finalize

@rgrp @zelima Please review

@kittdotech I still can't see test for

uploading README (without extensions)
Not uploading with wrong names E.g. README. or lowercased

Also Why do we have test_miscellaneous.py  in separate file?

Coverage increased (+3.8%) to 82.278% when pulling a3b92002295647afd3207d53d45abdfa5b53bd19 on kittdotech:test-end-to-end into 9f4ec5aaa1643cf23ce582fed693e5020bbb7f46 on frictionlessdata:master.

Added Test for Readme & Test in Readme Category.

Coverage increased (+4.06%) to 82.5% when pulling eb3a21490390c996a611048bd45fa64752c79e18 on kittdotech:test-end-to-end into 9f4ec5aaa1643cf23ce582fed693e5020bbb7f46 on frictionlessdata:master.

@kittdotech Looks good!
Could we include one more test: Check theres uploaded only one README if there is two or more valid README in directory.

Coverage increased (+4.06%) to 82.5% when pulling 0df909f414e824d322f66abb33ae6704e0e47bb1 on kittdotech:test-end-to-end into 9f4ec5aaa1643cf23ce582fed693e5020bbb7f46 on frictionlessdata:master.

Coverage increased (+4.06%) to 82.5% when pulling 4883b01a7dfc097e0715bbee30660a3e5b5b6429 on kittdotech:test-end-to-end into 9f4ec5aaa1643cf23ce582fed693e5020bbb7f46 on frictionlessdata:master.


Aggressive remove_dir_all

Let's see if this helps the flakiness.
cc @Diggsey @alexcrichton
cc @Aaronepower this PR contains a self-contained remove_dir_all module based on winapi.

Looks like we might need an agressive remove_file too...

I personally forget, but can you clarify how precisely this is more aggressive again?

It moves the files out of the subdirectory and up next to the directory being deleted, prior to deleting them. This means if the file deletion is delayed due to outstanding locks, then the directories can still be deleted successfully without waiting for other processes to close their file handles.
The failure here looks different - like a race condition in one of the self-updater tests possibly?

Updated to fix the linux build hopefully.

@bors r+

:pushpin: Commit b61370a has been approved by brson

:hourglass: Testing commit b61370a with merge 0acd9dd...

:broken_heart: Test failed - status-appveyor

Test failure was network error and i want to deploy this in next release.


Fixed Token Recognition Issue

Removed the \n token from one of the lines that seem to be causing
the issue

error: wso2/gmail:0.10.1::/src/gmail/gmail_data_mappings.bal:346:24: token recognition error at: '"Error occurred while converting sourceLabelJsonObject json to map of json. \n'
Resolves the Bad Sad Error that occurs when building the module as well

Related Issue : https://github.com/wso2/ballerina-integrator/issues/627

Tested with Gmail Connector Template. Fix works fine.


Fix Problems Running Heart Evaluation GUI Demo

docker/Dockerfile.tcf-dev: mention py-solc requires older solc compiler 0.4.25
docker-compose-sgx.yaml: add missing /tmp/.X11-unix volume for X Windows
docker-compose*.yaml: add missing DISPLAY environment variable
docker-compose*.yaml: set TCF_DEBUG_BUILD default to empty string
examples/.../heart_gui.py: make script executible

Signed-off-by: danintel<EMAIL_ADDRESS>

These problems were found while documenting the heart GUI demo (see PR #56 )


Build fails with "Maximum call stack size exceeded" when "concatenateModules" is enabled

Bug report

What is the current behavior?
In webpack optimization, with concatenateModules: true compilation fails with a Maximum call stack size exceeded error
If disabled, it builds as expected
If the current behavior is a bug, please provide the steps to reproduce.

The repo https://github.com/kuworking/webpack_concatenateModules_issue uses 11ty, and reproduces the problem in my computer
Please note than under Windows environment it does NOT work due to (presumably) a bug in 11ty
In my case (W10 user) I can run it successfully under WSL
git clone <<repo>>
yarn
npm run build
>>> edit `concatenateModules: false`
npm run build

What is the expected behavior?

With concatenateModules disabled the repo builds ok, but not when it is enabled
Perhaps there is a legitimate reason why the build fails (?)
Other relevant information:
webpack version:  5.4.0
Node.js version: v14.15.0
Operating System: W10 (WSL Ubuntu 20.04)
Additional tools:

webpack-remove-empty-js-chunks-plugin is unneeded with webpack@5, looks like without it build works.

Just to note that without the plugin activated ("webpack-remove-empty-scripts": "0.6.3") I do see empty js files, but this is likely an issue with webpack then

please create a separate issue


Implement vm heap

details:
Implement vm heap

[x] Test case

How about HAddress, Hdata?

How about HAddress, Hdata?

HData?

@hihiboss you mean free method exists to optimize?

@junbeomlee good!!


[test] Fix duplicate CHECK-DAG in reference dependencies test

An entry used to appear twice, but now it's nicely de-duplicated. However, FileCheck used to allow matching it twice (as a degenerate case of "overlapping CHECK-DAG"), so we never updated the test. Fix that, and check that the duplicate doesn't creep back in.
SR-8420

@swift-ci Please smoke test

Again, passed Swift tests before failing in LLDB tests for unrelated reasons. Merging.


could you share latent8_3d-lr1.00e-05-bs1-rs320-sr2-fr25/checkpoint-14000.ckpt

Hi team, great work. Wondering if you could share the checkpoint of causalVAE training. The current ckpt you are sharing only has the VAE part, but no discriminator, optimizer_state and scalar_state. I am thinking to resume from your ckpt to continue training. Thanks!

We are about to discard this version of VAE. as we will release a stronger, lower cost VAE.

ok, thanks!


[Question]  How can I install local client?

First of all, thanks for the great software.
I completed the installation based on Docker-compose and succeeded in creating a screenshot process using a simple Selenium example by following the tutorial.
However, when I run this process, there is no result. The problem seems to be that the local client is not installed, so I looked for a way to install the client, but I couldn't find a way to install the local client anywhere, so I'm posting this question.

Hi, you can find the local client here, including some setup instructions:
https://github.com/bptlab/ark_automate_local
However, since it is/was a prototype, it might require some more settings that are not documented to get it working.
Just reach out again if you require some assistance (even though I cannot promise that I will be able to solve every problem).
Would also be very happy to hear what you are up to and how you found this little project :)

Thank you for your response. It there any plans to develop a client with WebAssembly or automated installer? I found the Python runtime which is run on WebAssembly technology here - https://pyodide.org/en/stable/. Is it possible to run the robot framework including Desktop automation with this technology?

Sorry, currently, there are no plans for developing this research prototype any further.
It should be possible to use desktop automation features, but it would require extending the set of RPA Framework methods available for modeling in the software (there is currently only a limited set, like Browser and Excel automation).


Apply Gossipsub Params from the spec

Is your feature request related to a problem? Please describe.
Refer to https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md
D (topic stable mesh target count): 8
D_low (topic stable mesh low watermark): 6
D_high (topic stable mesh high watermark): 12
D_lazy (gossip target): 6
heartbeat_interval (frequency of heartbeat, seconds): 0.7
fanout_ttl (ttl for fanout maps for topics we are not subscribed to but have published to, seconds): 60
mcache_len (number of windows to retain full messages in cache for IWANT responses): 6
mcache_gossip (number of windows to gossip about): 3
seen_ttl (number of heartbeat intervals to retain message IDs): 550

Describe the solution you'd like

Apply respective params to lodestar
Prefer specifying params in lodestar instead of using the gossipsub default params
Probably wait for the new gossipsub work https://github.com/ChainSafe/js-libp2p-gossipsub/pull/206

@tuyennhv https://github.com/ChainSafe/js-libp2p-gossipsub/pull/206 is merged, issue unblocked


Feat/rayjob

What type of PR is this?
/kind feature

What this PR does / why we need it:
Adds RayJob support into kueue.
Which issue(s) this PR fixes:

Fixes #666
Special notes for your reviewer:
This implementation uses the not yet merged kuberay job suspend feature and is referencing the code from EPAM's fork.

/ok-to-test

/retest

could this be a flaky test in the existing suite?

could this be a flaky test in the existing suite?

100%, this PR has nothing to do with the scheduler, and none of the new code  (controller or webhooks) is executing in that test suite.

/retest

/cc @alculquicondor
/cc @mwielgus

Could you please open an issue and investigate?

Can you follow https://github.com/kubernetes-sigs/kueue/pull/668/files#diff-1dd4256189ca00ba75ebb5935fd7f7f403507b3d2e7033d8446c63b171715a52 to disable this by default?

Can you follow https://github.com/kubernetes-sigs/kueue/pull/668/files#diff-1dd4256189ca00ba75ebb5935fd7f7f403507b3d2e7033d8446c63b171715a52 to disable this integration by default?

done

The ray pr is here https://github.com/ray-project/kuberay/pull/926

@alculquicondor  the PR is rebased and  references to epam's fork removed, ready for review.
I'll do some extra in cluster manual test but I don't expect to have a big impact on this/

/retitle Integrate RayJob

/assign @kerthcet
Can you do a pass?

/test pull-kueue-test-e2e-main-1-24

Can we include ray to KnwonWorkloadOwner?
https://github.com/kubernetes-sigs/kueue/blob/74f45f69612a2d36ace91ae455f1e9fdbcd53037/pkg/controller/jobframework/known_frameworks.go#L25-L27

Can we include ray to KnwonWorkloadOwner?

For now, we don't need it , ray is not creating batch/jobs , going forward, maybe we should rework https://github.com/kubernetes-sigs/kueue/blob/74f45f69612a2d36ace91ae455f1e9fdbcd53037/pkg/controller/jobframework/integrationmanager.go#L41 to cover this for all registers frameworks.
Also  KnwonWorkloadOwner might not be the best name for this, we are not interested if we know the owner, we are interested if that Owner has it's own Workload  managed by kueue.

Also KnwonWorkloadOwner might not be the best name for this, we are not interested if we know the owner, we are interested if that Owner has it's own Workload managed by kueue.

Agree. Let's work on this at #782.
cc: @kerthcet


Class_weight with tf.dataset as input to model.fit will throw an error

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No Mobile device
TensorFlow installed from (source or binary):binary
TensorFlow version (use command below):1.10
Python version:2.7
Bazel version (if compiling from source): ---
GCC/Compiler version (if compiling from source):---
CUDA/cuDNN version: cuda-8.0
GPU model and memory: (Titan X and GeForce GTX 1080 )
Exact command to reproduce:

Describe the problem
I am usingtf.kerasin order to be able to feed the train_data using tf.dataset API through model.fit directly. It works fine whenever you didn't pass class_weight, but if you pass dict of class_weights, it will throw the following error :
AxisError: axis 1 is out of bounds for array of dimension 1
When I debug the error , I found the error happened exactly at line 531 :
> /usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_utils.py(530)standardize_weights()
    528       raise ValueError('`class_weight` not supported for '
    529                        '3+ dimensional targets.')
    530     if y.shape[1] > 1:
-->  531       y_classes = np.argmax(y, axis=1)
    532     elif y.shape[1] == 1:

The dimension of y is TensorShape([Dimension(None), Dimension(8)])
So y.shape[1] > 1 , but the problem y is a tensor now not an numpy array, that is why it throws the previous error.
So is there any solution to this situation?
EDIT: Adding sample code to reproduce the error.
import os, sys, logging
import numpy as np
import tensorflow as tf
import itertools as itt
logging.basicConfig(level=logging.INFO)

def _int64_feature(value):
  return tf.train.Feature(int64_list= tf.train.Int64List(value= [value]))

def _bytes_feature(value):
  return tf.train.Feature(bytes_list= tf.train.BytesList(value= [value]))

def _float32_feature(value):
  return tf.train.Feature(float_list= tf.train.FloatList(value= [value]))

def tf_records_creating(tfrecord_file):
    logging.info('Creating random tfrecord files for 100 sample')

    labels = np.random.uniform(0, num_classes, total_train).astype(np.int32)
    data = np.random.uniform(0, 255, total_train*224*224*3).reshape(total_train, 224, 224, 3).astype(np.int32)

    writer = tf.python_io.TFRecordWriter(tfrecord_file)

    for idx, (image, label) in enumerate(itt.izip(data, labels)):
        image = image.tostring()
        example = tf.train.Example(features=tf.train.Features(feature={
            'label': _int64_feature(int(label)),
            'image': _bytes_feature(image),
        }))
        writer.write(example.SerializeToString())
    writer.close()
    return

def decode(serialized_example):
  features = tf.parse_single_example(
      serialized_example,
      features={
          'image': tf.FixedLenFeature([], tf.string),
          'label': tf.FixedLenFeature([], tf.int64),
      })

  image = tf.decode_raw(features['image'], tf.float32)
  image.set_shape([224*224*3])

  image=tf.reshape(image, (224,224,3))

  label = tf.cast(features['label'], tf.int32)
  label_categorical = tf.one_hot(label,depth= num_classes, on_value=1,off_value=0,dtype=tf.int32,)
  label_categorical = tf.reshape(label_categorical, [num_classes])
  label_categorical.set_shape([num_classes])

  return image, label_categorical

def data_preparing(tfrecord_file):

    logging.info('Preparing the Training tf.dataset ')
    training_files = [tfrecord_file]
    dataset_train = tf.data.TFRecordDataset(training_files, num_parallel_reads=1)
    dataset_train = dataset_train.apply(tf.contrib.data.shuffle_and_repeat(buffer_size=4 * batch_size))
    dataset_train = dataset_train.map(decode, num_parallel_calls=1)
    dataset_train = dataset_train.batch(batch_size)
    dataset_train = dataset_train.prefetch(tf.contrib.data.AUTOTUNE)
    return dataset_train

def train_model(tfrecord_file):

    base_model = tf.keras.applications.VGG16(include_top=False, weights='imagenet',
                                                 input_shape=(224, 224, 3), pooling='avg')

    for layer in base_model.layers:
        layer.trainable = False

    logging.info('Building Our Classifier')
    x = base_model.output
    x = tf.keras.layers.Flatten()(x)
    x = tf.keras.layers.Dense(512, activation='relu')(x)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Dropout(0.5)(x)
    x = tf.keras.layers.Dense(num_classes, activation='sigmoid')(x)

    model = tf.keras.models.Model(inputs=base_model.input, outputs=x)
    model.summary()
    opt = tf.keras.optimizers.Adam(lr=0.001)
    model.compile(loss='categorical_crossentropy',
                  optimizer=opt, metrics=['accuracy'])
    dataset_train = data_preparing(tfrecord_file=tfrecord_file)

    weighted_array_train= np.array([0.01557266,0.00867447,0.04579864,0.08275284,0.18281397,
       0.30659676, 0.04686068, 0.31092999])
    class_weight_dict = dict(enumerate(weighted_array_train))

    if using_class_weight == True:
        model.fit(x=dataset_train, epochs=epochs, verbose=1,class_weight = class_weight_dict,
              steps_per_epoch=int(np.ceil(total_train / batch_size)))
    else:
        model.fit(x=dataset_train, epochs=epochs, verbose=1,
              steps_per_epoch=int(np.ceil(total_train / batch_size)))
    return

if __name__== '__main__':

    total_train = 100.
    num_classes= 8
    batch_size = 10
    epochs = 100

    #TODO (1) :Set the path to tfrecord file that we will create it.
    tfrecord_file = '~/train.tfrecords'
    tf_records_creating(tfrecord_file)                # Implement this only one time

    using_class_weight= False                         # if you set this to True, you will produce the error
    train_model(tfrecord_file)

@fchollet @mrry Would you mind taking a look?

@was84san I am not able to reproduce the scenario, as in case a tf.dataset is used, y is not specified in model.fit.
Can you provide a minimal example that trigger the error you encounter?

I know y is not specified, but then it will be extracted trough next element as shown below:
`> /home/wassan/tensorflow/venv/lib/python2.7/site-packages/tensorflow/python/keras/engine/training.py(755)_standardize_user_data()
753       try:
754         if is_x_iterator:
--> 755           next_element = self._get_iterator_get_next_tensors(x)
756         else:
757           next_element = x.get_next()
ipdb> n

/home/wassan/tensorflow/venv/lib/python2.7/site-packages/tensorflow/python/keras/engine/training.py(763)_standardize_user_data()
761                            'required number of samples.')
762
--> 763       if not isinstance(next_element, (list, tuple)) or len(next_element) != 2:
764         raise ValueError('Please provide data as a list or tuple of 2 elements '
765                          ' - input and target pair. Received %s' % next_element)

ipdb> n

/home/wassan/tensorflow/venv/lib/python2.7/site-packages/tensorflow/python/keras/engine/training.py(766)_standardize_user_data()
764         raise ValueError('Please provide data as a list or tuple of 2 elements '
765                          ' - input and target pair. Received %s' % next_element)
--> 766       x, y = next_element
767
768     # First, we build/compile the model on the fly if necessary.

`

@was84san Any chance you could provide a sample tfrecord file and a "complete" reproducible sample code (from first line of import tensorflow as tf to last line of model.fit)?
Many parameters in your code snippet are missing.

@yongtang I edited the code, you can run it now.

using_class_weight= True                        # if you set this to True, you will produce the error

@pavithrasv Could you please take a look at this issue? Thanks!

The only solution for me if this is not fixed, is to return back to use fit_generator instead and feed my data from own generator, and this one is much slower than model.fit. Like 10 times slower. I wish that can anyone just justify the condition in line 531 to make it work for both tensor and bumpy array.

Is there any update?

Nagging Assignee @pavithrasv: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

anyone can reproduce the error?

@was84san I could reproduce the error. I also could see where the issue is. However the fix might not be so straightforward so still think the best way to address it.

Added a PR #23381 for the fix. Please take a look.

It is the right fix. Thank you. When the new changes are going to be merged? Sorry, but I am not familiar about how this work , I mean when I am going to see the fix, like in the new version of tensor flow?

@was84san The PR #23381 is pending review. If the PR is approved it will be merged and picked up in the new version of TensorFlow.

Nagging Assignee @pavithrasv: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@yongtang Is there any possibility that this issue being reviewed soon :( . I really need to use it before my deadline.


Create a README.txt file in / and /root with Alpine image usage

[ ] How to sync kernel moduels
[ ] How to install package
[ ] Get the latest version of the README.txt (github link)
[ ] Mount NBD volumes
[ ] Links to Alpine Linux

Fixed with the dynamic motd, if needed, we will open new issues with specific needs


UX Improvements

MongoHacker-like UX improvements are very much appreciated. As integrating both extensions is a little bit tricky and we all prefer all-in-alls!
P.S. Thank you for this great contribution.

@sepehr yep, unfortunately we are pretty much mutually exclusive :disappointed:
What do you like to have in particular? What do you use?


GH2134: References latest Newtonsoft.Json from Cake.NuGet

Confirmed this results in Newtonsoft.Json 11.0.2 ending up in Cake bin for net46 target framework. This should resolve 90% of cases, but does not introduce a long-term fix of building a reference dependency graph and loading the newest (that'll take some time).
#2134

@daveaglick your changes have been merged, thanks for your contribution 👍


Ymap import not working

Steps to reproduce:

Open the Sollumz toolset
Attempt to import .ymap.xml file

--> Error will pop up showing the following:

This pops up no matter which ymap is being used, nor what other entities or geometry are in the scene.

Here are the files I used when testing.
Demo_FIles.zip


Easiest way to integrate zig-gamedev as a dependency?

Hey!
I've got zig-gamedev as a submodule to my src/deps/zig-gamedev
Assembled next build.zig
const std = @import("std");

const zig_gamedev = @import("src/deps/zig-gamedev/build.zig");

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "boids",
        .root_source_file = .{ .path = "src/boids/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    zig_gamedev.zwin32_pkg.link(exe, .{ .d3d12 = true });
    zig_gamedev.common_d2d_pkg.link(exe);
    zig_gamedev.zd3d12_d2d_pkg.link(exe);

    const install_cmd = b.addInstallArtifact(exe);

    const run_cmd = b.addRunArtifact(exe);

    run_cmd.step.dependOn(&install_cmd.step);

    const run_step = b.step("boids", "run example boids");

    run_step.dependOn(&run_cmd.step);
}

But when I run zig build I have
$ zig build
Segmentation fault at address 0x8
C:\Tools\Zig\lib\std\Build\Step\Compile.zig:1111:11: 0x7ff6595d5e7b in addRecursiveBuildDeps (build.exe.obj)
    module.source_file.addStepDependencies(&cs.step);
          ^
C:\Tools\Zig\lib\std\Build\Step\Compile.zig:1093:29: 0x7ff6595b104c in addModule (build.exe.obj)
    cs.addRecursiveBuildDeps(module, &done) catch @panic("OOM");
                            ^
C:\Work\Code\suprematism\src\deps\zig-gamedev\libs\zwin32\build.zig:17:22: 0x7ff65958ba8d in link (build.exe.obj)
        exe.addModule("zwin32", pkg.zwin32);
                     ^
C:\Work\Code\suprematism\build.zig:34:32: 0x7ff65954a57e in build (build.exe.obj)
    zig_gamedev.zwin32_pkg.link(exe, .{ .d3d12 = true });
                               ^
C:\Tools\Zig\lib\std\Build.zig:1602:33: 0x7ff659539c55 in runBuild__anon_7188 (build.exe.obj)
        .Void => build_zig.build(b),
                                ^
C:\Tools\Zig\lib\build_runner.zig:292:29: 0x7ff659535859 in main (build.exe.obj)
        try builder.runBuild(root);
                            ^
C:\Tools\Zig\lib\std\start.zig:379:65: 0x7ff65953a6d7 in WinStartup (build.exe.obj)
    std.os.windows.kernel32.ExitProcess(initEventLoopAndCallMain());
                                                                ^
???:?:?: 0x7ffe909e26ac in ??? (KERNEL32.DLL)
???:?:?: 0x7ffe9172aa07 in ??? (ntdll.dll)
error: the following build command failed with exit code 3:
C:\Work\Code\suprematism\zig-cache\o\742d7e9e69a94df5cae0156a2f0c7324\build.exe C:\Tools\Zig\zig.exe C:\Work\Code\suprematism C:\Work\Code\suprematism\zig-cache C:\Users\ichbi\AppData\Local\zig

Honestly I want to assemble clearest and smallest version of build.zig to make it work (I've took vector_graphics_test as an example)

Hey!
Take a look at a readme file here: https://github.com/michal-z/zig-gamedev/tree/main/libs/zwin32
You basically need to call package() and then link() for each lib.
Our internal samples do not call package() because it is already done in the root build.zig.

Thanks for a very quick response! 😎
This indeed helped to build a project, so code right now looks like this:
pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});

    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "boids",
        .root_source_file = .{ .path = "src/boids/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    b.installArtifact(exe);

    const run_cmd = b.addRunArtifact(exe);

    run_cmd.step.dependOn(b.getInstallStep());

    if (b.args) |args| {
        run_cmd.addArgs(args);
    }

    const zwin32_pkg = zwin32.package(b, target, optimize, .{});
    const zd3d12_pkg = zd3d12.package(b, target, optimize, .{
        .options = .{
            .enable_debug_layer = false,
            .enable_gbv = false,
        },
        .deps = .{ .zwin32 = zwin32_pkg.zwin32 },
    });
    const common_d2d_pkg = common.package(b, target, optimize, .{
        .deps = .{ .zwin32 = zwin32_pkg.zwin32, .zd3d12 = zd3d12_pkg.zd3d12 },
    });

    zwin32_pkg.link(exe, .{ .d3d12 = true });
    zd3d12_pkg.link(exe);
    common_d2d_pkg.link(exe);

    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);
}

It compiles and runs
But in runtime I have next issue and it crashing:
$ zig build run
thread 53348 panic: attempt to use null value
C:\Work\Code\suprematism\src\boids\main.zig:57:31: 0x7ff78caf279b in init (boids.exe.obj)
        hrPanicOnFail(gctx.d2d.?.context.CreateSolidColorBrush(
                              ^
C:\Work\Code\suprematism\src\boids\main.zig:709:24: 0x7ff78caf88a3 in main (boids.exe.obj)
    var demo = try init(allocator);
                       ^
C:\Tools\Zig\lib\std\start.zig:508:80: 0x7ff78caf96f3 in main (boids.exe.obj)
    return @call(.always_inline, callMainWithArgs, .{ @intCast(usize, c_argc), @ptrCast([*][*:0]u8, c_argv), envp });
                                                                               ^
???:?:?: 0x7ff78cb8f1e5 in ??? (boids.exe)
???:?:?: 0x7ff78cb8f24b in ??? (boids.exe)
???:?:?: 0x7ffe909e26ac in ??? (KERNEL32.DLL)
???:?:?: 0x7ffe9172aa07 in ??? (ntdll.dll)
run boids: error: the following command exited with error code 3:
C:\Work\Code\suprematism\zig-out\bin\boids.exe
Build Summary: 8/10 steps succeeded; 1 failed (disable with -fno-summary)
run transitive failure
+- run boids failure
   +- zig build-exe boids Debug native cached 15ms MaxRSS:28M
   |  +- zwin32-install-d3d12 cached
   |  |  +- install C:\Work\Code\suprematism\src\deps\zig-gamedev\libs\zwin32\bin\x64\D3D12Core.dll to bin/d3d12/D3D12Core.dll cached
   |  |  +- install C:\Work\Code\suprematism\src\deps\zig-gamedev\libs\zwin32\bin\x64\D3D12SDKLayers.dll to bin/d3d12/D3D12SDKLayers.dll cached
   |  +- options cached
   |  +- options (reused)
   |  +- options (reused)
   |  +- zig build-lib common Debug native cached 18ms MaxRSS:28M
   +- install cached
      +- install boids cached
         +- zig build-exe boids Debug native (+5 more reused dependencies)
error: the following build command failed with exit code 1:
C:\Work\Code\suprematism\zig-cache\o\d405c59f3f00f84a52b232c835f35fe0\build.exe C:\Tools\Zig\zig.exe C:\Work\Code\suprematism C:\Work\Code\suprematism\zig-cache C:\Users\ichbi\AppData\Local\zig run

If you use Direct2D you need to explicitly enable it:
    zd3d12_d2d_pkg = zd3d12.package(b, target, optimize, .{
        .options = .{
            .enable_debug_layer = options.zd3d12_enable_debug_layer,
            .enable_gbv = options.zd3d12_enable_gbv,
            .enable_d2d = true,
        },
        .deps = .{ .zwin32 = zwin32_pkg.zwin32 },
    });
    common_d2d_pkg = common.package(b, target, optimize, .{
        .deps = .{ .zwin32 = zwin32_pkg.zwin32, .zd3d12 = zd3d12_d2d_pkg.zd3d12 },
    });

Yes it worked, thanks!
So honestly this all look kinda obvious when you understand the process, but maybe there possibility to simplify it for a newbie?
Because when application crash on launch with segmentation fault or other same errors appears without explanation, you don't really know how to proceed.
If zig-gamedev open to proposals, I definitely like how raylib bindings are added to the program via  raylib.addTo(b, exe, target); helper function.
https://github.com/ryupold/raylib.zig
const raylib = @import("path/to/raylib.zig/build.zig");

pub fn build(b: *std.Build) !void {
    const target = b.standardTargetOptions(.{});
    const exe = ...;
    raylib.addTo(b, exe, target);
}

Maybe we can add couple of them to build.zig with a flags? Like zig_gamedev.addTo(..., flags: d3d12_2d, d3d12_3d, physics and etc)

What we currently have is a minimal low-level API which allows to build the lib once (package()) and link it with many executables (link()). I would like to keep it that way for now but we may add higher-level API in the future when Zig package manager will be fully-featured. Anyway, thanks for suggestion!

Sure! Thanks for a support!
Then maybe we can add some explanation right before Libraries kinda Get Started or Quick Start section
-Get submodule
Write such example code ... (with explanation you've wrote me above)
Run zig build
I can do a pull request if you are fine with that 🥸

PR with the instruction would be awesome, thanks!


Feature Request: Remember tree control states

A nice UI enhancement would be if the tree views in the constant buffer / UBO displays and the API Calls display remembered which nodes were expanded and which nodes remained collapsed when navigating between events. Currently, when switching between two (or more) events, all nodes in all open cbuffer displays and the API calls display are collapsed when a new event is selected, making it more difficult to compare the data used by those events as you will need to re-open all nodes you had opened before. Thanks!

That makes sense. To what extent should it maintain the setup? Would you expect it to only work for identical constant buffers, or should it try to handle constant buffers that have some similarities but aren't identical?

I'd like it best if the tree state was somehow bound to the tree control- i.e. when returning to a tree control you viewed before, regardless of the control's contents (e.g. cbuffer, API calls, event browser, etc), it retains the same expanded/collapsed state it had before. There's no need for comparing one control to another, each tree control has a state independent of any other controls. For my purposes, each tree's state does not need to be persistent if you load a previously saved log, but that might be a nice additional feature.

I don't plan to persist these between opens of a capture, but the controls that make sense to save state during a capture load are now saved: API Inspector, Resource Inspector, CBuffer view
I'll try to keep an eye on this to see if the memory/processing impact is significant but from initial tests it seems reasonable.


Http2FrameCodec should send WINDOW_UPDATE early

Expected behavior
Http2FrameCodec should send WINDOW_UPDATE before window size is exhausted (early).
Actual behavior
Http2FrameCodec sends WINDOW_UPDATE after the stream has timed out by HTTP/2 server and Reset Frame is sent by HTTP/2 Server.
Steps to reproduce
Set Initial Window size of 10000 and fetch content from HTTP/2 server of more than 10000 bytes. Reading of Data frames will stop after 10000 bytes are read. Once the server sends Reset Frame because no communication is received from Netty HTTP/2 client, Netty HTTP/2 client will then send HTTP/2 WINDOW_UPDATE frame.
Netty version
4.1.53.Final
JVM version (e.g. java -version)
openjdk version "11.0.9" 2020-10-20 LTS
OS version (e.g. uname -a)
Windows 10 Pro x64

Pcap File Link: https://drive.google.com/file/d/1F6ErZ-aNfHgjMeStpg8wBhZk3fK8wzs-/view?usp=sharing

I tried fetching www.google.com using Http2FrameCodec based Netty HTTP/2 Client.

Yes, we can send a manual WINDOW_UPDATE frame but we should have an automated way of doing it too. Maybe Http2FrameCodec can have a constructor option to enable auto increment of window size?

Alright, the problem is in DefaultHttp2LocalFlowController. I need to properly adjust Window update Ratio. Will open PR soon.


Add option for 2-digit numeric separators

This allows to generate passwords with 2-digit separators.
Example:
this44is44some44password
It is still easy to remember but adds digits which many password forms require.

Another idea I'll probably open a PR soon: Have the first letter of each word uppercase. Many websites require sth like one upper and at least one lowercase letter. I think that would be an elegant solution. Example:
This44Is44Some44Password

Hi Marc,
Thanks for the PR! I think the features you're adding already exist however:
 % xkcdpass --case first --delimiter 44

 Badness44Prelaunch44Empathic44Junior44Overcook44Speak

If I've missed something from your addition please let me know

Hi,
yes, I saw this feature before I started mine. The difference is that my changes create a random number instead of taking it as parameter. I need this for automation in my scripts.


Request interception not working in Chrome

Steps to reproduce
Using the example code for intercepting requests, the request and response events are not fired if request interception is enabled.
Tell us about your environment:

Puppeteer version: 1.1.1 (also tested with @next)
Platform / OS version: Windows 10 64 Bit
URLs (if applicable):
Node.js version: 8.10.0 / 9.8.0
Chrome version: 64.0.3282.186 64 Bit

What steps will reproduce the problem?
const PathToChrome = 'C:/Program Files (x86)/Google/Chrome/Application/chrome.exe';
const Width = 1920;
const Height = 1080;
const LauchOptions = {
  executablePath: PathToChrome,
  headless: false,
  args: [`--window-size=${Width},${Height}`]
};
  const browser = await puppeteer.launch('localhost:3000');
  const page = await browser.newPage();
  page.setViewport({
    width: Width - 30,
    height: Height - 135
  });

  await page.setRequestInterception(true);
  page.on('request', request => {
    console.log('test');
  });
  page.on('response', response => {
    console.log('test');
  });

What is the expected result?
test should be printed to the console.
What happens instead?
Nothing. The events are not fired.

Without executablePath the code seems to work. (puppeteer.launch('localhost:3000') should be puppeteer.launch(LauchOptions ) though).

Sorry, my bad, copy&paste error. I've actually had puppeteer.launch(LauchOptions).
You are right. Without executablePath it does work.
The issue is that it doesn't work in Chrome, but it does in Chromium, which is the default as I understand it. Is it actually supposed to work in Chrome? I couldn't find any information on that.

so, is there any solution, I need to use chrome to support video format, but chrome is not support puppeteer Request interception api

It's work, if i set headless:false; so great

Have the same problem, and I cannot use Chromium, because (for reasons unknown) Chromium does not render any fonts (including devtools). So, I cannot use Chromium for debugging.

@Domysee In every puppeteer release, we publish a version of chrome puppeteer is compatible with. For chrome 64, you can use<EMAIL_ADDRESS>