Codegen IntPtr-accepting overloads of methods with native pointers

#76 should be done first. See that issue for the rationale.
For now we'll use only a few pointers when deemed necessary and hand-craft IntPtr alternatives, but what would really be interesting is to only have to write the pointer versions and have the IntPtr overload auto-generated.
Code generation will also be needed for #55 so both issues would need the same infrastructure.

Code generation will also be needed for #55 so both issues would need the same infrastructure.

Just a note: #55 might be rather different from this one at least in how it's wired up. #55 may produce a separate assembly and that assembly might be in its own NuGet package as well, whereas the code gen for IntPtr would definitely produce code for the same assembly.

I'm going to do some experimentation with this using the basis for build-time code generation I developed here: https://github.com/AArnott/ImmutableObjectGraph/tree/master/src/ImmutableObjectGraph.CodeGeneration

OK, I've finished pulling out the basis of build-time code generation into this new project:
https://github.com/AArnott/CodeGeneration.Roslyn
I'll work on applying it to this project next.


Shared Object with defaultValue = null and isRequired = false are not .nullable()

I have a shared Address type. In all my collections, the address field uses this shared model which is not required and has a null default value. However, when saving a document, I do get an error about the null value being incorrect and about setting the schema to .nullable() if it should be correct. Though when I look at the generated input, it looks like so :
export class ProfileInsertInput {
  @Is(() => Schema.from(AddressInput))
  address?: AddressInput = null;

 // ...
}

Instead of :
export class ProfileInsertInput {
  @Is(() => Schema.from(AddressInput).nullable())
  address?: AddressInput = null;

 // ...
}

@qfl1ck32 can you look into this please?


Esc key to escape lazydocker

I think using just q to quit lazydocker is enough, as when we focus the main panel, and want to go back, we press Esc, sometimes we just spam the Esc key and lazydocker gets closed due to that.
Hopefully next change can just be using q to quit lazydocker.

Good point. There is currently a config value which allows you to get a prompt ensuring you indeed want to quit if you hit esc when at the top level:
confirmOnQuit: true

Would that satisfy your requirement?

I think this will be solvable once we implement configurable keybindings. An empty keybinding will represent that you want a key disabled, and doing something like:
keybindings:
  quit: ["q"] # as opposed to quit: ["q", "KeyEsc"]

will satisfy this requirement. So I'm closing in favour of https://github.com/jesseduffield/lazydocker/issues/48


Allow use qcow2 inline compression

Provide the .img qcow2 without wrapped compression by enabling the inline compression
#1135 by @tuxtof
Spoke with @pothos

Started a github PR build, we should also test on Jenkins

Jenkins CI http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/2493/cldsv/

There is one failure:
 --- FAIL: systemd.journal.user (16.95s)
         cluster.go:130: "journalctl --user --unit hello.service | grep Foo" failed: output , status Process exited with status 1

There is one failure:
 --- FAIL: systemd.journal.user (16.95s)
         cluster.go:130: "journalctl --user --unit hello.service | grep Foo" failed: output , status Process exited with status 1

That looks unrelated

The jenkins build got cleaned up, can't see the results but I think it's ok to merge if the reported failure was only about an unrelated kola test.

Let's merge.
I am fine with backports.

Let's see how the nightly build turns out before backporting.

Backported


Enable all thermostat client attributes in controller clusters

This will enable more attribute support within java and python.
Technically client-side clusters should have ignored enabling, however #25737 showed that there is more work to do here since codegen for java via zap also needs updating (work done by #25693).
Until then, I opened zap via ./scripts/tools/zap//run_zaptool.sh src/controller/data_model/controller-clusters.zap, switched thermostat to 'server', enabled all attributes and switched it back to client (odd workaround to mark client-side enabled in zap)

Fast tracking - restyler seems to fail here (unsure why ... large files?) so will have to merge bypassing it.


QC read files deleted too early

Run / system info
I'm running rotary commit 40381cc on Ubuntu 22.04.3, within the snakemake argument --until circularize (to run everything up until the start of the annotation module).
Problem description
After reaching checkpoint split_circular_and_linear_contigs, I encountered the following error:
BUG: Out of jobs ready to be started, but not all files built yet. Please check https://github.com/snakemake/snakemake/issues/823 for more information.
Remaining jobs:
 - search_contig_start: S18/circularize/identify/S18_circular.faa, S18/circularize/identify/S18_circular.ffn, S18/circularize/identify/S18_circular.gff, S18/circularize/identify/S18_hmmsearch_hits.txt, S18/circularize/identify/S18_hmmsearch_hits_no_comments.txt
 - finalize_circular_contig_rotation: S18/circularize/combine/S18_circular.fasta
 - combine_circular_and_linear_contigs: S18/circularize/combine/S18_combined.fasta
 - process_start_genes: S18/circularize/identify/S18_start_genes.list
 - polish_polypolish: S18/circularize/polypolish/S18_R1.clean.sam, S18/circularize/polypolish/S18_R2.clean.sam, S18/circularize/polypolish/S18_polypolish.fasta, S18/circularize/polypolish/polypolish.debug.log, S18/stats/circularize/polypolish_changes.log
 - map_short_reads_for_polishing: S18/circularize/polypolish/S18_R1.sam, S18/circularize/polypolish/S18_R2.sam, S18/circularize/polypolish/input/S18_input.fasta.0123, S18/circularize/polypolish/input/S18_input.fasta.amb, S18/circularize/polypolish/input/S18_input.fasta.ann, S18/circularize/polypolish/input/S18_input.fasta.bwt.2bit.64, S18/circularize/polypolish/input/S18_input.fasta.pac
 - circularize: checkpoints/circularize
 - get_start_genes: S18/circularize/identify/S18_start_gene.ffn
 - prepare_polypolish_circularize_input: S18/circularize/polypolish/input/S18_input.fasta
 - run_circlator: S18/circularize/circlator/S18_rotated.fasta, S18/circularize/circlator, S18/circularize/circlator/rotated.promer.contigs_with_ends.fa
 - symlink_circularization: S18/circularize/S18_circularize.fasta
 - get_polished_contigs: S18/circularize/filter/S18_circular.fasta
Complete log: .snakemake/log/2024-02-15T154852.164572.snakemake.log
Executing: snakemake --snakefile /2022_08_01_rotary_testing/rotary/rotary/rules/rotary.smk --configfile /2022_08_01_rotary_testing/output_dir/config.yaml --directory /2022_08_01_rotary_testing/output_dir --conda-prefix /2022_08_01_rotary_testing/rotary_db_dir/rotary_conda_env --conda-frontend mamba --use-conda --reason --rerun-incomplete --printshellcmds --jobs 48 --until circularize
Command 'snakemake --snakefile /2022_08_01_rotary_testing/rotary/rotary/rules/rotary.smk --configfile /2022_08_01_rotary_testing/output_dir/config.yaml --directory /2022_08_01_rotary_testing/output_dir --conda-prefix /2022_08_01_rotary_testing/rotary_db_dir/rotary_conda_env --conda-frontend mamba --use-conda --reason --rerun-incomplete --printshellcmds --jobs 48 --until circularize' returned non-zero exit status 1.

When I tried to re-start rotary, it wanted to re-built the QC read files from scratch -- these files had been deleted as temp() -- and essentially start the pipeline over. The deletion of the final QC read files is odd, because these QC read files were still needed for the downstream parts of the pipeline (e.g., for another round of polishing during the circularization module). I added fake final QC read files (and a couple intermediate files) back via touch (with appropriate timestamps), and the pipeline continued past the checkpoint error, but (of course) rotary ran into an error when it tried to use the empty QC read files.
Based on a quick skim of the GitHub issue flagged in the Snakemake error message (https://github.com/snakemake/snakemake/issues/823), I suspect that the issue I am running into has to do with a bug in checkpointing. After hitting the split_circular_and_linear_contigs checkpoint, if my understanding is correct, Snakemake re-evaluates the rest of the DAG for the run. Until that point, snakemake doesn't know if the QC'ed reads will be needed again... it depends on if there are any circular contigs that get passed into the circularization module. It seems that Snakemake pre-emptively deleted the temp() QC read files before knowing if it was really safe to do so. As a result, I think the pipeline crashed once it realized that the QC read files were actually necessary files but were missing for continuing the run.
Possible solution
Set the final QC read files as non-temp until we can find a better solution. An upgrade to Snakemake 8 might help fix this problem, or we might need to eventually contact the Snakemake authors about it.
The capability to set the QC read files as non-temp is already included in PR #130 . I would suggest that we change the default setting in that PR so that the final QC read files are non-temp by default.
@LeeBergstrand Any thoughts?

When I tried to re-start rotary, it wanted to re-built the QC read files from scratch -- these files had been deleted as temp() -- and essentially start the pipeline over. The deletion of the final QC read files is odd, because these QC read files were still needed for the downstream parts of the pipeline (e.g., for another round of polishing during the circularization module). I added fake final QC read files (and a couple intermediate files) back via touch (with appropriate timestamps), and the pipeline continued past the checkpoint error, but (of course) rotary ran into an error when it tried to use the empty QC read files.

This is expected behaviour. The rule 'circularize' is downstream of all the circularization methods. Because it no longer needs temp reads, it deletes them because all rules that use these reads have been running.

rotary run -s'--dag --until circularize' | grep -v 'Executing' | dot -Tpdf > until_circ_dag.pdf

My understanding is that Snakemake rebuilds the DAG every time it is run. So when you go --until circularize, it builds the DAG until the rule species by the --until without regard to the next steps. As you can see from the DAG graph above, it isn't aware of any of the steps, so it deletes the temp files it doesn't need.

When I tried to re-start rotary, it wanted to re-built the QC read files from scratch -- these files had been deleted as temp() -- and essentially start the pipeline over. The deletion of the final QC read files is odd, because these QC read files were still needed for the downstream parts of the pipeline (e.g., for another round of polishing during the circularization module). I added fake final QC read files (and a couple intermediate files) back via touch (with appropriate timestamps), and the pipeline continued past the checkpoint error, but (of course) rotary ran into an error when it tried to use the empty QC read files.

This makes sense to me --> it can't find the QCed reads --> says I need to make the QCed reads --> the new QCed reads affect the following steps (in case the read polishing is non-deterministic) --> re-does everything post assembly.

@LeeBergstrand Sorry, I think I didn't explain the problem clearly. You are right that it would be normal for snakemake to delete the short QC read files after reaching the end of the --until step specified. However, in my case, the pipeline is unable to finish the specified circularize rule. It gets stuck at the checkpoint a few steps before that and errors out early. I think the error is because temp() is deleting the QC read files too early.
So:
I start the pipeline and tell it --until circularize
The pipeline runs until the end of polish module normally (before the circularize module) -->
The pipeline doesn't know if short QC reads will be needed or not at the end of the circularize module due to the checkpoint in the middle of the module -->
The pipeline pre-emptively deletes the short QC reads (this is the problem, I think) -->
The pipeline evaluates the checkpoint partway through the circularize module and re-builds the DAG -->
The pipeline sees that short QC reads are actually needed to finish the circularize module -->
The pipeline produces the error shown above and refuses to finish the circularize module, because there are no short QC reads.
Does that make more sense? I think it is fundamentally a snakemake bug, but I think we could work around it temporarily by making the QC reads non-temp by default. However, I suspect that the bug will not appear if you have the pipeline run fully to the end. Let me know your thoughts here -- thanks!

Specifically, in the DAG figure you inserted, here is the circularization module:

split_circular_and_linear_contigs is a checkpoint. At that point, the DAG gets re-evaluated.
If you have any circular contigs, then before combine_circular_and_linear_contigs, the DAG will add a bunch of additional rules upon re-evaluation, including rotating to the start gene and performing one more round of Polypolish-based polishing. The polishing step needs the short QC reads. However, this possible use of the QC short reads seems to be hidden from snakemake until the checkpoint is evaluated.

@jmtsuji Thanks for the explanation. I'm testing on my machine to confirm the behaviour encountered. To clarify, does this error only occur if you run --until circularize and then restart the pipeline? Isn't --until circularize a corner case? I noticed you used it as an example in the README.

If you have any circular contigs, then before combine_circular_and_linear_contigs, the DAG will add a bunch of additional rules upon re-evaluation, including rotating to the start gene and performing one more round of Polypolish-based polishing. The polishing step needs the short QC reads. However, this possible use of the QC short reads seems to be hidden from snakemake until the checkpoint is evaluated.

Is there a separate rule for this polishing step? or is it calling polishing rules again somehow (is that why there is a {step} in some of the polishing rule strings)?

The capability to set the QC read files as non-temp is already included in PR https://github.com/rotary-genomics/rotary/pull/130 . I would suggest that we change the default setting in that PR so that the final QC read files are non-temp by default.

Let's do this until we move to Snakemake 8.

@jmtsuji Thanks for the explanation. I'm testing on my machine to confirm the behaviour encountered. To clarify, does this error only occur if you run --until circularize and then restart the pipeline? Isn't --until circularize a corner case? I noticed you used it as an example in the README.

@LeeBergstrand Thanks for testing -- I am also running a few more test on my end with different run params. For now, I assume the error only occurs if I run --until with something after the checkpoint. No re-start of the pipeline is needed -- the error occurs partway through the run.
Using --until circularize gets you the genome without any annotations. This is a nice way to avoid the most resource and time-intensive parts of the pipeline, e.g., for testing.

If you have any circular contigs, then before combine_circular_and_linear_contigs, the DAG will add a bunch of additional rules upon re-evaluation, including rotating to the start gene and performing one more round of Polypolish-based polishing. The polishing step needs the short QC reads. However, this possible use of the QC short reads seems to be hidden from snakemake until the checkpoint is evaluated.

Is there a separate rule for this polishing step? or is it calling polishing rules again somehow (is that why there is a {step} in some of the polishing rule strings)?

It's calling the Polypolish rule again -- this is why there is a {step}' wildcard (which can be either polishorcircularize`).

The capability to set the QC read files as non-temp is already included in PR #130 . I would suggest that we change the default setting in that PR so that the final QC read files are non-temp by default.

Let's do this until we move to Snakemake 8.

OK, sounds good. I am running a couple tests on my end to confirm that this fix works.

The capability to set the QC read files as non-temp is already included in PR #130 . I would suggest that we change the default setting in that PR so that the final QC read files are non-temp by default.

Let's do this until we move to Snakemake 8.

OK, sounds good. I am running a couple tests on my end to confirm that this fix works.

Sounds good. Thanks for giving me an overview of how checkpointing works and how it applies to our software. I send you the test results when I get them.

@jmtsuji I had to run --until circularize twice. It was because I had to stop it halfway through because I forgot to update the rotary using git. Interestingly, I ran into a very similar error running rotary with --until circularize again:
BUG: Out of jobs ready to be started, but not all files built yet. Please check https://github.com/snakemake/snakemake/issues/823 for more information.
Remaining jobs:
 - process_start_genes: cram/circularize/identify/cram_start_genes.list
 - prepare_polypolish_circularize_input: cram/circularize/polypolish/input/cram_input.fasta
 - get_polished_contigs: blam/circularize/filter/blam_circular.fasta
 - map_short_reads_for_polishing: cram/circularize/polypolish/cram_R1.sam, cram/circularize/polypolish/cram_R2.sam, cram/circularize/polypolish/input/cram_input.fasta.0123, cram/circularize/polypolish/input/cram_input.fasta.amb, cram/circularize/polypolish/input/cram_input.fasta.ann, cram/circularize/polypolish/input/cram_input.fasta.bwt.2bit.64, cram/circularize/polypolish/input/cram_input.fasta.pac
 - circularize: checkpoints/circularize
 - get_start_genes: blam/circularize/identify/blam_start_gene.ffn
 - run_circlator: cram/circularize/circlator/cram_rotated.fasta, cram/circularize/circlator, cram/circularize/circlator/rotated.promer.contigs_with_ends.fa
 - polish_polypolish: cram/circularize/polypolish/cram_R1.clean.sam, cram/circularize/polypolish/cram_R2.clean.sam, cram/circularize/polypolish/cram_polypolish.fasta, cram/circularize/polypolish/polypolish.debug.log, cram/stats/circularize/polypolish_changes.log
 - search_contig_start: blam/circularize/identify/blam_circular.faa, blam/circularize/identify/blam_circular.ffn, blam/circularize/identify/blam_circular.gff, blam/circularize/identify/blam_hmmsearch_hits.txt, blam/circularize/identify/blam_hmmsearch_hits_no_comments.txt
 - get_polished_contigs: cram/circularize/filter/cram_circular.fasta
 - combine_circular_and_linear_contigs: blam/circularize/combine/blam_combined.fasta
 - symlink_circularization: blam/circularize/blam_circularize.fasta
 - process_start_genes: blam/circularize/identify/blam_start_genes.list
 - get_start_genes: cram/circularize/identify/cram_start_gene.ffn
 - finalize_circular_contig_rotation: blam/circularize/combine/blam_circular.fasta
 - run_circlator: blam/circularize/circlator/blam_rotated.fasta, blam/circularize/circlator, blam/circularize/circlator/rotated.promer.contigs_with_ends.fa
 - symlink_circularization: cram/circularize/cram_circularize.fasta
 - map_short_reads_for_polishing: blam/circularize/polypolish/blam_R1.sam, blam/circularize/polypolish/blam_R2.sam, blam/circularize/polypolish/input/blam_input.fasta.0123, blam/circularize/polypolish/input/blam_input.fasta.amb, blam/circularize/polypolish/input/blam_input.fasta.ann, blam/circularize/polypolish/input/blam_input.fasta.bwt.2bit.64, blam/circularize/polypolish/input/blam_input.fasta.pac
 - search_contig_start: cram/circularize/identify/cram_circular.faa, cram/circularize/identify/cram_circular.ffn, cram/circularize/identify/cram_circular.gff, cram/circularize/identify/cram_hmmsearch_hits.txt, cram/circularize/identify/cram_hmmsearch_hits_no_comments.txt
 - polish_polypolish: blam/circularize/polypolish/blam_R1.clean.sam, blam/circularize/polypolish/blam_R2.clean.sam, blam/circularize/polypolish/blam_polypolish.fasta, blam/circularize/polypolish/polypolish.debug.log, blam/stats/circularize/polypolish_changes.log
 - combine_circular_and_linear_contigs: cram/circularize/combine/cram_combined.fasta
 - prepare_polypolish_circularize_input: blam/circularize/polypolish/input/blam_input.fasta
 - finalize_circular_contig_rotation: cram/circularize/combine/cram_circular.fasta
Complete log: .snakemake/log/2024-02-16T005846.034888.snakemake.log

@jmtsuji I had to run --until circularize twice. It was because I had to stop it halfway through because I forgot to update the rotary using git. Interestingly, I ran into a very similar error running rotary with --until circularize again:
BUG: Out of jobs ready to be started, but not all files built yet. Please check https://github.com/snakemake/snakemake/issues/823 for more information.
Remaining jobs:
 - process_start_genes: cram/circularize/identify/cram_start_genes.list
 - prepare_polypolish_circularize_input: cram/circularize/polypolish/input/cram_input.fasta
 - get_polished_contigs: blam/circularize/filter/blam_circular.fasta
 - map_short_reads_for_polishing: cram/circularize/polypolish/cram_R1.sam, cram/circularize/polypolish/cram_R2.sam, cram/circularize/polypolish/input/cram_input.fasta.0123, cram/circularize/polypolish/input/cram_input.fasta.amb, cram/circularize/polypolish/input/cram_input.fasta.ann, cram/circularize/polypolish/input/cram_input.fasta.bwt.2bit.64, cram/circularize/polypolish/input/cram_input.fasta.pac
 - circularize: checkpoints/circularize
 - get_start_genes: blam/circularize/identify/blam_start_gene.ffn
 - run_circlator: cram/circularize/circlator/cram_rotated.fasta, cram/circularize/circlator, cram/circularize/circlator/rotated.promer.contigs_with_ends.fa
 - polish_polypolish: cram/circularize/polypolish/cram_R1.clean.sam, cram/circularize/polypolish/cram_R2.clean.sam, cram/circularize/polypolish/cram_polypolish.fasta, cram/circularize/polypolish/polypolish.debug.log, cram/stats/circularize/polypolish_changes.log
 - search_contig_start: blam/circularize/identify/blam_circular.faa, blam/circularize/identify/blam_circular.ffn, blam/circularize/identify/blam_circular.gff, blam/circularize/identify/blam_hmmsearch_hits.txt, blam/circularize/identify/blam_hmmsearch_hits_no_comments.txt
 - get_polished_contigs: cram/circularize/filter/cram_circular.fasta
 - combine_circular_and_linear_contigs: blam/circularize/combine/blam_combined.fasta
 - symlink_circularization: blam/circularize/blam_circularize.fasta
 - process_start_genes: blam/circularize/identify/blam_start_genes.list
 - get_start_genes: cram/circularize/identify/cram_start_gene.ffn
 - finalize_circular_contig_rotation: blam/circularize/combine/blam_circular.fasta
 - run_circlator: blam/circularize/circlator/blam_rotated.fasta, blam/circularize/circlator, blam/circularize/circlator/rotated.promer.contigs_with_ends.fa
 - symlink_circularization: cram/circularize/cram_circularize.fasta
 - map_short_reads_for_polishing: blam/circularize/polypolish/blam_R1.sam, blam/circularize/polypolish/blam_R2.sam, blam/circularize/polypolish/input/blam_input.fasta.0123, blam/circularize/polypolish/input/blam_input.fasta.amb, blam/circularize/polypolish/input/blam_input.fasta.ann, blam/circularize/polypolish/input/blam_input.fasta.bwt.2bit.64, blam/circularize/polypolish/input/blam_input.fasta.pac
 - search_contig_start: cram/circularize/identify/cram_circular.faa, cram/circularize/identify/cram_circular.ffn, cram/circularize/identify/cram_circular.gff, cram/circularize/identify/cram_hmmsearch_hits.txt, cram/circularize/identify/cram_hmmsearch_hits_no_comments.txt
 - polish_polypolish: blam/circularize/polypolish/blam_R1.clean.sam, blam/circularize/polypolish/blam_R2.clean.sam, blam/circularize/polypolish/blam_polypolish.fasta, blam/circularize/polypolish/polypolish.debug.log, blam/stats/circularize/polypolish_changes.log
 - combine_circular_and_linear_contigs: cram/circularize/combine/cram_combined.fasta
 - prepare_polypolish_circularize_input: blam/circularize/polypolish/input/blam_input.fasta
 - finalize_circular_contig_rotation: cram/circularize/combine/cram_circular.fasta
Complete log: .snakemake/log/2024-02-16T005846.034888.snakemake.log

@jmtsuji I get the above error every time I run with --until circularize.

@jmtsuji I had to run --until circularize twice. It was because I had to stop it halfway through because I forgot to update the rotary using git. Interestingly, I ran into a very similar error running rotary with --until circularize again:
BUG: Out of jobs ready to be started, but not all files built yet. Please check https://github.com/snakemake/snakemake/issues/823 for more information.
Remaining jobs:
 - process_start_genes: cram/circularize/identify/cram_start_genes.list
 - prepare_polypolish_circularize_input: cram/circularize/polypolish/input/cram_input.fasta
 - get_polished_contigs: blam/circularize/filter/blam_circular.fasta
 - map_short_reads_for_polishing: cram/circularize/polypolish/cram_R1.sam, cram/circularize/polypolish/cram_R2.sam, cram/circularize/polypolish/input/cram_input.fasta.0123, cram/circularize/polypolish/input/cram_input.fasta.amb, cram/circularize/polypolish/input/cram_input.fasta.ann, cram/circularize/polypolish/input/cram_input.fasta.bwt.2bit.64, cram/circularize/polypolish/input/cram_input.fasta.pac
 - circularize: checkpoints/circularize
 - get_start_genes: blam/circularize/identify/blam_start_gene.ffn
 - run_circlator: cram/circularize/circlator/cram_rotated.fasta, cram/circularize/circlator, cram/circularize/circlator/rotated.promer.contigs_with_ends.fa
 - polish_polypolish: cram/circularize/polypolish/cram_R1.clean.sam, cram/circularize/polypolish/cram_R2.clean.sam, cram/circularize/polypolish/cram_polypolish.fasta, cram/circularize/polypolish/polypolish.debug.log, cram/stats/circularize/polypolish_changes.log
 - search_contig_start: blam/circularize/identify/blam_circular.faa, blam/circularize/identify/blam_circular.ffn, blam/circularize/identify/blam_circular.gff, blam/circularize/identify/blam_hmmsearch_hits.txt, blam/circularize/identify/blam_hmmsearch_hits_no_comments.txt
 - get_polished_contigs: cram/circularize/filter/cram_circular.fasta
 - combine_circular_and_linear_contigs: blam/circularize/combine/blam_combined.fasta
 - symlink_circularization: blam/circularize/blam_circularize.fasta
 - process_start_genes: blam/circularize/identify/blam_start_genes.list
 - get_start_genes: cram/circularize/identify/cram_start_gene.ffn
 - finalize_circular_contig_rotation: blam/circularize/combine/blam_circular.fasta
 - run_circlator: blam/circularize/circlator/blam_rotated.fasta, blam/circularize/circlator, blam/circularize/circlator/rotated.promer.contigs_with_ends.fa
 - symlink_circularization: cram/circularize/cram_circularize.fasta
 - map_short_reads_for_polishing: blam/circularize/polypolish/blam_R1.sam, blam/circularize/polypolish/blam_R2.sam, blam/circularize/polypolish/input/blam_input.fasta.0123, blam/circularize/polypolish/input/blam_input.fasta.amb, blam/circularize/polypolish/input/blam_input.fasta.ann, blam/circularize/polypolish/input/blam_input.fasta.bwt.2bit.64, blam/circularize/polypolish/input/blam_input.fasta.pac
 - search_contig_start: cram/circularize/identify/cram_circular.faa, cram/circularize/identify/cram_circular.ffn, cram/circularize/identify/cram_circular.gff, cram/circularize/identify/cram_hmmsearch_hits.txt, cram/circularize/identify/cram_hmmsearch_hits_no_comments.txt
 - polish_polypolish: blam/circularize/polypolish/blam_R1.clean.sam, blam/circularize/polypolish/blam_R2.clean.sam, blam/circularize/polypolish/blam_polypolish.fasta, blam/circularize/polypolish/polypolish.debug.log, blam/stats/circularize/polypolish_changes.log
 - combine_circular_and_linear_contigs: cram/circularize/combine/cram_combined.fasta
 - prepare_polypolish_circularize_input: blam/circularize/polypolish/input/blam_input.fasta
 - finalize_circular_contig_rotation: cram/circularize/combine/cram_circular.fasta
Complete log: .snakemake/log/2024-02-16T005846.034888.snakemake.log

@jmtsuji I get the above error every time I run with --until circularize.

Does that make more sense? I think it is fundamentally a snakemake bug, but I think we could work around it temporarily by making the QC reads non-temp by default. However, I suspect that the bug will not appear if you have the pipeline run fully to the end. Let me know your thoughts here -- thanks!

@jmtsuji Changing it to keep the qc reads fixes the above error. For me, the pipeline, when run end to end, still passes even with the keep qced reads set to false in the config.

@LeeBergstrand My tests are also finished :-)  Here are my results, using commit 864ca98 (pypolca branch):
Main rotary run with test dataset
If keep_final_qc_read_files is False (current default):

Whole pipeline: finishes successfully
--until circularize: FAILS with the bug above

If keep_final_qc_read_files as True (proposed temporary fix):

Whole pipeline: finishes successfully
--until circularize: finishes successfully

These results seem to match your tests results -- good.
Pipeline re-run
Interestingly, I also checked what would happen if I tried to re-run the pipeline after it finished for all 4 tests:
If keep_final_qc_read_files is False (current default):

Whole pipeline: tries to re-analyze the data, starting from long/short read QC
--until circularize: tries to re-analyze the data, starting from long/short read QC

If keep_final_qc_read_files is True (proposed temporary fix):

Whole pipeline: acknowledges the pipeline is finished and does not re-analyze the data (just re-generates the final checkpoints) <- this is the expected behaviour
--until circularize: tries to re-analyze the data, starting from long read QC (short read QC is OK). It claims it is looking for the missing file S18/qc/long/S18_nanopore_qc.fastq.gz (an early temp file from long read QC)

Summary
Setting keep_final_qc_read_files fixes two issues:

Allows the pipeline to run successfully to rule circularize
Allows full pipeline runs to acknowledge they are completely finished and do not need to re-run

However, the full pipeline still finishes, like you mentioned, even if keep_final_qc_read_files is False
Moving forward
I could go either way with changing keep_final_qc_read_files to True as default. Because the full pipeline still works with keep_final_qc_read_files as False, and discarding the QC read files saves a lot of space, I could see how keeping this feature as default could be nice. We could add a note to the README that keep_final_qc_read_files must be True to run --until circularize (i.e., to skip genome annotation). On the other hand, we could change our default to keep_final_qc_read_files as True to avoid people running into errors.
The re-run tests I did suggest there is a secondary bug with our temp file setup that is preventing the pipeline from properly acknowledging long read QC files. I'll open a second issue for this.
@LeeBergstrand What are your thoughts about how we ought to set keep_final_qc_read_files by default?

If keep_final_qc_read_files is False (current default):

Whole pipeline: tries to re-analyze the data, starting from long/short read QC
--until circularize: tries to re-analyze the data, starting from long/short read QC

If keep_final_qc_read_files is True (proposed temporary fix):

Whole pipeline: acknowledges the pipeline is finished and does not re-analyze the data (just re-generates the final checkpoints) <- this is the expected behaviour
--until circularize: tries to re-analyze the data, starting from long read QC (short read QC is OK). It claims it is looking for the missing file S18/qc/long/S18_nanopore_qc.fastq.gz (an early temp file from long read QC)

I got the same effect.

Moving forward
I could go either way with changing keep_final_qc_read_files to True as default. Because the full pipeline still works with keep_final_qc_read_files as False, and discarding the QC read files saves a lot of space, I could see how keeping this feature as default could be nice. We could add a note to the README that keep_final_qc_read_files must be True to run --until circularize (i.e., to skip genome annotation). On the other hand, we could change our default to keep_final_qc_read_files as True to avoid people running into errors.
The re-run tests I did suggest there is a secondary bug with our temp file setup that is preventing the pipeline from properly acknowledging long read QC files. I'll open a second issue for this.
@LeeBergstrand What are your thoughts about how we ought to set keep_final_qc_read_files by default?

@jmtsuji I recommend we do the following:

Create two separate issues for this.
Set If keep_final_qc_read_files to True by default.
Move to Snakemake 8 and reassess this issue.

@jmtsuji I noticed that you use a Symlinking paradigm throughout the pipeline. For example:
# Conditional based on whether short read polishing was performed
rule pre_coverage_filter:
    input:
        "{sample}/polish/medaka/{sample}_consensus.fasta" if POLISH_WITH_SHORT_READS == False else "{sample}/polish/polca/{sample}_polca.fasta"
    output:
        temp("{sample}/polish/cov_filter/{sample}_pre_filtered.fasta")
    run:
        source_relpath = os.path.relpath(str(input),os.path.dirname(str(output)))
        os.symlink(source_relpath,str(output))

This was quite troublesome when I made the temp() file path as Snakemake would start deleting files that the symlinks would point to. I have found that Snakemake doesn't like symlinking because it relies on the presence and absence of files.
Perhaps the rebuilding issues that came up above are related to the symlinking? For example, Snakemake is having trouble tracking what files are used by what rules. I'm not sure. I would bring up this symlinking issue later, but I wanted to put it here now to ensure you know about it.

If keep_final_qc_read_files is False (current default):

Whole pipeline: tries to re-analyze the data, starting from long/short read QC
--until circularize: tries to re-analyze the data, starting from long/short read QC

If keep_final_qc_read_files is True (proposed temporary fix):

Whole pipeline: acknowledges the pipeline is finished and does not re-analyze the data (just re-generates the final checkpoints) <- this is the expected behaviour
--until circularize: tries to re-analyze the data, starting from long read QC (short read QC is OK). It claims it is looking for the missing file S18/qc/long/S18_nanopore_qc.fastq.gz (an early temp file from long read QC)

I got the same effect.

@jmtsuji It might also rerun the entire pipeline if the config file is modified. My tests aren't really supporting that but I would also look into it. I'm getting the following for rule:  nanopore_qc_filter
Reason: Forced execution

Something is causing the forced execution of the very first rule.

@jmtsuji I recommend we do the following:

Create two separate issues for this.
Set If keep_final_qc_read_files to True by default.
Move to Snakemake 8 and reassess this issue.

@LeeBergstrand Thanks for the thoughts -- I agree. Second issue for checkpointing is already created at #134 , and keep_final_qc_read_files is now set as True by default in PR #138 (thanks for this). Let's keep this issue open and re-assess this bug after moving to Snakemake 8.

This was quite troublesome when I made the temp() file path as Snakemake would start deleting files that the symlinks would point to. I have found that Snakemake doesn't like symlinking because it relies on the presence and absence of files.
Perhaps the rebuilding issues that came up above are related to the symlinking? For example, Snakemake is having trouble tracking what files are used by what rules. I'm not sure. I would bring up this symlinking issue later, but I wanted to put it here now to ensure you know about it.

I've also run into symlinking issues while editing the pipeline and agree, it would be better to streamline these parts, if possible. Reducing the number of steps in the analysis this way might help with consistent DAG construction. (Just for reference for the future: for the polish steps, I think it's necessary to either copy or symlink the files before running the polish rules in order for the rules to be compatible with inputs from multiple modules. This is not the case for other rules, I think, so symlinking steps can probably be streamlined in many other cases.) I'll move this to a separate issue.

@jmtsuji It might also rerun the entire pipeline if the config file is modified. My tests aren't really supporting that but I would also look into it. I'm getting the following for rule: nanopore_qc_filter
Reason: Forced execution

Something is causing the forced execution of the very first rule.

Also interesting -- I haven't seen "Forced execution" as a reason to re-run nanopore_qc_filter in my tests, but I was running with a fresh install and config of rotary. I think snakemake also sometimes forces re-runs if it notices a snakefile has been updated compared to the last run.

@jmtsuji This may be related to this snakemake issue. https://github.com/snakemake/snakemake/pull/2737. The fix for this issue, in Snakemake 8, also caused other issues for me in https://github.com/snakemake/snakemake/issues/2982.


Shadow of Mordor not working

Thank you so much for this!
Like the other user, I also signed up just to say how awesome it was to be able to play Shadow of Mordor, and not feel like I wasted my money on a game that wouldn't invert both axes.
However, at the beginning of this month, it seems to have stopped working out of the blue.
I'm unsure if it was a game update, or something with Steam.

I realize you don't know me, or owe me anything...
I was hoping if you had the free time if you could give me a hand of some sort with it.
If there's any specific information that might help beyond the fact I use the Rock Candy XBox 360 Wired USB Controller, I'd be glad to make mention.
I tried to use X360Ce to avoid bothering you with this, and while it fixed the inversion, the triggers were boned.
At any rate, even if nothing can be done, thanks for helping me get some time into the game!

Alright, I'll use this instead of the steam forum as I get email notifications.
You mentioned the tool worked with the default filename. That's alright, as the tool only works with that name. Then x360ce works, so that means the wrappers in general seem to still work.
Just making sure of having the ini file made, and then the dll renamed to the one used with x360ce should've done the trick already. Are you using the 64bit or 32bit version of x360ce?

Yes I tried the 64bit version of x360ce after I noticed the controls weren't inverted anymore when I tried to play. Though it fixed the inversion problem, the triggers suddenly wouldn't work properly. (In Shadow of Mordor, the left trigger brings you to bow/arrow firing mode, and while depressed you can charge/fire shots with the right trigger.)
In the beginning with Durazno I had the exe, dll (renamed), and ini file in the game exe folder. It just seemed to have oddly changed back when I tried to play at the beginning of this month.

Can you make sure to be using the 64bit version of Durazno as well? Make sure the settings work on the test tab before renaming the dll.

Absolutely.
I went into the Durazno\x64 folder, and copied those files to my ShadowOfMordor\x64 folder.
The settings worked on the test tab. I renamed the dll, loaded the game, and nada.

Can you zip me that "ShadowOfMordor\x64" folder? I'll need to analyze that. Upload it somewhere and post the link please.

Absolutely can. I wanted to try it with Skyrim also. The ini/dll just go in the folder where the game's executable is? Also, aside photos, I've never really had to upload much. Hopefully this'll work. Link.

That worked, thanks. I'll let you know if I find something from it.
And yes, all you need is the dll and the ini for the settings. The tricky part can be finding out what dll name the game requires.
PS: You can delete that from your Dropbox now.

Apologies as I forgot to zip it. Thank you!

No problem, you can dowload it as a zip package from Dropbox.
I haven't found anything odd with the game as it does appear to load the durazno dll without issue. To see what happens ingame, I made this dll that has some logging. Durazno Test
Keep your settings, run the game, use the controlller and exit. See if the duranzo.log file is there and attach it here. Also, when you say that nothing happens when using Durazno, the gamepad still works right? Just not the customization?

This is the link to the log file on Dropbox.
This is the text in the log file.
== DURAZNO LOG BEGIN ==
XInputGetState : (0) durazno_init
LoadCustomDLL
SUCCESS
durazno_shutdown
== DURAZNO LOG CLOSE ==

The gamepad (Rock Candy XBox 360 Wired USB Controller) still works, just no inversion/customizations from Durazno.

It's like it's used once then never again. The game doesn't like my wrapper, then just ditches it. =(
New file, same test. Durazno Test 2

Log File Link
Log Text:
== DURAZNO LOG BEGIN ==
durazno_init
LoadCustomDLL :
durazno_init : SUCCESS, C:\WINDOWS\system32\XINPUT9_1_0.dll
XInputGetState : (0) SUCCESS
durazno_shutdown
== DURAZNO LOG CLOSE ==

Dang it. I have no idea why it suddenly stops working. I might be able to make code some testing application based on the game files you sent me, worth a try.
So far, consider the game unsupported. =S

Okay. It was working wonderfully until February. I've played about 12 hours of the game with it working through January. I know Steam had a few updates, but I doubt they're related to that.

I'll be out of town a few days to a week or so. If this is something you might pursue further, if there's anything I can do to help, please lmk. ^_^

Surething. I'll let you know if I have something to test.

I haven't been able to make a test case. I updated Durazno a bit before, I doubt it would make a difference but give it a try please.
Other than that, I have a few things that might give enough information. Reaching a gameplay point then exit, one would be using 360ce since it worked for you but with logging enabled; another one, would be using ApiMonitor with everything enabled while using Durazno and posting the capture here. Let me know if you're interested.

I'll download it and give it a go!
As far as the other ideas go, glad to give them a try.

With the new one I've downloaded, it doesn't show the inversion in the test area even after clicking the invert X and Y on the right stick.

I think it's fixed now. It was an issue in the GUI, so the wrapper should've worked (or not worked) the same.
Alright, so for the extra logs you already have x360ce so just enable logging or put Log=1 in the INI. Get to a gameplay point and exit, post the log in here.
As for ApiMonitor, you can get it from here: API Monitor and just get the v2 portable one. Make sure to be offline while doing this, as I dunno if it could be detected as a cheat. Use the x64 version, enable all the things to trace and use "Monitor new process" to run the game. While using Durazno reach the same point as the previous test, save the capture (File -> save) and post it here (big file warning).

Can you give this build a try please? durazno_test3
Beware of error messages.

Here's my Shadow of Morder/x64 folder.
When I rename the dll to xinput9_1_0.dll and reopen the application to look at the test tab, the upper area shows no movement of the stick at all, and the bottom area shows uninverted movement.

Thanks, and that's normal.
The application only loads the default dll. The idea was to just use the program to make the ini then copy the ini/dll and do the necessary renaming.

@Lynnifer I could not reproduce it. Can you please test extracting this files into your Mordor\x64 folder? Left stick should behave inverted.

Neither of the sticks were inverted.

I am clueless about this. I had hoped a game update changed the behavior to make Durazno no longer work with it, but nothing odd happened on my test.
There should be a Durazno.log file being created, delete it. Then please replace the files again with those given in the previous post and test again. Please post here the contents of the new log file.

= Durazno Log Opened =
DllMain: DLL_PROCESS_ATTACH
XInputLoadLibrary: C:\WINDOWS\system32\XINPUT9_1_0.dll
XInputGetProc: XInputGetState

Thank you very much. I'll try to make some better log messages on the weekend.

Thanks so much for trying to fix this.  I'd love to play the game someday.  :)

How about giving Xinput Plus a try? I never tried it, but might just work. Let me know if it works please,

It worked!  Thank you so much for suggesting it.


Could you release your pre-computed results?

Thanks for your work! Could you release your pre-computed results?

There is no plan to release the pre-computed results.


Improved testability of socket code

I implemented automatical notification of listeners, so that the test code does not need to have access to the productive socket instance to invoke the listeners.
Additionally it is now possible to fake a socket close and a message initialized by the backend.

I was using angular-websocket and when I tried to test my service that use the lib I realized that a lot of functionality was not implemented in the mock. So I started adding them when I bumped into your pull request. I implemented my service's test cases using your mock's version and they run very well. Thanks for sharing your work.

Thanks a lot for your feedback!
Hope it will get merged soon :-)

This PR is great! I hope this gets some attention and makes it into a release.


please use #import "UMMobClick/MobClick.h"  instead of #import "MobClick.h"

You're welcome to submit  a PR 👍


Unable to use @livekit/components-react hooks

Describe the bug
Im unable to use the @livekit/components-react hooks as I am unable to get the trackRef in my React app. Can anyone guide me on how to get the trackRef to be used in hooks like useTrackToggle.
Client

SDK: @livekit/components-react
Version: 1.2.2

This is resolved. Used the existing ToggleSource type from @livekit/components-core and modified it accordingly to the requirement (types from TrackToggle.tsx)


SS08 didn't appear

Hi I tried this on a GA-H270 Gaming 3, the 14th port I said yes to was SS08 and it is the only one that doesn't appear in the ioreg. So 13 are in ioreg.
Currently the install instructions are at the end of the command. It would be nice to have them in the readme so can refer to them after closing the terminal window. And perhaps some more explanation of exactly where to put the xml snippet, mine almost went inside the SSDT section. Thanks

Do you want to enable: SS08? [Y/n] y
You enabled: SS08, are you using it as an internally connected port (For example WiFi/Bluetooth module)? [y/N] n
  ```

Scope (_SB.PCI0.XHC.RHUB.SS08)
{
Method (_UPC, 0, NotSerialized)  // _UPC: USB Port Capabilities
{
If (LEqual (And (UMAP, 0x80), 0x80))
{
If (LEqual (And (HUBC, 0x80), 0x80))
{
Return (GUPC (Zero, 0x03))
}
Else
{
Return (GUPC (One, 0x03))
}
}
Else
{
Return (GUPC (Zero, 0x03))
}
}
        Method (_PLD, 0, NotSerialized)  // _PLD: Physical Location of Device
        {
            If (LEqual (And (UMAP, 0x80), 0x80))
            {
                If (LEqual (And (HUBC, 0x80), 0x80))
                {
                    Return (GPLD (Zero, Zero))
                }
                Else
                {
                    Return (GPLD (One, 0x08))
                }
            }
            Else
            {
                Return (GPLD (Zero, Zero))
            }
        }
    }

<img width="182" alt="screen shot 2017-08-24 at 04 18 32" src="https://user-images.githubusercontent.com/916912/29647872-51272bd0-8883-11e7-8eb4-21dc9fffc83b.png">


New test pipeline

Install deps script created from install-travis.sh
Requirements.txt created to support pip installation/testing along with install-test-deps-pip.sh

These will allow tests to be performed using the following:

Docker container with any version of omnisci core db
Docker container with python and cuda, currently using rapidsai/rapidsai:cuda10.0-runtime-ubuntu18.04
Nvidia docker runtime to allow GPU access to both containers

The tests will spin up the core db container, and a container to run the script in. It will link both containers in the same docker network and enable ipc sharing between them. From there either the install-test-deps-conda.sh or install-test-deps-pip.sh can be run for installation with conda or pip. And lastly the tests are kicked off with pytest tests.

Could you move most of the Jenkins scripts into the repo too? I often use the CI scripts for projects to figure out how to build & test.

LGTM, failures are only for Travis which is being disabled


use is_alive with thread objs (isAlive deprecated)

Fixes #11297
Status
not-tested
Description
python threads isAlive method disappeared in 3.9 after deprecation. is_alive should be use now.
Is it backward compatible (if not, which system it affects?)
YES
Related PRs
None
External dependencies / deployment changes
None

test this please

test this please


Doesn't Work From Installation

Issue Type: Bug
The Liveshare extension does not work on my machine from the start of installation. Unable to join or start a liveshare at all.
Extension version: 0.4.0
VS Code version: Code 1.49.2 (e5e9e69aed6e1984f7499b7af85b3d05f9a6883a, 2020-09-24T16:29:41.983Z)
OS version: Windows_NT x64 10.0.19041

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz (12 x 2592)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: disabled_offopengl: enabled_onprotected_video_decode: enabledrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
31.72GB (18.00GB free)

Process Argv
--crash-reporter-id 4dd576cd-7d04-4adc-b1bb-122351273b00

Screen Reader
no

VM
0%

Thanks for reaching out. Can you give us more details about your scenario? Also, your logs may provide us with helpful information. You can access them via the Live Share: Export Logs command from the command palette (ctrl + shift + p) and attach them to this issue.

Possible duplicate of https://github.com/MicrosoftDocs/live-share/issues/3875

Closing as a possible dupe. Please use the other bug for tracking.


Fix quick start resource calculate

For quick start script. The resource calculate is not accurate. Need to calculate based on PAI config

we need to document the resource requirement of each PAI service, starting from those deployed in worker nodes.

Double checked the PAI deploy script. For quick start, the qos-switch for PAI service is false. So if we use default config, we won't suffer this issue.(The QOS class for PAI daemonSet will be BestEfforts) But seems we changed the config when deploying INT bed,
Still working on quick start script, but seems not a blocking issue

Quick-start script change: https://github.com/microsoft/pai/pull/4855


update mutable objects in a callback?

Like how immutable-js does it, but maybe it passes a mutable copy into the function?
Like this?
var item = Immutable({name: "bob"})

// new Item is immutable, obj is mutable
var newItem = item.mutate(function(obj) {
   obj.name = "henry"
})

The rationale is that it's really handy to be able to make multiple changes to an object in one go, and get an immutable object back. I'd prefer to have that callback be an immutable function itself, but I can't think of a good way to set properties without breaking type safety (imagine trying to use your .merge with flow).

I haven't come across this use case yet, but it turns out you can already do this pretty cleanly with asMutable:
var item = Immutable({name: "bob"});

// new Item is immutable, obj is mutable
var newItem = (function(obj) {
   obj.name = "henry";
   return obj;
})(item.asMutable());

@seanhess Seems like this would address your use case, yeah? :smiley:

Cool! But wouldn't newItem be mutable afterward?

True! Fixed:
var item = Immutable({name: "bob"});

// new Item is immutable, obj is mutable
var newItem = (function(obj) {
   obj.name = "henry";
   return Immutable(obj);
})(item.asMutable());

I still think something more terse might be a good addition to the library, but nothing I can't fix with my own function. Thanks for being awesome!

Glad it was helpful! :smile_cat:


Remove the full option from homebrew_tap

Description
brew tap --full is no longer supported and there is no replacement.
Related Issue
Fixes #12345
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Chore (non-breaking change that does not add functionality or fix an issue)

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] I have run the pre-merge tests locally and they pass.
[x] I have updated the documentation accordingly.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.
[x] All commits have been signed-off for the Developer Certificate of Origin.

@tas50 am I right in thinking that the docs should get updated here too? Or do I need to do something extra?

We probably wanted to do this with the resource deprecation functionality. We'll also want a Cookstyle check for this too @damacus


[FLINK-11450][table] Port and move TableSource and TableSink to flink-table-common

What is the purpose of the change
This pull request ported the following classes to flink-table-common:
TableSource, TableSink, TableSinkBase, DefinedFieldMapping, NestedFieldsProjectableTableSource, ProjectableTableSource, TableConnectorUtil
Brief change log

Ported source/sink related classes to flink-table-common: TableSource, TableSink, TableSinkBase, DefinedFieldMapping, NestedFieldsProjectableTableSource, ProjectableTableSource, TableConnectorUtil

The following things should be noted:

The package name of TableConnectorUtil is changed from util to utils as there is already a package utils in flink-table-common. To keep the backward compatibility, I have created a package util also and forward the call to the implementation in utils. Please help to take a look at if this is needed?

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

Hi @dianfu ,
I have few comments before starting to review the PR:

I appreciate your work, but it would be better if you could keep the scope as it was originally planned in the corresponding JIRA? This PR actually covers multiple different JIRAs: FLINK-10755, FLINK-11451 and probably few more
Could you please split PRs into smaller chunks(commits) that make sense on their own. It simplifies reviewing enormously. Right now it is nearly impossible to review 4k changed lines at once. This somewhat relates to the previous point, as the JIRAs try to split the rework into those individual subtasks.

Thanks for the great work @dianfu. You ported a lot of classes. Maybe a bit too many. We talked about TableDescriptor but maybe there was some miscommunication. Not all subclasses of table descriptor must be ported at this point. Especially classes such as Schema has you have noticed need some rework. But we should finish the DDL discussion until we rework and break the API again.

@twalthr ExternalCatalog depends on TableDescriptor currently and do you mean that the port of ExternalCatalog should be done after FLINK-11240 is committed?

@dianfu The external catalog table can be ported but we could skip the ExternalCatalogBuilder to not pull in Schema.

But I think porting of external catalog should be done as part of https://issues.apache.org/jira/browse/FLINK-10755, it is not necessary to do it as part of this PR/JIRA.

@twalthr @dawidwys Thanks for your clarification and suggestion. Please correct me if I understand not correctly: we will port TableSource/TableSink related classes and ExternalCatalog, TableDescriptor, DescriptorBase, Descriptor, TableStats and ColumnStats to flink-table-common  as these descriptor classes are very simple and will not pull too much dependencies and port other descriptor related classes in FLINK-10755.

Thanks for working on this @dianfu. Here are my thoughts:

Add ColumnStats to this list.
SchematicDescriptor requires Schema. I would postpone this for now.
RegistrableDescriptor is more an API thing and must not be used by extension points in common.

@twalthr Thanks for the suggestion. Make sense to me. Regarding to StreamableDescriptor, I'd like to declare it an interface and doesn't extend TableDescriptor anymore. The reason is that TableDescriptor is an abstract class and StreamableDescriptor has to be declared as an abstract class if it extends TableDescriptor. Given that StreamTableDescriptor extends both ConnectTableDescriptor and StreamableDescriptor, it will be impossible. What's your thought about this change?

@dianfu that makes sense to me. The overall goal should be that the API mentioned in https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/connect.html is still valid.

OK. I'll keep it in mind and create another JIRA for this change. Thanks a lot.


Picture round two

I updated the button in the header to link to the blog post and changed out the  to  based on feedback.

Block it


"ShellExecuteEx failed: File not found." when running fusion360-launcher.sh in Manjaro

It seems like the installer script works in every way except actually installing Fusion 360, as when I run fusion360-launcher.sh it doesn't find Fusion360.exe (and nor can I, anywhere on my drive)
I've attached a log of the installation process.
f360.log
EDIT: Here's what happens whey I try to run Fusion360Installer.exe manually:
[g@moon INSTALLDIR]$ WINEPREFIX=/home/g/.wineprefixes/fusion360 wine data/fusion360/Fusion360installer.exe -p deploy -g -f log.txt  MESA-INTEL: warning: Haswell Vulkan support is incomplete 002c:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub 0034:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub 0024:fixme:imm:ImeSetActiveContext (0x49f700, 1): stub 0024:fixme:imm:ImmReleaseContext (00010054, 0049F700): stub 0070:fixme:imm:ImeSetActiveContext (0x269d00, 0): stub 0070:fixme:imm:ImmReleaseContext<PHONE_NUMBER>010020,<PHONE_NUMBER>269D00): stub 00fc:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub [g@moon INSTALLDIR]$ 003c:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub

Your graphics card driver is incomplete?

[g@moon INSTALLDIR]$ WINEPREFIX=/home/g/.wineprefixes/fusion360 wine data/fusion360/Fusion360installer.exe -p deploy -g -f log.txt MESA-INTEL: warning: Haswell Vulkan support is incomplete

And can you check here:

/home/g/.wineprefixes/fusion360/drive_c/Program Files/Autodesk/webdeploy/production

... if you have two folders and in one these you can find the Fusion360.exe.

Hi,
I selected OpenGL on install as I have an AMD GPU, so I'm not sure why the warning (not error) about incomplete Vulkan support. Maybe someone can clarify this ...
As for the folders, they don't exist:
    /home/g/.wineprefixes/fusion360/drive_c/Program Files/Autodesk/webdeploy/production

Okay, then it's so that the installation of Autodesk Fusion 360 failed! :-1:
Please do this:

cd $HOME/.wineprefixes/fusion360/INSTALLDIR/data/fusion360

WINEPREFIX=$HOME/.wineprefixes/fusion360 wine Fusion360installer.exe -p deploy -g -f log.txt --quiet

And then check again:

$HOME/.wineprefixes/fusion360/drive_c/Program Files/Autodesk/webdeploy/production

Hi, thanks for your reply,
Here's what happens when I do that:
`[g@moon fusion360]$ WINEPREFIX=$HOME/.wineprefixes/fusion360 wine Fusion360installer.exe -p deploy -g -f log.txt --quiet
MESA-INTEL: warning: Haswell Vulkan support is incomplete
MESA-INTEL: warning: Haswell Vulkan support is incomplete
002c:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub
0034:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub
0024:fixme:imm:ImeSetActiveContext (0x49f5c0, 1): stub
0024:fixme:imm:ImmReleaseContext (00010054, 0049F5C0): stub
0068:fixme:imm:ImeSetActiveContext (0x269cc0, 0): stub
0068:fixme:imm:ImmReleaseContext<PHONE_NUMBER>010020,<PHONE_NUMBER>269CC0): stub
00fc:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub
[g@moon fusion360]$ 003c:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub
[g@moon fusion360]$ ls -l $HOME/.wineprefixes/fusion360/drive_c/Program Files/Autodesk/webdeploy/production
ls: cannot access '/home/g/.wineprefixes/fusion360/drive_c/Program': No such file or directory
ls: cannot access 'Files/Autodesk/webdeploy/production': No such file or directory
[g@moon fusion360]$
`
I'm running wine-6.23 if this helps.
Cheers

I trieddisabling the Intel integrated graphics (which I am not using, the monitors are all plugged into my AMD card) in the BIOS, but when I do that, X fails to start, so maybe I have some messed up GPU configuration somehow, I will look into this.

Ok I've completely disabled the integrate graphics so I no longer have the Vulkan warning, but otherwise it's exactly the same:
`[g@moon fusion360]$ WINEPREFIX=$HOME/.wineprefixes/fusion360 wine Fusion360installer.exe -p deploy -g -f log.txt
002c:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub
0034:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub
0024:fixme:imm:ImeSetActiveContext (0x49f668, 1): stub
0024:fixme:imm:ImmReleaseContext (00010054, 0049F668): stub
0074:fixme:imm:ImeSetActiveContext (0x269e10, 0): stub
0074:fixme:imm:ImmReleaseContext<PHONE_NUMBER>010020,<PHONE_NUMBER>269E10): stub
00fc:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub
[g@moon fusion360]$ 003c:fixme:ver:GetCurrentPackageId<PHONE_NUMBER>21FD70<PHONE_NUMBER>000000): stub
[g@moon fusion360]$ ls -l "$HOME/.wineprefixes/fusion360/drive_c/Program Files/Autodesk/webdeploy/production"
ls: cannot access '/home/g/.wineprefixes/fusion360/drive_c/Program Files/Autodesk/webdeploy/production': No such file or directory
[g@moon fusion360]$`
It does seem to successfully extract though, as the little window comes up showing this

Can you check your /etc/hosts and /etc/hostname
For exemable (hosts):
<IP_ADDRESS> localhost
<IP_ADDRESS> YOUR-PC-NAME
For examble (hostname):
YOUR-PC-NAME

[g@moon ~]$ cat /etc/hosts  <IP_ADDRESS>	localhost <IP_ADDRESS>	moon ::1	localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters <redacted-for-privacy>	mothership [g@moon ~]$ cat /etc/hostname moon

Looks fine. :+1:

It just occurred to me, if you should have Flatpak on your system, then you could use my development branch.

https://github.com/cryinkfly/Autodesk-Fusion-360-for-Linux/tree/main/files/scripts/development-branch/flatpak

Thanks!
I'll try that now

Hi,
I can't figure out where (if anywhere) it installs to ...
Here's what happens when I run the script:
`./install-flatpak.sh
Looking for matches…
Skipping: org.freedesktop.Platform/x86_64/20.08 is already installed
Skipping: org.freedesktop.Platform.Compat.i386/x86_64/20.08 is already installed
Skipping: org.freedesktop.Platform.GL32.default/x86_64/20.08 is already installed
Skipping: org.freedesktop.Platform.GL.default/x86_64/20.08 is already installed
Skipping: org.freedesktop.Platform.VAAPI.Intel.i386/x86_64/20.08 is already installed
Skipping: org.freedesktop.Platform.VAAPI.Intel/x86_64/20.08 is already installed
Skipping: org.freedesktop.Platform.ffmpeg_full.i386/x86_64/20.08 is already installed
Skipping: org.freedesktop.Platform.ffmpeg-full/x86_64/20.08 is already installed
--2021-12-22 16:39:23--  https://github.com/fastrizwaan/flatpak-wine-releases/releases/download/6.19-20211009/org.winehq.flatpak-wine619-6.19-20211010.flatpak
SSL_INIT
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving github.com (github.com)... <IP_ADDRESS>
Connecting to github.com (github.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/361254011/646c70cf-2396-48e9-ad99-77dd5416447a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211222T163923Z&X-Amz-Expires=300&X-Amz-Signature=6ad0d4580ad82b77983c81e4b270692c47df45d4923075745fb0e23b491b7034&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=361254011&response-content-disposition=attachment%3B filename%3Dorg.winehq.flatpak-wine619-6.19-20211010.flatpak&response-content-type=application%2Foctet-stream [following]
--2021-12-22 16:39:23--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/361254011/646c70cf-2396-48e9-ad99-77dd5416447a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211222%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211222T163923Z&X-Amz-Expires=300&X-Amz-Signature=6ad0d4580ad82b77983c81e4b270692c47df45d4923075745fb0e23b491b7034&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=361254011&response-content-disposition=attachment%3B filename%3Dorg.winehq.flatpak-wine619-6.19-20211010.flatpak&response-content-type=application%2Foctet-stream
SSL_INIT
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 416 Range Not Satisfiable
The file is already fully retrieved; nothing to do.

org.winehq.flatpak-wine619 permissions:
ipc             network              pulseaudio           x11      devices      devel
multiarch       file access [1]      dbus access [2]
[1] host, ~/.local/share/flatpak-wine619:create
[2] org.freedesktop.Flatpak, org.freedesktop.Notifications, org.gnome.Mutter.DisplayConfig

    ID                                 Branch               Op        Remote                         Download

[✗] org.winehq.flatpak-wine619         6.19-20211010        i         flatpak-wine6191-origin        0 bytes

Error: org.winehq.flatpak-wine619 already installed
error: Failed to install bundle org.winehq.flatpak-wine619: This version of org.winehq.flatpak-wine619 is already installed
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Using winetricks 20210206-next - sha256sum: 4337110af45b0d770d0d847c916d016a432aa1fe7a9beb9880c5af2bfed76154 with wine-6.19 and WINEARCH=win64
Executing w_do_call corefonts
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_corefonts
Executing w_do_call andale
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_andale
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/andale32.exe
warning: Running /app/bin/wineserver -w. This will hang until all wine processes in prefix=/home/g/.local/share/flatpak-wine619/default terminate
Executing wine regedit /S C:\windows\Temp_register-font.reg
Executing wine64 regedit /S C:\windows\Temp_register-font.reg
Executing cp /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/_register-font.reg /tmp/winetricks.B4SPX3Pe/_reg_90ad6a9f_9.reg
Executing wine regedit /S C:\windows\Temp_register-font.reg
Executing wine64 regedit /S C:\windows\Temp_register-font.reg
Executing cp /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/_register-font.reg /tmp/winetricks.B4SPX3Pe/_reg_f6204652_9.reg
Executing w_do_call arial
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_arial
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/arial32.exe
warning: Running /app/bin/wineserver -w.  This will hang until all wine processes in prefix=/home/g/.local/share/flatpak-wine619/default terminate
Executing wine regedit /S C:\windows\Temp_register-font.reg
Executing wine64 regedit /S C:\windows\Temp_register-font.reg
Executing cp /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/_register-font.reg /tmp/winetricks.B4SPX3Pe/_reg_66d0267b_9.reg
Executing wine regedit /S C:\windows\Temp_register-font.reg
Executing wine64 regedit /S C:\windows\Temp_register-font.reg
Executing cp /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/_register-font.reg /tmp/winetricks.B4SPX3Pe/_reg_8067e78b_9.reg
Executing w_do_call comicsans
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_comicsans
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/comic32.exe
warning: Running /app/bin/wineserver -w.  Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_courier
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/courie32.exe
warning: Running /app/bin/wineserver -w.  Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_georgia
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/georgi32.exe
warning: Running /app/bin/wineserver -w.  Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_impact
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/impact32.exe
warning: Running /app/bin/wineserver -w. This will hang until all wine processes in prefix=/home/g/.local/share/flatpak-wine619/default terminate
Executing wine regedit /S C:\windows\Temp_register-font.reg
Executing wine64 regedit /S C:\windows\Temp_register-font.reg
Executing cp /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/_register-font.reg /tmp/winetricks.B4SPX3Pe/_reg_85408d52_9.reg
Executing wine regedit /S C:\windows\Temp_register-font.reg
Executing wine64 regedit /S C:\windows\Temp_register-font.reg
Executing cp /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/_register-font.reg /tmp/winetricks.B4SPX3Pe/_reg_83bd4a0f_9.reg
Executing w_do_call times
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_times
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/times32.exe
warning: Running /app/bin/wineserver -w.  Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_trebuchet
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/trebuc32.exe
warning: Running /app/bin/wineserver -w.  Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_verdana
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/verdan32.exe
warning: Running /app/bin/wineserver -w.  Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_webdings
Executing cabextract -q -d /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/corefonts/webdin32.exe
warning: Running /app/bin/wineserver -w. This will hang until all wine processes in prefix=/home/g/.local/share/flatpak-wine619/default terminate
0070:err:rpc:RpcAssoc_BindConnection receive failed with error 1726
Executing wine regedit /S C:\windows\Temp_register-font.reg
Executing wine64 regedit /S C:\windows\Temp_register-font.reg
Executing cp /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/_register-font.reg /tmp/winetricks.B4SPX3Pe/_reg_405b24a4_9.reg
Executing wine regedit /S C:\windows\Temp_register-font.reg
Executing wine64 regedit /S C:\windows\Temp_register-font.reg
Executing cp /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/_register-font.reg /tmp/winetricks.B4SPX3Pe/_reg_70fc14d8_9.reg
Executing w_do_call msxml4
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_msxml4
Using native,builtin override for following DLLs: msxml4
Executing wine regedit /S C:\windows\Temp\override-dll.reg
Executing wine64 regedit /S C:\windows\Temp\override-dll.reg
Executing cd /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/msxml4
Executing wine msiexec /i msxml.msi /q
0234:fixme:file:NtLockFile I/O completion on lock not implemented yet
0234:fixme:ntdll:NtQuerySystemInformation info_class SYSTEM_PERFORMANCE_INFORMATION
Executing w_do_call msxml6
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_msxml6
Executing cabextract -q --directory=/home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/msxml6/msxml6-KB973686-enu-amd64.exe
Executing cabextract -q --directory=/home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/msxml6.msi
Executing cp -f /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/msxml6.dll.86F857F6_A743_463D_B2FE_98CB5F727E09 /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/syswow64/msxml6.dll
Executing cp -f /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/msxml6r.dll.86F857F6_A743_463D_B2FE_98CB5F727E09 /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/syswow64/msxml6r.dll
Executing cp -f /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/msxml6.dll.1ECC0691_D2EB_4A33_9CBF_5487E5CB17DB /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/system32/msxml6.dll
Executing cp -f /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/temp/msxml6r.dll.1ECC0691_D2EB_4A33_9CBF_5487E5CB17DB /home/g/.local/share/flatpak-wine619/default/dosdevices/c:/windows/system32/msxml6r.dll
Using native,builtin override for following DLLs: msxml6
Executing wine regedit /S C:\windows\Temp\override-dll.reg
Executing wine64 regedit /S C:\windows\Temp\override-dll.reg
Executing w_do_call vcrun2019
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_vcrun2019
warning: ucrtbase.dll is no longer included in vcrun2019. For details see: https://github.com/Winetricks/winetricks/issues/1770
Using native,builtin override for following DLLs: api-ms-win-crt-private-l1-1-0 api-ms-win-crt-conio-l1-1-0 api-ms-win-crt-heap-l1-1-0 api-ms-win-crt-locale-l1-1-0 api-ms-win-crt-math-l1-1-0 api-ms-win-crt-runtime-l1-1-0 api-ms-win-crt-stdio-l1-1-0 api-ms-win-crt-time-l1-1-0 atl140 concrt140 msvcp140 msvcp140_1 msvcp140_2 msvcr140 vcomp140 vcruntime140
Executing wine regedit /S C:\windows\Temp\override-dll.reg
Executing wine64 regedit /S C:\windows\Temp\override-dll.reg
warning: Checksum for /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/vcrun2019/vc_redist.x86.exe did not match, but --force was used, so ignoring and trying anyway.

warning: Working around wine bug 50894 -- Working around failing wusa.exe lookup via C:\windows\SysNative
Executing wine winecfg -v winxp64
warning: Running /app/bin/wineserver -w. This will hang until all wine processes in prefix=/home/g/.local/share/flatpak-wine619/default terminate
Executing cd /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/vcrun2019
Executing wine vc_redist.x86.exe /q
0024:fixme:heap:RtlSetHeapInformation 00000000 1 00000000 0 stub
0024:fixme:advapi:DecryptFileW (L"C:\windows\Temp\{28703CCF-5838-44DB-AF9D-71C89BA449E0}\", 00000000): stub
0104:fixme:heap:RtlSetHeapInformation 00000000 1 00000000 0 stub
0104:fixme:advapi:DecryptFileW (L"C:\windows\Temp\{308DD496-BEFA-42D7-8EE0-472508180D33}\", 00000000): stub
0110:fixme:shell:SHAutoComplete stub
0104:fixme:advapi:DecryptFileW (L"C:\windows\Temp\{308DD496-BEFA-42D7-8EE0-472508180D33}\", 00000000): stub
0104:fixme:exec:SHELL_execute flags ignored: 0x00000100
0118:fixme:heap:RtlSetHeapInformation 00000000 1 00000000 0 stub
0118:fixme:ole:CoInitializeSecurity 0032F5F8, -1, 00000000, 00000000, 6, 2, 00000000, 12288, 00000000 stub
0118:fixme:wuapi:automatic_updates_Pause
0118:fixme:sfc:SRSetRestorePointW 0032F4C0 0032F6D0
0118:fixme:wuapi:automatic_updates_Resume
0118:fixme:ver:GetCurrentPackageId (0032FEA4 00000000): stub
0104:fixme:ver:GetCurrentPackageId (0032FEA4 00000000): stub
0024:fixme:ver:GetCurrentPackageId (0032FEA4 00000000): stub
Using native,builtin override for following DLLs: vcruntime140_1
Executing wine regedit /S C:\windows\Temp\override-dll.reg
Executing wine64 regedit /S C:\windows\Temp\override-dll.reg
warning: Checksum for /home/g/.var/app/org.winehq.flatpak-wine619/cache/winetricks/vcrun2019/vc_redist.x64.exe did not match, but --force was used, so ignoring and trying anyway.
Executing wine vc_redist.x64.exe /q
015c:fixme:heap:RtlSetHeapInformation 00000000 1 00000000 0 stub
015c:fixme:advapi:DecryptFileW (L"C:\windows\Temp\{B564628F-F102-4B9D-9EC5-7FCF9E49141A}\", 00000000): stub
0170:fixme:heap:RtlSetHeapInformation 00000000 1 00000000 0 stub
0170:fixme:advapi:DecryptFileW (L"C:\windows\Temp\{9B6EE78A-8D4F-4F92-A7A9-68655CEB8B1B}\", 00000000): stub
017c:fixme:shell:SHAutoComplete stub
0170:fixme:advapi:DecryptFileW (L"C:\windows\Temp\{9B6EE78A-8D4F-4F92-A7A9-68655CEB8B1B}\", 00000000): stub
0170:fixme:exec:SHELL_execute flags ignored: 0x00000100
0184:fixme:heap:RtlSetHeapInformation 00000000 1 00000000 0 stub
0184:fixme:ole:CoInitializeSecurity 0032F5F8, -1, 00000000, 00000000, 6, 2, 00000000, 12288, 00000000 stub
0184:fixme:wuapi:automatic_updates_Pause
0184:fixme:sfc:SRSetRestorePointW 0032F4C0 0032F6D0
0184:fixme:wuapi:automatic_updates_Resume
0184:fixme:ver:GetCurrentPackageId (0032FEA4 00000000): stub
0170:fixme:ver:GetCurrentPackageId (0032FEA4 00000000): stub
015c:fixme:ver:GetCurrentPackageId (0032FEA4 00000000): stub
Executing wine winecfg -v win7
warning: Running /app/bin/wineserver -w. This will hang until all wine processes in prefix=/home/g/.local/share/flatpak-wine619/default terminate
Executing w_do_call fontsmooth=rgb
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_fontsmooth rgb
Setting font smoothing to rgb
Executing wine regedit /S C:\windows\Temp\fontsmooth.reg
Executing wine64 regedit /S C:\windows\Temp\fontsmooth.reg
Executing w_do_call win8
Executing mkdir -p /home/g/.local/share/flatpak-wine619
warning: You are using a 64-bit WINEPREFIX. Note that many verbs only install 32-bit versions of packages. If you encounter problems, please retest in a clean 32-bit WINEPREFIX before reporting a bug.
Executing load_win8
Executing wine winecfg -v win8
warning: Running /app/bin/wineserver -w. This will hang until all wine processes in prefix=/home/g/.local/share/flatpak-wine619/default terminate
`

Sorry, there was a bug: https://github.com/cryinkfly/Autodesk-Fusion-360-for-Linux/pull/157

I tried with the updated script but still no installer appears ...
https://pastebin.com/TRR7qUQu

Run this command:
1.)
flatpak run org.winehq.flatpak-wine619 bash

2.)
mkdir -p "$HOME/.wineprefixes/fusion360/INSTALLDIR/data/fusion360"

3.)
cd "$HOME/.wineprefixes/fusion360/INSTALLDIR/data/fusion360"

4.)
wget https://dl.appstreaming.autodesk.com/production/installers/Fusion%20360%20Admin%20Install.exe -O Fusion360installer.exe

5.)
wine Fusion360installer.exe -p deploy -g -f log.txt --quiet

6.)
wine Fusion360installer.exe -p deploy -g -f log.txt --quiet

7.)
mkdir -p "$HOME/.local/share/flatpak-wine619/default/drive_c/users/steamuser/Application Data/Autodesk/Neutron Platform/"

8.)
wget -N -P $HOME/.local/share/flatpak-wine619/default/drive_c/users/steamuser/Application Data/Autodesk/Neutron Platform/Options https://github.com/cryinkfly/Autodesk-Fusion-360-for-Linux/raw/main/files/extras/opengl_dxvk/OpenGL.xml -O NMachineSpecificOptions.xml

Hi,
Thanks again for all your help. I can get it to install now but it just hangs for a while on the splash screen, then crashes:
https://pastebin.com/3SBZpWQ5

My distribution updated to wine 7.0-rc2 so I am using the non flatpak version now and it works! Thanks again!


Bump jupyterhub chart and hub image to version 1.1.3

This brings in authenticating with GitHub Teams feature from
OAuthenticator
first step towards #625
EDIT: closes #539

Happy to let someone else do that if it can't wait until I'm back? (About to go pick up my cat-sitter!) I don't think this is super urgent as we're not working on the OAuthenticator stuff yet :)

@sgibson91 absolutely! PLEASE GO VACATION! ROME NEEDS YOU!!!!!! SEND PICS :)

Update: We plan to merge this in ~12 hours (@GeorgianaElena will shepherd and others should be ready to assist if something unexpected happens!)

Before merging this, I tried to deploy it first manually only on the staging hub.
But, I bumped into this chart version mismatch when trying to do so. We declare a new version for the basehub chart, but the daskhub chart depends still on the old one.
My guess is that the basehub chart version was changed by mistake. @consideRatio, is this right or am I missing something?
Should we instead (of my code suggestion) have the daskhub chart depend on the new basehub chart version (version: "0.0.1-n1159.h5b045cd")?

Oh doh, great catch @GeorgianaElena! Either make daskhub reference basehubs new version, or make basehubs version be 0.1.0 again or whatever it was before.
Also update the inline comment about using chartpress about the need to keep basehubs chart version in sync with the dependency reference in daskhub.

Thanks @consideRatio! The staging deploy went ok and I was able to login and start a server on that hub afterwards.
So, I'll go ahead and push the merge button now :crossed_fingers:
Thanks everyone! :sun_with_face:

Thank you for stewarding @GeorgianaElena and @consideRatio for assisting! 🎉


Support for array of objects

It looks like consulator doesn't understand hot to import the following:
config:
  foo:
    rules:
      - foo: bar
        exp:
          type: A

You end up with an err rlike:
Error: Invalid type map[string]interface {} in array. Only strings, numbers and boolean values are supported.

Is this a KP or intentional?

I've narrowed it down to array of objects:
Example:
AAPL:
  - shares: -75.088
    date: 11/27/2015
  - shares: 75.088
    date: 11/26/2015

This fails to import.

Currently, it's intentionally not supported.
If it were to be supported, how would you expect it to be represented in Consul?

As a tree of key/value(s) of course :) -- My consumer here is Consul Template and I was trying to use the {{ tree "<prefix>" }} function but since consulator can't import such things I've opted for:
config:
  nested_json: |
    <json here>

What do you mean by a "a tree of key/value(s)"?
What would the structure look like? Are you suggesting it should create a path component from the index of your array, like this?
AAPL/0/shares: -75.088
AAPL/0/date: 11/27/2015
AAPL/1/shares: 75.088
AAPL/1/date: 11/26/2015

Or something different?
Consul can only store simple values at a given path, so I don't see a logical way to do this...
Sorry if I'm missing something obvious! :slightly_smiling_face:

No you are absolutely  right! I don't know what I was thinking :P -- Long day!
It makes no sense to import an array of objects. Thanks for confirming my silliness :D


feat!: publish in CJS and ESM

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

[ ] Make sure to open an issue as a bug/issue before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
[ ] Ensure the tests and linter pass
[ ] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

Need to merge #627 first


More than 480 streams in plex

Sorry if I am not using this correctly, but I can't seem to find an answer.
Is it possible using filters to have more than 480 streams loaded into Plex?

That is a plex restriction, plex can not handle more.

You can also filter more than 480 channels, only Plex will not show all channels.


index out of range error with select_taxa.py --chimera

Hey guys,
I have an issue running select_taxa.py. Right now I'm trying to combine several taxa into a new chimera, but select_taxa.py --chimeras chimeras_input.tsv gives me the following error:
Traceback (most recent call last):
  File "/opt/Anaconda3/envs/fisher_1.2.7/bin/select_taxa.py", line 182, in <module>
    make_subset_tsv()
  File "/opt/Anaconda3/envs/fisher_1.2.7/bin/select_taxa.py", line 42, in make_subset_tsv
    low_tax_list = [taxa[ind][2] for ind in df.index]
  File "/opt/Anaconda3/envs/fisher_1.2.7/bin/select_taxa.py", line 42, in <listcomp>
    low_tax_list = [taxa[ind][2] for ind in df.index]
IndexError: list index out of range

The chimeras_input.tsv file is formatted properly as far as I can tell:
newChimeraNameHigherTaxonomyLowerTaxonomytaxon1taxon2taxon3
I've tried it with several version of Phylofisher and keep getting the same error message. select_taxa.py without the --chimera flag works like it should and always has. If I remember correctly this used to work without issues about 1.5 years ago.
Any idea what the issue might be?
Cheers,
Gordon

Got it to work with version 1.1.2, and I now have a select_taxa.tsv. But that version doesn't generate a plot and gives me this:
Traceback (most recent call last):
  File "/opt/Anaconda3/envs/fisher_1.1.2/bin/select_taxa.py", line 181, in <module>
    tools.make_plot(taxa_comp, f'taxa_comp', y_count=gene_count, genes=False)
  File "/opt/Anaconda3/envs/fisher_1.1.2/lib/python3.7/site-packages/phylofisher/tools.py", line 170, in make_plot
    ax2.set_xticklabels(range(0, gene_num, 5))
  File "/opt/Anaconda3/envs/fisher_1.1.2/lib/python3.7/site-packages/matplotlib/axes/_base.py", line 73, in wrapper
    return get_method(self)(*args, **kwargs)
  File "/opt/Anaconda3/envs/fisher_1.1.2/lib/python3.7/site-packages/matplotlib/_api/deprecation.py", line 471, in wrapper
    return func(*args, **kwargs)
  File "/opt/Anaconda3/envs/fisher_1.1.2/lib/python3.7/site-packages/matplotlib/axis.py", line 1779, in _set_ticklabels
    return self.set_ticklabels(labels, minor=minor, **kwargs)
  File "/opt/Anaconda3/envs/fisher_1.1.2/lib/python3.7/site-packages/matplotlib/axis.py", line 1701, in set_ticklabels
    "The number of FixedLocator locations"
ValueError: The number of FixedLocator locations (78), usually from a call to set_ticks, does not match the number of ticklabels (77).

Hi Gordon,
Do you mind providing your chimeras_input.tsv? Each column in the file should be tab separated. I'm not able to find anything obviously wrong. This code hasn't been changed in several iterations now.
As far as the error in v1.1.2., that has now been handled. It stems from a plotting bug. You should be fine to move forward; you just won't be able to see the plot.
Let me know if there is anything else I can do to help.
Best,
Robert

Hi Robert,
absolutely, please see attached! That one worked with version 1.1.2 and I ran matrix_constructor.py successfully and already have a final tree.
For all I know this might be an issue on my side, but I found a workaround with the 1.1.2.
Cheers,
Gordon
chimeras_Jun7.tsv.zip

Hi Gordon,
This issue has now been resolved in v1.2.9. Which has been pushed and should be available from Bioconda in a few days.
Thank for pointing this out to us!
Best,
Robert

Thanks for fixing this Robert!
Cheers,
Gordon


Add html versions of our invite and signup emails.

The invite email is at templates/confirmation/invite_email_*, and the signup email is at templates/confirmation/preregistration_confirmation_email_*. You can find examples of our html emails under the folders at templates/zerver/emails/.
While there, it would also be good to rename the files to use our conventional naming, e.g. invite_email.txt, invite_email.html, and invite_email.subject.
To send the html versions of the email, you'll have to modify confirmation/models.ConfirmationManager.send_confirmation. The send_mail function takes a html_message argument; git grep send_mail to see other examples of this in our codebase.

@rishig I'll take this issue.

Great! Adding @rohithasrk for coordination (who is also working on email-related stuff).

Okay, @rishig made a PR :) it works for me, however I can only observer the mails in the dev servers output.


Lose Turn on Invalid Move

When a player tries to make an invalid move, their turn ends even though they didn't get to make a move.

fixed in aa4c342


[WIP] Implement kde for Series

This PR takes over https://github.com/databricks/koalas/pull/710

Codecov Report

Merging #767 into master will decrease coverage by 1.96%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #767      +/-   ##
==========================================
- Coverage   93.85%   91.89%   -1.97%
==========================================
  Files          32       32
  Lines        5681     5688       +7
==========================================
- Hits         5332     5227     -105
- Misses        349      461     +112

Impacted Files
Coverage Δ

databricks/koalas/plot.py
94.7% <100%> (-0.18%)
:arrow_down:

databricks/koalas/usage_logging/__init__.py
23.14% <0%> (-74.08%)
:arrow_down:

databricks/koalas/usage_logging/usage_logger.py
50% <0%> (-50%)
:arrow_down:

databricks/koalas/__init__.py
75% <0%> (-10%)
:arrow_down:

databricks/conftest.py
90.69% <0%> (-6.98%)
:arrow_down:

databricks/koalas/frame.py
95.53% <0%> (-0.71%)
:arrow_down:

databricks/koalas/generic.py
94.7% <0%> (-0.53%)
:arrow_down:

databricks/koalas/internal.py
95.83% <0%> (-0.47%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e476ed...6c34c7a. Read the comment docs.

Let me merge this to move forward.


Add a test case for verifying receipt in banking-app

https://github.com/microsoft/ccf-app-samples/pull/46


Release 2.1.0 to develop

What
Release 2.1.0 to develop.
Why
Sync PR #127 with the develop branch.

stellar-disbursement-platform-frontend-preview is available here:http://sdp-frontend-pr128.previews.kube001.services.stellar-ops.com/


guard against __setDefaultValues being called during teardown

This restores a guard that used to exist around trying to set a default value while the component is being torn down.  The changes from #152 made this necessary again as this.get('value') is null during teardown which was causing bad things to happen.

Looks good to me once tests pass


[Bug]: TypeError when playing a Youtube song with @discordx/music

What happened?
When playing a song with Queue.play, nothing is played. Looking at the stacktrace, it seems like Queue.processQueue throws an error that's caught and it tries to emit an error, and tries to index error.resource.metadata, but that causes an error (See below). The problem seems to be twofold: the song isn't being played, and the error handling causes an error.
node version: 16.6.0
OS: Arch Linux (5.14.14-arch1-1)
If there is any other relevant information I should include, let me know.
Reproduction
Link to repo
To reproduce, paste in the bot token to bot_token, get the bot running and run ~test in a server while connected to a voice channel. Key file to look at are youtubePlayer.ts
Package
@discordx/music
Version
Stable
Relevant log output
➜  BananaBotJS yarn run start
yarn run v1.22.17
$ nodemon --exec BOT_TOKEN=`cat bot_token` node --loader ts-node/esm/transpile-only src/index.ts
[nodemon] 2.0.15
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: ts,json
[nodemon] starting `BOT_TOKEN=<omitted> node --loader ts-node/esm/transpile-only src/index.ts`
(node:2134137) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
BananaTesting >> global >> command >> adding 0 []
BananaTesting >> global >> command >> deleting 0 []
BananaTesting >> global >> command >> updating 0 []
Bot started
TypeError: Cannot read property 'metadata' of undefined
    at MyQueue.processQueue (file:///home/tony/repos/BananaBotJS/node_modules/@discordx/music/src/YoutubePlayer/core/Queue.ts:272:56)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[nodemon] app crashed - waiting for file changes before starting...

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

before testing the repo, yous need to move the Player object outside slash command. check music example for more details.

please use discord-music-bot example and join our discord server for close discussion and support.

Thanks. I should have reached out on the discord server first; didn’t realize there was one. It might be helpful to reference the example in the music package’s README.

Thanks. I should have reached out on the discord server first; didn’t realize there was one. It might be helpful to reference the example in the music package’s README.

Alright, will add discord server link on all our package's readme.

Thanks. I should have reached out on the discord server first; didn’t realize there was one. It might be helpful to reference the example in the music package’s README.

Alright, will add discord server link on all our package's readme.

Ah sorry, my reply was confusing. I had meant adding the music bot example to the README, but that’s a good idea too

Thanks. I should have reached out on the discord server first; didn’t realize there was one. It might be helpful to reference the example in the music package’s README.

Alright, will add discord server link on all our package's readme.

Ah sorry, my reply was confusing. I had meant adding the music bot example to the README, but that’s a good idea too

still confusing, the bot example link is on readme but of discordx package, did you mean for @discordx/music?

Thanks. I should have reached out on the discord server first; didn’t realize there was one. It might be helpful to reference the example in the music package’s README.

Alright, will add discord server link on all our package's readme.

Ah sorry, my reply was confusing. I had meant adding the music bot example to the README, but that’s a good idea too

still confusing, the bot example link is on readme but of discordx package, did you mean for @discordx/music?

Yeah. As far as I can tell, I didn't see any mention of it for @discordx/music but I see you just added it. Thanks.


Lidar Lite V4 NVIDIA AGX

Hi! I am quite new at this and was wondering if you had a work around to using the Lidar Lite V4 with the AGX. I am not sure what to change (pin numbers) to get it to work

This code is very old. You are probably better served using a different library, such as the Adafruit  one. https://github.com/adafruit/Adafruit_CircuitPython_LIDARLite
Please refer to the pinout for the AGX Xavier for the I2C bus information:
https://www.jetsonhacks.com/nvidia-jetson-agx-xavier-gpio-header-pinout/


update README.md, fix make test, add circle.yml

Remove go.mod so the TiDB and parser repo won't be a circle
Add circle.yml so this repo will run CI later
Add README.md about "How to update parser for TiDB"

PTAL @lysu @zimulala

@CaitinChen PTAL