Empty vcf file produced instead of vcf.gz file

Hello,
I noticed that for some samples in my cohort, the final files produced were empty vcf files instead of vcf.gz files when these included CNVs. Is this the expected behavior when simply no CNV is called ?
There don't seem to be any errors in the logs.

Thanks.

Hi @Tintest
This should not happen, could you be so kind and confirm for me if there are any calls in the bed.gz file from sample1?
Do you have by any chance the log of sample1?
 Did you forget to use the -0 option?
spectre::2024-04-10 14:50:39,581::INFO::spectre.analysis.analysis>  Determined bin size from Mosdepth coverage: 1000
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
spectre::2024-04-10 14:50:43,310::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr1: 248957
spectre::2024-04-10 14:50:43,371::DEBUG::spectre.analysis.analysis>  nan_count 39965 v. 248957 self.coverage.size
spectre::2024-04-10 14:50:43,376::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr1:     avg: 16.495551552212532, std: 3.911249453813505, med: 16.3
spectre::2024-04-10 14:50:43,383::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr2: 242194
spectre::2024-04-10 14:50:43,439::DEBUG::spectre.analysis.analysis>  nan_count 16271 v. 242194 self.coverage.size
spectre::2024-04-10 14:50:43,444::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr2:     avg: 16.384813719718665, std: 3.6760824957335245, med: 16.22
spectre::2024-04-10 14:50:43,450::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr3: 198296
spectre::2024-04-10 14:50:43,496::DEBUG::spectre.analysis.analysis>  nan_count 6664 v. 198296 self.coverage.size
spectre::2024-04-10 14:50:43,500::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr3:     avg: 16.37970907781581, std: 3.600608108116106, med: 16.23
spectre::2024-04-10 14:50:43,506::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr4: 190215
spectre::2024-04-10 14:50:43,560::DEBUG::spectre.analysis.analysis>  nan_count 7521 v. 190215 self.coverage.size
spectre::2024-04-10 14:50:43,565::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr4:     avg: 16.28546744830153, std: 3.6505990022680552, med: 16.11
spectre::2024-04-10 14:50:43,570::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr5: 181539
spectre::2024-04-10 14:50:43,614::DEBUG::spectre.analysis.analysis>  nan_count 10145 v. 181539 self.coverage.size
spectre::2024-04-10 14:50:43,618::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr5:     avg: 16.3648838932518, std: 3.592289743140997, med: 16.21
spectre::2024-04-10 14:50:43,623::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr6: 170806
spectre::2024-04-10 14:50:43,664::DEBUG::spectre.analysis.analysis>  nan_count 9198 v. 170806 self.coverage.size
spectre::2024-04-10 14:50:43,667::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr6:     avg: 16.504869004009702, std: 4.107705071548058, med: 16.24
spectre::2024-04-10 14:50:43,672::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr7: 159346
spectre::2024-04-10 14:50:43,711::DEBUG::spectre.analysis.analysis>  nan_count 16339 v. 159346 self.coverage.size
spectre::2024-04-10 14:50:43,714::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr7:     avg: 16.393807435999634, std: 3.883543110869586, med: 16.19
spectre::2024-04-10 14:50:43,719::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr8: 145139
spectre::2024-04-10 14:50:43,758::DEBUG::spectre.analysis.analysis>  nan_count 11532 v. 145139 self.coverage.size
spectre::2024-04-10 14:50:43,760::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr8:     avg: 16.39050581182124, std: 3.5861119616763686, med: 16.23
spectre::2024-04-10 14:50:43,765::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr9: 138395
spectre::2024-04-10 14:50:43,800::DEBUG::spectre.analysis.analysis>  nan_count 34657 v. 138395 self.coverage.size
spectre::2024-04-10 14:50:43,803::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr9:     avg: 16.48295908924406, std: 3.828354034671187, med: 16.28
spectre::2024-04-10 14:50:43,808::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr10: 133798
spectre::2024-04-10 14:50:43,840::DEBUG::spectre.analysis.analysis>  nan_count 9556 v. 133798 self.coverage.size
spectre::2024-04-10 14:50:43,843::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr10:    avg: 16.452156919560213, std: 3.6900969462789814, med: 16.26
spectre::2024-04-10 14:50:43,847::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr11: 135087
spectre::2024-04-10 14:50:43,880::DEBUG::spectre.analysis.analysis>  nan_count 9712 v. 135087 self.coverage.size
spectre::2024-04-10 14:50:43,883::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr11:    avg: 16.464615274177465, std: 3.8841716871241454, med: 16.27
spectre::2024-04-10 14:50:43,887::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr12: 133276
spectre::2024-04-10 14:50:43,920::DEBUG::spectre.analysis.analysis>  nan_count 7009 v. 133276 self.coverage.size
spectre::2024-04-10 14:50:43,922::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr12:    avg: 16.405738158030207, std: 3.6570299328368594, med: 16.25
spectre::2024-04-10 14:50:43,927::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr13: 114365
spectre::2024-04-10 14:50:43,956::DEBUG::spectre.analysis.analysis>  nan_count 20732 v. 114365 self.coverage.size
spectre::2024-04-10 14:50:43,958::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr13:    avg: 16.25281941195946, std: 3.5331951862084674, med: 16.09
spectre::2024-04-10 14:50:43,962::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr14: 107044
spectre::2024-04-10 14:50:43,989::DEBUG::spectre.analysis.analysis>  nan_count 22399 v. 107044 self.coverage.size
spectre::2024-04-10 14:50:43,992::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr14:    avg: 16.342916888180046, std: 3.480057755307257, med: 16.21
spectre::2024-04-10 14:50:43,995::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr15: 101992
spectre::2024-04-10 14:50:44,023::DEBUG::spectre.analysis.analysis>  nan_count 31498 v. 101992 self.coverage.size
spectre::2024-04-10 14:50:44,025::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr15:    avg: 16.44266036825829, std: 3.8795770336510373, med: 16.29
spectre::2024-04-10 14:50:44,029::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr16: 90339
spectre::2024-04-10 14:50:44,053::DEBUG::spectre.analysis.analysis>  nan_count 24142 v. 90339 self.coverage.size
spectre::2024-04-10 14:50:44,055::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr16:    avg: 16.490465277882684, std: 3.802242794613207, med: 16.31
spectre::2024-04-10 14:50:44,058::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr17: 83258
spectre::2024-04-10 14:50:44,079::DEBUG::spectre.analysis.analysis>  nan_count 15698 v. 83258 self.coverage.size
spectre::2024-04-10 14:50:44,081::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr17:    avg: 16.57192007104796, std: 4.274941666058131, med: 16.3
spectre::2024-04-10 14:50:44,083::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr18: 80374
spectre::2024-04-10 14:50:44,103::DEBUG::spectre.analysis.analysis>  nan_count 8453 v. 80374 self.coverage.size
spectre::2024-04-10 14:50:44,105::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr18:    avg: 16.344201137359043, std: 3.655624211038385, med: 16.16
spectre::2024-04-10 14:50:44,107::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr19: 58618
spectre::2024-04-10 14:50:44,122::DEBUG::spectre.analysis.analysis>  nan_count 5916 v. 58618 self.coverage.size
spectre::2024-04-10 14:50:44,123::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr19:    avg: 16.764006679063414, std: 4.790208872549115, med: 16.39
spectre::2024-04-10 14:50:44,125::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr20: 64445
spectre::2024-04-10 14:50:44,141::DEBUG::spectre.analysis.analysis>  nan_count 7133 v. 64445 self.coverage.size
spectre::2024-04-10 14:50:44,142::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr20:    avg: 16.580571957007262, std: 3.6874092292418705, med: 16.38
spectre::2024-04-10 14:50:44,145::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr21: 46710
spectre::2024-04-10 14:50:44,157::DEBUG::spectre.analysis.analysis>  nan_count 18970 v. 46710 self.coverage.size
spectre::2024-04-10 14:50:44,159::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr21:    avg: 16.41108904109589, std: 3.772477922474992, med: 16.195
spectre::2024-04-10 14:50:44,160::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chr22: 50819
spectre::2024-04-10 14:50:44,174::DEBUG::spectre.analysis.analysis>  nan_count 19368 v. 50819 self.coverage.size
spectre::2024-04-10 14:50:44,175::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chr22:    avg: 16.636215700613654, std: 4.166625828987789, med: 16.38
spectre::2024-04-10 14:50:44,177::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chrX: 156041
spectre::2024-04-10 14:50:44,216::DEBUG::spectre.analysis.analysis>  nan_count 25036 v. 156041 self.coverage.size
spectre::2024-04-10 14:50:44,220::DEBUG::spectre.analysis.analysis>  Chromsomal Info @ chrX:     avg: 16.322508224876913, std: 3.7127365895852087, med: 16.21
spectre::2024-04-10 14:50:44,225::INFO::spectre.analysis.analysis>  Number positions to be tested on chromosome chrY: 57228
spectre::2024-04-10 14:50:44,245::DEBUG::spectre.analysis.analysis>  nan_count 57228 v. 57228 self.coverage.size
spectre::2024-04-10 14:50:44,245::WARNING::spectre.analysis.analysis>  Chromosome chrY: no data available!
spectre::2024-04-10 14:50:44,246::DEBUG::spectre.analysis.cnv_metrics>  Convert Genome analysis to coverage dataframe
spectre::2024-04-10 14:50:44,945::DEBUG::spectre.analysis.cnv_metrics>  Converting blacklist to exclusion zone
spectre::2024-04-10 14:50:45,001::DEBUG::spectre.analysis.cnv_metrics>  Preparing all parameters for the cnv evaluation
spectre::2024-04-10 14:50:59,733::DEBUG::spectre.analysis.cnv_metrics>  Getting exclusion zone indices in mosdepth coverage data
spectre::2024-04-10 14:51:03,683::DEBUG::spectre.analysis.cnv_metrics>  Generating random coverage sample indices
spectre::2024-04-10 14:51:03,915::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Writing random samples to file
spectre::2024-04-10 14:51:05,922::DEBUG::spectre.analysis.cnv_metrics>  Calculationg DEL/DUP thresholds based on quantiles: 0.1 and 0.9
spectre::2024-04-10 14:51:05,934::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.4809815950920244 and DUP: 2.510429447852761
spectre::2024-04-10 14:51:05,934::DEBUG::spectre.analysis.cnv_metrics>  Calculated thresholds for DEL: 1.4809815950920244 and DUP: 2.510429447852761
spectre::2024-04-10 14:51:05,935::INFO::spectre.analysis.analysis>  Selected threshold for DEL=1.481 and DUP=2.5104 for a ploidy of 2
spectre::2024-04-10 14:51:06,000::INFO::spectre.spectreCNV>  CNV calling - Coverage for sample: Sample2
spectre::2024-04-10 14:51:06,000::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr1
spectre::2024-04-10 14:51:06,273::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr2
spectre::2024-04-10 14:51:06,557::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr3
spectre::2024-04-10 14:51:06,800::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr4
spectre::2024-04-10 14:51:07,035::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr5
spectre::2024-04-10 14:51:07,274::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr6
spectre::2024-04-10 14:51:07,490::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr7
spectre::2024-04-10 14:51:07,701::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr8
spectre::2024-04-10 14:51:07,883::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr9
spectre::2024-04-10 14:51:08,049::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr10
spectre::2024-04-10 14:51:08,199::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr11
spectre::2024-04-10 14:51:08,352::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr12
spectre::2024-04-10 14:51:08,500::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr13
spectre::2024-04-10 14:51:08,624::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr14
spectre::2024-04-10 14:51:08,740::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr15
spectre::2024-04-10 14:51:08,849::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr16
spectre::2024-04-10 14:51:08,950::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr17
spectre::2024-04-10 14:51:09,041::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr18
spectre::2024-04-10 14:51:09,131::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr19
spectre::2024-04-10 14:51:09,200::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr20
spectre::2024-04-10 14:51:09,271::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr21
spectre::2024-04-10 14:51:09,317::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chr22
spectre::2024-04-10 14:51:09,373::INFO::spectre.analysis.analysis>  Calculating CNVs for Sample2 @ chromosome chrX
spectre::2024-04-10 14:51:09,543::DEBUG::spectre.analysis.cnv_metrics>  Evaluating all CNVs
spectre::2024-04-10 14:51:09,543::DEBUG::spectre.analysis.cnv_metrics>  Recalculating
spectre::2024-04-10 14:51:09,543::DEBUG::spectre.analysis.cnv_metrics>  Converting cnv calls to exclusion zone
spectre::2024-04-10 14:51:09,544::DEBUG::spectre.analysis.cnv_metrics>  Merging exclusion zones
spectre::2024-04-10 14:51:24,020::DEBUG::spectre.analysis.cnv_metrics>  Getting exclusion zone indices in mosdepth coverage data
spectre::2024-04-10 14:51:28,214::DEBUG::spectre.analysis.cnv_metrics>  Generating random coverage sample indices
spectre::2024-04-10 14:51:28,467::INFO::spectre.analysis.cnv_metrics>  DEL border:1.4809815950920244     DUP border:2.510429447852761
spectre::2024-04-10 14:51:28,479::DEBUG::spectre.analysis.cnv_metrics>  Creating CNV distribution plot
spectre::2024-04-10 14:51:28,573::DEBUG::spectre.analysis.cnv_metrics>  Plot path: /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/genome_coverage_distribution-Sample2.png
spectre::2024-04-10 14:51:28,653::INFO::spectre.analysis.analysis>  refining cnv calls
spectre::2024-04-10 14:51:28,658::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr1.csv
spectre::2024-04-10 14:51:29,484::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr1: 46 before merge
spectre::2024-04-10 14:51:29,484::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:29,489::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:29,493::DEBUG::spectre.analysis.analysis>  Total merge rounds: 2
spectre::2024-04-10 14:51:29,493::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr1
spectre::2024-04-10 14:51:29,493::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:29,496::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:29,876::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:29,877::DEBUG::spectre.analysis.analysis>  Clean by threshold: 41 -> 33. Cleaned: 8
spectre::2024-04-10 14:51:29,877::DEBUG::spectre.analysis.analysis>  Clean by size: 33 -> 0. Cleaned: 33
spectre::2024-04-10 14:51:29,877::DEBUG::spectre.analysis.analysis>  Clean by size: 41 -> 0. Cleaned: 41
spectre::2024-04-10 14:51:29,881::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr2.csv
spectre::2024-04-10 14:51:30,728::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr2: 24 before merge
spectre::2024-04-10 14:51:30,728::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:30,730::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:30,731::DEBUG::spectre.analysis.analysis>  Total merge rounds: 2
spectre::2024-04-10 14:51:30,731::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr2
spectre::2024-04-10 14:51:30,731::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:30,733::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:30,938::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:30,939::DEBUG::spectre.analysis.analysis>  Clean by threshold: 23 -> 15. Cleaned: 8
spectre::2024-04-10 14:51:30,939::DEBUG::spectre.analysis.analysis>  Clean by size: 15 -> 0. Cleaned: 15
spectre::2024-04-10 14:51:30,939::DEBUG::spectre.analysis.analysis>  Clean by size: 23 -> 0. Cleaned: 23
spectre::2024-04-10 14:51:30,943::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr3.csv
spectre::2024-04-10 14:51:31,648::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr3: 25 before merge
spectre::2024-04-10 14:51:31,648::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:31,650::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:31,652::DEBUG::spectre.analysis.analysis>  Current merge cycle 3
spectre::2024-04-10 14:51:31,653::DEBUG::spectre.analysis.analysis>  Total merge rounds: 3
spectre::2024-04-10 14:51:31,653::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr3
spectre::2024-04-10 14:51:31,653::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:31,654::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:31,848::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:31,848::DEBUG::spectre.analysis.analysis>  Clean by threshold: 21 -> 14. Cleaned: 7
spectre::2024-04-10 14:51:31,848::DEBUG::spectre.analysis.analysis>  Clean by size: 14 -> 0. Cleaned: 14
spectre::2024-04-10 14:51:31,848::DEBUG::spectre.analysis.analysis>  Clean by size: 21 -> 0. Cleaned: 21
spectre::2024-04-10 14:51:31,852::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr4.csv
spectre::2024-04-10 14:51:32,530::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr4: 24 before merge
spectre::2024-04-10 14:51:32,530::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:32,532::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:32,533::DEBUG::spectre.analysis.analysis>  Current merge cycle 3
spectre::2024-04-10 14:51:32,535::DEBUG::spectre.analysis.analysis>  Total merge rounds: 3
spectre::2024-04-10 14:51:32,535::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr4
spectre::2024-04-10 14:51:32,535::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:32,536::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:32,720::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:32,721::DEBUG::spectre.analysis.analysis>  Clean by threshold: 21 -> 16. Cleaned: 5
spectre::2024-04-10 14:51:32,721::DEBUG::spectre.analysis.analysis>  Clean by size: 16 -> 0. Cleaned: 16
spectre::2024-04-10 14:51:32,721::DEBUG::spectre.analysis.analysis>  Clean by size: 21 -> 0. Cleaned: 21
spectre::2024-04-10 14:51:32,724::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr5.csv
spectre::2024-04-10 14:51:33,367::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr5: 24 before merge
spectre::2024-04-10 14:51:33,367::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:33,368::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:33,370::DEBUG::spectre.analysis.analysis>  Current merge cycle 3
spectre::2024-04-10 14:51:33,371::DEBUG::spectre.analysis.analysis>  Total merge rounds: 3
spectre::2024-04-10 14:51:33,371::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr5
spectre::2024-04-10 14:51:33,371::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:33,373::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:33,556::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:33,557::DEBUG::spectre.analysis.analysis>  Clean by threshold: 20 -> 16. Cleaned: 4
spectre::2024-04-10 14:51:33,557::DEBUG::spectre.analysis.analysis>  Clean by size: 16 -> 0. Cleaned: 16
spectre::2024-04-10 14:51:33,557::DEBUG::spectre.analysis.analysis>  Clean by size: 20 -> 0. Cleaned: 20
spectre::2024-04-10 14:51:33,560::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr6.csv
spectre::2024-04-10 14:51:34,169::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr6: 38 before merge
spectre::2024-04-10 14:51:34,169::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:34,172::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:34,174::DEBUG::spectre.analysis.analysis>  Current merge cycle 3
spectre::2024-04-10 14:51:34,176::DEBUG::spectre.analysis.analysis>  Total merge rounds: 3
spectre::2024-04-10 14:51:34,176::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr6
spectre::2024-04-10 14:51:34,176::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:34,178::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:34,472::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:34,473::DEBUG::spectre.analysis.analysis>  Clean by threshold: 33 -> 24. Cleaned: 9
spectre::2024-04-10 14:51:34,473::DEBUG::spectre.analysis.analysis>  Clean by size: 24 -> 0. Cleaned: 24
spectre::2024-04-10 14:51:34,473::DEBUG::spectre.analysis.analysis>  Clean by size: 33 -> 0. Cleaned: 33
spectre::2024-04-10 14:51:34,476::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr7.csv
spectre::2024-04-10 14:51:35,024::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr7: 37 before merge
spectre::2024-04-10 14:51:35,024::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:35,027::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:35,029::DEBUG::spectre.analysis.analysis>  Current merge cycle 3
spectre::2024-04-10 14:51:35,031::DEBUG::spectre.analysis.analysis>  Total merge rounds: 3
spectre::2024-04-10 14:51:35,031::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr7
spectre::2024-04-10 14:51:35,031::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:35,033::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:35,339::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:35,339::DEBUG::spectre.analysis.analysis>  Clean by threshold: 30 -> 25. Cleaned: 5
spectre::2024-04-10 14:51:35,339::DEBUG::spectre.analysis.analysis>  Clean by size: 25 -> 0. Cleaned: 25
spectre::2024-04-10 14:51:35,339::DEBUG::spectre.analysis.analysis>  Clean by size: 30 -> 0. Cleaned: 30
spectre::2024-04-10 14:51:35,343::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr8.csv
spectre::2024-04-10 14:51:35,848::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr8: 14 before merge
spectre::2024-04-10 14:51:35,848::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:35,849::DEBUG::spectre.analysis.analysis>  Total merge rounds: 1
spectre::2024-04-10 14:51:35,849::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr8
spectre::2024-04-10 14:51:35,849::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:35,850::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:35,971::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:35,972::DEBUG::spectre.analysis.analysis>  Clean by threshold: 14 -> 6. Cleaned: 8
spectre::2024-04-10 14:51:35,972::DEBUG::spectre.analysis.analysis>  Clean by size: 6 -> 0. Cleaned: 6
spectre::2024-04-10 14:51:35,972::DEBUG::spectre.analysis.analysis>  Clean by size: 14 -> 0. Cleaned: 14
spectre::2024-04-10 14:51:35,975::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr9.csv
spectre::2024-04-10 14:51:36,404::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr9: 26 before merge
spectre::2024-04-10 14:51:36,404::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:36,406::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:36,408::DEBUG::spectre.analysis.analysis>  Total merge rounds: 2
spectre::2024-04-10 14:51:36,408::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr9
spectre::2024-04-10 14:51:36,408::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:36,410::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:36,607::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:36,608::DEBUG::spectre.analysis.analysis>  Clean by threshold: 22 -> 17. Cleaned: 5
spectre::2024-04-10 14:51:36,608::DEBUG::spectre.analysis.analysis>  Clean by size: 17 -> 0. Cleaned: 17
spectre::2024-04-10 14:51:36,608::DEBUG::spectre.analysis.analysis>  Clean by size: 22 -> 0. Cleaned: 22
spectre::2024-04-10 14:51:36,611::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr10.csv
spectre::2024-04-10 14:51:37,085::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr10: 20 before merge
spectre::2024-04-10 14:51:37,085::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:37,086::DEBUG::spectre.analysis.analysis>  Total merge rounds: 1
spectre::2024-04-10 14:51:37,087::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr10
spectre::2024-04-10 14:51:37,087::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:37,088::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:37,259::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:37,260::DEBUG::spectre.analysis.analysis>  Clean by threshold: 19 -> 13. Cleaned: 6
spectre::2024-04-10 14:51:37,260::DEBUG::spectre.analysis.analysis>  Clean by size: 13 -> 0. Cleaned: 13
spectre::2024-04-10 14:51:37,260::DEBUG::spectre.analysis.analysis>  Clean by size: 19 -> 0. Cleaned: 19
spectre::2024-04-10 14:51:37,263::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr11.csv
spectre::2024-04-10 14:51:37,743::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr11: 30 before merge
spectre::2024-04-10 14:51:37,743::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:37,745::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:37,747::DEBUG::spectre.analysis.analysis>  Total merge rounds: 2
spectre::2024-04-10 14:51:37,747::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr11
spectre::2024-04-10 14:51:37,747::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:37,749::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:37,995::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:37,996::DEBUG::spectre.analysis.analysis>  Clean by threshold: 29 -> 23. Cleaned: 6
spectre::2024-04-10 14:51:37,996::DEBUG::spectre.analysis.analysis>  Clean by size: 23 -> 0. Cleaned: 23
spectre::2024-04-10 14:51:37,996::DEBUG::spectre.analysis.analysis>  Clean by size: 29 -> 0. Cleaned: 29
spectre::2024-04-10 14:51:37,999::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr12.csv
spectre::2024-04-10 14:51:38,478::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr12: 15 before merge
spectre::2024-04-10 14:51:38,478::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:38,479::DEBUG::spectre.analysis.analysis>  Total merge rounds: 1
spectre::2024-04-10 14:51:38,479::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr12
spectre::2024-04-10 14:51:38,479::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:38,480::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:38,609::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:38,610::DEBUG::spectre.analysis.analysis>  Clean by threshold: 15 -> 7. Cleaned: 8
spectre::2024-04-10 14:51:38,610::DEBUG::spectre.analysis.analysis>  Clean by size: 7 -> 0. Cleaned: 7
spectre::2024-04-10 14:51:38,610::DEBUG::spectre.analysis.analysis>  Clean by size: 15 -> 0. Cleaned: 15
spectre::2024-04-10 14:51:38,613::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr13.csv
spectre::2024-04-10 14:51:38,992::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr13: 7 before merge
spectre::2024-04-10 14:51:38,992::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:38,992::DEBUG::spectre.analysis.analysis>  Total merge rounds: 1
spectre::2024-04-10 14:51:38,992::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr13
spectre::2024-04-10 14:51:38,992::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:38,993::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:39,055::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:39,055::DEBUG::spectre.analysis.analysis>  Clean by threshold: 7 -> 5. Cleaned: 2
spectre::2024-04-10 14:51:39,055::DEBUG::spectre.analysis.analysis>  Clean by size: 5 -> 0. Cleaned: 5
spectre::2024-04-10 14:51:39,055::DEBUG::spectre.analysis.analysis>  Clean by size: 7 -> 0. Cleaned: 7
spectre::2024-04-10 14:51:39,057::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr14.csv
spectre::2024-04-10 14:51:39,404::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr14: 7 before merge
spectre::2024-04-10 14:51:39,404::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:39,405::DEBUG::spectre.analysis.analysis>  Total merge rounds: 1
spectre::2024-04-10 14:51:39,405::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr14
spectre::2024-04-10 14:51:39,405::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:39,405::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:39,467::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:39,467::DEBUG::spectre.analysis.analysis>  Clean by threshold: 7 -> 3. Cleaned: 4
spectre::2024-04-10 14:51:39,468::DEBUG::spectre.analysis.analysis>  Clean by size: 3 -> 0. Cleaned: 3
spectre::2024-04-10 14:51:39,468::DEBUG::spectre.analysis.analysis>  Clean by size: 7 -> 0. Cleaned: 7
spectre::2024-04-10 14:51:39,470::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr15.csv
spectre::2024-04-10 14:51:39,777::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr15: 13 before merge
spectre::2024-04-10 14:51:39,777::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:39,778::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:39,779::DEBUG::spectre.analysis.analysis>  Current merge cycle 3
spectre::2024-04-10 14:51:39,779::DEBUG::spectre.analysis.analysis>  Total merge rounds: 3
spectre::2024-04-10 14:51:39,779::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr15
spectre::2024-04-10 14:51:39,779::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:39,780::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:39,868::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:39,868::DEBUG::spectre.analysis.analysis>  Clean by threshold: 9 -> 7. Cleaned: 2
spectre::2024-04-10 14:51:39,868::DEBUG::spectre.analysis.analysis>  Clean by size: 7 -> 0. Cleaned: 7
spectre::2024-04-10 14:51:39,868::DEBUG::spectre.analysis.analysis>  Clean by size: 9 -> 0. Cleaned: 9
spectre::2024-04-10 14:51:39,870::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr16.csv
spectre::2024-04-10 14:51:40,160::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr16: 21 before merge
spectre::2024-04-10 14:51:40,160::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:40,162::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:40,163::DEBUG::spectre.analysis.analysis>  Current merge cycle 3
spectre::2024-04-10 14:51:40,164::DEBUG::spectre.analysis.analysis>  Total merge rounds: 3
spectre::2024-04-10 14:51:40,164::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr16
spectre::2024-04-10 14:51:40,164::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:40,166::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:40,326::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:40,327::DEBUG::spectre.analysis.analysis>  Clean by threshold: 17 -> 12. Cleaned: 5
spectre::2024-04-10 14:51:40,327::DEBUG::spectre.analysis.analysis>  Clean by size: 12 -> 0. Cleaned: 12
spectre::2024-04-10 14:51:40,327::DEBUG::spectre.analysis.analysis>  Clean by size: 17 -> 0. Cleaned: 17
spectre::2024-04-10 14:51:40,329::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr17.csv
spectre::2024-04-10 14:51:40,606::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr17: 34 before merge
spectre::2024-04-10 14:51:40,606::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:40,609::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:40,611::DEBUG::spectre.analysis.analysis>  Current merge cycle 3
spectre::2024-04-10 14:51:40,613::DEBUG::spectre.analysis.analysis>  Total merge rounds: 3
spectre::2024-04-10 14:51:40,613::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr17
spectre::2024-04-10 14:51:40,613::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:40,615::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:40,890::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:40,891::DEBUG::spectre.analysis.analysis>  Clean by threshold: 27 -> 18. Cleaned: 9
spectre::2024-04-10 14:51:40,891::DEBUG::spectre.analysis.analysis>  Clean by size: 18 -> 0. Cleaned: 18
spectre::2024-04-10 14:51:40,891::DEBUG::spectre.analysis.analysis>  Clean by size: 27 -> 0. Cleaned: 27
spectre::2024-04-10 14:51:40,893::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr18.csv
spectre::2024-04-10 14:51:41,201::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr18: 9 before merge
spectre::2024-04-10 14:51:41,202::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:41,202::DEBUG::spectre.analysis.analysis>  Total merge rounds: 1
spectre::2024-04-10 14:51:41,202::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr18
spectre::2024-04-10 14:51:41,202::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:41,203::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:41,280::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:41,281::DEBUG::spectre.analysis.analysis>  Clean by threshold: 9 -> 7. Cleaned: 2
spectre::2024-04-10 14:51:41,281::DEBUG::spectre.analysis.analysis>  Clean by size: 7 -> 0. Cleaned: 7
spectre::2024-04-10 14:51:41,281::DEBUG::spectre.analysis.analysis>  Clean by size: 9 -> 0. Cleaned: 9
spectre::2024-04-10 14:51:41,282::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr19.csv
spectre::2024-04-10 14:51:41,489::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr19: 48 before merge
spectre::2024-04-10 14:51:41,489::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:41,492::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:41,495::DEBUG::spectre.analysis.analysis>  Total merge rounds: 2
spectre::2024-04-10 14:51:41,495::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr19
spectre::2024-04-10 14:51:41,495::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:41,497::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:41,874::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:41,875::DEBUG::spectre.analysis.analysis>  Clean by threshold: 45 -> 32. Cleaned: 13
spectre::2024-04-10 14:51:41,875::DEBUG::spectre.analysis.analysis>  Clean by size: 32 -> 0. Cleaned: 32
spectre::2024-04-10 14:51:41,875::DEBUG::spectre.analysis.analysis>  Clean by size: 45 -> 0. Cleaned: 45
spectre::2024-04-10 14:51:41,877::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr20.csv
spectre::2024-04-10 14:51:42,104::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr20: 5 before merge
spectre::2024-04-10 14:51:42,105::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:42,105::DEBUG::spectre.analysis.analysis>  Total merge rounds: 1
spectre::2024-04-10 14:51:42,105::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr20
spectre::2024-04-10 14:51:42,105::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:42,106::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:42,151::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:42,151::DEBUG::spectre.analysis.analysis>  Clean by threshold: 5 -> 5. Cleaned: 0
spectre::2024-04-10 14:51:42,151::DEBUG::spectre.analysis.analysis>  Clean by size: 5 -> 0. Cleaned: 5
spectre::2024-04-10 14:51:42,151::DEBUG::spectre.analysis.analysis>  Clean by size: 5 -> 0. Cleaned: 5
spectre::2024-04-10 14:51:42,152::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr21.csv
spectre::2024-04-10 14:51:42,285::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr21: 3 before merge
spectre::2024-04-10 14:51:42,285::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:42,286::DEBUG::spectre.analysis.analysis>  Total merge rounds: 1
spectre::2024-04-10 14:51:42,286::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr21
spectre::2024-04-10 14:51:42,286::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:42,286::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:42,313::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:42,313::DEBUG::spectre.analysis.analysis>  Clean by threshold: 3 -> 1. Cleaned: 2
spectre::2024-04-10 14:51:42,313::DEBUG::spectre.analysis.analysis>  Clean by size: 1 -> 0. Cleaned: 1
spectre::2024-04-10 14:51:42,313::DEBUG::spectre.analysis.analysis>  Clean by size: 3 -> 0. Cleaned: 3
spectre::2024-04-10 14:51:42,315::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchr22.csv
spectre::2024-04-10 14:51:42,461::DEBUG::spectre.analysis.analysis>  total cnv candidates in chr22: 18 before merge
spectre::2024-04-10 14:51:42,461::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:42,462::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:42,464::DEBUG::spectre.analysis.analysis>  Total merge rounds: 2
spectre::2024-04-10 14:51:42,464::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chr22
spectre::2024-04-10 14:51:42,464::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:42,465::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:42,588::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:42,588::DEBUG::spectre.analysis.analysis>  Clean by threshold: 14 -> 11. Cleaned: 3
spectre::2024-04-10 14:51:42,588::DEBUG::spectre.analysis.analysis>  Clean by size: 11 -> 0. Cleaned: 11
spectre::2024-04-10 14:51:42,588::DEBUG::spectre.analysis.analysis>  Clean by size: 14 -> 0. Cleaned: 14
spectre::2024-04-10 14:51:42,591::DEBUG::spectre.analysis.analysis>  Writing coverage to /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/cnv_Sample2_byCoverage_chrchrX.csv
spectre::2024-04-10 14:51:43,113::DEBUG::spectre.analysis.analysis>  total cnv candidates in chrX: 31 before merge
spectre::2024-04-10 14:51:43,113::DEBUG::spectre.analysis.analysis>  Current merge cycle 1
spectre::2024-04-10 14:51:43,116::DEBUG::spectre.analysis.analysis>  Current merge cycle 2
spectre::2024-04-10 14:51:43,118::DEBUG::spectre.analysis.analysis>  Total merge rounds: 2
spectre::2024-04-10 14:51:43,118::DEBUG::spectre.analysis.analysis>  Trying to merge over blacklisted regions for chrX
spectre::2024-04-10 14:51:43,118::DEBUG::spectre.analysis.analysis>  Current blacklist merge cycle 1
spectre::2024-04-10 14:51:43,120::DEBUG::spectre.analysis.analysis>  Total merge rounds over blacklisted region: 1
spectre::2024-04-10 14:51:43,363::DEBUG::spectre.analysis.cnv_metrics>  CNV-Metrics: Calculated thresholds for DEL: 1.153806708360109 and DUP: 2.87418633303406
spectre::2024-04-10 14:51:43,364::DEBUG::spectre.analysis.analysis>  Clean by threshold: 27 -> 25. Cleaned: 2
spectre::2024-04-10 14:51:43,364::DEBUG::spectre.analysis.analysis>  Clean by size: 25 -> 0. Cleaned: 25
spectre::2024-04-10 14:51:43,364::DEBUG::spectre.analysis.analysis>  Clean by size: 27 -> 0. Cleaned: 27
spectre::2024-04-10 14:51:43,364::INFO::spectre.spectreCNV>  Calculate CNV metrics
spectre::2024-04-10 14:51:43,364::DEBUG::spectre.analysis.cnv_metrics>  Evaluating all CNVs
spectre::2024-04-10 14:51:43,364::DEBUG::spectre.analysis.cnv_metrics>  Recalculating
spectre::2024-04-10 14:51:43,364::DEBUG::spectre.analysis.cnv_metrics>  Converting cnv calls to exclusion zone
spectre::2024-04-10 14:51:43,364::DEBUG::spectre.analysis.cnv_metrics>  Merging exclusion zones
spectre::2024-04-10 14:51:57,744::DEBUG::spectre.analysis.cnv_metrics>  Getting exclusion zone indices in mosdepth coverage data
spectre::2024-04-10 14:52:01,500::DEBUG::spectre.analysis.cnv_metrics>  Generating random coverage sample indices
spectre::2024-04-10 14:52:01,749::INFO::spectre.analysis.cnv_metrics>  DEL border:1.4809815950920244     DUP border:2.510429447852761
spectre::2024-04-10 14:52:01,750::DEBUG::spectre.analysis.cnv_metrics>  Creating CNV distribution plot
spectre::2024-04-10 14:52:01,825::DEBUG::spectre.analysis.cnv_metrics>  Plot path: /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2/debug/genome_coverage_distribution-Sample2.png
spectre::2024-04-10 14:52:01,888::INFO::spectre.spectreCNV>  Merge CNV and LoH candidates
spectre::2024-04-10 14:52:01,888::DEBUG::spectre.classes.loh_candidate>  CNV cand: 0
spectre::2024-04-10 14:52:01,888::INFO::spectre.spectreCNV>  Final candidates are written to .spc file
spectre::2024-04-10 14:52:02,024::INFO::spectre.spectreCNV>  Results are writen to bed file
spectre::2024-04-10 14:52:02,047::INFO::spectre.spectreCNV>  Results are writen to VCF file
spectre::2024-04-10 14:52:02,047::INFO::spectre.spectreCNV>  Result plot in progress
spectre::2024-04-10 14:52:02,076::INFO::spectre.plots.plot>  CNVs in chromosome: chr1
spectre::2024-04-10 14:52:02,940::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr1.png
spectre::2024-04-10 14:52:02,997::INFO::spectre.plots.plot>  CNVs in chromosome: chr2
spectre::2024-04-10 14:52:04,138::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr2.png
spectre::2024-04-10 14:52:04,195::INFO::spectre.plots.plot>  CNVs in chromosome: chr3
spectre::2024-04-10 14:52:05,332::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr3.png
spectre::2024-04-10 14:52:05,461::INFO::spectre.plots.plot>  CNVs in chromosome: chr4
spectre::2024-04-10 14:52:06,519::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr4.png
spectre::2024-04-10 14:52:06,577::INFO::spectre.plots.plot>  CNVs in chromosome: chr5
spectre::2024-04-10 14:52:07,554::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr5.png
spectre::2024-04-10 14:52:07,610::INFO::spectre.plots.plot>  CNVs in chromosome: chr6
spectre::2024-04-10 14:52:08,544::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr6.png
spectre::2024-04-10 14:52:08,600::INFO::spectre.plots.plot>  CNVs in chromosome: chr7
spectre::2024-04-10 14:52:09,472::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr7.png
spectre::2024-04-10 14:52:09,526::INFO::spectre.plots.plot>  CNVs in chromosome: chr8
spectre::2024-04-10 14:52:10,351::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr8.png
spectre::2024-04-10 14:52:10,404::INFO::spectre.plots.plot>  CNVs in chromosome: chr9
spectre::2024-04-10 14:52:11,098::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr9.png
spectre::2024-04-10 14:52:11,151::INFO::spectre.plots.plot>  CNVs in chromosome: chr10
spectre::2024-04-10 14:52:11,965::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr10.png
spectre::2024-04-10 14:52:12,018::INFO::spectre.plots.plot>  CNVs in chromosome: chr11
spectre::2024-04-10 14:52:12,813::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr11.png
spectre::2024-04-10 14:52:12,867::INFO::spectre.plots.plot>  CNVs in chromosome: chr12
spectre::2024-04-10 14:52:13,660::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr12.png
spectre::2024-04-10 14:52:13,728::INFO::spectre.plots.plot>  CNVs in chromosome: chr13
spectre::2024-04-10 14:52:14,384::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr13.png
spectre::2024-04-10 14:52:14,453::INFO::spectre.plots.plot>  CNVs in chromosome: chr14
spectre::2024-04-10 14:52:15,096::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr14.png
spectre::2024-04-10 14:52:15,168::INFO::spectre.plots.plot>  CNVs in chromosome: chr15
spectre::2024-04-10 14:52:15,767::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr15.png
spectre::2024-04-10 14:52:15,819::INFO::spectre.plots.plot>  CNVs in chromosome: chr16
spectre::2024-04-10 14:52:16,351::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr16.png
spectre::2024-04-10 14:52:16,403::INFO::spectre.plots.plot>  CNVs in chromosome: chr17
spectre::2024-04-10 14:52:16,950::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr17.png
spectre::2024-04-10 14:52:17,002::INFO::spectre.plots.plot>  CNVs in chromosome: chr18
spectre::2024-04-10 14:52:17,604::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr18.png
/usr/local/lib/python3.11/site-packages/spectre/plots/plot.py:39: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
  self.figure = plot_engine.figure(figsize=(8, 4))
spectre::2024-04-10 14:52:17,657::INFO::spectre.plots.plot>  CNVs in chromosome: chr19
spectre::2024-04-10 14:52:18,150::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr19.png
spectre::2024-04-10 14:52:18,202::INFO::spectre.plots.plot>  CNVs in chromosome: chr20
spectre::2024-04-10 14:52:18,720::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr20.png
spectre::2024-04-10 14:52:18,784::INFO::spectre.plots.plot>  CNVs in chromosome: chr21
spectre::2024-04-10 14:52:19,119::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr21.png
spectre::2024-04-10 14:52:19,185::INFO::spectre.plots.plot>  CNVs in chromosome: chr22
spectre::2024-04-10 14:52:19,550::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chr22.png
spectre::2024-04-10 14:52:19,609::INFO::spectre.plots.plot>  CNVs in chromosome: chrX
spectre::2024-04-10 14:52:20,434::INFO::spectre.plots.plot>  Plot saved: img/Sample2_plot_cnv_chrX.png
spectre::2024-04-10 14:52:20,465::INFO::spectre.spectreCNV>  Output dir: /srv/scratch/testardqu/tickets/28452/Spectre/results/Sample2
spectre::2024-04-10 14:52:20,465::INFO::spectre.spectreCNV>  Done with sample Sample2
spectre::2024-04-10 14:52:20,465::INFO::spectre.spectre>  Spectre finished

Thanks.

Hi @Tintest,
The log looks normal to me.
Can could you be so kind and check the BED file for me if it contains any calls, since it is directly linked to the VCF output?

The Sample2_cnv.bed.gz is empty. Do you need the debug bed files ?

I just noticed that every sample with an empty VCF file instead of an VCF.GZ file are females. All the other VCF.GZ files are males except for two samples. One is non-empty with a CNV on chr15 and the other has numerous deletions on the X chromosome, in the same way as in #16. This sample may have a particular sexual phenotype that I'll have to investigate.
This "bug" would therefore seem to occur when no CNV is detected for the sample in question, and therefore when the VCF is empty.

Hi @Tintest my apologies for the delay, I am currently slammed with tasks.
I have pushed some changes 0e96275, that will address your issues. I also added instructions in the README.md on how to install the latest Spectre version using a local installation with pip.
I have reviewed your command again and found that you were running Spectre in the single sample mode. Our intention was only to provide multiple samples when used with the population mode. The main idea was to give the user the option to bundle multiple similar samples together, thus all the results from the samples are located in the one output directory. As a little add-on we offer to run the standalone population mode on the batch on samples when the --population flag has been provided.
Please keep in mind, all flags that have been set are applied to all provided samples. This could become an issue when running male or female samples on, e.g. chrX.
If you want to run Spectre on multiple samples with different properties (e.g. male or female), please run Spectre in single sample mode by providing only one Mosdepth coverage file at each time. This will give you the flexibility to adjust the setting according to the sample. Furthermore,
In the end of each run, Spectre produces an SPC file that you can then use with the population mode in a later step of your workflow.
I hope this will help you.
Feel free to reach out anytime.
Cheers,
Philippe


Modified Cobalt browser to play secondary video on PIP window.

This pull request needs to be tested on Broadcom platform.

This code is platform independent. Let's leave it like this

Hi Pawel :-)
There is a dual video test in YTLR.
On Broadcom, second video works only in PIP window. Where this change can be done?

It has to be abstracted out and done in brcm directory.


Drop genesis block if not block-chain-id

Enforce the value of the block chain id (found in the data dir) by
dropping a received genesis block if it does not match.
Signed-off-by: Peter Schwarz<EMAIL_ADDRESS>

I don't get the three cases in _set_genesis in this PR. Case 1, if the validator doesn't have the block-chain-id in it's data directory it creates it. Case 2, it does have it but it isn't the same as the genesis block it received. Case 3, It does have it and it is equal to the genesis block it received, so it validates the block. For the Poet Smoke test and any situation where a validator is joining a network where it doesn't already know about the genesis block it would have to receive the genesis block twice for that block to get into the block store.

Updated to address this comment:

drop block if there is a block id mismatch
verify block:
a. If valid, store block id
b. if not, drop block


Add Unicesumar to domains

Add the domain: cesumar.br
Unicesumar - Centro Universitário Cesumar

@fxcoudert If you wish for a school to be added to the repository, please fork the repository, add the school and create a pull request.
Please see the contribution guidelines for further information


Unexpected Error

Sometimes prlint has an error even if there is reviewer on the pull request. It gets better when I add someone new

We have also been facing this issue. Anyone maintaining this project anymore?

It's down, I don't know if anyone is actually trying to fix it.
https://stats.uptimerobot.com/ZzYnEf2BW

@manpreetsgarha @maor-rosenfeld
Is there any alternative tool ?

Seems like support of the tool has gone.
@ewolfe Can you comment this ?

There's Semantic PR but it only allows you to define one regex.
I guess I can develop a PRLint compatible app without too much effort... would prefer if this one will just work :)

We are checking pull request title,assignees,labels,requested_reviewers
Semantic PR only checks title :(

I created an alternative implementation, can you please help me test it? it's by no means production ready, I simply took the code in this project and wrapped it in a serverless deployment.
https://github.com/apps/prlintreloaded


Cannot handle objects without a properties method

I'm trying to find the shortest path with:
MATCH (a:Node {}), (b:Node {}), p = shortestPath((a)-[*]-(b)) RETURN p

but I'm getting error:
Cannot handle objects without a properties method. Class given: Bolt\structures\Path

Are these kind of operations supported?

I found out that it works over Http Connection but not over Bolt Connection.

I found out that it works over Http Connection but not over Bolt Connection.

Thank you, I have figured out the problem and am working on a solution for bolt. It shouldn't take too long to fix this.

Thank you, I have figured out the problem and am working on a solution for bolt. It shouldn't take too long to fix this.

Hello va5ja,
Thank you for being patient; I was quite busy. The relationship types now properly get translated. Please update to version 1.1.2 if you wish to use it.
Have a good day,
Ghlen

Hello va5ja,
Thank you for being patient; I was quite busy. The relationship types now properly get translated. Please update to version 1.1.2 if you wish to use it.
Have a good day,
Ghlen

That's great, thank you!


Unpaid Orders still go through

This plugin seems to have a flaw to it or I am doing something wrong. The problem is that once a shopper places an order (hits the confirm button) the shopper is presented with the bitpay screen and offered a chance to make payment. If he or she decides they do not want to continue with the purchase, perhaps they don't have enough bitcoin and didn't realize it, there is no way to cancel the order and if they just close out the screen the order is still pending in zen cart. The module should not be generating an order until payment has been made or have some means to cancel the order and clear the cart. The bitpay api seems to have enough flexibility to accomplish this as in checking the status of an invoice.

Thanks for reporting this issue, @WilsonMF.  Are you still having this issue?

This is a known issue in the plugin that will be fixed in version 2.0.

Roger that!  Closing this issue then.


graphql tester not autoconfigured

I imagine what I really have here is a missing dependency, but the reference documentation suggests I only need spring-graphql-test, although I've tried a fair number of dependency permutations based on the GS guide and it still isn't finding the bean. I've manually tested standing a server up and it runs the controller. I feel like this is a red herring error, like there's a reason this bean isn't available that's not erroring obviously.
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.xenoterracide.controller.registration.RegistrationCtrlrTest': Unsatisfied dependency expressed through field 'graphQlTester': No qualifying bean of type 'org.springframework.graphql.test.tester.GraphQlTester' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}

  api(platform(libs.spring.bom))
  api(libs.spring.context)
  api(libs.jakarta.validation)

  implementation(platform(libs.spring.bom))
  implementation(libs.spring.graphql.core)
  implementation(projects.modelSecurity)

  runtimeOnly(platform(libs.spring.bom))
  runtimeOnly(libs.starter.graphql)
  runtimeOnly(libs.starter.validation)
  runtimeOnly(libs.starter.web)
  runtimeOnly(libs.starter.log4j2)

  testImplementation(platform(libs.spring.bom))
  testImplementation(libs.spring.boot.test.autoconfigure)
  testImplementation(libs.spring.boot.test.core)
  testImplementation(libs.spring.graphql.test)

  testRuntimeOnly(platform(libs.spring.bom))
  testRuntimeOnly(projects.testApp)
  testRuntimeOnly(libs.h2)
  testRuntimeOnly(libs.mockito)
  testRuntimeOnly(libs.starter.webflux)
  testRuntimeOnly(libs.starter.test)

// © Copyright 2024 Caleb Cushing
// SPDX-License-Identifier: AGPL-3.0-or-later

package com.xenoterracide.controller.registration;

import static org.assertj.core.api.Assertions.assertThat;

import com.xenoterracide.model.security.UserRepository;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.graphql.GraphQlTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.graphql.test.tester.GraphQlTester;

@GraphQlTest( RegistrationCtrlr.class)
class RegistrationCtrlrTest {

  @MockBean
  UserRepository userRepository;

  @Autowired
  GraphQlTester graphQlTester;

  @Test
  void register() {
    assertThat(true).isTrue();
  }
}

spring-bom = { module = "org.springframework.boot:spring-boot-dependencies", version.ref = "spring" }
spring-boot-actuator = { module = "org.springframework.boot:spring-boot-actuator" }
spring-boot-autoconfigure = { module = "org.springframework.boot:spring-boot-autoconfigure" }
spring-boot-configuration-processor = { module = "org.springframework.boot:spring-boot-configuration-processor" }
spring-boot-core = { module = "org.springframework.boot:spring-boot" }
spring-boot-test-core = { module = "org.springframework.boot:spring-boot-test" }
spring-boot-test-autoconfigure = { module = "org.springframework.boot:spring-boot-test-autoconfigure" }
spring-boot-devtools = { module = "org.springframework.boot:spring-boot-devtools" }
spring-beans = { module = "org.springframework:spring-beans" }
spring-context = { module = "org.springframework:spring-context" }
spring-core = { module = "org.springframework:spring-core" }
spring-data-jpa = { module = "org.springframework.data:spring-data-jpa" }
spring-data-envers = { module = "org.springframework.data:spring-data-envers" }
spring-graphql-core = { module = "org.springframework.graphql:spring-graphql" }
spring-graphql-test = { module = "org.springframework.graphql:spring-graphql-test" }
spring-instrument = { module = "org.springframework:spring-instrument" }
spring-orm = { module = "org.springframework:spring-orm" }
spring-test = { module = "org.springframework:spring-test" }
spring-transaction = { module = "org.springframework:spring-tx" }
spring-security-config = { module = "org.springframework.security:spring-security-config" }
spring-security-core = { module = "org.springframework.security:spring-security-core" }
spring-security-oauth2-authorization-server = { module = "org.springframework.security:spring-security-oauth2-authorization-server" }
spring-security-oauth2-core = { module = "org.springframework.security:spring-security-oauth2-core" }
spring-security-web = { module = "org.springframework.security:spring-security-web" }
spring-data-commons = { module = "org.springframework.data:spring-data-commons" }
spring-data-neo4j = { module = "org.springframework.data:spring-data-neo4j" }
spring-web = { module = "org.springframework:spring-web" }
spring-webflux = { module = "org.springframework:spring-webflux" }
spring-webmvc = { module = "org.springframework:spring-webmvc" }
starter-actuator = { module = "org.springframework.boot:spring-boot-starter-actuator" }
starter-aop = { module = "org.springframework.boot:spring-boot-starter-aop" }
starter-core = { module = "org.springframework.boot:spring-boot-starter" }
starter-data-jpa = { module = "org.springframework.boot:spring-boot-starter-data-jpa" }
starter-graphql = { module = "org.springframework.boot:spring-boot-starter-graphql" }
starter-json = { module = "org.springframework.boot:spring-boot-starter-json" }
starter-log4j2 = { module = "org.springframework.boot:spring-boot-starter-log4j2" }
starter-neo4j = { module = "org.springframework.boot:spring-boot-starter-data-neo4j" }
starter-oauth2-authorization-server = { module = "org.springframework.boot:spring-boot-starter-oauth2-authorization-server" }
starter-oauth2-client = { module = "org.springframework.boot:spring-boot-starter-oauth2-client" }
starter-oauth2-resource-server = { module = "org.springframework.boot:spring-boot-starter-oauth2-resource-server" }
starter-security = { module = "org.springframework.boot:spring-boot-starter-security" }
starter-validation = { module = "org.springframework.boot:spring-boot-starter-validation" }
starter-web = { module = "org.springframework.boot:spring-boot-starter-web" }
starter-webflux = { module = "org.springframework.boot:spring-boot-starter-webflux" }
spring-test = ["spring-test", "spring-boot-test-core", "spring-boot-test-autoconfigure"]
spring-boot = { id = "org.springframework.boot", version.ref = "spring" }

I initially thought this was because the webflux dependency was missing from tests, but removing it from my sample works.
This works for me with the following dependencies:
dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-graphql'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testImplementation 'org.springframework.graphql:spring-graphql-test'
    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}

You'll get the correct setup if you create a project on start.spring.io: https://start.spring.io/#!type=gradle-project&language=java&platformVersion=3.3.0&packaging=jar&jvmVersion=17&groupId=com.example&artifactId=graphqltester&name=graphqltester&description=Demo project for Spring Boot&packageName=com.example.graphqltester&dependencies=graphql,web
I guess this is linked with your build setup or your dependency scopes? It's hard to tell without a minimal sample to look at. Or maybe your sample doesn't have a graphql schema file?
You can run your test from the IDE and add the DEBUG=true environment variable. This should show you the auto-configuration report in the console; look for the GraphQlTesterAutoConfiguration and why the condition doesn't match.

urgh... I just realized my schema was in my demoServer "feature/sourceset". Maybe it's a ludicrous ask because spring MVC doesn't do that... but then it doesn't need a json schema to operate. would it be possible to log something like this even if there's no schema? I had trouble figuring out the missing schema thing even before because from the feature to output the schema I thought maybe it could output schema from java.
 INFO 1131748 - pri.grap.exec.DefaultSchemaResourceGraphQlSourceBuilder : Loaded 1 resource(s) in the GraphQL schema.                                                                             	at org.springframework.graphql.execution.DefaultSchemaResourceGraphQlSourceBuilder.initGraphQlSchema(DefaultSchemaResourceGraphQlSourceBuilder.java:147)
DEBUG 1131748 - pri.grap.exec.DefaultSchemaResourceGraphQlSourceBuilder : Loaded GraphQL schema resources: (class path resource [graphql/registration.graphqls])
INFO 1131748 - pri.grap.exec.DefaultSchemaResourceGraphQlSourceBuilder : GraphQL schema inspection:
	Unmapped fields: {Query=[_dummy]}
	Unmapped registrations: {}
	Unmapped arguments: {}
	Skipped types: []

If I remember correctly, we didn't consider logging a warning in the past because:

applications can use the spring-graphql dependency for the client only, not needing a schema at all
applications can contribute a schema programmatically

We'll discuss this as a team and reconsider the question. The auto-configuration now lives in Spring Boot so we would have to create a new issue there if we think it's the right call.

Being that I'm not sure how it works... Should the tester fail to inject? Like that feels extra weird for the test to fail to start up in this way... One thing where the tester to start up and then do a query and have that fail but to have the test fail to start up...
The graphql server on the other hand is silent in its failures.
This is just an additional thought for that discussion. Maybe the test shouldn't fail to start or it should have an assertion failure of some kind.
I think I would want a warning that now schema is present for the server and that this Bean should be injected regardless. The test should basically start and continue on to an actual test which could then fail with the graphql endpoint or schema not being found.

@xenoterracide with @GraphQlTest executing directly against an ExecutionGraphQlService (i.e. without no transport or running server), there is no good way to create an ExecutionGraphQlServiceTester, which is what we have in this case, if there is no ExecutionGraphQlService to begin with, and the Boot GraphQL starter only creates the ExecutionGraphQlService bean if there is a schema. Or else it's possibly not a GraphQL server to begin with.
I understand that an autowiring failure is a rather indirect outcome, but it's something that would happen once only when you're first setting up your application. For that the DefaultGraphQlSchemaCondition does log a debug message that no schema files were found.
All in all there isn't much further we can do here, except consider a slightly more specific message that indicates no schema files are found and therefore our assumption is, it is not a server application. We'll consider a WARN level for higher visibility.

I'm closing this issue in favor of spring-projects/spring-boot#41320. We'll see there if we can log something at the appropriate time in the console.

I mean I found the whole whole not setting up the server a little weird if no schema files are found. Kind of feels like a do it anyway... Maybe something should always be there

This is not to say that client only scenarios don't exist. It's only a question of how to make the determination.

So is there no way for spring boot to report in its configuration report that it didn't find any resources matching pattern?


Access userbase through this

Is it possible to add access to the userbase global through this? Right now, the only way to access userbase is through context, which is not accessible in methods.
An example of a use case would be calling this.$userbase.signUp in a method to register accounts:
<template>

Create an account

    <form id="signup-form" @submit.prevent="createAccount">
      <input
        id="signup-username"
        v-model="username"
        type="text"
        required
        placeholder="Username"
      />
      <input
        id="signup-password"
        v-model="password"
        type="password"
        required
        placeholder="Password"
      />
      <input type="submit" value="Create an account" />
    </form>

</template>

Thanks!

@mujahidfa Hello, thank you for the issue!
I add this functionality in 0.4.0, just upgrade it.
But you can always use context from asyncData, or anonymous middleware in pages, and pass data to components via props.
https://nuxtjs.org/api
https://nuxtjs.org/guides/components-glossary/pages-middleware#anonymous-middleware


Change email_alert_api to use new adapter method

Trello: https://trello.com/c/9FmE3rPn/281-double-opt-in-create-api-adapter-to-send-double-opt-in-email

Should await merge and publication of this branch on gds-api-adaptors.
After which this PR should increment the version of that gem.

A part of the refactor work to improve double opt in across GOV.uk

Superseded by https://github.com/alphagov/email-alert-frontend/pull/631


Broken Link to PubMed Abstracts dataset .

Describe the bug
The link provided for the dataset is broken,
data_files =
https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst
The
Steps to reproduce the bug
Steps to reproduce:

Head over to https://huggingface.co/learn/nlp-course/chapter5/4?fw=pt#big-data-datasets-to-the-rescue

In the Section  "What is the Pile?", you can see a code snippet that contains the broken link.

Expected behavior
The link should Redirect to the  "PubMed Abstracts dataset" as expected .
Environment info
.

This has already been reported in the HF Course repo (https://github.com/huggingface/course/issues/623).

@lhoestq @albertvillanova @lewtun I don't think we are allowed to host these data files on the Hub (due to DMCA), which means the only option is to use a different dataset in the course (and to re-record the video 🙂)

Keeping the video is maybe fine, we can add a note on youtube to suggest to load a dataset with a different name. Maybe C4 ?

Maybe you want to try it with the PUBMED dataset that I created based on the The [PubMed Abstract GitHub (https://github.com/thoppe/The-Pile-PubMed)](http://xn--(https:-2j53ao22b9r0a//github.com/thoppe/The-Pile-PubMed) and uploaded on the HuggingFace repository:
from datasets import load_dataset
pubmed_dataset = load_dataset("qualis2006/PUBMED_title_abstracts_2020_baseline")
pubmed_dataset
Downloading data: 100%
7.98G/7.98G [11:47<00:00, 9.68MB/s]
Generating train split: 17722096/0 [00:36<00:00, 505376.37 examples/s]
DatasetDict({
train: Dataset({
features: ['meta', 'text'],
num_rows: 17722096
})
})

孔令涛说感谢感谢


Cannot find module 'broccoli-funnel' and 'broccoli-merge-trees'

I recently upgraded from 0.1.x to the latest stable version and I could not run my ember project because of these missing npm modules. I noticed that they are there in this project's devDependencies, should they be in dependencies instead? index.js requires these modules.

No, I don't think these particular packages should need to be in dependencies. After upgrading, have you tried the following steps?
npm cache clean
bower cache clean
rm -rf node_modules bower_components dist tmp
npm install
bower install

I tried this and it didn't install these two dev dependencies. After running npm install, I checked the /node_modules/fixtable-ember folder and its own node_modules folder and the only thing that was installed is font-awesome. My local NODE_ENV is set to development and all other packages install correctly.

What are your node and npm versions? Are you consuming fixtable-ember inside of another addon, or from a regular Ember app?

I'm on Node 4.4.5 and NPM 2.15.5. I'm consuming fixtable-ember from inside a regular Ember app. If it helps, I've also tried removing fixtable-ember from my bower.json and package.json, removing the folder from my node_modules, and running ember install fixtable-ember from the project root; however, before this command succeeds, it returns with the error "cannot find modules broccoli-funnel and broccoli-merge-trees".

Gotcha. Unfortunately, I believe that NPM 3+ is required for these dependencies to work correctly. You could try using a version prior to 1.0.2 -- I believe version 1.0.0 does not include those dependencies. But that shuts off the upgrade path for you until you upgrade NPM.

Thanks for your help! Other than a few deprecation warnings, 1.0.0 works well for our use.


⚠ Samsung has degraded performance

In f8f284a, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2552 ms

Resolved: Samsung performance has improved in 44bf6d6.


mio: Retry poll if interrupted

Apparently on Linux, epoll_wait can return EINTR if a ptracer
attaches. The SIGSTOP is suppressed, but the call is still interrupted.
This retries the call to make it easier to do things like observe what's
going on with strace.

ugh EINTR


Raise error | IndexError: list index out of range

I was just walking a mount point. Nothing special.
  File "diskover.py", line 2268, in <module>
    post_crawl_tasks()
  File "diskover.py", line 2030, in post_crawl_tasks
    upload_stats()
  File "diskover.py", line 1995, in upload_stats
    size = es.search(index=index, doc_type='directory', body=body)['hits']['hits'][0]['_source']['filesize']
IndexError: list index out of range

is this still an issue?

Yes

What is the full cli command you are using to run diskover? Is the index getting created in elasticsearch and is there data getting added? Can you view the index in Kibana or diskover-web okay?

I reinstall everything with a new python3.8 install and work great!

Bad news. Like the monthy python would say about a rabbot it did it again
Here is more information:
python /srv/dockers/web/diskover/diskover.py -d /mnt/GFS/PRODS -i diskover-2020-09-18 -a -O -A

   ___       ___       ___       ___       ___       ___       ___       ___
  /\  \     /\  \     /\  \     /\__\     /\  \     /\__\     /\  \     /\  \
 /::\  \   _\:\  \   /::\  \   /:/ _/_   /::\  \   /:/ _/_   /::\  \   /::\  \
/:/\:\__\ /\/::\__\ /\:\:\__\ /::-"\__\ /:/\:\__\ |::L/\__\ /::\:\__\ /::\:\__\
\:\/:/  / \::/\/__/ \:\:\/__/ \;:;-",-" \:\/:/  / |::::/  / \:\:\/  / \;:::/  /
 \::/  /   \:\__\    \::/  /   |:|  |    \::/  /   L;;/__/   \:\/  /   |:\/__/
  \/__/     \/__/     \/__/     \|__|     \/__/               \/__/     \|__|
                                      v<IP_ADDRESS>
                                      https://shirosaidev.github.io/diskover
                                      Bringing light to the darkness.
                                      Support diskover on Patreon or PayPal :)

2020-09-18 07:52:24,691 [INFO][diskover] Using config file: /srv/dockers/web/diskover/diskover.cfg
2020-09-18 07:52:24,694 [INFO][diskover] Checking es index: diskover-2020-09-18
Drop existing index? [y/n]: y
2020-09-18 07:52:31,265 [WARNING][diskover] es index exists, deleting
2020-09-18 07:52:31,390 [INFO][diskover] Creating es index
2020-09-18 07:52:32,241 [INFO][diskover] Tuning ES index settings for crawl
2020-09-18 07:52:32,315 [INFO][diskover] Adding disk space info to es index
2020-09-18 07:52:32,382 [INFO][diskover] Found 16 diskover RQ worker bots
2020-09-18 07:52:32,382 [INFO][diskover] Enqueueing crawl to diskover worker bots for /mnt/GFS/PRODS...
2020-09-18 07:52:32,382 [INFO][diskover] Worker bots set to auto-tag (-A)
2020-09-18 07:52:32,382 [INFO][diskover] Sending adaptive batches to worker bots (-a)
2020-09-18 07:52:32,382 [INFO][diskover] Starting crawl using 80 treewalk threads (maxdepth None)
| Crawling (Queue: 0, 1144.284 dirs/sec) Elapsed Time: 0:00:15
| |           #                                                                                | 0 Elapsed Time: 0:04:48
2020-09-18 07:57:36,749 [INFO][diskover] Finished crawling in 0d:0h:05m:04s, dirs walked 16372 (53.791 dirs/sec)
2020-09-18 07:57:36,791 [INFO][diskover] Waiting for diskover worker bots to be done with any jobs in rq...
2020-09-18 07:57:36,819 [INFO][diskover] Getting diskover bots to calculate directory sizes (maxdepth None)...
/ Calculating (Queue: 0) Elapsed Time: 0:00:00                                                                                2020-09-18 07:57:36,821 [INFO][diskover] Searching for all directory docs in diskover-2020-09-18...
2020-09-18 07:57:37,248 [INFO][diskover] Found 650 directory docs
| Calculating (Queue: 0) Elapsed Time: 0:00:00
| |          #                                                                                | 0 Elapsed Time: 0:00:01
2020-09-18 07:57:38,294 [INFO][diskover] Finished calculating 650 directory sizes in 0d:0h:00m:01s
2020-09-18 07:57:38,344 [INFO][diskover] Setting ES index settings back to defaults
2020-09-18 07:57:38,428 [INFO][diskover] Force merging ES index...
2020-09-18 07:57:39,269 [INFO][diskover] Optimizing ES index... this could take a while... (-O)
Traceback (most recent call last):
  File "/srv/dockers/web/diskover/diskover.py", line 2267, in <module>
    post_crawl_tasks()
  File "/srv/dockers/web/diskover/diskover.py", line 2029, in post_crawl_tasks
    upload_stats()
  File "/srv/dockers/web/diskover/diskover.py", line 1994, in upload_stats
    size = es.search(index=index, doc_type='directory', body=body)['hits']['hits'][0]['_source']['filesize']
IndexError: list index out of range

The index is create in ES. And visible in the web interface but incomplete.

i had time to run in debug finaly:
2020-09-26 16:36:41,071 [INFO][diskover] batchsize set to: 50
2020-09-26 16:36:41,082 [INFO][diskover] enqueued batchsize: 50 (batchsize: 50)
2020-09-26 16:36:41,083 [INFO][diskover] batchsize set to: 50
2020-09-26 16:36:41,094 [INFO][diskover] enqueued batchsize: 50 (batchsize: 50)
2020-09-26 16:36:41,094 [INFO][diskover] batchsize set to: 50
2020-09-26 16:36:41,105 [INFO][diskover] enqueued batchsize: 50 (batchsize: 50)
2020-09-26 16:36:41,105 [INFO][diskover] batchsize set to: 50
2020-09-26 16:36:41,116 [INFO][diskover] enqueued batchsize: 50 (batchsize: 50)
2020-09-26 16:36:41,117 [INFO][diskover] batchsize set to: 50
2020-09-26 16:36:41,120 [DEBUG][urllib3.util.retry] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2020-09-26 16:36:41,122 [DEBUG][urllib3.connectionpool] http://localhost:9200 "GET /_search/scroll?scroll=1m HTTP/1.1" 200 240
2020-09-26 16:36:41,123 [INFO][elasticsearch] GET http://localhost:9200/_search/scroll?scroll=1m [status:200 request:0.003s]
2020-09-26 16:36:41,123 [DEBUG][elasticsearch] > {"scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAACAMWdW1MNVptOGtTMVd4YzNNVWxISUhQUQ=="}
2020-09-26 16:36:41,123 [DEBUG][elasticsearch] < {"_scroll_id":"DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAACAMWdW1MNVptOGtTMVd4YzNNVWxISUhQUQ==","took":1,"timed_out":false,"terminated_early":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":865,"max_score":1.0,"hits":[]}}
2020-09-26 16:36:41,126 [INFO][diskover] Found 865 directory docs
2020-09-26 16:36:42,163 [INFO][diskover] Finished calculating 865 directory sizes in 0d:0h:00m:01s
2020-09-26 16:36:42,163 [DEBUG][urllib3.util.retry] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2020-09-26 16:36:42,239 [DEBUG][urllib3.connectionpool] http://localhost:9200 "POST /diskover-20200926-gfs/crawlstat HTTP/1.1" 201 178
2020-09-26 16:36:42,239 [INFO][elasticsearch] POST http://localhost:9200/diskover-20200926-gfs/crawlstat [status:201 request:0.076s]
2020-09-26 16:36:42,240 [DEBUG][elasticsearch] > {"path": "/mnt/GFS/PRODS", "state": "finished_dircalc", "crawl_time": 54.814962, "indexing_date": "2020-09-26T14:36:42.163338"}
2020-09-26 16:36:42,240 [DEBUG][elasticsearch] < {"_index":"diskover-20200926-gfs","_type":"crawlstat","_id":"AXTK2Dk1atzIJIIT4fRd","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"created":true}
2020-09-26 16:36:42,240 [INFO][diskover] Setting ES index settings back to defaults
2020-09-26 16:36:42,240 [DEBUG][urllib3.util.retry] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2020-09-26 16:36:42,311 [DEBUG][urllib3.connectionpool] http://localhost:9200 "PUT /diskover-20200926-gfs/_settings HTTP/1.1" 200 21
2020-09-26 16:36:42,311 [INFO][elasticsearch] PUT http://localhost:9200/diskover-20200926-gfs/_settings [status:200 request:0.071s]
2020-09-26 16:36:42,312 [DEBUG][elasticsearch] > {"index": {"refresh_interval": "1s", "number_of_replicas": 0, "translog.flush_threshold_size": "512mb"}}
2020-09-26 16:36:42,312 [DEBUG][elasticsearch] < {"acknowledged":true}
2020-09-26 16:36:42,312 [INFO][diskover] Force merging ES index...
2020-09-26 16:36:42,312 [DEBUG][urllib3.util.retry] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2020-09-26 16:36:43,162 [DEBUG][urllib3.connectionpool] http://localhost:9200 "POST /diskover-20200926-gfs/_forcemerge HTTP/1.1" 200 49
2020-09-26 16:36:43,163 [INFO][diskover] Optimizing ES index... this could take a while... (-O)
2020-09-26 16:36:43,163 [DEBUG][urllib3.util.retry] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2020-09-26 16:36:45,532 [DEBUG][urllib3.connectionpool] http://localhost:9200 "POST /diskover-20200926-gfs/_forcemerge?max_num_segments=1 HTTP/1.1" 200 49
2020-09-26 16:36:45,533 [DEBUG][urllib3.util.retry] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2020-09-26 16:36:45,537 [DEBUG][urllib3.connectionpool] http://localhost:9200 "POST /diskover-20200926-gfs/_refresh HTTP/1.1" 200 49
2020-09-26 16:36:45,537 [DEBUG][urllib3.util.retry] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2020-09-26 16:36:45,541 [DEBUG][urllib3.connectionpool] http://localhost:9200 "GET /diskover-20200926-gfs/directory/_count HTTP/1.1" 200 73
2020-09-26 16:36:45,542 [DEBUG][urllib3.util.retry] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2020-09-26 16:36:45,545 [DEBUG][urllib3.connectionpool] http://localhost:9200 "GET /diskover-20200926-gfs/file/_count HTTP/1.1" 200 75
2020-09-26 16:36:45,545 [DEBUG][urllib3.util.retry] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2020-09-26 16:36:45,549 [DEBUG][urllib3.connectionpool] http://localhost:9200 "GET /diskover-20200926-gfs/directory/_search HTTP/1.1" 200 134
Traceback (most recent call last):
  File "/srv/dockers/web/diskover/diskover.py", line 2267, in <module>
    post_crawl_tasks()
  File "/srv/dockers/web/diskover/diskover.py", line 2029, in post_crawl_tasks
    upload_stats()
  File "/srv/dockers/web/diskover/diskover.py", line 1994, in upload_stats
    size = es.search(index=index, doc_type='directory', body=body)['hits']['hits'][0]['_source']['filesize']
IndexError: list index out of range

It looks like there is an issue getting the rootdir doc size from the es index. Do you see in diskover-web or kibana the correct size for the rootdir doc directory for /mnt/GFS/PRODS directory?

In the "Disk Space Overview" i can see the root dir size.
Is there somewhere where i can see this size to confirm you the information?

Do a search in diskover-web for all directories and see if that directory (the top rootdir one /mnt/GFS/PRODS) is listed and has the correct size, items counts. If you can, please attach screenshot of diskover-web all directory search results.

The top directory  /mnt/GFS/PRODS is not listed

With the same user you run diskover.py and the worker bots, can you run ls -la /mnt/GFS/PRODS. Do you get a directory listing? The user you are using, does it have permission to stat the files/dirs in that directory? Check the worker bots are running python3, same version of python as you are running diskover.py. Take a look here at python version https://github.com/shirosaidev/diskover/wiki/Troubleshooting and other troubleshooting.

 Do you get a directory listing?
Yes
The user you are using, does it have permission to stat the files/dirs in that directory?
Yes it's root user ( i know it's not safe but i'm in testing phase)
Check the worker bots are running python3, same version of python as you are running diskover.py.
I run diskover in virtualenv and start the bot with the sh maybe it's from here.
Can override the python path in the sh. From what i saw it use the current python exe which is my virtualenv python nope?

So i run Diskover and bots in the same specific virtualenv, with the exact same version of python 3.8.5
Same issues.

when you run ls -la /mnt/GFS/PRODS do you see any permission denied lines?

can you please send output of ls -la /mnt/GFS if you can

Off course ! I removed some informations.
ls -la /mnt/GFS/PRODS
total 6
drwxrwxrwx 12 root   root           12 Oct  2 10:58 .
drwxr-xr-x  6 root   root            6 Mar 12  2020 ..
drwxrwxrwx  XXX
drwxrwxrwx  XXX
drwxrwxrwx  XXX
drwxrwxrwx  XXX
drwxrwxr-x  XXX
drwxrwxrwx  XXX
dr-xrwx---  XXX
drwxrwxrwx  XXX
drwxrwxr-x  XXX
drwxrwxrwx  XXX

ls -la /mnt/GFS
total 6
drwxr-xr-x   6 root root    6 Mar 12  2020 .
drwxr-xr-x.  6 root root 4096 Sep 10 19:28 ..
drwxrwxr-x   3 root  root     3 Dec  3  2019 DEMO
drwxrwxrwx  12 root root   12 Oct  2 10:58 PRODS
drwxrwxrwx   2 root root    2 Mar 12  2020 SANDBOX

with adding the additional cli flags I mentioned above, do you see /mnt/GFS/PRODS directory doc in the index now?

I recommend to stop all the bots, run 1 bot in foreground so you can see its log output (or redirect log to disk) and see if there are any warnings/errors when trying to index the top rootdir doc PRODS. Also run diskover.py in debug mode and see if there are any errors/warnings related to that directory doc.

So i made multiple tests in my spare time along this week.
The only case i'm able to reproduce the bug. It's when i'm running more than 3 bots.
And i don't seem to have error in the bot log.
I'm gonna try new test this week end.


add test helper

cc @Fryguy

Looks good to me other than the very minor comment.   :+1:

@Fryguy @simon3z pls have a look on the last commit before I merge.

Looks great! :+1:

I'll wait until this is in before fixing #33


Investigate broken ci

@SeanBryan51, is this ready to merge. Seems like a simple fix...

Im going to go ahead and approve this, but we will need to think about what happens if/when setup tools is deprecated...
Chances are this is just a delay between setuptools and python versions, but you never know.


CITA v0.25.0 version info is incorrect

Description
CITA v0.25.0 version info is incorrect
Found by CITA-Monitor:

using CITA version  v0.25.0 88b90bb

Steps to Reproduce

[First Step]
[Second Step]
[and so on …]

Expected behavior: should be v0.25.0 88b90bb
Actual behavior:  check the screen shot
Reproduce how often: always
Versions
CITA v0.25.0
Additional Information

Yeah. i've tried the released package, it does show the wrong version info.
And when i build the tag v0.25.0 on my own pc, it shows right vrsion.
Maybe the package's problem.
I'll check it.

Check the patch version:
https://github.com/cryptape/cita/releases/tag/v0.25.1


Some packages don't get published on myget

See this trace:
http://aspnetci/viewLog.html?buildId=609574&buildTypeId=AspNetCore_Publish&tab=buildLog&_focus=704

cc @natemcmaster

Duplicate of https://github.com/aspnet/AspNetCore-Internal/issues/1459


Failure to build --requirements.txt when it uses google protobuf

I am running with --requirements_file requirements.txt, which contains:

google-cloud-datastore

Unfortunately, when attempting to run this on the cloud dataflow, I get the following error trying to build the requirements:

Collecting setuptools (from protobuf>=3.0.0->google-cloud-core<0.24dev,>=0.23.1->google-cloud-datastore->-r
requirements.txt (line 3))
  File was already downloaded /var/folders/94/wngs1jw91_n2_jjjrfljtqrc0000gn/T/dataflow-requirements-cache/setuptools-34.3.2.zip

   Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

     File "<string>", line 1, in <module>
      File "setuptools/__init__.py", line 12, in <module>

       import setuptools.version
      File "setuptools/version.py", line 1, in <module>
        import
pkg_resources
      File "pkg_resources/__init__.py", line 70, in <module>
        import packaging.version

   ImportError: No module named packaging.version

Looking online https://github.com/pypa/setuptools/issues/937 , it appears this is due to "pip asking setuptools to build itself (from source dist), which is no longer supported."
I'm not sure what the correct fix is here...since protobuf depends on setuptools, and a lot of Google libraries depend on protobuf. Seems there is no way to list protobuf/setuptools as being "provided" by the beam runtime (ie https://github.com/pypa/pip/issues/3090).
I'm going to try using my own setup.py next and see if I can skirt around the issue, but this definitely seems like a bug with beam's requirements packager asking for too much?
In the case of GCE, I compile my dependencies into a docker image that extends the base GCE images (and lets me use binary installs), not sure something like that would work here?
Imported from Jira BEAM-1790. Original Jira may contain additional context.
Reported by: mlambert.

obsolete for now


oas3-valid-schema-example does not support non-string enums

Vacuum appears to use treat all enum values as strings when validating examples, regardless of their actual type.
Example Config
components:
  schemas:
    NotAuthorized:
      type: integer
      enum: [401]
      example: 401

Expected
No error pertaining to the example, as it is a member of the enum and the specified type
Actual
I get an error for oas3-valid-schema-example with the following message:

Example for component NotAuthorized is not valid: (root) must be one of the following: "401". Value 401 is not compatible

Interesting, I will investigate this - it may be a dependency issue.

github.com/xeipuuv/gojsonschema is throwing this error.
vaccum also uses github.com/santhosh-tekuri/jsonschema, so I will see if that handles the schema correctly.

Resolved in v0.1.0


Azure DevOps git repo via Azure CLI : Command group 'staticwebapp' is in preview and under development.

Describe the bug
I came across this question in a forum on how to deploy a react application to Azure static web apps with Azure Devops.
To Reproduce
Execute the following command,
az staticwebapp create \
    -n my-first-static-web-app \
    -g <RESOURCE_GROUP_NAME> \
    -s https://github.com/<YOUR_GITHUB_ACCOUNT_NAME>/my-first-static-web-app \
    -l <LOCATION> \
    -b main \
    --app-artifact-location "build" \
    --token <YOUR_GITHUB_PERSONAL_ACCESS_TOKEN>

az staticwebapp create -l westus2 -n appNameTest1 -g TestPrEnvResourceGroup -s "https://dev.azure.com/myOrganisationName/myProjectName/_git/myRepoName -b "main" --token "generatedTokenValuefwfsdgsgsd"
Error
Command group 'staticwebapp' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus Operation returned an invalid status 'Bad Request'
https://i.stack.imgur.com/d22hy.png
Additional Context
It works well when it is created in portal

This issue isn't related to the SWA CLI. Please transfer it to https://github.com/Azure/static-web-apps.