error while running a multitarget designer

I set up a multitarget designer calculation using an edited form of the Ablaffinitytestsystem class. It looks like the systems are being set up appropriately, but I keep getting the following error:
Minimizing 'explicit-src-imatinib'...
Initial energy is   -58550.422 kcal/mol
Final energy is    -137454.317 kcal/mol
Minimizing 'explicit-abl-imatinib'...
Initial energy is   -31761.928 kcal/mol
Final energy is     -31756.639 kcal/mol
********************************************************************************
MultiTargetDesign sampler iteration        0
================================================================================
SAMS sampler iteration     0
--------------------------------------------------------------------------------
Expanded Ensemble sampler iteration        0
................................................................................
MCMC sampler iteration 0
Taking 5 steps of Langevin dynamics...
potential  = -53268.502 kT
kinetic    =<PHONE_NUMBER>13.024 kT
force norm = 411813.866 kT/A/dof
Final energy is          nan kT
................................................................................
Updating chemical state with ncmc-geometry-ncmc scheme...
Proposing new topology...
Warning: MCS search truncated
No handlers could be found for logger "alchemy.alchemy"
Proposed transformation: Cc1ccc(cc1Nc2nccc(n2)c3cccnc3)NC(=O)c4ccc(cc4)C[NH+]5CCN(CC5)C => CN1CCC(CC1)COc2cc3c(cc2OC)c(ncn3)Nc4ccc(cc4F)Br
Generating new system...
Performing NCMC annihilation
Traceback (most recent call last):
  File "abl_src_affinity.py", line 215, in <module>
    system.designer.run(niterations=100)
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 1094, in run
    self.update()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 1076, in update
    self.update_samplers()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 1040, in update_samplers
    sampler.update()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 956, in update
    self.update_sampler()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 924, in update_sampler
    self.sampler.update()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 799, in update
    self.update_state()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 719, in update_state
    [ncmc_old_positions, ncmc_elimination_logp, potential_delete] = self.ncmc_engine.integrate(topology_proposal, positions, direction='delete')
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/annihilation/ncmc_switching.py", line 315, in integrate
    context.setPositions(initial_positions)
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/simtk/openmm/openmm.py", line 13580, in setPositions
    return _openmm.Context_setPositions(self, positions)
Exception: Particle coordinate is nan
Compute mode is already set to DEFAULT for GPU 0000:03:00.0.

Looks like I'm having trouble with the NCMC moves. The script that I'm using can be found here

It is exploding even at the MCMC stage before it gets to NCMC. It looks like the system was minimized, but either it isn't enough minimization or the minimized coordinates aren't being used.
Can you open a PR with your changes and we can help test/debug?

I just added my script to PR #201, also added the PDB files I'm working from.
It seems that resolved the clashes in the abl structure fixed the problems with the MCMC step. I encountered a different error that I referenced in issue #171
I seem to consistently get this line:
No handlers could be found for logger "alchemy.alchemy"

Is that a problem with my alchemy installation? I'm pretty sure I have alchemy-dev installed through conda

No handlers could be found for logger "alchemy.alchemy"

We have a logger in alchemy.py, but I'm not sure if we want that or, if we do, how to suppress this message by default. @andrrizzi introduced these---maybe he can help?
It is a known issue, and doesn't mean that you have a problem with your installation.
Any other issues with the test example now?

No immediate problems, still at the geometry engine proposal stage ~40 minutes in

Whoa, really?  Where are you running this?

A single GPU (in shared mode) on the cluster. job ID 7082042
I'm inferring from log file, which has been at the Geometry engine proposal... for a while

Hmm, I'm not sure I understand that logging message. If you don't define a handler it should just fall back to the default one (which basically doesn't report anything I think). The logging was introduced exactly because we could control the amount of output without using verbose parameters that needed to be passed everywhere. In general, logging can be configured by using one of the following anywhere in the script (listed in order of decreasing verbosity)
import logging
logging.basicConfig(level=logging.DEBUG)
logging.basicConfig(level=logging.INFO)
logging.basicConfig(level=logging.WARNING)
logging.basicConfig(level=logging.ERROR)
logging.basicConfig(level=logging.CRITICAL)

I'm not sure how perses handles logging configuration though, so there may be something there.

@steven-albanese I did increase the number of steps/torsion in an effort to get molecules to come out a bit better. Not sure if that's the cause of your hang-up...do you know what the molecule pair was?

@pgrinaway it was imatinib --> ibrutinib

Ok. When I have a chance I'll take a closer look at that case. I'm really hoping that this conference can shed some light on how to do this geometry stuff better + faster.

Not sure if this would be an issue, but the src pdb file seems to be drawing a bond between imatinib to one of the Phe (and messing up the hybridization of one of the Phe carbons). That might be a problem with how I combined the original mol2 file with the pdb file I'm using. is there a better, more programmatic way to combine a mol2 file with the receptor pdb than using PyMol? Or a way to manually edit the bonds in the pdb file?

By default, I believe pymol bonds HETATM atoms by geometry. There is an option to show only bonds with CONECT residues instead.
If you start with PDB and mol2 files, maybe we should have the system setup scripts load in these two files instead to create the system, rather than hacking together a PDB file. Or maybe we should work out a better way of overlaying a small molecule with whatever is bound in the PDB file?  When you extend this to other kinases, what kind of system setup pipeline would be most useful?

I think a better system for overlay would be most useful, since it seems to me that we're looking to start PERSES calculations with the ligand in the binding pocket at the moment

I found the connect settings, but they don't seem to help. Is that going to be a problem for perses?

Looks like you have to change the connect settings before loading the PDB. I've played with this a bit, and I am convinced this is just a wacky PyMOL thing that is bonding two nearby atoms. The PDB file itself is fine.  You should be OK if you minimize.
For now, I'd just write a separate script to generate the systems from PDB files, minimize them, and store them as XML files so you can more rapidly test.

Ok, I let a calculation run overnight and ran into this error:
********************************************************************************
MultiTargetDesign sampler iteration        0
================================================================================
SAMS sampler iteration     0
--------------------------------------------------------------------------------
Expanded Ensemble sampler iteration        0
................................................................................
MCMC sampler iteration 0
Taking 5 steps of Langevin dynamics...
potential  = -209256.348 kT
kinetic    = 47664.825 kT
force norm = 22.576 kT/A/dof
Final energy is  -206219.210 kT
................................................................................
Updating chemical state with ncmc-geometry-ncmc scheme...
Proposing new topology...
No handlers could be found for logger "alchemy.alchemy"
Proposed transformation: Cc1ccc(cc1Nc2nccc(n2)c3cccnc3)NC(=O)c4ccc(cc4)C[NH+]5CCN(CC5)C => CC[C@@H](c1nc2cccc(c2c(=O)n1c3ccccc3)F)Nc4c-5ncnc5nc[nH]4
Generating new system...
Performing NCMC annihilation
Geometry engine proposal...
WARNING:alchemy.alchemy:particle 291 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 350 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 364 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 469 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 545 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 577 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 980 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1033 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1117 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1162 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1202 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1230 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1270 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1413 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1467 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1701 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1756 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1863 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2400 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2410 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2613 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2716 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2777 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2833 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2896 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2990 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3038 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3052 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3160 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3414 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3574 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3874 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3934 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 4067 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
Performing NCMC insertion
Traceback (most recent call last):
  File "abl_src_affinity.py", line 228, in <module>
    system.designer.run(niterations=5)
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 1094, in run
    self.update()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 1076, in update
    self.update_samplers()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 1040, in update_samplers
    sampler.update()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 956, in update
    self.update_sampler()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 924, in update_sampler
    self.sampler.update()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 799, in update
    self.update_state()
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/samplers/samplers.py", line 742, in update_state
    [ncmc_new_positions, ncmc_introduction_logp, potential_insert] = self.ncmc_engine.integrate(topology_proposal, geometry_new_positions, direction='insert')
  File "/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/annihilation/ncmc_switching.py", line 394, in integrate
    raise NaNException("Final potential of 'delete' operation is NaN")
perses.annihilation.ncmc_switching.NaNException: Final potential of 'delete' operation is NaN
Compute mode is already set to DEFAULT for GPU 0000:03:00.0.
All done.

That error looks similar to the one @jchodera mentioned in issue #202

I think we need the GHMC NCMC from @pgrinaway to make the code robust to these kinds of issues.

Did we ever figure out why startup was taking so long?

I wasn't able to figure it out, maybe it's just longer because it needs to set up two kinase systems? Also I was able to reproduce this error

How about we merge this and I can take a look?

ok can do

Using the current master branch of this PERSES repo, I set up another calculation, and it looks like I was able to run until I hit the walltime (12 hours on a single GPU)
creating system
constructing environments list
reading SMILES files
sanitizing SMILES
generating system generator
loading topologies
/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/data/abl-src/src-imatinib.pdb
/cbio/jclab/home/albaness/miniconda/lib/python2.7/site-packages/perses-0.1-py2.7.egg/perses/data/abl-src/abl-imatinib.pdb
creating all solvent environments
generating system
generating thermodynamic state
creating SAMS samplers
creating MultiTargetDesign sampler
minimizing system
Minimizing 'explicit-src-imatinib'...
Initial energy is   -58547.167 kcal/mol
Final energy is    -124192.872 kcal/mol
Minimizing 'explicit-abl-imatinib'...
Initial energy is   -87279.170 kcal/mol
Final energy is    -165953.073 kcal/mol
running system
********************************************************************************
MultiTargetDesign sampler iteration        0
================================================================================
SAMS sampler iteration     0
--------------------------------------------------------------------------------
Expanded Ensemble sampler iteration        0
................................................................................
MCMC sampler iteration 0
Taking 5 steps of Langevin dynamics...
potential  = -208320.692 kT
kinetic    = 47806.057 kT
force norm = 23.123 kT/A/dof
Final energy is  -205592.274 kT
................................................................................
Updating chemical state with ncmc-geometry-ncmc scheme...
Proposing new topology...
No handlers could be found for logger "alchemy.alchemy"
Proposed transformation: Cc1ccc(cc1Nc2nccc(n2)c3cccnc3)NC(=O)c4ccc(cc4)C[NH+]5CCN(CC5)C => COc1cc2c(cc1OCCCN3CCOCC3)c(ncn2)Nc4ccc(c(c4)Cl)F
Generating new system...
Performing NCMC annihilation
Geometry engine proposal...
WARNING:alchemy.alchemy:particle 291 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 350 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 364 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 469 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 545 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 577 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 980 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1033 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1117 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1162 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1202 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1230 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1270 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1413 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1467 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1701 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1756 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1863 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2400 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2410 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2613 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2716 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2777 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2833 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2896 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2990 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3038 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3052 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3160 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3414 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3574 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3874 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3934 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 4067 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
Performing NCMC insertion
potential before geometry  :  -202379.282 kT
potential after geometry   :  -196003.240 kT
---------------------------------------------------------
switch_logp                :    -6376.042
geometry_logp_propose      :       50.605
geometry_logp_reverse      : -1460152.142
logp_accept = -9.0162e+11 [logp_proposal -4.5977e-03 geometry_logp -1.4602e+06 switch_logp -6.3760e+03 ncmc_elimination_logp +7.5566e+01 ncmc_introduction_logp -9.0162e+11 old_log_weight +0.0000e+00 new_log_weight +0.0000e+00]
    rejected
--------------------------------------------------------------------------------
================================================================================
================================================================================
SAMS sampler iteration     0
--------------------------------------------------------------------------------
Expanded Ensemble sampler iteration        0
................................................................................
MCMC sampler iteration 0
Taking 5 steps of Langevin dynamics...
potential  = -278369.085 kT
kinetic    = 55653.080 kT
force norm = 22.541 kT/A/dof
Final energy is  -270272.586 kT
................................................................................
Updating chemical state with ncmc-geometry-ncmc scheme...
Proposing new topology...
WARNING:alchemy.alchemy:particle 31 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 190 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 249 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 376 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 387 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 417 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 495 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 577 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1045 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1156 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1205 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1271 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1295 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1387 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1657 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1695 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1745 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1756 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 1819 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2331 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2402 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2435 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2459 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2470 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2659 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2712 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2762 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2829 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 2881 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3103 has Lennard-Jones sigma = 0 (charge=0.4102 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3127 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3162 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3197 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3279 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3333 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3456 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3671 has Lennard-Jones sigma = 0 (charge=0.3992 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3871 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
WARNING:alchemy.alchemy:particle 3932 has Lennard-Jones sigma = 0 (charge=0.4275 e, sigma=0.0 nm, epsilon=0.0 kJ/mol); setting sigma=1A
Proposed transformation: Cc1ccc(cc1Nc2nccc(n2)c3cccnc3)NC(=O)c4ccc(cc4)C[NH+]5CCN(CC5)C => CCCS(=O)(=O)Nc1ccc(c(c1F)C(=O)c2c[nH]c3c2cc(cn3)c4ccc(cc4)Cl)F
Generating new system...
Performing NCMC annihilation
Geometry engine proposal...
=>> PBS: job killed: walltime 43215 exceeded limit 43200
Compute mode is already set to DEFAULT for GPU 0000:03:00.0.
All done.

So you only made it through one geometry proposal in 12h?  Yikes.
It looks like you have the verbose settings off for the different samplers.  Can you turn them on, like in this example?
For now, since this is so slow, can you test the kinase inhibitors test system in explicit solvent (instead of vacuum)?
I think @pgrinaway is working on the particle filter MC right now, which will eventually lead to optimizations.

@pgrinaway said that some fixed need to be checked in to prevent this from
taking forever.

Unless there are any objections, I'm going to close this issue because the original issue seems to have been addressed

I think this is fixed.


iPhone 4s and 5 write issues

Hi guys
I am doing a basic write of information to realm, Id, artists, facebookId and the write works on all phones.  However in my simulator with iPhone 5 and 4s, for some reason the Id is getting altered when writing to the realm.  For example I run the same code on an IPhone 6s and up i get (random object):
FacebookArtists {
id = 100;
artist = Indie-Go;
facebookId =<PHONE_NUMBER>86033;
}
and on an iPhone 5 and 4s i get:
FacebookArtists {
id = 100;
artist = Indie-Go;
facebookId = -1643586703;
}
It is altering every facebookId, that is previously determined by a JSON file.  The JSON file stays the same however the when writing to realm on iPhone 5 and 4s it seems to changing the information.  Can you please assist?  I am using the latest version of Realm currently (0.96.2).

@WayneNotTaken Because<PHONE_NUMBER>86033 is overflowed in a 32-bit integer. If you'd like to store the ID as an integer in 32-bit devices, use long long instead NSInteger, int or long. Or you can convert an integer to a string and store the value as String.

Awesome thanks.  All sorted out.  Used a string to write the facebookId and its works perfectly.  Thanks for your quick help.


PERF:  We should read files in chunks rather than lines

The tokenizer has universal newline support and we basically only support those newlines anyway.  Since we removed the "raw" file reader, we are always reading line-by-line.  This is not optimal!
For very narrow files (few integer fields), in this extreme a single integer per line <1000, the difference is pretty big:
>>> d = pd.read_csv("test.csv", delimiter=",", dtype=int, float_precision="round_trip")
36.4 ms ± 427 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
>>> e = _loadtxt("test.csv", delimiter=",", dtype=int)
67.6 ms ± 478 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

when read by line, and:
>>> d = pd.read_csv("test.csv", delimiter=",", dtype=int, float_precision="round_trip")
36.5 ms ± 589 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
>>> e = _loadtxt("test.csv", delimiter=",", dtype=int)
26.2 ms ± 368 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

when read in chunks of 2<<14 ~bytes~ characters.
EDIT: To be clear, this is probably as simple as a small flag, after querying whether the file-like object supports a read method.  It may also make sense to just the iterator API in C if it does not (instead of readline).  There is one serious problem though: loadtxt never reads too much data with max_rows.  So we can't do this unless we opened the file if max_rows is given.

Now I am wondering if I should do the opposite, and double-down on line parsing...

Closing, since I put gh-60, so this is the current state.  There is still the issue especially of stray newlines when using the "iterator" mode: gh-28.


limit the FPS

How can I set a max FPS?

My guess would be to set vsync to true in window construction parameters. Or if that doesn't help, to generate frames "manually".
see https://github.com/node-3d/3d-core-raub#initopts--

opts.vsync
requestAnimationFrame

vsync is now enabled by default


Never mutate user errors

Some libraries freezes an error object before rejection (Object.freeze(err)). Because code which adds context and code to the error object throws an exception, the router unable to call error handler and some exceptions may become unhandled.
Solution: never mutate third-party error object.
ref #152

Could there be a way to disable adding context/code to the error completely?  For example, I log errors to a service and have to delete these properties before sending.

@frenzzy thanks for the feedback. #152 looks like a good solution to me, and provides the context incase you want it.
code feels like something that should be handled by user code, but maybe I'm missing something.
Perhaps just document that adding {  path: '(.*)'  } last will capture "404s" and catch will capture the "500s"

@gpoitch correct, I just created a PR with this change. Would be cool if you could take a look: #158


Skosgi util

Critical Changes
Changes
Issues Closed
New Metadata
Deleted Metadata
Testing Notes

@skosgi1 @lifewithryan I noticed this PR is pointing from skosgiUtil to feature/skosgi/indentationDescribeUtil. Ideally, the commits would have been made on the feature/skosgi/indentationDescribeUtil branch and the PR would be pointing from feature/skosgi/indentationDescribeUtil to master.
I think this might be okay for right now - once this PR is merged, then the commits will all be in the feature/skosgi/indentationDescribeUtil branch. Then we can create another pull request to merge feature/skosgi/indentationDescribeUtil into master.

@skosgi1 @lifewithryan I noticed this PR is pointing from skosgiUtil to feature/skosgi/indentationDescribeUtil. Ideally, the commits would have been made on the feature/skosgi/indentationDescribeUtil branch and the PR would be pointing from feature/skosgi/indentationDescribeUtil to master.
I think this might be okay for right now - once this PR is merged, then the commits will all be in the feature/skosgi/indentationDescribeUtil branch. Then we can create another pull request to merge feature/skosgi/indentationDescribeUtil into master.

Yep. That's exactly what we have to do :) Its alot like when we get a PR from the community. We pull it down and put it into a EDA branch and merge to master from there. In the future we should just create the branch directly from EDA. I am a command line person so from master: git checkout -b feature//featureName, do work, commit, push with --set-upstream if necessary (usually is on the first one) then submit PR.

@lifewithryan I have updated the requested changes, can you please give it another shot and let me know? @robertbwatson @lifewithryan Thank you for letting me know. I will follow the process you mentioned from my next WI onwards.

@lifewithryan Do you think we need any release notes for this?

Going to merge this into the feature branch. Then I will create a PR from that and perhaps have someone review from there.


MVD Deployment @ 8 Feb 2023

Bug Report
Describe the Bug
Starting from scratch, by deploying MVD and start learning the concepts around Data Spaces.
Expected Behavior
Easier to follow documentation around quick deployment into Azure. Goal is to try it, understand comments, and have a working demo.
Observed Behavior
Documentation had a few glitches, and ended up looking at GH Actions, repo. Not able to deploy it on Azure using documentation.
Was able to follow documentation and run the local environment.
Steps to Reproduce

Follow Documentation

OS: Windows 11, WSL2 with Ubuntu 20.04

Detailed Description
Running the Initialize CD failed as the ACR_NAME var was not added to GH env vars
Documentation mentioned a Deploy action which I found it being removed in a ~recent commit. I presume WIP.
Running the system-tests went ok. All based on docker-compose.
Ended up pushing images to ACR manually and put it to work.

I can confirm that the current documentation doesn't allow to deploy, like you said, the Deploy action no longer exists. I still think it would be valuable to have the ability to deploy quickly after setting up Azure secrets are set up (like in Cloud-CD).
I understand for testing that maybe it complicated things, but I think that a general cloud deployment workflow would be convenient.  Can I get more information about why the deploy action was removed? (I have read #112, but I don't think I understood it perfectly so happy to learn more!)
I'm also looking at this issue, and I'm interested in knowing if:

would it be okay to bring the Deploy workflow back?
if not, what is the best way to deploy to cloud as of now? (is it create_dataspaces.sh + the rest of the testing flow in run_azure_dataspaces_tests.sh?)

I agree with @robertdevlin-ms.
Please let me mention @paullatzelsperger here. We would like to have your opinion regarding questions above.

@paullatzelsperger Thank you for answering our questions. Those reasons make sense for me.
@joalmeid As the deploy workflow will not be back again I wrote up a blog post about Azure deployment for MVD. (contributor : @loubigard)
I think we can close this issue if you two feel that the article above covers how to deploy MVD to Azure as a workaround.

Hello @paullatzelsperger, @joalmeid,
We have created the PRs #139 and #138 in order to close this issue.
Could you please give us your feedback?


Error after e2e test 'Feature could not be found in feature map`

Rasa version:
Rasa 1.1.4
Rasa X version (if used & relevant):
Rasa X 0.19.5
Python version:
Python 3.6.8
Operating system (windows, osx, ...):
Windows 8.1
Issue:
I have the file e2e_stories.md for e2e test. I run the command rasa test --stories data/e2e_stories.md --e2e .
After the test, I run the training command 'rasa train'.
As a result, I had a lot of messages in logs with phrases from e2e_stories.md: Feature 'intent_deny: no'(value: '1.0') could not be found in feature map.
Also phrases from e2e_stories.md were added to domain.yml as new intents.
Error (including full traceback):
2019-08-23 18:24:05 DEBUG    rasa.core.featurizers  - Feature 'intent_deny: no'(value: '1.0') could not be found in feature map. Make sure you added all intents and entities to the domain

Command or request that led to error:
rasa test --stories data/e2e_stories.md --e2e
rasa train

Thanks for the issue, @dakshvar22 will get back to you about it soon!
You may find help in the docs and the forum, too 🤗

Hi,
Can you post a snippet of your data as well please?

Hi @dakshvar22 ,
I have reproduced the error with default content that was created by rasa init.
Then I run
rasa test --stories data/e2e_stories.md --e2e
Then I run rasa train.
Errors in the log during training:
2019-09-08 23:18:26 DEBUG    rasa.core.featurizers  - Feature 'intent_goodbye: bye' (value: '1.0') could not be found in feature map. Make sure you added all intents and entities to the domain
Then I run rasa x
After the Rasa X running, domain.yml file started to have new values:
intents:
- greet
- goodbye
- affirm
- deny
- mood_great
- mood_unhappy
- 'greet: hello bot'
- 'affirm: yes'
- 'goodbye: bye'
- 'mood_great: perfect'
- 'mood_unhappy: sad'
- 'deny: no'

@dakshvar22 were you able to reproduce the issue or should I provide more information?

Hi @TatianaParshina, I noticed this issue was created in August, are you still using the same Rasa and Rasa X versions? If not, would it be possible to update them to the newest one? (Rasa X 0.21.1)

@TatianaParshina looks like you have end-to-end stories in the same data directory that you are passing to rasa train. As of now, e2e format is only supported for testing, not training, but it's interpreting the end-to-end format as a new intent. I think this is something we should pick up on and warn/error on our part, but in the mean time, probably best if you keep your test data in a different folder like test_data.

I'm getting this issue even though I have not included end to end stories


Add tests on Linux ARM

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

@grondo Do you have an idea what could cause the  t1016-nest-namespace.t test to fail on Linux ARM? It might be related to us cross-compiling.
90/90 Test #90: python/t10001-resourcegraph.py ........   Passed    1.72 sec

99% tests passed, 1 tests failed out of 90

Total Test time (real) = 4917.02 sec

The following tests FAILED:
	 23 - t1016-nest-namespace.t (Timeout)
Errors while running CTest
make[3]: *** [CMakeFiles/check.dir/build.make:70: CMakeFiles/check] Error 8
make[2]: *** [CMakeFiles/Makefile2:1381: CMakeFiles/check.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:1388: CMakeFiles/check.dir/rule] Error 2
make: *** [Makefile:543: check] Error 2

No, I have no idea. Maybe a question for @trws (it doesn't fail in our aarch64 ci afaik). Do you know if it is possible to reproduce locally?


feat: zeta supply checks

Description

Adds helper functions and a thread to the zetaclient to perform the zeta supply checks.

Closes: <#1352>
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Include instructions and any relevant details so others can reproduce.

[ ] Tested CCTX in localnet
[ ] Tested in development environment
[ ] Go unit tests
[ ] Go integration tests
[ ] Tested via GitHub Actions

Checklist:

[ ] I have added unit tests that prove my fix feature works

As far as I can tell CctxByStatus would be called every ticks? It would be a very intensive operation iterating the cctxs in this query.
If we just need this query for aborted CCTXs we should think about a new indexing like we do for pending cctxs, or storing the result we want for the aborted cctxs directly onchain.
Another solution is to use a backend indexer for the data but in this case we should no longer consider observers posting invariant failure to the protocol because they should only rely on blockchain data in order to do this.

Yeah, I thought of this; since we only really need the amount, storing that separately might be a good idea. But we can let this run for sometime on athens3 to see how well it holds and implement a service to keep track of this amount only.
The GetCctxByStatus is just a good-to-have query, and I agree that it should not be used for this purpose.


chore: graphite and gitpod init

Current dependencies on/for this PR:

master

PR #2

PR #3   👈

PR #4

This comment was auto-generated by Graphite.

⏳ This pull request is set to merge as part of a Graphite merge job
Stack job ID: ig6nty8pUfH6VXaZuEcF.
See details on graphite.dev


Fix syntax error because of missing catch error name

This PR https://github.com/Microsoft/types-publisher/pull/608 occurred syntax error.
This broke DefinitelyTyped CI pipeline, like https://github.com/DefinitelyTyped/DefinitelyTyped/pull/34973

@rbuckton Cloud you take a look?

Not a syntax error I think https://mariusschulz.com/blog/typescript-2-5-optional-catch-binding

Problem seems to be that DefinitelyTyped travis uses node 8, support for the optional catch binding proposal (stage 4 now, so will likely be part of ES2019) was only added in node 10:
https://h3manth.com/new/blog/2018/whats-new-in-node-v10/

@pluma Thanks for the further investigation. I updated title and description of this PR.

This got fixed in https://github.com/microsoft/types-publisher/commit/9de71ee4db1979e62c8030cd7776c2899990938e.

Thanks all!


ERROR:config.ini file not found. Copy or rename the config.sample.ini to config.ini.

Hi Manuel
Thanks alot for sharing your software.  I have already view times tried to run your driver but I get alway this error message. Venus OS is fresh installed (2.93) on a RPI 4 and nothing is connected. MQTT is running. The config.ini is in the folder.

What I'm doing wrong?
Best regards
Harald

Hi @harald7,
can you post your config.ini? Please remove the password before, if set.
Can you also post the result of ls -l /data/etc/dbus-mqtt-battery?

Sure!

`; CONFIG FILE
; GitHub reporitory: https://github.com/mr-manuel/venus-os_dbus-mqtt-battery
; remove semicolon ; to enable desired setting
[DEFAULT]
; Set logging level
; ERROR = shows errors only
; WARNING = shows ERROR and warnings
; INFO = shows WARNING and running functions
; DEBUG = shows INFO and data/values
; default: WARNING
logging = DEBUG
; Device name
; default: MQTT Battery
device_name = MQTT Battery
; Device VRM instance
; default: 41
device_instance = 41
; Specify after how many seconds the driver should exit (disconnect), if no new MQTT message was received
; default: 60
; value to disable timeout: 0
timeout = 60
[MQTT]
; IP addess or FQDN from MQTT server
broker_address = <IP_ADDRESS>
; Port of the MQTT server
; default plaintext: 1883
; default TLS port: 8883
broker_port = 1883
; Enables TLS
; 0 = Disabled
; 1 = Enabled
;tls_enabled = 1
; Absolute path to the Certificate Authority certificate file that is to be treated as trusted by this client
;tls_path_to_ca = /data/keys/mosquitto.crt
; Disables verification of the server hostname in the server certificate
; 0 = Disabled
; 1 = Enabled
;tls_insecure = 1
; Username used for connection
;username = myuser
; Password used for connection
;password = mypassword
; Topic where the meters data as JSON string is published
; minimum required JSON payload: { "Dc": { "Power": 321.6, "Voltage": 52.6  }, "Soc": 63 }
topic = enphase/battery
{
"Dc": {
"Power": 321.6,
"Voltage": 52.7,
"Current": 6.10,
"Temperature": 23
},
"InstalledCapacity": 200.0,
"ConsumedAmphours": 74.5,
"Capacity": 125.5,
"Soc": 63,
"TimeToGo": 43967,
"Balancing": 0,
"SystemSwitch": 0,
"Alarms": {
"LowVoltage": 0,
"HighVoltage": 0,
"LowSoc": 0,
"HighChargeCurrent": 0,
"HighDischargeCurrent": 0,
"HighCurrent": 0,
"CellImbalance": 0,
"HighChargeTemperature": 0,
"LowChargeTemperature": 0,
"LowCellVoltage": 0,
"LowTemperature": 0,
"HighTemperature": 0,
"FuseBlown": 0
},
"Info": {
"ChargeRequest": 0,
"MaxChargeVoltage": 55.2,
"MaxChargeCurrent": 80.0,
"MaxDischargeCurrent": 120.0
},
"History": {
"ChargeCycles": 5,
"MinimumVoltage": 40.8,
"MaximumVoltage": 58.4,
"TotalAhDrawn": 1057.3
},
"System": {
"MinVoltageCellId": "C3",
"MinCellVoltage": 3.392,
"MaxVoltageCellId": "C15",
"MaxCellVoltage": 3.417,
    "MinTemperatureCellId": "C2",
    "MinCellTemperature": 22.5,
    "MaxTemperatureCellId": "C9",
    "MaxCellTemperature": 23.5,
    "MOSTemperature": 23.5,

    "NrOfModulesOnline": 0,
    "NrOfModulesOffline": 0,

    "NrOfModulesBlockingCharge": 0,
    "NrOfModulesBlockingDischarge": 0
},
"Voltages": {
    "Cell1":  3.201,
    "Cell2":  3.202,
    "Cell3":  3.203,
    "Cell4":  3.204,
    "Cell5":  3.205,
    "Cell6":  3.206,
    "Cell7":  3.207,
    "Cell8":  3.208,
    "Cell9":  3.209,
    "Cell10": 3.210,
    "Cell11": 3.211,
    "Cell12": 3.212,
    "Cell13": 3.213,
    "Cell14": 3.214,
    "Cell15": 3.215,
    "Cell16": 3.216,
    "Cell17": 3.217,
    "Cell18": 3.218,
    "Cell19": 3.219,
    "Cell20": 3.220,
    "Cell21": 3.221,
    "Cell22": 3.222,
    "Cell23": 3.223,
    "Cell24": 3.224
},
"Balances": {
    "Cell1":  0,
    "Cell2":  0,
    "Cell3":  0,
    "Cell4":  0,
    "Cell5":  0,
    "Cell6":  0,
    "Cell7":  0,
    "Cell8":  0,
    "Cell9":  0,
    "Cell10": 0,
    "Cell11": 0,
    "Cell12": 0,
    "Cell13": 0,
    "Cell14": 0,
    "Cell15": 0,
    "Cell16": 0,
    "Cell17": 0,
    "Cell18": 0,
    "Cell19": 0,
    "Cell20": 0,
    "Cell21": 0,
    "Cell22": 0,
    "Cell23": 0,
    "Cell24": 0
},
"Io": {
    "AllowToCharge": 0,
    "AllowToDischarge": 0,
    "AllowToBalance": 0,
    "ExternalRelay": 0
}

}
`

Could you also download the dbus-mqtt-battery.py from the master branch and use it? I changed something for better error handling to understand better the problem. You can run this command to do so: wget -O /data/etc/dbus-mqtt-battery/dbus-mqtt-battery.py https://raw.githubusercontent.com/mr-manuel/venus-os_dbus-mqtt-battery/master/dbus-mqtt-battery/dbus-mqtt-battery.py

Have you uploaded the script by FTP/SFTP or downloaded directly in Venus OS?

I have uploaded with WinSCP.

2023-04-04 18:34:56.325368500 ERROR: Source contains parsing errors: '/data/etc/dbus-mqtt-battery/config.ini' 2023-04-04 18:34:56.325374500   [line 59]: '{\n' 2023-04-04 18:34:56.325377500   [line 176]: '}\n' 2023-04-04 18:34:56.325380500 If config.ini file is not found then copy or rename the config.sample.ini to config.ini. The driver restarts in 60 seconds. 2023-04-04 18:34:56.407045500 *** starting dbus-mqtt-battery ***

Thats the problem. The line endings where changed during upload. From \n to \r\n.
See https://winscp.net/eng/docs/faq_line_breaks.

Should I download direct in VenusOS?
If yes which version?

Yes, download directly in Venus OS. You can download from the master branch.

Like this?
wget -O /data/etc/dbus-mqtt-battery/dbus-mqtt-battery.py https://raw.githubusercontent.com/mr-manuel/venus-os_dbus-mqtt-battery/master/venus-os_dbus-mqtt-battery-master.zip

Try this, haven't tested it.
cd /tmp

wget https://github.com/mr-manuel/venus-os_dbus-mqtt-battery/archive/refs/heads/master.zip
unzip master.zip

cp -rf /tmp/venus-os_dbus-mqtt-battery-master/etc/dbus-mqtt-battery/ /data/etc

chmod +x /data/etc/dbus-mqtt-battery/*.sh
chmod +x /data/etc/dbus-mqtt-battery/*.py
chmod +x /data/etc/dbus-mqtt-battery/service/run
chmod +x /data/etc/dbus-mqtt-battery/service/log/run

bash /data/etc/dbus-mqtt-battery/install.sh

Hi Manuel
I managed to install the driver with your instructions. Thank you for that.
cp -rf /tmp/venus-os_dbus-mqtt-battery-master/etc/dbus-mqtt-battery/ /data/etc
The first etc was wrong.
That's what I have done:
cd /tmp
wget https://github.com/mr-manuel/venus-os_dbus-mqtt-battery/archive/refs/heads/master.zip
unzip master.zip
cp -rf /tmp/venus-os_dbus-mqtt-battery-master/dbus-mqtt-battery/ /data/etc
chmod +x /data/etc/dbus-mqtt-battery/*.sh
chmod +x /data/etc/dbus-mqtt-battery/*.py
chmod +x /data/etc/dbus-mqtt-battery/service/run
chmod +x /data/etc/dbus-mqtt-battery/service/log/run
cd /data/etc/dbus-mqtt-battery
ls
mv config.sample.ini config.ini
nano config.ini #Made the adjustments and insert the json
bash /data/etc/dbus-mqtt-battery/install.sh
svstat /service/dbus-mqtt-battery
tail -n 100 -f /data/log/dbus-mqtt-battery/current | tai64nlocal
Unfortunately I get the same error.
2023-04-04 20:37:55.075887500 *** starting dbus-mqtt-battery *** 2023-04-04 20:38:55.900223500 ERROR: Source contains parsing errors: '/data/etc/dbus-mqtt-battery/config.ini' 2023-04-04 20:38:55.900229500   [line 59]: '{\n' 2023-04-04 20:38:55.900232500   [line 176]: '}\n' 2023-04-04 20:38:55.900234500 If config.ini file is not found then copy or rename the config.sample.ini to config.ini. The driver restarts in 60 seconds.

; CONFIG FILE
; GitHub reporitory: https://github.com/mr-manuel/venus-os_dbus-mqtt-battery
; remove semicolon ; to enable desired setting

[DEFAULT]
; Set logging level
; ERROR = shows errors only
; WARNING = shows ERROR and warnings
; INFO = shows WARNING and running functions
; DEBUG = shows INFO and data/values
; default: WARNING
logging = DEBUG

; Device name
; default: MQTT Battery
device_name = MQTT Battery

; Device VRM instance
; default: 41
device_instance = 41

; Specify after how many seconds the driver should exit (disconnect), if no new MQTT message was received
; default: 60
; value to disable timeout: 0
timeout = 60

[MQTT]
; IP addess or FQDN from MQTT server
broker_address = <IP_ADDRESS>

; Port of the MQTT server
; default plaintext: 1883
; default TLS port: 8883
broker_port = 1883

; Enables TLS
; 0 = Disabled
; 1 = Enabled
;tls_enabled = 1

; Absolute path to the Certificate Authority certificate file that is to be treated as trusted by this client
;tls_path_to_ca = /data/keys/mosquitto.crt

; Disables verification of the server hostname in the server certificate
; 0 = Disabled
; 1 = Enabled
;tls_insecure = 1

; Username used for connection
;username = myuser

; Password used for connection
;password = mypassword

; Topic where the meters data as JSON string is published
; minimum required JSON payload: { "Dc": { "Power": 321.6, "Voltage": 52.6  }, "Soc": 63 }
topic = enphase/battery

{
   "Dc": {
       "Power": 321.6,
       "Voltage": 52.7,
       "Current": 6.10,
       "Temperature": 23
   },
   "InstalledCapacity": 200.0,
   "ConsumedAmphours": 74.5,
   "Capacity": 125.5,
   "Soc": 63,
   "TimeToGo": 43967,
   "Balancing": 0,
   "SystemSwitch": 0,
   "Alarms": {
       "LowVoltage": 0,
       "HighVoltage": 0,
       "LowSoc": 0,
       "HighChargeCurrent": 0,
       "HighDischargeCurrent": 0,
       "HighCurrent": 0,
       "CellImbalance": 0,
       "HighChargeTemperature": 0,
       "LowChargeTemperature": 0,
       "LowCellVoltage": 0,
       "LowTemperature": 0,
       "HighTemperature": 0,
       "FuseBlown": 0
   },
   "Info": {
       "ChargeRequest": 0,
       "MaxChargeVoltage": 55.2,
       "MaxChargeCurrent": 80.0,
       "MaxDischargeCurrent": 120.0
   },
   "History": {
       "ChargeCycles": 5,
       "MinimumVoltage": 40.8,
       "MaximumVoltage": 58.4,
       "TotalAhDrawn": 1057.3
   },
   "System": {
       "MinVoltageCellId": "C3",
       "MinCellVoltage": 3.392,
       "MaxVoltageCellId": "C15",
       "MaxCellVoltage": 3.417,

       "MinTemperatureCellId": "C2",
       "MinCellTemperature": 22.5,
       "MaxTemperatureCellId": "C9",
       "MaxCellTemperature": 23.5,
       "MOSTemperature": 23.5,

       "NrOfModulesOnline": 0,
       "NrOfModulesOffline": 0,

       "NrOfModulesBlockingCharge": 0,
       "NrOfModulesBlockingDischarge": 0
   },
   "Voltages": {
       "Cell1":  3.201,
       "Cell2":  3.202,
       "Cell3":  3.203,
       "Cell4":  3.204,
       "Cell5":  3.205,
       "Cell6":  3.206,
       "Cell7":  3.207,
       "Cell8":  3.208,
       "Cell9":  3.209,
       "Cell10": 3.210,
       "Cell11": 3.211,
       "Cell12": 3.212,
       "Cell13": 3.213,
       "Cell14": 3.214,
       "Cell15": 3.215,
       "Cell16": 3.216,
       "Cell17": 3.217,
       "Cell18": 3.218,
       "Cell19": 3.219,
       "Cell20": 3.220,
       "Cell21": 3.221,
       "Cell22": 3.222,
       "Cell23": 3.223,
       "Cell24": 3.224
   },
   "Balances": {
       "Cell1":  0,
       "Cell2":  0,
       "Cell3":  0,
       "Cell4":  0,
       "Cell5":  0,
       "Cell6":  0,
       "Cell7":  0,
       "Cell8":  0,
       "Cell9":  0,
       "Cell10": 0,
       "Cell11": 0,
       "Cell12": 0,
       "Cell13": 0,
       "Cell14": 0,
       "Cell15": 0,
       "Cell16": 0,
       "Cell17": 0,
       "Cell18": 0,
       "Cell19": 0,
       "Cell20": 0,
       "Cell21": 0,
       "Cell22": 0,
       "Cell23": 0,
       "Cell24": 0
   },
   "Io": {
       "AllowToCharge": 0,
       "AllowToDischarge": 0,
       "AllowToBalance": 0,
       "ExternalRelay": 0
   }
}

Wait… is this all one file? If yes you have to remove the JSON. This is only an example you have to send via MQTT.

Good morning Manuel
Yes, it was all in the config.ini
After removing the json it's working.
2023-04-05 07:44:40.042999500 *** starting dbus-mqtt-battery *** 2023-04-05 07:44:41.234346500 INFO:root:MQTT client: Connected to MQTT broker! 2023-04-05 07:44:41.238671500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:44:46.244143500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:44:51.248650500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:44:56.253985500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:45:01.259554500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:45:06.263210500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:45:11.268788500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:45:33.707125500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:45:38.712683500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:45:43.718254500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:45:48.723811500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:45:53.729375500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:45:58.735009500 WARNING:root:Waiting since 60 seconds for receiving first data... 2023-04-05 07:46:03.740567500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:46:08.746130500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:46:13.751693500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:46:18.757253500 INFO:root:Waiting 5 seconds for receiving first data... 2023-04-05 07:46:23.762821500 INFO:root:Waiting 5 seconds for receiving first data...
Thanks a lot!
Another question if it's OK to ask in this issue.
I have all the data in ioBroker, the NodeRed adapter is also installed.
What's the easiest way to create the json string?

Unfortunately I don't know ioBroker, but it should also be possible to generate the JSON directly there. I would use Node-RED, if you don't know both.

Do you have a link for a example in Node-Red?

Unfortunately not, but if you search for "node red send json over mqtt" on Google you find a lot.
Probably you have also to join the data you get from ioBroker.


feat: HuggingFace Embedding Inference Server EF

Go-version bump to 1.21 to support testcontainers

#50

@bjwswang, I know I've bumped the go-version, but this seems to be required when using testcontainers. I've also noted that langchaingo also uses 1.21. Wdyt?

When langchaingo bumped golang to 1.21,I asked why . The author didn't make it(why go1.21) that clear.

ok then, let's keep the bump to 1.21. It is unfortunate, but langchaingo is probably the largest user of the lib so we might as well go forward.

ok then, let's keep the bump to 1.21. It is unfortunate, but langchaingo is probably the largest user of the lib so we might as well go forward.

that's right.


PfxImport:  Cannot validate argument on parameter Thumbprint

Details of the scenario you tried and the problem that is occurring
Importing DigiCert Wildcard / GeoTrust RSA CA 2018 certificate fails with "Cannot validate argument on parameter Thumbprint: System.InvalidOperationException: 'c0ba334eb65b7942e9c57c116a804dc44a269a28' is
not a valid hash.
Verbose logs showing the problem

VERBOSE: [localhost]:[SP201902]: LCM:  [ Start  Set      ]
VERBOSE: [localhost]:[SP201902]: LCM:  [ Start  Resource ]  [[PfxImport]PfxImport_]
VERBOSE: [localhost]:[SP201902]: LCM:  [ Start  Test     ]  [[PfxImport]PfxImport_]
VERBOSE: [localhost]:[SP201902]: LCM:  [ End    Test     ]  [[PfxImport]PfxImport_]  in 1.4690 seconds.
PowerShell DSC resource MSFT_PfxImport  failed to execute Test-TargetResource functionality with error message: Cannot
validate argument on parameter 'Thumbprint'. System.InvalidOperationException: 'c0ba334eb65b7942e9c57c116a804dc44a269a28' is
not a valid hash.
+ CategoryInfo          : InvalidOperation: (:) [], CimException
+ FullyQualifiedErrorId : ProviderOperationExecutionFailure
+ PSComputerName        : [localhost]
Suggested solution to the issue
Review/resolve code.
The DSC configuration that is used to reproduce the issue (as detailed as possible)
PfxImport ('PfxImport_' + $Certificate.CertThumbprint) {
                Ensure = $Certificate.Ensure
                Thumbprint = $Certificate.Thumbprint
                Path = $Certificate.DestinationPath
                Location = $Certificate.Location
                Store = $Certificate.Store
                Credential = $objCertCredential
                Exportable = $Certificate.Exportable
            }

The operating system the target node is running
OsName               : Microsoft Windows 10 Pro
OsOperatingSystemSKU : 48
OsArchitecture       : 64-bit
WindowsVersion       : 1909
WindowsBuildLabEx    : 18362.1.amd64fre.19h1_release.190318-1202
OsLanguage           : en-US
OsMuiLanguages       : {en-US}
Version and build of PowerShell the target node is running
Name                           Value

PSVersion                      5.1.18362.628
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.18362.628
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
Version of the DSC module that was used ('dev' if using current dev branch)
<IP_ADDRESS>

Added verbose logging to view comparison, trying to determine why it doesn't match when Hex Length 40.

if ("C0BA334EB65B7942E9C57C116A804DC44A269A28" -cmatch "^[a-fA-F0-9]{40}$")
{
    Write-Host "Match!" -ForegroundColor Green
}
else
{
    Write-Host "Not a Match!" -ForegroundColor Red
}

Yields no match:

Hi @ruandersMSFT - I've found the issue. When you copy the Thumbprint of a certificate out of Windows Certificate Manager, it seems to include an extra character at the beginning (something to do with char encoding).
If I Copy/Paste the code you put in above into a PowerShell window it shows like this:

So delete the 'C' at the beginning and add it back in.

Verified, resolved.

Ran into the same issue. Copy/Paste from cert manager added a special character. Took action recommended by PlagueHO about and it resolved the issue.


Repo is hacktoberfest excluded

@abhisek247767 please look into it, the repo is hacktoberfest excluded.

sorry . we are failed to back our repo.


Gh 1715 namedview creatorid

Codecov Report

:exclamation: No coverage uploaded for pull request base (develop@897b6e9). Click here to learn what that means.
The diff coverage is 50%.

@@            Coverage Diff             @@
##             develop    #1716   +/-   ##
==========================================
  Coverage           ?   65.88%
  Complexity         ?     2010
==========================================
  Files              ?      815
  Lines              ?    26484
  Branches           ?     3403
==========================================
  Hits               ?    17450
  Misses             ?     7665
  Partials           ?     1369

Impacted Files
Coverage Δ
Complexity Δ

...r/data/elementdefinition/view/NamedViewDetail.java
0% <0%> (ø)
0 <0> (?)

...e/operation/handler/named/AddNamedViewHandler.java
70.96% <100%> (ø)
0 <0> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 897b6e9...b6d3ba8. Read the comment docs.


Add informations to cargo order form

About the PR
Displays the product name, unit cost and description in the small cargo order form window so you actually see what you are ordering.
It also fix the descriptions for ATV and syndie secway
Screenshots

Changelog
:cl:

add: More detailed informations are now displayed on the cargo order form

The setsize no longer fits the entire control so should be expanded vertically. Also the other text is white but this is grey (should it be rich text?)

It is grey because this is default color of richtext and richtext is the only way i found to get the text on multiple lines. I will force it to white.


Composibility

Sorry for the back-to-back issues, but these seems separate, so I didn't want to bog down the first issue.
I'm using untwine to write a command parser for a "CLI" running inside of a TUI. Its been nice so far, but there is an annoyance with calling parsers from other parsers. What I would like to do is parse the top level command and then parse subcommands for that command in the function block. However, pulling out the tail of a tail of a string (via rest=.*) returns a Vec<char>, which then has to be parsed into a String in order to get a &str.
It would be nice to have syntax for "everything else", perhaps ... If possible, it would be even better if this could be placed anywhere, not just at the end of a sequence (for example, if your data is suffixed).

I would recommend using rest=<.*> in this case, it will capture as a &str

Note that repeating patterns are greedy in untwine, so there's no good way to do something like "start" .* "end". If you want to do that, you need to use a greedy match which cannot consume the terminal token.

Excellent, thanks! rest=<.*> fits my needs.


Python 3.8.0

Trying to pip install azure-cognitiveservices-speech on Windows 10 64-bit with Python 3.8.0 64-bit.
Getting:
Could not find a version that satisfies the requirement azure-cognitiveservices-speech (from versions: )
No matching distribution found for azure-cognitiveservices-speech

Thanks for feedback, we are aware of the issue and working on to get support for Python 3.8.0
Please install earlier version of Python 3.7 and let me know if you see still issues.

This is a known issue.
Python  3.8 went public just as we released 1.8 of our SDK.
We looked I to adding support for Python 3.8 in the current release, but at the moment are waiting on Azure hosted agent to support the latest python on all our supported platform, so I do not have an exact ETA for when this will be added, but I hope to see it in the next few releases...

Thank you. I'll monitor the releases and watch for support.


RAT-358: first documentation update

Major documentation addition and update.  Some restructuring to the left hand menu to make navigation easier.
This PR is manually deployed to be seen at:
https://creadur.apache.org/rat017/

If you are happy with the text let's get this merged and start on the reviews.  Unless you want to complete the migration to the new generation system first.

@Claudenw pls have a look if you are okay with my changes. I will push the current version to https://creadur.apache.org/rat017/

LGTM - go ahead an push at your convenience.


Replace JsonFx.NET with Json.NET

Replace JsonFx.NET with Json.NET.

Tests has been updated to support Json.NET

I'm going to run a few more tests.

Nice work, JSON.NET is the way to go. I would merge this PR, but it seems your indentation is mixed up in the files. It is clearly visible if you review the files in GitHub, not VS.

You are right, I will try to cleanup the indentation as well. Will re-open after I have finish testing a few more stuffs, I got an exception last night when it tries to convert object[] to in[]


build: migrate to Starlark Java rules

For now, they are just wrappers around the native rules, but their usage will become manadatory in Bazel 1.0 (issue bazel#8741).

The Starlark rules are just using native.java_* calls so should be backwards compatible through most Bazel versions.


chore(package): update @types/node to version 7.0.27

Closes #69

Codecov Report

Merging #78 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #78   +/-   ##
=======================================
  Coverage   96.55%   96.55%
=======================================
  Files           6        6
  Lines          58       58
  Branches        3        3
=======================================
  Hits           56       56
  Partials        2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update adf405b...20a666a. Read the comment docs.


Improve UG for v1.3

Further improvements to the UG.

Codecov Report

Merging #335 (165ab61) into master (f85ba0c) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #335   +/-   ##
=========================================
  Coverage     71.28%   71.28%
  Complexity      695      695
=========================================
  Files           128      128
  Lines          2302     2302
  Branches        258      258
=========================================
  Hits           1641     1641
  Misses          548      548
  Partials        113      113

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


refactor(projects): refactor projects route with value objects (DSP-1555)

In addition to DSP-1555 implementing escaping characters, which has been already done, we have added the value objects handling for projects route.

looks good.

@subotic all comments reflected in a0d5f97.


THRIFT-1805 Provide option for handling RTEs

Adds a Java option to the generator to generate code which lets Thrift
handle RuntimeExceptions from a service, and present them as
TApplicationException to the client.
This addresses a back-and-forth debate over the expected behavior of
this in Java services, and the resulting breakage, as described in
THRIFT-1805 and related issues, by making the behavior configurable.
Throwable types are not handled, because it is generally ill-advised to
handle Error types within a Java application. So, this only allows
configuration of handling RuntimeException.
This patch preserves current behavior as the default, and works with
previously generated code.
This patch does not affect AsyncProcessFunction exception handling.
Related JIRAs: THRIFT-378, THRIFT-1658, THRIFT-1805, THRIFT-3607

I'm not familiar enough with Thrift's testing framework to add tests, but this works tested locally.
I don't know if this makes sense for other languages, but it seems to make a lot of sense for Java.

My guess is there would be fairly easy ways to unit test this behavior so it doesn't need to be an integration test.  Also, +1 for using the term Mocket.  Love that.

@ctubbsii: Could you please have a look at THRIFT-4239? Not sure if it has sth. to do with your patch, but it came up while looking for the cause, so maybe you do have some input.

@Jens-G I will look and comment on THRIFT-4239.


[RFE] - [RFE for upcoming 2.0] Upload analysis results/report to associated git repo

What is your persona?
I'm an architect tasked with large scale application modernisation engagements.
What is your story / use case?
As an architect, I want to share the analysis report with the development teams, so that the developers have access to the reports from their project repository.
What is your specific intent, not necessarily the feature you want, but what are
you trying to achieve?
If an analysis is done "remotely" (the application is configured to download the code from a git repo for analysis), it would be nice to upload the report into that same git repo (into a configurable directory, on main or automatically creating a tackle-<timestamp> branch.
This would allow developers access to the report without having to access Tackle.
Furthermore, it would also allow for a delta comparison of reports between application versions (since the report is versioned along with the application source).
How has the number of story points changed between 0.0.1 and 0.0.5 would be an easy comparison in git, since developers would now have a versioned report, alongside the code that has been analysed.
How does your immediate need fit into your bigger picture? What's the overall
benefit?

Ease of use (access to the report directly from your repo)
Version controlled "snapshots" of reports associated with their code versions
Ability to compare / visualise progress over time (comparison of reports across branches/versions/tags)

Do you have a suggestion for implementation?
Since Tackle 2.0 will incorporate the git/svn download/clone of code, "uploading" aka adding/committing the reports shouldn't be too much of an effort, I hope.
Additionally, to make diffs/comparisons easier, maybe a simple text representation of the report dashboard would be nice, so a git compare would not show the diff of the generated HTML but rather changes in the key metrics (number of issues, story points, percentage in different areas, etc.)
Additionally, an analysis config text file containing the parameters added to the uploaded report would be necessary, so developers would see which analysis was actually performed and would not compare results of two different analysis targets.

@PhilipCattanach we should start looking into enabling the Windup CLI to provide an overview of the execution results aside from the reports themselves. A set of high level metrics should be fine, maybe collocated with the HTML reports in a JSON or YAML file. This would enable both what Markus is requesting and the analysis history/burndown charts feature we were discussing some time ago. @jortel what do you think?


Bug: SNARK Jobs tab has fees in Nanomina

Closed by https://github.com/Granola-Team/mina-block-explorer/pull/484


Auto detect system glyph ranges by system lang

This looks much better in my opinion 👍
Just add that fallthrough and if pipeline runs, I think this could be merged.

Just add that fallthrough and if pipeline runs
Can I go into more detail?


Organize project

If you log in as a user that doesn’t exist the app crashes.

When you log into your newly activated account you immediately get an error as it takes you to http://<IP_ADDRESS>:8000/app. Though, if you change the url to http://<IP_ADDRESS>:8000/ this works fine.

From http://<IP_ADDRESS>:8000/authors/  if you click on an author you get an error when you try to view that author’s page: http://<IP_ADDRESS>:8000/authors/5ec32f5e-8e5e-495f-b38e-3cf340d18b2c

@arbitraryconstants I fixed the bugs you reported but I could not reproduce the first bug you mentioned relating to logging in with a user that does not exist.

Yeah, neither can I.

I can reproduce the first bug I mentioned either... so I guess either it's fixed or it never was...


chore(main): release 0.37.0

:robot: I have created a release beep boop
0.37.0 (2023-11-24)
Features

Pipelines: add CTA when pipeline screen is empty (#484) (7688f8e)

Bug Fixes

Connections: hide s3 connection secret key (#482) (38b9924)
PipelineDialog: prevent display of empty value on picker (#481) (435118b)

Miscellaneous

release please bump policy change (#485) (e9151c1)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/BLSQ/openhexa-frontend/releases/tag/0.37.0 :sunflower:


[Forwardport ] Generate and Publish sha256sum.txt in RKE Releases

Forwardport for https://github.com/rancher/rancher/issues/46677

RKE branch for 1.7 does not exist yet. Changes will get forward ported from 1.6 as and when the branch gets created, since its already merged there. So does not need to be handled separately.


How to set in proguard-rules ?

help me.

this library needs no proguard rules. you can use library without progaurd


If I build a standard HTML table and 'hard code' the table data .. then TableEdit works and gives me edit and delete buttons on all rows.
However - when I load data from mysql json and build rows in javascript  ... there are no edit and delete buttons.
It seems that the TableEdit "ondraw()" function completes before the data is retrieved and loaded in the table.
Can someone please help to give me the right way to load data from DB ... an get TableEdit to add "editable" buttons.
Help will be very much appreciated.

OK - so I got it figured.
You do a  $.ajax to get the table data and only on ".done" do you call TableEdit().


Added PerfectBugReport.io under others

Link to resource: PerfectBugReport.io
A simple checklist of the essential items to include in bug reports.

LGTM. :)
Be sure to visit web-dev-resources.com and check if everything is ok. Thanks!


clean instance value extraction

When extracting the instance label values, there is no need to use the
regex, instead use the label_values function, which is cleaner and
safer.
This patch change versions 2.1, 2.2, 2018.1 and master.
Fixes #351

@amnonh please check the legend


fix polish grammar

@eLGi  Please approve, thanks :)

PR #162 is correct. Typically polish typos.

👍 Thanks @jsciga for the PR.


feat: add support for watermeter

Closes #14
Add support for Homewizard watermeter
Since the total_liter_m3 does not match the actual value on the analog meter, but rather the cumulative value since installation, the value can be corrected by setting the offset argument.

:tada: This PR is included in version 1.3.0-alpha.1 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


⚠ Cioos Pacific CKAN Sitemap has degraded performance

In b724a28, Cioos Pacific CKAN Sitemap (https://catalogue.cioospacific.ca/sitemap/sitemap-1.xml) experienced degraded performance:

HTTP code: 200
Response time: 7152 ms

Resolved: Cioos Pacific CKAN Sitemap performance has improved in 68092d9 after 36 minutes.


Uploading large files:  files get "stuck" partially uploaded and never complete

Preflight Checklist

[X] I have installed the latest version of Storage Explorer.
[X] I have checked existing resources, including the troubleshooting guide and the release notes.
[X] I have searched for similar issues.

Storage Explorer Version
1.35.0
Regression From
1.19.1
Architecture
i86
Storage Explorer Build Number
No response
Platform
Windows
OS Version
Windows 11
Bug Description
When uploading large files, the file gets "stuck" after partially uploading.  Progress never goes beyond a certain point and the file never finishes uploading.  In my most recent scenario the file was around 750 GB, but I have seen this sporadically in the past for files as small as 1 GB.
The issue seems likely related to the use of OAuth for authentication, which was added somewhat recently.  If I roll back to version 1.19.1, it uses Active Directory authentication, and I don't see the issue.
The repro is 100% with the 750 GB file.
I've attached an AzCopy log file from the failure case
9978b3ad-4aa4-3944-7646-d12ca3c1bcdc.log
Steps to Reproduce

Launch Azure Storage Explorer
Connect to blob container storage using OAuth
Upload a 750 GB file to blob storage container

Actual Experience
The upload partially completes then gets "stuck" and never makes any further progress.
Expected Experience
File should upload completely
Additional Context
No response

Can you also share your Storage Explorer app logs? How long does it take for the hang to occur?
Can't seem to find them right now, but we have a few other issues other users have opened related to this.
We have an ongoing discussion with AzCopy developers about an authentication question. Essentially, AzCopy clients are expected to insert an authentication token in the system credential store, and it reads the token to do its work. For long-running operations, clients also need to refresh the token. As far as we can tell, Storage Explorer does this as expected. What remains to be determined is whether AzCopy is checking the credential store for refreshed tokens at regular intervals.

Here are the app logs.  I don't know exactly how long it takes for the hang to occur, but it seems like you can compute it from the AzCopy log.  It seems to be under an hour from rough observation.
AppLogs.zip

Yeah, that sounds about right. Tokens are normally good for about an hour before they need to be refreshed. So, it definitely sounds like AzCopy is not getting the refreshed token.
One way to test this is by running AzCopy directly yourself. You can copy the command Storage Explorer uses from the link in the Activity Log. If AzCopy can't complete on its own, it isn't doing something right.

I ran the AzCopy command manually and I think I can confirm the authentication problem.  Here's the last part of the AzCopy log:
2024/09/25 19:05:28 PERF: primary performance constraint is Disk. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  1, W:  0, F:  0, B:  1, E:  0, T:  2, GRs:  5
2024/09/25 19:05:28 10.7 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 134.1982 (Disk may be limiting speed)
2024/09/25 19:05:28 ==> REQUEST/RESPONSE (Try=1/306.3515ms, OpTime=469.0319ms) -- RESPONSE SUCCESSFULLY RECEIVED
PUT https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blockid=MDAwMDBsRMGle8NE2GLVLVZQMZkSMDAwMDAwMDAwMDA0OTUy&comp=block
X-Ms-Request-Id: [5f0b258a-c01e-0111-037d-0f3e52000000]
2024/09/25 19:05:29 ==> REQUEST/RESPONSE (Try=1/290.7196ms, OpTime=564.8591ms) -- RESPONSE SUCCESSFULLY RECEIVED
PUT https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blockid=MDAwMDBsRMGle8NE2GLVLVZQMZkSMDAwMDAwMDAwMDA0OTUz&comp=block
X-Ms-Request-Id: [5f0b2750-c01e-0111-157d-0f3e52000000]
2024/09/25 19:05:30 PERF: primary performance constraint is Disk. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  1, W:  1, F:  0, B:  0, E:  0, T:  2, GRs:  5
2024/09/25 19:05:30 10.7 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 66.8998 (Disk may be limiting speed)
2024/09/25 19:05:31 ==> REQUEST/RESPONSE (Try=1/228.61ms, OpTime=525.8384ms) -- RESPONSE SUCCESSFULLY RECEIVED
PUT https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blockid=MDAwMDBsRMGle8NE2GLVLVZQMZkSMDAwMDAwMDAwMDA0OTU0&comp=block
X-Ms-Request-Id: [5f0b28e6-c01e-0111-7f7d-0f3e52000000]
2024/09/25 19:05:32 PERF: primary performance constraint is Disk. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  1, W:  0, F:  0, B:  1, E:  0, T:  2, GRs:  5
2024/09/25 19:05:32 10.7 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 129.4926 (Disk may be limiting speed)
2024/09/25 19:05:32 ==> REQUEST/RESPONSE (Try=1/434.9037ms, OpTime=693.251ms) -- RESPONSE SUCCESSFULLY RECEIVED
PUT https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blockid=MDAwMDBsRMGle8NE2GLVLVZQMZkSMDAwMDAwMDAwMDA0OTU1&comp=block
X-Ms-Request-Id: [5f0b2b1a-c01e-0111-777d-0f3e52000000]
2024/09/25 19:05:33 ==> REQUEST/RESPONSE (Try=1/307.093ms, OpTime=551.3149ms) -- RESPONSE SUCCESSFULLY RECEIVED
PUT https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blockid=MDAwMDBsRMGle8NE2GLVLVZQMZkSMDAwMDAwMDAwMDA0OTU2&comp=block
X-Ms-Request-Id: [5f0b2cbd-c01e-0111-797d-0f3e52000000]
2024/09/25 19:05:34 PERF: primary performance constraint is Disk. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  1, W:  0, F:  0, B:  1, E:  0, T:  2, GRs:  5
2024/09/25 19:05:34 10.7 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 71.2911 (Disk may be limiting speed)
2024/09/25 19:05:35 ==> REQUEST/RESPONSE (Try=1/324.4066ms, OpTime=623.817ms) -- RESPONSE SUCCESSFULLY RECEIVED
PUT https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blockid=MDAwMDBsRMGle8NE2GLVLVZQMZkSMDAwMDAwMDAwMDA0OTU3&comp=block
X-Ms-Request-Id: [5f0b2e6a-c01e-0111-707d-0f3e52000000]
2024/09/25 19:05:35 ==> REQUEST/RESPONSE (Try=1/0s, OpTime=62.7333ms) -- RESPONSE STATUS CODE ERROR
PUT https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blockid=MDAwMDBsRMGle8NE2GLVLVZQMZkSMDAwMDAwMDAwMDA0OTU4&comp=block
Accept: application/xml
Authorization: REDACTED
Content-Length: 16777216
Content-Type: application/octet-stream
User-Agent: AzCopy/10.26.0 azsdk-go-azblob/v1.4.0 (go1.22.5; Windows_NT)
X-Ms-Client-Request-Id: d4462e6d-5baf-4df7-6977-1171eaf0106e
x-ms-version: 2023-08-03
RESPONSE Status: 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
Content-Length: 404
Content-Type: application/xml
Date: Wed, 25 Sep 2024 19:05:37 GMT
Server: Microsoft-HTTPAPI/2.0
Www-Authenticate: Bearer authorization_uri=https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/authorize resource_id=https://storage.azure.com
X-Ms-Error-Code: InvalidAuthenticationInfo
X-Ms-Request-Id: 5f0b3046-c01e-0111-327d-0f3e52000000
Response Details: InvalidAuthenticationInfoServer failed to authenticate the request. Please refer to the information in the www-authenticate header. Lifetime validation failed. The token is expired.
2024/09/25 19:05:36 ==> REQUEST/RESPONSE (Try=1/66.8347ms, OpTime=66.8347ms) -- RESPONSE STATUS CODE ERROR
PUT https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blockid=MDAwMDBsRMGle8NE2GLVLVZQMZkSMDAwMDAwMDAwMDA0OTU4&comp=block
Accept: application/xml
Authorization: REDACTED
Content-Length: 16777216
Content-Type: application/octet-stream
User-Agent: AzCopy/10.26.0 azsdk-go-azblob/v1.4.0 (go1.22.5; Windows_NT)
X-Ms-Client-Request-Id: d4462e6d-5baf-4df7-6977-1171eaf0106e
x-ms-version: 2023-08-03
RESPONSE Status: 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
Content-Length: 404
Content-Type: application/xml
Date: Wed, 25 Sep 2024 19:05:37 GMT
Server: Microsoft-HTTPAPI/2.0
Www-Authenticate: Bearer authorization_uri=https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/authorize resource_id=https://storage.azure.com
X-Ms-Error-Code: InvalidAuthenticationInfo
X-Ms-Request-Id: 0ebdb9b9-301e-0047-6a7d-0f31b0000000
Response Details: InvalidAuthenticationInfoServer failed to authenticate the request. Please refer to the information in the www-authenticate header. Lifetime validation failed. The token is expired.
2024/09/25 19:05:36 INFO: [P#0-T#0] Early close of chunk in singleChunkReader with context still active
2024/09/25 19:05:36 ERR: [P#0-T#0] UPLOADFAILED: \?\Z:\downsample_text_bilinear.hdf5 : 401 : 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.. When Staging block. X-Ms-Request-Id: 0ebdb9b9-301e-0047-6a7d-0f31b0000000
Dst: https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload/test2/downsample_text_bilinear.hdf5
2024/09/25 19:05:36 PERF: primary performance constraint is Disk. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  1, W:  0, F:  0, B:  0, E:  0, T:  1, GRs:  5
2024/09/25 19:05:36 10.7 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 130.1875 (Disk may be limiting speed)
2024/09/25 19:05:37 ==> REQUEST/RESPONSE (Try=1/10.7782ms, OpTime=76.9684ms) -- RESPONSE STATUS CODE ERROR
GET https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blocklisttype=all&comp=blocklist
Accept: application/xml
Authorization: REDACTED
User-Agent: AzCopy/10.26.0 azsdk-go-azblob/v1.4.0 (go1.22.5; Windows_NT)
X-Ms-Client-Request-Id: 795c7943-17e5-4c71-62f9-e46512d1b621
x-ms-version: 2023-08-03
RESPONSE Status: 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
Content-Length: 404
Content-Type: application/xml
Date: Wed, 25 Sep 2024 19:05:39 GMT
Server: Microsoft-HTTPAPI/2.0
Vary: Origin
Www-Authenticate: Bearer authorization_uri=https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/authorize resource_id=https://storage.azure.com
X-Ms-Error-Code: InvalidAuthenticationInfo
X-Ms-Request-Id: dc134f59-401e-0000-597d-0f5aeb000000
Response Details: InvalidAuthenticationInfoServer failed to authenticate the request. Please refer to the information in the www-authenticate header. Lifetime validation failed. The token is expired.
2024/09/25 19:05:37 ==> REQUEST/RESPONSE (Try=1/502.9µs, OpTime=502.9µs) -- RESPONSE STATUS CODE ERROR
GET https://gfxmltrainingstore1.blob.core.windows.net/cadmus/TestLargeUpload%2Ftest2%2Fdownsample_text_bilinear.hdf5?blocklisttype=all&comp=blocklist
Accept: application/xml
Authorization: REDACTED
User-Agent: AzCopy/10.26.0 azsdk-go-azblob/v1.4.0 (go1.22.5; Windows_NT)
X-Ms-Client-Request-Id: 795c7943-17e5-4c71-62f9-e46512d1b621
x-ms-version: 2023-08-03
RESPONSE Status: 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
Content-Length: 404
Content-Type: application/xml
Date: Wed, 25 Sep 2024 19:05:39 GMT
Server: Microsoft-HTTPAPI/2.0
Vary: Origin
Www-Authenticate: Bearer authorization_uri=https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/authorize resource_id=https://storage.azure.com
X-Ms-Error-Code: InvalidAuthenticationInfo
X-Ms-Request-Id: dc134f5f-401e-0000-5e7d-0f5aeb000000
Response Details: InvalidAuthenticationInfoServer failed to authenticate the request. Please refer to the information in the www-authenticate header. Lifetime validation failed. The token is expired.
2024/09/25 19:05:37 JobID=a5c1446c-c37b-d844-62d5-2d5650319912, Part#=0, TransfersDone=1 of 1
2024/09/25 19:05:37 all parts of entire Job a5c1446c-c37b-d844-62d5-2d5650319912 successfully completed, cancelled or paused
2024/09/25 19:05:37 is part of Job which 1 total number of parts done
2024/09/25 19:05:38 PERF: primary performance constraint is Unknown. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  0, GRs:  5
2024/09/25 19:05:38 0.0 %, 0 Done, 1 Failed, 0 Pending, 0 Skipped, 1 Total,
2024/09/25 19:05:38
Diagnostic stats:
IOPS: 1
End-to-end ms per request: 669
Network Errors: 0.00%
Server Busy: 0.00%
Job a5c1446c-c37b-d844-62d5-2d5650319912 summary
Elapsed Time (Minutes): 71.5707
Number of File Transfers: 1
Number of Folder Property Transfers: 0
Number of Symlink Transfers: 0
Total Number of Transfers: 1
Number of File Transfers Completed: 0
Number of Folder Transfers Completed: 0
Number of File Transfers Failed: 1
Number of Folder Transfers Failed: 0
Number of File Transfers Skipped: 0
Number of Folder Transfers Skipped: 0
Total Number of Bytes Transferred: 0
Final Job Status: Failed
2024/09/25 19:05:38 Closing Log

Yep, error message says the token is expired. So I highly suspect that's why the job is hanging. I'm pretty sure something in AzCopy isn't checking for a refreshed token. I'll keep reaching out to the AzCopy team to get something moving.

Any update on this?  Our team would really benefit from a fix as we use Azure Storage for all our ML training data.

We have no significant updates. The AzCopy team is aware of the issue and are currently looking into it.

My team is facing the same issue with large files in a major customer project. Can you point me to AzCopy issue related to this bug?
To some extent we were able to circumvent this OAuth expiration issue by using SAS-tokens for authentication - but unfortunately Storage Explorer appears to default to SAS-token authentication only if user has Storage account level Microsoft.Storage/storageAccounts/listkeys/action permission - something we can't grant to out end users. User can create SAS token manually without that Storage account level permission and use the token to do large transfers successfully - but instructing this manual method to all end users is not doable. Maybe storage explorer could default to sas-tokens even without said listkeys permission?

We've been using version 1.19.1 as a workaround for now:  https://github.com/microsoft/AzureStorageExplorer/releases/tag/V1.19.1

@jarkko-siitonen Keep in mind there are two different kinds of SAS: key-based SAS and user-delegated SAS. Storage Explorer supports both. So you do not need to give out List Keys permissions. If the user has permissions to generate user-delegated SAS, Storage Explorer should be able to use that method instead.
@JasonYeMSFT Correct me, if I'm wrong?

@craxal Yes, but how do you make Storage explorer to actually use SAS tokens without
a) generating the SAS token manually and connecting the explorer to Shared access signature URL
b) granting the listkeys action
My testing seems to indicate Storage explorer passes AZCOPY_CRED_TYPE = "OAuthToken" parameter to AzCopy unless either of those 2 is true. Would be delighted if there was 3rd way to make Storage Exlorer use SAS.

@jarkko-siitonen What Craig said is right. Storage Explorer can generate user delegation SAS. But it can only do so after you are signed in because it needs your Azure AD credential. Storage Explorer is implemented to prefer user delegation SAS because it is more flexible. If your user can already sign in, get an OAuth token and use it to do data transfer, there is a good chance that you can take advantage of user delegation SAS to achieve easier access control. If you can tell us more about your scenario we may better help you by giving recommendations, for example

How do your users connect to their storage in Storage Explorer
How do you distribute permissions to your users
What are the operations your users need
Any additional requirements

@JasonYeMSFT Short description: As a fragment of larger project, we are replacing a legacy sftp server with Azure storage account(s) to enforce Entra ID authentication with MFA and Geoblocking. The service is used to transfer files with sizes in hundreds of GB, thus transfer times can be quite long (more than OAuth token lifetime).
How do your users connect to their storage in Storage Explorer:

Users access the storage using Azure Storage Explorer and/or Azure Portal
** Storage Explorer is more complex than your average sftp client, but not too much so.
** Using Storage Explorer version 1.19.1 might work, but launching a new service which requires 2021 version of the client software sounds scary
The users connect to storage accounts using Entra Id authentication
There are both member user accounts and guest user accounts

What are the operations your users need

There are some users with upload and download permission (read+write+delete files in a blob container)
Some users have download permission only (read files)
There are several blob containers
** one user might have write access to several containers
** another user might have only read permission to one or two of them

How do you distribute permissions to your users

Users are created (or invited in case of guest account) to Entra ID and assigned to security groups
Each storage accounts has two groups, one with read+write permission, one with read permission only
Read+write users get now Storage Account Contributor Role
** For user with contributor role Storage explorer uses account key-based SAS in uploads and downloads
** This solved the upload problems but forced us to place each blob container in separate storage account
Read only users are assigned Storage Blob Data Reader role (or a custom role if that helps)
** For these users Storage Explorer download uses OAuth authentication - and transfers fail after 60-90 minutes
** Attempts to solve this with custom role did not work out - as soon as explorer started using account key-based SAS the users also got undesired write permission
** The read only users can generate User delegated SAS manually using Storage Explorer tool
** If user connects storage explorer to self generated User delegated SAS url, transfer succeeds
*** Unfortunately, this adds a lot of extra complexity to end user
*** Users with sftp mindset, try to download directly without generating and mounting a User delegated SAS first - and transfer fails

Is the AzCopy team just aware of the problem or is fixing this bug on the roadmap?

@jarkko-siitonen They are actively working on a fix for the issue. They've given us an estimated release date in January.
@JasonYeMSFT

For user with contributor role Storage explorer uses account key-based SAS in uploads and downloads

To my knowledge, assuming they also have permission to create user-delegated SAS, this should not be happening. Any idea why this would?

@jarkko-siitonen As Craig has said, AzCopy is working on a fix and once it's done, direct OAuth access token based transfer should no longer have any issues. However, you might be able to stop relying on account key access today in Storage Explorer.
For your read-only case, I think you should keep giving your users Storage Blob Data Reader role. For your read-write case, have you considered giving your users Storage Blob Data Contributor role instead and turn off the storage account key access for your storage accounts?  For users with the Storage Blob Data Contributor role, Storage Explorer can generate a user delegation SAS for them which can have a lifetime much longer than a regular OAuth access token (up to 7 days). It has a maximum lifetime though so it would become an issue if your data transfer takes more time to complete. You can try setting this up for a new test storage account and upload a blob to see how it works. We would like to help you stop retaining storage account key access for your users. If you run into any issues, please let us know and we can help.

Thank you @craxal  and @JasonYeMSFT for the bug fix release date indication. It's a lot easier to decide what to do next, when there is a clear understanding, the bug is actively worked on!
Seven days maximum duration for OAuth file transfers should not be an issue for us.
And yes, we would indeed prefer not to give storage account key access to users. However, a new test did not yield any better results for us than it did before. Guest user has Storage Blob Data Contributor role on container level and Reader role for storage account level, with additional custom role containing Generate a user delegation key permission on storage account level. The user can generate User delegated SAS manually, but Storage Explorer upload, download and delete functions all use OAuth authentication - which would be just fine for us, if this token renewal bug didn’t exist.
If this Storage Blob Data Contributor behavior is not the expected/desired outcome with Storage Blob Data Contributor role, maybe I should create a new issue for Storage Explorer team to inspect?

I read the code and I found that our upload code doesn't use user delegation SAS because it doesn't have the limitation of service to service copy where the source and destination cannot use different OAuth credentials. I added this issue to tracking adding this user delegation SAS to our upload/download code https://github.com/microsoft/AzureStorageExplorer/issues/8299