Dask with jobqueue not using multiple nodes

I am trying to use Dask to do parallel processing on multiple nodes on supercomputing resources - yet the Dask-distributed map only takes advantage of one of the nodes. Note that I put this up on stackoverflow  but didn't get attention so now I'm giving here a go.
Here is a test script I am using to set up the client and perform a simple operation:
import time
from distributed import Client
from dask_jobqueue import SLURMCluster
from socket import gethostname

def slow_increment(x):
    time.sleep(10)
    return [x + 1, gethostname(), time.time()]

cluster = SLURMCluster(
    queue='somequeue',
    cores=2,
    memory='128GB',
    project='someproject',
    walltime='00:05:00',
    job_extra=['-o myjob.%j.%N.out',
               '-e myjob.%j.%N.error'],
    env_extra=['export I_MPI_FABRICS=dapl',
               'source activate dask-jobqueue'])

cluster.scale(2)

client = Client(cluster)

A = client.map(slow_increment, range(8))
B = client.gather(A)

print(client)

for res in B:
    print(res)

client.close()

And here is the output:
<Client: scheduler='tcp://someip' processes=2 cores=4>
[1, 'bdw-0478',<PHONE_NUMBER>.6744401]
[2, 'bdw-0478',<PHONE_NUMBER>.67487]
[3, 'bdw-0478',<PHONE_NUMBER>.68666]
[4, 'bdw-0478',<PHONE_NUMBER>.6879778]
[5, 'bdw-0478',<PHONE_NUMBER>.6986163]
[6, 'bdw-0478',<PHONE_NUMBER>.6997452]
[7, 'bdw-0478',<PHONE_NUMBER>.7100565]
[8, 'bdw-0478',<PHONE_NUMBER>.711296]

While printing out the client info indicates that Dask has the correct number of nodes (processes) and tasks per node (cores), the socket.gethostname() output and time-stamps indicate that the second node isn't used. I do know that dask-jobqueue successfully requested two nodes, and that both jobs complete at the same time. I tried using different MPI Fabrics for inter- and intra-node communication (e.g. tcp, shm:tcp, shm:ofa, ofa, ofi, dapl) but this did not change the result. I also tried removing the "export I_MPI_FABRICS" command and using the "interface" option, but this caused the code to hang.
Thanks in advance for any assistance.
-Noah

Transferred.  Also gave all org members with the ability to delete issues, which I hope implies the ability to move them.

Thank you for taking an interest in my issue!
@guillaumeeb, below I'm including short descriptions of my investigations into your various debugging suggestions and after the full code, it's output and stderror.

Dask does not use MPI for communication, only plain TCP, so either ethernet interfaces, or IP over IB. Did you try without any fabrics, and without interface option ?

When I don't explicitly define the fabrics and don't include an interface option the code runs similarly as before.

You should print(client) before the map just to be sure, but I imagine the cluster was formed before the end of the gather, especially if jobs ended at the same time.

To check this I printed the client before the gather operation and it appears to have formed properly.

Did you try opening the Dask Dashboard to watch the computation?

I haven't yet figured out how do to this, though I'm looking into it next.

Do you have something interesting in Slurm stdout or stderr?

The stdout is empty but there are details in stderr.

You can also look at http://jobqueue.dask.org/en/latest/debug.html to see if there is something relevant. Thinking about debug mode for instance.

I have enabled debugging mode and included the output below. Some of the other suggestions seem to deal with getting the jobs themselves through SLURM, though I don't seem to be having a problem in that regard.
Here is the updated version of my code:
import time
import logging
from distributed import Client
from dask_jobqueue import SLURMCluster
from socket import gethostname

logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

def slow_increment(x):
    time.sleep(10)
    return [x + 1, gethostname(), time.time()]

cluster = SLURMCluster(
    queue='bdwall',
    cores=2,
    memory='128GB',
    project='STARTUP-NPAULSON',
    walltime='00:05:00',
    job_extra=['-o myjob.%j.%N.out',
               '-e myjob.%j.%N.error'])

cluster.scale(2)

client = Client(cluster)

A = client.map(slow_increment, range(8))

time.sleep(15)
print('\n::: start user output :::')
print(client)
print('::: end user output :::\n')

B = client.gather(A)

print('\n::: start user output :::')
print(client)
print('::: end user output :::\n')

for res in B:
    print(res)

client.close()

Here is the output of my code with logging enabled:
DEBUG:Using selector: EpollSelector
DEBUG:Using selector: EpollSelector
DEBUG:Using selector: EpollSelector
DEBUG:Job script:
 #!/bin/bash

#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -p bdwall
#SBATCH -A STARTUP-NPAULSON
#SBATCH -n 1
#SBATCH --cpus-per-task=2
#SBATCH --mem=120G
#SBATCH -t 00:05:00
#SBATCH -o myjob.%j.%N.out
#SBATCH -e myjob.%j.%N.error
JOB_ID=${SLURM_JOB_ID%;*}

/home/npaulson/miniconda/envs/dask-jobqueue/bin/python -m distributed.cli.dask_worker tcp://<IP_ADDRESS>:38499 --nthreads 2 --memory-limit 128.00GB --name dask-worker--${JOB_ID}-- --death-timeout 60

DEBUG:Found 0 active/pending workers.
DEBUG:Scaling up to 2 workers.
DEBUG:Using selector: EpollSelector
DEBUG:Found 0 active/pending workers.
DEBUG:starting 2 workers
DEBUG:Using selector: EpollSelector
DEBUG:writing job script:
#!/bin/bash

#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -p bdwall
#SBATCH -A STARTUP-NPAULSON
#SBATCH -n 1
#SBATCH --cpus-per-task=2
#SBATCH --mem=120G
#SBATCH -t 00:05:00
#SBATCH -o myjob.%j.%N.out
#SBATCH -e myjob.%j.%N.error
JOB_ID=${SLURM_JOB_ID%;*}

/home/npaulson/miniconda/envs/dask-jobqueue/bin/python -m distributed.cli.dask_worker tcp://<IP_ADDRESS>:38499 --nthreads 2 --memory-limit 128.00GB --name dask-worker--${JOB_ID}-- --death-timeout 60

DEBUG:Executing the following command to command line
sbatch --parsable /tmp/tmpl5nkitoz.sh
DEBUG:started job: 802327
DEBUG:writing job script:
#!/bin/bash

#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -p bdwall
#SBATCH -A STARTUP-NPAULSON
#SBATCH -n 1
#SBATCH --cpus-per-task=2
#SBATCH --mem=120G
#SBATCH -t 00:05:00
#SBATCH -o myjob.%j.%N.out
#SBATCH -e myjob.%j.%N.error
JOB_ID=${SLURM_JOB_ID%;*}

/home/npaulson/miniconda/envs/dask-jobqueue/bin/python -m distributed.cli.dask_worker tcp://<IP_ADDRESS>:38499 --nthreads 2 --memory-limit 128.00GB --name dask-worker--${JOB_ID}-- --death-timeout 60

DEBUG:Executing the following command to command line
sbatch --parsable /tmp/tmp_8knxogn.sh
DEBUG:started job: 802328
DEBUG:adding worker tcp://<IP_ADDRESS>:32853
DEBUG:job id for new worker: 802328
DEBUG:802328 is a new job or restarting worker
DEBUG:802328 is a new job, adding to running_jobs
DEBUG:adding worker tcp://<IP_ADDRESS>:33998
DEBUG:job id for new worker: 802327
DEBUG:802327 is a new job or restarting worker
DEBUG:802327 is a new job, adding to running_jobs

::: start user output :::
<Client: scheduler='tcp://<IP_ADDRESS>:38499' processes=2 cores=4>
::: end user output :::

::: start user output :::
<Client: scheduler='tcp://<IP_ADDRESS>:38499' processes=2 cores=4>
::: end user output :::

[1, 'bdw-0285',<PHONE_NUMBER>.1772363]
[2, 'bdw-0285',<PHONE_NUMBER>.1834495]
[3, 'bdw-0285',<PHONE_NUMBER>.187123]
[4, 'bdw-0285',<PHONE_NUMBER>.1882021]
[5, 'bdw-0285',<PHONE_NUMBER>.198275]
[6, 'bdw-0285',<PHONE_NUMBER>.199521]
[7, 'bdw-0285',<PHONE_NUMBER>.209588]
[8, 'bdw-0285',<PHONE_NUMBER>.210915]

Here is the stderror for node 00048:
Warning: Your terminal does not set locales.

If you use unicode text inputs for command line options then this may cause
undesired behavior.  This is rare.

If you don't use unicode characters in command line options then you can safely
ignore this message.  This is the common case.

You can support unicode inputs by specifying encoding environment variables,
though exact solutions may depend on your system:

    $ export LC_ALL=C.UTF-8
    $ export LANG=C.UTF-8

For more information see: http://click.pocoo.org/5/python3/

distributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:40144'
distributed.diskutils - INFO - Found stale lock file and directory '/blues/gpfs/home/npaulson/2018_10_24_dask/dask-worker-space/worker-bqgulcd7', purging
distributed.worker - INFO -       Start worker at:  tcp://<IP_ADDRESS>:33998
distributed.worker - INFO -          Listening to:  tcp://<IP_ADDRESS>:33998
distributed.worker - INFO -              nanny at:        <IP_ADDRESS>:40144
distributed.worker - INFO -              bokeh at:        <IP_ADDRESS>:36591
distributed.worker - INFO - Waiting to connect to:   tcp://<IP_ADDRESS>:38499
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          2
distributed.worker - INFO -                Memory:                  128.00 GB
distributed.worker - INFO -       Local Directory: /blues/gpfs/home/npaulson/2018_10_24_dask/dask-worker-space/worker-si3_68ly
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to:   tcp://<IP_ADDRESS>:38499
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Stopping worker at tcp://<IP_ADDRESS>:33998
distributed.worker - INFO - Connection to scheduler broken.  Reconnecting...
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO - Closing Nanny at 'tcp://<IP_ADDRESS>:40144'
distributed.dask_worker - INFO - End worker

Here is the stderror for node 00285:
Warning: Your terminal does not set locales.

If you use unicode text inputs for command line options then this may cause
undesired behavior.  This is rare.

If you don't use unicode characters in command line options then you can safely
ignore this message.  This is the common case.

You can support unicode inputs by specifying encoding environment variables,
though exact solutions may depend on your system:

    $ export LC_ALL=C.UTF-8
    $ export LANG=C.UTF-8

For more information see: http://click.pocoo.org/5/python3/

distributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:43857'
distributed.worker - INFO -       Start worker at:   tcp://<IP_ADDRESS>:32853
distributed.worker - INFO -          Listening to:   tcp://<IP_ADDRESS>:32853
distributed.worker - INFO -              nanny at:         <IP_ADDRESS>:43857
distributed.worker - INFO -              bokeh at:         <IP_ADDRESS>:43568
distributed.worker - INFO - Waiting to connect to:   tcp://<IP_ADDRESS>:38499
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          2
distributed.worker - INFO -                Memory:                  128.00 GB
distributed.worker - INFO -       Local Directory: /blues/gpfs/home/npaulson/2018_10_24_dask/dask-worker-space/worker-bqgulcd7
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -         Registered to:   tcp://<IP_ADDRESS>:38499
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Stopping worker at tcp://<IP_ADDRESS>:32853
distributed.diskutils - ERROR - Failed to remove '/blues/gpfs/home/npaulson/2018_10_24_dask/dask-worker-space/worker-bqgulcd7' (failed in <built-in function lstat>): [Errno 2] No such file or directory: '/blues/gpfs/home/npaulson/2018_10_24_dask/dask-worker-space/worker-bqgulcd7'
distributed.worker - INFO - Connection to scheduler broken.  Reconnecting...
distributed.worker - INFO - -------------------------------------------------
distributed.nanny - INFO - Closing Nanny at 'tcp://<IP_ADDRESS>:43857'
distributed.dask_worker - INFO - End worker

I've tried accessing the dashboard, which sometimes populates with the workers but other times does not. When it does populate with the workers, it only shows two - which I assume are the workers associated with the single node that reports back.

My guess is that once the first worker arrives it gets all of the work, and that this work finishes well before the second worker arrives a subsecond later.  Maybe try waiting until all of the workers have arrived?

@mrocklin Yes that would make sense. However the workers all spawn and run nearly instantly and at the same time.
Still when submitting long running jobs they still all get submitted to the same node and run on the same PID.
Sleeping between scale and submit seems to solves this. I assume my idea of the scheduler is wrong. The workers all spawn and run well before the first job is processed. And still all of them are pushed to that node without the sleep between scale and submit.

I modified my code to include @mrocklin 's suggestion but still only one of the nodes is doing any work. I verified this on the dashboard by having my task instantiate a large numpy array. On the dashboard I see both nodes being initialized, but processing is only done on one of the nodes.
@guillaumeeb , I also simplified the script so that only one process is started per node.
The updated script is included below:
import time
import logging
import numpy as np
from distributed import Client
from dask_jobqueue import SLURMCluster
from socket import gethostname

logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

def slow_increment(x):
    m = np.random.random((100000, 10000))
    time.sleep(10)
    return [x + 1, gethostname(), time.time()]

cluster = SLURMCluster(
    queue='bdwall',
    cores=1,
    memory='128GB',
    project='BAYES_THERMO',
    walltime='00:05:00',
    job_extra=['-o myjob.%j.%N.out',
               '-e myjob.%j.%N.error'])

cluster.scale(2)

client = Client(cluster)

A = client.map(slow_increment, range(8))

time.sleep(10)

B = client.gather(A)

print('\n::: start user output :::')
print(client)
print('::: end user output :::\n')

for res in B:
    print(res)

client.close()

Again, if there are more than eight threads in your first worker that
worker will get all of the tasks and probably won't share with the others
when they come on.
On Wed, Dec 12, 2018 at 11:39 AM Noah Paulson<EMAIL_ADDRESS>wrote:

I modified my code to include @mrocklin https://github.com/mrocklin 's
suggestion but still only one of the nodes is doing any work. I verified
this on the dashboard by having my task instantiate a large numpy array. On
the dashboard I see both nodes being initialized, but processing is only
done on one of the nodes.
@guillaumeeb https://github.com/guillaumeeb , I also simplified the
script so that only one process is started per node.
The updated script is included below:
import time
import logging
import numpy as np
from distributed import Client
from dask_jobqueue import SLURMCluster
from socket import gethostname
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
def slow_increment(x):
m = np.random.random((100000, 10000))
time.sleep(10)
return [x + 1, gethostname(), time.time()]
cluster = SLURMCluster(
queue='bdwall',
cores=1,
memory='128GB',
project='BAYES_THERMO',
walltime='00:05:00',
job_extra=['-o myjob.%j.%N.out',
'-e myjob.%j.%N.error'])
cluster.scale(2)
client = Client(cluster)
A = client.map(slow_increment, range(8))
time.sleep(10)
B = client.gather(A)
print('\n::: start user output :::')
print(client)
print('::: end user output :::\n')
for res in B:
print(res)
client.close()
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask-jobqueue/issues/206#issuecomment-446654857,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AASszDOVWX8DyWqjwIt_qnW8qELj9GzAks5u4TE8gaJpZM4ZBsKj
.

There are two different situations here.
@goraj why do you want your tasks to be submitted to different workers? I think @mrocklin comments are for you. There are other solutions if you want only one task per node than using sleep and waiting for all the cluster to be up.
@npaulson I still don't see where the problem can be. I will try to execute your code, but I don't expect to run on the same issue. There should be scheduling error if no tasks can be sent to one of your worker. @mrocklin is there a way to see this kind of errors?

@guillaumeeb so I have hundreds of very long running jobs that are not thread-safe.
I also want each to run on its own cluster node and use all available cpus there.
Unfortunately I cannot figure out how to get each job to one cluster node using dask-jobqueue without using the wait like so:
while use_cluster and (client.status == 'running') and (len(client.scheduler_info()['workers']) < n_workers):
            time.sleep(1.0)

It is likely that my mental model is wrong and I would highly appreciate any tips or changes I should consider.  Thank you all for the help and sorry for hijacking this issue, I thought I had the same problem as @npaulson.

I have hundreds of very long running jobs that are not thread-safe.
I also want each to run on its own cluster node and use all available cpus there.

That's what I imagined. Please look at #181. Seems like we should definitly document this!

@guillaumeeb Thank you very much!

@npaulson any update on this? I tried your simple scrit and as expected everything looks fine:
::: start user output :::
<Client: scheduler='tcp://<IP_ADDRESS>:46417' processes=2 cores=2>
::: end user output :::

[1, 'node117.sis.cnes.fr',<PHONE_NUMBER>.2696798]
[2, 'node118.sis.cnes.fr',<PHONE_NUMBER>.313745]
[3, 'node117.sis.cnes.fr',<PHONE_NUMBER>.3178716]
[4, 'node118.sis.cnes.fr',<PHONE_NUMBER>.4473453]
[5, 'node117.sis.cnes.fr',<PHONE_NUMBER>.3758643]
[6, 'node118.sis.cnes.fr',<PHONE_NUMBER>.4892852]
[7, 'node117.sis.cnes.fr',<PHONE_NUMBER>.4334419]
[8, 'node118.sis.cnes.fr',<PHONE_NUMBER>.56492]

There's probably something weird with your network or system configuration. I cannot understand why the scheduler cannot address one of the worker.

@guillaumeeb Thank you for running the script - your outputs look perfectly fine to me as well.
I did try specifying various interfaces as a keyword in SLURMCluster from a list of interface options I obtained from a node on my cluster, but these either gave a key error:
Traceback (most recent call last):
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 259, in <module>
    go()
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 255, in go
    main()
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/distributed/cli/dask_worker.py", line 206, in main
    host = get_ip_interface(interface)
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/distributed/utils.py", line 166, in get_ip_interface
    for info in psutil.net_if_addrs()[ifname]:
KeyError: 'ib1'

or this error:
Traceback (most recent call last):
  File "jobqueue_test1.py", line 25, in <module>
    '-e myjob.%j.%N.error'])
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/dask_jobqueue/slurm.py", line 69, in __init__
    super(SLURMCluster, self).__init__(**kwargs)
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/dask_jobqueue/core.py", line 223, in __init__
    kwargs.setdefault('ip', get_ip_interface(interface))
  File "/home/npaulson/miniconda/envs/dask-jobqueue/lib/python3.6/site-packages/distributed/utils.py", line 169, in get_ip_interface
    raise ValueError("interface %r doesn't have an IPv4 address" % (ifname,))
ValueError: interface 'ib0' doesn't have an IPv4 address

@guillaumeeb I recreated my script in an interactive job and it is now appearing to use both nodes
@lesteve Your last suggestion also works! I guess my script wasn't giving SLURM enough time to spin up the nodes I requested?

Well, that's great! Guess we can finally close this one?

Yes, thank you both for your help!

@lesteve Your last suggestion also works! I guess my script wasn't giving SLURM enough time to spin up the nodes I requested?

I guess, this is what was happening indeed.

This just cost me 4 hours of debugging since #304 and #230 are similar but completely unhelpful and show up high on Google, with lots of other unrelated noise. Plus with distributed's #3741 it's not easy to figure out what's actually going on. Shouldn't cluster.scale() do the waiting for you? At least the docs could be a little bit more helpful.

I don't think this should be closed as long as cluster.scale() doesn't wait for the workers to spin up.
This is super hard to debug for someone new to dask, since the first assumption is that the scheduling works differently than expected (which wouldn't be surprising as the documentation is a bit terse).
Currently, there is another closed (!) issue (#304) probably due to the same issue, which doesn't contain a resolution but contains a misleading statement regarding scale():

I'll recommand to use scale instead of start_worker, but mainly to make sure all the workers are started before submitting the tasks.

client.scale submit jobs to your job scheduler (SLURM, PBS, etc ...) and indeed does not wait for jobs to start, you can use client.wait_for_workers if you want to wait before some (or all) of your jobs actually start running.
You are probably right that this could be made more clear in the documentation. I invite you to open a PR to improve the documentation.

I understand that now, but is that really sensible behavior? The API looks analogous to the standard library's multiprocessing, i.e.
p = multiprocessing.Pool(num_workers)
p.map(...)

... but that assumption is going to waste you a day, sort of invalidating the whole idea of using dask as a clean wrapper around the shellscript hell of classic job schedulers.
I could see how you might want this behavior if you are starting thousands of workers and want to start submitting before they are all ready, but even then the scheduling is going to be unpredictable and may well not be what you want.
If you want to preserve this possibility, a good solution might be to have a wait_for_workers=False keyword argument to scale().

Thank you for your interest in the project @fabiansvara .  If you'd like the community to change this behavior then I encourage you to raise a new issue at https://github.com/dask/distributed/issues/new . The scale behavior is shared among all Dask deployment projects (hpc, cloud, kubernetes, yarn) and so isn't a decision that will be made in the dask-jobqueue repository.

I understand that now, but is that really sensible behavior?
.. but that assumption is going to waste you a day, sort of invalidating the whole idea of using dask as a clean wrapper around the shellscript hell of classic job schedulers

It sounds like you've had a frustrating experience, for which I apologize.  Even with that though I encourage you to keep a positive tone.  Most people who work on Dask are volunteers and they read dozens of messages a day (sometimes hundreds).  Keeping a positive tone can make their volunteer activities a lot more pleasant, which reduces OSS maintainer fatigue.

Apologies, I have edited my message to be more neutral and to the point.

Thanks a lot for editing your mesage! Of course we all get frustrated and we all wish things were simpler and documentation made the caveat we bumped into more obvious but yeah maintainer fatigue is a real thing. On this topic, I really like Setting expectations for open source participation by Brett Cannon.
About Dask-Jobqueue in general and the change you are proposing, I don't think waiting for all jobs to start after .start is an acceptable default. How long your job takese to start depends on lot on your cluster (also you may be billed for hours when your job does nothing). In general I feel that in most cases people are fine for the computation to start before all the jobs start and then continue on the Dask workers as they connect to the scheduler (Dask scheduler does this automatically). If that's not the case for you (or if you hit issues with this approach) I would be interested to hear more about your use case.
I am not an expert on this but it feels like Dask-MPI has more an approach where you get all your workers running at the same time so maybe that would be something to investigate for you (being familiar with MPI first would probably be a prerequisite)

My use case is pretty simple: Every job (CNN inference) needs to run on a different cluster node, since they need a GPU and each node has exactly one. I know how many SLURM jobs (dask workers) I can request without having to wait. SLURM starts all the jobs within less than a second.
However, when calling map right after scale, only one worker ever gets any jobs.
As it is currently, doing anything other than waiting after calling scale is basically guaranteed to be incorrect. Even if you didn't want to wait for all the workers to be ready, you'd have to wait in a loop to monitor the number of active workers and submit the jobs in manually generated batches - basically rolling your own scheduling for the initial start-up phase.
An alternative to having scale wait for worker startup could be for map to schedule jobs to workers that are expected to start up soon and / or to reschedule jobs from busy workers to idle workers.

It sounds like you either want to use client.wait_for_workers or dask-mpi,
as @lesteve suggested.
On Sun, Jun 21, 2020 at 4:53 PM fabiansvara<EMAIL_ADDRESS>wrote:

My use case is pretty simple: Every job (CNN inference) needs to run on a
different cluster node, since they need a GPU and each node has exactly
one. I know how many SLURM jobs (dask workers) I can request without having
to wait. SLURM starts all the jobs within less than a second.
However, when calling map right after scale, only one worker ever gets
any jobs.
As it is currently, doing anything other than waiting after calling scale
is basically guaranteed to be incorrect. Even if you didn't want to wait
for all the workers to be ready, you'd have to wait in a loop to
monitor the number of active workers and submit the jobs in manually
generated batches - basically rolling your own scheduling for the initial
start-up phase.
An alternative to having scale wait for worker startup could be for map
to schedule jobs to workers that are expected to start up soon and / or to
reschedule jobs from busy workers to idle workers.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask-jobqueue/issues/206#issuecomment-647198193,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTB23ZLJ4UAU7UA2TMLRX2MN5ANCNFSM4GIGYKRQ
.

(CNN inference)

You may be interested to look at: https://examples.dask.org/machine-learning/torch-prediction.html. It is with a LocalCluster but easy to translate to a SLURMCluster.

I know how many SLURM jobs (dask workers) I can request without having to wait.

I would say in a typical HPC cluster this is rather unusual but I may be biased by the HPC clusters I am familiar with ... generally you have no idea how long you have to wait for your one job to start (it can depend on how people are using the cluster, on your past usage and probably on other factors) and even less for many jobs to start.

reschedule jobs from busy workers to idle workers.

Work stealing does that in principle automatically for you. If all your jobs do start running (i.e. squeue shows them running) as you say, this does point towards work stealing not kicking in your use case, hard to tell why but maybe the doc can help.

I also think dask-mpi is what you want for your use case, the workers will start instantly and you can tweak your SLURM job requirements to give you the distribution you want (1 MPI task per node would mean one worker per node, no need for a map step)

Work stealing does that in principle automatically for you. If all your jobs do start running (i.e. squeue shows them running) as you say, this does point towards work stealing not kicking in your use case, hard to tell why but maybe the doc can help.

If that worked it would make the whole issue moot. According to the work stealing docs:

If a task has been specifically restricted to run on particular workers (such as is the case when special hardware is required) then we do not steal.

... depending on how exactly I interpret this sentence, it might explain what's going on. I instantiate the SLURMCluster with extra=['--resources GPU=1'] and call map with resources={'GPU': 1}. This is because the GPU nodes also have a lot of CPUs, so they could also be used for running multiple CPU-bound jobs in other cases.
If I interpret the sentence from the doc to mean "jobs where resources were specified will not be stolen", then that's the problem. Here in my case, it would be perfectly fine for other workers that have the same resource available to steal the jobs.

If I interpret the sentence from the doc to mean "jobs where resources were specified will not be stolen", then that's the problem.

This was indeed a limitation but a PR supposed to fix this (https://github.com/dask/distributed/pull/3069) has been merged. It seems like this is in 2.12 so maybe check that you distributed version is recent enough.
If that still does not work for you this could be a problem with the PR or work stealing that does not kick in for some reason in your particular case ... hard to tell ...
And of course the distributed doc may also need to be fixed as well ...

Hm, I have 2.18.0. Anything I can do to narrow down what's going on?

Yeah that was my guess that your version was recent enough.
could you first post a snippet to show how you are creating your cluster and using it. In an ideal world a stand-alone snippet reproducing the problem would be great.
Here are a list of suggestions at this point:

can you make sure that without resource work-stealing does happen
do you need to actually use resource? For embarassingly parallel use cases, I would think that this is not a requirement but maybe I am missing something.
how to debug work stealing in general I don't know there may be some debugging options or a dashboard that can help. Searching in the distributed issue tracker or asking there would be a good idea.

Finally found some time to experiment with this.
Here's my complete testing code:
from dask_jobqueue import SLURMCluster
from dask.distributed import Client
import time
import itertools as it

def _worker(worker_id):
    print(f'{worker_id=} starting.')
    time.sleep(10)
    print(f'{worker_id=} done.')

def dask_map(wait=True, with_resource=True, cores=20, workers=5):
    if with_resource:
        extra = ['--resources GPU=1']
        resources = {'GPU': 1}
    else:
        extra = []
        resources = {}

    cluster = SLURMCluster(
        cores=cores,
        processes=1,
        memory='188GB',
        walltime='24:00:00',
        queue='p.gpu',
        shebang='#!/bin/bash -l',
        local_directory='/my/local/dir/',
        extra=extra,
        job_extra=['--gres=gpu:1', ],
        env_extra=[
            'module load cuda',
            'module load anaconda',
            'conda init bash',
            'conda activate /my/conda/env',])
    cluster.scale(workers)
    client = Client(cluster)

    if wait:
        client.wait_for_workers(workers)
    f = client.map(_worker, list(range(workers)), resources=resources)
    [xx.result() for xx in f]

    cluster.scale(n=0, jobs=0)
    cluster.close()

if __name__ == '__main__':
    for wait, with_resource, cores in it.product([False, True], [True, False], [20, 1]):
        t0 = time.time()
        dask_map(wait=wait, with_resource=with_resource, cores=cores)
        total_t = time.time() - t0
        print(f'{wait=}, {with_resource=}, {cores=} took {total_t:.2f} s')

This prints the following:
wait=False, with_resource=True, cores=20 took 55.11 s
wait=False, with_resource=True, cores=1 took 14.09 s
wait=False, with_resource=False, cores=20 took 13.84 s
wait=False, with_resource=False, cores=1 took 14.29 s
wait=True, with_resource=True, cores=20 took 15.28 s
wait=True, with_resource=True, cores=1 took 14.45 s
wait=True, with_resource=False, cores=20 took 14.58 s
wait=True, with_resource=False, cores=1 took 14.43 s

(the individual lines are separated by
distributed.client - ERROR - Failed to reconnect to scheduler after 10.00 seconds, closing client
_GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=CancelledError()>
asyncio.exceptions.CancelledError

... I removed that from the output above for clarity. This seems to be caused be the creation / closing of the cluster in the loop; it doesn't influence the results, which are identical when this is not run in a loop.)
Anyway, as you can see, the issue only occurs in the wait=False, with_resource=True, cores=20 case, where the jobs run sequentially. The workers are all there, but only one of them is doing anything. This is what it looks like in the dashboard:

Thanks a lot for doing this! So it looks like work-stealing with resources does happen when cores=1, which is good to confirm.
The case where it does not work is when you have cores=20. I am not an expert on the work stealing logic, but I can only guess that work-stealing does not kick in because the scheduler sees 20 slots on the first node (one slot for each core) with 5 tasks to execute so work stealing does not seem worth it. It could be interesting to confirm this hypothesis by submitting many more short tasks and see whether work stealing happens. It could also be worth getting some inputs by @mrocklin to make sure my hypothesis seems reasonable to him ...
Maybe this is something that can be improved in distributed, not sure. I guess it should be possible to create a reproducer with distributed.SSHCluster so that more people are likely to look at it.

I looked into the documentation but couldn't find out how to create an SSHCluster with resources, not sure I can be of any help there. Should I create an issue specifically about work stealing on the distributed tracker?

@appassionate We created a library that does what you are asking for but I admit it requires quite a bit of configuration since you need to tell it about the system and how you launch MPI jobs there. The library is at https://github.com/E-CAM/jobqueue_features and there's a tutorial at https://github.com/E-CAM/jobqueue_features_workshop_materials (and you can find a recording of the tutorial at https://www.youtube.com/watch?v=FpMua8iJeTk&ab_channel=E-CAM).
I haven't touched it in a few months, need to check if our CI is still passing.

@appassionate We created a library that does what you are asking for but I admit it requires quite a bit of configuration since you need to tell it about the system and how you launch MPI jobs there.

@ocaisa  waow cool! Many thanks!!

@appassionate, with your comment, it is not really clear to me what you're trying to achieve.
Anyway, if @ocaisa answer suits you, this is perfect, if not, I encourage you to open a new issue and try to make your issue a bit clearer.

@appassionate, with your comment, it is not really clear to me what you're trying to achieve.
Anyway, if @ocaisa answer suits you, this is perfect, if not, I encourage you to open a new issue and try to make your issue a bit clearer.
Thanks for your suggestion! jobqueue_features have customized the "SlurmCluster" for some MPI using, i believe there will be some using in such as "more nodes" in slurm which is suitable for me.


fmSetupAssistant: negated boolean negated twice

A Boolean preference, gets compared to its bit value twice. It should only be compared once.
For normal boolean values this is (unnecessary, but) not a problem, but by inverted boolean values (where the preference value is the inverted GUI-value) the not-inversion gets applied twice and thus is annulled.
Fix: Apply inversion only once.

Implemented


Better fix VC14 compilation

When I tried to compile PHP7 without --disable-redis-igbinary I ran into another fatal error. I took a good look at stdbool.h on Windows. VC12 and VC14 have their own stdbool.h that does all that is needed.So I put this in front of the original code:
# if _MSC_VER >= 1900
#  include <stdbool.h>
# else

And a #endif at the end. 1900 is the version number of VC14. I also changed the indentation a little bit to make clear what belongs to what.

Merged as 93496f7a85a676205d7bab53046a1003109fa212 (I fixed the line endings in ig_win32.h so the diff was a tiny bit different)
Thank you so much for putting all this work into this, I will try to have all the tests passing really soon (at 40% ! )


Some improvements to TileSetScenesCollectionSource docs

Adds some clarifications to TileSetScenesCollectionSource description regarding how scenes are indexed. Also adds a code example how to get scene ID from a tile, because it's not obvious.
Closes #73645

Still insane to me that one would have to follow that relatively verbose example because the API is not exactly accessible.

It can be reduced to single (less safe) line, but it's not exactly friendly:
tile_set.get_source(tile_map_layer.get_cell_source_id(cell)).get_scene_tile_scene(tile_map_layer.get_cell_alternative_tile(cell))

We could add a method similar to get_cell_tile_data(), which is only relevant to TileSetAtlasSource, but there is no high demand for it.

Strictly speaking the fact that it takes so many operations to get the "starting information" that you need is insane. It's not just for getting Scene Collections, it's for getting information on tiles in general. I guess it's also a byproduct of cell data being split in 3 like this.

Thanks!


[numerics] rework NEWTONMIN solver

use newton_LSA
understand and document the C-function used -> it does not correspond to the classical min-based alg
many tests are failing, understand why.

transferred in https://github.com/siconos/siconos/wiki/Wish-list-and-enhancement


Error Creating Observable From MediaQueryList

Bug Report
Current Behavior
Creating an Observable from a MediaQueryList throws an Error in Safari.

The new version of the MediaQueryList interface inherits methods from its parent interface, EventTarget.

In Safari MediaQueryList does not inherit from EventTarget.
The problem is this line of code, which calls addListener with the wrong arity for a MediaQueryList.
https://github.com/ReactiveX/rxjs/blob/cd5895d0e44008714ccd702af959ae40c8262f16/src/internal/observable/fromEvent.ts#L222
Reproduction

REPL or Repo link:
(you can use https://stackblitz.com/ to create one to attach here)

const mediaQuery = matchMedia('screen and (min-width: 768px)');
const mediaObserver = fromEvent<MediaQueryListEvent>(mediaQuery, 'change');

Expected behavior
Should correctly create an observable from the MediaQueryList change event.
Environment

Runtime: Safari 12 (latest)
RxJS version: 6.3.3

Possible Solution
Maybe check the length property before calling addListener? Another option might be to make the event name option.
// In Safari MediaQueryList.addListener.length is 0, which is wrong but less than 2
// In Chrome and FireFox MediaQueryList.addListener.length is 1, which is right
// In Node EventEmitter.addListener.length is 2
// 🤷♂
if (sourceObj.addListener.length < 2) {
  sourceObj.addListener(handler as NodeEventHandler);
} else {
  sourceObj.addListener(eventName, handler as NodeEventHandler);
}

// or
if (!eventName) {
  sourceObj.addListener(handler as NodeEventHandler);
}

// then
const mediaObserver = fromEvent<MediaQueryListEvent>(mediaQuery);

So per https://developer.mozilla.org/en-US/docs/Web/API/MediaQueryList, it's safari & IE behavior doesn't inherit eventTarget, right? meanwhile fromEvent explicitly require eventTarget-like object to be passed.
I feel in this case using fromEventPattern seems more feasible, instead of trying to add polymorphic behavior into fromEvent - afaik we don't expect to have known list of object to work in fromEvent but only accepts satisfying object, and recommend fromEventPattern for other object.

Didn't know about fromEventPattern. I'm ok with this.
const mediaObserver = fromEventPattern<MediaQueryListEvent>(
  mediaQuery.addListener.bind(mediaQuery),
  mediaQuery.removeListener.bind(mediaQuery)
);

Thanks @kwonoj!


suspending task makes timer and fps counter inaccurate

just as stated in the issue title. I guess it calculates all those parameters since the computation start time or a big time period? This is NOT critical, just a note on the side.

I recognized this problem when I developed Suspend feature, but left the issue unaddressed.
I fixed it now.


Bump abacus

Signed-off-by: Vasil Petkov<EMAIL_ADDRESS>

Coverage remained the same at 91.864% when pulling cea659a7b5bd828d7fc14f85e403577365d07725 on SAP:bump into c27104549c0cb20d3282cda67e78b951c6bce6c9 on cloudfoundry-incubator:master.


Confusing 'Show Command Actions...' text in command decoration hover

Testing #143433
The 'Show Command Actions...' text that appears on hovering a terminal command decoration seems to suggest a context menu of some sort. Should I be able to click on this somehow?

Miguel had this feedback too
@Tyriar I thought you had resolved this?

/duplicate https://github.com/microsoft/vscode/issues/143703


v1.1.50 CLI not fully working

[x] I have searched open and closed issues for duplicates
[x] I am submitting a bug report for existing functionality that does not work as intended
[x] I have read https://vigem.org/Community-Support/
[x] This isn't a feature request or a discussion topic

Bug description
Using the command line interface to temporarily hide 6 USB controllers.
The changes are reflected in the HidHide Configuration Client interface afterwards but not in (for example) the Windows gamepad control panel. So the changes aren't fully enacted.
But, when I change the status of any controller in the HidHide Configuration Client after the steps above, then all changes made through the CLI do also show up in the gamepad control panel.
Changes made directly through the HidHide Configuration Client work perfectly fine. And Windows/the gamepad control panel are not allowed through the cloack
Screenshots

Machine info

x64,
Windows version:** Windows 10 20H2
Software/driver version(s):** 1.1.50

Careful! Hiding the USB instance only works for XInput devices! For your general purpose joystick, pad, wheel etc. you need to block the HID-Instance, e.g. HID\VID_054C&PID_0268
Cheers

Thank you for the clarification. Using the HID-Instance it works like a charm!


BDM/Coldfire support

I have come across a Freescale chip that uses BDM, instead of JTAG, which I wasn't familiar with since before. I noticed OpenOCD has it on its todo-list but has yet to produce anything.
Would it make sense to have this supported by Hydrabus?

Hi !
The BDM communication could be easily added to Hydrabus (according to http://www.nxp.com/files-static/training_pdf/27642_HCS08_BDM_ICE_WBT.pdf). However, I couldn't find any opensource tool that uses a serial interface for the BDM probe. It would therefore be difficult to interface it with GDB right away.
But still, creating a flash/dump script using BBIO commands in this mode should be doable.
Would you be able to help on testing this ?

It appears that the chip I am looking at is locked, so at the moment it would prove difficult as I don't have a board to test on. I will see if I can get my hands on one for debugging purposes.

@Baldanos So I've started writing up a script with BBIO commands. But with BDM being asynchronous between host and target clock, timing is everything.
E.g for transmitting a logical 0/1, if the target is running at 8 MHz, I need to initiate communication with a negative flank and then set the BKGD HIGH/LOW within 8 cycles (=1 us). Is this possible with BBIO commands?
All according to:

@csholmq This protocol looks a bit similar to 1-wire, except the clock is not fixed. There is currently no way to use this protocol with Hydrabus, but it doesn't look too difficult to implement.
I think the best way to start would be to implement the SYNC command, and use the calculated clock to program a timer. That timer can then be used to implement a read_bit and write_bit primitive, then write_byte and read_byte.
Take a look at src/hydrabus/hydrabus_mode_onewire.c for an example of that kind of protocol.
I'd be glad to help if you want to implement this on Hydrabus.

Any news on this subject as so far it seems no anyone have the hardware (or hardware and time to work on it) to do test and implement it ?

Issue closed as there is no new to support that specific Target


Already fixed by #3802


CollectionView's dequeueReusableSupplementaryView(ofKind:for:viewType) not available

Hello :)
I'm trying to use the UICollectionView's dequeueReusableSupplementaryView(ofKind:for:viewType) function but I struggle with the viewType parameter.
From the function definition, the viewType parameter must be a UICollectionReusableView, and also a Reusable.
This first snippet compiles correctly:
let headerType = TitleHeaderView.self
      let view = collectionView.dequeueReusableSupplementaryView(ofKind: kind, for: indexPath, viewType: headerType)

But this next one doesn't compiles and gives me a Argument 'viewType' must precede argument 'for' error.
//This func is responsible for returning the correct header class according to the given indexPath
func headerType() -> (UICollectionReusableView & Reusable).Type { ... }

let headerType = headerType()
let view = collectionView.dequeueReusableSupplementaryView(ofKind: kind, for: indexPath, viewType: headerType)

As I understand it, this error comes up because my headerType doesn't have the func's required type, and the compilator think I want to use the classic dequeue func.
But isn't a UICollectionReusableView & Reusable a correct type for dequeueReusableSupplementaryView(ofKind:for:viewType) ?
What type should my headerType func return in order to work?
Reusable version: 4.0.2 (imported with cocoapods)
Xcode 9.3
App project in Swift 4.1
Reusable project in Swift 3.3

Hello
I see two potential reasons for the error you see

You pass viewType: headerType but headerType is a function returning your type, and you didn't call it. Didn't you want to use viewType: headerType() instead, to call the function so that it returns the expected (UICollectionReusableView & Reusable).Type?

Sadly I'm not sure if the current version of Swift is capable of going to such levels of abstraction just yet. Tbh I'm actually quite surprised that (UICollectionReusableView & Reusable).Type would compile (meaning that Swift would somehow reference a metatype built dynamically from two protocols? reminds me of similar limitations of current version of Swift like not being able to make a meta type conform to a protocol, and other limitations like the ones forcing us to use type erasure, etc)

I think if you make your headerType() function's return type be concrete type instead of (UICollectionReusableView & Reusable).Type it would work. e.g. if you have class CommonRV: UICollectionReusableView, Reusable {} then use class MyRV1: CommonRV & class MyRV2: CommonRV in your code, you could have func headerType() -> CommonRV.Type { /* return MyRV1.Type or MyRV2.Type depending on some condition */ } and I think in that case it would work.
tl;dr I think what make Swift compiler unhappy here is that (UICollectionReusableView & Reusable).Type is maybe a little too complex for it to grasp, being a meta-type of a combination of a class and a protocol, not sure it's able to handle that just yet.
If that test confirms my hypothesis n°2, then your solution is either to do what I described above (make all your concrete classes of UICollectionReusableView that you want to use with Reusable be a subclass of class CommonRV: UICollectionReusableView, Reusable and not subclasses of UICollectionReusableView directly), or if you don't want to use inheritance, maybe we could find a workaround solution using type erasure? I'm not sure

The first potential reason you point is in fact a typo. In my example, the var and the func have the same name, but it's not the case in my code. I'll update my post to better reflect that.
About your second potential reason, I've tried to simplify like that:
typealias MyHeader = UICollectionReusableView & Reusable

//This func is responsible for returning the correct header class according to the given indexPath
func retrieveHeaderType() -> MyHeader.Type { ... }

But I've got the same issue as the typealias doesn't change anything from the compiler pov.
I've also thought about having all my headers inherit from MyHeader but I wonder if it will correctly dequeue a MyHeader or its subclass... I'll try.

if you have class CommonRV: UICollectionReusableView, Reusable {} then use class MyRV1: CommonRV & class MyRV2: CommonRV in your code, you could have func headerType() -> CommonRV.Type { /* return MyRV1.Type or MyRV2.Type depending on some condition */ } and I think in that case it would work.

It indeed compiles and work as expected :). Thanks!

Thanks for the follow up!
(And I hope a future version of Swift will allow such composed metatypes construct!)


Custom data type with Generics

Feature Request
It would be really cool if it were possible to create custom data types with support for generics.
I am thinking of something like this
import pydantic
from pydantic import BaseModel

class Foo(BaseModel):
    bar: Custom[int]

Currently I extended the ModelMetaclass to do some custom type parsing (and basically transform the Custom[int] into a class Custom with Custom.dtype == int), but it would be nice if there were a less hacky way to do this.

I suspect this is already possible, but I'm not entirely clear what you're looking for.

Ok maybe it will become more clear with the actual thing I want to get done.
I want to build a pydantic Model for specifying a hyperparameter space of a machine learning model. One "dimension" type of that space can be a categorical hyperparameter that consists of a list of values and prior for drawing each value from the list.
Ideally I would like to write the pydantic Model like so
import pydantic
from pydantic import BaseModel

class HSpace(BaseModel):
    num_layers: Categorical[int]

where
HSpace(num_layers=[1,2,5,10])

validates that each value in the list is of type int and initializes my custom type Categorical with these values and a uniform prior.
I could of course write a custom data type for each data type (like IntCategorical and FloatCategorical) but I would much rather leave this up to the user to decide what kind of type this categorical dimension has.
Does that make any sense?

For completion's sake the way I currently do this is by extending the BaseModel and giving it a new Metaclass that parses the Categorical[int] annotation and dynamically creates an IntCategorical class out of it (that pydantic then handles normally). I was just wondering if there is a less hacky way to get this done.

Currently if you try to use a custom generic class, pydantic will say that fields of this type are not supported.
E.g.
from typing import Generic
from typing import TypeVar

from pydantic import BaseModel

T = TypeVar("T")

class Categorical(Generic[T]):
    pass

class MyModel(BaseModel):
    foo: Categorical[int]

gives
Traceback (most recent call last):
  File "<input>", line 11, in <module>
  File "/Users/timon/Library/Caches/pypoetry/virtualenvs/hspace-py3.7/lib/python3.7/site-packages/pydantic/main.py", line 197, in __new__
    config=config,
  File "/Users/timon/Library/Caches/pypoetry/virtualenvs/hspace-py3.7/lib/python3.7/site-packages/pydantic/fields.py", line 267, in infer
    field_info=field_info,
  File "/Users/timon/Library/Caches/pypoetry/virtualenvs/hspace-py3.7/lib/python3.7/site-packages/pydantic/fields.py", line 236, in __init__
    self.prepare()
  File "/Users/timon/Library/Caches/pypoetry/virtualenvs/hspace-py3.7/lib/python3.7/site-packages/pydantic/fields.py", line 301, in prepare
    self._type_analysis()
  File "/Users/timon/Library/Caches/pypoetry/virtualenvs/hspace-py3.7/lib/python3.7/site-packages/pydantic/fields.py", line 383, in _type_analysis
    raise TypeError(f'Fields of type "{origin}" are not supported.')
TypeError: Fields of type "<class '__main__.Categorical'>" are not supported.

Maybe one could introduce an interface for custom generics, e.g. sth along the lines
class CustomGeneric:
    @abstractmethod
    @classmethod
    def substitute_type(type_args: Tuple[Any]) -> Any:
        raise NotImplementedError

class ModelField(Representation):
    def _type_analysis(self) -> None:
        ...
        elif issubclass(origin, CustomeGeneric):
            self.type_ =  origin.substitute_type(self.type_.__args__)
        ...

which would then allow you to implement a custom generic type class (e.g. like so
class Categorical(CustomGeneric, Generic[T]):
    @classmethod
    def substitute_type(type_args: Tuple[Any]) -> Any:
        """Returns a type customized by the parameter of the annotation."
        type_arg = type_args[0]
        # do some stuff and return a new type
        return new_class(
             to_title_case(get_class_name(type_arg) + "Categorical"),
            (Categorical,),
            {},
            lambda ns: ns.update({"dtype": type_arg]}),
        )

You have thought much more about the typing system and probably know much better if this makes any sense at all, so curious to hear your thoughts. :)

@timonbimon Have you looked at GenericModel? I think this might cover what you are trying to do.
Docs here

oh wow, sorry for not finding this myself.
Thanks a lot for the help and merry christmas 🎄:)


Improve documentation

Current documentation is somewhat terse
Guide/instructions for contributors

@shon , Whilst the documentation is a valid thing for contributors to work on, I believe that Contributing.md should specifically be written by the author to avoid confusions. Thoughts? :)

@sachinkamath you are right. This was for myself.


Error about reference index

my command was
bamsnap -bam ./NA12878.final.cram -pos chr11:12314 -out ./test.png
It worked well but suddenly it didn't work with error below.
Traceback (most recent call last):
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/sylee/.local/lib/python3.6/site-packages/bamsnap/bamsnap.py", line 227, in run_process_drawplot_bamlist
refseq = rseq.get_refseq(pos1)
File "/home/sylee/.local/lib/python3.6/site-packages/bamsnap/bamsnap.py", line 535, in get_refseq
refseq = self.get_refseq_from_ucsc(pos1)
File "/home/sylee/.local/lib/python3.6/site-packages/bamsnap/bamsnap.py", line 575, in get_refseq_from_ucsc
refseq[gpos] = seq[i]
IndexError: string index out of range
how can I fix this problem?

I am facing the same issue as the OP. When I run the command
bamsnap -bam ./BAMs/NA18-281_CID21-7767_B21-613.BAM_ETC-RUO.bam -pos 1:816687-818057 -out ./snapshots -save_image_only
I get an error as following:
Process proc 1:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/Users/ankit/Desktop/ETCR-BMO-206/venv/lib/python3.8/site-packages/bamsnap/bamsnap.py", line 235, in run_process_drawplot_bamlist
imagefname = bsplot.drawplot_bamlist(pos1, image_w, bamlist, xscale, refseq)
File "/Users/ankit/Desktop/ETCR-BMO-206/venv/lib/python3.8/site-packages/bamsnap/bamsnap.py", line 473, in drawplot_bamlist
ia_sub = self.get_bamplot_image(bam, pos1, image_w, xscale, refseq)
File "/Users/ankit/Desktop/ETCR-BMO-206/venv/lib/python3.8/site-packages/bamsnap/bamsnap.py", line 357, in get_bamplot_image
rset.calculate_readmap(is_strand_group=True)
File "/Users/ankit/Desktop/ETCR-BMO-206/venv/lib/python3.8/site-packages/bamsnap/drawreadset.py", line 300, in calculate_readmap
for a in self.samAlign.fetch(self.chrom, self.g_spos-self.read_gap_w-500, self.g_epos+500):
File "pysam/libcalignmentfile.pyx", line 1091, in pysam.libcalignmentfile.AlignmentFile.fetch
File "pysam/libchtslib.pyx", line 685, in pysam.libchtslib.HTSFile.parse_region
ValueError: invalid contig 1
2021-11-17 13:42:43,666 : [INFO] Total running time: 0.8 sec
Additionally, while using bed files to give regions for snapshots using the command:
bamsnap -bam BAMs/NA18-281_CID21-7767_B21-613.BAM_ETC-RUO.bam -bed regions.bed -out snapshots
I get the error as follows:
Traceback (most recent call last):
File "/Users/ankit/Desktop/ETCR-BMO-206/venv/bin/bamsnap", line 8, in
sys.exit(cli())
File "/Users/ankit/Desktop/ETCR-BMO-206/venv/lib/python3.8/site-packages/bamsnap/init.py", line 7, in cli
opt = get_options()
File "/Users/ankit/Desktop/ETCR-BMO-206/venv/lib/python3.8/site-packages/bamsnap/_options.py", line 209, in get_options
check_option(opt)
File "/Users/ankit/Desktop/ETCR-BMO-206/venv/lib/python3.8/site-packages/bamsnap/_options.py", line 165, in check_option
opt['poslist'] = set_pos_list(opt)
File "/Users/ankit/Desktop/ETCR-BMO-206/venv/lib/python3.8/site-packages/bamsnap/_options.py", line 131, in set_pos_list
p1['t_pos'] = int(arr[1])
IndexError: list index out of range
I have tried both the dockerized container and running locally by installing bamsnap from the pypi. I get the same errors for both the cases. Any information/help regarding the issue will be helpful.
Thanks.

Apparently this issue was caused by the bed file I was using as input. The formatting of bed file was not appropriate and when the lines were split based on tabs (in the _options.py file line 127), the function wasn't able to split the lines properly. By reformatting the bed files, I was able to resolve this issue. Once again, thanks for this awesome tool! :)


Adapt dialog for round screens (Android wear)

[x] I have verified there are no duplicate active or recent bugs, questions, or requests
[x] I have verified that I am using the latest version of Material Dialogs.
[x] I have given my issue a non-generic title.

Include the following:

Material Dialogs version: <IP_ADDRESS>
Device OS version: 6.0.1
Device Manufacturer: Huawei
Device Name: Watch

Is it possible to adapt the dialogs for round screens? I am trying to use the dialog in my huawei watch, but the dialog goes out of the screen. Is there a way, to shrink the dialog, or maybe make it round? As much i know you own huawei watch too, so maybe there is something possible to make about it? 👍 Thanks in advance!

That's a little past the scope of this library. The stock dialogs look fine on Android Wear devices though.

Well, i don't think so. The stock dialog looks like Holo style. But when i use appcompat theme, the dialog goes off the screen (like your material dialog too). Cause of it, i thought, maybe it would be nice to have nice material dialog for round screens too(i know, the rect screen with stock dialog using appcompat theme looks good. But thanks anyway.


How to create an app with python?

I want to create a flask app to get access to dex.
I tried different libraries. The last one was Flask-Dance.
After Authorization i got the error MissingTokenError which means that the URL parameter access_token is missing.
Does dex doesn't provides this token?
There are 4 different OAuth2 methods. Does dex support all 4 or just one which Flask Dance doesn't handle? :(
Does anyone know how to implement a client with python?

It was a proxy fail...


Add "conflict" option with old http-interop/http-middleware

This PR fixes #5 by stating that middleware/utils 0.13+ should not be used if "http-interop/http-middleware" is used.
This will force a fallback to older versions of middleware/utils.

Fixed issue in previous commit.
First commit was failing because http-server-middleware replaces http-middleware for version >=0.5.
Hence, this was causing a conflict.
https://github.com/http-interop/http-server-middleware/blob/master/composer.json#L25-L27

Thank you!


[gardening] Fix accidental trailing whitespace.

@swift-ci please smoke test

@swift-ci please test

We have a policy against cluttering commit history and creating spurious diffs by stripping trailing whitespace.  IMO this should never have been merged.


Incorrect SSIS instructions

https://docs.microsoft.com/en-us/sql/integration-services/lesson-1-4-adding-package-configurations?view=sql-server-ver15
The section "To create indirect configuration for the DataTransfer package" contains the following instruction "3. On the SSIS menu, click Package Configurations.".
This appears to be incorrect, there is no such option, I assumed this would be found under Extensions, SSIS.
I have actually found the "Package Configurations Organizer" in the Properties of the Control Flow, Misc, Configurations.
Does the doco need changing or am I doing something strange?
Cheers

@rothja this might be feedback for someone on your team.

@JasonWHowell Thank you. @HamishBrowne Thanks for the feedback. @chugugrace is our main contact for this documentation, and I will assign this question to her. #assign:chugugrace

@HamishBrowne have you converted to package deployment model? instructions before step one as below:
"Check the project's current Deployment Model, and set it to Package Deployment Model, if needed. On the Project menu, click Convert to Package Deployment Model"


Lack of tea available

There are multiple non-coffee drinkers in desperate need of caffeine !

How water on the way!

resolved


[OAP-0.5] orc-core-1.5.2 is compile by java8, but spark 2.1 need support java7 runtime

we dep orc-core-1.5.2 is compile by jdk 1.8.0_121 as following:
Manifest-Version: 1.0
Implementation-Title: ORC Core
Implementation-Version: 1.5.2
Archiver-Version: Plexus Archiver
Built-By: pjayachandran
Specification-Vendor: The Apache Software Foundation
Specification-Title: ORC Core
Implementation-Vendor-Id: org.apache.orc
Implementation-Vendor: The Apache Software Foundation
Created-By: Apache Maven 3.5.3
Build-Jdk: 1.8.0_121
Specification-Version: 1.5
Implementation-URL: http://orc.apache.org/orc-core

we can down load jar form http://central.maven.org/maven2/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar to vertify this.
Oap with spark-2.1 version need support Java7（compile & run）because Spark-2.1.* build by Java7, we should not use third-party dependencies compiled by Java 8 here, so we need downgrade orc version of spark-2.1 profile

[WARNING] one warning found
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/common/type/HiveDecimal.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/serde2/io/HiveDecimalWritable.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/ql/exec/vector/VectorizedRowBatch.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/ql/exec/vector/ColumnVector.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/ql/exec/vector/TimestampColumnVector.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/ql/exec/vector/LongColumnVector.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/ql/exec/vector/DoubleColumnVector.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/ql/exec/vector/BytesColumnVector.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/ql/exec/vector/DecimalColumnVector.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] warning: /home/work/m2/repository/org/apache/orc/orc-core/1.5.2/orc-core-1.5.2-nohive.jar(org/apache/orc/storage/common/type/FastHiveDecimal.class): major version 52 is newer than 51, the highest major version supported by this compiler.
[WARNING]   It is recommended that the compiler be upgraded.
[WARNING] /home/work/oap-src/oap-mine/src/main/java/org/apache/spark/sql/execution/datasources/oap/orc/OrcMapreduceRecordReader.java:47: warning: [rawtypes] found raw type: WritableComparable
[WARNING] public class OrcMapreduceRecordReader<V extends WritableComparable>
[WARNING]                                                 ^
[WARNING]   missing type arguments for generic class WritableComparable<T>
[WARNING]   where T is a type-variable:
[WARNING]     T extends Object declared in interface WritableComparable
[WARNING] /home/work/oap-src/oap-mine/src/main/java/org/apache/spark/sql/execution/datasources/oap/orc/OrcMapreduceRecordReader.java:75: warning: [unchecked] unchecked cast
[WARNING]     this.row = (V) OrcStruct.createValue(schema);
[WARNING]                                         ^
[WARNING]   required: V
[WARNING]   found:    WritableComparable
[WARNING]   where V is a type-variable:
[WARNING]     V extends WritableComparable declared in class OrcMapreduceRecordReader
[WARNING] /home/work/oap-src/oap-mine/src/main/java/org/apache/spark/sql/execution/datasources/oap/orc/IndexedOrcMapreduceRecordReader.java:30: warning: [rawtypes] found raw type: WritableComparable
[WARNING] public class IndexedOrcMapreduceRecordReader<V extends WritableComparable>
[WARNING]                                                        ^
[WARNING]   missing type arguments for generic class WritableComparable<T>
[WARNING]   where T is a type-variable:
[WARNING]     T extends Object declared in interface WritableComparable
[WARNING] 13 warnings
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 59.713 s
[INFO] Finished at: 2018-12-06T11:32:46+08:00
[INFO] Final Memory: 43M/1976M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project oap: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed: An API incompatibility was encountered while executing net.alchim31.maven:scala-maven-plugin:3.2.2:compile: java.lang.UnsupportedClassVersionError: org/apache/orc/storage/ql/exec/vector/VectorizedRowBatch : Unsupported major.minor version 52.0

bytecode version of org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch in orc-core-1.5.2-nohive.jar is 52.0(Java8)


With State & Country Picklists enabled, can't save Contact from Canada

have State & Country picklists enabled
have United States specified as the default country (not sure if this is required to reproduce the bug)
create a new contact, and specify BC, Canada in their picklists
save

Apex trigger djhcumulus2.TDTM_Contact caused an unexpected exception, contact your administrator: djhcumulus2.TDTM_Contact: execution of AfterInsert caused by: System.DmlException: Update failed. First exception on row 0 with id 001G0000022jmK9IAI; first error: FIELD_INTEGRITY_EXCEPTION, The existing country/territory doesn't recognize the state value for field: Billing State/Province Code: [BillingStateCode]: Class.djhcumulus2.TDTM_TriggerHandler.processDMLAllOrNothing: line 231, column 1

Tracking W-017307

if default country is empty, or address management is disabled, this problem will not reproduce.

As per this thread, it may be related this known issue.

on that thread, person from Classy also added some useful info.
https://powerofus.force.com/0D58000003O8TdT

It also appears that you only get the error if you choose a non-default Country that has a defined list of States. For example, if you follow the steps to reproduce above and choose Algeria, you don't get an error. If you choose Canada and British Columbia you get the error. You don't get it though if you just choose Canada and leave mailing State blank.

This Issue was fixed in a previously release, and is no longer reproducible.


Question: redis not replicating "removing conn ... from active"

This project sounds perfect for my 2 server, 2 datacenter situation. Each DC would have 2 racks of 1 node each (no sharding). But I can't get replication to work on a new install. I add keys to either node and they don't show up on the other.

CentOS 7, selinux off, VMs in 1 in-house private datacenter, iptables off
Fresh redis instances, 3.2.11, port 6380. Direct usage proven OK.
redis-cli works OK between each other.
dynamite stats /info doesn't show any non-zero redis stats after trial usage; seems suspicious...

I've tried many variations...

.5.9, .6.3 and dev branches
tokens 0 or<PHONE_NUMBER> or<PHONE_NUMBER>
w/wo env network
0 IPs and private IPs
w/wo gossip
debug mode and not (w/configure&make in between)
2 DCs of 1 rack 1 node each, and 1 DC of 2 racks 1 node each
I've tried with -m 100000 or -d
when I don't use debug mode, the logs hang on "conn_connect:321 <CONN_LOCAL_PEER_SERVER 0x246a070 14 to '<IP_ADDRESS>:8101:rack2:dc:0'> connecting....." and this on the other node "dnode_accept:169 Accepting client connection from <IP_ADDRESS>:56696 on sd 14"

I'm hoping there's just something simple I'm missing. I would really love to use this solution vs a massive IMDG like hazelcast, infinispan or couchbase.
Partial logs from a few attempts:
[2017-12-17 19:04:24.089] conn_pool_notify_conn_close:171 <CONN_POOL 0x1c08e90 active_conn:1 in array 1 max 1> Removing <CONN_SERVER 0x1bf8680 -1 to '<IP_ADDRESS>:6380:1'>
[2017-12-17 19:04:24.089] conn_pool_notify_conn_errored:211 <DATASTORE 0x1bf7df0 <IP_ADDRESS>:6380:1> <CONN_POOL 0x1c08e90 active_conn:0 in array 1 max 1> Scheduling reconnect task after 1 secs
[2017-12-17 19:04:25.091] conn_pool_preconnect:105 <DATASTORE 0x1bf7df0 <IP_ADDRESS>:6380:1> <CONN_POOL 0x1c08e90 active_conn:0 in array 1 max 1> Preconnecting
[2017-12-17 19:04:25.091] _create_missing_connections:52 <DATASTORE 0x1bf7df0 <IP_ADDRESS>:6380:1> <CONN_POOL 0x1c08e90 active_conn:0 in array 1 max 1> created <CONN_SERVER 0x1bf8680 -1 to '<IP_ADDRESS>:6380:1'>
[2017-12-17 19:04:25.091] conn_connect:321 <CONN_SERVER 0x1bf8680 10 to '<IP_ADDRESS>:6380:1'> connecting.....
[2017-12-17 19:04:25.091] server_connected:309 <CONN_SERVER 0x1bf8680 10 to '<IP_ADDRESS>:6380:1'> connected

[2017-12-17 19:13:46.397] _stats_pool_incr:1694 incr field 'dnode_client_connections' to 1
[2017-12-17 19:13:46.397] event_add_conn:207 adding conn <CONN_LOCAL_PEER_CLIENT 0x16e4f30 14 from '<IP_ADDRESS>:38382'> to active
[2017-12-17 19:13:46.397] dnode_accept:211 <CONN_PEER_PROXY 0x16e25c0 12 listening on '<IP_ADDRESS>:8101'> accepted <CONN_LOCAL_PEER_CLIENT 0x16e4f30 14 from '<IP_ADDRESS>:38382'>
[2017-12-17 19:13:46.397] stats_swap:1592 skip swap of current 0x16d79c0 shadow 0x16d7a20 as aggregator is busy
[2017-12-17 19:13:46.397] core_process_messages:615 length of C2G_OutQ : 0
[2017-12-17 19:13:46.397] event_wait:264 epoll 0004 triggered on conn 0x16e4f30
[2017-12-17 19:13:46.397] core_core:508 event FF00 on <CONN_LOCAL_PEER_CLIENT 0x16e4f30 14 from '<IP_ADDRESS>:38382'>

[2017-12-17 19:19:16.195] gossip_loop:797 Gossip is running ...
[2017-12-17 19:19:16.405] stats_aggregate:743 skip aggregate of shadow to sum as generator is slow
[2017-12-17 19:19:16.416] stats_swap:1598 skip swap of current 0x16d79c0 shadow 0x16d7a20 as there is nothing new
[2017-12-17 19:19:16.416] core_process_messages:615 length of C2G_OutQ : 0

[2017-12-24 17:31:16.820] conn_connect:321 <CONN_LOCAL_PEER_SERVER 0xd74010 14 to '<IP_ADDRESS>:8101:rack1:dc:0'> connecting.....
[2017-12-24 17:31:16.820] event_add_conn:207 adding conn <CONN_LOCAL_PEER_SERVER 0xd74010 14 to '<IP_ADDRESS>:8101:rack1:dc:0'> to active
[2017-12-24 17:31:16.820] _dn_free:305 free(0xd73ed0) @ dyn_task.c:106
[2017-12-24 17:31:16.820] event_wait:264 epoll 0004 triggered on conn 0xd74010
[2017-12-24 17:31:16.820] core_core:508 event FF00 on <CONN_LOCAL_PEER_SERVER 0xd74010 14 to '<IP_ADDRESS>:8101:rack1:dc:0'>
[2017-12-24 17:31:16.820] _stats_pool_incr:1694 incr field 'peer_connections' to 1
[2017-12-24 17:31:16.820] dnode_peer_connected:780 <CONN_LOCAL_PEER_SERVER 0xd74010 14 to '<IP_ADDRESS>:8101:rack1:dc:0'> connected
[2017-12-24 17:31:16.820] event_del_out:181 removing conn <CONN_LOCAL_PEER_SERVER 0xd74010 14 to '<IP_ADDRESS>:8101:rack1:dc:0'> from active
[2017-12-24 17:31:16.820] core_process_messages:615 length of C2G_OutQ : 0
[2017-12-24 17:31:45.819] gossip_loop:797 Gossip is running ...
[2017-12-24 17:31:45.846] core_process_messages:615 length of C2G_OutQ : 0
[2017-12-24 17:31:45.846] core_print_peer_status:63 1)0xd71a70 dc rack1 <IP_ADDRESS>:8101:rack1:dc:0 NORMAL
[2017-12-24 17:31:45.846] core_print_peer_status:63 0)0xd71830 dc rack2 <IP_ADDRESS>:8101 NORMAL
[2017-12-24 17:31:45.846] _dn_free:305 free(0xd73e80) @ dyn_task.c:106

[2017-12-24 17:31:16.485] core_debug:566 =====================Peers info=====================
[2017-12-24 17:31:16.485] core_debug:568 Server pool          : 'dyn_o_mite'
[2017-12-24 17:31:16.485] core_debug:571 ==============================================
[2017-12-24 17:31:16.485] core_debug:573        Peer DC            : 'dc'
[2017-12-24 17:31:16.485] core_debug:574        Peer Rack          : 'rack1'
[2017-12-24 17:31:16.485] core_debug:576        Peer name          : '<IP_ADDRESS>'
[2017-12-24 17:31:16.485] core_debug:577        Peer pname         : '<IP_ADDRESS>:8101'
[2017-12-24 17:31:16.485] core_debug:579        Peer state         : JOINING
[2017-12-24 17:31:16.485] core_debug:580        Peer port          : 8101
[2017-12-24 17:31:16.485] core_debug:581        Peer is_local      : 1
[2017-12-24 17:31:16.485] core_debug:582        Peer failure_count : 0
[2017-12-24 17:31:16.485] core_debug:583        Peer num tokens    : 1
[2017-12-24 17:31:16.485] print_dyn_token:86                    Token : 0 0 1
[2017-12-24 17:31:16.485] _dn_alloc:259 malloc(1048640) at 0x7fc21c0008c0 @ dyn_mbuf.c:420
[2017-12-24 17:31:16.485] core_debug:571 ==============================================
[2017-12-24 17:31:16.485] gossip_loop:793 gossip_interval : 30000 msecs
[2017-12-24 17:31:16.485] core_debug:573        Peer DC            : 'dc'
[2017-12-24 17:31:16.485] core_debug:574        Peer Rack          : 'rack2'
[2017-12-24 17:31:16.485] core_debug:576        Peer name          : '<IP_ADDRESS>'
[2017-12-24 17:31:16.485] core_debug:577        Peer pname         : '<IP_ADDRESS>:8101:rack2:dc:0'
[2017-12-24 17:31:16.485] core_debug:579        Peer state         : DOWN
[2017-12-24 17:31:16.485] core_debug:580        Peer port          : 8101
[2017-12-24 17:31:16.485] core_debug:581        Peer is_local      : 0
[2017-12-24 17:31:16.485] core_debug:582        Peer failure_count : 0
[2017-12-24 17:31:16.485] core_debug:583        Peer num tokens    : 1
[2017-12-24 17:31:16.485] print_dyn_token:86                    Token : 0 0 1
[2017-12-24 17:31:16.485] core_debug:592 Peers Datacenters/racks/nodes .................................................
[2017-12-24 17:31:16.485] core_debug:596 Peer datacenter........'dc'
[2017-12-24 17:31:16.485] core_debug:600        Peer rack........'rack1'
[2017-12-24 17:31:16.485] core_debug:601        Peer rack ncontinuumm    : 1
[2017-12-24 17:31:16.485] core_debug:602        Peer rack nserver_continuum    : 1
[2017-12-24 17:31:16.485] core_debug:600        Peer rack........'rack2'
[2017-12-24 17:31:16.485] core_debug:601        Peer rack ncontinuumm    : 1
[2017-12-24 17:31:16.485] core_debug:602        Peer rack nserver_continuum    : 1
[2017-12-24 17:31:16.485] core_debug:605 ...............................................................................

[2017-12-24 20:53:46.381] event_add_conn:207 adding conn <CONN_PEER_PROXY 0x16dc360 13 listening on '<IP_ADDRESS>:8101'> to active
[2017-12-24 20:53:46.381] event_del_out:181 removing conn <CONN_PEER_PROXY 0x16dc360 13 listening on '<IP_ADDRESS>:8101'> from active

[2017-12-24 20:53:52.917] _stats_pool_incr:1694 incr field 'peer_connections' to 1
[2017-12-24 20:53:52.917] dnode_peer_connected:780 <CONN_LOCAL_PEER_SERVER 0x2133020 15 to '<IP_ADDRESS>:8101:rack1:dc:0'> connected
[2017-12-24 20:53:52.917] event_del_out:181 removing conn <CONN_LOCAL_PEER_SERVER 0x2133020 15 to '<IP_ADDRESS>:8101:rack1:dc:0'> from active
[2017-12-24 20:53:52.917] stats_swap:1592 skip swap of current 0x21256e0 shadow 0x2125740 as aggregator is busy

configs for both nodes
<IP_ADDRESS>:

dyn_o_mite:
  env: network
  datacenter: dc
  rack: rack1
  dyn_listen: <IP_ADDRESS>:8101
  dyn_seeds:
    - <IP_ADDRESS>:8101:rack2:dc:0
  listen: <IP_ADDRESS>:8102
  servers:
    - <IP_ADDRESS>:6380:1
  tokens: '0'
  secure_server_option: none
  pem_key_file: /opt/dynomite/conf/dynomite.pem
  data_store: 0
  stats_listen: <IP_ADDRESS>:22222
  timeout: 5000
  read_consistency : DC_ONE
  write_consistency : DC_ONE
  server_retry_timeout: 30000
  server_failure_limit: 3
  enable_gossip: false

<IP_ADDRESS>:

dyn_o_mite:
  env: network
  datacenter: dc
  rack: rack2
  dyn_listen: <IP_ADDRESS>:8101
  dyn_seeds:
    - <IP_ADDRESS>:8101:rack1:dc:0
  listen: <IP_ADDRESS>:8102
  servers:
    - <IP_ADDRESS>:6380:1
  tokens: '0'
  secure_server_option: none
  pem_key_file: /opt/dynomite/conf/dynomite.pem
  data_store: 0
  stats_listen: <IP_ADDRESS>:22222
  timeout: 5000
  read_consistency : DC_ONE
  write_consistency : DC_ONE
  server_retry_timeout: 30000
  server_failure_limit: 3
  enable_gossip: false

In my opinion, the yml file looks fine to me. The removing from active indicatest that this node does not have any outgoing data to that peer.
Now at this level, when you do any redis-cli command, do you use 8102 or 6380? The later is talking directly to redis (no replication). Check with 8102, and see what you see in the logs at that time.

That's exactly the kind of easy solution I was hoping for! Haha, thank you very much.
Now that I'm connecting to the dynomite listen port, 8102, and making changes to data using that redis-cli console, changes are correctly replicated when viewed from the other host on 8102. And when viewing data using the internal redis port, 6380, I see replicated changes on both hosts correctly.
OK to close!


Add extremely common word sequences?

TLDR:
123456 is pretty much the most common password in the world and also has no entropy due to being an obvious sequence.
zxcvbn-ts falls on it's face with onetwothreefourfivesix, rating it as maximum strength.
Let's fix that?

Just an idea, not sure if this is commonly done with passwords.
But just like 123456789 or 987654321 or abcdefg, etc is seen as completely lacking entropy... what about
Months
januaryfebruarymarch
julyjunemay
Written numbers
onetwothree
nineeightseven
Seasons
springsummerautumn
winterspringsummer
Bible chapters
genesisexoduswhatever etc
Sizes
smallmediumlarge
largemediumsmall
Greek whatever
alphabeta etc
Phonetic alphabet
alphabravocharliedelta
tangosierraromeo
zxcvbn-ts currently thinks all this sort of junk is a strong password (might need to add an extra word in some cases, but normally 3-4 words, and it thinks you're golden), when you've basically got no entropy if you're using any of the above.
Obviously there's an endless amount of common sequences people could put into a password.
Like listing the characters of a popular tv series.
But I figured the categories I wrote above should be standard, because regardless of a person's preferences or personality, they'll deal with (or be familiar with) most, if not all of the above. With the exception of maybe awareness of the bible chapter names.

Hey, thanks for the suggestion.
I like the idea, the problem is that this would be a combination from the dictionary matcher and the sequence matcher.
Basically you need a dictionary for every language that has those sequences. For example like this:
{
  "numbers": [
    "one",
    "two",
    "three"
...
  ],
  "seasons": [
    "spring",
    "summer",
    "autumn",
    "winter"
  ]
...
}

This would mean you need to use the dictionary matcher to identify all those different words and then you need to use some kind of sequence matcher to go through all those matches to check if they are in a row.
I like the general idea of this but i don't see the solution right now. If you have an idea feel free to open a PR or create your own package, since 1.0.0-beta-1 custom matchers are possible but i think it would be easier to add it to the repo to reuse the dictionary and add a custom DictionarySequence matcher.

Youre welcome. This is what immediately comes to mind. I haven't given it any deep thought so there may be dragons.
One idea that comes to mind, is if these words exist in an order less dictionary, they can be moved out into an array of sequence arrays. (As your example)
(Then we aren't loading duplicate words into the browser and keeping the bundle small)
Then the regular word matcher can be tweaked to look for words in the seuqnece arrays just like it looks for words on the order less array currently.
Then once thats done... Its a simple matter of writing an algorithm to look for words in the sequence arrays within the password. If a match is found, check the following word or previous word, if that matches check the next word (while loop) then remove that string from the password and mark it as score 1 or zero or whatever. Then repeat the process with whatever's left of the password.
It seems relatively trivial to add this functionality but of course time is precious and would take a little bit of time. Currently I have no free time. Just contributing the idea at this point.
No pressure.
Thanks for the library :)
On 22 July 2021 3:19:11 PM SAST, MrWook @.***> wrote:

Hey, thanks for the suggestion.
I like the idea, the problem is that this would be a combination from
the dictionary matcher and the sequence matcher.
Basically you need a dictionary for every language that has those
sequences. For example like this:
{
 "numbers": [
   "one",
   "two",
   "three"
...
 ],
 "seasons": [
   "spring",
   "summer",
   "autumn",
   "winter"
 ]
...
}

This would mean you need to use the dictionary matcher to identify all
those different words and then you need to use some kind of sequence
matcher to go through all those matches to check if they are in a row.
I like the general idea of this but i don't see the solution right now.
If you have an idea feel free to open a PR or create your own package,
since 1.0.0-beta-1 custom matchers are possible but i think it would
be easier to add it to the repo to reuse the dictionary and add a
custom DictionarySequence matcher.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/zxcvbn-ts/zxcvbn/issues/63#issuecomment-884905511

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

I had a similar, broader idea:
Since Dropbox kicked off this project, there have been some public leaks of unhashed password lists that should be game-changing data sources for a project like this.  Instead of assuming that passwords use common words in the same frequency as written text ("you, to, it, that, ..."), we can rank them based on their actual usage in passwords.
Based on actual leaked password lists, we can improve entropy scoring based on (1) the popularity of the password structure (set of patterns; e.g. (word)(number)(symbol) > (symbol)(word)(symbol)) and (2) the rank/weight of each particular pattern within those sets (e.g. onetwothreefour > correcthorsebatterystaple).  That first exercise – determining the entropy of the password structure itself – was waived by the original project due to lack of data.
Of course, this exercise is the same as improving the efficiency of a password cracker.  But that was essentially the point of zxcvbn to begin with – to help password strength meters "catch up" to password cracking libraries.
(I understand that this fork is focused on cleanup, tech debt, and other higher priority things :) Hopefully it is flattering and not annoying that the suggestions are coming here now.)

@modest this fork isn't just a clean up.
I would love to see more matchers and contribution. Which means your idea could be a extendet version of the password dictionary as a separated matcher with a new password list.
Feel free to open an own issue for your idea and if you have the time you can even make a PR :)


Add dots to slider

create a new optional prop named dots ( default false ) that increment dots for each slide that we have on the slider.
when the user click on the dot ( eg 3 ) should go to the slide 3
use the slide Single Item as exemple of behaviour
create beautiful dots, follow or current style-guide ( colors, shadows, etc )
here is our slider component
note: single image should not render the dot

Hi @vczb
can i work on this issue?

Hi @vczb can i work on this issue?

Sure! I'll sign you on it

Hi @vczb can i work on this issue?

hi @AsmaKacem1 are you still working on that?

Let's assign it to me, I will work on it. Thanks! @vczb

Ive created a PR #65 for this issue, please take a look. Thanks @vczb

Hello, I will review ASAP

Closed by #65


Dependencies fail to be injected into validation static override methods

✏ Describe the bug
The v3 and v4 documentation are identical for the section titled USING DEPENDENCIES IN OVERWRITTEN FUNCTIONALITY under the validation. Specifically, the ValidationContext can be DI'd totally fine into the rules() static method override, but cannot in the attributes() static method override.
Error: Unresolvable dependency resolving [Parameter #0 [ <required> mixed $payload ]] in class Spatie\LaravelData\Support\Validation\ValidationContext
In case it helps, I noticed the Spatie\LaravelData\Resolvers\DataValidationRulesResolver is passing the context into rules(), yet nothing is being passed into attributes() (in Spatie\LaravelData\Resolvers\DataValidationMessagesAndAttributesResolver). See below:

// In `Spatie\LaravelData\Resolvers\DataValidationRulesResolver` around line 243
$overwrittenRules = app()->call([$class->name, 'rules'], ['context' => $validationContext]);

vs

// In `Spatie\LaravelData\Resolvers\DataValidationMessagesAndAttributesResolver` around line 79
$attributes = collect(app()->call([$class, 'attributes']))
                ->keyBy(fn (mixed $messages, string $key) => $path->property($key)->get())
                ->merge($attributes)
                ->all();

↪ To Reproduce
Add the following attributes() static method override to any DTO and attempt to run the validation. NOTE: the following code is copied directly from the v4 documentation:

    public static function attributes(
        ValidationAttributesLanguageRepository $validationAttributesLanguageRepository
    ): array
    {
        return [
            'title' => $validationAttributesLanguageRepository->get('title'),
            'artist' => $validationAttributesLanguageRepository->get('artist'),
        ];
    }

NOTE: In v4 the ValidationAttributesLanguageRepository class does not exist, but that is not the problem because if you change the DI to ValidationContext, the error still persists.
✅ Expected behavior
Access to the DI'd object/value.
🖥 Versions
Laravel: 11.5.0
Laravel Data: 4.5.1
PHP: 8.3.6

I don't really see this as an issue, the docs specify that DI is possible in the attributes method not that a ValidationContext can be injected that's only reserved for the rules method. As for ValidationAttributesLanguageRepository it is just an example of something that can be injected, it does not exist in any data version.
I think a feature like this can be added quite easily, so I always welcome PR's adding this but at the moment I'm going to close the issue since we do not advertise such a feature in the docs.


Rename package.json name

Let's rename nest-typescript-starter to nestjs-starter-rest-api and update the package.json and the lock file. Thanks.

@Saad-Amjad
I think this issue is already fixed now.
I am closing this here, please re-open if I am mistaken.