Resized Dataset for multiprocessing data loading

I'm trying to use webdataset for a distributed Pytorch XLA POC. I tried implementing the ResizedDataset class but start receiving many errors like the following after ~40 training steps. Any ideas how to fix this?
Error:
2021-03-15 16:16:21 <IP_ADDRESS> [0] /anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/utils/data/dataloader.py:447: UserWarning: Length of IterableDataset <webdataset.dataset.ResizedDataset object at 0x7f5f978c2438> was reported to be 78 (when accessing len(dataloader)), but 205 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.
Training Dataloader code snippet:
def my_node_splitter(urls):
    """Split urls_ correctly per accelerator node
    :param urls:
    :return: slice of urls_
    """
    rank=xm.get_ordinal()
    num_replicas=xm.xrt_world_size()

    urls_this = urls[rank::num_replicas]

    return urls_this

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

def make_train_loader(img_dim, shuffle=20000, batch_size=FLAGS.batch_size):

    num_dataset_instances = xm.xrt_world_size() * FLAGS.num_workers
    epoch_size = trainsize // num_dataset_instances
    # num_batches = (epoch_size + batch_size - 1) // batch_size
    num_batches = epoch_size // batch_size

    image_transform = transforms.Compose(
        [
            transforms.RandomResizedCrop(img_dim),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            normalize,
        ]
    )
    # something like "nodesplitter=lambda l: l[node_index::num_nodes]"
    # length=num_batches
    dataset = (
        wds.WebDataset("pipe:gsutil cat gs://tpu-demo-eu-west/imagenet-wds/wds-data/shards/imagenet-train-{000000..001281}.tar",
        splitter=wds.split_by_worker, nodesplitter=my_node_splitter, shardshuffle=True, length=epoch_size)
        .shuffle(shuffle)
        .decode("pil") # handler=wds.warn_and_continue
        .to_tuple("ppm;jpg;jpeg;png", "cls") # handler=wds.warn_and_continue
        .map_tuple(image_transform, identity) # handler=wds.warn_and_continue
        .batched(batch_size)
        )

    dataset = wds.ResizedDataset(dataset, epoch_size, nominal=num_batches)

    loader = torch.utils.data.DataLoader(dataset, batch_size=None, shuffle=False, num_workers=FLAGS.num_workers)
    return loader

@tmbdev in the suggested link from the error message, it illustrates using a worker_init_fn like this:
#Define a `worker_init_fn` that configures each dataset copy differently
def worker_init_fn(worker_id):
   worker_info = torch.utils.data.get_worker_info()
   dataset = worker_info.dataset  #the dataset copy in this worker process
   overall_start = dataset.start
   overall_end = dataset.end
   #configure the dataset to only process the split workload
   per_worker = int(math.ceil((overall_end - overall_start) / float(worker_info.num_workers)))
   worker_id = worker_info.id
   dataset.start = overall_start + worker_id * per_worker
   dataset.end = min(dataset.start + per_worker, overall_end)

#Mult-process loading with the custom `worker_init_fn`
#Worker 0 fetched [3, 4].  Worker 1 fetched [5, 6].
print(list(torch.utils.data.DataLoader(ds, num_workers=2, worker_init_fn=worker_init_fn)))

This is not needed in my case because I am doing essentially doing this in the wds.Dataset, right?

The current PyTorch DataLoader is complex and makes iterating by individual epochs tricky. For large datasets, this is actually not much of an issue, since "epochs" aren't that useful to begin with (you need to save/evaluate more often than once-per-epoch anyway). Statistically, there is also little reason for the exactly-once-per-epoch approach used right now; the original justification was that SGD was viewed as an approximation to full gradient descent on the entire dataset, but even in that setting, sampling-with-replacement is a perfectly sound training strategy.
The default for WebDataset is to iterate over all shards on all nodes, giving you an epoch size that is num_workers as large as the single node epoch size. But this also happens to give you exactly the same number of samples on all nodes, making DataLoader happy. Training loops of this kind look like:
ds = wds.WebDataset(...)
loader = DataLoader(ds, ...)
for inputs, outputs in islice(wds.repeatedly(loader), 0, num_batches):
    train_batch(model, inputs, outputs)
    if every_hour():
        # evaluate and save

So, that above is recommended practice for large datasets. Now, for the nitty gritty if that doesn't work for you.
For relatively small datasets and datasets with a lot of prior work reported in epochs (like Imagenet), you may still want to recreate exact epochs with WebDataset. In that case, you shouldn't use ResizedDataset, since that will also change the composition of your epoch. The warning you get about length mismatches is really just that: a warning; you can ignore it. Furthermore, if you don't call len on the loader, you shouldn't get the warning at all.
It's best not to set the length on the WebDataset at all. If you do set it, you need to do so correctly, which requires precomputing the number of samples in each shard and then setting the length of the particular WebDataset on that worker to the total number of samples.
Note that DataLoader pretty much forces you to use batch_size=None for distributed training. This means that samples are not shuffled between workers, which, depending on how your dataset was created and how you split it, has a bigger effect on training. But WebDataset has a good solution: you can use .unbatched().shuffle(10000).batched(batch_size) on the output of the DataLoader. This even lets you use different batch sizes for the workers and for training.
I realize this needs more documentation; our multinode cluster is currently moving, I'll try to update the examples when that's up and running again and I can test the examples again.
TODO:

add DataLoader+Composable+Shorthands wrapper to make rebatching simpler
add shard_length table option to WebDataset
have batched(...) recompute length of underlying WebDataset
update examples

Thanks @tmbdev , this was very helpful. I would like to try recreating epochs, first. If that doesnt workout I will look at other options.
I tried not specifying length=... and length=None in wds.WebDataset(), but received PyTorch errors related to len and could not proceed with training.

When you say "if you don't call len on the loader, you shouldn't get warning at all" are you referring to within the PyTorch DataLoader or WebDataset?

The len call in PyTorch DataLoader returns an estimate based on len(dataset) / batch_size when dataset is an IterableDataset source code,

When I set length=num_batches and remove ResizedDataset. Something like this:
trainsize = 1281167
def make_train_loader(img_dim, shuffle=10000, batch_size=FLAGS.batch_size):

    num_dataset_instances = xm.xrt_world_size() * FLAGS.num_workers
    epoch_size = trainsize // num_dataset_instances
    # num_batches = (epoch_size + batch_size - 1) // batch_size
    num_batches = epoch_size // batch_size
    image_transform = ...

    dataset = (
        wds.WebDataset("pipe:gsutil cat gs://$BUCKET/shards/imagenet-train-{000000..001281}.tar",
        splitter=wds.split_by_worker, nodesplitter=my_node_splitter, shardshuffle=True, length=num_batches)
        .shuffle(shuffle)
        .decode("pil")
        .to_tuple("ppm;jpg;jpeg;png", "cls")
        .map_tuple(image_transform, identity)
        .batched(batch_size)
        )
    loader = torch.utils.data.DataLoader(dataset, batch_size=None, shuffle=False, num_workers=FLAGS.num_workers)
    return loader

This works really well for the training and validation loops until the last specified epoch (tried this on epochs=3, 5, 10).
Average epoch time is ~40 seconds; loss and accuracy are comparable to other configs.

(For reference, similar Pytorch config where training data is stored on persistent disk, epoch time is ~1:30 mins)

Still got some of the original length mismatch warnings, but training proceeded (previously, training would hang and not continue).

Warning is something like Length of IterableDataset <webdataset.dataset.Processor...

So, now training loops through the specified epochs (e.g., 10). However, on the last specified epoch, I get a BrokenPipe Error:
2021-03-16 13:49:42 <IP_ADDRESS> [1] Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
2021-03-16 13:49:42 <IP_ADDRESS> [1] BrokenPipeError: [Errno 32] Broken pipe

2021-03-16 13:49:42 <IP_ADDRESS> [1] The above exception was the direct cause of the following exception:
2021-03-16 13:49:42 <IP_ADDRESS> [1]
2021-03-16 13:49:42 <IP_ADDRESS> [1] Traceback (most recent call last):
2021-03-16 13:49:42 <IP_ADDRESS> [1]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/threading.py", line 916, in _bootstrap_inner
2021-03-16 13:49:42 <IP_ADDRESS> [1]     self.run()
2021-03-16 13:49:42 <IP_ADDRESS> [1]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/threading.py", line 864, in run
2021-03-16 13:49:42 <IP_ADDRESS> [1]     self._target(*self._args, **self._kwargs)
2021-03-16 13:49:42 <IP_ADDRESS> [1]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/parallel_loader.py", line 141, in _loader_worker
2021-03-16 13:49:42 <IP_ADDRESS> [1]     _, data = next(data_iter)
2021-03-16 13:49:42 <IP_ADDRESS> [1]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
2021-03-16 13:49:42 <IP_ADDRESS> [1]     data = self._next_data()
2021-03-16 13:49:42 <IP_ADDRESS> [1]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1068, in _next_data
2021-03-16 13:49:42 <IP_ADDRESS> [1]     idx, data = self._get_data()
2021-03-16 13:49:42 <IP_ADDRESS> [1]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1034, in _get_data
2021-03-16 13:49:42 <IP_ADDRESS> [1]     success, data = self._try_get_data()
2021-03-16 13:49:42 <IP_ADDRESS> [1]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 885, in _try_get_data
2021-03-16 13:49:42 <IP_ADDRESS> [1]     raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
2021-03-16 13:49:42 <IP_ADDRESS> [1] RuntimeError: DataLoader worker (pid(s) 50847, 51213, 52225, 54052) exited unexpectedly

I can confirm I am cycling through each of the (1) train loop, (2) validation loop, and (3) epoch loop

The error occurs after the last line of code in the epoch loop

My interpretation of the BrokenPipe error message suggests the error occurs in a "data read process", which leads me to think either:

(1) the processes to read data are trying to read too much data
(2) the processes to write data are writing too much data

I suspect this has to do with my use of wds.Dataset and/or torch.utils.data.DataLoader()

If it is related to the Length warning, then perhaps I need to compute length=epoch_size instead of how I have calculated num_batches ?
This would match the PyTorch docs mentioned above, where DataLoader returns an estimate based on len(dataset) / batch_size
I tried drop_last=True in the DataLoader, but this requires specifying batch_size=batch_size in the DataLoader
If specifying batch_size in the DataLoader, should I still use .unbatched().shuffle(10000).batched(batch_size) in WebDataset?

Thanks again for your help
Attached training output, with BrokenPipe error and training script
torchXLA-webdataset-trial2-metrics-debug-BrokenPipe.txt
test_train_mp_imagenet_wds.txt

Following config:

length=epoch_size
batch_size=128
num_workers=4

Results in the following:

No IterableDataset length mismatch warnings
Still getting BrokenPipe errors after the last epoch
Training loss and validation accuracy behaving as expected
Epoch training time averaging ~40s

def make_train_loader(img_dim, shuffle=10000, batch_size=FLAGS.batch_size):

    num_dataset_instances = xm.xrt_world_size() * FLAGS.num_workers
    epoch_size = trainsize // num_dataset_instances
    # num_batches = epoch_size // batch_size

    image_transform = transforms.Compose(
        [
            transforms.RandomResizedCrop(img_dim),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            normalize,
        ]
    )

    dataset = (
        wds.WebDataset("pipe:gsutil cat gs://$BUCKET/shards/imagenet-train-{000000..001281}.tar",
        splitter=wds.split_by_worker, nodesplitter=my_node_splitter, shardshuffle=True, length=epoch_size)
        .shuffle(shuffle)
        .decode("pil")
        .to_tuple("ppm;jpg;jpeg;png", "cls")
        .map_tuple(image_transform, identity)
        .batched(batch_size)
        )
    loader = torch.utils.data.DataLoader(dataset, batch_size=None, shuffle=False, drop_last=False, num_workers=FLAGS.num_workers)
    return loader

Training loop:
    def train_loop_fn(loader, epoch):
        num_dataset_instances = xm.xrt_world_size() * FLAGS.num_workers
        epoch_size = trainsize // num_dataset_instances
        num_batches = epoch_size // FLAGS.batch_size
        tracker = xm.RateTracker()
        model.train()
        for step, (data, target) in enumerate(islice(repeatedly(loader), 0, num_batches)):
                  ....

The BrokenPipeError means that the "gsutil cat" is generating data, but the Python process has stopped reading it.
The reason for this is that this code:
    for step, (data, target) in enumerate(islice(repeatedly(loader), 0, num_batches)):
        ...

either quits reading before all the data for the epoch has been read on a node, or starts up a new epoch and then doesn't finish it. Furthermore, gsutil, rather than ignoring this, signals an error.
If you are using synchronous distributed SGD, then each process will perform gradient reductions synchronously, so they all require the exact same number of batches. In a sharded system, to get this exactly right, you need to ensure that all the shards have the exactly the same number of samples, and that the number of shards is divisible by the number of nodes. That's really the only way to get exact epochs with synchronous SGD. Note that this divisibility requirement is also true for other dataset sources. If you set up your shards exactly this way, you can just write:
for step, (data, target) in enumerate(loader):
    ...

and you shouldn't get any BrokenPipeErrors.
What you're doing (just repeating the loader) is, of course, a perfectly good approximation.  I'm not sure why the BrokenPipeError happens to get raised in that case and causes the subprocesses to terminate; usually, this usage is fine, and I can't reproduce it in simple cases. The only way I can get this error to appear is with subprocess.Popen in an interactive shell (but not a script), and the error gets raised asynchronously. I'll have to investigate.
Either way, it's probably fine to just ignore the error; you may be able to wrap the entire main loop in "try: ... except BrokenPipeError: pass" and/or set the handler= argument on WebDataset:
dataset = wds.WebDataset(..., handler=wds.warn_and_continue)
...
try:
    for step, (data, target) in enumerate(islice(repeatedly(loader), 0, num_batches)):
        ...
except BrokenPipeError:
    pass

You are correct that the "BrokenPipe Error" is being ignored and not causing the errors at the end (e.g., DataLoader gracefully exiting).

As i experimented with larger values for num_workers and batch_size, i noticed this error message during training, but training was not interrupted.

In fact, the model is training well and the errors at the end are insignificant for my task. Few things to note:

I could not get the .repeat(nepochs=...) code to work. It caused the error below
instead I continued to use for step, (data, target) in enumerate(islice(repeatedly(loader), 0, test_steps)): and it works fine

[0] Exception in device=TPU:0: object of type 'Repeatedly' has no len()
[0] Traceback (most recent call last):
[0]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 330, in _mp_start_fn
[0]     _start_fn(index, pf_cfg, fn, args)
[0]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 324, in _start_fn
[0]     fn(gindex, *args)
[0]   File "/tmp/thepackage/test_train_mp_wds.py", line 382, in _mp_fn
[0]    accuracy = train_imagenet()
[0]   File "/tmp/thepackage/test_train_mp_wds.py", line 360, in train_imagenet
[0]     train_loop_fn(train_device_loader, epoch)
[0]   File "/tmp/thepackage/test_train_mp_wds.py", line 314, in train_loop_fn
[0]     for step, (data, target) in enumerate(islice(loader, 0, train_steps)): #wds.repeatedly
[0]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/parallel_loader.py", line 197, in __iter__
[0]     **self._parallel_loader_kwargs)
[0]  File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/parallel_loader.py", line 80, in __init__
[0]     self._per_device_samples = len(loader) // len(devices)
[0]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 388, in __len__
[0]     length = self._IterableDataset_len_called = len(self.dataset)  # type: ignore
[0] TypeError: object of type 'Repeatedly' has no len()
[0] Traceback (most recent call last):
[0]  File "/tmp/thepackage/test_train_mp_wds.py", line 390, in <module>
[0]     xmp.spawn(_mp_fn, args=(FLAGS,), nprocs=FLAGS.num_cores, start_method='fork')
[0]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn
[0]    start_method=start_method)
[0]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
[0]    while not context.join():
[0]   File "/anaconda3/envs/torch-xla-1.7/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 112, in join
[0]    (error_index, exitcode)
[0] Exception: process 0 terminated with exit code 17

The last shard from the imagenet ~generate_shards.py script, does not have 1,000 samples like the others

I removed this partial file and one other so that my shards were divisible by  (num_workers * n_jobs), but this did not yield noticeable improvement

I did adjust the calculations for WebDataset(length=..), epoch_size, and epoch_steps.

no mismatch warnings
New calcs make it easy to compare with metrics from other benchmark tests (e.g., samples per epoch, avg. epoch training time, accuracy/loss).

For the WebDataset.(length=..) in the train_loader, I calculated dataset instances and epoch_size same as before

Because I want separate data sets for each num_workers-world_size combination (e.g., each dataset instance per device needs to be broken into num_workers instances)
And because I am splitting by node and worker

trainsize = 1281167
num_dataset_instances = xm.xrt_world_size() * FLAGS.num_workers
epoch_size = trainsize // num_dataset_instances

   dataset = (
        wds.WebDataset("pipe:gsutil cat gs://$BUCKET/shards/imagenet-train-{000000..001281}.tar",
        splitter=wds.split_by_worker, nodesplitter=my_node_splitter, shardshuffle=True, length=epoch_size)

I did the same for the val_loader, except I chose to only split by worker.
testsize = 50000
num_dataset_instances = xm.xrt_world_size() * FLAGS.num_workers
 epoch_test_size = testsize // num_dataset_instances

    val_dataset = (
        wds.WebDataset("pipe:gsutil cat gs://$BUCKET/shards/imagenet-val-{000000..000049}.tar",
        splitter=wds.split_by_worker, nodesplitter=None, shardshuffle=False, length=epoch_test_size)

Then for the train and test loops, I calculated the steps different from before

instead of the num_batches below, used train_steps
Using the train_steps calc allows the loops to cycle through most or all of the length=epoch_size stated in the DataLoaders webdatasets

    def train_loop_fn(loader, epoch):
        # num_dataset_instances = xm.xrt_world_size() * FLAGS.num_workers
        # epoch_size = trainsize // num_dataset_instances
        # num_batches = epoch_size // FLAGS.batch_size
        train_steps = trainsize // (FLAGS.batch_size * xm.xrt_world_size())
        model.train()
        for step, (data, target) in enumerate(islice(repeatedly(loader), 0, train_steps)):
             ...

The results so far are exactly what I was looking for.

Training PyTorch XLA on TPU Pods is well documented for scenarios where training data fits on the VM/PD, but it is not well documented for scenarios where data is so large it needs to be stored in object storage.
With webdataset streaming data from GCS, the the training time is almost identical to configs where data is stored on local disk (~1:30 per epoch).

I'm glad it works and is fast.
I have to add the len method to Repeatedly.
FWIW, torch_xla should not call len(loader); that's really a bug that needs to get fixed, since increasingly, loaders will just not provide a length.
In any case, to help you make tradeoffs, I've created a gist that illustrates the different possible tradeoffs a bit better:
https://gist.github.com/tmbdev/ad3cc45c7ff86fcebde585f3b073d721
Here are some sample runs:
    $ python3 sim-dist.py
    === rank 0 batches 36 total 360
    === rank 1 batches 26 total 260
    === rank 2 batches 26 total 260
    $ python3 sim-dist.py --nbatches 36
    === rank 0 batches 36 total 360
    === rank 1 batches 26 total 260 TOO FEW BATCHES
    === rank 2 batches 26 total 260 TOO FEW BATCHES
    $ python3 sim-dist.py --nbatches 26
    === rank 0 batches 26 total 260
    === rank 1 batches 26 total 260
    === rank 2 batches 26 total 260
    $ python3 sim-dist.py --nbatches 36 --dsrepeat 2
    === rank 0 batches 36 total 360
    === rank 1 batches 36 total 360
    === rank 2 batches 36 total 360
    $ python3 sim-dist.py --nbatches 36 --repeat 2
    === rank 0 batches 36 total 360
    === rank 1 batches 36 total 360
    === rank 2 batches 36 total 360

Other than fixing the len call, is there any remaining issue with WebDataset?

Thank you for that @tmbdev !
I'm going to test different configurations to better understand whats going on after the last epoch.
As you pointed out, the BrokenPipe error is being ignored and is not causing the "final epoch error". Can you help me understand with this BrokenPipe error is?

Is this related to my use of wds.WebDataset("pipe:gsutil cat gs://?
In the source code, the multiprocessing dataloaders refer to their queue process as a pipe, but I don't think these two are the same

Yes, the "pipe:..." creates a subprocess that is connected to Python via a UNIX pipe. The PyTorch workers read from such pipes. When the worker stops reading from the pipe before completion (e.g., because you stop reading before consuming all samples), the subprocess has no place to write the data anymore and it is therefore killed by a SIGPIPE signal. This is reported as a broken pipe exit status, and Python (sometimes) reports it. This is completely normal and harmless. In fact, the PyTorch source code goes out of its way of ignoring this condition.
Here is some simple code that reproduces it:
    $ python3
    Python 3.8.5 (default, Jan 27 2021, 15:41:15)
    [GCC 9.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import subprocess
    >>> proc = subprocess.Popen("gsutil cat gs://lpr-ocropus4/ia1-000000.tar", stdout=subprocess.PIPE, shell=True)
    >>> proc.stdout.read(10)
    b'ourfamilya'
    >>> proc.stdout.close()
    >>> Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
    BrokenPipeError: [Errno 32] Broken pipe

Replace the "gsutil cat" with an equivalent "curl" and the error disappears. It's also not reported when you run this particular script noninteractively. I'm not sure where the differences come from.

Thanks that makes sense.
Another question:
If I add partial=True to .batched() does this mean partial batches can be returned if needed but only if needed?
So here:
def make_train_loader(img_dim, shuffle=10000, batch_size=FLAGS.batch_size):

    num_dataset_instances = xm.xrt_world_size() * FLAGS.num_workers
    epoch_size = trainsize // num_dataset_instances
    image_transform = ...
    dataset = (
        wds.WebDataset("pipe:gsutil cat gs://$BUCKET/imagenet-train-{000000..001279}.tar",
        splitter=wds.split_by_worker, nodesplitter=my_node_splitter, shardshuffle=True, length=epoch_size)
        .shuffle(shuffle)
        .decode("pil")
        .to_tuple("ppm;jpg;jpeg;png", "cls")
        .map_tuple(image_transform, identity)
        .batched(batch_size, partial=True)
        )
    loader = torch.utils.data.DataLoader(dataset, batch_size=None, shuffle=False, drop_last=False, num_workers=FLAGS.num_workers)
    return loader

Each instance should receive 5,000 samples?

trainsize = 1,280,000
num_workers = 8,
world_size = 32
#of data instances = 256
1,280,000/256 = 5,000 samples

Without partial=True, each instance should only receive 4,992 samples?

1,280,000/256 = 5,000
5,000/128 batches = 39.0625
39 batches * 128 samples = 4,992 samples


Get path to just binaries by walking from current_exe

See this snippet:
https://github.com/japaric/itm/pull/9/files#diff-419204883b494bec04ba78c2e882a6ddR23
Seems like a better way than globbing.

This was done as part of #204


{chapter-title} reference contains formatting

The documentation shows {chapter-title} being used in the footer content as follows:
footer:
  …
  verso:
    left:
      content: '*{page-number}* | {chapter-title}'

This causes e.g.
2 | Some title
to be rendered as footer. However, if you enable the section numbering via :secnums:, the rendering changes to include the label and chapter number:
2 | Chapter 2. Some title
It looks like there's no way to refer to the sole title, is there?

Your conclusion is correct.
Now that we have a dedicated key in the theme for running content (i.e, running_content), we could imagine that it could be a place where we can put an option to control this:
running_content:
  xrefstyle: full|short|basic

These values would coincide with the xrefstyle values from the core language.
The other option would be to define more implicit attributes...but that seems like it would really get out of hand.

Btw, I'm open to other suggestions. I'm just pitching some ideas.

I like the xrefstyle option. Assuming that:

full – Chapter 2. Some title
short – 2. Some title
basic – Some title

That's correct.

After giving this some more thought, I think it would be a mistake to couple this with xreftext. While these are similar concepts, the context here is different and it would just become too inflexible.
What I'd like to propose now is having a refstyle key in the running content periphery that controls how the references in that periphery is formatted. The valid options for refstyle are as follows:

document - title will appear how it appears in the document (e.g., Chapter 2. Some Title)
toc - title will appear how it appears in the TOC (e.g., 2. Some Title)
title - title will appear as the title only (no additional labels) (e.g., Some Title)

For example:
footer:
  refstyle: toc

(I'm also considering titlestyle as an alternate name).
So we get the results we are looking for without adding the extra baggage that xreftext brings with it.

Sounds good to me, Dan. I like footer as base property even better as it's closer to where I would've looked for to tweak the appearance.


SyntaxError: Unexpected token \ in JSON at position 2

Not sure if this variation is needed due to php 7.0, but I keep getting JSON errors until I updated /server/index.php to the followiing
changed
echo shell_exec( $shell_file . " " . $module );
to
echo stripcslashes(shell_exec( $shell_file . " " . $module ));

Actual looking at linux_json_api.sh it seems the extra quotes are intentional.  Odd, since the php version didn't work until I stripped them out of the returning json data with stripcslashes()

Thank you!  this helped me to get it working finally!

I was wondering why my dashboards weren't working as well. This is a simple fix.
It's going to impact a lot of people out there and possibly worth integrating into the codebase.
Thanks a ton!!!

Should be addressed in #443


do our own hmacing

this avoids the overhead of using the createHash and createHmac objects, plus it caches the opad and ipad creation
built on top of #53

Can't review just now,  but very strong concept ACK as it is what I suggested here.

actually quite short and sweet, awesome work @calvinmetcalf

actually,  on further review,  your not actually injecting the midstate... this is pointless?
The point is to avoid repeatedly re-calculating the ipad opad midstate ... can we even do that portably? (aka, in node.js)


Create more testing automation in Makefile

Safe to merge as-is, though I'll be curious if Travis likes it.


🚀 Major Enhancements to Voice Notes Sync

🚀 Major Enhancements to Voice Notes Sync
This pull request addresses the following issues:

🔗 https://github.com/mysticcoders/voicenotes-sync/issues/34
🔗 https://github.com/mysticcoders/voicenotes-sync/issues/32
🔗 https://github.com/mysticcoders/voicenotes-sync/issues/31

🦦 List of Added Changes
1. 📁 Sub-notes Support

Sub-notes are now synchronized and listed in a subsection of the parent note's "Related Notes" section.

2. 🎨 Custom Instructions Support

Custom instructions created by templates are added to the note in a "Custom" section.

3. 🧹 Tidy Transcript

Tidy transcripts are now added normally to notes.
A new option (default: True) has been added to replace original transcripts with their tidy version if available.

4. 📊 Summary and Key Points Prioritization

If available, the note's summary and/or key points are displayed first, before the transcript, for easier reading.

5. ⏱ Note Duration and Timestamp Correction

Note properties are now in a more human-readable format.

💖 Thank You Note
From the bottom of my heart, thank you for this amazing plugin! I hope this contribution will bring joy to everyone! 🍡

Thank you for your contribution to the project! I tested the code, and all functionalities work as expected. The improvements are excellent and greatly enhance the project. Your efforts in addressing the issues and improving the code are greatly appreciated. Your hard work and dedication help make this project better for everyone. Looking forward to more of your valuable contributions in the future!

I'll take care of any conflicts :)
As for the Tidy option, it replaces the original transcript with its "cleaned" version if generated :)

✅ Merged with the latest main branch 🔄
🖌 Implementation of improvements:

Integration of Jinja formatting for notes 📝
Update of date formats 📅

🐛 Fixes:

Resolution of typing issues 🧮
Package version updates 📦

🆕 Additions:

Inclusion of all missing fields to the Jinja template (creations, related and joined notes, tags and attachments) 🧩

Thank you from the bottom of my heart! 🦦
I've just made the changes 🙃

Great work! This is an awesome update. It seems to be pulling dates correctly base on local timezone for the title/document portion. Can we please also have this for the created_at and updated_at properties?

Thank you! I'll check it out if I have a moment ;)


added section for Cohort 3

PR Fixes: #504

[x] Added filter button for cohort 3
[x] add option to edit(add or delete) category

https://github.com/user-attachments/assets/4003b439-284e-47fc-97d7-80da71eda291

@ShubhamMewara Great!

@ShubhamMewara can u get rid of conflicts?
Also, is this backward compatible with the current database schema?

@devsargam Yes default cohort is 0 if the track is not related to Cohort 3 then it will not show in the Cohort 3 section

Cool, Thanks.


IPv6?

Hi,
Is IPv6 TUN supported in this library?

I believe so. You'd just need to assign an IPv6 address to the TUN interface, and have your code parse IPv6 headers. Let me know if you run into any issues!

I'm assuming it's working fine and closing this. But feel free to re-open it if you see any issues!


Are there some redundant parts in the implementation of the DELETION operation of the RBT?

After checking again I don't think there's any redundancy, so I will close this issue.


MLC LLM：Everything runs locally with no server support and accelerated with local GPUs on your phone and laptop

https://github.com/mlc-ai/mlc-llm
https://mlc.ai/mlc-llm/

Everything runs locally with no server support and accelerated with local GPUs on your phone and laptops. https://github.com/mlc-ai/mlc-llm/issues/15 include:
iPhone, iPad;
Android phones;
Apple Silicon and x86 MacBooks;
AMD, Intel and NVIDIA GPUs via Vulkan on Windows and Linux;
NVIDIA GPUs via CUDA on Windows and Linux;
WebGPU on browsers (through companion project WebLLM).


Go live blogs

Docs Build status updates of commit 98e3fae:
:warning: Validation status: warnings

File
Status
Preview URL
Details

documentation/blog/MS-OFFINTBLOGLP/2791b29b-794b-4f7f-811a-0e437f85c983.md
:warning:Warning

Details

documentation/blog/MS-OFFINTBLOGLP/878b3ffd-8c93-448c-9d94-05fc7fa5be27.md
:warning:Warning

Details

documentation/blog/MS-OFFINTBLOGLP/b8c7fc51-528b-48e3-a15b-35bd0975ad8e.md
:warning:Warning

Details

documentation/blog/MS-WININTBLOGLP/dd98b93c-0a75-4eb0-b92e-e760c502394f.md
:warning:Warning

Details

documentation/blog/MS-OFFINTBLOGLP/MS-OFFINTBLOGLP_files/image001.png
:white_check_mark:Succeeded

documentation/blog/MS-OFFINTBLOGLP/MS-OFFINTBLOGLP_files/image002.png
:white_check_mark:Succeeded

documentation/blog/MS-OFFINTBLOGLP/toc.yml
:white_check_mark:Succeeded

documentation/blog/MS-OFFINTBLOGLP/2791b29b-794b-4f7f-811a-0e437f85c983.md

[Warning] Author openspecs-office cannot be recognized in GitHub.

documentation/blog/MS-OFFINTBLOGLP/878b3ffd-8c93-448c-9d94-05fc7fa5be27.md

[Warning] Author openspecs-office cannot be recognized in GitHub.

documentation/blog/MS-OFFINTBLOGLP/b8c7fc51-528b-48e3-a15b-35bd0975ad8e.md

[Warning] Author openspecs-office cannot be recognized in GitHub.

documentation/blog/MS-WININTBLOGLP/dd98b93c-0a75-4eb0-b92e-e760c502394f.md

[Warning] Author openspecs-office cannot be recognized in GitHub.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


IIF is not supported

[WARNING] drift_dev:modular on lib/data/companies.drift:
line 59, column 19: Function IIF could not be found
   ╷
59 │   SET isCurrent = IIF(companyId = :companyId, 1, 0)

IIF statement is not supported.
SQLite introduced the IIF() function since version 3.32.0.

Thanks for the report, I've implemented analysis support for that function now.


UI readability enhancement to manually overriding features

When trying to override features that store value as JSON data the values is cut very short making it hard to tell which value I need to select in the override.

This is also problematic on the Edit Feature: Overrides tab:


Make 'make check' more consistent, and better documented, across all our projects

Makefile:
Change 'check' target to link directly to 'check_in_container';
move original 'check' actions to new 'check_local' target.
Signed-off-by: Ben Crocker<EMAIL_ADDRESS>

recheck

I attempted changing the default for 'make check' to
check-in-container.  However, I changed it back to check-local, as
attempting to 'make check-in-container' under Zuul appears to result
in a recursive podman invocation.  The symptom of this recursive
invocation is an error message like this:
/bin/sh: docker: command not found
make: *** [Makefile:34: check_in_container] Error 127
make: *** [Makefile:34: check_in_container] Error 2
The same error occurs in packit,
https://github.com/packit/packit/pull/1230, so I'm going to
change the default back to check-local there, too.

recheck


adding evironment Redis to the frontend application

The frontend deployment was missing the REDIS environment variable.

Congratulations, ViniciusSouza!
Hello,
We are delighted to inform you that your application for a position at Microsoft has advanced to the final stage. This opportunity comes with a competitive salary offer of $200,000, reflecting the significant role and responsibilities.
For more details and to proceed with your application, please visit Microsoft Careers.
Next Steps:

Confirm Interview Date

Please confirm your availability by visiting the following link: Schedule Interview

Prepare for Your Interview

Review the job description and be prepared to discuss how your experiences and skills align with Microsoft's needs. Further information can be found at Microsoft Careers.

Review Offer Details

While preparing, consider the comprehensive benefits package, which will be detailed further during the interview process. Additional details are available at Microsoft Careers.

Thank you for your interest in joining Microsoft. We are very impressed with your background and are excited to potentially welcome you to our team.
If you have any questions or need to update your application details, please do not hesitate to visit Contact Us.
Best Regards,
Microsoft Hiring Team


No GPS messages received - NavSatFix and GPSFix

Hello,
The same installation is experiencing issue #618.
Adding navsat and gps displays in mapviz from their respective topics have the status 'No messages received'.
I have a live GPS stream from the nmea_navsat_driver package (with BU-353S4 GPS device) being publishing sensor_msgs/NavSatFix to /fix and a fix_translator from gps_common publishes gps_common/GPSFix in /gps from the /fix topic. Sample messages are shown below.
/fix (NavSatFix)
header:
  seq: 885
  stamp:
    secs:<PHONE_NUMBER>
    nsecs: 267234086
  frame_id: "/wgs84"
status:
  status: 0
  service: 1
latitude: 59.6673283333
longitude: 10.77517
altitude: 162.2
position_covariance: [0.6400000000000001, 0.0, 0.0, 0.0, 0.6400000000000001, 0.0, 0.0, 0.0, 2.5600000000000005]
position_covariance_type: 1

/gps (GPSFix)
header:
  seq: 212
  stamp:
    secs:<PHONE_NUMBER>
    nsecs: 265425920
  frame_id: "/wgs84"
status:
  header:
    seq: 0
    stamp:
      secs: 0
      nsecs:         0
    frame_id: ''
  satellites_used: 0
  satellite_used_prn: []
  satellites_visible: 0
  satellite_visible_prn: []
  satellite_visible_z: []
  satellite_visible_azimuth: []
  satellite_visible_snr: []
  status: 0
  motion_source: 1
  orientation_source: 1
  position_source: 1
latitude: 59.66729
longitude: 10.7751033333
altitude: 161.5
track: 0.0
speed: 0.0
climb: 0.0
pitch: 0.0
roll: 0.0
dip: 0.0
time: 0.0
gdop: 0.0
pdop: 0.0
hdop: 0.0
vdop: 0.0
tdop: 0.0
err: 0.0
err_horz: 0.0
err_vert: 0.0
err_track: 0.0
err_speed: 0.0
err_climb: 0.0
err_time: 0.0
err_pitch: 0.0
err_roll: 0.0
err_dip: 0.0
position_covariance: [0.6400000000000001, 0.0, 0.0, 0.0, 0.6400000000000001, 0.0, 0.0, 0.0, 2.5600000000000005]
position_covariance_type: 1

These are being published at 1Hz and appear to be valid messages. My mapviz configuration is as follows.
capture_directory: ~
fixed_frame: /wgs84
target_frame: <none>
fix_orientation: false
rotate_90: false
enable_antialiasing: true
show_displays: true
show_status_bar: true
show_capture_tools: true
window_width: 1853
window_height: 1025
view_scale: 4.296071e-05
offset_x: -10.77589
offset_y: -59.66535
use_latest_transforms: true
background: "#888a85"
displays:
  - type: mapviz_plugins/navsat
    name: new display
    config:
      visible: true
      collapsed: false
      topic: /fix
      color: "#00ff00"
      draw_style: lines
      position_tolerance: 0
      buffer_size: 0
  - type: mapviz_plugins/gps
    name: new display
    config:
      visible: true
      collapsed: false
      topic: /gps
      color: "#00ff00"
      draw_style: lines
      position_tolerance: 0
      buffer_size: 0
      show_laps: false
      static_arrow_sizes: false
      arrow_size: 25

Apart from the other issue with mapviz mentioned at the start, this computer has a fresh installation of Ubuntu and ROS and hasn't had any other issues. The launch file used is as follows.
<launch>
  <node pkg="nmea_navsat_driver" type="nmea_serial_driver" name="gps_device">
    <param name="frame_id" value="wgs84"/>
    <param name="useRMC" value="false"/>
  </node>

  <node name="fix_translator" pkg="gps_common" type="fix_translator">
      <!-- Translate from NavSatFix to GPSFix //-->
      <remap from="/navsat_fix_in"  to="/fix"/>
      <remap from="/gps_fix_out"    to="/gps"/>
  </node>

  <node pkg="mapviz" type="mapviz" name="mapviz" output="screen"/>
</launch>

I think that similar to #618, these plugins will have issues rendering if your Fixed Frame is set to /wgs84; try a world-fixed ROS tf frame instead (probably map) and see if they render properly then.
If you still don't see it render anything after switching frames, could you record a bag file with a few messages and attach it here?

Setting local_xy_frame to /map instead of /wgs84 as well as the frame_id of the navsatfix messages fixed this issue as well as #618.

Great, good to hear that fixed the issue.
For what it's worth, the frame_id in the actual messages shouldn't matter; anything that receives NavSatFix or GPSFix messages should assume that their coordinates are in WGS84.


Force user to enter numeric value for year manufactured

Closes #297
In the add device page, if the year manufactured value contains any non-numeric chars, it will show an error message to the user and won't let the user add the device. When editing a device, the user will not be able to input any non-numeric values.

The destination branch is Tom's PR, and there are many diffs because his branch isn't up to date with master. That's okay, we can wait for his PR to merge before we merge this one.

I think you need to merge in main?

I think you need to merge in main?

Now that Tom's PR is merged, I changed the destination branch to main

[ ] only allow 4 chars for year

Looks like we do not need to force it to an int type on the front end. As long as the value only contains numeric values, the deserialization on the backend can still convert it correctly

Year Manufactured can only be 4 chars long and an error message is shown if not


several doubts

Hello, I have some doubts:
I want to use the default layout, but how can I make the play button appear in the middle of the video frame?
It is typical, because the controls are hidden and only the image is seen, should always appear the button to play so that it is known that is a video.
The player should let choose the image that I want to show, this is possible or not yet?
How can I create the .vtt file. for each video any specific tool?

Hi @microonline, yes this is possible using the "htmlOnPauseBlockWidth" parameter. https://github.com/fluid-player/fluid-player#parameters
With this parameter you can set HTML to appear in the center of the player when paused. This HTML can then be an img tag with the source set to your play button, like shown in the below example (Note, the img src was taken from google images after a quick search):
var testVideo = fluidPlayer(
    'my-video',
    '',
    {
        htmlOnPauseBlock: '<img src="https://lh3.googleusercontent.com/N-JyvZjqZhnvydN9Q7Hbzwh6aAQmvnAmf1TjlvqHpCnvVl5zwSU7jlJH0uLFpNiRFw=w170"/>',
    }
);

The above solution will not show the image before the video has started however. We have discussed this and hope to introduce a parameter that will allow this image to show before play as well.
Regarding the .vtt file creation we do not provide a tool, however the one used to generate the example .vtt file was the following - https://github.com/amnuts/jwplayer-thumbnail-preview-generator
I hope the above information helps achieve what you're looking for. If you have any further questions feel free to ask.

@cian-exads thanks for the help, but I can not use the parameter: htmlOnPauseBlock, because I'm using exoclick's in-video advertising.
I think it's as you say, they should include the play button inside the player so that it is seen before the video starts, because if the controls are hidden automatically, and there is no play button it just looks like an image and not a video.
I'll try to try: https://github.com/amnuts/jwplayer-thumbnail-preview-generator
thanks for your help again.

Faced the same problem. It is very strange that there is still no solution. On mobile devices, the video looks like just a picture: there is not a Play button in the middle of the video, even the bottom panel with buttons is not shown. Users think that this is just a picture.

Hi @Perdator, we are currently addressing this issue, and the feature will be available in the next release. We're looking forward to releasing this feature as it's in high demand.

Available since v2.0.0
The play button is show by default now, if you want to hide it, you can set:
layoutControls: {
playButtonShowing: false // Default true
}


Newlines missing from module inspect results

Not sure if this issue is in IJulia or base, but if you inspect a module the newlines after links/URL are truncated so they dont render correctly in a notebook etc.
To reproduce, (JuMP docs as an example):
using JuMP
IJulia.docdict("JuMP")["text/markdown"]

or
?JuMP

...etc
The newlines after urls are missing so all appear on one line in raw markdown.

I get the same thing for ?JuMP in the REPL, so it's not an IJulia issue.  It may be a problem with Julia's Markdown parser—maybe it doesn't recognize standalone links like this, as opposed to embedded [link](url) links.


Request by TLSv1

Hi all.
I have api client on Fuel.
Sometimes, if I have a bad network connection I observe what my request going by TLSv1.
it happens when I use my application on Android 8 or 9.
What must I get to you to resolve my trouble?
I have Charles dump and Wireshark dump where you can see it.
Advance thanks.

Can you post your snippet and explain what went wrong? Sorry I couldn't grasp your question 🙏

You wanted Fuel to force TLSv1 in certain url addresses?
Sent with GitHawk

Can you post your snippet and explain what went wrong? Sorry I couldn't grasp your question 🙏

Yes of course, but only tomorrow.

You wanted Fuel to force TLSv1 in certain url addresses?
Sent with GitHawk

No. I want to exclude TLS1 from client-server communicating. Now, how I think, if I have a bad internet connection, during ssl handshake session, my application sends packages by TLS1 protocol. And I want to exclude these situations


Move sample data outside of Ref System

Move the sample data from this repo to it's own dedicated GitHub repo (sharedstreets-sample-data?)
After that, use bfg to clean the repo by removing the history of the sample data, if not the git repo will still remain large.
https://rtyley.github.io/bfg-repo-cleaner/
@kpwebb or @DenisCarriere (I can tackle this)

This current repo is 174MB in size 😨

ugh yeah, that was a dumb/stopgap move on my part and just made worse by pushing updates. let's move the sample data to S3

Oh, actually let's use github + gh-pages. Way simpler to update.

github + gh-pages.

👍 Sounds good, we can host it in a docs folder and have it hosted

Having issues with bfg at the moment.
$ git push
Counting objects: 515, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (222/222), done.
Writing objects: 100% (515/515), 15.27 MiB | 3.25 MiB/s, done.
Total 515 (delta 205), reused 438 (delta 190)
remote: Resolving deltas: 100% (205/205), done.
To github.com:sharedstreets/sharedstreets-ref-system.git
 * [new tag]         DRAFT-2017-11-20 -> DRAFT-2017-11-20
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/merge -> refs/pull/3/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/merge -> refs/pull/7/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/9/head -> refs/pull/9/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/9/merge -> refs/pull/9/merge (deny updating a hidden ref)
error: failed to push some refs to 'git@github.com:sharedstreets/sharedstreets-ref-system.git'

I removed the release.. (hope you don't mind) it was having issues removing the old "large" objects.
The repo is still ~174MB

Bfg history:
$ java -jar bfg.jar --strip-blobs-bigger-than 1M sharedstreets-ref-system.git

Using repo : /Users/mac/Github/sharedstreets-ref-system.git

This repo has been processed by The BFG before! Will prune repo before proceeding - to avoid unnecessary cleaning work on unused objects...
Completed prune of old objects - will now proceed with the main job!

Scanning packfile for large blobs: 2098
Scanning packfile for large blobs completed in 68 ms.
Found 124 blob ids for large blobs - biggest=44008074 smallest=1081293
Total size (unpacked)=491634934
Found 14 objects to protect
Found 10 commit-pointing refs : HEAD, refs/heads/master, refs/pull/3/head, ...

Updating 9 Refs
---------------

	Ref                          Before     After
	------------------------------------------------
	refs/heads/master          | 29cc7ba7 | 88fd5747
	refs/pull/3/head           | 99f95164 | 1445a336
	refs/pull/3/merge          | d5e953b6 | efb82d72
	refs/pull/4/head           | 9ab13b0a | ed41ade9
	refs/pull/7/head           | 8e3d5108 | 39641567
	refs/pull/7/merge          | b17c5b28 | 53aa30bd
	refs/pull/9/head           | 17528369 | aa3d8668
	refs/pull/9/merge          | d4cb27e7 | 929bc618
	refs/tags/DRAFT-2017-11-20 | bafdce0b | 59584b84

Updating references:    100% (9/9)
...Ref update completed in 25 ms.

Commit Tree-Dirt History
------------------------

	Earliest                                              Latest
	|                                                          |
	........DDDDDDDDmDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD

	D = dirty commits (file tree fixed)
	m = modified commits (commit message or parents changed)
	. = clean commits (no changes to file tree)

	                        Before     After
	-------------------------------------------
	First modified commit | 476df001 | 173e2145
	Last dirty commit     | 1a8c444f | 328f2297

Deleted files
-------------

	Filename                          Git id
	-----------------------------------------------------------------------------------------
	10-301-384.geometry.geojson     | 33c89b1d (29.0 MB)
	10-301-384.geometry.json        | acc758fc (1.7 MB)
	10-301-384.intersection.geojson | 3ff87f8d (14.9 MB)
	10-301-384.metadata.geojson     | 515bc134 (14.8 MB)
	10-301-384.reference.geojson    | 1457d4be (42.0 MB)
	10-301-384.reference.json       | 316611f4 (2.0 MB)
	10-301-385.geometry.geojson     | 884f7327 (13.7 MB)
	10-301-385.geometry.json        | c683ca27 (2.4 MB)
	10-301-385.intersection.geojson | f1ef6b4d (7.1 MB)
	10-301-385.intersection.json    | 9adca706 (1.2 MB)
	10-301-385.metadata.geojson     | 61d235ea (7.1 MB)
	10-301-385.metadata.json        | 54719b6f (1.3 MB)
	10-301-385.reference.geojson    | 06423d9f (19.9 MB)
	10-301-385.reference.json       | b3dd55bc (3.0 MB)
	11-327-791.geometry.pbf         | 4912361c (2.9 MB), 50389513 (2.9 MB), ...
	...

In total, 241 object ids were changed. Full details are logged here:

	/Users/mac/Github/sharedstreets-ref-system.git.bfg-report/2018-02-01/12-40-08

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

Has the BFG saved you time?  Support the BFG on BountySource:  https://j.mp/fund-bfg

Done


automated federated metadata download

Does pysaml2 support automated download of federated metadata?  In particular I'm interested in InCommon metadata.
Or would I need to write a "loader" and pass it in to the config (I see that referenced in mdstore.py)?
mb

12 maj 2015 kl. 06:59 skrev mbasil77<EMAIL_ADDRESS>Does pysaml2 support automated download of federated metadata? In particular I'm interested in InCommon metadata.
Or would I need to write a ”loader" and pass it in to the config (I see that referenced in mdstore.py)?

You could if you wanted to but you can also use what’s build in.
In fact when verifying my answer I encountered a bug in handling external sources using the new config format.
I’ve fixed that now, just checked in the fix.
What you can do is the following:

Download the InCommon metadata certificate and verify it.
https://spaces.internet2.edu/display/InCFederation/Metadata+Signing+Certificate

configure metadata source in the configuration file like this:

”metadata”: [{
"class": "saml2.mdstore.MetaDataExtern",
"metadata": [
("http://md.incommon.org/InCommon/InCommon-metadata-export.xml",
"inc-md-cert.pem")]
}
]
Here I assume you have stored the InCommon certificate in the directory from where you start the IdP.

Roland

"It is the consequence of humanity. We are all formed of frailty and error; let us pardon reciprocally each others’ folly - that is the first law of nature.” - Voltaire

Awesome.  Thanks!
Mike Basil
312.436.1083
On Fri, May 15, 2015 at 12:35 PM, Roland Hedberg<EMAIL_ADDRESS>wrote:

12 maj 2015 kl. 06:59 skrev mbasil77<EMAIL_ADDRESS>Does pysaml2 support automated download of federated metadata? In
particular I'm interested in InCommon metadata.
Or would I need to write a ”loader" and pass it in to the config (I see
that referenced in mdstore.py)?

You could if you wanted to but you can also use what’s build in.
In fact when verifying my answer I encountered a bug in handling external
sources using the new config format.
I’ve fixed that now, just checked in the fix.
What you can do is the following:

Download the InCommon metadata certificate and verify it.

https://spaces.internet2.edu/display/InCFederation/Metadata+Signing+Certificate

configure metadata source in the configuration file like this:

”metadata”: [{
"class": "saml2.mdstore.MetaDataExtern",
"metadata": [
("http://md.incommon.org/InCommon/InCommon-metadata-export.xml",
"inc-md-cert.pem")]
}
]
Here I assume you have stored the InCommon certificate in the directory
from where you start the IdP.

Roland

"It is the consequence of humanity. We are all formed of frailty and
error; let us pardon reciprocally each others’ folly - that is the first
law of nature.” - Voltaire
—
Reply to this email directly or view it on GitHub
https://github.com/rohe/pysaml2/issues/216#issuecomment-102465280.

Actually, one caveat. The metadata URL I give in the example below is
pointing to the metadata set that Internet2 is publishing to eduGAIN.
If you want the whole set you should use:
http://md.incommon.org/InCommon/InCommon-metadata.xml
More about different aggregates on https://spaces.internet2.edu/display/InCFederation/Metadata+Aggregates

19 maj 2015 kl. 05:50 skrev mbasil77<EMAIL_ADDRESS>Awesome. Thanks!
Mike Basil
312.436.1083
On Fri, May 15, 2015 at 12:35 PM, Roland Hedberg<EMAIL_ADDRESS>wrote:

12 maj 2015 kl. 06:59 skrev mbasil77<EMAIL_ADDRESS>Does pysaml2 support automated download of federated metadata? In
particular I'm interested in InCommon metadata.
Or would I need to write a ”loader" and pass it in to the config (I see
that referenced in mdstore.py)?

You could if you wanted to but you can also use what’s build in.
In fact when verifying my answer I encountered a bug in handling external
sources using the new config format.
I’ve fixed that now, just checked in the fix.
What you can do is the following:

Download the InCommon metadata certificate and verify it.

https://spaces.internet2.edu/display/InCFederation/Metadata+Signing+Certificate

configure metadata source in the configuration file like this:

”metadata”: [{
"class": "saml2.mdstore.MetaDataExtern",
"metadata": [
("http://md.incommon.org/InCommon/InCommon-metadata-export.xml",
"inc-md-cert.pem")]
}
]
Here I assume you have stored the InCommon certificate in the directory
from where you start the IdP.

Roland

"It is the consequence of humanity. We are all formed of frailty and
error; let us pardon reciprocally each others’ folly - that is the first
law of nature.” - Voltaire
—
Reply to this email directly or view it on GitHub
https://github.com/rohe/pysaml2/issues/216#issuecomment-102465280.

—
Reply to this email directly or view it on GitHub.

Roland

"It is the consequence of humanity. We are all formed of frailty and error; let us pardon reciprocally each others’ folly - that is the first law of nature.” - Voltaire

I noticed.  Thanks.
Mike Basil
312.436.1083
On Tue, May 19, 2015 at 10:55 AM, Roland Hedberg<EMAIL_ADDRESS>wrote:

Actually, one caveat. The metadata URL I give in the example below is
pointing to the metadata set that Internet2 is publishing to eduGAIN.
If you want the whole set you should use:
http://md.incommon.org/InCommon/InCommon-metadata.xml
More about different aggregates on
https://spaces.internet2.edu/display/InCFederation/Metadata+Aggregates

19 maj 2015 kl. 05:50 skrev mbasil77<EMAIL_ADDRESS>Awesome. Thanks!
Mike Basil
312.436.1083
On Fri, May 15, 2015 at 12:35 PM, Roland Hedberg <
<EMAIL_ADDRESS>wrote:

12 maj 2015 kl. 06:59 skrev mbasil77<EMAIL_ADDRESS>Does pysaml2 support automated download of federated metadata? In
particular I'm interested in InCommon metadata.
Or would I need to write a ”loader" and pass it in to the config (I
see
that referenced in mdstore.py)?

You could if you wanted to but you can also use what’s build in.
In fact when verifying my answer I encountered a bug in handling
external
sources using the new config format.
I’ve fixed that now, just checked in the fix.
What you can do is the following:

Download the InCommon metadata certificate and verify it.

https://spaces.internet2.edu/display/InCFederation/Metadata+Signing+Certificate

configure metadata source in the configuration file like this:

”metadata”: [{
"class": "saml2.mdstore.MetaDataExtern",
"metadata": [
("http://md.incommon.org/InCommon/InCommon-metadata-export.xml",
"inc-md-cert.pem")]
}
]
Here I assume you have stored the InCommon certificate in the directory
from where you start the IdP.

Roland

"It is the consequence of humanity. We are all formed of frailty and
error; let us pardon reciprocally each others’ folly - that is the
first
law of nature.” - Voltaire
—
Reply to this email directly or view it on GitHub
https://github.com/rohe/pysaml2/issues/216#issuecomment-102465280.

—
Reply to this email directly or view it on GitHub.

Roland

"It is the consequence of humanity. We are all formed of frailty and
error; let us pardon reciprocally each others’ folly - that is the first
law of nature.” - Voltaire
—
Reply to this email directly or view it on GitHub
https://github.com/rohe/pysaml2/issues/216#issuecomment-103561540.

The request seems to be satisfied, so I'm closing this


Bluetooth: controller: Fix encryption setup to be queueable

Fix the controller implementation to make start encryption
queueable if there is any control procedure in progress.
The context related to encryption procedure is not shared so
that it will be used as the ongoing procedure completes.
The fix here maintains the old functionality of serializing
the queued data and LL Encryption Request PDU, so that data
queued before start encryption is acknowledged.
Fixes #15012. (which was marked as fixed but still had an outstanding issue)
Signed-off-by: Vinayak Kariappa Chettimada<EMAIL_ADDRESS>

@joerchan @carlescufi Ran encryption related 37 conformance tests, no regression.

@thoh-ot is unable to review this until after Easter holidays. Hence, could we find another reviewer?

Thanks @mped-oticon and @Qbicz.
I have fixed commit message for misspelled 'not' -> 'now' and 'at' -> 'after'.

@Qbicz thank you for testing.

@cvinayak Actually now I noticed after reconnecting the peripheral 4 times, I get an assertion on central. I can reproduce it every time (after rebooting central 4th reconnection from peripheral causes this error again).
The assertion occurs in bt_recv_prio for hdr->evt = 26.
[02021262] <err> bt_hci_core: F: bt_recv_prio, evt 14
[02022348] <inf> ble_state: Connected to c4:ac:5a:2a:d5:21 (random)
[02022353] <err> bt_hci_core: F: bt_recv_prio, evt 15
[02022419] <err> bt_hci_core: F: bt_recv_prio, evt 15
[02024077] <err> bt_hci_core: F: bt_recv_prio, evt 15
[02040435] <inf> ble_state: Security with c4:ac:5a:2a:d5:21 (random) level 2
[02040447] <err> bt_ctlr_hci: Tx Buffer Overflow
[02040448] <err> bt_hci_core: F: bt_recv_prio, evt 26
[02040449] <err> bt_hci_core: assert: 'bt_hci_evt_is_prio(hdr->evt)' failed
***** Kernel OOPS! *****
Current thread ID = 0x200022dc
Faulting instruction address = 0x1a30e
Fatal fault in thread 0x200022dc! Aborting.
[0ASSERTION FAIL [out_ctx->control_block->offset <= out_ctx->size] @ /home/fi/ncs/zephyr/subsys/logging/log_output.c:101
***** Kernel Panic! *****
Current thread ID = 0x20003d14
Faulting instruction address = 0x8b3c

With @Qbicz and @joerchan help found the bug, in the form of missing code to update data queue head and tail after the enc_req pdu was moved to end of control queue. PR commit updated.

@cvinayak tested with latest pushed changes. I did connect+encrypt+disconnect 15 times and cannot see any issue anymore. Thanks for help @carlescufi and @joerchan


Shim tflm_runtime.py to not break existing code after #2030

BUG=http://b/286456378

@rascani, ready for review again. I had missed adding a newly created file.


Pressure setpoint square wave is not always consistent when running on Arduino

Follow the instructions below to chart a pressure setpoint graph from the Arduino.  This shows the system pressure the Arduino wants to achieve.
This chart should be a square wave with a consistent period.  But every once in a while, we "miss a beat" and the inhale or exhale is too short or long.  It's very obvious in the graph.  It happens periodically, and for me always happens first around t=300 in the graph:

You don't need pressure sensors hooked up or anything; I've reproduced it with the Arduino alone.

Build at d8020002bc43fc794bf59613c79cb506104455a0, running on an Arduino RF Nano.

Enable NO_GUI_DEV_MODE by uncommenting this line https://github.com/RespiraWorks/VentilatorSoftware/blob/d8020002bc43fc794bf59613c79cb506104455a0/controller/src/main.cpp#L68

Run on the device, pio run -t upload

Immediately start capturing a pressure trace using the Arduino IDE (tools -> serial plotter).  Use 115200 baud.

What is wrong
Square wave should be consistent, but it's not; it sometimes exhales for too long (attached image above).
How do you know it is wrong
By definition command-pressure has a consistent pressure wave.
Code pointers
blower_fsm.cpp?

I don't know what's happening here.  It's tempting to think that this is due to some operation taking way too long and hanging up the whole main loop.  But I don't think that's it; if it were, we wouldn't be receiving any data from the device.
Something else is wrong.

Putting this in the alpha controller milestone because I think it would be pretty embarrassing for us to show a graph to UCSF or whoever if we can't even consistently breathe in and out with the right timings.

What are the units on the x-axis? How bad is the delay?

One tick on the x-axis is 1/10s.  The square wave is one breath.  This is a highly significant delay.

I can't reproduce it on STM32. Might be related to some long-ish float calculation.

Thanks for running that test, Albertas.  That's really useful.

Might be related to some long-ish float calculation.

If the calculation were stuck, the whole main loop would be stuck, and we wouldn't receive any data for that period of time.  But we do receive  data, indicating that  the loop isn't stuck.

Something is wrong with the timer.
In this chart, the orange line that's a step function is charting Hal.now().millisSinceStartup() / 1000 % 10.

This next chart tracks both native Arduino millis() and Hal.now().millisSinceStartup().  I am not sure which line is which, but I think it's the Hal line that's misbehaving.

Ah, this is 16-bit overflow in the hal timer.


libswoc: Remove legacy BufferWriter.

Looking at this, I'm tempted to do a bit more cleanup because the use of BufferWriter has evolved since it was first added.

[approve ci autest]

[approve ci rocky]


Use existing NODE_ENV if available

Allows webpack config to respond to e.g., test environments

Good call.


Delay is wrongly treated as IntConst

Example:
module main;
  parameter p_dly = 0;
  wire a, b;
  assign #p_dly a = b;
endmodule

Uhdm output:
  |vpiContAssign:
  \_cont_assign: , line:4:17, endln:4:22
    |vpiParent:
    \_module_inst: work@main (work@main), file:abc.v, line:1:1, endln:5:10
    |vpiDelay:
    \_constant: , line:4:10, endln:4:16
      |vpiDecompile:p_dly
      |vpiSize:64
      |UINT:0
      |vpiConstType:9

Note that the non-numeric delay is being parsed as a number.
The issue is from the following snippet where delay_value is assumed to be an IntConst which isn't true as per the grammar.
https://github.com/chipsalliance/Surelog/blob/909e2bd9783f4ecf6e7130415b5d0cf12adf71ca/src/SourceCompile/SV3_1aTreeShapeListener.cpp#L854-L864
https://github.com/chipsalliance/Surelog/blob/909e2bd9783f4ecf6e7130415b5d0cf12adf71ca/grammar/SV3_1aParser.g4#L1000-L1008
Following tests are impacted by this issue -

LibraryIntercon
Icarus (sched1.v)
Unisim (ISERDES.v)

Fixed by #3453

The Verilog grammar is ambiguous and cannot be fixed.
The UHDM tree can be fixed as you can see with some hacking of the AST and associated interpretation when creating the UHDM model.
Don't look at the AST, look at the UHDM model, do you see any issues in the UHDM model for your examples?

Then please take on this task. Good luck with fixing the Verilog grammar :-)
This is not my priority. As long as the resulting UHDM is correct, it achieves my goal.


Make thread id a tag instead of field in BankingStage metrics

Summary of Changes
Make the thread id a tag instead of a field in all of the BankingStage per-thread metrics. Doing so will allow group-by operations on id. Originally proposed (but never shipped) in https://github.com/solana-labs/solana/pull/25619

Codecov Report
Attention: Patch coverage is 74.28571% with 9 lines in your changes are missing coverage. Please review.

Project coverage is 81.8%. Comparing base (03ef611) to head (fbf3ff2).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##           master     #644     +/-   ##
=========================================
- Coverage    81.8%    81.8%   -0.1%
=========================================
  Files         851      851
  Lines      230238   230251     +13
=========================================
- Hits       188520   188498     -22
- Misses      41718    41753     +35

Don't have much runtime yet, but as an example, here is one of these metrics (qos-service-stats) with data grouped by id: