# Keeping IterableDataset node-wise split fixed during DDP

**URL:** <https://discuss.huggingface.co/t/keeping-iterabledataset-node-wise-split-fixed-during-ddp/58713>\
**Category:** 🤗Datasets\
**Created:** [October 16, 2023, 11:05am UTC](https://discuss.huggingface.co/t/keeping-iterabledataset-node-wise-split-fixed-during-ddp/58713 "2023-10-16T11:05:07Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![s9ngularity](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@s9ngularity](https://discuss.huggingface.co/u/s9ngularity)\
**Post date:** [October 16, 2023, 5:33pm UTC](https://discuss.huggingface.co/t/keeping-iterabledataset-node-wise-split-fixed-during-ddp/58713/3 "2023-10-16T17:33:03Z")

</div>

Thanks, yes, the dataset has many shards.

But could you give the example how to keep the dataset split fixed for each node during the whole DDP run?

The documentation had the following example for `split_dataset_by_node`:

```python
from datasets.distributed import split_dataset_by_node
ids = ds.to_iterable_dataset(num_shards=512)
ids = ids.shuffle(buffer_size=10_000) # will shuffle the shards order and use a shuffle buffer when you start iterating
ids = split_dataset_by_node(ids, world_size=8, rank=0) # will keep only 512 / 8 = 64 shards from the shuffled lists of shards when you start iterating
dataloader = torch.utils.data.DataLoader(ids, num_workers=4) # will assign 64 / 4 = 16 shards from this node's list of shards to each worker when you start iterating
for example in ids:
    pass

```

However, if I understand correctly, in this example each node would get a new random subset of shards every epoch, and then my nodes cannot to re-use the previous local cache.

Thanks again!

---

_[View the full topic](https://discuss.huggingface.co/t/keeping-iterabledataset-node-wise-split-fixed-during-ddp/58713)._