# Nested datasets and oversampling

**URL:** <https://discuss.huggingface.co/t/nested-datasets-and-oversampling/6897>\
**Category:** 🤗Datasets\
**Created:** [June 16, 2021, 3:19pm UTC](https://discuss.huggingface.co/t/nested-datasets-and-oversampling/6897 "2021-06-16T15:19:53Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![mralexis](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mralexis](https://discuss.huggingface.co/u/mralexis)\
**Post date:** [June 16, 2021, 3:19pm UTC](https://discuss.huggingface.co/t/nested-datasets-and-oversampling/6897/1 "2021-06-16T15:19:53Z")

</div>

Does `datasets` support nested datasets? I have a nested dataset path dict, e.g.,

```auto
{
	a: {
		train:
		dev:
		test:
	},
	b: {
		train:
		dev:
		test:
	}
}

```

and `load_dataset` seems unhappy with this structure (`TypeError: expected str, bytes or os.PathLike object, not dict`). Does that imply I have to do `load_dataset` for each sub-dataset and then merge them together?

Further, is there any support for over/undersampling given multiple (sub-)datasets? Thanks a lot!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [June 28, 2021, 10:09am UTC](https://discuss.huggingface.co/t/nested-datasets-and-oversampling/6897/2 "2021-06-28T10:09:00Z")

</div>

Hi !  
Indeed `load_dataset` is used to load only one dataset. This dataset can have several splits though.  
In your case you have to call `load_dataset` once per dataset.

Regarding over/undersampling, we are adding an `interleave_datasets` functions that creates a new dataset from several datasets. By default it alternates between each original dataset, but you can specify sampling probabilities to over/undersample from the original datasets 🙂  
It will be available in the next release of the library.

What you can do in the meantime is use a mix of `shuffle` and `conatenate_datasets`.  
If you have `dataset1` and `dataset2` and want to get `dataset3` with 10 examples from `datasets1` and 90 examples from `dataset2` you can do:

```python
from datasets import concatenate_datasets

seed = 42
dataset3 = concatenate_datasets([
    dataset1.shuffle(seed=seed).select(range(10)),
    dataset2.shuffle(seed=seed).select(range(90)),
])
```

---

<div class="post-metadata">

**Author:** ![mralexis](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mralexis](https://discuss.huggingface.co/u/mralexis)\
**Post date:** [June 29, 2021, 6:11am UTC](https://discuss.huggingface.co/t/nested-datasets-and-oversampling/6897/3 "2021-06-29T06:11:41Z")

</div>

Thanks for the reply! Regarding shuffling, my understanding is that by default when using trainer the dataset will be shuffled and then batched. So if I don’t need sampling, I could just concat all datasets and pass it to the trainer. Is that right?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [June 30, 2021, 12:54pm UTC](https://discuss.huggingface.co/t/nested-datasets-and-oversampling/6897/4 "2021-06-30T12:54:53Z")

</div>

I don’t think the trainer shuffles the data, so you should probably shuffle the dataset after concatenation

EDIT: actually it does, see thom’s comment

---

<div class="post-metadata">

**Author:** ![mralexis](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mralexis](https://discuss.huggingface.co/u/mralexis)\
**Post date:** [July 1, 2021, 4:00am UTC](https://discuss.huggingface.co/t/nested-datasets-and-oversampling/6897/5 "2021-07-01T04:00:44Z")

</div>

But it doesn’t seem to have any explicit mention of shuffling in the official examples, for which if there is no default shuffling in trainer then there should be issues (e.g., there is no “shuffl\*” in [this example](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization.py))?

---

<div class="post-metadata">

**Author:** ![thomwolf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomwolf/32/2198_2.png) [@thomwolf](https://discuss.huggingface.co/u/thomwolf)\
**Post date:** [July 5, 2021, 8:00pm UTC](https://discuss.huggingface.co/t/nested-datasets-and-oversampling/6897/6 "2021-07-05T20:00:29Z")

</div>

Yes the HF trainer shuffle the examples (see here for instance: [transformers/trainer.py at 4605b2b8ec5512a5ea125773bcaa4b0014b32d50 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/4605b2b8ec5512a5ea125773bcaa4b0014b32d50/src/transformers/trainer.py#L575))