# Using interleave\_datasets with probabilities

**URL:** <https://discuss.huggingface.co/t/using-interleave-datasets-with-probabilities/70741>\
**Category:** 🤗Datasets\
**Created:** [January 26, 2024, 4:46pm UTC](https://discuss.huggingface.co/t/using-interleave-datasets-with-probabilities/70741 "2024-01-26T16:46:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![pvversteeg](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@pvversteeg](https://discuss.huggingface.co/u/pvversteeg)\
**Post date:** [January 26, 2024, 4:46pm UTC](https://discuss.huggingface.co/t/using-interleave-datasets-with-probabilities/70741/1 "2024-01-26T16:46:24Z")

</div>

Hi,

I’m trying to use the `interleave_datasets` function with probabilities. Now, my use case some probabilities might be 0. For example, `probabilities=[0.5, 0.5, 0]` where I would like to sample from the first and second dataset at 50% likelihood, and never from the third.

Reading the documentation I figured this would be supported. But when trying it, the code seems to loop forever. Does anyone have any experience with this, or some more knowledge of the inner workings that could clear this up?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![pvversteeg](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@pvversteeg](https://discuss.huggingface.co/u/pvversteeg)\
**Post date:** [January 27, 2024, 6:13am UTC](https://discuss.huggingface.co/t/using-interleave-datasets-with-probabilities/70741/2 "2024-01-27T06:13:29Z")

</div>

Ok, I think I figured out why it was infinitely looping. I’m posting this in case it helps someone in the future, but just know it’s not more than an educated guess.

I was using `stopping_strategy='all_exhausted'`, so it would keep trying sample until it had fully sampled all datasets. This causes a problem when certain datasets had probability 0, so there was never any sampling from these. I think this caused the infinite loop.

I ended up solving it by doing a little preprocessing: delete all 0s from `probabilities` as well as their corresponding datasets from the `datasets` list.

Hope this helps someone in the future. If someone has a better understanding of the `interleave_datasets` function and thinks this is wrong, please let me know.