Loading partial dataset when debugging

Hi ! `load_dataset` downloads the full dataset once and caches it, so that subsequent calls to `load_dataset` just reloads the dataset from your disk.
Then when you specify a `split` in `load_dataset`, it will just load the requested split from the disk. If your specified split is a sliced split (e.g. `"train[:10]"`), then it will load the 10 first rows of the train split that you have on disk.

Therefore, as long as you don't delete your cache, all your calls to `load_dataset` will be very fast. Except the first call that downloads the dataset of course ^^

That’s a use case for the new streaming feature, no?

Hi @reachtarunhere.

Besides the above insights provided by @lhoestq and @thomwolf, there is also a Dataset feature in progress (I plan to finish it this week): #2249, which will allow you, when calling `load_dataset`, to pass the option to download/preprocess/cache only some specific split(s), which will definitely speed up your workflow.

If this feature is interesting for you, I can ping you once it will be merged into the master branch.

Thanks all for responding.

Hey @albertvillanova

Thanks. Yes, I would be interested.

@lhoestq I think even if a small split is specified it loads up the full dataset from the disk (please correct me if this is not the case). Because it does seem to be slow to me even on subsequent calls. There is no repeated downloading so it seems that the cache is working.

I am not aware of the streaming feature @thomwolf mentioned. So I might need to read up on it.

@reshinthadithyan  I use the .select function to have a fraction of indices.

I am using PyTorch Lightning along with datasets (thanks for so many datasets already prepared and the great splits).

Every time I execute load_dataset  for the imdb dataset it takes some time even if I specify a split involving very few samples. I guess this due to hashing as per the other issues.

Is there a way to only load part of the dataset on load_dataset? This would really speed up my workflow.
Something like a debug mode would really help. Thanks!


Add Parquet loader + from_parquet and to_parquet

`pyarrow` 1.0.0 doesn't support some types in parquet, we'll have to bump its minimum version.

Also I still need to add dummy data to test the parquet builder.

I had to bump the minimum pyarrow version to 3.0.0 to properly support parquet.

Everything is ready for review now :)
I reused pretty much the same tests we had for CSV

Done !
Now we're still allowing pyarrow>=1.0.0, but when users want to use parquet features they're asked to update to pyarrow>=3.0.0

Continuation of #2247

I added a "parquet" dataset builder, as well as the methods `Dataset.from_parquet` and `Dataset.to_parquet`.
As usual, the data are converted to arrow in a batched way to avoid loading everything in memory.


Use `Audio` features for `AutomaticSpeechRecognition` task template

enhancement

New feature or request

I'm just retaking and working on #2324. 😉

In #2533 we added a task template for speech recognition that relies on the file paths to the audio files. As pointed out by @SBrandeis this is brittle as it doesn't port easily across different OS'.

The solution is to use dedicated `Audio` features when casting the dataset. These features are not yet available in `datasets`, but should be included in the `AutomaticSpeechRecognition` template once they are.


Improve Features docs

- Fix rendering and cross-references in Features docs
- Add docstrings to Features methods


Sync with transformers disabling NOTSET

Nice thanks ! I think there are other places with
```python
not_verbose = bool(logger.getEffectiveLevel() > WARNING)
```

Could you replace them as well ?

Sure @lhoestq! I was not sure if this change should only be circumscribed to `http_get`...

Close #2528.


Add task template for automatic speech recognition

@SBrandeis @lhoestq i've integrated your suggestions, so this is ready for another review :)

This PR adds a task template for automatic speech recognition. In this task, the input is a path to an audio file which the model consumes to produce a transcription.

Usage:

```python
from datasets import load_dataset
from datasets.tasks import AutomaticSpeechRecognition

ds = load_dataset("timit_asr", split="train[:10]")
# Dataset({
#     features: ['file', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
#     num_rows: 10
# })

task = AutomaticSpeechRecognition(audio_file_column="file", transcription_column="text")
ds.prepare_for_task(task)
# Dataset({
#     features: ['audio_file', 'transcription'],
#     num_rows: 10
# })
```


Tokenizer's normalization preprocessor cause misalignment in return_offsets_mapping for tokenizer classification task

bug

Something isn't working

> Hi @jerryIsHere, thanks for reporting the issue. But are you sure this is a bug in HuggingFace **Datasets**?

Oh, I am sorry
I would reopen the post on huggingface/transformers

[This colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) implements a token classification input pipeline extending the logic from [this hugging example](https://huggingface.co/transformers/custom_datasets.html#tok-ner).

The pipeline works fine with most instance in different languages, but unfortunately, [the Japanese Kana ligature (a form of abbreviation? I don't know Japanese well)](https://en.wikipedia.org/wiki/Kana_ligature) break the alignment of `return_offsets_mapping`:
![image](https://user-images.githubusercontent.com/50871412/122904371-db192700-d382-11eb-8917-1775db76db69.png)

Without the try catch block, it riase `ValueError: NumPy boolean array indexing assignment cannot assign 88 input values to the 87 output values where the mask is true`, example shown here [(another colab notebook)](https://colab.research.google.com/drive/1MmOqf3ppzzdKKyMWkn0bJy6DqzOO0SSm?usp=sharing)

It is clear that the normalizer is the process that break the alignment, as it is observed that `tokenizer._tokenizer.normalizer.normalize_str('ヿ')` return 'コト'.

One workaround is to include `tokenizer._tokenizer.normalizer.normalize_str` before the tokenizer preprocessing pipeline, which is also provided in the [first colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) with the name `udposTestDatasetWorkaround`.

I guess similar logics should be included inside the tokenizer and the offsets_mapping generation process such that user don't need to include them in their code. But I don't understand the code of tokenizer well that I think I am not able to do this.

p.s.
**I am using my own dataset building script in the provided example, but the script should be equivalent to the changes made by this [update](https://github.com/huggingface/datasets/pull/2466)**
`get_dataset `is just a simple wrapping for `load_dataset`
and the `tokenizer` is just `XLMRobertaTokenizerFast.from_pretrained("xlm-roberta-large")`


Fix dev version

The dev version that ends in `.dev0` should be greater than the current version.
However it happens that `1.8.0 > 1.8.0.dev0` for example.
Therefore we need to use `1.8.1.dev0` for example in this case.

I updated the dev version to use `1.8.1.dev0`, and I also added a comment in the setup.py in the release steps about this.


Fixed label parsing in the ProductReviews dataset

@lhoestq, can you please review this PR?
What exactly is the problem in the test case? Should it matter?

Hi ! Thanks for fixing this :)

The CI fails for two reasons:
- the `pretty_name` tag is missing in yaml tags in ./datasets/turkish_product_reviews/README.md. You can fix that by adding this in the yaml tags:
```yaml
pretty_name: Turkish Product Reviews
```
- The test that runs the turkish_product_reviews.py file on the dummy_data.zip data returned 0 examples. Indeed it looks like you changed dummy_data.zip file and now it is an empty zip file. I think you can fix that by reverting your change to the dummy_data.zip file

> Hi ! Thanks for fixing this :)
>
> The CI fails for two reasons:
>
>     * the `pretty_name` tag is missing in yaml tags in ./datasets/turkish_product_reviews/README.md. You can fix that by adding this in the yaml tags:
>
>
> ```yaml
> pretty_name: Turkish Product Reviews
> ```
>
>     * The test that runs the turkish_product_reviews.py file on the dummy_data.zip data returned 0 examples. Indeed it looks like you changed dummy_data.zip file and now it is an empty zip file. I think you can fix that by reverting your change to the dummy_data.zip file

Many thanks for the quick feedback.
I made the relevant fixes but still got the error :(

> Thanks !
> The CI was failing because of the dataset card that was missing some sections. I fixed that.
>
> It's all good now

Super.  Thanks for the support.

Fixed issue with parsing dataset labels.


Add summarization template

> Nice thanks !
> Could you just move the test outside of the BaseDatasetTest class please ? Otherwise it will unnecessarily be run twice.

sure, on it! thanks for the explanations about the `self._to` method :)

@lhoestq i've moved all the task template tests outside of `BaseDatasetTest` and collected them in their dedicated test case. (at some point i'll revisit this so we can just use `pytest` natively, but the PR is already getting out-of-scope :))

This PR adds a task template for text summarization. As far as I can tell, we do not need to distinguish between "extractive" or "abstractive" summarization - both can be handled with this template.

Usage:

```python
from datasets import load_dataset
from datasets.tasks import Summarization

ds = load_dataset("xsum", split="train")
# Dataset({
#     features: ['document', 'summary', 'id'],
#     num_rows: 204045
# })

summarization = Summarization(text_column="document", summary_column="summary")
ds.prepare_for_task(summarization)
# Dataset({
#     features: ['text', 'summary'],
#     num_rows: 204045
# })
```


Logging cannot be set to NOTSET similar to transformers

Something isn't working

Hi @joshzwiebel, thanks for reporting. We are going to align with `transformers`.

## Describe the bug
In the transformers library you can set the verbosity level to logging.NOTSET to work around the usage of tqdm and IPywidgets, however in Datasets this is no longer possible. This is because transformers set the verbosity level of tqdm with [this](https://github.com/huggingface/transformers/blob/b53bc55ba9bb10d5ee279eab51a2f0acc5af2a6b/src/transformers/file_utils.py#L1449)
`disable=bool(logging.get_verbosity() == logging.NOTSET)`
and datasets accomplishes this like [so](https://github.com/huggingface/datasets/blob/83554e410e1ab8c6f705cfbb2df7953638ad3ac1/src/datasets/utils/file_utils.py#L493)
`not_verbose = bool(logger.getEffectiveLevel() > WARNING)`
## Steps to reproduce the bug
```python
import datasets
import logging
datasets.logging.get_verbosity = lambda : logging.NOTSET
datasets.load_dataset("patrickvonplaten/librispeech_asr_dummy")
```

## Expected results
The code should download and load the dataset as normal without displaying progress bars

## Actual results
```ImportError                               Traceback (most recent call last)
<ipython-input-4-aec65c0509c6> in <module>
----> 1 datasets.load_dataset("patrickvonplaten/librispeech_asr_dummy")

~/venv/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, **config_kwargs)
    713         dataset=True,
    714         return_resolved_file_path=True,
--> 715         use_auth_token=use_auth_token,
    716     )
    717     # Set the base path for downloads as the parent of the script location

~/venv/lib/python3.7/site-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, **download_kwargs)
    350                     file_path = hf_bucket_url(path, filename=name, dataset=False)
    351                 try:
--> 352                     local_path = cached_path(file_path, download_config=download_config)
    353                 except FileNotFoundError:
    354                     raise FileNotFoundError(

~/venv/lib/python3.7/site-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    289             use_etag=download_config.use_etag,
    290             max_retries=download_config.max_retries,
--> 291             use_auth_token=download_config.use_auth_token,
    292         )
    293     elif os.path.exists(url_or_filename):

~/venv/lib/python3.7/site-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    668                     headers=headers,
    669                     cookies=cookies,
--> 670                     max_retries=max_retries,
    671                 )
    672

~/venv/lib/python3.7/site-packages/datasets/utils/file_utils.py in http_get(url, temp_file, proxies, resume_size, headers, cookies, timeout, max_retries)
    493         initial=resume_size,
    494         desc="Downloading",
--> 495         disable=not_verbose,
    496     )
    497     for chunk in response.iter_content(chunk_size=1024):

~/venv/lib/python3.7/site-packages/tqdm/notebook.py in __init__(self, *args, **kwargs)
    217         total = self.total * unit_scale if self.total else self.total
    218         self.container = self.status_printer(
--> 219             self.fp, total, self.desc, self.ncols)
    220         self.sp = self.display
    221

~/venv/lib/python3.7/site-packages/tqdm/notebook.py in status_printer(_, total, desc, ncols)
     95         if IProgress is None:  # #187 #451 #558 #872
     96             raise ImportError(
---> 97                 "IProgress not found. Please update jupyter and ipywidgets."
     98                 " See https://ipywidgets.readthedocs.io/en/stable"
     99                 "/user_install.html")

ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.95-42.163.amzn2.x86_64-x86_64-with-debian-10.8
- Python version: 3.7.10
- PyArrow version: 3.0.0
I am running this code on Deepnote and which important to this issue **does not** support IPywidgets


Replace bad `n>1M` size tag

Some datasets were still using the old `n>1M` tag which has been replaced with tags `1M<n<10M`, etc.
This resulted in unexpected results when searching for datasets bigger than 1M on the hub, since it was only showing the ones with the tag `n>1M`.


Add COCO datasets

dataset request

Requesting to add a new dataset

vision

Vision datasets

I'm currently adding it, the entire dataset is quite big around 30 GB so I add splits separately. You can take a look here https://huggingface.co/datasets/merve/coco

I talked to @lhoestq and it's best if I download this dataset through TensorFlow datasets instead, so I'll be implementing that one really soon.
@NielsRogge

I started adding COCO, will be done tomorrow EOD
my work so far https://github.com/merveenoyan/datasets (my fork)

Hi Merve @merveenoyan , thank you so much for your great contribution! May I ask about the current progress of your implementation? Cuz I see the pull request is still in progess here. Or can I just run the COCO scripts in your fork repo?

> Hello @yixuanren I had another prioritized project about to be merged, but I'll start continuing today will finish up soon.

It's really nice of you!! I see you've commited another version just now

@yixuanren we're working on it, will be available soon, thanks a lot for your patience

## Adding a Dataset
- **Name:** COCO
- **Description:** COCO is a large-scale object detection, segmentation, and captioning dataset.
- **Paper + website:** https://cocodataset.org/#home
- **Data:** https://cocodataset.org/#download
- **Motivation:** It would be great to have COCO available in HuggingFace datasets, as we are moving beyond just text. COCO includes multi-modalities (images + text), as well as a huge amount of images annotated with objects, segmentation masks, keypoints etc., on which models like DETR (which I recently added to HuggingFace Transformers) are trained. Currently, one needs to download everything from the website and place it in a local folder, but it would be much easier if we can directly access it through the datasets API.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Use scikit-learn package rather than sklearn in setup.py

The sklearn package is an historical thing and should probably not be used by anyone, see https://github.com/scikit-learn/scikit-learn/issues/8215#issuecomment-344679114 for some caveats.

Note: this affects only TESTS_REQUIRE so I guess only developers not end users.


Raise FileNotFoundError in WindowsFileLock

Hi ! Could you clarify what it fixes exactly and give more details please ? Especially why this is related to the windows hanging error ?

This has already been merged, but I'll clarify the idea of this PR. Before this merge, FileLock was the only component affected by the max path limit on Windows (that came to my notice) because of its infinite loop that would suppress errors. So instead of suppressing the `FileNotFoundError` that is thrown by `os.open` if the file name is longer than the max allowed path length, this PR reraises it to notify the user.

Closes #2443


Fr

__Originally posted by @lewtun in https://github.com/huggingface/datasets/pull/2469__


Documentation Mistakes in Dataset: emotion

bug

Something isn't working

Hi,

this issue has been already reported in the dataset repo (https://github.com/dair-ai/emotion_dataset/issues/2), so this is a bug on their side.

The documentation has another bug in the dataset card [here](https://huggingface.co/datasets/emotion).

In the dataset summary **six** emotions are mentioned: *"six basic emotions: anger, fear, joy, love, sadness, and surprise"*, however, in the datafields section we have only **five**:
```
label: a classification label, with possible values including sadness (0), joy (1), love (2), anger (3), fear (4).
```

As per documentation,
Dataset: emotion
Homepage: https://github.com/dair-ai/emotion_dataset

Dataset: https://github.com/huggingface/datasets/blob/master/datasets/emotion/emotion.py

Permalink: https://huggingface.co/datasets/viewer/?dataset=emotion

Emotion is a dataset of English Twitter messages with eight basic emotions: anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. For more detailed information please refer to the paper.

But when we view the data, there are only 6 emotions, anger, fear, joy, sadness, surprise, and trust.


Insert text classification template for Emotion dataset

This PR includes a template and updated `dataset_infos.json` for the `emotion` dataset.


Datasets with tricky task templates

Dataset discussion

Discussions on the datasets

I'm collecting a list of datasets here that don't follow the "standard" taxonomy and require further investigation to implement task templates for.

## Text classification

* [hatexplain](https://huggingface.co/datasets/hatexplain): ostensibly a form of text classification, but not in the standard `(text, target)` format and each sample appears to be tokenized.
* [muchocine](https://huggingface.co/datasets/muchocine): contains two candidate text columns (long-form and summary) which in principle requires two `TextClassification` templates which is not currently supported


Improve performance of pandas arrow extractor

Looks like this change
```
pa_table[pa_table.column_names[0]].to_pandas(types_mapper=pandas_types_mapper)
```
doesn't return a Series with the correct type.
This is related to https://issues.apache.org/jira/browse/ARROW-9664

Since the types_mapper isn't taken into account, the ArrayXD types are not converted to the correct pandas extension dtype

@lhoestq I think I found a workaround... 😉

For some reason the benchmarks are not run Oo

Anyway, merging.
We'll see on master how much speed ups we got

While reviewing PR #2505, I noticed that pandas arrow extractor could be refactored to be faster.


Add task templates for tydiqa and xquad

This PR adds question-answering templates to the remaining datasets that are linked to a model on the Hub.

Notes:

* I could not test the tydiqa implementation since I don't have enough disk space 😢 . But I am confident the template works :)
* there exist other datasets like `fquad` and `mlqa` which are candidates for question-answering templates, but some work is needed to handle the ordering of nested column described in #2434


Fix typo in MatthewsCorrelation class name

Close #2513.


datasets.map pickle issue resulting in invalid mapping function

bug

Something isn't working

Hi ! `map` calls `__getstate__` using `dill` to hash your map function. This is used by the caching mechanism to recover previously computed results. That's why you don't see any `__setstate__` call.

Why do you change an attribute of your tokenizer when `__getstate__` is called ?

@lhoestq because if I try to pickle my custom tokenizer (it contains a pure python pretokenization step in an otherwise rust backed tokenizer) I get

> Exception: Error while attempting to pickle Tokenizer: Custom PreTokenizer cannot be serialized

So I remove the Custom PreTokenizer in `__getstate__` and then restore it in `__setstate__` (since it doesn't contain any state). This is what my `__getstate__`  / `__setstate__` looks like:

    def __getstate__(self):
        """
        Removes pre_tokenizer since it cannot be pickled
        """
        logger.debug("Copy state dict")
        out = self.__dict__.copy()
        logger.debug("Detaching pre_tokenizer")
        out['_tokenizer'].pre_tokenizer = tokenizers.pre_tokenizers.Sequence([])
        return out

    def __setstate__(self, d):
        """
        Reinstates pre_tokenizer
        """
        logger.debug("Reattaching pre_tokenizer")
        self.__dict__ = d
        self.backend_tokenizer.pre_tokenizer = self._pre_tokenizer()

If this is the case can you think of another way of avoiding my issue?

Actually, maybe I need to deep copy `self.__dict__`? That way `self` isn't modified. That was my intention and I thought it was working - I'll double-check after the weekend.

Doing a deep copy results in the warning:

> 06/20/2021 16:02:15 - WARNING - datasets.fingerprint -   Parameter 'function'=<function tokenize_function at 0x7f1e95f05d40> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.

```
def __getstate__(self):
    """
    Removes pre_tokenizer since it cannot be pickled
    """
    logger.debug("Copy state dict")
    out = copy.deepcopy(self.__dict__)
    logger.debug("Detaching pre_tokenizer")
    out['_tokenizer'].pre_tokenizer = tokenizers.pre_tokenizers.Sequence([])
    return out
```

Looks like there is still an object that is not pickable in your `tokenize_function` function.

You can test if an object can be pickled and hashed by using
```python
from datasets.fingerprint import Hasher

Hasher.hash(my_object)
```

Under the hood it pickles the object to compute its hash, so it calls `__getstate__` when applicable.

I figured it out, the problem is deep copy itself uses pickle (unless you implement `__deepcopy__`). So when I changed `__getstate__` it started throwing an error.

I'm sure there's a better way of doing this, but in order to return the `__dict__` without the non-pikelable pre-tokeniser and without modifying self I removed the pre-tokenizers, did a deep copy and then re-generated it.

It does work -  although I noticed Hasher doesn't call `__hash__` if the object being hashed implements it which I feel it should? If it did I could return a hash of the tokenizers.json file instead.

```
    def __getstate__(self):
        """
        Removes pre_tokenizer since it cannot be pickled
        """
        logger.debug("Copy state dict")
        self.backend_tokenizer.pre_tokenizer = tokenizers.pre_tokenizers.Sequence([])
        out = copy.deepcopy(self.__dict__)  #self.__dict__.copy()
        self.backend_tokenizer.pre_tokenizer = self._pre_tokenizer()

        return out
```

I'm glad you figured something out :)

Regarding hashing: we're not using hashing for the same purpose as the python `__hash__` purpose (which is in general for dictionary lookups). For example it is allowed for python hashing to not return the same hash across sessions, while our hashing must return the same hashes across sessions for the caching to work properly.

I trained my own tokenizer, and I needed to use a python custom class. Because of this I have to detach the custom step before saving and reattach after restore. I did this using the standard pickle `__get_state__` / `__set_state__` mechanism. I think it's correct but it fails when I use it inside a function which is mapped to a dataset, i.e. in the manner of run_mlm.py and other huggingface scripts.

The following reproduces the issue - most likely I'm missing something

A simulated tokeniser which can be pickled

```
class CustomTokenizer:
    def __init__(self):
        self.state = "init"

    def __getstate__(self):
        print("__getstate__ called")
        out = self.__dict__.copy()
        self.state = "pickled"
        return out

    def __setstate__(self, d):
        print("__setstate__ called")
        self.__dict__ = d
        self.state = "restored"

tokenizer = CustomTokenizer()
```

Test that it actually works - prints "__getstate__ called" and "__setstate__ called"
```
import pickle
serialized = pickle.dumps(tokenizer)
restored = pickle.loads(serialized)
assert restored.state == "restored"
```

Simulate a function that tokenises examples, when dataset.map is called, this function
```
def tokenize_function(examples):
    assert tokenizer.state == "restored" # this shouldn't fail but it does
    output = tokenizer(examples)           # this will fail as tokenizer isn't really a tokenizer
    return output
```

Use map to simulate tokenization
```
import glob
from datasets import load_dataset

assert tokenizer.state == "restored"
train_files = glob.glob('train*.csv')
validation_files = glob.glob('validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
)
```

What's happening is I can see that __getstate__ is called but not __setstate__, so the state of `tokenize_function` is invalid at the point that it's actually executed. This doesn't matter as far as I can see for the standard tokenizers as they don't use __getstate__ / __setstate__. I'm not sure if there's another hook I'm supposed to implement as well?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-22-a2aef4f74aaa> in <module>
      8 tokenized_datasets = datasets.map(
      9     tokenize_function,
---> 10     batched=True,
     11 )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    487                     desc=desc,
    488                 )
--> 489                 for k, dataset in self.items()
    490             }
    491         )

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1633                 fn_kwargs=fn_kwargs,
   1634                 new_fingerprint=new_fingerprint,
-> 1635                 desc=desc,
   1636             )
   1637         else:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    184         }
    185         # apply actual function
--> 186         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    187         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    188         # re-apply format to the output

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    395             # Call actual function
    396
--> 397             out = func(self, *args, **kwargs)
    398
    399             # Update fingerprint of in-place transforms + update in-place history of transforms

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, desc)
   1961                                 indices,
   1962                                 check_same_num_examples=len(input_dataset.list_indexes()) > 0,
-> 1963                                 offset=offset,
   1964                             )
   1965                         except NumExamplesMismatch:

~/.pyenv/versions/3.7.6/envs/xxx/lib/python3.7/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   1853                 effective_indices = [i + offset for i in indices] if isinstance(indices, list) else indices + offset
   1854             processed_inputs = (
-> 1855                 function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1856             )
   1857             if update_data is None:

<ipython-input-21-8ee4a8ba5b1b> in tokenize_function(examples)
      1 def tokenize_function(examples):
----> 2     assert tokenizer.state == "restored"
      3     tokenizer(examples)
      4     return examples


CRD3 dataset card

This PR adds additional information to the CRD3 dataset card.


Can datasets remove duplicated rows?

enhancement

New feature or request

Hi ! For now this is probably the best option.
We might add a feature like this in the feature as well.

Do you know any deduplication method that works on arbitrary big datasets without filling up RAM ?
Otherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases

Yes, I'd like to work on this feature once I'm done with #2500, but first I have to do some research, and see if the implementation wouldn't be too complex.

In the meantime, maybe [this lib](https://github.com/TomScheffers/pyarrow_ops) can help. However, note that this lib operates directly on pyarrow tables and relies only on `hash` to find duplicates (e.g. `-1` and `-2` have the same hash in Python 3, so this lib will treat them as duplicates), which doesn't make much sense.

> Hi ! For now this is probably the best option.
> We might add a feature like this in the feature as well.
>
> Do you know any deduplication method that works on arbitrary big datasets without filling up RAM ?
> Otherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases

Great if this is can be done. Thanks!!

Not sure if you are asking me. In any case I don't know of any unfortunately :( in practice if data is really large we normally do it with spark (only for info. I understand this is not useful in developing this library..)

Hello,

I'm also interested in this feature.
Has there been progress on this issue?

Could we use a similar trick as above, but with a better hashing algorithm like SHA?

We could also use a [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter), should we care a lot about collision in this case?

For reference, we can get a solution fairly easily if we assume that we can hold in memory all unique values.

```python
from datasets import Dataset
from itertools import cycle
from functools import partial

memory = set()
def is_unique(elem:Any , column: str, memory: set) -> bool:
    if elem[column] in memory:
        return False
    else:
        memory.add(elem[column])
        return True

# Example dataset
ds = Dataset.from_dict({"col1" : [sent for i, sent in zip(range(10), cycle(["apple", "orange", "pear"]))],
                                      "col2": [i % 5 for i in range(10)]})

# Drop duplicates in `ds` on "col1"
ds2 = ds.filter(partial(is_unique, column="col1", memory=memory))
```

Of course, we can improve the API so that we can introduce `Dataset.drop_duplicates`.
For the parallel version, we can use a shared memory set.

An approach that works assuming you can hold the all the unique document hashes in memory:
```python
from datasets import load_dataset

def get_hash(example):
    """Get hash of content field."""
    return {"hash": hash(example["content"])} # can use any hashing function here

def check_uniques(example, uniques):
    """Check if current hash is still in set of unique hashes and remove if true."""
    if example["hash"] in uniques:
        uniques.remove(example["hash"])
        return True
    else:
        return False

ds = load_dataset("some_dataset")
ds = ds.map(get_hash)
uniques = set(ds.unique("hash"))
ds_filter = ds.filter(check_uniques, fn_kwargs={"uniques": uniques})
```
If the `uniques` could be stored in arrow then no additional memory would used at all but I don't know if this is possible.

**Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..

**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...

**Additional context**
no


Corelation should be Correlation

Hi @colbym-MM, thanks for reporting. We are fixing it.


seqeval metric does not work with a recent version of sklearn: classification_report() got an unexpected keyword argument 'output_dict'

bug

Something isn't working

Sorry, I was using an old version of sequeval

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset, load_metric
seqeval = load_metric("seqeval")
seqeval.compute(predictions=[['A']], references=[['A']])
```

## Expected results
The function computes a dict with metrics

## Actual results
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-39-69a57f5cf06f> in <module>
      1 from datasets import load_dataset, load_metric
      2 seqeval = load_metric("seqeval")
----> 3 seqeval.compute(predictions=[['A']], references=[['A']])

~/p3/lib/python3.7/site-packages/datasets/metric.py in compute(self, *args, **kwargs)
    396             references = self.data["references"]
    397             with temp_seed(self.seed):
--> 398                 output = self._compute(predictions=predictions, references=references, **kwargs)
    399
    400             if self.buf_writer is not None:

~/.cache/huggingface/modules/datasets_modules/metrics/seqeval/81eda1ff004361d4fa48754a446ec69bb7aa9cf4d14c7215f407d1475941c5ff/seqeval.py in _compute(self, predictions, references, suffix)
     95
     96     def _compute(self, predictions, references, suffix=False):
---> 97         report = classification_report(y_true=references, y_pred=predictions, suffix=suffix, output_dict=True)
     98         report.pop("macro avg")
     99         report.pop("weighted avg")

TypeError: classification_report() got an unexpected keyword argument 'output_dict'
```

## Environment info
sklearn=0.24
datasets=1.1.3


Add C4

dataset request

Requesting to add a new dataset

Update on this: I'm computing the checksums of the data files. It will be available soon

## Adding a Dataset
- **Name:** *C4*
- **Description:** *https://github.com/allenai/allennlp/discussions/5056*
- **Paper:** *https://arxiv.org/abs/1910.10683*
- **Data:** *https://huggingface.co/datasets/allenai/c4*
- **Motivation:** *Used a lot for pretraining*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Should fix https://github.com/huggingface/datasets/issues/1710


Add align_labels_with_mapping to DatasetDict

https://github.com/huggingface/datasets/pull/2457 added the `Dataset.align_labels_with_mapping` method.
In this PR I also added `DatasetDict.align_labels_with_mapping`


Fix fingerprint when moving cache dir

Thanks @lhoestq, I'm starting reviewing this PR.

Yea issues on windows are about long paths, not long filenames.
We can make sure the lock filenames are not too long, but not for the paths

Took your suggestions into account @albertvillanova :)

The fingerprint of a dataset changes if the cache directory is moved.
I fixed that by setting the fingerprint to be the hash of:
- the relative cache dir (dataset_name/version/config_id)
- the requested split

Close #2496

I had to fix an issue with the filelock filename that was too long (>255). It prevented the tests to run on my machine. I just added `hash_filename_if_too_long` in case this happens, to not get filenames longer than 255.
We usually have long filenames for filelocks because they are named after the path that is being locked. In case the path is a cache directory that has long directory names, then the filelock filename could en up being very long.


Load Image Classification Dataset from Local

enhancement

New feature or request

Hi ! Is this folder structure a standard, a bit like imagenet ?
In this case maybe we can consider having a dataset loader for cifar-like, imagenet-like, squad-like, conll-like etc. datasets ?
```python
from datasets import load_dataset

my_custom_cifar = load_dataset("cifar_like", data_dir="path/to/data/dir")
```

Let me know what you think

Yep that would be sweet - closing for now as we found a workaround.

@lhoestq I think we'll want a generic `image-folder` dataset (same as 'imagenet-like'). This is like `torchvision.datasets.ImageFolder`, and is something vision folks are used to seeing.

Opening this back up, since I'm planning on tackling this. Already posted a quick version of it on my account on the hub.

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
```

Bumping this one following our recent discussion @mariosasko @nateraw :)

**Is your feature request related to a problem? Please describe.**
Yes - we would like to load an image classification dataset with datasets without having to write a custom data loader.

**Describe the solution you'd like**

Given a folder structure with images of each class in each folder, the ability to load these folders into a HuggingFace dataset like "cifar10".

**Describe alternatives you've considered**

Implement ViT training outside of the HuggingFace Trainer and without datasets (we did this but prefer to stay on the main path)

Write custom data loader logic

**Additional context**

We're training ViT on custom dataset


Rearrange JSON field names to match passed features schema field names

1.9

Next minor release

This PR depends on PR #2453 (which must be merged first).

Close #2366.


Add course banner

This PR adds a course banner similar to the one you can now see in the [Transformers repo](https://github.com/huggingface/transformers) that links to the course. Let me know if placement seems right to you or not, I can move it just below the badges too.


Make numpy arrow extractor faster

Looks like we have a nice speed up in some benchmarks. For example:
- `read_formatted numpy 5000`:  4.584777 sec -> 0.487113 sec
- `read_formatted torch 5000`:  4.565676 sec -> 1.289514 sec

Can we convert this draft to PR @lhoestq ?

@lhoestq I tried the branch and it works for me. Although performance trace now shows a speedup, the overall pre-training speed up is minimal. But that's on my plate to explore further.

Thanks for investigating @vblagoje

@albertvillanova , do you have any comments on this PR ? Otherwise I think we can merge it

I changed the NumpyArrowExtractor to call directly to_numpy and see if it can lead to speed-ups as discussed in https://github.com/huggingface/datasets/issues/2498

This could make the numpy/torch/tf/jax formatting faster


SubjQA wrong boolean values in entries

bug

Something isn't working

Hi @arnaudstiegler, thanks for reporting. I'm investigating it.

@arnaudstiegler I have just checked that these mismatches are already present in the original dataset: https://github.com/megagonlabs/SubjQA

We are going to contact the dataset owners to report this.

I have:
- opened an issue in their repo: https://github.com/megagonlabs/SubjQA/issues/3
- written an email to all the paper authors

Please [see my response](https://github.com/megagonlabs/SubjQA/issues/3#issuecomment-905160010). There will be a fix in a couple of days.

## Describe the bug
SubjQA seems to have a boolean that's consistently wrong.

It defines:
- question_subj_level: The subjectiviy level of the question (on a 1 to 5 scale with 1 being the most subjective).
- is_ques_subjective: A boolean subjectivity label derived from question_subj_level (i.e., scores below 4 are considered as subjective)

However, `is_ques_subjective` seems to have wrong values in the entire dataset.

For instance, in the example in the dataset card, we have:
- "question_subj_level": 2
- "is_ques_subjective": false

However, according to the description, the question should be subjective since the `question_subj_level` is below 4


JAX integration

Hi !

I just added the "jax" formatting, as we already have for pytorch, tensorflow, numpy (and also pandas and arrow).
It does pretty much the same thing as the pytorch formatter except it creates jax.numpy.ndarray objects.

```python
from datasets import Dataset

d = Dataset.from_dict({"foo": [[0., 1., 2.]]})
d = d.with_format("jax")
d[0]
# {'foo': DeviceArray([0., 1., 2.], dtype=float32)}
```

A few details:
- The default integer precision for jax depends on the jax configuration `jax_enable_x64` (see [here](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#double-64bit-precision)), I took that into account. Unless `jax_enable_x64` is specified, it is int32 by default
- AFAIK it's not possible to do a full conversion from arrow data to jax data. We are doing arrow -> numpy -> jax but the numpy -> jax part doesn't do zero copy unfortutanely (see [here](https://github.com/google/jax/issues/4486))
- the env var for disabling JAX is `USE_JAX`. However I noticed that in `transformers` it is `USE_FLAX`. This is not an issue though IMO

I also updated `convert_to_python_objects` to allow users to pass jax.numpy.ndarray objects to build a dataset.

Since the `convert_to_python_objects` method became slow because it's the time when pytorch, tf (and now jax) are imported, I fixed it by checking the `sys.modules` to avoid unecessary import of pytorch, tf or jax.

Close #2495


Add Zenodo metadata file with license

1.9

Next minor release

This Zenodo metadata file fixes the name of the `Datasets` license appearing in the DOI as `"Apache-2.0"`, which otherwise by default is `"other-open"`.

Close #2472.


Add load_dataset_builder

Hi @mariosasko, thanks for taking on this issue.

Just a few logistic suggestions, as you are one of our most active contributors ❤ :
- When you start working on an issue, you can self-assign it to you by commenting on the issue page with the keyword: `#self-assign`; we have implemented a GitHub Action to take care of that... 😉
- When you are still working on your Pull Request, instead of using the `[WIP]` in the PR name, you can instead create a *draft* pull request: use the drop-down (on the right of the *Create Pull Request* button) and select **Create Draft Pull Request**, then click **Draft Pull Request**.

I hope you find these hints useful. 🤗

@albertvillanova Thanks for the tips. When creating this PR, it slipped my mind that this should be a draft. GH has an option to convert already created PRs to draft PRs, but this requires write access for the repo, so maybe you can help.

Ready for the review!

One additional change. I've modified the `camelcase_to_snakecase`/`snakecase_to_camelcase` conversion functions to fix conversion of the names with 2 or more underscores (e.g. `camelcase_to_snakecase("__DummyDataset__")` would return `___dummy_dataset__`; notice one extra underscore at the beginning). The implementation is based on the [inflection](https://pypi.org/project/inflection/) library.

Thank you for adding this feature, @mariosasko - this is really awesome!

Tried with:
```
python -c "from datasets import load_dataset_builder; b = load_dataset_builder('openwebtext-10k'); print(b.cache_dir)"
Using the latest cached version of the module from /home/stas/.cache/huggingface/modules/datasets_modules/datasets
/openwebtext-10k/3a8df094c671b4cb63ed0b41f40fb3bd855e9ce2e3765e5df50abcdfb5ec144b (last modified on Wed May 12
20:22:53 2021)

since it couldn't be found locally at openwebtext-10k/openwebtext-10k.py

or remotely (FileNotFoundError).

/home/stas/.cache/huggingface/datasets/openwebtext10k/plain_text/1.0.0/3a8df094c671b4cb63ed0b41f40fb3bd855e9ce2e3765e5df50abcdfb5ec144b
```

The logger message (edited by me to add new lines to point the issues out) is a bit confusing to the user - that is what does `FileNotFoundError` refer to?

1. May be replace  `FileNotFoundError` with where it was looking for a file online. But then the remote file is there - it's found
2. I'm not sure why it says "since it couldn't be found locally" - as it is locally found at the cache folder and again what does " locally at openwebtext-10k/openwebtext-10k.py" mean - i.e. where does it look for it? Is it `./openwebtext-10k/openwebtext-10k.py` it's looking for? or in some specific dir?

If the cached version always supersedes any other versions perhaps this is what it should say?
```
found cached version at xxx, not looking for a local at yyy, not downloading remote at zzz
```

Hi ! Thanks for the comments

Regarding your last message:
You must pass `stas/openwebtext-10k` as in `load_dataset` instead of `openwebtext-10k`. Otherwise it doesn't know how to retrieve the builder from the HF Hub.

When you specify a dataset name without a slash, it tries to load a canonical dataset or it looks locally at ./openwebtext-10k/openwebtext-10k.py
Here since `openwebtext-10k` is not a canonical dataset and doesn't exist locally at ./openwebtext-10k/openwebtext-10k.py: it raised a FileNotFoundError.
As a fallback it managed to find the dataset script in your cache and it used this one.

Oh, I see, so I actually used an incorrect input. so it was a user error. Correcting it:

```
python -c "from datasets import load_dataset_builder; b = load_dataset_builder('stas/openwebtext-10k'); print(b.cache_dir)"
/home/stas/.cache/huggingface/datasets/openwebtext10k/plain_text/1.0.0/3a8df094c671b4cb63ed0b41f40fb3bd855e9ce2e3765e5df50abcdfb5ec144b
```

Now there is no logger message. Got it!

OK, I'm not sure the magical recovery it did in first place is most beneficial in the long run. I'd have rather it failed and said: "incorrect input there is no such dataset as 'openwebtext-10k' at <this path> or <this url>" - because if it doesn't fail I may leave it in the code and it'll fail later when another user tries to use my code and won't have the cache. Does it make sense? Giving me `this url` allows me to go to the datasets hub and realize that the dataset is missing the username qualifier.

> Here since openwebtext-10k is not a canonical dataset and doesn't exist locally at ./openwebtext-10k/openwebtext-10k.py: it raised a FileNotFoundError.

Except it slapped the exception name to ` remotely (FileNotFoundError).` which makes no sense.

Plus for the local it's not clear where is it looking relatively too when it gets `FileNotFoundError` - perhaps it'd help to use absolute path and use it in the message?

---------------

Finally, the logger format is not set up so the user gets a warning w/o knowing it's a warning. As you can see it's missing the WARNING pre-amble in  https://github.com/huggingface/datasets/pull/2500#issuecomment-874250500

i.e. I had no idea it was warning me of something, I was just trying to make sense of the message that's why I started the discussion and otherwise I'd have completely missed the point of me making an error.

Adds the `load_dataset_builder` function. The good thing is that we can reuse this function to load the dataset info without downloading the dataset itself.

TODOs:
- [x] Add docstring and entry in the docs
- [x] Add tests

Closes #2484


Python Programming Puzzles

dataset request

Requesting to add a new dataset

Thanks @VictorSanh!
There's also a [notebook](https://aka.ms/python_puzzles) and [demo](https://aka.ms/python_puzzles_study) available now to try out some of the puzzles

## Adding a Dataset
- **Name:** Python Programming Puzzles
- **Description:** Programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis
- **Paper:** https://arxiv.org/pdf/2106.05784.pdf
- **Data:** https://github.com/microsoft/PythonProgrammingPuzzles ([Scrolling through the data](https://github.com/microsoft/PythonProgrammingPuzzles/blob/main/problems/README.md))
- **Motivation:** Spans a large range of difficulty, problems, and domains. A useful resource for evaluation as we don't have a clear understanding of the abilities and skills of extremely large LMs.

Note: it's a growing dataset (contributions are welcome), so we'll need careful versioning for this dataset.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Improve torch formatting performance

enhancement

New feature or request

That’s interesting thanks, let’s see what we can do. Can you detail your last sentence? I’m not sure I understand it well.

Hi ! I just re-ran a quick benchmark and using `to_numpy()` seems to be faster now:

```python
import pyarrow as pa  # I used pyarrow 3.0.0
import numpy as np

n, max_length = 1_000, 512
low, high, size = 0, 2 << 16, (n, max_length)

table = pa.Table.from_pydict({
    "input_ids": np.random.default_rng(42).integers(low=low, high=high, size=size).tolist()
})

%%timeit
_ = table.to_pandas()["input_ids"].to_numpy()
# 1.44 ms ± 80.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
_ = table["input_ids"].to_pandas().to_numpy()
# 461 µs ± 14.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
_ = table["input_ids"].to_numpy()
# 317 µs ± 5.06 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

Currently the conversion from arrow to numpy is done in the NumpyArrowExtractor here:

https://github.com/huggingface/datasets/blob/d6d0ede9486ffad7944642ca9a326e058b676788/src/datasets/formatting/formatting.py#L143-L166

Let's update the NumpyArrowExtractor to call `to_numpy` directly and see how our github benchmarks evolve ?__

Sounds like a plan @lhoestq  If you create a PR I'll pick it up and try it out right away!

@lhoestq I can also prepare the PR, just lmk.

I created https://github.com/huggingface/datasets/pull/2505 if you want to play with it @vblagoje

> I’m not exactly sure how to read the graph but it seems that to_categorical take a lot of time here. Could you share more informations on the features/stats of your datasets so we could maybe design a synthetic datasets that looks more similar for debugging testing?

@thomwolf starting from the top, each rectangle represents the cumulative amount of it takes to execute the method call. Therefore, format_batch in torch_formatter.py takes ~20 sec, and the largest portion of that call is taken by to_pandas call and the smaller portion (grey rectangle) by the other method invocation(s) in format_batch (series_to_numpy etc).

Features of the dataset are BERT pre-training model input columns i.e:
```
f = Features({
 "input_ids": Sequence(feature=Value(dtype="int32")),
 "attention_mask": Sequence(feature=Value(dtype="int8")),
 "token_type_ids": Sequence(feature=Value(dtype="int8")),
 "labels": Sequence(feature=Value(dtype="int32")),
 "next_sentence_label": Value(dtype="int8")
})
```

I'll work with @lhoestq till we get to the bottom of this one.

@lhoestq the proposed branch is faster, but overall training speedup is a few percentage points. I couldn't figure out how to include the GitHub branch into setup.py, so I couldn't start NVidia optimized Docker-based pre-training run.  But on bare metal, there is a slight improvement. I'll do some more performance traces.

Hi @vblagoje, to install Datasets from @lhoestq PR reference #2505, you can use:
```shell
pip install git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head#egg=datasets
```

Hey @albertvillanova yes thank you, I am aware, I can easily pull it from a terminal command line but then I can't automate docker image builds as dependencies are picked up from setup.py and for some reason setup.py doesn't accept this string format.

@vblagoje in that case, you can add this to your `setup.py`:
```python
    install_requires=[
        "datasets @ git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head",
```

@lhoestq @thomwolf @albertvillanova The new approach is definitely faster, dataloader now takes less than 3% cumulative time (pink rectangle two rectangles to the right of tensor.py backward invocation)

![Screen Shot 2021-06-16 at 3 05 06 PM](https://user-images.githubusercontent.com/458335/122224432-19de4700-ce82-11eb-982f-d45d4bcc1e41.png)

When we drill down into dataloader next invocation we get:

![Screen Shot 2021-06-16 at 3 09 56 PM](https://user-images.githubusercontent.com/458335/122224976-a1c45100-ce82-11eb-8d40-59194740d616.png)

And finally format_batch:

![Screen Shot 2021-06-16 at 3 11 07 PM](https://user-images.githubusercontent.com/458335/122225132-cae4e180-ce82-11eb-8a16-967ab7c1c2aa.png)

Not sure this could be further improved but this is definitely a decent step forward.

> ```python
> datasets @ git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head
> ```

@albertvillanova how would I replace datasets dependency in https://github.com/huggingface/transformers/blob/master/setup.py as the above approach is not working.

@vblagoje I tested my proposed approach before posting it here and it worked for me.

Is it not working in your case because of the SSH protocol? In that case you could try the same approach but using HTTPS:
```
"datasets @ git+https://github.com/huggingface/datasets.git@refs/pull/2505/head",
```

Also note the blanks before and after the `@`.

@albertvillanova of course it works. Apologies. I needed to change datasets in all deps references , like [here](https://github.com/huggingface/transformers/blob/master/setup.py#L235) for example.

Is time spent casting an issue here? See https://github.com/huggingface/datasets/issues/4676 that Datasets can spend huge amounts of time repeatedly casting to Python objects.

**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors.

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded.

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call.

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion.

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach.

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%.


Use default cast for sliced list arrays if pyarrow >= 4

1.9

Next minor release

I believe we don't use PyArrow >= 4.0.0 because of some segfault issues:
https://github.com/huggingface/datasets/blob/1206ffbcd42dda415f6bfb3d5040708f50413c93/setup.py#L78
Can you confirm @lhoestq ?

@SBrandeis pyarrow version 4.0.1 has fixed that issue: #2489 😉

From pyarrow version 4, it is supported to cast sliced lists.

This PR uses default pyarrow cast in Datasets to cast sliced list arrays if pyarrow version is >= 4.

In relation with PR #2461 and #2490.

cc: @lhoestq, @abhi1thakur, @SBrandeis


Dataset fingerprint changes after moving the cache directory, which prevent cache reload when using `map`

`Dataset.map` uses the dataset fingerprint (a hash) for caching.
However the fingerprint seems to change when someone moves the cache directory of the dataset.

This is because it uses the default fingerprint generation:
1. the dataset path is used to get the fingerprint
2. the modification times of the arrow file is also used to get the fingerprint

To fix that we could set the fingerprint of the dataset to be a hash of (<dataset_name>, <config_name>, <version>, <script_hash>), i.e. a hash of the the cache path relative to the cache directory.


JAX formatting

We already support pytorch, tensorflow, numpy, pandas and arrow dataset formatting. Let's add jax as well


Improve docs on Enhancing performance

documentation

Improvements or additions to documentation

In the ["Enhancing performance"](https://huggingface.co/docs/datasets/loading_datasets.html#enhancing-performance) section of docs, add specific use cases:
- How to make datasets the fastest
- How to make datasets take the less RAM
- How to make datasets take the less hard drive mem

cc: @thomwolf


add tensorflow-macos support

ref - https://github.com/huggingface/datasets/issues/2068


Eduge

Hi, awesome folks behind the huggingface!

Here is my PR for the text classification dataset in Mongolian.

Please do let me know in case you have anything to clarify.

Thanks & Regards,
Enod


add eduge classification dataset

Closing this PR as I'll submit a new one - bug free


Allow latest pyarrow version

1.9

Next minor release

Allow latest pyarrow version, once that version 4.0.1 fixes the segfault bug introduced in version 4.0.0.

Close #2489.


Allow latest pyarrow version once segfault bug is fixed

enhancement

New feature or request

As pointed out by @symeneses (see https://github.com/huggingface/datasets/pull/2268#issuecomment-860048613), pyarrow has fixed the segfault bug present in version 4.0.0 (see https://issues.apache.org/jira/browse/ARROW-12568):
- it was fixed on 3 May 2021
- version 4.0.1 was released on 19 May 2021 with the bug fix


Set configurable downloaded datasets path

1.9

Next minor release

Part of #2480.


Set configurable extracted datasets path

1.9

Next minor release

Part of #2480.


Add Rico Dataset

Hi ! Thanks for adding this dataset :)

Regarding your questions:
1. We can have them as different configuration of the `rico` dataset
2. Yes please use the path to the image and not open the image directly, so that we can let users open the image one at at time during training if they want to for example. In the future we'll have an Image feature type that will decode the encoded image data on the fly when accessing the examples.
3. Feel free to keep the hierarchies as strings if they don't follow a fixed format
4. You can just return the path

Hi there!

I'm wanting to add the Rico datasets for software engineering type data to y'alls awesome library. However, as I have started coding, I've ran into a few hiccups so I thought it best to open the PR early to get a bit of discussion on how the Rico datasets should be added to the `datasets` lib.

1) There are 7 different datasets under Rico and so I was wondering, should I make a folder for each or should I put them as different configurations of a single dataset?
You can see the datasets available for Rico here: http://interactionmining.org/rico

2) As of right now, I have a semi working version of the first dataset which has pairs of screenshots and hierarchies from android applications. However, these screenshots are very large (1440, 2560, 3) and there are 66,000 of them so I am not able to perform the processing that  the `datasets` lib does after downloading and extracting the dataset since I run out of memory very fast. Is there a way to have `datasets` lib not put everything into memory while it is processing the dataset?

2.1) If there is not a way, would it be better to just return the path to the screenshots instead of the actual image?

3) The hierarchies are JSON objects and looking through the documentation of `datasets`, I didn't see any feature that I could use for this type of data. So, for now I just have it being read in as a string, is this okay or should I be doing it differently?

4) One of the Rico datasets is a bunch of animations (GIFs), is there a `datasets` feature that I can put this type of data into or should I just return the path as a string?

I appreciate any and all help I can get for this PR, I think the Rico datasets will be an awesome addition to the library :nerd_face: !


Implement layered building

New feature or request

As discussed with @stas00 and @lhoestq (see also here https://github.com/huggingface/datasets/issues/2481#issuecomment-859712190):

> My suggestion for this would be to have this enabled by default.
>
> Plus I don't know if there should be a dedicated issue to that is another functionality. But I propose layered building rather than all at once. That is:
>
> 1. uncompress a handful of files via a generator enough to generate one arrow file
> 2. process arrow file 1
> 3. delete all the files that went in and aren't needed anymore.
>
> rinse and repeat.
>
> 1. This way much less disc space will be required - e.g. on JZ we won't be running into inode limitation, also it'd help with the collaborative hub training project
> 2. The user doesn't need to go and manually clean up all the huge files that were left after pre-processing
> 3. It would already include deleting temp files this issue is talking about
>
> I wonder if the new streaming API would be of help, except here the streaming would be into arrow files as the destination, rather than dataloaders.


Implement loading a dataset builder

New feature or request

As discussed with @stas00 and @lhoestq, this would allow things like:

```python
from datasets import load_dataset_builder
dataset_name = "openwebtext"
builder = load_dataset_builder(dataset_name)
print(builder.cache_dir)
```


Use gc.collect only when needed to avoid slow downs

I continue thinking that the origin of the issue has to do with tqdm (and not with Arrow): this issue only arises for version 4.50.0 (and later) of tqdm, not for previous versions of tqdm.

My guess is that tqdm made a change from version 4.50.0 that does not properly release the iterable.

In https://github.com/huggingface/datasets/commit/42320a110d9d072703814e1f630a0d90d626a1e6 we added a call to gc.collect to resolve some issues on windows (see https://github.com/huggingface/datasets/pull/2482)

However calling gc.collect too often causes significant slow downs (the CI run time doubled).
So I just moved the gc.collect call to the exact place where it's actually needed: when post-processing a dataset


Allow to use tqdm>=4.50.0

We used to have permission errors on windows whith the latest versions of tqdm (see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/6365/workflows/24f7c960-3176-43a5-9652-7830a23a981e/jobs/39232))

They were due to open arrow files not properly closed by pyarrow.
Since https://github.com/huggingface/datasets/commit/42320a110d9d072703814e1f630a0d90d626a1e6 gc.collect is called each time we don't need an arrow file to make sure that the files are closed.

close https://github.com/huggingface/datasets/issues/2471

cc @lewtun


Delete extracted files to save disk space

New feature or request

1.10

Next minor release

My suggestion for this would be to have this enabled by default.

Plus I don't know if there should be a dedicated issue to that is another functionality. But I propose layered building rather than all at once. That is:

1. uncompress a handful of files via a generator enough to generate one arrow file
2. process arrow file 1
3. delete all the files that went in and aren't needed anymore.

rinse and repeat.

1. This way much less disc space will be required - e.g. on JZ we won't be running into inode limitation, also it'd help with the collaborative hub training project
2. The user doesn't need to go and manually clean up all the huge files that were left after pre-processing
3. It would already include deleting temp files this issue is talking about

I wonder if the new streaming API would be of help, except here the streaming would be into arrow files as the destination, rather than dataloaders.

As discussed with @stas00 and @lhoestq, allowing the deletion of extracted files would save a great amount of disk space to typical user.


Set download/extracted paths configurable

New feature or request

For example to be able to send uncompressed and temp  build files to another volume/partition, so that the user gets the minimal disk usage on their primary setup - and ends up with just the downloaded compressed data + arrow files, but outsourcing the huge files and building to another partition. e.g. on JZ there is a special partition for fast data, but it's also volatile, so only temp files should go there.

Think of it as `TMPDIR` so we need the equivalent for `datasets`.

As discussed with @stas00 and @lhoestq, setting these paths configurable may allow to overcome disk space limitation on different partitions/drives.

TODO:
- [x] Set configurable extracted datasets path: #2487
- [x] Set configurable downloaded datasets path: #2488
- [ ] Set configurable "incomplete" datasets path?


❌ load_datasets ❌


Create release script

New feature or request

Create a script so that releases can be done automatically (as done in `transformers`).


Fix docs custom stable version

1.9

Next minor release

I see that @lhoestq overlooked this PR with his commit 07e2b05. 😢

I'm adding a script so that this issue does not happen again.

For the moment, the script only includes `update_custom_js`, but in a follow-up PR I will include all the required steps to make a package release.

I think we just need to clarify the release process in setup.py instead of adding a script that does the replacement

@lhoestq I really think we should implement a script that performs the release (instead of doing it manually as it is done now), as it is already the case in `transformers`. I will do it in a next PR.

For the moment, this PR includes one of the steps of the release script.

Currently docs default version is 1.5.0. This PR fixes this and sets the latest version instead.


Add TimeDial

Hi @lhoestq,
I've pushed the updated README and tags. Let me know if anything is missing/needs some improvement!

~PS. I don't know why it's not triggering the build~

Dataset: https://github.com/google-research-datasets/TimeDial

To-Do: Update README.md and add YAML tags


Issue in timit_asr database

bug

Something isn't working

This bug was fixed in #1995. Upgrading datasets to version 1.6 fixes the issue!

Indeed was a fixed bug.
Works on version 1.8
Thanks

## Describe the bug
I am trying to load the timit_asr dataset however only the first record is shown (duplicated over all the rows).
I am using the next code line
dataset = load_dataset(“timit_asr”, split=“test”).shuffle().select(range(10))

The above code result with the same sentence duplicated ten times.
It also happens when I use the dataset viewer at Streamlit .

## Steps to reproduce the bug
from datasets import load_dataset
dataset = load_dataset(“timit_asr”, split=“test”).shuffle().select(range(10))
data = dataset.to_pandas()

# Sample code to reproduce the bug
```

## Expected results
table with different row information

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.4.1 (also occur in the latest version)
- Platform: Linux-4.15.0-143-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 1.15.3 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


cache_dir parameter for load_from_disk ?

enhancement

New feature or request

Hi ! `load_from_disk` doesn't move the data. If you specify a local path to your mounted drive, then the dataset is going to be loaded directly from the arrow file in this directory. The cache files that result from `map` operations are also stored in the same directory by default.

However note than writing data to your google drive actually fills the VM's disk (see https://github.com/huggingface/datasets/issues/643)

Given that, I don't think that changing the cache directory changes anything.

Let me know what you think

Thanks for your answer! I am a little surprised since I just want to read the dataset.

After debugging a bit, I noticed that the VM’s disk fills up when the tables (generator) are converted to a list:

https://github.com/huggingface/datasets/blob/5ba149773d23369617563d752aca922081277ec2/src/datasets/table.py#L850

If I try to iterate through the table’s generator e.g.:

`length = sum(1 for x in tables)`

the VM’s disk fills up as well.

I’m running out of Ideas 😄

Indeed reading the data shouldn't increase the VM's disk. Not sure what google colab does under the hood for that to happen

Apparently, Colab uses a local cache of the data files read/written from Google Drive. See:
- https://github.com/googlecolab/colabtools/issues/2087#issuecomment-860818457
- https://github.com/googlecolab/colabtools/issues/1915#issuecomment-804234540
- https://github.com/googlecolab/colabtools/issues/2147#issuecomment-885052636

**Is your feature request related to a problem? Please describe.**
When using Google Colab big datasets can be an issue, as they won't fit on the VM's disk. Therefore mounting google drive could be a possible solution. Unfortunatly when loading my own dataset by using the _load_from_disk_ function, the data gets cached to the VM's disk:

`
from datasets import load_from_disk

myPreprocessedData = load_from_disk("/content/gdrive/MyDrive/ASR_data/myPreprocessedData")

`
I know that chaching on google drive could slow down learning. But at least it would run.

**Describe the solution you'd like**
Add cache_Dir parameter to the load_from_disk function.

**Describe alternatives you've considered**
It looks like you could write a custom loading script for the load_dataset function. But this seems to be much too complex for my use case. Is there perhaps a template here that uses the load_from_disk function?


Add Disfl-QA

Sounds great! It'll make things easier for the user while accessing the dataset. I'll make some changes to the current file then.

I've updated with the suggested changes. Updated the README, YAML tags as well (not sure of Size category tag as I couldn't pass the path of `dataset_infos.json` for this dataset)

Dataset: https://github.com/google-research-datasets/disfl-qa

To-Do: Update README.md and add YAML tags


Fix automatic generation of Zenodo DOI

bug

Something isn't working

1.9

Next minor release

I have received a reply from Zenodo support:
> We are currently investigating and fixing this issue related to GitHub releases. As soon as we have solved it we will reach back to you.

Other repo maintainers had the same problem with Zenodo.

There is an open issue on their GitHub repo: zenodo/zenodo#2181

I have received the following request from Zenodo support:
> Could you send us the link to the repository as well as the release tag?

My reply:
> Sure, here it is:
> - Link to the repository: https://github.com/huggingface/datasets
> - Link to the repository at the release tag: https://github.com/huggingface/datasets/releases/tag/1.8.0
> - Release tag: 1.8.0

Zenodo issue has been fixed. The 1.8.0 release DOI can be found here: https://zenodo.org/record/4946100#.YMd6vKj7RPY

After the last release of Datasets (1.8.0), the automatic generation of the Zenodo DOI failed: it appears in yellow as "Received", instead of in green as "Published".

I have contacted Zenodo support to fix this issue.

TODO:
- [x] Check with Zenodo to fix the issue
- [x] Check BibTeX entry is right


Fix PermissionError on Windows when using tqdm >=4.50.0

Something isn't working

1.9

Next minor release

See: https://app.circleci.com/pipelines/github/huggingface/datasets/235/workflows/cfb6a39f-68eb-4802-8b17-2cd5e8ea7369/jobs/1111

```
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process
```


Crash when `num_proc` > dataset length for `map()` on a `datasets.Dataset`.

Something isn't working

Hi ! It looks like the issue comes from pyarrow. What version of pyarrow are you using ? How did you install it ?

Thank you for the quick reply! I have `pyarrow==4.0.0`, and I am installing with `pip`. It's not one of my explicit dependencies, so I assume it came along with something else.

Could you trying reinstalling pyarrow with pip ?
I'm not sure why it would check in your multicurtural-sc directory for source files.

Sure! I tried reinstalling to get latest. pip was mad because it looks like Datasets currently wants <4.0.0 (which is interesting, because apparently I ended up with 4.0.0 already?), but I gave it a shot anyway:

```bash
$ pip install --upgrade --force-reinstall pyarrow
Collecting pyarrow
  Downloading pyarrow-4.0.1-cp39-cp39-manylinux2014_x86_64.whl (21.9 MB)
     |████████████████████████████████| 21.9 MB 23.8 MB/s
Collecting numpy>=1.16.6
  Using cached numpy-1.20.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.4 MB)
Installing collected packages: numpy, pyarrow
  Attempting uninstall: numpy
    Found existing installation: numpy 1.20.3
    Uninstalling numpy-1.20.3:
      Successfully uninstalled numpy-1.20.3
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 3.0.0
    Uninstalling pyarrow-3.0.0:
      Successfully uninstalled pyarrow-3.0.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 1.8.0 requires pyarrow<4.0.0,>=1.0.0, but you have pyarrow 4.0.1 which is incompatible.
Successfully installed numpy-1.20.3 pyarrow-4.0.1
```

Trying it, the same issue:

![image](https://user-images.githubusercontent.com/1170062/121730226-3f470b80-caa4-11eb-85a5-684c44c816da.png)

I tried installing `"pyarrow<4.0.0"`, which gave me 3.0.0. Running, still, same issue.

I agree it's weird that pyarrow is checking the source code directory for its files. (There is no `pyarrow/` directory there.) To me, that makes it seem like an issue with how pyarrow is called.

Out of curiosity, I tried running this with fewer workers to see when the error arises:

- 1: ✅
- 2: ✅
- 4: ✅
- 8: ✅
- 10: ✅
- 11: ❌  🤔
- 12: ❌
- 16: ❌
- 32: ❌

checking my datasets:

```python
>>> datasets
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 389290
    })
    validation.sc: Dataset({
        features: ['text'],
        num_rows: 10  # 🤔
    })
    validation.wvs: Dataset({
        features: ['text'],
        num_rows: 93928
    })
})
```

New hypothesis: crash if `num_proc` > length of a dataset? 😅

If so, this might be totally my fault, as the caller. Could be a docs fix, or maybe this library could do a check to limit `num_proc` for this case?

Good catch ! Not sure why it could raise such a weird issue from pyarrow though
We should definitely reduce num_proc to the length of the dataset if needed and log a warning.

This has been fixed in #2566, thanks @connor-mccarthy !
We'll make a new release soon that includes the fix ;)

## Describe the bug
Crash if when using `num_proc` > 1 (I used 16) for `map()` on a `datasets.Dataset`.

I believe I've had cases where `num_proc` > 1 works before, but now it seems either inconsistent, or depends on my data. I'm not sure whether the issue is on my end, because it's difficult for me to debug! Any tips greatly appreciated, I'm happy to provide more info if it would helps us diagnose.

## Steps to reproduce the bug
```python
# this function will be applied with map()
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        padding=PaddingStrategy.DO_NOT_PAD,
        truncation=True,
    )

# data_files is a Dict[str, str] mapping name -> path
datasets = load_dataset("text", data_files={...})

# this is where the error happens if num_proc = 16,
# but is fine if num_proc = 1
tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=num_workers,
)
```

## Expected results
The `map()` function succeeds with `num_proc` > 1.

## Actual results
![image](https://user-images.githubusercontent.com/1170062/121404271-a6cc5200-c910-11eb-8e27-5c893bd04042.png)
![image](https://user-images.githubusercontent.com/1170062/121404362-be0b3f80-c910-11eb-9117-658943029aef.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.2
- Platform: Linux-5.4.0-73-generic-x86_64-with-glibc2.31
- Python version: 3.9.5
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, but I think N/A for this issue
- Using distributed or parallel set-up in script?: Multi-GPU on one machine, but I think also N/A for this issue


Bump tqdm version

i tried both the latest version of `tqdm` and the version required by `autonlp` - no luck with windows 😞

it's very weird that a progress bar would trigger these kind of errors, so i'll have a look to see if it's something unique to `datasets`

Closing since this is now fixed in #2482


Implement ClassLabel encoding in JSON loader

1.9

Next minor release

No, nevermind @lhoestq. Thanks to you for your reviews!

Close #2365.


change udpos features structure

Let's add the tags in another PR. Thanks again !

The structure is change such that each example is a sentence
The change is done for issues:
#2061
#2444

Close #2061 , close #2444.


adding masahaner dataset

Thanks a lot for the corrections and comments.

I have resolved point 2. The make style still throws some errors, please see below

black --line-length 119 --target-version py36 tests src benchmarks datasets/**/*.py metrics
/bin/sh: 1: black: not found
Makefile:13: recipe for target 'style' failed
make: *** [style] Error 127

Can you help to resolve this?

Thank you very much @lhoestq for the help.

Adding Masakhane dataset https://github.com/masakhane-io/masakhane-ner

@lhoestq , can you please review


fix: adjusting indexing for the labels.

> Good catch ! Thanks for fixing it

My pleasure🙏

The labels index were mismatching the actual ones used in the dataset. Specifically `0` is used  for `SUPPORTS` and `1` is used for `REFUTES`
After this change, the `README.md` now reflects the content of `dataset_infos.json`.

Signed-off-by: Matteo Manica <drugilsberg@gmail.com>


Fix proto_qa download link

Fixes #2459

Instead of updating the path, this PR fixes a commit hash as suggested by @lhoestq.


Merge DatasetDict and Dataset

enhancement

New feature or request

generic discussion

Generic discussion on the library

1.12

Next minor release

As discussed in #2424 and #2437 (please see there for detailed conversation):
- It would be desirable to improve UX with respect the confusion between DatasetDict and Dataset.
- The difference between Dataset and DatasetDict is an additional abstraction complexity that confuses "typical" end users.
- A user expects a "Dataset" (whatever it contains multiple or a single split) and maybe it could be interesting to try to simplify the user-facing API as much as possible to hide this complexity from the end user.

Here is a proposal for discussion and refined (and potential abandon if it's not good enough):
- let's consider that a DatasetDict is also a Dataset with the various split concatenated one after the other
- let's disallow the use of integers in split names (probably not a very big breaking change)
- when you index with integers you access the examples progressively in split after the other is finished (in a deterministic order)
- when you index with strings/split name you have the same behavior as now (full backward compat)
- let's then also have all the methods of a Dataset on the DatasetDict

The end goal would be to merge both Dataset and DatasetDict object in a single object that would be (pretty much totally) backward compatible with both.

There are a few things that we could discuss if we want to merge Dataset and DatasetDict:

1. what happens if you index by a string ? Does it return the column or the split ? We could disallow conflicts between column names and split names to avoid ambiguities. It can be surprising to be able to get a column or a split using the same indexing feature
   ```
   from datasets import load_dataset

   dataset = load_dataset(...)
   dataset["train"]
   dataset["input_ids"]
   ```
2. what happens when you iterate over the object ? I guess it should iterate over the examples as a Dataset object, but a DatasetDict used to iterate over the splits as they are the dictionary keys. This is a breaking change that we can discuss.

Moreover regarding your points:

- integers are not allowed as split names already
- it's definitely doable to have all the methods. Maybe some of them like train_test_split that is currently only available for Dataset can be tweaked to work for a split dataset

cc: @thomwolf @lhoestq


Support sliced list arrays in cast

There is this issue in pyarrow:
```python
import pyarrow as pa

arr = pa.array([[i * 10] for i in range(4)])
arr.cast(pa.list_(pa.int32()))  # works

arr = arr.slice(1)
arr.cast(pa.list_(pa.int32()))  # fails
# ArrowNotImplementedError("Casting sliced lists (non-zero offset) not yet implemented")
```

However in `Dataset.cast` we slice tables to cast their types (it's memory intensive), so we have the same issue.
Because of this it is currently not possible to cast a Dataset with a Sequence feature type (unless the table is small enough to not be sliced).

In this PR I fixed this by resetting the offset of `pyarrow.ListArray` arrays to zero in the table before casting.
I used `pyarrow.compute.subtract` function to update the offsets of the ListArray.

cc @abhi1thakur @SBrandeis


Revert default in-memory for small datasets

enhancement

New feature or request

1.8

Next minor release

Close #2458


`Proto_qa` hosting seems to be broken

bug

Something isn't working

@VictorSanh , I think @mariosasko is already working on it.

## Describe the bug
The hosting (on Github) of the `proto_qa` dataset seems broken. I haven't investigated more yet, just flagging it for now.

@zaidalyafeai if you want to dive into it, I think it's just a matter of changing the links in `proto_qa.py`

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("proto_qa")
```

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/load.py", line 751, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 630, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/hf/.cache/huggingface/modules/datasets_modules/datasets/proto_qa/445346efaad5c5f200ecda4aa7f0fb50ff1b55edde3003be424a2112c3e8102e/proto_qa.py", line 131, in _split_generators
    train_fpath = dl_manager.download(_URLs[self.config.name]["train"])
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 199, in download
    num_proc=download_config.num_proc,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 195, in map_nested
    return function(data_struct)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 218, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/iesl/protoqa-data/master/data/train/protoqa_train.jsonl
```


enhancement

New feature or request

1.8

Next minor release

Users are reporting issues and confusion about setting default in-memory to True for small datasets.

We see 2 clear use cases of Datasets:
- the "canonical" way, where you can work with very large datasets, as they are memory-mapped and cached (after every transformation)
- some edge cases (speed benchmarks, interactive/exploratory analysis,...), where default in-memory can explicitly be enabled, and no caching will be done

After discussing with @lhoestq we have agreed to:
- revert this feature (implemented in #2182)
- explain in the docs how to optimize speed/performance by setting default in-memory

cc: @stas00 https://github.com/huggingface/datasets/pull/2409#issuecomment-856210552


Add align_labels_with_mapping function

@lhoestq i think this is ready for another review 🙂

@lhoestq thanks for the feedback - it's now integrated :)

i also added a comment about sorting the input label IDs

Created the PR here: https://github.com/huggingface/datasets/pull/2510

> Thanks ! Looks all good now :)
>
> We will also need to have the `DatasetDict.align_labels_with_mapping` method. Let me quickly add it

thanks a lot! i always forget about `DatasetDict` - will be happy when it's just one "dataset" object :)

So, there seems to be a problem with the function align_labels_with_mapping for models like this: https://huggingface.co/huggingface/distilbert-base-uncased-finetuned-mnli]. At least with this model, but perhaps also with others, the model.config.label2id values are of type str not int, which crashes said function. After manually converting the model.config.label2id values to int, the script runs smoothly.

This PR adds a helper function to align the `label2id` mapping between a `datasets.Dataset` and a classifier (e.g. a transformer with a `PretrainedConfig.label2id` dict), with the alignment performed on the dataset itself.

This will help us with the Hub evaluation, where we won't know in advance whether a model that is fine-tuned on say MNLI has the same mappings as the MNLI dataset we load from `datasets`.

An example where this is needed is if we naively try to evaluate `microsoft/deberta-base-mnli` on `mnli` because the model config has the following mappings:

```python
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  },
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  }
```

while the `mnli` dataset has the `contradiction` and `neutral` labels swapped:

```python
id2label = {0: 'entailment', 1: 'neutral', 2: 'contradiction'}
label2id = {'contradiction': 2, 'entailment': 0, 'neutral': 1}
```

As a result, we get a much lower accuracy during evaluation:

```python
from datasets import load_dataset
from transformers.trainer_utils import EvalPrediction
from transformers import AutoModelForSequenceClassification, Trainer

# load dataset for evaluation
mnli = load_dataset("glue", "mnli", split="test")
# load model
model_ckpt = "microsoft/deberta-base-mnli"
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
# preprocess, create trainer ...
mnli_enc = ...
trainer = Trainer(model, args=args, tokenizer=tokenizer)
# generate preds
preds = trainer.predict(mnli_enc)
# preds.label_ids misalinged with model.config => returns wrong accuracy (too low)!
compute_metrics(EvalPrediction(preds.predictions, preds.label_ids))
```

The fix is to use the helper function before running the evaluation to make sure the label IDs are aligned:

```python
mnli_enc_aligned = mnli_enc.align_labels_with_mapping(label2id=config.label2id, label_column="label")
# preds now aligned and everyone is happy :)
preds = trainer.predict(mnli_enc_aligned)
```

cc @thomwolf @lhoestq


Fix cross-reference typos in documentation

Fix some minor typos in docs that avoid the creation of cross-reference links.


Update version in xor_tydi_qa.py

Hi ! Thanks for updating the version

> Should I revert to the old dummy/1.0.0 or delete it and keep only dummy/1.1.0?

Feel free to delete the old dummy data files

Fix #2449

@lhoestq Should I revert to the old `dummy/1.0.0` or  delete it and keep only `dummy/1.1.0`?


Rename config and environment variable for in memory max size

Thank you for the rename, @albertvillanova!

As discussed in #2409, both config and environment variable have been renamed.

cc: @stas00, huggingface/transformers#12056


Keep original features order

1.9

Next minor release

The arrow writer was supposing that the columns were always in the sorted order. I just pushed a fix to reorder the arrays accordingly to the schema. It was failing for many datasets like squad

Feel free to revert my commit. I can investigate this in the coming days

@lhoestq I do not understand when you say:
> It was failing for many datasets like squad

All the tests were green after my last commit.

> All the tests were green after my last commit.

Yes but loading the actual squad dataset was failing :/

When loading a Dataset from a JSON file whose column names are not sorted alphabetically, we should get the same column name order, whether we pass features (in the same order as in the file) or not.

I found this issue while working on #2366.


MRPC test set differences between torch and tensorflow datasets

bug

Something isn't working

Realized that `tensorflow_datasets` is not provided by Huggingface and should therefore raise the issue there.

## Describe the bug
When using `load_dataset("glue", "mrpc")` to load the MRPC dataset, the test set includes the labels. When using `tensorflow_datasets.load('glue/{}'.format('mrpc'))` to load the dataset the test set does not contain the labels. There should be consistency between torch and tensorflow ways of importing the GLUE datasets.

## Steps to reproduce the bug

Minimal working code
```python
from datasets import load_dataset
import tensorflow as tf
import tensorflow_datasets

# torch
dataset = load_dataset("glue", "mrpc")
# tf
data = tensorflow_datasets.load('glue/{}'.format('mrpc'))
data = list(data['test'].as_numpy_iterator())
for i in range(40,50):
  tf_sentence1 = data[i]['sentence1'].decode("utf-8")
  tf_sentence2 = data[i]['sentence2'].decode("utf-8")

  tf_label = data[i]['label']

  index = data[i]['idx']
  print('Index {}'.format(index))
  torch_sentence1 = dataset['test']['sentence1'][index]
  torch_sentence2 = dataset['test']['sentence2'][index]

  torch_label = dataset['test']['label'][index]
  print('Tensorflow: \n\tSentence1 {}\n\tSentence2 {}\n\tLabel {}'.format(tf_sentence1, tf_sentence2, tf_label))
  print('Torch: \n\tSentence1 {}\n\tSentence2 {}\n\tLabel {}'.format(torch_sentence1, torch_sentence2, torch_label))
```

Sample output
```
Index 954
Tensorflow:
	Sentence1 Sabri Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate yesterday on charges of violating U.S. arms-control laws .
	Sentence2 The elder Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate Wednesday on charges of violating U.S. arms control laws .
	Label -1
Torch:
	Sentence1 Sabri Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate yesterday on charges of violating U.S. arms-control laws .
	Sentence2 The elder Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate Wednesday on charges of violating U.S. arms control laws .
	Label 1
Index 711
Tensorflow:
	Sentence1 Others keep records sealed for as little as five years or as much as 30 .
	Sentence2 Some states make them available immediately ; others keep them sealed for as much as 30 years .
	Label -1
Torch:
	Sentence1 Others keep records sealed for as little as five years or as much as 30 .
	Sentence2 Some states make them available immediately ; others keep them sealed for as much as 30 years .
	Label 0
```

## Expected results
I would expect the datasets to be independent of whether I am working with torch or tensorflow.

## Actual results
Test set labels are provided in the `datasets.load_datasets()` for MRPC. However MRPC is the only task where the test set labels are not -1.

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0


Mention that there are no answers in adversarial_qa test set

As mention in issue https://github.com/huggingface/datasets/issues/2447, there are no answers in the test set