Update text classification template labels in DatasetInfo __post_init__

If I'm not mistaken, one way to fix this would be to drop the task templates when copying the info by inserting `dataset.info.task_templates = None` before the `Dataset.cast` call in `Dataset.prepare_for_task`. Moreover, we should do this change independently of the KeyError being raised because currently the following is possible:
```python
dset = load_dataset("some_dataset") # let's say 'some_dataset' supports text classification and question answering
dset_tc = dset.prepare_for_task("text-classification")
dset_tc.preprare_for_task("question-answering") # this should raise an error because the schema is no longer valid for this task; currently this fails on 'rename_columns'
```
I see 2 options:
1. to drop the task templates after the first `Dataset.prepare_for_task` call
2. to save only the tasks compatible with the new schema after Dataset.prepare_for_task` (but then we have to update the column names of the compatible tasks to make sure the column mapping is still valid)

> If I'm not mistaken, one way to fix this would be to drop the task templates when copying the info by inserting `dataset.info.task_templates = None` before the `Dataset.cast` call in `Dataset.prepare_for_task`. Moreover, we should do this change independently of the KeyError being raised because currently the following is possible:
>
> ```python
> dset = load_dataset("some_dataset") # let's say 'some_dataset' supports text classification and question answering
> dset_tc = dset.prepare_for_task("text-classification")
> dset_tc.preprare_for_task("question-answering") # this should raise an error because the schema is no longer valid for this task; currently this fails on 'rename_columns'
> ```
>
> I see 2 options:
>
> 1. to drop the task templates after the first `Dataset.prepare_for_task` call
> 2. to save only the tasks compatible with the new schema after Dataset.prepare_for_task` (but then we have to update the column names of the compatible tasks to make sure the column mapping is still valid)

thanks for the great idea @mariosasko and for spotting the problem with sequential task preparation! i am in favour of your option (1) since it is simple and saves us from having to keep track of the column mappings across multiple steps.

i've implemented the change and refactored the tests to account for the new approach (including a new test that the templates are flushed after we call `prepare_for_task`). perhaps the slightly inelegant aspect here is that if we want to allow the user to set `labels` in the `TextClassification` template, then we have two places (`DatasetInfo.__post_init__` and `TextClassification.__post_init__`) where we need to update `label_schema`.

on the other hand, dropping `labels` from the `TextClassification` signature would have the nice effect that users only have to think about column names when defining their tasks.

in any case, i think it would be a good idea to merge #2376 soon as the current PR is touching a lot of the same places in the codebase 😄

cc @SBrandeis who might also be interested in this feature :)

Tests are failing only because the `emotion` dataset card doesn't pass our dataset card validator (tags are missing), you can ignore this since it's unrelated to this PR.

@lhoestq @SBrandeis i've fixed the tests and think this is now in a good state for another review :)

Maybe @SBrandeis you can also take a look to make sure you're fine with it ?

This PR implements the idea discussed in #2389 to update the `labels` of the `TextClassification` template in the `DatasetInfo.__post_init__`. The main reason for doing so is so avoid duplicating the label definitions in both `DatasetInfo.features` and `DatasetInfo.task_templates`.

To avoid storing state in `DatasetInfo.__post_init__`, the current implementation flushes `DatasetInfo.task_templates` before the features are cast in `Dataset.prepare_for_task` (thanks to @mariosasko for this idea!).

Here is an example of the current workflow:

```python
ds1 = load_dataset("./datasets/emotion/")
# cast features and flush templates
ds2 = ds1.prepare_for_task("text-classification")
assert ds2.info.task_templates is None
```

Note that if users want to pass a `TextClassification` template to `prepare_for_task`, we require them to set `TextClassification.labels` to match the dataset's features corresponding to `label_column`:

```python
ds1 = load_dataset("./datasets/emotion/")
# TextClassification.labels is None by default => invalid template
task = TextClassification(text_column="text", label_column="label")
# Raises ValueError
ds1.prepare_for_task(task)
# Specifying the labels => valid template
task = TextClassification(text_column="text", label_column="label", labels=['anger', 'fear', 'joy', 'love', 'sadness', 'surprise'])
ds1.prepare_for_task(task)
```

This PR also adds:

* New tests + fixed some old tests that weren't testing `assertRaises` properly
* A decorator to share docstrings across common functions. This allows us to document `DatasetDict.prepare_for_task` and `Dataset.prepare_for_task` in one place.
* Fixes to avoid side-effects from in-place replacements of `DatasetInfo.task_templates` in `DatasetInfo.__post_init__`. Thanks to @lhoestq for figuring this out!
* Removal of `FeaturesWithLazyClassLabel` since we now create a new instance of `TextClassification` in `DatasetInfo.__post_init__` and avoid the side-effects first pointed out by @mariosasko

### PR Description from original WIP

Hi @yjernite and @lhoestq, here's a first stab at the suggestion discussed in #2389 to update the `labels` of the `TextClassification` template in the `DatasetInfo.__post_init__`.

One problem I've spotted is that my current implementation introduces state into the `__post_init__`:

* When we call `load_dataset`, `DatasetInfo.features` are the "raw" features without any casting so we can access the column names by the `label_column` specified in `TextClassification`
* When we call `Dataset.prepare_for_task` we run into a problem because the `DatasetInfo.features` are first cast into the new schema which triggers a `KeyError` when we update the infos [here](https://github.com/huggingface/datasets/blob/8b2a78520828e0cc13c14a31f413a5395ef25110/src/datasets/arrow_dataset.py#L1959).

Here's an explicit example of what I mean with the stack trace appended below:

```python
from datasets import load_dataset

# this works
ds = load_dataset("emotion")
# we can verify the task template is correctly set
ds["train"].info.task_templates # returns [TextClassification(labels=('sadness', 'joy', 'love', 'anger', 'fear', 'surprise'), text_column='text', label_column='label')]
# but this fails because the _post_init__ is looking for the original column names
ds.prepare_for_task("text-classification")
```
```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-4-54a43019b319> in <module>
----> 1 ds.prepare_for_task("text-classification")

~/git/datasets/src/datasets/dataset_dict.py in prepare_for_task(self, task)
    807         """
    808         self._check_values_type()
--> 809         return DatasetDict({k: dataset.prepare_for_task(task=task) for k, dataset in self.items()})

~/git/datasets/src/datasets/dataset_dict.py in <dictcomp>(.0)
    807         """
    808         self._check_values_type()
--> 809         return DatasetDict({k: dataset.prepare_for_task(task=task) for k, dataset in self.items()})

~/git/datasets/src/datasets/arrow_dataset.py in prepare_for_task(self, task)
   1421         dataset = self.remove_columns(columns_to_drop)
   1422         dataset = dataset.rename_columns(column_mapping)
-> 1423         dataset = dataset.cast(features=template.features)
   1424         return dataset
   1425

~/git/datasets/src/datasets/arrow_dataset.py in cast(self, features, batch_size, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, num_proc)
    970         format = self.format
    971         dataset = self.with_format("arrow")
--> 972         dataset = dataset.map(
    973             lambda t: t.cast(schema),
    974             batched=True,

~/git/datasets/src/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1583
   1584         if num_proc is None or num_proc == 1:
-> 1585             return self._map_single(
   1586                 function=function,
   1587                 with_indices=with_indices,

~/git/datasets/src/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    173         }
    174         # apply actual function
--> 175         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    176         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    177         # re-apply format to the output

~/git/datasets/src/datasets/fingerprint.py in wrapper(*args, **kwargs)
    338             # Call actual function
    339
--> 340             out = func(self, *args, **kwargs)
    341
    342             # Update fingerprint of in-place transforms + update in-place history of transforms

~/git/datasets/src/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset)
   1959         if update_data:
   1960             # Create new Dataset from buffer or file
-> 1961             info = self.info.copy()
   1962             info.features = writer._features
   1963             if buf_writer is None:

~/git/datasets/src/datasets/info.py in copy(self)
    274
    275     def copy(self) -> "DatasetInfo":
--> 276         return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
    277
    278

~/git/datasets/src/datasets/info.py in __init__(self, description, citation, homepage, license, features, post_processed, supervised_keys, task_templates, builder_name, config_name, version, splits, download_checksums, download_size, post_processing_size, dataset_size, size_in_bytes)

~/git/datasets/src/datasets/info.py in __post_init__(self)
    174                     # The reason is that Dataset.prepare_for_task calls Dataset.cast which converts the
    175                     # DatasetInfo.features to the new schema and thus template.label_column is no longer a valid key
--> 176                     object.__setattr__(template, "labels", tuple(self.features[template.label_column].names))
    177                     template.label_schema["labels"] = ClassLabel(names=template.labels)
    178                     self.task_templates[idx] = template

KeyError: 'label'
```

What do you think? I did this a bit quickly, so maybe I'm overlooking something obvious :) One thing would be to only update the labels of the task template on load, but this seems a bit hacky IMO


Missing original answers in kilt-TriviaQA

bug

Something isn't working

That could be useful indeed! Feel free to open a PR on the dataset card if you already have some code that runs, otherwise we'll take care of it soon :)

I can open a PR but there is 2 details to fix:
- the name for the corresponding key (e.g. `original_answer`)
- how to implement it: I’m not sure what happens when you map `lambda x: {'input': ...}` as it keeps the other keys (e.g. `output`) intact but here since we want to set a nested value (e.g. `x['output']['original_answer']`) I implemented it with a regular function (not lambda), see below

```py
def add_original_answer(x, trivia_qa, triviaqa_map):
    i = triviaqa_map[x['id']]
    x['output']['original_answer'] = trivia_qa['validation'][i]['answer']['value']
    return x
```

I previously opened an issue at https://github.com/facebookresearch/KILT/issues/42 but from the answer of @fabiopetroni it seems that the problem comes from HF-datasets

## Describe the bug
The `answer` field in kilt-TriviaQA, e.g. `kilt_tasks['train_triviaqa'][0]['output']['answer']` contains a list of alternative answer which are accepted for the question.
However it'd be nice to know the original answer to the question (the only fields in `output` are `'answer', 'meta', 'provenance'`)

## How to fix
It can be fixed by retrieving the original answer from the original TriviaQA (e.g. `trivia_qa['train'][0]['answer']['value']`), perhaps at the same place as here where one retrieves the questions https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md#loading-the-kilt-knowledge-source-and-task-data

cc @yjernite who previously answered to an issue about KILT and TriviaQA :)


Add check for task templates on dataset load

This PR adds a check that the features of a dataset match the schema of each compatible task template.


Insert task templates for text classification

Update: found a few datasets that slipped through the net. Adding them shortly!

> You might have thought about this already, but would it make sense to use the `datasets.features.ClassLabel` values when possible instead of declaring the list once for the `feature` and once for the `template`?

hi @yjernite, these code insertions are auto-generated so could certainly be improved :)

just so i understand, your idea is that instead of doing something like

```python
class AGNews(datasets.GeneratorBasedBuilder):
    """AG News topic classification dataset."""

    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            features=datasets.Features(
                {
                    "text": datasets.Value("string"),
                    "label": datasets.features.ClassLabel(
                        names=["World", "Sports", "Business", "Sci/Tech"]
                    ),
                }
            ),
            homepage="http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html",
            citation=_CITATION,
            task_templates=[
                TextClassification(
                    labels=("Business", "Sci/Tech", "Sports", "World"),
                    text_column="text",
                    label_column="label",
                )
            ],
        )
```

we could do the following:

```python
class AGNews(datasets.GeneratorBasedBuilder):
    """AG News topic classification dataset."""

    def _info(self):
        info = datasets.DatasetInfo(
            description=_DESCRIPTION,
            features=datasets.Features(
                {
                    "text": datasets.Value("string"),
                    "label": datasets.features.ClassLabel(
                        names=["World", "Sports", "Business", "Sci/Tech"]
                    ),
                }
            ),
            homepage="http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html",
            citation=_CITATION,
        )

        info.task_templates = [
            TextClassification(
                labels=info.features.names,
                text_column="text",
                label_column="label",
            )
        ]
        return info
```

> Or we could simply not specify the labels and update the template in the DatasetInfo postinit to give it the labels ?

Oh yes, that would be great! It does mean enforcing that people use the right feature type (sometimes people still use a `string` feature still because they don't want to enumerate the classes, but I guess you've been catching most of those in reviews @lhoestq )

There might be reasons where there should be a legitimate difference, but I can't really think of nay right now, and we can always duplicate the feature

Let's ignore the CI fails since they are unrelated to your changes. They're about dataset cards issues

This PR inserts text-classification templates for datasets with the following properties:

* Only one config
* At most two features of `(Value, ClassLabel)` type

Note that this misses datasets like `sentiment140` which only has `Value` type features - these will be handled in a separate PR


Incorrect URLs for some datasets

Something isn't working

## Describe the bug
It seems that the URLs for the following datasets are invalid:

- [ ]  `bn_hate_speech` has been renamed: https://github.com/rezacsedu/Bengali-Hate-Speech-Dataset/commit/c67ecfc4184911e12814f6b36901f9828df8a63a
- [ ] `covid_tweets_japanese` has been renamed: http://www.db.info.gifu-u.ac.jp/covid-19-twitter-dataset/

As a result we can no longer load these datasets using `load_dataset`. The simple fix is to rename the URL in the dataset script - will do this asap.

## Steps to reproduce the bug
```python
from datasets import load_dataset
# pick one of the datasets from the list above
ds = load_dataset("bn_hate_speech")
```

## Expected results
Dataset loads without error.

## Actual results
```
Downloading: 3.36kB [00:00, 1.07MB/s]
Downloading: 2.03kB [00:00, 678kB/s]
Using custom data configuration default
Downloading and preparing dataset bn_hate_speech/default (download: 951.48 KiB, generated: 949.84 KiB, post-processed: Unknown size, total: 1.86 MiB) to /Users/lewtun/.cache/huggingface/datasets/bn_hate_speech/default/0.0.0/a2dc726e511a2177523301bcad196af05d4d8a2cff30d2769ba8aacc1f5fdb5c...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lewtun/miniconda3/envs/hf-hub_eval/lib/python3.8/site-packages/datasets/load.py", line 744, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/lewtun/miniconda3/envs/hf-hub_eval/lib/python3.8/site-packages/datasets/builder.py", line 574, in download_and_prepare
    self._download_and_prepare(
  File "/Users/lewtun/miniconda3/envs/hf-hub_eval/lib/python3.8/site-packages/datasets/builder.py", line 630, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/lewtun/.cache/huggingface/modules/datasets_modules/datasets/bn_hate_speech/a2dc726e511a2177523301bcad196af05d4d8a2cff30d2769ba8aacc1f5fdb5c/bn_hate_speech.py", line 76, in _split_generators
    train_path = dl_manager.download_and_extract(_URL)
  File "/Users/lewtun/miniconda3/envs/hf-hub_eval/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 287, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/Users/lewtun/miniconda3/envs/hf-hub_eval/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 195, in download
    downloaded_path_or_paths = map_nested(
  File "/Users/lewtun/miniconda3/envs/hf-hub_eval/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 195, in map_nested
    return function(data_struct)
  File "/Users/lewtun/miniconda3/envs/hf-hub_eval/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 218, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/Users/lewtun/miniconda3/envs/hf-hub_eval/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 281, in cached_path
    output_path = get_from_cache(
  File "/Users/lewtun/miniconda3/envs/hf-hub_eval/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/rezacsedu/Bengali-Hate-Speech-Dataset/main/Bengali_%20Hate_Speech_Dataset_Subset.csv
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.2.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 3.0.0


datasets 1.6 ignores cache

Something isn't working

Looks like there are multiple issues regarding this (#2386, #2322) and it's a WIP #2329. Currently these datasets are being loaded in-memory which is causing this issue. Quoting @mariosasko here for a quick fix:

> set `keep_in_memory` to `False` when loading a dataset (`sst = load_dataset("sst", keep_in_memory=False)`) to prevent it from loading in-memory. Currently, in-memory datasets fail to find cached files due to this check (always False for them)

Hi ! Since `datasets` 1.6.0 we no longer keep small datasets (<250MB) on disk and load them in RAM instead by default. This makes data processing and iterating on data faster. However datasets in RAM currently have no way to reload previous results from the cache (since nothing is written on disk). We are working on making the caching work for datasets in RAM.

Until then, I'd recommend passing `keep_in_memory=False` to the calls to `load_dataset` like here:

https://github.com/huggingface/transformers/blob/223943872e8c9c3fc11db3c6e93da07f5177423f/examples/pytorch/language-modeling/run_clm.py#L233

This way you say explicitly that you want your dataset to stay on the disk, and it will be able to recover previously computed results from the cache.

OK, It doesn't look like we can use the proposed workaround - see https://github.com/huggingface/transformers/issues/11801

Could you please add an env var for us to be able to turn off this unwanted in our situation behavior? It is really problematic for dev work, when one needs to restart the training very often and needs a quick startup time. Manual editing of standard scripts is not a practical option when one uses examples.

This could also be a problem for tests, which will be slower because of lack of cache, albeit usually we use tiny datasets there. I think we want caching for tests.

Thank you.

Hi @stas00,

You are right: an env variable is needed to turn off this behavior. I am adding it.

For the moment there is a config parameter to turn off this behavior: `datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES = None`

You can find this info in the docs:
- in the docstring of the parameter `keep_in_memory` of the function [`load_datasets`](https://huggingface.co/docs/datasets/package_reference/loading_methods.html#datasets.load_dataset):
- in a Note in the docs about [Loading a Dataset](https://huggingface.co/docs/datasets/loading_datasets.html#from-the-huggingface-hub)

> The default in 🤗Datasets is to memory-map the dataset on drive if its size is larger than datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES (default 250 MiB); otherwise, the dataset is copied in-memory. This behavior can be disabled by setting datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES = None, and in this case the dataset is not loaded in memory.

Yes, but this still requires one to edit the standard example scripts, so if I'm doing that already I just as well can add `keep_in_memory=False`.

May be the low hanging fruit is to add `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES` env var to match the config, and if the user sets it to 0, then it'll be the same as `keep_in_memory=False` or `datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=0`?

@stas00, however, for the moment, setting the value to `0` is equivalent to the opposite, i.e. `keep_in_memory=True`. This means the max size until which I load in memory is 0 bytes.

Tell me if this is logical/convenient, or I should change it.

In my PR, to turn off current default bahavior, you should set env variable to one of: `{"", "OFF", "NO", "FALSE"}`.

For example:
```
MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=
```

IMHO, this behaviour is not very intuitive, as 0 is a normal quantity of bytes. So `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=0` to me reads as don't cache ever.

Also "SIZE_IN_BYTES" that can take one of `{"", "OFF", "NO", "FALSE"}` is also quite odd.

I think supporting a very simple `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES` that can accept any numerical value to match the name of the variable, requires minimal logic and is very straightforward.

So if you could adjust this logic - then `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=0` is all that's needed to not do in-memory datasets.

Does it make sense?

I understand your point @stas00, as I am not very convinced with current implementation.

My concern is: which numerical value should then pass a user who wants `keep_in_memory=True` by default, independently of dataset size? Currently it is `0` for this case.

That's a good question, and again the normal bytes can be used for that:
```
MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES=1e12 # (~2**40)
```
Since it's unlikely that anybody will have more than 1TB RAM.

It's also silly that it uses BYTES and not MBYTES - that level of refinement doesn't seem to be of a practical use in this context.

Not sure when it was added and if there are back-compat issues here, but perhaps it could be renamed `MAX_IN_MEMORY_DATASET_SIZE` and support 1M, 1G, 1T, etc.

But scientific notation is quite intuitive too, as each 000 zeros is the next M, G, T multiplier. Minus the discrepancy of 1024 vs 1000, which adds up. And it is easy to write down `1e12`, as compared to `1099511627776` (2**40).  (`1.1e12` is more exact).

Great! Thanks, @stas00.

I am implementing your suggestion to turn off default value when set to `0`.

For the other suggestion (allowing different metric prefixes), I will discuss with @lhoestq to agree on its implementation.

Moving from https://github.com/huggingface/transformers/issues/11801#issuecomment-845546612

Quoting @VictorSanh:

>
> I downgraded datasets to `1.5.0` and printed `tokenized_datasets.cache_files` (L335):
>
> > `{'train': [{'filename': '/home/victor/.cache/huggingface/datasets/openwebtext10k/plain_text/1.0.0/3a8df094c671b4cb63ed0b41f40fb3bd855e9ce2e3765e5df50abcdfb5ec144b/cache-c6aefe81ca4e5152.arrow'}], 'validation': [{'filename': '/home/victor/.cache/huggingface/datasets/openwebtext10k/plain_text/1.0.0/3a8df094c671b4cb63ed0b41f40fb3bd855e9ce2e3765e5df50abcdfb5ec144b/cache-97cf4c813e6469c6.arrow'}]}`
>
> while the same command with the latest version of datasets (actually starting at `1.6.0`) gives:
> > `{'train': [], 'validation': []}`
>

I also confirm that downgrading to `datasets==1.5.0` makes things fast again - i.e. cache is used.

to reproduce:
```
USE_TF=0 python  examples/pytorch/language-modeling/run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name "stas/openwebtext-10k" \
    --output_dir output_dir \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --max_train_samples 1000 \
    --max_eval_samples 200 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --num_train_epochs 1 \
    --warmup_steps 8 \
    --block_size 64 \
    --fp16 \
    --report_to none
```

the first time the startup is slow and some 5 tqdm bars. It shouldn't do it on consequent runs. but with `datasets>1.5.0` it rebuilds on every run.

@lhoestq


Accessing Arrow dataset cache_files

Something isn't working

Thanks @bhavitvyamalik for referencing the workaround. Setting `keep_in_memory=False` is working.

## Describe the bug
In datasets 1.5.0 the following code snippet would have printed the cache_files:

```
train_data = load_dataset('conll2003', split='train', cache_dir='data')
print(train_data.cache_files[0]['filename'])

```

However, in the newest release (1.6.1), it prints an empty list.

I also tried loading the dataset with `keep_in_memory=True` argument but still `cache_files` is empty.

Was wondering if this is a bug or I need to pass additional arguments so I can access the cache_files.


update citations

To update citations for [Offenseval_dravidiain](https://huggingface.co/datasets/offenseval_dravidian)


Add args description to DatasetInfo

Thanks for the suggestions! I've included them and made a few minor tweaks along the way

Please merge master into this branch to fix the CI, I just fixed metadata validation tests.

Closes #2354

I am not sure what `post_processed` and `post_processing_size` correspond to, so have left them empty for now. I also took a guess at some of the other fields like `dataset_size` vs `size_in_bytes`, so might have misunderstood their meaning.


Improve example in rounding docs

Improves the example in the rounding subsection of the Split API docs. With this change, it should more clear what's the difference between the `closest` and the `pct1_dropremainder` rounding.


DuplicatedKeysError: FAILURE TO GENERATE DATASET ! load_dataset('head_qa', 'en')

Hello everyone,

I try to use head_qa dataset in [https://huggingface.co/datasets/viewer/?dataset=head_qa&config=en](url)

```
!pip install datasets
from datasets import load_dataset
dataset = load_dataset(
   'head_qa', 'en')
```
When I write above load_dataset(.), it throws the following:

```
DuplicatedKeysError                       Traceback (most recent call last)

<ipython-input-6-ea87002d32f0> in <module>()
      2 from datasets import load_dataset
      3 dataset = load_dataset(
----> 4    'head_qa', 'en')

5 frames

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
    347         for hash, key in self.hkey_record:
    348             if hash in tmp_record:
--> 349                 raise DuplicatedKeysError(key)
    350             else:
    351                 tmp_record.add(hash)

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 1
Keys should be unique and deterministic in nature
```
How can I fix the error? Thanks


add dataset card title

few of them were missed by me earlier which I've added now


maintain YAML structure reading from README

How YAML used be loaded earlier in the string (structure of YAML was affected because of this and YAML for datasets with multiple configs was not being loaded correctly):
```
annotations_creators:
labeled_final:
- expert-generated
labeled_swap:
- expert-generated
unlabeled_final:
- machine-generated
language_creators:
- machine-generated
languages:
- en
licenses:
- other
multilinguality:
- monolingual
size_categories:
labeled_final:
- 10K<n<100K
labeled_swap:
- 10K<n<100K
unlabeled_final:
- 100K<n<1M
source_datasets:
- original
task_categories:
- text-classification
- text-scoring
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring-other-paraphrase-identification
```
How YAML is loaded in string now:
```
annotations_creators:
  labeled_final:
  - expert-generated
  labeled_swap:
  - expert-generated
  unlabeled_final:
  - machine-generated
language_creators:
- machine-generated
languages:
- en
licenses:
- other
multilinguality:
- monolingual
size_categories:
  labeled_final:
  - 10K<n<100K
  labeled_swap:
  - 10K<n<100K
  unlabeled_final:
  - 100K<n<1M
source_datasets:
- original
task_categories:
- text-classification
- text-scoring
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring-other-paraphrase-identification
```


Disallow duplicate keys in yaml tags

Make sure that there's no duplidate keys in yaml tags.
I added the check in the yaml tree constructor's method, so that the verification is done at every level in the yaml structure.

cc @julien-c


Add missing dataset_infos.json files

enhancement

New feature or request

Some of the datasets in `datasets` are missing a `dataset_infos.json` file, e.g.

```
[PosixPath('datasets/chr_en/chr_en.py'), PosixPath('datasets/chr_en/README.md')]
[PosixPath('datasets/telugu_books/README.md'), PosixPath('datasets/telugu_books/telugu_books.py')]
[PosixPath('datasets/reclor/README.md'), PosixPath('datasets/reclor/reclor.py')]
[PosixPath('datasets/json/README.md')]
[PosixPath('datasets/csv/README.md')]
[PosixPath('datasets/wikihow/wikihow.py'), PosixPath('datasets/wikihow/README.md')]
[PosixPath('datasets/c4/c4.py'), PosixPath('datasets/c4/README.md')]
[PosixPath('datasets/text/README.md')]
[PosixPath('datasets/lm1b/README.md'), PosixPath('datasets/lm1b/lm1b.py')]
[PosixPath('datasets/pandas/README.md')]
```

For `json`, `text`, csv`, and `pandas` this is expected, but not for the others which should be fixed


ArrowDataset.save_to_disk produces files that cannot be read using pyarrow.feather

bug

Something isn't working

Hi ! This is because we are actually using the arrow streaming format. We plan to switch to the arrow IPC format.
More info at #1933

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from pyarrow import feather

dataset = load_dataset('imdb', split='train')
dataset.save_to_disk('dataset_dir')
table = feather.read_table('dataset_dir/dataset.arrow')
```

## Expected results
I expect that the saved dataset can be read by the official Apache Arrow methods.

## Actual results
```
  File "/usr/local/lib/python3.7/site-packages/pyarrow/feather.py", line 236, in read_table
    reader.open(source, use_memory_map=memory_map)
  File "pyarrow/feather.pxi", line 67, in pyarrow.lib.FeatherReader.open
  File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not a Feather V1 or Arrow IPC file
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-1.6.2
- Platform: Linux
- Python version: 3.7
- PyArrow version: 0.17.1, also 2.0.0


Improve task api code quality

Looks good thanks, what do you think @lewtun ?

thanks for including the lazy `ClassLabel` class @mariosasko ! from my side this LGTM!

Improves the code quality of the `TaskTemplate` dataclasses.

Changes:
* replaces `return NotImplemented` with raise `NotImplementedError`
* replaces `sorted` with `len` in the uniqueness check
* defines `label2id` and `id2label` in the `TextClassification` template as properties
* replaces the `object.__setattr__(self, attr, value)` syntax with (IMO nicer) `self.__dict__[attr] = value`


Dataset Streaming

# Dataset Streaming

## API

Current API is

```python
from datasets import load_dataset

# Load an IterableDataset without downloading data
snli = load_dataset("snli", streaming=True)

# Access examples by streaming data
print(next(iter(snli["train"])))
# {'premise': 'A person on a horse jumps over a broken down airplane.',
#  'hypothesis': 'A person is training his horse for a competition.',
#  'label': 1}
```

I already implemented a few methods:
- IterableDataset.map: apply transforms on-the-fly to the examples
- IterableDataset.shuffle: shuffle the data _a la_ TFDS, i.e. with a shuffling buffer
- IterableDataset.with_format: set the format to `"torch"` to get a `torch.utils.data.IterableDataset`
- merge_datasets: merge two iterable datasets by alternating one or the other (you can specify the probabilities)

I would love to have your opinion on the API design :)

## Implementation details

### Streaming

Data streaming is done using `fsspec` which has nice caching features.

To make dataset streaming work I extend the `open` function of dataset scripts to support opening remote files without downloading them entirely. It also works with remote compressed archives (currently only zip is supported):

```python
# Get a file-like object by streaming data from a remote file
open("https://github.com/davidsbatista/NER-datasets/raw/master/CONLL2003/train.txt")

# Get a file-like object by streaming data from a remote compressed archive by using the hop separator "::"
open("zip://snli_1.0_train.txt::https://nlp.stanford.edu/projects/snli/snli_1.0.zip")
```

I also extend the `os.path.join` function to support navigation in remote compressed archives, since it has to deal with the `"::"` separator. This separator is used by `fsspec`.

Finally I also added a retry mechanism in case the connection fails during data streaming.

### Transforms

An IterableDataset wraps an ExamplesIterable instance. There are different subclasses depending on the transforms we want to apply:
- ExamplesIterable: the basic one
- MappedExamplesIterable: an iterable with a `map` function applied on the fly
- BufferShuffledExamplesIterable: an iterable with a shuffling buffer
- CyclingMultiSourcesExamplesIterable: alternates between several ExamplesIterable
- RandomlyCyclingMultiSourcesExamplesIterable: randomly alternates between several ExamplesIterable

### DatasetBuilder

I use the same builders as usual. I just added a new method `_get_examples_iterable_for_split` to get an ExamplesIterable for a given split. Currently only the GeneratorBasedBuilder and the ArrowBasedBuilder implement it.

The BeamBasedBuilder doesn't implement it yet.
It means that datasets like wikipedia and natural_questions can't be loaded as IterableDataset for now.

## Other details

<S>I may have to do some changes in many dataset script to use `download` instead of `download_and_extract` when extraction is not needed. This will avoid errors for streaming.</s>

EDIT: Actually I just check for the extension of the file to do extraction only if needed.

EDIT2: It's not possible to stream from .tar.gz files without downloading the file completely. For now I raise an error if one want to get a streaming dataset based on .tar.gz files.

## TODO

usual stuff:

- [x] make streaming dependency "aiohttp" optional: `pip install datasets[streaming]`
- [x] tests
- [x] docs


add `desc` to `tqdm` in `Dataset.map()`

Once this is merged, let's update `transformers` examples to use this new code. As currently all those tqdm bars are who knows what they are....

https://github.com/huggingface/transformers/issues/11797

Sure @stas00! Once this is merged let's discuss what all changes can be done on `transformers` side

@bhavitvyamalik, as it has been merged would you like to tackle https://github.com/huggingface/transformers/issues/11797?

Definitely @stas00. From what I could gather, you guys want more meaningful `.map` calls for all examples [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch)?

That's exactly right, @bhavitvyamalik

Perhaps the best approach is to do one example, see that other maintainers agree on it. and then replicate to other.

Fixes #2330. Please let me know if anything is also required in this


Loading dataset from local path

Version below works, checked again in the docs, and data_files should be a path.
```
ds = datasets.load_dataset('my_script.py',
                           data_files='/data/dir/corpus.txt',
                           cache_dir='.')
```

I'm trying to load a local dataset with the code below

```
ds = datasets.load_dataset('my_script.py',
                           data_files='corpus.txt',
                           data_dir='/data/dir',
                           cache_dir='.')
```
But internally a BuilderConfig is created, which tries to use getmtime on the data_files string, without using data_dir. Is this a bug or am I not using the load_dataset correctly?

https://github.com/huggingface/datasets/blob/bc61954083f74e6460688202e9f77dde2475319c/src/datasets/builder.py#L153


ConvQuestions benchmark added

Thanks for your helpful comments and suggestions! :)
I integrated the additional fields, and extended some of the README/dataset card.
And I actually realized that we had the cc-by-4.0 for the dataset, so this was also changed.

I added the answers to the test set actually :)

Hello,
I would like to integrate our dataset on conversational QA. The answers are grounded in the KG.
The work was published in CIKM 2019 (https://dl.acm.org/doi/10.1145/3357384.3358016).
We hope for further research on how to deal with the challenges of factoid conversational QA.
Thanks! :)


Align question answering tasks with sub-domains

enhancement

New feature or request

As pointed out by @thomwolf in #2255 we should consider breaking with the pipeline taxonomy of `transformers` to account for the various types of question-answering domains:

> `question-answering` exists in two forms: abstractive and extractive question answering.
>
> we can keep a generic `question-answering` but then it will probably mean diferrent schema of input/output for both (abstractive will have text for both while extractive can use spans indication as well as text).
>
> Or we can also propose to use `abstractive-question-answering` and `extractive-question-answering` for instance.
> Maybe we could have `question-answering-abstractive` and `question-answering-extractive` if somehow we can use a for a completion or search in the future (detail).
> Actually I see that people are more organizing in terms of general and sub-tasks, for instance on paperwithcode: https://paperswithcode.com/area/natural-language-processing and on nlpprogress: https://github.com/sebastianruder/NLP-progress/blob/master/english/question_answering.md#squad
>
> Probably the best is to align with one of these in terms of denomination, PaperWithCode is probably the most active and maintained and we work with them as well.
> Maybe you want to check with a few QA datasets that this schema make sense. Typically NaturalQuestions, TriviaQA and can be good second datasets to compare to and be sure of the generality of the schema.
>
> A good recent list of QA datasets to compare the schemas among, is for instance in the UnitedQA paper: https://arxiv.org/abs/2101.00178

Investigate which grouping of QA is best suited for `datasets` and adapt / extend the QA task template accordingly.


Adding HendrycksTest dataset

@lhoestq Thank you for the review. I've made the suggested changes. There still might be some problems with dummy data though due to some csv loading issues (which I haven't found the cause to).

I took a look at the dummy data and some csv lines were cropped. I fixed them :)

Adding Hendrycks test from https://arxiv.org/abs/2009.03300.
I'm having a bit of trouble with dummy data creation because some lines in the csv files aren't being loaded properly (only the first entry loaded in a row of length 6). The dataset is loading just fine. Hope you can kindly help!
Thank you!


correct labels of conll2003

# What does this PR

It fixes/extends the `ner_tags` for conll2003 to include all.
Paper reference https://arxiv.org/pdf/cs/0306050v1.pdf
Model reference https://huggingface.co/elastic/distilbert-base-cased-finetuned-conll03-english/blob/main/config.json


Allow "other-X" in licenses

This PR allows "other-X" licenses during metadata validation.

@lhoestq


Remove getchildren from hyperpartisan news detection

`Element.getchildren()` is now deprecated in the ElementTree library (I think in python 3.9, so it still passes the automated tests which are using 3.6. But for those of us on bleeding-edge distros it now fails).

https://bugs.python.org/issue29209


Json loader fails if user-specified features don't match the json data fields order

bug

Something isn't working

If you do
```python
dataset = load_dataset("json", data_files=data_files, features=features)
```
Then depending on the order of the features in the json data field it fails:
```python
[...]
~/Desktop/hf/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files)
     94             if self.config.schema:
     95                 # Cast allows str <-> int/float, while parse_option explicit_schema does NOT
---> 96                 pa_table = pa_table.cast(self.config.schema)
     97             yield i, pa_table
[...]
ValueError: Target schema's field names are not matching the table's field names: ['tokens', 'ner_tags'], ['ner_tags', 'tokens']
```

This is because one must first re-order the columns of the table to match the `self.config.schema` before calling cast.

One way to fix the `cast` would be to replace it with:
```python
# reorder the arrays if necessary + cast to schema
# we can't simply use .cast here because we may need to change the order of the columns
pa_table = pa.Table.from_arrays([pa_table[name] for name in schema.names], schema=schema)
```


Missing ClassLabel encoding in Json loader

Something isn't working

1.9

Next minor release

Currently if you want to load a json dataset this way
```python
dataset = load_dataset("json", data_files=data_files, features=features)
```
Then if your features has ClassLabel types and if your json data needs class label encoding (i.e. if the labels in the json files are strings and not integers), then it would fail:
```python
[...]
~/Desktop/hf/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files)
     94             if self.config.schema:
     95                 # Cast allows str <-> int/float, while parse_option explicit_schema does NOT
---> 96                 pa_table = pa_table.cast(self.config.schema)
     97             yield i, pa_table
[...]
ArrowInvalid: Failed to parse string: 'O' as a scalar of type int64
```

This is because it just tries to cast the string data to integers, without applying the mapping str->int first

The current workaround is to do instead
```python
dataset = load_dataset("json", data_files=data_files)
dataset = dataset.map(features.encode_example, features=features)
```


README updated for SNLI, MNLI

Regarding the license issue, I think we should allow it since it starts with `other-`. Cc @gchhablani what do you think ?

Closes #2275. Mentioned about -1 labels in MNLI, SNLI and how they should be removed before training. @lhoestq `check_code_quality` test might fail for MNLI as the license name `other-Open Portion of the American National Corpus`  is not a registered tag for 'licenses'


Fix web_nlg metadata

Hi ! `release_v2.1` and the others are dataset configuration names.

The configuration names are used to show the right code snippet in the UI to load the dataset.
For example if the parsing of the web_nlg tags worked correctly we would have:
![image](https://user-images.githubusercontent.com/42851186/118475444-8d1e5d00-b70c-11eb-98e9-844d4daf6139.png)

Therefore I don't think it's a good idea to rename the configurations from `release_v2.1` to `release_v2_1` as the code snippet would be wrong in this case.

Moreover we can't really disallow dots in configuration names and rename the configurations since it would be a big breaking change. It's commonly used, especially with multilingual datasets. For example `load_dataset("indic_glue", "sna.bn")`.

Is this something that can be fixed on the moonlanding side instead ?

> Is this something that can be fixed on the moonlanding side instead ?

Not really unless we change database:)

We'll maybe try to find another workaround, but super low-prio given that it's the only dataset that has those dotted keys in the YAML metadata

Our metadata storage system does not support `.` inside keys. cc @Pierrci


Preserve dtype for numpy/torch/tf/jax arrays

Hi @lhoestq,
It turns out that pyarrow `ListArray` are not recognized as list-like when we get output from `numpy_to_pyarrow_listarray`. This might cause tests to fail. If possible can we convert that `ListArray` output to list inorder for tests to pass? Under the hood it'll maintain the dtype as that of numpy array passed during input only

Brought down the failing tests from 7 to 4. Let me know if that part looks good. Failing tests are looking quite similar. In `test_map_torch` https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1039 and `test_map_tf`https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1056
they're expecting `float64`. Shouldn't that be `float32` now?

It's normal: pytorch and tensorflow use `float32` by default, unlike numpy which uses `float64`.

I think that we should always keep the precision of the original tensor (torch/tf/numpy).
It means that as it is in this PR it's fine (the precision is conserved when doing the torch/tf -> numpy conversion).

This is a breaking change but in my opinion the fact that we had Value("float64") for torch.float32 tensors was an issue already.

Let me know what you think. Cc @albertvillanova if you have an opinion on this

If we agree on doing this breaking change, we can just change the test.

Hi @lhoestq,
Merged master into this branch. Only changing the test is left for now (mentioned below) after which all tests should pass.

> Brought down the failing tests from 7 to 4. Let me know if that part looks good. Failing tests are looking quite similar. In `test_map_torch`
>
> https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1039
>
> and `test_map_tf`
> https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1056
>
>
> they're expecting `float64`. Shouldn't that be `float32` now?

> they're expecting float64. Shouldn't that be float32 now?

Yes feel free to update those tests :)

It would be nice to have the same test for JAX as well

Added same test for for JAX too. Also, I saw that I missed changing `test_cast_to_python_objects_jax` like I did for TF and PyTorch. Finished that as well

Fixes #625. This lets the user preserve the dtype of numpy array to pyarrow array which was getting lost due to conversion of numpy array -> list -> pyarrow array.


Automatically detect datasets with compatible task schemas

enhancement

New feature or request

See description of #2255 for details.


Allow model labels to be passed during task preparation

Models have a config with label2id. And we have the same for datasets with the ClassLabel feature type. At one point either the model or the dataset must sync with the other. It would be great to do that on the dataset side.

For example for sentiment classification on amazon reviews with you could have these labels:
- "1 star", "2 stars", "3 stars", "4 stars", "5 stars"
- "1", "2", "3", "4", "5"

Some models may use the first set, while other models use the second set.

Here in the `TextClassification` class, the user can only specify one set of labels, while many models could actually be compatible but have different sets of labels. Should we allow users to pass a list of compatible labels sets ?

Then in terms of API, users could use `dataset.prepare_for_task("text-classification", labels=model.labels)` or something like that.

The label set could also be the same but not in the same order. For NLI for example, some models use `["neutral", "entailment", "contradiction"]` and some others use `["neutral", "contradiction", "entailment"]`, so we should take care of updating the order of the labels in the dataset to match the labels order of the model.

Let me know what you think ! This can be done in a future PR

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/pull/2255#discussion_r632412792_


Roman Urdu Stopwords List

Hi ! Thanks for sharing :)
I think the best place to share this is probably the `Languages at Hugging Face` section of the forum:
https://discuss.huggingface.co/c/languages-at-hugging-face/15

Since this is not a dataset, I'm closing this PR if you don't mind

Thank you I will look into the link that you have shared with me.

On Mon, May 17, 2021 at 7:05 PM Quentin Lhoest ***@***.***>
wrote:

> Closed #2358 <https://github.com/huggingface/datasets/pull/2358>.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/2358#event-4754836267>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AN7SJYJVY4C5XQRDNET743DTOEPC7ANCNFSM443AZ3MA>
> .
>

A list of most frequently used Roman Urdu words with different spellings and usages.
This is a very basic effort to collect some basic stopwords for Roman Urdu to help efforts of analyzing text data in roman Urdu which makes up a huge part of daily internet interaction of Roman-Urdu users.


Adding Microsoft CodeXGlue Datasets

Oh one other thing. Mentioned in the PR was that I would need to regenerate the dataset_infos.json once the camel casing was done. However, I am unsure why this is the case since there is no reference to any object names in the dataset_infos.json file.

If it needs to be reran, I can try it do it on my own machine, but I've had a memory issues with a previous dataset due to my compute constraints so I'd prefer to hopefully avoid it all together if not necessary to regenerate.

Was just reviewing the `builder_name`s of each dataset and it seems like it is already following this format:

`CodeXGlueCcCloneDetectionBigCloneBenchMain -> code_x_glue_cc_clone_detection_big_clone_bench_main` Is there a location I am missing?

> Was just reviewing the `builder_name`s of each dataset and it seems like it is already following this format:
>
> `CodeXGlueCcCloneDetectionBigCloneBenchMain -> code_x_glue_cc_clone_detection_big_clone_bench_main` Is there a location I am missing?

If it's already in this format then it's fine thanks ! It's all good then

To fix the CI you just need to add the `encoding=` parameters to the `open()` calls

@lhoestq I think everything should be good to go besides the code styling, which seem to be due to missing or unsupported metadata tags for the READMEs, is this something I should worry about since all the other datasets seem to be failing as well?

Awesome! Just committed your changes and I will begin on adding the TOCs and filling in the content for the new sections/subsections.

Also, I see that we are having to only use the `code` tag instead of individual langs and I get that is required for indexing or showing available tags on the datasets hub. However, as a future feature, it might be good to add tags for individual programming languages to make it easier to search.

> Also, I see that we are having to only use the code tag instead of individual langs and I get that is required for indexing or showing available tags on the datasets hub. However, as a future feature, it might be good to add tags for individual programming languages to make it easier to search.

Yes I agree. We'll be able to reuse the tags per programming language from this PR when we allow this feature

cc @yjernite what do you think about extending our languages taxonomy to programming languages ?

Hey @lhoestq, just finalizing the READMEs and testing them against the automated test. For the non, WIN tests, it seems like there is some dependency issue that doesn't have to do with the new datasets. For the WIN tests, it looks like some of the headings are mislabeled such as "Supported Tasks and Leaderboards" -> "Supported Tasks" in the TOC you posted. Should I base my TOC on the one you posted or on the one that the test script is using? Also, it throws errors for some of the fields being empty, such as "Source Data" in the `code_x_glue_tt_text_to_text` dataset. However, I am not familiar with this dataset, so I put the `[More Information Needed]` stub, similar to the other sections I couldn't easily answer. For some of the sections like "Source Data", is this info required?

Yes you're right, it is `Supported Tasks and Leaderboards` that we need to use, sorry about that

I also noticed the same for the splits section: we have to use `Data Splits` (not Data Splits Sample Size)

Some subsections are also missing: `Initial Data Collection and Normalization`, `Who are the source language producers?`.
If you are interested you can fill those sections as well, or leave them empty for now.
This will also fix the error regarding "Source Data"

You can see the template of the readme here:
https://github.com/huggingface/datasets/blob/9d8bf36fdb861d9b2922d7c782fb58f9f542997c/templates/README.md

> > Also, I see that we are having to only use the code tag instead of individual langs and I get that is required for indexing or showing available tags on the datasets hub. However, as a future feature, it might be good to add tags for individual programming languages to make it easier to search.
>
> Yes I agree. We'll be able to reuse the tags per programming language from this PR when we allow this feature
>
> cc @yjernite what do you think about extending our languages taxonomy to programming languages ?

Sounds good, as long as they all share a prefix! maybe `code_cpp`, `code_java`, etc. ?

I don't think we currently have `_` in language codes/names, but also don't see what it would break *a priori*

We don't use `_` but there are some languages that use `-` though like `en-US`. Let's use `-` maybe, to match the same hierarchy pattern ?

Hi guys, I just started working on https://github.com/huggingface/datasets/pull/997 this morning and I just realized that you were finishing it... You may want to get the dataset cards from https://github.com/madlag/datasets, and maybe some code too, as I did a few things like moving _CITATION and _DESCRIPTION to globals.

I am renaming the main classes to match the dataset names, for example : CodeXGlueTcTextToCodeMain -> CodeXGlueTcTextToCode . And I am regenerating the dataset_infos.json accordingly.

Thanks for renaming the classes and updating the dataset_infos.json ! This looks all clean now :)

This PR looks all good to me :) One just needs to merge master into this branch to make sure the CI is green with the latest changes. It should also fix the current CI issues that are not related to this PR

Woot woot :rocket:! All green, looks like it is ready for showtime. Thank you both @lhoestq and especially @madlag, I think these datasets are going to be a great new addition to :hugs: datasets and I can't wait to use them in my research :nerd_face:.

Thanks @ncoop57 for you contribution! It will be really cool to see those datasets used as soon as they are released !

Hi there, this is a new pull request to get the CodeXGlue datasets into the awesome HF datasets lib. Most of the work has been done in this PR #997 by the awesome @madlag. However, that PR has been stale for a while now and so I spoke with @lhoestq about finishing up the final mile and so he told me to open a new PR with the final changes :smile:.

I believe I've met all of the changes still left in the old PR to do, except for the change to the languages. I believe the READMEs should include the different programming languages used rather than just using the tag "code" as when searching for datasets, SE researchers may specifically be looking only for what type of programming language and so being able to quickly filter will be very valuable. Let me know what you think of that or if you still believe it should be the "code" tag @lhoestq.


normalized TOCs and titles in data cards

Oh right! I'd be in favor of still having the same TOC across the board, we can either leave it as is or add a `[More Info Needed]` `Contributions` Section wherever it's currently missing, wdyt?

(I thought those were programmatically updated based on git history :D )

Merging for now to avoid conflict since there are so many changes but let's figure out the contributions section next ;)

I started fixing some of the READMEs that were failing the tests introduced by @gchhablani but then realized that there were some consistent differences between earlier and newer versions of some of the titles (e.g. Data Splits vs Data Splits Sample Size, Supported Tasks vs Supported Tasks and Leaderboards). We also had different versions of the Table of Content

This PR normalizes all of them to the newer version


Document DatasetInfo attributes

New feature or request

**Is your feature request related to a problem? Please describe.**
As noted in PR #2255, the attributes of `DatasetInfo` are not documented in the [docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=datasetinfo#datasetinfo). It would be nice to do so :)


Update README vallidation rules

This PR allows unexpected subsections under third-level headings. All except `Contributions`.

@lhoestq


Set to_json default to JSON lines

This is perfect, @albertvillanova - thank you! Tested it to work.

Might it be a good idea to document the args to `to_json`?

and also even a very basic progress bar? took 10min for 8M large records for `openwebtext` so perhaps some indication of it's being alive every min or so?

@lhoestq I added tests for both `lines` and `orient`.

With this PR, the method `Dataset.to_json`:
- is added to the docs
- defaults to JSON lines


simpllify faiss index save

Fixes #2350

In some cases, Faiss GPU index objects do not have neither "device" nor "getDevice". Possibly this happens when some part of the index is computed on CPU.

In particular, this would happen with the index `OPQ16_128,IVF512,PQ32` (issue #2350). I did check it, but it is likely that `OPQ` or `PQ` transforms cause it.

I propose, instead of using the index object to get the device, to infer it form the `FaissIndex.device` field as it is done in `.add_vectors`. Here we assume that `.device` always corresponds to the index placement and it seems reasonable.


`FaissIndex.save` throws error on GPU

bug

Something isn't working

Just in case, this is a workaround that I use in my code and it seems to do the job.

```python
if use_gpu_index:
    data["train"]._indexes["text_emb"].faiss_index = faiss.index_gpu_to_cpu(data["train"]._indexes["text_emb"].faiss_index)
```

## Describe the bug

After training an index with a factory string `OPQ16_128,IVF512,PQ32` on GPU, `.save_faiss_index` throws this error.

```
  File "index_wikipedia.py", line 119, in <module>
    data["train"].save_faiss_index("text_emb", index_save_path)
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/datasets/search.py", line 470, in save_faiss_index
    index.save(file)
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/datasets/search.py", line 334, in save
    faiss.write_index(index, str(file))
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/faiss/swigfaiss_avx2.py", line 5654, in write_index
    return _swigfaiss.write_index(*args)
RuntimeError: Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /root/miniconda3/conda-bld/faiss-pkg_1613235005464/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index
```

## Steps to reproduce the bug

Any dataset will do, I just selected a familiar one.

```python
import numpy as np
import datasets
INDEX_STR = "OPQ16_128,IVF512,PQ32"
INDEX_SAVE_PATH = "will_not_save.faiss"

data = datasets.load_dataset("Fraser/news-category-dataset", split=f"train[:10000]")

def encode(item):
    return {"text_emb": np.random.randn(768).astype(np.float32)}

data = data.map(encode)

data.add_faiss_index(column="text_emb", string_factory=INDEX_STR, train_size=10_000, device=0)
data.save_faiss_index("text_emb", INDEX_SAVE_PATH)
```

## Expected results
Saving the index

## Actual results
Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) ... don't know how to serialize this type of index

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

I will be proposing a fix in a couple of minutes


Update task_ids for Ascent KB

This "other-other-knowledge-base" task is better suited for the dataset.


Add tests for dataset cards

@lhoestq

Should I remove the scripts? or atleast remove running them from the CircleCI config?

Also, I hope it is okay that the combined method (metadata+content) is only a slow test, and for the Circle CI, I assume only non-slow tests are run? If yes, this would mean separate tests for content and metadata.

Also feel free to remove the scripts from the CI and also remove the scripts files :)

Adding tests for dataset cards

This PR will potentially remove the scripts being used for dataset tags and readme validation.

Additionally, this will allow testing dataset readmes by providing the name as follows:

```bash
pytest tests/test_dataset_cards.py::test_dataset_tags[fashion_mnist]
```
and

```bash
pytest tests/test_dataset_cards.py::test_readme_content[fashion_mnist]
```
or a combined test as:

```bash
pytest tests/test_dataset_cards.py::test_dataset_card[fashion_mnist]
```
@lhoestq


Add an API to access the language and pretty name of a dataset

enhancement

New feature or request

Hi ! With @bhavitvyamalik we discussed about having something like
```python
from datasets import load_dataset_card

dataset_card = load_dataset_card("squad")
print(dataset_card.metadata.pretty_name)
# Stanford Question Answering Dataset (SQuAD)
print(dataset_card.metadata.languages)
# ["en"]

```
What do you think ?

I don't know if you already have a way to load the model tags in `transformers` but we can agree on the API to have something consistent.

Also note that the pretty name would only be used to show users something prettier than a dataset id, but in the end the source of truth will stay the dataset id (here `squad`).

maybe use the hub-backed dataset_info method? (so there's only one parser of README.md metadata)?

What dataset_info method are you talking about @julien-c ? In `huggingface_hub` I can only see `model_info`.

hmm the equivalent method in `datasets` (which could go into `huggingface_hub` at some point)

It would be super nice to have an API to get some metadata of the dataset from the name and args passed to `load_dataset`. This way we could programmatically infer the language and the name of a dataset when creating model cards automatically in the Transformers examples scripts.


Add Qasper Dataset

I saw that the README [template](https://github.com/huggingface/datasets/blob/master/templates/README.md) changed while I was working on this 😅 Some TOC titles may be different but I filled it to the best of my knowledge & readme quality check passes now.
ready for review @lhoestq

[Question Answering on Scientific Research Papers](https://allenai.org/project/qasper/home)

Doing NLP on NLP papers to do NLP ♻ I had to add it~

- [x] Add README (just gotta fill out some more )
- [x] Dataloader code
- [x] Make dummy dataset
- [x] generate dataset infos
- [x] Tests


[Question] How to move and reuse preprocessed dataset?

<s>Hi :) Can you share with us the code you used ?</s>

EDIT: from https://github.com/huggingface/transformers/issues/11665#issuecomment-838348291 I understand you're using the run_clm.py script. Can you share your logs ?

> Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same

I only changed the `preprocessing_num_workers` maybe it is the problem~ I will try again~

Hi, I am training a gpt-2 from scratch using run_clm.py.

I want to move and reuse the preprocessed dataset (It take 2 hour to preprocess),

I tried to :

copy path_to_cache_dir/datasets to new_cache_dir/datasets
set export HF_DATASETS_CACHE="new_cache_dir/"
but the program still re-preprocess the whole dataset without loading cache.

I also tried to torch.save(lm_datasets, fw), but the saved file is only 14M.

What is the proper way to do this?


Is there a way to join multiple datasets in one?

New feature or request

Hi ! We don't have `join`/`merge` on a certain column as in pandas.
Maybe you can just use the [concatenate_datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets) function.

**Is your feature request related to a problem? Please describe.**
I need to join 2 datasets, one that is in the hub and another I've created from my files. Is there an easy way to join these 2?

**Describe the solution you'd like**
Id like to join them with a merge or join method, just like pandas dataframes.

**Additional context**
If you want to extend an existing dataset with more data, for example for training a language model, you need that functionality. I've not found it in the documentation.


Columns are removed before or after map function applied?

bug

Something isn't working

Hi! Columns are removed **after** applying the function and **before** updating the examples with the function's output (as per the docs [here](https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.map.remove_columns)). I agree the docs on this should be more clear.

## Describe the bug
According to the documentation when applying map function the [remove_columns ](https://huggingface.co/docs/datasets/processing.html#removing-columns) will be removed after they are passed to the function, but in the [source code](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map) it's documented that they are removed before applying function. I thinks the source code doc is more accurate, right?


Docs - CER above 1

CER can actually be greater than 1.


Added the Ascent KB

Added the Ascent Commonsense KB of 8.9M assertions.

- Paper: [Advanced Semantics for Commonsense Knowledge Extraction (WWW'21)](https://arxiv.org/abs/2011.00905)
- Website: https://ascent.mpi-inf.mpg.de/

(I am the author of the dataset)


More consistent copy logic

Use `info.copy()` instead of `copy.deepcopy(info)`.
`Features.copy` now creates a deep copy.


fixed download link for web_science

Fixes #2337. Should work with:
`dataset = load_dataset("web_of_science", "WOS11967", ignore_verifications=True)`


NonMatchingChecksumError for web_of_science dataset

Something isn't working

I've raised a PR for this. Should work with `dataset = load_dataset("web_of_science", "WOS11967", ignore_verifications=True)`once it gets merged into the main branch. Thanks for reporting this!

NonMatchingChecksumError when trying to download the web_of_science dataset.

>NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://data.mendeley.com/datasets/9rw3vkcfy4/6/files/c9ea673d-5542-44c0-ab7b-f1311f7d61df/WebOfScience.zip?dl=1']

Setting `ignore_verfications=True` results in OSError.

>OSError: Cannot find data file.
Original error:
[Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/37ab2c42f50d553c1d0ea432baca3e9e11fedea4aeec63a81e6b7e25dd10d4e7/WOS5736/X.txt'

```python
dataset = load_dataset('web_of_science', 'WOS5736')
```
There are 3 data instances and they all don't work. 'WOS5736', 'WOS11967', 'WOS46985'

datasets 1.6.2
python 3.7.10
Ubuntu 18.04.5 LTS


Fix overflow issue in interpolation search

~~Seems like the CI failure is unrelated to this PR~~ (fixed with the merge).

@lhoestq Can you please verify that everything is OK in terms of speed? Another solution is to change the offsets array dtype to np.int64 (but this doesn't scale in theory compared to Python integer which is unbound). I'm not sure why on my 64-bit machine the default numpy dtype is np.int32 tho.

Hi ! Thanks for the fix.
Unfortunately in terms of speed this is not acceptable :/
The `get_batch_of_1024_random_rows` metric or the `benchmark_getitem_100B ` benchmark is almost at 1sec instead of a few milliseconds.

Would it be possible to avoid the overflow by simply passing `dtype=np.int64` to `np.cumsum` ?
On windows machines the default is int32 unfortunately so we have to force the dtype to be int64

Yes, casting the array to np.int64 should work as well. Another option would be to cast the array elements (`arr[i], arr[j]`) in interpolation search to Python integers (bound only with memory) before multiplication (the error stems from this part: `(j - i) * (x - arr[i])`) when working with big values. But for now, the first option is OK for the sake of simplicity.

Fixes #2335

More info about this error can be found [here](https://stackoverflow.com/questions/53239890/why-do-i-keep-getting-this-error-runtimewarning-overflow-encountered-in-int-sc/53240100).


Index error in Dataset.map

Something isn't working

The following code, if executed on master, raises an IndexError (due to overflow):
```python
>>> from datasets import *
>>> d = load_dataset("bookcorpus", split="train")
Reusing dataset bookcorpus (C:\Users\Mario\.cache\huggingface\datasets\bookcorpus\plain_text\1.0.0\44662c4a114441c35200992bea923b170e6f13f2f0beb7c14e43759cec498700)
2021-05-08 21:23:46.859818: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
>>> d.map(lambda ex: ex)
  0%|▎                                                                    | 289430/74004228 [00:13<58:41, 20935.33ex/s]c:\users\mario\desktop\projects\datasets-1\src\datasets\table.py:84: RuntimeWarning: overflow encountered in int_scalars
  k = i + ((j - i) * (x - arr[i]) // (arr[j] - arr[i]))
  0%|▎                                                                    | 290162/74004228 [00:13<59:11, 20757.23ex/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 1498, in map
    new_fingerprint=new_fingerprint,
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 174, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\fingerprint.py", line 340, in wrapper
    out = func(self, *args, **kwargs)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 1799, in _map_single
    for i, example in enumerate(pbar):
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\site-packages\tqdm\std.py", line 1133, in __iter__
    for obj in iterable:
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 1145, in __iter__
    format_kwargs=format_kwargs,
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 1337, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\formatting\formatting.py", line 368, in query_table
    pa_subtable = _query_table(table, key)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\formatting\formatting.py", line 79, in _query_table
    return table.fast_slice(key % table.num_rows, 1)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\table.py", line 128, in fast_slice
    i = _interpolation_search(self._offsets, offset)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\table.py", line 91, in _interpolation_search
    raise IndexError(f"Invalid query '{x}' for size {arr[-1] if len(arr) else 'none'}.")
IndexError: Invalid query '290162' for size 74004228.
```
Tested on Windows, can run on Linux if needed.

EDIT:
It seems like for this to happen, the default NumPy dtype has to be np.int32.


Updating the DART file checksums in GEM

The DART files were just updated on the source GitHub

https://github.com/Yale-LILY/dart/commit/34b3c872da4811523e334f1631e54ca8105dffab


Fix duplicate keys

As noticed in https://github.com/huggingface/datasets/pull/2245, many datasets yield duplicate keys.
Most of the time it was because the counter used for ids were reset at each new data file.


Add note about indices mapping in save_to_disk docstring


Add Topical-Chat

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Topical-Chat
- **Description:** a knowledge-grounded human-human conversation dataset where the underlying knowledge spans 8 broad topics and conversation partners don’t have explicitly defined roles
- **Paper:** https://www.isca-speech.org/archive/Interspeech_2019/pdfs/3079.pdf
- **Data:** https://github.com/alexa/Topical-Chat
- **Motivation:** Good quality, knowledge-grounded dataset that spans a broad range of topics

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Allow passing `desc` to `tqdm` in `Dataset.map()`

enhancement

New feature or request

good first issue

Good for newcomers

Hi @lhoestq,
Should we change `desc` in [pbar](https://github.com/huggingface/datasets/blob/81fcf88172ed5e3026ef68aed4c0ec6980372333/src/datasets/arrow_dataset.py#L1860) to something meaningful?

I think the user could pass the `desc` parameter to `map` so that it can be displayed in the tqdm progress bar, as suggested by @cccntu.

When there's no multiprocessing, the `desc` of the progress bar could be the `desc` passed by the user.
In multiprocessing, we were already using a `desc` equal to `"#" + str(rank)`.
We can change it to be `(desc or "") + "#" + str(rank)` instead.

In the end, since both `desc` and `rank` could be None, we can have:
```python
pbar_desc = (desc or "") + "#" + str(rank) if rank is not None else desc
```

Finally let's remember that if we add `desc` as a new parameter to `map`, we should add it to the `ignore_kwargs` list of the `@fingerprint_transform` decorator of `Dataset._map_single` since we don't want this parameter to affect the fingerprint of the resulting dataset.

It's normal to have many `map()` calls, and some of them can take a few minutes,
it would be nice to have a description on the progress bar.

Alternative solution:
Print the description before/after the `map()` call.


Add cache dir for in-memory datasets

Yes, having `cache_dir` as an attribute looks cleaner.

Good job! Looking forward to this new feature! 🥂

@lhoestq Sorry for the late reply. Yes, I'll start working on tests. Thanks for the detailed explanation of the current issues with caching (like the idea of adding the `use_caching` parameter to `load_dataset`)

@lhoestq Sure. I'm aware this is a high-priority issue to some extent, so feel free to take over.

Few suggestions I have:
* there is a slight difference between setting `use_caching` to `False` in `load_dataset` and disabling caching globally with `set_caching_enabled(False)` because the former will never execute the following code (`self._cache_dir` is always `False`):
https://github.com/huggingface/datasets/blob/c231abdb174987419bbde3360b5b9d6a4672c736/src/datasets/arrow_dataset.py#L1807-L1824
, so I'm just checking whether this is intended (if yes, maybe the docs should mention this) or not?
* think we should add the `use_caching` parameter to every method that has the `keep_in_memory` (and `in_memory` 😃) parameter in its signature for better consistency, but I say let's address this in a separate PR. IMO we need one more PR that will deal exclusively with consistency in the caching logic.

Hi @mariosasko
We discussed internally and we think that this feature might not be the direction we're doing to take for these reasons:

- it goes against our simple definition of caching: on-disk == uses file cache, and in-memory == nothing is written to disk. I think it adds too much complexity just for a minimal flexibility addition
- there are a few edge cases which are really confusing:
    - map on an in memory dataset with a cache_file_name specified by the user -> should the result be in memory or from disk ?
    - it would require a special cache directory just for in memory datasets, since they don’t have a preferred directory for caching
- it would break a lot of stuff and would require to rewrite significant parts of the core code and the tests

So in the end we're probably going to close this PR.
Let me know what you think, and thank you anyway for your help on this !

Hi,

I'm fine with that. I agree this adds too much complexity. Btw, I like the idea of reverting default in-memory for small datasets that led to this PR.

Superseded by #2460 (to close issue #2458).

Adds the cache dir attribute to DatasetInfo as suggested by @lhoestq.

Should fix #2322


Add Matthews/Pearson/Spearman correlation metrics

Added three metrics:
- The Matthews correlation coefficient (from sklearn)
- The Pearson correlation coefficient (from scipy)
- The Spearman correlation coefficient (from scipy)

cc @sgugger


A syntax error in example

bug

Something isn't working

cc @beurkinger but I think this has been fixed internally and will soon be updated right ?

![image](https://user-images.githubusercontent.com/6883957/117315905-b47a5c00-aeba-11eb-91eb-b2a4a0212a56.png)

Sorry to report with an image, I can't find the template source code of this snippet.


Enable auto-download for PAN-X / Wikiann domain in XTREME

This PR replaces the manual download of the `PAN-X.lang` domains with an auto-download from a Dropbox link provided by the Wikiann author. We also add the relevant dummy data for these domains.

While re-generating `dataset_infos.json` I ran into a `KeyError` in the `udpos.Arabic` domain so have included a fix for this as well.


Added the HLGD dataset

Is there anything else needed from my end?

Thanks Bhavitvya and Quentin, this was very streamlined!

Added the Headline Grouping Dataset (HLGD), from the NAACL2021 paper: News Headline Grouping as a Challenging NLU Task
Dataset Link: https://github.com/tingofurro/headline_grouping
Paper link: https://people.eecs.berkeley.edu/~phillab/pdfs/NAACL2021_HLG.pdf


Create Audio feature

1.12

Next minor release

For optimal storage, it would be better to:
- store only the audio file path in the cache Arrow file
- perform decoding of the audio file (into audio array and sample rate) on the fly, while loading the dataset from cache (or by adding a convenient `load_audio` function)

Thanks a lot @lhoestq for your helpful insights! 🤗

Just one step before having a first running example to benchmark.

Decision to make: how to call the function `dataset.features.decode_example`:
- The usual approach until now in speech applications: call it in a subsequent `.map` function
  - Pros: multiprocessing can be used out of the box
  - Cons: large disk storage required for caching decoded audio files, although having it cached will enhance speed for further usage
- Approach suggested by @lhoestq (see above: https://github.com/huggingface/datasets/pull/2324#discussion_r660758683): doing it in formatting
  - Pros: no large disk storage required, as it will be done on the fly while iterating on the dataset
  - Cons: it is not cached; need to implement multiprocessing for this case
- Other pros/cons for the previous options?
- Other options?

cc: @lhoestq @patrickvonplaten @anton-l

@albertvillanova I'm in two minds about this, to be honest. For example, if we consider CommonVoice, which is encoded in lossy `mp3`:

- If we decompress `mp3` into raw `wav` arrays,  loading a batch will speed up about 40x.
- However, a 60gb English mp3 dataset will blow up to about 600gb raw (iirc), which is why loading on-the-fly (optionally?) could be very beneficial as well.

Users can do the conversion from mp3 to wav by themselves if they want to using `map`.

IMO it's better if we can keep the decoding part with the minimal features to be both easy to understand and flexible, i.e. just having the on-the-fly decoding of the audio data (with the sampling rate parameter)

Decompressing from mp3 to wav sounds like an optimization that depends on the problem that the user wants to solve, the constrains from its environment (disk space, IO speed), and other parameters (optimal training speed for example). Therefore I would leave this to the user to decide whether it has to do it or not.

Let me know what you think about this

@albertvillanova, In my opinion the pros strongly outweigh the cons in the @lhoestq's suggestion which is why I think we should go forward with it.

The cons:
- "the operation won't be cached" is not to important as the user will most likely access just a couple of audio array to see how it looks like and then for the "full" feature extraction she/he will make use of `.map(...)` anyways which means that the result will be cached.
- Regarding the multi-processing - if I understand correctly it'll follow the same logic here -> the user will only access some audio arrays for testing playing around with the model but use `.map(...)` for larger operations where multi-processing would still work as before.

The advantages mostly solve the main poinpoints being:
- exploding disk space
- bad user experience since the audio is not loaded on the go

=> So I'm very much in favor of the "direct-access" feature

Update: I've retaken this issue.

If the decoding logic is implemented when "examples are accessed", then if afterwards we use the `.map`, it tries to apply the decoding twice (as maps iterates over the examples, thus "accessing them", before trying to apply the map function)...

I'm thinking on some other approach...

I have reimplemented the previous approach, so that we can discuss about it: examples are decoded when accessed.

What about creating a new specific formatting, just for decoding? This would be only active within a context manager.

Hi @lhoestq, as we discussed, I've followed your suggestion of implementing the decoding step within the formatting logic: extract-decode-format. Feel free to tell me what you think.

@patrickvonplaten and @anton-l, could you have a look at the use case in the test (https://github.com/huggingface/datasets/pull/2324/files#diff-58e348f6e4deaa5f3119e420a5d48ebb82875a78c28628831748fb54f59b2c78R34-R50) and tell me if this is aligned with your needs? Thanks.

Hi @lhoestq, if you validate this approach, we could merge the Audio feature this (or early next) week.

Sure it looks nice this way :) Feel free to continue !

As discussed, we should pay attention when applying `map` to a dataset with `Audio` feature, in order to avoid decoding the audio data twice.

One proposed solution is to pass `input_columns` to `map`. Just, note that the field containing the Audio feature should not be passed in `input_columns` (not possible, for example, to map the audio file path to a new directory).

I suggest again (3rd time, sorry, lol) using a formatting context manager (as we already use for PyTorch/TensorFlow: https://huggingface.co/docs/datasets/torch_tensorflow.html).

Above (https://github.com/huggingface/datasets/pull/2324#issuecomment-915244003), I suggested to define a formatting just for decoding: the decoding of the audio data is only performed if this specific formatting is set (`ds.set_format("decoding")`) or within a context manager (`with ds.formatted_as("decoding"): ...`)

Now, I would like also to suggest an alternative formatting for **non-decoding** (if decoding is the default behavior), for a use case like this:
```python
def change_dir(example):
    example["audio"] = "dir/" + example["audio"]

with ds.formatted_as("no_decoding"):
    print(ds[0])  # {"audio": "path/to/file.wav"}
    ds.map(change_dir)
    print(ds[0])  # {"audio": "dir/path/to/file.wav"}

print(ds[0])  # {"audio": {"path": "dir/path/to/file.wav", "array": np.array([1., 2., 3...]), "sampling_rate": 44100}}
```

Please, just tell me what you think.
CC: @lhoestq @patrickvonplaten @anton-l

> As discussed, we should pay attention when applying `map` to a dataset with `Audio` feature, in order to avoid decoding the audio data twice.
>
> One proposed solution is to pass `input_columns` to `map`. Just, note that the field containing the Audio feature should not be passed in `input_columns` (not possible, for example, to map the audio file path to a new directory).
>
> I suggest again (3rd time, sorry, lol) using a formatting context manager (as we already use for PyTorch/TensorFlow: https://huggingface.co/docs/datasets/torch_tensorflow.html).
>
> Above ([#2324 (comment)](https://github.com/huggingface/datasets/pull/2324#issuecomment-915244003)), I suggested to define a formatting just for decoding: the decoding of the audio data is only performed if this specific formatting is set (`ds.set_format("decoding")`) or within a context manager (`with ds.formatted_as("decoding"): ...`)
>
> Now, I would like also to suggest an alternative formatting for **non-decoding** (if decoding is the default behavior), for a use case like this:
>
> ```python
> def change_dir(example):
>     example["audio"] = "dir/" + example["audio"]
>
>
> with ds.formatted_as("no_decoding"):
>     print(ds[0])  # {"audio": "path/to/file.wav"}
>     ds.map(change_dir)
>     print(ds[0])  # {"audio": "dir/path/to/file.wav"}
>
> print(ds[0])  # {"audio": {"path": "dir/path/to/file.wav", "array": np.array([1., 2., 3...]), "sampling_rate": 44100}}
> ```
>
> Please, just tell me what you think.
> CC: @lhoestq @patrickvonplaten @anton-l

I'm fine with a context manager! There is no way to **not** decode the audio if its key is not accessed no?

E.g.

```python
def load(batch):
    batch["speech_array"] = torchaudio.load(batch["file"])
    return batch

ds.map(load)
```

does *e.g.* not access the "audio" key `batch["audio"}` but there is no way to not decode it without major changes no?

=> I'm happy with both the context manager and using `input_colmuns`. Both of those solutions are equally good to me if a "not-access-key-no-decoding" solution is just not feasible. I let you guys decide :-)

>
> There is no way to **not** decode the audio if its key is not accessed no?
>
> E.g...
>
> does _e.g._ not access the "audio" key `batch["audio"}` but there is no way to not decode it without major changes no?

@patrickvonplaten I think therefore we should rethink the implementation of the Audio feature: its goal is to enrich/simplify the user experience when working with audio files. If on the other hand, you see that the current implementation may be problematic/unsatisfying/not-optimal, then we miss the point of creating this feature. This feature should be useful to users, not inconvenient.

> > There is no way to **not** decode the audio if its key is not accessed no?
> > E.g...
> > does _e.g._ not access the "audio" key `batch["audio"}` but there is no way to not decode it without major changes no?
>
> @patrickvonplaten I think therefore we should rethink the implementation of the Audio feature: its goal is to enrich/simplify the user experience when working with audio files. If on the other hand, you see that the current implementation may be problematic/unsatisfying/not-optimal, then we miss the point of creating this feature. This feature should be useful to users, not inconvenient.

Thanks a lot for the message! I'm discussing a bit with @anton-l at the moment - will share our results as soon as possible

Current implementation: see use cases in file https://github.com/huggingface/datasets/blob/0f80e6eaa6f596ff6287eb33587e2d9c69af0e73/tests/features/test_audio.py

Automatic decoding:
- when directly accessing an example or a batch
  ```python
  dset[0]
  dset[:2]
  ```
- during map, only if audio field is accessed:
  ```python
  def process_audio_sampling_rate(example):
      example["double_sampling_rate"] = 2 * example["audio"]["sampling_rate"]
      return example

  decoded_dset = dset.map(process_audio_sampling_rate)
  ```

No automatic decoding:
- during map if audio field is not accessed:
  ```python
  def process_text(example):
      example["text"] = example["text"] + " World!"
      return example

  decoded_dset = dset.map(process_text)
  ```

The types of example and batch are kept as usual, `dict[str, Any]` and `dict[str, list[Any]]` respectively.

CC: @patrickvonplaten @anton-l @lhoestq

That's awesome! Thanks so much for your work on this @albertvillanova!

Oh and maybe have a test to make sure that casting the Audio feature to change the sampling rate works as expected ?

@lhoestq the test for the resampling is already in place in `test_audio_resampling`:
https://github.com/huggingface/datasets/pull/2324/files#diff-58e348f6e4deaa5f3119e420a5d48ebb82875a78c28628831748fb54f59b2c78R48-R56

Please note that we should agree in the API: see 53d6d73

This is just a proposal implementation:
- Create a new method named `cast_column`, which performs a shallow kind of cast (without using `map()` or caching)

We should agree in the name, because as it is, it might be confused with `cast` (and users might think `cast_column` caches the result as `cast`)

CC: @lhoestq @patrickvonplaten @anton-l

IMO cast and cast_column should have the exact same behavior, to make the experience simple for the user (no distinction between shallow or deep cast).

Maybe we should change `cast` to use `cast_column` on every column and make `cast_column` use `map` if and only if it's necessary. For Audio for example `map` is not needed.

We just need to do some tests to know which casts always need map and which ones don't. This implies either looking at the PyArrow source code (the documentation doesn't mention all these details) or playing with PyArrow to figure it out.

I guess for now we can just have the simplest `cast_column` which always uses map unless it's an Audio feature type.

Let me know what you think !

@lhoestq I totally agree: `cast` and `cast_column` should be analog to each other.

For the implementation, let me try something simpler than the one suggested by you...

@lhoestq what do you think of an approach like this 633ef09?

If it's OK, then we should implement passing parameters to `cast`.

@lhoestq maybe for now we could make a simple implementation and finish this PR. Then we could make a follow-up PR to deal specifically with the optimal implementation of `cast_column` and `cast`, as this issue is not specific to the Audio feature.

> @lhoestq what do you think of an approach like this 633ef09?

Yea that's good enough for the time being :)

I think the last thing we need to do is make sure that `cast_column` changes the fingerprint of the dataset. Feel free to use the `fingerprint_transform` decorator, as for `remove_columns`.

(note that cast currently doesn't use the decorator since it's based on `map` that already updates the fingerprint).

>
> I think the last thing we need to do is make sure that `cast_column` changes the fingerprint of the dataset. Feel free to use the `fingerprint_transform` decorator, as for `remove_columns`.
>
> (note that cast currently doesn't use the decorator since it's based on `map` that already updates the fingerprint).

@lhoestq note that `cast_column` may call `cast` in some cases, and the decorator would not be necessary for these cases...
- I did it by setting `inplace=False`, and updating fingerprint explicitly only when `cast` is not called.

I think current state of this PR could be included in our next release, as experimental feature, for stress testing it and try to find all potential issues. What do you think?

CC: @lhoestq @patrickvonplaten @anton-l

Looks great! Ready to try it out on the transformers examples after the release :)

Create `Audio` feature to handle raw audio files.

Some decisions to be further discussed:
- I have chosen `soundfile` as the audio library; another interesting library is `librosa`, but this requires `soundfile` (see [here](https://github.com/librosa/librosa/blob/main/setup.cfg#L53)). If we require some more advanced functionalities, we could eventually switch the library.
- I have implemented the audio feature as an extra: `pip install datasets[audio]`. For the moment, the typical datasets user uses only text datasets, and there is no need for them for additional package requirements for audio/image if they do not need them.
- For tests, I require audio dependencies (so that all audio functionalities are checked with our CI test suite); I exclude Linux platforms, which require an additional library to be installed with the distribution package manager
  - I also require `pytest-datadir`, which allow to have (audio) data files for tests
- The audio data contain: array and sample_rate.
- The array is reshaped as 1D array (expected input for `Wav2Vec2`).

Note that to install `soundfile` on Linux, you need to install `libsndfile` using your distribution’s package manager, for example `sudo apt-get install libsndfile1`.

## Requirements Specification
- Access example with audio loading and resampling:
  ```python
  ds[0]["audio"]
  ```
- Map with audio loading & resampling:
  ```python
  def preprocess(batch):
       batch["input_values"] = processor(batch["audio"]).input_values
       return batch

  ds = ds.map(preprocess)
  ```
- Map without audio loading and resampling:
  ```python
  def preprocess(batch):
       batch["labels"] = processor(batch["target_text"]).input_values
       return batch

  ds = ds.map(preprocess)
  ```
- Additional requirement specification (see https://github.com/huggingface/datasets/pull/2324#pullrequestreview-768864998): Cast audio column to change sampling sate:
  ```python
  ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
  ```


load_dataset("timit_asr") gives back duplicates of just one sample text

Something isn't working

Upgrading datasets to version 1.6 fixes the issue

This bug was fixed in #1995. Upgrading the `datasets` should work!

Thanks @ekeleshian for having reported.

I am closing this issue once that you updated `datasets`. Feel free to reopen it if the problem persists.

## Describe the bug
When you look up on key ["train"] and then ['text'], you get back a list  with just one sentence duplicated 4620 times. Namely, the sentence "Would such an act of refusal be useful?". Similarly when you look up ['test'] and then ['text'], the list is one sentence repeated "The bungalow was pleasantly situated near the shore." 1680 times.

I tried to work around the issue by downgrading to datasets version 1.3.0, inspired by [this post](https://www.gitmemory.com/issue/huggingface/datasets/2052/798904836) and removing the entire huggingface directory from ~/.cache, but I still get the same issue.

## Steps to reproduce the bug
```python
from datasets import load_dataset
timit = load_dataset("timit_asr")
print(timit['train']['text'])
print(timit['test']['text'])
```

## Expected Result
Rows of diverse text, like how it is shown in the [wav2vec2.0 tutorial](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tuning_Wav2Vec2_for_English_ASR.ipynb)

## Actual results
Rows of repeated text.

## Versions
- Datasets: 1.3.0
- Python: 3.9.1
- Platform: macOS-11.2.1-x86_64-i386-64bit}