Metrics documentation is not accessible in the datasets doc UI

enhancement

New feature or request

Metric discussion

Discussions on the metrics

Hey @fxmarty :) Yes we are working on showing the docs of all the metrics on the Hugging face website. If you want to follow the advancements you can check the [evaluate](https://github.com/huggingface/evaluate) repository cc @lvwerra @sashavor

**Is your feature request related to a problem? Please describe.**
Search for a metric name like "seqeval" yields no results on https://huggingface.co/docs/datasets/master/en/index . One needs to go look in `datasets/metrics/README.md` to find the doc. Even in the `README.md`, it can be hard to understand what the metric expects as an input, for example for `squad` there is a [key `id`](https://github.com/huggingface/datasets/blob/1a4c185663a6958f48ec69624473fdc154a36a9d/metrics/squad/squad.py#L42) documented only in the function doc but not in the `README.md`, and one needs to go look into the code to understand what the metric expects.

**Describe the solution you'd like**
Have the documentation for metrics appear as well in the doc UI, e.g. this https://github.com/huggingface/datasets/blob/1a4c185663a6958f48ec69624473fdc154a36a9d/metrics/squad/squad.py#L21-L63

I know there are plans to migrate metrics to the evaluate library, but just pointing this out.


Fix failing CI on Windows for sari and wiki_split metrics

This PR adds `sacremoses` as explicit tests dependency (required by sari and wiki_split metrics).

Before, this library was installed as a third-party dependency, but this is no longer the case for Windows.

Fix #4341.


Failing CI on Windows for sari and wiki_split metrics

bug

Something isn't working

## Describe the bug
Our CI is failing from yesterday on Windows for metrics: sari and wiki_split
```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_sari - ...
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_wiki_split
```

See: https://app.circleci.com/pipelines/github/huggingface/datasets/11928/workflows/79daa5e7-65c9-4e85-829b-00d2bfbd076a/jobs/71594


Fix irc_disentangle dataset script

Thanks ! This has been fixed in https://github.com/huggingface/datasets/pull/4377, we can close this PR

updated extracted dataset's repo's latest commit hash (included in tarball's name), and updated the related data_infos.json


Dataset loader for the MSLR2022 shared task

I think the underlying issue is in https://github.com/huggingface/datasets/blob/c0ed6fdc29675b3565b01b77fde5ab5d9d8b60ec/src/datasets/commands/dummy_data.py#L124 - where `CSV`s are considered to be in the same class of file as text, jsonl, and tsv.

I think this is an error because CSVs can have newlines within the rows of a file. I'm happy to make a PR to change how this handling works, or make the change within this PR.

We should figure out:
1. Does this dummy data need to be generated more than once? (It looks like no)
2. Should this be fixed generally? (needs a HF person to weigh in here)
3. What is the right way for such a fix to exist permanently here; the [Contributing document](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md) doesn't provide guidance on any tests. Writing a test is several times more effort than fixing the underlying issue. (again needs a HF person)

Would someone from HF mind taking a look at this PR? (@lhoestq)

Hi ! Sorry for the delay in responding :)

I don't think there's a big need to fix this in the general case for now, feel free to just generate the dummy data for this specific dataset :)

The `datasets-cli dummy_data datasets/mslr2022` command should tell you what dummy files to generate. In each dummy file you just need to include enough data to generate 4 or 5 examples

_The documentation is not available anymore as the PR was closed or merged._

Awesome! Generated the dummy data and the tests now pass. @jayded thanks for your help! If you and @lucylw are happy with this I think it's ready to be merged. @lhoestq this is ready for another look :)

Hi @lhoestq, is there anything blocking this from being merged that I can address?

Hi @JohnGiorgi ! Thanks for the changes, it looks all good now :)

I think this dataset can be under the AllenAI page here: https://huggingface.co/allenai What do you think ?
Feel free to create a new dataset repository on huggingface.co and upload your files (dataset script, readme, etc.)

Once the dataset is under the AllenAI org, we can close this PR

> Hi @JohnGiorgi ! Thanks for the changes, it looks all good now :)
>
> I think this dataset can be under the AllenAI page here: https://huggingface.co/allenai What do you think ? Feel free to create a new dataset repository on huggingface.co and upload your files (dataset script, readme, etc.)
>
> Once the dataset is under the AllenAI org, we can close this PR

Sweet! It is uploaded here: https://huggingface.co/datasets/allenai/mslr2022

Nice ! Thanks :)

I think we can close this PR then.

I noticed that the dataset preview is not available on this dataset, this is because we require datasets to work in streaming mode to show a preview. However TAR archives don't work well in streaming mode (you can't know in advance what files are inside a TAR archive without reading it completely). This can be fixed by using a ZIP archive instead.

Let me know if you have questions or if I can help.

This PR adds a dataset loader for the [MSLR2022 Shared Task](https://github.com/allenai/mslr-shared-task). Both the MS^2 and Cochrane datasets can be loaded with this dataloader:

```python
from datasets import load_dataset

ms2 = load_dataset("mslr2022", "ms2")
cochrane = load_dataset("mslr2022", "cochrane")
```

Usage looks like:

```python
>>> ms2 = load_dataset("mslr2022", "ms2", split="validation")
>>> ms2.keys()
dict_keys(['review_id', 'pmid', 'title', 'abstract', 'target', 'background', 'reviews_info'])
>>> ms2[0].target
'Conclusions SC therapy is effective for PAH in pre clinical studies .\nThese results may help to st and ardise pre clinical animal studies and provide a theoretical basis for clinical trial design in the future .'
```

I have tested this works with the following command:

```bash
datasets-cli test datasets/mslr2022 --save_infos --all_configs
```

However I have having a little trouble generating the dummy data

```bash
datasets-cli dummy_data datasets/mslr2022 --auto_generate
```

errors out with the following stack trace:

```
Couldn't generate dummy file 'datasets/mslr2022/dummy/ms2/1.0.0/dummy_data/mslr_data.tar.gz/mslr_data/ms2/convert_to_cochrane.py'. Ignore that if this file is not useful for dummy data.
Traceback (most recent call last):
  File "/Users/johngiorgi/.pyenv/versions/datasets/bin/datasets-cli", line 11, in <module>
    load_entry_point('datasets', 'console_scripts', 'datasets-cli')()
  File "/Users/johngiorgi/Documents/dev/datasets/src/datasets/commands/datasets_cli.py", line 39, in main
    service.run()
  File "/Users/johngiorgi/Documents/dev/datasets/src/datasets/commands/dummy_data.py", line 319, in run
    keep_uncompressed=self._keep_uncompressed,
  File "/Users/johngiorgi/Documents/dev/datasets/src/datasets/commands/dummy_data.py", line 361, in _autogenerate_dummy_data
    dataset_builder._prepare_split(split_generator, check_duplicate_keys=False)
  File "/Users/johngiorgi/Documents/dev/datasets/src/datasets/builder.py", line 1146, in _prepare_split
    desc=f"Generating {split_info.name} split",
  File "/Users/johngiorgi/.pyenv/versions/3.7.13/envs/datasets/lib/python3.7/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/Users/johngiorgi/.cache/huggingface/modules/datasets_modules/datasets/mslr2022/b4becd2f52cf18255d4934d7154c2a1127fb393371b87b3c1fc2c8b35a777cea/mslr2022.py", line 149, in _generate_examples
    reviews_info_df = pd.read_csv(reviews_info_filepath, index_col=0)
  File "/Users/johngiorgi/.pyenv/versions/3.7.13/envs/datasets/lib/python3.7/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/Users/johngiorgi/.pyenv/versions/3.7.13/envs/datasets/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/Users/johngiorgi/.pyenv/versions/3.7.13/envs/datasets/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 488, in _read
    return parser.read(nrows)
  File "/Users/johngiorgi/.pyenv/versions/3.7.13/envs/datasets/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 1047, in read
    index, columns, col_dict = self._engine.read(nrows)
  File "/Users/johngiorgi/.pyenv/versions/3.7.13/envs/datasets/lib/python3.7/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 224, in read
    chunks = self._reader.read_low_memory(nrows)
  File "pandas/_libs/parsers.pyx", line 801, in pandas._libs.parsers.TextReader.read_low_memory
  File "pandas/_libs/parsers.pyx", line 857, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 843, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 1925, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: EOF inside string starting at row 2
```

I think this may have to do with unusual line terminators in the original data. When I open it in VSCode, it complains:

```
The file 'dev-inputs.csv' contains one or more unusual line terminator characters, like Line Separator (LS) or Paragraph Separator (PS).

It is recommended to remove them from the file. This can be configured via `editor.unusualLineTerminators`.
```

Tagging the organizers of the shared task in case they want to sanity check this or add any info to the model card :) @lucylw  @jayded


Eval metadata Batch 4: Tweet Eval, Tweets Hate Speech Detection, VCTK, Weibo NER, Wisesight Sentiment, XSum, Yahoo Answers Topics, Yelp Polarity, Yelp Review Full

Summary of CircleCI errors:

- **XSum**: missing 6 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', and 'source_datasets'
- **Yelp_polarity**: missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'

Adding evaluation metadata for:

- Tweet Eval
- Tweets Hate Speech Detection
- VCTK
- Weibo NER
- Wisesight Sentiment
- XSum
- Yahoo Answers Topics
- Yelp Polarity
- Yelp Review Full


Eval metadata batch 3: Reddit, Rotten Tomatoes, SemEval 2010, Sentiment 140, SMS Spam, Snips, SQuAD, SQuAD v2, Timit ASR

Summary of CircleCI errors:

- **sem_eval_2010_task_8**: missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
- **sms_spam**: `Data Instances` and`Data Splits` are empty.
- **Quora** : Expected some content in section `Citation Information` but it is empty,  missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
- **sentiment140**:  missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'

There are also some timeout errors, I don't really understand the source though :confused:

Adding evaluation metadata for:
- Reddit
- Rotten Tomatoes
- SemEval 2010
- Sentiment 140
- SMS Spam
- Snips
- SQuAD
- SQuAD v2
- Timit ASR


Eval metadata batch 2 : Health Fact, Jigsaw Toxicity, LIAR, LJ Speech, MSRA NER, Multi News, NCBI Disease, Poem Sentiment

Summary of CircleCI errors:
- **Jjigsaw_toxicity_pred**:  `Citation Information` but it is empty.
- **LIAR** :  `Data Instances`,`Data Fields`,  `Data Splits`,  `Citation Information` are empty.
- **MSRA NER** : Dataset Summary`, `Data Instances`, `Data Fields`,  `Data Splits`, `Citation Information` are empty.

The CI errors about missing content in the dataset cards can be ignored in this PR btw

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4336). All of your documentation changes will be reflected on that endpoint.

Adding evaluation metadata for :
- Health Fact
- Jigsaw Toxicity
- LIAR
- LJ Speech
- MSRA NER
- Multi News
- NCBI Diseas
- Poem Sentiment


Eval metadata batch 1: BillSum, CoNLL2003, CoNLLPP, CUAD, Emotion, GigaWord, GLUE, Hate Speech 18,  Hate Speech

Summary of CircleCI errors:
- **BoolQ**: missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
- **Conllpp**: expected some content in section `Citation Information` but it is empty.
- **GLUE**:   'annotations_creators', 'language_creators', 'source_datasets' :['unknown'] are not registered tags
- **ConLL2003**:  field 'task_ids':  ['part-of-speech-tagging'] are not registered tags for 'task_ids'
-  **Hate_speech18:**   Expected some content in section `Data Instances` but it is empty, Expected some content in section `Data Splits` but it is empty

And yes we can ignore all the CI errors related to missing content in the dataset cards, these issues can be fixed in other PRs

Adding evaluation metadata for:
- BillSum
- CoNLL2003
- CoNLLPP
- CUAD
- Emotion
- GigaWord
- GLUE
- Hate Speech 18
- Hate Speech Offensive


Adding eval metadata for billsum

Adding eval metadata for billsum


Adding eval metadata for Banking 77

@lhoestq , Circle CI is giving me an error, saying that ['extended'] is a key that shouldn't be in the dataset metadata, but it was there before my modification (so I don't want to remove it)

Adding eval metadata for Banking 77


Adding eval metadata for arabic speech corpus


Adding eval metadata to Amazon Polarity

Adding eval metadata to Amazon Polarity


Adding eval metadata to Allociné dataset


Adding eval metadata for AG News

Adding eval metadata for AG News


Fix and clean Apache Beam functionality


`wikipedia` pre-processed datasets

Something isn't working

Hi @vpj, thanks for reporting.

I'm sorry, but I can't reproduce your bug: I load "20220301.simple"in 9 seconds:
```shell
time python -c "from datasets import load_dataset; load_dataset('wikipedia', '20220301.simple')"

Downloading and preparing dataset wikipedia/20220301.simple (download: 228.58 MiB, generated: 224.18 MiB, post-processed: Unknown size, total: 452.76 MiB) to .../.cache/huggingface/datasets/wikipedia/20220301.simple/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.66k/1.66k [00:00<00:00, 1.02MB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 235M/235M [00:02<00:00, 82.8MB/s]
Dataset wikipedia downloaded and prepared to .../.cache/huggingface/datasets/wikipedia/20220301.simple/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 290.75it/s]

real	0m9.693s
user	0m6.002s
sys	0m3.260s
```

Could you please check your environment info, as requested when opening this issue?
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
```
Maybe you are using an old version of `datasets`...

Downloading and processing `wikipedia simple` dataset completed in under 11sec on M1 Mac. Could you please check `dataset` version as mentioned by @albertvillanova? Also check system specs, if system is under load processing could take some time I guess.

## Describe the bug
[Wikipedia](https://huggingface.co/datasets/wikipedia) dataset readme says that certain subsets are preprocessed. However it seems like they are not available. When I try to load them it takes a really long time, and it seems like it's processing them.

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset("wikipedia", "20220301.en")
```

## Expected results
To load the dataset

## Actual results
Takes a very long time to load (after downloading)

After `Downloading data files: 100%`. It takes hours and gets killed.
Tried `wikipedia.simple` and it got processed after ~30mins.


Fix type hint and documentation for `new_fingerprint`

Currently, there are no type hints nor `Optional` for the argument `new_fingerprint` in several methods of `datasets.arrow_dataset.Dataset`.

There was some documentation missing as well.

Note that pylance is happy with the type hints, but pyright does not detect that `new_fingerprint` is set within the decorator.

The modifications in this PR are fine since here https://github.com/huggingface/datasets/blob/aa743886221d76afb409d263e1b136e7a71fe2b4/src/datasets/fingerprint.py#L446-L454

for the non-inplace case we make sure to auto-generate a new fingerprint (as indicated in the doc).


Dataset Viewer issue for strombergnlp/offenseval_2020, strombergnlp/polstance

dataset-viewer

Related to the dataset viewer on huggingface.co

Not sure if it's related... I was going to raise an issue for https://huggingface.co/datasets/domenicrosati/TruthfulQA which also has the same issue... https://huggingface.co/datasets/domenicrosati/TruthfulQA/viewer/domenicrosati--TruthfulQA/train

Yes, it's related. The backend behind the dataset viewer is currently under too much load, and these datasets are still in the jobs queue. We're actively working on this issue, and we expect to fix the issue permanently soon. Thanks for your patience 🙏

Thanks @severo and no worries! - a suggestion for a UI usability thing maybe is to indicate that the dataset processing is in the job queue (rather than no data?)

Thanks, these are working great now (including @domenicrosati 's, afaics!)

### Link

https://huggingface.co/datasets/strombergnlp/offenseval_2020/viewer/ar/train

### Description

The viewer isn't running for these two datasets. I left it overnight because a wait sometimes helps things get loaded, and the error messages have all gone, but the datasets are still turning up blank in viewer. Maybe it needs a bit more time.

* https://huggingface.co/datasets/strombergnlp/polstance/viewer/PolStance/train

* https://huggingface.co/datasets/strombergnlp/offenseval_2020/viewer/ar/train

While offenseval_2020 is gated w. prompt, the other gated previews I have run fine in Viewer, e.g. https://huggingface.co/datasets/strombergnlp/shaj , so I'm a bit stumped!

### Owner

Yes


Support >1 PWC dataset per dataset card

enhancement

New feature or request

Hi @leondz, I agree it would be nice. We'll see what we can do ;)

**Is your feature request related to a problem? Please describe.**
Some datasets cover more than one dataset on PapersWithCode. For example, the OffensEval 2020 challenge involved five languages, and there's one dataset to cover all five datasets, [`strombergnlp/offenseval_2020`](https://huggingface.co/datasets/strombergnlp/offenseval_2020). However, the yaml `paperswithcode_id:` dataset card entry only supports one value; when multiple are added, the PWC link disappears from the dataset page.

Because the link from a PapersWithCode dataset to a Hugging Face Hub entry can't be entered manually and seems to be scraped, this means end users don't have a way of getting a dataset reader link to appear on all the PWC datasets supported by one HF Hub Dataset reader.

It's not super unusual to have papers introduce multiple parallel variants of a dataset and would be handy to reflect this, so e.g. dataset maintainers can DRY, and so dataset users can keep what they're doing simple.

**Describe the solution you'd like**
I'd like `paperswithcode_id:` to support lists and be able to connect with multiple PWC datasets.

**Describe alternatives you've considered**
De-normalising the datasets on HF Hub to create multiple readers for each variation on a task, i.e. instead of a single `offenseval_2020`, having `offenseval_2020_ar`, `offenseval_2020_da`, `offenseval_2020_gr`, ...

**Additional context**
Hope that's enough

**Priority**
Low


Audio can not find value["bytes"]

bug

Something isn't working

![image](https://user-images.githubusercontent.com/34292279/168063684-fff5c12a-8b1e-4c65-b18b-36100ab8a1af.png)

that is reason my bytes`s empty
but i have some confused why path prior is higher than bytes?

if you can make bytes in _generate_examples , you don`t have to make bytes to path?
because we have path and bytes already

> but i have some confused why path prior is higher than bytes?

If the audio file is already available locally, we don't need to store the bytes again.

If you don't specify a "path" to a local file, then the bytes are stored. You can set "path" to None for example.

> if you can make bytes in _generate_examples , you don`t have to make bytes to path?
> because we have path and bytes already

It's useful to pass both "path" and "bytes" in `_generate_examples`:
- when the dataset has been downloaded, then the "path" to the audio files are stored and we can ignore "bytes" in order to save disk space.
- when the dataset is loaded in streaming mode, the audio files are not available on your disk and therefore we use the "bytes"

@lhoestq
First of all, thx for reply

but, if i put in "bytes" and "path"
ex) {"bytes":"blah blah~", "path":"blah blah~"}

that source working that my bytes to empty first,
and then, re-calculate my bytes!
![image](https://user-images.githubusercontent.com/34292279/168534687-1fb60d8c-d369-47d2-a4bb-db68f95194b4.png)

if you have some pcm file, pcm is can read bytes.
so, i put in bytes and paths.
but bytes is been None why encode_example func make None
and then, on decode_example func, we no have bytes. so, calculate bytes to path.
pcm is not support librosa or soundfile, error occured!

the most important thing is not announced anywhere this situation can be reproduced

is that truly right process flow?

I don't think we support PCM files, feel free to convert your data to WAV for now.

It would be awesome to support PCM files though, let me know if you'd like to contribute this feature, I'd be happy to help

You can clone the repository (see the guide on [how to contribute](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-create-a-pull-request)) and see how we can make the `Image.encode_example` method work with PCM data.

There might be other ways to approach this problem, but here is what I think is a reasonable one:

I think `Image.encode_example` should be able to take PCM bytes as input and the sampling rate, and return the WAV bytes (built by combining the PCM bytes and the sampling rate info), so that `Image.decode_example` can read it.

To check if the input bytes are PCM data, you can just check if the extension of the `path` is ".pcm".

maybe i can start to contribute on this sunday!
@lhoestq

@lhoestq plz check my pr #4409

am i wrong somting?

## Describe the bug
I wrote down _generate_examples like:
![image](https://user-images.githubusercontent.com/34292279/168027186-2fe8b255-2cd8-4b9b-ab1e-8d5a7182979b.png)

but where is the bytes?
![image](https://user-images.githubusercontent.com/34292279/168027330-f2496dd0-1d99-464c-b15c-bc57eee0415a.png)

## Expected results
value["bytes"] is not None, so i can make datasets with bytes, not path

## bytes looks like:
blah blah~~
\xfe\x03\x00\xfb\x06\x1c\x0bo\x074\x03\xaf\x01\x13\x04\xbc\x06\x8c\x05y\x05,\t7\x08\xaf\x03\xc0\xfe\xe8\xfc\x94\xfe\xb7\xfd\xea\xfa\xd5\xf9$\xf9>\xf9\x1f\xf8\r\xf5F\xf49\xf4\xda\xf5-\xf8\n\xf8k\xf8\x07\xfb\x18\xfd\xd9\xfdv\xfd"\xfe\xcc\x01\x1c\x04\x08\x04@\x04{\x06^\tf\t\x1e\x07\x8b\x06\x02\x08\x13\t\x07\x08 \x06g\x06"\x06\xa0\x03\xc6\x002\xff \xff\x1d\xff\x19\xfd?\xfb\xdb\xfa\xfc\xfa$\xfb}\xf9\xe5\xf7\xf9\xf7\xce\xf8.\xf9b\xf9\xc5\xf9\xc0\xfb\xfa\xfcP\xfc\xba\xfbQ\xfc1\xfe\x9f\xff\x12\x00\xa2\x00\x18\x02Z\x03\x02\x04\xb1\x03\xc5\x03W\x04\x82\x04\x8f\x04U\x04\xb6\x04\x10\x05{\x04\x83\x02\x17\x01\x1d\x00\xa0\xff\xec\xfe\x03\xfe#\xfe\xc2\xfe2\xff\xe6\xfe\x9a\xfe~\x01\x91\x08\xb3\tU\x05\x10\x024\x02\xe4\x05\xa8\x07\xa7\x053\x07I\n\x91\x07v\x02\x95\xfd\xbb\xfd\x96\xff\x01\xfe\x1e\xfb\xbb\xf9S\xf8!\xf8\xf4\xf5\xd6\xf3\xf7\xf3l\xf4d\xf6l\xf7d\xf6b\xf7\xc1\xfa(\xfd\xcf\xfd*\xfdq\xfe\xe9\x01\xa8\x03t\x03\x17\x04B\x07\xce\t\t\t\xeb\x06\x0c\x07\x95\x08\x92\t\xbc\x07O\x06\xfb\x06\xd2\x06U\x04\x00\x02\x92\x00\xdc\x00\x84\x00 \xfeT\xfc\xf1\xfb\x82\xfc\x97\xfb}\xf9\x00\xf8_\xf8\x0b\xf9\xe5\xf8\xe2\xf7\xaa\xf8\xb2\xfa\x10\xfbl\xfa\xf5\xf9Y\xfb\xc0\xfd\xe8\xfe\xec\xfe1\x00\xad\x01\xec\x02E\x03\x13\x03\x9b\x03o\x04\xce\x04\xa8\x04\xb2\x04\x1b\x05\xc0\x05\xd2\x04\xe8\x02z\x01\xbe\x00\xae\x00\x07\x00$\xff|\xff\x8e\x00\x13\x00\x10\xff\x98\xff0\x05{\x0b\x05\t\xaa\x03\x82\x01n\x03
blah blah~~

that function not return None

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:2.2.1
- Platform:ubuntu 18.04
- Python version:3.6.9
- PyArrow version:6.0.1


Added stratify option to train_test_split function.

> Nice thank you ! This will be super useful :)
>
> Could you also add some tests in test_arrow_dataset.py and add an example of usage in the `Example:` section of the `train_test_split` docstring ?

I will try to do it, is there any documentation for adding test cases? I have never done it before.

Thanks for the changes !

> I will try to do it, is there any documentation for adding test cases? I have never done it before.

You can just add a function `test_train_test_split_startify` in `test_arrow_dataset.py`.

In this function you can define a dataset and make sure that `train_test_split` with the `stratify` argument works as expected.

You can do `pytest tests/test_arrow_dataset.py::test_train_test_split_startify` to run your test.

Feel free to get some inspiration from other tests like `test_interleave_datasets` for example

I have added tests for stratified train_test_split in `test_arrow_dataset.py` file inside `test_train_test_split_startify` function. I have also added example usage with `stratify` arg in `Example:` section of the `train_test_split` docstring.
Results of tests:
```
(data) nandwalritik@hp:~/datasets$ pytest tests/test_arrow_dataset.py::test_train_test_split_startify -W ignore
============================================================================ test session starts ============================================================================
platform linux -- Python 3.9.5, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/nandwalritik/datasets
plugins: datadir-1.3.1, forked-1.4.0, xdist-2.5.0
collected 1 item

tests/test_arrow_dataset.py .                                                                                [100%]

============================================================================= 1 passed in 0.12s =============================================================================

```

Thanks a lot !

`utils/stratify.py` sounds good yes :)

Also feel free to merge `master` into your branch to fix the CI ;)

Added all the changes as were suggested and rebased with `main`.

_The documentation is not available anymore as the PR was closed or merged._

This PR adds `stratify` option to `train_test_split` method. I took reference from scikit-learn's `StratifiedShuffleSplit` class for implementing stratified split and integrated the changes as were suggested by @lhoestq.

It fixes #3452.

@lhoestq Please review and let me know, if any changes are required.


Adding dataset enwik8

@lhoestq Thank you for the great feedback! Looks like all tests are passing now :)

Because I regularly work with enwik8, I would like to contribute the dataset loader 🤗


Multi-news dataset loader attempts to strip wrong character from beginning of summaries

Something isn't working

Hi ! Thanks for reporting :)

This dataset was simply converted from [tensorflow datasets](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/summarization/multi_news.py)

I think we can just remove the `.strip("- ")` and keep this character

## Describe the bug

The `multi_news.py` data loader has [a line which attempts to strip `"- "` from the beginning of summaries](https://github.com/huggingface/datasets/blob/aa743886221d76afb409d263e1b136e7a71fe2b4/datasets/multi_news/multi_news.py#L97). The actual character in the multi-news dataset, however, is `"– "`, which is different, e.g. `"– " != "- "`.

I would have just opened a PR to fix the mistake, but I am wondering what the motivation for stripping this character is? AFAICT most approaches just leave it in, e.g. the current SOTA on this dataset, [PRIMERA](https://huggingface.co/allenai/PRIMERA-multinews) (you can see its in the generated summaries of the model in their [example notebook](https://github.com/allenai/PRIMER/blob/main/Evaluation_Example.ipynb)).

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Adding eval metadata for ade v2

Adding metadata to allow evaluation


Don't check f.loc in _get_extraction_protocol_with_magic_number

`f.loc` doesn't always exist for file-like objects in python. I removed it since it was not necessary anyway (we always seek the file to 0 after reading the magic number)

Fix https://github.com/huggingface/datasets/issues/4310


Fix cnn_dailymail (dm stories were ignored)

https://github.com/huggingface/datasets/pull/4188 introduced a bug in `datasets` 2.2.0: DailyMail stories are ignored when generating the dataset.

I fixed that, and removed the google drive link (it has annoying quota limitations issues)

We can do a patch release after this is merged


Support passing config_kwargs to CLI run_beam

This PR supports passing `config_kwargs` to CLI run_beam, so that for example for "wikipedia" dataset, we can pass:
```
--date 20220501 --language ca
```


Fix CLI run_beam namespace

Currently, it raises TypeError:
```
TypeError: __init__() got an unexpected keyword argument 'namespace'
```


Catch pull error when mirroring

Catch pull errors when mirroring so that the script continues to update the other datasets.

The error will still be printed at the end of the job. In this case the job also fails, and asks to manually update the datasets that failed.


Add API code examples for Builder classes

documentation

Improvements or additions to documentation

This PR adds API code examples for the Builder classes.


added TR-News dataset


[Imagefolder] Docs + Don't infer labels from file names when there are metadata + Error messages when metadata and images aren't linked correctly

Merging this one since mario is off, I took care of adding some tests to make sure everything is fine. Will do the release after it

I updated the `docs/source/image_process.mdx` documentation and added an example for image captioning and object detection using `ImageFolder`.

While doing so I also improved a few aspects:
- we don't need to infer labels from file names when there are metadata - they can just be in the metadata if necessary
- raise informative error messages when metadata and images aren't linked correctly:
  - when an image is missing a metadata file
  - when a metadata file is missing an image

I added some tests for these changes as well

cc @mariosasko


Loading dataset with streaming: '_io.BufferedReader' object has no attribute 'loc'

bug

Something isn't working

## Describe the bug
Loading a datasets with `load_dataset` and `streaming=True` returns `AttributeError: '_io.BufferedReader' object has no attribute 'loc'`. Notice that loading with `streaming=False` works fine.

In the following steps we load parquet files but the same happens with pickle files. The problem seems to come from `fsspec` lib, I put in the environment info also `s3fs` and `fsspec` versions since I'm loading from an s3 bucket.

## Steps to reproduce the bug
```python
from datasets import load_dataset
# path is the path to parquet files
data_files = {"train": path + "meta_train.parquet.gzip", "test": path + "meta_test.parquet.gzip"}
dataset = load_dataset("parquet", data_files=data_files, streaming=True)
```

## Expected results
A dataset object `datasets.dataset_dict.DatasetDict`

## Actual results
```
AttributeError                            Traceback (most recent call last)
<command-562086> in <module>
     11
     12 data_files = {"train": path + "meta_train.parquet.gzip", "test": path + "meta_test.parquet.gzip"}
---> 13 dataset = load_dataset("parquet", data_files=data_files, streaming=True)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1679     if streaming:
   1680         extend_dataset_builder_for_streaming(builder_instance, use_auth_token=use_auth_token)
-> 1681         return builder_instance.as_streaming_dataset(
   1682             split=split,
   1683             use_auth_token=use_auth_token,

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/builder.py in as_streaming_dataset(self, split, base_path, use_auth_token)
    904         )
    905         self._check_manual_download(dl_manager)
--> 906         splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
    907         # By default, return all splits
    908         if split is None:

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/packaged_modules/parquet/parquet.py in _split_generators(self, dl_manager)
     30         if not self.config.data_files:
     31             raise ValueError(f"At least one data file must be specified, but got data_files={self.config.data_files}")
---> 32         data_files = dl_manager.download_and_extract(self.config.data_files)
     33         if isinstance(data_files, (str, list, tuple)):
     34             files = data_files

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in download_and_extract(self, url_or_urls)
    798
    799     def download_and_extract(self, url_or_urls):
--> 800         return self.extract(self.download(url_or_urls))
    801
    802     def iter_archive(self, urlpath_or_buf: Union[str, io.BufferedReader]) -> Iterable[Tuple]:

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in extract(self, path_or_paths)
    776
    777     def extract(self, path_or_paths):
--> 778         urlpaths = map_nested(self._extract, path_or_paths, map_tuple=True)
    779         return urlpaths
    780

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types, disable_tqdm, desc)
    312         num_proc = 1
    313     if num_proc <= 1 or len(iterable) <= num_proc:
--> 314         mapped = [
    315             _single_map_nested((function, obj, types, None, True, None))
    316             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in <listcomp>(.0)
    313     if num_proc <= 1 or len(iterable) <= num_proc:
    314         mapped = [
--> 315             _single_map_nested((function, obj, types, None, True, None))
    316             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)
    317         ]

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in _single_map_nested(args)
    267         return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
    268     else:
--> 269         mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
    270         if isinstance(data_struct, list):
    271             return mapped

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in <listcomp>(.0)
    267         return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
    268     else:
--> 269         mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
    270         if isinstance(data_struct, list):
    271             return mapped

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/py_utils.py in _single_map_nested(args)
    249     # Singleton first to spare some computation
    250     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 251         return function(data_struct)
    252
    253     # Reduce logging to keep things readable in multiprocessing with tqdm

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _extract(self, urlpath)
    781     def _extract(self, urlpath: str) -> str:
    782         urlpath = str(urlpath)
--> 783         protocol = _get_extraction_protocol(urlpath, use_auth_token=self.download_config.use_auth_token)
    784         if protocol is None:
    785             # no extraction

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _get_extraction_protocol(urlpath, use_auth_token)
    371         urlpath, kwargs = urlpath, {}
    372     with fsspec.open(urlpath, **kwargs) as f:
--> 373         return _get_extraction_protocol_with_magic_number(f)
    374
    375

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/datasets/utils/streaming_download_manager.py in _get_extraction_protocol_with_magic_number(f)
    335 def _get_extraction_protocol_with_magic_number(f) -> Optional[str]:
    336     """read the magic number from a file-like object and return the compression protocol"""
--> 337     prev_loc = f.loc
    338     magic_number = f.read(MAGIC_NUMBER_MAX_LENGTH)
    339     f.seek(prev_loc)

/local_disk0/.ephemeral_nfs/envs/pythonEnv-a7e72260-221c-472b-85f4-bec801aee66d/lib/python3.8/site-packages/fsspec/implementations/local.py in __getattr__(self, item)
    337
    338     def __getattr__(self, item):
--> 339         return getattr(self.f, item)
    340
    341     def __enter__(self):

AttributeError: '_io.BufferedReader' object has no attribute 'loc'
```
## Environment info
- `datasets` version: 2.1.0
- Platform: Linux-5.4.0-1071-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.2
- `fsspec` version: 2021.08.1
- `s3fs` version: 2021.08.1


[WIP] Add TEDLIUM dataset

dataset request

Requesting to add a new dataset

```python
from datasets import load_dataset

ds = load_dataset('./datasets/tedlium', 'release1', cache_dir='/home/sanchitgandhi/cache')
```

```
Downloading and preparing dataset tedlium/release1 to /home/sanchitgandhi/cache/tedlium/release1/1.0.1/5a9fcb97b4b52d5a1c9dc7bde4b1d5994cd89c4a3425ea36c789bf6096fee4f0...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/sanchit_huggingface_co/datasets/src/datasets/load.py", line 1703, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/sanchit_huggingface_co/datasets/src/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/home/sanchit_huggingface_co/datasets/src/datasets/builder.py", line 1240, in _download_and_prepare
    raise MissingBeamOptions(
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
        `load_dataset('tedlium', 'release1', beam_runner='DirectRunner')`
```
Specifying the `beam_runner='DirectRunner'` works:
```python
from datasets import load_dataset

ds = load_dataset('./datasets/tedlium', 'release1', cache_dir='/home/sanchitgandhi/cache', beam_runner='DirectRunner')
```

Extra Python imports/Linux packages:
```
pip install pydub
sudo apt install ffmpeg
```

Script heavily inspired by the TF datasets script at: https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/tedlium.py

The TF datasets script uses the module AudioSegment from the package `pydub` (https://github.com/jiaaro/pydub), which is used to to open the audio files (stored in .sph format):
https://github.com/huggingface/datasets/blob/61bf6123634bf6e7c7287cd6097909eb26118c58/datasets/tedlium/tedlium.py#L167-L170
This package requires the pip install of `pydub` and the system installation of `ffmpeg`: https://github.com/jiaaro/pydub#installation
Is it ok to use these packages? Or do we tend to avoid introducing additional dependencies?

The TF datasets script also uses `_build_pcollection`:
https://github.com/huggingface/datasets/blob/8afbbb6fe66b40d05574e2e72e65e974c72ae769/datasets/tedlium/tedlium.py#L200-L206
However, I was advised against using `beam` logic. Thus, I have reverted to generating the examples file-by-file: https://github.com/huggingface/datasets/blob/61bf6123634bf6e7c7287cd6097909eb26118c58/datasets/tedlium/tedlium.py#L112-L138

I am now able to generate examples by running the `load_dataset` command:
```python
from datasets import load_dataset

ds = load_dataset('./datasets/tedlium', 'release1', cache_dir='/home/sanchitgandhi/cache')
```

Here, generating examples is **extremely** slow: it takes ~1 second per example, so ~60k seconds for the train set (~16 hours). Is there a way of paralleling this to make it faster?

> This package requires the pip install of pydub and the system installation of ffmpeg: https://github.com/jiaaro/pydub#installation
Is it ok to use these packages? Or do we tend to avoid introducing additional dependencies?

It's ok, windows users will have have a bad time but I'm not sure we can do much about it.

> Here, generating examples is extremely slow: it takes ~1 second per example, so ~60k seconds for the train set (~16 hours). Is there a way of paralleling this to make it faster?

Not at the moment. For such cases we advise hosting the dataset ourselves in a processed format. The license doesn't allow this since the license is "NoDerivatives". Currently the only way to parallelize it is by keeping is as a beam dataset and let users pay Google Dataflow to process it (or use spark or whatever).

Thanks for your super speedy reply @lhoestq!

I’ve uploaded the script and README.md to the org here: https://huggingface.co/datasets/LIUM/tedlium
Is any modification of the script required to be able to use it from the Hub? When I run:
```python
from datasets import load_dataset

tedlium = load_dataset("LIUM/tedlium", "release1") # for Release 1
```
I get the following error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 load_dataset("LIUM/tedlium", "release1")

File ~/datasets/src/datasets/load.py:1676, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1673 ignore_verifications = ignore_verifications or save_infos
   1675 # Create a dataset builder
-> 1676 builder_instance = load_dataset_builder(
   1677     path=path,
   1678     name=name,
   1679     data_dir=data_dir,
   1680     data_files=data_files,
   1681     cache_dir=cache_dir,
   1682     features=features,
   1683     download_config=download_config,
   1684     download_mode=download_mode,
   1685     revision=revision,
   1686     use_auth_token=use_auth_token,
   1687     **config_kwargs,
   1688 )
   1690 # Return iterable dataset in case of streaming
   1691 if streaming:

File ~/datasets/src/datasets/load.py:1502, in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, **config_kwargs)
   1500     download_config = download_config.copy() if download_config else DownloadConfig()
   1501     download_config.use_auth_token = use_auth_token
-> 1502 dataset_module = dataset_module_factory(
   1503     path,
   1504     revision=revision,
   1505     download_config=download_config,
   1506     download_mode=download_mode,
   1507     data_dir=data_dir,
   1508     data_files=data_files,
   1509 )
   1511 # Get dataset builder class from the processing script
   1512 builder_cls = import_main_class(dataset_module.module_path)

File ~/datasets/src/datasets/load.py:1254, in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1249             if isinstance(e1, FileNotFoundError):
   1250                 raise FileNotFoundError(
   1251                     f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1252                     f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1253                 ) from None
-> 1254             raise e1 from None
   1255 else:
   1256     raise FileNotFoundError(
   1257         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory."
   1258     )

File ~/datasets/src/datasets/load.py:1227, in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1225         raise e
   1226 if filename in [sibling.rfilename for sibling in dataset_info.siblings]:
-> 1227     return HubDatasetModuleFactoryWithScript(
   1228         path,
   1229         revision=revision,
   1230         download_config=download_config,
   1231         download_mode=download_mode,
   1232         dynamic_modules_path=dynamic_modules_path,
   1233     ).get_module()
   1234 else:
   1235     return HubDatasetModuleFactoryWithoutScript(
   1236         path,
   1237         revision=revision,
   (...)
   1241         download_mode=download_mode,
   1242     ).get_module()

File ~/datasets/src/datasets/load.py:940, in HubDatasetModuleFactoryWithScript.get_module(self)
    938 def get_module(self) -> DatasetModule:
    939     # get script and other files
--> 940     local_path = self.download_loading_script()
    941     dataset_infos_path = self.download_dataset_infos_file()
    942     imports = get_imports(local_path)

File ~/datasets/src/datasets/load.py:918, in HubDatasetModuleFactoryWithScript.download_loading_script(self)
    917 def download_loading_script(self) -> str:
--> 918     file_path = hf_hub_url(path=self.name, name=self.name.split("/")[1] + ".py", revision=self.revision)
    919     download_config = self.download_config.copy()
    920     if download_config.download_desc is None:

TypeError: hf_hub_url() got an unexpected keyword argument 'name'
```

Note that I am able to load the dataset from the `datasets` repo with the following lines of code:
```python
from datasets import load_dataset

ds = load_dataset('./datasets/tedlium', 'release1', cache_dir='/home/sanchitgandhi/cache')
```

What version of `datasets` do you have ?
Updating `datasets` should fix the error ;)

> This package requires the pip install of pydub and the system installation of ffmpeg: https://github.com/jiaaro/pydub#installation
Is it ok to use these packages? Or do we tend to avoid introducing additional dependencies?

`soundfile`, which is a required audio dependency, should also work with `.sph` files, no?

> `soundfile`, which is a required audio dependency, should also work with `.sph` files, no?

Awesome, thanks for the pointer @mariosasko! Switched `pydub` to `soundfile`, and having specifying the `dtype` argument in `soundfile.read` as `np.int16`, the arrays match with those from `pydub` ✅

I also did some heavy optimising of the script with the processing of the `.stm` and `.sph` files - it now runs 2000x faster than before, so there probably isn't a need to upload the data to the Hub @lhoestq. The total processing time is just ~2mins now 🚀

TEDLIUM completed and uploaded to the HF Hub: https://huggingface.co/datasets/LIUM/tedlium

Adds the TED-LIUM dataset https://www.tensorflow.org/datasets/catalog/tedlium#tedliumrelease3

TODO:

- [x] Port `tedium.py` from TF datasets using `convert_dataset.sh` script
- [x] Make `load_dataset` work
- [ ] ~~Run `datasets-cli` command to generate `dataset_infos.json`~~
- [ ] ~~Create dummy data for continuous testing~~
- [ ] ~~Dummy data tests~~
- [ ] ~~Real data tests~~
- [ ] Create the metadata JSON
- [ ] Close PR and add directly to the Hub under LIUM org


Remove unused multiprocessing args from test CLI

Multiprocessing is not used in the test CLI.


Add packaged builder configs to the documentation

Add the packaged builders configurations to the docs reference is useful to show the list of all parameters one can use when loading data in many formats: CSV, JSON, etc.


`load_dataset` does not work with certain filename.

bug

Something isn't working

Never mind. It is because of the caching of datasets...

## Describe the bug
This is a weird bug that took me some time to find out.

I have a JSON dataset that I want to load with `load_dataset` like this:

```
data_files = dict(train="train.json.zip", val="val.json.zip")
dataset = load_dataset("json", data_files=data_files, field="data")
```

## Expected results
No error.

## Actual results
The val file is loaded as expected, but the train file throws JSON decoding error:

```
╭──────────────────────────── Traceback (most recent call last) ────────────────────────────╮
│ <ipython-input-74-97947e92c100>:5 in <module>                                             │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/site-packages/datasets/load.py:1687 in    │
│ load_dataset                                                                              │
│                                                                                │
│   1684 │   try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES                       │
│   1685 │                                                                                │
│   1686 │   # Download and prepare data                                                    │
│ ❱ 1687 │   builder_instance.download_and_prepare(                                         │
│   1688 │   │   download_config=download_config,                                           │
│   1689 │   │   download_mode=download_mode,                                               │
│   1690 │   │   ignore_verifications=ignore_verifications,                                 │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/site-packages/datasets/builder.py:605 in  │
│ download_and_prepare                                                                      │
│                                                                                │
│    602 │   │   │   │   │   │   except ConnectionError:                                    │
│    603 │   │   │   │   │   │   │   logger.warning("HF google storage unreachable. Downloa │
│    604 │   │   │   │   │   if not downloaded_from_gcs:                                    │
│ ❱  605 │   │   │   │   │   │   self._download_and_prepare(                                │
│    606 │   │   │   │   │   │   │   dl_manager=dl_manager, verify_infos=verify_infos, **do │
│    607 │   │   │   │   │   │   )                                                          │
│    608 │   │   │   │   │   # Sync info                                                    │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/site-packages/datasets/builder.py:694 in  │
│ _download_and_prepare                                                                     │
│                                                                                │
│    691 │   │   │                                                                          │
│    692 │   │   │   try:                                                                   │
│    693 │   │   │   │   # Prepare split will record examples associated to the split       │
│ ❱  694 │   │   │   │   self._prepare_split(split_generator, **prepare_split_kwargs)       │
│    695 │   │   │   except OSError as e:                                                   │
│    696 │   │   │   │   raise OSError(                                                     │
│    697 │   │   │   │   │   "Cannot find data file. "                                      │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/site-packages/datasets/builder.py:1151 in │
│ _prepare_split                                                                            │
│                                                                                │
│   1148 │   │                                                                              │
│   1149 │   │   generator = self._generate_tables(**split_generator.gen_kwargs)            │
│   1150 │   │   with ArrowWriter(features=self.info.features, path=fpath) as writer:       │
│ ❱ 1151 │   │   │   for key, table in logging.tqdm(                                        │
│   1152 │   │   │   │   generator, unit=" tables", leave=False, disable=True  # not loggin │
│   1153 │   │   │   ):                                                                     │
│   1154 │   │   │   │   writer.write_table(table)                                          │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/site-packages/tqdm/notebook.py:257 in     │
│ __iter__                                                                                │
│                                                                                │
│   254 │                                                                                │
│   255 │   def __iter__(self):                                                             │
│   256 │   │   try:                                                                        │
│ ❱ 257 │   │   │   for obj in super(tqdm_notebook, self).__iter__():                       │
│   258 │   │   │   │   # return super(tqdm...) will not catch exception                    │
│   259 │   │   │   │   yield obj                                                           │
│   260 │   │   # NB: except ... [ as ...] breaks IPython async KeyboardInterrupt           │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/site-packages/tqdm/std.py:1183 in         │
│ __iter__                                                                                │
│                                                                                │
│   1180 │   │   # If the bar is disabled, then just walk the iterable                      │
│   1181 │   │   # (note: keep this check outside the loop for performance)                 │
│   1182 │   │   if self.disable:                                                           │
│ ❱ 1183 │   │   │   for obj in iterable:                                                   │
│   1184 │   │   │   │   yield obj                                                          │
│   1185 │   │   │   return                                                                 │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/site-packages/datasets/packaged_modules/j │
│ son/json.py:90 in _generate_tables                                                        │
│                                                                                │
│    87 │   │   │   # If the file is one json object and if we need to look at the list of  │
│    88 │   │   │   if self.config.field is not None:                                       │
│    89 │   │   │   │   with open(file, encoding="utf-8") as f:                             │
│ ❱  90 │   │   │   │   │   dataset = json.load(f)                                          │
│    91 │   │   │   │                                                                       │
│    92 │   │   │   │   # We keep only the field we are interested in                       │
│    93 │   │   │   │   dataset = dataset[self.config.field]                                │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/json/__init__.py:293 in load              │
│                                                                                │
│   290 │   To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``           │
│   291 │   kwarg; otherwise ``JSONDecoder`` is used.                                       │
│   292 │   """                                                                             │
│ ❱ 293 │   return loads(fp.read(),                                                         │
│   294 │   │   cls=cls, object_hook=object_hook,                                           │
│   295 │   │   parse_float=parse_float, parse_int=parse_int,                               │
│   296 │   │   parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)   │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/json/__init__.py:357 in loads             │
│                                                                                │
│   354 │   if (cls is None and object_hook is None and                                     │
│   355 │   │   │   parse_int is None and parse_float is None and                           │
│   356 │   │   │   parse_constant is None and object_pairs_hook is None and not kw):       │
│ ❱ 357 │   │   return _default_decoder.decode(s)                                           │
│   358 │   if cls is None:                                                                 │
│   359 │   │   cls = JSONDecoder                                                           │
│   360 │   if object_hook is not None:                                                     │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/json/decoder.py:337 in decode             │
│                                                                                │
│   334 │   │   containing a JSON document).                                                │
│   335 │   │                                                                               │
│   336 │   │   """                                                                         │
│ ❱ 337 │   │   obj, end = self.raw_decode(s, idx=_w(s, 0).end())                           │
│   338 │   │   end = _w(s, end).end()                                                      │
│   339 │   │   if end != len(s):                                                           │
│   340 │   │   │   raise JSONDecodeError("Extra data", s, end)                             │
│                                                                                │
│ /home/tiankang/software/anaconda3/lib/python3.8/json/decoder.py:353 in raw_decode         │
│                                                                                │
│   350 │   │                                                                               │
│   351 │   │   """                                                                         │
│   352 │   │   try:                                                                        │
│ ❱ 353 │   │   │   obj, end = self.scan_once(s, idx)                                       │
│   354 │   │   except StopIteration as err:                                                │
│   355 │   │   │   raise JSONDecodeError("Expecting value", s, err.value) from None        │
│   356 │   │   return obj, end                                                             │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
JSONDecodeError: Unterminated string starting at: line 85 column 20 (char 60051)
```

However, when I rename the `train.json.zip` to other names (like `training.json.zip`, or even to `train.json`), everything works fine; when I unzip the file to `train.json`, it works as well.

## Environment info
```
- `datasets` version: 2.1.0
- Platform: Linux-4.4.0-131-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 7.0.0
- Pandas version: 1.4.2
```


Fixes FrugalScore

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4305). All of your documentation changes will be reflected on that endpoint.

> predictions and references are swapped. Basically Frugalscore is commutative, however some tiny differences can occur if we swap the references and the predictions. I decided to swap them just to obtain the exact results as reported in the paper.

What is the order of magnitude of the difference ? Do you know what causes this ?

> I switched to dynamic padding that was was used in the training, forcing the padding to max_length introduces errors for some reason that I ignore.

What error ?

There are two minor modifications in this PR:
1) `predictions` and `references` are swapped. Basically Frugalscore is commutative, however some tiny differences can occur if we swap the references and the predictions. I decided to swap them just to obtain the exact results as reported in the paper.
2) I switched to dynamic padding that was was used in the training, forcing the padding to `max_length`  introduces errors for some reason that I ignore.

@lhoestq


Language code search does direct matches

Something isn't working

Thanks for reporting ! I forwarded the issue to the front-end team :)

Will keep you posted !

I also changed the tagging app to suggest two letters code for now.

## Describe the bug

Hi. Searching for bcp47 tags that are just the language prefix (e.g. `sq` or `da`) excludes datasets that have added extra information in their language metadata (e.g. `sq-AL` or `da-bornholm`). The example codes given in the [tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging) encourages addition of the additional codes ("_expected format is BCP47 tags separated for ';' e.g. 'en-US;fr-FR'_") but this would lead to those datasets being hidden in datasets search.

## Steps to reproduce the bug
1. Add a dataset using a variant tag (e.g. [`sq-AL`](https://huggingface.co/datasets?languages=languages:sq-AL))
2. Look for datasets using the full code
3. Note that they're missing when just the language is searched for (e.g. [`sq`](https://huggingface.co/datasets?languages=languages:sq))

Some datasets are already affected by this - e.g. `AmazonScience/massive` is listed under `sq-AL` but not `sq`.

One workaround is for dataset creators to add an additional root language tag to dataset YAML metadata, but it's unclear how to communicate this. It might be possible to index the search on `languagecode.split('-')[0]` but I wanted to float this issue before trying to write any code :)

## Expected results
Datasets using longer bcp47 tags also appear under searches for just the language code; e.g. Quebecois datasets (`fr-CA`) would come up when looking for French datasets with no region specification (`fr`), or US English (`en-US`) datasets would come up when searching for English datasets (`en`).

## Actual results
The language codes seem to be directly string matched, excluding datasets with specific language tags from non-specific searches.

## Environment info
(web app)


Fix: Add missing comma

The CI failure is unrelated to this PR and fixed on master, merging :)


Remove hacking license tags when mirroring datasets on the Hub

_The documentation is not available anymore as the PR was closed or merged._

The Hub doesn't allow these characters in the YAML tags, and git push fails if you want to push a dataset card containing these characters.

Ok, let me rename the bad config names :) I think I can also keep backward compatibility with a warning

Almost done with it btw, will submit a PR that shows all the configuration name changes (from a bit more than 20 datasets)

Please, let me know when the renaming of configs is done. If not enough bandwidth, I can take care of it...

I realized when renaming all the configurations with dots in https://github.com/huggingface/datasets/pull/4365 that it's not ideal for certain cases. For example:
- many configurations have a version like "1.0.0" in their names
- to avoid breaking changes we need to replace dots with underscores in the user input and show a warning, which hurts the experience
- our second most downloaded dataset at the moment is affected: `newsgroup`
- if we disallow dots, then we'll never be able to make the [allenai/c4](https://huggingface.co/datasets/allenai/c4) work with its different configurations since they contain dots, and we can't rename them because they are the official download links

I was thinking of other alternatives:
1. just stop separating tags per config name completely, and have a single flat YAML for all configurations. Dataset search doesn't use this info anyway
2. use another YAML structure to avoid having config names as keys, such as
```yaml
languages:
- config: 20220301_en
  values:
  - en
```

I'm down for 1, to keep things simple

@lhoestq I agree:
- better not changing config names (so that we do not introduce any braking change)
- therefore, we should not use them as keys

In relation with the proposed solutions, I have no strong opinion:
- option 1 is simpler and aligns better with current usage on the Hub (configs are ignored)
- however:
  - we will lose all the information per config we already have (for those datasets containing config keys; contributors made an effort to put that information per config)
  - and this information might be useful on the Hub in the future, in case we would like to enrich the search feature with more granularity; this is only applicable if this feature could eventually make sense

So, no strong opinion...

Closing in favor of https://github.com/huggingface/datasets/pull/4367

Currently, when mirroring datasets on the Hub, the license tags are hacked: removed of characters "." and "$". On the contrary, this hacking is not applied to community datasets on the Hub. This generates multiple variants of the same tag on the Hub.

I guess this hacking is no longer necessary:
- it is not applied to community datasets
- all canonical datasets are validated by maintainers before being merged: CI + maintainers make sure license tags are the right ones

Fix #4298.


Add ImageNet-Sketch dataset

I think you can go ahead with uploading the data, and also ping the author in parallel. I think the images may subject to copyright anyway (scrapped from google image) so the dataset author is not allowed to set a license to the data.

I think it's fine to upload the dataset as soon as we mention explicitly that the images may be subject to copyright.

This PR adds the ImageNet-Sketch dataset and resolves #3953 .


Add API code examples for loading methods

This PR adds API code examples for loading methods, let me know if I've missed any important parameters we should showcase :)

I was a bit confused about `inspect_dataset` and `inspect_metric`. The `path` parameter says it will accept a dataset identifier from the Hub. But when I try the identifier `rotten_tomatoes`, it gives me:

```py
from datasets import inspect_dataset
inspect_dataset('rotten_tomatoes', local_path='/content/rotten_tomatoes')

FileNotFoundError: Couldn't find a dataset script at /content/rotten_tomatoes/rotten_tomatoes.py or any data file in the same directory.
```

Does the user need to have an existing copy of `rotten_tomatoes.py` on their local drive (in which case, it seems like the same option as the first option in `path`)?


Remove manual download from imagenet-1k

Thanks for the reviews @apsdehal and @lhoestq! As suggested by @lhoestq, I'll separate the train/val/test splits, apply the validation split fixes and shuffle the images files to simplify the script and make streaming faster.

@apsdehal I dismissed your review as it's no longer relevant after the data files changes suggested by @lhoestq.

Remove the manual download code from `imagenet-1k` to make it a regular dataset.


Normalise license names

enhancement

New feature or request

we'll add the same server-side metadata validation system as for hf.co/models soon-ish

(you can check on hf.co/models that licenses are "clean")

**Is your feature request related to a problem? Please describe.**
When browsing datasets, the Licenses tag cloud (bottom left of e.g. https://huggingface.co/datasets) has multiple variants of the same license. This means the options exclude datasets arbitrarily, giving users artificially low recall. The cause of the dupes is probably due to a bit of variation in metadata.

**Describe the solution you'd like**
I'd like the licenses in metadata to follow the same standard as much as possible, to remove this problem. I'd like to go ahead and normalise the dataset metadata to follow the format & values given in [src/datasets/utils/resources/licenses.json](https://github.com/huggingface/datasets/blob/master/src/datasets/utils/resources/licenses.json) .

**Describe alternatives you've considered**
None

**Additional context**
None

**Priority**
Low


Datasets YAML tagging space is down

bug

Something isn't working

@lhoestq @albertvillanova `update-task-list` branch does not exist anymore, should point to `main` now i guess

## Describe the bug
The neat hf spaces app for generating YAML tags for dataset `README.md`s is down

## Steps to reproduce the bug
1. Visit https://huggingface.co/spaces/huggingface/datasets-tagging

## Expected results
There'll be a HF spaces web app for generating dataset metadata YAML

## Actual results
There's an error message; here's the step where it breaks:

```
Step 18/29 : RUN pip install -r requirements.txt
 ---> Running in e88bfe7e7e0c
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/huggingface/datasets.git@update-task-list (from -r requirements.txt (line 4))
  Cloning https://github.com/huggingface/datasets.git (to revision update-task-list) to /tmp/pip-req-build-bm8t0r0k
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/datasets.git /tmp/pip-req-build-bm8t0r0k
  WARNING: Did not find branch or tag 'update-task-list', assuming revision or ref.
  Running command git checkout -q update-task-list
  error: pathspec 'update-task-list' did not match any file(s) known to git
  error: subprocess-exited-with-error

  × git checkout -q update-task-list did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git checkout -q update-task-list did not run successfully.
│ exit code: 1
╰─> See above for output.
```

## Environment info

- Platform: Linux / Brave


Fix URL query parameters in compression hop path when streaming

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4296). All of your documentation changes will be reflected on that endpoint.

Fix #3488.


Fix missing lz4 dependency for tests

Currently, `lz4` is not defined as a dependency for tests. Therefore, all tests marked with `@require_lz4` are skipped.


Fix CLI run_beam save_infos

Currently, it raises TypeError:
```
TypeError: _download_and_prepare() got an unexpected keyword argument 'save_infos'
```


Fix wrong map parameter name in cache docs

The `load_from_cache` parameter of `map` should be `load_from_cache_file`.


Add API code examples for remaining main classes

This PR adds API code examples for the remaining functions in the Main classes. I wasn't too familiar with some of the functions (`decode_batch`, `decode_column`, `decode_example`, etc.) so please feel free to add an example of usage and I can fill in the rest :)


Dataset Viewer issue for strombergnlp/ipm_nel : preview is empty, no error message

Hi @leondz, thanks for reporting.

Indeed, the dataset viewer relies on the dataset being streamable (passing `streaming=True` to `load_dataset`). Whereas most of the datastes are streamable out of the box (thanks to our implementation of streaming), there are still some exceptions.

In particular, in your case, that is due to the data file being TAR. This format is not streamable out of the box (it does not allow random access to the archived files), but we use a trick to allow streaming: using `dl_manager.iter_archive`.

Let me know if you need some help: I could push a commit to your repo with the fix.

Ah, right! The preview is working now, but this explanation is good to know, thank you. I'll prefer formats with random file access supported in datasets.utils.extract in future, and try out this fix for the tarfiles :)

### Link

https://huggingface.co/datasets/strombergnlp/ipm_nel/viewer/ipm_nel/train

### Description

The viewer is blank. I tried my best to emulate a dataset with a working viewer, but this one just doesn't seem to want to come up. What did I miss?

### Owner

Yes


Update README.md

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4290). All of your documentation changes will be reflected on that endpoint.

Updating readme in medmcqa dataset.


Add missing `faiss` import to fix https://github.com/huggingface/datasets/issues/4287

This PR fixes the issue recently mentioned in https://github.com/huggingface/datasets/issues/4287 🤗


"NameError: name 'faiss' is not defined" on `.add_faiss_index` when `device` is not None

Something isn't working

So I managed to solve this by adding a missing `import faiss` in the `@staticmethod` defined in https://github.com/huggingface/datasets/blob/f51b6994db27ea69261ef919fb7775928f9ec10b/src/datasets/search.py#L305, triggered from https://github.com/huggingface/datasets/blob/f51b6994db27ea69261ef919fb7775928f9ec10b/src/datasets/search.py#L249 when trying to `ds_with_embeddings.add_faiss_index(column='embeddings', device=0)` with the code above.

As it seems that the `@staticmethod` doesn't recognize the `import faiss` defined in https://github.com/huggingface/datasets/blob/f51b6994db27ea69261ef919fb7775928f9ec10b/src/datasets/search.py#L261, so whenever the value of `device` is not None in https://github.com/huggingface/datasets/blob/71f76e0bdeaddadedc4f9c8d15cfff5a36d62f66/src/datasets/search.py#L438, that exception is triggered.

So on, adding `import faiss` inside https://github.com/huggingface/datasets/blob/71f76e0bdeaddadedc4f9c8d15cfff5a36d62f66/src/datasets/search.py#L305 right after the check of `device`'s value, solves the issue and lets you calculate the indices in GPU.

I'll add the code in a PR linked to this issue in case you want to merge it!

Adding here the complete error traceback!

```
Traceback (most recent call last):
  File "/home/alvarobartt/lol.py", line 12, in <module>
    ds_with_embeddings.add_faiss_index(column='embeddings', device=0) # default `device=None`
  File "/home/alvarobartt/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3656, in add_faiss_index
    super().add_faiss_index(
  File "/home/alvarobartt/.local/lib/python3.9/site-packages/datasets/search.py", line 478, in add_faiss_index
    faiss_index.add_vectors(self, column=column, train_size=train_size, faiss_verbose=True)
  File "/home/alvarobartt/.local/lib/python3.9/site-packages/datasets/search.py", line 281, in add_vectors
    self.faiss_index = self._faiss_index_to_device(index, self.device)
  File "/home/alvarobartt/.local/lib/python3.9/site-packages/datasets/search.py", line 327, in _faiss_index_to_device
    faiss_res = faiss.StandardGpuResources()
NameError: name 'faiss' is not defined
```

Closed as https://github.com/huggingface/datasets/pull/4288 already merged! :hugs:

## Describe the bug

When using `datasets` to calculate the FAISS indices of a dataset, the exception `NameError: name 'faiss' is not defined` is triggered when trying to calculate those on a device (GPU), so `.add_faiss_index(..., device=0)` fails with that exception.

All that assuming that `datasets` is properly installed and `faiss-gpu` too, as well as all the CUDA drivers required.

## Steps to reproduce the bug

```python
# Sample code to reproduce the bug
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
import torch
torch.set_grad_enabled(False)
ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

from datasets import load_dataset
ds = load_dataset('crime_and_punish', split='train[:100]')
ds_with_embeddings = ds.map(lambda example: {'embeddings': ctx_encoder(**ctx_tokenizer(example["line"], return_tensors="pt"))[0][0].numpy()})

ds_with_embeddings.add_faiss_index(column='embeddings', device=0) # default `device=None`
```

## Expected results

A new column named `embeddings` in the dataset that we're adding the index to.

## Actual results

An exception is triggered with the following message `NameError: name 'faiss' is not defined`.

## Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.13.0-1022-azure-x86_64-with-glibc2.31
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.2


Add Lahnda language tag

_The documentation is not available anymore as the PR was closed or merged._

This language is present in [Wikimedia's WIT](https://huggingface.co/datasets/wikimedia/wit_base) dataset.


Update LexGLUE README.md

Update the leaderboard based on the latest results presented in the ACL 2022 version of the article.


Issues in processing very large datasets

Something isn't working

Hi ! `datasets` doesn't load the dataset in memory. Instead it uses memory mapping to load your dataset from your disk (it is stored as arrow files). Do you know at what point you have RAM issues exactly ?

How big are your graph_data_train dictionaries btw ?

## Describe the bug
I'm trying to add a feature called "subgraph" to CNN/DM dataset (modifications on run_summarization.py of Huggingface Transformers script) --- I'm not quite sure if I'm doing it the right way, though--- but the main problem appears when the training starts where the error ` [OSError: [Errno 12] Cannot allocate memory]`  appears. I suppose this problem roots in RAM issues and how the dataset is loaded during training, but I have no clue of what I can do to fix it.  Observing the dataset's cache directory, I see that it takes ~600GB of memory and that's why I believe special care is needed when loading it into the memory.

Here are my modifications to `run_summarization.py` code.

```
# loading pre-computed dictionary where keys are 'id' of article and values are corresponding subgraph
graph_data_train = get_graph_data('train')
graph_data_validation = get_graph_data('val')
...
...

with training_args.main_process_first(desc="train dataset map pre-processing"):
    train_dataset = train_dataset.map(
        preprocess_function_train,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not data_args.overwrite_cache,
        desc="Running tokenizer on train dataset",
    )

```

And here is the modified preprocessed function:

```
def preprocess_function_train(examples):
        inputs, targets, sub_graphs, ids = [], [], [], []
        for i in range(len(examples[text_column])):
            if examples[text_column][i] is not None and examples[summary_column][i] is not None:
                # if examples['doc_id'][i] in graph_data.keys():
                inputs.append(examples[text_column][i])
                targets.append(examples[summary_column][i])
                sub_graphs.append(graph_data_train[examples['id'][i]])
                ids.append(examples['id'][i])

        inputs = [prefix + inp for inp in inputs]
        model_inputs = tokenizer(inputs, max_length=data_args.max_source_length, padding=padding, truncation=True,
                                 sub_graphs=sub_graphs, ids=ids)

            # Setup the tokenizer for targets
        with tokenizer.as_target_tokenizer():
            labels = tokenizer(targets, max_length=max_target_length, padding=padding, truncation=True)

        # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
        # padding in the loss.
        if padding == "max_length" and data_args.ignore_pad_token_for_loss:
            labels["input_ids"] = [
                [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
            ]

        model_inputs["labels"] = labels["input_ids"]
        return model_inputs
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:  2.1.0
- Platform: Linux Ubuntu
- Python version: 3.6
- PyArrow version: 6.0.1


Fix filesystem docstring

This PR untangles the `S3FileSystem` docstring so the [parameters](https://huggingface.co/docs/datasets/master/en/package_reference/main_classes#parameters) are properly displayed.


Don't do unnecessary list type casting to avoid replacing None values by empty lists

Quick question about the message in the warning. You say "will be fixed in a future major version" but don't you mean "will raise an error in a future major version"?

Right ! Good catch, thanks, I updated the message to say "will raise an error in a future major version"

In certain cases, `None` values are replaced by empty lists when casting feature types.

It happens every time you cast an array of nested lists like [None, [0, 1, 2, 3]] to a different type (to change the integer precision for example). In this case you'd get [[], [0, 1, 2, 3]] for example. This issue comes from PyArrow, see the discussion in https://github.com/huggingface/datasets/issues/3676

This issue also happens when no type casting is needed, because casting is supposed to be a no-op in this case. But as https://github.com/huggingface/datasets/issues/3676 shown, it's not the case and `None` are replaced by empty lists even if we cast to the exact same type.

In this PR I just workaround this bug in the case where no type casting is needed. In particular, I only call `pa.ListArray.from_arrays` only when necessary.

I also added a warning when some `None` are effectively replaced by empty lists. I wanted to raise an error in this case, but maybe we should wait a major update to do so

This PR fixes this particular case, that is occurring in `run_qa.py` in `transformers`:
```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(4)})
ds = ds.map(lambda x: {"b": [[None, [0]]]}, batched=True, batch_size=1, remove_columns=["a"])
print(ds.to_pandas())
# before:
#              b
# 0  [None, [0]]
# 1    [[], [0]]
# 2    [[], [0]]
# 3    [[], [0]]
#
# now:
#              b
# 0  [None, [0]]
# 1  [None, [0]]
# 2  [None, [0]]
# 3  [None, [0]]
```

cc @sgugger


Remove a copy-paste sentence in dataset cards

The non-passing tests have nothing to do with this PR.

Remove the following copy-paste sentence from dataset cards:
```
We show detailed information for up to 5 configurations of the dataset.
```


Add missing features to commonsense_qa dataset

@albertvillanova it adds question_concept and id which is great. I suppose we'll talk about staying true to the format on another PR.

Yes, let's merge this PR as it is: it adds missing features.

A subsequent PR may address the request on changing the dataset feature structure.

Fix partially #4275.


Update minimal PyArrow version warning

Update the minimal PyArrow version warning (should've been part of #4250).


Add missing features to openbookqa dataset for additional config

Let's merge this PR as it is: it adds missing features.

A subsequent PR may address the request on changing the data feature structure.

Fix partially #4276.


Enable label alignment for token classification datasets

Hmm, not sure why the Windows tests are failing with:

```
Did not find path entry C:\tools\miniconda3\bin
C:\tools\miniconda3\envs\py37\python.exe: No module named pytest
```

Edit: running the CI again fixed the problem 🙃

> One last nit and we can merge then

Thanks, done!

This PR extends the `Dataset.align_labels_with_mapping()` method to support alignment of label mappings between datasets and models for token classification (e.g. NER).

Example of usage:

```python
from datasets import load_dataset

ner_ds = load_dataset("conll2003", split="train")
# returns [3, 0, 7, 0, 0, 0, 7, 0, 0]
ner_ds[0]["ner_tags"]
# hypothetical model mapping with O <--> B-LOC
label2id = {
    "B-LOC": "0",
    "B-MISC": "7",
    "B-ORG": "3",
    "B-PER": "1",
    "I-LOC": "6",
    "I-MISC": "8",
    "I-ORG": "4",
    "I-PER": "2",
    "O": "5"
  }
ner_aligned_ds = ner_ds.align_labels_with_mapping(label2id, "ner_tags")
# returns [3, 5, 7, 5, 5, 5, 7, 5, 5]
ner_aligned_ds[0]["ner_tags"]
```

Context: we need this in AutoTrain to automatically align datasets / models during evaluation. cc @abhishekkrthakur


OpenBookQA has missing and inconsistent field names

dataset bug

A bug in a dataset script provided in the library

Thanks for reporting, @vblagoje.

Indeed, I noticed some of these issues while reviewing this PR:
- #4259

This is in my TODO list.

Ok, awesome @albertvillanova How about #4275 ?

On the other hand, I am not sure if we should always preserve the original nested structure. I think we should also consider other factors as convenience or consistency.

For example, other datasets also flatten "question.stem" into "question":
- ai2_arc:
  ```python
  question = data["question"]["stem"]
  choices = data["question"]["choices"]
  text_choices = [choice["text"] for choice in choices]
  label_choices = [choice["label"] for choice in choices]
  yield id_, {
      "id": id_,
      "answerKey": answerkey,
      "question": question,
      "choices": {"text": text_choices, "label": label_choices},
  }
  ```
- commonsense_qa:
  ```python
  question = data["question"]
  stem = question["stem"]
  yield id_, {
      "answerKey": answerkey,
      "question": stem,
      "choices": {"label": labels, "text": texts},
  }
  ```
- cos_e:
  ```python
  "question": cqa["question"]["stem"],
  ```
- qasc
- quartz
- wiqa

Exceptions:
- exams

I think we should agree on a CONVENIENT format for QA and use always CONSISTENTLY the same.

@albertvillanova I agree that we should be consistent. In the last month, I have come across tons of code that deals with OpenBookQA and CommonSenseQA and all of that code relies on the original data format structure. We can't expect users to adopt HF Datasets if we arbitrarily change the structure of the format just because we think something makes more sense. I am in that position now (downloading original data rather than using HF Datasets) and undoubtedly it hinders HF Datasets' widespread use and adoption. Missing fields like in the case of #4275 is definitely bad and not even up for a discussion IMHO! cc @lhoestq

I'm opening a PR that adds the missing fields.

Let's agree on the feature structure: @lhoestq @mariosasko @polinaeterna

IMO we should always try to preserve the original structure unless there is a good reason not to (and I don't see one in this case).

I agree with @mariosasko . The transition to the original format could be done in one PR for the next minor release, clearly documenting all dataset changes just as @albertvillanova outlined them above and perhaps even providing a per dataset util method to convert the new valid format to the old for backward compatibility. Users who relied on the old format will update their code with either the util method for a quick fix or slightly more elaborate for the new.

I don't have a strong opinion on this, besides the fact that whatever decision we agree on, should be applied to all datasets.

There is always the tension between:
- preserving each dataset original structure (which has the advantage of not forcing users to learn other structure for the same dataset),
-  and on the other hand performing some king of standardization/harmonization depending on the task (this has the advantage that once learnt, the same structure applies to all datasets; this has been done for e.g.  POS tagging: all datasets have been adapted to a certain "standard" structure).
   - Another advantage: datasets can easily be interchanged (or joined) to be used by the same model

Recently, in the BigScience BioMedical hackathon, they adopted a different approach:
- they implement a "source" config, respecting the original structure as much as possible
- they implement additional config for each task, with a "standard" nested structure per task, which is most useful for users.

@albertvillanova, thanks for the detailed answer and the new perspectives. I understand the friction for the best design approach much better now.  Ultimately, it is essential to include all the missing fields and the correct data first. Whatever approach is determined to be optimal is important but not as crucial once all the data is there, and users can create lambda functions to create whatever structure serves them best.

## Describe the bug
OpenBookQA implementation is inconsistent with the original dataset.

We need to:

1. The dataset field [question][stem] is flattened into question_stem. Unflatten it to match the original format.
2. Add missing additional fields:
    - 'fact1': row['fact1'],
    - 'humanScore': row['humanScore'],
    - 'clarity': row['clarity'],
    - 'turkIdAnonymized': row['turkIdAnonymized']
3. Ensure the structure and every data item in the original OpenBookQA matches our OpenBookQA version.

## Expected results
The structure and every data item in the original OpenBookQA matches our OpenBookQA version.

## Actual results
TBD

## Environment info
- `datasets` version: 2.1.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.2


CommonSenseQA has missing and inconsistent field names

Thanks for reporting, @vblagoje.

I'm opening a PR to address this.

## Describe the bug
In short, CommonSenseQA implementation is inconsistent with the original dataset.

More precisely, we need to:

1. Add the dataset matching "id" field. The current dataset, instead, regenerates monotonically increasing id.
2. The [“question”][“stem”] field is flattened into "question". We should match the original dataset and unflatten it
3. Add the missing "question_concept" field in the question tree node
4. Anything else? Go over the data structure of the newly repaired CommonSenseQA and make sure it matches the original

## Expected results
Every data item of the CommonSenseQA should structurally and data-wise match the original CommonSenseQA dataset.

## Actual results
TBD

## Environment info
- `datasets` version: 2.1.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.2


Add API code examples for IterableDataset

_The documentation is not available anymore as the PR was closed or merged._

This PR adds API code examples for `IterableDataset` and `IterableDatasetDicts`.


leadboard info added for TNE


Fix typo in logging docs

> This PR fixes #4271.

Things have not changed when searching "tqdm" in the Dataset document. The second result still performs as "Enable".

Hi @jiangwy99, the fix will appear on the `main` version of the docs:

![Screen Shot 2022-05-04 at 8 38 29 AM](https://user-images.githubusercontent.com/59462357/166718225-6848ab91-87d1-4572-9912-40a909af6cb9.png)

This PR fixes #4271.


A typo in docs of datasets.disable_progress_bar

bug

Something isn't working

Hi! Thanks for catching and reporting the typo, a PR has been opened to fix it :)

## Describe the bug
in the docs of V2.1.0 datasets.disable_progress_bar, we should replace "enable" with "disable".


Fix style in openbookqa dataset

CI in PR:
- #4259

was green, but after merging it to master, a code quality error appeared.


Add license and point of contact to big_patent dataset

Update metadata of big_patent dataset with:
- license
- point of contact