Add wiki dpr multiset embeddings

I still have to compute the dataset_infos, and build + host the indexes

update: I'm computing the metadata, will update the PR soon

There are two DPR encoders, one trained on Natural Questions and one trained on a multiset/hybrid dataset.
Previously only the embeddings from the encoder trained on NQ were available. I'm adding the ones from the encoder trained on the multiset/hybrid dataset.
In the configuration you can now specify `embeddings_name="nq"` or `embeddings_name="multiset"`


Replace seqeval code with original classification_report for simplicity

Hello,

I ran https://github.com/huggingface/transformers/blob/master/examples/token-classification/run.sh

And received this error:
```
100%|██████████| 407/407 [21:37<00:00,  3.44s/it]Traceback (most recent call last):
  File "run_ner.py", line 445, in <module>
    main()
  File "run_ner.py", line 398, in main
    results = trainer.evaluate()
  File "/data/2021/transformers/src/transformers/trainer.py", line 1470, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/data/2021/transformers/src/transformers/trainer.py", line 1622, in prediction_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
  File "run_ner.py", line 345, in compute_metrics
    results = metric.compute(predictions=true_predictions, references=true_labels)
  File "/usr/local/lib/python3.6/dist-packages/datasets/metric.py", line 398, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/root/.cache/huggingface/modules/datasets_modules/metrics/seqeval/81eda1ff004361d4fa48754a446ec69bb7aa9cf4d14c7215f407d1475941c5ff/seqeval.py", line 97, in _compute
    report = classification_report(y_true=references, y_pred=predictions, suffix=suffix, output_dict=True)
TypeError: classification_report() got an unexpected keyword argument 'output_dict'
```

I'm still trying multiple things to see if I can work around this, but I thought it might be useful to mention it here.

```
Name: transformers
Version: 4.3.0.dev0

Name: datasets
Version: 1.2.1
```

Hi, can you try to update your local installation of `seqeval` ?

```
pip install --upgrade seqeval
```

@lhoestq thanks for the reply. Indeed it was some issue with my setup. I removed the "transformers" and "datasets" (that I had previously installed from the source code), cleared the cache and installed everything again. It works great now!

Recently, the original seqeval has enabled us to get per type scores and overall scores as a dictionary.

This PR replaces the current code with the original function(`classification_report`) to simplify it.

Also, the original code has been updated to fix #352.
- Related issue: https://github.com/chakki-works/seqeval/pull/38

```python
from datasets import load_metric
metric = load_metric("seqeval")
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
metric.compute(predictions=y_pred, references=y_true)
# Output: {'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.8}
```


Trec Dataset Connection Error

Thanks for reporting.
That's because the download url has changed. The old url now redirects to the new one but we don't support redirection for downloads.

I'm opening a PR to update the url

**Datasets Version:**
1.1.2

**Python Version:**
3.6/3.7

**Code:**
```python
from datasets import load_dataset
load_dataset("trec")
```

**Expected behavior:**
Download Trec dataset and load Dataset object

**Current Behavior:**
Get a connection error saying it couldn't reach http://cogcomp.org/Data/QA/QC/train_5500.label (but the link doesn't seem broken)

<details>
  <summary>Error Logs</summary>

Using custom data configuration default
Downloading and preparing dataset trec/default (download: 350.79 KiB, generated: 403.39 KiB, post-processed: Unknown size, total: 754.18 KiB) to /root/.cache/huggingface/datasets/trec/default/1.1.0/ca4248481ad244f235f4cf277186cad2ee8769f975119a2bbfc41b8932b88bd7...
---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-8-66bf1242096e> in <module>()
----> 1 load_dataset("trec")

10 frames
/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag)
    473         elif response is not None and response.status_code == 404:
    474             raise FileNotFoundError("Couldn't find file at {}".format(url))
--> 475         raise ConnectionError("Couldn't reach {}".format(url))
    476
    477     # Try a second time

ConnectionError: Couldn't reach http://cogcomp.org/Data/QA/QC/train_5500.label

</details>


Start community-provided dataset docs

can you also reference the `--organization` flag like in https://github.com/huggingface/transformers/blob/master/docs/source/model_sharing.rst#upload-your-model-with-the-cli ?

Not sure if the changes in `datasets/wmt_t2t/wmt_utils.py` are intentional.
If you want to add more configs to wmt, could you do it in a serapate PR ?

I don't think I changed wmt_utils (I think github is wrong or my setup is poorly configured).

Locally git diff master --name-only says one file. Master is up to date.
Tried to make a new PR #755 and the same thing happened.

This is one I did to get the pseudo-labels updated. Not sure if it generalizes, but I figured I would write it down. It was pretty easy because all I had to do was make properly formatted directories and change URLs.

+ In slack @thomwolf  called it a `user-namespace` dataset, but the docs call it `community dataset`.
I think the first naming is clearer, but I didn't address that here.

+ I didn't add metadata, will try that.


Throw error when an unexpected key is used in data_files

Thanks for reporting !
We'll add support for other keys

I have found that only "train", "validation" and "test" are valid keys in the `data_files` argument. When you use any other ones, those attached files are silently ignored - leading to unexpected behaviour for the users.

So the following, unintuitively, returns only one key (namely `train`).

```python
datasets = load_dataset("text", data_files={"train": train_f, "valid": valid_f})
print(datasets.keys())
# dict_keys(['train'])
```

whereas using `validation` instead, does return the expected result:

```python
datasets = load_dataset("text", data_files={"train": train_f, "validation": valid_f})
print(datasets.keys())
# dict_keys(['train', 'validation'])
```

I would like to see more freedom in which keys one can use, but if that is not possible at least an error should be thrown when using an unexpected key.


Fix GLUE metric description

Small typo: the description says translation instead of prediction.


Update link to dataset viewer

Change 404 error links in quick tour to working ones


dataset(wlasl): initial loading script

Followup:
From the info in https://github.com/huggingface/datasets/pull/722, I probably should load the videos as array of frames directly into the database.
This will make the dataset generation time very long, but will make working with the dataset much easier.

When I run:
```
python datasets-cli dummy_data datasets/wlasl
```

I get:
```
Checking datasets/wlasl/wlasl.py for additional imports.
Found main folder for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl
Found specific version folder for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786
Found script file from datasets/wlasl/wlasl.py to /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/wlasl.py
Found dataset infos file from datasets/wlasl/dataset_infos.json to /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/dataset_infos.json
Found metadata file for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/wlasl.json
Using custom data configuration default
Loading Dataset Infos from /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786
Creating dummy folder structure for datasets/wlasl/dummy/0.3.0...
Dataset datasets with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data.
Traceback (most recent call last):
File "datasets-cli", line 36, in
service.run()                                                                                File "/home/nlp/amit/anaconda2/envs/meta-scholar/lib/python3.7/site-packages/datasets-1.1.2-py3.7.egg/datasets/commands/dummy_data.py", line 73, in run
for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

takes like 9-10 hours to download all of the videos for the dataset, but it does finish :)


dataset(aslg_pc12): initial loading script

Thanks @lhoestq
Are there any guidelines for the dummy data?
In this particular case for example, the dataset fetches from two hardcoded URLs.
Do I just `head -n 10` both files and zip them?

> Thanks @lhoestq
> Are there any guidelines for the dummy data?
> In this particular case for example, the dataset fetches from two hardcoded URLs.
> Do I just `head -n 10` both files and zip them?

Yes the idea is just to have a few examples to properly test the script and make sure it keeps working in the long run.

And FYI there's a command to help you name the dummy data files correctly. More info in the documentation [here](https://huggingface.co/docs/datasets/share_dataset.html#adding-dummy-data)

This contains the only current public part of this corpus.

The rest of the corpus is not yet been made public, but this sample is still being used by researchers.


Possible caching bug

bug

Something isn't working

Thanks for reporting. That's a bug indeed.
Apparently only the `data_files` parameter is taken into account right now in `DatasetBuilder._create_builder_config` but it should also be the case for `config_kwargs` (or at least the instantiated `builder_config`)

Hi, does this bug be fixed? when I load JSON files, I get the same errors by the command
`!python3 run.py --do_train --task qa --dataset squad-retrain-data/train-v2.0.json --output_dir ./re_trained_model/`

change the dateset to load json by refering to https://huggingface.co/docs/datasets/loading.html
`dataset = datasets.load_dataset('json', data_files=args.dataset)`

Errors:
`Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/json/default-c1e124ad488911b8/0.0.0/45636811569ec4a6630521c18235dfbbab83b7ab572e3393c5ba68ccabe98264...
`

The following code with `test1.txt` containing just "🤗🤗🤗":
```
dataset = datasets.load_dataset('text', data_files=['test1.txt'], split="train", encoding="latin_1")
print(dataset[0])
dataset = datasets.load_dataset('text', data_files=['test1.txt'], split="train", encoding="utf-8")
print(dataset[0])
```
produces this output:
```
Downloading and preparing dataset text/default-15600e4d83254059 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155...
Dataset text downloaded and prepared to /home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155. Subsequent calls will reuse this data.
{'text': 'ð\x9f¤\x97ð\x9f¤\x97ð\x9f¤\x97'}
Using custom data configuration default
Reusing dataset text (/home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155)
{'text': 'ð\x9f¤\x97ð\x9f¤\x97ð\x9f¤\x97'}
```
Just changing the order (and deleting the temp files):
```
dataset = datasets.load_dataset('text', data_files=['test1.txt'], split="train", encoding="utf-8")
print(dataset[0])
dataset = datasets.load_dataset('text', data_files=['test1.txt'], split="train", encoding="latin_1")
print(dataset[0])
```
produces this:
```
Using custom data configuration default
Downloading and preparing dataset text/default-15600e4d83254059 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155...
Dataset text downloaded and prepared to /home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155. Subsequent calls will reuse this data.
{'text': '🤗🤗🤗'}
Using custom data configuration default
Reusing dataset text (/home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155)
{'text': '🤗🤗🤗'}
```

Is it intended that the cache path does not depend on the config entries?

tested with datasets==1.1.2 and python==3.8.5


Better error message when one forgets to call `add_batch` before `compute`

When using metrics, if for some reason a user forgets to call `add_batch` to a metric before `compute` (with no arguments), the error message is a bit cryptic and could probably be made clearer.

## Reproducer

```python
import datasets
import torch
from datasets import Metric

class GatherMetric(Metric):
    def _info(self):
        return datasets.MetricInfo(
            description="description",
            citation="citation",
            inputs_description="kwargs",
            features=datasets.Features({
                'predictions': datasets.Value('int64'),
                'references': datasets.Value('int64'),
            }),
            codebase_urls=[],
            reference_urls=[],
            format='numpy'
        )

    def _compute(self, predictions, references):
        return {"predictions": predictions, "labels": references}

metric = GatherMetric(cache_dir="test-metric")
inputs = torch.randint(0, 2, (1024,))
targets = torch.randint(0, 2, (1024,))

batch_size = 8
for i in range(0, 1024, batch_size):
     pass # User forgets to call `add_batch`
result = metric.compute()
```

## Stack trace:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-267729d187fa> in <module>
      3      pass
      4     # metric.add_batch(predictions=inputs[i:i+batch_size], references=targets[i:i+batch_size])
----> 5 result = metric.compute()

~/git/datasets/src/datasets/metric.py in compute(self, *args, **kwargs)
    380         if predictions is not None:
    381             self.add_batch(predictions=predictions, references=references)
--> 382         self._finalize()
    383
    384         self.cache_file_name = None

~/git/datasets/src/datasets/metric.py in _finalize(self)
    343         elif self.process_id == 0:
    344             # Let's acquire a lock on each node files to be sure they are finished writing
--> 345             file_paths, filelocks = self._get_all_cache_files()
    346
    347             # Read the predictions and references

~/git/datasets/src/datasets/metric.py in _get_all_cache_files(self)
    280         filelocks = []
    281         for process_id, file_path in enumerate(file_paths):
--> 282             filelock = FileLock(file_path + ".lock")
    283             try:
    284                 filelock.acquire(timeout=self.timeout)

TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
```


Passing `cache_dir` to a metric does not work

When passing `cache_dir` to a custom metric, the folder is concatenated to itself at some point and this results in a FileNotFoundError:

## Reproducer

```python
import datasets
import torch
from datasets import Metric

class GatherMetric(Metric):
    def _info(self):
        return datasets.MetricInfo(
            description="description",
            citation="citation",
            inputs_description="kwargs",
            features=datasets.Features({
                'predictions': datasets.Value('int64'),
                'references': datasets.Value('int64'),
            }),
            codebase_urls=[],
            reference_urls=[],
            format='numpy'
        )

    def _compute(self, predictions, references):
        return {"predictions": predictions, "labels": references}

metric = GatherMetric(cache_dir="test-metric")
inputs = torch.randint(0, 2, (1024,))
targets = torch.randint(0, 2, (1024,))

batch_size = 8
for i in range(0, 1024, batch_size):
    metric.add_batch(predictions=inputs[i:i+batch_size], references=targets[i:i+batch_size])
result = metric.compute()
```

## Stack trace:

```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
~/git/datasets/src/datasets/metric.py in _finalize(self)
    349                 reader = ArrowReader(path=self.data_dir, info=DatasetInfo(features=self.features))
--> 350                 self.data = Dataset(**reader.read_files([{"filename": f} for f in file_paths]))
    351             except FileNotFoundError:

~/git/datasets/src/datasets/arrow_reader.py in read_files(self, files, original_instructions)
    227         # Prepend path to filename
--> 228         pa_table = self._read_files(files)
    229         files = copy.deepcopy(files)

~/git/datasets/src/datasets/arrow_reader.py in _read_files(self, files)
    166         for f_dict in files:
--> 167             pa_table: pa.Table = self._get_dataset_from_filename(f_dict)
    168             pa_tables.append(pa_table)

~/git/datasets/src/datasets/arrow_reader.py in _get_dataset_from_filename(self, filename_skip_take)
    291         )
--> 292         mmap = pa.memory_map(filename)
    293         f = pa.ipc.open_stream(mmap)

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.memory_map()

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.MemoryMappedFile._open()

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

FileNotFoundError: [Errno 2] Failed to open local file 'test-metric/gather_metric/default/test-metric/gather_metric/default/default_experiment-1-0.arrow'. Detail: [errno 2] No such file or directory

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-17-e42d43cc981f> in <module>
      2 for i in range(0, 1024, batch_size):
      3     metric.add_batch(predictions=inputs[i:i+batch_size], references=targets[i:i+batch_size])
----> 4 result = metric.compute()

~/git/datasets/src/datasets/metric.py in compute(self, *args, **kwargs)
    380         if predictions is not None:
    381             self.add_batch(predictions=predictions, references=references)
--> 382         self._finalize()
    383
    384         self.cache_file_name = None

~/git/datasets/src/datasets/metric.py in _finalize(self)
    351             except FileNotFoundError:
    352                 raise ValueError(
--> 353                     "Error in finalize: another metric instance is already using the local cache file. "
    354                     "Please specify an experiment_id to avoid colision between distributed metric instances."
    355                 )

ValueError: Error in finalize: another metric instance is already using the local cache file. Please specify an experiment_id to avoid colision between distributed metric instances.
```

The code works when we remove the `cache_dir=...` from the metric.


Parallel downloads progress bar flickers

When there are parallel downloads using the download manager, the tqdm progress bar flickers since all the progress bars are on the same line.

To fix that we could simply specify `position=i` for i=0 to n the number of files to download when instantiating the tqdm progress bar.

Another way would be to have one "master" progress bar that tracks the number of finished downloads, and then one progress bar per process that show the current downloads.


"Checksums didn't match for dataset source files" error while loading openwebtext dataset

Hi try, to provide more information please.

Example code in a colab to reproduce the error, details on what you are trying to do and what you were expected and details on your environment (OS, PyPi packages version).

> Hi try, to provide more information please.
>
> Example code in a colab to reproduce the error, details on what you are trying to do and what you were expected and details on your environment (OS, PyPi packages version).

I have update the description, sorry for the incomplete issue by mistake.

Hi, I have manually downloaded the compressed dataset `openwebtext.tar.xz' and use the following command to preprocess the examples:
```
>>> dataset = load_dataset('/home/admin/workspace/datasets/datasets-master/datasets-master/datasets/openwebtext', data_dir='/home/admin/workspace/datasets')
Using custom data configuration default
Downloading and preparing dataset openwebtext/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/admin/.cache/huggingface/datasets/openwebtext/default/0.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02...
Dataset openwebtext downloaded and prepared to /home/admin/.cache/huggingface/datasets/openwebtext/default/0.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02. Subsequent calls will reuse this data.
>>> len(dataset['train'])
74571
>>>
```
The size of the pre-processed example file is only 354MB, however the processed bookcorpus dataset is 4.6g. Are there any problems?

NonMatchingChecksumError: Checksums didn't match for dataset source files:

i got this issue when i try to work on my own datasets kindly tell me, from where i can get checksums of train and dev file in my github repo

Hi,  I got the similar issue for xnli dataset while working on colab with python3.7.

`nlp.load_dataset(path = 'xnli')`

The above command resulted in following issue :
```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip']
```

Any idea how to fix this  ?

Did anyone figure out how to fix this error?

Says fixed but I'm still getting it.

command:

 dataset = load_dataset("ted_talks_iwslt", language_pair=("en", "es"), year="2014",download_mode="force_redownload")

got:

Using custom data configuration en_es_2014-35a2d3350a0f9823
Downloading and preparing dataset ted_talks_iwslt/en_es_2014 (download: 2.15 KiB, generated: Unknown size, post-processed: Unknown size, total: 2.15 KiB) to /home/ken/.cache/huggingface/datasets/ted_talks_iwslt/en_es_2014-35a2d3350a0f9823/1.1.0/43935b3fe470c753a023642e1f54b068c590847f9928bd3f2ec99f15702ad6a6...
Downloading:
2.21k/? [00:00<00:00, 141kB/s]

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?id=1Cz1Un9p8Xn9IpEMMrg2kXSDt0dnjxc4z&export=download']

Hi,
I have encountered this problem during loading the openwebtext dataset:
```
>>> dataset = load_dataset('openwebtext')
Downloading and preparing dataset openwebtext/plain_text (download: 12.00 GiB, generated: 37.04 GiB, post-processed: Unknown size, total: 49.03 GiB) to /home/admin/.cache/huggingface/datasets/openwebtext/plain_text/1.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 536, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 39, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://zenodo.org/record/3834942/files/openwebtext.tar.xz']
```
I think this problem is caused because the released dataset has changed. Or I should download the dataset manually?

Sorry for release the unfinised issue by mistake.


pretty print dataset objects

Great, as you found it useful I improved the code a bit to automate indentation in the parent class, so that the child repr doesn't need to guess the indentation level, while repr'ing nicely on its own.

- do we want indent=4 or 2?
- do we want `{` ... `}` or w/o?

currently it's indent4 and w/ curly braces, so it looks:

```
DatasetDict({
    train: Dataset({
        features: ['text', 'headline', 'title'],
        num_rows: 157252
    })
    validation: Dataset({
        features: ['text', 'headline', 'title'],
        num_rows: 5599
    })
    test: Dataset({
        features: ['text', 'headline', 'title'],
        num_rows: 5577
    })
})
```
just child:
```
Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 5577
})
```

Currently, if I do:
```
from datasets import load_dataset
load_dataset("wikihow", 'all', data_dir="/hf/pegasus-datasets/wikihow/")
```
I get:
```

DatasetDict({'train': Dataset(features: {'text': Value(dtype='string', id=None),
'headline': Value(dtype='string', id=None), 'title': Value(dtype='string',
id=None)}, num_rows: 157252), 'validation': Dataset(features: {'text':
Value(dtype='string', id=None), 'headline': Value(dtype='string', id=None),
'title': Value(dtype='string', id=None)}, num_rows: 5599), 'test':
Dataset(features: {'text': Value(dtype='string', id=None), 'headline':
Value(dtype='string', id=None), 'title': Value(dtype='string', id=None)},
num_rows: 5577)})
```

This is not very readable.

Can we either have a better `__repr__` or have a custom method to nicely pprint the dataset object?

Here is my very simple attempt. With this PR, it produces:
```
DatasetDict({
  train:   Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 157252
  })
  validation:   Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 5599
  })
  test:   Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 5577
  })
})
```
I did omit the data types on purpose to make it more readable, but it shouldn't be too difficult to integrate those too.

note that this PR also fixes the inconsistency in output that in master misses enclosing `{}` for Dataset, but it is there for `DatasetDict` - or perhaps it was by design.

I'm totally not attached to this format, just wanting something more readable. One approach could be to serialize to `json.dumps` or something similar. It'd make the indentation simpler.

Thank you.


need to redirect /nlp to /datasets and remove outdated info

Should be fixed now:

![image](https://user-images.githubusercontent.com/35882/95917301-040b0600-0d78-11eb-9655-c4ac0e788089.png)

Not sure I understand what you mean by the second part?

Thank you!

> Not sure I understand what you mean by the second part?

Compare the 2:
* https://huggingface.co/datasets/wikihow
* https://huggingface.co/nlp/viewer/?dataset=wikihow&config=all
Can you see the difference? 2nd has formatting, 1st doesn't.

For context, those are two different pages (not an old vs new one), one is from the dataset viewer (you can browse data inside the datasets) while the other is just a basic reference page displayed some metadata about the dataset.

For the second one, we'll move to markdown parsing soon, so it'll be formatted better.

I understand. I was just flagging the lack of markup issue.

It looks like the website still has all the `nlp` data, e.g.: https://huggingface.co/nlp/viewer/?dataset=wikihow&config=all

should probably redirect to: https://huggingface.co/datasets/wikihow

also for some reason the new information is slightly borked. If you look at the old one it was nicely formatted and had the links marked up, the new one is just a jumble of text in one chunk and no markup for links (i.e. not clickable).


Adding pseudo-labels to datasets

Nice ! :)
It's indeed the first time we have such contributions so we'll have to figure out the appropriate way to integrate them.
Could you add details on what they could be used for ?

They can be used as training data for a smaller model.

Sounds just like a regular dataset to me then, no?

A new configuration for those datasets should do the job then.
Note that until now datasets like xsum only had one configuration. It means that users didn't have to specify the configuration name when loading the dataset. If we add new configs, users that update the lib will have to update their code to specify the default/standard configuration name (not the one with pseudo labels).

Could also be a `user-namespace` dataset maybe?

Oh yes why not. I'm more in favor of this actually since pseudo labels are things that users (not dataset authors in general) can compute by themselves and share with the community

![image](https://user-images.githubusercontent.com/6045025/96045248-b528a380-0e3f-11eb-9124-bd55afa031bb.png)

I assume I should (for example) rename the xsum dir, change the URL, and put the modified dir somewhere in S3?

You can use the `datasets-cli` to upload the folder with your version of xsum with the pseudo labels.

```
datasets-cli upload_dataset path/to/xsum
```

I recently [uploaded pseudo-labels](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/precomputed_pseudo_labels.md) for CNN/DM, XSUM and WMT16-en-ro to s3, and thom mentioned I should add them to this repo.
Since pseudo-labels are just a large model's generations on an existing dataset, what is the right way to structure this contribution.
I read https://huggingface.co/docs/datasets/add_dataset.html, but it doesn't really cover this type of contribution.

I could, for example, make a new directory, `xsum_bart_pseudolabels` for each set of pseudolabels or add some sort of parametrization to `xsum.py`: https://github.com/huggingface/datasets/blob/5f4c6e830f603830117877b8990a0e65a2386aa6/datasets/xsum/xsum.py

What do you think @lhoestq ?


datasets(RWTH-PHOENIX-Weather 2014 T): add initial loading script

This might be interesting to @kayoyin the author of https://github.com/kayoyin/transformer-slt – pinging you just in case :)

Thanks Amit, this is a great idea! I'm thinking of porting the SLT models from my paper here as well, having this dataset would be perfect for that :)

This is the first sign language dataset in this repo as far as I know.
Following an old issue I opened https://github.com/huggingface/datasets/issues/302.

I added the dataset official REAMDE file, but I see it's not very standard, so it can be removed.


feat(dl_manager): add support for ftp downloads

We only support http by default for downloading.
If you really need to use ftp, then feel free to use a library that allows to download through ftp in your dataset script (I see that you've started working on #722 , that's awesome !). The users will get a message to install the extra library when they load the dataset.

To make the download_manager work with a custom downloader, you can call `download_manager.download_custom` instead of `download_manager.download_and_extract`. The expected arguments are the following:
```
url_or_urls: url or `list`/`dict` of urls to download and extract. Each
        url is a `str`.
custom_download: Callable with signature (src_url: str, dst_path: str) -> Any
        as for example `tf.io.gfile.copy`, that lets you download from google storage
```

Also maybe it coud be interesting to have a direct support of ftp inside the `datasets` library. Do you know any good libraries that we might consider adding as a (optional ?) dependency ?

Downloading an `ftp` file is as simple as:
```python
import urllib
urllib.urlretrieve('ftp://server/path/to/file', 'file')
```

I believe this should be supported by the library, as its not using any dependency and is trivial amount of code.

I know its unorthodox, but I added `ftp` download support to `file_utils` in the same PR https://github.com/huggingface/datasets/pull/722
So its possible to understand the interaction of the download component with the ftp download ability

@AmitMY Can you now download the Phoenix2014 Dataset?

@hoanganhpham1006 yes.
See pull request https://github.com/huggingface/datasets/pull/722 , it has a loader for this dataset, mostly ready.
There's one issue that delays it being merged - https://github.com/huggingface/datasets/issues/741 - regarding memory consumption.

The problem which I have now is that this dataset seems does not allow to download? Can you share it with me pls

The dataset loader is not yet ready, because of that issue.
If you want to just download the dataset the old-fashioned way, just go to: https://www-i6.informatik.rwth-aachen.de/ftp/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz (the ftp link is now broken, and its available over https)

I am working on a new dataset (#302) and encounter a problem downloading it.

```python
# This is the official download link from https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/
_URL = "ftp://wasserstoff.informatik.rwth-aachen.de/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz"

dl_manager.download_and_extract(_URL)
```

I get an error:

> ValueError: unable to parse ftp://wasserstoff.informatik.rwth-aachen.de/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz as a URL or as a local path

I checked, and indeed you don't consider `ftp` as a remote file.
https://github.com/huggingface/datasets/blob/4c2af707a6955cf4b45f83ac67990395327c5725/src/datasets/utils/file_utils.py#L188

Adding `ftp` to that list does not immediately solve the issue, so there probably needs to be some extra work.


OSError: Cannot find data file when not using the dummy dataset in RAG

Same issue here. I will be digging further, but it looks like the [script](https://github.com/huggingface/datasets/blob/master/datasets/wiki_dpr/wiki_dpr.py#L132) is attempting to open a file that is not downloaded yet.

```
99dcbca09109e58502e6b9271d4d3f3791b43f61f3161a76b25d2775ab1a4498.lock
```

```
---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
~/anaconda3/envs/eqa/lib/python3.7/site-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    446             try:
--> 447                 return pickle.load(fid, **pickle_kwargs)
    448             except Exception:

UnpicklingError: pickle data was truncated

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
~/src/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    559
--> 560         if verify_infos:
    561             verify_splits(self.info.splits, split_dict)

~/src/datasets/src/datasets/builder.py in _prepare_split(self, split_generator)
    847                 writer.write(example)
--> 848         finally:
    849             num_examples, num_bytes = writer.finalize()

~/anaconda3/envs/eqa/lib/python3.7/site-packages/tqdm/notebook.py in __iter__(self, *args, **kwargs)
    227         try:
--> 228             for obj in super(tqdm_notebook, self).__iter__(*args, **kwargs):
    229                 # return super(tqdm...) will not catch exception

~/anaconda3/envs/eqa/lib/python3.7/site-packages/tqdm/std.py in __iter__(self)
   1132         try:
-> 1133             for obj in iterable:
   1134                 yield obj

/hdd/rag/cache/huggingface/modules/datasets_modules/datasets/wiki_dpr/14b973bf2a456087ff69c0fd34526684eed22e48e0dfce4338f9a22b965ce7c2/wiki_dpr.py in _generate_examples(self, data_file, vectors_files)
    131                         break
--> 132                     vecs = np.load(open(vectors_files.pop(0), "rb"), allow_pickle=True)
    133                     vec_idx = 0

~/anaconda3/envs/eqa/lib/python3.7/site-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    449                 raise IOError(
--> 450                     "Failed to interpret file %s as a pickle" % repr(file))
    451

OSError: Failed to interpret file <_io.BufferedReader name='/hdd/rag/downloads/99dcbca09109e58502e6b9271d4d3f3791b43f61f3161a76b25d2775ab1a4498'> as a pickle

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-8-24351ff8ce44> in <module>
      4 retriever = RagRetriever.from_pretrained("facebook/rag-sequence-nq",
      5                                          index_name="exact",
----> 6                                          use_dummy_dataset=False)

~/src/transformers/src/transformers/retrieval_rag.py in from_pretrained(cls, retriever_name_or_path, **kwargs)
    321         generator_tokenizer = rag_tokenizer.generator
    322         return cls(
--> 323             config, question_encoder_tokenizer=question_encoder_tokenizer, generator_tokenizer=generator_tokenizer
    324         )
    325

~/src/transformers/src/transformers/retrieval_rag.py in __init__(self, config, question_encoder_tokenizer, generator_tokenizer)
    310         self.config = config
    311         if self._init_retrieval:
--> 312             self.init_retrieval()
    313
    314     @classmethod

~/src/transformers/src/transformers/retrieval_rag.py in init_retrieval(self)
    338
    339         logger.info("initializing retrieval")
--> 340         self.index.init_index()
    341
    342     def postprocess_docs(self, docs, input_strings, prefix, n_docs, return_tensors=None):

~/src/transformers/src/transformers/retrieval_rag.py in init_index(self)
    248                 split=self.dataset_split,
    249                 index_name=self.index_name,
--> 250                 dummy=self.use_dummy_dataset,
    251             )
    252             self.dataset.set_format("numpy", columns=["embeddings"], output_all_columns=True)

~/src/datasets/src/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    615     builder_instance.download_and_prepare(
    616         download_config=download_config,
--> 617         download_mode=download_mode,
    618         ignore_verifications=ignore_verifications,
    619     )

~/src/datasets/src/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    481                     # Sync info
    482                     self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())
--> 483                     self.info.download_checksums = dl_manager.get_recorded_sizes_checksums()
    484                     self.info.size_in_bytes = self.info.dataset_size + self.info.download_size
    485                     # Save info

~/src/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    560         if verify_infos:
    561             verify_splits(self.info.splits, split_dict)
--> 562
    563         # Update the info object with the splits.
    564         self.info.splits = split_dict

OSError: Cannot find data file.
```

Thank you.

An update on my end. This seems like a transient issue. Reran the script from scratch overnight with no errors.

Closing this one. Feel free to re-open if you have other questions about this issue

## Environment info

    transformers version: 3.3.1
    Platform: Linux-4.19
    Python version: 3.7.7
    PyTorch version (GPU?): 1.6.0
    Tensorflow version (GPU?): No
    Using GPU in script?: Yes
    Using distributed or parallel set-up in script?: No

## To reproduce

Steps to reproduce the behaviour:
```
import os
os.environ['HF_DATASETS_CACHE'] = '/workspace/notebooks/POCs/cache'

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=False)
```

Plese note that I'm using the whole dataset: **use_dummy_dataset=False**
After around 4 hours (downloading and some other things) this is returned:

```
Downloading and preparing dataset wiki_dpr/psgs_w100.nq.exact (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /workspace/notebooks/POCs/cache/wiki_dpr/psgs_w100.nq.exact/0.0.0/14b973bf2a456087ff69c0fd34526684eed22e48e0dfce4338f9a22b965ce7c2...

---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    459             try:
--> 460                 return pickle.load(fid, **pickle_kwargs)
    461             except Exception:

UnpicklingError: pickle data was truncated

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    552                 # Prepare split will record examples associated to the split
--> 553                 self._prepare_split(split_generator, **prepare_split_kwargs)
    554             except OSError:

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _prepare_split(self, split_generator)
    840             for key, record in utils.tqdm(
--> 841                 generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
    842             ):

/opt/conda/lib/python3.7/site-packages/tqdm/notebook.py in __iter__(self, *args, **kwargs)
    217         try:
--> 218             for obj in super(tqdm_notebook, self).__iter__(*args, **kwargs):
    219                 # return super(tqdm...) will not catch exception

/opt/conda/lib/python3.7/site-packages/tqdm/std.py in __iter__(self)
   1128         try:
-> 1129             for obj in iterable:
   1130                 yield obj

~/.cache/huggingface/modules/datasets_modules/datasets/wiki_dpr/14b973bf2a456087ff69c0fd34526684eed22e48e0dfce4338f9a22b965ce7c2/wiki_dpr.py in _generate_examples(self, data_file, vectors_files)
    131                         break
--> 132                     vecs = np.load(open(vectors_files.pop(0), "rb"), allow_pickle=True)
    133                     vec_idx = 0

/opt/conda/lib/python3.7/site-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    462                 raise IOError(
--> 463                     "Failed to interpret file %s as a pickle" % repr(file))
    464     finally:

OSError: Failed to interpret file <_io.BufferedReader name='/workspace/notebooks/POCs/cache/downloads/f34d5f091294259b4ca90e813631e69a6ded660d71b6cbedf89ddba50df94448'> as a pickle

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-10-f28df370ac47> in <module>
      1 # ln -s /workspace/notebooks/POCs/cache /root/.cache/huggingface/datasets
----> 2 retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=False)

/opt/conda/lib/python3.7/site-packages/transformers/retrieval_rag.py in from_pretrained(cls, retriever_name_or_path, **kwargs)
    307         generator_tokenizer = rag_tokenizer.generator
    308         return cls(
--> 309             config, question_encoder_tokenizer=question_encoder_tokenizer, generator_tokenizer=generator_tokenizer
    310         )
    311

/opt/conda/lib/python3.7/site-packages/transformers/retrieval_rag.py in __init__(self, config, question_encoder_tokenizer, generator_tokenizer)
    298         self.config = config
    299         if self._init_retrieval:
--> 300             self.init_retrieval()
    301
    302     @classmethod

/opt/conda/lib/python3.7/site-packages/transformers/retrieval_rag.py in init_retrieval(self)
    324
    325         logger.info("initializing retrieval")
--> 326         self.index.init_index()
    327
    328     def postprocess_docs(self, docs, input_strings, prefix, n_docs, return_tensors=None):

/opt/conda/lib/python3.7/site-packages/transformers/retrieval_rag.py in init_index(self)
    238                 split=self.dataset_split,
    239                 index_name=self.index_name,
--> 240                 dummy=self.use_dummy_dataset,
    241             )
    242             self.dataset.set_format("numpy", columns=["embeddings"], output_all_columns=True)

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    609         download_config=download_config,
    610         download_mode=download_mode,
--> 611         ignore_verifications=ignore_verifications,
    612     )
    613

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    474                     if not downloaded_from_gcs:
    475                         self._download_and_prepare(
--> 476                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    477                         )
    478                     # Sync info

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    553                 self._prepare_split(split_generator, **prepare_split_kwargs)
    554             except OSError:
--> 555                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))
    556
    557         if verify_infos:

OSError: Cannot find data file.

```

Thanks


Fix train_test_split output format

There was an issue in the `transmit_format` wrapper that returned bad formats when using train_test_split.
This was due to `column_names` being handled as a List[str] instead of Dict[str, List[str]] when the dataset transform (train_test_split) returns a DatasetDict (one set of column names per split).

This should fix @timothyjlaurent 's issue in #620 and fix #676

I added tests for `transmit_format` so that it doesn't happen again


Don't use tqdm 4.50.0

tqdm 4.50.0 introduced permission errors on windows
see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/235/workflows/cfb6a39f-68eb-4802-8b17-2cd5e8ea7369/jobs/1111) for the error details.

For now I just added `<4.50.0` in the setup.py
Hopefully we can find what's wrong with this version soon


Fixes #712 Error in the Overview.ipynb notebook by adding `with_details=True` parameter to `list_datasets` function in Cell 3 of **overview** notebook


Fixes #712 Attribute error in cell 3 of the overview notebook

Referencing the wrong issue # in the commit message. Closing this to fix it again.

Fixes the Attribute error in cell 3 of the overview notebook


Use python read for text dataset

One thing though, could we try to read the files in parallel?

We could but I'm not sure this would help a lot since the bottleneck is the drive IO if the files are big enough.
It could make sense for very small files.

Looks like windows is not a big fan of this approach
I'm working on a fix

I remember issue https://github.com/huggingface/datasets/issues/546 where this was kinda requested (but maybe IO would bottleneck). What do you think?

I think it's worth testing multiprocessing. It could also be something we add to our speed benchmarks

> I remember issue #546 where this was kinda requested (but maybe IO would bottleneck). What do you think?

It still would be interesting I think, especially in scenarios where IO is less of an issue (SSDs particularly) and where there are many smaller files. Wrapping this function in a `pool.map` is perhaps an easy thing to try.

Merging this one for now for the patch release

As mentioned in #622 the pandas reader used for text dataset doesn't work properly when there are \r characters in the text file.

Instead I switched to pure python using `open` and `read`.
From my benchmark on a 100MB text file, it's the same speed as the previous pandas reader.


Add the official dependabot implementation

This will keep dependencies up to date. This will require a pr label `dependencies` being created in order to function correctly.


Fix reading text files with carriage return symbols

Discussed in #622, fixed in #715. Closing the issue. Thanks @lhoestq, it works now! 👍

The new pandas-based text reader isn't able to work properly with files that contain carriage return symbols (`\r`).

It fails with the following error message:

```
...
  File "pandas/_libs/parsers.pyx", line 847, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 874, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 918, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 905, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2042, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.
```

___
I figured out the pandas uses those symbols as line terminators and this eventually causes the error. Explicitly specifying the `lineterminator` fixes that issue and everything works fine.

Please, consider this PR as it seems to be a common issue to solve.


Error in the notebooks/Overview.ipynb notebook

Do this:
``` python
squad_dataset = list_datasets(with_details=True)[datasets.index('squad')]
pprint(squad_dataset.__dict__)  # It's a simple python dataclass
```

Thanks! This worked. I have created a PR to fix this in the notebook.

Hi,

I got the following error in **cell number 3** while exploring the **Overview.ipynb** notebook in google colab. I used the [link ](https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb) provided in the main README file to open it in colab.

```python
# You can access various attributes of the datasets before downloading them
squad_dataset = list_datasets()[datasets.index('squad')]

pprint(squad_dataset.__dict__)  # It's a simple python dataclass
```

Error message
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-8dc805c4949c> in <module>()
      2 squad_dataset = list_datasets()[datasets.index('squad')]
      3
 ----> 4 pprint(squad_dataset.__dict__)  # It's a simple python dataclass

AttributeError: 'str' object has no attribute '__dict__'
```

The object `squad_dataset` is a `str` not a `dataclass` .


New Update bertscore.py


fix README typos/ consistency


How to use similarity settings  other then "BM25" in Elasticsearch index ?

Datasets does not use elasticsearch API to define custom similarity. If you want to use a custom similarity, the best would be to run a curl request directly to your elasticsearch instance (see sample hereafter, directly from ES documentation), then you should be able to use `my_similarity` in your configuration passed to datasets

```
curl -X PUT "localhost:9200/index?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index": {
      "similarity": {
        "my_similarity": {
          "type": "DFR",
          "basic_model": "g",
          "after_effect": "l",
          "normalization": "h2",
          "normalization.h2.c": "3.0"
        }
      }
    }
  }
}
'

```

**QUESTION : How should we use other similarity algorithms supported by Elasticsearch other than "BM25"  ?**
**ES Reference**
https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html
**HF doc reference:**
https://huggingface.co/docs/datasets/faiss_and_ea.html

**context :**
========

I used the latest Elasticsearch server  version 7.9.2
When I set DFR  which is one of the other similarity algorithms supported by elasticsearch  in the mapping, I get an error

For example DFR that I had tried in the first instance in mappings as below.,
`"mappings": {"properties": {"text": {"type": "text", "analyzer": "standard", "similarity": "DFR"}}},`

I get the following error
RequestError: RequestError(400, 'mapper_parsing_exception', 'Unknown Similarity type [DFR] for field [text]')

The other thing as another option I had tried was to declare "similarity": "my_similarity" within settings and then assigning "my_similarity" inside the mappings as below

`es_config = {
        "settings": {
            "number_of_shards": 1,
             **"similarity":  "my_similarity"**: {
          "type": "DFR",
          "basic_model": "g",
          "after_effect": "l",
          "normalization": "h2",
          "normalization.h2.c": "3.0"
        } ,
            "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},

        },
        "mappings": {"properties": {"text": {"type": "text", "analyzer": "standard", "similarity": "my_similarity"}}},
    }`

For this , I got the following error
RequestError: RequestError(400, 'illegal_argument_exception', 'unknown setting [index.similarity] please check that any required plugins are installed, or check the breaking changes documentation for removed settings')


Datasets performance slow? - 6.4x slower than in memory dataset

Facing a similar issue here. My model using SQuAD dataset takes about 1h to process with in memory data and more than 2h with datasets directly.

And if you use in-memory-data with datasets with `load_dataset(..., keep_in_memory=True)`?

Thanks for the tip @thomwolf ! I did not see that flag in the docs. I'll try with that.

We should add it indeed and also maybe a specific section with all the tips for maximal speed. What do you think @lhoestq @SBrandeis @yjernite ?

By default the datasets loaded with `load_dataset` live on disk.
It's possible to load them in memory by using some transforms like `.map(..., keep_in_memory=True)`.

Small correction to @thomwolf 's comment above: currently we don't have the `keep_in_memory` parameter for `load_dataset` AFAIK but it would be nice to add it indeed :)

Great! Thanks a lot.

I did a test using `map(..., keep_in_memory=True)` and also a test using in-memory only data.

```python
features = dataset.map(tokenize, batched=True, remove_columns=dataset['train'].column_names)
features.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask'])

features_in_memory = dataset.map(tokenize, batched=True, keep_in_memory=True, remove_columns=dataset['train'].column_names)
features_in_memory.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask'])

in_memory = [features['train'][i] for i in range(len(features['train']))]
```

For using the features without any tweak, I got **1min17s** for copying the entire DataLoader to CUDA:

```
%%time

for i, batch in enumerate(DataLoader(features['train'], batch_size=16, num_workers=4)):
    batch['input_ids'].to(device)
```

For using the features mapped with `keep_in_memory=True`, I also got **1min17s** for copying the entire DataLoader to CUDA:

```
%%time

for i, batch in enumerate(DataLoader(features_in_memory['train'], batch_size=16, num_workers=4)):
    batch['input_ids'].to(device)
```

And for the case using every element in memory, converted from the original dataset, I got **12.5s**:

```
%%time

for i, batch in enumerate(DataLoader(in_memory, batch_size=16, num_workers=4)):
    batch['input_ids'].to(device)
```

Taking a closer look in my SQuAD code, using a profiler, I see a lot of calls to `posix read` api. It seems that it is really reliying on disk, which results in a very high train time.

I am having the same issue here. When loading from memory I can get the GPU up to 70% util but when loading after mapping I can only get 40%.

In disk:
```
book_corpus = load_dataset('bookcorpus', 'plain_text', cache_dir='/home/ad/Desktop/bookcorpus', split='train[:20%]')
book_corpus = book_corpus.map(encode, batched=True, num_proc=20, load_from_cache_file=True, batch_size=2500)
book_corpus.set_format(type='torch', columns=['text', "input_ids", "attention_mask", "token_type_ids"])

training_args = TrainingArguments(
    output_dir="./mobile_bert_big",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=16,
    save_steps=50,
    save_total_limit=2,
    logging_first_step=True,
    warmup_steps=100,
    logging_steps=50,
    eval_steps=100,
    no_cuda=False,
    gradient_accumulation_steps=16,
    fp16=True)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=book_corpus,
    tokenizer=tokenizer)
```

In disk I can only get 0,17 it/s:
`[ 13/28907 01:03 < 46:03:27, 0.17 it/s, Epoch 0.00/1] `

If I load it with torch.utils.data.Dataset()
```
class BCorpusDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        self.encodings = encodings

    def __getitem__(self, idx):
        item = [torch.tensor(val[idx]) for key, val in self.encodings.items()][0]
        return item

    def __len__(self):
        length = [len(val) for key, val in self.encodings.items()][0]
        return length

**book_corpus = book_corpus.select([i for i in range(16*2000)])** # filtering to not have 20% of BC in memory...
book_corpus = book_corpus(book_corpus)
```
I can get:
` [ 5/62 00:09 < 03:03, 0.31 it/s, Epoch 0.06/1]`

But obviously I can not get BookCorpus in memory xD

EDIT: it is something weird. If i load in disk 1% of bookcorpus:
```
book_corpus = load_dataset('bookcorpus', 'plain_text', cache_dir='/home/ad/Desktop/bookcorpus', split='train[:1%]')
```

I can get 0.28 it/s, (the same that in memory) but if I load 20% of bookcorpus:
```
book_corpus = load_dataset('bookcorpus', 'plain_text', cache_dir='/home/ad/Desktop/bookcorpus', split='train[:20%]')
```
I get again 0.17 it/s.

I am missing something? I think it is something related to size, and not disk or in-memory.

There is a way to increase the batches read from memory? or multiprocessed it? I think that one of two or it is reading with just 1 core o it is reading very small chunks from disk and left my GPU at 0 between batches

My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

I've been very excited about this amazing datasets project. However, I've noticed that the performance can be substantially slower than using an in-memory dataset.

Now, this is expected I guess, due to memory mapping data using arrow files, and you don't get anything for free. But I was surprised at how much slower.

For example, in the `yelp_polarity` dataset (560000 datapoints, or 17500 batches of 32), it was taking me 3:31 to just get process the data and get it on the GPU (no model involved). Whereas, the equivalent in-memory dataset would finish in just 0:33.

Is this expected? Given that one of the goals of this project is also accelerate dataset processing, this seems a bit slower than I would expect. I understand the advantages of being able to work on datasets that exceed memory, and that's very exciting to me, but thought I'd open this issue to discuss.

For reference I'm running a AMD Ryzen Threadripper 1900X 8-Core Processor CPU, with 128 GB of RAM and an NVME SSD Samsung 960 EVO. I'm running with an RTX Titan 24GB GPU.

I can see with `iotop` that the dataset gets quickly loaded into the system read buffers, and thus doesn't incur any additional IO reads. Thus in theory, all the data *should* be in RAM, but in my benchmark code below it's still 6.4 times slower.

What am I doing wrong? And is there a way to force the datasets to completely load into memory instead of being memory mapped in cases where you want maximum performance?

At 3:31 for 17500 batches, that's 12ms per batch. Does this 12ms just become insignificant as a proportion of forward and backward passes in practice, and thus it's not worth worrying about this in practice?

In any case, here's my code `benchmark.py`. If you run it with an argument of `memory` it will copy the data into memory before executing the same test.

``` py
import sys
from datasets import load_dataset
from transformers import DataCollatorWithPadding, BertTokenizerFast
from torch.utils.data import DataLoader
from tqdm import tqdm

if __name__ == '__main__':
    tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
    collate_fn = DataCollatorWithPadding(tokenizer, padding=True)

    ds = load_dataset('yelp_polarity')

    def do_tokenize(x):
        return tokenizer(x['text'], truncation=True)

    ds = ds.map(do_tokenize, batched=True)
    ds.set_format('torch', ['input_ids', 'token_type_ids', 'attention_mask'])

    if len(sys.argv) == 2 and sys.argv[1] == 'memory':
        # copy to memory - probably a faster way to do this - but demonstrates the point
        # approximately 530 batches per second - 17500 batches in 0:33
        print('using memory')
        _ds = [data for data in tqdm(ds['train'])]
    else:
        # approximately 83 batches per second - 17500 batches in 3:31
        print('using datasets')
        _ds = ds['train']

    dl = DataLoader(_ds, shuffle=True, collate_fn=collate_fn, batch_size=32, num_workers=4)

    for data in tqdm(dl):
        for k, v in data.items():
            data[k] = v.to('cuda')
```

For reference, my conda environment is [here](https://gist.github.com/05b6101518ff70ed42a858b302a0405d)

Once again, I'm very excited about this library, and how easy it is to load datasets, and to do so without worrying about system memory constraints.

Thanks for all your great work.


Requirements should specify pyarrow<1

Hello @mathcass I would want to work on this issue. May I do the same?

@punitaojha, certainly. Feel free to work on this. Let me know if you need any help or clarity.

Hello @mathcass
1. I did fork the repository and clone the same on my local system.

2. Then learnt about how we can publish our package on pypi.org. Also, found some instructions on same in setup.py documentation.

3. Then I Perplexity document link that you shared above. I created a colab link from there keep both tensorflow and pytorch means a mixed option and tried to run it in colab but I encountered no errors at a point where you mentioned. Can you help me to figure out the issue.

4.Here is the link of the colab file with my saved responses.
https://colab.research.google.com/drive/1hfYz8Ira39FnREbxgwa_goZWpOojp2NH?usp=sharing

Also, please share some links which made you conclude that pyarrow < 1 would help.

Thanks for looking at this @punitaojha and thanks for sharing the notebook.

I just tried to reproduce this on my own (based on the environment where I had this issue) and I can't reproduce it somehow. If I run into this again, I'll include some steps to reproduce it. I'll close this as invalid.

Thanks again.

I am sorry for hijacking this closed issue, but I believe I was able to reproduce this very issue. Strangely enough, it also turned out that running `pip install "pyarrow<1" --upgrade` did indeed fix the issue (PyArrow was installed in version `0.14.1` in my case).

Please see the Colab below:

https://colab.research.google.com/drive/15QQS3xWjlKW2aK0J74eEcRFuhXUddUST

Thanks!

I was looking at the docs on [Perplexity](https://huggingface.co/transformers/perplexity.html) via GPT2. When you load datasets and try to load Wikitext, you get the error,

```
module 'pyarrow' has no attribute 'PyExtensionType'
```
I traced it back to datasets having installed PyArrow 1.0.1 but there's not pinning in the setup file.

https://github.com/huggingface/datasets/blob/e86a2a8f869b91654e782c9133d810bb82783200/setup.py#L68

Downgrading by installing `pip install "pyarrow<1"` resolved the issue.


Fix config creation for data files with NamedSplit

During config creation, we need to iterate through the data files of all the splits to compute a hash.
To make sure the hash is unique given a certain combination of files/splits, we sort the split names.
However the `NamedSplit` objects can't be passed to `sorted` and currently it raises an error: we need to sort the string of their names instead.

Fix #705


Hi !
Thanks for reporting :)
Indeed this is an issue on the `datasets` side.
I'm creating a PR

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1 (installed from master)
- `datasets` version: 1.0.2 (installed as a dependency from transformers)
- Platform: Linux-4.15.0-118-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.9

I'm testing my own text classification dataset using [this example](https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-generic-text-classification-script-in-tensorflow) from transformers. The dataset is split into train / dev / test, and in csv format, containing just a text and a label columns, using comma as sep. Here's a sample:
```
text,label
"Registra-se a presença do acadêmico <name> . <REL_SEP> Ao me deparar com a descrição de dois autores no polo ativo da ação junto ao PJe , margem esquerda foi informado pela procuradora do reclamante que se trata de uma reclamação trabalhista individual . <REL_SEP> Diante disso , face a ausência injustificada do autor <name> , determina-se o ARQUIVAMENTO do presente processo , com relação a este , nos termos do [[ art . 844 da CLT ]] . <REL_SEP> CUSTAS AUTOR - DISPENSADO <REL_SEP> Custas pelo autor no importe de R $326,82 , calculadas sobre R $16.341,03 , dispensadas na forma da lei , em virtude da concessão dos benefícios da Justiça Gratuita , ora deferida . <REL_SEP> Cientes os presentes . <REL_SEP> Audiência encerrada às 8h42min . <REL_SEP> <name> <REL_SEP> Juíza do Trabalho <REL_SEP> Ata redigida por << <name> >> , Secretário de Audiência .",NO_RELATION
```

However, @Santosh-Gupta reported in #7351 that he had the exact same problem using the ChemProt dataset. His colab notebook is referenced in the following section.

## To reproduce

Steps to reproduce the behavior:

1. Created a new conda environment using conda env -n transformers python=3.7
2. Cloned transformers master, `cd` into it and installed using pip install --editable .  -r examples/requirements.txt
3. Installed tensorflow with `pip install tensorflow`
3. Ran `run_tf_text_classification.py` with the following parameters:

```
--train_file <DATASET_PATH>/train.csv \
--dev_file <DATASET_PATH>/dev.csv \
--test_file <DATASET_PATH>/test.csv \
--label_column_id 1 \
--model_name_or_path neuralmind/bert-base-portuguese-cased \
--output_dir <OUTPUT_PATH> \
--num_train_epochs 4 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--do_train \
--do_eval \
--do_predict \
--logging_steps 1000 \
--evaluate_during_training \
--save_steps 1000 \
--overwrite_output_dir \
--overwrite_cache
```

I have also copied [@Santosh-Gupta 's colab notebook](https://colab.research.google.com/drive/11APei6GjphCZbH5wD9yVlfGvpIkh8pwr?usp=sharing) as a reference.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Here is the stack trace:

```
2020-10-02 07:33:41.622011: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
/media/discoD/repositorios/transformers_pedro/src/transformers/training_args.py:333: FutureWarning: The `evaluate_during_training` argument is deprecated in favor of `evaluation_strategy` (which has more options)
  FutureWarning,
2020-10-02 07:33:43.471648: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-10-02 07:33:43.471791: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.472664: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 15 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 238.66GiB/s
2020-10-02 07:33:43.472684: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-02 07:33:43.472765: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-02 07:33:43.472809: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-02 07:33:43.472848: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-02 07:33:43.474209: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-02 07:33:43.474276: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-02 07:33:43.561219: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-02 07:33:43.561397: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.562345: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.563219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-02 07:33:43.563595: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-02 07:33:43.570091: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3591830000 Hz
2020-10-02 07:33:43.570494: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560842432400 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-02 07:33:43.570511: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-02 07:33:43.570702: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.571599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 15 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 238.66GiB/s
2020-10-02 07:33:43.571633: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-02 07:33:43.571645: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-02 07:33:43.571654: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-02 07:33:43.571664: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-02 07:33:43.571691: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-02 07:33:43.571704: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-02 07:33:43.571718: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-02 07:33:43.571770: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.572641: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.573475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-02 07:33:47.139227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-02 07:33:47.139265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2020-10-02 07:33:47.139272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-10-02 07:33:47.140323: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:47.141248: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:47.142085: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:47.142854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5371 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-10-02 07:33:47.146317: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5608b95dc5c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-10-02 07:33:47.146336: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1
10/02/2020 07:33:47 - INFO - __main__ -   n_replicas: 1, distributed training: False, 16-bits training: False
10/02/2020 07:33:47 - INFO - __main__ -   Training/evaluation parameters TFTrainingArguments(output_dir='/media/discoD/models/datalawyer/pedidos/transformers_tf', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=True, evaluate_during_training=True, evaluation_strategy=<EvaluationStrategy.STEPS: 'steps'>, prediction_loss_only=False, per_device_train_batch_size=4, per_device_eval_batch_size=4, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=4.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Oct02_07-33-43_user-XPS-8700', logging_first_step=False, logging_steps=1000, save_steps=1000, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, dataloader_num_workers=0, past_index=-1, run_name='/media/discoD/models/datalawyer/pedidos/transformers_tf', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=False, tpu_name=None, xla=False)
10/02/2020 07:33:53 - INFO - filelock -   Lock 140407857405776 acquired on /home/user/.cache/huggingface/datasets/e0f1e9ed46db1e2429189f06b479cbd4075c0976104c1aacf8f77d9a53d2ad87.03756fef6da334f50a7ff73608e21b5018229944ca250416ce7352e25d84a552.py.lock
10/02/2020 07:33:53 - INFO - filelock -   Lock 140407857405776 released on /home/user/.cache/huggingface/datasets/e0f1e9ed46db1e2429189f06b479cbd4075c0976104c1aacf8f77d9a53d2ad87.03756fef6da334f50a7ff73608e21b5018229944ca250416ce7352e25d84a552.py.lock
Using custom data configuration default
Traceback (most recent call last):
  File "run_tf_text_classification.py", line 283, in <module>
    main()
  File "run_tf_text_classification.py", line 222, in main
    max_seq_length=data_args.max_seq_length,
  File "run_tf_text_classification.py", line 43, in get_tfds
    ds = datasets.load_dataset("csv", data_files=files)
  File "/media/discoD/anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/load.py", line 604, in load_dataset
    **config_kwargs,
  File "/media/discoD/anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/builder.py", line 158, in __init__
    **config_kwargs,
  File "/media/discoD/anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/builder.py", line 269, in _create_builder_config
    for key in sorted(data_files.keys()):
TypeError: '<' not supported between instances of 'NamedSplit' and 'NamedSplit'
```

## Expected behavior

Should be able to run the text-classification example as described in [https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-generic-text-classification-script-in-tensorflow](https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-generic-text-classification-script-in-tensorflow)

Originally opened this issue at transformers' repository: [https://github.com/huggingface/transformers/issues/7535](https://github.com/huggingface/transformers/issues/7535). @jplu instructed me to open here, since according to [this](https://github.com/huggingface/transformers/issues/7535#issuecomment-702778885) evidence, the problem is from datasets.

Thanks!


Fix remote tests for new datasets

When adding a new dataset, the remote tests fail because they try to get the new dataset from the master branch (i.e., where the dataset doesn't exist yet)
To fix that I reverted to the use of the HF API that fetch the available datasets on S3 that is synced with the master branch


Add hotpot QA

Awesome :)

Don't pay attention to the RemoteDatasetTest error, I'm fixing it right now

You can rebase from master to fix the CI test :)

If we're lucky we can even include this dataset in today's release

> Just thinking since `type` can only be `comparison` or `bridge` and `level` can only be `easy`, `medium`, `hard` should they be `ClassLabel`?

I think it's more a tag than a label. I guess a string is fine

Added the [HotpotQA](https://github.com/hotpotqa/hotpot) multi-hop question answering dataset.


Complete rouge kwargs

In #701 we noticed that some kwargs were missing for rouge


Add rouge 2 and rouge Lsum to rouge metric outputs

Continuation of #700

Rouge 2 and Rouge Lsum were missing in Rouge's outputs.
Rouge Lsum is also useful to evaluate Rouge L for sentences with `\n`

Fix #617


Add rouge-2 in rouge_types for metric calculation

Indeed there's currently a mismatch between the description and what it rouge actually returns.
Thanks for proposing this fix :)

I think it's better to return rouge 1-2-L.
Was there a reason to only include rouge 1 and rouge L @thomwolf ?

rougeLsum is also missing, could you add it ?

Adding `RougeLSum` would fix https://github.com/huggingface/datasets/issues/617

I am opening a PR with both of them right now actually :)

Also the format of the output isn't exactly ideal, It's usually only the F-1 score that is cared about.

Formatting the output to reflect how `ROUGE-1-5-5` (the perl version thats usually used and pyrouge is a wrapper over it), would be better.

I'll close this since you seem to have already added it in another PR. Sorry for the delay in responding to you @lhoestq.

What do you mean by "Formatting the output to reflect how ROUGE-1-5-5" @Shashi456 ?

I like the idea of returning all the scores for two reason:
- Rouge's aggregator does sampling and therefore it returns "low" "mid" and "high" scores
- It is interesting to have the precision and recall to see how the F1 score was computed
But I understand your point that returning only the F1 score makes sense since it's the one that's always used

Ok I see.
I think it's also important to follow one of the existing output format (there are already too many different formats, let's try not to add another different one)
I'd still stick with the current format and not transform the output of the python implementation of rouge since it's already widely used.
What do you think ?

Maybe we could convert the dataclasses in dictionnaries, would that help @Shashi456 ?

@thomwolf yeah I think that would help. I initially didn't understand the high low mid categories. Dictionaries could help in this case I guess, and if we allow the user to choose what they want i.e F1 and precision or recall.

The description of the ROUGE metric says,
```
_KWARGS_DESCRIPTION = """
Calculates average rouge scores for a list of hypotheses and references
Args:
    predictions: list of predictions to score. Each predictions
        should be a string with tokens separated by spaces.
    references: list of reference for each prediction. Each
        reference should be a string with tokens separated by spaces.
Returns:
    rouge1: rouge_1 f1,
    rouge2: rouge_2 f1,
    rougeL: rouge_l f1,
    rougeLsum: rouge_l precision
"""
```

but the `rouge_types` argument defaults to  `rouge_types = ["rouge1", "rougeL"]`, this PR updates and add `rouge2` to the list so as to reflect the description card.


XNLI dataset is not loading

also i tried below code to solve checksum error
`datasets-cli test ./datasets/xnli --save_infos --all_configs`

and it shows

```
2020-10-02 07:06:16.588760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 268, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 474, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/./datasets/xnli/xnli.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 279, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 474, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/./datasets/xnli/xnli.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/datasets-cli", line 36, in <module>
    service.run()
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 76, in run
    module_path, hash = prepare_module(path)
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 283, in prepare_module
    combined_path, github_file_path, file_path
FileNotFoundError: Couldn't find file locally at ./datasets/xnli/xnli.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/./datasets/xnli/xnli.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/./datasets/xnli/xnli.py
```

Hi !
Yes the download url changed.
It's updated on the master branch. I'm doing a release today to fix that :)

`dataset = datasets.load_dataset(path='xnli')`

showing below error
```
/opt/conda/lib/python3.7/site-packages/nlp/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     36     if len(bad_urls) > 0:
     37         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 38         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     39     logger.info("All the checksums matched successfully" + for_verification_name)
     40

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip']
```

I think URL is now changed to "https://cims.nyu.edu/~sbowman/xnli/XNLI-MT-1.0.zip"


Update README.md

Hey I was just telling my subscribers to check out your repositories
Thank you


Elasticsearch index docs

I added the docs for ES indexes.

I also added a `load_elasticsearch_index` method to load an index that has already been built.

I checked the tests for the ES index and we have tests that mock ElasticSearch.
I think this is good for now but at some point it would be cool to have an end-to-end test with a real ES running.


Update XNLI download link

The old link isn't working anymore. I updated it with the new official link.
Fix #690


Use GitHub instead of aws in remote dataset tests

Recently we switched from aws s3 to github to download dataset scripts.
However in the tests, the dummy data were still downloaded from s3.
So I changed that to download them from github instead, in the MockDownloadManager.

Moreover I noticed that `anli`'s dummy data were quite heavy (18MB compressed, i.e. the entire dataset) so I replaced them with dummy data with few examples.


Rachel ker add dataset/mlsum

It looks like an outdated PR (we've already added mlsum). Closing it


Update README.md

To enhance its readability.....not Hacktoberfest spam

How is adding a punctuation to the end of a sentence justified as "To enhance its readability".
Considering that this is not your first "README enhancement '' please don't spam the open source community with useless PR to get a free T-Shirt it just hurts the maintainers.

//Joey


Add UI filter to filter datasets based on task

enhancement

New feature or request

This is great work, so huge shoutout to contributors and huggingface.

The [/nlp/viewer](https://huggingface.co/nlp/viewer/) is great and the [/datasets](https://huggingface.co/datasets) page is great. I was wondering if in both or either places we can have a filter that selects if a dataset is good for the following tasks (non exhaustive list)

- Classification
	- Multi label
	- Multi class
- Q&A
- Summarization
- Translation

I believe this feature might have some value, for folks trying to find datasets for a particular task, and then testing their model capabilities.

Thank you :)


XNLI dataset: NonMatchingChecksumError

Thanks for reporting.
The data file must have been updated by the host.
I'll update the checksum with the new one.

Well actually it looks like the link isn't working anymore :(

The new link is https://cims.nyu.edu/~sbowman/xnli/XNLI-1.0.zip
I'll update the dataset script

I'll do a release in the next few days to make the fix available for everyone.
In the meantime you can load `xnli` with
```
xnli = load_dataset('xnli', script_version="master")
```
This will use the latest version of the xnli script (available on master branch), instead of the old one.

Hi,
I tried to download "xnli" dataset in colab using
`xnli = load_dataset(path='xnli')`
but got 'NonMatchingChecksumError' error

`NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-27-a87bedc82eeb> in <module>()
----> 1 xnli = load_dataset(path='xnli')

3 frames
/usr/local/lib/python3.6/dist-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     37     if len(bad_urls) > 0:
     38         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 39         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     40     logger.info("All the checksums matched successfully" + for_verification_name)
     41

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip']`

The same code worked well several days ago in colab but stopped working now. Thanks!


Switch to pandas reader for text dataset

If the windows tests in the CI pass, today will be a happy day

Following the discussion in #622 , it appears that there's no appropriate ways to use the payrrow csv reader to read text files because of the separator.

In this PR I switched to pandas to read the file.

Moreover pandas allows to read the file by chunk, which means that you can build the arrow dataset from a text file that is bigger than RAM (we used to have to shard text files an mentioned in https://github.com/huggingface/datasets/issues/610#issuecomment-691672919)

From a test that I did locally on a 1GB text file, the pyarrow reader used to run in 150ms while the new one takes 650ms (multithreading off for pyarrow). This is probably due to chunking since I am having the same speed difference by calling `read()` and calling `read(chunksize)` + `readline()` to read the text file.


Disable tokenizers parallelism in multiprocessed map

It was reported in #620 that using multiprocessing with a tokenizers shows this message:
```
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
```
This message is shown when TOKENIZERS_PARALLELISM is unset.
Moreover if it is set to `true`, then the program just hangs.

To hide the message (if TOKENIZERS_PARALLELISM is unset) and avoid hanging (if TOKENIZERS_PARALLELISM is `true`), then I set TOKENIZERS_PARALLELISM to `false` when forking the process. After forking is gets back to its original value.

Also I added a warning if TOKENIZERS_PARALLELISM was `true` and is set to `false`:
```
Setting TOKENIZERS_PARALLELISM=false for forked processes.
```

cc @n1t0


`ArrowInvalid` occurs while running `Dataset.map()` function

Hi !

This is because `encode` expects one single text as input (str), or one tokenized text (List[str]).
I believe that you actually wanted to use `encode_batch` which expects a batch of texts.
However this method is only available for our "fast" tokenizers (ex: BertTokenizerFast).
BertJapanese is not one of them unfortunately and I don't think it will be added for now (see https://github.com/huggingface/transformers/pull/7141)...
cc @thomwolf for confirmation.

Therefore what I'd suggest for now is disable batching and process one text at a time using `encode`.
Note that you can make it faster by using multiprocessing:

```python
num_proc = None  # Specify here the number of processes if you want to use multiprocessing. ex: num_proc = 4
encoded = train_ds.map(
    lambda example: {'tokens': t.encode(example['title'], max_length=1000)}, num_proc=num_proc
)
```

Thank you very much for the kind and precise suggestion!
I'm looking forward to seeing BertJapaneseTokenizer built into the "fast" tokenizers.

I tried `map` with multiprocessing as follows, and it worked!

```python
# There was a Pickle problem if I use `lambda` for multiprocessing
def encode(examples):
    return {'tokens': t.encode(examples['title'], max_length=1000)}

num_proc = 8
encoded = train_ds.map(encode, num_proc=num_proc)
```

Thank you again!

It seems to fail to process the final batch. This [colab](https://colab.research.google.com/drive/1_byLZRHwGP13PHMkJWo62Wp50S_Z2HMD?usp=sharing) can reproduce the error.

Code:

```python
# train_ds = Dataset(features: {
#     'title': Value(dtype='string', id=None),
#     'score': Value(dtype='float64', id=None)
# }, num_rows: 99999)

# suggested in #665
class PicklableTokenizer(BertJapaneseTokenizer):
    def __getstate__(self):
        state = dict(self.__dict__)
        state['do_lower_case'] = self.word_tokenizer.do_lower_case
        state['never_split'] = self.word_tokenizer.never_split
        del state['word_tokenizer']
        return state

    def __setstate(self):
        do_lower_case = state.pop('do_lower_case')
        never_split = state.pop('never_split')
        self.__dict__ = state
        self.word_tokenizer = MecabTokenizer(
            do_lower_case=do_lower_case, never_split=never_split
        )

t = PicklableTokenizer.from_pretrained('bert-base-japanese-whole-word-masking')

encoded = train_ds.map(
    lambda examples: {'tokens': t.encode(examples['title'], max_length=1000)}, batched=True, batch_size=1000
)
```

Error Message:

```
 99% 99/100 [00:22<00:00, 39.07ba/s]
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
<timed exec> in <module>

/usr/local/lib/python3.6/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1242                 fn_kwargs=fn_kwargs,
   1243                 new_fingerprint=new_fingerprint,
-> 1244                 update_data=update_data,
   1245             )
   1246         else:

/usr/local/lib/python3.6/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    151             "output_all_columns": self._output_all_columns,
    152         }
--> 153         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    154         if new_format["columns"] is not None:
    155             new_format["columns"] = list(set(new_format["columns"]) & set(out.column_names))

/usr/local/lib/python3.6/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    161             # Call actual function
    162
--> 163             out = func(self, *args, **kwargs)
    164
    165             # Update fingerprint of in-place transforms + update in-place history of transforms

/usr/local/lib/python3.6/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, update_data)
   1496                     if update_data:
   1497                         batch = cast_to_python_objects(batch)
-> 1498                         writer.write_batch(batch)
   1499             if update_data:
   1500                 writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file

/usr/local/lib/python3.6/site-packages/datasets/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    271             typed_sequence = TypedSequence(batch_examples[col], type=col_type, try_type=col_try_type)
    272             typed_sequence_examples[col] = typed_sequence
--> 273         pa_table = pa.Table.from_pydict(typed_sequence_examples)
    274         self.write_table(pa_table)
    275

/usr/local/lib/python3.6/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_pydict()

/usr/local/lib/python3.6/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_arrays()

/usr/local/lib/python3.6/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.validate()

/usr/local/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Column 4 named tokens expected length 999 but got length 1000
```


Dataset browser url is still https://huggingface.co/nlp/viewer/

Yes! might do it with @srush one of these days. Hopefully it won't break too many links (we can always redirect from old url to new)

This was fixed but forgot to close the issue. cc @lhoestq @yjernite

Thanks @jarednielsen!

Might be worth updating to https://huggingface.co/datasets/viewer/


Add features parameter to CSV

Add support for the `features` parameter when loading a csv dataset:

```python
from datasets import load_dataset, Features

features = Features({...})
csv_dataset = load_dataset("csv", data_files=["path/to/my/file.csv"], features=features)
```

I added tests to make sure that it is also compatible with the caching system

Fix #623