Add CoNLL-2003 shared task dataset

I think we should somewhere mention, that is the dataset in IOB2 tagging scheme, whereas the original dataset uses IOB1 :)

Indeed this is something we want to mention.

If would want to add more details about the IOB1->2 change, feel free to ignore my suggestions and edit the description + update the dataset_info

@lhoestq do you want me to update it or you'll update it. I am ok either way

The best would be to mention this change in the description and then update the dataset_info.json file.
Could you do that if you don't mind ?

Then it should be ready to merge :)

Thanks again for adding this dataset !

@lhoestq please check if 847addf is exactly what we want

Is the German task also part of this? If not, can it be accessed via the Datasets library?

Please consider adding CoNLL-2003 shared task dataset as it's beneficial for token classification tasks. The motivation behind this PR is the [PR](https://github.com/huggingface/transformers/pull/7041) in the transformers project. This dataset would be not only useful for the usual run-of-the-mill NER tasks but also for syntactic chunking and part-of-speech (POS) tagging.


add multi-proc to dataset dict

Add multi-proc to `DatasetDict`


Can you give us stats/information on your pandas DataFrame?

```
<class 'pandas.core.frame.DataFrame'>
Int64Index: 17136104 entries, 0 to 17136103
Data columns (total 6 columns):
 #   Column        Dtype
---  ------        -----
 0   item_id       int64
 1   item_titl     object
 2   start_price   float64
 3   shipping_fee  float64
 4   picture_url   object
 5   embeddings    object
dtypes: float64(2), int64(1), object(3)
memory usage: 915.2+ MB
```

Thanks and some more on the `embeddings` and `picture_url` would be nice as well (type and max lengths of the elements)

`embedding` is `np.array` of shape `(128,)`. `picture_url` is url, such as 'https://i.ebayimg.com/00/s/MTE5OVgxNjAw/z/ZOsAAOSwAG9fHQq5/$_12.JPG?set_id=880000500F;https://i.ebayimg.com/00/s/MTE5OVgxNjAw/z/OSgAAOSwokBfHQq8/$_12.JPG?set_id=880000500F'

It looks like a Pyarrow limitation.
I was able to reproduce the error with

```python
import pandas as pd
import numpy as np
import pyarrow as pa

 n = 1713614
df = pd.DataFrame.from_dict({"a": list(np.zeros((n, 128))), "b": range(n)})
pa.Table.from_pandas(df)
```

I also tried with 50% of the dataframe and it actually works.
I created an issue on Apache Arrow's JIRA [here](https://issues.apache.org/jira/browse/ARROW-9976)

One way to fix that would be to chunk the dataframe and concatenate arrow tables.

It looks like it's going to be fixed in pyarrow 2.0.0 :)

In the meantime I suggest to chunk big dataframes to create several small datasets, and then concatenate them using [concatenate_datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets)

Hi, I'm trying to load a dataset from Dataframe, but I get the error:
```bash
---------------------------------------------------------------------------
ArrowCapacityError                        Traceback (most recent call last)
<ipython-input-7-146b6b495963> in <module>
----> 1 dataset = Dataset.from_pandas(emb)

~/miniconda3/envs/dev/lib/python3.7/site-packages/nlp/arrow_dataset.py in from_pandas(cls, df, features, info, split)
    223         info.features = features
    224         pa_table: pa.Table = pa.Table.from_pandas(
--> 225             df=df, schema=pa.schema(features.type) if features is not None else None
    226         )
    227         return cls(pa_table, info=info, split=split)

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_pandas()

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/pandas_compat.py in dataframe_to_arrays(df, schema, preserve_index, nthreads, columns, safe)
    591         for i, maybe_fut in enumerate(arrays):
    592             if isinstance(maybe_fut, futures.Future):
--> 593                 arrays[i] = maybe_fut.result()
    594
    595     types = [x.type for x in arrays]

~/miniconda3/envs/dev/lib/python3.7/concurrent/futures/_base.py in result(self, timeout)
    426                 raise CancelledError()
    427             elif self._state == FINISHED:
--> 428                 return self.__get_result()
    429
    430             self._condition.wait(timeout)

~/miniconda3/envs/dev/lib/python3.7/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

~/miniconda3/envs/dev/lib/python3.7/concurrent/futures/thread.py in run(self)
     55
     56         try:
---> 57             result = self.fn(*self.args, **self.kwargs)
     58         except BaseException as exc:
     59             self.future.set_exception(exc)

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/pandas_compat.py in convert_column(col, field)
    557
    558         try:
--> 559             result = pa.array(col, type=type_, from_pandas=True, safe=safe)
    560         except (pa.ArrowInvalid,
    561                 pa.ArrowNotImplementedError,

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._ndarray_to_array()

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowCapacityError: List array cannot contain more than 2147483646 child elements, have 2147483648
```
My code is :
```python
from nlp import Dataset
dataset = Dataset.from_pandas(emb)
```


Load text file for RoBERTa pre-training.

Could you try
```python
load_dataset('text', data_files='test.txt',cache_dir="./", split="train")
```
?

`load_dataset` returns a dictionary by default, like {"train": your_dataset}

Hi @lhoestq
Thanks for your suggestion.

I tried
```
dataset = load_dataset('text', data_files='test.txt',cache_dir="./", split="train")
print(dataset)
dataset.set_format(type='torch',columns=["text"])
dataloader = torch.utils.data.DataLoader(dataset, batch_size=8)
next(iter(dataloader))
```

But it still doesn't work and got error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-388aca337e2f> in <module>
----> 1 next(iter(dataloader))

/Library/Python/3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    361
    362     def __next__(self):
--> 363         data = self._next_data()
    364         self._num_yielded += 1
    365         if self._dataset_kind == _DatasetKind.Iterable and \

/Library/Python/3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    401     def _next_data(self):
    402         index = self._next_index()  # may raise StopIteration
--> 403         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    404         if self._pin_memory:
    405             data = _utils.pin_memory.pin_memory(data)

/Library/Python/3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/Library/Python/3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/arrow_dataset.py in __getitem__(self, key)
   1069             format_columns=self._format_columns,
   1070             output_all_columns=self._output_all_columns,
-> 1071             format_kwargs=self._format_kwargs,
   1072         )
   1073

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/arrow_dataset.py in _getitem(self, key, format_type, format_columns, output_all_columns, format_kwargs)
   1056                 format_columns=format_columns,
   1057                 output_all_columns=output_all_columns,
-> 1058                 format_kwargs=format_kwargs,
   1059             )
   1060         return outputs

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/arrow_dataset.py in _convert_outputs(self, outputs, format_type, format_columns, output_all_columns, format_kwargs)
    872                     continue
    873                 if format_columns is None or k in format_columns:
--> 874                     v = map_nested(command, v, **map_nested_kwargs)
    875                 output_dict[k] = v
    876         return output_dict

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types)
    214     # Singleton
    215     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 216         return function(data_struct)
    217
    218     disable_tqdm = bool(logger.getEffectiveLevel() > INFO)

/Library/Python/3.7/site-packages/datasets-0.4.0-py3.7.egg/datasets/arrow_dataset.py in command(x)
    833                     if x.dtype == np.object:  # pytorch tensors cannot be instantied from an array of objects
    834                         return [map_nested(command, i, **map_nested_kwargs) for i in x]
--> 835                 return torch.tensor(x, **format_kwargs)
    836
    837         elif format_type == "tensorflow":

TypeError: new(): invalid data type 'str'
```

I found type can be ['numpy', 'torch', 'tensorflow', 'pandas'] only, how can I deal with the string type?

You need to tokenize the string inputs to convert them in integers before you can feed them to a pytorch dataloader.

You can read the quicktour of the datasets or the transformers libraries to know more about that:
- transformers: https://huggingface.co/transformers/quicktour.html
- dataset: https://huggingface.co/docs/datasets/quicktour.html

Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
But finally got it working. This is what I did after looking into the documentation.

1. split the whole dataset file into smaller files
```bash
mkdir ./shards
split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_
````
2. Pass paths of small data files to `load_dataset`
```python
files = glob.glob('shards/*')
from datasets import load_dataset
dataset = load_dataset('text', data_files=files, split='train')
```
(On passing the whole dataset file (11GB) directly to `load_dataset` was resulting into RAM issue)

3. Tokenization
```python
def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')
dataset = dataset.map(encode, batched=True)
dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
```
 Now you can pass `dataset` to `Trainer` or `pytorch DataLoader`
```python
dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
next(iter(dataloader))
```
Hope this helps

Thanks, @thomwolf  and @sipah00 ,

I tried to implement your suggestions in my scripts.
Now, I am facing some connection time-out error. I am using my local file, I have no idea why the module request s3 database.

The log is:
```
Traceback (most recent call last):
  File "/home/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    raise err
  File "/home/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    timeout=timeout
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

Traceback (most recent call last):
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    chunked=chunked,
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/text/text.py (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection obj
ect at 0x7fff401e0e48>: Failed to establish a new connection: [Errno 110] Connection timed out',))

Traceback (most recent call last):
  File "/scratch/roberta_emohash/run_language_modeling.py", line 1019, in <module>
    main()
  File "/scratch/roberta_emohash/run_language_modeling.py", line 962, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/roberta_emohash/run_language_modeling.py", line 177, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/roberta_emohash/run_language_modeling.py", line 117, in HG_Datasets
    dataset = load_dataset('text', data_files=files, cache_dir = args.data_cache_dir, split="train")
  File "/arc/project/evn_py36/datasets/datasets/src/datasets/load.py", line 590, in load_dataset
    self._validate_conn(conn)
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/.local/lib/python3.6/site-packages/urllib3/connection.py", line 300, in connect
    conn = self._new_conn()
  File "/home/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fff401e0da0>: Failed to establish a new connection: [Errno 110] Connection timed out

```

Do you have any experience on this issue?

No, I didn't encounter this problem, it seems to me a network problem

I noticed this is because I use a cloud server where does not provide for connections from our standard compute nodes to outside resources.

For the `datasets` package, it seems that if the loading script is not already cached in the library it will attempt to connect to an AWS resource to download the dataset loading script.

I am wondering why the package works in this way. Do you have any suggestions to solve this issue?

I solved the above issue by downloading text.py manually and passing the path to the `load_dataset` function.

Now, I have a new issue with the Read-only file system.

The error is:
```
I0916 22:14:38.453380 140737353971520 filelock.py:274] Lock 140734268996072 acquired on /scratch/chiyuzh/roberta/text.py.lock
Found main folder for dataset /scratch/chiyuzh/roberta/text.py at /home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text
Creating specific version folder for dataset /scratch/chiyuzh/roberta/text.py at /home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014
I0916 22:14:38.530371 140737353971520 filelock.py:318] Lock 140734268996072 released on /scratch/chiyuzh/roberta/text.py.lock
Traceback (most recent call last):
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 1019, in <module>
    main()
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 962, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 177, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 117, in HG_Datasets
    dataset = load_dataset('/scratch/chiyuzh/roberta/text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")
  File "/arc/project/chiyuzh/evn_py36/datasets/src/datasets/load.py", line 590, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/arc/project/chiyuzh/evn_py36/datasets/src/datasets/load.py", line 385, in prepare_module
    os.makedirs(hash_folder_path)
  File "/project/chiyuzh/evn_py36/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014'

```

I installed datasets at /project/chiyuzh/evn_py36/datasets/src where is a writable directory.
I also tried change the environment variables to the writable directory:
`export HF_MODULES_PATH=/project/chiyuzh/evn_py36/datasets/cache_dir/`
`export HF_DATASETS_CACHE=/project/chiyuzh/evn_py36/datasets/cache_dir/`

In my scripts, I also changed to:
`dataset = load_dataset('/scratch/chiyuzh/roberta/text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")`
`data_cache_dir = $TMPDIR/data/` that also a writable directory.

But it still try to make directory at /home/chiyuzh/.cache/huggingface/modules/.
Do you have any idea about this issue? @thomwolf

> Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
> But finally got it working. This is what I did after looking into the documentation.
>
> 1. split the whole dataset file into smaller files
>
> ```shell
> mkdir ./shards
> split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_
> ```
>
> 1. Pass paths of small data files to `load_dataset`
>
> ```python
> files = glob.glob('shards/*')
> from datasets import load_dataset
> dataset = load_dataset('text', data_files=files, split='train')
> ```
>
> (On passing the whole dataset file (11GB) directly to `load_dataset` was resulting into RAM issue)
>
> 1. Tokenization
>
> ```python
> def encode(examples):
>   return tokenizer(examples['text'], truncation=True, padding='max_length')
> dataset = dataset.map(encode, batched=True)
> dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
> ```
>
> Now you can pass `dataset` to `Trainer` or `pytorch DataLoader`
>
> ```python
> dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
> next(iter(dataloader))
> ```
>
> Hope this helps

When I run 'dataset = dataset.map(encode, batched=True)',
I encountered a problem like this:

> Testing the mapped function outputs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in map
    for k, dataset in self.items()
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in <dictcomp>
    for k, dataset in self.items()
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1224, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1195, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "<stdin>", line 3, in encode
TypeError: __init__() takes 1 positional argument but 2 were given

> > Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
> > But finally got it working. This is what I did after looking into the documentation.
> >
> > 1. split the whole dataset file into smaller files
> >
> > ```shell
> > mkdir ./shards
> > split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_
> > ```
> >
> >
> >
> > 1. Pass paths of small data files to `load_dataset`
> >
> > ```python
> > files = glob.glob('shards/*')
> > from datasets import load_dataset
> > dataset = load_dataset('text', data_files=files, split='train')
> > ```
> >
> >
> > (On passing the whole dataset file (11GB) directly to `load_dataset` was resulting into RAM issue)
> >
> > 1. Tokenization
> >
> > ```python
> > def encode(examples):
> >   return tokenizer(examples['text'], truncation=True, padding='max_length')
> > dataset = dataset.map(encode, batched=True)
> > dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
> > ```
> >
> >
> > Now you can pass `dataset` to `Trainer` or `pytorch DataLoader`
> > ```python
> > dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
> > next(iter(dataloader))
> > ```
> >
> >
> > Hope this helps
>
> When I run 'dataset = dataset.map(encode, batched=True)',
> I encountered a problem like this:
>
> > Testing the mapped function outputs
> > Traceback (most recent call last):
> > File "", line 1, in
> > File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in map
> > for k, dataset in self.items()
> > File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in
> > for k, dataset in self.items()
> > File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1224, in map
> > update_data = does_function_return_dict(test_inputs, test_indices)
> > File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1195, in does_function_return_dict
> > function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
> > File "", line 3, in encode
> > TypeError: **init**() takes 1 positional argument but 2 were given

What is your encoder function?

> ```python
> def encode(examples):
>   return tokenizer(examples['text'], truncation=True, padding='max_length')
> ```

It is the same as suggested:

> def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')

> > ```python
> > def encode(examples):
> >   return tokenizer(examples['text'], truncation=True, padding='max_length')
> > ```
>
> It is the same as suggested:
>
> > def encode(examples):
> > return tokenizer(examples['text'], truncation=True, padding='max_length')

Do you use this function in a `class` object?

init() takes 1 positional argument but 2 were given. I guess the additional argument is self?

> > > ```python
> > > def encode(examples):
> > >   return tokenizer(examples['text'], truncation=True, padding='max_length')
> > > ```
> >
> >
> > It is the same as suggested:
> > > def encode(examples):
> > > return tokenizer(examples['text'], truncation=True, padding='max_length')
>
> Do you use this function in a `class` object?
>
> init() takes 1 positional argument but 2 were given. I guess the additional argument is self?

Thanks for your reply.
Could you provide some simple example here?
Currently, I do not use this function in a class object.
I think you are right and I was wondering how to construct this class.
I try to modify it based on transformers' LineByLineTextDataset. Am I correct?

> class LineByLineTextDataset(Dataset):
    """
    This will be superseded by a framework-agnostic approach
    soon.
    """

    def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
        assert os.path.isfile(file_path), f"Input file path {file_path} not found"
        # Here, we do not cache the features, operating under the assumption
        # that we will soon use fast multithreaded tokenizers from the
        # `tokenizers` repo everywhere =)
        #logger.info("Creating features from dataset file at %s", file_path)
        #with open(file_path, encoding="utf-8") as f:
        #    lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
        #batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)

	import glob
	files = glob.glob('/home/mtzhang111/fairseq/cs_doc/shards/shard_003*')
	from datasets import load_dataset
	dataset = load_dataset('text', data_files=files)
        batch_encoding= dataset.map(encode, batched=True)
        self.examples = batch_encoding["input_ids"]

    def encode(examples):
        return tokenizer(examples['text'], truncation=True, padding='max_length')

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i) -> torch.Tensor:
        return torch.tensor(self.examples[i], dtype=torch.long)

> > > > ```python
> > > > def encode(examples):
> > > >   return tokenizer(examples['text'], truncation=True, padding='max_length')
> > > > ```
> > >
> > >
> > > It is the same as suggested:
> > > > def encode(examples):
> > > > return tokenizer(examples['text'], truncation=True, padding='max_length')
> >
> >
> > Do you use this function in a `class` object?
> > init() takes 1 positional argument but 2 were given. I guess the additional argument is self?
>
> Thanks for your reply.
> Could you provide some simple example here?
> Currently, I do not use this function in a class object.
> I think you are right and I was wondering how to construct this class.
> I try to modify it based on transformers' LineByLineTextDataset. Am I correct?
>
> > class LineByLineTextDataset(Dataset):
> > """
> > This will be superseded by a framework-agnostic approach
> > soon.
> > """
>
> ```
> def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
>     assert os.path.isfile(file_path), f"Input file path {file_path} not found"
>     # Here, we do not cache the features, operating under the assumption
>     # that we will soon use fast multithreaded tokenizers from the
>     # `tokenizers` repo everywhere =)
>     #logger.info("Creating features from dataset file at %s", file_path)
>     #with open(file_path, encoding="utf-8") as f:
>     #    lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
>     #batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)
>
> import glob
> files = glob.glob('/home/mtzhang111/fairseq/cs_doc/shards/shard_003*')
> from datasets import load_dataset
> dataset = load_dataset('text', data_files=files)
>     batch_encoding= dataset.map(encode, batched=True)
>     self.examples = batch_encoding["input_ids"]
>
>
> def encode(examples):
>     return tokenizer(examples['text'], truncation=True, padding='max_length')
>
> def __len__(self):
>     return len(self.examples)
>
> def __getitem__(self, i) -> torch.Tensor:
>     return torch.tensor(self.examples[i], dtype=torch.long)
> ```

I am also struggling with this adaptation.
I am not sure whether I am right.

I think you don't need to construct `class LazyLineByLineTextDataset(Dataset)` at all.
torch.utils.data.Dataset is a generator.

Now, we use `dataset = dataset.map(encode, batched=True)` as a generator. So we just pass dataset to torch.utils.data.DataLoader.

@chiyuzhang94 Thanks for your reply. After some changes, currently, I managed to make the data loading process running.
I published it in case you might want to take a look. Thanks for your help!
https://github.com/shizhediao/Transformers_TPU

Hi @shizhediao ,

Thanks! It looks great!

But my problem still is the cache directory is a read-only file system.
[As I mentioned](https://github.com/huggingface/datasets/issues/610#issuecomment-693912285), I tried to change the cache directory but it didn't work.

Do you have any suggestions?

> I installed datasets at /project/chiyuzh/evn_py36/datasets/src where is a writable directory.
> I also tried change the environment variables to the writable directory:
> `export HF_MODULES_PATH=/project/chiyuzh/evn_py36/datasets/cache_dir/`

I think it is `HF_MODULES_CACHE` and not `HF_MODULES_PATH` @chiyuzhang94 .
Could you try again and let me know if it fixes your issue ?

We should probably add a section in the doc on the caching system with the env variables in particular.

Hi @thomwolf , @lhoestq ,

Thanks for your suggestions. With the latest version of this package, I can load text data without Internet.

But I found the speed of dataset loading is very slow.

My scrips like this:
```
    def token_encode(examples):
        tokenizer_out = tokenizer(examples['text'], truncation=True,  padding="max_length", add_special_tokens=True, max_length=args.block_size)
        return tokenizer_out

    path = Path(file_path)
    files = sorted(path.glob('*'))
    dataset = load_dataset('./text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")
    dataset = dataset.map(token_encode, batched=True)

    dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
```

I have 1,123,870,657 lines in my input directory.
I can find the processing speed as following. It is very slow.
```
| 13/1123871 [00:02<62:37:39,  4.98ba/s]^M  0%|
| 14/1123871 [00:03<61:27:31,  5.08ba/s]^M  0%|
| 15/1123871 [00:03<66:34:19,  4.69ba/s]^M  0%|
| 16/1123871 [00:03<68:25:01,  4.56ba/s]^M  0%|
| 17/1123871 [00:03<72:00:03,  4.34ba/s]^M  0%|
```
Do you have any suggestions to accelerate this loading process?

You can use multiprocessing by specifying `num_proc=` in `.map()`

Also it looks like you have `1123871` batches of 1000 elements (default batch size), i.e. 1,123,871,000 lines in total.
Am I right ?

> You can use multiprocessing by specifying `num_proc=` in `.map()`
>
> Also it looks like you have `1123871` batches of 1000 elements (default batch size), i.e. 1,123,871,000 lines in total.
> Am I right ?

Hi @lhoestq ,

Thanks. I will try it.

You are right. I have 1,123,870,657 lines totally in the path. I split the large file into 440 small files. Each file has 2,560,000 lines.

I have another question. Because I am using a cloud server where only allows running a job up to 7 days. Hence, I need to resume my model every week. If the script needs to load and process the dataset every time. It is very low efficient based on the current processing speed. Is it possible that I process the dataset once and use the process cache to in the future work?

Hi @lhoestq ,

I tried to use multi-processor, but I got errors as follow:
Because I am using python distributed training, it seems some conflicts with the distributed job.

Do you have any suggestions?
```
I0925 10:19:35.603023 140737353971520 filelock.py:318] Lock 140737229443368 released on /tmp/pbs.1120510.pbsha.ib.sockeye/cache/_tmp_pbs.1120510.pbsha.ib.sockeye_cache_text_default-7fb934ed6fac5d01_0.0.0_512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7
fcc649178b014.lock
Traceback (most recent call last):
  File "/scratch/chiyuzh/roberta/run_language_modeling.py", line 1024, in <module>
    main()
  File "/scratch/chiyuzh/roberta/run_language_modeling.py", line 967, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/chiyuzh/roberta/run_language_modeling.py", line 180, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/chiyuzh/roberta/run_language_modeling.py", line 119, in HG_Datasets
    dataset = dataset.map(token_encode, batched=True, batch_size = 10000, num_proc = 16)
  File "/project/chiyuzh/evn_py36/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1287, in map
    transformed_shards = [r.get() for r in results]
  File "/project/chiyuzh/evn_py36/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1287, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/project/chiyuzh/evn_py36/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/project/chiyuzh/evn_py36/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/project/chiyuzh/evn_py36/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/project/chiyuzh/evn_py36/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'HG_Datasets.<locals>.token_encode'
```

For multiprocessing, the function given to `map` must be picklable.
Maybe you could try to define `token_encode` outside `HG_Datasets` ?

Also maybe #656 could make functions defined locally picklable for multiprocessing, once it's merged.

> I have another question. Because I am using a cloud server where only allows running a job up to 7 days. Hence, I need to resume my model every week. If the script needs to load and process the dataset every time. It is very low efficient based on the current processing speed. Is it possible that I process the dataset once and use the process cache to in the future work?

Feel free to save your processed dataset using `dataset.save_to_disk("path/to/save/directory")`.

Then you'll be able to reload it again using
```python
from datasets import load_from_disk

dataset = load_from_disk("path/to/save/directory")
```

Hi @lhoestq ,

Thanks for your suggestion.
I tried to process the dataset and save it to disk.
I have 1.12B samples in the raw dataset. I used 16 processors.
I run this process job for 7 days. But it didn't finish. I don't why the processing is such slow.

The log shows that some processors (\#12, \#14, \#15) are very slow. The different processor has a different speed. These slow processors look like a bottleneck.

Could you please give me any suggestion to improve the processing speed?

Thanks.
Chiyu

Here is my code:
```
def token_encode(examples):
        tokenizer_out = tokenizer(examples['text'], truncation=True,  padding="max_length", add_special_tokens=True, max_length=args.block_size)
        return tokenizer_out

path = Path(file_path)
files = sorted(path.glob('*'))
dataset = load_dataset('./text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")
dataset = dataset.map(token_encode, batched=True, batch_size = 16384, num_proc = 16)
dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
dataset.save_to_disk(output_dir)
```
Here is the log.
```
^M#6:   1%|▏         | 59/4288 [55:10<66:11:58, 56.35s/ba]
^M#1:   8%|▊         | 356/4288 [55:39<10:40:02,  9.77s/ba]
^M#2:   5%|▍         | 210/4288 [55:33<17:47:19, 15.70s/ba]
^M#0:  19%|█▉        | 836/4288 [55:53<4:08:56,  4.33s/ba]
^M#0:  20%|█▉        | 837/4288 [55:57<4:01:52,  4.21s/ba]
^M#1:   8%|▊         | 357/4288 [55:48<10:38:09,  9.74s/ba]
^M#0:  20%|█▉        | 838/4288 [56:01<4:02:56,  4.23s/ba]
^M#3:   4%|▎         | 155/4288 [55:43<24:41:20, 21.51s/ba]
^M#0:  20%|█▉        | 839/4288 [56:05<4:04:48,  4.26s/ba]
^M#12:   1%|          | 29/4288 [54:50<133:20:53, 112.72s/ba]
^M#2:   5%|▍         | 211/4288 [55:48<17:40:33, 15.61s/ba]
^M#14:   0%|          | 2/4288 [04:24<157:17:50, 132.12s/ba]
^M#15:   0%|          | 1/4288 [02:24<172:11:37, 144.60s/ba]
```

Hi !

As far as I can tell, there could be several reasons for your processes to have different speeds:
- some parts of your dataset have short passages while some have longer passages, that take more time to be processed
- OR there are other processes running that prevent some of them to run at full speed
- OR the value of `num_proc` is higher than the number of actual processes that you can run in parallel at full speed.

So I'd suggest you to check that you have nothing else running in parallel to your processing job, and also maybe take a look at the slow parts of the datasets.
When doing multiprocessing, the dataset is sharded in `num_proc` contiguous parts that are processed individually in each process. If you want to take a look at the dataset processed in the 12th shard of 16 for example, you can do:

```python
my_shard = dataset.shard(num_shards=16, index=12, contiguous=True)
```

Hope this helps, let me know if you find what is causing this slow down.

> Do you use a fast or a slow tokenizer from the `transformers` library @chiyuzhang94?

Hi @thomwolf ,
 I use this:
```
from transformers import
AutoTokenizer.from_pretrained(args.model_name_or_path, cache_dir=args.cache_dir)
```

I guess this is a slow one, let me explore the fast tokenizer.

> Hi !
>
> As far as I can tell, there could be several reasons for your processes to have different speeds:
>
> * some parts of your dataset have short passages while some have longer passages, that take more time to be processed
> * OR there are other processes running that prevent some of them to run at full speed
> * OR the value of `num_proc` is higher than the number of actual processes that you can run in parallel at full speed.
>
> So I'd suggest you to check that you have nothing else running in parallel to your processing job, and also maybe take a look at the slow parts of the datasets.
> When doing multiprocessing, the dataset is sharded in `num_proc` contiguous parts that are processed individually in each process. If you want to take a look at the dataset processed in the 12th shard of 16 for example, you can do:
>
> ```python
> my_shard = dataset.shard(num_shards=16, index=12, contiguous=True)
> ```
>
> Hope this helps, let me know if you find what is causing this slow down.

Hi @lhoestq ,

Thanks for your suggestions.
I don't think my problem is due to any one of these seasons.

1.  I have 1,123,870,657 lines totally in the path. I split the large file into 440 small files. Each file has 2,560,000 lines. The last file is smaller a little bit. But they are similar. I randomly shuffled all the 1,123,870,657 lines. Hence, the sequences should also be similar across all the files.

2. I run this script on the entire node. I requested all the resources on the nodes (40 CPUs, 384GB memory). Hence, these were not any other processes.

 3. As I say, the node has 40 CPUs, but I set num_proc = 16. This should not be a problem.

Hi @thomwolf
I am using `RobertaTokenizerFast` now.

But the speed is still imbalanced, some processors are still slow.
Here is the part of the log. #0 is always much fast than lower rank processors.

```
#15:   3%|▎         | 115/3513 [3:18:36<98:01:33, 103.85s/ba]
#2:  24%|██▍       | 847/3513 [3:20:43<11:06:49, 15.01s/ba]
#1:  37%|███▋      | 1287/3513 [3:20:52<6:19:02, 10.22s/ba]
#0:  72%|███████▏  | 2546/3513 [3:20:52<1:51:03,  6.89s/ba]
#3:  18%|█▊        | 617/3513 [3:20:36<15:50:30, 19.69s/ba]
#0:  73%|███████▎  | 2547/3513 [3:20:59<1:50:25,  6.86s/ba]
#1:  37%|███▋      | 1288/3513 [3:21:02<6:21:13, 10.28s/ba]
#7:   7%|▋         | 252/3513 [3:20:09<44:09:03, 48.74s/ba]
#12:   4%|▍         | 144/3513 [3:19:19<78:00:54, 83.36s/ba]
#4:  14%|█▍        | 494/3513 [3:20:37<20:46:06, 24.77s/ba]
#0:  73%|███████▎  | 2548/3513 [3:21:06<1:49:26,  6.80s/ba]
#2:  24%|██▍       | 848/3513 [3:20:58<11:06:17, 15.00s/ba]
```
Here is my script related to the datasets processing,

```
tokenizer = RobertaTokenizerFast.from_pretrained(args.model_name_or_path, cache_dir=args.cache_dir)

def token_encode(examples):
    tokenizer_out = tokenizer(examples['text'], truncation=True,  padding="max_length", add_special_tokens=True, max_length=128)
    return tokenizer_out

def HG_Datasets(tokenizer, file_path, args):

    path = Path(file_path)
    files = sorted(path.glob('*'))
    dataset = load_dataset('./text.py', data_files=files, cache_dir = ""./, split="train")
    dataset = dataset.map(token_encode, batched=True, batch_size = 20000, num_proc = 16)

    dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
    return dataset

```
I have 1,123,870,657 lines totally in the path. I split the large file into 440 small files. Each file has 2,560,000 lines.

Could you please give any suggestion? Thanks very much!!

I migrate my question from https://github.com/huggingface/transformers/pull/4009#issuecomment-690039444

I tried to train a Roberta from scratch using transformers. But I got OOM issues with loading a large text file.
According to the suggestion from @thomwolf , I tried to implement `datasets` to load my text file. This test.txt is a simple sample where each line is a sentence.
```
from datasets import load_dataset
dataset = load_dataset('text', data_files='test.txt',cache_dir="./")
dataset.set_format(type='torch',columns=["text"])
dataloader = torch.utils.data.DataLoader(dataset, batch_size=8)
next(iter(dataloader))
```

But dataload cannot yield sample and error is:
```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-12-388aca337e2f> in <module>
----> 1 next(iter(dataloader))

/Library/Python/3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    361
    362     def __next__(self):
--> 363         data = self._next_data()
    364         self._num_yielded += 1
    365         if self._dataset_kind == _DatasetKind.Iterable and \

/Library/Python/3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    401     def _next_data(self):
    402         index = self._next_index()  # may raise StopIteration
--> 403         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    404         if self._pin_memory:
    405             data = _utils.pin_memory.pin_memory(data)

/Library/Python/3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/Library/Python/3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

KeyError: 0
```

`dataset.set_format(type='torch',columns=["text"])` returns a log says:
```
Set __getitem__(key) output type to torch for ['text'] columns (when key is int or slice) and don't output other (un-formatted) columns.
```

I noticed the dataset is `DatasetDict({'train': Dataset(features: {'text': Value(dtype='string', id=None)}, num_rows: 44)})`.
Each sample can be accessed by `dataset["train"]["text"]` instead of `dataset["text"]`.

Could you please give me any suggestions on how to modify this code to load the text file?

Versions:
Python version 3.7.3
PyTorch version 1.6.0
TensorFlow version 2.3.0
datasets version: 1.0.1


Update GLUE URLs (now hosted on FB)

Thanks for opening this PR :)

We changed the name of the lib from nlp to datasets yesterday.
Could you rebase from master and re-generate the dataset_info.json file to fix the name changes ?

Rebased changes here: https://github.com/huggingface/datasets/pull/626

NYU is switching dataset hosting from Google to FB. This PR closes https://github.com/huggingface/datasets/issues/608 and is necessary for https://github.com/jiant-dev/jiant/issues/161. This PR updates the data URLs based on changes made in https://github.com/nyu-mll/jiant/pull/1112.


Don't use the old NYU GLUE dataset URLs

Feel free to open the PR ;)
Thanks for updating the dataset_info.json file !

NYU is switching dataset hosting from Google to FB. Initial changes to `datasets` are in https://github.com/jeswan/nlp/commit/b7d4a071d432592ded971e30ef73330529de25ce. What tests do you suggest I run before opening a PR?

See: https://github.com/jiant-dev/jiant/issues/161 and https://github.com/nyu-mll/jiant/pull/1112


Add transmit_format wrapper and tests

Same as #605 but using a decorator on-top of dataset transforms that are not in place


Quick fix :)

`nlp` => `datasets`


[Datasets] Transmit format to children

Transmit format to children obtained when processing a dataset.

Added a test.

When concatenating datasets, if the formats are disparate, the concatenated dataset has a format reset to defaults.


Update bucket prefix

cc @julien-c


Set scripts version to master

By default the scripts version is master, so that if the library is installed with
```
pip install git+http://github.com/huggingface/nlp.git
```
or
```
git clone http://github.com/huggingface/nlp.git
pip install -e ./nlp
```

will use the latest scripts, and not the ones from the previous version.


apply offset to indices in multiprocessed map

Fix #597

I fixed the indices by applying an offset.
I added the case to our tests to make sure it doesn't happen again.

I also added the message proposed by @thomwolf in #597

```python
>>> d.select(range(10)).map(fn, with_indices=True, batched=True, num_proc=2, load_from_cache_file=False)
Done writing 10 indices in 80 bytes .
Testing the mapped function outputs
[0, 1]
Testing finished, running the mapping function on the dataset
Done writing 5 indices in 41 bytes .
Done writing 5 indices in 41 bytes .
Spawning 2 processes
[0, 1, 2, 3, 4]
[5, 6, 7, 8, 9]
#0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 377.90ba/s]
#1: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 378.92ba/s]
Concatenating 2 shards from multiprocessing

# Dataset(features: {'label': ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None), 'text': Value(dtype='string', id=None)}, num_rows: 10)
```


check if trasnformers has PreTrainedTokenizerBase

Fix #598


Pickling error when loading dataset

When I change from python3.6 to python3.8, it works!

Does it work when you install `nlp` from source on python 3.6?

I wasn't able to reproduce on google colab (python 3.6.9 as well) with

pickle==4.0
dill=0.3.2
transformers==3.1.0
datasets=1.0.1 (also tried nlp 0.4.0)

If I try

```python
from datasets import load_dataset  # or from nlp
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                        truncation=True, max_length=512), batched=True)
dataset.set_format(type='torch', columns=['input_ids'])
```
It runs without error

Closing since it looks like it's working on >= 3.6.9
Feel free to re-open if you have other questions :)

Hi,

I modified line 136 in the original [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py) as:

```
# line 136: return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                        truncation=True, max_length=args.block_size), batched=True)
dataset.set_format(type='torch', columns=['input_ids'])
return dataset
```

When I run this with transformers (3.1.0) and nlp (0.4.0), I get the following error:

```
Traceback (most recent call last):
  File "src/run_language_modeling.py", line 319, in <module>
    main()
  File "src/run_language_modeling.py", line 248, in main
    get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None
  File "src/run_language_modeling.py", line 139, in get_dataset
    dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True, truncation=True, max_length=args.block_size), batched=True)
  File "/data/nlp/src/nlp/arrow_dataset.py", line 1136, in map
    new_fingerprint=new_fingerprint,
  File "/data/nlp/src/nlp/fingerprint.py", line 158, in wrapper
    self._fingerprint, transform, kwargs_for_fingerprint
  File "/data/nlp/src/nlp/fingerprint.py", line 105, in update_fingerprint
    hasher.update(transform_args[key])
  File "/data/nlp/src/nlp/fingerprint.py", line 57, in update
    self.m.update(self.hash(value).encode("utf-8"))
  File "/data/nlp/src/nlp/fingerprint.py", line 53, in hash
    return cls.hash_default(value)
  File "/data/nlp/src/nlp/fingerprint.py", line 46, in hash_default
    return cls.hash_bytes(dumps(value))
  File "/data/nlp/src/nlp/utils/py_utils.py", line 362, in dumps
    dump(obj, file)
  File "/data/nlp/src/nlp/utils/py_utils.py", line 339, in dump
    Pickler(file, recurse=True).dump(obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 446, in dump
    StockPickler.dump(self, obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 1438, in save_function
    obj.__dict__, fkwdefaults), obj=obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 1170, in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 736, in save_tuple
    save(element)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 605, in save_reduce
    save(cls)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 1365, in save_type
    obj.__bases__, _dict), obj=obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 933, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/root/miniconda3/envs/py3.6/lib/python3.6/site-packages/dill/_dill.py", line 933, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 507, in save
    self.save_global(obj, rv)
  File "/root/miniconda3/envs/py3.6/lib/python3.6/pickle.py", line 927, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle typing.Union[str, NoneType]: it's not the same object as typing.Union
```


Add MATINF dataset

Hi ! sorry for the late response

Could you try to rebase from master ? We changed the named of the library last week so you have to include this change in your code.

Can you give me more details about the error you get when running the cli command ?

Note that in case of a manual download you have to specify the directory where you downloaded the data with `--data_dir <path/to/the/directory>`

I fucked up the Git rebase lol. Closing it.

@lhoestq The command to create metadata failed. I guess it's because the zip is not downloaded from a remote address? How to solve that? Also the CI fails and I don't know how to fix that :(


The current version of the package on github has an error when loading dataset

Thanks for reporting !
Which version of transformers are you using ?
It looks like it doesn't have the PreTrainedTokenizerBase class

I was using transformer 2.9. And I switch to the latest transformer package. Everything works just fine!!

Thanks for helping! I should look more carefully next time. Didn't realize loading the data part requires using tokenizer.

Yes it shouldn’t fail with older version of transformers since this is only a special feature to make caching more efficient when using transformers for tokenization.
We’ll update this.

Instead of downloading the package from pip, downloading the version from source will result in an error when loading dataset (the pip version is completely fine):

To recreate the error:
First, installing nlp directly from source:
```
git clone https://github.com/huggingface/nlp.git
cd nlp
pip install -e .
```
Then run:
```
from nlp import load_dataset
dataset = load_dataset('wikitext', 'wikitext-2-v1',split = 'train')
```
will give error:

```
>>> dataset = load_dataset('wikitext', 'wikitext-2-v1',split = 'train')
Checking /home/zeyuy/.cache/huggingface/datasets/84a754b488511b109e2904672d809c041008416ae74e38f9ee0c80a8dffa1383.2e21f48d63b5572d19c97e441fbb802257cf6a4c03fbc5ed8fae3d2c2273f59e.py for additional imports.
Found main folder for dataset https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/wikitext.py at /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext
Found specific version folder for dataset https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/wikitext.py at /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d
Found script file from https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/wikitext.py to /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d/wikitext.py
Found dataset infos file from https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/dataset_infos.json to /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d/dataset_infos.json
Found metadata file for dataset https://raw.githubusercontent.com/huggingface/nlp/0.4.0/datasets/wikitext/wikitext.py at /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d/wikitext.json
Loading Dataset Infos from /home/zeyuy/.cache/huggingface/modules/nlp_modules/datasets/wikitext/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d
Overwrite dataset info from restored data version.
Loading Dataset info from /home/zeyuy/.cache/huggingface/datasets/wikitext/wikitext-2-v1/1.0.0/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d
Reusing dataset wikitext (/home/zeyuy/.cache/huggingface/datasets/wikitext/wikitext-2-v1/1.0.0/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d)
Constructing Dataset for split train, from /home/zeyuy/.cache/huggingface/datasets/wikitext/wikitext-2-v1/1.0.0/5de6e79516446f747fcccc09aa2614fa159053b75909594d28d262395f72d89d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/load.py", line 600, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/builder.py", line 611, in as_dataset
    datasets = utils.map_nested(
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/utils/py_utils.py", line 216, in map_nested
    return function(data_struct)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/builder.py", line 631, in _build_single_dataset
    ds = self._as_dataset(
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/builder.py", line 704, in _as_dataset
    return Dataset(**dataset_kwargs)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/arrow_dataset.py", line 188, in __init__
    self._fingerprint = generate_fingerprint(self)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/fingerprint.py", line 91, in generate_fingerprint
    hasher.update(key)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/fingerprint.py", line 57, in update
    self.m.update(self.hash(value).encode("utf-8"))
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/fingerprint.py", line 53, in hash
    return cls.hash_default(value)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/fingerprint.py", line 46, in hash_default
    return cls.hash_bytes(dumps(value))
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/utils/py_utils.py", line 361, in dumps
    with _no_cache_fields(obj):
  File "/home/zeyuy/miniconda3/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/zeyuy/transformers/examples/language-modeling/nlp/src/nlp/utils/py_utils.py", line 348, in _no_cache_fields
    if isinstance(obj, tr.PreTrainedTokenizerBase) and hasattr(obj, "cache") and isinstance(obj.cache, dict):
AttributeError: module 'transformers' has no attribute 'PreTrainedTokenizerBase'

```


Indices incorrect with multiprocessing

I fixed a bug that could cause this issue earlier today. Could you pull the latest version and try again ?

Still the case on master.
I guess we should have an offset in the multi-procs indeed (hopefully it's enough).

Also, side note is that we should add some logging before the "test" to say we are testing the function otherwise its confusing for the user to see two outputs I think. Proposal (see the "Testing the mapped function outputs:" lines):
```
>>> d.select(range(10)).map(fn, with_indices=True, batched=True, num_proc=2)
Done writing 10 indices in 80 bytes .
Done writing 5 indices in 41 bytes .
Done writing 5 indices in 41 bytes .
Spawning 2 processes
Testing the mapped function outputs:
inds: [0, 1]
inds: [0, 1]
Testing finished, running the mapped function on the dataset:
#0:   0%|                                                                                | 0/1 [00:00<?, ?ba/s]
inds: [0, 1, 2, 3, 4]                                                                                inds: [0, 1, 2, 3, 4]                                                                                | 0/1 [00:00<?, ?ba/s]
#0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1321.04ba/s]
#1: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1841.22ba/s]
Concatenating 2 shards from multiprocessing
Dataset(features: {'text': Value(dtype='string', id=None), 'label': ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None)}, num_rows: 10)
```

When `num_proc` > 1, the indices argument passed to the map function is incorrect:

```python
d = load_dataset('imdb', split='test[:1%]')

def fn(x, inds):
    print(inds)
    return x

d.select(range(10)).map(fn, with_indices=True, batched=True)
# [0, 1]
# [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

d.select(range(10)).map(fn, with_indices=True, batched=True, num_proc=2)
# [0, 1]
# [0, 1]
# [0, 1, 2, 3, 4]
# [0, 1, 2, 3, 4]
```

As you can see, the subset passed to each thread is indexed from 0 to N which doesn't reflect their positions in `d`.


[style/quality] Moving to isort 5.0.0 + style/quality on datasets and metrics

Ready for review @lhoestq, just updated a few 156 files here

Move the repo to isort 5.0.0.

Also start testing style/quality on datasets and metrics.

Specific rule: we allow F401 (unused imports) in metrics to be able to add imports to detect early on missing dependencies.
Maybe we could add this in datasets but while cleaning this I've seen many example of really unused imports in dataset so maybe it's better to have it as a line-by-line nova instead of a general rule like in metrics.


`Dataset`/`DatasetDict` has no attribute 'save_to_disk'

`pip install git+https://github.com/huggingface/nlp.git` should have done the job.

Did you uninstall `nlp` before installing from github ?

> Did you uninstall `nlp` before installing from github ?

I did not. I created a new environment and installed `nlp` directly from `github` and it worked!

Thanks.

Hi,

As the title indicates, both `Dataset` and `DatasetDict` classes don't seem to have the `save_to_disk` method.  While the file [`arrow_dataset.py`](https://github.com/huggingface/nlp/blob/34bf0b03bfe03e7f77b8fec1cd48f5452c4fc7c1/src/nlp/arrow_dataset.py) in the repo here has the method, the file `arrow_dataset.py` which is saved after `pip install nlp -U` in my `conda` environment DOES NOT contain the `save_to_disk` method. I even tried `pip install git+https://github.com/huggingface/nlp.git ` and still no luck. Do I need to install the library in another way?


Fix germeval url

Continuation of #593 but without the dummy data hack


GermEval 2014: new download urls

Closing this one as #594 is merged (same changes except the dummy data hack)

Awesome @stefan-it ! @lhoestq how soon can I use the fixed GermEval dataset in HF token classification examples?

I've manually updated the script on S3, so you can actually use it right now with
```python
from nlp import load_dataset

germeval = load_dataset("germeval_14")
```

not sure if it's used in token classification examples already

Awesome. Not used yet but I am going to use it now. I've been working on an update for token classification examples and this was a missing piece. Thanks @stefan-it @lhoestq

Hi,

unfortunately, the download links for the GermEval 2014 dataset have changed: they're now located on a Google Drive.

I changed the URLs and bump version from 1.0.0 to 2.0.0.


Test in memory and on disk

I added test parameters to do every test both in memory and on disk.
I also found a bug in concatenate_dataset thanks to the new tests and fixed it.


fix #589 (backward compat)

Fix #589


The process cannot access the file because it is being used by another process (windows)

Hi, which version of `nlp` are you using?

By the way we'll be releasing today a significant update fixing many issues (but also comprising a few breaking changes).
You can see more informations here #545 and try it by installing from source from the master branch.

Ok, it's probably fixed on master. Otherwise if you can give me a fully self-contained exemple to reproduce the error, I can try to investigate.

I get the same behavior, on Windows, when `map`ping a function to a loaded dataset.
The error doesn't occur if I re-run the cell a second time though!
I'm on version 1.0.1.

@saareliad I got the same issue that troubled me quite a while. Unfortunately, there are no good answers to this issue online, I tried it on Linux and that's absolutely fine. After hacking the source code, I solved this problem as follows.

In the source code file: arrow_dataset.py -> _map_single(...)

change
```python
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```
to
```python
tmp_file.close()
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```

Then it works without needing multiple times runs to avoid the permission error.
I know this solution is unusual since it changes the source code. Hopefully, the lib's contributors can have better solutions in the future.

@wangcongcong123 thanks  for sharing.
(BTW I also solved it locally on windows by putting the problematic line under try except and not using cache... On windows I just needed 1% of the dataset anyway)

Hi, I consistently get the following error when developing in my PC (windows 10):

```
    train_dataset = train_dataset.map(convert_to_features, batched=True)
  File "C:\Users\saareliad\AppData\Local\Continuum\miniconda3\envs\py38\lib\site-packages\nlp\arrow_dataset.py", line 970, in map
    shutil.move(tmp_file.name, cache_file_name)
  File "C:\Users\saareliad\AppData\Local\Continuum\miniconda3\envs\py38\lib\shutil.py", line 803, in move
    os.unlink(src)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\saareliad\\.cache\\huggingface\\datasets\\squad\\plain_text\\1.0.0\\408a8fa46a1e2805445b793f1022e743428ca739a34809fce872f0c7f17b44ab\\tmpsau1bep1'

```


Cannot use nlp.load_dataset text, AttributeError: module 'nlp.utils' has no attribute 'logging'

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/nlp/load.py", line 533, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/nlp/load.py", line 61, in import_main_class
    module = importlib.import_module(module_path)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/nlp/datasets/text/5dc629379536c4037d9c2063e1caa829a1676cf795f8e030cd90a537eba20c08/text.py", line 9, in <module>
    logger = nlp.utils.logging.get_logger(__name__)
AttributeError: module 'nlp.utils' has no attribute 'logging'
```

Occurs on the following code, or any code including the load_dataset('text'):
```
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                        truncation=True, max_length=args.block_size), batched=True)
dataset.set_format(type='torch', columns=['input_ids'])
return dataset
```


Support pathlike obj in load dataset

Fix #582

(I recreated the PR, I got an issue with git)


Support pathlike obj in load dataset

Fix #582


Better message when data files is empty

Fix #581


Fix select for pyarrow < 1.0.0

Fix #583


Use github versioning

I noticed that datasets like `cnn_dailymail` need the `version` parameter to be passed to its `config_kwargs`.
Shall we rename the `version` paramater in `load_dataset` ? Maybe `repo_version` or `script_version` ?

Right now dataset scripts and metrics are downloaded from S3 which is in sync with master. It means that it's not currently possible to pin the dataset/metric script version.

To fix that I changed the download url from S3 to github, and adding a `version` parameter in `load_dataset` and `load_metric` to pin a certain version of the lib, as in #562


ArrowIndexError on Dataset.select

If the indices table consists in several chunks, then `dataset.select` results in an `ArrowIndexError` error for pyarrow < 1.0.0

Example:

```python
from nlp import load_dataset

mnli = load_dataset("glue", "mnli", split="train")
shuffled = mnli.shuffle(seed=42)
mnli.select(list(range(len(mnli))))
```

raises:
```python
---------------------------------------------------------------------------
ArrowIndexError                           Traceback (most recent call last)
<ipython-input-64-006a5d38d418> in <module>
----> 1 mnli.shuffle(seed=42).select(list(range(len(mnli))))

~/Desktop/hf/nlp/src/nlp/fingerprint.py in wrapper(*args, **kwargs)
    161             # Call actual function
    162
--> 163             out = func(self, *args, **kwargs)
    164
    165             # Update fingerprint of in-place transforms + update in-place history of transforms

~/Desktop/hf/nlp/src/nlp/arrow_dataset.py in select(self, indices, keep_in_memory, indices_cache_file_name, writer_batch_size, new_fingerprint)
   1653         if self._indices is not None:
   1654             if PYARROW_V0:
-> 1655                 indices_array = self._indices.column(0).chunk(0).take(indices_array)
   1656             else:
   1657                 indices_array = self._indices.column(0).take(indices_array)

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.Array.take()

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowIndexError: take index out of bounds
```

This is because the `take` method is only done on the first chunk which only contains 1000 elements by default (mnli has ~400 000 elements).

Shall we change that to use
```python
pa.concat_tables(self._indices._indices.slice(i, 1) for i in indices_array)
```
instead of `take` ? @thomwolf


Allow for PathLike objects

Using PathLike objects as input for `load_dataset` does not seem to work. The following will throw an error.

```python
files = list(Path(r"D:\corpora\yourcorpus").glob("*.txt"))
dataset = load_dataset("text", data_files=files)
```

Traceback:

```
Traceback (most recent call last):
  File "C:/dev/python/dutch-simplification/main.py", line 7, in <module>
    dataset = load_dataset("text", data_files=files)
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\load.py", line 548, in load_dataset
    builder_instance.download_and_prepare(
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\builder.py", line 470, in download_and_prepare
    self._save_info()
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\builder.py", line 564, in _save_info
    self.info.write_to_directory(self._cache_dir)
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\info.py", line 149, in write_to_directory
    self._dump_info(f)
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\info.py", line 156, in _dump_info
    file.write(json.dumps(asdict(self)).encode("utf-8"))
  File "c:\users\bramv\appdata\local\programs\python\python38\lib\json\__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "c:\users\bramv\appdata\local\programs\python\python38\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "c:\users\bramv\appdata\local\programs\python\python38\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
TypeError: keys must be str, int, float, bool or None, not WindowsPath
```

We have to cast to a string explicitly to make this work. It would be nicer if we could actually use PathLike objects.

```python
files = [str(f) for f in Path(r"D:\corpora\wablieft").glob("*.txt")]
```


Better error message when input file does not exist

In the following scenario, when `data_files` is an empty list, the stack trace and error message could be improved. This can probably be solved by checking for each file whether it actually exists and/or whether the argument is not false-y.

```python
dataset = load_dataset("text", data_files=[])
```

Example error trace.

```
Using custom data configuration default
Downloading and preparing dataset text/default-d18f9b6611eb8e16 (download: Unknown size, generated: Unknown size, post-processed: Unknown sizetotal: Unknown size) to C:\Users\bramv\.cache\huggingface\datasets\text\default-d18f9b6611eb8e16\0.0.0\3a79870d85f1982d6a2af884fde86a71c771747b4b161fd302d28ad22adf985b...
Traceback (most recent call last):
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\builder.py", line 424, in incomplete_dir
    yield tmp_dir
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\builder.py", line 462, in download_and_prepare
    self._download_and_prepare(
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\builder.py", line 537, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\builder.py", line 813, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\arrow_writer.py", line 217, in finalize
    self.pa_writer.close()
AttributeError: 'NoneType' object has no attribute 'close'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/dev/python/dutch-simplification/main.py", line 7, in <module>
    dataset = load_dataset("text", data_files=files)
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\load.py", line 548, in load_dataset
    builder_instance.download_and_prepare(
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\builder.py", line 470, in download_and_prepare
    self._save_info()
  File "c:\users\bramv\appdata\local\programs\python\python38\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\nlp\builder.py", line 430, in incomplete_dir
    shutil.rmtree(tmp_dir)
  File "c:\users\bramv\appdata\local\programs\python\python38\lib\shutil.py", line 737, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\users\bramv\appdata\local\programs\python\python38\lib\shutil.py", line 615, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\users\bramv\appdata\local\programs\python\python38\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\bramv\\.cache\\huggingface\\datasets\\text\\default-d18f9b6611eb8e16\\0.0.0\\3a79870d85f1982d6a2af884fde86a71c771747b4b161fd302d28ad22adf985b.incomplete\\text-train.arrow'
```


nlp re-creates already-there caches when using a script, but not within a shell

Couln't reproduce on my side :/
let me know if you manage to reproduce on another env (colab for example)

`nlp` keeps creating new caches for the same file when launching `filter` from a script, and behaves correctly from within the shell.

Example: try running

```
import nlp

hans_easy_data = nlp.load_dataset('hans', split="validation").filter(lambda x: x['label'] == 0)
hans_hard_data = nlp.load_dataset('hans', split="validation").filter(lambda x: x['label'] == 1)
```

twice. If launched from a `file.py` script, the cache will be re-created the second time. If launched as 3 shell/`ipython` commands, `nlp` will correctly re-use the cache.
As observed with @lhoestq.


Doc metrics

Adding documentation on metrics loading/using/sharing


Add CommonGen Dataset

CC Authors:
@yuchenlin @MichaelZhouwang