"NonMatchingChecksumError" when importing 'spider' dataset

Thanks for reporting, @OmarAlaaeldein.

Datasets hosted at Google Drive give problems quite often due to a change in their service:
- #3786

Related to:
- #3906

I'm having a look.

We have made a Pull Request to replace the Google Drive URL. This fix will be accessible in our next `datasets` library release.

In the meantime, once the PR merged into master, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

## Describe the bug
When importing 'spider' dataset [https://huggingface.co/datasets/spider] an error occurs

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('spider')
```

## Expected results
Dataset object

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']

## Environment info
- `datasets` version: 2.2.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.11
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


load dataset wikitext-2-raw-v1 failed. Could not reach wikitext-2-raw-v1.py.

bug

Something isn't working

## Describe the bug
Could not reach wikitext-2-raw-v1.py

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("wikitext-2-raw-v1")
```

## Expected results
Download `wikitext-2-raw-v1` dataset successfully.

## Actual results
```
  File "load_datasets.py", line 13, in <module>
    load_dataset("wikitext-2-raw-v1")
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1715, in load_dataset
    **config_kwargs,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1536, in load_dataset_builder
    data_files=data_files,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1282, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1224, in dataset_module_factory
    dynamic_modules_path=dynamic_modules_path,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 559, in get_module
    local_path = self.download_loading_script(revision)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 539, in download_loading_script
    return cached_path(file_path, download_config=download_config)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 246, in cached_path
    download_desc=download_config.download_desc,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 582, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.2.2/datasets/wikitext-2-raw-v1/wikitext-2-raw-v1.py (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Read timed out. (read timeout=100)",),))
```
I tried to download wikitext-2-raw-v1.py by chrome and got:
![image](https://user-images.githubusercontent.com/20658907/170171595-0ca9f1da-c05a-4b57-861e-9530bfa3bdb9.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2
- Platform: CentOS 7
- Python version: 3.6
- PyArrow version: 3.0.0


LocalDatasetModuleFactoryWithoutScript extracts invalid builder name

Something isn't working

Ok, so
```
os.path.basename("/home/user/")
```
gives `''` while
```
os.path.basename("/home/user")
```
gives `user`.
The code should check if the last char is a slash.

The fix is:
```
"name": os.path.basename(self.path[:-1] if self.path[-1] == "/" else self.path)
```

## Describe the bug
Trying to load a local dataset raises an error indicating that the config builder has to have a name.
No error should be reported, since the call is completly valid.

## Steps to reproduce the bug
```python
load_dataset("./data/some-dataset/", name="some-name")
```

## Expected results
The dataset should be loaded.

## Actual results
```
Traceback (most recent call last):
  File "train_lquad.py", line 19, in <module>
    load(tokenize_target_function, tokenize_target_function, {}, tokenizer)
  File "train_lquad.py", line 14, in load
    dataset = load_dataset("./data/lquad/", name="lquad")
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/load.py", line 1708, in load_dataset
    builder_instance = load_dataset_builder(
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/load.py", line 1560, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/builder.py", line 269, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/builder.py", line 403, in _create_builder_config
    raise ValueError(f"BuilderConfig must have a name, got {builder_config.name}")
ValueError: BuilderConfig must have a name, got
```

## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-4.18.0-348.20.1.el8_5.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.6
- PyArrow version: 8.0.0
- Pandas version: 1.4.2

The error is probably in line 795 in load.py:

```
 builder_kwargs = {
     "hash": hash,
     "data_files": data_files,
     "name": os.path.basename(self.path),
     "base_path": self.path,
     **builder_kwargs,
 }
```

`os.path.basename` for a directory returns an empty string, rather than the name of the directory.


Calling `cast_column`/`remove_columns` and a sequence of `map` operations ends up making `faiss` fail with `ValueError`

Something isn't working

It works if we either remove the `ds = ds.cast_column("id", Value("int32"))` line from the code above, or if instead calling `ds.remove_columns()` we remove the columns inside each mapping as `ds.map(..., remove_columns=[...])` instead of right after the mapping.

Both of those solutions seem to fix the issue, so the root cause of it may be around that. Sorry I cannot provide you more insights, in case I get to fix it I'll submit a PR, in the meanwhile the code that I'm using as a workaround is the following:

```python
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
import torch

torch.set_grad_enabled(False)
ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

from datasets import load_dataset, Value

ds = load_dataset("csv", data_files=["sample.csv"], split="train")
ds = ds.cast_column("id", Value("int32"))
ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

def generate_embeddings(x):
    return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

ds = ds.map(generate_embeddings, remove_columns=["inputs"])
ds.add_faiss_index(column="embeddings")
```

FYI the main reason I want to use `dataset.remove_columns` rather than the function inside `dataset.map` is because according to the 🤗 Datasets documentation, it's faster.

"🤗 Datasets also has a [Dataset.remove_columns()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.Dataset.remove_columns) method that is functionally identical, but faster, because it doesn’t copy the data of the remaining columns."

More information at https://huggingface.co/docs/datasets/process#map

Here I'm presenting all the scenarios so that you can further investigate the issue:

- ✅ `cast_column` -> `map` with `remove_columns` -> `map` with `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings, remove_columns=["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ❌ `cast_column` -> `map` -> `remove_columns` -> `map` -> `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
  ds = ds.remove_columns(["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings)
  ds = ds.remove_columns(["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ❌ `cast_column` -> `map` with `remove_columns` -> `map` -> `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings)
  ds = ds.remove_columns(["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ✅ `cast_column` -> `map` -> `remove_columns` -> `map` with `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
  ds = ds.remove_columns(["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings, remove_columns=["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ✅ `map` -> `remove_columns` -> `map` -> `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
  ds = ds.remove_columns(["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings)
  ds = ds.remove_columns(["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

So on, I've created #4411 so as to fix the bug with `remove_columns` under certain conditions before `add_faiss_index`, which means that the scenarios not working above are already working fine.

First of all, sorry in advance for the unclear title, but this bug is weird to explain (at least for me), so I tried my best to summarize all the information in this issue.

## Describe the bug

Calling a certain combination of operations over a 🤗 `Dataset` and then trying to calculate the `faiss` index with `.add_faiss_index` ends up throwing an exception while trying to set the format back of a previously removed column. But this just happens over certain conditions... I'll present some scenarios below!

## Steps to reproduce the bug

Assuming the following dataset named `sample.csv` with some IMDb data:

```csv
id,title,summary
1877830,"The Batman","When a sadistic serial killer begins murdering key political figures in Gotham, Batman is forced to investigate the city's hidden corruption and question his family's involvement."
9419884,"Doctor Strange in the Multiverse of Madness","Doctor Strange teams up with a mysterious teenage girl from his dreams who can travel across multiverses, to battle multiple threats, including other-universe versions of himself, which threaten to wipe out millions across the multiverse. They seek help from Wanda the Scarlet Witch, Wong and others."
11138512,"The Northman","From visionary director Robert Eggers comes The Northman, an action-filled epic that follows a young Viking prince on his quest to avenge his father's murder."
1745960,"Top Gun: Maverick","After more than thirty years of service as one of the Navy's top aviators, Pete Mitchell is where he belongs, pushing the envelope as a courageous test pilot and dodging the advancement in rank that would ground him."
```

We'll be able to reproduce the bug using the following piece of code:

```python
# Sample code to reproduce the bug
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
import torch

torch.set_grad_enabled(False)
ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

from datasets import load_dataset, Value

ds = load_dataset("csv", data_files=["sample.csv"], split="train")
ds = ds.cast_column("id", Value("int32")) # from `int64` to `int32`
ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
ds = ds.remove_columns(["title", "summary"])

def generate_embeddings(x):
    return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

ds = ds.map(generate_embeddings)
ds = ds.remove_columns("inputs")
ds.add_faiss_index(column="embeddings") # It fails here!
```

The code above is an adaptation of https://huggingface.co/docs/datasets/faiss_es, for the sake of presenting the bug with a simple example.

## Expected results

Ideally, the `faiss` index should be calculated over the 🤗 `Dataset` and no exception should be triggered.

## Actual results

But what happens instead is that a `ValueError: Columns ['inputs'] not in the dataset. Current columns in the dataset: ['id', 'embeddings']`, which makes no sense as that column has been previously dropped.

## Environment info

<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-1074-azure-x86_64-with-glibc2.31
- Python version: 3.9.5
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Fix dependency on dill version

_The documentation is not available anymore as the PR was closed or merged._

We had to make a hotfix by pinning dill:
- #4380

because from version 0.3.5, our custom `save_function` pickling function was raising an exception:
- #4379

This PR fixes this by implementing our custom `save_function` depending on the version of dill.

CC: @anivegesana

This PR needs first being merged:
- [x] #4384
  - so that a circular import is fixed

It is also convenient to merge first:
- [x] #4385


Fix URL in gem dataset for totto config

As commented in:
- https://github.com/huggingface/datasets/issues/4386#issuecomment-1134902372

CC: @StevenTang1998


Add Pascal VOC dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4395). All of your documentation changes will be reflected on that endpoint.

Some CI fails are unrelated to your PR and fixed on master, feel free to merge master into your branch :)

This PR adds the Pascal VOC dataset in the same way TFDS has it added. I believe we can iterate on this dataset and in future versions include more data, such as segmentation masks, but for now I think it is a good idea to just add it the same way as TFDS to get a solid first version out there.


trainer became extremely slow after reload dataset by `load_from_disk`

Something isn't working

I tried to make the dataset much more smaller (100000 rows) , then the speed became `33.88it/s` from`8.62s/it`.  It's nearly 200 times... Do you have any idea? Thank you!

Similar issue: https://github.com/huggingface/transformers/issues/8818

I changed `RandomSampler` to  `SequentialSampler` in the `trainer.py`, but the speed didn't become faster.

I changed
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
```
to
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset", keep_in_memory=True
    )
```
and obtained normal speed. It's seems that the problem is on the os's speed limit.

Hi ! Currently `save_to_disk` saves one big Arrow file, which causes some slow downs. This has been discussed in #3735 and we'll implement sharding pretty soon to solve this

For now you can try splitting and saving your dataset in several Arrow files. Then you can load them one by one and use `concatenate_datasets` to have your big dataset again and hopefully with a better speed

## Describe the bug
Due to memory problem, I need to save my tokenized datasets locally by CPU and reload it by multi GPU for running training script. However, after I reload it by `load_from_disk` and start training, the speed is extremely slow. It says I need about 1500 hours with 8 A100 cards. Before this, I can run the whole script in one day with a single A100 card.
Since I am try to pre-train a BERT, **my dataset is very large(29058165 rows)**

## Steps to reproduce the bug
```python
tokenized_datasets.save_to_disk(
        "/pathto/dataset"
    )
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"] if training_args.do_train else None,
        eval_dataset=tokenized_datasets["validation"]
        if training_args.do_eval
        else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
 train_result = trainer.train(resume_from_checkpoint=checkpoint)
```

## Expected results
Without the save and reload process, I only need about one day to run the whole script with one A100 card.

## Actual results
```
[INFO|trainer.py:1290] 2022-05-23 22:49:46,266 >> ***** Running training *****
[INFO|trainer.py:1291] 2022-05-23 22:49:46,266 >>   Num examples = 29058165
[INFO|trainer.py:1292] 2022-05-23 22:49:46,266 >>   Num Epochs = 5
[INFO|trainer.py:1293] 2022-05-23 22:49:46,266 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1294] 2022-05-23 22:49:46,266 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:1295] 2022-05-23 22:49:46,266 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:1296] 2022-05-23 22:49:46,266 >>   Total optimization steps = 567540
  0%|          | 1/567540 [00:09<1544:49:04,  9.80s/it]
  0%|          | 2/567540 [00:17<1320:00:17,  8.37s/it]
  0%|          | 3/567540 [00:26<1393:10:17,  8.84s/it]
  0%|          | 4/567540 [00:34<1344:56:33,  8.53s/it]
  0%|          | 5/567540 [00:43<1359:36:12,  8.62s/it]
```

## Environment info
```
torch              1.11.0+cu113
torchaudio         0.11.0+cu113
torchvision        0.12.0+cu113
transformers       4.18.0
datasets           2.2.2
```


Update CI deprecated legacy image

Now our CI still uses a deprecated legacy image:
> You’re using a [deprecated Docker convenience image.](https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034) Upgrade to a next-gen Docker convenience image.

This PR updates to next-generation convenience image.

Related to:
- #2955


remove int documentation from logging docs

Removes the `int` documentation from the [logging section](https://huggingface.co/docs/datasets/package_reference/logging_methods#levels) of the docs.


Refactor column mappings for question answering datasets

> Thanks.
>
> I have no visibility about this, but if you say it is more useful for AutoTrain this way...

Thanks for the review @albertvillanova ! Yes, I need some way to reconstruct the original column names with a period because that's how they appear after we flatten the nested columns. In any case, we can adjust this later if needed :)

> Does that mean that we need to change the metadata?

Yes, but this PR takes care of it :)

This PR tweaks the keys in the metadata that are used to define the column mapping for question answering datasets. This is needed in order to faithfully reconstruct column names like `answers.text` and `answers.answer_start` from the keys in AutoTrain.

As observed in https://github.com/huggingface/datasets/pull/4367 we cannot use periods `.` in the keys of the YAML tags, so a decision was made to use a flat mapping with underscores. For QA datasets, however, it's handy to be able to reconstruct the nesting -- hence this PR.

cc @sashavor


Fix metadata validation

Since Python 3.8, the typing module:
- raises an AttributeError when trying to access `__args__` on any type, e.g.: `List.__args__`
- provides the `get_args` function instead: `get_args(List)`

This PR implements a fix for Python >=3.8 whereas maintaining backward compatibility.


Fix bug in gem dataset for wiki_auto_asset_turk config

This PR fixes some URLs.

Fix #4386.


Set builder name from module instead of class

Now the builder name attribute is set from from the builder class name.

This PR sets the builder name attribute from the module name instead. Some motivating reasons:
- The dataset ID is relevant and unique among all datasets and this is directly related to the repository name, i.e., the name of the directory containing the dataset
- The name of the module (i.e. the file containing the loading loading script) is already relevant for loading: it must have the same name as its containing directory (related to the dataset ID), as we search for it using its directory name
- On the other hand, the name of the builder class is not relevant for loading: in our code, we just search for a class which is subclass of `DatasetBuilder` (independently of its name). We do not put any constraint on the naming of the builder class and indeed it can have a name completely different from its module/direcotry/dataset_id

IMO it makes more sense to align the caching directory name with the dataset_id/directory/module name instead of the builder class name.

Fix #4381.


device/google/accessory/adk2012 - Git at Google

"git clone https://android.googlesource.com/device/google/accessory/adk2012"
 https://android.googlesource.com/device/google/accessory/adk2012/#:~:text=git%20clone%20https%3A//android.googlesource.com/device/google/accessory/adk2012


Bug for wiki_auto_asset_turk from GEM

Something isn't working

Thanks for reporting, @StevenTang1998.

I'm looking into it.

Hi @StevenTang1998,

We have fixed the issue:
- #4389

The fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by installing `datasets` from our GitHub repo:
```
pip install git+https://github.com/huggingface/datasets#egg=datasets
```

Thanks for your reply!!
And the totto dataset has the same problem. The url should be change to [https://storage.googleapis.com/totto-public/totto_data.zip](https://storage.googleapis.com/totto-public/totto_data.zip).

Hi again @StevenTang1998,

I don't see any problem when loading `totto` dataset:
```python
In [4]: import datasets
   ...: ds = datasets.load_dataset("totto")
Downloading builder script: 5.58kB [00:00, 5.33MB/s]
Downloading metadata: 2.78kB [00:00, 2.96MB/s]
Using custom data configuration default
Downloading and preparing dataset totto/default (download: 179.03 MiB, generated: 706.59 MiB, post-processed: Unknown size, total: 885.62 MiB) to .../.cache/huggingface/datasets/totto/default/1.0.0/263c85871e5451bc892c65ca0306c0629eb7beb161e0eb998f56231562335dd2...
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188M/188M [00:32<00:00, 5.77MB/s]
Dataset totto downloaded and prepared to .../.cache/huggingface/datasets/totto/default/1.0.0/263c85871e5451bc892c65ca0306c0629eb7beb161e0eb998f56231562335dd2. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 147.95it/s]

In [5]: ds
Out[5]:
DatasetDict({
    train: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 120761
    })
    validation: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 7700
    })
    test: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 7700
    })
})
```

Sorry, I didn't express it clearly. It's the totto dataset from gem.
datasets.load_dataset('gem', 'totto')

## Describe the bug
The script of wiki_auto_asset_turk for GEM may be out of date.

## Steps to reproduce the bug
```python
import datasets
datasets.load_dataset('gem', 'wiki_auto_asset_turk')
```

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/load.py", line 1731, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 640, in download_and_prepare
    self._download_and_prepare(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 1158, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 707, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/tangtianyi/.cache/huggingface/modules/datasets_modules/datasets/gem/982a54473b12c6a6e40d4356e025fb7172a5bb2065e655e2c1af51f2b3cf4ca1/gem.py", line 538, in _split_generators
    dl_dir = dl_manager.download_and_extract(_URLs[self.config.name])
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 416, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 294, in download
    downloaded_path_or_paths = map_nested(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 351, in map_nested
    mapped = [
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 352, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 288, in _single_map_nested
    return function(data_struct)
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 320, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 234, in cached_path
    output_path = get_from_cache(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 579, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://github.com/facebookresearch/asset/raw/master/dataset/asset.test.orig
```


Test dill

I should point out that the hash will be the same if computed twice with the same code on the same version of dill (after adding huggingface's code that removes line numbers and file names, and sorts globals.) My changes in dill 0.3.5 and ones that I will make in 0.3.6 will result in different pickles than the ones dill 0.3.4 was making. This should still be fine for caching.

Just some comments @lhoestq:

The best practice for testing is to have a `test_<filename>.py` for each `<filename>.py`. Therefore in order to have the filenames aligned, I would propose:
- either renaming `fingerprint.py` to `caching.py`
- or renaming `test_caching.py` to `test_fingerprint.py`

On the other hand, my idea when implementing this test was not to test all the functionalities of the `Hasher`, but just to have a regression test that fails if dill version is > 0.3.4 and the pin in our `setup.py` is not present. Just recall that we had no failing test in our CI when the issue with dill was found on `transformers`.

The objective of this PR is just to have a regression test for that case: I tested and I got `AttributeError: module 'dill._dill' has no attribute 'stack'`

For this regression test, I took into account this comment by @gugarosa: https://github.com/huggingface/datasets/issues/4379#issuecomment-1133131825

There is no equivalent test in `test_caching.py` because our CI did not fail before pinning dill.

Ok I see, renaming it to `test_fingerprint.py` sounds like a good idea :)

Regression test for future releases of `dill`.

Related to #4379.


Refactor download

_The documentation is not available anymore as the PR was closed or merged._

This looks like a breaking change no ?
Also could you explain why it would be better this way ?

The might be only there to help type checkers, but I am not too familiar with the code base to know for sure. I think this might be useful:

https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING

> This looks like a breaking change no ?
> Also could you explain why it would be better this way ?

Sorry, @lhoestq, I naively thought it was obvious. I have tried to give some arguments in the motivation of this PR (see above). I can give additional arguments if needed.

This PR performs a refactoring of the download functionalities, by proposing a modular solution and moving them to their own package "download". Some motivating arguments:
- understandability: from a logical partitioning of the library, it makes sense to have all download functionalities grouped together instead of scattered in a much larger directory containing many more different functionalities
- abstraction: the level of abstraction of "download" (higher) is not the same as "utils" (lower); putting different levels of abstraction together, makes dependencies more intricate (potential circular dependencies) and the system more tightly coupled; when the levels of abstraction are clearly separated, the dependencies flow in a neat direction from higher to lower
- architectural: "download" is a domain-specific functionality of our library/application (a dataset builder performs several actions: download, generate dataset and cache it); these functionalities are at the core of our library; on the other hand, "utils" are always a low-level set of functionalities, not directly related to our domain/business core logic (all libraries have "utils"), thus at the periphery of our lib architecture

Also note that when a library is not architecturally designed following simple, neat, clean principles, this has a negative impact on extensibility, making more and more difficult to make enhancements.

As a concrete example in this case, please see: https://app.circleci.com/pipelines/github/huggingface/datasets/12185/workflows/ff25a790-8e3f-45a1-aadd-9d79dfb73c4d/jobs/72860
- After an extension, a circular import is found
- Diving into the cause of this circular import, see the dependency flow, which should be from higher to lower levels of abstraction:
```
ImportError while loading conftest '/home/circleci/datasets/tests/conftest.py'.
tests/conftest.py:12: in <module>
    import datasets
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/__init__.py:37: in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/arrow_dataset.py:59: in <module>
    from . import config
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/config.py:8: in <module>
    from .utils.logging import get_logger
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/utils/__init__.py:30: in <module>
    from .download_manager import DownloadConfig, DownloadManager, DownloadMode
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/utils/download_manager.py:39: in <module>
    from .py_utils import NestedDataStructure, map_nested, size_str
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/utils/py_utils.py:608: in <module>
    if config.DILL_VERSION < version.parse("0.3.5"):
E   AttributeError: module 'datasets.config' has no attribute 'DILL_VERSION'
```

Imports:
- datasets
  - Dataset: lower level than datasets
    - config: lower level than Dataset
      - logger: lower level than config
        - DownloadManager: !!! HIGHER level of abstraction than logger!!

Why when importing logger we require importing DownloadManager?!?
- Logically, it does not make sense
- This is due to an error in the design/architecture of our library:
  - To import the logger, we need to import it from `.utils.logging`
  - To import `.utils.logging` we need to import `.utils`
  - The import of `.utils` require the import of all its submodules defined in `utils.__init__.py`, among them: `.utils.download_manager`!

When putting `logging` and `download_manager` both inside `utils`, in order to import `logging` we need to import `download_manager` first: this is a strong coupling between modules and moreover between modules at different levels of abstraction (to import a lower level module, we require to import a higher level module). Additionally, it is clear that is makes no sense that in order to import `logging` we require to import `download_manager` first.


L

Something isn't working

## Describe the   L
L
## Expected   L
A clear and concise lmll
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


First time trying

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Bug in caching 2 datasets both with the same builder class name

bug

Something isn't working

Hi @NouamaneTazi, thanks for reporting.

Please note that both datasets are cached in the same directory because their loading builder classes have the same name: `class MTOP(datasets.GeneratorBasedBuilder)`.

You should name their builder classes differently, e.g.:
- `MtopDomain`
- `MtopIntent`

Hi @NouamaneTazi, please note that after our fix:
- #4388

we do not consider the class name anymore, but the name of the file where the loading builder class is implemented.

## Describe the bug
The two datasets `mteb/mtop_intent` and `mteb/mtop_domain `use both the same cache folder `.cache/huggingface/datasets/mteb___mtop`. So if you first load `mteb/mtop_intent` then datasets will not load `mteb/mtop_domain`.
If you delete this cache folder and flip the order how you load the two datasets , you will get the opposite datasets loaded (difference is here in terms of the label and label_text).

## Steps to reproduce the bug
```python
import datasets

dataset = datasets.load_dataset("mteb/mtop_intent", "en")
print(dataset['train'][0])
dataset = datasets.load_dataset("mteb/mtop_domain", "en")
print(dataset['train'][0])
```

## Expected results
```
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop_intent/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 920.14it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 1, 'label_text': 'GET_MESSAGE'}
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop_domain/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1307.59it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 0, 'label_text': 'messaging'}
```

## Actual results
```
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 920.14it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 1, 'label_text': 'GET_MESSAGE'}
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1307.59it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 1, 'label_text': 'GET_MESSAGE'}
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.1
- Platform: macOS-12.1-arm64-arm-64bit
- Python version: 3.9.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Pin dill

Hotfix #4379.

CC: @sgugger


Latest dill release raises exception

Something isn't working

Just an additional insight, the latest dill (either 0.3.5 or 0.3.5.1) also broke the hashing/fingerprinting of any mapping function.

For example:
```
from datasets import load_dataset

d = load_dataset("rotten_tomatoes")
d.map(lambda x: x)
```

Returns the standard non-dillable error:
```
Parameter 'function'=<function <lambda> at 0x7fe7d18c9560> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly....
```

@albertvillanova ExamplesTests.test_run_speech_recognition_seq2seq is in which file?

Thanks a lot @gugarosa for the insight: we will incorporate it in our CI as regression testing for future dill releases.

Hi @anivegesana, that test is in `transformers` library:
- https://github.com/huggingface/transformers/blob/main/examples/pytorch/test_pytorch_examples.py#L449
- https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py

@albertvillanova

I did a deep dive into @gugarosa's problem and found the issue and it might be related to the one @sgugger discovered. In dill 0.3.5(.1), I created a new `save_function` that fixes a bug in dill that prevented the pickling of recursive inner functions. It was a more complete solution to the problem that `dill._dill.stack` tried to solve in the internal API of dill. Since `dill._dill.stack` was no longer needed, I removed it. Since datasets copies the `save_function` directly from the dill API, it stops working with the new dill version since `dill._dill.stack` is no longer present and the `save_function` has been updated with new code.

https://github.com/huggingface/datasets/blob/95193ae61e92aa537d0c65d37a1fd9d2393aae89/src/datasets/utils/py_utils.py#L607-L678

~If the dill version is below 0.3.5, you should keep this function. If it is after, you would need to update your copy of `save_function` to use the code I introduced, or manually add a `stack` variable to `dill._dill` if it doesn't exist. Fortunately, in any version of Python 3.7+, dictionaries are always in insertion order and dill no longer supports Python 3.6 or older. So, any globals dictionary saved by dill 0.3.5+ will be deterministic given that the version of dill is held constant and this save_function is unnecessary for newer versions of dill.~

Ah. I see what is happening. I guess a different copy of the function code is needed that sorts the global variables by name.

```py
if dill.__version__.split('.') < ['0', '3', '5']:
    # current save_function code inside here
else:
    # new save_function code inside here with the following line inserted after creating the globals
    globs = {k: globs[k] for k in sorted(globs.keys())}
```

Will look into the test case @sgugger pointed out after that and verify if this is causing the problem.

I am actually looking into rewriting the global variables code in uqfoundation/dill#466 and will keep this in mind and will try to create an easy way to modify the global variables in dill 0.3.6 (for example, sort them by key like datasets does).

Thanks a lot for your investigation @anivegesana.

Yes, we copied-pasted the old `save_function` function from `dill`, just adding a line to make deterministic the order of global variables `globs`.

However, this function has changed a lot from version 0.3.5, after your PR (thank you for the fix in recursiveness, indeed):
- uqfoundation/dill#443

We have to address this change.

If finally your PR to sort global variables is merged into dill 0.3.6, that will make our life easier, as the tweak will no longer be necessary. ;)

I have included a regression test so that we are sure future releases of dill do not break `datasets`:
- #4385

I should note that because Python 3.6 and older are now deprecated and Python 3.7 has insertion order dictionaries, the globals in dill will have a deterministic order, just not sorted. I would still keep it sorted like you have it to help with stability (for example, if someone reorders variables in a file, then sorting the globals would not invalidate the cache.)

It seems that the order is not quite deterministic in IPython. Huggingface datasets seems to do well in Jupyter regardless, so it is not a good idea to remove the sorting. uqfoundation/dill#19

## Describe the bug
As reported by @sgugger, latest dill release is breaking things with Datasets.

```
______________ ExamplesTests.test_run_speech_recognition_seq2seq _______________

self = <multiprocess.pool.ApplyResult object at 0x7fa5981a1cd0>, timeout = None

    def get(self, timeout=None):
        self.wait(timeout)
        if not self.ready():
            raise TimeoutError
        if self._success:
            return self._value
        else:
>           raise self._value
E           TypeError: '>' not supported between instances of 'NoneType' and 'float'
```


Tidy up license metadata for google_wellformed_query, newspop, sick

Amend three licenses on datasets to fit naming convention (lower case, cc licenses include sub-version number). I think that's it - everything else on datasets looks great & super-searchable now!


Fix checksum and bug in irc_disentangle dataset

There was a bug in filepath segment:
- wrong: `jkkummerfeld-irc-disentanglement-fd379e9`
- right: `jkkummerfeld-irc-disentanglement-35f0a40`

Also there was a bug in the checksum of the downloaded file.

This PR fixes these issues.

Fix partially #4376.


irc_disentagle viewer error

DUPLICATED comment from https://github.com/huggingface/datasets/issues/3807:

my code:
```
from datasets import load_dataset

dataset = load_dataset("irc_disentangle", download_mode="force_redownload")
```
however, it produces the same error
```
[38](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=37) if len(bad_urls) > 0:
     [39](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=38)     error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> [40](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=39)     raise NonMatchingChecksumError(error_msg + str(bad_urls))
     [41](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=40) logger.info("All the checksums matched successfully" + for_verification_name)

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/jkkummerfeld/irc-disentanglement/tarball/master']
```
I attempted to use the `ignore_verifications' as such:

```
ds = datasets.load_dataset('irc_disentangle', download_mode="force_redownload", ignore_verifications=True)

Downloading builder script: 12.0kB [00:00, 5.92MB/s]
Downloading metadata: 7.58kB [00:00, 3.48MB/s]
No config specified, defaulting to: irc_disentangle/ubuntu
Downloading and preparing dataset irc_disentangle/ubuntu (download: 112.98 MiB, generated: 60.05 MiB, post-processed: Unknown size, total: 173.03 MiB) to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5...
Downloading data: 118MB [00:09, 12.1MB/s]

Dataset irc_disentangle downloaded and prepared to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5. Subsequent calls will reuse this data.
100%|██████████| 3/3 [00:00<00:00, 675.38it/s]
```
but, this returns an empty set?
```
DatasetDict({
    train: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    test: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
})
```
not sure what else to try at this point?
Thanks in advanced🤗

Thanks for reporting, @labouz. I'm addressing it.

The issue with checksum and empty dataset has been fixed by:
- #4377

To load the dataset, you should force the re-generation of the dataset from the downloaded file by passing `download_mode="reuse_cache_if_exists"` to `load_dataset`.

In relation with the issue with the dataset viewer, first the dataset should be refactored to support streaming.

the dataviewer shows this message for "ubuntu" - "train", "test", and "validation" splits:
```
Server error
Status code:   400
Exception:     ValueError
Message:       Cannot seek streaming HTTP file

```
it appears to give the same message for the "channel_two" data as well.

I get a Checksums error when using `load_data()` with this dataset. Even with the `download_mode` and `ignore_verifications` options set. i referenced the issue here: https://github.com/huggingface/datasets/issues/3807


Support DataLoader with num_workers > 0 in streaming mode

Alright this is finally ready for review ! It's quite long I'm sorry, but it's not easy to disentangle everything ^^'

The main additions are in
- src/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py
- src/datasets/iterable_dataset.py
- src/datasets/utils/patching.py

Added some comments and an error when lists have different lengths for sharding :)

Let's resolve the merge conflict and the CI error (if it's related to the changes), and I can review the PR again.

Feel free to review again :) The CI fail is unrelated to this PR and will be fixed by https://github.com/huggingface/datasets/pull/4472 (the hub now returns 401 instead of 404 for unauthenticated requests to non-existing repos)

CI failures are unrelated to this PR - merging :)

(CI fails are a mix of pip install fails and Hub fails)

### Issue

It's currently not possible to properly stream a dataset using multiple `torch.utils.data.DataLoader` workers:

- the `TorchIterableDataset` can't be pickled and passed to the subprocesses: https://github.com/huggingface/datasets/issues/3950
- streaming extension is failing: https://github.com/huggingface/datasets/issues/3951
- `fsspec` doesn't work out of the box in subprocesses

### Solution in this PR

I fixed these to enable passing an `IterableDataset` to a  `torch.utils.data.DataLoader` with `num_workers > 0`.

I also had to shard the `IterableDataset` to give each worker a shard, otherwise data would be duplicated. This is implemented in `TorchIterableDataset.__iter__` and uses the new `IterableDataset._iter_shard(shard_idx)` method

I also had to do a few changes the patching that enable streaming in dataset scripts:
- the patches are now always applied - not just for streaming mode. They're applied when a builder is instantiated
- I improved it to also check for renamed modules or attributes (ex: pandas vs pd)
- I grouped all the patches of pathlib.Path into a class `xPath`, so that `Path` outside of dataset scripts stay unchanged - otherwise I didn't change the content of the extended Path methods for streaming
- I fixed a bug with the `pd.read_csv` patch, opening the file in "rb" mode was missing and causing some datasets to not work in streaming mode, and compression inference was missing

### A few details regarding `fsspec` in multiprocessing

From https://github.com/fsspec/filesystem_spec/pull/963#issuecomment-1131709948 :
> Non-async instances might be safe in the forked child, if they hold no open files/sockets etc.; I'm not sure any implementations pass this test!
> If any async instance has been created, the newly forked processes must:
> 1. discard references to locks, threads and event loops and make new ones
> 2. not use any async fsspec instances from the parent process
> 3. clear all class instance caches

Therefore in a DataLoader's worker, I clear the reference to the loop and thread (1). We should be fine for 2 and 3 already since we don't use fsspec class instances from the parent process.

Fix https://github.com/huggingface/datasets/issues/3950
Fix https://github.com/huggingface/datasets/issues/3951

TODO:
- [x] fix tests


extremely slow processing when using a custom dataset

Something isn't working

question

Further information is requested

Hi !

My guess is that some examples in your dataset are bigger than your RAM, and therefore loading them in RAM to pass them to `remove_non_indic_sentences` takes forever because it might use SWAP memory.

Maybe several examples in your dataset are grouped together, can you check `len(lang_dataset["train"])` and `lang_dataset["train"].data.nbytes` of both datasets please ? It can also be helpful to check the distribution of lengths of each examples in your dataset.

## processing a custom dataset loaded as .txt file  is extremely slow,  compared to a dataset of similar volume from the hub

I have a large .txt file of 22 GB which i load into HF dataset

`lang_dataset = datasets.load_dataset("text", data_files="hi.txt")`

further i use a pre-processing function to clean the dataset

 `lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)`

the following processing takes astronomical time to process, while hoging all the ram.

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data.
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi",  use_auth_token=True)`

the hours predicted to preprocess are as follows:

huggingface hub dataset:  6.5 hrs
custom loaded dataset: 7000 hrs

note: both the datasets are almost actually same, just provided by different sources with has +/- some samples, only one is hosted on the HF hub and the other is downloaded in a text format.

## Steps to reproduce the bug
```
import datasets
import psutil
import sys
import glob
from fastcore.utils import listify
import re
import gc

def remove_non_indic_sentences(example):
    tmp_ls = []
    eng_regex = r'[. a-zA-Z0-9ÖÄÅöäå _.,!"\'\/$]*'
    for e in listify(example['text']):
        matches = re.findall(eng_regex, e)
        for match in (str(match).strip() for match in matches if match not in [""," ", "  ", ",", " ,", ", ", " , "]):
            if len(list(match.split(" "))) > 2:
                e = re.sub(match," ",e,count=1)
        tmp_ls.append(e)
        gc.collect()
    example['clean_text'] = tmp_ls
    return example

lang_dataset = datasets.load_dataset("text", data_files="hi.txt")

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)

## same thing work much faster when loading similar dataset from hub

lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", split="train", use_auth_token=True)

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)

```

## Actual results

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data.
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", use_auth_token=True)

**the hours predicted to preprocess are as follows:**
huggingface hub dataset:  6.5 hrs
custom loaded dataset: 7000 hrs

**i even tried the following:**

-  sharding the large 22gb text files into smaller files and loading
- saving the file to disk and then loading
- using lesser num_proc
- using smaller batch size
- processing without batches ie :  without `batched=True`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2.dev0
- Platform: Ubuntu 20.04 LTS
- Python version: 3.9.7
- PyArrow version:8.0.0


Remove links in docs to old dataset viewer

_The documentation is not available anymore as the PR was closed or merged._

Remove the links in the docs to the no longer maintained dataset viewer.


Check if dataset features match before push in `DatasetDict.push_to_hub`

Fix #4211


Add missing language tags for udhr dataset

Related to #4362.


Add redirect to dataset script in the repo structure page

Following https://github.com/huggingface/hub-docs/pull/146 I added a redirection to the dataset scripts documentation in the repository structure page.


Add long answer candidates to natural questions dataset

Once we have added `long_answer_candidates` maybe it would be worth to also add the missing `candidate_index` (inside `long_answer`). What do you think, @seirasto ?

Also note the "Data Fields" section in the README is missing the `long_answer` field.

Moreover, there is no instance example in "Data Instances" section.

We could either make these fixes in this PR or in a subsequent PR.

@albertvillanova I've added the missing fields and updated the README to include a data instance and some other things.

Great! I've made the updates to align the README. Please let me know if I missed anything.

As there were many minor little fixes, I thought it would be easier to fix them directly.

I think the loading script is OK now. If it is also validated by another datasets maintainer, I could run the generation of the pre-processed data and then merge this PR into master (once all the tests are green).

CC: @lhoestq

I have merged the master branch, so that we include all the fixes on Apache Beam + Google Dataflow.

Pre-processing is running!

Already finished for "dev" config:
```python
In [2]: ds = load_dataset("datasets/natural_questions", "dev")

In [3]: ds
Out[3]:
DatasetDict({
    validation: Dataset({
        features: ['id', 'document', 'question', 'long_answer_candidates', 'annotations'],
        num_rows: 7830
    })
})
```

There is an issue while running the preprocessing for the "default" (train+dev) config. Train data files are larger than than dev ones and workers run out of memory.

I'm opening a separate issue to handle this problem: #4525

@seirasto is proposing uploading their preprocessed data files to our Datasets bucket.

I think @lhoestq can give a more informed answer about authentication requirements.

Now that the data fiels are uploaded, can you merge the `main` branch into yours to re-trigger the CI @seirasto please ? :) Then I think we can merge if it's good for you @albertvillanova

Merge is done! I think someone needs to approve the CI to run :)

Can you run `make style` to fix the code formatting required by the CI please ?

Thanks @albertvillanova! I've committed all your suggestions.

This is a modification of the Natural Questions dataset to include missing information specifically related to long answer candidates. (See here: https://github.com/google-research-datasets/natural-questions#long-answer-candidates). This information is important to ensure consistent comparison with prior work. It does not disturb the rest of the format . @lhoestq @albertvillanova


Remove config names as yaml keys

I included the change from https://github.com/huggingface/datasets/pull/4302 directly in this PR, this way the datasets will be updated right away in the CI (the CI is only triggered when a dataset card is changed)

_The documentation is not available anymore as the PR was closed or merged._

Alright it's ready now :)

Here is an example for the `ade_corpus_v2` dataset card. Notice the new `configs` key:

https://github.com/huggingface/datasets/blob/76d9a141740a03f6836feb251f6059894b8d8046/datasets/ade_corpus_v2/README.md#L1-L78

CI failures are only related to dataset cards missing some content.

Many datasets have dots in their config names. However it causes issues with the YAML tags of the dataset cards since we can't have dots in YAML keys.

I fix this, I removed the tags separations per config name completely, and have a single flat YAML for all configurations. Dataset search doesn't use this info anyway. I removed all the config names used as YAML keys, and I moved them in under a new `config:` key.

This is related to https://github.com/huggingface/datasets/pull/2362 (internal https://github.com/huggingface/moon-landing/issues/946).

Also removing the dots in the YAML keys would allow us to do as in https://github.com/huggingface/datasets/pull/4302 which removes a hack that replaces all the dots by underscores in the YAML tags.

I also added a test in the CI that checks that all the YAML tags to make sure that:

- they can be parsed using a YAML parser
- they contain only valid YAML tags like languages or task_ids


duplicate

This issue or pull request already exists

Duplicate of:
- #3956

I think you should report that issue to `elasticsearch` library: https://github.com/elastic/elasticsearch-py

  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"

when I run the order:
nohup python3 custom_service.pyc > service.log 2>&1&

the log:
nohup: 忽略输入
Traceback (most recent call last):
  File "/home/xfz/p3_custom_test/custom_service.py", line 55, in <module>
  File "/home/xfz/p3_custom_test/custom_service.py", line 48, in doInitialize
  File "custom_impl.py", line 286, in custom_setup
  File "custom_impl.py", line 127, in create_es_index
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/__init__.py", line 345, in __init__
    ssl_show_warn=ssl_show_warn,
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 105, in client_node_configs
    node_configs = hosts_to_node_configs(hosts)
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 154, in hosts_to_node_configs
    node_configs.append(host_mapping_to_node_config(host))
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 221, in host_mapping_to_node_config
    return NodeConfig(**options)  # type: ignore
TypeError: __init__() missing 1 required positional argument: 'scheme'
[1]+  退出 1                nohup python3 custom_service.pyc > service.log 2>&1

custom_service_pyc can't running


Remove dots in config names

Closing in favor of https://github.com/huggingface/datasets/pull/4367

20+ datasets have dots in their config names. However it causes issues with the YAML tags of the dataset cards since we can't have dots in YAML keys.

This is related to https://github.com/huggingface/datasets/pull/2362 (internal https://github.com/huggingface/moon-landing/issues/946).

Also removing the dots in the config names would allow us to merge https://github.com/huggingface/datasets/pull/4302 which removes a hack that replaces all the dots by underscores in the YAML tags.

I also added a test in the CI that checks that all the YAML tags to make sure that:
- they can be parsed using a YAML parser
- they contain only valid YAML tags like `languages` or `task_ids`
- they contain valid config names (no invalid characters `<>:/\|?*.`)


Support complex feature types as `features` in packaged loaders

This PR adds `table_cast` to the packaged loaders to fix casting to the `Image`/`Audio`, `ArrayND` and `ClassLabel` types. If these types are not present in the `builder.config.features` dictionary, the built-in `pa.Table.cast` is used for better performance. Additionally, this PR adds `cast_storage` to `ClassLabel` to support the string to int conversion in `table_cast` and ensure that integer labels are in a valid range.

Fix https://github.com/huggingface/datasets/issues/4210

This PR is also a solution for these (popular) discussions: https://discuss.huggingface.co/t/converting-string-label-to-int/2816 and https://discuss.huggingface.co/t/class-labels-for-custom-datasets/15130/2

TODO:
* [x] tests


The dataset preview is not available for this split.

dataset-viewer

Related to the dataset viewer on huggingface.co

Hi!  A dataset has to be streamable to work with the viewer. I did a quick test, and yours is, so this might be a bug in the viewer. cc @severo

Looking at it. The message is now:

```
Message:       cannot cache function '__shear_dense': no locator available for file '/src/services/worker/.venv/lib/python3.9/site-packages/librosa/util/utils.py'
```

so possibly it's related to the libraries versions?

Maybe this SO thread can help: https://stackoverflow.com/questions/59290386/runtimeerror-at-cannot-cache-function-shear-dense-no-locator-available-fo

Same error for https://huggingface.co/datasets/LIUM/tedlium/viewer/release1/test. cc @sanchit-gandhi . I'm on it

Fixed in the datasets viewer, by setting the `NUMBA_CACHE_DIR` env var to a writable directory.

https://huggingface.co/datasets/Roh/ryanspeech/viewer/male/train

<img width="1538" alt="Capture d’écran 2022-06-08 à 11 30 08" src="https://user-images.githubusercontent.com/1676121/172583285-4cd49a0f-5715-423b-95dd-5f6ace3b2416.png">

https://huggingface.co/datasets/LIUM/tedlium/viewer/

<img width="1538" alt="Capture d’écran 2022-06-08 à 14 31 52" src="https://user-images.githubusercontent.com/1676121/172616897-fbcb7df7-0308-4d09-a17d-48826bc91374.png">

I have uploaded the corpus developed by our lab in the speech domain to huggingface [datasets](https://huggingface.co/datasets/Roh/ryanspeech). You can read about the companion paper accepted in interspeech 2021 [here](https://arxiv.org/abs/2106.08468). The dataset works fine but I can't make the dataset preview work. It gives me the following error that I don't understand. Can you help me to begin debugging it?

```
Status code:   400
Exception:     AttributeError
Message:       'NoneType' object has no attribute 'split'
```


Update dataset_infos for UDHN/udhr dataset

Thanks for contributing @leondz.

The checksums of the files have changed because more languages have been added:
- the new language codes need to be added to the dataset card (README file)
- I think the dataset version number should also be increased, so that users who had previously cached it, get a new dataset download (with the additional languages)

Yep! All done (also fixed the language tags in the README which were iso639-3 instead of the expected bcp47)

I guess the language code CI failure is due to languages.json being a subset of bcp47 (see issue #4304), happy to contribute a solution here, e.g. autogeneration of the lang list from the relevant isos and the ietf bcp47 subtag register or full code for validation

> Thanks again for your contribution, @leondz.
>
> Yes, I think it is OK to set version 1.0.0 (as previous was 0.0.0).
>
> One of the CI failures is related to dummy data: once you have updated the dataset version, the dummy_data ZIP file should be moved from "dummy/0.0.0/dummy_data.zip" to "dummy/1.0.0/dummy_data.zip".

Oh, thanks, I missed that one

> Other CI failure is related to missing languages in our resources file. This has been addressed in this PR:
>
> * #4371
>
> You should merge master branch into your feature branch to incorporate that fix.

Yeah, I saw this :) I already have the merge, thanks. I'm talking about the longer-term picture: every time another language code comes up (e.g. da-bornholm or es-VE), the json will need updating, because the current approach is non-exhaustive manual whitelisting instead of relying on the established bcp standard.

Checksum update to `udhr` for issue #4361


`udhr` doesn't load, dataset checksum mismatch

bug

Something isn't working

## Describe the bug
Loading `udhr` fails due to a checksum mismatch for some source files. Looks like both of the source files on unicode.org have changed:

size + checksum in datasets repo:
```
(hfdev) leon@blade:~/datasets/datasets/udhr$ jq .default.download_checksums < dataset_infos.json
{
  "https://unicode.org/udhr/assemblies/udhr_xml.zip": {
    "num_bytes": 2273633,
    "checksum": "0565fa62c2ff155b84123198bcc967edd8c5eb9679eadc01e6fb44a5cf730fee"
  },
  "https://unicode.org/udhr/assemblies/udhr_txt.zip": {
    "num_bytes": 2107471,
    "checksum": "087b474a070dd4096ae3028f9ee0b30dcdcb030cc85a1ca02e143be46327e5e5"
  }
}
```

size + checksum regenerated from current source files:
```
(hfdev) leon@blade:~/datasets/datasets/udhr$ rm dataset_infos.json
(hfdev) leon@blade:~/datasets/datasets/udhr$ datasets-cli test --save_infos udhr.py
Using custom data configuration default
Testing builder 'default' (1/1)
Downloading and preparing dataset udhn/default (download: 4.18 MiB, generated: 6.15 MiB, post-processed: Unknown size, total: 10.33 MiB) to /home/leon/.cache/huggingface/datasets/udhn/default/0.0.0/ad74b91fa2b3c386e5751b0c52bdfda76d334f76731142fd432d4acc2e2fde66...
Dataset udhn downloaded and prepared to /home/leon/.cache/huggingface/datasets/udhn/default/0.0.0/ad74b91fa2b3c386e5751b0c52bdfda76d334f76731142fd432d4acc2e2fde66. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 686.69it/s]
Dataset Infos file saved at dataset_infos.json
Test successful.
(hfdev) leon@blade:~/datasets/datasets/udhr$ jq .default.download_checksums < dataset_infos.json
{
  "https://unicode.org/udhr/assemblies/udhr_xml.zip": {
    "num_bytes": 2389690,
    "checksum": "a3350912790196c6e1b26bfd1c8a50e8575f5cf185922ecd9bd15713d7d21438"
  },
  "https://unicode.org/udhr/assemblies/udhr_txt.zip": {
    "num_bytes": 2215441,
    "checksum": "cb87ecb25b56f34e4fd6f22b323000524fd9c06ae2a29f122b048789cf17e9fe"
  }
}
(hfdev) leon@blade:~/datasets/datasets/udhr$

```

--- is unicode.org a sustainable hosting solution for this dataset?

## Steps to reproduce the bug
```python
from datasets import load_dataset
udhr = load_dataset("udhr")
```

## Expected results
That a Dataset object containing the UDHR data will be returned.

## Actual results
```
>>> d = load_dataset('udhr')
Using custom data configuration default
Downloading and preparing dataset udhn/default (download: 4.18 MiB, generated: 6.15 MiB, post-processed: Unknown size, total: 10.33 MiB) to /home/leon/.cache/huggingface/datasets/udhn/default/0.0.0/ad74b91fa2b3c386e5751b0c52bdfda76d334f76731142fd432d4acc2e2fde66...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/leon/.local/lib/python3.9/site-packages/datasets/load.py", line 1731, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/leon/.local/lib/python3.9/site-packages/datasets/builder.py", line 613, in download_and_prepare
    self._download_and_prepare(
  File "/home/leon/.local/lib/python3.9/site-packages/datasets/builder.py", line 1117, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/leon/.local/lib/python3.9/site-packages/datasets/builder.py", line 684, in _download_and_prepare
    verify_checksums(
  File "/home/leon/.local/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://unicode.org/udhr/assemblies/udhr_xml.zip', 'https://unicode.org/udhr/assemblies/udhr_txt.zip']
>>>
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.1 commit/4110fb6034f79c5fb470cf1043ff52180e9c63b7
- Platform: Linux Ubuntu 20.04
- Python version: 3.9.12
- PyArrow version: 8.0.0


Fix example in opus_ubuntu, Add license info

CI seems to fail due to languages incorrectly being flagged as invalid, I guess that's related to the currently-broken bcp47 validation (see #4304)

This PR
* fixes a typo in the example for the`opus_ubuntu` dataset where it's mistakenly referred to as `ubuntu`
* adds the declared license info for this corpus' origin
* adds an example instance
* updates the data origin type


Fix Version equality

I think `Version` equality should align with other similar cases in Python, like:
```python
In [1]: "a" == 5, "a" == None
Out[1]: (False, False)

In [2]: "a" != 5, "a" != None
Out[2]: (True, True)
```

With this PR, we will get:
```python
In [3]: Version("1.0.0") == 5, Version("1.0.0") == None
Out[3]: (False, False)

In [4]: Version("1.0.0") != 5, Version("1.0.0") != None
Out[4]: (True, True)
```

Note I found this issue when `doc-builder` tried to compare:
```python
if param.default != inspect._empty
```
where `param.default` is an instance of `Version`.


Missing dataset tags and sections in some dataset cards

Something isn't working

@lhoestq I can take this issue. Please can you point out to me where I can find the other positional arguments?

Hi @RohitRathore1 :)

You can find all the YAML tags in the tagging app here: https://hf.co/spaces/huggingface/datasets-tagging). They're all passed as arguments to a DatasetMetadata object used to validate the tags.

Summary of CircleCI errors for different dataset metadata:

- **BoolQ**: missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
- **Conllpp**: expected some content in section `Citation Information` but it is empty.
- **GLUE**:   'annotations_creators', 'language_creators', 'source_datasets' :['unknown'] are not registered tags
- **ConLL2003**:  field 'task_ids':  ['part-of-speech-tagging'] are not registered tags for 'task_ids'
-  **Hate_speech18:**   Expected some content in section `Data Instances` but it is empty, Expected some content in section `Data Splits` but it is empty
- **Jjigsaw_toxicity_pred**:  `Citation Information` but it is empty.
- **LIAR** :  `Data Instances`,`Data Fields`,  `Data Splits`,  `Citation Information` are empty.
- **MSRA NER** : Dataset Summary`, `Data Instances`, `Data Fields`,  `Data Splits`, `Citation Information` are empty.
- **sem_eval_2010_task_8**: missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
- **sms_spam**: `Data Instances` and`Data Splits` are empty.
- **Quora** : Expected some content in section `Citation Information` but it is empty,  missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
- **sentiment140**:  missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'


Fix warning in push_to_hub

_The documentation is not available anymore as the PR was closed or merged._

Fix warning:
```
FutureWarning: 'shard_size' was renamed to 'max_shard_size' in version 2.1.1 and will be removed in 2.4.0.
```


Fix dataset builder default version

This PR requires one of these other PRs being merged first:
- #4359
- huggingface/doc-builder#211

Currently, when using a custom config (subclass of `BuilderConfig`), default version set at the builder level is ignored: we must set default version in the custom config class.

However, when loading a dataset with `config_kwargs` (for a configuration not present in `BUILDER_CONFIGS`), the default version set in the custom config is ignored and "0.0.0" is used instead:
```python
ds = load_dataset("wikipedia", language="co", date="20220501", beam_runner="DirectRunner")
```
generates the following config:
```python
WikipediaConfig(name='20220501.co', version=0.0.0, data_dir=None, data_files=None, description='Wikipedia dataset for co, parsed from 20220501 dump.')
```
with version "0.0.0" instead of "2.0.0".

See as a counter-example, when the config is present in `BUILDER_CONFIGS`:
```python
ds = load_dataset("wikipedia", "20220301.fr", beam_runner="DirectRunner")
```
generates the following config:
```python
WikipediaConfig(name='20220301.fr', version=2.0.0, data_dir=None, data_files=None, description='Wikipedia dataset for fr, parsed from 20220301 dump.')
```
with correct version "2.0.0", as set in the custom config class.

The reason for this is that `DatasetBuilder` has a default VERSION ("0.0.0") that overwrites the default version set at the custom config class.

This PR:
- Removes the default VERSION at `DatasetBuilder` (set to None, so that the class attribute exists but it does not override the custom config default version).
- Note that the `BuilderConfig` class already sets a default version = "0.0.0"; no need to pass this from the builder.


Fix warning in upload_file

Fix warning:
```
FutureWarning: Pass path_or_fileobj='...' as keyword args. From version 0.7 passing these as positional arguments will result in an error
```


Problems with WMT dataset

Something isn't working

dataset bug

A bug in a dataset script provided in the library

Hi! Yes, the docs are outdated. Expect this to be fixed soon.

In the meantime, you can try to fix the issue yourself.

These are the configs/language pairs supported by `wmt15` from which you can choose:
* `cs-en` (Czech - English)
* `de-en` (German - English)
* `fi-en` (Finnish- English)
* `fr-en` (French - English)
* `ru-en` (Russian - English)

And the current implementation always uses all the subsets available for a language, so to define custom subsets, you'll have to clone the repo from the Hub and replace the line https://huggingface.co/datasets/wmt15/blob/main/wmt_utils.py#L688 with:
`for split, ss_names in (self._subsets if self.config.subsets is None else self.config.subsets).items()`

Then, you can load the dataset as follows:
```python
from datasets import load_dataset
dset = load_dataset("path/to/local/wmt15_folder", "<one of 5 available configs>", subsets=...)

@mariosasko thanks a lot for the suggested fix!

Hi @mariosasko

Are the docs updated? If not, I would like to get on it. I am new around here, would we helpful, if you can guide.

Thanks

Hi @khushmeeet! The docs haven't been updated, so feel free to work on this issue. This is a tricky issue, so I'll give the steps you can follow to fix this:

First, this code:
https://github.com/huggingface/datasets/blob/7cff5b9726a223509dbd6224de3f5f452c8d924f/src/datasets/load.py#L113-L118

needs to be replaced with (makes the dataset builder search more robust and allows us to remove the ABC stuff from `wmt_utils.py`):
```python
    for name, obj in module.__dict__.items():
        if inspect.isclass(obj) and issubclass(obj, main_cls_type):
            if inspect.isabstract(obj):
                continue
            module_main_cls = obj
            obj_module = inspect.getmodule(obj)
            if obj_module is not None and module == obj_module:
               break
```

Then, all the `wmt_utils.py` scripts need to be updated as follows (these are the diffs with the requiered changes):
````diff
 import os
 import re
 import xml.etree.cElementTree as ElementTree
-from abc import ABC, abstractmethod

 import datasets
````

````diff
logger = datasets.logging.get_logger(__name__)

 _DESCRIPTION = """\
-Translate dataset based on the data from statmt.org.
+Translation dataset based on the data from statmt.org.

-Versions exists for the different years using a combination of multiple data
-sources. The base `wmt_translate` allows you to create your own config to choose
-your own data/language pair by creating a custom `datasets.translate.wmt.WmtConfig`.
+Versions exist for different years using a combination of data
+sources. The base `wmt` allows you to create a custom dataset by choosing
+your own data/language pair. This can be done as follows:

 ```
-config = datasets.wmt.WmtConfig(
-    version="0.0.1",
+from datasets import inspect_dataset, load_dataset_builder
+
+inspect_dataset("<insert the dataset name", "path/to/scripts")
+builder = load_dataset_builder(
+    "path/to/scripts/wmt_utils.py",
     language_pair=("fr", "de"),
     subsets={
         datasets.Split.TRAIN: ["commoncrawl_frde"],
         datasets.Split.VALIDATION: ["euelections_dev2019"],
     },
 )
-builder = datasets.builder("wmt_translate", config=config)
-```

+# Standard version
+builder.download_and_prepare()
+ds = builder.as_dataset()
+
+# Streamable version
+ds = builder.as_streaming_dataset()
+```
 """
````

````diff
+class Wmt(datasets.GeneratorBasedBuilder):
     """WMT translation dataset."""
+
+    BUILDER_CONFIG_CLASS = WmtConfig

     def __init__(self, *args, **kwargs):
-        if type(self) == Wmt and "config" not in kwargs:  # pylint: disable=unidiomatic-typecheck
-            raise ValueError(
-                "The raw `wmt_translate` can only be instantiated with the config "
-                "kwargs. You may want to use one of the `wmtYY_translate` "
-                "implementation instead to get the WMT dataset for a specific year."
-            )
         super(Wmt, self).__init__(*args, **kwargs)

     @property
-    @abstractmethod
     def _subsets(self):
         """Subsets that make up each split of the dataset."""
````
```diff
         """Subsets that make up each split of the dataset for the language pair."""
         source, target = self.config.language_pair
         filtered_subsets = {}
-        for split, ss_names in self._subsets.items():
+        subsets = self._subsets if self.config.subsets is None else self.config.subsets
+        for split, ss_names in subsets.items():
             filtered_subsets[split] = []
             for ss_name in ss_names:
                 dataset = DATASET_MAP[ss_name]
```

`wmt14`, `wmt15`, `wmt16`, `wmt17`, `wmt18`, `wmt19` and `wmt_t2t` have this script, so all of them need to be updated. Also, the dataset summaries from the READMEs of these datasets need to be updated to match the new `_DESCRIPTION` string. And that's it! Let me know if you need additional help.

Hi @mariosasko ,

I have made the changes as suggested by you and have opened a PR #4537.

Thanks

## Describe the bug
I am trying to load WMT15 dataset and to define which data-sources to use for train/validation/test splits, but unfortunately it seems that the official documentation at [https://huggingface.co/datasets/wmt15#:~:text=Versions%20exists%20for,wmt_translate%22%2C%20config%3Dconfig)](https://huggingface.co/datasets/wmt15#:~:text=Versions%20exists%20for,wmt_translate%22%2C%20config%3Dconfig)) doesn't work anymore.

## Steps to reproduce the bug
```shell
>>> import datasets
>>> a = datasets.translate.wmt.WmtConfig()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'datasets' has no attribute 'translate'
>>> a = datasets.wmt.WmtConfig()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'datasets' has no attribute 'wmt'
```

## Expected results
To load WMT15 with given data-sources.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.10.0-10-amd64-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Don't strip proceeding hyphen

Closes #4320.


When using `dataset.map()` if passed `Features` types do not match what is returned from the mapped function, execution does not except in an obvious way

bug

Something isn't working

Hi ! Thanks for reporting :) `datasets` usually returns a `pa.lib.ArrowInvalid` error if the feature types don't match.

It would be awesome if we had a way to reproduce the `OverflowError` in this case, to better understand what happened and be able to provide the best error message

## Describe the bug
Recently I was trying to using `.map()` to preprocess a dataset. I defined the expected Features and passed them into `.map()` like `dataset.map(preprocess_data, features=features)`. My expected `Features` keys matched what came out of `preprocess_data`, but the types i had defined for them did not match the types that came back. Because of this, i ended up in tracebacks deep inside arrow_dataset.py and arrow_writer.py with exceptions that [did not make clear what the problem was](https://github.com/huggingface/datasets/issues/4349). In short i ended up with overflows and the OS killing processes when Arrow was attempting to write. It wasn't until I dug into `def write_batch` and the loop that loops over cols that I figured out what was going on.

It seems like `.map()` could set a boolean that it's checked that for at least 1 instance from the dataset, the returned data's types match the types provided by the `features` param and error out with a clear exception if they don't. This would make the cause of the issue much more understandable and save people time. This could be construed as a feature but it feels more like a bug to me.

## Steps to reproduce the bug
I don't have explicit code to repro the bug, but ill show an example

Code prior to the fix:
```python
def preprocess(examples):
    # returns an encoded data dict with keys that match the features, but the types do not match
...

def get_encoded_data(data):
  dataset = Dataset.from_pandas(data)
      unique_labels = data['audit_type'].unique().tolist()
      features = Features({
          'image': Array3D(dtype="uint8", shape=(3, 224, 224))),
          'input_ids': Sequence(feature=Value(dtype='int64'))),
          'attention_mask': Sequence(Value(dtype='int64'))),
          'token_type_ids': Sequence(Value(dtype='int64'))),
          'bbox': Array2D(dtype="int64", shape=(512, 4))),
          'label': ClassLabel(num_classes=len(unique_labels), names=unique_labels),
      })

      encoded_dataset = dataset.map(preprocess_data, features=features, remove_columns=dataset.column_names)
```

The Features set that fixed it:
```python
    features = Features({
        'image': Sequence(Array3D(dtype="uint8", shape=(3, 224, 224))),
        'input_ids': Sequence(Sequence(feature=Value(dtype='int64'))),
        'attention_mask': Sequence(Sequence(Value(dtype='int64'))),
        'token_type_ids': Sequence(Sequence(Value(dtype='int64'))),
        'bbox': Sequence(Array2D(dtype="int64", shape=(512, 4))),
        'label': ClassLabel(num_classes=len(unique_labels), names=unique_labels),
    })
```
The difference between my original code (which was based on documentation) and the working code is the addition of the `Sequence(...)` to 4/5 features as I am working with paginated data and the doc examples are not.

## Expected results
Dataset.map() attempts to validate the data types for each Feature on the first iteration and errors out if they are not validated.

## Actual results
Specify the actual results or traceback.
Based on the value of `writer_batch_size`, execution errors out when Arrow attempts to write because the types do not match, though its error messages dont make this obvious

Example errors:
```
OverflowError: There was an overflow with type <class 'list'>. Try to reduce writer_batch_size to have batches smaller than 2GB.
(offset overflow while concatenating arrays)
```

```
zsh: killed     python doc_classification.py

UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
datasets version: 2.1.0
Platform: macOS-12.2.1-arm64-arm-64bit
Python version: 3.9.12
PyArrow version: 6.0.1
Pandas version: 1.4.2


Add optional progress bar for .save_to_disk(..) and .load_from_disk(..) when working with remote filesystems

enhancement

New feature or request

Hi! I like this idea. For consistency with `load_dataset`, we can use `fsspec`'s `TqdmCallback` in  `.load_from_disk` to monitor the number of bytes downloaded, and in `.save_to_disk`, we can track the number of saved shards for consistency with `push_to_hub` (after we implement https://github.com/huggingface/datasets/issues/4196).

**Is your feature request related to a problem? Please describe.**
When working with large datasets stored on remote filesystems(such as s3), the process of uploading a dataset could take really long time. For instance: I was uploading a re-processed version of wmt17 en-ru to my s3 bucket and it took like 35 minutes(and that's given that I have a fiber optic connection). The only output during that process was a progress bar for flattening indices and then ~35 minutes of complete silence.

**Describe the solution you'd like**
I want to be able to enable a progress bar when calling .save_to_disk(..) and .load_from_disk(..), it would track either amount of bytes sent/received or amount of records written/loaded, and will give some ETA. Basically just tqdm.

**Describe alternatives you've considered**
- Save dataset to tmp folder at the disk and then upload it using custom wrapper over botocore, which will work with progress bar, like [this](https://alexwlchan.net/2021/04/s3-progress-bars/).


Add a new metric: CTC_Consistency

Thanks for your contribution, @YEdenZ.

Please note that our old `metrics` module is in the process of being incorporated to a separate library called `evaluate`: https://github.com/huggingface/evaluate

Therefore, I would ask you to transfer your PR to that repository. Thank you.

Add CTC_Consistency metric
Do I also need to modify the `test_metric_common.py` file to make it run on test?


Dataset.map()'s fails at any value of parameter writer_batch_size

bug

Something isn't working

Note that this same issue occurs even if i preprocess with the more default way of tokenizing that uses LayoutLMv2Processor's internal OCR:

```python
        feature_extractor = LayoutLMv2FeatureExtractor()
        tokenizer = LayoutLMv2Tokenizer.from_pretrained("microsoft/layoutlmv2-base-uncased")
        processor = LayoutLMv2Processor(feature_extractor, tokenizer)
        encoded_inputs = processor(images, padding="max_length", truncation=True)
        encoded_inputs["image"] = np.array(encoded_inputs["image"])
        encoded_inputs["label"] = examples['label_id']
```

Wanted to make sure anyone that finds this also finds my other report: https://github.com/huggingface/datasets/issues/4352

Did you close it because you found that it was due to the incorrect Feature types ?

Yeah-- my analysis of the issue was wrong in this one so I just closed it while linking to the new issue

I met with the same problem when doing some experiments about layoutlm. I tried to set the writer_batch_size to 1, and the error still exists. Is there any solutions to this problem?

The problem lies in how your Features are defined. It's erroring out when it actually goes to write them to disk

## Describe the bug
If the the value of `writer_batch_size` is less than the total number of instances in the dataset it will fail at that same number of instances. If it is greater than the total number of instances, it fails on the last instance.

Context:
I am attempting to fine-tune a pre-trained HuggingFace transformers model called LayoutLMv2. This model takes three inputs: document images, words and word bounding boxes. [The Processor for this model has two options](https://huggingface.co/docs/transformers/model_doc/layoutlmv2#usage-layoutlmv2processor), the default is passing a document to the Processor and allowing it to create images of the document and use PyTesseract to perform OCR and generate words/bounding boxes. The other option is to provide `revision="no_ocr"` to the pre-trained model which allows you to use your own OCR results (in my case, Amazon Textract) so you have to provide the image, words and bounding boxes yourself. I am using this second option which might be good context for the bug.

I am using the Dataset.map() paradigm to create these three inputs, encode them and save the dataset. Note that my documents (data instances) on average are fairly large and can range from 1 page up to 300 pages.
Code I am using is provided below

## Steps to reproduce the bug
I do not have explicit sample code, but I will paste the code I'm using in case reading it helps. When `.map()` is called, the dataset has 2933 rows, many of which represent large pdf documents.
```python
def get_encoded_data(data):
    dataset = Dataset.from_pandas(data)
    unique_labels = data['label'].unique()
    features = Features({
        'image': Array3D(dtype="int64", shape=(3, 224, 224)),
        'input_ids': Sequence(feature=Value(dtype='int64')),
        'attention_mask': Sequence(Value(dtype='int64')),
        'token_type_ids': Sequence(Value(dtype='int64')),
        'bbox': Array2D(dtype="int64", shape=(512, 4)),
        'label': ClassLabel(num_classes=len(unique_labels), names=unique_labels),
    })

    encoded_dataset = dataset.map(preprocess_data, features=features, remove_columns=dataset.column_names, writer_batch_size=dataset.num_rows+1)
    encoded_dataset.save_to_disk(TRAINING_DATA_PATH + ENCODED_DATASET_NAME)
    encoded_dataset.set_format(type="torch")
    return encoded_dataset
```
```python
PROCESSOR = LayoutLMv2Processor.from_pretrained(MODEL_PATH, revision="no_ocr", use_fast=False)

def preprocess_data(examples):
    directory = os.path.join(FILES_PATH, examples['file_location'])
    images_dir = os.path.join(directory, PDF_IMAGE_DIR)
    textract_response_path = os.path.join(directory, 'textract.json')
    doc_meta_path = os.path.join(directory, 'doc_meta.json')
    textract_document = get_textract_document(textract_response_path, doc_meta_path)
    images, words, bboxes = get_doc_training_data(images_dir, textract_document)
    encoded_inputs = PROCESSOR(images, words, boxes=bboxes, padding="max_length", truncation=True)
    # https://github.com/NielsRogge/Transformers-Tutorials/issues/36
    encoded_inputs["image"] = np.array(encoded_inputs["image"])
    encoded_inputs["label"] = examples['label_id']
    return encoded_inputs
```

## Expected results
My expectation is that `writer_batch_size` allows one to simply trade off performance and memory requirements, not that it must be a specific number for `.map()` to function correctly.

## Actual results
If writer_batch_size is set to a value less than the number of rows, I get either:

```
OverflowError: There was an overflow with type <class 'list'>. Try to reduce writer_batch_size to have batches smaller than 2GB.
(offset overflow while concatenating arrays)
```
or simply

```
zsh: killed     python doc_classification.py

UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
```

If it is greater than the number of rows, i get the `zsh: killed` error above

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.1.0
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.12
- PyArrow version: 6.0.1
- Pandas version: 1.4.2


`inspect` functions can't fetch dataset script from the Hub

Something isn't working

Hi, thanks for reporting! `git bisect` points to #2986 as the PR that introduced the bug. Since then, there have been some additional changes to the loading logic, and in the current state, `force_local_path` (set via `local_path`) forbids pulling a script from the internet instead of downloading it: https://github.com/huggingface/datasets/blob/cfae0545b2ba05452e16136cacc7d370b4b186a1/src/datasets/inspect.py#L89-L91

cc @lhoestq: `force_local_path` is only used in `inspect_dataset` and `inspect_metric`. Is it OK if we revert the behavior to match the old one?

The `inspect_dataset` and `inspect_metric` functions are unable to retrieve a dataset or metric script from the Hub and store it locally at the specified `local_path`:

```py
>>> from datasets import inspect_dataset
>>> inspect_dataset('rotten_tomatoes', local_path='path/to/my/local/folder')

FileNotFoundError: Couldn't find a dataset script at /content/rotten_tomatoes/rotten_tomatoes.py or any data file in the same directory.
```


Support remote cache_dir

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq thanks for your review.

Please note that `xjoin` cannot be used in this context, as it always returns a POSIX path string and this is not suitable on Windows machines.

<s>`xjoin` returns windows paths (not posix) on windows, since it just extends`os.path.join` </s>

Actually you are right.

https://github.com/huggingface/datasets/blob/08ec04ccb59630a3029b2ecd8a14d327bddd0c4a/src/datasets/utils/streaming_download_manager.py#L104-L105

Though this is not an issue because posix paths (as returned by Path().as_posix()) work on windows. That's why we can replace `os.path.join` with `xjoin` in streaming mode. They look like `c:/Program Files/` or something (can't confirm right now, I don't have a windows with me)

Until now, we have always replaced "/" in paths with `os.path.join` (`os.sep`,...) in order to support Windows paths (that contain r"\\").

Now, you suggest ignoring this and work with POSIX strings (with "/").

As an example, when passing `cache_dir=r"C:\Users\Username\.mycache"`:
- Until now, it results in `self._cache_downloaded_dir = r"C:\Users\Username\.mycache\downloads"`
- If we use `xjoin`, it will give `self._cache_downloaded_dir = "C:/Users/Username/.mycache/downloads"`

You say this is OK and we don't care if we work with POSIX strings on Windows machines.

I'm incorporating your suggested changes then...

Also note that using `xjoin`, if we pass `cache_dir="C:\\Users\\Username\\.mycache"`, we get:
- `self._cache_dir_root = "C:\\Users\\Username\\.mycache"`
- `self._cache_downloaded_dir = "C:/Users/Username/.mycache/downloads"`

It looks like it broke the CI on windows :/ maybe this was not a good idea, sorry

This PR implements complete support for remote `cache_dir`. Before, the support was just partial.

This is useful to create datasets using Apache Beam (parallel data processing) builder with `cache_dir` in a remote bucket, e.g., for Wikipedia dataset.


GH Action to build documentation never ends

Something isn't working

## Describe the bug
See: https://github.com/huggingface/datasets/runs/6418035586?check_suite_focus=true

I finally forced the cancel of the workflow.


Fix never ending GH Action to build documentation

There was an unclosed code block introduced by:
- #4313

https://github.com/huggingface/datasets/pull/4313/files#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R538

This causes the "Make documentation" step in the "Build documentation" workflow to never finish.
- I think this issue should also be addressed in the `doc-builder` lib.

Fix #4346.


Fix docstring in DatasetDict::shuffle

I think  due to #1626, the docstring contained this error ever since `seed` was added.