500 Server error with pushing a dataset

[martinmatak]

Describe the bug

Suddenly, I started getting this error message saying it was an internal error.

`Error creating/pushing dataset: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/ll4ma-lab/grasp-dataset/commit/main (Request ID: Root=1-6787f0b7-66d5bd45413e481c4c2fb22d;670d04ff-65f5-4741-a353-2eacc47a3928)

Internal Error - We're working hard to fix this as soon as possible!
Traceback (most recent call last):
File "/uufs/chpc.utah.edu/common/home/hermans-group1/martin/software/pkg/miniforge3/envs/myenv2/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 406, in hf_raise_for_status
response.raise_for_status()
File "/uufs/chpc.utah.edu/common/home/hermans-group1/martin/software/pkg/miniforge3/envs/myenv2/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/ll4ma-lab/grasp-dataset/commit/main

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/uufs/chpc.utah.edu/common/home/u1295595/grasp_dataset_converter/src/grasp_dataset_converter/main.py", line 142, in main
subset_train.push_to_hub(dataset_name, split='train')
File "/uufs/chpc.utah.edu/common/home/hermans-group1/martin/software/pkg/miniforge3/envs/myenv2/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 5624, in push_to_hub
commit_info = api.create_commit(
File "/uufs/chpc.utah.edu/common/home/hermans-group1/martin/software/pkg/miniforge3/envs/myenv2/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/uufs/chpc.utah.edu/common/home/hermans-group1/martin/software/pkg/miniforge3/envs/myenv2/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1518, in _inner
return fn(self, *args, **kwargs)
File "/uufs/chpc.utah.edu/common/home/hermans-group1/martin/software/pkg/miniforge3/envs/myenv2/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 4087, in create_commit
hf_raise_for_status(commit_resp, endpoint_name="commit")
File "/uufs/chpc.utah.edu/common/home/hermans-group1/martin/software/pkg/miniforge3/envs/myenv2/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 477, in hf_raise_for_status
raise _format(HfHubHTTPError, str(e), response) from e
huggingface_hub.errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/ll4ma-lab/grasp-dataset/commit/main (Request ID: Root=1-6787f0b7-66d5bd45413e481c4c2fb22d;670d04ff-65f5-4741-a353-2eacc47a3928)

Internal Error - We're working hard to fix this as soon as possible!`

Steps to reproduce the bug

I am pushing a Dataset in a loop via push_to_hub API

Expected behavior

It worked fine until it stopped working suddenly.
Expected behavior: It should start working again

Environment info

• datasets version: 3.2.0
• Platform: Linux-4.18.0-477.15.1.el8_8.x86_64-x86_64-with-glibc2.28
• Python version: 3.10.0
• huggingface_hub version: 0.27.1
• PyArrow version: 18.1.0
• Pandas version: 2.2.3
• fsspec version: 2024.9.0

[martinmatak]

EDIT: seems to be all good now. I'll add a comment if the error happens again within the next 48 hours. If it doesn't, I'll just close the topic.