Filename too long error

[adamkarvonen]

I got this error when using snapshot_download():

OSError: [Errno 36] File name too long: 'pythia-4k-backup/.cache/huggingface/download/autointerp_with_generations/saebench_pythia-160m-deduped_width-2pow12_date-0108/saebench_pythia-160m-deduped_width-2pow12_date-0108_StandardTrainerAprilUpdate_EleutherAI_pythia-160m-deduped_ctx1024_0108_resid_post_layer_8_trainer_10_custom_sae_eval_results.json.81ca464035a619f824d841360a423337f239b86c07092010f6df8e64fef98b74.incomplete'

My original filename is around ~180 characters, and the limit on my OS was 255. huggingface-hub added ~80 characters to the intermediate filename, causing the Filename too long error. I think huggingface-hub would be fine with using only 20 or 30 characters for the hash and incomplete filename extension, which would limit filename length errors.

Possibly some logic could be added so the hash could be shortened if the filename length is > 255 characters.

[hanouticelina]

Hi @adamkarvonen, sorry you encountered this issue.
the hash is a SHA256 (64 characters) returned by the server and is a critical part used for security and consistency checks.
the filepath alone is already quite long, which is of size 261:

>>> len("autointerp_with_generations/saebench_pythia-160m-deduped_width-2pow12_date-0108/saebench_pythia-160m-deduped_width-2pow12_date-0108_StandardTrainerAprilUpdate_EleutherAI_pythia-160m-deduped_ctx1024_0108_resid_post_layer_8_trainer_10_custom_sae_eval_results.json")
261

The filepath alone exceeds the 255 character limit on most systems, even before adding the necessary cache management components. The best way to fix this would be to reduce the filepath length on your side.