runtime error

Exit code: 1. Reason: tokenizer.json: 100%|██████████| 11.4M/11.4M [00:00<00:00, 113MB/s] tokenizer_config.json: 0%| | 0.00/9.38k [00:00<?, ?B/s] tokenizer_config.json: 100%|██████████| 9.38k/9.38k [00:00<00:00, 1.29MB/s] vocab.json: 0%| | 0.00/2.78M [00:00<?, ?B/s] vocab.json: 100%|██████████| 2.78M/2.78M [00:00<00:00, 77.4MB/s] Traceback (most recent call last): File "/home/user/app/app.py", line 333, in <module> model = AutoModelForCausalLM.from_pretrained( File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 558, in from_pretrained model_class = get_class_from_dynamic_module( File "/usr/local/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 570, in get_class_from_dynamic_module return get_class_in_module(class_name, final_module, force_reload=force_download) File "/usr/local/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 267, in get_class_in_module module_spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/home/user/.cache/huggingface/modules/transformers_modules/dots-ocr-local/modeling_dots_ocr.py", line 8, in <module> from .modeling_dots_vision import DotsVisionTransformer File "/home/user/.cache/huggingface/modules/transformers_modules/dots-ocr-local/modeling_dots_vision.py", line 7, in <module> from flash_attn import flash_attn_varlen_func File "/usr/local/lib/python3.10/site-packages/flash_attn/__init__.py", line 3, in <module> from flash_attn.flash_attn_interface import ( File "/usr/local/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 15, in <module> import flash_attn_2_cuda as flash_attn_gpu ImportError: /usr/local/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa

Container logs: