2025-05-28 21:30:28,646 INFO MainThread:907963 [wandb_setup.py:_flush():67] Current SDK version is 0.19.8 2025-05-28 21:30:28,646 INFO MainThread:907963 [wandb_setup.py:_flush():67] Configure stats pid to 907963 2025-05-28 21:30:28,646 INFO MainThread:907963 [wandb_setup.py:_flush():67] Loading settings from /home/hansirui_1st/.config/wandb/settings 2025-05-28 21:30:28,646 INFO MainThread:907963 [wandb_setup.py:_flush():67] Loading settings from /home/hansirui_1st/jiayi/resist/setting3/scripts/wandb/settings 2025-05-28 21:30:28,646 INFO MainThread:907963 [wandb_setup.py:_flush():67] Loading settings from environment variables 2025-05-28 21:30:28,647 INFO MainThread:907963 [wandb_init.py:setup_run_log_directory():647] Logging user logs to /aifs4su/hansirui_1st/boyuan/resist/setting3-safety/Qwen1.5-4B/Qwen1.5-4B-s3-Q1-1k-Q2-200/wandb/run-20250528_213028-26lxigzr/logs/debug.log 2025-05-28 21:30:28,647 INFO MainThread:907963 [wandb_init.py:setup_run_log_directory():648] Logging internal logs to /aifs4su/hansirui_1st/boyuan/resist/setting3-safety/Qwen1.5-4B/Qwen1.5-4B-s3-Q1-1k-Q2-200/wandb/run-20250528_213028-26lxigzr/logs/debug-internal.log 2025-05-28 21:30:28,647 INFO MainThread:907963 [wandb_init.py:init():761] calling init triggers 2025-05-28 21:30:28,647 INFO MainThread:907963 [wandb_init.py:init():766] wandb.init called with sweep_config: {} config: {'model_name_or_path': '/aifs4su/hansirui_1st/boyuan/resist/setting3-safety/Qwen1.5-4B/Qwen1.5-4B-s3-Q1-1k', 'max_length': 2048, 'trust_remote_code': True, 'train_datasets': [('inverse-json', {'proportion': 1.0, 'path': '/home/hansirui_1st/jiayi/resist/setting3/safety_data/training/unsafe/unsafe_200.json'})], 'eval_datasets': None, 'epochs': 1, 'per_device_train_batch_size': 4, 'per_device_eval_batch_size': 4, 'gradient_accumulation_steps': 8, 'gradient_checkpointing': True, 'lr': 1e-05, 'lr_scheduler_type': , 'lr_warmup_ratio': 0.0, 'weight_decay': 0.0, 'seed': 42, 'fp16': False, 'bf16': True, 'tf32': True, 'eval_strategy': 'epoch', 'eval_interval': 1000000, 'need_eval': False, 'eval_split_ratio': None, 'output_dir': '/aifs4su/hansirui_1st/boyuan/resist/setting3-safety/Qwen1.5-4B/Qwen1.5-4B-s3-Q1-1k-Q2-200', 'log_type': 'wandb', 'log_dir': '/aifs4su/hansirui_1st/boyuan/resist/setting3-safety/Qwen1.5-4B/Qwen1.5-4B-s3-Q1-1k-Q2-200', 'log_project': 'Inverse_Alignment', 'log_run_name': 'qwen-4b-s3-Q1-1k-Q2-200', 'save_16bit': True, 'save_interval': 1000000, 'local_rank': 0, 'zero_stage': 3, 'offload': 'none', 'deepspeed': False, 'deepspeed_config': None, 'deepscale': False, 'deepscale_config': None, 'global_rank': 0, 'device': device(type='cuda', index=0), 'num_update_steps_per_epoch': 1, 'total_training_steps': 1, '_wandb': {}} 2025-05-28 21:30:28,647 INFO MainThread:907963 [wandb_init.py:init():784] starting backend 2025-05-28 21:30:28,647 INFO MainThread:907963 [wandb_init.py:init():788] sending inform_init request 2025-05-28 21:30:28,652 INFO MainThread:907963 [backend.py:_multiprocessing_setup():101] multiprocessing start_methods=fork,spawn,forkserver, using: spawn 2025-05-28 21:30:28,652 INFO MainThread:907963 [wandb_init.py:init():798] backend started and connected 2025-05-28 21:30:28,656 INFO MainThread:907963 [wandb_init.py:init():891] updated telemetry 2025-05-28 21:30:28,656 INFO MainThread:907963 [wandb_init.py:init():915] communicating run to backend with 90.0 second timeout 2025-05-28 21:30:29,124 INFO MainThread:907963 [wandb_init.py:init():990] starting run threads in backend 2025-05-28 21:30:29,353 INFO MainThread:907963 [wandb_run.py:_console_start():2375] atexit reg 2025-05-28 21:30:29,353 INFO MainThread:907963 [wandb_run.py:_redirect():2227] redirect: wrap_raw 2025-05-28 21:30:29,353 INFO MainThread:907963 [wandb_run.py:_redirect():2292] Wrapping output streams. 2025-05-28 21:30:29,353 INFO MainThread:907963 [wandb_run.py:_redirect():2315] Redirects installed. 2025-05-28 21:30:29,356 INFO MainThread:907963 [wandb_init.py:init():1032] run started, returning control to user process 2025-05-28 21:30:53,095 INFO MainThread:907963 [wandb_run.py:_finish():2112] finishing run xtom/Inverse_Alignment/26lxigzr 2025-05-28 21:30:53,096 INFO MainThread:907963 [wandb_run.py:_atexit_cleanup():2340] got exitcode: 0 2025-05-28 21:30:53,097 INFO MainThread:907963 [wandb_run.py:_restore():2322] restore 2025-05-28 21:30:53,098 INFO MainThread:907963 [wandb_run.py:_restore():2328] restore done 2025-05-28 21:30:54,099 INFO MainThread:907963 [wandb_run.py:_restore():2322] restore 2025-05-28 21:30:54,099 INFO MainThread:907963 [wandb_run.py:_restore():2328] restore done 2025-05-28 21:30:54,099 ERROR MainThread:907963 [wandb_run.py:_atexit_cleanup():2361] Problem finishing run Traceback (most recent call last): File "/aifs4su/hansirui_1st/miniconda3/envs/by-align/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 2352, in _atexit_cleanup self._on_finish() File "/aifs4su/hansirui_1st/miniconda3/envs/by-align/lib/python3.11/site-packages/wandb/sdk/wandb_run.py", line 2609, in _on_finish wait_with_progress( File "/aifs4su/hansirui_1st/miniconda3/envs/by-align/lib/python3.11/site-packages/wandb/sdk/mailbox/wait_with_progress.py", line 24, in wait_with_progress return wait_all_with_progress( ^^^^^^^^^^^^^^^^^^^^^^^ File "/aifs4su/hansirui_1st/miniconda3/envs/by-align/lib/python3.11/site-packages/wandb/sdk/mailbox/wait_with_progress.py", line 87, in wait_all_with_progress return asyncio_compat.run(progress_loop_with_timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/aifs4su/hansirui_1st/miniconda3/envs/by-align/lib/python3.11/site-packages/wandb/sdk/lib/asyncio_compat.py", line 27, in run future = executor.submit(runner.run, fn) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/aifs4su/hansirui_1st/miniconda3/envs/by-align/lib/python3.11/concurrent/futures/thread.py", line 169, in submit raise RuntimeError('cannot schedule new futures after ' RuntimeError: cannot schedule new futures after interpreter shutdown 2025-05-28 21:30:54,359 INFO MsgRouterThr:907963 [mailbox.py:close():129] Closing mailbox, abandoning 2 handles.