vLLM 0.13.0 causes error due to deprecated VLLM_ATTENTION_BACKEND environment variable

[3manifold]

Cmdline

  VLLM_USE_V1=0 \
  FORCE_QWENVL_VIDEO_READER=torchcodec  \
  USE_AUDIO_IN_VIDEO=True \
  ENABLE_AUDIO_OUTPUT=False \
  FPS_MAX_FRAMES=4 \
  swift infer \
      --model "Qwen3-Omni-30B-A3B-Instruct" 
      --infer_backend vllm \
      --val_dataset "$WORK_DIR/query.jsonl" \
      --result_path "$WORK_DIR/captions_Qwen3-Omni-30B-A3B-Instruct.jsonl"

Error

[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 253, in flash_attn_varlen_func
[rank0]:     out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
[rank0]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
[rank0]:     return self._op(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.AcceleratorError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
[rank0]: Search for `cudaErrorUnsupportedPtxVersion' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[rank0]:[W1219 15:23:53.162823414 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
E1219 15:23:55.158000 1229 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 0 (pid: 1295) of binary: /opt/conda/bin/python3.11
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/run.py", line 940, in <module>
    main()
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/run.py", line 936, in main
    run(args)
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/opt/conda/lib/python3.11/site-packages/swift/cli/infer.py FAILED

Notes

It relates to vllm-project/vllm#26315 and

https://github.com/vllm-project/vllm/releases/tag/v0.13.0

Breaking Changes & Deprecations
PassConfig flags renamed per RFC vllm-project/vllm#27995 (vllm-project/vllm#29646)
Attention env vars → CLI args: VLLM_ATTENTION_BACKEND replaced with --attention-backend (vllm-project/vllm#26315)

System specs

vllm 0.13.0
ms_swift 3.11.1

Python version: 3.11.14 | packaged by conda-forge | (main, Oct 13 2025, 14:09:32) [GCC 14.3.0]
PyTorch version: 2.9.0+cu128 
CUDA version: 12.8
CUDNN version: 91002 
Name of current CUDA device: NVIDIA H200

[alanshaoTT]

hi！i got the same problem, did you solved this issue? i notice you post #7133, do you mean we can add the --vllm_engine_kwargs to solve this problem.
my issue is like following, i will be very thankful if you can give me some suggestion, i have tried several ways but not helpful...
my scripts is like this:

export CUDA_VISIBLE_DEVICES=0,1
export VLLM_USE_V1=0
export VLLM_ATTENTION_BACKEND=FLASHINFER
VLLM_USE_V1=0 \
swift rollout \
    --model "qwen3omni" \
    --vllm_tensor_parallel_size 2 \
    --vllm_max_model_len 16384 \
    --vllm_gpu_memory_utilization 0.6 \
    --multi_turn_scheduler AudioAgentScheduler \
    --vllm_limit_mm_per_prompt '{"audio": 10}' \
    --vllm_enforce_eager true \
    --external_plugins "${SWIFT_WORKSPACE}/examples/train/multimodal/my-RL/exp9_plugin.py" \
    --port 8000 \

and error like this

(Worker_TP0 pid=2421860) INFO 02-12 23:57:17 [gpu_model_runner.py:3562] Starting to load model /mnt/data/share-oss/user/shaomingchen/ckpt/long-audio/time-aware/ceshi/exp6-agentic-TAG/v1-20260207-183313/checkpoint-900...
(Worker_TP1 pid=2421861) INFO 02-12 23:57:17 [mm_encoder_attention.py:104] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP0 pid=2421860) INFO 02-12 23:57:17 [mm_encoder_attention.py:104] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(Worker_TP1 pid=2421861) WARNING 02-12 23:57:17 [vllm.py:629] Inductor compilation was disabled by user settings,Optimizations settings that are only active
duringInductor compilation will be ignored.
(Worker_TP1 pid=2421861) INFO 02-12 23:57:17 [vllm.py:722] Cudagraph is disabled under eager mode
(Worker_TP0 pid=2421860) WARNING 02-12 23:57:17 [vllm.py:629] Inductor compilation was disabled by user settings,Optimizations settings that are only active
duringInductor compilation will be ignored.
(Worker_TP0 pid=2421860) INFO 02-12 23:57:17 [vllm.py:722] Cudagraph is disabled under eager mode
(Worker_TP1 pid=2421861) INFO 02-12 23:57:18 [cuda.py:315] Using AttentionBackendEnum.FLASHINFER backend.
(Worker_TP0 pid=2421860) INFO 02-12 23:57:18 [cuda.py:315] Using AttentionBackendEnum.FLASHINFER backend.
(Worker_TP0 pid=2421860) INFO 02-12 23:57:18 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
(Worker_TP0 pid=2421860) INFO 02-12 23:57:18 [gpu_model_runner.py:3659] Model loading took 30.2648 GiB memory and 0.679824 seconds
(Worker_TP0 pid=2421860) INFO 02-12 23:57:19 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 12544 tokens, and profiled with 1
image items of the maximum feature size.
(Worker_TP1 pid=2421861) INFO 02-12 23:57:19 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 12544 tokens, and profiled with 1
image items of the maximum feature size.
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824] WorkerProc hit an exception.
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824] Traceback (most recent call last):
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 819, in worker_busy_loop
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     output = func(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return func(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 340, in determine_available_memory
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     self.model_runner.profile_run()
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4462, in profile_run
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     dummy_encoder_outputs = self.model.embed_multimodal(
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_omni_moe_thinker.py", line 1324, in embed_multimodal
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     image_embeddings = self._process_image_input(multimodal_input)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 734, in _process_image_input
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     image_embeds = self.visual(pixel_values, grid_thw=grid_thw)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_omni_moe_thinker.py", line 543, in forward
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     hidden_states = blk(
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_omni_moe_thinker.py", line 229, in forward
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     x = x + self.attn(
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 401, in forward
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     context_layer = self.attn(
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/custom_op.py", line 47, in forward
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._forward_method(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/attention/layers/mm_encoder_attention.py", line 229, in forward_cuda
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._forward_fa(query, key, value, cu_seqlens, max_seqlen)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/attention/layers/mm_encoder_attention.py", line 199, in _forward_fa
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     output = vit_flash_attn_wrapper(
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/attention/ops/vit_attn_wrappers.py", line 80, in vit_flash_attn_wrapper
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return torch.ops.vllm.flash_attn_maxseqlen_wrapper(
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/_ops.py", line 1255, in __call__
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._op(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/attention/ops/vit_attn_wrappers.py", line 37, in flash_attn_maxseqlen_wrapper
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824] WorkerProc hit an exception.
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     output = flash_attn_varlen_func(
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 253, in flash_attn_varlen_func
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824] Traceback (most recent call last):
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 819, in worker_busy_loop
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/_ops.py", line 1255, in __call__
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     output = func(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._op(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824] torch.AcceleratorError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return func(*args, **kwargs)
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824] Search for `cudaErrorUnsupportedPtxVersion' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 340, in determine_available_memory
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824] CUDA kernel errors might be asynchronously reported at some other API call, so the
stacktrace below might be incorrect.
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     self.model_runner.profile_run()
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4462, in profile_run
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     dummy_encoder_outputs = self.model.embed_multimodal(
(Worker_TP1 pid=2421861) ERROR 02-12 23:57:21 [multiproc_executor.py:824]
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_omni_moe_thinker.py", line 1324, in embed_multimodal
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     image_embeddings = self._process_image_input(multimodal_input)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_omni_thinker.py", line 734, in _process_image_input
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     image_embeds = self.visual(pixel_values, grid_thw=grid_thw)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_omni_moe_thinker.py", line 543, in forward
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     hidden_states = blk(
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_omni_moe_thinker.py", line 229, in forward
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     x = x + self.attn(
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 401, in forward
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     context_layer = self.attn(
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/model_executor/custom_op.py", line 47, in forward
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._forward_method(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/attention/layers/mm_encoder_attention.py", line 229, in forward_cuda
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._forward_fa(query, key, value, cu_seqlens, max_seqlen)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/attention/layers/mm_encoder_attention.py", line 199, in _forward_fa
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     output = vit_flash_attn_wrapper(
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/attention/ops/vit_attn_wrappers.py", line 80, in vit_flash_attn_wrapper
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return torch.ops.vllm.flash_attn_maxseqlen_wrapper(
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/_ops.py", line 1255, in __call__
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._op(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/attention/ops/vit_attn_wrappers.py", line 37, in flash_attn_maxseqlen_wrapper
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     output = flash_attn_varlen_func(
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 253, in flash_attn_varlen_func
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd(
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/torch/_ops.py", line 1255, in __call__
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]     return self._op(*args, **kwargs)
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824] torch.AcceleratorError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824] Search for `cudaErrorUnsupportedPtxVersion' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824] CUDA kernel errors might be asynchronously reported at some other API call, so the
stacktrace below might be incorrect.
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(Worker_TP0 pid=2421860) ERROR 02-12 23:57:21 [multiproc_executor.py:824]
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866] EngineCore failed to start.
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866] Traceback (most recent call last):
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]     super().__init__(
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 240, in _initialize_kv_caches
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 126, in determine_available_memory
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 359, in collective_rpc
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]     return aggregate(get_response())
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 342, in get_response
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866]     raise RuntimeError(
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866] RuntimeError: Worker failed with error 'CUDA error: the provided PTX was compiled with an unsupported toolchain.
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866] Search for `cudaErrorUnsupportedPtxVersion' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:21 [core.py:866] ', please check the stack trace above for the root cause
(EngineCore_DP0 pid=2421703) ERROR 02-12 23:57:23 [multiproc_executor.py:231] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.
(EngineCore_DP0 pid=2421703) Process EngineCore_DP0:
(EngineCore_DP0 pid=2421703) Traceback (most recent call last):
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/multiprocessing/process.py", line 314, in
_bootstrap
(EngineCore_DP0 pid=2421703)     self.run()
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/multiprocessing/process.py", line 108, in
run
(EngineCore_DP0 pid=2421703)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 870, in run_engine_core
(EngineCore_DP0 pid=2421703)     raise e
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=2421703)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=2421703)     super().__init__(
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=2421703)     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 240, in _initialize_kv_caches
(EngineCore_DP0 pid=2421703)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 126, in determine_available_memory
(EngineCore_DP0 pid=2421703)     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 359, in collective_rpc
(EngineCore_DP0 pid=2421703)     return aggregate(get_response())
(EngineCore_DP0 pid=2421703)   File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 342, in get_response
(EngineCore_DP0 pid=2421703)     raise RuntimeError(
(EngineCore_DP0 pid=2421703) RuntimeError: Worker failed with error 'CUDA error: the provided PTX was compiled with an unsupported toolchain.
(EngineCore_DP0 pid=2421703) Search for `cudaErrorUnsupportedPtxVersion' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more
information.
(EngineCore_DP0 pid=2421703) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=2421703) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=2421703) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=2421703) ', please check the stack trace above for the root cause
Process Process-1:
Traceback (most recent call last):
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/mnt/data/share-ssd/user/shaomingchen/code/long-audio/time-aware/ft_omni/ms-swift-2-9/swift/pipelines/infer/rollout.py", line 314, in llm_worker_entry
    asyncio.run(async_llm_worker(*args, **kwargs))
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/mnt/data/share-ssd/user/shaomingchen/code/long-audio/time-aware/ft_omni/ms-swift-2-9/swift/pipelines/infer/rollout.py", line 282, in async_llm_worker
    engine = SwiftRolloutDeploy.get_infer_engine(args, template=args.get_template(), seed=worker_seed)
  File "/mnt/data/share-ssd/user/shaomingchen/code/long-audio/time-aware/ft_omni/ms-swift-2-9/swift/pipelines/infer/rollout.py", line 412, in get_infer_engine
    return GRPOVllmEngine(**kwargs)
  File "/mnt/data/share-ssd/user/shaomingchen/code/long-audio/time-aware/ft_omni/ms-swift-2-9/swift/infer_engine/grpo_vllm_engine.py", line 65, in __init__
    super().__init__(
  File "/mnt/data/share-ssd/user/shaomingchen/code/long-audio/time-aware/ft_omni/ms-swift-2-9/swift/infer_engine/vllm_engine.py", line 153, in __init__
    self._prepare_engine()
  File "/mnt/data/share-ssd/user/shaomingchen/code/long-audio/time-aware/ft_omni/ms-swift-2-9/swift/infer_engine/vllm_engine.py", line 175, in _prepare_engine
    engine = llm_engine_cls.from_engine_args(self.engine_args)
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 244, in from_engine_args
    return cls(
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 134, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client
    return AsyncMPClient(*client_args)
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 820, in __init__
    super().__init__(
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 477, in __init__
    with launch_core_engines(vllm_config, executor_class, log_stats) as (
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 903, in launch_core_engines
    wait_for_engine_startup(
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 960, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
ERROR:    Traceback (most recent call last):
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/site-packages/starlette/routing.py", line 694, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/mnt/data/share-ssd/user/shaomingchen/code/long-audio/time-aware/ft_omni/ms-swift-2-9/swift/pipelines/infer/rollout.py", line 363, in lifespan
    msg = connection.recv()
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/mnt/data/share-ssd/user/shaomingchen/miniconda3/envs/swift-rl/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

ERROR:    Application startup failed. Exiting.

my platform is h200 with cuda 12.8 and Driver Version: 570.124.06.
environment is vllm 0.13.0 msswift 0.4.0.dev0 transformers 4.57.6 torch 2.9.0 torchaudio 2.9.0 torchvision 0.24.0.
i tried the command line in your PR, tried
--vllm_engine_kwargs '{"attention_config": {"backend": "FLASHINFER"}}' and
--vllm_engine_kwargs '{"attention_config": {"backend": "FLASH_ATTN"}}' but failed again....

[3manifold]

You can have a look at the source code of vllm for the version you use & check how FLASH_ATTN is handled.