What is the meaning of the statement 'only enumeration of a single MIG instance is supported'?

AI & Data Science Deep Learning (Training & Inference)
1

Hi,Please see the illustration below:

image
image623×671 80.3 KB

I know that it is not possible to use both MIG instance ① and MIG instance ② on a single GPU simultaneously. However, why does the code torch.cuda.device_count() only return the first MIG instance when I specify MIG instance ① and ③ (or ① and ②) using the CUDA_VISIBLE_DEVICES environment variable? Does this mean that only one MIG instance can be used per GPU (or across all GPUs) ? If so, aren’t the other GPUs being wasted?

Host Environment:
GPU:NVIDIA H20
Driver Version: 550.163.01
Docker version 27.3.1, build ce12230
NVIDIA Container Runtime Hook version 1.16.2

Container Environment:
CUDA release 12.1, V12.1.66
torch==2.2.1+cu121

When I tested using the code torch.cuda.device_count() on NVIDIA drivers 565 and 570, I was able to see all the MIG instances I specified. Does this mean that driver version 550 does not support this functionality ?