nvidia-smi

Verified against the NVIDIA System Management Interface documentation.

Commands

What cards are in the box, with stable identifiers:

nvidia-smi -L

Watch, rather than sample once:

nvidia-smi -l 5

Machine readable, which is the only form worth scripting against:

nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu \
  --format=csv,noheader,nounits

Which processes hold GPU memory:

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

Per process and per device monitoring:

nvidia-smi pmon -i 0
nvidia-smi dmon -s pucvmet

How the cards are wired to each other, which decides multi GPU throughput:

nvidia-smi topo -m

Keep the driver loaded so the first job does not pay the initialisation cost:

sudo nvidia-smi -pm 1

Every field name the query options accept:

nvidia-smi --help-query-gpu

Notes

--format=csv is mandatory with --query-gpu. The noheader and nounits modifiers are what make the output safe to parse.

-i accepts an index, a UUID or a PCI bus id. Indexes are reassigned across reboots, so anything durable should use the UUID from nvidia-smi -L.

Persistence mode needs root, applies only on Linux, and does not survive a reboot. Set it from a unit file rather than by hand.

topo -m prints a legend: NV# is an NVLink connection, SYS crosses the CPU socket, and a job spread across SYS pairs will be slower than the same job on one node.

Links

• nvidia-smi documentation