GPU inference host setup

GlobalStacks runs managed GPU inference in private containers on the connected host. The host owner installs and maintains the NVIDIA driver and container runtime; the agent reports whether that configuration is usable before GPU capacity can be scheduled.

Compatibility

Choose a worker image for the exact NVIDIA architecture reported by the host agent. The report uses CUDA SM features such as cuda:sm_61.

• GTX 1080 Ti is Pascal, compute capability 6.1, so use sm_61 images built with CUDA 12.x. The CUDA image must still be supported by the installed driver: for example, NVIDIA driver 550.163.01 was verified with CUDA 12.0 and cannot run a CUDA 12.6 image (that image requires a newer driver).
• CUDA 13 does not support building applications for Pascal, Maxwell, or Volta.
• Modern vLLM and PyTorch images may require newer architectures or Tensor Cores; use a reviewed llama.cpp CUDA image for Pascal rather than assuming a current generic image will work.
• Current Ollama releases require NVIDIA driver 570 or newer for compute capability 5.0 through 6.2. On a GTX 1080 Ti with driver 550, use a pinned legacy runtime such as Ollama 0.5.7 or a CUDA 12.0 llama.cpp build compiled for sm_61; verify the model is actually offloaded before exposing it to users.

Configure Docker GPU access

Install an NVIDIA driver that supports the selected CUDA worker image. Driver kernel modules and user-space NVIDIA libraries must be the same version. Do not mix a host-mounted driver library with a newer Ubuntu NVIDIA library package.

Install NVIDIA Container Toolkit from NVIDIA’s signed repository, configure Docker, then restart Docker:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list >/dev/null
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
sudo systemctl restart docker

Verify the exact container path GlobalStacks needs:

docker run --rm --gpus all nvidia/cuda:12.0.1-base-ubuntu22.04 nvidia-smi

If CDI generation reports Driver/library version mismatch, repair the host NVIDIA installation first. The driver shown by nvidia-smi and the libnvidia-ml.so selected by ldconfig -p must match. Do not schedule GPU workers until the container probe passes.

What GlobalStacks reports

After the host agent reconnects, a healthy card is reported as ready with a feature such as cuda:sm_61. A visible card without a working Docker NVIDIA runtime is reported as unavailable, with the remediation trait install-nvidia-container-toolkit-and-generate-cdi.

This is intentional: host GPU visibility is not enough to make tenant inference capacity safe or usable.

Proxmox LXC hosts

For a Docker GPU worker, use a dedicated privileged LXC with nesting=1 and keyctl=1. Pass through /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-modeset, /dev/nvidia-uvm, and /dev/nvidia-uvm-tools; allow character-device majors 195, 511, and 226. Bind the matching host NVIDIA user-space library directory into the LXC and add it to ld.so.conf.d before running ldconfig.

Do not start the CT until the Proxmox host has created the UVM nodes. After a CT restart, confirm that /dev/nvidia-uvm exists inside it. Docker-in-LXC also needs the outer LXC AppArmor profile set to unconfined; this is appropriate only for a dedicated, trusted worker CT. If the workload needs micro-VMs, pass /dev/kvm through as well.