Proxmox
Proxmox home cluster
Proxmox Backup Server
ZFS stuck?
Terminate process with SSH tunnel if terminating zfs send process doesn’t work
Change VM ID range, next free ID for VM.
Datacenter → Options → Next Free VMID Range
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_next_id_range
Helpers and scripts
https://github.com/tteck/Proxmox
Host key verification failed
If migration fails, check if ssh_known_hosts is ok, also restart pveproxy service.
It also can cause ZFS replication errors.
https://forum.proxmox.com/threads/host-key-verification-failed-when-migrate.41666/
FS freeze thaw
# https://pve.proxmox.com/wiki/Qemu-guest-agent
qm guest cmd VMID fsfreeze-statusRAM balloon
if RAM usage on the host is below 80%, will dynamically add memory to the guest up to the maximum memory specified.
https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_memory
https://forum.proxmox.com/threads/ballooning-issues.69048/#post-309631
GPU passthrough
https://pve.proxmox.com/wiki/PCI_Passthrough
https://bobcares.com/blog/proxmox-gpu-passthrough/
https://forum.proxmox.com/threads/audio-is-same-iommu-group-as-ethernet-how-pass-through.128404/
https://forum.proxmox.com/threads/proxmox-8-1-3-gpu-passthrough-soooo-close.139341/
# nano /etc/default/grub
from
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
to
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
or
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"
# Intel VT-d in BIOS should be enabled
# IOMMU should be enabled looking at dmesg
dmesg | grep -i 'IOMMU enabled'
# check current kernel launch cmd if iommu enabled
cat /proc/cmdline
# add intel iommu to kernel cmd without modfification of files
echo 'GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT intel_iommu=on"' > /etc/default/grub.d/iommu.cfg
update-grub
echo -e 'vfio\nvfio_iommu_type1\nvfio_pci\nvfio_virqfd' > /etc/modules-load.d/vfio.conf
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "blacklist radeon" > /etc/modprobe.d/blacklist_gpu.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist_gpu.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist_gpu.conf
# check vendor id codes for vga and audio
lspci | grep -i nvidia
lspci -n -s 01:00
# 10de:1cb3
# 10de:0fb9
echo "options vfio-pci ids=10de:1cb3,10de:0fb9 disable_vga=1" > /etc/modprobe.d/vfio.conf
update-initramfs -u
reboot
# TODO check if it's necessary
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.confroot@mango:~# lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
root@mango:~# lspci -n -s 01:00
01:00.0 0300: 10de:1cb3 (rev a1)
01:00.1 0403: 10de:0fb9 (rev a1)
Debian 12 bookworm VM with nvidia GPU for Docker
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt-get update
apt-get install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker
# test if it works
docker run --rm --gpus all nvidia/cuda:11.5.2-base-ubuntu20.04 nvidia-smi
# /etc/apt/sources.list.d/debian.sources
# Components: main contrib non-free non-free-firmware
apt update
# https://old.reddit.com/r/debian/comments/17vdj3c/install_nvidia_headless_driver/
apt install --no-install-recommends firmware-misc-nonfree nvidia-smi nvidia-driver nvidia-kernel-dkms linux-headers-amd64 libcuda1 nvidia-persistenced
# probably we need nvidia container toolkit too
# apt install nvidia-cuda-dev nvidia-cuda-toolkitUbuntu 22.04 VM with Nvidia GPU for Docker
# install clear Ubuntu Server 22.04, login as root and upgrade it to latest version
apt-get update
apt-get dist-upgrade
# blacklist default GPU drivers after checking
echo "blacklist nouveau" > /etc/modprobe.d/blacklist-nouveau.conf
echo "options nouveau modeset=0" >> /etc/modprobe.d/blacklist-nouveau.conf
# search for latest proprietary Nvidia GPU driver in Ubuntu repo and install it
apt search nvidia-driver-5 | grep 'nvidia-driver-.*-server'
apt-get install -y nvidia-driver-535-server
update-initramfs -u
reboot
# check GPU status after rebooting
nvidia-smi
# install standard Docker
curl https://get.docker.com | sh && sudo systemctl --now enable docker
# install Docker with Nvidia GPU support from offical repo
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu22.04/nvidia-docker.list > /etc/apt/sources.list.d/nvidia-docker.list
apt update
apt -y install nvidia-container-toolkit
systemctl restart docker
# test if GPU is visible and working from inside of container
docker run --rm --gpus all nvidia/cuda:11.5.2-base-ubuntu20.04 nvidia-smi
# deploy ready to use containerized SD with easy to use Web GUI from
git clone https://github.com/AbdBarho/stable-diffusion-webui-docker
cd stable-diffusion-webui-docker/
docker compose --profile download up --build
docker compose --profile auto up -d --build
#
# visit http://IP:7860 of your Docker host for WebGUI
#
# Extras
# limit power to prevent faster wear of card
# setting is not persistent, add this to systemd service after boot if needed
nvidia-smi -pl 180
nvidia-smi -q -d POWERVM io-error
If you are getting yellow triangle and “io-error” for VM in Proxmox, probably it’s just disk full.
In my case I made some snapshots for VM before upgrade, didn’t notice that ZFS replication would fill up disk of my smallest node.
Source: https://forum.proxmox.com/threads/crash-vm-with-io-error.118692/
Misc
Migrate to remote cluster
https://forum.proxmox.com/threads/sync-zfs-to-remote-zfs-datastore.120424/post-561444