I ran into several issues when trying to create a Fedora CoreOS template on Proxmox using Packer, so here’s how I got it working.
coreos-installer to install CoreOS to a disk attached to that VM, and finally Packer will convert that VM to a template.qemu-guest-agent installed for Packer to read its IP and run commands on it for further setup.CoreOS uses a tool called Ignition to perform first-boot configuration. While you can write Ignition configurations by hand (it’s JSON after all), Fedora recommends writing your configs in Butane and using the eponymous CLI to convert them to the Ignition format. We need our Ignition/Butane config to do two things:
qemu-guest-agent on startupsudo privileges and our SSH public keyFedora CoreOS does not have a package manager, therefore qemu-guest-agent is not easily obtainable. The only way I could think of was to run it inside a docker container with --network host so that it would report the host VM’s IP address and not the container’s internal one. There is no official docker image for qemu-guest-agent; I opted to use this one. You can either use it (the Dockerfile looks trustworthy) or build your own.
Additionally, I ran into a third thing that’s needed specifically for the Packer VM: docker needs to be given a different data-root. By default, the docker daemon writes data to a subdirectory in /var. However, the qemu-guest-agent docker container proved too large for whatever storage device the ISO mounted at /var. After logging in, I noticed that /tmp had nearly a gigabyte of free space, so I changed the config to configure docker to use /tmp/docker as its data directory specifically for the Packer VM.
All that said, here’s the butane config file for the Packer VM:
variant: fcos
version: 1.4.0
storage:
files:
- path: /etc/docker/daemon.json
mode: 0600
contents:
inline: |
{
"data-root": "/tmp/docker"
}
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB3...
groups:
- wheel
- sudo
systemd:
units:
- name: qemu-guest-agent.service
enabled: true
contents: |
[Unit]
Description=Runs qemu guest agent inside docker
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=docker run -d --privileged --network host -v /dev/virtio-ports/org.qemu.guest_agent.0:/dev/virtio-ports/org.qemu.guest_agent.0 eleh/qemu-guest-agent
[Install]
WantedBy=multi-user.target
Next up, one final VM (the one Packer will convert into a template) needs its own Ignition config file. This will be usesd by Ignition on first boot of any VM you create by cloning the template. It is largely identical to the config file used for installation, minus the changes to Docker’s data root.
variant: fcos
version: 1.4.0
storage:
files:
- path: /etc/docker/daemon.json
mode: 0600
contents:
inline: |
{
"data-root": "/tmp/docker"
}
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB3...
groups:
- wheel
- sudo
systemd:
units:
- name: qemu-guest-agent.service
enabled: true
contents: |
[Unit]
Description=Runs qemu guest agent inside docker
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=docker run -d --privileged --network host -v /dev/virtio-ports/org.qemu.guest_agent.0:/dev/virtio-ports/org.qemu.guest_agent.0 eleh/qemu-guest-agent
[Install]
WantedBy=multi-user.target
You can convert a butane file to the ignition format by running
butane --pretty --strict config/input.bu > output.ign
We’re going to use a few features of Packer to achieve our goal:
additional_iso_files provided by the Proxmox builder to serve our template’s Ignition configCreate a directory with the following files:
packer-root
.
|- config
| |- installer.bu
| |- template.bu
|- proxmox-coreos.pkr.hcl
In the below config, I’ve embedded secrets inline for readability. You should use Variables to store your secrets (such as your Proxmox token/password) and keep those files private (aka off GitHub). Change the values of fields like the vm_id and iso_file to taste, and note that you will have to manually download that ISO to your Proxmox node first. If your storage pools on Proxmox have different names, you’ll have to change the local to your pool’s name in several places too.
packer {
required_plugins {
proxmox = {
version = ">= 1.1.0"
source = "github.com/hashicorp/proxmox"
}
}
}
source "proxmox" "coreos" {
// proxmox configuration
insecure_skip_tls_verify = true
node = var.proxmox.node
username = var.proxmox.username
token = var.proxmox.token
proxmox_url = var.proxmox.api_url
# Commands packer enters to boot and start the auto install
boot_wait = "2s"
boot_command = [
"<spacebar><wait><spacebar><wait><spacebar><wait><spacebar><wait><spacebar><wait>",
"<tab><wait>",
"<down><down><end>",
" ignition.config.url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/installer.ign",
"<enter>"
]
# This supplies our installer ignition file
http_directory = "config"
# This supplies our template ignition file
additional_iso_files {
cd_files = ["./config/template.ign"]
iso_storage_pool = "local"
unmount = true
}
# CoreOS does not support CloudInit
cloud_init = false
qemu_agent = true
scsi_controller = "virtio-scsi-pci"
cpu_type = "host"
cores = "2"
memory = "2048"
os = "l26"
vga {
type = "qxl"
memory = "16"
}
network_adapters {
model = "virtio"
bridge = "vmbr0"
}
disks {
disk_size = "45G"
storage_pool = "local-lvm"
storage_pool_type = "lvm"
type = "virtio"
}
iso_file = "local:iso/fedora-coreos-37.20221106.3.0-live.x86_64.iso"
unmount_iso = true
template_name = "coreos-37.20221106.3.0"
template_description = "Fedora CoreOS"
ssh_username = "core"
ssh_private_key_file = "~/.ssh/id_rsa"
ssh_timeout = "20m"
}
build {
sources = ["source.proxmox.coreos"]
provisioner "shell" {
inline = [
"sudo mkdir /tmp/iso",
"sudo mount /dev/sr1 /tmp/iso -o ro",
"sudo coreos-installer install /dev/vda --ignition-file /tmp/iso/template.ign",
# Packer's shutdown command doesn't seem to work, likely because we run qemu-guest-agent
# inside a docker container.
# This will shutdown the VM after 1 minute, which is less than the duration that Packer
# waits for its shutdown command to complete, so it works out.
"sudo shutdown -h +1"
]
}
}
Go to packer-root and generate the ignition configs:
butane --pretty --strict config/installer.bu > config/installer.ign
butane --pretty --strict config/template.bu > config/template.ign
Build the template using packer:
packer build --on-error=ask .
Assuming the build succeeds, you should see your new VM template on your Proxmox node. If it fails, check the console of the VM created by Packer for errors. You can also get detailed Packer logs by exporting the following environment variables before running packer build:
export PACKER_LOG_PATH="/tmp/packer.log"
export PACKER_LOG=10
Running packer after exporting these variables will create a detailed log file at /tmp/packer.log.
If you don’t want to use an unofficial docker image for qemu-guest-agent, you can assign static IPs to the Packer VM and to the template. Additionally, you can install qemu-guest-agent in the template (but not in the Packer VM because its root filesystem is mounted as read-only) using rpm-ostree.
To boot the Packer VM with a static IP, change the boot_command line that starts with ignition.config.url to:
" ignition.config.url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/installer.ign net.ifnames=0",
and change your installer.bu file to:
variant: fcos
version: 1.4.0
storage:
files:
- path: /etc/NetworkManager/system-connections/eth0.nmconnection
mode: 0600
contents:
inline: |
[connection]
id=eth0
type=ethernet
interface-name=eth0
[ipv4]
address1=192.168.1.200/24,192.168.1.1
dhcp-hostname=k3s-test-controller
dns=192.168.1.1;
dns-search=
may-fail=false
method=manual
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkayHzoWIWE4P1z3+qOoyfdnapU8ATcYUriXDsdGkyncEZpnz4jHqZsp0EVZhtSg668H8+aEDd4RSYHvmprXWZJQe+CUIQRIfazch8mCmlVYpRVqtjms3ya7S6WWl96+jwecEwQf0eDYojFry+S5A8+cZmIZfsQb6PkRr350OxzufH2dii96zS9aIOFz7NiVn/qB+mhyMuicrPqzx0HJjK4t8p2WFMAQsPrFqWwWlX/nDr0xFDmPUZlh4SEhznSB+ai99B0FFsjaHyhlSGBL56Sy0TL3CGXWcaW5kwQhzf9P1n/WK+83j8CLkD/xwxhB5MdhNUWIY7c02QWIeU9RPOU6Y8Qf4sgKpd6/CKROJC/SkBDFpE6MMX24/UejR1PPFP+qwg6XnX2g08gIonfI9tKBTsMAPib2D13ZSUK/QgxmOV33hfbiDPXmyXFeLuzW/GIuP9PWbe6qNYoDL2ZUk/BK3kgLWd4gXtVS3Gtu/DEiw+3kCwjP85VBW0NUx7GbM= amey@ubuntu
groups:
- wheel
- sudo
Change your template.bu file to:
variant: fcos
version: 1.4.0
kernel_arguments:
should_exist:
- net.ifnames=0
storage:
files:
- path: /etc/NetworkManager/system-connections/eth0.nmconnection
mode: 0600
contents:
inline: |
[connection]
id=eth0
type=ethernet
interface-name=eth0
[ipv4]
address1=192.168.1.201/24,192.168.1.1
dhcp-hostname=k3s-test-controller
dns=192.168.1.1;
dns-search=
may-fail=false
method=manual
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkayHzoWIWE4P1z3+qOoyfdnapU8ATcYUriXDsdGkyncEZpnz4jHqZsp0EVZhtSg668H8+aEDd4RSYHvmprXWZJQe+CUIQRIfazch8mCmlVYpRVqtjms3ya7S6WWl96+jwecEwQf0eDYojFry+S5A8+cZmIZfsQb6PkRr350OxzufH2dii96zS9aIOFz7NiVn/qB+mhyMuicrPqzx0HJjK4t8p2WFMAQsPrFqWwWlX/nDr0xFDmPUZlh4SEhznSB+ai99B0FFsjaHyhlSGBL56Sy0TL3CGXWcaW5kwQhzf9P1n/WK+83j8CLkD/xwxhB5MdhNUWIY7c02QWIeU9RPOU6Y8Qf4sgKpd6/CKROJC/SkBDFpE6MMX24/UejR1PPFP+qwg6XnX2g08gIonfI9tKBTsMAPib2D13ZSUK/QgxmOV33hfbiDPXmyXFeLuzW/GIuP9PWbe6qNYoDL2ZUk/BK3kgLWd4gXtVS3Gtu/DEiw+3kCwjP85VBW0NUx7GbM= amey@ubuntu
groups:
- wheel
- sudo
Full credit for this approach goes to the author of this blog post. Add the following to the relevant sections of your template.bu file:
storage:
files:
- path: /usr/local/bin/install-qemu-guest-agent
mode: 0755
contents:
inline: |
#!/usr/bin/env bash
set -euo pipefail
rpm-ostree install qemu-guest-agent
systemd:
units:
- name: install-qemu-guest-agent.service
enabled: true
contents: |
[Unit]
After=network-online.target
Wants=network-online.target
Before=systemd-user-sessions.service
OnFailure=emergency.target
OnFailureJobMode=replace-irreversibly
ConditionPathExists=!/var/lib/qemu-guest-agent-installed
[Service]
RemainAfterExit=yes
Type=oneshot
ExecStart=/usr/local/bin/install-qemu-guest-agent
ExecStartPost=/usr/bin/touch /var/lib/qemu-guest-agent-installed
ExecStartPost=/usr/bin/systemctl --no-block reboot
StandardOutput=kmsg+console
StandardError=kmsg+console
[Install]
WantedBy=multi-user.target