Home Lab

Building a LXC in Proxmox with Automation

Are you tired of clicking buttons in a WebGUI to create an LXC? Wish you could start having things scripted?

Phillip Tarrant

Phillip Tarrant

— 7 min read
Building a LXC in Proxmox with Automation

Automation is the path forward but knowing where to start can be daunting. This guide will help you get started with automating the process of building LXCs in Proxmox.

Assumptions and Don't blame me language

This is a very advanced topic. Executing the wrong thing CAN blow up a LXC on your system, or destroy everything. I will be doing my best to explain everything, however, please take the time to understand these commands we are scripting, double check your syntax, make backups, burn sage, whatever you need to do to ensure the best results for this operation. I'm not responsible for borked systems. There Be Dragons Ahead.

Requirements

Let's start talking about requirements to start on this journey. When you start looking into automated deployments you most likely saw something about Infrastructure as Code (IaC for short). IaC is something that started taking over the dev-ops world when server admins needed a reliable way to give devs what they needed, without devs actually building the systems. There are about 30 different ways to use frameworks and scripting to deploy things. Just to name a few off the top of my head, Jenkins, Ansible, Puppet, Chef, Terraform and the list goes on. Some people are using Drone runners to do this (I do this in some area's of my network). Today, we will lesson the need for these tools and frameworks by sticking to just shell scripts.

The Tools

You will need the following if you wish to follow along on this tutorial.

  • Proxmox fully stoodup
  • Root access to your proxmox
  • Being comfortable with the command line and scripting

Optional:

  • Git repos
  • ssh keys

The Goal

The goal of this tutorial / article is to deploy a full LXC, with a fully patched Debian 11 LXC image, with basic software deployment, and in this case, a reverse proxy called Caddy. Caddy will be a reverse proxy for my domain, and be configured with cloudflare certs out the gate. I'll include an example of my Caddyfile as well for completeness sake.

Files Layout

Before we start, understand we will be doing this entirely via the terminal, no GUI. If you want, you can create these scripts on a remote computer using the IDE of your choice (VSCode anyone?), and then upload it to your Proxmox server.

We are first going to create a new directory for our project, and one for our secrets. I will be making this a git repo and don't want keys in my scripts, or synced to a git server.

Let's make our 2 directories in a base directory under root

mkdir -p /root/deploy_scripts/caddy
mkdir -p /root/deploy_scripts/secrets

Secrets configuration

Let's start with our secrets config file. This will be a basic bash script that will serve our secrets to the other bash scripts.

#!/bin/bash
export ROOT_PASS=SUPER_SECURE_P4$$WORD!

That's it. Easy right? This is creating an environment variable called "ROOT_PASS" with the defined value of a password for use later. Please for the love that is holy, don't use this password, substitute your own. My secrets file has a ton more secrets in it, such as my tailscale auth key, database credentials, and other things for various LXC's I use this for.

Understanding the flow

We are creating a few files here, some for config, some for execution. Here is a very broad overview of the process.

  • build_vm.sh will build a LXC on the proxmox host, and copy over some files.

    • Stops the old LXC
    • Destroys the old LXC
    • Creates a new LXC
    • Edits the config files of the LXC on the proxmox to allow tunneling
    • Copys over files into the LXC to be ran inside the container

  • software.sh will be executed inside the LXC to do the software config.

    • Updates all packages
    • Adds the caddy repo
    • Installs Caddy
    • Installs UFW rules for the reverse proxy

Build_VM script

This first script will be executed on the Proxmox server itself to create the LXC.

Loading and Defining more variables

Let's start a new script and call it build_vm.sh. The first few lines of our script are simply setting environment variables. Note these work for my system, your system/network might be different. Go ahead and start your script with these lines.

# loads our secrets from file we made earlier
source ../secrets/secrets.sh

#our container ID, number of cores, IP setup, Hard drive size (in GB), LXC Name and network bridge to use.

CT_ID=201
CORES=2
MEM=128
IP=192.168.1.175
GW=192.168.1.1
HD=2
NAME=caddy
BRIDGE=vmbr0

Removing old containers

This line below uses pct (a Proxmox helper script) to kill any container that might be running on 201, and then destroy it

echo "Stopping old container"
pct stop $CT_ID
echo "Waiting 2 sec for container to die"
sleep 2

echo "Deleting old container"
pct destroy $CT_ID

Building a new container

Add the following next to your script, this is actually building the new Debian 11 LXC.

echo "Building new container"
pct create $CT_ID /var/lib/vz/template/cache/debian-11-standard_11.3-1_amd64.tar.zst \
--cores $CORES \
--hostname $NAME \
--memory $MEM \
--net0 name=eth0,bridge=$BRIDGE,firewall=1,gw=$GW,ip=$IP/24,type=veth \
--storage local-lvm \
--rootfs local-lvm:$HD \
--unprivileged 1 \
--features keyctl=1,nesting=1,fuse=1 \
--ostype debian \
--password=$ROOT_PASS \
--start 1 \
--onboot 1 \
--ssh-public-keys /root/.ssh/authorized_keys

Explaination of above

Most of the above is simply defining things using variables we already made earlier, however, notice that $ROOT_PASS? That's not in this file...because it's already been loaded from the secrets.sh file earlier. This way the password is never in this file, and since we keep our secrets file out of our repo, it's safe from anyone without root access to the server.

A Special note about feature flags -
Notice the line --features keyctl=1,nesting=1,fuse=1 \? This is because I need these settings for my LXC for other things going on. I left them in here so you can see how to enable them via the CLI.

Allowing Tunneling

If you need tunneling support in the LXC (tailscale, zerotier, wireguard etc) you will need to add this to the end of the script we have so far -

#Allow Tunneling
echo "lxc.cgroup2.devices.allow: c 10:200 rwm" >> /etc/pve/lxc/$CT_ID.conf
echo "lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file" >> /etc/pve/lxc/$CT_ID.conf

Rebooting the LXC for the tunneling to take affect

Add the following to your script so far.

#Reboot to allow tunneling
echo "Rebooting $CT_ID"
pct reboot $CT_ID
sleep 3

Copying over some files into our LXC from the Proxmox host

Again, using the PCT script, we are going to push some files into our LXC. We haven't made some of these files yet, but we will. Let's add the following to our script. I've added comments to the script to help explain

#copy over software/secret scripts INTO the LXC via PCT push command
pct push $CT_ID /root/deploy_scripts/caddy/software.sh /root/software.sh
pct push $CT_ID /root/deploy_scripts/secrets/secrets.sh /root/secrets.sh

#execute the chmod command inside the LXC to make .sh files exc
pct exec $CT_ID chmod +x /root/software.sh
pct exec $CT_ID chmod +x /root/secrets.sh

#execute the software.sh file inside the LXC run install processes and such.
pct exec $CT_ID /root/software.sh


# download your origin cert from cloudflare for your domain and save them to /root/deploy_scripts/caddy/certs/

#copy over SSL certs via PCT push
pct push $CT_ID /root/deploy_scripts/caddy/certs/cert.pem /etc/ssl/cert.pem 
pct push $CT_ID /root/deploy_scripts/caddy/certs/key.pem /etc/ssl/key.pem

Caddyfile

Here is my caddyfile, located at /root/deploy_scripts/caddy/Caddyfile

*.chickenbabiesrus.com {

	#Cloudflare origin certs
	tls /etc/ssl/cert.pem /etc/ssl/key.pem

	@shop host shop.chickenbabiesrus.com
	handle @shop {
		reverse_proxy 192.168.1.176:80
	}
	
	handle {
		respond "Yeah I got nothing boss, unable to find a site for that subdomain!"
	}
}

pushing the Caddyfile into the container and one final reboot

#finally copy over the caddyfile
pct push $CT_ID Caddyfile /etc/caddy/Caddyfile

#reboot the LXC one last time, so services start up and we can make sure things work.
pct reboot $CT_ID

WHOOT - First part is done

Now that the initial script for the creation of the LXC is complete, let's create our software.sh file that will be executed inside the LXC container.

Software updates, install, and other configs inside the container

Now we get to the fun part. Let's make another script at /root/deploy_scripts/caddy/software.sh. I'll comment below to explain it as we read through it.

#!/bin/bash
#load our env variables and delete the file for security
source /root/secrets.sh
rm -f /root/secrets.sh

### Update, upgrade, and install basic packages i use all the time.
apt update
apt upgrade -y
apt install sudo curl wget psmisc fail2ban ca-certificates gnupg lsb-release ufw -y

# Caddy install instructions
apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list

#yet another update to apt
apt update

# installing caddy
apt install caddy -y

# UFW rules for a reverse proxy, allowing tailcale and local network
ufw allow from 100.64.0.0/10 proto tcp to any
ufw allow from 192.168.1.0/24 proto tcp to any

# allow everything else to 80/443
ufw allow 80/tcp
ufw allow 443/tcp

#start UFW
ufw enable

Finishing up

Now we have several files ready to copy over and run on the proxmox server.
Copy over your files, and do the following:

cd /root/deploy_scripts/caddy
./build_vm.sh

This will build a new container (ID 201), install all the required software and have a reverse proxy setup.

Hit a snag? Something broke?

Hit us up on Discord.

Read more