Introduction
Previously, we looked at building a custom caddy with an L4 addon, which will be used later. We will only use Caddy’s auto HTTPS cert generation to add the proxmox management interface as a load balance.
Cloudflare Changes
In Cloudflare, we will add a wildcard A and an AAAA record to our Edge VM (BuyVM)
Caddy.json file
We will upload a new config file on the Edge VM to test it out. It will check any incoming connection to see if it’s on proxmox.ata.al and if matching will load-balance between the four nodes. There is a section to ignore the SSL cert, as Proxmox uses a self-signed one.
{
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"proxmox.ata.al"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "reverse_proxy",
"health_checks": {
"active": {
"expect_status": 200,
"interval": 10000000000,
"timeout": 2000000000,
"uri": "/"
}
},
"load_balancing": {
"selection_policy": {
"policy": "ip_hash"
},
"try_duration": 1000000000,
"try_interval": 250000000
},
"transport": {
"protocol": "http",
"tls": {
"insecure_skip_verify": true
}
},
"upstreams": [
{
"dial": "10.0.200.101:8006"
},
{
"dial": "10.0.200.102:8006"
},
{
"dial": "10.0.200.103:8006"
},
{
"dial": "10.0.200.104:8006"
}
]
}
]
}
]
}
],
"terminal": true
}
]
}
}
}
}
}On the VM, after SSH’ing into it, stop the running caddy background instance with Caddy stop, then upload the new caddy.json file. After that, you can launch caddy with caddy run --config caddy.json. A lot will be going on in the terminal while Caddy requests new Certs for the subdomain.
root@localhost:~/caddy# caddy run --config caddy.json
2023/11/04 16:13:54.196 INFO using provided configuration {"config_file": "caddy.json", "config_adapter": ""}
2023/11/04 16:13:54.199 INFO admin admin endpoint started {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/11/04 16:13:54.200 INFO http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2023/11/04 16:13:54.200 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2023/11/04 16:13:54.202 INFO http.log server running {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/11/04 16:13:54.202 INFO http enabling HTTP/3 listener {"addr": ":443"}
2023/11/04 16:13:54.203 INFO http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/11/04 16:13:54.203 INFO http enabling automatic TLS certificate management {"domains": ["proxmox.ata.al"]}
2023/11/04 16:13:54.204 INFO autosaved config (load with --resume flag) {"file": "/root/.config/caddy/autosave.json"}
2023/11/04 16:13:54.204 INFO serving initial configuration
2023/11/04 16:13:54.204 INFO tls.obtain acquiring lock {"identifier": "proxmox.ata.al"}
2023/11/04 16:13:54.208 INFO tls.obtain lock acquired {"identifier": "proxmox.ata.al"}
2023/11/04 16:13:54.209 INFO tls.obtain obtaining certificate {"identifier": "proxmox.ata.al"}
2023/11/04 16:13:54.210 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc00036bf80"}
2023/11/04 16:13:54.213 INFO tls cleaning storage unit {"description": "FileStorage:/root/.local/share/caddy"}
2023/11/04 16:13:54.213 INFO tls finished cleaning storage unitsAfter it stops generating logs, you can visit the new subdomain in a new web browser, and it will choose one of the hosts and reverse proxy to it. In my case, it choose pve02.
To write the changes to the background service, replace the config file in /etc/caddy/caddy.json, then run sudo systemctl restart caddy. You can check the status by running systemctl status caddy.service
root@localhost:~/caddy# systemctl status caddy.service
● caddy.service - Caddy
Loaded: loaded (/etc/systemd/system/caddy.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/caddy.service.d
└─override.conf
Active: active (running) since Sat 2023-11-04 16:47:04 UTC; 21s ago
Docs: https://caddyserver.com/docs/
Main PID: 37392 (caddy)
Tasks: 7 (limit: 1089)
Memory: 12.1M
CPU: 217ms
CGroup: /system.slice/caddy.service
└─37392 /usr/bin/caddy run --environ --config /etc/caddy/caddy.json
Nov 04 16:47:04 localhost caddy[37392]: {"level":"info","ts":1699116424.6689034,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["proxmox.ata.al"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Nov 04 16:47:04 localhost caddy[37392]: {"level":"info","ts":1699116424.923021,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"proxmox.ata.al","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/di>
Nov 04 16:47:05 localhost caddy[37392]: {"level":"info","ts":1699116425.3406816,"logger":"tls","msg":"served key authentication certificate","server_name":"proxmox.ata.al","challenge":"tls-alpn-01","remote":"[2600:1f14:804:fd02:7fd:aac5:50aa:ba32]>
Nov 04 16:47:05 localhost caddy[37392]: {"level":"info","ts":1699116425.3983707,"logger":"tls","msg":"served key authentication certificate","server_name":"proxmox.ata.al","challenge":"tls-alpn-01","remote":"[2600:3000:1511:200::83]:41751","distri>
Nov 04 16:47:06 localhost caddy[37392]: {"level":"info","ts":1699116426.6794689,"logger":"tls","msg":"served key authentication certificate","server_name":"proxmox.ata.al","challenge":"tls-alpn-01","remote":"[2600:1f16:269:da02:e154:9a15:b37a:1139>
Nov 04 16:47:08 localhost caddy[37392]: {"level":"info","ts":1699116428.820731,"logger":"http.acme_client","msg":"authorization finalized","identifier":"proxmox.ata.al","authz_status":"valid"}
Nov 04 16:47:08 localhost caddy[37392]: {"level":"info","ts":1699116428.8213177,"logger":"http.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/1395765976/219865018386"}
Nov 04 16:47:09 localhost caddy[37392]: {"level":"info","ts":1699116429.621099,"logger":"http.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/04fba25e1>
Nov 04 16:47:09 localhost caddy[37392]: {"level":"info","ts":1699116429.6218455,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"proxmox.ata.al"}
Nov 04 16:47:09 localhost caddy[37392]: {"level":"info","ts":1699116429.6220608,"logger":"tls.obtain","msg":"releasing lock","identifier":"proxmox.ata.al"}Another way to test it out is to reboot the chosen server, in my case, pve02. After it goes down and you refresh the page, it will auto switch to another node. In my case pve03
