The seven things that broke deploying to a free VM

This is the companion to the step-by-step deployment guide. That post is the happy path. This one is everything that went wrong on the way, in the order it surfaced.

Seven distinct failures got between "CI is green" and "the domain loads." Not one was in the application code. They were spread across GitHub secrets, Linux groups, token types, CPU architectures, DNS, and two different firewalls that look identical from outside.

I'm writing the symptoms rather than just the fixes, because the fixes are all one search away once you know which layer is lying to you. Identifying the layer is the hard part.

One pattern worth internalising before the list: each failure hid the next one. Fixing the firewall revealed a second firewall. Nothing was ever the last problem, and every time I thought I was done, I wasn't.

1. Secrets that were never there

Symptom. The deploy job failed in six seconds. The log printed its environment:

DEPLOY_HOST:
DEPLOY_USER:
DEPLOY_KEY:

Not wrong values — empty ones. ssh-keyscan then ran against an empty host and died instantly.

The diagnostic. GitHub Actions masks any secret that resolved, printing three asterisks instead of the value. So a masked value means the secret exists and might be wrong; a blank value means the reference resolved to nothing at all. That distinction stops you debugging a value that doesn't exist.

Cause. They were never saved as repository secrets. Worth knowing that GitHub has two kinds: Repository secrets (Settings → Secrets and variables → Actions) and Environment secrets (a separate Environments page). A job without an environment: key cannot see environment secrets — and they fail exactly like this, silently, as empty strings, with no warning that you put them somewhere the job can't reach.

2. Docker installed, but not usable

Symptom.

permission denied while trying to connect to the docker API
at unix:///var/run/docker.sock

Cause. The deploy user wasn't in the docker group.

Why it's easy to miss. Every guide puts usermod -aG docker $USER in its install Docker section. If Docker came preinstalled on your cloud image — as it often does — you skip that whole section, and with it the one line you needed. Docker works fine under sudo, so it looks installed and healthy right up until an automated, non-sudo deploy tries to use it.

sudo usermod -aG docker ubuntu
exit    # group membership only applies to a NEW session

Test with docker ps in a fresh session. The shell you ran usermod in will never see the change.

3. A token type that cannot do the job

Symptom. Login succeeds. Pull is denied.

$ docker login ghcr.io -u username
Login Succeeded

$ docker pull ghcr.io/username/app:main
Error response from daemon: error from registry: denied

I regenerated the token twice, hunting for a "Packages" permission I'd missed.

Cause. There is no such permission. Fine-grained personal access tokens cannot access GitHub Packages at all. It isn't a scope you forgot; it's a capability that token type doesn't have. GitHub lists it plainly under fine-grained token limitations: "Using fine-grained personal access token to access Packages."

Fix. A classic token with read:packages. Or make the package public and skip VM-side auth entirely — for an image with no baked-in secrets, that's simpler and one less credential to rotate.

The trap inside the trap. Three different things get called "the GitHub token," and I conflated two of them for a full round-trip:

I updated the third and expected docker pull to behave differently. It doesn't. Docker's registry credentials are written only by docker login — nothing in Compose, .env, or your app config touches them.

4. The wrong CPU architecture

Symptom.

no matching manifest for linux/arm64/v8 in the manifest list entries

Cause. Oracle's Always Free tier is Ampere A1 — arm64. GitHub's ubuntu-latest runners are x86. docker/build-push-action defaults to building for the runner's own architecture, so CI produced a perfectly valid amd64 image that the VM had no way to run.

This is the least-advertised property of Oracle's free tier and it invalidates a lot of copy-pasted CI config.

Two fixes, and the better one isn't the popular one. Most guides reach for QEMU emulation:

- uses: docker/setup-qemu-action@v3
- uses: docker/build-push-action@v5
  with:
    platforms: linux/amd64,linux/arm64

That works, but emulated arm64 builds commonly run 3–5× slower. If the only target is an arm VM, build natively on an arm runner instead:

jobs:
  build-and-push:
    runs-on: ubuntu-24.04-arm
    ...
      - uses: docker/build-push-action@v5
        with:
          platforms: linux/arm64

No emulation, one architecture, dramatically faster. Multi-arch is only worth the cost if something else actually pulls the amd64 variant.

5. DNS that answered five different ways

Symptom. The site loaded for some people and showed the registrar's "coming soon" page for others. dig returned four IPs that weren't mine, plus mine.

Cause. The registrar had parked the domain on four A records under a preset (Squarespace calls it "Squarespace Defaults"). I added my own @ record assuming it would replace them. It didn't — it added a fifth. DNS is then free to hand different resolvers different answers.

Fix. Delete the registrar's default preset in the same visit you add your own record.

And then wait, because you can't undo caching. Those defaults carried a 4-hour TTL. Public resolvers cache independently and expire on their own schedule, so deleting the records late doesn't reset that clock:

dig +short imrpm.com @8.8.8.8    # re-run until only your IP comes back

Set a low TTL on your own record now, before you ever need to repoint it.

5b. The browser lies after DNS is already fixed

Worth its own heading, because it looks like a server problem and isn't.

Symptom. dig returned exactly the right IP. curl https://imrpm.com returned my real site with correct headers. The browser still showed the registrar's parked page — on desktop and phone. Incognito worked perfectly.

Cause. Chrome keeps its own per-profile DNS cache, independent of your OS resolver and of whatever dig queries. Incognito bypasses it, which is exactly why incognito working is the giveaway.

Fix. chrome://net-internals/#dns → Clear host cache.

The rule: when the terminal and the browser disagree about whether your site is up, believe the terminal. I spent real time convinced the deployment had broken when it had been correct for an hour.

6 and 7. Two firewalls, and the one that isn't ufw

The last two, and the pair that took longest, because fixing the first made no visible difference.

Symptom. Nginx was running and correct:

# on the VM
curl -I http://localhost      # 200 OK
 
# from anywhere else
curl -I http://imrpm.com      # nothing

Cause 6: the cloud-level VCN Security List. Oracle firewalls your instance before traffic reaches the machine. The default allows port 22 and nothing else. Add ingress for TCP 80 and 443 from 0.0.0.0/0, and leave the rules stateful — a stateless rule needs a matching egress rule for return traffic, which is its own silent failure.

Cause 7: iptables on the VM itself. After fixing the Security List it still didn't work, because Oracle's images also ship a default-reject INPUT chain:

1  ACCEPT  state RELATED,ESTABLISHED
2  ACCEPT  icmp
3  ACCEPT  lo
4  ACCEPT  tcp dpt:22
5  REJECT  reject-with icmp-host-prohibited

The trap. sudo ufw status returns ufw: command not found on Oracle's minimized images. That reads like "there's no host firewall here." It means the opposite: there is one, it just isn't managed by ufw. Check the real thing:

sudo iptables -L INPUT -n -v --line-numbers
sudo iptables -I INPUT 5 -m state --state NEW -p tcp --dport 80 -j ACCEPT
sudo iptables -I INPUT 5 -m state --state NEW -p tcp --dport 443 -j ACCEPT
sudo netfilter-persistent save

Insert before the REJECT line — iptables stops at the first match. And persist, or the rules disappear on the next reboot and you get to debug this twice.

The one diagnostic worth memorising

These two firewalls produce the same user-visible failure — "the site doesn't load" — but they're distinguishable by latency:

A DROP leaves your client waiting for a reply that never comes. A REJECT sends an ICMP response immediately. When I finally noticed the failure had gone from a 28-second timeout to a 17-millisecond refusal, that single change told me I'd fixed one firewall and found another.

What I'd tell myself before starting

Verify each layer in isolation before adding the next. Most of my lost time went to debugging composite failures that would have been obvious alone. Concretely: curl localhost on the VM before touching DNS; curl the raw IP from your laptop before touching Nginx; make DNS resolve cleanly before running Certbot.

Prefer diagnostics that distinguish, not just detect. "The site is down" is worthless. "It fails in 17ms rather than timing out" identifies the layer. Latency, blank-versus-masked, works-in-incognito — each of those pinned a cause faster than any amount of config re-reading.

When two sources disagree, find out which one is closer to the truth. Terminal over browser. Server logs over user reports. dig @8.8.8.8 over whatever your ISP's resolver claims.

Assume every "done" is provisional. Seven times I thought I'd finished. The pipeline has many independent layers, and green at one is not green overall — right up until the moment you load the real domain in a fresh browser and it works.

The full walkthrough, with all of this already accounted for, is in the step-by-step guide.