podman container hardening

[0xlua]

• Don't run containers as root
• Don't run containers privileged
• use pinned versions, see Automatic Podman Updates #138

[0xlua]

[source]

---

After my last post blew up, I audited my Docker security. It was worse than I thought.

A week ago I posted here about dockerizing my self-hosted stack on a single VPS. A lot of you rightfully called me out on some bad advice, especially the "put everything on one Docker network" part. I owned that in the comments.

But it kept nagging at me. If the networking was wrong, what else was I getting wrong? So I went through all 19 containers one by one and yeah, it was bad.

Capabilities First thing I checked. I ran docker inspect and every single container had the full default Linux capability set. NET_RAW, SYS_CHROOT, MKNOD, the works. None of my services needed any of that.

I added cap_drop: ALL to everything, restarted one at a time. Most came back fine with zero capabilities. PostgreSQL was the exception, its entrypoint needs to chown data directories so it needed a handful back (CHOWN, SETUID, SETGID, a couple others). Traefik needed NET_BIND_SERVICE for 80/443. That was it. Everything else ran with nothing.

Honestly the whole thing took maybe an hour. Add it, restart, read the error if it crashes, add back the minimum.

Resource limits None of my containers had memory limits. 19 containers on a 4GB VPS and any one of them could eat all the RAM and swap if it felt like it.

Set explicit limits on everything. Disabled swap per container (memswap_limit = mem_limit) so if a service hits its ceiling it gets OOM killed cleanly instead of taking the whole box down with it. Added PID limits too because I don't want to find out what a fork bomb does to a shared host.

The CPU I just tiered with cpu_shares. Reverse proxy and databases get highest priority. App services get medium. Background workers get lowest. My headless browser container got a hard CPU cap on top of that because it absolutely will eat an entire core if you let it.

Health checks Had health checks on most containers already but they were all basically "is the process alive." Which tells you nothing. A web server can have a running process and be returning 500s on every request.

Replaced them with real HTTP probes. The annoying part: each runtime needs its own approach. Node containers don't have curl, so I used Node's http module inline. Python slim doesn't have curl either (spent an embarrassing amount of time debugging that one), so urllib. Postgres has pg_isready which just works.

Not glamorous work but now when docker says a container is healthy, it actually means something.

Network segmentation Ok this was the big one. All 19 containers on one flat network. Databases reachable from web-facing services. Mail server can talk to the URL shortener. Nothing needed to talk to everything but everything could.

I basically ripped it out. Each database now sits on its own network marked internal: true so it has zero internet access. Only the specific app that uses it can reach it. Reverse proxy gets its own network. Inter-service communication goes through a separate mesh.

    # before: everything on one network
    networks:
      default:
        name: shared_network

    # after: database isolated, no internet
    networks:
      default:
        name: myapp_db
        internal: true
      web_ingress:
        external: true

My postgres containers literally cannot see the internet anymore. Can't see Traefik. Can only talk to their one app.

The shared database I didn't even realize this was a problem until I started mapping out the networks. Three separate services, all connecting to the same PostgreSQL container, all using the same superuser account. A URL shortener, an API gateway, and a web app. They have nothing in common except I set them all up pointing at the same database and never thought about it again.

If any one of them leaked connections or ran a bad query, it would exhaust the pool for all four. Classic noisy neighbor.

I can't afford separate postgres containers on my VPS so I did logical separation. Dedicated database + role per service, connection limits per role, and then revoked CONNECT from PUBLIC on every database. Now psql -U serviceA -d serviceB_db gets "permission denied." Each service is walled off.

Migration was mostly fine. pg_dump per table, restore, reassign ownership. One gotcha though: per-table dumps don't include trigger functions. Had a full-text search trigger that just silently didn't make it over. Only noticed because searches started coming back empty. Had to recreate it manually.

Secrets This was the one that made me cringe. My Cloudflare key? The Global API Key. Full account access. Plaintext env var. Visible to anyone who runs docker inspect.

Database passwords? Inline in DATABASE_URL. Also visible in docker inspect.

Replaced the CF key with a scoped token (DNS edit only, single zone). Moved DB passwords to Docker secrets so they're mounted as files, not env vars. Also pinned every image to SHA256 digests while I was at it. No more :latest. Tradeoff is manual updates but honestly I'd rather decide when to update.

Traefik TLS 1.2 minimum. Restricted ciphers. Catch-all that returns nothing for unknown hostnames (stops bots from enumerating subdomains). Blocked .env, .git, wp-admin, phpmyadmin at high priority so they never reach any backend. Rate limiting on all public routers. Moved Traefik's own ping endpoint to a private port.

Still on my list Not going to pretend I'm done. Haven't moved all containers to non-root users. Postgres especially needs host directory ownership sorted first and I haven't gotten around to it. read_only filesystems are only on some containers because the rest need tmpfs paths I haven't mapped yet. And tbh my memory limits are educated guesses from docker stats, not real profiling.

Was it worth it? None of this had caused an actual incident. Everything was "working." But now if something does go wrong, the blast radius is one container instead of the whole box. A compromised web service can't pivot to another service's database. A memory leak gets OOM killed instead of swapping the host to death.

Biggest time sink was the network segmentation and database migration. The per-container stuff was pretty quick once I had the pattern.

Still figuring things out. If anyone's actually gotten postgres running as non-root in Docker or has a good approach to read_only with complex entrypoints, would genuinely like to know how you did it.

[0xlua]

see also: https://mroczek.dev/articles/hardened-arr-stack/

[0xlua]

If Docker - Use Reverse Proxy for HTTPs, expose only Dockers which are made for being exposed (Like Seer etc.) where it is known that part of the community is trying to maintain a proper and secure code base for that part.

For Docker: Container has no privilege, they dont run as root PUID/GUID “0:0” case, and you have own Docker Network, nor the default bridge.

For the Host: enable firewall and block all access from countries you are not from. Use 2FA.

That is the baseline.

More I don’t have myself..

source: https://reddit.lua.one/r/selfhosted/comments/1sqi04i/beyond_the_basics_what_are_your_nonnegotiable/