Self cross-posting from: https://lemmy.zip/post/70909658
Intention to have slightly better visibility from the self-hosted crowd and I’m interested in more general feedback on this too.
Hey everyone! I’m trying to find a solution to a really confusing problem…
I have the following simple nginx docker compose configuration on my Fedora home server that I can run without issue on my uid
1000user, lets call this user “userA”.services: nginx: container_name: nginx-alt image: docker.io/library/nginx restart: unless-stopped ports: - 8181:80This exposes internal port 80 as 8181 and can be accessed in a lan in the expected matter.
However, for security reasons, I want to actually host this service eventually on a completely different user with less permissions. Let’s call this user “userB” who has a very limited scope of the file system. This is to prevent potential escaping of the rootless container causing major file system havoc (i.e. reduce the scope of the user to a very limited network of containers.)
The problem is really simple: For some reason, when userB runs this service (uid
1001), the nginx service suddenly complains about privileges. As a result, I get a “Forbidden 403” error when hosting. Turning off selinux has no affect (sosetenforce 0does nothing, meaning I can rule out secure linux interruption.)The errors look like the following:
nginx-alt | 2026/09/04 20:03:34 [error] 25#25: *1 "/usr/share/nginx/html/index.html" is forbidden (13: Permission denied), client: xx.xx.x.x, server: localhost, request: "GET / HTTP/1.1", host: "xxx.xxx.xxx.xxx:8181" nginx-alt | 10.89.0.2 - - [04/Sep/2026:20:03:34 +0000] "GET / HTTP/1.1" 403 153 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:155.0) Gecko/20100101 Firefox/155.0" "-"For what it’s worth, both users should be relatively vanilla and all ports are appropriately exported. There shouldn’t be anything, for example, that is making userA run as “privileged” over the other users and podman should be running rootless in both containers.
I did see a note on the nginx image about running in rootless that I might try, but it doesn’t solve my bigger issue here which is the lack of consistency between the two users. Additionally,
userns_mode: keep-idsonly caused the container to fail to boot for other reason entirely.There must be something fundamentally wrong with my configuration of my system. Has anyone had any experience running two podman containers on two different users simultaneously that can provide feedback?
Obviously, I’m not trying to run just an nginx server, but I found this to be the easiest configuration to reproduce.
You might want to switch to rootless quadlets with subuids
https://www.golinuxcloud.com/rootless-podman-quadlet/
You’ll get unified logging through systemctl and journalctl, you’ll also get segmented user space and if you need persistent storage that you can gate at subuid.
Complete shot in the dark, but I would start by comparing
podman system infobetween the two user accounts, with the most importance on theidMappings. Podman is usually configured to use rootless by default, and this can result in wildly different behavior based on whether/etc/subuidand/etc/subgidare configured to give your user IDs or not. See tutorial if your user 1001 is missing.Modern distro installers tend to give the user created during setup 65536 IDs, but
useraddusually doesn’t do that by default. This is really distro-specific stuff though and I don’t know how modern Fedora handles it, so I might be grasping at nothing here.I think I have uids set up, but I’m not positive they’re set up correctly.
Here’s an example of
/etc/subuidfor example, with it a bit anonymised for my specific user setup:core:524288:65536 syncthing:589824:65536 userA:655360:65536 userB:720896:65536 userC:786432:65536Syncthing in this case isn’t running any podman containers, all sharing is done bare-metal. userA is working, but userB and userC do look like they’re configured “correctly” at first glance.
podman system infoseems to be outputting the results aboveidMappings: gidmap: - container_id: 0 host_id: 1101 size: 1 - container_id: 1 host_id: 786432 size: 65536 uidmap: - container_id: 0 host_id: 1101 size: 1 - container_id: 1 host_id: 786432 size: 65536Note: The user in question here is
uid 1101, another new user I created for testing purposes.That looks correct to me if uid 1101 == userC. Sorry, don’t really know what it might be then.
You use podman-compose with the Podman backend?
And there are no volumes mounted, just to be sure?
No volumes mounted and yes, it’s using the podman backend. Just double checked it wasn’t using the docker one.
The service starts as it should, right? The errors appear only on request, right? Any other sections in the compose logs that seem relevant?
This is due to a security design deicison of Podman. Each user’s network(s) is only available to that user. This is great for most services, but can cause issues for some services - especially reverse proxies. Unfortunately, I’m not aware of an ideal solution. The only solution I’ve seen is moving the reverse proxy to another host and exposing the services’ ports on the localhost. I hope someone can provide a better solution!
No expert but don’t the logs look like the requests are coming through?
I didn’t see the logs when I originally posted. However, I’m not sure what the logs indicate. It could be that nginx successfully received the request and received an authorization error from Podman’s networking stack then returned a 503 error to the client (or logged the 503 response that was returned to nginx).
OK now this would make some sense but would definitely be a bit of a show stopper for me. I’ll give it a try though.
Edit: Unfortunately this didn’t solve the issue. I thought it did for a minute, but it was the browser cache I think auto-filling the correct result.
The solution to this can be multiple reverse proxies. Each user runs it’s own reverse proxy (if you actually need one, otherwise you let the container bind directly to a port on the host system). Then you run one main reverse proxy on port 80/443 that proxies to those other ports based on hostname.
The upside of this construction is that containers running as different users can’t directly access each others internal networks, which is much better for security, as those networks often contain barely protected services, which is why you’re using a reverse proxy in the first place.
Edit: there are different options with custom networks and such, but they’re even more complicated so I wouldn’t advise them.
I think this is what OP was originally trying, but this approach breaks when each service’s Podman service runs on its own localhost user.
You can use the per-user reverse proxies as a bridge between the host network and the user-specific internal network. So for a user coming from outside the path looks like this:
User -> main reverse proxy on main IP and port 80/443 -> user-specific proxy listening on main IP with port 8080 and proxying to user-specific internal network -> destination container listening on user-specific internal network.
And for a container running as a different user the path will be the same, but the user-specific reverse proxy will be listening on port 8081 and higher for example.
This would require the main proxy running as root or with some other sort of elevated privileges to allow cross-user network access though, right? If so, wouldn’t that essentially make the user-specific reverse proxy unnecessary in most cases?
It can be run as root, but it doesn’t have to, as it is only accessing the host network, not any user-specific network. Crossing the boundaries from host network to user-specific networks is left to the reverse proxies running as those specific users.
Port 80 and 443 can normally only be bound as root, but you can work around that with either firewall rules or by using something like this in your sysctl config:
net.ipv4.ip_unprivileged_port_start = 80Do NOT do the latter! It allows unprivileged users in the system to spawn processes to take privileged ports starting with port 80 all over to port 1024, including relevant ports like 443 (HTTPS)!
I’m not trying to convince you of this solution (I personally don’t like it), but I am curious what didn’t work for you. Were you unable to get the reverse proxy to serve each service? Were you unable to have the services behind the reverse proxy to talk to each other?
"/usr/share/nginx/html/index.html" is forbidden (13: Permission denied)Does the user have permission to that file?
As far as I understand, it should as it’s embedded inside the actual image file and not mounted.
I’m having other strange behaviour on non-1000 uid accounts. For example, running
docker.io/library/httpdusing the podman examples page on an account that isn’t userid 1000 will create an issue where it complains that libgcc_0 isn’t installed. This works flawlessly on the uid1000user.It does make me think that something needs to be configured for non-uid 1000 users on podman. This is not replicable on arch linux on my raspberry pi, fwiw.
Shell into the container and find out.
What are the assigned subuids and subgids for userA and userB?
Sorry for not answering to your question directly, but why not just use subuids for separation instead?
podman run -d --restart unless-stopped \ --uidmap "+0:@100001:1" \ --name nginx-1 \ --port 8081:80 \ docker.io/library/nginxpodman run -d --restart unless-stopped \ --uidmap "+0:@100002:1" \ --name nginx-2 \ --port 8082:80 \ docker.io/library/nginxI mean, I might be wrong, but this only changes the mapping right? In the case that user broke containment, I would assume that they would have the user id of the user that’s running the process?
Granted this is all in theory so I’m not sure exactly – that might be a perfectly reasonable way to do it.
