As an alternative to mounting full /dev/dri, there are symlinks under /dev/dri/by-path/ which you can use. IIUC they are linked by the used PCI slot, hence should be as stable as the physical order of the cards.

I use this pattern in a compose file:

...
devices:
  - source: /dev/dri/by-path/pci-XXX-render
    target: /dev/dri/renderD128
    permissions: rwm

In case you are unaware, friends don’t let friends use ollama. The llama.cpp CLI has become pretty much equally easy to use, plus since you use Docker there’s no need to compile anything.

Edit: typos and missing words

The service starts as it should, right? The errors appear only on request, right? Any other sections in the compose logs that seem relevant?

You use podman-compose with the Podman backend?

And there are no volumes mounted, just to be sure?

No expert but don’t the logs look like the requests are coming through?

Yeah right? At least a layer of VPN on top.

Hugging Face used the incident to promote running open-weights LLMs on-prem, which a large part of their business focusses on.

Their blog post

Your biggest issue with 2010 cards will be software (inference engine) support, I assume.

To add some practical advice:

It depends on what you mean by more advanced models. I run Qwen3.6-27b on 48GB VRAM across 3 cards (RTX 2000e Ada), and with the recent software optimizations merged into llama.cpp (tensor parallelism & MTP) I get around 30 tokens per second in generation. I use the model through openwebui for (agentic) web research and simple Q&A mostly and I’m quite happy with what it can do.

If you want something similar, maybe look at one or two second hand V100 PCIE 32GB. Or something from the Intel Arc Pro series, if you don’t mind the software support lacking behind a bit (as in less optimized).

Also it might be worth reading into the difference of dense vs MoE models, if you’re new to that. For MoE models, if your system RAM is fast enough, it’s often viable to offload the “experts” (largest parts of such models) to RAM, reducing VRAM capacity needs. Note that server motherboards with e.g. octa-channel RAM have a huge advantage over consumer boards (making DDR4 interesting despite slower speed per module).

And to adress your last question, while I have no direct experience, I’ve seen posts online about people connecting Strix Halo or DGX Spark devices, but usually via a 10+Gbit/s switch as interconnect is crucial (except if you just want to load balance).

Self-hosting LLMs is a very fun thing to do, but also a time- and money-consuming rabbit hole. You might wanna check out the LocalLlama community over at shitjustworks.

Edit: typos

SFTPGo is such an awesome project, never had any problems with it.

It really depends on how much you enjoy to set things up for yourself and how much it hurts you to give up control over your data with managed solutions.

If you want to do it yourself, I recommend taking a look at ZFS and its RAIDZ configurations, snapshots and replication capabilities. It’s probably the most solid setup you will achieve, but possibly also a bit complicated to wrap your head around at first.

But there are a ton of options as beautifully represented by all the comments.

That sounds awesome! No issues at all so far?

Thanks for the list! Do you use Pangolin yourself?

I’ve been testing Zed for the last couple weeks for some Vue / Nuxt projects. It works great for that and seems very stable so far, but is also developed by a for-profit. Curious to see how the Zedless project works out.

I would recommed to use redundant storage, such as a RAID 1 (or 5 or 6, if you want a more advanced setup). This way your data doesn’t die with your SSD.

“Independent” browsers. Yeah right.

That’s really helpful, thank you. I’ve ordered an AX23 which will arrive tomorrow. I’ll try to figure it out in the next few days and report back.

Thank you! I’ll evaluate and report back.

And openwrt is capable enough?

Yeah it’s insane right? Every address is reachable when I open a port range. And it’s like there are ~ 10 predefined services (HTTP/S, SMTP, …) and the category “All other ports” where also 22 is part of. So I really have the choice to either keep everything shut or leave everything wide open.

I think I can’t use my own modem but I’ll have to double check with my ISP. But yes the Wi-Fi is also provided by that router and it’s also quite crappy.

Thank you! Do you have an example for such a firewall device? Could something like the TP-Link Archer AX55 in IPv6 “pass-through” mode do the job? Or would you go for a standalone firewall? My budget is around a hundret bucks.