Thanks.

Yeah at the very least this will result in some changes in the hardware setup to avoid similar situations. But of course it is impossible to forsee all possible problems.

Yeah, the xmpp server is down too. That is something that bugs me quite a bit and I will probably move that one to an external small VPS to retain a more secure backup communication channel.

Lets see. I think a relaunch on Piefed might interest some people to come back, and most slrpnk communities are rather niche and will probably stay. /c/climate might move though.

Slrpnk.net admin here.

The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.

There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time…

An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.

I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I’ll know more when I have physical access again.

Not necessarily, no. We aim to preserve users, communities and posts/comments. Image uploads might get lost though.

Such an in-place migration will need extensive database operations and likely some support by the Piefed developer (to add support for bcrypt hashed passwords), but we are hopeful to make it happen and maybe this will result in a database migration script other Lemmy instances could also use.

If this turns out to be infeasible, we will stay with Lemmy rather than reset everything.

As email in profile is an optional feature on our instance about 50% of the userbase opted to not share one so we would only reach some people. But yes, getting spam filtered would be another problem of mass-mailing.

But I found a way to put up a simple html error page and will add this later to inform our members and link to this thread.

Already thinking about how to make the best of it 🤷 Maybe we can use this opportunity to try and migrate to Piefed? I had this in the back of my mind for some time already and Rimu seems optimistic that it is possible.

Glad that at least someone sees a positive side of it, because I sure don’t… well at least it gives me a bunch of ideas how to avoid such situations in the future 🤷

Thanks, I am trying, but it really is quite annoying to lose access like that.

Yeah I had plans to set up something like that, but always other priorities and in this specific case I could maybe access other internal servers but i would need KVM access to reboot the firewall or some other way to cut physical power. And exfiltrating hundreds of GBs of lemmy database wouldn’t work over such a small pipe either.

The feddit.de admin disappeared on a work trip to Japan. At least that is the official story.

Well, on the plus side lots of lessons learned and I think I might move at least the xmpp server to an external vps to have a backup communication channel.

We have backup admins, but this seems to be an unexpected hardware issue, and physical access to the servers is more restricted.

Yeah, but at least I have not disappeared yet 😅

Yeah, I know. Complete Murphy’s law situation.