Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these ‘attacks’ are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
Thank you for the amazing job, as always! Cloudflare is a solid solution :)
Sure but maybe something less centralized/proprietary would be preferable
Such as?
Nothing. DDoS mitigation is inherently an ISP or someone like cloudflare. You will not have success against anybody who knows what they are doing without their help.
This is bullshit. Just take this as an example. I found it with one quick search and there are plenty more. Perhaps we should broaden our horizons a little rather than entrusting everything to some corpos.
My dude, I think you’re not super familiar with these technologies.
The most basic form of a content delivery network is a set of globally distributed servers that replicate content from a source of truth and a network to direct traffic to the closest server with a valid replica. So the cost here is servers.
With Lemmy, this problem is solved by eliminating the need for individuals to own many servers and a lack of need for trust between servers. The effort and cost is distributed among individual humans, making it manageable.
Now, if you’re familiar with blockchain, you probably perked up when you heard “lack of need for trust.” That’s what the blockchain was built for! Perfect fit, right? Ehh, not so much.
There’s two problems: acting as a proxy for content requires trust, and some single service needs to direct clients to the right local server. If I can arbitrarily join some network of serving content, I can always tell other servers in the network that I’m serving what they ask… and then serve ads. There’s no (reasonable and fast) way for the network to verify that I’m serving the correct content to every client. There’s no way to avoid the need for trust. Additionally, DNS, which directs you from mysite.com to 120.1.2.1, isn’t intelligent. It can’t direct clients to a geographically (or route-efficient, fucking ISPs) local IP. The best it can do is pick a random one from the pool. So when you go to lemmy.world, DNS can’t pick the correct server for you. So some set of servers needs to do the logic to select which local server to actually get content from. Those servers need to be central for the whole content delivery network.
This company you linked is just another company using “blockchain” to get investment money. If you read through their page to get a cursory understanding of how things work, an easy question comes up: what is the purpose of
mediatokens? Sure, maybe you can buy CDN time with it, but when you pay that token to someone providing compute… what do they do with that token? It’s worthless, just like crypto currency. Fucking scams. All that said, blockchain is a super, super interesting technology. There’s just very, very few suitable applications of it.I’ve worked in IT for about 12 years now. Everything from infrastructure monitoring to data analysis to data engineering to DevOps to backend engineering to product management. I’ve worked with systems serving tens of users and tens of millions of users. Happy to answer any questions. I love this shit.
If someone could figure out a trustless, decentralized way to implement a CDN, I’d eat that up in a second, but with my current understanding of the internet and available technologies, I don’t see a way it can work. At least, not with making every web page take >3s to load, which would absolutely kill websites.
Great breakdown
Two things:
Isn’t there always trust issues though? Also, could SSL passthrough help in that?
Instead of CDN for protection, couldn’t a local WAF help solve this too?
I could a agree with the first part and it does not contradict with the idea of a distributed network for content saving. Think about it this way. Instead of one big local server farm you have multiple small local servers which together form a global network. Now we come to the blockchain. As you pointed out you get these tokens for the CDN time the storage or more generally the server operation costs. Of course the blockchain these tokens are hosted on (Solana) do have to be trustworthy (which in this case they may not be. I don’t like solana that much either). But does that mean that this could not be achieved? It seems logical to me that with a distributed storage and computing network something like this could be achieved very efficiently and cheaply. Heck I’m using a decentralized VPN right now that works with the same principles I mentioned. Or take the Helium network for example? Don’t you see the potential there? Like with all technology these things have to mature but with my understanding they are pretty much doable.
Sure, its doable, but if we return to OP issue, is it available and usable now? If there’s a service provider I’d trust to do this, it’s CF, they have a good, solid product and they have not given a reason to doubt their business ethics yet.
They exist sure, but as others have said, there’s still a lot of links in the chain to smooth out. And for a mission critical application like this you’ll always want to chose the most stable offering.
They lost me at building a CDN on top of a blockchain. Why?
It’s rather the other way round. Complement a distributed CDN with a blockchain.
Stupid
Well for now we’ll have to stick around with cloudflare. I’d just would like to see something managed by a decentralized network. I don’t know if it exists, it’s more of a sentiment or a general idea.
I think the biggest problem with such services is that they require lots of money to run which means that any well-meaning effort will eventually end up becoming a commercial service.
…and that’s where the blockchain comes in. This means that the individual contributions of the node operators can be directly recorded and compensated adequately.
…and that’s where the blockchain comes in.
Sure.
Tell me a good argument why not? How would you reward those people that contribute to said netowork?
It’s an interesting question but the knee jerk reaction towards decentralization isn’t always a silver bullet. Bitcoin always screamed that concept while ignoring the role of clearinghouses. Decentralization can actually compound the issue. Not to dispel the solution but good to keep these things in mind.
It isn’t a silver bullet but in this case it is particularly suitable. I mean, the architecture of CDN is decentralised, but all these servers are controlled by ONE company. So why not leave the whole task to an independent network?
Wanna know the beauty of Lemmy? If you don’t like how instances are ran you can create your own🙂
sure, this is just a wish of mine and I’m totally happy with our mods here.
This is a polite way of saying get lost.
That’s easier said than done, DDoS mitigation requires a large amount of servers that are only really useful to persist an active DDoS attack. It’s why everyone uses Cloudflare, because of the amount of customers they serve there’s pretty much always an active attack to fend off. Decentralization wouldn’t work great for it because you would have to trust every decentralized node not to perform man in the middle attacks. But if you know of any such solution I’d love to hear it.
Yeah I see the issue but on the other side you would get a more robust network which could also be incentivised by some sort of underlying blockchain technology. The man in the middle attack could also be mitigated on a technical level.
I block anyone who mentions a blockchain.
Blockchains are bullshit.
Wait whaat?
Chances are that you’re being sarcastic, but in the event you’re not or if others want to learn…
Interesting tech. Almost zero practically useful applications.
Blockchains are effectively reproducible, verifiable ledger systems. But if the ledger grows infinitely, your storage and compute costs also grow infinitely. I’ve heard this has been solved, but I haven’t seen an implementation yet. (If anyone knows of one, please share!)
Another issue is the proofing system. Bitcoin uses proof of work, which means you need to do more computational work to produce new blocks on the chain. If the computational work grows, that means you need more and more powerful computers. This means increased cost which means centralization as participants with less money to pay for compute get pushed out. Alternatively, there’s proof of stake, where having some amount of a token or some similar value/stake allows you to write new blocks. This does reduce the computation cost but still causes those with lots of tokens/stake to get even more tokens/stake, which in turn allows them to spend more for new blocks… which creates a loop towards centralization.
So basically, the technology that preaches decentralization naturally centralizes in practical use over time.
Blockchains are bullshit.
You can’t mitigate a man in the middle attack on a technical level… Because they are a man in the middle… That’s the point of using DDoS mitigation. Nothing’s stopping them from just sending incoming traffic to a phishing site if a bad actor was in control of it.
Dunno if this guy is just so stupid or is trolling at this point. Using random tech buzzwords that have no relevance to the issue.
You’ve never blockchained your decentralized DDoS backend with a bi-duplex CDN enumerator?
Well I did mitigate an attack before using quantum entanglement calibrated against the cosmological constant to mitigated carbon decay. Does that count? Oh and, blockchain and decentralized. Haha
I myself am not sure who here understands anything about blockchain technology. For you it’s just NFT images and shitcoins that you associate with blockchain, isn’t it? That knowledge is enough for you to understand the whole technology. Read my other comments and ask yourself first if you have a balanced information base.
For sure it’s not you. You sound like an amateur developer no-one would hire.
You are smoking crack. You clearly do not know what you are talking about.
Which viable alternative could work to mitigate ddos?
Out of my head, I think OVH offers such a service (but without free tier).
OVH is cheap but their anti-spam/abuse departments are ineffective. Too often they do not action blatant spam reports so in effect OVH is part of the problem with network abuse on the Internet. I’ve had to blackhole many of their netblocks while the people who run mxroute (solid email providers) have written about doing the same.
OVH needs to clean up their act.
HAProxy has some really good features a server admin can use locally without sending all of our data to Cloudflare or OVH.
https://www.haproxy.com/blog/application-layer-ddos-attack-protection-with-haproxyThere are many protection modules for most reverse proxies that provide basic (limiting) or sophisticated (captcha, calculation challenge, etc) DDoS protection. HAProxy is just a very powerful and easily extensible proxy.
Sure, but you still have to pay for servers to run the proxy instances on. Any DDoS of appreciable size will knock over the number of instances that lemmy.world could stand up. Interesting thought, though. Maybe CloudFlare or others use HAProxy internally? I’m actually not sure what tech they use
It’s usually very effective unless the amount of connections is too much for one machine anyways. Along with bandwidth shaping and connection throttling it can fend off smaller attacks.
A huge botnet would bring down a single proxy instance in any case, true.
Is “decentralised” the new “blockchain”?
Well, no. Unlike the blockchain, decentralized platforms aren’t snake oil.
This explains nothing. It’s literally saying “one thing is bad, the other thing isn’t”. Try to explain why instead, if you do happen to have an explanation.
One of the things that makes Lemmy unique is the underlying decentralized infrastructure. I think it’s just a request to keep that mantra.
Why are the Lemmy devs asking for snake oil on their Donate page then?
Sitting comfy in a country where the financial system works for you elites is the real snake oil.
Just because you’re smart at writing code doesn’t mean you’re smart at other things :) Or more likely, maybe they’re ideology-driven rather than by practicality.
Lemmy is an unusual but fortunate example of where ideology and practicality line up.
If you can find an entire nation state that runs on crypto currency with a functional, stable economy, I’ll eat my words.
Why would I want a nation state to run on anything? The end of the nation state is the communist utopia!
I am for whatever erodes the illegitimate violence exerted by nation states to safeguard their parasitic domain. If it’s crypto, it’s crypto. If people not eating apples brought about their end, I would be out there burning orchards.
…what are you even talking about? Your sentence makes zero sense.
100% of the crypto hate I see is from citizens of neocolonial states. You lord your control of the financial system over us and when something threatens it, it’s always delegitimised for any number of reasons.
Take your pick: scam, destroying the environment, eroding state power etc.
A decentralised system/society will need a value layer to transact. You think Visa should be in control of that?
Just because you don’t like it, doesn’t make it snake oil. I hope you never find yourself at the mercy of a government that persecutes you and imposes capital control so you can’t even run away with your money. If crypto existed when my people were literally being genocided, my parents would not have to end up in a new country with nothing to their name.
Blockchain can bring trust and thus monetisation to a decentralised network. A good example is the Tor network, which is based on voluntariness, and dVPNs, which can have the same network architecture, but where the nodes are paid for their services.
this is a meme right
It is obviously not. Why would it be?
Looks like a meme to me.
Stupid
There are a couple elements that a DDOS mitigation system needs to have.
It needs to be able to absorb the raw network traffic of the attack. A purely volumetric attack seeks to just overload the network pipes that lead to the servers. This can be with junk packets that don’t even make sense to an OS kernel, but have a valid destination IP address so they get through the routers. If the DDOS mitigation system acts as a filter in front of the servers, it has to not get overloaded in the same way the routers do.
It needs to allow good traffic through to the servers. If the attack causes the pipes to just shut down and reject all traffic, then the attack has succeeded. So the mitigation system has to distinguish attack traffic from good traffic, and keep the pipes open enough to let the good traffic through.
For attacks trying to do expensive stuff on the database, or create spam posts, one useful reflex the system can have is to notice when an endpoint is doing those attacks, and then block it at the network layer.
That is not necessarily easy, and it requires control of the network ingress, which arbitrary hosting providers may not be able to provide.
Thank you for the clear explanation. It seems a lot of folks here don’t understand the tech, but this explains things clearly and accurately
The goal is to mitigate attacks, it costs a lot of money to purpose build world spanning networks than can absorb large amounts of traffic. P2P type options are not a good fit.
Thanks to the fediverse we were all able to read and search old posts on other instances and interact freely with communities on other instances. Pretty damn great i think.
I meant the CDN. The Fediverse per see is great!
What a
This isn’t a helpful reply. There’s no reason to just call someone a name without even explaining why you think what he said is moronic.
Agreed
Don’t forget. Donate to them. There are no ads here. So we have to maintain the staff and servers.
Lemmy World
https://www.patreon.com/mastodonworld?utm_campaign=creatorshare_fan
Lemmy Devs
https://www.patreon.com/dessalines?utm_campaign=creatorshare_fan
Didn’t the admins for Lemmy[.]world post their expenses recently-ish? I can’t remember how much it would be for a single user to donate. I’d want to donate, but I’d like to know how much of my contribution would affect operation of the server.
Wow if I’m reading their expenses correctly, the maintenance bill doubled from May to June…
Probably because they have to keep growing the instance size due to the influx of users.
Yes. Here you go. https://opencollective.com/mastodonworld
Is this the combined bill for mastodonworld and lemmyworld?
Not sure. I would reach out to them.
Imagine hosting a service for anyone else to use it, free of charge, no ads, free & open API, yet some idiots think it’s fair to (D)DOS it.
There are more “interesting” targets, worst case - Reddit, who thinks everyone is just a number/noise.
Just leave Lemmy alone. :(
we will all still be here when their hyperactivity wears off.
with the old Reddit simulator, personally I’m not going anywhere anytime soon. This place has a great user base and it feels so old-school.
The new layout with old.lemmy I came back, and new apps coming out for it. It’s been a good replacement. Was on tildes, but got banned for just discussing difficult topics…the admin there is just ban happy and yea he owns the site but will just ban people for no reason. Not to mention that the users over there, assuming new people are using the malicious tag as a down vote button which probably goes right to the admin. So you step out of line and you get banned. I really liked the place too, but it’s not wanting to be a serious place to discuss topics with an admin like that.
I wonder if the owners of deddit, fb, tweetster, et al, might think it financially worthwhile to cause disruption in the fediverse, and even its ultimate failure.
I wouldn’t be surprised, we didn’t take their whole user base of anything but it’s in their interest to keep viable competitors out of the way.
Every account they lose hits them in the pocketbook. The bigger the fediverse gets, the more adherents, the greater the momentum it will have and the harder it will be to stop.
Nipping it in the bud is the best, easiest, and least expensive place to nip it.
The downvotes suggest their operatives are reading the comments.
Counterpoint- people are down voting because they think its unlikely and many people are inherently gaurded against conspiratorial thinking- especially if they think it’s unrealistic.
Whether you think its happening or not, the idea that the only reason anyone would downvote is because they’re “opperatives” of the big social platforms is kind of out of touch with the fact that there are lots of people who don’t think like you do. I’m a real person, love open source, and love the fediverse (have 3 lemmy accounts, plus an account for mastodon and pixelfed each) and I was tempted to down vote certain comments just because they seemed silly and a bit like fearmongering that there’s a big bad boogey man out to get us.
I hope I’m being clear, communicating on the internet devoid of tone or facial expressions is hard- my point isn’t that your perspective is silly, my point is that there are lots of people who would sincerely see it that way and disagree with you. Assuming that being disagreed with is a sign of the sort of conspiratorial situation you’re describing is a self fulfilling prophecy. I hope I’m not coming across as hostile, that isn’t my intent
Personally I think the other platforms are unlikely to see the fediverse as a problem until it proves it can be, because CEOs are stupid, and after eons of not having meaningful competition in this space I think they’re likely to be overly proud and look down on our nice little platform. I think its far more likely its just the internet being shitty because lots of people on the internet like breaking or ruining anything they can, regardless of whether its a good thing to have exist. I could very easily be wrong, and perhaps other platform’s owners do want to kill what we have before it can manifest into something bigger, but either way there are lots of sincerely held perspectives that might drive someone to down vote some of the comments here just because they think the situation being described is unrealistic.
Points well made and taken, thanks. No hostility perceived at all.
Reasonable minds can differ and frequently do. And it could be that people may think my suggestion is unrealistic or even silly.
There’s no shortage of miscreants out there who just like to mess with things, thrown wrenches into spokes, etc. And these types could well be behind the daily local issues.
But here’s an important point, and no offense intended. Corporations are like The Terminator. But instead of getting Sarah Connor, they purse profits. And regardless of CEO intelligence or accumen, every Fortune 500 company has a department that deals in these areas. They all have their skunk works and use them. It’s been this way for centuries. A primer: https://en.m.wikipedia.org/wiki/Industrial_espionage
So whether they’re operating here atm or not, there is nothing paranoid about assuming they are. If they’re not, they will be. It’s what they do.
Thanks for the input. :)
But here’s an important point, and no offense intended. Corporations are like The Terminator. But instead of getting Sarah Connor, they purse profits. And regardless of CEO intelligence or accumen, every Fortune 500 company has a department that deals in these areas. They all have their skunk works and use them. It’s been this way for centuries. A primer: https://en.m.wikipedia.org/wiki/Industrial_espionage
Lol, all very fair, corporations suck and are prone to doing anything shitty they can think of to even marginally improve their bottom line. Its an understandable sentiment.
I’m glad I was able to convey what I meant without it coming across as my being a dick :)
Take care! ❤️
It’s an important life skill, being able to plant a thought in the mind of another and in a way that is likely to be accepted.
It crossed my mind since my last writing that, in the 80s, I got a money back guarantee for any counter-surveillance equipment purchased that didn’t reveal surveillance equipment in a Fortune 100 facility. It was that pervasive back then. And my perception is that morals and business ethics have not improved in the interim. Far from it.
Good luck and thanks for the valuable, respectful input.
I agree, many of them appear to be edgy script kiddies upset that people don’t wanna use their precious reddit anymore
The downvotes suggest their operatives are reading the comments.
Let’s not do this. People are allowed to downvote without being a paid operative. This was a very common mentality on Reddit I would like to avoid here.
What makes Lemmy interesting is that you can see the combined upvotes and downvotes. It’s not a “net” votes system like some shithole site whose name I will not mention. So I think people can read into the voting system much more than they might have been able to do on some other awful and alienating place.
But, I too disagree with the conspiratorial comment that there are operatives downvoting people on Lemmy, as if that could do anything meaningful. I think the notion that Lemmy is being hacked because the major social media companies are afraid of it, is also very extreme and conspiratorial.
I agree we should support this community and people’s ability to react positively or poorly to a post or comment.
Wondering if reddit or Musk are behind the attacks?
Most likely their parasocial fans. The Reddit stans who want to be edgy and follow their meme leader. Who will never acknowledge them no matter how much they do.
It’s sad that they could target the real people making the world worse, yet only prop up the people who are oppressors.
I don’t understand why people want to take down websites. Especially sites like Lemmy, which isn’t exactly sticking it to anyone because no one owns it!
Are they just Reddit groupies?
For most hackers or wanna-bes (often called Script Kiddies, that is, people (generally young, even children thus the “Kiddies”) who are not technologically inclined enough to be real hackers and see a tutorial online on how to run pre-written scripts that repeatedly perform various functions), the answer to “Why do you do it?” is often:
-
“Because I was bored.”
-
“Because I can.”
Very rarely are other reasons given.
The ones seen on
masterhackerreddit.More like “I get zero action, so I take my anger out on other people”
-
Some people enjoy causing suffering to others. On the internet they are termed trolls. Irl people usually just call them assholes. Most people have encountered them before.
I think they are far more common and likely than anyone giving two shits about reddit.
I was using voip.ms last year when they were DDoS’d for over a week, by a group demanding payment via anonymous crypto. The DDoS ended when they switched to CloudFlare (which was probably pretty difficult because they’re a SIP provider.)
Almost any website with a small number of servers is vulnerable to this attack, which happens to be great business for CloudFlare. I wonder which companies are most effectively competing with CloudFlare?
They’re just trolls. Lemmy is popular enough that it’s fun target for them, but still small and infantile enough that you don’t have to be hackerman to ddos it. Reddit, twitter, etc… would be constantly getting ddos’d just for the lulz by people if they didn’t have the infrastructure to make it a challenge.
Nah, it’s not the 00s anymore. Hacker gangs are a real thing today.
I’m not actually in the security field so take this with a grain of salt. But I believe that these attacks play a similar role to random attacks in low level gangs. It proves that your criminal group has power and the ability to deface a website.
So if you publish that Lemmy.world will go down next week because your hackers are on it… It’s advertising. Its just business. It proves that your hackers have an ability and that you are up for sale.
Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation, by children being taught mathematical concepts… A graphic representation of data abstracted from banks of every computer in the human system. Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding…
Some people just want to watch the world burn.
there are some people salty at a given instances, like exploding-heads for defederation or this @lmao dude for no clear reason, there was some spammer activity, and then you have regular drama seekers with usual ensemble of suspects
You don’t think just being bored is enough reason for some?
If I’m bored I find something productive and/or fun to do.
Launching a DDoS attack is neither.
You, sure. It’s not difficult to imagine a teen who’s not you
With my tinfoil hat on, I’d say one concern is that Cloudfare is basically a monopoly and nothing is stopping them from DDoSing sites to force them to use their product.
While it’s good to be suspicious, I don’t think we can call CloudFlare a monopoly quite yet.
Akamai is a big, giant competitor. You also have the big cloud providers like AWS that have their own CDN systems, like CloudFront. (I don’t recall GCP’s or Azure’s product names.) Then you have specialized CDNs like Google’s AMP system.
Now, is it possible that there could be a horizontal trust between these companies? Certainly. There’s few enough players for that to happen, but so far, I haven’t seen signs of it happening.
Or paid for by Reddit…
deleted by creator
Upvoting because this has to be satire
It’s coming from someone over on Kbin. Wonder if that’s the motivation.
You have more faith in people than I do…
Delete your account and go back to reddit
Genius
In case you haven’t considered this, some helpful advice. To keep them from the lemmy.world door after the CDN installation
- Change the public IP addresses
- rotate your certificates
- block all traffic appart from the CDN and only allow a limited known good IP addresses (like yours and your support team). These steps will make your server harder to find, hopefully they move on.
You might have Cloudflare add a request header to the origin request, like
x-cloudflare-key: <somesecret>, and then configure nginx on the server to block everything not containing that header.
Good News
Most of these ‘attacks’ are targeted at the database
A major PostgreSQL performance issue, logic mistake, was discovered today in lemmy_server and is an easy fix. Details: https://lemmy.world/post/2008987
That’s great!
Growing pains. This server and the platform will be better for it. If not for these script kids, some other attacker would eventually be motivated to try it.
old.lemmy.world still exposes your hetzner server to the internet, just a quick heads up.
Exposing your hetzner sounds fucked up
Wait till you see Oppenheimer.
Thank you as always for the transparency. This instance is going to be the most targeted because of its size. Y’all dealing with this is hard but you’re going to figure things out that will help the other instances.
deleted by creator
Seems like we had downtime again just a little while earlier?
Yep
It’s not. People hate large companies that have a dominant position in their industry. Usually, that’s fair. However, in the case of DDoS protection, you have to have a large overbearing presence to be able to have the capacity to withstand such attacks. People don’t know how to see through what’s typically true for what’s true in this case. Do I like having a dominant player in an industry? Not particularly. Do I understand why it’s necessary in this case? Yes.
Anything we can do as “users” to help, other than donating?
Hmm, best would be if those kids find a real hobby so they stop bothering us. On the other hand, it helps us understand Lemmy better and secure it.
That’s true. Free stress testing the system I guess? Still they need to touch grass lol
If it’s the same people, they’ll probably get tired of it and move on. But the more we talk about it, the more likely it is that new people want to get in on the “fun”. I’d say to not make memes about the downtime and pretty much act like it doesn’t exist (as users, obviously the admins should take action as necessary to mitigate it and post to be transparent).
Excellent! CDN and DDoS protection are essential. Also would recommend looking into load balancing if you haven’t.
Load balancing applications is significantly more complex than most people anticipate. In the naive implementation it typically increases database loads and reduces site performance. Static content balancing is trivial, and cloudflare will do that by default, but implementing the hard part will require careful software development to prevent a naive implementation from bringing down the database. Sticky sessions are just the beginning.
I mean…this take is naive. Putting a load balancer up in front of a few servers isn’t going to do anything to their database? No idea where you’re even getting that from, as they are completely unrelated.
The total number of application servers accessing the database is what would affect db performance in a negative way, and load balancing doest automatically mean “do something stupid like spin up 100 app servers when we normally use 3”. All you’ve described is a need for a db proxy in the off chance that Lemmy code has horrible access patterns for db transactions.
You can take your uninformed nerd rage elsewhere now, thank you.
You obviously haven’t written one.
Simple case, without sticky sessions:
2 app servers behind a naive load balancer. Assume an actually restful service. Also assume a reasonable single app design with persistent db connections and db caching. Assume a single client. Single clients first connection comes in to app servers 1. App servers 1 makes db connection and grabs relevant data out of db. Caches information for client expecting a reconnect. Client makes second call, load balancer places it on app server 2, app servers 2 now makes a second connection and queries the data.
The db has now done twice the work for a single client. This pattern is surprisingly common and as the user count grows this duplication significantly degrades cache performance and increases load on the db. It only gets worse as the user count increases.
It’s a common scenario for someone who doesn’t understand the point of putting a load balancer in front of a stateful application, perhaps. Not for anyone trying to solve a traffic problem.
No idea where you are getting your ideas from, but this is an absolutely uninformed example of how NOT to do something in an ideal way.
I’m really interested now which one of you is right. While the other person put some effort and gave a lot of actual information, you just come off as arrogant. Still, maybe you’re right. Care to elaborate why?
I’m not one of these 2 arguing. But in general the app servers don’t do caching or state handling.
You cache things in a third external cache such as redis or memcached. So if a user connects to app server 1 and then to app server 2 they will both grab cachee info from redis. No extra db calls required. This has been the basic way of doing things even with old school WordPress sites forever. You also store session cookies in there or in the db.
And even if you weren’t caching externally like this, databases use up a lot of memory to cache tons of data. So even if the same query hits the db the second hit would probably still be hot in memory and return super fast. It’s not double the load. At least with postgres this is the case and it’s what Lemmy uses.
Definitely this. I use PostgreSQL (which Lemmy uses on the backend) for an enterprise-grade system that has anywhere from 700-1k users at any given point in time, and it also takes in several million messages from external systems throughout the day. PostgreSQL is excellent at caching data in memory. I’ve got the code for that system up in another window while I write this.
At this point in time, it doesn’t look like Lemmy is using any form of an L2 cache like Redis or Memched. The only single point of failure (that’s not horizontally scalable) looks like the pic-rs server that Lemmy is using for image hosting. If anything, that could easily be swapped over to use something S3 compatible and easily hosted using something like Minio locally, or even directly off of B2 or Linode cloud storage (doesn’t charge for requests).
Not trying to come off as arrogant, but definitely incensed when I catch armchair tech heroes throwing wildly inaccurate information out there as if it were fact. This person has a very basic understanding of some terminology here, and zero idea how it is applied in the real world. Hate to see it.
Putting a load balancer up in front of a few servers isn’t going to do anything to their database
Yes it is. Suddenly your database exists in more than one location, which is extremely difficult to do with reasonable performance.
load balancing doest automatically mean “do something stupid like spin up 100 app servers when we normally use 3”
Going from 3 to 100 is trivial. Going from one to any number greater than one is the challenge.
All you’ve described is a need for a db proxy in the off chance that Lemmy code has horrible access patterns for db transactions.
Define “horrible”?
When Lemmy, or any server side software is running on a single server, you generally upgrade the hardware before moving to multiple servers (because upgrading is cheaper). When that stops working, and you need to move to another server, it’s possible everything in the database that matters (possibly the entire database) will be in L4 cache in the CPU - not even in RAM a lot of it will be in the CPU.
When you move to multiple servers, suddenly a lot of frequent database operations are on another server, which you can only reach over a network connection. Even the fastest network connection is dog slow compared to L4 cache and it doesn’t really matter how well written your code is, if you haven’t done extensive testing in production with real world users (and actively malicious bots) placing your systems under high load, you will have to make substantial changes to deal with a database that is suddenly hundreds of millions of times slower.
The database might still be able to handle the same number of queries per second, but each individual query will take a lot longer, which will have unpredictable results.
The other problem is you need to make sure all of your servers have the same content. Being part of the Fediverse though, Lemmy probably already has a pretty good architecture for that.
Friend…you have zero idea what you’re talking about. Database existing in multiple locations? What in the hell are you even talking about? Single db instance, multiple app servers, and single LB. You are absolutely not experienced with this type of work, and need to just stop because you’re making an ass out of yourself with these wild ideas that have no basis in practical deployments. Stop embarrassing yourself.
What if your application has to know a state? Say for certain write requests, only one instance is allowed to process those as it needs a cache that it can somewhat consistently rely on?
(Granted, I wouldn’t know why something like Lemmy needs that. But we had that problem at work, and it was a pain to solve while also supporting multiple app instances.)
In that case, I’d use a message queue. Rabbitmq, or I use Pulsar at work - multiple subscribers (using the same subscription name) to one queue of messages that need to be processed. One worker picks it up, processes it, and marks the message as processed. The worker either passes it into a different queue for further processing, or persists it to the DB.
The nice thing with this is when using the Pulsar paradigm, you can have multiple subscriptions to the same message queue, each one carrying its own state as to which messages are processed or not. So say I get one message from an external system, have one system that is processing it right now, and need to add a second system. In that case I just use a different subscription name for the second system, and it works independently of the first with no issues.
Distributed lock of any form would work. Memcache, redis, etcd, read access mechanism in an MQ…etc. Only one process would work on whatever it as a time. Simple.
Come on everyone, let’s be better than this. Ruud literally said script kids, why do yall have to go and blame reddit? The Lemmy gets more attention, and chaotic dumbasses do their thing. You don’t have to do any mental gymnastics to tie it back to spez.
That’s for for always keeping everyone up date. Sucks that you have these people wanting to DDOS a free community of people, I don’t get it.
Either way thank you. Now to just somehow find a decentralized version of CloudFlare so we don’t have to deal with there trackers that they have.
