I never even thought about this before, but what kind of hardware is a root level DNS server like 1.1.1.1 or quad 8 run on? Are they all roughly the same, or do they take different approaches to the load?
In a previous role at an Internet provider, I was the network engineer in charge of the Anycast DNS cache servers. The company made it some sort of point of pride to buy the cheapest hardware possible. I guess that is the beauty of anycast, failure recovery is pretty much automatic.
Anyways, the servers were dead cheap, single power supply, single SSD. They ran BSD, but my system engineer and I wanted to convert them to Linux, so we ran a proof of concept for switching. We ran performance tests against the same server hardware with BSD and with Ubuntu, I think we saw loss on the BSD server at 325k QPS and Ubuntu made it closer to 500k QPS. Oh, we had like 16 of these caches and our peak load network wide across all of them was something like 36k QPS.
It was a pretty cool setup. If I am remembering correctly we used unbound, ExaBGP, IPTables, and a health check script that would trigger the advertisements to our core network. Although we really wanted to switch to Linux (or rather away from BSD), we were primarily driven by the fact that Cisco IOS-XR does not support DNS query IP SLA tests, which we used previously to trigger the route advertisements.
Bit of a nitpick, but these providers are not running root servers. They’re just recursive caching servers, probably running unbound.
If you want to find out root servers - dig for . - that’s a dot. A full stop. That’s the root. Those can then answer where to find your .coms, .nets, etc. Those, in turn, handle the domain you rent; so on and so forth.
Look up how Cloudflare does anycast BGP. They have servers in data centers everywhere and leverage the fact that they can all be 1.1.1.1 so people’s queries only hit the physically closest server, it’s pretty cool
Most probably off-the-shelf hardware, but with software optimized out the wazoo. Like, reading directly from SSDs instead of using a filesystem. Or more likely using an in-memory database like Redis, perhaps with custom additions.
I knew a guy who went on to work for CloudFlare. Back at that job, he made a write-only static-file storage in Node.js, dumping the contents into a giant file bypassing the filesystem and its metadata overhead. That’s the kind of optimization you want — pretty similar to what database engines like MySQL/PostgreSQL do (iirc one of them does in fact support using a plain disk partition for the database).
Back in early-mid 2010s I’ve read an article on Pornhub’s architecture. They ran Redis behind Haproxy load balancers. An in-memory database, behind load balancers. Some people say that it’s normal, but I’ve never had conditions calling for such a thing instead of sharding harder.
I’m actually surprised that they use some beefy servers. Like, Google famously use shitty and cheap commonplace hardware, but in large numbers, so it’s easy to replace when it croaks.
I never even thought about this before, but what kind of hardware is a root level DNS server like 1.1.1.1 or quad 8 run on? Are they all roughly the same, or do they take different approaches to the load?
In a previous role at an Internet provider, I was the network engineer in charge of the Anycast DNS cache servers. The company made it some sort of point of pride to buy the cheapest hardware possible. I guess that is the beauty of anycast, failure recovery is pretty much automatic.
Anyways, the servers were dead cheap, single power supply, single SSD. They ran BSD, but my system engineer and I wanted to convert them to Linux, so we ran a proof of concept for switching. We ran performance tests against the same server hardware with BSD and with Ubuntu, I think we saw loss on the BSD server at 325k QPS and Ubuntu made it closer to 500k QPS. Oh, we had like 16 of these caches and our peak load network wide across all of them was something like 36k QPS.
It was a pretty cool setup. If I am remembering correctly we used unbound, ExaBGP, IPTables, and a health check script that would trigger the advertisements to our core network. Although we really wanted to switch to Linux (or rather away from BSD), we were primarily driven by the fact that Cisco IOS-XR does not support DNS query IP SLA tests, which we used previously to trigger the route advertisements.
Bit of a nitpick, but these providers are not running root servers. They’re just recursive caching servers, probably running unbound.
If you want to find out root servers - dig for
.- that’s a dot. A full stop. That’s the root. Those can then answer where to find your .coms, .nets, etc. Those, in turn, handle the domain you rent; so on and so forth.Look up how Cloudflare does anycast BGP. They have servers in data centers everywhere and leverage the fact that they can all be 1.1.1.1 so people’s queries only hit the physically closest server, it’s pretty cool
Most probably off-the-shelf hardware, but with software optimized out the wazoo. Like, reading directly from SSDs instead of using a filesystem. Or more likely using an in-memory database like Redis, perhaps with custom additions.
I knew a guy who went on to work for CloudFlare. Back at that job, he made a write-only static-file storage in Node.js, dumping the contents into a giant file bypassing the filesystem and its metadata overhead. That’s the kind of optimization you want — pretty similar to what database engines like MySQL/PostgreSQL do (iirc one of them does in fact support using a plain disk partition for the database).
Back in early-mid 2010s I’ve read an article on Pornhub’s architecture. They ran Redis behind Haproxy load balancers. An in-memory database, behind load balancers. Some people say that it’s normal, but I’ve never had conditions calling for such a thing instead of sharding harder.
https://blog.cloudflare.com/gen13-config/
Dang. They do it all in one Pizza box. I figured it would be a whole rack per unit.
I’m actually surprised that they use some beefy servers. Like, Google famously use shitty and cheap commonplace hardware, but in large numbers, so it’s easy to replace when it croaks.