%%excerpt%% Reddit has commenced its assault on search engines, blocking those that don’t have a commercial relationship with the company, like Google.
They’re likely blocking user agents too, which I think also doesn’t have legal enforcement (as in DuckDuckGo can just use “Google” unless they said otherwise.
this is just going to cause indexers to ignore robots.txt
“We always obey the robots.txt”
They’re likely blocking user agents too, which I think also doesn’t have legal enforcement (as in DuckDuckGo can just use “Google” unless they said otherwise.
Rate limiting could “fix” that unfortunately.