hmm, i though websites just blocked crawler traffic directly? I know one site in particular has rules about it, and will even go so far as to ban you permanently if you continually ignore them.
There are more crawlers than I have fucks to give, you’ll be in a pissing match forever. robots.txt was supposed to be the norm to tell crawlers what they can and cannot access. Its not on you to block them. Its on them, and its sadly a legislative issues at this point.
I wish it wasn’t, but legislative fixes are always the most robust and complied against.
yes but also there’s a point where it’s blatantly obvious. And i can’t imagine it’s hard to get rid of the obviously offending ones. Respectful crawlers are going to be imitating humans, so who cares, disrespectful crawlers will ddos your site, that can’t be that hard to implement.
Though if we’re talking “hey please dont scrape this particular data” Yeah nobody was ever respecting that lol.
hmm, i though websites just blocked crawler traffic directly? I know one site in particular has rules about it, and will even go so far as to ban you permanently if you continually ignore them.
You cannot simply block crawlers lol
Well you can if you know the IPs that come in from but that’s of course the trick.
last i checked humans dont access every page on a website nearly simultaneously…
And if you imitate a human then honestly who cares.
Detecting crawlers can be easier said than done 🙁
i mean yeah, but at a certain point you just have to accept that it’s going to be crawled. The obviously negligent ones are easy to block.
There are more crawlers than I have fucks to give, you’ll be in a pissing match forever. robots.txt was supposed to be the norm to tell crawlers what they can and cannot access. Its not on you to block them. Its on them, and its sadly a legislative issues at this point.
I wish it wasn’t, but legislative fixes are always the most robust and complied against.
yes but also there’s a point where it’s blatantly obvious. And i can’t imagine it’s hard to get rid of the obviously offending ones. Respectful crawlers are going to be imitating humans, so who cares, disrespectful crawlers will ddos your site, that can’t be that hard to implement.
Though if we’re talking “hey please dont scrape this particular data” Yeah nobody was ever respecting that lol.
Both paragraphs demonstrate gross ignorance