Davriellelouna@lemmy.world to Technology@lemmy.worldEnglish · edit-22 个月前The AI company Perplexity is complaining their bots can't bypass Cloudflare's firewallwww.searchenginejournal.comexternal-linkmessage-square193fedilinkarrow-up1750arrow-down15
arrow-up1745arrow-down1external-linkThe AI company Perplexity is complaining their bots can't bypass Cloudflare's firewallwww.searchenginejournal.comDavriellelouna@lemmy.world to Technology@lemmy.worldEnglish · edit-22 个月前message-square193fedilink
minus-squareubergeek@lemmy.todaylinkfedilinkEnglisharrow-up3·2 个月前Except, it’s not a live user hitting 10 sights all the same time, trying to crawl the entire site… Live users cannot do that. That said, if my robots.txt forbids them from hitting my site, as a proxy, they obey that, right?
Except, it’s not a live user hitting 10 sights all the same time, trying to crawl the entire site… Live users cannot do that.
That said, if my robots.txt forbids them from hitting my site, as a proxy, they obey that, right?