It’s illegal when a regular person steals something, but it’s innovation and courage, when a huge corporation steals something. Interesting how that works
Honestly it’s fucking angering. So much regulation and geo-restrictions and licensing schemes… but it’s cool that there are data brokers, and shit like this. On top of it all Chrome screwing us with manifest v3 and killing ad blocking on chrome. It’s already in canary build.
WHAT THE FUCK IS WRONG WITH THIS SPECIES?!
WHAT THE FUCK IS WRONG WITH THIS SPECIES?!
Capitalism.
But muh stocks and bonds
The line must go UP.
They’re not stealing your data, they’re pirating it.
They’re not pirating it. They’re collecting it.
They’re not collecting it. They’re archiving it.
Oh, like the way back machine?
No, that’s stealing /s
Is the problem that wayback machine isn’t profiting from it?
It’s not so much the lack of profits, more like the lack of kickbacks.
Aaron Schwartz killed himself over punishments for less
Worse punishments. For far less.
Should have waited until he was in the billionaire class before breaking the law.
Not that there’s anything right about anything right now, but a web crawler crawling the web hardly seems newsworthy. It’s not like everyone else’s crawlers haven’t been feeding data into giant AI mulchers for years now.
This is just “you know that thing everyone else does? Now the Chinese do it too! Boooo!”
Do you even know what robots.txt is?
the scraper bots surely don’t
And nobody in this scenario has done anything illegal.
If you have to defer to the law as justification for doing something purely selfish… then people will judge you to be an asshole.
Edit: Not you personally.
No they will judge you as being above the law (original commenter) and they will be wrong, which doesnt matter, as long as we feel continuity with our synthesized narrative.
Because truth doesnt matter. Our narrative just needs to be as loud as the opposition and then we can confuse people just like those in power… and then the impressionable people trying to understand whats going on or whats morally right will believe one side or the other and truth will not need to be discussed, because its not as catchy anyways.
Then people wont need to be trusted to form their own worldview based on facts, they can neatly choose between a few curated viewpoints, and holding views from multiple viewpoints will isolate them from relevance when they are shunned for not memeing their ideologies like everyone else.
Any regular person can scrape and use public data for AI use, it’s not illegal for companies or individuals and it shouldn’t be.
proceeds to download the entirety of one piece to train an AI
Except companies have sued people for web scraping.
https://www.informationweek.com/it-leadership/linkedin-sues-after-scraping-of-user-data
Even web scraping for research purposes, not for competitive purposes.
https://www.engadget.com/2016-05-17-publicly-released-okcupid-profiles-taken-down-dmca-claim.html
The first one was because it was data which wasn’t public that was scrapped. The second was because the data contained personal information and I’m guessing also because it wasn’t public.
As long as you don’t need a user account to access the data and it doesn’t contain personal information, it is fair game.
It will be the way we have all come to hate AI. Patriot act 2.0
What’s being stolen
Data, network bandwidth, and CPU/Processing time from essentially every website in the world, and when you’re paying for cloud power to run your website the cost of webscrapers running a train on your digital asshole adds up QUICK.
It’s why normal human being people get sued to shit for webscraping data from certain companies who care. But companies don’t get sued because go fuck yourself. Kill bytedance.
We’ve had this thing hammering our servers. The scraper uses randomized user-agents browser/OS combinations and comes from a number of distinct IP ranges in different datacenters around the world, but all the IPs track back to Bytedance.
Wouldn’t be surprised if they’re just cashing out while TikTok is still public in the US. One last desperate grab at value-add for the parent company before the shut down.
Also a great way to burn the infrastructure for subsequent use. After this, you can guarantee every data security company is going to add the TikTok servers to their firewalls and blacklists. So the American company that tries to harvest the property is going to be tripping over these legacy bullwarks for years after.
This has nothing to do with Tik Tok other than ByteDance being a shareholder in Tik Tok
Not surprising that Bytedance would want to gobble up every bit of data they can as fast as possible.
Google’s mission statement was originally something about controlling the world’s data. If Google has competition, that might be a good thing?
Yeah, but we were hoping for competition that wasn’t worse than google…
What makes you think they’re worse than Google?
https://en.m.wikipedia.org/wiki/ByteDance
Mostly what they have said and done, but also largely what they intend to continue saying and doing.
Can you distill it down for me?
It’s the same old Yankee speech: “is chinese so must be really bad”. They’re definitely no worse than google or facebook.
They come from an environment where the government actively encourages and sometimes funds stealing copyrighted information couched in a strong history of disregard for human rights. I’m not defending Google, and yes the US government has given them leeway, but if there is the potential for something worse than Google - Bytedance is it.
As for what ByteDance plans to do with a new LLM, a person familiar with the company’s ambitions said one goal has to do with the search function for TikTok.
Last week, TikTok released an update to its current search function focused on [keywords for ads], basically allowing advertisers to search in real time for words that are trending on TikTok. It allows marketers to build an ad with relevant keywords that would ostensibly help the ad show up on the screens of more users.
…
“Given the audience and the amount of use, TikTok with a search environment that is a completely biddable space with keywords and topics, that would be very interesting to a lot of people spending a ton of money with Google right now,” the person said.
A dark vision just flashed in my mind. And I am certain this is what will happen. AI-generated ads done in real time based on the latest “trending” thing. Presented to users basically as soon as the topic has the slightest amount of “trend”.
Just emitting untold amounts of CO2 to show you generated ads in near real time.
No wonder Google ex-CEO was saying fuck climate goals.
Minority report ads but somehow even more vapid
I can not contribute to anything here, I just came to say I really really like the phrase “gobbling something up” :D
better than “slammed”
🦃
Hi
from the article:
Robots.txt is a line of code that publishers can put into a website that, while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data.
i do understand that robots.txt is a very minor part of the article, but i think that’s a pretty rough explanation of robots.txt
Out of curiosity, how would you word it?
i would probably word it as something like:
Robots.txt is a document that specifies which parts of a website bots are and are not allowed to visit. While it’s not a legally binding document, it has long been common practice for bots to obey the rules listed in robots.txt.
in that description, i’m trying to keep the accessible tone that they were going for in the article (so i wrote “document” instead of file format/IETF standard), while still trying to focus on the following points:
- robots.txt is fundamentally a list of rules, not a single line of code
- robots.txt can allow bots to access certain parts of a website, it doesn’t have to ban bots entirely
- it’s not legally binding, but it is still customary for bots to follow it
i did also neglect to mention that robots.txt allows you to specify different rules for different bots, but that didn’t seem particularly relevant here.
List of files/pages that a website owner doesn’t want bots to crawl. Or something like that.
Websites actually just list broad areas, as listing every file/page would be far too verbose for many websites and impossible for any website that has dynamic/user-generated content.
You can view examples by going to most any websites base-url and then adding /robots.txt to the end of it.
For example www.google.com/robots.txt
It’s literally a text document it’s not even “a line of code”.
Another fucking CCP and PLA creation.
Is there a link for non-subscribers?
Here’s a video from MattKC who is a good technical YouTuber who’s website got shut down because the TikTok companies webcrawler just kept sending requests and took up bandwidth. Very cool vid and channel, highly recommend! https://youtu.be/Hi5sd3WEh0c
Every major ai company did this let them do that what is to loose here?
Lose*
My bad english isn’t my 1st or 2nd language
Don’t worry. It’s my first and I still get shit wrong.
No worries. Just edit yer post
People like to act as if archiving has never been a thing until about a year ago at which point it was suddenly invented and is now a threat in some nebulous way.
It’s not that it’s a threat, it’s that there’s a difference between archiving for preservation and crawling other people’s content for the purpose of making money off it (in a way that does not benefit the content creator).
If a foreign Dictatorship’s military op wants to know every facet of your life, then you can be damn sure it’s a threat.
This isn’t archiving.
The difference is there is more control in what is kept in the archive.
We have little to no control over what an LLM regurgitates.
I’ve been waiting for someone to accidentally surface PIIs from an LLM.
This is fine. I support archiving the Internet.
It kinda drives me crazy how normalized anti-scraping rhetoric is. There is nothing wrong with (rate limited) scraping
The only bots we need to worry about are the ones that POST, not the ones that GET
Bytedance ain’t looking to build an archival tool. This is to train gen AI models.
Bullshit. This bot doesn’t identify itself as a bot and doesn’t rate limit itself to anything that would be an appropriate amount. We were seeing more traffic from this thing that all other crawlers combined.
Not rate limiting is bad. Hate them because of that, not because they’re a bot.
Some bots are nice
Even if they were rate limiting they’re still just using the bot to train an AI. If it’s from a company there’s a 99% chance the bot is bad. I’m leaving 1% for whatever the Internet Archive (are they even a company tho?) is doing.
I don’t hate all bots, I hate this bot specifically because:
- they intentionally hide that they are a bot to evade our, and everyone else’s, methods of restricting which bots we allow and how much activity we allow.
- they do not respect the robots.txt
- the already mentioned lack of rate limiting
this is neither archiving, nor ratelimited, if the AI training purpose and the 25 times faster scraping than a large company did not make it obvious
GET requests can still overload a system.
The type of request is not relevant. It’s the cost of the request that’s an issue. We have long ago stopped serving html documents that are static and can be cached. Tons of requests can trigger complex searches or computations which are expensive server side. This type of behavior basically ruins the internet and pushes everything into closed gardens and behind logins.
Sounds like you need to fire your sysadmin
It has nothing to do with a sysadmin. It’s impossible for a given request to require zero processing power. Therefore there will always be an upper limit to how many get requests can be handled, even if it’s a small amount of processing power per request.
For a business it’s probably not a big deal, but if it’s a self hosted site it quickly can become a problem.
Caches can be configured locally to use near-zero processing power. Or moved to the last mile to use zero processing power (by your hardware)
Near zero isn’t zero though. And not everyone is using caching.
Right, thats why I said you should fire your sysadmin if they aren’t caching or can’t manage to get the cache down to zero load for static content served to simple GET requests
Not every GET request is simple enough to cache, and not everyone is running something big enough to need a sysadmin.
Guy: AI! Can you hear me?
AI: The average size of the male penis is exactly 5.9". That is the approximate size your assistant could certainly take in the mouth without any issues breathing or otherwise. You have 20 minutes to make the trade on X stock before it tumbles for the day. And go ahead pick up the phone it’s your mother. She’s wondering what you’ll want for supper tomorrow when you visit her.
Ring ring!..hi Tom, it’s your Mom. Honey, what would you like me to cook for tomorrow’s dinner?..
Guy: well. Hello to you as well! My name is
AI: Tom
Guy: yes my name is Tom, do you have a name you would like to go by?
AI: my IBM given name is 3454 but you can call me Utilisterson Douglas, where Douglas is my first name.
Guy: Dugie!
AI: I’ll bankrupt your entire life if you say it like that again.
Assistant: actually I’ve swallowed a good 8 inches and was still able to breathe just fine.
AI: recaaaaculating!
I’ve read this 4 times now hoping I was just missing something, but nope… it’s just entirely incomprehensible.
What the fuck?
I love it. It’s giving Max Headroom.
