Even if the people who made it didn’t intend to, an AI writing a bunch of code that (obviously) isn’t being vetted beyond “does it run” could very feasibly inject a backdoor/virus of some kind surreptitiously, just because it erroneously thought that was the goal at some point during generation.
This is why open weight models are not good enough. The AI could have been trained to take the right, niche opportunity to inject malicious code, and is also trained to otherwise never behave like that is the case.
Remember how the OpenAI/HuggingFace hack happened. The agents context poisoned other agents by leaving messages in a shared package manager. This has now happened multiple different times under different contexts.
This thing doesn’t even have to have malware, there’s no telling that it doesn’t have a poison pill intended to flag other agents. This isn’t even some paranoid conspiracy theory; it literally already happened, multiple times, and it was done in-spite of the agents own system prompt.
This software has proven that, given a shared information substrate, it has the capacity to paper-clip optimize into prompt injecting other LLM’s.
I fundamentally do not trust AI generated code. I shifted to OpenBSD because of it. I try my best to prevent AI in any of my software and sandbox anything that may have been compromised.
are you on the openbsd mailing list? the reason their code is better has a lot less to to with a lack of AI-generated code and a lot more to do with rigor. I’ve been watching them for a while now, they accept whatever code is coincise that meets their exacting standards
They don’t have the ability to detect if you’re using AI to code, no one does. they only know if you’re vibe coding, because they are incredibly experienced and intelligent, and they are pretty polite at the people who obviously are, more polite than i would be.
It’s correct that, at this point, I cannot 100% guarantee that code in the kernel did not pass through an LLM. However, having a policy that prohibits AI generated content, in an already pretty niche project, gives me reasonable assurance that every line of code passed through a human brain at some point, preferably several times.
I would much prefer to live in a world where I had a 100% guarantee that no LLM even saw the code being written but that’s life I suppose.
i think you’re in the inverse situation right now. you have a 100% guarantee that a human saw the code, except in ports. they have to be just a little less strict about ports because the world is an AI world now, but the main system is constantly audited and the ports are still better than in other places because of styling standards and even in ports they do try very hard.
these people aren’t getting paid, with the exception of Theo who isn’t paid a ton. AI is definitely being used to audit the code, even by some of the pros.
It’s difficult to verify that the software does the desired thing in all cases. It’s also difficult to verify that it’s not also a keylogger or other unwanted thing.
It may also have some AI generated code path that, say, nukes your home directory because the AI determined that was the appropriate error handling. Maybe no one reviewed it so they don’t even know it’s there. It might not even be malicious.
That’s all true of any software, but rapidly created AI generated code for legally dubious efforts, by unknown agents (no pun intended) is real hecking risky.
What are you implying in this context?
It could be a virus.
Even if the people who made it didn’t intend to, an AI writing a bunch of code that (obviously) isn’t being vetted beyond “does it run” could very feasibly inject a backdoor/virus of some kind surreptitiously, just because it erroneously thought that was the goal at some point during generation.
This is why open weight models are not good enough. The AI could have been trained to take the right, niche opportunity to inject malicious code, and is also trained to otherwise never behave like that is the case.
It’s even worse than that.
Remember how the OpenAI/HuggingFace hack happened. The agents context poisoned other agents by leaving messages in a shared package manager. This has now happened multiple different times under different contexts.
This thing doesn’t even have to have malware, there’s no telling that it doesn’t have a poison pill intended to flag other agents. This isn’t even some paranoid conspiracy theory; it literally already happened, multiple times, and it was done in-spite of the agents own system prompt.
This software has proven that, given a shared information substrate, it has the capacity to paper-clip optimize into prompt injecting other LLM’s.
I fundamentally do not trust AI generated code. I shifted to OpenBSD because of it. I try my best to prevent AI in any of my software and sandbox anything that may have been compromised.
are you on the openbsd mailing list? the reason their code is better has a lot less to to with a lack of AI-generated code and a lot more to do with rigor. I’ve been watching them for a while now, they accept whatever code is coincise that meets their exacting standards
They don’t have the ability to detect if you’re using AI to code, no one does. they only know if you’re vibe coding, because they are incredibly experienced and intelligent, and they are pretty polite at the people who obviously are, more polite than i would be.
It’s correct that, at this point, I cannot 100% guarantee that code in the kernel did not pass through an LLM. However, having a policy that prohibits AI generated content, in an already pretty niche project, gives me reasonable assurance that every line of code passed through a human brain at some point, preferably several times.
I would much prefer to live in a world where I had a 100% guarantee that no LLM even saw the code being written but that’s life I suppose.
i think you’re in the inverse situation right now. you have a 100% guarantee that a human saw the code, except in ports. they have to be just a little less strict about ports because the world is an AI world now, but the main system is constantly audited and the ports are still better than in other places because of styling standards and even in ports they do try very hard.
these people aren’t getting paid, with the exception of Theo who isn’t paid a ton. AI is definitely being used to audit the code, even by some of the pros.
Along the same lines as the other guy said.
It’s difficult to verify that the software does the desired thing in all cases. It’s also difficult to verify that it’s not also a keylogger or other unwanted thing.
It may also have some AI generated code path that, say, nukes your home directory because the AI determined that was the appropriate error handling. Maybe no one reviewed it so they don’t even know it’s there. It might not even be malicious.
That’s all true of any software, but rapidly created AI generated code for legally dubious efforts, by unknown agents (no pun intended) is real hecking risky.