Price drop caused in majority by open weight models. No wonder Altman wants regulations against open models and to take everybody’s home computer’s RAM.
People running local models would RUIN their profit margins.
And nVidia is now selling hardware for local models specifically. For cheap.
There is a clear winner in the AI race and its hardware makers. To me, its almost like Jensen Huang calculated this outcome quitea while ago, and fed into the bubble with the loans to the software companies.
During a gold rush, you want to be the guy selling shovels.
You do, however Nvidia got greedy and started to circular invest in the billions if not trillions into those buying the shovels. Now, when the bubble bursts, Nvidia will be hit massively as well.
Eh somewhat. I think they were careful enough to just have the loans not actually be in their name if you follow it as deeply as Ed Zitron has. But Nvidia is screwed from a different angle as they’ve set up expectations of selling more GPUs to data center builders each quarter in order to justify their share price.
That’s still much better than holding the actual bag though.
Nvidia did a massive share buyback with a portion of the ridiculous profits they made last year. That serves to temper expectations, as they’re signaling that they’re not going to be investing all their profit in yet more production (which is difficult anyway given they don’t have their own fabs and TSMC seems to be running at full capacity).
Likely they’ll be doing another one next year and shareholders will be happy.
If they can refinance their own customers to buy their own products without getting caught in the middle that would be even more scandalous than the circular financing itself but then, the financial system has been completely broken for a good while.
They’re basically operating as a kinda/sorta backstop on the loans by saying they’ll rent them back if everything falls through and be a guaranteed customer…but I think they did so in snaky legalese that lets them ultimately get away without actually renting them. I think it’s through like subsidiaries and shit. Oracle is one of the actual bag holders and I think there are several others. They set up shell companies for each specific data center project and shit. It’s quite the fucking racket and should pretty obviously be illegal from multiple angles.
That’s the best way to describe this.
They don’t have profit margins though
This is the logical endgame IMO so I get why the losers are fighting it.
They can only afford to buy the worlds compute for so long though.
The world at large is nothing like us here on lemmy. Few people even have a PC, let alone interest in local models.
What profit margins? :)
Seriously though, home AI is never going to compete. Few people have PCs in their home any longer and only a tiny subset of those users will fiddle with local models. I’d like to, but I don’t have the compute. :(
If hardware prices weren’t insane, we would have prosumer level turnkey home AI already.
Now it’s just for hobbyists with too much money.
And I don’t see that changing. :( And in no case are loads of people going to get into local AI. Lotta nerds around here, but Joe on the street barely knows what AI is, and he certainly doesn’t have a PC any longer.
It’d be Apple packaging local AI capabilities to something like an Apple TV Pro or any other manufacturer with enough integration.
Google might do it too, but less private - ad revenue you know.
But with these prices a decent local AI box would be closer to 1k, ain’t nobody buying that.
At best a macbook air/neo or equivalent on windows good enough to browse the average web and the gamer joe will have a gpu with ~8GB vram for most games (aka 3070 or something like that)
Honestly, if these shit heads hadn’t inflated RAM prices with their speculative datacenter building, I could probably afford to run a quantized GLM-5.3 on local hardware right now
That’s a lot of VRAM! What would be your dream build in that scenario?
I love how we went from 1. AI will make your computer obsolete to 2. rent out your computer to AI to 3. sorry, AI doesn’t pay enough to rent your computer. That’s an entire tech boom and bust cycle squeezed into one Mac’s idle time.
Unfortunately in addition, they took all the money for like three tech booms and slammed it all into that one cycle because the US economy was very stagnant and it seemed like their only move that made line go up. Whole thing has been exhausting.
I’m tired, boss.
With the amount of debt (new money) they’ve created out of thin air, I imagine inflation over the next 5 years will be similar to the last 5 years.
A global distributed AI network using currently unused local RAM, should put Altman’s concerns about efficiently to bed. I’m sure he’s absolutely delighted.
Isn’t that bad for the big US AI companies? The only thing they are doing that actually makes money is selling their inference service on their private models and they need to make a much larger profit on it to even start paying off the interest on the debt they have accrued on infrastructure and training costs.
This is in my opinion why there is all the scare mongering to try and lock out open weight competition to prevent decoupling inference from the models being run. Otherwise inference just becomes a low margin utility service and the big tech companies that have bet big on AI lose big.
Yes, it is. Their costs aren’t going down, only what they can charge and they struggle to be profitable already now. Their only hope is outlawing open models from frontier applications and coding.
Confusing headline/article. Darkbloom does not serve proprietary models. It is competitive for smaller models. best price for qwen 3.8 27b.
The actual competition problem is from deepseek which has open sourced its inference server architecture on Nvidia (and Huawei) datacenter cards, and deepseek v4.1 flash is one of the best combos of good, fast, cheap. But it needs a mac ultra for q4, and datacenters much faster/cheaper with batch service. Qwen 4 is coming out next week with 2 good models that fit under 128gb.
As I understand, only 27B is known to come out as open weight, Max and Plus are API-only anyway and it’s suspected that so is Flash. Makes sense tbh, they’ve established their position in the market, now it’s time to stop giving people better and better models they can run locally. It eats into their profits.
Max has always been API/proprietary. Plus has shifted between the 2, but could very well be API only. Flash the 120b parameter model was new in 3.8. Current king with strata, and I’d be surprised if it wasn’t open. 27b is what drove 5090 to $7000, though flash next is the real 3.8 king.
Cheaper Tokens is a good thing. Maybe not for people who want to sell them to you, but if you buy them, that is awesome
This can’t possibly be acceptable from a privacy perspective
Edit: Was wrong. They actually have a plausible privacy story. Inference happens in the local machine’s Trusted Execution Environment. https://www.eigenlabs.org/blog/project-darkbloom-unlocking-idle-compute-for-ai/
There’s no way that’s possible
My reading of this is the TEE is used to validate the binary hasn’t been tampered with, but the TEE is definitely not powerful enough to do any inference.
The provider process is hardened against common local inspection paths, including debugger attachment and external memory inspection. The integrity of the running binary is also part of the trust model, helping ensure that the software serving requests matches what the network expects.
The system also uses hardware-backed attestation rooted in Apple’s security architecture. Secure Enclave keys, attestation signals, and recurring challenge-response checks are used to verify that a participating node is running with the expected protections and software state.
I haven’t fact-checked this, but the TEE could protect GPU memory in addition to main memory. Apple attests that the correct binary is running in the TEE, et voila.
Attesting is very different from protecting
The binary performing the inference isn’t running in the TEE, because the TEE runs on a weaker processor. The TEE is validating that the binary running on the main processors hasn’t been tampered with
That’s decent, but doesn’t match the statement that inference is happening in the TEE
I think the phrase “running in the TEE” is the idiomatic way to describe a program running on the main CPU, attested by the TEE processor, and protected/isolated from other processes on the system (modulo exploits, lol). I’m vaguely familiar with AMD’s SEV-SNP, mainly from reading security papers rather than any hands-on experience. I make no claims about Apple’s Secure Enclave, though.
I very much disagree with
think the phrase “running in the TEE” is the idiomatic way to describe a program running on the main CPU
In the Confidentiality/Integrity/Availability triangle, a secondary process running in another operating system gives you integrity, but not confidentiality
I really like the idea of home users renting out compute.
(But I don’t like that only being specialized scam-connected compute.)
If you like distributed compute that isn’t running a scam, you should check out https://boinc.berkeley.edu/
Also folding at home. I basically figure you’re donating your power consumption costs to science. I don’t know that there will ever be a way to make that profitable to the user renting their computer, considering power costs.
There are some crypto projects attached to boinc and f@h, like Gridcoin, that provide rewards for compute power. They are very much an added on layer that isn’t really worth much in real world dollars though.
I wish we could boinc. ( ͡° ͜ʖ ͡°)
Already
sounds good. keep it up
FAFO?
Why Mac instead of beefier hardware alternatives or better OSs?
Macs have good chips and unified memory. There is a reason everyone wants a Mac Mini or Mac Studio for OpenClaw. Also, for AI token generation, OS is irrelevant, but Mac is based on Unix.
