As far as energy goes, its a matter of degree. LLMs are mainly bad emissions-wise because of the volume of calls being made. If you’re running it on your GPU, you could have been playing a game or something similarly emitting.
A moderately sized text-to-text model that you would run locally is about 10g of carbon for 1000 inferences which is driving a car about 1/40th of a mile. Even assuming your model is running in some kind of agentic loop, maybe 5 inferences / actual response (though it could be dozens depending on the architecture) that gets to you, that’s 10gcarbon / 200 messages to your model which is at least 2-3 sessions on the heavy end I would think. You could use it for a year and its equivalent to driving 3 miles if you do that every day.
Image generation, however, is 1000-1500x that so just chatting with your GF isn’t that bad. Generating images is where it really adds up.
I wouldn’t trust these numbers exactly, they’re more ball-park. There’s optimizations that they don’t include and there’s a million other variables that could make it more expensive. I doubt it would be more than 10-20 miles in a car / year for really heavy usage though.
I mean, you can run an LLM locally, its not that hard.
And you can run such a local machine off of solar power, if you have an energy efficient setup.
It is possible to use this tech in a way that is not horrendously evil, and instead merely somewhat questionable, lol.
Hell, I guess you could arguably literally warm a room of your home with your conversations.
As far as energy goes, its a matter of degree. LLMs are mainly bad emissions-wise because of the volume of calls being made. If you’re running it on your GPU, you could have been playing a game or something similarly emitting.
The issue is more image generation models which are 1000 times worse https://www.technologyreview.com/2023/12/01/1084189/making-an-image-with-generative-ai-uses-as-much-energy-as-charging-your-phone/
Original Paper: https://arxiv.org/pdf/2311.16863
A moderately sized text-to-text model that you would run locally is about 10g of carbon for 1000 inferences which is driving a car about 1/40th of a mile. Even assuming your model is running in some kind of agentic loop, maybe 5 inferences / actual response (though it could be dozens depending on the architecture) that gets to you, that’s 10gcarbon / 200 messages to your model which is at least 2-3 sessions on the heavy end I would think. You could use it for a year and its equivalent to driving 3 miles if you do that every day.
Image generation, however, is 1000-1500x that so just chatting with your GF isn’t that bad. Generating images is where it really adds up.
I wouldn’t trust these numbers exactly, they’re more ball-park. There’s optimizations that they don’t include and there’s a million other variables that could make it more expensive. I doubt it would be more than 10-20 miles in a car / year for really heavy usage though.