I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?

  • Mercer@lemmy.zip
    2·
    1 天前

    I use tinfoil.sh with locally hosted models, you are giving tinfoil money for the service but they aren’t paying the model trainers for open source models nor are they training models with your data

  • heartSagan5@lemmy.zip
    72·
    2 天前

    The LLMs would be of minor use without the ingestion of the Internet to update its weights. It’s a clanker so it has no ideas; you’ve got to feed it whatever you’re purpose is with it.

  • mesa@piefed.socialEnglish
    51·
    2 天前

    I donate my compute to https://aihorde.net/ once in a while. got to play around with some cool AI bullshit there. Its fun to play around with.

    Its all volunteer run. I got my steam deck pushing small updates to peoples like dnd rollplay kinda stuff. koboldcpp makes it easy to set up and you can use hugging face models or your own.

    Theres some really excellent open models out there if you care to look! Most are hyper specialized better than say GPT and such.

  • OscarRobin@lemmy.world
    81·
    2 天前

    Use open weight models either small enough to run yourself or via your diverse online host of choice.

      • traxex@lemmy.dbzer0.com
        43·
        2 天前

        I believe there are options to host open models on hardware you don’t own without having to give money to the “evil” data centers. Getting away from data centers entirely is impossible as that is how the internet has worked for a very long time now.

  • assaultpotato@sh.itjust.works
    594·
    3 天前

    Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.

    If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…

      • bumblebeer@lemmy.world
        4·
        1 天前

        ollama

        Shittiest local inference engine. Use llama.cpp instead.

        qwen 3:4b-instruct

        Old version 3 model from Alibaba. 4 billion parameters total. It will struggle with anything other than natural language generation.

        on 6vCPU and no GPU

        Running on a 6 vcore VM with no GPU pass through.

        TTFT

        Time.To.First.Token. How long it takes the model to start responding after you hit enter. Dependent on not only processing speed but also context length.

        prompt is uncached

        Cold start = no cache = slow. But faster when mid conversation and the key-value cache is hot.

        100tps

        100 tokens per second. 1 token is roughly 1/4 word.

      • assaultpotato@sh.itjust.works
        2·
        2 天前

        So… I’m with you/them about the vendor lock in but I think they’re slightly underestimating what ollama brings to the table. For example, ollama bundles CUDA libraries so I don’t need to worry about that as I wrangle with my GPU. It’s also well supported by other ecosystems like home assistant in a way that raw openai compatible API server isn’t.

        Overall I don’t know if I find these arguments compelling at this time.

  • HogGirl@sh.itjust.worksEnglish
    5528·
    3 天前

    You can, but it’s not worth the effort. The technology itself is set up for people of power, such as governments and corporations. That’s why they steal our hardware and make it unaffordable. The only genuine use is accessibility, but it’s purely audio/text recognition. Everything else is for targeting people with drones and surveillance state.

    Edit: how could I forget deepfakes? You practically can’t live without them nowadays /s

  • Lovable Sidekick@lemmy.worldEnglish
    144·
    3 天前

    Srsly, unless you grow all your food and make all your possessions, you’re constantly supporting billionaires. I mean sure, run your own LLM, but don’t let it create a delusion of ethical purity.

    • boonhet@sopuli.xyz
      3·
      2 天前

      There are 3 or 4 billionaires in my country. Founders of Wise and Bolt. None of them own farms or grocery stores.

      Matter of fact, the most popular grocery store chain is coop so most of the “owners” (members) of your local cooperative aren’t even millionaires.

  • keimevo@lemmy.world
    221·
    3 天前

    I run a “server” (old 9-year gamer laptop) with an 8 GB Geforce 1070 Max-Q. It’s currently running full GPU-offloaded stock llama.cpp, with Unsloth’s Qwen3.5 9B at UD Q4_K_XL (good enough quantization for coding assistance and short agentic tasks) and a 90k context at around 300 t/s prefill and 25 t/s decode, which are good enough speeds for daily use.

    It can’t vibecode GTA6, but it can help with quick scripting, research and things like that.

  • anus@lemmy.world
    72·
    2 天前

    The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier

    Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age

    What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai

    Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware

    • boonhet@sopuli.xyz
      6·
      2 天前

      trying to code

      copy pasting to and from duck

      That’s, uh… The worst way to do any of this. And GPT-6 Luna is the weakest GPT-6, if you’re going to run something that’s on someone else’s hardware anyway, you can just use openrouter and choose ZDR hosters of open-weight models that are somewhere between GPT-6.1 Sol and Terra in accuracy.

      • anus@lemmy.world
        2·
        2 天前

        OP said they didn’t want to support billionaires and data centers

        The money used to make zdr requests on openrouter to good models like GLM 5.3 flows into large inference providers like fireworks and baseten

        Yes, copy pasting is a terrible way to vibe code. But my point is it’s still going to be better than trying to ollama with a 1060 and 16 GB RAM

  • FlexibleToast@lemmy.worldEnglish
    152·
    3 天前

    You can run very small models with Ollama or Llama.cpp. It really depends on the type of work you want to do, but these small models won’t do anything super complex. In my opinion the models don’t start getting really good until around the 30GB size like Qwen 3.8 27b. You could probably still use a small model for some chat, search, rag, maybe simple script writing.

    • Riskable@programming.devEnglish
      61·
      3 天前

      You can get the TernaryQuench version of Qwen3.8:27b to run great on a GPU with just 16GB of VRAM. I started using it last week and it’s been amazing.

      I have a 4060 Ti 16GB and was using the 14b version for a long while but that ternary conversion tech is just enough to squeeze the full 27b version into my card with plenty of room for context 👍

      • Nighed@feddit.ukEnglish
        21·
        3 天前

        Is that by penkia on hugging face? (How safe is running random models?)

    • Rhaedas@fedia.io
      21·
      3 天前

      Note that you would need to use Llama.cpp for the Mix of Expert type models like the Qwen mentioned, Ollama doesn’t have the flexibility of commands.

      • FlexibleToast@lemmy.worldEnglish
        1·
        3 天前

        Ollama can run MoE. It just decides for you where to place things. Also the Qwen model mentioned is a sense model.

        • Rhaedas@fedia.io
          11·
          3 天前

          That wasn’t my experience, but maybe I just didn’t run across the right instructions, or it was the particular model. Llama lets you adjust parameters far more. Two sides of the same coin, really.

          • FlexibleToast@lemmy.worldEnglish
            11·
            3 天前

            Ollama is just llamacpp under the hood. Ollama is just an opinionated version that automatically sets things up that will work for most people. Llamacpp definitely has more of those “nerd knobs” that you can configure for better performance. The flipside is that it is far more complicated. Granted using another LLM to help you makes it not so bad.

  • inspxtr@lemmy.world
    3·
    2 天前

    I don’t think I’ve seen this recommended yet, but maybe you can check out CoCore https://www.cocore.dev/. I saw a demo of this some time ago and it’s still pretty much an experimental project. Essentially this is more of a community shared way to run local LLMs, especially for those without necessary compute. You are basically using an LLM hosted on another person’s computer in an encrypted manner. There’s a lot of technical detail about it that goes over my head. It is slow, and of course, still requires trust, or some auditing to verify it. But I think it’s kinda a cool way to crowdsource resource to run. The network of users is on fediverse btw.