I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?

  • Riskable@programming.devEnglish
    61·
    3 天前

    You can get the TernaryQuench version of Qwen3.8:27b to run great on a GPU with just 16GB of VRAM. I started using it last week and it’s been amazing.

    I have a 4060 Ti 16GB and was using the 14b version for a long while but that ternary conversion tech is just enough to squeeze the full 27b version into my card with plenty of room for context 👍

    • Nighed@feddit.ukEnglish
      21·
      3 天前

      Is that by penkia on hugging face? (How safe is running random models?)