Hello internet users. I have tried gpt4all and like it, but it is very slow on my laptop. I was wondering if anyone here knows of any solutions I could run on my server (debian 12, amd cpu, intel a380 gpu) through a web interface. Has anyone found any good way to do this?
Ollama is a nice server base, they lots of projects that plug on top of that.
I’ve had pretty good luck running llamafile on my laptop. The speeds aren’t super fast, and I can only use the models that are Mistral 7B and smaller, but the results are good enough for casual use and general R and Python code.
Edit: my laptop doesn’t have a dedicated GPU, and I don’t think llamafile has support for Intel GPUs yet. CPU inference is still pretty quick.
Thanks to this post, and the other comments in here, I’ve discovered that the ultimate ui for ai-models may well be
https://github.com/ParisNeo/lollms-webui
and on HuggingFace ( that name is aweful: to me it is the creepy-horrible FaceHugger, from the movie Alien, that I saw so many decades ago ) TheBloke has some models which are smaller
https://huggingface.co/TheBloke/
so you can choose a model that will actually-work on your hardware.
I think Llama-2 for brainstorming & CodeLlama-instruct for learning programming examples seems to be the cleanest pair, from what I’ve read, and he’s got GGUF versions with different quantizations, so you can choose what will actually-fit on your hardware.
There are other models on huggingface which seem very useful, like
- whisper-large-v3 for speech-to-text,
- whisperspeech for text-to-speech,
- sdxl-turbo for image-making ( for some copyright-free subjects to practice drawing with ), and so-on…
Some models require GPU, not all.
Damn things moved fast!
Did you try LM studio?
Its proprietary
OP should try H2OGPT, it is somewhat technical but the UI makes it easy to configure. You can select many models and prompt types, and you can even input your own documents so that the AI uses them to answer
deleted by creator
