Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
You must log in or # to comment.
Found vLLM to be the most efficient local runtime service. And “ray” as a good (but complicated) way to distribute the load: https://docs.ray.io/
I’ve fiddled around with a few models on ollama and opencode but more for the sake of seeing what I can run as ive yet to really find a use for it in my home usage.
Yup, ollama, various models. I initially downloaded it because I, along with thousands of other people, wanted to see what would happen if I made models debate with each other after RAGging them with various books (The Prince, The Art of War, The complete works of Shakespeare, etc.).
The results were uninteresting and I abandoned the project pretty quickly. I’ll sometimes use them for code analysis but they’re too slow on my rig to be really useful.
Did you use OWUIs native “call simultaneous models to answer” feature for that or one of the AI debate harnesses?
Nothing so fancy. I just made a little python script to prompt the first model, wait for a response, then prompt the next model with the initial prompt + the response, and so on. It was very hacky and slow.
Ah - I thought you might have used something like this
Oh neat. Yeah, if something like that had existed (and I’d been aware of it) I probably would have used it instead of building my own shoestring version.
wanted to see what would happen if I made models debate
LOL I kind of do that…sort of. I’ll ask several AI the very same question to see what they spit out.
You’ll like this then
Well I’ll be damned. Of course the law of large numbers dictates someone, somewhere has the same thought.
One of the projects I started and never got to a satisfactory end state was basically that, plus a judging round. Every model would respond to the same prompt, then every model would evaluate every other model’s response for accuracy and completeness. Then the results would get logged to a spreadsheet.
It’s simple enough, but for N models it requires N + N^2 model calls so it takes forever to run any decent dataset on consumer hardware. If I had the resources and a way to run it that didn’t fry the planet, I think it would be a cool running set of comparative benchmarks. IDK if it’d be useful at all but I’m still interested to see the data.
Every model would respond to the same prompt, then every model would evaluate every other model’s response for accuracy and completeness
If I understand correctly I sorta kinda do that. I’ll copy and paste one AI’s response into another and prompt something like 'Validate AI response: and paste it in. HAHA I thought I was being tricky but you’re already on it.
I think it’s tricky. It’s kind of like adding LLMs like vectors, and hopefully the effect can soften or at least reveal the shortcomings of individual models. Is it a good idea? I don’t know, I think there are good reasons to think it’s a waste of time and resources. I certainly think I’d need a better explanation of what use it would be before I spent more time building it. But I still think about what use it would be from time to time; I haven’t decided that it’s a bad idea yet.
at least reveal the shortcomings of individual models. Is it a good idea? I don’t know,
I mean I do it, in my rudimentary way, to check for some semblance of consistency. I’m unclear why you think that not a good idea?
P.S. This is a hypothesis, I haven’t even designed the test for it, much less run it. What follow are my suppositions.
I think whether or not it’s a good idea depends on how similar all the models are. I don’t have a rigorous definition of “similar” but things like similar training data, similar design methodologies, similar QA processes would all contribute. Theoretically (I think), if they’re all dissimilar, they should each catch errors the others miss. However, the more similar they are, the more likely they have the same biases and weak spots, and your error rate from a response + verification may be the same or even higher than the error rate for just the original prompt, and you’d be unlikely to detect those errors using just two similar models. It can instill false confidence in the results because you’re doing something that should in theory increase the validity of the data, but in practice might make no difference or even make the quality of responses worse.
Yes. My Actual Intelligence lives in my head, and runs mostly on coffee.
I’ll make sure to send you flowers, Algernon lol
@[email protected] this comment is not (directly) for you, I just want it in context.
Before you report someone for breaking rule 1, please look a the context. Specifically, the username someone may be replying to.
LOL.
https://en.wikipedia.org/wiki/Flowers_for_Algernon
Looks like someone got big mad over a harmless joke. You love to see it. Sorry they wasted your time.
Eh, its fine. Certainly better than the “I don’t like this so I’m going to report it” approach.
Just coffee?!? That’s cool.
Mine runs on:
- coffee
- spite
- tortilla chips
- & shame
Mostly on coffee, not exclusively. Noticable amounts of spite & tortilla chips are also present, yes, but… no shame.
Nice!
If that’s not already on a shirt it should be
deleted by creator
With sufficient coffee, mine shows considerable artifice.
Plastic flowers.
critical security bug: if coffee is taken away my head hurts :(
As we know AI stands for “An Indian”, so if you’re not from India, its actually impossible to self host.
Well, unless you manage to trap one in your basement, but that would violate human rights and hopefully also break the laws of your country.
You may be confusing Indians with gremlins. Which might explain ChatGPTs obsession with gremlins
Do you get many hallucinations?
Only when I’m deprived of coffee.
Would flowers work instead?
I tried but I only have 16g of ram and it wouldn’t complete a thought alas
Why would I?
I ran through lmstudio because it really eazy, I ran some kind of qwen 3.6 27b imatrix neo code DI, it is the best local model for coding I tried, I think it can be better than some cloud model
Yep! on my gaming Laptop I use Qwen 3.5 and Gemma 4 with an RTX 4080 Mobile, I’m also just able to run video generation on it to thanks to a quant of LTX 2.3. On this AMD AI laptop I am able to run a few LLMs via the NPU thanks to FastflowLM and the Lemonade server.
I use it for messing about pretty much. I haven’t found any serious uses for AI so it’s still a toy.
Myself - I’ve self hosted LLMs before, but with only 4-8GB vram (depending which card is in place), I can’t run the good stuff at acceptable enough speeds.
(Don’t @ me - I know all the tricks with turbo quants, spec decoding, MoE etc. 192GB/s is 192GB/s)
I do use Handy (STT) which is amazing (my fingers are arthritic and typing hurts after a while).
My personal use case for LLM is quite simple - a trumped up super google and / or self reflection / journalling / sound board. Despite being glib about it, that’s actually very useful to me.
Work wise, I use the big winking orange asshole (Claude) when I have to. I have moral tension with with it, so am seriously looking at other options. I hear good things about GLM 5.2, but if I can’t run Qwen 35B at any kind of decent speed, well…self hosted GLM is a pipe dream.
I’ve tried a few times but with only 8gig of vram it’s simply not worth it.
Have you tried qwen3.5-9b? It’s pretty solid for its size.
Yeah, it’s “good for its size” but it’s just too flaky for me to use for any significant coding.
Yeah, I wouldn’t use it for coding. It’s a bit dumb unfortunately.
How much CPU RAM do you have?
64G. But CPU inference is painfully slow.
Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read.
You just need to use the right framework, and the right model.
I’d suggest trying ik_llama.cpp and a MoE like this: https://huggingface.co/sokann/Qwen3.6-27B-GGUF-5.076bpw
And speculative decoding like DFlash or MTP (which you can also get specific models for).
I’ll check that out - speed isn’t my biggest issue so much as coding performance… The qwen 3.5 model I was using can write code, but it’s… Meh? Like sometimes it doesn’t even compile.
I did try tweaking llama.cpp to do some cpu offloading and it does seem to allow for much larger contexts at a modest performance loss. I’ll check out larger models.
Try qwen3.6-35b-a3b with a lightweight harness like pi.dev
Having it be able to run commands and try to compile or run the code and see the output helps especially on the “doesn’t compile” part of things
Yeah - I’ve been playing around more with the Qwen3-Coder-30B-A3B-Instruct MoE model and it’s still quite… Meh. I’ve been using llama.cpp and I’ve tried a bunch of tuning. It works and performs well enough (15t/s) but the output is just garbage. I can do some simple coding but I’m finding I’m fighting with it more than if I just wrote the code myself. Maybe I just have standards that are too high. Claude Opus 3.7 is just in an entirely different league…
When you run it, do you use unsloth’s recommended settings for coding?
https://unsloth.ai/docs/models/qwen3.6
Also have preserve thinking on, it helps it stay consistent in multi turn work.
Which model version you’re using can also affect results, usually unsloth’s ones are good.
With all that said, it’s of course a small model so it’s not a super coder. The 27b is better (I’d guess 25-35% better), but of course still a small model so…
So it’ll maybe not be good enough still, but should give it the chance to let it do the best it can :)
CPU offloading is too slow unless you use a hybrid MoE model, with the --n-cpu-moe parameter, specifically.
This only offloads “sparse” parts of the model to the CPU, which take up a lot of RAM but are very compute-lite to run. In practice, thats most of the size of modern MoE LLMs.
Since implementation of the
--fitparameter and its relatives, and--fit onbecoming the default, llama.cpp intelligently decides what to offload. For me, it made--n-cpu-moeobsolete.Mostly, yeah.
Sometimes it’s better to “cut it close,” with (for instance) a 27B model that’s nearly OOMing your VRAM fully offloaded, but you know will be fine in regular use without too many programs open.
In my case, with MiMo 2.5, it fills both my CPU and GPU RAM rather completely, so it’s best to set a static value so I don’t swap CPU RAM, and don’t OOM on the GPU either.
Bought b70 with egpu enclosure and usb4 connection wasn’t really planning to actually run anything but now ended up with llama.cpp with openwebui - kids/parents want to/have to use chat, might as well provide local solution than them using industry options. Also started with ollama and Gemma 4 26b a4b - asked it to write script to setup llama.cpp in container.
Yeah, I’ve heard the B70 is good bang for buck. My kids love using chat GPT to generate images and I’m aware that there are some really capable local models that can do that as well now - B70 should make short work of it.
That may be something for me to look at later on if I decide to keep self hosting.
OTOH, I’m also aware that I may end up building something that they don’t actually use. Been there, done that, and I don’t want to do it again.
Actually, on that topic, one interesting use case for me is my youngest one wants to have a YouTube channel.
So obviously, I’m not going to let her become a YouTuber, but what I’m thinking of doing is providing her my old phone (properly locked down) so that she can video record clips of what she wants.
Then - have those clips sent automatically to our jellyfin server so it appears like a channel. Code a fake YT plugin so that AI can do likes, positive comments etc.
It’s… work. I dunno…maybe a good enough AI can vibe code the entire project for me.
I started running LLMs a couple months ago on my own hardware. I have a Framework Desktop that I ordered last year and also recently picked up a refurbished 24GB AMD RX 7900 XTX which I’m doing some performance testing against. The dGPU is much better for dense models, and slightly faster for MoE if I’m willing to run them at a lower quant – but uses more power and has annoying coil whine. The Framework Desktop uses ~100W under load, is quieter, and for the MoE models already runs them fast enough for most of my needs – so most of my LLM use happens on that system still.
For software: I’m using ollama on the Framework currently, but I want to replace it with just using llama.cpp directly eventually. I’ve been using llama-cli for testing the dGPU. I wrote my own chat client to interact with ollama as well as a few other programs for specific tasks.
I’ve been using the LLMs for a mix of research (both personal and professional), entertainment, practical coding tasks (mostly debugging and brainstorming, plus a bit of UI prototyping, automatic generation of sequence diagrams for documentation, and light scripting), as well as automation of tedious tasks.
As an example of the latter, people often send me requests to prepare data sets by email but don’t specify the sources they want precisely so I have to go match the name against the real name in our archives; LLMs are great for mapping the imperfect name – with typos, missing prefixes, incorrect addition of spaces, addition/removal of hyphens, etc. – to the exact name I actually need to pull the data off disk when given a lookup table to compare against.
As far as models go, I’m mostly using various Qwen 3.6 and Gemma4 variants. I have multiple versions of each for different purposes. llmfan46’s uncensored Qwen 3.6 35B-A3B @ Q6_K (from Hugging Face) is my default model currently.
Ollama with gemma 4 for LLM stuff, coding brainstorming, etc.
Comfy ui with z-image or stable diffusion for images.
How is Gemma compared to others?
Technically, TTS/STT are mostly MLs; I’m pretty sure many people run these. I have a setup but I’m better with buttons that with spoken words, and I listen to ambient sounds or music. I think some day I’ll make voice assistant for talking to while driving, but that’s not a trivial task hardware-wise, even if I used cloud LLM layer, which I won’t. Putting AI on baremetal sounds like an interesting project.
I have a homemade “local agent” that can actually “code” somewhat, I use it just to figure out how this thing works on the inside practically. Mostly useless otherwise (also I have GPU that’s older than AI, so it’s kind of fun technical task to run this stuff on pure RAM+swap). Feels like the whole hype is greatly overrated, but I appreciate a chance to learn something new anyway.
My server is way to weak for that unfortunately. I run some llms on my laptop with ollama but it’s not particularly effective. I use it to run dolphin series models when k need an uncensored LLM I have tried running some of the coding models but they just aren’t smart enough on my level of compute for any useful work so Ive ended up just paying api prices on open routers.
