Kagamino's Blog

Qwen 3.8 27B

This week was interesting for local models with the open-weights release of Qwen 3.8 27B. It was highly anticipated, because Qwen 3.6 held the crown for a long time in open models of a size that is runnable on almost consumer hardware.

The Artificial Analysis score is pretty damn good! 52 with xhigh is in the ballpark of GPT-5.6 Luna (max) and DeepSeek V4 Flash 0731, which were my go to models for august, given their relatively low price. To think that I could run as powerful a model locally feels insane. For reference, it sits between a Claude Opus 4.6 and a Claude Opus 4.7, and that was the best available in February in Claude Code.

Artificial Analysis pareto frontier

Artificial Analysis is my reference for model comparison. It actually compiles scores from many benchmarks. I know there are many pitfalls to these benchmarks, but hey, I have to reduce the world’s complexity, right? I found my preferred charts when removing all models. Then AAI highlights the pareto frontier, with the best option at each tradeoff.

Int vs Cost/Time per task - What frontier model to use in what case. Here Luna looks pretty neat for cost, and that’s what I used in august.

Int vs Active/Total params - This helps me understand what I could run on my hardware. For this one, filtering on small (and maybe tiny) models helps. This shows Qwen 3.8 27B as much better than the previous Gemma 4 26B A4B. And I can see a new contender Ling 3.0 Tiny that looks promising.

Running modern models

I wonder if local inference got better since my previous attempt. Back then, I had decent inference performance with Gemma 4 E2B (AAI 15), but it felt too dumb to do meaningful work. I wanted to test Qwen 3.5 9B, but this time it was too slow, on my Desktop as well as on my Mac.

As I ran out of my OpenCode Go subscription for the last ten days of august, it’s time to check if I can work locally! Maybe there were new smaller and smarter models, like Qwen or Ling ? Or better inference, like oMLX on Mac?

Last time was a one shot scenario, so this time I want to test a coding agentic workflow with Pi - it has a smaller system prompt. I go by feel, because I don’t have clear metrics to target.

Sadly, Qwen 3.8 27B is too large, even with oMLX. That’s also because I need a larger context of 100k tokens. And I couldn’t run Ling 3.0 Tiny, because of unsupported architecture on both oMLX and LMStudio. So much for modern models!

New contenders

But the AAI pareto frontier gives me new ideas of old models to run. Qwen 3.5 4B is still on this frontier with a AAI 20, and a smaller size than the Gemma 4 E2B (AAI 10) I tested last time.

With oMLX and an 8-bit version, it runs using around 10GB unified memory. The first prompt is always a bit slow, but then the KV cache kicks in and it feels snappy enough. But after some time it ended up in a locked grep loop. I think this is usable for targeted changes.

I had an attempt on my desktop, which is better a prefill (input) than the Mac. But this time it didn’t feel really smart.

In the end, I’m a bit disappointed I couldn’t run the newer models, even though Ling 3.0 Tiny was supposed to fit. But maybe I can run it later when the tooling has caught up. I think Qwen 3.5 4B may have a spot for offline use, as I can run it at 50 tok/s on Mac. I’d like to investigate using it as an autocomplete model for IDE.