ninfer: 16GB VRAM to run Qwen 3.8 27B
 

https://github.com/Neroued/ninfer

NInfer is a from-scratch C++/CUDA inference engine for Qwen3.5 Dense and MoE architectures on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs. The runtime is deliberately specialized: one GPU, one resident model, and one to eight execution lanes fixed at startup.


https://github.com/ruwwww/ninfer-5060ti

High-performance single-GPU inference for selected model checkpoints and GPUs (rtx 5060ti 16gb).

https://github.com/roofkid/ninfer-4080

Qwen3.8-27B-GSQ3 on one RTX 4080: 100k context with rk4v4-e8, up to 2720 tok/s prefill and 262 tok/s generation

It seems today China is putting export controls on many models and also 2T or more open models cannot be released without authorization. Of course 2T isnt effecting normal rigs but I dont like this direction. I was hoping to see open weight models hitting the current frontier soon.