ninfer: 16GB VRAM to run Qwen 3.8 27B

[WhoCares]

https://github.com/Neroued/ninfer

NInfer is a from-scratch C++/CUDA inference engine for Qwen3.5 Dense and MoE architectures on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs. The runtime is deliberately specialized: one GPU, one resident model, and one to eight execution lanes fixed at startup.


https://github.com/ruwwww/ninfer-5060ti

High-performance single-GPU inference for selected model checkpoints and GPUs (rtx 5060ti 16gb).

https://github.com/roofkid/ninfer-4080

Qwen3.8-27B-GSQ3 on one RTX 4080: 100k context with rk4v4-e8, up to 2720 tok/s prefill and 262 tok/s generation

[chants]

It seems today China is putting export controls on many models and also 2T or more open models cannot be released without authorization. Of course 2T isnt effecting normal rigs but I dont like this direction. I was hoping to see open weight models hitting the current frontier soon.

https://www.reuters.com/world/beijing-is-looking-curbing-overseas-access-chinas-top-ai-models-sources-say-2026-07-07/