ninfer: 16GB VRAM to run Qwen 3.8 27B

[WhoCares]

https://github.com/Neroued/ninfer

NInfer is a from-scratch C++/CUDA inference engine for Qwen3.5 Dense and MoE architectures on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs. The runtime is deliberately specialized: one GPU, one resident model, and one to eight execution lanes fixed at startup.


https://github.com/ruwwww/ninfer-5060ti

High-performance single-GPU inference for selected model checkpoints and GPUs (rtx 5060ti 16gb).

https://github.com/roofkid/ninfer-4080

Qwen3.8-27B-GSQ3 on one RTX 4080: 100k context with rk4v4-e8, up to 2720 tok/s prefill and 262 tok/s generation