|
ninfer: 16GB VRAM to run Qwen 3.8 27B
https://github.com/Neroued/ninfer
NInfer is a from-scratch C++/CUDA inference engine for Qwen3.5 Dense and MoE architectures on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs. The runtime is deliberately specialized: one GPU, one resident model, and one to eight execution lanes fixed at startup.
https://github.com/ruwwww/ninfer-5060ti
High-performance single-GPU inference for selected model checkpoints and GPUs (rtx 5060ti 16gb).
https://github.com/roofkid/ninfer-4080
Qwen3.8-27B-GSQ3 on one RTX 4080: 100k context with rk4v4-e8, up to 2720 tok/s prefill and 262 tok/s generation
__________________
AKA Solomon/blowfish.
|