Start with total weight memory, then budget for the runtime, context and concurrent users. The candidates below are versioned examples for testing; the fit labels describe estimated weight headroom, not measured throughput or a benchmark winner.
Fit matrix| GPU | VRAM | Candidate | Precision | Weight headroom | Evidence |
|---|---|---|---|---|---|
| RTX 3060 12GB | 12 GB | Qwen3-8B (versioned baseline) | Q5 where supported; context and batch must be measured | tight estimate | Model card ↗ Weight-only estimate: 5.1 GB at 5 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| RTX 4060 Ti 16GB | 16 GB | Qwen3-14B (versioned baseline) | Q4 where supported; context and batch must be measured | tight estimate | Model card ↗ Weight-only estimate: 7.4 GB at 4 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| RTX 5080 16GB | 16 GB | Qwen3-14B (versioned baseline) | Q4 where supported; context and batch must be measured | tight estimate | Model card ↗ Weight-only estimate: 7.4 GB at 4 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| RTX 3090 24GB | 24 GB | Qwen3.6-35B-A3B (evaluation candidate) | Q4 where supported; context and batch must be measured | tight estimate | Model card ↗ Weight-only estimate: 17.5 GB at 4 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| RTX 4090 24GB | 24 GB | Qwen3.6-35B-A3B (evaluation candidate) | Q4 where supported; context and batch must be measured | tight estimate | Model card ↗ Weight-only estimate: 17.5 GB at 4 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| RTX 5090 32GB | 32 GB | Qwen3.6-35B-A3B (evaluation candidate) | Q6 where supported; context and batch must be measured | tight estimate | Model card ↗ Weight-only estimate: 26.3 GB at 6 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| A100 40GB | 40 GB | Qwen3.6-35B-A3B (evaluation candidate) | Q6 where supported; context and batch must be measured | comfortable estimate | Model card ↗ Weight-only estimate: 26.3 GB at 6 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| A100 80GB | 80 GB | Qwen3.6-35B-A3B (evaluation candidate) | BF16; context and batch must be measured | comfortable estimate | Model card ↗ Weight-only estimate: 70.0 GB at 16 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| H100 80GB | 80 GB | Qwen3.6-35B-A3B (evaluation candidate) | BF16; context and batch must be measured | comfortable estimate | Model card ↗ Weight-only estimate: 70.0 GB at 16 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| H200 141GB | 141 GB | Qwen3.6-35B-A3B (evaluation candidate) | BF16; context and batch must be measured | comfortable estimate | Model card ↗ Weight-only estimate: 70.0 GB at 16 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| B200 | 192 GB | Qwen3.6-35B-A3B (evaluation candidate) | BF16; context and batch must be measured | comfortable estimate | Model card ↗ Weight-only estimate: 70.0 GB at 16 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
| MI300X 192GB | 192 GB | Qwen3.6-35B-A3B (evaluation candidate) | BF16; context and batch must be measured | comfortable estimate | Model card ↗ Weight-only estimate: 70.0 GB at 16 bits, before scales, unquantized tensors, vision components, KV cache and runtime buffers. |
35 billion parameters at two bytes each need about 70 GB before any overhead. Quantization, sharding or offload is required on a 40 GB A100. Check backend support rather than assuming every GPU accelerates every precision.
Kimi K2.6 has 1T total parameters and 32B active. Raw 4-bit weights alone are about 500 GB; FP8 about 1 TB. It cannot reside entirely on one 141 or 192 GB GPU at those precisions.
Kimi model card ↗Measure peak allocated memory at the intended prompt length, output length and concurrency. Reserve headroom for vision inputs, KV cache, quantization metadata and runtime buffers. Published maximum context is not a promise that it fits your card.
The previous composite-quality and relative-throughput bubbles were withdrawn because they were not measured results.
Notes and comments in this board are stored in this browser. They are not submitted to a public moderation queue.
RTX 3060 12GB - Q5 where supported; context and batch must be measured - tight fit
Edits are saved in this browser. Use comments below to send corrections for moderation.
Active MoE parameters describe compute, not resident weights. Kimi K2.6 has 1T total parameters: raw 4-bit weights are about 500 GB and FP8 about 1 TB, exceeding a single 141 or 192 GB device.
No local comments yet for this GPU.