Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Inkling GPU requirements: VRAM and cheapest GPU

Inkling has 952.4B parameters, published in BF16. It is a mixture-of-experts model: 6 of its 256 experts work on each token, about 18.1B parameters, so it generates at roughly the speed of a model that size. Every expert still has to sit in GPU memory, which is why the memory figures below are for the full model. It is released under apache-2.0, which allows commercial use.

Parameters
952.4B
Smallest footprint (INT4)
about 553 GB
Cheapest GPU per hour (FP16)
–
Type
Vision
  • Parameters 952.4B
  • Active per token 18.1B
  • Type Vision
  • Task image-text-to-text
  • Licence apache-2.0
  • Downloads 1.3M
  • Likes 1.9K

Calculating for Inkling

Precision
Workload
More settings
Serving engine
Requests at once

Inkling needs about 2,099 GB of GPU memory

Inference (vLLM), FP16 / BF16, 8K tokens context, one request at a time.

Where the memory goes

  • Weights 1,905 GB
  • KV cache 2.2 GB
  • Runtime overhead 192 GB
  • Total 2,099 GB

GPU counts keep 10% of each card free for spikes. Plan extra if you expect long prompts or many users at once.

No tracked offer fits this setup on one node. Try a lower precision or a shorter context.

No tracked offer fits this setup on one node. Try a lower precision or a shorter context.

Prices collected Nov 16, 2026. Every cost is approximate: on-demand, US dollars, mostly US regions (marketplaces such as Vast.ai and Lium are worldwide).

Memory by precision and workload

With a 8K-token context and one request at a time. LoRA and full fine-tuning start from 16-bit weights and QLoRA from 4-bit, so the other cells are blank.
PrecisionInferenceLoRA fine-tuneQLoRA fine-tuneFull fine-tune
FP16 / BF162,099 GB2,441 GB–16,772 GB
INT81,103 GB–––
INT4553 GB–895 GB–

Cheapest GPU by precision

Inference with an 8K context and one request at a time.

FP16 / BF16

about 2,099 GB · 8K tokens

The weights as published. Nothing lost, but the largest footprint.

No single-node offer fits

INT8

about 1,103 GB · 8K tokens

Weights at half the size. Most chat and RAG work shows no visible difference.

7 × MI300X 192GB

RunPod (opens in a new tab) · 8x MI300X

sold as 8 GPUs; you need 7
Per GPU / hr
$2.39
GPUs needed
7
Total / hr
about $19.12
per month, 24/7, approx.
$13,958

INT4

about 553 GB · 8K tokens

Weights at a quarter of the size. Small drops show up first in maths and code.

4 × MI300X 192GB

RunPod (opens in a new tab) · 4x MI300X

Per GPU / hr
$2.39
GPUs needed
4
Total / hr
about $9.56
per month, 24/7, approx.
$6,979

Inference vs fine-tuning

  • Inference (FP16)

    1×

    of the memory inference needs

    2,099 GB

    more than one 8-GPU node

  • LoRA fine-tune

    1.2×

    of the memory inference needs

    2,441 GB

    more than one 8-GPU node

  • QLoRA fine-tune

    0.4×

    of the memory inference needs

    895 GB

    6 × MI300X 192GB · about $19.12

  • Full fine-tune

    8×

    of the memory inference needs

    16,772 GB

    more than one 8-GPU node

Serving it at FP16 fits on more than one 8-GPU node. LoRA needs more than one 8-GPU node, QLoRA 6 × MI300X 192GB (RunPod) and a full fine-tune more than one 8-GPU node. Training keeps activations and optimiser state in memory as well as the weights, which is why it so often needs more GPUs than serving the same model.

Will it run on a Mac or a single RTX 4090?

RTX 4090 (24 GB)

Not on its own. Even at INT4 it needs about 553 GB, more than the card’s 24 GB. Two or more cards, or a bigger data-centre GPU, will do it.

Mac with Apple silicon

Not realistically. Even at INT4 it needs about 553 GB, more than the largest Mac can give its GPU.

Similar models

Open models of a similar size.

Calculate another model

Inkling: common questions

How much VRAM does Inkling need?

About 2,099 GB at FP16, 1,103 GB at INT8 and 553 GB at INT4, with an 8K context and one request at a time. Longer contexts and more parallel requests add KV cache on top.

What is the cheapest GPU to run Inkling?

7 × MI300X 192GB (about $2.39 per GPU / hour; sold as 8 GPUs; you need 7) on RunPod, at about $19.12 an hour for INT8 inference with an 8K context. Prices collected Nov 16, 2026; they change often.

Can I fine-tune Inkling on a single GPU?

Not on one GPU. Even QLoRA needs about 895 GB, more than an 80 GB GPU holds. LoRA needs about 2,441 GB and a full fine-tune about 16,772 GB.

How are these numbers calculated?

Weights are parameters times bytes per parameter (2 at FP16, 1 at INT8, 0.5 at INT4). The KV cache comes from the model’s layers, KV heads, head size and context length. We add about 10% plus 1 GB for the runtime and fill each GPU to 90%. Prices are on-demand list prices in US regions, collected from each provider’s public pricing.

Can Inkling run on a consumer GPU like the RTX 4090?

Not on its own. Even at INT4 it needs about 553 GB, more than the card’s 24 GB. Two or more cards, or a bigger data-centre GPU, will do it.

Does Inkling fit on a single H100 80GB?

No. Even at INT4 it needs about 553 GB, so you need several GPUs.

How many GPUs do I need for Inkling?

At FP16 with an 8K context: 30 × 80 GB, 49 × 48 GB or 98 × 24 GB GPUs. At INT4 you need about a quarter of the memory for the weights.

How much VRAM does Inkling need at INT8?

About 1,103 GB: 1,000 GB for the weights, 2.2 GB for an 8K KV cache and 101 GB of runtime overhead.

How much does it cost to run Inkling 24/7 for a month?

About $13,958 on the cheapest setup we found (7 × MI300X 192GB (about $2.39 per GPU / hour; sold as 8 GPUs; you need 7), $19.12 an hour for 730 hours). The cheapest on AWS, Google Cloud or Azure is about $83,171 a month.

FP16 vs INT8 vs INT4: which should I use?

FP16 keeps full quality and needs the most memory (about 2,099 GB here). INT8 halves the weights with little loss for most chat work. INT4 (AWQ, GPTQ or GGUF Q4) cuts them to a quarter, about 553 GB in total, with a small drop in quality that shows most in maths and code.

Can Inkling run on a Mac?

Not realistically. Even at INT4 it needs about 553 GB, more than the largest Mac can give its GPU.

vLLM or llama.cpp?

vLLM is built for serving many requests at once on NVIDIA and AMD GPUs and reserves most of the GPU memory for KV cache. llama.cpp runs GGUF files on almost anything, including Macs and CPUs, and suits one user or a small team. Memory needs are similar for the same precision.

Prices collected Nov 16, 2026. Every cost is approximate: on-demand, US dollars, mostly US regions (marketplaces such as Vast.ai and Lium are worldwide).

How we calculate

  • Weights. Parameters times bytes per parameter: 2 at FP16, 1 at INT8, 0.5 at INT4, plus about 5% for the quantisation scales at INT8 and INT4.
  • KV cache. 2 × layers × KV heads × head size × context × requests × 2 bytes, from the model’s config.json. Where a setting is missing we estimate it and say so.
  • Overhead. About 10% plus 1 GB for the runtime, in line with what vLLM reserves.
  • Fine-tuning. LoRA keeps 16-bit weights and adds a small adapter, its optimiser and activations. QLoRA does the same over 4-bit weights. A full fine-tune needs about 16 bytes per parameter with mixed-precision Adam (weights, gradients and optimiser states), plus activations.
  • GPUs. We fill each GPU to 90% of its memory and stop at 8 per node. Costs are on-demand list prices in US regions; marketplace clouds show the median offer.