Question 1

How much VRAM does starcoder need?

Accepted Answer

starcoder (15.8B) requires approximately 35.35921783447265 GB of VRAM at full precision (FP16), 17.7 GB at 8-bit quantization (Q8), and 8.8 GB at 4-bit quantization (Q4). The actual VRAM usage may be slightly higher due to KV cache and runtime overhead.

Question 2

What GPUs can run starcoder?

Accepted Answer

We have 10 compatible GPUs indexed for starcoder, including A40, L40S, A100 40GB, L40, A100 80GB PCIe and 5 more. Compatibility is determined by whether the GPU's VRAM can fit the model's memory footprint, potentially using quantization to reduce memory usage.

Question 3

What is the cheapest GPU to run starcoder?

Accepted Answer

The cheapest GPU to run starcoder in the cloud is the A40 at $0.44/hr. Other affordable options include L40S ($0.60/hr) and A100 40GB ($0.93/hr). Prices reflect the lowest on-demand rate across cloud providers.

Question 4

Can starcoder run on consumer hardware?

Accepted Answer

starcoder may be able to run on high-end consumer GPUs like the RTX 4090 (24 GB) with 4-bit quantization (Q4), which reduces VRAM usage to approximately 8.8 GB. However, performance may be limited compared to datacenter GPUs.

Question 5

What quantization options are available for starcoder?

Accepted Answer

starcoder supports the following quantization levels: FP16 (full precision): ~35.35921783447265 GB, Q8 (8-bit): ~17.7 GB, Q6 (6-bit): ~13.3 GB, Q4 (4-bit): ~8.8 GB. Lower quantization reduces VRAM usage at the cost of some quality loss. Q8 offers a good balance between quality and memory savings, while Q4 provides maximum compression.

GPU	VRAM	Best Quant	From	Rent
A40	48 GB	FP16	$0.44/hr	Rent
L40S	48 GB	FP16	$0.60/hr	Rent
A100 40GB	40 GB	Q8	$0.93/hr	Rent
L40	48 GB	FP16	$0.99/hr	Rent
A100 80GB PCIe	80 GB	FP16	$1.39/hr	Rent
A100 80GB	80 GB	FP16	$1.39/hr	Rent
H100 MEGA	80 GB	FP16	$1.80/hr	Rent
H100 SXM	80 GB	FP16	$1.80/hr	Rent
H100 NVL	94 GB	FP16	$1.80/hr	Rent
H100 PCIe	80 GB	FP16	$1.80/hr	Rent

starcoder

VRAM Requirements

Compatible GPUs

starcoder FAQ

VRAM requirements for starcoder at different quantization levels
Quantization	VRAM Required
FP16	35.4 GB
Q8	17.7 GB
Q6	13.3 GB
Q4	8.8 GB