Question 1

How much VRAM does MediPhi need?

Accepted Answer

MediPhi (3.8B) requires approximately 8.540778350830077 GB of VRAM at full precision (FP16), 4.3 GB at 8-bit quantization (Q8), and 2.1 GB at 4-bit quantization (Q4). The actual VRAM usage may be slightly higher due to KV cache and runtime overhead.

Question 2

What GPUs can run MediPhi?

Accepted Answer

We have 10 compatible GPUs indexed for MediPhi, including K80, L4, A40, A100 40GB, A10G and 5 more. Compatibility is determined by whether the GPU's VRAM can fit the model's memory footprint, potentially using quantization to reduce memory usage.

Question 3

What is the cheapest GPU to run MediPhi?

Accepted Answer

The cheapest GPU to run MediPhi in the cloud is the K80 at $0.18/hr. Other affordable options include L4 ($0.20/hr) and A40 ($0.44/hr). Prices reflect the lowest on-demand rate across cloud providers.

Question 4

Can MediPhi run on consumer hardware?

Accepted Answer

MediPhi may be able to run on high-end consumer GPUs like the RTX 4090 (24 GB) with 4-bit quantization (Q4), which reduces VRAM usage to approximately 2.1 GB. However, performance may be limited compared to datacenter GPUs.

Question 5

What quantization options are available for MediPhi?

Accepted Answer

MediPhi supports the following quantization levels: FP16 (full precision): ~8.540778350830077 GB, Q8 (8-bit): ~4.3 GB, Q6 (6-bit): ~3.2 GB, Q4 (4-bit): ~2.1 GB. Lower quantization reduces VRAM usage at the cost of some quality loss. Q8 offers a good balance between quality and memory savings, while Q4 provides maximum compression.

GPU	VRAM	Best Quant	From	Rent
K80	12 GB	FP16	$0.18/hr
L4	24 GB	FP16	$0.20/hr
A40	48 GB	FP16	$0.44/hr	Rent
A100 40GB	40 GB	FP16	$0.93/hr	Rent
A10G	24 GB	FP16	$1.01/hr
A10	24 GB	FP16	$1.29/hr
A100 80GB PCIe	80 GB	FP16	$1.39/hr	Rent
A100 80GB	80 GB	FP16	$1.39/hr	Rent
H100 MEGA	80 GB	FP16	$1.80/hr	Rent
H100 SXM	80 GB	FP16	$1.80/hr	Rent

MediPhi

VRAM Requirements

Compatible GPUs

MediPhi FAQ

VRAM requirements for MediPhi at different quantization levels
Quantization	VRAM Required
FP16	8.5 GB
Q8	4.3 GB
Q6	3.2 GB
Q4	2.1 GB