Question 1

How much VRAM does wildguard need?

Accepted Answer

wildguard (7.2B) requires approximately 16.20061340332031 GB of VRAM at full precision (FP16), 8.1 GB at 8-bit quantization (Q8), and 4.1 GB at 4-bit quantization (Q4). The actual VRAM usage may be slightly higher due to KV cache and runtime overhead.

Question 2

What GPUs can run wildguard?

Accepted Answer

We have 10 compatible GPUs indexed for wildguard, including L4, A40, A100 40GB, A10G, A10 and 5 more. Compatibility is determined by whether the GPU's VRAM can fit the model's memory footprint, potentially using quantization to reduce memory usage.

Question 3

What is the cheapest GPU to run wildguard?

Accepted Answer

The cheapest GPU to run wildguard in the cloud is the L4 at $0.20/hr. Other affordable options include A40 ($0.44/hr) and A100 40GB ($0.93/hr). Prices reflect the lowest on-demand rate across cloud providers.

Question 4

Can wildguard run on consumer hardware?

Accepted Answer

wildguard may be able to run on high-end consumer GPUs like the RTX 4090 (24 GB) with 4-bit quantization (Q4), which reduces VRAM usage to approximately 4.1 GB. However, performance may be limited compared to datacenter GPUs.

Question 5

What quantization options are available for wildguard?

Accepted Answer

wildguard supports the following quantization levels: FP16 (full precision): ~16.20061340332031 GB, Q8 (8-bit): ~8.1 GB, Q6 (6-bit): ~6.1 GB, Q4 (4-bit): ~4.1 GB. Lower quantization reduces VRAM usage at the cost of some quality loss. Q8 offers a good balance between quality and memory savings, while Q4 provides maximum compression.

GPU	VRAM	Best Quant	From	Rent
L4	24 GB	FP16	$0.20/hr
A40	48 GB	FP16	$0.44/hr	Rent
A100 40GB	40 GB	FP16	$0.93/hr	Rent
A10G	24 GB	FP16	$1.01/hr
A10	24 GB	FP16	$1.29/hr
A100 80GB PCIe	80 GB	FP16	$1.39/hr	Rent
A100 80GB	80 GB	FP16	$1.39/hr	Rent
H100 MEGA	80 GB	FP16	$1.80/hr	Rent
H100 SXM	80 GB	FP16	$1.80/hr	Rent
H100 NVL	94 GB	FP16	$1.80/hr	Rent

wildguard

VRAM Requirements

Compatible GPUs

wildguard FAQ

VRAM requirements for wildguard at different quantization levels
Quantization	VRAM Required
FP16	16.2 GB
Q8	8.1 GB
Q6	6.1 GB
Q4	4.1 GB