LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

Open benchmark

LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

Zishan Shao^1,*, Lixun Zhang^1,*, Kangning Cui^2,*, Wenhao Wu^1,*, Jinhee Kim¹, Yixiao Wang¹, Ting Jiang¹, Hancheng Ye¹, Qinsi Wang¹, Fan Yang², Danyang Zhuo¹, Yiran Chen¹, Hai Li¹

¹Duke University ²Wake Forest University ^*Equal contribution

Paper Code Checkpoints & Results Reproduce

8model variants

9method tracks

329checkpoint artifacts

5 TiB+hosted artifacts

Why LowRankArena

From fragmented claims to matched evidence.

LowRankArena overview connecting four problems in prior SVD evaluation to a standardized comparison framework and four main takeaways — LowRankArena aligns comparison regimes, broadens tasks and models, measures end-to-end inference, and releases an open checkpoint zoo.

Standardized results

Main quality leaderboard

Compare within the shared LowRankArena protocol. Perplexity is lower-is-better; accuracy and dense-normalized quality retention are higher-is-better.

Dense reference Best compressed result

Backbone

Keep ratio

Columns

MCQ Avg. is the unweighted mean over BoolQ, ARC-Easy, ARC-Challenge, WinoGrande, PIQA, HellaSwag, and OpenBookQA. Q. Ret. averages dense-normalized retention on WT2 PPL, C4 PPL, MCQ Avg., MathQA, and MMLU-Math.

Matched compression families

Structured pruning vs. low-rank compression

Llama-3.1-8B under matched keep ratios without expensive retraining. Accuracy can remain competitive while perplexity reveals a sharper language-modeling capability cliff.

Metric

Best compressed or pruned result at each keep ratio is highlighted.

End-to-end inference

Inference gains depend on the workload phase.

A100 inference results for Llama-3.1-8B at 80% keep in BF16. Each SVD row uses the same vLLM request profile and scheduler settings as the dense reference.

TTFT and E2E report p50 milliseconds. Throughput is tokens per second. †Pruning rows come from a separate recovered-artifact, batch-size-1 feasibility audit and are not directly comparable with the dense/SVD full-sweep rows.

Reproduce

Start with a registered checkpoint, then run one shared path.

The repository exposes separate accuracy, active-memory, and vLLM inference entrypoints. Normalized outputs share one project-owned top-level schema.

Checkpoint registry ↗ Result schema ↗ Audit reproductions ↗

quick start

conda env create -f environment.yml
conda activate lowrankarena

# Accuracy
python scripts/run_eval.py \
  llama31-8b-svdllm-v1-update-0.6 \
  --suite accuracy/mcq

# Active GPU memory
python scripts/run_memory.py \
  llama31-8b-svdllm-v1-update-0.6

# vLLM speed
python scripts/run_speed.py \
  llama31-8b-svdllm-v1-update-0.6

Citation

Cite LowRankArena

@misc{shao2026lowrankarenastandardizedevaluationplatform,
  title         = {LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression},
  author        = {Zishan Shao and Lixun Zhang and Kangning Cui and Wenhao Wu and Jinhee Kim and Yixiao Wang and Ting Jiang and Hancheng Ye and Qinsi Wang and Fan Yang and Danyang Zhuo and Yiran Chen and Hai Li},
  year          = {2026},
  eprint        = {2608.26389},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.26389}
}