Output-based saliency
Estimate the first-order effect of removing each weight, then aggregate the scores within a structural unit.
A name inspired by the band.Celebrating Nirvana ↘
COLM 2026LLM compression
NIRVANAStructured Pruning Reimagined
for Large Language Model Compression
University of Illinois Urbana-Champaign
Prune whole structures.
Keep the model’s useful behavior.
2026
23.8%
LLM-Pruner: 4.4%.
MBPP pass@1
20% target sparsity, before LoRA.LLM-Pruner: 4.4%.
62.11%
Before recovery: 55.09%.
Average accuracy after LoRA
Llama3.1-8B, 20% target sparsity.Before recovery: 55.09%.
1.57×
12.41 → 19.50 ktokens/s.
Inference throughput
50% target sparsity, A100, batch 32.12.41 → 19.50 ktokens/s.
A little love for our namesake
Congratulations, Nirvana.
Celebrating the band’s 2026 MTV Video Vanguard Award. Thanks for the music that accompanies me through a long time.
The official announcement ↗01 / The question
What should a smaller
language model keep?
LLMs are expensive to deploy. Structured pruning removes entire attention heads and MLP neurons, reducing model dimensions so the savings can translate into practical acceleration.
But compression can damage reasoning and code generation. NIRVANA asks how to remove whole structures while limiting changes to model outputs and retaining the ability to recover through fine-tuning.
Three things a pruning decision needs to respect.
- 01
What the model outputs
A unit’s importance depends on how much removing it changes the model function.
- 02
Where the unit lives
Layers and attention/MLP modules play different roles. Their pruning budgets should reflect those differences.
- 03
Which data tells the story
Calibration data affects saliency estimates. Select it by the behavior of the resulting pruned model.
02 / The method
From output impact
to structured decisions.
Score individual weights, group dependent neurons or heads, then allocate pruning across layers and modules.
Flexible attention / MLP allocation
Use global unit ranking for nonuniform layer sparsity. Balance the pruning rates of MLP neurons and attention heads through γ.
KL-guided calibration
Prune with candidate calibration batches. Choose the batch whose pruned model has the smallest output KL divergence from the dense model on a fixed held-out anchor set.
Select data by the behavior it preserves.
03 / The evidence
Smaller models.
Useful capabilities.
Evaluation on Llama3.1-8B spans language modeling, commonsense reasoning, math, and code generation, before and after LoRA recovery.
A closer look
Compare NIRVANA with LLM-Pruner.
Llama3.1-8B · 20% sparsity · Before LoRA
| Benchmark | NIRVANA | LLM-Pruner |
|---|---|---|
| MBPP | 23.80 | 4.40 |
| SVAMP | 49.00 | 22.33 |
| Avg. accuracy | 55.09 | 45.65 |
+9.44 points in average accuracy over LLM-Pruner.
MBPP: 3-shot pass@1. Sparsity is the pruning target, not the percentage reduction in total parameters. Full baseline comparisons are in the original table.
Practical inference gains
Llama3.1-8B · 50% target sparsity
Single A100 · batch size 32
Throughput
12.4119.50
ktokens/s · 1.57× the dense modelMemory
17.9511.27
GB · dense → NIRVANA04 / Take it with you
The paper.
The code.
The conversation.
Read the full method and experiments, explore the implementation, or download the COLM poster.
Questions, ideas, or just saying hello?mai10@illinois.edu
@misc{ai2025nirvana,
title = {NIRVANA: Structured Pruning Reimagined
for Large Language Model Compression},
author = {Mengting Ai and Tianxin Wei and
Sirui Chen and Jingrui He},
year = {2025},
eprint = {2509.14230},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2509.14230}
}