Open Problems

Quantization & Low-Precision Training

Characterizing and Mitigating Low-Bit Quantization Breakdown in Small-Scale Language Models

Effect to explainOpen
Possible candidate · 3/5 runs12 papers report this58% from 2025+

Generated automatically from the limitations stated in 12 papers (ICML, ICLR, ACL), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Post-training quantization consistently fails in the sub-3-bit and 2-bit regimes, suffering catastrophic perplexity degradation that disproportionately impacts small and mid-sized models (e.g., OPT-125M, LLaMA-7B/8B). Because smaller base models have less parameter redundancy, aggressive bit-width reductions push perplexity beyond functional utility (often exceeding 100+ PPL), preventing local deployment of compact models on edge hardware. Today, practitioners are forced to choose between unquantized models that exceed device memory budgets or severely degraded low-bit variants that produce gibberish.

Why it matters

Enables deployable sub-3-bit compression for compact LLMs on microcontrollers and edge devices without catastrophic output degeneration.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Layer-wise Capacity and Sensitivity Audit: Profile activation dynamic ranges, outlier channels, and Hessian eigenvalue spectra across model scales (125M to 70B) under 2-bit and 3-bit regimes to measure exactly which architectural components trigger catastrophic quantization error in small models.

  2. 2

    Mixed-Precision and Outlier Retention Allocation: Implement an automated integer linear programming (ILP) or greedy sensitivity-based bit-allocation framework that protects critical layers (e.g., early layers, down-projections) at higher precision while compressing the rest to sub-2-bit, measuring perplexity recovery against uniform baseline bit budgets.

  3. 3

    Low-Rank Error Compensation: Apply low-rank residual adapters (e.g., FP16/INT4 low-rank correction terms) specifically tuned to recover the high-magnitude quantization errors observed in capacity-constrained smaller architectures, measuring language modeling perplexity and downstream zero-shot accuracy.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

The capacity limit of small models under 2-bit representations may be an inescapable information-theoretic bottleneck where weight representation requires retraining or substantial architectural changes rather than post-training quantization schemes. Furthermore, rapid concurrent development in post-training quantization (PTQ) techniques creates high competition risk.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 12 papers

Nearest existing work

Related open problems

Quantization & Low-Precision Training

Barrier to removeOpen

Post-Training Realization of Training-Time Quantization Objectives on Fixed Pre-Trained Models

High-performance low-precision quantization techniques depend heavily on active training cycles, relying on mechanisms such as iterative knowledge distillation, multi-stage noise optimization, and auxiliary loss terms (e.g., kurtosis minimization). Because these methods require backpropagation through the model and access to full training infrastructure, they cannot be applied to fixed, pre-trained checkpoints, closed-source models, or settings where original training pipelines and datasets are inaccessible. Practitioners constrained to frozen models are therefore blocked from utilizing these advanced quantization objectives and must fall back to basic post-training heuristics. Bridging this gap enables applying sophisticated distribution-shaping and distillation objectives directly to fixed models.

Possible candidate · 2/5 runs3 papers report this33% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.