Open Problems

Quantization & Low-Precision Training

Post-Training Realization of Training-Time Quantization Objectives on Fixed Pre-Trained Models

Barrier to removeOpen
Possible candidate · 2/5 runs3 papers report this33% from 2025+

Generated automatically from the limitations stated in 3 papers (NeurIPS, ICML), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

High-performance low-precision quantization techniques depend heavily on active training cycles, relying on mechanisms such as iterative knowledge distillation, multi-stage noise optimization, and auxiliary loss terms (e.g., kurtosis minimization). Because these methods require backpropagation through the model and access to full training infrastructure, they cannot be applied to fixed, pre-trained checkpoints, closed-source models, or settings where original training pipelines and datasets are inaccessible. Practitioners constrained to frozen models are therefore blocked from utilizing these advanced quantization objectives and must fall back to basic post-training heuristics. Bridging this gap enables applying sophisticated distribution-shaping and distillation objectives directly to fixed models.

Why it matters

Advanced distribution-regularized and distillation-guided quantization becomes directly applicable to frozen, pre-trained model weights without requiring full training cycles, loss backpropagation, or access to the original training setup.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Analytical coordinate transformations: Derive closed-form weight-activation equivalences (e.g., channel-wise scaling or rotations) that directly reduce activation kurtosis and shape outlier distributions during a single forward-only calibration pass, measuring calibration runtime and post-quantization task accuracy.

  2. 2

    Layer-wise reconstruction matching: Reformulate the multi-stage noise optimization and distillation objectives into isolated, layer-by-layer quadratic reconstruction problems solved over small calibration sets without global backpropagation, measuring memory overhead, speedup over full fine-tuning, and downstream accuracy across 4-bit and 3-bit regimes.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If distribution-shaping regularizers like kurtosis suppression fundamentally depend on non-local, cross-layer parameter interactions that cannot be captured locally, feed-forward calibration approximations will fail to close the accuracy gap with full fine-tuning at low bit-widths.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Quantization & Low-Precision Training

Effect to explainOpen

Characterizing and Mitigating Low-Bit Quantization Breakdown in Small-Scale Language Models

Post-training quantization consistently fails in the sub-3-bit and 2-bit regimes, suffering catastrophic perplexity degradation that disproportionately impacts small and mid-sized models (e.g., OPT-125M, LLaMA-7B/8B). Because smaller base models have less parameter redundancy, aggressive bit-width reductions push perplexity beyond functional utility (often exceeding 100+ PPL), preventing local deployment of compact models on edge hardware. Today, practitioners are forced to choose between unquantized models that exceed device memory budgets or severely degraded low-bit variants that produce gibberish.

Possible candidate · 3/5 runs12 papers report this58% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.