Empirical Benchmarking of Generalization Failure in Alternative Optimization Dynamics

The problem

Alternative optimization methods, such as exact Gauss-Newton (GN) and non-backpropagation dynamics (e.g., NMNC), exhibit severe generalization gaps and early loss saturation when scaled to deep networks and mini-batch settings. Standard regularizers (dropout, weight decay, data augmentation, pseudoinverse regularization) and initialization heuristics developed on shallow models consistently fail to close these train-test gaps. Without a controlled comparative evaluation across these distinct settings, it remains unknown whether these generalization failures share common optimization dynamics or require fundamentally different stabilization interventions.

Why it matters

Provides empirical boundary conditions identifying where non-standard and second-order update dynamics break down in deep networks, enabling optimization researchers to test targeted remedies against an established baseline.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

1. 1

   Construct a standardized benchmark evaluating NMNC, mini-batch exact Gauss-Newton, and standard backpropagation across identical ResNet and ConvNet backbones on CIFAR-100 and ImageNet, measuring train-test generalization gaps, batch-level loss trajectories, and curvature metrics.

   Check for prior work
2. 2

   Systematically test stochastic regularization and damping mechanisms (e.g., batch-size scaling, stochastic damping, gradient clipping) across all three training regimes to measure whether any single intervention reliably prevents mini-batch overfitting.

   Check for prior work

Why it might fail

The generalization gaps may be trivial consequences of fundamental sample-complexity lower bounds for non-gradient updates, or the community may abandon these alternative update rules entirely in favor of standard first-order backpropagation variants.

Evidence

Each paper's own statement of the limitation, verbatim.

• Light Differentiable Logic Gate NetworksICLR 2026

  A substantial generalization gap between train and test accuracy remains, which standard regularizers (dropout, residual connections, data augmentations) do not resolve.

• Credit Assignment via Neural Manifold Noise CorrelationICML 2026

  A substantial accuracy gap remains between NMNC (~45%) and exact backpropagation (~57%) on ImageNet-scale benchmarks.

• Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor GeneralizationNeurIPS 2024

  Exact GN generalizes poorly in mini-batch training: it overfits each mini-batch and training/test losses saturate early; smaller learning rates, weight decay, and pseudoinverse regularization all fail to fix this, and remedies suggested by prior work (small learning rates, better initializations) do not carry over to deeper networks

Nearest existing work

• Train faster, generalize better: Stability of stochastic gradient descentICML 2016
• Understanding Batch NormalizationNeurIPS 2018
• The Marginal Value of Adaptive Gradient Methods in Machine LearningNeurIPS 2017
• On the generalization of learning algorithms that do not convergeNeurIPS 2022
• The Implicit Regularization of Dynamical Stability in Stochastic Gradient DescentICML 2023
• Surprising Instabilities in Training Deep Networks and a Theoretical AnalysisNeurIPS 2022
• A Quantitative Analysis of the Effect of Batch Normalization on Gradient DescentICML 2019
• Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor GeneralizationNeurIPS 2024
• Gradient Descent Finds Global Minima of Deep Neural NetworksICML 2019
• SGD with Large Step Sizes Learns Sparse FeaturesICML 2023
• On Lazy Training in Differentiable ProgrammingNeurIPS 2019
• Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying RegularizationNeurIPS 2025
• Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural NetworksNeurIPS 2022
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural NetworksNeurIPS 2016
• Adversarially Robust Neural Networks via Optimal Control: Bridging Robustness with Lyapunov StabilityICLR 2020

Related open problems