Deep Learning Theory & Optimization Dynamics
Bridging the Activation Divide in Neural Network Optimization Dynamics Theory
Generated automatically from the limitations stated in 13 papers (NeurIPS, ICML, ICLR), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Current theoretical analyses of neural network training dynamics are split across incompatible mathematical preconditions: one major body of literature strictly requires high-order smoothness ($C^2$, $C^\infty$, bounded derivatives, or odd symmetry), which explicitly excludes standard piecewise-linear activations like ReLU, while another body relies strictly on piecewise linearity and positive homogeneity, failing to transfer to smooth activations like GELU, SiLU, or tanh. As a result, foundational theoretical predictions—such as edge-of-stability behavior, width-independent convergence rates, and representation learning bounds—cannot be generalized across activation families. Without analytical tools that either handle subgradient boundary crossings or bound the divergence between smooth approximations and exact piecewise-linear trajectories during training, optimization theory remains fragmented into mutually exclusive activation regimes.
Why it matters
Enables optimization and generalization theorems developed exclusively for smooth activations to be transferred rigorously to ReLU networks (and vice versa). It also establishes explicit mathematical boundaries for when smooth approximations faithfully represent non-smooth neural network training dynamics.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Trajectory-Dependent Smoothing Error Bounds: Analyze the discrepancy between gradient descent trajectories on exact piecewise-linear networks and parameterized smooth approximations (e.g., smoothed ReLU$_\iota$ or GELU), measuring the cumulative error and iteration complexity penalty as a function of the smoothing scale $\iota$ and the rate of neuron activation-state transitions.
- 2
Differential Inclusions vs. Regularized Gradient Flows: Formulate neural network training dynamics using Clarke subdifferentials and compare the resulting non-smooth flow directly against the zero-temperature limit of smooth ODE dynamics, measuring convergence rate bounds and stability thresholds on synthetic two-layer networks.
- 3
Unified Weak-Smoothness Lyapunov Framework: Construct Lyapunov functionals for gradient descent dynamics that replace strict $C^2$ requirements with directional or almost-everywhere differentiability, measuring whether width-independent convergence guarantees hold across ReLU, LeakyReLU, and smooth activations under identical optimizer settings.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
If gradient descent dynamics for ReLU networks exhibit pathological activation-switching behavior that causes smooth surrogate trajectories to diverge exponentially rather than polynomially over training time. Additionally, technical barriers in non-smooth dynamical systems (differential inclusions) may prevent closing the gap without introducing equally restrictive geometric assumptions.
Evidence
Each paper's own statement of the limitation, verbatim.
- Width Independent Bounds for the Local Lipschitz Constant of Deep Neural Networks at Random Initialization and after Lazy TrainingICML 2026
Analysis requires smooth activations (ϕ(0)=0, 1-Lipschitz, β_φ-smooth); ReLU networks are not covered, though the authors conjecture analogous width-independence holds
- A Derandomization Framework for Structure Discovery: Applications in Neural Networks and BeyondICLR 2026
Framework requires smooth activations (ReLU handled only via smooth ReLU_ι approximation, with iteration complexity degrading polynomially in ι)
- Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimeCOLT 2025
Requires odd, twice-differentiable activation and readout functions with bounded first and second derivatives, excluding e.g. ReLU.
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$ ParametrizationICML 2025
Analysis assumes infinite width and mild but real activation restrictions (twice continuously differentiable, bounded derivatives, GOOD); ReLU-family activations lacking these properties are not covered
- Entropy-based Activation Function Optimization: A Method on Searching Better Activation FunctionsICLR 2025
The framework requires the activation function to be invertible; non-invertible functions need ad hoc regional treatment and approximations (as done for ReLU)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU NetworksNeurIPS 2024
Evaluated only on ReLU networks; does not address other activation functions
- Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast OptimizationNeurIPS 2024
The activation must be continuously differentiable (ruling out ReLU) and, for the EoS analysis, its derivative must be bounded away from zero — requiring 'leaky' modifications of standard activations like GELU or softplus
- Implicit regularization of deep residual networks towards neural ODEsICLR 2024
Requires smooth (C², Lipschitz, bounded, σ(0)=0) activation functions; ReLU destroys weight smoothness during training (proven in a simple setting in Appendix D)
Show all 13 papers
- Training Fully Connected Neural Networks is $\exists\mathbb{R}$-CompleteNeurIPS 2023
The result applies only to piecewise-linear activation functions; the methods do not transfer to Sigmoid, soft ReLU, or step functions, and decidability for Sigmoid/softplus is unknown.
- Precise Asymptotic Analysis of Deep Random Feature ModelsCOLT 2023
Requires odd activation functions with bounded derivatives satisfying differentiability of eta_1, eta_2 (tanh, erf qualify; ReLU violates the oddness requirement as stated)
- Gradient Descent in Neural Networks as Sequential Learning in Reproducing Kernel Banach SpaceICML 2023
Cannot handle non-smooth activations such as ReLU, since the construction fundamentally requires C∞ activations with power-series representations; polynomial approximation of ReLU is conjectured but unproven as mathematically reasonable when a weight-step crosses the non-smooth point
- Global Convergence of Over-parameterized Deep Equilibrium ModelsAISTATS 2023
Analysis is specific to ReLU activations; no extension to other nonlinearities is provided
- Exact Verification of ReLU Neural Control Barrier FunctionsNeurIPS 2023
Restricted to feedforward ReLU networks; other activations and continuous differentiability are not handled
Nearest existing work
- Continuous vs. Discrete Optimization of Deep Neural NetworksNeurIPS 2021
- Global Convergence in Neural ODEs: Impact of Activation FunctionsICLR 2025
- A Minimalist Example of Edge-of-Stability and Progressive SharpeningNeurIPS 2025
- Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic ActivationsNeurIPS 2025
- Understanding Gradient Descent on the Edge of Stability in Deep LearningICML 2022
- Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth OptimizationICML 2026
- Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation TheoryNeurIPS 2023
- A Dynamical View on Optimization Algorithms of Overparameterized Neural NetworksAISTATS 2021
- Surprising Instabilities in Training Deep Networks and a Theoretical AnalysisNeurIPS 2022
- SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal StructuresNeurIPS 2025
- The phase diagram of approximation rates for deep neural networksNeurIPS 2020
- Learning Stable Deep Dynamics ModelsNeurIPS 2019
- Scaling Properties of Deep Residual NetworksICML 2021
- Dynamics of Deep Neural Networks and Neural Tangent HierarchyICML 2020
- Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient DescentNeurIPS 2019
Related open problems
Deep Learning Theory & Optimization Dynamics
Extending Two-Layer Network Theory to Deep and Structured Architectures
A substantial body of rigorous results—on training dynamics, predictor statistics, generalization, and under-fitting—is proven only for one- or two-layer fully-connected networks, often with fixed or special activations. Practitioners use deep residual, convolutional, and attention models, so none of these guarantees can be checked against the systems that matter. Until the theory carries over, each paper's conclusions remain unvalidated in practice, and the specific mechanisms they identify may be artifacts of the shallow, unstructured setting.
Deep Learning Theory & Optimization Dynamics
Cross-Architecture Empirical Validation of Deep Learning Theory Predictors
Theoretical analysis and optimization dynamic guarantees in deep learning are overwhelmingly developed and proved on shallow fully connected ReLU models or two-layer networks. As a result, theoretical claims regarding convergence rates, generalization bounds, and optimization statistics are rarely verified on non-feedforward or deeper architectures such as CNNs, ResNets, and Transformers. Without systematic empirical evaluation across these structural settings, it remains unknown whether these theoretical quantities correlate with actual training dynamics or are merely artifacts of idealized shallow feedforward assumptions.
Deep Learning Theory & Optimization Dynamics
Empirical Benchmarking of Generalization Failure in Alternative Optimization Dynamics
Alternative optimization methods, such as exact Gauss-Newton (GN) and non-backpropagation dynamics (e.g., NMNC), exhibit severe generalization gaps and early loss saturation when scaled to deep networks and mini-batch settings. Standard regularizers (dropout, weight decay, data augmentation, pseudoinverse regularization) and initialization heuristics developed on shallow models consistently fail to close these train-test gaps. Without a controlled comparative evaluation across these distinct settings, it remains unknown whether these generalization failures share common optimization dynamics or require fundamentally different stabilization interventions.