Deep Learning Theory & Optimization Dynamics
Cross-Architecture Empirical Validation of Deep Learning Theory Predictors
Generated automatically from the limitations stated in 6 papers (ICML, ICLR, COLT), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Theoretical analysis and optimization dynamic guarantees in deep learning are overwhelmingly developed and proved on shallow fully connected ReLU models or two-layer networks. As a result, theoretical claims regarding convergence rates, generalization bounds, and optimization statistics are rarely verified on non-feedforward or deeper architectures such as CNNs, ResNets, and Transformers. Without systematic empirical evaluation across these structural settings, it remains unknown whether these theoretical quantities correlate with actual training dynamics or are merely artifacts of idealized shallow feedforward assumptions.
Why it matters
Establishes an empirical baseline determining which deep learning theory diagnostics reliably hold across modern architectures, enabling theorists to target mechanisms that genuinely break down in practical model families.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Benchmark existing theoretical metrics (e.g., neural tangent kernel spectral properties, gradient alignment, PAC-Bayes bounds) across standardized convolutional, recurrent, and self-attention models on standard vision and sequential datasets to measure predictive correlation with final generalization.
- 2
Construct controlled architectural ablation ladders (from 2-layer MLP to deep MLP, CNN, ResNet, and Vision Transformer) to measure exactly where existing theoretical guarantees and dynamics tracking fail to predict empirical behavior.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
High training instability or architecture-specific hyperparameter confounders could obscure underlying theoretical trends, or computing kernel-based theoretical metrics across large deep architectures could prove computationally prohibitive without significant approximations.
Evidence
Each paper's own statement of the limitation, verbatim.
- The impact of allocation strategies in subset learning on the expressive power of neural networksICLR 2025
Main theoretical guarantees are limited to linear RNNs and linear feedforward networks; the ReLU extension covers only one-hidden-layer networks and is partly heuristic/conjectural
- Forward Learning with Top-Down Feedback: Empirical and Analytical CharacterizationICLR 2024
Experiments are limited to fully connected networks with up to five hidden layers; convolutional architectures are not addressed, and the theoretical analysis covers only two-layer networks
- Defining Neural Network Architecture through Polytope Structures of DatasetsICML 2024
Analysis restricted to fully connected ReLU networks; convolutional and other architectures not covered
- Critical feature learning in deep neural networksICML 2024
Only fully-connected feed-forward networks analyzed; extension to RNNs, CNNs, ResNets, and transformers is proposed but not demonstrated; predictor statistics not yet computed
- The Expressive Power of Tuning Only the Normalization LayersCOLT 2023
Results are for fully connected ReLU networks only; applicability to transformers/convolutional architectures with attention or other nonlinearities is not shown
- Benign Overfitting in Two-layer ReLU Convolutional Neural NetworksICML 2023
Limited to two-layer CNNs; authors state extension to deep ReLU networks is open.
Nearest existing work
- Understanding Generalization and Optimization Performance of Deep CNNsICML 2018
- Learning by Turning: Neural Architecture Aware OptimisationICML 2021
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural NetworksICML 2019
- On the generalization of learning algorithms that do not convergeNeurIPS 2022
- Analyzing Generalization of Neural Networks through Loss Path KernelsNeurIPS 2023
- When Representations Align: Universality in Representation Learning DynamicsICML 2024
- Gradient Descent Finds Global Minima of Deep Neural NetworksICML 2019
- Optimal Rates for Generalization of Gradient Descent for Deep ReLU ClassificationNeurIPS 2025
- A Convergence Theory for Deep Learning via Over-ParameterizationICML 2019
- GradSign: Model Performance Inference with Theoretical InsightsICLR 2022
- An Investigation into Neural Net Optimization via Hessian Eigenvalue DensityICML 2019
- Block Coordinate Descent for Neural Networks Provably Finds Global MinimaNeurIPS 2025
- Learning Trajectories are Generalization IndicatorsNeurIPS 2023
- Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent KernelNeurIPS 2025
- On the Convergence Rate of Training Recurrent Neural NetworksNeurIPS 2019
Related open problems
Deep Learning Theory & Optimization Dynamics
Bridging the Activation Divide in Neural Network Optimization Dynamics Theory
Current theoretical analyses of neural network training dynamics are split across incompatible mathematical preconditions: one major body of literature strictly requires high-order smoothness ($C^2$, $C^\infty$, bounded derivatives, or odd symmetry), which explicitly excludes standard piecewise-linear activations like ReLU, while another body relies strictly on piecewise linearity and positive homogeneity, failing to transfer to smooth activations like GELU, SiLU, or tanh. As a result, foundational theoretical predictions—such as edge-of-stability behavior, width-independent convergence rates, and representation learning bounds—cannot be generalized across activation families. Without analytical tools that either handle subgradient boundary crossings or bound the divergence between smooth approximations and exact piecewise-linear trajectories during training, optimization theory remains fragmented into mutually exclusive activation regimes.
Deep Learning Theory & Optimization Dynamics
Extending Two-Layer Network Theory to Deep and Structured Architectures
A substantial body of rigorous results—on training dynamics, predictor statistics, generalization, and under-fitting—is proven only for one- or two-layer fully-connected networks, often with fixed or special activations. Practitioners use deep residual, convolutional, and attention models, so none of these guarantees can be checked against the systems that matter. Until the theory carries over, each paper's conclusions remain unvalidated in practice, and the specific mechanisms they identify may be artifacts of the shallow, unstructured setting.
Deep Learning Theory & Optimization Dynamics
Empirical Benchmarking of Generalization Failure in Alternative Optimization Dynamics
Alternative optimization methods, such as exact Gauss-Newton (GN) and non-backpropagation dynamics (e.g., NMNC), exhibit severe generalization gaps and early loss saturation when scaled to deep networks and mini-batch settings. Standard regularizers (dropout, weight decay, data augmentation, pseudoinverse regularization) and initialization heuristics developed on shallow models consistently fail to close these train-test gaps. Without a controlled comparative evaluation across these distinct settings, it remains unknown whether these generalization failures share common optimization dynamics or require fundamentally different stabilization interventions.