Client-Scalable Federated Learning Under Severe Data Partitioning and Heterogeneity

The problem

Federated learning algorithms consistently exhibit severe accuracy collapses when the client population scales from small toy cohorts (5–10 clients) to realistic deployments (30–100+ clients). As the client count increases for a fixed dataset, each client receives fewer training samples while cross-client data heterogeneity and gradient conflict intensify, causing standard aggregation operators to produce destructive model updates. This scaling wall structurally limits federated systems from deploying in environments where data is naturally fragmented across large numbers of low-volume edge devices. Current literature masks this limitation by evaluating primarily on small, well-provisioned client counts where local sample sizes remain artificially high.

Why it matters

Enables federated learning systems to scale to hundreds of decentralized, low-sample participants without suffering performance collapse, eliminating the need to artificially restrict federation size or downscale local model architectures.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

1. 1

   Controlled Scaling Diagnostic & Gradient Conflict Analysis: Construct an empirical testbed evaluating standard FL algorithms (FedAvg, FedProx, SCAFFOLD) across client scales ($K \in \{5, 10, 20, 50, 100, 200\}$) across vision and language benchmarks; measure gradient cosine similarity distributions, local drift norms, and effective rank of aggregated updates to pinpoint the exact transition point where aggregation fails.

   Check for prior work
2. 2

   Subspace Projection and Conflict-Aware Aggregation: Implement a server-side aggregation mechanism that projects client updates into common gradient subspaces and downweights orthogonal or opposing directional updates from data-starved clients; evaluate validation accuracy and round-to-convergence as $K$ increases from 5 to 100.

   Check for prior work
3. 3

   Adaptive Representation Anchoring: Implement a prototype-regularized local objective where clients align feature representations to globally shared anchor points to prevent local models with small sample sizes from overfitting to spurious local features; measure test accuracy across Dirichlet non-IID partitions at $K=80$ and $K=100$.

   Check for prior work

Why it might fail

If the performance collapse is fundamentally driven by an information-theoretic sample-complexity floor—where local clients have too few examples to produce any valid local gradient direction under extreme non-IID splits—server-side aggregation techniques will fail without introducing auxiliary shared public data or client clustering.

Evidence

Each paper's own statement of the limitation, verbatim.

• RAFed: Responsive Augmentation and Approximate Update Method for Federated Learning with Non-IID DataTheWebConference 2026

  Performance degrades as client count grows (e.g., 62.50% at C=5 down to 49.23% at C=100 on CIFAR-10, alpha=0.05), reflecting increased heterogeneity and aggregation difficulty

• Noise-Free Dynamic Rank-Adaptation via Riemannian Methods in Federated Fine-TuningAISTATS 2026

  Accuracy degrades substantially as the number of clients grows (notably dropping when N reaches 80), because per-client data shrinks and heterogeneity intensifies

• Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated LearningICML 2026

  Performance degrades as client count grows (e.g., CIFAR-10-LT20 F1 drops from 0.8568 at K=5 to 0.8010 at K=20, IID), though less than FedCGN's drop

• FedTMOS: Efficient One-Shot Federated Learning with Tsetlin MachineICLR 2025

  Performance declines as client count grows because local TM model size must be scaled down (2.9×–4× fewer clauses for 20–80 clients) to constrain server model size

• Does One-shot Give the Best Shot? Mitigating Model Inconsistency in One-shot Federated LearningICML 2025

  Performance degrades as client count grows (e.g., 88.74% at 5 clients down to 75.37% at 100 on CIFAR-10)

• Can Textual Gradient Work in Federated Learning?ICLR 2025

  Performance degrades sharply as the number of clients or local steps increases, and heterogeneous clients can produce conflicting contexts that lead to failed or ambiguous aggregation; classical FL conflict-resolution methods do not apply to text.

• Resource-Aware Federated Self-Supervised Learning with Global Class RepresentationsNeurIPS 2024

  Performance degrades as client count increases (5→30 clients drops linear accuracy by ~14 points) since per-client data shrinks

• PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental LearningECCV 2024

  Performance degrades as the number of clients increases (K=10→20 causes a slight decline) since heterogeneity deepens; robustness is only shown for 10 clients

Nearest existing work

• FedScale: Benchmarking Model and System Performance of Federated Learning at ScaleICML 2022
• Communication-Efficient Federated Learning with Accelerated Client GradientCVPR 2024
• Towards Instance-adaptive Inference for Federated LearningICCV 2023
• Local Learning Matters: Rethinking Data Heterogeneity in Federated LearningCVPR 2022
• FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated LearningNeurIPS 2025
• Low Precision Local Training is Enough for Federated LearningNeurIPS 2024
• ScaleFL: Resource-Adaptive Federated Learning With Heterogeneous ClientsCVPR 2023
• Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated LearningICML 2021
• Elastic Aggregation for Federated OptimizationCVPR 2023
• Anchor Sampling for Federated Learning with Partial Client ParticipationICML 2023
• An Aggregation-Free Federated Learning for Tackling Data HeterogeneityCVPR 2024
• FedGMKD: An Efficient Prototype Federated Learning Framework through Knowledge Distillation and Discrepancy-Aware AggregationNeurIPS 2024
• Fedvarp: Tackling the variance due to partial client participation in federated learningUAI 2022
• FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client VectorsCVPR 2025
• FedARA: Resource-adaptive Low-rank Personalized Federated Learning via Anchor-driven Representation Alignment on Heterogeneous Edge DevicesCVPR 2026

Related open problems