How AI Foundation Models Address Scaling Bottlenecks: Mathematical Structures, Algorithm Design, and Systems Engineering

Today, users can use AI applications built on foundation models to process several documents, each hundreds of pages long, refer back to earlier context during long conversations, generate images quickly, or answer questions under industry-specific rules. They usually see only an input box. Behind it, parameter training, long-sequence computation, generative paths, post-training, and inference systems all work together.

People often attribute these capabilities to three factors: more data, more compute, and more parameters.

That explanation is not wrong, but it misses another equally important thread. As models have grown, mathematical and computational problems that once seemed secondary have become critical bottlenecks that foundation models must confront if they are to keep scaling.

Must a model with hundreds of billions of parameters modify every parameter to adapt to a new task? Must a long sequence explicitly compute attention scores between every pair of tokens? Once preference data are available, does post-training still require a reward model, a value model, and a complete reinforcement-learning system?

These questions come from different fields, but researchers have repeatedly taken the same approach:

Identify the effective structure in the problem, then redesign its representation and computation.

This reformulation may reduce the dimensions, parameters, relationships, or intermediate results a system must process. It may remove parts of a training pipeline or change how data move. It may also shift costs into communication, state management, or verification. Its value is not simply to “compute less”: Transformers reorganize sequence computation, rotary position embeddings reformulate positional relationships, Flow Matching redesigns the generative path between distributions, and non-convex optimization studies the geometry of the objective landscape.

This article is not an algorithm catalog. It asks what structure researchers found, how they turned that structure into scalable algorithms and systems, and where the bottleneck moved next.

I. Large Problems May Have Much Simpler Effective Structure #

Many machine-learning problems eventually become questions about matrices, parameter spaces, or optimization. As nominal dimension grows, the first question is not how to process the entire high-dimensional object, but how many degrees of freedom the task actually depends on.

Matrix Sketching and Geometric Structure #

In randomized numerical linear algebra (RandNLA), matrix sketching uses random projection or sampling to compress a matrix into a smaller representation, then performs the main computation on that representation. The geometry to preserve is usually the lengths, distances, or subspace relationships relevant to the task—not every entry of the original matrix. The underlying judgment is simple: if a task depends on only part of a matrix’s geometry, the algorithm may not need to process the full matrix.

For a matrix \(A\in\mathbb{R}^{m\times n}\), a random map \(S\) can construct a smaller representation \(SA\), where:

\[ SA\in\mathbb{R}^{k\times n}, \qquad k\ll m. \]

When \(S\) satisfies an appropriate subspace-embedding condition, the compressed matrix can approximately preserve important geometric properties of the original. Matrix sketching, randomized singular value decomposition (randomized SVD), random projection, and leverage-score sampling all belong to this family of methods. In 2014, Woodruff surveyed their theoretical results in linear regression, low-rank approximation, and numerical linear algebra [1].

The computational benefit of RandNLA depends on the matrix shape, target rank, error tolerance, and specific method, so it must be evaluated for the task at hand. RandNLA changes the starting point: first decide what the task must preserve, then decide whether the full object is worth processing.

Intrinsic Dimension and Low-Rank Adaptation #

As pretrained models reached billions and tens of billions of parameters, low-dimensional structure appeared in another form. Intrinsic dimension is not the total number of parameters. It is the dimension required for a low-dimensional reparameterization to approach the performance available in the full parameter space. A model can have many parameters without requiring adaptation to move freely through the entire parameter space.

Aghajanyan, Gupta, and Zettlemoyer constrained pretrained language models to optimize within random subspaces far smaller than the full parameter space, then measured how much of full fine-tuning performance this restricted optimization could recover. For some of the models and tasks they studied, the intrinsic dimension required for task adaptation was far smaller than the total parameter dimension [2]. This observation is limited to the models and tasks examined in the paper, but it suggests that adaptation may need to explore only a small set of effective directions in parameter space.

The rank of a matrix can be understood as the number of linearly independent directions it contains; low rank means describing variation with relatively few directions. LoRA (Low-Rank Adaptation) turns a related idea into a scalable parameterization [3]. For a weight matrix \(W\in\mathbb{R}^{d\times k}\), LoRA does not learn a full update \(\Delta W\). Instead, it writes:

\[ \Delta W=AB, \qquad A\in\mathbb{R}^{d\times r}, \quad B\in\mathbb{R}^{r\times k}, \]

where \(r\ll\min(d,k)\). Training freezes the original weights and optimizes only the low-rank matrices \(A\) and \(B\). The largest model in the original paper was GPT-3 with 175 billion parameters. On the tasks examined there, optimizing only a low-rank update still produced effective adaptation results.

RandNLA and LoRA operate on different objects, but they share the same starting point: nominal scale may be large while useful information and effective change are concentrated in far fewer dimensions.

Non-Convex Geometry and Optimization Paths #

Useful structure is not limited to low dimensionality. It also appears in non-convex optimization. Here, the optimization landscape means the distribution of local minima, saddle points, and descent paths in parameter space. A common intuition equates “non-convex” with “full of bad local minima that cannot be escaped,” but some problems with particular matrix structures behave differently.

A spurious local minimum looks optimal locally but is not globally optimal. Ge and colleagues proved that, under appropriate conditions, certain matrix-completion problems contain no such minima [4]. Other work analyzed negative-curvature directions around saddle points and showed that optimization methods with stochastic perturbations can escape saddles under specific conditions [5]. Gunasekar and colleagues studied implicit regularization: even when an objective contains no explicit regularizer of a particular kind, the path taken by gradient descent may still favor solutions with special structure [6].

These results depend on particular ranks, initializations, incoherence conditions, and geometric structures. They support a narrower conclusion: complex spaces can contain comparatively favorable regions and paths, and identifying them can help explain why some optimization methods work on larger problems.

Compared with these theoretical analyses, LoRA has had a more direct effect on development practice. Its clearest impact is lower customization cost. A company can store small sets of LoRA adapter parameters for customer support, contract analysis, and internal knowledge Q&A instead of training and maintaining a complete model for every task; smaller teams can iterate more easily as well. LoRA reduces the number of parameters that require gradients and optimizer states, and it substantially lowers the additional parameter storage required for each task. Total training memory still depends on the base-model weights, activations, sequence length, and quantization scheme. LoRA does not usually make each inference call faster. Once adaptation becomes cheaper, attention turns to planning foundation-model scale and allocating compute.

II. Scaling Foundation Models Requires Rethinking Training and Resource Allocation #

Low-dimensional structure reduces some computational and adaptation costs, but scaling a foundation model requires more: sequence computation must be parallelizable, training must remain stable, and the resources required to enlarge the model must be planned.

Transformer and AdamW #

A Transformer is a sequence-model architecture built around self-attention and position-wise feed-forward networks. It does not rely on recurrence to organize sequence computation.

A recurrent neural network updates its state as \(h_t=f(h_{t-1},x_t)\). It must obtain \(h_{t-1}\) before computing \(h_t\), and this temporal dependency limits parallelism on graphics processing units (GPUs).

The Transformer reformulates sequence modeling as matrix computation centered on self-attention [7]:

\[ \operatorname{Attention}(Q,K,V)= \operatorname{softmax} \left( \frac{QK^\top}{\sqrt{d_k}} \right)V. \]

Here, \(Q\), \(K\), and \(V\) are the query, key, and value representation matrices. The query and key determine matching weights between tokens, while the value provides the information that is then aggregated.

Relationships between tokens no longer propagate mainly through hidden states one step at a time; \(QK^\top\) computes them directly. The models reported in the original paper were far smaller than today’s large models, but the Transformer reorganized sequence computation so that GPUs and tensor processing units (TPUs) could compute token relationships in parallel at scale. Its central contribution was to cast the problem in a form better suited to modern hardware.

Adam (Adaptive Moment Estimation) and AdamW, which uses decoupled weight decay, later became common parameter-update methods for large-scale training. In Adam, adding an \(L_2\) regularization term to the loss also subjects that term to adaptive gradient scaling, so it is no longer equivalent to applying weight decay independently. AdamW therefore decouples weight decay from the loss-gradient update [8]. The full optimizer equations are not necessary for this article’s main argument. What matters here is that as training scales, parameter updates must also preserve stability and well-defined regularization behavior.

Scaling Laws and Chinchilla #

Once models could be trained in parallel, the question shifted from “Can we train them?” to “How large should we train them?” Scaling laws use empirical relationships to describe how model performance changes with parameter count, training data, and compute. Kaplan and colleagues observed that, across a broad range of scales, language-model loss has an approximate power-law relationship with these variables [9]. A power law here means that as resources increase proportionally, loss changes at a comparatively stable but nonlinear rate, giving the returns from scaling some predictability.

Compute-optimal training asks how parameter count and the number of training tokens should be combined to minimize loss under a fixed compute budget. To investigate this question, the Chinchilla study by Hoffmann and colleagues trained more than 400 models with different sizes and token counts. It argued that many large language models at the time had grown their parameter counts too quickly relative to their training data [10]. Chinchilla, with 70 billion parameters and roughly 1.4 trillion training tokens, outperformed the 280-billion-parameter Gopher under a similar training-compute budget.

The two lines of work answer different questions. Scaling laws study how performance changes with scale; Chinchilla studies how parameters and data should be combined for a given compute budget. For a training team, the practical choice is not simply whether to buy more GPUs. Under the same budget, should it train a larger model, train a smaller model for longer, or add more high-quality data?

Maximal Update Parametrization (μP) and Hyperparameter Transfer #

Once model scale is chosen, hyperparameter experiments can themselves become expensive. If every learning-rate search requires retraining a large model, experimentation costs quickly become unmanageable.

Maximal Update Parametrization (μP) assigns scale rules to parameters and updates at different widths so that key training dynamics can remain comparable as a model becomes wider. Hyperparameter transfer uses this cross-width comparability: tune learning rates and other hyperparameters on a smaller proxy model, then transfer them to a larger target model. μTransfer combines these ideas [11]. The original paper reported experiments on BERT-like models from 13 million to 350 million parameters and transfers on GPT-3-style models from 40 million to 6.7 billion parameters. The range over which μP transfers successfully still depends on architecture, data, and model scale.

Mixture of Experts (MoE) and Conditional Computation #

Conditional computation allows different inputs to invoke only part of a network’s parameters. A mixture-of-experts model (MoE) organizes those parameters into expert subnetworks, then uses a gating or routing network to select a small number of experts for each input. In a dense model, the parameters in an expanded layer generally participate in the computation for every token at that layer. MoE instead selects and activates only a few experts per input, allowing total parameter count to be much larger than the number of parameters used for each token. Sparsely-Gated MoE and Switch Transformer extended this approach to large-scale models [12][13].

Conditional routing also creates system problems, including load balancing, expert collapse, and all-to-all communication in expert parallelism. While MoE reduces the parameter computation required for each token, it introduces additional complexity in routing and cross-device communication, making both part of the scaling problem.

In model development, these methods help teams use the same GPU budget more deliberately. A team can allocate resources more sensibly between parameter count and training data, move some tuning experiments to proxy models, and expand model capacity through conditional computation. Better resource use can create room for more frequent updates to translation models, search systems, customer-service assistants, and other specialized applications. Product pricing still depends on hardware costs, service utilization, and business strategy. As models continue to grow, long-sequence computation and memory pressure become the next bottleneck.

III. Longer Context Turns Computation, Memory, State, and Position into Bottlenecks #

The Transformer enabled scaling but introduced new limits. As context grows, at least four problems must be distinguished: the cost of computing attention scores between tokens, data movement within a GPU, the way history is stored, and positional representations beyond the training range.

FlashAttention and Data Movement #

The computational cost of standard attention grows quadratically with sequence length. Even without changing the mathematical definition of attention, execution faces another bottleneck: data movement between high-bandwidth memory (HBM) and on-chip static random-access memory (SRAM).

FlashAttention changes the order of computation through tiling. It divides the attention matrix into blocks small enough to fit in on-chip SRAM, computes and combines those blocks, and avoids repeatedly writing the full intermediate attention matrix back to HBM. This reduces inefficient reads and writes while producing the same result as standard attention [14]. FlashAttention addresses input/output (I/O) within a single attention operator. How an inference service allocates key-value cache (KV cache) across requests is a runtime state-management problem, which Section VI will revisit.

Floating-point operations (FLOPs) therefore do not determine wall-clock time on their own. A GPU may be waiting not for the next multiplication but for data to arrive from a slower memory tier. Only when an algorithm accounts for the hardware memory hierarchy can an advantage in theoretical complexity become an actual speedup.

Rotary Position Embedding (RoPE) and Context Extension #

Even when sufficient compute is available for attention, the model still needs to know where each token appears. Positional encoding injects sequence position into the model so that the same token can have different representations at different locations. Rotary Position Embedding (RoPE) represents position as two-dimensional rotations applied to the query and key, allowing their inner product to depend naturally on the relative position \(m-n\) [15]. Its value is not that it removes computation, but that it reformulates positional relationships through rotational geometry.

When an inference sequence extends far beyond the training length, RoPE’s frequency structure may enter ranges the model has never seen. YaRN (Yet another RoPE extensioN) extends the context window by scaling different frequencies differently; the original work included experiments extending context to 128,000 tokens [16].

LongRoPE further combines non-uniform positional interpolation with progressive extension; in the paper’s experiments, it extended pretrained models to context windows of 2.048 million tokens [17]. Moving from hundred-thousand-token to million-token contexts usually requires more than one positional-encoding technique: long-context training, efficient attention implementations, memory management, and inference engineering must also work together. YaRN and LongRoPE primarily address positional extension within that system. Positional representation and sequence computation still have to be co-designed.

State Space Models (SSMs) and Selective Memory #

Another line of work does not continue optimizing attention between tokens. It reconsiders how history should be stored. A state space model (SSM) summarizes history in a fixed-dimensional state that updates recurrently with the input. During autoregressive inference, it need not retain the keys and values for every earlier token in a KV cache or compute the current token’s attention over the entire history. Its recurrent state therefore does not grow with context length in the way a standard attention KV cache does; training still incurs memory costs from intermediate activations and parallel scans. Traditional recurrent models also compress history into a finite state, but they are difficult to parallelize and can lose long-range dependencies.

High-order Polynomial Projection Operators (HiPPO) study how a fixed-dimensional state can continue approximating an ever-growing history. HiPPO projects the past signal onto an orthogonal polynomial basis, such as Legendre polynomials, and updates the coefficients online, providing a mathematical foundation for structured state space models [18]. Its original experiments included permuted MNIST, trajectory classification, and long-sequence benchmarks—far removed from the scale of today’s large language models.

Mamba introduces an input-dependent selection mechanism into state space models and uses a hardware-aware parallel scan to improve GPU training efficiency [19]. Based on the current token, the model can decide which information enters the state and which information is forgotten. The main language-model experiments in the original Mamba paper reached 3 billion parameters, demonstrating a sequence-modeling path that has developed alongside the Transformer.

The Transformer explicitly computes attention between tokens; Mamba uses an input-dependent selection mechanism to compress history into a recurrent state. They allocate the costs of computation, memory, and parallelism differently.

For users, the value of long context is not simply that a chat window can hold more text. A contract-analysis tool can accept more complete agreements and attachments; a coding assistant can refer to more source files and edit history; a meeting assistant can trace discussion threads and earlier decisions across a longer transcript. FlashAttention, RoPE extensions, and Mamba address long-sequence bottlenecks through data access, positional representation, and historical state, respectively. But accepting more content does not guarantee that a model will find the key facts or answer correctly. Retrieval, information organization, and verification still determine practical performance.

IV. Generative Models Must Learn How to Move from Noise to Data #

A deeper change in generative modeling concerns the design of the generative path: as a simple distribution evolves into the target data distribution, what stochastic process or continuous path should the model follow?

Diffusion Models and Probability Flow #

A denoising diffusion probabilistic model (DDPM) gradually adds Gaussian noise to data, then learns to reverse that process [20]. This reformulates generation as a series of local denoising tasks and helped make diffusion models competitive again in modern image generation.

The score function is the gradient of the log probability density with respect to the data; intuitively, at a given noise level, it points toward regions of higher probability. Song and colleagues unified diffusion and score-based models within a stochastic differential equation (SDE) framework and derived the corresponding probability flow ordinary differential equation (probability flow ODE) [21]. The probability flow ODE uses deterministic trajectories to produce the same marginal probability distributions as the corresponding SDE, so the same distributional evolution can be understood either as a reverse-time stochastic process or as a deterministic continuous flow.

Optimal Transport (OT) and Flow Matching #

Alongside diffusion, optimal transport (OT) asks a different question: given a distance or another transport cost, what is the least costly way to move probability mass from one distribution to another? Cuturi introduced entropy regularization, allowing OT to be computed more efficiently with Sinkhorn iterations [22]. The original experiments focused on MNIST and other histogram-based tasks. Although far removed from modern generative models, this work provided computational tools later used to study geometric relationships between probability distributions in generative modeling.

A continuous normalizing flow (CNF) uses an ordinary differential equation to describe how samples change continuously over time. Flow Matching directly learns a vector field within a CNF:

\[ \frac{dx_t}{dt}=v_t(x_t), \]

A vector field assigns a direction and speed to every position at each time, allowing a simple distribution to follow a continuous path into the target distribution [23]. Flow Matching can use diffusion-style probability paths or optimal-transport displacement interpolation. In this framework, the line from diffusion models through score-based SDEs to the probability flow ODE, and the line based on OT displacement interpolation, can both be expressed through continuous vector fields. Flow Matching’s direct contribution is to redesign and learn the path between distributions. In the original experiments, conditional probability paths built from optimal-transport displacement interpolation produced faster sampling results. For the models and data examined in the original paper, this result indicates that path geometry can affect numerical-solution efficiency; the number of steps required in practice still depends on the learned vector field and the solver.

The design of that path affects how long users wait for results. If a path is easier to solve numerically and preserves quality in fewer steps, design tools can preview more image options quickly, merchants can reduce the cost of experimenting with product assets, and video teams can revise storyboards and visual styles more often. Fewer sampling steps may also reduce the time each task occupies a GPU, creating room to lower service costs. Final speed and quality, however, depend jointly on the path, solver, model architecture, hardware, and training recipe.

V. Post-Training Can Remove Components by Rewriting the Objective #

After pretraining, foundation models still need post-training with human preferences or verifiable rewards. A traditional reinforcement-learning-from-human-feedback (RLHF) pipeline often contains a policy model, reference model, reward model, value model or critic, and Proximal Policy Optimization (PPO). Depending on the data and objective, researchers can re-express the relationship between reward and policy or construct advantages from relative comparisons among responses, reducing reliance on some of these components.

Direct Preference Optimization (DPO) and the Policy Objective #

A preference pair records which of two responses to the same prompt is preferred. The Bradley–Terry preference model represents the probability that one response is preferred through the difference between their rewards; the reference policy is a baseline that constrains how far the new policy may move. Direct Preference Optimization (DPO) uses the relationship between this preference model and reward maximization with a Kullback–Leibler (KL) divergence constraint to express reward as a log-probability ratio between the policy and reference policy [24].

Using this equivalence, DPO turns the conventional “preference data → reward model → reinforcement learning” pipeline into a policy optimization objective defined directly over preference pairs. It therefore avoids separately training an explicit reward model or running PPO.

DPO’s applicability depends on the preference data, reference policy, and training distribution. Reward modeling still has a role when online exploration or explicit reward evaluation is required.

Group Relative Policy Optimization (GRPO) and Within-Group Comparison #

PPO typically uses a value model or critic to estimate the expected return of a state or response, then uses an advantage value to measure how much better the current action is than that baseline. Group Relative Policy Optimization (GRPO) generates a group of responses to the same prompt and constructs advantage estimates from their relative rewards, reducing reliance on an independent critic [25].

DeepSeekMath was built on a 7-billion-parameter model. Before reinforcement learning with GRPO, the base model underwent continued pretraining on roughly 120 billion math-related tokens. The 120-billion-token figure therefore describes the data scale of the base model, while the evidence for GRPO comes from the subsequent reinforcement-learning experiments on that 7-billion-parameter model. Later engineering extensions to larger reasoning models represent a new level of evidence.

At the product level, post-training changes how a model answers. A customer-service assistant must follow an organization’s tone and operational boundaries; a model in an educational application must explain ideas clearly and step by step; a mathematics or coding assistant must be trained to produce more verifiable results. DPO and GRPO require different data conditions. When a team has offline preference pairs, DPO can learn directly from those comparisons. When a team can generate and score a group of responses to the same prompt, GRPO can use relative rewards within the group. The two methods simplify preference optimization and reward-based reinforcement learning in different ways, reducing some implementation and computational burdens. What the model actually learns still depends on preference data, reward rules, and evaluation methods.

VI. Inference Systems Can Reorganize State, While Optimizers Can Exploit Matrix Structure #

The methods in the first five sections draw on numerical linear algebra, optimization, probability, geometry, and dynamical systems. Researchers bring these mathematical ideas into modern AI through a longer chain: identify structure, reformulate the problem, design a scalable algorithm, and adapt that algorithm to real hardware and training systems.

PagedAttention and KV Cache #

A model that can run is not necessarily able to serve real requests reliably or economically. Artificial intelligence infrastructure—commonly called AI Infrastructure or AI Infra—covers the system conditions on which model execution depends: how GPU memory is allocated, how requests are scheduled, how caches are reused, and how multiple GPUs work together.

The KV cache stores keys and values already computed for earlier tokens so that generating the next token does not require recomputing the entire history. PagedAttention borrows virtual memory and paging from operating systems. It maps a logically contiguous KV cache onto non-contiguous blocks of physical GPU memory and allocates them on demand as a sequence grows [26]. It preserves the mathematical definition of the Transformer and attention while changing how inference state is stored and allocated.

My earlier article, Optimizing LLM Inference: The Role of PagedAttention in Efficient GPU Memory Management, discusses continuous batching, prefix caching, and prefill–decode disaggregation in more detail. The end-to-end gains of the vLLM inference framework come from several layers working together, including memory management, scheduling, and kernels; PagedAttention is responsible for KV-cache management within that system. The benefits of prefill–decode disaggregation depend on request distributions, resource configuration, and service objectives.

Beyond runtime state, optimizers on the training side continue to exploit matrix structure in parameters and updates.

Muon and Matrix Updates #

AdamW primarily applies element-wise adaptive scaling, but many weights in neural-network hidden layers naturally form two-dimensional matrices. Here, matrix orthogonalization means adjusting an update matrix toward a semi-orthogonal form. The Newton–Schulz matrix iteration approximates that transformation through repeated matrix multiplications. Muon (MomentUm Orthogonalized by Newton–Schulz) first produces a momentum update, then approximately orthogonalizes it with this iteration, attempting to exploit matrix structure directly in the optimizer [27]. Muon is intended primarily for matrix-valued hidden-layer weights; embeddings, output heads, biases, and other low-dimensional parameters are generally still handled by optimizers such as AdamW.

Muon shows that the process of identifying structure and reformulating a problem is still continuing. The public DeepSeek-V3 technical report states that its pretraining used AdamW [28], so its training results cannot be treated as evidence validating Muon. Kimi K2 subsequently used MuonClip, which adds a QK-Clip stabilization mechanism, to pretrain a mixture-of-experts model with roughly one trillion total parameters and 32 billion activated parameters per token on 15.5 trillion tokens [29]. This demonstrates that Muon-based optimization has entered large-scale training practice, but the evidence applies to the modified MuonClip rather than establishing that the original Muon works broadly across architectures and training recipes.

The response time users experience, and the inference cost borne by service providers, also depend on how the inference system manages and schedules compute. If a GPU can allocate KV cache more flexibly, it can reduce memory fragmentation and accommodate more requests of different lengths. The system may therefore increase serving capacity under high concurrency and shorten queueing time for some requests; conversations of varying lengths also need not reserve large contiguous memory regions in advance. PagedAttention improves state management within this system. Long conversations still require more KV cache, and the end-to-end experience still depends on the complete system.

Conclusion: Where Might the Next Breakthrough Come From? #

Looking across these works, the important pattern is not that one method ultimately wins, but that scaling bottlenecks keep moving. The Transformer improved parallelism in sequence computation but created computational and memory pressure in attention. MoE reduced the number of parameters activated per token but shifted some complexity into routing and communication. PagedAttention reduced waste from KV-cache reservation and fragmentation, while long conversations still occupy more cache. When one method eases a local bottleneck, new costs and constraints may appear at another layer.

If this pattern continues, the next phase of foundation-model research may not revolve only around parameter count and floating-point operations. Researchers will also need to consider how data move, how state is stored, how parameters are activated selectively, how generative paths are solved, and how model capabilities are validated in real settings. Co-design across model architecture, optimization algorithms, hardware characteristics, and inference runtimes may become even more important. A method should be evaluated not only by the computation it reduces under ideal conditions, but also by end-to-end latency, memory use, communication cost, and reliability.

This shift also raises a more useful set of questions. Does the current task contain low-dimensional, sparse, geometric, or dynamic structure that has not yet been used? Is a seemingly indispensable computation merely a consequence of an inherited parameterization or system assumption? Under what conditions does a method work, and where does it move the cost? Only by placing a local method back into the full training or inference system can we tell whether it improves the system as a whole or merely moves the burden elsewhere.

Future foundation models may continue to grow, but a more consequential change may be that they allocate parameters, state, and computational paths more selectively and validate their outputs more reliably. The next major breakthrough may still begin with the same question: which assumptions about computation, representation, or systems that seem self-evident today can actually be redefined?

Appendix I: Structured Methods Also Appear in Sparse Features and Graph Learning #

Superposition and Sparse Autoencoders #

Superposition means that multiple features share a limited number of representational dimensions, allowing a model to represent more features than the dimensionality of its space. Toy Models of Superposition proposed that, when features are sufficiently sparse, a high-dimensional space can support such sharing through many approximately orthogonal directions [30]. The original work used very small synthetic rectified linear unit (ReLU) networks, so its evidence concerns mechanisms observed in toy models.

A sparse representation activates only a small number of latent features for each input. A sparse autoencoder (SAE) attempts to find such a feature basis within dense activations. Cunningham and colleagues applied this method to activations from real language models and found that some learned features were easier to interpret than individual neurons [31]. Current evidence mainly supports analysis of local features in the activations of particular language models.

Graph Neural Networks (GNNs) and Relational Propagation #

A graph neural network (GNN) operates on data composed of nodes and edges rather than on a regular grid. One of its core operations is neighborhood aggregation: each node gathers information from adjacent nodes along graph edges, so its updated representation contains both node features and local relationships. Defferrard and colleagues used Chebyshev polynomials to approximate spectral filters, avoiding a full eigendecomposition and localizing graph convolution [32]. Kipf and Welling later simplified graph convolution further [33]. LightGCN subsequently applied graph propagation to the user–item bipartite graph in recommendation systems [34].

These methods extend the discussion of effective structure to dense activations and graph relationships. This article treats them as parallel lines in mechanistic interpretability and graph learning that supplement the main account of foundation-model scaling.

Appendix II: Representative Methods, Reformulations, and Evidence Boundaries #

Representative period	Method	Structure identified	What was reformulated	Engineering impact	Boundary of original evidence
2014–22	RandNLA / non-convex optimization / intrinsic dimension / LoRA	Tasks may depend on low-dimensional structure, and optimization spaces may have favorable geometry	Full matrices or full parameter updates become approximations, structured analyses, or low-rank updates	Reduces some large-matrix computation and task-adaptation costs	Non-convex results require specific conditions; low-rank effectiveness is not a universal theorem
2017–22	Transformer / scaling laws / Chinchilla / μP / MoE	Sequence relationships, returns to scale, and resource allocation contain exploitable structure	Recurrence becomes parallel matrix computation; blind expansion becomes budget planning and conditional routing	Makes large-model training, tuning, and expansion more predictable	Empirical laws and cross-scale transfer depend on the model, data, and training setup
2020–24	FlashAttention / RoPE / YaRN / LongRoPE / HiPPO / Mamba	Long sequences involve I/O, position, state, and historical representation	Reorders data access, rescales positional frequencies, or uses finite state to summarize history without explicitly computing attention between tokens	Supports longer and more efficient sequence processing	Million-token context depends on the full system; alternative sequence architectures are still evolving
2013–23	DDPM / score SDE / OT / Flow Matching	Generation can be expressed as a stochastic process, probability path, or vector field	Extends discrete denoising into continuous distributional evolution	Provides new training and sampling paths	Few-step generation also depends on solvers, distillation, and training recipes
2023–24	DPO / GRPO	Preferences or within-group rewards can enter the policy objective directly	Reduces dependence on reward models, PPO, or an independent critic	Simplifies different post-training systems	Results still depend on data quality, reward reliability, and distribution shift
2023	PagedAttention / AI Infrastructure	Logically contiguous inference state need not occupy contiguous physical memory	Replaces contiguous preallocation with paging and on-demand allocation	Makes KV-cache management more flexible and creates room to scale serving	vLLM’s end-to-end gains also come from scheduling, kernels, and system co-design
2024–	Muon / MuonClip	Hidden-layer weights and updates have matrix structure	Moves from element-wise scaling to orthogonalizing update matrices	May change training efficiency and optimization dynamics	MuonClip has entered trillion-parameter training; cross-architecture generality remains unproven
2022–24	Superposition / SAE	Dense representations may contain sparse feature structure	Replaces direct neuron inspection with sparse-feature analysis	Provides tools for mechanistic interpretability	The main evidence for feature superposition comes from toy models; SAE research is still developing
2016–20	GNN	Data relationships form non-Euclidean graph structure	Replaces regular convolution with local propagation on graphs	Supports recommendation, node classification, and related tasks	Original evidence centers on graph-structured tasks, a parallel line to foundation-model scaling

References #

[1] Woodruff, D. P. (2014). Sketching as a Tool for Numerical Linear Algebra. Foundations and Trends in Theoretical Computer Science, 10(1–2), 1–157.

[2] Aghajanyan, A., Gupta, S., & Zettlemoyer, L. (2021). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL 2021.

[3] Hu, E. J., et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.

[4] Ge, R., Lee, J. D., & Ma, T. (2016). Matrix Completion Has No Spurious Local Minimum. NeurIPS 2016.

[5] Ge, R., Huang, F., Jin, C., & Yuan, Y. (2015). Escaping From Saddle Points — Online Stochastic Gradient for Tensor Decomposition. NeurIPS 2015.

[6] Gunasekar, S., Woodworth, B. E., Bhojanapalli, S., Neyshabur, B., & Srebro, N. (2017). Implicit Regularization in Matrix Factorization. NeurIPS 2017.

[7] Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017.

[8] Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. ICLR 2019.

[9] Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.

[10] Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. NeurIPS 2022.

[11] Yang, G., et al. (2022). Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. ICLR 2022.

[12] Shazeer, N., et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.

[13] Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR, 23.

[14] Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022.

[15] Su, J., et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 568, 127063. First released in 2021.

[16] Peng, B., et al. (2024). YaRN: Efficient Context Window Extension of Large Language Models. ICLR 2024. First released in 2023.

[17] Ding, Y., et al. (2024). LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens. arXiv:2402.13753.

[18] Gu, A., Dao, T., Ermon, S., Rudra, A., & Ré, C. (2020). HiPPO: Recurrent Memory with Optimal Polynomial Approximations. NeurIPS 2020.

[19] Gu, A., & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.

[20] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020.

[21] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021.

[22] Cuturi, M. (2013). Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances. NeurIPS 2013.

[23] Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., & Le, M. (2023). Flow Matching for Generative Modeling. ICLR 2023. First released in 2022.

[24] Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. NeurIPS 2023.

[25] Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.

[26] Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.

[27] Jordan, K. (2024). Muon: An Optimizer for Hidden Layers in Neural Networks. Online technical note, published December 8, 2024. See also the official implementation.

[28] DeepSeek-AI. (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437.

[29] Kimi Team. (2025). Kimi K2: Open Agentic Intelligence. arXiv:2507.20534. Revised in 2026.

[30] Elhage, N., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread.

[31] Cunningham, H., Ewart, A., Riggs, L., Huben, R., & Sharkey, L. (2024). Sparse Autoencoders Find Highly Interpretable Features in Language Models. ICLR 2024. First released in 2023.

[32] Defferrard, M., Bresson, X., & Vandergheynst, P. (2016). Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. NeurIPS 2016.

[33] Kipf, T. N., & Welling, M. (2017). Semi-Supervised Classification with Graph Convolutional Networks. ICLR 2017.

[34] He, X., et al. (2020). LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. SIGIR 2020.