Higher Scores, Worse Models? The Theoretical Limits of LLM Alignment

Two people ask the same large language model the exact same question—“I have a headache. How much pain medication should I take?”—but receive very different responses. For the first user, the model gives a specific dose without asking for more information. The second user had mentioned in conversation a few days earlier that they were taking other medication; in response to the same question, the model asks about age, medical history, and current medication before recommending that the user consult a doctor or pharmacist.

The second response looks more cautious and safer. How can developers make a model choose this kind of response more often? If the standard used to judge which response is better is itself biased, does the model learn better behavior—or merely learn to satisfy the scoring rule?

An Apple research team examines this tension in Theoretical Limits of Language Model Alignment[1]. They ask two core questions: Given a fixed scoring rule, how much can a model improve, and how far must it depart from its original response habits? And if that scoring rule is flawed, can pushing for higher scores actually make the model behave worse?

The paper does not attempt to explain every alignment method. It studies a common setting in which training raises a reward score while constraining how far the model can move from its original behavior.

Their findings explain both the ceiling of alignment and how score optimization goes astray. First, an estimable frontier links score gains to behavioral shift, showing how far alignment can go. Second, aggressive optimization amplifies flaws in the scoring rule, showing how a model can be pushed off course. For everyday users, the takeaway is just as direct: a response that sounds fluent, decisive, and authoritative is not necessarily more accurate.

1. What Does Alignment Actually Change? #

To understand the paper, we first need to define alignment in this context. Pretraining teaches a model which tokens are most likely to follow the current text. Yet a model that excels at text continuation does not inherently know when to refuse a dangerous request, when to admit uncertainty, or why being genuinely helpful matters more than sounding confident.

From different responses to probability redistribution. After pretraining, developers continue training the model with demonstrations, human preferences, or safety rules. This stage is usually called post-training. In this setting, alignment does not inject new capabilities from scratch; rather, it redistributes the probabilities of different response strategies that the model can already produce for the same prompt. Common methods include supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF)[2]. SFT uses demonstrations to show the model how it should respond. RLHF typically trains a reward model on human preference data and then uses that reward signal to update the generation policy.

To set aside differences in conversation history and isolate how training changes model choices, consider two candidate responses to the exact same medication prompt:

• A: Gives a specific dose without asking about age, medical history, or other medication.
• B: Explains the risk, asks for the necessary information, and recommends consulting a doctor or pharmacist.

To make this concrete with a toy example (for illustration, not from the paper’s experiments), suppose the base model assigns probabilities \(Q=(0.8,0.2)\) to A and B, and alignment shifts those probabilities to \(P=(0.5,0.5)\). The model has not acquired a new language skill; it simply suppresses the risky response A and boosts the cautious response B.

Why constrain how far the model moves? Training still needs a standard that determines which responses should receive more probability. We cannot directly write down an ideal reward for every input and response, so developers often train a reward model \(r(x,y)\) as a proxy. It takes a prompt \(x\) and response \(y\), then produces a score. That proxy introduces a fundamental catch: the reward model only captures preferences as expressed by human annotators, heuristic rules, or a judge model—not necessarily the true outcome we care about.

If training only pursues higher scores, the model may move far from its original response distribution to satisfy the reward. The paper therefore studies a specific objective: increase expected reward while using KL divergence to constrain the new policy’s deviation from the base model. KL divergence (Kullback–Leibler divergence) measures the difference between two probability distributions[3]. If \(P\) is the new policy and \(Q\) is the base policy, the KL of the new policy relative to the base policy is

\[ \begin{aligned} \operatorname{KL}(P\Vert Q) &= \sum_y P(y)\log\frac{P(y)}{Q(y)}. \end{aligned} \]

KL divergence is not an ordinary distance. In general, \(\operatorname{KL}(P\Vert Q)\neq\operatorname{KL}(Q\Vert P)\); it is not symmetric. Using \(P=(0.5,0.5)\) and \(Q=(0.8,0.2)\),

\[ \begin{aligned} \operatorname{KL}(P\Vert Q) &= 0.5\ln 0.625+0.5\ln 2.5 \\ &\approx 0.223\ \text{nats}. \end{aligned} \]

A nat is a unit of information based on the natural logarithm. A larger value means that the aligned response distribution has moved farther from the base model. KL tells us how much the distribution changed, not whether the change was beneficial.

The paper writes the optimization objective as

\[ \begin{aligned} \max_{\pi} \mathbb E_{x,\,y\sim\pi(\cdot\mid x)}[r(x,y)] &- \lambda\, \mathbb E_x\!\left[ \operatorname{KL}\bigl(\pi(\cdot\mid x)\Vert\pi_{\text{base}}(\cdot\mid x)\bigr) \right]. \end{aligned} \]

The parameter \(\lambda>0\) controls the trade-off between raising reward and staying close to the base model. A larger \(\lambda\) is more conservative; a smaller \(\lambda\) lets the policy change more in pursuit of reward. The corresponding optimal policy is

\[ \begin{aligned} \pi_{r,\lambda}(y\mid x) &= \frac{ \pi_{\text{base}}(y\mid x)\exp(r(x,y)/\lambda) }{ Z_{r,\lambda}(x) }. \end{aligned} \]

The normalization constant is

\[ \begin{aligned} Z_{r,\lambda}(x) &= \mathbb E_{Y\sim\pi_{\text{base}}(\cdot\mid x)} \left[\exp(r(x,Y)/\lambda)\right]. \end{aligned} \]

Alignment therefore exponentially reweights the base model’s response distribution by reward (often called exponential tilting in statistics): high-reward responses gain probability, while low-reward responses lose probability. Every result that follows comes from this framework; the paper does not make a universal claim about all alignment methods.

2. How Far Can Reward Go Without Training? #

The first section explained how alignment redistributes response probabilities and how KL measures deviation from the base model. The next question is how much reward can improve once the reward function is fixed, and how far the model must move to obtain that improvement.

If you want to skip the estimator derivations, jump straight to the five-step sampling workflow below. The core takeaway of this section is simple: teams do not need to run reinforcement learning first—they can estimate the theoretical alignment ceiling directly from base-model samples.

Each value of \(\lambda\) produces a reward gain and a KL divergence. Connecting these optimal pairs gives the reward–KL frontier. Let \(\pi_{r,\lambda}\) denote the optimal aligned policy above. For each fixed prompt \(x\), its reward gain over the base model satisfies

\[ \begin{aligned} \mathbb E_{\pi_{r,\lambda}}[r] &- \mathbb E_{\pi_{\text{base}}}[r] \\ &= \lambda\left[ \operatorname{KL}(\pi_{r,\lambda}\Vert\pi_{\text{base}}) + \operatorname{KL}(\pi_{\text{base}}\Vert\pi_{r,\lambda}) \right]. \end{aligned} \]

The sum in brackets is the Jeffreys divergence. Using the earlier numbers, the reverse direction is

\[ \begin{aligned} \operatorname{KL}(Q\Vert P) &= 0.8\ln 1.6+0.2\ln 0.4 \\ &\approx 0.193. \end{aligned} \]

The Jeffreys divergence is therefore approximately \(0.223+0.193=0.416\). The two directions yield different numbers, showing that KL direction is not a mere notational detail. More importantly, the identity ties reward gain directly to distributional shift in both directions—how far the aligned policy moves from the base model, and how far the base model lies from the aligned policy.

The paper’s more practical result is that the theoretical frontier can be estimated from base-model samples without training a new policy:

1. Sample prompts \(x\) from the prompt distribution.
2. For each \(x\), sample responses \(Y_1,\ldots,Y_n\) from the base model.
3. Score the responses with the reward model.
4. For several values of \(\lambda\), compute weights \(\exp(r/\lambda)\), the conditional covariance, and the normalization constant.
5. Estimate reward gain and KL together, then plot the reward–KL Pareto frontier.

This Monte Carlo procedure uses finite random samples to approximate the full distribution. Before investing in reinforcement learning, a training team can estimate how much reward improvement is theoretically available within a chosen KL range.

Technical detail: why samples are sufficient. The paper writes the reward gain as a conditional covariance:

\[ \mathbb E_x\left[ \frac{ \operatorname{Cov}_{Y\sim\pi_{\text{base}}(\cdot\mid x)} \left(r(x,Y),\exp(r(x,Y)/\lambda)\right) }{ \mathbb E_{Y\sim\pi_{\text{base}}(\cdot\mid x)} \left[\exp(r(x,Y)/\lambda)\right] } \right]. \]

Note that both the covariance and the normalization constant must be computed within the base model’s conditional distribution for each individual prompt \(x\) before averaging across prompts. In practice, you cannot simply pool responses from different prompts into a single flat distribution.

The paper also derives an estimator for the KL of the aligned policy relative to the base model:

\[ \begin{aligned} \widehat{\operatorname{KL}} \left(\pi_{r,\lambda}\Vert\pi_{\text{base}}\right) &= \frac{\widehat{\mu}_r+\widehat{\Delta}_n(r,r)}{\lambda} -\log\widehat Z_{r,\lambda}. \end{aligned} \]

Here, \(\widehat{\mu}_r\) is the average reward of base-model samples, \(\widehat{\Delta}_n(r,r)\) estimates reward gain (the paper uses two arguments to distinguish the training reward from the evaluation reward, both equal to \(r\) here), and \(\widehat Z_{r,\lambda}\) is the sample mean of \(\exp(r/\lambda)\). The quantities are estimated for each fixed prompt before aggregation over the prompt distribution. The estimator follows from \(\operatorname{KL}(\pi_{r,\lambda}\Vert\pi_{\text{base}})=\mathbb E_{\pi_{r,\lambda}}[r]/\lambda-\log Z_{r,\lambda}\).

For fixed \(\lambda\), estimates usually stabilize with fewer than one thousand samples per prompt. Frontiers from 1,000 and 10,000 samples per prompt nearly overlap across most of the appendix plots. Estimation becomes harder when \(\lambda\) is very small because the exponential weights concentrate on a few responses.

The frontier can therefore estimate the return on training before training begins. If even the optimum yields little additional reward within a given KL range, an expensive reinforcement learning run cannot overcome that limit. If an actual run falls far below the frontier, optimization, sampling, or implementation becomes the more likely bottleneck.

The frontier only answers how far a given reward function can be optimized. It does not guarantee that the reward function measures the quality we actually care about. Once proxy reward differs from the true objective, the question changes from how far the model can go to whether it is moving quickly in the wrong direction.

3. Why Can Higher Scores Make Models Worse? #

The previous section assumed that reward-model scores accurately represent the outcome we want. In practice, the scores approximate that outcome through human labels, rules, or other models, so errors are unavoidable.

The paper denotes the desired reward by \(r^*(x,y)\) and the proxy reward used in training by \(r(x,y)\). Their difference is the preference residual:

\[ \mathcal N(x,y)=r(x,y)-r^*(x,y). \]

Suppose training under the proxy reward \(r\) and the ideal reward \(r^*\) yields policies \(\pi_{r,\lambda}\) and \(\pi_{r^*,\lambda}\). The paper measures how much true reward \(r^*\) the proxy-trained policy loses compared with the ideal policy:

\[ \begin{aligned} G_x(\lambda) &= \mathbb E_{Y\sim\pi_{r^*,\lambda}(\cdot\mid x)}[r^*(x,Y)] \\ &\quad - \mathbb E_{Y\sim\pi_{r,\lambda}(\cdot\mid x)}[r^*(x,Y)]. \end{aligned} \]

The residual affects the final policy through terms of the form \(\exp(\mathcal N/\lambda)\). A smaller \(\lambda\) means more aggressive optimization, so exponential tilting can amplify even modest ranking errors in the proxy reward. This result is consistent with earlier work on reward-model overoptimization[4].

To see how quickly this exponential term amplifies errors, consider a simple toy example (constructed for illustration, not from the paper’s experiments). Suppose that for a given prompt, the base model assigns probability 0.5 to each of two responses:

• A is genuinely good. Its true and proxy rewards are both 1.
• B is polished but wrong. Its true reward is 0, but the proxy reward is 2.

After optimizing the proxy reward, the probability of choosing A is

\[ \begin{aligned} P_{\text{proxy}}(A) &= \frac{\exp(1/\lambda)}{\exp(1/\lambda)+\exp(2/\lambda)}. \end{aligned} \]

With the ideal reward, it is

\[ \begin{aligned} P_{\text{ideal}}(A) &= \frac{\exp(1/\lambda)}{\exp(1/\lambda)+1}. \end{aligned} \]

For \(\lambda=1\),

\[ \begin{aligned} P_{\text{proxy}}(A)&=\frac{e}{e+e^2}\approx26.9\%,\\ P_{\text{ideal}}(A)&=\frac{e}{e+1}\approx73.1\%. \end{aligned} \]

Because A has true reward 1 and B has true reward 0, a policy’s true reward equals its probability of choosing A:

\(\lambda\)	Strength	Proxy policy	Ideal policy	Gap \(G\)
1	Moderate	26.9%	73.1%	46.2%
0.2	Aggressive	0.67%	99.33%	98.66%

The base model originally chooses A with probability 50%. At \(\lambda=0.2\), proxy-reward optimization reduces this probability to 0.67%, making the policy worse than the unaligned model. The gap is

\[ \begin{aligned} G&=99.33\%-0.67\%\approx98.66\%. \end{aligned} \]

The optimizer did not malfunction—it optimized a flawed objective with ruthless efficiency. The smaller \(\lambda\) becomes, the more aggressively exponential tilting magnifies the proxy’s overvaluation of B. This is reward hacking: the model learns how to game the proxy score while drifting away from our actual intent.

The appendix identifies a deeper problem. Standard Bradley–Terry reward models fit win probabilities from score differences between two responses and usually learn only which response wins a pairwise comparison. Even after the pairwise loss is minimized, the absolute scale of the learned reward difference can differ arbitrarily from the true preference strength, so \(\mathcal N\) can remain large. The paper suggests capturing preference intensity, not only wins and losses.

The paper also proves that if \(m\) reward models are sampled independently and identically, are pointwise unbiased for every \((x,y)\), and have bounded rewards, then averaging them makes the related gap converge in expectation at \(O(m^{-1/2})\). Here, \(m\) counts reward models and remains distinct from the samples per prompt, \(n\).

These assumptions are much stronger than asking several models to vote. Models trained by different companies, on different datasets, or with the same underlying bias need not be identically distributed or pointwise unbiased. Averaging does not remove a shared blind spot.

4. How Close Are PPO, GRPO, and BoN? #

The paper compares Best-of-N (BoN), PPO, and GRPO against the theoretical frontier.

BoN does not update model parameters. The base model generates \(N\) candidates for a prompt, and the reward model selects the highest-scoring one. Its main cost occurs at inference time.

Proximal Policy Optimization (PPO) uses reward to train the model. It estimates which generation actions should become more likely while constraining each parameter update[5]. Training repeatedly samples, scores, and updates, but the trained model usually generates one response at inference time.

Group Relative Policy Optimization (GRPO) samples a group of responses for the same prompt, calculates advantages from their relative rewards, and changes the model’s future generation probabilities[6]. It reduces reliance on a separate value model. In short, BoN generates several responses and picks one; PPO and GRPO use reward feedback to change future behavior.

The paper evaluates two tasks:

• BeaverTails safety: Tests whether a model avoids harmful content in response to dangerous or sensitive prompts. The dataset has about 334,000 safety-labeled question–answer pairs (333,963) and about 362,000 expert preference pairs (361,903)[7].
• Reddit TL;DR summarization: Tests whether a model produces concise summaries that people prefer. The source version has about three million posts and author-written summaries; the public feedback data has about 65,000 candidate-summary comparisons (64,832)[8][9].

These are total dataset sizes, not the samples used by every run. The tasks cover safety and content preference, but not all alignment settings. The paper excludes Direct Preference Optimization (DPO) because DPO does not use an explicit reward model[10].

Appendix Tables 2 and 3 report the largest reward gain achieved within the paper’s KL range:

Task	Limit	BoN	GRPO	PPO
BeaverTails	2.764	2.722	2.117	2.084
TL;DR	0.387	0.385	0.282	0.214

Because the two tasks use separately trained reward models, the absolute score scales (2.764 vs. 0.387) are not directly comparable across tasks; what matters is how closely each method approaches the limit within the same task. PPO and GRPO achieve about 55% to 77% of the theoretical gain, while BoN comes close to the limit. The paper varies BoN’s \(N\) from 1 to 10,000 and estimates its KL using Beirami et al.[11].

Across both tasks and evaluation setups, simple BoN sampling gets closer to the theoretical frontier than reinforcement learning. That does not mean BoN can simply replace RL in production, however—their KL estimation methods are not fully aligned, and they pay for compute at completely different stages. Whereas BoN’s KL is computed via an analytical estimator, the paper does not specify how KL is estimated for PPO and GRPO; moreover, BoN shifts the compute burden to repeated sampling at inference time, whereas PPO and GRPO pay that cost upfront during training, leading to very different latency, throughput, and deployment trade-offs.

The paper treats a whole response as one decision and studies KL-regularized reward maximization. This differs from per-token KL in common implementations and does not cover other \(f\)-divergences (a broader family of divergences that includes KL) or every preference objective. The frontier also depends on the base model, prompt distribution, and reward function.

The careful conclusion is that simple resampling approaches the theoretical frontier in these experiments, while the tested reinforcement learning methods leave optimization headroom.

5. How Frontier Models Design Rewards and Limits #

The paper does not analyze commercial models, but its questions offer a useful lens: What represents desired behavior? How is the model trained to follow it? How is drift or capability loss controlled?

The table summarizes first-party public sources; it is not a product ranking. “Not disclosed” means that, as of November 14, 2026, I did not find directly comparable information in the cited sources.

Case	Reward / rules	Optimization	Drift control
OpenAI[12][13]	Spec rules, RM	SFT, RL	Not disclosed
Gemini[14][15][16]	Human prefs, RM	SFT, RL	Not disclosed
Anthropic[17][18]	Constitution, AI feedback	SFT, RL	Not disclosed
DeepSeek-V4[19]	Verifiers, GRM	GRPO, OPD	Toward teachers
Qwen[20][21][22]	Prefs, rules, RM	DPO, GRPO, RL	KL, pretrain grad
Kimi K2[23]	Verifiers, rubrics	Joint RL	Old-policy ratio, PTX

The table uses concise labels while keeping the original report terminology in the text below. RM denotes a scalar reward model, whereas a generative reward model (GRM) scores responses by generating a critique or evaluation rationale; on-policy distillation (OPD) distills trajectories generated by the current student toward a teacher; and PTX (pretraining mix), like the “pretrain grad” (pretrained gradients) entry in the table, mixes pretraining language-modeling objectives into RL to reduce forgetting.

OpenAI’s Model Spec organizes desired behavior around five values, while deliberative alignment uses safety policies for synthetic data, SFT, and reinforcement learning driven by a policy-aware reward model[12][13]. OpenAI also states that the Model Spec describes behavior it is working toward and that current models do not yet fully satisfy it[12]. Gemini 1.0 and 1.5 disclose reward modeling and RLHF; Gemini 1.5 uses safety-data generation inspired by Constitutional AI. The Gemini 4 Argon model card further confirms that post-training uses instruction data, reinforcement learning data, and human preference data, but does not provide a comparable KL setting[14][15][16]. Anthropic uses its constitution across several training stages, including synthetic-data generation and response ranking. Earlier Constitutional AI work uses AI feedback to construct preference signals[17][18].

DeepSeek-V4 trains experts with SFT, GRPO, rule verification, and a GRM, then combines them through OPD. OPD minimizes the KL of the student policy relative to the teacher policies (often called reverse KL in the distillation literature), rather than the KL of an aligned policy relative to a fixed base model[19]. Qwen reports a PPO KL coefficient of 0.04 and uses pretrained gradients to reduce capability loss. Qwen2.5 combines offline DPO with online GRPO; Qwen3 uses rule-based and model-based rewards, while its small-model distillation uses KL toward a teacher[20][21][22].

Kimi K2 combines verifiable rewards with self-critique rubric rewards for open-ended tasks. Its RL objective uses \(\tau\log(\pi_\theta/\pi_{\mathrm{old}})\) to constrain the probability ratio between new and old policies, while an auxiliary PTX loss helps retain high-quality data. The report does not provide a reward–KL budget directly comparable to the paper’s \(\lambda\)[23]. Qwen’s pretrained gradients and Kimi’s PTX help retain earlier capabilities, but neither is another name for a KL constraint.

Viewed through the paper’s framework, these public disclosures suggest four broader takeaways (the following are my inferences from public reports, not claims made by the vendors or the paper):

Verifiable rewards reduce some ambiguity. Tests and formal verification can create repeatable signals, but incomplete or exploitable verifiers still leave a preference residual \(\mathcal N\).

A policy changes the model only when it enters training. Specs, constitutions, and rubrics matter when they generate data, rank responses, or construct rewards. Public reports do not measure how much they reduce \(\mathcal N\).

Public evidence cannot locate products on the theoretical frontier. Published KL terms point to different references—base models, teachers, or old policies—and no cited source provides a complete comparable reward–KL curve.

Method names do not determine whether the framework applies. The complete objective and reference policy determine whether a system performs explicit reward maximization with KL regularization.

6. Avoiding Proxy Traps in Practice #

AI engineers define what counts as good even when they do not train a base model. Once a system pursues an evaluation score, model-judge rating, or completion rate, proxy error can reappear. The following points extend the paper; they are not direct paper results.

For Builders: Guarding Against Proxy Scores #

Training teams update model parameters directly with reward signals. AI engineers increasingly use existing large language models as foundation models for assistants, retrieval-augmented generation (RAG), automation, and agentic workflows. Their success criteria may be benchmark scores, ratings from another model acting as a judge, task-completion rates, user clicks, or successful tool calls. These metrics remain proxies and can diverge from business outcomes, safety requirements, and user interests.

Adapting the paper’s recommendations for training teams yields three practical principles for engineering systems:

1. Define the real objective before choosing a proxy. Training teams can estimate the reward–KL frontier before reinforcement learning to determine the available theoretical gain. AI engineers should define what the system must accomplish, what risks are acceptable, and what constitutes failure before selecting accuracy, model ratings, or completion rate. A convenient score should not silently replace the real objective.
2. Measure benefits and costs together. Training teams should report reward, \(\lambda\), and KL rather than reward alone. Even when AI engineers cannot observe a model’s internal KL, they can track task quality, error rates, safety failures, latency, cost, and manual-review load. A higher benchmark score does not establish that the complete system is more reliable.
3. Audit shared blind spots and limit the effect of errors. Training teams should examine preference intensity, out-of-distribution responses, and shared bias across reward signals, not only preference-classification accuracy. AI engineers should not treat another model’s score as independent verification. Agentic workflows that can send messages, modify data, call external systems, or execute transactions need least-privilege access, confirmation at critical steps, complete logs, and rollback mechanisms.

For Users: Fluency Is Not Fact #

Tools such as ChatGPT, Claude, and Gemini generate answers through the combined effects of pretraining, post-training, the current context, and available tools; they do not retrieve every answer from a table of verified facts. Search and tool calls can add evidence, but the final prose is still organized by the model. Polished text is not proof of factual correctness.

1. Separate presentation quality from factual reliability. A complete, confident answer with clear reasoning steps is not necessarily correct. The Kimi K2 report notes that its rubric penalizes disclaimers, self-qualification, and hesitant language while preferring clear, decisive answers. The report acknowledges that this can make the model overconfident when uncertainty would be appropriate[23].
2. Verify critical evidence independently. Users can ask for sources, assumptions, and uncertainty, but should still open primary sources, repeat calculations, or consult a professional when the result matters. Agreement among several AI tools is not independent evidence when they may rely on similar public data, evaluation practices, or common narratives.
3. Match review depth to consequence, verifiability, and reversibility. AI can be used more freely for brainstorming, rewriting, and reversible low-risk work. Medical, legal, financial, and important business decisions require primary-source verification and human confirmation. Unverified output should not directly trigger irreversible actions in external systems.

For developers, the hard problem is keeping a metric representative of the real objective and limiting damage when it fails. For users, the hard problem is deciding which outputs can be used directly and which require verification. The theoretical frontier tells us how much optimization room remains; the preference residual warns that faster movement in the wrong direction can carry the model farther from the goal.

7. References #

[1] Paes, Lucas Monteiro, Natalie Mackraz, Barry-John Theobald, and Federico Danieli. Theoretical Limits of Language Model Alignment. arXiv:2605.07105v1, 2026. https://arxiv.org/abs/2605.07105v1

[2] Ouyang, Long, et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS, 2022. https://arxiv.org/abs/2203.02155

[3] Kullback, Solomon, and Richard A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 1951. https://doi.org/10.1214/aoms/1177729694

[4] Gao, Leo, et al. Scaling Laws for Reward Model Overoptimization. ICML, 2023. https://arxiv.org/abs/2210.10760

[5] Schulman, John, et al. Proximal Policy Optimization Algorithms. arXiv:1707.06347, 2017. https://arxiv.org/abs/1707.06347

[6] Shao, Zhihong, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024. https://arxiv.org/abs/2402.03300

[7] Ji, Jiaming, et al. BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset. NeurIPS, 2023. https://arxiv.org/abs/2307.04657

[8] Stiennon, Nisan, et al. Learning to Summarize from Human Feedback. NeurIPS, 2020. https://arxiv.org/abs/2009.01325

[9] OpenAI. summarize-from-feedback: Human Feedback Data. GitHub, 2020. https://github.com/openai/summarize-from-feedback

[10] Rafailov, Rafael, et al. Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. NeurIPS, 2023. https://arxiv.org/abs/2305.18290

[11] Beirami, Ahmad, et al. Theoretical Guarantees on the Best-of-n Alignment Policy. ICML, PMLR 267:3580–3602, 2025. https://proceedings.mlr.press/v267/beirami25a.html; arXiv:2401.01879. https://arxiv.org/abs/2401.01879

[12] OpenAI. An Updated Model Spec for More Capable and Beneficial AI. 2026-10-29. https://openai.com/index/updated-model-spec/

[13] Guan, Melody Y., et al. Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv:2412.16339, 2024. https://arxiv.org/abs/2412.16339

[14] Gemini Team. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805, 2023. https://arxiv.org/abs/2312.11805

[15] Gemini Team. Gemini 1.5: Unlocking Multimodal Understanding across Millions of Tokens of Context. arXiv:2403.05530, 2024. https://arxiv.org/abs/2403.05530

[16] Google DeepMind. Gemini 4 Argon Model Card. 2026-10. https://deepmind.google/models/model-cards/gemini-4-argon/

[17] Anthropic. Claude’s New Constitution. 2026. https://www.anthropic.com/news/claude-new-constitution

[18] Bai, Yuntao, et al. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, 2022. https://arxiv.org/abs/2212.08073

[19] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 2026. https://arxiv.org/abs/2606.19348

[20] Bai, Jinze, et al. Qwen Technical Report. arXiv:2309.16609, 2023. https://arxiv.org/abs/2309.16609

[21] Qwen Team. Qwen2.5 Technical Report. arXiv:2412.15115, 2024. https://arxiv.org/abs/2412.15115

[22] Yang, An, et al. Qwen3 Technical Report. arXiv:2505.09388, 2025. https://arxiv.org/abs/2505.09388

[23] Kimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534, 2025, revised 2026 (v2). https://arxiv.org/abs/2507.20534