OpenAI has published how well its models predict code and is asking other AI labs to report the same measure. Perplexity here is a metric, not the company Perplexity. In a post titled "Perplexity and Intelligence" dated Nov. 22, Chief Scientist Jakub Pachocki argues that perplexity, or equivalently cross-entropy loss, should be a metric to compare between developers and to inform shared safety standards.

What perplexity means

Perplexity measures how well a model predicts the next word-piece (token) of a text on average. Lower perplexity means better predictions. The post says OpenAI generally tracks the log of perplexity, also called cross-entropy loss. In plain words, it is a score for how surprised the model is by real text, and a lower score is better.

The numbers OpenAI published

All figures are log-perplexity on OpenAI's internal codebase, which the post says is the dataset OpenAI relies on most today to gauge pretraining progress. The post notes that code is a poor proxy for the general ability of older models like GPT-2 and GPT-3:

  • GPT-3.5: about 0.99. The post says it was capable of fluent conversation and formed the basis of the original ChatGPT.
  • GPT-4: about 0.91. The post says it was proficient at chain-of-thought reasoning.
  • An unnamed model: about 0.76. The post says that once OpenAI upgraded to a model at this level, it "started producing groundbreaking results" on open math problems. It does not name the model.

The post says that below 0.8, OpenAI has seen "transformative capabilities" in fields like cybersecurity and mathematics. It says to expect more as models enter a 0.68 to 0.7 window, and adds: "I don't know what those capabilities will be."

Why he proposes limits

The post warns that as models enter that 0.68 to 0.7 window, "we should also expect further degradation in currently dominant forms of alignment auditing and further increases in evaluation and training awareness." That is a concern the post raises alongside its proposal for limits. It also calls for shared norms and transparency standards for evaluating such models and red-teaming safety cases before first use.

The proposal: common limits

Pachocki proposes "common limits, based on perplexity and other measurements," that developers commit not to cross until the ability to understand and monitor AI behavior reaches a concrete milestone. His example milestone is a successor to chain-of-thought (CoT) monitors, tools that watch a model's written-out reasoning. This is OpenAI's proposal, not an agreement between labs. The post gives no numeric limit, no timeline and no list of labs that have signed on. It fits Pachocki's Sept. 6 essay, where he wrote that he expects and hopes "voluntary slowdowns" will become commonplace until shared safety bars are established.

The 2027 point is hedged

The post says "as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements." That is a possibility Pachocki raises, not a forecast. OpenAI says the steep acceleration in its loss curve in 2026 is mostly the result of long-term research and compute investment, and only in a smaller part of automation, though the most recent models have already made meaningful design contributions. He also says he expects AI to help with alignment and monitoring but thinks that progress is unlikely to be fast enough to allow the full potential acceleration in pretraining.

The limits the post names

The post lists several weaknesses of the metric:

  • Comparing labs is hard. Developers use different evaluation datasets and data mixtures.
  • It is detached from tasks people care about. The post says perplexity does not directly correspond to a task we care about.
  • It misses advances outside pretraining. The post says chain-of-thought reasoning does not reduce perplexity, even though it sharply raises capability.
  • The floor is unknown. The post says a loss of zero is not expected to be reachable and that the inherent entropy of the data is unknown.

OpenAI's own numbers come from its internal codebase. The post says publishing the numbers gives competitors a clearer picture of where OpenAI stands.

What this means

Our read: this is a company publishing a measurement and asking rivals to match it, with no outside check yet. For readers, the practical point is that the headline claims, the 0.76 level and the 2027 possibility, come from OpenAI's own post and are not independently verified. For related coverage, see our reports on OpenAI moving frontier training offline and on the Claude Riemann hypothesis claim.

What we checked

We read OpenAI's post and Pachocki's Sept. 6 essay on Nov. 22. We did not check the figures independently, and we did not use social-media reposts or aggregator summaries for any claim.