SOURCE-LINKED INTELLIGENCE

Next-token functional estimation

arXiv · AI, language, vision and robotics · article · Sep 17, 2026 · UTC

Suppose we observe the first $n$ points of a sequence of random variables having length $n+1$, and wish to estimate a functional of the unobserved final point and the empirical measure of the $n$ observed training points. Such next-token functionals include the probability that the next token is novel (also known as the surprise probability), the tail probability of the minimum distance between the next token and training points, and the test error of a classifier trained on the observed points. All of these quantities are classically estimated by the leave-one-out method, which is inconsisten

Read original source ↗ Open in workspace

recordType

paper

region

Global