Each question is embedded in a document set totalling roughly 100,000 tokens under the cl100k_base tokenizer. The answer is not in one place โ the model must locate several relevant passages scattered across the input, relate them to one another and derive a conclusion. Answers are graded by a GPT-based equality checker that compares the model's output against a reference answer; scoring is pass@1, so only the first attempt counts. The set contains 100 questions, and the result enters the Artificial Analysis Intelligence Index with a 5% weighting.
Advertised context windows grow faster than models' real ability to use them. Popular needle-in-a-haystack tests check only the retrieval of a single fact, which a model can do without understanding the whole. AA-LCR forces the integration of scattered premises โ the thing that actually separates a useful long context from a nominal one.
Length measured with the cl100k_base tokenizer; the premises needed for the answer are deliberately scattered across multiple passages.
Questions constructed so that a single passage is not enough to answer correctly.
Compares the model's answer with a reference; pass@1 scoring without repeats.
Official
Input length is counted with cl100k_base; models using a different tokenizer will see a different count of their own tokens for the same text.
Grading by a GPT checker introduces its own noise and may treat substantively correct but differently phrased answers inconsistently.
AA-LCR v1.1 comprises 100 questions and enters the Artificial Analysis Intelligence Index with a 5% weighting.
