Research · NVIDIA Research

Long context: testing document understanding

What the source says

RULER tests retrieval, connections and aggregation across long texts. In the study, evaluated models lost accuracy as context grew, even after succeeding on simple retrieval tests.

What this means for a buyer

Test answers to relevant questions and their references to source passages. The amount of text a model accepts does not by itself establish the usefulness of its answers.

What to check before choosing

  • Include both short and long documents.
  • Check information from different parts of the text.
  • Record omissions, errors and unsupported answers.

Buyer implications are our editorial interpretation. The publication does not establish the speed or compatibility of a particular catalogue system.