Needle in a haystack for Time Series Language Models

Language models are now routinely tested on "needle in a haystack" tasks: hide a sentence inside a long document and ask the model to find it. The test exposes whether a model actually attends across its full context window, or quietly forgets the middle. Time series deserve the same scrutiny — but the needle is not a sentence, it is a transient: a three-beat arrhythmia in an hour of ECG, a pressure spike in a week of pump telemetry, a single fraudulent transaction in a year of account history.

What the benchmark measures

TS-Haystack embeds a labelled event at a known position inside a long, otherwise unremarkable signal, then asks a model two things: locate it, and justify it. Retrieval alone is not enough. A model that flags the right window but cannot say why it stood out is no more trustworthy than a threshold alarm. We score both the localization error and the quality of the natural-language rationale, graded against expert references.

The failure mode we care about is not missing the needle. It is finding it and being unable to explain it.

Why length breaks most approaches

Serializing a million-point signal into text tokens is a non-starter: the sequence overflows the context window long before the needle is in view. Rendering the signal as a plot collapses the resolution that made the event detectable in the first place. Both workarounds fail quietly — the model produces a fluent answer about data it never actually inspected. Query-conditioned fusion lets the model scan broadly and then request full-resolution detail only where its own reasoning points, which is exactly the behavior TS-Haystack is designed to reward.

Early results

Explicit cross-attention models hold their localization accuracy roughly flat as the haystack grows, while implicit soft-prompt models degrade sharply once the signal exceeds a few thousand points — the same scaling wall we see in training-time memory. The gap widens with the length of the haystack, which is precisely the regime that matters for real deployments on continuous monitoring.

TS-Haystack, its generator, and the reference rationales are being released open-source alongside the OpenTSLM models. We want temporal retrieval measured in the open, the same way text retrieval is.