Loading…
OpenTSLM
Teaching Large Language Models to Understand Time
OpenTSLM: Time Series Language Models for Reasoning over Multivariate Medical Text- and Time Series Data
Patrick Langer1, 2, 3, Thomas Kaar1, 3, Max Rosenblattl1, 3, Maxwell A. Xu4, 5, Winnie Chow6, Martin Maritsch7, Robert Jakob3, Ning Wang3, Juncheng Liu8, 9, Aradhana Verma10, Brian Han11, Daniel Seung Kim12, Henry Chubb11, Scott Ceresnak11, Aydin Zahedivash1, Alexander Tarlochan Singh Sandhu10, Fatima Rodriguez10, Daniel McDuff5, 13, Elgar Fleisch2, 3, 14, Oliver Aalami1, Filipe Barata2, *, Paul Schmiedmayer1, *
AUTHOR AFFILIATIONS
1STANFORD MUSSALLEM CENTER FOR BIODESIGN, STANFORD UNIVERSITY, USA, 2CENTRE FOR DIGITAL HEALTH INTERVENTIONS, ETH ZURICH, SWITZERLAND, 3AGENTIC SYSTEMS LAB, ETH ZURICH, SWITZERLAND, 4UNIVERSITY OF ILLINOIS URBANA-CHAMPAIGN, USA, 5GOOGLE RESEARCH, USA, 6STANFORD UNIVERSITY, USA, 7AMAZON, USA, 8MICROSOFT, 9NATIONAL UNIVERSITY OF SINGAPORE, SINGAPORE, 10DIVISION OF CARDIOVASCULAR MEDICINE, STANFORD UNIVERSITY, USA, 11PEDIATRIC CARDIOLOGY, STANFORD UNIVERSITY, USA, 12DIVISION OF CARDIOLOGY, UNIVERSITY OF WASHINGTON, USA, 13UNIVERSITY OF WASHINGTON, USA, 14CENTRE FOR DIGITAL HEALTH INTERVENTIONS, UNIVERSITY OF ST. GALLEN, SWITZERLAND, *EQUAL CONTRIBUTION
ACCEPTED AT
What are Time Series Language Models?
Time Series Language Models (TSLMs) are language models that can take raw time series as input alongside text. Instead of converting signals into plots, summaries, or long strings of numbers, a TSLM reads the signal directly and answers in natural language.
This makes them flexible in the same way multimodal chat models are flexible for images: you can attach one or many signals, combine signals of different lengths and sampling rates, ask open-ended questions, request explanations, compare windows, or ask for a forecast without training a separate model for each task.
How OpenTSLM works
With OpenTSLM, we present a multimodal LLM that treats time series as a native modality. It can reason over raw multivariate signals alongside text, and it is designed to scale to multiple very long time series without forcing the signals through the language-model token stream.
The paper compares two ways to fuse text and time series, OpenTSLM-SoftPrompt and OpenTSLM-Flamingo, on Gemma3 and Llama3.2 backbones at 270M, 1B and 3B scale. SoftPrompt places time series tokens in the language-model input. Flamingo uses cross-attention, allowing the language model to query compact time series representations instead of processing every signal point as a text token.
OPENTSLM-FLAMINGO ARCHITECTURE
Experiments
We compare OpenTSLM against the main ways people try to make language models handle time series today: serialize the values as text, fine tune on serialized values, show the model a plot, or use GPT-4o with image inputs.
OpenTSLM outperforms every baseline across time series question answering, activity recognition, sleep staging, and ECG question answering. Even 1B-parameter OpenTSLM models surpass GPT-4o, showing that time series capability comes from the model interface, not just from making a general model larger.
BEST MACRO-F1 PER METHOD FAMILY · HIGHER IS BETTER ↑
OpenTSLM
Image-based
Tokenized
GPT-4o
TSQATIME SERIES QUESTION ANSWERING
OpenTSLM
99.71
Image-based
94.94
Tokenized FT
84.54
GPT-4o
59.24
Tokenized
16.24
HAR-CoTACTIVITY RECOGNITION
OpenTSLM
69.31
Tokenized FT
60.44
Image-based
44.01
GPT-4o
10.83
Tokenized
0.00
Sleep-CoTSLEEP STAGING
OpenTSLM
69.88
Image-based
18.56
GPT-4o
15.47
Tokenized FT
9.05
Tokenized
5.66
ECG-QA-CoTECG QUESTION ANSWERING
OpenTSLM
41.92
Image-based
26.17
GPT-4o
24.95
Tokenized FT
OOM
—
Tokenized
0.00
BARS SHOW BEST MACRO-F1 WITHIN EACH METHOD FAMILY · OOM = OUT OF MEMORY | LANGER ET AL., 2025
ECG-reasoning validated with 7 cardiologists
We evaluated 84 rationales of our best model, OpenTSLM/Llama3.2-3B, through expert review by 7 Stanford Hospital clinicians, finding the reasoning correct or partially correct 97% of the time. They used a rubric based on ACC/AHA ECG competence guidelines and the RIME framework, focusing on ECG feature identification, reasoning quality, and use of patient context.
ECG REASONING VALIDATED BY SEVEN CARDIOLOGISTS | STANFORD HOSPITAL
POSITIVE
PARTIAL
NEGATIVE
97%
of 84 reviewed rationales from OpenTSLM/Llama3.2-3B rated correct or partially correct, using a rubric based on ACC/AHA ECG competence guidelines and the RIME framework.
PATTERN RECOGNITION
CLINICAL REASONING
CONTEXT INTEGRATION
Scalability unlocked
We, for the first time, present an architecture for TSLMs that enables scaling to multiple multivariate and long time series. While prior methods typically scale quadratically in memory with longer time series, our implementation achieves comparable performance with near constant memory.
SIMULATION DATASETS | LANGER ET AL., 2025
OPENTSLM-FLAMINGO
PRIOR METHODS / SOFTPROMPT
OpenTSLM-Flamingo keeps VRAM usage nearly constant when inputting more or longer time series.
All code, datasets, and models from the paper are released open-source.