More Data or More Sensors?
What Time-Series Foundation Models Can and Cannot Learn
Associate Professor & Graduate Studies Chair, ECE | The Ohio State University
Co-Director, IITB-OSU Frontier Center
Founder, Ensemble Control Inc.
The world is a partially observed dynamical system
Every gas turbine, wind farm, and grid segment is a partially observed dynamical system streaming sensor data.
Promise of Foundation Models:
- Pretrain one big model on sensor logs
- Forecast zero-shot on a machine it has never seen
Assumptions from the field:
- “More data will lead to better forecasting.”
- “Any data can be tokenized and predicted.”
- “Heterogeneous data will generalize better.”
The thesis
A single computable quantity — the model’s Fisher information — reveals what your models can learn, how fast, and what they can never learn.
Types of dynamics: Stochastic, periodic, or chaotic?
Stochastic
- Wind-farm power — turbulent wind, second to second
- Reactor neutron-flux noise — fluctuating neutron population
Periodic
- 1 GW synchronous generator — rotor locked to 50/60 Hz
- Rotating machinery — shaft & blade-pass vibration signatures
Chaotic
- Combustion / thermoacoustic instability — gas turbines
- Density-wave oscillations — boiling-water reactors
What is a time-series foundation model?
Chronos Model:
1 · Scale
mean-scale the context
\tilde{o} = \frac{o - \text{mean}(o)}{\text{std}(o)}
→
2 · Quantize
bin into 2^m tokens
q : \mathbb{R}^n \to \{0,\dots,1\}^m
q(\tilde{o}_t) = z_t
→
3 · Train & Predict
categorical transformer
p_\theta(z_{t+1} | z_{1:t}), trained using cross-entropy loss
→
4 · Dequantize
map back to real values
o_{t+1} = \text{std}(o)q^{-1}(z_{t+1}) + \text{mean}(o)
Other Representative models
- TimesFM (Google)
- Moirai, Lag-Llama, MOMENT
Widely believed hypothesis: scale the corpus and forecasting improves.
Timeseries foundation model connected to World Action Model and VLAs
| World-Action | Time-Series FM | Vision-Lang-Action | |
|---|---|---|---|
| Observation o_t | image / video | numerical value | image + proprioception |
| Conditions on | past frames + action | past values only | image + instruction |
| Emits | next frame | next value | next action |
| Takes actions? | conditions on them | none (autonomous) | produces them |
| Examples | Genie, GAIA-1, Dreamer | Chronos, TimesFM | RT-2, OpenVLA, \pi_0 |
To see the connection, note that: P\big(o_{1:T},a_{1:T}\big| \ell\big)=\prod_{t=1}^{T} \underbrace{P_\theta(o_t\mid h_t^-,a_{t-1})}_{\textsf{world / TSFM channel }(\theta)}\; \underbrace{\pi_\phi(a_t\mid h_t,\ell)}_{\textsf{VLA channel }(\phi)}
Three questions this talk answers
1. How does more data help?
We derive the observability ceiling of a time-series foundation model and determine the data needed to bring the estimate within a desirable error bound.
2. Where should the next sensor go?
The Fisher spectrum ranks candidate measurements by the forecasting power each would unlock.
3. Which signals are forecastable?
Observation: Slowly drifting signals forecast with tiny error;
Random, chaotic, and fast states do not.
Fisher Information Matrix and the Cramér-Rao Bound
For a model p_\theta(o_{k+1}|o_{1:k}) with score function s_\theta = \nabla_\theta \log p_\theta(o_{k+1}|o_{1:k})\;, the Fisher information matrix (FIM) is \;\mathcal{I}(\theta) = \mathbb E\!\left[ s_\theta s_\theta^\top \right] = -\,\mathbb E\!\left[\nabla_\theta^2 \log p_\theta\right] where k is the context length, typically k = 128 or k=256.
Assume \theta^* is the true parameter and \widehat{\theta}_T is the unbiased estimate of \theta^* using a dataset of T samples.
Cramér–Rao Bound
For large T, any unbiased estimator \widehat{\theta}_T of \theta^* obeys \operatorname{Cov}(\widehat{\theta}_T)\approx \frac{1}{T} \mathcal{I}^{-1}(\theta^*)\implies \mathbb E\Big[(v_i^\top (\widehat{\theta}_T - \theta^*))^2\Big] \approx O\Big(\frac{1}{\lambda_i T}\Big) where v_i and \lambda_i are the i-th eigenvector and eigenvalue of \mathcal{I}(\theta^*)
Linear Process and The Observability Ceiling
No amount of data can fix the observability of partially-observed linear systems
Linear System with fewer sensors than states
X_{t+1}=A X_t + w_t,\qquad O_t = C X_t + v_t, \qquad \theta^* = x_0, \qquad C\in\mathbb R^{m\times n} A = \begin{bmatrix} A_o & 0\\ A_{21} & A_{\bar o} \end{bmatrix},\qquad C=[I_m\ 0] \qquad \qquad \text{(provably unobservable for all $m<n$)}
Assume w_t=0\: – no actuation noise. Then p_\theta(o_{1:n}|x_0) = \mathcal{N}(\mathcal{O}x_0, \sigma_v^2 I_{nm})
Observability Grammian: \mathcal{O} = [C^\top, (CA)^\top, \dots, (CA^{n-1})^\top]^\top
Fisher Information Matrix: \mathcal{I}(\theta^*)=\frac{1}{\sigma_v^2} \mathcal{O}^\top \mathcal{O}
Observability ceiling No corpus of dataset can teach the model to forecast unobservable subspace (\texttt{Null}(\mathcal{O})) of the dynamics. Consequently, O_{n+1:T|1:n} will always have errors
Spectrum of Fisher Information Matrix
Predicting x_0 from context using Ridge Regression with w_t Gaussian
- Training data: Centered dataset (\widetilde{Y}_{tr}, \widetilde{X}_{0,tr})
W = (\widetilde{Y}_{tr}^T \widetilde{Y}_{tr} + \lambda I)^{-1} \widetilde{Y}_{tr}^T \widetilde{X}_{0,tr}
- State Prediction (Testing)
\widehat{x}_{0,te} = \Big(Y_{te} - Mean({Y_{tr}})\Big) W + Mean(x_{0,tr}),\quad \text{PredictionError} = \widehat{x}_{0,te} - x_{0,te}
- Empirical error variance along j^{th} eigenvector is \sigma^2_j = \text{Var}(v_j ^T \text{PredictionError})
Initial State Estimation Error
Does context window help prediction accuracy?
Does measurement noise impact prediction accuracy?
For context window = 10:
High Dimensional Stochastic, Periodic and Chaotic systems
Empirical Performance of Time-series foundation models (Chronos) on high dimensional systems
Goal
Can we train a single foundation model to forecast multiple high dimensional deterministic and stochastic systems?
- Periodic systems (Stuart-Landau Oscillators)
- Noisy linear autoregressive systems (same as the linear systems example above, but observable with high magnitude noise)
- Chaotic systems (Kuramoto, Kuramoto-Sivashinsky, Lorenz)
64-dimensional real-valued states for all the systems
Periodic System Timeseries Training
High Dimensional Autoregressive Timeseries with High Actuation Noise
Chaotic System (Kuramoto Oscillator) Timeseries
