More Data or More Sensors?
What Time-Series Foundation Models Can and Cannot Learn
Associate Professor & Graduate Studies Chair, ECE | The Ohio State University
Co-Director, IITB-OSU Frontier Center
Founder, Ensemble Control Inc.
The world is a partially observed dynamical system
Every gas turbine, wind farm, and grid segment is a partially observed dynamical system streaming sensor data.
Promise of Foundation Models:
- Pretrain one big model on sensor logs
- Forecast zero-shot on a machine it has never seen
Assumptions from the field:
- “More data will lead to better forecasting.”
- “Any data can be tokenized and predicted.”
- “Heterogeneous data will generalize better.”
The thesis
A single computable quantity — the model’s Fisher information — reveals what your models can learn, how fast, and what they can never learn.
Empirical observation: Chaotic dynamics cannot be predicted by foundation models.
Types of dynamics: Stochastic, periodic, or chaotic?
Stochastic
- Wind-farm power — turbulent wind, second to second
- Reactor neutron-flux noise — fluctuating neutron population
Periodic
- 1 GW synchronous generator — rotor locked to 50/60 Hz
- Rotating machinery — shaft & blade-pass vibration signatures
Chaotic
- Combustion / thermoacoustic instability — gas turbines
- Density-wave oscillations — boiling-water reactors
Three questions this talk answers
1. How does more data help?
We derive the observability ceiling of a time-series foundation model and determine the data needed to bring the estimate within a desirable error bound.
2. Where should the next sensor go?
The Fisher spectrum ranks candidate measurements by the forecasting power each would unlock.
3. Which signals are forecastable?
Observation: Slowly drifting signals forecast with tiny error;
Random, chaotic, and fast states do not.
1 · Foundation Models
Their training pipeline, and their promise to technology
What is a time-series foundation model?
Chronos Model:
mean-scale the context
õ = o / s, s = mean|o|
bin into B tokens
q : ℝ → {1,…,B}
categorical transformer
pθ(zt+1 | z1:t), trained using cross-entropy loss
map back to real values
q-1(z) × s
Other Representative models
- TimesFM (Google)
- Moirai, Lag-Llama, MOMENT
Widely believed hypothesis: scale the corpus and forecasting improves.
Side Remark: Timeseries foundation model connected to World Action Model and VLAs
| World-Action | Time-Series FM | Vision-Lang-Action | |
|---|---|---|---|
| Observation \(o_t\) | image / video | numerical value | image + proprioception |
| Conditions on | past frames + action | past values only | image + instruction |
| Emits | next frame | next value | next action |
| Takes actions? | conditions on them | none (autonomous) | produces them |
| Examples | Genie, GAIA-1, Dreamer | Chronos, TimesFM | RT-2, OpenVLA, \(\pi_0\) |
To see the connection, note that: \[ P\big(o_{1:T},a_{1:T}\big| \ell\big)=\prod_{t=1}^{T} \underbrace{P_\theta(o_t\mid h_t^-,a_{t-1})}_{\textsf{world / TSFM channel }(\theta)}\; \underbrace{\pi_\phi(a_t\mid h_t,\ell)}_{\textsf{VLA channel }(\phi)} \]
Fisher Information Matrix and the Cramér-Rao Bound
For a model \(p_\theta(o)\) with score function \(s_\theta = \nabla_\theta \log p_\theta(o)\;\), the Fisher information matrix (FIM) is \[\;\mathbf I(\theta) = \mathbb E\!\left[ s_\theta s_\theta^\top \right] = -\,\mathbb E\!\left[\nabla_\theta^2 \log p_\theta\right]\]
Assume \(\theta^*\) is the true parameter and \(\widehat{\theta}_T\) is the estimate of \(\theta^*\) using \(T\) samples.
Cramér–Rao Bound
For large \(T\), any unbiased estimator \(\widehat{\theta}_T\) of \(\theta^*\) obeys \[\operatorname{Cov}(\widehat{\theta}_T)\approx \frac{1}{T} \mathbf I^{-1}(\theta^*)\implies \mathbb E\Big[(v_i^\top (\widehat{\theta}_T - \theta^*))^2\Big] \approx O\Big(\frac{1}{\lambda_i \sqrt{T}}\Big)\] where \(v_i\) and \(\lambda_i\) are the i-th eigenvector and eigenvalue of \(\mathbf I(\theta^*)\)
2 · Linear Process and The Observability Ceiling
Partially observed MDPs — what no amount of data can fix
Linear System with fewer sensors than states
\[X_{t+1}=A X_t + w_t,\qquad O_t = C X_t + v_t, \qquad \theta^* = x_0, \qquad C\in\mathbb R^{m\times n}\] \[ A = \begin{bmatrix} A_o & 0\\ A_{21} & A_{\bar o} \end{bmatrix},\qquad C=[I_m\ 0] \qquad \qquad \text{(provably unobservable for all $m<n$)} \]
Assume \(w_t=0\:\) – no actuation noise.
Observability Grammian: \(O = [C^\top, (CA)^\top, \dots, (CA^{n-1})^\top]^\top\)
Fisher Information Matrix: \(I(\theta^*) \approx \frac{1}{\sigma_v^2} O^\top O\)
Observability ceiling There is always a subspace of the dynamics that no corpus can teach the model to forecast, because there is an unobservable subspace (\(\texttt{Null}(O)\)) about the initial state in the data.
Spectrum of Fisher Information Matrix
Hallucination
How does context window help?
How does context window help LQG forecasts?
3 · High Dimensional Stochastic, Periodic and Chaotic systems
Empirical Performance of Time-series foundation models (Chronos) on high dimensional systems
Goal
Can we train a single foundation model to forecast multiple high dimensional deterministic and stochastic systems?
- Periodic systems (Stuart-Landau Oscillators)
- Noisy linear autoregressive systems (same as the linear systems example above, but observable with high magnitude noise)
- Chaotic systems (Kuramoto, Kuramoto-Sivashinsky, Lorenz)
Periodic System Timeseries Training
High Dimensional Autoregressive Timeseries with High Actuation Noise
Chaotic System (Kuramoto Oscillator) Timeseries
