let's do it like they did in 1924 and draw forecasts by hand

Ensembles Are Called Agents Now

Posts / 29 October, 2026

The GIFT-eval forecast benchmark allows submissions to be listed under model type “agentic”. From its README:

agentic: Multi-step systems that use agents or LLMs to reason, generate or select forecasts.

Its leaderboard is now dominated by models tagged as such (indicated by the yellow circle emoji). Not being able to check all of the submissions (as some did not even provide links to documentation let alone replication code), at least a good chunk of the top submissions are barely using “reasoning”. Instead, they are ensembles, combining models such as TimesFM, Chronos, Toto, and TiRex.

TW3Cast is a complicated ensemble. From its README:

Listed in the leaderboard’s multi-model (agentic) category, but it runs no agent and no LLM, and makes no inference-time decisions: a frozen per-configuration router over specialized models.

While there is a bit more to it, TW3Cast is a uniform ensemble for some datasets, as listed in its router.csv file. For example, for the Car Parts dataset, it uses a uniform ensemble of Chronos-2, TimesFM, and TiRex-2.

PRISM by Birla AI Labs is an ensemble:

Prism is a leakage-free probabilistic forecasting ensemble developed by the Frontier Research and DeepTech Team at Birla AI Labs. Rather than training a new model, Prism combines the quantile outputs of several pretrained foundation models, learning how to blend them from a held-out validation window that never touches test data. The key insight is that different models have complementary strengths: learning a separate blend at each quantile level means the model best at the median does not have to be the model that covers the tails.

RacineCast-1 and RacineCast-1-NC (i.e. non-commercial) are weighted ensembles of models. RacineAI summarizes their approach in this recipe:

0.55 · Toto-2.0-FnF + 0.16 · Toto-2.0-2.5B + 0.16 · TiRex-2 + 0.045 · Chronos-2 + 0.045 · FlowState-r1.1 + 0.03 · PatchTST-FM-r1

Back in July of this year, it ranked them first on the leaderboard.

55% of the RacineCast approach is Toto-2.0-FnF (as in Family-and-Friends). Toto-2.0-FnF is an ensemble of different Toto checkpoints and other models—meta-learned by a FFORMA-style XGBoost. It, too, is submitted as “agentic” to the GIFT-eval leaderboard.

My personal favorite, however, is IK-QuadTSFM. It is a uniform ensemble of Chronos-2, TimesFM-3, TiRex-2, and Toto-2.0-FT, and as such questions the additional complexity of any other “agentic” submission in its vicinity.

Screenshot of GIFT-eval leaderboard.
Right-click to open the full-size image. Screenshot of the GIFT-eval overall leaderboard ranked by the default MASE_Rank. The yellow emoji stands for GIFT-eval’s “agentic” model type, red stands for “zero-shot”, green for “pretrained”.

The above screenshot was taken on October 29, 2026. It shows the default sorting based on the MASE_Rank column. Sorting by MASE would shuffle the results somewhat, though “Stride w/ Synapse” and “EXAONE-Forecast-Agent” would continue to rank at the top. The rest would cover the range from 0.650 MASE (for LS-MoE and TIMEHEDGE) over 0.663 for the aforementioned IK-QuadTSFM and 0.667 for TimesFM, to 0.698 for Chronos-2.

Notes / 2025-12-01

Similar-Day Naive Method

Like the Chronos-2 technical report, Uniejewski and Weron compare the approaches implemented in their aforementioned paper against a naive baseline. Unlike the technical report which uses seasonal naive for all data sets, Uniejewski and Weron, to their credit, employ a baseline that, while naive, is tuned to their use case of electricity price forecasting and employs as much knowledge as possible:

The first benchmark, the so-called naive method, belongs to the class of similar-day techniques. It is defined by $\hat{P}_{d,h} = \hat{P}_{d-7,h}$ for Monday, Saturday and Sunday, and $\hat{P}_{d,h} = \hat{P}_{d-1,h}$ otherwise. As Contreras et al. argued, forecasting procedures that are not calibrated carefully fail to outperform this extremely simple method surprisingly often.

So given the different human patterns in electricity usage on workdays and on weekends, they use the naive method for workdays and seasonal naive for the weekend and Monday (because Sunday can’t be used to predict Monday).

Reminds me of this footnote.

Notes / 2025-11-30

Scaling Time Series Using the Inverse Hyperbolic Sine

This one’s new to me: The Chronos-2 tokenization pipeline follows up an initial standardization of input time series with a variance stabilizing transformation in the inverse hyperbolic sine function, $\sinh^{-1}$, as described in the model’s technical report.

The inverse hyperbolic sine function is defined as:

$$ \sinh^{-1}(x) = \ln(x + \sqrt{x^2+1}). $$

As such it is an alternative to the $\ln(x+1)$ transform with the added benefit of being applicable to negative values.

While news to me, the technical report points out that the inverse hyperbolic sine is no news to the electricity price forecasting literature, citing “Efficient Forecasting of Electricity Spot Prices with Expert and LASSO Models” by Bartosz Uniejewski and Rafał Weron, which describes the inverse hyperbolic sine and other variance stabilizing transformations in more detail.

Notes / 2025-11-23

Stacking Simple Forecasts by Classification

This paper got me thinking about stacking and ensembling of forecasts again.

Different from what the paper proposes but in the stacking models-sense, consider some multi-class classification model used to predict which base model forecasts a given observation from a set of many time series best. Something more akin to the FFORMA approach.

But use simple forecast methods to keep the overhead of first creating base forecasts at a minimum. Use the seasonal lag, the seasonal average, the most recent observation, zero, year-to-date average, etc. as base forecasts. Anything, really, that makes for a strong baseline for part of your dataset and can be calculated “without training”. If nothing else, this avoids the need for expensive cross-validation loads (as those described in the aforementioned paper).

Then, for each observation of your time series data, derive a label based on the base forecast that comes closest to the observation.1

Finally, train the classifer model to predict the labels, using the base models as features, but also any other kind of feature. Now well and truly spitballing: Have some features summarize base predictions relative to one another, and relative to recent observations (“The seasonal lag is 190% of the year-to-date average”) to help the model distinguish patters.

Given that the classifier model would only pick between simple forecast methods to arrive at the final prediction, the overall forecast accuracy will always be limited. I can imagine scenarios, however, where the prediction ends up being more stable and at the right level compared to models trained to predict the time series directly. Constrained by the quality of the base forecasts, this stacking approach could pay off handsomely if the optimal base learner differs across the set of time series, or frequently changes within the same series. That is, when there is no dominant baseline.

For example, it could work well in scenarios where time series have multi-modal distributions due to zeroes, or frequently exhibit structural breaks.

In a way, the classifier would learn the sort of “item routing” described here. Instead of assigning time series to different forecast models based on rules, the classifier will split the data into subsets and assign the model.

One more nice side effect: There is no need for target scaling when base forecasts are non-trained methods and when the meta learner only needs to classify.

The overall approach is also fairly explainable if not interpretable. Especially if one picks the features for additional context wisely.

Finally, while the base forecasts will limit the overall forecast quality that can be achieved, we can measure the best possible forecast quality the stacking approach can reach in the hypothetical case of a perfect classification of every observation. So before putting any effort into training a good classifier, we will know whether the effort could hypothetically be worth it.

Though there is a limit to the number of base forecasts you should use to derive the labels for the classifier. Dare to use the range of real numbers and you’ll end up back with a (discretized) regression problem.

  1. Why not throw in an additional label category indicating that none of the methods came even close to the actual observation? To predict that a data point is not predictable by the base forecasts. ↩︎

Links / 2025-05-10

'Data Engineers Should Be Held to the Same Standards as Bakers' ∞

Over at Hermit Tech, Jordan Andersen laments over embarrassingly low project delivery expectations within the software and data industry when compared to other professions such as the one of a baker. His concern, however, is not only with the customers of the project, but with the developers:

The latter consequence of low expectations is the more serious one. It’s something that some people may never recover from. […] The trajectory of a data or software engineer seems to be a bit different: they burn out from the mental stress of working in dishonest and fraudulent cultures that produce unsatisfying work, but the engineer stays in the profession. Only they’re a shell of a human.

Notes / 2025-03-16

Inherently Coherent Forecasts

Suppose you offer a single API endpoint that takes historical observations of one time series at a time (not unlike Nixtla’s TimeGPT). Suppose further that everyone at your company uses this endpoint to generate any business forecast they need. What behaviors should the API’s forecast exhibit?

An obvious one would be forecast coherence across aggregation levels. Say your company sells three products. You have the historical revenue for the three products, $Y_1$, $Y_2$, $Y_3$, as well as the historical revenue for the company as a whole, $Y$, where $Y=Y_1+Y_2+Y_3$. When you feed $Y_1$ into the API, the API returns forecast $\text{API}(Y_1)=\hat{Y}_1$. It returns $\hat{Y}_2$ given $Y_2$, $\hat{Y}_3$ given $Y_3$, and given $Y$ it returns $\hat{Y}$. If we want to see a forecast at company-level that “makes sense” given product-level forecasts, we want the forecasts to be coherent: $\hat{Y}=\hat{Y}_1 + \hat{Y}_2 + \hat{Y}_3$. But given that the API only ever sees one time series at a time, what model could be used to provide coherent forecasts?

Again, keep in mind that the API doesn’t have access to more information than its current input. For example, when you feed it $Y_2$, it doesn’t know that $Y_1$ and $Y_3$ exist. It also doesn’t know whether $Y_2$ is at the hierarchy’s bottom- or top-level. The API does not have access to all possible time series that exist at the company. Consequently, the API does not have the information required to apply the methods from the vast literature on coherent forecasts. What modeling approach will return coherent forecasts nevertheless?

The seasonal naive method would. As would the naive method, of course. The mean forecast, too. And any other linear combination of the past observations with a pre-determined context length and coefficients.1 Any pre-trained—the coefficients must not change depending on the input provided to the API—linear autoregressive model of order p will do.

That’s a fairly flexible model family. It also includes the Non-Parametric Time Series Forecaster and Threedx. Thus, the API could do an OK job for most series passed to it. It would have an obvious weakness in new or recently discontinued time series, though. The API could not, for example, override its usual forecast for a recently discontinued product and forecast zero instead. The same holds for any other kind of structural break as well as anomalies. That would make it tough to rely exclusively on such an API for operational decisions happening at the low hierarchy level ridden with discontinuities.

The above combined with recent evidence make me inclined to offer the seasonal naive method as Zero-Shot Foundation Model with Inherently Coherent Forecasts available as API.

  1. For example, the mean forecast must not be the mean of the provided observations, but the mean of $k$ observations resulting in coefficients $1/k$. A time series with $n \lt k$ observations needs to be padded with zeros. A time series with $n \gt k$ observations is shortened to the $k$ most recent observations. ↩︎

Links / 2025-02-23

'Performance of Zero-Shot Time Series Foundation Models on Cloud Data' ∞

Toner et al. compare time series foundation models such as Chronos, Mamba4Cast, and TimesFM on data from Huawei data centers (data is available on Github!):

To examine the behaviour of (zero-shot) FMs on cloud time series, we perform experiments on function demand data drawn from real-world usage. Our results show that FMs perform poorly, being outperformed by the simple baselines of a linear model and a naive seasonal forecaster.

If you think about it, the seasonal naive method is the original zero-shot foundation model.

For example, the naive seasonal forecaster performs better than all the FMs across all datasets and forecast horizons. Moreover, the performance difference is often large; for example, the naive seasonal forecaster incurs a MASE typically half that of TimesFM.

Alas, the authors make little attempt to explain why the foundation models are outperformed by the seasonal naive method beyond the data’s spikiness. But perhaps that’s just it. Chronos, for example, is known to fail on spiky data due to its mean scaling and quantization (see figure 16 in the corresponding paper).

We also present evidence of pathological behaviours of FMs, demonstrating they can produce chaotic (as in Figure 1) or illogical (as in Figure 2) forecasts.

Check out the mentioned plots in figures 1, 2, 4 and 5. They’re why I don’t trust any paper that doesn’t visualize its method’s predictions.

Notes / 2025-01-12

R-squared as Forecast Evaluation Metric

When Jane Street’s \$120,000 “Real-Time Market Data Forecasting” Kaggle competition closes tomorrow, submitted forecasts will be evaluated using $R^2$:

Submissions are evaluated on a scoring function defined as the sample weighted zero-mean R-squared score ($R^2$) of responder_6. The formula is give by: $R^2 = 1 - \frac{\sum w_i(y_i - \hat{y}_i)^2}{\sum w_i y_i^2}$ where $y$ and $\hat{y}$ are the ground-truth and predicted value vectors of responder_6, respectively; $w$ is the sample weight vector.

Stats 101 courses have left $R^2$ with a bad rep as gameable regression metric. But I’ve come to appreciate it as a metric in forecasting—when calculated on the test set with the total-sum-of-squares denominator based on the test period’s empirical mean, $\bar{y}_{T,h} = \sum_{t = T+1}^{T+h}y_t$, where $T$ is the training set’s last observation and $h$ the number of observations in the test set. Then:

$$ R^2 = 1 - \frac{\sum_{t = T+1}^{T+h}(y_t - \hat{y}_t)^2}{\sum_{t = T+1}^{T+h}(y_t - \bar{y}_{T,h})^2} $$

Above definition gives the total-sum-of-squares the advantage of future knowledge included in $\bar{y}_{T,h}$ that the predictions $\hat{y}_t$ used for the residual-sum-of-squares do not have. In comparison to its linear regression origin, this test set-focused definition is not going to improve as parameters are added to the forecast model. But it does keep its interpretability:

For example, when predicting the final twelve months of AirPassengers data, the obviously poor prediction given by the training set mean results in an $R^2$ of -8.24 whereas a seasonal naive prediction achieves an $R^2$ of 0.54.

This shouldn’t come as a suprise. Interpreting $R^2$ as the fraction of (test) variance explained, the seasonal naive prediction can explain more than half of the variance that the test mean left unexplained. The model captures a large part of the signal hidden in the data’s variation.

Just like Jane Street replaced the test set empirical mean in their definition of $R^2$ by zero, which in their application of predicting expected financial returns is a hard to beat expected value, one could replace the total sum of squares denominator by the squared errors of a different benchmark such as seasonal naive predictions when it is known that due to the data’s seasonality the seasonal naive prediction will consistently outperform the test set mean despite its future knowledge. Thus, $R^2$ is not so different from the more common relative measures such as the Relative Mean Squared Error described in section 6.1.4 of Hewamalage et al. (2022), or Skill Scores employed in weather prediction described by Murphy (1988).

Then again, when a time series’ variance is not dominated by its seasonal component, $R^2$ and its interpretation as fraction of variance explained can leave a damning picture of a model’s forecasts not actually being all that skillful. It casts doubt by asking how much signal your model has picked up to meaningfully predict future variation. Suddenly any value larger than zero will feel like an enormous success.

Links / 2024-10-06

Explosion Back to Its Roots ∞

In 2016, Matthew Honnibal presented at the first PyData Berlin meetup that I attended. He had already started spaCy and was training models on Reddit comments, whereas I wasn’t really into NLP and heard about spaCy for the first time that evening. And while I’m still not really into NLP, I never stopped keeping tabs on what he and Ines Montani were building at Explosion.

Honnibal recounts Explosion’s history in view of recent changes:

For most of Explosion’s life we’ve been a very small company, running off revenues. In 2021 that changed, and we became a slightly less small company running off venture capital. We’ve been unable to make that configuration work, so we’re back to running Explosion as an independent-minded self-sufficient company. We’re going to stay small and not look for any more venture capital. spaCy and Prodigy will continue.

With a focus on “industrial-strength”, Explosion has built opinionated data science tooling with beautiful documentation. SpaCy is beloved open-source software (with the community coming together for spaCy IRL—a real treat of a conference) that convinces data scientists to spend company budget on Prodigy. This combination of spaCy and Prodigy is the ingredient to Explosion’s unique success as small, self-sufficient company in a venture-funded AI environment. Already familiar with spaCy, data scientists are comfortable with purchasing Prodigy licenses to ease their annotation workflows common to NLP. And being technical expert users, they also are capable of hosting the software themselves. Explosion doesn’t have to handle customers’ data.

License revenues, no hosting, no data: Enablers of a profitable business run by a small team. I wish they continue to thrive!

In his post, Honnibal shares realities of maintaining software that companies and developers rarely admit to, yet which are determinants of a team’s success:

Engineering for spaCy and our other projects was also very challenging to hand over. spaCy is implemented in Cython, and big chunks of the project are essentially C code with funny syntax. We pass around pointers to arrays of structs, and if you access them out of bounds, well, hopefully it crashes. You have to just not do that. And then in addition to this memory-managed code, there’s all the GPU-specific considerations, all the numpy minutiae, and maintaining compatibility with a big matrix of Python versions, operating systems and hardware. It’s a lot.

The infrastructure required for machine learning doesn’t make it any easier:

I’ve been finding the transition back to the way things were quite difficult. I still know our codebases well, but the associated infrastructure isn’t easy to wrangle. Overall I haven’t been very productive over the last few months, but it’s getting better now.

On top come unexpected team dynamics as the previous architect shifts his focus:

As I became less involved in the hands-on work, I struggled to be effective as a decision-maker. A lot of the bigger questions got deferred, and we had an increasing bias towards whichever approach was least committal.

On a different note, I am fascinated that Hugging Face has the funds to provide a quarter-million grant for open-source developers. How many of these funds do they provide?1

We considered selling the company, but we weren’t able to find a good fit. Instead, we’re back at the same sort of size we had before the investment. We’re very grateful to Hugging Face for a $250,000 grant to support our open-source work as our funding ran out, and we’ve applied successfully for a German R&D reimbursement grant that will give us up to €1.5m in unconditional funding.

To me, Explosion is one of the coolest exports that Berlin and Germany have to offer. Great to see them receive such grant.

  1. At roughly 0.1% of their Series D funding round, there might be a few. ↩︎

Links / 2024-10-06

The New Internet ∞

Avery Pennarun, CEO of Tailscale, in a company all-hands meeting:

In modern computing, we tolerate long builds, and then docker builds, and uploading to container stores, and multi-minute deploy times before the program runs, and even longer times before the log output gets uploaded to somewhere you can see it, all because we’ve been tricked into this idea that everything has to scale. People get excited about deploying to the latest upstart container hosting service because it only takes tens of seconds to roll out, instead of minutes. But on my slow computer in the 1990s, I could run a perl or python program that started in milliseconds and served way more than 0.2 requests per second, and printed logs to stderr right away so I could edit-run-debug over and over again, multiple times per minute.

Links / 2024-07-27

Trained Random Forests Completely Reveal Your Dataset ∞

The paper’s title is a small portion of clickbait: As of yet not all but some trained random forests completely reveal your dataset. Still, using constraint programming the authors completely reconstruct the data used to train binary classification random forests without bagging on only binary features.

I imagine a lot of random forests having been trained on sensitive data in the past and their model files been more loosely handeled than the data. What private information could the model possibly reveal? Yeah.

Watch Julien Ferry present his paper in a video recorded for ICML 2024.

Links / 2024-06-16

How The Economist's Presidential Forecast Works ∞

The Economist is back with a forecast for the 2024 US presidential election in collaboration with Andrew Gelman and others. One detail in the write-up of their approach stood out to me:

The ultimate result is a list of 10,001 hypothetical paths that the election could take.

Not 10,000, but 10,000 and one MCMC samples. I can’t remember seeing any reference for this choice before (packages love an even number as default), but I have been adding a single additional sample as tie breaker for a long time: If nothing else, it comes in handy to have a dedicated path represent the median to prevent an awkward median estimate of 269.5 electoral votes.

The extra sample is especially helpful when the main outcome of interest is a sum of more granular outcomes. In the case of the presidential election, the main outcome is the sum of electoral votes provided by the states. One can first identify the median of the main outcome (currently 237 Democratic electoral votes). Given the extra sample, there will be one MCMC sample that results in the median. From here, one can work backwards and identify this sample index and the corresponding value for every state, for example. The value might not be a state’s most “representative” outcome and it is unlikely to be the state’s median number of electoral votes. But the sum across states will be the median of the main outcome. Great for a visualization depicting what scenario would lead to this projected constellation of the electoral college.

In contrast, summing up the median outcome of each state, there would be only 226 Democratic electoral votes as of today.1

  1. CA, DC, HI, MA, MD, ME 1, NY, VT, CO, CT, DE, IL, NJ, OR, RI, WA, NM, ME, NE 2, VA, NH, MN. ↩︎

Links / 2024-05-06

(Deep) Non-Parametric Time Series Forecaster ∞

If you read The History of Amazon’s Forecasting Algorithm, you’ll hear about fantastic models such as Quantile Random Forests, and the MQTransformer. In GluonTS you’ll find DeepAR and DeepVARHierarchical. But the real hero is the simple model that does the work when all else fails. Tim Januschowski on Linkedin:

One of the baselines that we’ve developed over the years is the non-parametric forecaster or NPTS for short. Jan Gasthaus invented it probably a decade ago and Valentin Flunkert made it seasonality aware and to the best of my knowledge it’s been re-written a number of times and still runs for #amazon retail (when other surrounding systems were switched off long ago).

Januschowski mentions this to celebrate the Arxiv paper describing NPTS and its DeepNPTS variant with additional “bells and whistles”. Which I celebrate as I no longer have to refer people to section 4.3 of the GluonTS paper.

Notes / 2024-04-21

Debug Forecasts with Animated Plots

Speaking of GIFs, animated visualizations of rolling forecasts are eye-opening to the impact of individual observations, the number of observations, and default settings on a model’s forecasts. In the example below, the default forecast::auto.arima() transitions between poor model specifications until it can finally pick up the seasonality after 24 observations, only to generate a negative point forecast despite purely non-negative observations.

Fantastic way to understand forecast methods’ edge-case behavior.

My favorite frame? After nine observations, when a model specification with trend is picked and returns an exploding forecast based on very little evidence.

The code for this GIF in this small Github repo.

Links / 2024-04-15

Reliably Forecasting Time-Series in Real Time ∞

Straight from my YouTube recommendations, a PyData London 2018 (!) presentation by Charles Masson of Datadog. To predict whether server metrics cross a threshold, he builds a method model that focuses on being robust to all the usual issues of anomalies and structural breaks. He keeps it simple, interpretable, and–for the sake of real-time forecasting–fast. Good stuff all around. The GIFs are the cherry on top.

Links / 2024-03-27

Chronos: Learning the Language of Time Series ∞

Ansari et al. (2024) introduce their Chronos model family on Github:

Chronos is a family of pretrained time series forecasting models based on language model architectures. A time series is transformed into a sequence of tokens via scaling and quantization, and a language model is trained on these tokens using the cross-entropy loss. Once trained, probabilistic forecasts are obtained by sampling multiple future trajectories given the historical context.

The whole thing is very neat. The repository can be pip-installed as package wrapping the pre-trained models on HuggingFace so that the installation and “Hello, world” example just work, and the paper is extensive at 40 pages overall. I commend the authors for using that space to include section 5.7 “Qualitative Analysis and Limitations” discussing and visualizing plenty of examples. The limitation arising from the quantization approach (Figure 16) would not have been as clear otherwise.

Speaking of quantization, the approach used to tokenize time series onto a fixed vocabulary reminds me of the 2020 paper “The Effectiveness of Discretization in Forecasting” by Rabanser et al., a related group of (former) Amazon researchers.

The large set of authors of Chronos also points to the NeurIPS 2023 paper “Large Language Models Are Zero-Shot Time Series Forecasters”, though the approach of letting GPT-3 or LLaMA-2 predict a sequence of numeric values directly is very different.

Links / 2024-03-25

Average Temperatures by Month Instead of Year ∞

This tweet is a prime example for why it’s hard to analyze one signal in a time series (here, its trend) without simultaneously adjusting for its other signal components (here, its seasonality).

If the tweet gets taken down, perhaps this screenshot on Mastodon remains.