In the world of quantitative trading, one of the most important tools for evaluating the effectiveness of a strategy is backtesting. But there is a more robust approach, less subject to the distortions of rigid models:
non-parametric backtesting. In this article, we will discover what it is, what advantages it offers, and how to concretely apply it in your operational analyses.
What is non-parametric backtesting
To explain what
non-parametric backtesting is, we must first explain what backtesting is
in general.
Well, backtesting refers to a procedure that allows verifying the validity of a trading strategy by applying it to historical data. The purpose is to estimate how it would have behaved in the past and, by extension,
predict how it might behave in the future.
A typical example of backtesting might consist of applying a strategy based on crossed moving averages on data from the last five years to evaluate profits, drawdowns, and success rates.
At this point, we can answer the question:
what is non-parametric backtesting? Basically, it is a backtesting that is not based on rigid models or optimized parameters.
In traditional backtesting (e.g., 20-period moving average), specific parameters are often selected and optimized to fit the historical data. This can lead to a problem called curve fitting: the strategy only seems to work because it has been "tailor-made" for that dataset.
Non-parametric backtesting, therefore, uses statistical techniques that analyze results without assuming any predefined form of relationship between variables. It focuses on
empirical evidence, not on fitting the model to the data.
In other words, non-parametric backtesting looks for repeatable and statistically significant patterns without forcing them into a predetermined form.
But let's make
a concrete example of non-parametric backtesting.
Imagine we want to test whether days when the market opens with a bearish gap greater than 1% offer a long-term advantage in the short term.
In parametric backtesting, we could use a regression or set defined thresholds for entering/exiting. In non-parametric backtesting, on the other hand,
we would analyze all days with gaps > -1%, observe the distribution of returns in the following days, and compare this distribution with that of days without gaps.
We are not looking for a formula: we are looking for
a real difference in data behaviors.
The advantages of non-parametric backtesting
Non-parametric backtesting is gaining popularity among the most attentive traders. Its strength lies in the ability to reduce statistical illusions and focus on real data. But here are three significant advantages of this technique.
Lower risk of overfitting
In parametric backtesting, each new parameter inserted is a potential additional error. You can end up building a perfect system... for the past. Non-parametric backtesting avoids this trap because it does not try to "force" a model to work.
More robustness to market variations
Non-parametric strategies tend to be more adaptable to changes in market regimes. Since they are not constrained by fixed models,
they are less sensitive to distortions caused by temporary conditions.
This is not a minor detail: only a strategy that is based on empirical evidence of behavior can better survive sideways, hyper-volatile, or irregular markets.
More visual and statistical approach
Non-parametric backtesting allows working on distributions, heatmaps, boxplots, analysis of variance, making the decision-making process more intuitive and transparent. This type of visualization is particularly useful when working with behavioral patterns and qualitative signals.
How to use the evidence of non-parametric backtesting
Using the evidence of non-parametric backtesting means validating hypotheses with greater statistical honesty. This means, for example...
- Avoiding arbitrary thresholds: instead of testing "if RSI < 30", it is good to evaluate the distribution of returns as a function of the RSI value, observing where the best results are concentrated.
- Using visual statistical tools: return histograms, boxplots, and kernel density estimation are much more meaningful than a simple equity curve based on fixed rules.
- Working on different frequencies: you should not limit yourself to daily or H1. After all, intraday data and non-parametric analyses on minute windows can reveal valuable anomalies.
Let's now make a concrete example of using non-parametric backtesting.
Imagine we want to understand if upside breakouts in the first 30 minutes of S&P500 trading bring an operational advantage. Here is the most correct procedure.
- Phase 1: Data collection. Take 5 years of intraday data (1-minute), identify all the days when the S&P500 breaks the high of the first 30 minutes, and measure the return until closing.
- Phase 2: Distributional analysis. Compare the distribution of returns of these "breakout days" with that of normal days (without breakouts). Do not set a fixed threshold or model: observe the shape of the distribution, calculate the mean, median, and variance.
- Phase 3: Statistical validation. With a Kolmogorov–Smirnov test or with a simple visual analysis, verify if the two distributions differ significantly. If so, we have evidence – not a model – on which to build our strategy, perhaps even discretionary.
Here, the result will not be a rule "buy at 9:31 and sell at 3:47", but rather a statistical awareness on which to base more robust choices.
