Some people will probably disagree with this. But what I’m going to share is something that has helped me understand the limitations of backtesting and how I can get a much better idea of how a trading system will perform in the real world.
But as they say, past performance does not guarantee future results. So how can you protect yourself from the dangers of backtesting?
In this post, I will show you why you should backtest your strategy multiple times, at least 8, to be exact.
1. You Might be Distracted When You Backtest a System
In spite of your best efforts to focus on your testing, things can distract you. Your dog might have crapped on your carpet, your kids could be yelling, or a movie could be running the background.
I could tell you to turn off the porn, but let’s face it, shit happens. Some things are under your control, some aren’t.
Even the things that are under your control can get away from you from time-to-time…
Bottom line: There will be days when you are more focused on your testing than others. Therefore, if you only test a system once, you may be having a good testing day or a bad one.
Here are my actual results for one system. Notice how much the results can vary.
The last two tests were done on the same day. Guess how focused I was then.
If you discard an idea, solely based on one unfocused test, then you could potentially be throwing away a very profitable idea.
Vice versa, if you are uber focused and have a great testing session, that may not reflect real-life conditions. You might miss trades because you are sleeping. You might misread a trade.
Testing 8 times will give you a good sample of how you may trade in both distracted and focused states.
2. You Can Compare Individual Trades
A series of 8 backtests will give you a fairly large sample size of trades to study. You can then match them up and try to figure out why you took a trade on one test and why you didn’t the same trade on another round of testing.
From there, you can find out things like:
- Are your rules not specific enough?
- Was there hindsight bias involved in your trading decision (AKA, did you peek)?
- Are you really concentrating on your testing?
- Why did the same setup produce different results?
The reality is that if you are really following the same set of rules, your results should be pretty similar. If they aren’t, then you have to figure out why.
Having 8 rounds of testing to examine will help you do this.
3. You Can Create a Library of Good Trades
A large sample size of trades will help you compile a library of past successful trades. One thing that I’m experimenting with is creating flash cards of trades that worked in the past.
This is something that I learned in this interview. Here is a small sample of some flash cards that I created to examine possible patterns in the EURUSD London Session.
Think about how engraining successful patterns in your mind can help you hone your instinct. This might not work for everyone, but I find that it helps me.
4. You Will Notice Possible Improvements
After looking at 8 rounds of testing, you will notice ways that you can possibly improve your results. For example, I noticed that one improvement that I could make to the trading results in #1 above, is that I could add an RSI to the method.
As I’m writing this, the improvement looks good. It has taken my average win rate from 62.63% to about 85%, with about the same return on capital. There is still a lot more testing to do, but I’m hopeful.
If it turns out that this change does improve my win rate, then all I have to do is risk more per trade. Regardless of the ultimate outcome, I might not have thought to test this if I didn’t put in the repetitions.
5. Practice Makes…One Hell of a Lot Better
You probably think that being funny is an innate talent, but you might be surprised at how much work goes into being a great stand-up comedian. They put in a ton of time to improve every single joke.
This is the same for anything in life. You already know all the sports analogies.
Backtesting is the “gym” for traders. Will being a “gym fighter” automatically make you a good live trader?
But you won’t get anywhere without practice. So put in the time.
Yeah, it is hard, I struggle with it too.
Alright, I know what you are thinking…
What if you are testing a 1 minute system? Backtesting 10+ years of data 8 times, is almost impossible.
In that case, I would recommend choosing random time periods over the total span of your data and testing that time period 8 times. For example, if you have data from 2001 to 2015, pick a random week to test from 2001, 2003, 2005, 2009 and 2013.
Obviously, the more time periods you can test, the better. But you get the idea.
Now get to work!
Do you disagree? Let me know in the comments below…