There was an article on mechanicalforex site where Daniel explained why he thinks that you should always backtest on at least 10 years of data. But, I can't find it anymore, I'm afraid you'll have to dig yourself. While we all have our opinions, I tend to mostly agree that strategy that is profitable in longer term is better, because there's a greater chance that it will continue to be profitable in the future.
I'd love to read the article if you do find it.
But I would argue that one-size-does-not-fit-all, and that includes backtesting with "as much data as possible". Personally my theory is that there is an "appropriate amount" of recent data to test with, but that too little is dangerous, and also perhaps too much is as well (at least that's my theory).
Consider how the trader is planning to utilize the bot in the bigger picture. In my case, I am not necessarily trying to build the supreme-holy-grail bot that will lay golden eggs and make me rich enough to buy an island country (though I wouldn't complain if I stumbled upon such a gem).
Instead, I envision more of a dynamic/rolling portfolio of bots (ie, the legion) which actually grows and improves with... my brain. Yep, it would be myself that coaches the legion, and thus myself that would be contributing to its (hopefully ever increasing) success over time.
Therefore, I envision a different route. A route of safety and measured risk. But perhaps one with only "mediocre" bots on the playing field. That's ok, because each player in the legion will be judged, and if they don't make the grade ... they're soon replaced.
So in this scenario, I hope you can understand that consistency is critical... I think it is extremely important to have a consistent way to test and compare bots.
