Hunting a Systemic Training (or other?) Bias?

Hi jcl. Anecdotally, it seems scripts, both Z & home-grown, in general perform worse than Tested - speaking in terms of AR. I realize your perspective seems to be so long as DD<CR and the account hasn't blown up the script is working and will most likely eventually prove out; but most of us would like to make a few extra $ sooner than "eventually"...

I've been wondering if this would be a reasonable/possible way to investigate at least part of this:

Using the Z systems since they have a fairly regular release cycle, they each have a Tested equity curve and a "run life" of a few months after release before being superseded by the next version. I presume you run internally each version with the same default 50/10 setting? Would it be possible to add to each version's equity curve 2 "tails": Test and actual Trade results over the run life?

There would be 2 objectives:

1) Are the Test results over the run life consistently better or worse than the original Test period results revealing some bias?

2) Are the Trade results over the run life randomly better & worse than Test as they should be? Or is there a systemic Training (or other) bias one way or the other?

Either way, any bias found would be a candidate for identification & removal from the development process...

Does this make sense? Has enough Z version history accumulated to make it worth doing? I suspect only you folks would have enough consistent (same Margin/Risk) history to do this...

Thanks.

In fact DD < CR is not the best criteria for comparing live trading with the backtest. You can find some methods here: http://www.financial-hacker.com/the-cold-blood-index/.

The next Zorro update will also contain a "retest" mode for backtesting the current live period. But we can't add tails to the Z system backtest curves. We don't publish live results anyway, and even if we did, the tails could be only for a past version and not for the current version. It's just not worth the effort.

Thanks jcl. I will re-read that article, but as I recall it was also a lot of effort.

Have you undertaken to study in some other way how the different Z versions have Traded vs. Tested to see if there is any better/worse bias?

Thanks.

Yes, with most of our accounts so far the Z3 and Z7 results have been in the same range as the backtest, Z12 is slightly worse, and for Z8 it's too early to tell, but it seems to also match the backtest.