Strategy decline after update to Zorro 1.51

Posted By: savvy

Strategy decline after update to Zorro 1.51 - 11/07/16 09:18

Hi. I am testing a trading system that performed fairly well with Zorro 1.46. With and without setting the TICK flag, Profit factor, Sharpe ratio, R2 coeff., Ulcer index and Cycle performance were all in a good range. Even with parameters that deviated quite a bit from their optimum the system appeared robust for the whole test period, although less profitable.

Now, after updating to Zorro 1.51 the system performance is entirely gone. It's almost as if I am trading a different system. The trades of both versions are hardly comparable anymore. Is it because of the changed data formate from .bar to .t6?

What might be the reason for such a drastic decline? Maybe my system was futile from the beginning but how can I know that the current test results with the new data formate are any more reliable than the old ones?

Posted By: jcl

Re: Strategy decline after update to Zorro 1.51 - 11/07/16 09:36

See the "Updating" remarks under "What's New". If your script is scalping and exploits tiny intrabar price changes, you should set the EXTRADATA flag. Otherwise the bar open and close prices are simulated only.

Posted By: savvy

Re: Strategy decline after update to Zorro 1.51 - 11/07/16 10:35

Thank you. You are right it's all there. I am using tight stop losses and the missing EXTRADATA flag was the problem.

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/07/16 13:31

Wow, seems like it was the source of my problem too. My BarPeriod is 60 and I use only SMA of close prices with period 5 for entries. Without EXTRADATA flag I got AR 41% and SR 0.46, with the flag I get AR 72% and SR 0.75. I guess I'll use the flag as default.

Posted By: Sphin

Re: Strategy decline after update to Zorro 1.51 - 11/07/16 18:18

I just asked for this in another thread and was told that the difference should be marginal. If it is not, there could be a large randomness in the strategy. I face this problem even on daily bars and I'm not quite sure how to handle it.

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/07/16 18:28

Originally Posted By: Sphin

I just asked for this in another thread and was told that the difference should be marginal. If it is not, there could be a large randomness in the strategy. I face this problem even on daily bars and I'm not quite sure how to handle it.

Don't know which thread it is so I'll continue in this one...

I don't think there's much randomness in the strategy I'm testing. It's a simple mean reversion strategy based on SMA and hourly data. It makes hundreds of trades per year so some minor differences should be lost in the large sample.

It's this one (you can check it and judge for yourself):

http://people.brandeis.edu/~blebaron/wps/fxnyc.pdf

If anyone's interested I can paste the code too.

Posted By: Sphin

Re: Strategy decline after update to Zorro 1.51 - 11/07/16 19:01

It was not my opinion but jcl's. For me it's more interessting what comes closer to reality. If a strategy includes calculations based on open and close prices of a bar and without EXTRADATA both are simulated I guess setting this flag includes the real prices - so I think setting this flag might be more realistic in this case. But maybe I didn't understand it, as said, I'm not sure.

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/07/16 19:17

Originally Posted By: Sphin

It was not my opinion but jcl's. For me it's more interessting what comes closer to reality. If a strategy includes calculations based on open and close prices of a bar and without EXTRADATA both are simulated I guess setting this flag includes the real prices - so I think setting this flag might be more realistic in this case. But maybe I didn't understand it, as said, I'm not sure.

I know - I hope he'll read it here eventually

And I agree with the rest you wrote - seems like with the flag the results should be more realistic. Because of that I would prefer if it was default (implicit). IMHO this new feature has potential to cause lot of confusion in the future (and among new users).

Posted By: jcl

Re: Strategy decline after update to Zorro 1.51 - 11/08/16 12:53

You have a point in making that flag default. Maybe we'll do that in the next update. We found not much difference with the workshops and Z systems, but it might be different with other systems.

Posted By: Finstratech

Re: Strategy decline after update to Zorro 1.51 - 11/08/16 16:09

I use range bars and was not aware aware that I need EXTRADATA flag to rely on precise closing prices. My entries did act "wacky" without the flag. I will test it with the flag now.

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/09/16 20:46

Now I have the feeling that for one of my (relatively) high-frequency strategies the backtest produces much better results than in the previous version of Zorro. Even with the EXTRADATA flag set. For example Sharpe Ratio has almost doubled in one year compared with the old results I saved while the volume and transaction costs remained almost the same.

Unfortunately I don't have the old version anymore. Can anyone please share an old version of Zorro (1.4X) so I can find out if there's really a difference or whether it was my fault? Or is there an official place to download previous releases?

Posted By: MIGI

Re: Strategy decline after update to Zorro 1.51 - 11/10/16 12:03

Here is a Zorro 1.44 https://www.dropbox.com/s/0rxfj4uuyrggh1q/Zorro%201.44.exe?dl=0

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/10/16 14:07

Thank you very much, MIGI.

It seems that there is a big difference between the versions indeed. This is a backtest of the strategy for 2014. Same code, same data (FXCM .bar files), same asset configuration, very different results. There is no training involved and it's very simple mean reversion strategy based on one indicator with no parameters. Before I investigate further - any ideas why is this happening?

Code:

Test IsIbs EUR/USD

Simulated account   AssetsFix 
Bar period          1 hour (avg 86 min)
Test period         07.01.2014-31.12.2014 (5963 bars)
Lookback period     80 bars (3 days)
Monte Carlo cycles  200
Simulation mode     Realistic ticks (slippage 5.0 sec)
Avg ticks per bar   61.3
Spread              0.5 pips (roll -0.42/0.06)
Contracts per lot   100000.0

Gross win/loss      97530$ / -78307$ (+1922p)
Average profit      19605$/year, 1634$/month, 75$/day
Max drawdown        -5875$ 31% (MAE -5875$ 31%)
Total down time     92% (TAE 94%)
Max down time       26 days from Feb 2014
Max open margin     2236$
Max open risk       1400$
Trade volume        173272874$ (176720192$/year)
Transaction costs   -6525$ spr, 8668$ slp, -116$ rol
Capital required    12512$

Number of trades    1305 (1331/year, 26/week, 6/day)
Percent winning     65.8%
Max win/loss        1030$ / -1962$
Avg trade profit    15$ 1.5p (+11.4p / -17.6p)
Avg trade slippage  6.64$ 0.7p (+1.4p / -0.7p)
Avg trade bars      3 (+2 / -5)
Max trade bars      35 (35 hours)
Time in market      83%
Max open trades     1
Max loss streak     5 (uncorrelated 7)

Annual return       157%
Profit factor       1.25 (PRR 1.15)
Sharpe ratio        2.48
Kelly criterion     3.92
R2 coefficient      0.873
Ulcer index         9.0%

Code:

Test IsIbs EUR/USD

Simulated account   AssetsFix 
Bar period          1 hour (avg 86 min)
Test period         07.01.2014-31.12.2014 (5963 bars)
Lookback period     80 bars (3 days)
Monte Carlo cycles  200
Assumed slippage    10.0 sec
Avg ticks per bar   61.3
Spread              0.5 pips (roll -0.42/0.06)
Contracts per lot   100000.0

Gross win/loss      94183$ / -81382$ (+1280p)
Average profit      13056$/year, 1088$/month, 50$/day
Max drawdown        -6644$ 52% (MAE -6639$ 52%)
Total down time     96% (TAE 95%)
Max down time       7 weeks from Oct 2014
Max open margin     2236$
Max open risk       1400$
Trade volume        173272829$ (176720146$/year)
Transaction costs   -6525$ spr, 2238$ slp, -116$ rol
Capital required    13858$

Number of trades    1305 (1331/year, 26/week, 6/day)
Percent winning     64.4%
Max win/loss        1041$ / -1984$
Avg trade profit    9.81$ 1.0p (+11.2p / -17.5p)
Avg trade slippage  1.71$ 0.2p (+0.4p / -0.2p)
Avg trade bars      3 (+2 / -5)
Max trade bars      35 (35 hours)
Time in market      83%
Max open trades     1
Max loss streak     8 (uncorrelated 7)

Annual return       94%
Profit factor       1.16 (PRR 1.07)
Sharpe ratio        1.64
Kelly criterion     2.85
R2 coefficient      0.705
Ulcer index         16.4%
Prediction error    122%

Posted By: Sphin

Re: Strategy decline after update to Zorro 1.51 - 11/10/16 15:19

How funny! In the thread I already mentioned before I began my posting with 'different results in different versions'. The last one was caused by the EXTRADATA flag but the cause in prior versions I don't know. You can find my post and jcl's answers here: http://www.opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=461906#Post461906.

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/10/16 17:04

OK, lets go deeper

I exported all the prices and indicator values into .csv files and compared them. They are all the same. The trade entry and exit times match exactly and rollover as well. So the only difference are entry and exit prices and the corresponding profit.

Example trade from the trade log (the second number is from the newer Zorro version):

Code:

Open 2014-01-08 14:00 @ 1.3595 and @ 1.3596
Close 2014-01-08 16:00 @ 1.3601 and @ 1.3600
Profit: -55.72 and -37.66

Exported prices (OHLC):

Code:

2014-01-08 13:00, 1.35843, 1.35950, 1.35827, 1.35903
2014-01-08 14:00, 1.35903, 1.35989, 1.35685, 1.35955
2014-01-08 15:00, 1.35955, 1.36093, 1.35907, 1.36023
2014-01-08 16:00, 1.36023, 1.36194, 1.36003, 1.36012
2014-01-08 17:00, 1.36012, 1.36056, 1.35858, 1.35873

Why is there the big difference in profit? And why are entry and exit prices rounded differently in each version? Why is there any rounding at all? Wouldn't it be better for everyone if the logs contained prices with maximum possible precision?

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/10/16 17:29

I think it has something to do with slippage. With the new Zorro there's 5 seconds default slippage. It's funny actually, because with increasing slippage the profit always increases:

Code:

slippage=0: AR 73%, SR 1.35
slippage=1: AR 89%, SR 1.57
slippage=2: AR 107%, SR 1.80
slippage=3: AR 125%, SR 2.02
slippage=4: AR 141%, SR 2.25
slippage=5: AR 157%, SR 2.48
slippage=6: AR 163%, SR 2.70
...
slippage=2000: AR 2504%, SR 15.78

Wish it worked that way:) But even with slippage=0 the results don't match the results from the old Zorro version so I guess there has to be another difference.

Btw. this should be fixed as soon as possible, the backtests with slippage are useless and people can lose money because of it. It's especially unfortunate because of the default settings (5 second slippage) so maybe an email to all current users should be considered.

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/10/16 17:36

So the new version with Slippage set to 0 matches the old version with Slippage set to 0 (maybe there was also default slippage > 0 in the old version?). Here are the values I get with Zorro 1.44:

Code:

slippage=0: AR 73%, SR 1.35
slippage=1: AR 75%, SR 1.37
slippage=2: AR 77%, SR 1.40
slippage=3: AR 79%, SR 1.43
slippage=4: AR 81%, SR 1.46
slippage=5: AR 83%, SR 1.49
slippage=6: AR 85%, SR 1.52
...
slippage=2000: AR 2663%, SR 15.91

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/10/16 18:15

Another note - maybe it has to do something with the fact that the strategy is mean reverting. When testing different strategy (Intraday Seasonality) it works more like one would expect.

Posted By: Sphin

Re: Strategy decline after update to Zorro 1.51 - 11/10/16 19:49

Referring slippage I just saw the actual last point on the bug list:

> Slippage simulation was not 100% neutral in some cases, but affected by the price slope inside the current bar (fixed in V 1.52.0).

But the 1.52.0 ist not yet available, it's the 1.51.9 as beta on download page.

Posted By: pcz

Re: Strategy decline after update to Zorro 1.51 - 11/11/16 13:13

Originally Posted By: Sphin

Referring slippage I just saw the actual last point on the bug list:

> Slippage simulation was not 100% neutral in some cases, but affected by the price slope inside the current bar (fixed in V 1.52.0).

But the 1.52.0 ist not yet available, it's the 1.51.9 as beta on download page.

I've just tried it - the problem is not fixed in the beta version.