Gamestudio Links
Zorro Links
Newest Posts
Blobsculptor tools and objects download here
by NeoDumont. 03/28/24 03:01
Issue with Multi-Core WFO Training
by aliswee. 03/24/24 20:20
Why Zorro supports up to 72 cores?
by Edgar_Herrera. 03/23/24 21:41
Zorro Trader GPT
by TipmyPip. 03/06/24 09:27
VSCode instead of SED
by 3run. 03/01/24 19:06
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
2 registered members (degenerate_762, Nymphodora), 1,012 guests, and 3 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
Page 1 of 2 1 2
How to adapt Lowpass cutoff period to market conditions? #445480
09/11/14 05:09
09/11/14 05:09
Joined: Nov 2013
Posts: 123
Mithrandir77 Offline OP
Member
Mithrandir77  Offline OP
Member

Joined: Nov 2013
Posts: 123
I have been trying to advance in coding some trend following systems and using this great contribution from jcl: MMI I found that with EUR/USD in the period going from 2009-2014 it makes 231% AR,Sharpe Ratio 1.14 , UI 5% but from 2002-2008 it 'makes' -9% AR, Sharpe Ratio -0.17 , UI 1960% -that's what I call a stomach-ache- laugh

How can there be so many differences? I have tried using many tools (MAMA, ShannonGain, KAMA, Hurst) to try to adapt to the market conditions or filter out when the market is not trending but didn't have much luck, if you want to look at everything I tried:

Click to reveal..

Code:
// Workshop 4: Trend Trading ///////////////////
#include <profile.c>

var MMI(vars Data,int TimePeriod)
//credits to jcl
{
	var m = Median(Data,TimePeriod);
	int i, nh=0, nl=0;
	for(i=1; i<TimePeriod; i++) {
		if(Data[i] > m && Data[i] > Data[i-1])
			nl++;
		else if(Data[i] < m && Data[i] < Data[i-1])
			nh++;
	}
	return 100.*(nl+nh)/(TimePeriod-1);
}

function run()
{
	//StartDate = 2002;
	//EndDate = 2008;	
	vars Price = series(price());		
	vars Trend = series(LowPass(Price,500));
	//vars Trendindicator = series(KAMA(Trend,500));	
	vars Signals = series(0);
	
	Stop = 4*ATR(100);
/*	
	if(valley(Trend)) {
		Signals[0] = 1;
		if(Sum(Signals+1,4) == 0)  // no signals in the previous 3 bars?
			enterLong();
		exitShort();  // close opposite position
	} else if(peak(Trend)) {
		Signals[0] = 1;
		if(Sum(Signals+1,4) == 0)
			enterShort();
		exitLong();
	}
*/	
	//vars Hursts = series(Hurst(Price,100));
	var HurstExponent = Hurst(Price,100);
/*
	if(HurstExponent > 0.5) {
		if(valley(Trend))
			enterLong();
		else if(peak(Trend))
			enterShort();
	}	else {
		exitShort();
		exitLong();
	}
*/
	
	/*if(valley(Trend)) {
		if(HurstExponent > 5)  // no signals in the previous 3 bars?
			enterLong();
		exitShort();  // close opposite position
	} else if(peak(Trend)) {
		if(HurstExponent > 5)
			enterShort();
		exitLong();
	}*/

	
	vars Meanness = series(MMI(Price,200));
	vars Filter = series(LowPass(Meanness,500));
	
	if( valley(Trend) ){
		exitShort();  // close opposite position
		if(falling(Filter))
			enterLong();
	} else if( peak(Trend) ) {
		exitLong();
		if(falling(Filter))
			enterShort();
	}
	
	
	/*MAMA(Price,0.05,0.05);
	vars MAMAs = series(rMAMA);
	vars FAMAs = series(rFAMA);*/
	
	/*if( crossUnder(FAMAs,MAMAs) ){
		exitShort();  // close opposite position
		if(falling(Filter))
			enterLong();
	} else if( crossOver(FAMAs,MAMAs) ) {
		exitLong();
		if(falling(Filter))
			enterShort();
	}*/
	
	//vars FilterShannon = series(LowPass(series(ShannonGain(Price,500)),500));
	/*vars FilterShannon = series(ShannonGain(Price,500));
	if( valley(Trend) ){
		exitShort();  // close opposite position
		if(crossOver(FilterShannon,0))
			enterLong();
	} else if( peak(Trend) ) {
		exitLong();
		if(crossUnder(FilterShannon,0))
			enterShort();
	}*/
	
	
	/*BBands(Price,30,2,2,MAType_MAMA);
	plot("price",Price[0],MAIN|LINE,BLACK);
	plot("Bollinger1",rRealUpperBand,BAND1,0x000000CC);
  	plot("Bollinger2",rRealLowerBand,BAND2,0x800000FF);
  	plot("Trend",Trend[0],LINE,RED);
  	plot("Filter",Filter[0],NEW|LINE,BLACK);
	plot("Hurst",HurstExponent*100,NEW|LINE,BLACK);
	plot("Shannon",FilterShannon[0],NEW|LINE,BLACK);*/
	
	/*plot("price",Price[0],MAIN|LINE,BLACK);
	plot("lowpass",Trend,LINE,RED);
	plot("kama",kavg,LINE,BLUE);*/
	
	//PlotWidth = 600;
	//PlotHeight1 = 300;
//	plotTradeProfile(50);
	set(LOGFILE); // log all trades
}




So, in addition to solve this issue, I would like to know in general how to adapt a LowPass to the market conditions. I understood in the tutorial that they use 1000 as cutoff to catch the 2-month or more trends, but is there a way to set this according to the market like the way it is done in workshop 5 using DominantPeriod as the cutoff of the HighPass? Actually the code for that part is:

Code:
vars Period = series(DominantPeriod(Price, 30));
var LowPeriod = LowPass(Period, 500);



and again the doubt: why using 500? In this case maybe because it's an example but in designing an -almost at least- parameter free strategy I would like to know if there's a way or some heuristic to choose the cutoff period of the LowPass.

Thanks beforehand!

Re: How to adapt Lowpass cutoff period to market conditions? [Re: Mithrandir77] #445508
09/11/14 22:05
09/11/14 22:05
Joined: Nov 2013
Posts: 123
Mithrandir77 Offline OP
Member
Mithrandir77  Offline OP
Member

Joined: Nov 2013
Posts: 123
Just in case, I add that I have used fxcm historical data.

Re: How to adapt Lowpass cutoff period to market conditions? [Re: Mithrandir77] #445518
09/12/14 08:31
09/12/14 08:31
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
The 500 is indeed an arbitrary value, for keeping the example simple. In a real system you would check how the 500 affect the result, and whether the system becomes more robust when the parameter is optimized.

Re: How to adapt Lowpass cutoff period to market conditions? [Re: jcl] #445649
09/18/14 22:00
09/18/14 22:00
Joined: Nov 2013
Posts: 123
Mithrandir77 Offline OP
Member
Mithrandir77  Offline OP
Member

Joined: Nov 2013
Posts: 123
Originally Posted By: jcl
The 500 is indeed an arbitrary value, for keeping the example simple. In a real system you would check how the 500 affect the result, and whether the system becomes more robust when the parameter is optimized.


Thanks for your input jcl. I have tried to understand your advice and read a bit more meanwhile and tried different indicators that try to detect when the market is not in 'trend mode' but without success.

I don't want to be spoon-fed but could you elaborate a bit more how to 'check how the 500 affect the result'? , the only thing I can think of is a signficance statistical test but even in that case I am lost in how to go about performing it or which one should I use.

Re: How to adapt Lowpass cutoff period to market conditions? [Re: Mithrandir77] #445653
09/19/14 07:58
09/19/14 07:58
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Use the optimize function for this. You'll then get a performance chart as in Workshop 5, which shows you how the lowpass period affects the result. If it's more or less flat, leave it at 500. If it has a clear peak, it's a parameter to be optimized.

Re: How to adapt Lowpass cutoff period to market conditions? [Re: jcl] #445756
09/23/14 14:20
09/23/14 14:20
Joined: Nov 2013
Posts: 123
Mithrandir77 Offline OP
Member
Mithrandir77  Offline OP
Member

Joined: Nov 2013
Posts: 123
Originally Posted By: jcl
Use the optimize function for this. You'll then get a performance chart as in Workshop 5, which shows you how the lowpass period affects the result. If it's more or less flat, leave it at 500. If it has a clear peak, it's a parameter to be optimized.


Ok, I have used this code for optimization:

Click to reveal..

Code:
var MMI(vars Data,int TimePeriod)
{
	var m = Median(Data,TimePeriod);
	int i, nh=0, nl=0;
	for(i=1; i<TimePeriod; i++) {
		if(Data[i] > m && Data[i] > Data[i-1])
			nl++;
		else if(Data[i] < m && Data[i] < Data[i-1])
			nh++;
	}
	return 100.*(nl+nh)/(TimePeriod-1);
}

function run()
{
	set(PARAMETERS);
	StartDate = 2002;
	EndDate = 2008;	
	//NumWFOCycles = 10;
	LookBack = 3000;
	vars Price = series(price());
	var period = optimize(500,50,3000,50);
	var mmiperiod = optimize(200,50,1200);
	vars Trend = series(LowPass(Price,period));	
	
	Stop = ATR(100)*optimize(4, 2, 8);
	
	vars Meanness = series(MMI(Price,period));
	vars Filter = series(LowPass(Meanness,mmiperiod));	
	
	if( valley(Trend) ){		
		exitShort();  // close opposite position
		if(falling(Filter))
			enterLong();
	} else if( peak(Trend) ) {		
		exitLong();
		if(falling(Filter))
			enterShort();
	}
}




And here are the parameters charts:

Lowpass period (parameter 1):
Click to reveal..



MMI lowpass period (parameter 2):
Click to reveal..



Stop factor (parameter 3):
Click to reveal..




So they are more or less flat except for the lowpass period, is it because it is the first parameter to be optimized and the others were optimized after the first was optimized?

Then I used WFO:
(I simply added NumWFOCycles = 10; after LookBack = 3000; )

and this is the result of the strategy:

Click to reveal..

Walk-Forward Test Workshop4_2Modfied EUR/USD - performance report

Simulation period 01.07.2002-31.12.2008
Test period 06.01.2005-31.12.2008
WFO test cycles 9 x 2674 bars (23 weeks)
Training cycles 10 x 15152 bars (130 weeks)
Monte Carlo cycles 200
Lookback time 3000 bars (25 weeks)
Assumed slippage 10.0 sec
Spread 2.5 pips (roll 0.01/-0.02)
Contracts per lot 1000.0

Gross win/loss 663$ / -428$ (+3282p)
Average profit 59$/year, 4.93$/month, 0.23$/day
Max drawdown -94$ 40% (MAE -102$ 43%)
Total down time 84% (TAE 43%)
Max down time 159 weeks from Mar 2005
Largest margin 5.00$
Trade volume 254213$ (63784$/year)
Transaction costs -48$ spr, -0.27$ slp, -0.27$ rol
Capital required 87$

Number of trades 265 (67/year, 2/week, 1/day)
Percent winning 21%
Max win/loss 100$ / -9.74$
Avg trade profit 0.89$ 12.4p (+167.9p / -28.4p)
Avg trade slippage -0.00$ -0.0p (+0.6p / -0.2p)
Avg trade bars 42 (+138 / -17)
Max trade bars 902 (7 weeks)
Time in market 47%
Max open trades 1
Max loss streak 32 (uncorrelated 26)

Annual return 68%
Profit factor 1.55 (PRR 1.26)
Sharpe ratio 0.85
Kelly criterion 1.05
R2 coefficient 0.000
Ulcer index 21.8%
Prediction error 55%

Confidence level AR DDMax Capital

10% 85% 74$ 69$
20% 76% 84$ 78$
30% 69% 93$ 86$
40% 64% 100$ 92$
50% 61% 107$ 97$
60% 55% 119$ 108$
70% 49% 133$ 120$
80% 43% 152$ 137$
90% 37% 179$ 160$
95% 33% 204$ 182$
100% 24% 273$ 242$

Portfolio analysis OptF ProF Win/Loss Wgt% Cycles

EUR/USD .062 1.55 55/210 100.0 /\\XXXXX/
EUR/USD:L .093 1.69 26/108 60.5 /\\//\///
EUR/USD:S .039 1.42 29/102 39.5 /\\\\/\\/


And the equity curve. What is the meaning of the blue equity curve below zero?



Anyway, It improved but I have some concerns:

-I used a big optimize range which according to the manual can lead to overfitting but how can I narrow the range without knowing a priori where the optimal value? -This is precisely where all the problem started-

-As I asked before, I don't like that blue and red equity curve below 0. What is the meaning of that? Should I worry?

Thanks!

Re: How to adapt Lowpass cutoff period to market conditions? [Re: Mithrandir77] #445765
09/23/14 17:45
09/23/14 17:45
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Last question first - you can see that this system is not profitable when volatility is low, as in the test period except for the last part.

I would reduce the optimization range of the lowpass period, and not optimize the other 2 parameters.

Re: How to adapt Lowpass cutoff period to market conditions? [Re: jcl] #445773
09/24/14 03:45
09/24/14 03:45
Joined: Nov 2013
Posts: 123
Mithrandir77 Offline OP
Member
Mithrandir77  Offline OP
Member

Joined: Nov 2013
Posts: 123
Originally Posted By: jcl
Last question first - you can see that this system is not profitable when volatility is low, as in the test period except for the last part.

I would reduce the optimization range of the lowpass period, and not optimize the other 2 parameters.


Thank you jcl, I guess that the blue equity below 0 is a margin call isn't it? And the red below 0 are actually 'draw ups'

Anyway I trained as you said, only optimizing lowpass period, with this code:
Click to reveal..

Code:
var MMI(vars Data,int TimePeriod)
{
	var m = Median(Data,TimePeriod);
	int i, nh=0, nl=0;
	for(i=1; i<TimePeriod; i++) {
		if(Data[i] > m && Data[i] > Data[i-1])
			nl++;
		else if(Data[i] < m && Data[i] < Data[i-1])
			nh++;
	}
	return 100.*(nl+nh)/(TimePeriod-1);
}

function run()
{
	set(PARAMETERS);
	StartDate = 2002;
	EndDate = 2008;	
	NumWFOCycles = 10;
	LookBack = 3000;
	vars Price = series(price());
	var period = optimize(1200,1000,2000,50);//optimize(500,50,3000,50);
	var mmiperiod = 100;//optimize(200,50,1200);
	vars Trend = series(LowPass(Price,period));	
	
	Stop = ATR(100)*4;//optimize(4, 2, 8);
	
	vars Meanness = series(MMI(Price,period));
	vars Filter = series(LowPass(Meanness,mmiperiod));	
	
	if( valley(Trend) ){		
		exitShort();  // close opposite position
		if(falling(Filter))
			enterLong();
	} else if( peak(Trend) ) {		
		exitLong();
		if(falling(Filter))
			enterShort();
	}
}




Which gave this result:
Click to reveal..

Walk-Forward Test Workshop4_2Pb EUR/USD - performance report

Simulation period 01.07.2002-31.12.2008
Test period 06.01.2005-31.12.2008
WFO test cycles 9 x 2674 bars (23 weeks)
Training cycles 10 x 15152 bars (130 weeks)
Monte Carlo cycles 200
Lookback time 3000 bars (25 weeks)
Assumed slippage 10.0 sec
Spread 2.5 pips (roll 0.01/-0.02)
Contracts per lot 1000.0

Gross win/loss 220$ / -177$ (+588p)
Average profit 11$/year, 0.88$/month, 0.04$/day
Max drawdown -109$ 258% (MAE -133$ 314%)
Total down time 80% (TAE 31%)
Max down time 130 weeks from Jun 2006
Largest margin 5.00$
Trade volume 88337$ (22165$/year)
Transaction costs -17$ spr, 0.36$ slp, -0.16$ rol
Capital required 100$

Number of trades 92 (24/year, 1/week, 1/day)
Percent winning 16%
Max win/loss 70$ / -7.81$
Avg trade profit 0.46$ 6.4p (+203.8p / -32.1p)
Avg trade slippage 0.00$ 0.1p (+0.9p / -0.1p)
Avg trade bars 86 (+371 / -30)
Max trade bars 1316 (11 weeks)
Time in market 33%
Max open trades 1
Max loss streak 20 (uncorrelated 29)

Annual return 11%
Profit factor 1.24 (PRR 0.82)
Sharpe ratio 0.21
Kelly criterion 0.40
R2 coefficient 0.000
Ulcer index 69.5%
Prediction error 86%

Confidence level AR DDMax Capital

10% 26% 41$ 40$
20% 24% 45$ 44$
30% 22% 49$ 48$
40% 19% 57$ 55$
50% 18% 62$ 58$
60% 16% 69$ 65$
70% 15% 76$ 71$
80% 13% 92$ 85$
90% 11% 107$ 98$
95% 10% 121$ 110$
100% 8% 151$ 136$

Portfolio analysis OptF ProF Win/Loss Wgt% Cycles

EUR/USD .023 1.24 15/77 100.0 \X\X\\\\X
EUR/USD:L .047 1.49 9/39 107.2 \\\/\\\\/
EUR/USD:S .000 0.96 6/38 -7.2 \/\\\\\\\


And this equity I don't understand, now the unprofitable period became profitable and viceversa, how can be that? frown


Re: How to adapt Lowpass cutoff period to market conditions? [Re: Mithrandir77] #445859
09/27/14 17:47
09/27/14 17:47
Joined: Sep 2003
Posts: 929
Spirit Offline

Moderator
Spirit  Offline

Moderator

Joined: Sep 2003
Posts: 929
Equity below 0 means your strategy makes no profit but a loss, this is not a margin call. Margin call happens when you have not enough capital and can not cover the margin of open trades.

Your last strategy looks as if it makes very few trades, maybe you should adjust your parameters for more trades?

Re: How to adapt Lowpass cutoff period to market conditions? [Re: Spirit] #446118
10/06/14 01:43
10/06/14 01:43
Joined: Nov 2013
Posts: 123
Mithrandir77 Offline OP
Member
Mithrandir77  Offline OP
Member

Joined: Nov 2013
Posts: 123
Spirit, thanks for your response. I have been busy with studies and work the last weeks. I have decided to reaproach this script by adding a filter that measures volatility. I could have used ATR but for some reason I found Alligator interesting so here it is the script and the results:

script:
Click to reveal..

function run()
{
//set(PARAMETERS);
StartDate = 2002;
EndDate = 2008;
NumWFOCycles = 10;
LookBack = 2000;
vars Price = series(price());
var period = 500;
vars Trend = series(LowPass(Price,period));

Stop = ATR(100)*4;//optimize(4, 2, 8);

vars Meanness = series(MMI(Price,200));
vars Filter = series(LowPass(Meanness,period));

Alligator(series(MedPrice()));
var diff = FisherN(series(abs(rBlue-rGreen) + abs(rGreen-rRed)),500);
vars diffs = series(diff);
var Threshold = 2.5;//0.1;//1.0;//1.5;//2.5

static bool Filter2;
if (crossOver(diffs,Threshold) or crossUnder(diffs,-Threshold))
Filter2 = true;
if (crossUnder(diffs,Threshold) or crossOver(diffs,-Threshold))
Filter2 = false;

if( valley(Trend) ){
exitShort(); // close opposite position
if(falling(Filter) and Filter2)
enterLong();
} else if( peak(Trend) ) {
exitLong();
if(falling(Filter) and Filter2)
enterShort();
}
set(LOGFILE); // log all trades
}


performance report:
Click to reveal..

Walk-Forward Test Workshop4_2Pc EUR/USD - performance report

Simulation period 01.05.2002-31.12.2008
Test period 30.11.2004-31.12.2008
WFO test cycles 9 x 2742 bars (23 weeks)
Training cycles 10 x 15538 bars (134 weeks)
Monte Carlo cycles 200
Lookback time 2000 bars (17 weeks)
Assumed slippage 10.0 sec
Spread 2.3 pips (roll -0.10/0.04)
Contracts per lot 1000.0

Gross win/loss 117$ / -44$ (+941p)
Average profit 18$/year, 1.48$/month, 0.07$/day
Max drawdown -34$ 47% (MAE -42$ 58%)
Total down time 41% (TAE 9%)
Max down time 68 weeks from Jul 2007
Largest margin 5.00$
Trade volume 17479$ (4278$/year)
Transaction costs -3.02$ spr, 0.10$ slp, -0.88$ rol
Capital required 34$

Number of trades 17 (5/year, 1/week, 1/day)
Percent winning 41%
Max win/loss 42$ / -16$
Avg trade profit 4.28$ 55.4p (+215.8p / -56.9p)
Avg trade slippage 0.01$ 0.1p (+0.6p / -0.3p)
Avg trade bars 151 (+346 / -14)
Max trade bars 542 (4 weeks)
Time in market 10%
Max open trades 1
Max loss streak 6 (uncorrelated 6)

Annual return 52%
Profit factor 2.65 (PRR 1.25)
Sharpe ratio 0.75
Kelly criterion 1.08
R2 coefficient 0.385
Ulcer index 6.0%
Prediction error 125%

Confidence level AR DDMax Capital

10% 103% 14$ 17$
20% 93% 17$ 19$
30% 85% 18$ 21$
40% 82% 20$ 22$
50% 77% 21$ 23$
60% 73% 22$ 24$
70% 67% 25$ 27$
80% 62% 28$ 29$
90% 55% 32$ 33$
95% 50% 36$ 36$
100% 33% 58$ 54$

Portfolio analysis OptF ProF Win/Loss Wgt% Cycles

EUR/USD .166 2.65 7/10 100.0 ./XX//\\\
EUR/USD:L .169 3.14 5/6 88.4 ..\///\\\
EUR/USD:S .147 1.60 2/4 11.6 .//\...\.

plot:
Click to reveal..




As you can see, I used Alligator lines closeness as a measure of volatility, I normalize their differences and when they go out of a threshold (which is higher the less volatile the market is) I consider it a signal to buy or sell.

This has turned the loser system into a winning one with good Sharpe Ratio for a trend system and an ulcer of 6%. What do you think? Do you think It has any bias or curve fitting? I look forward to hearing your inputs. Thanks!

Page 1 of 2 1 2

Moderated by  Petra 

Powered by UBB.threads™ PHP Forum Software 7.7.1