How to adapt Lowpass cutoff period to market conditions?

Posted By: Mithrandir77

How to adapt Lowpass cutoff period to market conditions? - 09/11/14 05:09

I have been trying to advance in coding some trend following systems and using this great contribution from jcl: MMI I found that with EUR/USD in the period going from 2009-2014 it makes 231% AR,Sharpe Ratio 1.14 , UI 5% but from 2002-2008 it 'makes' -9% AR, Sharpe Ratio -0.17 , UI 1960% -that's what I call a stomach-ache- laugh

How can there be so many differences? I have tried using many tools (MAMA, ShannonGain, KAMA, Hurst) to try to adapt to the market conditions or filter out when the market is not trending but didn't have much luck, if you want to look at everything I tried:

Click to reveal..

Code:
// Workshop 4: Trend Trading ///////////////////
#include <profile.c>

var MMI(vars Data,int TimePeriod)
//credits to jcl
{
	var m = Median(Data,TimePeriod);
	int i, nh=0, nl=0;
	for(i=1; i<TimePeriod; i++) {
		if(Data[i] > m && Data[i] > Data[i-1])
			nl++;
		else if(Data[i] < m && Data[i] < Data[i-1])
			nh++;
	}
	return 100.*(nl+nh)/(TimePeriod-1);
}

function run()
{
	//StartDate = 2002;
	//EndDate = 2008;	
	vars Price = series(price());		
	vars Trend = series(LowPass(Price,500));
	//vars Trendindicator = series(KAMA(Trend,500));	
	vars Signals = series(0);
	
	Stop = 4*ATR(100);
/*	
	if(valley(Trend)) {
		Signals[0] = 1;
		if(Sum(Signals+1,4) == 0)  // no signals in the previous 3 bars?
			enterLong();
		exitShort();  // close opposite position
	} else if(peak(Trend)) {
		Signals[0] = 1;
		if(Sum(Signals+1,4) == 0)
			enterShort();
		exitLong();
	}
*/	
	//vars Hursts = series(Hurst(Price,100));
	var HurstExponent = Hurst(Price,100);
/*
	if(HurstExponent > 0.5) {
		if(valley(Trend))
			enterLong();
		else if(peak(Trend))
			enterShort();
	}	else {
		exitShort();
		exitLong();
	}
*/
	
	/*if(valley(Trend)) {
		if(HurstExponent > 5)  // no signals in the previous 3 bars?
			enterLong();
		exitShort();  // close opposite position
	} else if(peak(Trend)) {
		if(HurstExponent > 5)
			enterShort();
		exitLong();
	}*/

	
	vars Meanness = series(MMI(Price,200));
	vars Filter = series(LowPass(Meanness,500));
	
	if( valley(Trend) ){
		exitShort();  // close opposite position
		if(falling(Filter))
			enterLong();
	} else if( peak(Trend) ) {
		exitLong();
		if(falling(Filter))
			enterShort();
	}
	
	
	/*MAMA(Price,0.05,0.05);
	vars MAMAs = series(rMAMA);
	vars FAMAs = series(rFAMA);*/
	
	/*if( crossUnder(FAMAs,MAMAs) ){
		exitShort();  // close opposite position
		if(falling(Filter))
			enterLong();
	} else if( crossOver(FAMAs,MAMAs) ) {
		exitLong();
		if(falling(Filter))
			enterShort();
	}*/
	
	//vars FilterShannon = series(LowPass(series(ShannonGain(Price,500)),500));
	/*vars FilterShannon = series(ShannonGain(Price,500));
	if( valley(Trend) ){
		exitShort();  // close opposite position
		if(crossOver(FilterShannon,0))
			enterLong();
	} else if( peak(Trend) ) {
		exitLong();
		if(crossUnder(FilterShannon,0))
			enterShort();
	}*/
	
	
	/*BBands(Price,30,2,2,MAType_MAMA);
	plot("price",Price[0],MAIN|LINE,BLACK);
	plot("Bollinger1",rRealUpperBand,BAND1,0x000000CC);
  	plot("Bollinger2",rRealLowerBand,BAND2,0x800000FF);
  	plot("Trend",Trend[0],LINE,RED);
  	plot("Filter",Filter[0],NEW|LINE,BLACK);
	plot("Hurst",HurstExponent*100,NEW|LINE,BLACK);
	plot("Shannon",FilterShannon[0],NEW|LINE,BLACK);*/
	
	/*plot("price",Price[0],MAIN|LINE,BLACK);
	plot("lowpass",Trend,LINE,RED);
	plot("kama",kavg,LINE,BLUE);*/
	
	//PlotWidth = 600;
	//PlotHeight1 = 300;
//	plotTradeProfile(50);
	set(LOGFILE); // log all trades
}



So, in addition to solve this issue, I would like to know in general how to adapt a LowPass to the market conditions. I understood in the tutorial that they use 1000 as cutoff to catch the 2-month or more trends, but is there a way to set this according to the market like the way it is done in workshop 5 using DominantPeriod as the cutoff of the HighPass? Actually the code for that part is:

Code:
vars Period = series(DominantPeriod(Price, 30));
var LowPeriod = LowPass(Period, 500);


and again the doubt: why using 500? In this case maybe because it's an example but in designing an -almost at least- parameter free strategy I would like to know if there's a way or some heuristic to choose the cutoff period of the LowPass.

Thanks beforehand!
Posted By: Mithrandir77

Re: How to adapt Lowpass cutoff period to market conditions? - 09/11/14 22:05

Just in case, I add that I have used fxcm historical data.
Posted By: jcl

Re: How to adapt Lowpass cutoff period to market conditions? - 09/12/14 08:31

The 500 is indeed an arbitrary value, for keeping the example simple. In a real system you would check how the 500 affect the result, and whether the system becomes more robust when the parameter is optimized.
Posted By: Mithrandir77

Re: How to adapt Lowpass cutoff period to market conditions? - 09/18/14 22:00

Originally Posted By: jcl
The 500 is indeed an arbitrary value, for keeping the example simple. In a real system you would check how the 500 affect the result, and whether the system becomes more robust when the parameter is optimized.


Thanks for your input jcl. I have tried to understand your advice and read a bit more meanwhile and tried different indicators that try to detect when the market is not in 'trend mode' but without success.

I don't want to be spoon-fed but could you elaborate a bit more how to 'check how the 500 affect the result'? , the only thing I can think of is a signficance statistical test but even in that case I am lost in how to go about performing it or which one should I use.
Posted By: jcl

Re: How to adapt Lowpass cutoff period to market conditions? - 09/19/14 07:58

Use the optimize function for this. You'll then get a performance chart as in Workshop 5, which shows you how the lowpass period affects the result. If it's more or less flat, leave it at 500. If it has a clear peak, it's a parameter to be optimized.
Posted By: Mithrandir77

Re: How to adapt Lowpass cutoff period to market conditions? - 09/23/14 14:20

Originally Posted By: jcl
Use the optimize function for this. You'll then get a performance chart as in Workshop 5, which shows you how the lowpass period affects the result. If it's more or less flat, leave it at 500. If it has a clear peak, it's a parameter to be optimized.


Ok, I have used this code for optimization:

Click to reveal..

Code:
var MMI(vars Data,int TimePeriod)
{
	var m = Median(Data,TimePeriod);
	int i, nh=0, nl=0;
	for(i=1; i<TimePeriod; i++) {
		if(Data[i] > m && Data[i] > Data[i-1])
			nl++;
		else if(Data[i] < m && Data[i] < Data[i-1])
			nh++;
	}
	return 100.*(nl+nh)/(TimePeriod-1);
}

function run()
{
	set(PARAMETERS);
	StartDate = 2002;
	EndDate = 2008;	
	//NumWFOCycles = 10;
	LookBack = 3000;
	vars Price = series(price());
	var period = optimize(500,50,3000,50);
	var mmiperiod = optimize(200,50,1200);
	vars Trend = series(LowPass(Price,period));	
	
	Stop = ATR(100)*optimize(4, 2, 8);
	
	vars Meanness = series(MMI(Price,period));
	vars Filter = series(LowPass(Meanness,mmiperiod));	
	
	if( valley(Trend) ){		
		exitShort();  // close opposite position
		if(falling(Filter))
			enterLong();
	} else if( peak(Trend) ) {		
		exitLong();
		if(falling(Filter))
			enterShort();
	}
}



And here are the parameters charts:

Lowpass period (parameter 1):
Click to reveal..



MMI lowpass period (parameter 2):
Click to reveal..



Stop factor (parameter 3):
Click to reveal..




So they are more or less flat except for the lowpass period, is it because it is the first parameter to be optimized and the others were optimized after the first was optimized?

Then I used WFO:
(I simply added NumWFOCycles = 10; after LookBack = 3000; )

and this is the result of the strategy:

Click to reveal..

Walk-Forward Test Workshop4_2Modfied EUR/USD - performance report

Simulation period 01.07.2002-31.12.2008
Test period 06.01.2005-31.12.2008
WFO test cycles 9 x 2674 bars (23 weeks)
Training cycles 10 x 15152 bars (130 weeks)
Monte Carlo cycles 200
Lookback time 3000 bars (25 weeks)
Assumed slippage 10.0 sec
Spread 2.5 pips (roll 0.01/-0.02)
Contracts per lot 1000.0

Gross win/loss 663$ / -428$ (+3282p)
Average profit 59$/year, 4.93$/month, 0.23$/day
Max drawdown -94$ 40% (MAE -102$ 43%)
Total down time 84% (TAE 43%)
Max down time 159 weeks from Mar 2005
Largest margin 5.00$
Trade volume 254213$ (63784$/year)
Transaction costs -48$ spr, -0.27$ slp, -0.27$ rol
Capital required 87$

Number of trades 265 (67/year, 2/week, 1/day)
Percent winning 21%
Max win/loss 100$ / -9.74$
Avg trade profit 0.89$ 12.4p (+167.9p / -28.4p)
Avg trade slippage -0.00$ -0.0p (+0.6p / -0.2p)
Avg trade bars 42 (+138 / -17)
Max trade bars 902 (7 weeks)
Time in market 47%
Max open trades 1
Max loss streak 32 (uncorrelated 26)

Annual return 68%
Profit factor 1.55 (PRR 1.26)
Sharpe ratio 0.85
Kelly criterion 1.05
R2 coefficient 0.000
Ulcer index 21.8%
Prediction error 55%

Confidence level AR DDMax Capital

10% 85% 74$ 69$
20% 76% 84$ 78$
30% 69% 93$ 86$
40% 64% 100$ 92$
50% 61% 107$ 97$
60% 55% 119$ 108$
70% 49% 133$ 120$
80% 43% 152$ 137$
90% 37% 179$ 160$
95% 33% 204$ 182$
100% 24% 273$ 242$

Portfolio analysis OptF ProF Win/Loss Wgt% Cycles

EUR/USD .062 1.55 55/210 100.0 /\\XXXXX/
EUR/USD:L .093 1.69 26/108 60.5 /\\//\///
EUR/USD:S .039 1.42 29/102 39.5 /\\\\/\\/


And the equity curve. What is the meaning of the blue equity curve below zero?



Anyway, It improved but I have some concerns:

-I used a big optimize range which according to the manual can lead to overfitting but how can I narrow the range without knowing a priori where the optimal value? -This is precisely where all the problem started-

-As I asked before, I don't like that blue and red equity curve below 0. What is the meaning of that? Should I worry?

Thanks!
Posted By: jcl

Re: How to adapt Lowpass cutoff period to market conditions? - 09/23/14 17:45

Last question first - you can see that this system is not profitable when volatility is low, as in the test period except for the last part.

I would reduce the optimization range of the lowpass period, and not optimize the other 2 parameters.
Posted By: Mithrandir77

Re: How to adapt Lowpass cutoff period to market conditions? - 09/24/14 03:45

Originally Posted By: jcl
Last question first - you can see that this system is not profitable when volatility is low, as in the test period except for the last part.

I would reduce the optimization range of the lowpass period, and not optimize the other 2 parameters.


Thank you jcl, I guess that the blue equity below 0 is a margin call isn't it? And the red below 0 are actually 'draw ups'

Anyway I trained as you said, only optimizing lowpass period, with this code:
Click to reveal..

Code:
var MMI(vars Data,int TimePeriod)
{
	var m = Median(Data,TimePeriod);
	int i, nh=0, nl=0;
	for(i=1; i<TimePeriod; i++) {
		if(Data[i] > m && Data[i] > Data[i-1])
			nl++;
		else if(Data[i] < m && Data[i] < Data[i-1])
			nh++;
	}
	return 100.*(nl+nh)/(TimePeriod-1);
}

function run()
{
	set(PARAMETERS);
	StartDate = 2002;
	EndDate = 2008;	
	NumWFOCycles = 10;
	LookBack = 3000;
	vars Price = series(price());
	var period = optimize(1200,1000,2000,50);//optimize(500,50,3000,50);
	var mmiperiod = 100;//optimize(200,50,1200);
	vars Trend = series(LowPass(Price,period));	
	
	Stop = ATR(100)*4;//optimize(4, 2, 8);
	
	vars Meanness = series(MMI(Price,period));
	vars Filter = series(LowPass(Meanness,mmiperiod));	
	
	if( valley(Trend) ){		
		exitShort();  // close opposite position
		if(falling(Filter))
			enterLong();
	} else if( peak(Trend) ) {		
		exitLong();
		if(falling(Filter))
			enterShort();
	}
}



Which gave this result:
Click to reveal..

Walk-Forward Test Workshop4_2Pb EUR/USD - performance report

Simulation period 01.07.2002-31.12.2008
Test period 06.01.2005-31.12.2008
WFO test cycles 9 x 2674 bars (23 weeks)
Training cycles 10 x 15152 bars (130 weeks)
Monte Carlo cycles 200
Lookback time 3000 bars (25 weeks)
Assumed slippage 10.0 sec
Spread 2.5 pips (roll 0.01/-0.02)
Contracts per lot 1000.0

Gross win/loss 220$ / -177$ (+588p)
Average profit 11$/year, 0.88$/month, 0.04$/day
Max drawdown -109$ 258% (MAE -133$ 314%)
Total down time 80% (TAE 31%)
Max down time 130 weeks from Jun 2006
Largest margin 5.00$
Trade volume 88337$ (22165$/year)
Transaction costs -17$ spr, 0.36$ slp, -0.16$ rol
Capital required 100$

Number of trades 92 (24/year, 1/week, 1/day)
Percent winning 16%
Max win/loss 70$ / -7.81$
Avg trade profit 0.46$ 6.4p (+203.8p / -32.1p)
Avg trade slippage 0.00$ 0.1p (+0.9p / -0.1p)
Avg trade bars 86 (+371 / -30)
Max trade bars 1316 (11 weeks)
Time in market 33%
Max open trades 1
Max loss streak 20 (uncorrelated 29)

Annual return 11%
Profit factor 1.24 (PRR 0.82)
Sharpe ratio 0.21
Kelly criterion 0.40
R2 coefficient 0.000
Ulcer index 69.5%
Prediction error 86%

Confidence level AR DDMax Capital

10% 26% 41$ 40$
20% 24% 45$ 44$
30% 22% 49$ 48$
40% 19% 57$ 55$
50% 18% 62$ 58$
60% 16% 69$ 65$
70% 15% 76$ 71$
80% 13% 92$ 85$
90% 11% 107$ 98$
95% 10% 121$ 110$
100% 8% 151$ 136$

Portfolio analysis OptF ProF Win/Loss Wgt% Cycles

EUR/USD .023 1.24 15/77 100.0 \X\X\\\\X
EUR/USD:L .047 1.49 9/39 107.2 \\\/\\\\/
EUR/USD:S .000 0.96 6/38 -7.2 \/\\\\\\\


And this equity I don't understand, now the unprofitable period became profitable and viceversa, how can be that? frown

Posted By: Spirit

Re: How to adapt Lowpass cutoff period to market conditions? - 09/27/14 17:47

Equity below 0 means your strategy makes no profit but a loss, this is not a margin call. Margin call happens when you have not enough capital and can not cover the margin of open trades.

Your last strategy looks as if it makes very few trades, maybe you should adjust your parameters for more trades?
Posted By: Mithrandir77

Re: How to adapt Lowpass cutoff period to market conditions? - 10/06/14 01:43

Spirit, thanks for your response. I have been busy with studies and work the last weeks. I have decided to reaproach this script by adding a filter that measures volatility. I could have used ATR but for some reason I found Alligator interesting so here it is the script and the results:

script:
Click to reveal..

function run()
{
//set(PARAMETERS);
StartDate = 2002;
EndDate = 2008;
NumWFOCycles = 10;
LookBack = 2000;
vars Price = series(price());
var period = 500;
vars Trend = series(LowPass(Price,period));

Stop = ATR(100)*4;//optimize(4, 2, 8);

vars Meanness = series(MMI(Price,200));
vars Filter = series(LowPass(Meanness,period));

Alligator(series(MedPrice()));
var diff = FisherN(series(abs(rBlue-rGreen) + abs(rGreen-rRed)),500);
vars diffs = series(diff);
var Threshold = 2.5;//0.1;//1.0;//1.5;//2.5

static bool Filter2;
if (crossOver(diffs,Threshold) or crossUnder(diffs,-Threshold))
Filter2 = true;
if (crossUnder(diffs,Threshold) or crossOver(diffs,-Threshold))
Filter2 = false;

if( valley(Trend) ){
exitShort(); // close opposite position
if(falling(Filter) and Filter2)
enterLong();
} else if( peak(Trend) ) {
exitLong();
if(falling(Filter) and Filter2)
enterShort();
}
set(LOGFILE); // log all trades
}


performance report:
Click to reveal..

Walk-Forward Test Workshop4_2Pc EUR/USD - performance report

Simulation period 01.05.2002-31.12.2008
Test period 30.11.2004-31.12.2008
WFO test cycles 9 x 2742 bars (23 weeks)
Training cycles 10 x 15538 bars (134 weeks)
Monte Carlo cycles 200
Lookback time 2000 bars (17 weeks)
Assumed slippage 10.0 sec
Spread 2.3 pips (roll -0.10/0.04)
Contracts per lot 1000.0

Gross win/loss 117$ / -44$ (+941p)
Average profit 18$/year, 1.48$/month, 0.07$/day
Max drawdown -34$ 47% (MAE -42$ 58%)
Total down time 41% (TAE 9%)
Max down time 68 weeks from Jul 2007
Largest margin 5.00$
Trade volume 17479$ (4278$/year)
Transaction costs -3.02$ spr, 0.10$ slp, -0.88$ rol
Capital required 34$

Number of trades 17 (5/year, 1/week, 1/day)
Percent winning 41%
Max win/loss 42$ / -16$
Avg trade profit 4.28$ 55.4p (+215.8p / -56.9p)
Avg trade slippage 0.01$ 0.1p (+0.6p / -0.3p)
Avg trade bars 151 (+346 / -14)
Max trade bars 542 (4 weeks)
Time in market 10%
Max open trades 1
Max loss streak 6 (uncorrelated 6)

Annual return 52%
Profit factor 2.65 (PRR 1.25)
Sharpe ratio 0.75
Kelly criterion 1.08
R2 coefficient 0.385
Ulcer index 6.0%
Prediction error 125%

Confidence level AR DDMax Capital

10% 103% 14$ 17$
20% 93% 17$ 19$
30% 85% 18$ 21$
40% 82% 20$ 22$
50% 77% 21$ 23$
60% 73% 22$ 24$
70% 67% 25$ 27$
80% 62% 28$ 29$
90% 55% 32$ 33$
95% 50% 36$ 36$
100% 33% 58$ 54$

Portfolio analysis OptF ProF Win/Loss Wgt% Cycles

EUR/USD .166 2.65 7/10 100.0 ./XX//\\\
EUR/USD:L .169 3.14 5/6 88.4 ..\///\\\
EUR/USD:S .147 1.60 2/4 11.6 .//\...\.

plot:
Click to reveal..




As you can see, I used Alligator lines closeness as a measure of volatility, I normalize their differences and when they go out of a threshold (which is higher the less volatile the market is) I consider it a signal to buy or sell.

This has turned the loser system into a winning one with good Sharpe Ratio for a trend system and an ulcer of 6%. What do you think? Do you think It has any bias or curve fitting? I look forward to hearing your inputs. Thanks!
Posted By: royal

Re: How to adapt Lowpass cutoff period to market conditions? - 10/06/14 08:33

5 trades/year doesn't seen to be very statistically significant for me.
Posted By: Thirstywolf

Re: How to adapt Lowpass cutoff period to market conditions? - 10/07/14 04:14

Hi Mithrandir

I would be a little bit concerned about the low number of trades. Is it enough data?
Posted By: Mithrandir77

Re: How to adapt Lowpass cutoff period to market conditions? - 10/10/14 15:02

Originally Posted By: royal
5 trades/year doesn't seen to be very statistically significant for me.


Originally Posted By: Thirstywolf
Hi Mithrandir

I would be a little bit concerned about the low number of trades. Is it enough data?


Thanks for your inputs mates. I used oversampling, with NumSampleCycles=4 and 6 but the number of trades per year doesn't increase. I attached the reports. How can I obtain a statistically significant result? Or is it filtering the bad trades for a specific period a tradeoff with getting enough trades for the result to be statistically significant?

I think I am more or less curve fitting but it is a 'good' curve fitting because I am adapting to the volatility of a specific period of time (in particular one which has low volatility thus not good for trend strategies) . Even though as an engineering student I am afraid of a not statistically significant result as you frown

Attached File
Attached File
Posted By: Thirstywolf

Re: How to adapt Lowpass cutoff period to market conditions? - 10/10/14 16:50

Hey Mith

I'm an absolute newbie on the stats/testing side of things, however I would probably want to see how the revisions impact the previously profitable 2009 onwards period. Whatever you do, can't reduce profitability of this more recent period. I myself would be tempted to accept that the markets have changed, and a previously unprofitable strategy is now profitable, rather than attempting to make this strategy profitable for that period. If you take this path, then all you need is a way to identify conditions have changed enough for the strategy not to work, and stop the system, or alternatively see if you can train the system out of taking any or many trades during that period, so long as profitable periods stay profitable.

I am going to finish here by again stating that I know nothing, and I am sure that there are more knowledgeable people to comment on this.
© 2024 lite-C Forums