Additional testing to be undertaken...

Posted By: 1ND1G0

Additional testing to be undertaken... - 03/01/18 17:30

I'm currently running an algo on a demo account as part of a final check prior to going live. However, I also wanted to make sure there were no other obvious tests I could undertake directly in Zorro before funding this with actual capital.

The strategy is based on 60 minute bars, so I have been using M1 data rather than Tick. I have tried to only optimise relevant parameters.

I've tested with both fixed lot sizes and also margin amounts. While developing, I have set NumWFOCycles to 10 and NumSampleCycles to 10. MonteCarlo to 1000.

I've trained and tested the algo using both SKIP1/2/3 settings and then DataSplit / DataSlope.

I've used various Detrend settings including CURVE, INVERT, and SHUFFLE.

I've used various start and end dates for training / testing as well as the default 6 year view.

I've updated the AssetsFix file with realistic spread, commission and roll amounts, using worst case numbers provided by my broker.

I've tried it with and without phantom trades but found it best without any equity curve trading.

Any additional points to consider?




Posted By: AndrewAMD

Re: Additional testing to be undertaken... - 03/01/18 17:57

Some boilerplate questions:
* Is this algo going to get used on Forex/Cryptos?
* If so, is your historical data source straight from the broker you intend to use?
* Is this a portfolio setup or a single-asset setup?
* How are your positions sized?
Posted By: 1ND1G0

Re: Additional testing to be undertaken... - 03/01/18 18:12

Hi Andrew,

It's forex. Single asset for now.

I haven't been able to get historical data from the broker, but tested on both the Zorro provided data initially, and then also created new t6 files from histdata for the relevant pair.

Positions are sized based on the account Margin and OptimalF factor. It also reinvests profits.
Posted By: MatPed

Re: Additional testing to be undertaken... - 03/02/18 10:12

How many trades for single optimized parameter does the algo place?
The rule of thumb is 20 trade for each parameter in single wfocycle
Establish in advance a set a parameters or indicator that will determine the "Stop the TS condition" and be ready to stick with it.
i.e PF of the last 30 trades < 85% of PF of the TS in the backtest and number of consecutive loss > maximum of consecutive loss of the TS.
Reinvesting will complicate this topic a bit
Posted By: AndrewAMD

Re: Additional testing to be undertaken... - 03/02/18 12:38

Add this to the end of your script and show us the plot:

Code:
plotMAEPercentGraph(50);
Posted By: Hredot

Re: Additional testing to be undertaken... - 03/02/18 14:03

Weird, I get an error "undeclared identifier" about plotMAEPercentGraph even though it is listed in the manual under the "Help" button...
Posted By: AndrewAMD

Re: Additional testing to be undertaken... - 03/02/18 14:25

Originally Posted By: Hredot
Weird, I get an error "undeclared identifier" about plotMAEPercentGraph even though it is listed in the manual under the "Help" button...
Oh, right, add this to the top:
Code:
#include<profile.c>
Posted By: 1ND1G0

Re: Additional testing to be undertaken... - 03/02/18 17:50

Hi Andrew,

I've attached an image (I could not see an easy way to insert directly into the post).

I've also removed the performance / risk metrics from my original post as they are not actually relevant to the discussion. My main aim was to understand how best to improve my test approach during algo development.



Description: plotMAEPercentGraph(50); results
Attached picture 2018-03-02 17_43_31-Chart Viewer - [Algo_XXXX.png - Image 1 of 1 [100%]].png
Posted By: AndrewAMD

Re: Additional testing to be undertaken... - 03/02/18 19:06

A quick look at the plot says a couple of things:

* You let your winners run. This is good.
* Once you have a trade that draws down more than 1.1% of your position, you never have a winning trade. You can add one more filter to improve risk management. May I offer a suggestion? Close your trades when you have 1.0% nominal position drawdown, and optimize it for 0.5% to 1.5% position drawdown.
* After you make the above adjustment, re-plot.

As it says in the book Trading Systems by Tomasini and Jaekle, your losses are within your realm of control and not your profits. Yet, it is by controlling your losses that you will create more opportunities for profit.
Posted By: Hredot

Re: Additional testing to be undertaken... - 03/03/18 07:15

I don't understand, what is the x-axis and the y-axis? How should one read this plot?

Posted By: MatPed

Re: Additional testing to be undertaken... - 03/03/18 16:10

In case of use of oversampling the graph will plot the distribution of all trades or only of one sample?
Posted By: AndrewAMD

Re: Additional testing to be undertaken... - 03/03/18 16:20

MAE stands for Maximum Adverse Excursion. It's a nice tool to visually determine if and where to place a stop.

Every trade made by your system is represented by a dot. If the trade was profitable, it is green and above zero. If it was unprofitable, it is red and below zero. So the higher the dot, the better.

The X axis is maximum excursion of a trade, in percentage of the trade's position.

The Y axis is profit/loss of a trade, in percentage of the trade's position.

Say you buy a stock at $100.00, it reaches a low of $99.00, and it sells at $120.00. Your trade will plot at (1.0,20.0). That is, 1% drawdown, 20% profit.

To see visually the effect of adding a percentage-based stop loss, draw a vertical line. Any greens to the right of the line become losses, and any reds become less unprofitable. Thus, it appears to be useful for 1ND1G0 and not for Hredot.

But we have Zorro, so we can easily optimize after using our visual cue.

Hredot: I like your plot, you have some very profitable trades. What I don't understand is why your system isn't profitable when it sees very small drawdowns (<=0.6% per trade). In any case, I don't think you need to add stop losses because it can hurt your profitability.
Posted By: AndrewAMD

Re: Additional testing to be undertaken... - 03/03/18 16:23

Originally Posted By: MatPed
In case of use of oversampling the graph will plot the distribution of all trades or only of one sample?
All trades. The code is in your profile.c header, so you can tweak it for your own purposes if necessary.
Code:
void plotMAEPercentGraph(int bars)
{
	if(!is(TESTMODE)) return; 
	g->dwStatus |= PLOTSTATS;
	if(is(EXITRUN))
	{
		if(!bars) bars = 50;
	
		var vMaxMAE = 0;
		for(all_trades) // calculate maximum MAE relative to open
			vMaxMAE = max(vMaxMAE,(100.*TradeMAE/(TradePriceOpen)));
		//printf("nMaxMAE: %.0f",vMaxMAE);
	
		var vStep;
		if(bars < 0) // bucket size in thousandths of percentage points (i.e. 0.001%)
			vStep = (bars * (-1.) / (1000.));
		else
			vStep = max(0.05, roundto((vMaxMAE/bars),0.001));
		printf("  Step: %.3f",vStep);
		
		for(all_trades) 
		{
			var vResult = (100*TradeProfit/(TradeUnits*TradePriceOpen));
			var vMAE = (100.*TradeMAE/(TradePriceOpen))/vStep;
			int n = floor(vMAE);
			plotBar("Profit",n,n*vStep,0,AVG|BARS|LBL2,COLOR_DEV);
			if(vResult > 0)
				plotGraph("Win",vMAE,vResult,DOT,GREEN);
			else
				plotGraph("Loss",vMAE,vResult,DOT,RED);
		}
	}
}
Posted By: Hredot

Re: Additional testing to be undertaken... - 03/03/18 16:52

Thanks for the explanation!

Btw, that plot was not from a real system. Rather an ideal one that strives to achieve optimal performance by using PEEK. It is actually surprising how hard it is to code an ideal system using only traditional indicators even when it knows the future... The idea is, if an approach does poorly when future is known, there is no chance it will succeed with only present information. A way to discard garbage ideas.
Posted By: 1ND1G0

Re: Additional testing to be undertaken... - 03/03/18 17:36

Many thanks for this information, Andrew. It's very much appreciated.

In terms of limiting my max drawdown so somewhere around 1%, is it just a case of adding in a line along the lines of Risk = 1; or is there more to it than that?
Posted By: AndrewAMD

Re: Additional testing to be undertaken... - 03/03/18 18:03

I don't think that's how the Risk variable works. Anyways, I would take a different approach using trade management functions.

Slightly modifying the code above, below is your percentage MAE...
Code:
//for a given trade...
var vPercentMAE = 100.*TradeMAE/(TradePriceOpen);
If you close a trade once the above variable reaches 1.0 or higher, you will have drawn a vertical line on your chart at 1.0. See how it affects your chart. And then attempt an optimization of this parameter, such as 0.5 to 1.5 in 0.1 increments, and see what you end up with on the chart.

See also...
(TMF, TradeMAE): http://zorro-project.com/manual/en/trade.htm
(optimize): http://zorro-project.com/manual/en/optimize.htm

EDIT: In retrospect, don't use TMF, use Stop.
Posted By: 1ND1G0

Re: Additional testing to be undertaken... - 03/03/18 18:21

Thanks again, Andrew.

I'm probably doing something wrong but it seems to only make things worse when I try and add this type of trade management into the mix.

I added your code to create the var, then the below:

Code:
for(open_trades)
   if(vPercentMAE > 1.0)
   exitTrade(ThisTrade);


Even optimising this parameter does not help.

In any case, this is certainly very good to know and I will certainly use this (i.e. the MAE graph) in future development.
Posted By: AndrewAMD

Re: Additional testing to be undertaken... - 03/03/18 18:34

Did you define the variable inside the for loop?
Code:
for(open_trades) 
{
   // assuming no virtual hedging / pool trading
   var vPercentMAE = 100.*TradeMAE/(TradePriceOpen);   
   if(vPercentMAE > 1.0)
      exitTrade(ThisTrade);
}
Posted By: 1ND1G0

Re: Additional testing to be undertaken... - 03/04/18 15:42

Alas, I have not been able to get this working as expected.

This is however certainly something I will work on figuring out eventually, as now that I can see the results on the graph, it is very clear this will improve my overall performance (for this particular algo).

Thanks again for your insight in this area, Andrew.
Posted By: 1ND1G0

Re: Additional testing to be undertaken... - 03/04/18 16:01

Ah, scratch that - It's not the most elegant solution, but I did the following:

- instead of generating the graph with percentages, I used plotMAEGraph(50); instead, so I could see the actual pip values
- I then set a basic stop at the 120 pip level as that represented the vertical line that Andrew alluded to
- this improved all metrics in the test period

One thing that you can see in the newly generated output, is that even with a stop at 120 pips, there is a number (albeit just a few) losing trades that have a value further right that this level. Which seems strange.

Attached picture 2018-03-04 16_00_12-Chart Viewer - [Algo_One_XXXX_s.png - Image 1 of 1 [100%]].png
Posted By: Dalla

Re: Additional testing to be undertaken... - 03/04/18 18:12

It's not that strange. Probably from slippage, or gaps if your market is not trading around the clock
Posted By: Sphin

Re: Additional testing to be undertaken... - 03/04/18 19:51

Quote:
The strategy is based on 60 minute bars, so I have been using M1 data rather than Tick. I have tried to only optimise relevant parameters.

set(TICKS) sometimes helps to get the stops closer in simulations.
Posted By: 1ND1G0

Re: Additional testing to be undertaken... - 03/04/18 21:00

Thanks, Dalla, Sphin - that helps clear that up.
Posted By: AndrewAMD

Re: Additional testing to be undertaken... - 03/04/18 23:24

Aha! I figured out what I did wrong, instead of exiting a trade via TMF, set Stop.

Here's the percentage approach:
Code:
// to place a stop 1% away from the current price
Stop = price() * 1.0 / 100.;
enterLong();
Below is a before-and-after of applying a 0.5% stop to the Luxor system.




We do have outliers, but 99% of the issue has been addressed.

Quote:
set(TICKS) sometimes helps to get the stops closer in simulations
I did attempt this, I don’t think it eliminated my outlier.

I'm happy my theory was well-received, though.

Attached picture before.png
Attached picture after.png
© 2024 lite-C Forums