Forums Calendar Active Threads Search Forum Help
Gamestudio Links
► Features
► Download
► Online Manual
► Online Tutorial
► Free Resources
► F.A.Q.
Zorro Links
► Download
► Online Manual
► Online Tutorial
► Blog
► Support
► What's New?
Newest Posts
Data from CSV not parsed correctly
by EternallyCurious. 04/18/24 10:45
StartWeek not working as it should
by Zheka. 04/18/24 10:11
folder management functions
by VoroneTZ. 04/17/24 06:52
lookback setting performance issue
by 7th_zorro. 04/16/24 03:08
zorro 64bit command line support
by 7th_zorro. 04/15/24 09:36
Zorro FIX plugin - Experimental
by flink. 04/14/24 07:48
Zorro FIX plugin - Experimental
by flink. 04/14/24 07:46
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
1 registered members (AndrewAMD), 552 guests, and 1 spider.
Key: Admin, Global Mod, Mod
Newest Members
EternallyCurious, 11honza11, ccorrea, sakolin, rajesh7827
19046 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
Really really annoying problem with histories #464377
02/08/17 15:45 02/08/17 15:45
Joined: Apr 2013
Posts: 35
J
Joaquin Offline OP
Newbie
Joaquin  Offline OP
Newbie
J

Joined: Apr 2013
Posts: 35
I keep strugling with my Pairs Trading Strategy.

This time I a suffering a problem with what I think it is a problem with histories.

I download my histories from an Oanda Demo account, which runs much better and faster than MT4.

This is a snippet of the code I am running:

Code:
function run()
{
	BarPeriod =1;
	StartDate = 20160615; 
	EndDate = 20170201; 
	LookBack = 1000;
	UnstablePeriod = 50;
	vars pairRatio, priceClose1, priceClose2;
	//---------1 ----------//
	asset(asset1);
	Spread =	Slippage = RollShort = RollLong = 0; 
	priceClose1 = series(log(priceClose()) / log(10));
	//---------2 ----------//
	asset(asset2);
	Spread =	Slippage = RollShort = RollLong = 0; 
	priceClose2 = series(1/log(priceClose()) / log(10));
	pairRatio = series((priceClose1[0]) * (priceClose2[0]));
	
	if(is(FIRSTINITRUN)) { 
		while(loop(asset1,asset2)) assetHistory(Loop1,1);
	}
	
	int halfLife = LookBack;
	vars zScore = series((pairRatio[0] - SMA(pairRatio,halfLife)) / StdDev(pairRatio, halfLife));
	if (Bar > LookBack) printf("nzScore: %f",zScore[0]);

With this code snippet, it runs fine in "Trade" mode for LookBack smaller than 1000 (or so, I don't know exactly where is the frontier). If I try with LookBack = 2000 and bigger, then I get

Code:
zScore: -1.#IND00
zScore: -1.#IND00
zScore: -1.#IND00
zScore: -1.#IND00
zScore: -1.#IND00
zScore: -1.#IND00
zScore: -1.#IND00
zScore: -1.#IND00

[2093: Wed 08.02.17 14:55]  0.99387
zScore: -1.#IND00
Logout.. ok


I guess there must be a problem with histories: either I don't have enough histories, or there's a gap and at some point one of the asset's price is zero, or something like that.

I would like to know what could I do to
1) First, to be sure that this issue is due to a histories problem
2) Second, if it is actually a history problem, to solve it somehow... like using latest known price if there's a gap, or another intermediate solution.

If I run this same script in Trading mode with Oanda, it runs wonderfully.

I think that what is happening is that MT4 history sucks horribly, and when I'm in trading mode with MT4 accounts, it loads MT4 history, it does not get correct histories, and then it gets this error...

How do I solve this nightmare?
Re: Really really annoying problem with histories [Re: Joaquin] #464378
02/08/17 16:03 02/08/17 16:03
Joined: Apr 2013
Posts: 35
J
Joaquin Offline OP
Newbie
Joaquin  Offline OP
Newbie
J

Joined: Apr 2013
Posts: 35
I'm just looking for a better way to program my strategies, at least better than mql4. I thought that Zorro would be better, but I'm running through lots of problems... I will still not go for the Sponsor version, as I don't see Zorro Trader able to run properly a real account...

Maybe you'll say that these problems are not related to Zorro, but to MT4 and its bad histories.
I guess that after a few weeks fighting against a lot of issues I'm a bit fed up... I've seen Zorro 'freeze' my MT4 demo account (and that's not acceptable in real trading), I've seen unstable behaviour in exactly same conditions but different "runs", I've seen strange memory problems although the computer has 12 GB RAM and more than 8 GB free...

This makes me think that I'd better stick to what I know that works... mql4.

Sorry for the critics jcl, but I've been trading in demo accounts for weeks with Zorro and almost every day I find another different bad surprise. Maybe it's the normal way of getting to know any new platform / technology, but I want to do things more efficiently with Zorro, not to lose more time trying to guess why it is (mis)behaving the way it does.
Re: Really really annoying problem with histories [Re: Joaquin] #464385
02/09/17 09:24 02/09/17 09:24
Joined: Apr 2013
Posts: 35
J
Joaquin Offline OP
Newbie
Joaquin  Offline OP
Newbie
J

Joined: Apr 2013
Posts: 35
I don't like to leave my researching projects abandoned, so I just keep looking for an answer of what is happening with the histories.
It's very difficult to explain in just a few lines the strange things that I have seen. But I will try.

As I keep growing LookBack, probabilities of getting an error are higher and higher.

I've noticed that MT4 passes histories to Zorro in batches of 300 candles each time.
This is the log I get in MT4 with LookBack = 7000 and BarPeriod = 1, loading 2 assets:
Code:
2017.02.09 10:01:01.481	Zorro AUDUSD,M1: Zorro Disconnected!
2017.02.09 10:01:00.172	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.02 15:53 (6886 - 7186)
2017.02.09 10:00:59.828	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.02 20:53 (6585 - 6885)
2017.02.09 10:00:59.500	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.03 01:54 (6285 - 6585)
2017.02.09 10:00:59.188	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.03 06:53 (5986 - 6286)
2017.02.09 10:00:58.860	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.03 11:53 (5686 - 5986)
2017.02.09 10:00:58.563	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.03 16:55 (5387 - 5687)
2017.02.09 10:00:58.282	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.03 21:54 (5088 - 5388)
2017.02.09 10:00:58.000	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.06 03:04 (4788 - 5088)
2017.02.09 10:00:57.688	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.06 08:03 (4489 - 4789)
2017.02.09 10:00:57.406	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.06 13:02 (4190 - 4490)
2017.02.09 10:00:57.125	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.06 18:01 (3891 - 4191)
2017.02.09 10:00:56.844	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.06 23:01 (3591 - 3891)
2017.02.09 10:00:56.578	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.07 04:02 (3292 - 3592)
2017.02.09 10:00:56.281	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.07 09:01 (2993 - 3293)
2017.02.09 10:00:55.985	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.07 14:01 (2693 - 2993)
2017.02.09 10:00:55.672	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.07 19:00 (2394 - 2694)
2017.02.09 10:00:55.375	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.08 00:01 (2094 - 2394)
2017.02.09 10:00:55.110	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.08 05:04 (1795 - 2095)
2017.02.09 10:00:54.766	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.08 10:03 (1496 - 1796)
2017.02.09 10:00:54.485	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.08 15:02 (1197 - 1497)
2017.02.09 10:00:54.141	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.08 20:02 (897 - 1197)
2017.02.09 10:00:53.797	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.09 01:02 (598 - 898)
2017.02.09 10:00:53.438	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.09 06:01 (299 - 599)
2017.02.09 10:00:53.063	Zorro AUDUSD,M1: Send 300 USDCHF bars at 2017.02.09 11:00 (0 - 300)
2017.02.09 10:00:52.563	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.02 16:05 (6881 - 7181)
2017.02.09 10:00:52.203	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.02 21:04 (6582 - 6882)
2017.02.09 10:00:51.875	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.03 02:03 (6283 - 6583)
2017.02.09 10:00:51.532	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.03 07:02 (5984 - 6284)
2017.02.09 10:00:51.203	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.03 12:01 (5685 - 5985)
2017.02.09 10:00:50.891	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.03 17:04 (5385 - 5685)
2017.02.09 10:00:50.594	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.03 22:04 (5085 - 5385)
2017.02.09 10:00:50.281	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.06 03:11 (4786 - 5086)
2017.02.09 10:00:50.000	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.06 08:10 (4487 - 4787)
2017.02.09 10:00:49.719	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.06 13:09 (4188 - 4488)
2017.02.09 10:00:49.422	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.06 18:08 (3889 - 4189)
2017.02.09 10:00:49.125	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.06 23:08 (3589 - 3889)
2017.02.09 10:00:48.844	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.07 04:08 (3290 - 3590)
2017.02.09 10:00:48.547	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.07 09:07 (2991 - 3291)
2017.02.09 10:00:48.250	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.07 14:06 (2692 - 2992)
2017.02.09 10:00:47.969	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.07 19:05 (2393 - 2693)
2017.02.09 10:00:47.685	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.08 00:05 (2094 - 2394)
2017.02.09 10:00:47.364	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.08 05:04 (1795 - 2095)
2017.02.09 10:00:46.921	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.08 10:03 (1496 - 1796)
2017.02.09 10:00:46.530	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.08 15:02 (1197 - 1497)
2017.02.09 10:00:46.189	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.08 20:02 (897 - 1197)
2017.02.09 10:00:45.824	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.09 01:02 (598 - 898)
2017.02.09 10:00:45.449	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.09 06:01 (299 - 599)
2017.02.09 10:00:45.188	Zorro AUDUSD,M1: Send 300 EURUSD bars at 2017.02.09 11:00 (0 - 300)
2017.02.09 10:00:45.078	Zorro AUDUSD,M1: Zorro Connected!

The histories are there. I would not tell that the problem is in bad MT4 histories. The problem, I think, is in the DLL, in the "bridge" between MT4 and Zorro. I think that this bridge tries in some way to synchronize both timeseries, and because there are not exactly the same number of candles for the same period of time, the script disconnects in the MT4 end, and in the Zorro end it just says that it lacks X candles.

This is a problem.

Another one is that between different runs of the same script, sometimes it works, sometimes it doesn't, sometimes it says that it lacks X candles and other times it says it lacks Y candles.

I think that when LookBack is long enough and you are loading several assets, there should be a better check of histories correctly synchronized. This test could be done in the MT4 end (the DLL) or in the Zorro part. Personally, I would do it in the DLL... The sooner the problem is solved, the better.
Re: Really really annoying problem with histories [Re: Joaquin] #464386
02/09/17 10:51 02/09/17 10:51
Joined: Apr 2013
Posts: 35
J
Joaquin Offline OP
Newbie
Joaquin  Offline OP
Newbie
J

Joined: Apr 2013
Posts: 35
Bu the way, it is not a matter of 2 different assets... If I have only one asset to load, but a long LookBack, I get the same weird and random errors.

I think there's a bug either in the Zorro DLL or expert advisor, or in the Zorro Trader end.

I will try similar tests in a different computer this afternoon, in order to know if there's some hardware dependant issue.
Re: Really really annoying problem with histories [Re: Joaquin] #464391
02/09/17 17:08 02/09/17 17:08
Joined: Apr 2013
Posts: 35
J
Joaquin Offline OP
Newbie
Joaquin  Offline OP
Newbie
J

Joined: Apr 2013
Posts: 35
Well, I have solved something... I just noticed that the error I was getting with my zScore

Code:
zScore: -1.#IND00
was because some candles before this error I was getting something like #INF00 error, and I assumed that there was a division by zero somewhere. And yes, in fact there's a division of the two price series.

It turns out that priceClose() sometimes returns zero. Don't ask me why, because I think it shouldn't. So I've just coded an indicator to eliminate these zeroes:

Code:
var myFilter(var* Data, double number) 
{
	int __i = 0;
	while (Data[__i] == number) __i++;
	if (Data[__i] != number) return(Data[__i]);
}


Passing the price series as first parameter and 0.0 as second parameter I have solved this error of zero division.

Also, I had a lot of outliers in the zScore, maybe due to incredibly huge gaps in history or just due to flash crashes. To eliminate these outliers, I've written another function:

Code:
var removeOutliers(var* Data, double number) 
{
	int __i = 0;
	while (abs(Data[__i]) >= abs(number)) __i++;
	if (abs(Data[__i]) <= abs(number)) return(Data[__i]);
}


being "number" the outlier level you want to chop off. Previously I had seen in a chart what I had in Data, in order to see intuitively where I should set the threshold for the outliers.

I'm happier now, after having found a solution to some of my problems.

I will keep on my research with Zorro Trader, I think it is one the best options I have for doing multiasset backtests...
Re: Really really annoying problem with histories [Re: Joaquin] #464396
02/10/17 12:39 02/10/17 12:39
Joined: Jul 2000
Posts: 27,978
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,978
Frankfurt
Price functions return zero when either asset() was not yet called or the price history contains no data at or before that bar. In all other cases they should not be zero. Or else it would be a bug.

Moderated by  Petra 
Forum Rules · Mark All Read ·

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de
Powered by UBB.threads™ PHP Forum Software 7.7.1