Brokers incompatible with MT4 bridge?

Is it possible for the MT4 platforms of individual brokers to be incompatible with the Zorro MT4 bridge?

I am asking because I have found that when connected to a certain broker in live trading mode, Zorro detects obviously erroneous ticks. I can find no trace of these erroneous ticks in the MT4 chart, nor in the MT4 history centre. No trades have so far been executed at or close to these levels, which makes me question whether these ticks actually existed. Is it possible that the Zorro bridge could somehow register a tick that never existed?

Although no trades have been executed at these erroneous tick levels, my system uses the recent high and recent low by way of the HH() and LL() functions. These erroneous ticks have a significant impact on the values calculated by these functions (verified by printing indicator values to the log file) and several trades have been executed as a result, when they clearly shouldn't have.

This issue first came to my attention last night on EUR/USD. Attached is the image that Zorro created - although the quality is not great (I am running my VPS using 15-bit color to free up as much RAM as possible) you can see the erroneous ticks occurring between 7 and 8 PM at values of approximately 2.3 and 1.7. These ticks dwarf the scale and therefore the rest of the price curve is barely visible. Clearly EUR/USD never traded at these levels and there is no sign of these ticks in the broker's MT4 feed.



I have started running the same system with another broker and have yet to observe this behavior.

The broker that I am having trouble with is Katalon Markets. Here is the link to download their MT4 platform: https://download.mql5.com/cdn/web/7580/mt4/katalonmarket4setup.exe

I would very much appreciate any insight into what could cause this and what the possible workarounds could be. I do understand that it is possible that the broker widened the spread and created these ticks artificially, and while I won't rule that out, I consider it highly unlikely. The GM of this firm is personally known to me and I would be extremely surprised if this turned out to be the case. So for the purposes of this exercise, could we please just focus on what, if anything, related to the Zorro MT4 bridge could cause this.

Thanks in advance

This does not look like a incompatibility, it looks like an outlier. It can be prevented by software, and there is indeed an outlier filter build in Zorro, it's just not activated since we did not have such a case so far. Does it happen often with this broker, or was it only this one time?

It has happened several times today with this broker. In addition, I tested the same script on a second broker (Royal Financial) and the same thing happened. It didn't happen often - just once and I ran the live script for approximately 4 hours during the day today. It happened both on my VPS and on my desktop PC, if that makes any difference.

Ok, we'll look into that. Inbetween you can set

TickSmooth = 100;

This will suppress outliers.

Hi jcl, boatman,

This brings up some interesting questions...

From what I read of TickSmooth, a value of 100 would "suppress" by ~99%? Does TickSmooth drop the tick or cause the tick's actual value to be replaced by the EMA value? Besides changing the tick stream, the EMA-introduced delay could of course be problematic also depending on the strategy...

Assuming that Zorro isn't creating these ticks, they're coming from the broker - either directly if DD or as a pass-through from the liquidity provider if NDD? In either case, why then wouldn't they be in the broker's MT4 history? Do brokers regularly "scrub" their history? Is there something in the Terms of Service that say these outliers are the client's problem?

boatman, have you asked the brokers about this & what's going on? Since it's happened at more than 1 broker, is it some sort of "industry standard" that all are using?

Thanks.

Hi DdlV

I haven't yet approached the broker about this. Before I do that, I would like to have some more evidence that these ticks are actually coming from the broker. While I am keeping an open mind about the source of the problem, I think that it is more likely to be a bug either with the script (unlikely, since no such ticks are evident in the backtest), my own set up (although I have reproduced the issue on two different machines with different installations of Zorro - both V1.42) or with the communication between MT4 and Zorro.

Thanks for the workaround jcl, I'll give that a try.

Update: this is the result of running the same script (sans trade entries) last night on two different brokers from my desktop PC. I also reinstalled both Zorro 1.42 and MT4 bridge 1.8 prior to starting this test run.





The 'bad ticks' seem to come at random. I'm scratching my head over this one - got me stumped.

If someone wants to test if his broker has a similar problem, here's an outlier test script:



We had no outliers so far with the brokers we've tested. But we're looking into Katalon now.

Thanks. I've not yet received any strange outlying ticks from trading your test script.

Could this issue be related to my script? I sent the script to support - could you possibly look over it, or at least suggest what could cause the issue that I am having so that I can solve it myself?

Thanks

Yes, we got that script and will check it. The problem is not outliers, but probably something else in the script overwriting prices or otherwise producing the lines in the chart. The prices are stored in arrays in the ASSET struct - you can see that in the trading.h header.

This can be hard for you to solve, since it's possibly not a bug in your script, but a bug in a used indicator. What you could do in such a situation is removing indicators and other elements step by step until the problem disappears. But wait for the support, they have better tools to look into those problems.

OK. Thanks for the information. I'll look forward to support's response, but will probably tinker with this myself in the mean time...I am too impatient not to. Thanks for looking into it for me.

Hi

I had similar problem with my script and another mt4 broker. I thought it was a bug on it. But yesterday I opened the performance report of Z7 and I have similar high peaks on the chart.

We could still not reproduce the problem here, but it's not related to the broker and also not to the script. Current theory is a bug in the chart library. We'll know more next week.

I'm getting a lot of peaks on ICMarkets as well: http://i.imgur.com/V5FNY0f.png

I'll note that I'm getting it on Z12 and Z7 but not on Z3 for some reason.

I'm not convinced its an issue with the chart library. The values of indicators based on the HH() function are affected following one of these peaks. Is this consistent with what would happen if only the chart library were affected?

@Veratyr: you are getting the same errors that I am seeing here. Can you test whether the values of HH() are affected by the peaks?

I believe we have found the problem. It is no chart bug, but a bug in the mt4 bridge that was present all the time, but only surfaced recently due to the higher speed of the last version. It produces random outliers with some mt4 versions and asset combinations.

Sorry for that, and thanks to anyone who helped finding the problem. We'll do some more tests and will then upload a new version.

Fantastic news that the bug has been found and is soon to be squashed.

I'd also like to add that Zorro's support was very professional and objective in identifying this bug. I had quite a lot of email communication with them about this issue, and they took what I said seriously and kept me informed with regular and honest progress updates. Zorro gets a massive vote of confidence from me because of this.

Please test the new bridge:

http://server.conitec.net/down/ZorroMT4.zip

This is bridge version 1.9. Please install the DLL also in the Plugin folder. If no issues are found with this version in the next days, we'll include it in the release.

Does it work only for version 1.42 and above?

I switched over to the new bridge (unzipped, refreshed in MT4) and my Zorro instances all started displaying multiple prices in the Server Window while trading. It flicks rapidly between different assets and consistently has an exclamation mark next to it. Switching the tracked asset around a few times seems to stop the flickering but the exclamation mark remains.

I had a look at my server and it turns out MT4 is murdering all the CPUs with the new plugin, which may be related to the exclamation mark. I downgraded all of the MT4 instances to bridge 1.8 and CPU usage went back to normal.

I am testing the new bridge on my VPS. Running my own script with which I detected this issue originally. Seems to be running OK thus far. Will report back later.

Also, I am trying to run the test script that support supplied. It will no longer run on the same MT4 platform that it was running on with the old bridge and returns the following error after clicking 'Trade':

testScript compiling..............
Login MT4...
!Katalon Market Holding Pty Limited connected at UTC 22.03. 01:14

Loading EUR/USD prices.. no history!
No bars generated
Logout.. ok

Not sure what's going on there. Something of a mystery since this script was running just prior to updating the bridge. In addition, my own script is running on the same MT4 terminal using the new bridge at the same time. I am running Zorro S V1.42. And I confirm that I have updated the DLL in the Plugin folder.

For me too the CPU usage looks unusually high after installing the new bridge. Zorro is now consuming 50% more memory than the MT4 terminal (!), which is itself a very resource hungry application.

This implies that I wouldn't be able to run more than 1 or 2 Zorro instances on my VPS.

jcl, can this be improved?

To be clear, for me at least, the CPU usage is all on the MT4 side of the bridge. Zorro didn't seem to be unusually bad itself but each MT4 instance had a full CPU core pegged (and these aren't pushovers, they're all on a Xeon E5-2687W).

The Zorro memory consumption can not be improved by a bridge version, but by using less series in the script.

The CPU usage by MT4 with 1.9 is indeed 30% higher - with Bridge 1.9 it's 5.5% CPU load, compared to 4.1% with bridge 1.8. This can be reduced on the MT4 side by setting eventsetmillisecondtimer() to a higher value, like 100. And on the Zorro side you could reduce the load with a higher ticktime, but that would not be noticable since the Zorro CPU load is normally in the 1% area.

Katalon did not work for us anymore since Sunday, unrelated to the bridge. We assumed their server is down. But we'll check that again.

We're still having the same problems with Katalon. Sometimes it connects to their MT4 server, but after a few minutes it reports "Invalid account". This is unrelated to the bridge, it happens with just plain MT4. A new account can also not be opened. Whatever it is, it could be the reason why the historical data was unavailable.

I have been connected to Katalon via my original script all day. Nothing strange printed in the log file. But when I hit 'Result" I get the following chart, which is really strange:



It looks like it is now registering a bad tick on the downside. The chart is also printed from 17 March - the script was only started earlier today (22 March). The EUR/JPY chart at least shows some clean price history, but again it doesn't show the price history from when the script was started. I am not zooming the plots in the script.

I get the same message as before when trying to run the test script that support provided on a different broker via MT4 bridge 1.9:

testScript compiling...............
Login MT4...
!Royal Financial Trading Pty Ltd connected at UTC 22.03. 10:33

Error 053: EUR/USD unavailable!
No bars generated
Logout.. ok

I can not see a downtick in your chart, but it is hard to tell due to the low resolution. I see a yellow line from an indicator starting at zero. Can you check if there is really a downtick or not? Maybe during the weekend? The start date of your chart depends on your lookback period.

So why does the test script that worked fine on V 1.8 no longer work on V 1.9? What has changed? Its not the broker since I am connected to them via the bridge (different script) right now. Plus I have tried it on a different broker and got the same result.

I'm just checking that test script that you got. I have also an Outliertest.c here, but it is probably very different to yours.

I agree that chart isn't a great example. The resolution is too low. I can't actually see a downtick either but assumed there must be one based on the scale. Those indicators are the same as the ones I was using with the previous version and their values never started at zero (they are based on HH() and LL()).

Thanks, I'd appreciate that. I'm running my original script on another broker right now on a VPS with a better resolution. So far so good...the indicator values didn't start from zero. I'm very confused - the plotted results on Katalon were vastly different. I'll keep running the test and see what turns up.

Ok, I've just tested it with your Outliertest script that you got last Friday. I also got the "history missing" message when I started it the first time, due to the usual server delay for uploading a new history. But afterwards the script ran ok.

Do you get the error message consistently with the Outliertest.c? And only with it, not with other scripts such as Z7?

-Update: I just noticed that the script you got has a Lookback period of 1 bar. This could be a hint that there's still something wrong. We'll look into that.

Its just with that test script. And it happens even if I repeatedly click 'Trade'. Doesn't seem to matter which broker I use either.

Update: I just got that script to connect and start trading! I can't explain why it worked, I didn't do anything differently other than close Zorro and restart it, which I had done several times previously. Seems really random and I can't isolate anything that would have changed the bahavior.

Ok. We'll look into that. After all, if such bugs were easy to reproduce, anyone could find them...

Thank you. I really appreciate you taking this seriously and responding so quickly to my comments and questions.

That's not at all what I'm seeing. I'm seeing ~20% CPU usage on bridge 1.8 and >100% CPU usage on bridge 1.9. Two instances running bridge 1.9 bring my server to its knees. Far more than a 30% increase.

100% usage is not normal, with no bridge. Even 20% is a rather extreme value when you have no gigantic portfolio. Can you post that script?

The bridge script or the Zorro script? Zorro 1.42 is running Z3, Z7 and Z12 each with a separate instance of MT4 limited to the assets those systems trade connected with bridge 1.8 (or 1.9 when I want to roast a CPU core or two).

The CPU usage may be affected by the fact that these are running on Linux through WINE and rendering to a framebuffer so I wouldn't be too concerned about the absolute value. I'm more worried about the relative impact of bridge 1.9 (2x usage).

If it's just the Z systems, then the relative impact of 1.9 does not explain the CPU load. None of the Z systems consumes any noticeable CPU resources with 1.9. So the CPU load is generated elsewhere. Can you test it on a normal Windows PC or have you only Linux?

Another test that you could do: close all sessions and remove the EA from the chart, then replace Libraries\ZorroMT4.ex4 with the previous ZorroMT4.ex4 from version 1.8. This is the only file that uses a little more CPU resources than before due to a faster reaction on commands. Leave the other files unchanged. The EA will then report bridge version 1.8, but it's in fact 1.9. Please let me know if this has an effect on that 100% CPU load.

Only have Linux at the moment I'm afraid.



I tried this on Z7 and no, it has the same effect as entirely 1.9. 100% CPU usage.

Here's a pair of screenshots of my server load with a 1.9 instance (top) and without (bottom): https://imgur.com/a/0NZOi

The good news is I got no bad ticks running my original script and the test script the last few hours.

The bad news is that I am also seeing significantly higher CPU usage. Running two instances of Zorro (test script and a fairly basic portfolio script) shows up as 196,000K private working set memory in windows task manager.

Ok, we got a Linux system and can now confirm the CPU Load problem. We also know the reason. We'll upload a new version 1.9 later today. I'm afraid that won't help with a memory issue, though, but maybe this was also a CPU load issue.

Great news. Thanks jcl.

Here's the next attempt:

http://server.conitec.net/down/ZorroMT4.zip

Again, please install the DLL also in the Plugin folder. There are two changes. The high CPU Load was due to a too-high event timer rate dependent on the OS configuration. This should not happen anymore with this version. The other modification was enforcing an asset subscription when it is not available at first. This could solve the issue of unavailable assets at start. Since this problem didn't happen here with any broker, it's a "blind fix".

Please test and let me know how this version works. Don't forget to remove the old EA before putting the new one on the chart.

New one works great, thanks! Was the issue the different clock resolutions on Linux/Windows? Interesting that WINE didn't deal with that itself.

I have also just started testing this one. Working great so far. CPU usage looks a lot more reasonable too.

The error loading prices that I was experiencing with the last version has disappeared too.

I will run this throughout the Asian trading session today and report back once Europe has started up.

Thanks jcl for getting this update out so quickly.

I can report what I believe is success with the bridge. The test script provided by Zorro support returns the exact results I would expect. However, I am still encountering some inexplicable behavior with my original script (support assured me that the script itself is fine), but I am wondering if this may be an issue unrelated to the bridge.

The script relies on the HH() and LL() functions to calculate some of its indicator values. The indicators in questions are the Ichimoku indicators (don't judge me...this is something that I was asked to code by someone else). At start-up, there are a number of instances that return 0 for the Ichimoku indicators sometimes for some pairs in the asset loop. I have verified in the log file that the returned value is sometimes 0 - so its not merely a plotting error.

Here's an example of an obviously incorrect calculation of the values, and one that behaves as it should:


In the image above, the price curve is barely visible around the 85 level. The orange lines represent the Senku values from the Ichimoku indicator - clearly they have reacted to a price level of 0. Come to think of it, the blue line is an EMA, and it has also clearly been affected by a zero value - so its not restricted to just the Ichimoku indicator.

This one looks as it should and there are no zeros in the log file:


I will next run the script via both the MT4 bridge and the FXCM plugin (demo account only) and compare the results.

jcl (or anyone else), I'm wondering if this points to a problem with this particular indicator and the way it handles price quotes in real time, possibly being affected by the asset(loop() call in the script (I have noticed that results are a lot more stable when using only a single asset). This issue never appeared in any backtest, and I ran loads of them.

Sorry to keep prolonging this issue, but I would really like to get to the bottom of this one.

The general problem with those indicators is that they normally retun 0 when accessing nonexisting data, as is the case during the lookback period. This is sometimes tricky to work around, since it is unknown what they shall return instead.

Maybe this is not properly solved for the Ichimoku. I'll check.

Update: It is indeed so. The Ichimoku has no precautions against returning zero due to nonexisting data. This will be fixed in the next update, but inbetween you can simply solve it by plotting the Ichimoku only when the lookback period is over.

Well that explains a lot! I'm quite relieved that it turned out to be something so simple.

So I can work around this in the mean time by doing something like:



Would you also mind checking the situation for the EMA indicator? I couldn't find its code in the indicators.c file. I am using the EMA indicator that uses a cutoff period rather than an alpha value.

I would actually prefer to replace the EMA with a lowpass filter in my script, but it would still be useful to know if the EMA can handle non-existent prices too.

Thanks again for your help with this.

Actually, the code I wrote above won't do at all. It won't calculate any indicator values for startup.

Its no good just wrapping the 'plot' calls in an if(!is(LOOKBACK)) statement either since the problem is not a charting issue - the issue is that the incorrect indicator values are being used to trigger trades incorrectly.

What about this for a simple workaround:


Not an ideal solution since some information is lost (the actual lowest price of the previous bar), but at least it would prevent drastic changes to the way the script trades.

Any other ideas?

How can incorrect indicator values trigger trades? You must make sure with a sufficient lookback period that only valid indicator values trigger trades in your script.

I understood the issue as a chart problem, not a problem that affects the trading behavior. PriceLow is not supposed to be fixed. It's the indicator that is wrong, not PriceLow.

Here's an example of how an incorrect indicator value can trigger a trade:



In this case, if the value for senkuB[0] is incorrectly returned as zero, a trade will be triggered. This is precisely the behavior I am seeing. And my LookBack period is more than long enough. The problem is that Zorro registers an incorrect price, passes this price to an indicator value, and then triggers trades that would otherwise not be taken.

Hmm. I see no incorrect trade in your code. Zorro does not trade in the lookback period. When did that happen? Do you have a log of how Zorro registers an incorrect price and triggers a wrong trade?

That's my point - there is nothing wrong with the code. The problem is that Zorro was registering a price that never existed and that incorrect price was used to calculate the value of my indicators. The resulting incorrect indicator caused trades to be taken when they shouldn't have.

This is the very issue that I initially contacted support about. I've also stated a number of times in this thread that the detection of non-existent prices is causing incorrect indicator calculations.


Another way of explaining this is that if the price that Zorro registers is not the correct price (for example, the wild ticks in the charts shown throughout the thread), then any indicator that uses that price in its input will also not be correct.

I have the log file from when the issue first came up back on March 15, but I wasn't printing indicator values to the log file, so it is probably not much use.

I have been running the same script with printfs for some hours today and tonight, and so far I haven't seen it occur again. If/when it does happen, I will post the log file.

Ok. Support does not know anything about this new problem, nor do I. If you see it happening, please save the log and post it here, or contact Support. They'll look into it, so we can determine what the problem is. We already have your script.

There's obviously been some miscommunication (and I accept some responsibility for that) - this is not a new problem; its the same problem that I initially contacted support about. I guess it is just framed slightly differently: in terms of an incorrect indicator calculation rather than a problem with the MT4 bridge (which I assumed to be the root cause).

In any event, I haven't been able to reproduce the error from post #458662. I ran my script on 10 assets up until the Friday close and it never occurred once.

If it happens again, I'll be sure to post the log.

For the record, and so that we are all on the same page, here is my initial email to support. It clearly mentions the incorrect indicator calculations:

Hello

I'm a Zorro S user (invoice number XXXXXXX) and I'm hoping to get some help.

Is it possible for the MT4 platforms of individual brokers to be incompatible with the Zorro MT4 bridge?

I am asking because I have found that when connected to a certain broker in live trading mode, Zorro detects obviously erroneous ticks. I can find no trace of these erroneous ticks in the MT4 chart, nor in the MT4 history centre. No trades have so far been executed at or close to these levels, which makes me question whether these ticks actually existed. Is it possible that the Zorro bridge could somehow register a tick that never existed?

Although no trades have been executed at these erroneous tick levels, my system uses the recent high and recent low by way of the HH() and LL() functions. These erroneous ticks have a significant impact on the values calculated by these functions (verified by printing indicator values to the log file) and several trades have been executed as a result, when they clearly shouldn't have.

This issue first came to my attention last night on EUR/USD. Attached is the image that Zorro created - although the quality is not great (I am running my VPS using 15-bit color to free up as much RAM as possible) you can see the erroneous ticks occurring between 7 and 8 PM at values of approximately 2.3 and 1.7. These ticks dwarf the scale and therefore the rest of the price curve is barely visible. Clearly EUR/USD never traded at these levels and there is no sign of these ticks in the broker's MT4 feed.

I guess this was becoming a large confusion.

I'm not sure what problems we're talking about. You originally reported MT4 outliers that were indeed confirmed and fixed. Then you got a bad looking chart due to Ichimoku starting at 0, which I suggested to fix by not plotting during lookback. Then you told me that this would still trigger wrong trades. That's where you lost me. Was this the original bug that was fixed, or is there still another problem?

This is what I am trying to determine. I think this is the original bug that was fixed, since I can't seem to reproduce the error that I described in post #458662. But I'd like to be sure.

My understanding is that your explanation for the errors I described in post #458662 is:



There's a very good chance that I'm missing something here, but I don't see how simply not plotting the indicator is a workaround for the incorrect values returned by the Ichimoku. My trading logic depends on these values, whether they are plotted or not. Could you perhaps provide some more detail that would help me understand?

Certainly. Most indicators have a validity range. Look at the simple moving average, SMA(100). It is invalid between bars 0-98 and valid from bar 99 on. This is no bug and has nothing to do with wrong trades. It is just due to the definition of the indicator and the fact that data does not exist before bar 0.

What I suggested was simply not plotting invalid indicator values. I hope that you're not trading them anyway.

Thanks, that explanation helped me understand how you are viewing my problem. Following that logic, the error that I posted could not possibly occur after the lookback period if it was related to the validity range of the indicator. Since it did occur after the lookback period, I have to conclude that the error had another source. I will continue running the script on the next trading session and see if it happens again.

I have an update on the error I have been experiencing. There is still something not right, and I believe I have narrowed it down. Firstly, some context:

I have been testing a script that plots the indicators that control the trade logic, but which does not take any trades. The trade logic is commented out. It is the same script I sent to support with one minor alteration (details below). The script looks at two time frames (5-minute and 1-hour) and plots the Ichimoku senku spans from each time frame. I am now using TimeFrame = frameSync(60) to control the H1 indicators, rather than just TimeFrame = 60 in the original script sent to support.

I have been running the script on Katalon markets MT4 via the MT4 brdige V1.9. In parallel, I am running a copy of the script on an FXCM demo account via the API. I have printed out the indicator values from the 1-hour time frame into a log file.

The indicators from the 5-minute time frame look perfectly fine. The values closely match between the FXCM API and the MT4 bridge instances. The plots of these indicators match each other and the equivalent indicators plotted in the Katalon MT4 terminal.

The indicators from the 1-hour time frame are a different story. They almost match, but with some very obvious differences. Below I show plots from the FXCM API instance, the MT4 bridge instance and finally from the Katalon MT4 terminal. The former two show both the 5-minute indicators (orange) and price action and the 1-hour indicators (blue). The chart from the MT4 terminal shows the 1-hour indicators and price action only. The period of interest is between the two red vertical lines.

You can see that the MT4 bridge version doesn't quite match the FXCM API version and the MT4 terminal verion. It can't be explained by differences in the broker feed, since the MT4 terminal is the same broker being accessed via the MT4 bridge.

Curious that I can't see any errors on the 5-minute indicators, only in the 1-hour indicators. Does this give a clue as to the source of the error?

MT4 Bridge version:


FXCM API version:


MT4 terminal version:


I will upload the log files in a separate post.

Is there a way to upload text files to the forum? I don't see the "File Manager" link that is referred to in the FAQ.

It was deactivated, but should be visible now. - We got your logs, but the difference between FXCM and MT4 is the server time. While the FXCM server time reflects the real time, the MT4 server time has only the time of the last price update. This leads to the time frame series being shifted at different times. For this reason we had already thought about removing the MT4 server time function altogether.

For a quick fix, can you replace TimeFrame = frameSync(12) just by TimeFrame = 12? Or is there a reason for trading exactly at hour boundaries?

Update: I've just looked in your script again. In fact we have a worse problem. I didn't notice it first, but I believe your H1 Ichimoku signals probably won't work at all - regardless of FXCM or MT4. I'll contact you by email about this and we'll work out a solution.

Thanks jcl, much appreciated.