Gamestudio Links
Zorro Links
Newest Posts
Blobsculptor tools and objects download here
by NeoDumont. 03/28/24 03:01
Issue with Multi-Core WFO Training
by aliswee. 03/24/24 20:20
Why Zorro supports up to 72 cores?
by Edgar_Herrera. 03/23/24 21:41
Zorro Trader GPT
by TipmyPip. 03/06/24 09:27
VSCode instead of SED
by 3run. 03/01/24 19:06
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
3 registered members (AndrewAMD, Quad, TipmyPip), 865 guests, and 6 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rating: 4
Page 4 of 7 1 2 3 4 5 6 7
Re: New Zorro version 1.96 [Re: DdlV] #475030
11/19/18 12:13
11/19/18 12:13
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Thanks, I've put this on the todo list.

- New version:

http://opserver.de/down/Zorro_196.exe

It's still 1.96.4 since Zorro was unchanged, but wrong parameters in the GP asset list and a wrong variable definition have been fixed.

Re: New Zorro version 1.96 [Re: jcl] #475079
11/21/18 20:28
11/21/18 20:28
Joined: Sep 2017
Posts: 235
H
Hredot Offline
Member
Hredot  Offline
Member
H

Joined: Sep 2017
Posts: 235
There is a problem with downloading historical data from IB Gateway with this version. Only the API connection field becomes green, while historical data farm field does not even appear -- which means Zorro does not even send the corresponding request. Instead Zorro shows "error 056". This error implies that the broker does not offer price data, but this is clearly wrong, since it worked before just fine.

Could you please fix the historical data download from IB in the newest zorro version?

Re: New Zorro version 1.96 [Re: Hredot] #475086
11/22/18 07:55
11/22/18 07:55
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
IB download has not changed in 1.96, but the usual error 056 reasons are a closed market or no market data permission.

First check with the Download script with which assets the download works, and with which not. Then check your data subscriptions, and also check if you have a recent Gateway version. The Gateway log normally shows you what the problem is. If you still cannot find the reason, IB support might help. You can also contact Zorro support with a detailed problem description and the Zorro and Gateway logs, and they'll help too.

Re: New Zorro version 1.96 [Re: jcl] #475093
11/22/18 20:20
11/22/18 20:20
Joined: Sep 2017
Posts: 235
H
Hredot Offline
Member
Hredot  Offline
Member
H

Joined: Sep 2017
Posts: 235
I got the newest version of IB Gateway, necessary Asset subscriptions with IB are in place. Moreover, Multicharts .net downloads the prices from that same IB Gateway just fine.

I invite you to take your favorite asset list and see for yourself:

Code:
function run()
{
  NumYears = 10;
  assetList("History\AssetsMinutes.csv"); // load asset list
  while(loop(Assets))
    assetHistory(Loop1,1);
  quit();
}



Seems to be a problem with Zorro after all.

Re: New Zorro version 1.96 [Re: Hredot] #475101
11/23/18 10:43
11/23/18 10:43
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
If you used this script for your download attempts, I readily believe that you had no luck. IB has lifted some hard limits, but they still disconnect you when you download too much too fast.

Do as I suggested, make sure that at least your symbols are correct, set a proper request rate, and if you then still have problems, let Support help.

Re: New Zorro version 1.96 [Re: jcl] #475109
11/23/18 14:03
11/23/18 14:03
Joined: Jul 2017
Posts: 783
Z
Zheka Offline
User
Zheka  Offline
User
Z

Joined: Jul 2017
Posts: 783
The script works on my side, at least with FX pairs. What symbols cause problems?

@JCL,
Why would this script not work with IB? Do you mean it will be slow to load data?
Probably, but why would it crash?

Here are 2 subsequent requests to the API when I just tested the script:
Quote:
2018-11-23 07:56:45.805 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:DET] ReqHistoricalData(20)::[version=6,ID=418,action=null,reqDesc=Symbol=AUD Type=CASH Expiry=null Strike=0.0 Put/Call=? Exchange=IDEALPRO CompExch=null Currency=USD Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false newsSource=null Legs=null Special Info=null,endDateTimeStr=20101215 07:00:30GMT,backfillDuration=2 D ,whatToShow=4,barSizeSettingStr=1 min,formatDate=2,combo=null]
.....................
2018-11-23 07:56:47.024 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:INFO] Handling incoming UnSet(0) message.
2018-11-23 07:56:47.024 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:ERR] Invalid incoming request type - 0
2018-11-23 07:56:49.045 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:INFO] Handling incoming UnSet(0) message.
2018-11-23 07:56:49.045 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:ERR] Invalid incoming request type - 0
2018-11-23 07:56:51.014 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:INFO] Handling incoming UnSet(0) message.
2018-11-23 07:56:51.014 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:ERR] Invalid incoming request type - 0
2018-11-23 07:56:53.031 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:INFO] Handling incoming UnSet(0) message.
2018-11-23 07:56:53.031 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:ERR] Invalid incoming request type - 0
2018-11-23 07:56:55.000 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:INFO] Handling incoming UnSet(0) message.
2018-11-23 07:56:55.000 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:ERR] Invalid incoming request type - 0
2018-11-23 07:56:57.016 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:INFO] Handling incoming UnSet(0) message.
2018-11-23 07:56:57.016 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:ERR] Invalid incoming request type - 0

...................................
2018-11-23 07:56:58.172 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:DET] ReqHistoricalData(20)::[version=6,ID=419,action=null,reqDesc=Symbol=AUD Type=CASH Expiry=null Strike=0.0 Put/Call=? Exchange=IDEALPRO CompExch=null Currency=USD Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false newsSource=null Legs=null Special Info=null,endDateTimeStr=20101214 15:45:30 GMT,backfillDuration=2 D,whatToShow=4,barSizeSettingStr=1 min,formatDate=2,combo=null]

2018-11-23 07:56:59.360 [QF] INFO [JTS-EServerSocket-948] - [1:63:76:1:0:0:0:DET] ReqHistoricalData(20)::[version=6,ID=420,action=null,reqDesc=Symbol=AUD Type=CASH Expiry=null Strike=0.0 Put/Call=? Exchange=IDEALPRO CompExch=null Currency=USD Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false newsSource=null Legs=null Special Info=null,endDateTimeStr=20101214 00:45:30 GMT,backfillDuration=2 D,whatToShow=4,barSizeSettingStr=1 min,formatDate=2,combo=null]

And this is how MC requests same data from IB:
Quote:
2018-11-23 08:40:46.031 [QF] INFO [JTS-EServerSocket-1017] - [31116:139:139:1:0:0:0:DET] ReqHistoricalData(20)::[version=139,ID=1000005,action=null,reqDesc=Symbol=AUD.USD Type=CASH Expiry=null Strike=0.0 Put/Call=?
Exchange=IDEALPRO CompExch=null Currency=USD Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false
newsSource=null Legs=null Special Info=null,endDateTimeStr=20181108 22:00:00 UTC,backfillDuration=5D, whatToShow=4,barSizeSettingStr=1 min,formatDate=2,combo=null]
.......
2018-11-23 08:40:47.172 [QF] INFO [JTS-EServerSocket-1017] - [31116:139:139:1:0:0:0:DET] ReqHistoricalData(20)::[version=139,ID=1000007,action=null,reqDesc=Symbol=AUD.USD Type=CASH Expiry=null Strike=0.0 Put/Call=?
Exchange=IDEALPRO CompExch=null Currency=USD Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false
newsSource=null Legs=null Special Info=null,endDateTimeStr=20181101 21:00:00 UTC,backfillDuration=5 D, whatToShow=4,barSizeSettingStr=1 min,formatDate=2,combo=null]

No erroneous requests taking full 10sec and EndDateTime is calculated properly.

It took Zorro 14-15+ sec to download 1 day of data (in 3 requests), while it took MC 1 sec to download 5.

So much for "slow" IB API.

Re: New Zorro version 1.96 [Re: Zheka] #475111
11/23/18 14:35
11/23/18 14:35
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
The script won't work with unavailable or too many assets in the list. So it's no good idea to try downloading all assets from some list, unless it was checked before that all can be really downloaded. Otherwise the IB API can disconnect.

"Though IB has lifted the "hard" limit, we still implement a "soft" slow to load-balance client requests vs. server response. Requesting too much historical data can lead to throttling and eventual disconnect of the API client. If a request requires more than several minutes to return data, it would be best to cancel the request using the IBApi.EClient.cancelHistoricalData function".

Re: New Zorro version 1.96 [Re: jcl] #475116
11/23/18 15:08
11/23/18 15:08
Joined: Jul 2017
Posts: 783
Z
Zheka Offline
User
Zheka  Offline
User
Z

Joined: Jul 2017
Posts: 783
Quote:
too many assets in the list..
we still implement a "soft" slow to load-balance client requests vs. server response
Pacing of requests is exactly the job of a plug-in, cannot be the user's script.

Though as of TWS/IBG v974 IB API - finally! - implements automatic pacing.
Quote:
Better Pacing: API messages sent at a higher rate than 50/second can now be paced by TWS at the 50/second rate instead of potentially causing a disconnection. This is now done automatically by the RTD Server API and can be done with other API technologies by invoking SetConnectOptions("+PACEAPI") prior to eConnect.
https://www.interactivebrokers.com/en/index.php?f=5061

This has been tested by some people at IBAPI forum and it does work, allowing to update prices for an option chain of 1860 contracts in under 40sec.

Quote:
Requesting too much historical data can lead to throttling and eventual disconnect of the API client.
And as is clear from the logs above, Zorro's IB plug-in is grossly inefficient at determining how much to request in one go (and is by far not pushing the limits).

But what can potentially cause "throttling" by IB is that the same hist.data is re-requested several times (which is a no-no) because of incorrectly formed EndDates..

Re: New Zorro version 1.96 [Re: Zheka] #475119
11/23/18 19:38
11/23/18 19:38
Joined: Sep 2017
Posts: 235
H
Hredot Offline
Member
Hredot  Offline
Member
H

Joined: Sep 2017
Posts: 235
Hi Zheka, interesting to see that it works for you, while for me it just bounces back error messages. Perhaps I should put fewer assets into the list at once.

I agree, proper pacing is the task of the plugin, not the user. Zorro just being Zorro again... Even the Zorro manual still wrongly states that one cannot load historical data from IB.

I'm curious, Zheka, you seem to be rather experienced and certainly have access to multicharts, which by the looks of it is a far superior option. May I ask why you bother to waste time on Zorro?

Last edited by Hredot; 11/23/18 19:43.
Re: New Zorro version 1.96 [Re: Hredot] #475123
11/23/18 23:53
11/23/18 23:53
Joined: Jul 2017
Posts: 783
Z
Zheka Offline
User
Zheka  Offline
User
Z

Joined: Jul 2017
Posts: 783
Mainly to trade Options and to access MT4 brokers (without having to learn and program in MQL).

Quote:
Perhaps I should put fewer assets into the list at once
I dont think this has anything to do with the length of a list. Zorro will not start processing the next asset until it is finished with the current one. And there is no limitation in the API as to the number of paced requests..

Set IBG to a logging level "Detail" and try to see which commands come in as Zorro tries to connect and how API responds...

Last edited by Zheka; 11/24/18 00:06.
Page 4 of 7 1 2 3 4 5 6 7

Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1