Gamestudio Links
Zorro Links
Newest Posts
Blobsculptor tools and objects download here
by NeoDumont. 03/28/24 03:01
Issue with Multi-Core WFO Training
by aliswee. 03/24/24 20:20
Why Zorro supports up to 72 cores?
by Edgar_Herrera. 03/23/24 21:41
Zorro Trader GPT
by TipmyPip. 03/06/24 09:27
VSCode instead of SED
by 3run. 03/01/24 19:06
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
2 registered members (monk12, Quad), 830 guests, and 4 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
Page 1 of 5 1 2 3 4 5
IB Plugin needs a fix #468480
10/06/17 15:45
10/06/17 15:45
Joined: Jul 2017
Posts: 783
Z
Zheka Offline OP
User
Zheka  Offline OP
User
Z

Joined: Jul 2017
Posts: 783
Jcl,

It is stated numerous times all over the manual that
Quote:
Historical price data for backtests can not be downloaded from IB
or that they are very slow and limited.

This proves true - with Zorro. But not with other platforms.
I have been using both Multicharts and Sierra with IB, and historical data download has always happened quite swiftly.

Just did a test, downloading 5 days of EUR.JPY 1 min data.
MultiCharts- <1min
Zorro - 8min - stopped after loading 1 day..

Looking at the IB Gateway logs, the reasons are as follows:

1. Zorro requests data in 36000s(=10hrs) chunks, and re-requsts every 5hrs (??!)
Quote:
2017-10-06 09:20:46.293 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:DET] ReqHistoricalData(20)::[version=6,ID=4,action=null,reqDesc=Symbol=EUR Type=CASH Expiry=null Strike=0.0 Put/Call=? Exchange=IDEALPRO CompExch=null Currency=JPY Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false newsSource=null Legs=null Special Info=null,endDateTimeStr=20171006 13:20:46 GMT,backfillDuration=36000 S, whatToShow=4,barSizeSettingStr=1 min,formatDate=2,combo=null]

Quote:
2017-10-06 09:21:57.391 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:DET] ReqHistoricalData(20)::[version=6,ID=6,action=null,reqDesc=Symbol=EUR Type=CASH Expiry=null Strike=0.0 Put/Call=? Exchange=IDEALPRO CompExch=null Currency=JPY Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false newsSource=null Legs=null Special Info=null,endDateTimeStr=20171006 08:21:00 GMT,backfillDuration=36000 S, whatToShow=4,barSizeSettingStr=1 min,formatDate=2,combo=null]

while MC requests data for 5d in one go.
Quote:
2017-10-06 09:54:39.228 [IL] INFO [JTS-EServerSocket-3491] - [21500:63:76:1:0:0:0:DET] ReqHistoricalData(20)::[version=6,ID=1000007,action=null,reqDesc=Symbol=EUR.JPY Type=CASH Expiry=null Strike=0.0 Put/Call=? Exchange=IDEALPRO CompExch=null Currency=JPY Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false newsSource=null Legs=null Special Info=null,endDateTimeStr=20170929 21:00:00 UTC,backfillDuration=5 D, whatToShow=4,barSizeSettingStr=1 min,formatDate=2,combo=null]

From IB API guide(https://interactivebrokers.github.io/tws-api/historical_limitations.html#gsc.tab=0):
Quote:
Historical Data requests need to be assembled in such a way that only a few thousand bars are returned at a time.

And so, it takes MC 15sec from the request to closing the historical data socket.

2. besides "Ask", Zorro also requests data for "Trades" - which do not exist for FX.
Quote:
2017-10-06 09:20:47.152 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:DET] ReqHistoricalData(20)::[version=6,ID=5,action=null,reqDesc=Symbol=EUR Type=CASH Expiry=null Strike=0.0 Put/Call=? Exchange=IDEALPRO CompExch=null Currency=JPY Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false newsSource=null Legs=null Special Info=null,endDateTimeStr=20171006 13:20:46 GMT,backfillDuration=36000 S, whatToShow=0, barSizeSettingStr=1 min,formatDate=2,combo=null]
2017-10-06 09:20:47.152 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:DET] [6;5;0;EUR;CASH;null;0;2;null;IDEALPRO;JPY;null;null;false;false;20171006 13:20:46 GMT;1 min;36000 S;0;TRADES;false]
.......
2017-10-06 09:20:47.152 [IL] INFO [JTS-cashhmdsDispatcherS42-3413S42-3414] - cdebug: QUERY | WARNING | qm@305bef71 | Query error | 1755;;EUR.JPY@IDEALPRO Trades;;1;;true;;0;;I | No historical market data for EUR/CASH@FXSUBPIP Last 60

It should have been "Bid" (whattoshow=3), correct?

3. There is a huge lot of strange UnSet() messages generating errors.
Quote:
2017-10-06 09:20:49.031 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:INFO] Handling incoming UnSet(0) message.
2017-10-06 09:20:49.031 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:ERR] Invalid incoming request type - 0

There are also numerous ReqMrktData calls in the first 27 seconds of connection and ReqAccData in the very beginning (even though I pressed [Test], not [Trade]. Script is only "UpdateDays=5"). The first call for historical data is 27sec after pressing the "Test" button (2sec for MC).

Log files are attached.

Btw: MC opens 2 client connections in IB Gateway

Attached Files
MC_IB_log_10062017.txt (580 downloads)
Zorro_IB_log_10062017.txt (530 downloads)
Last edited by Zheka; 10/06/17 18:26.
Re: IB Plugin needs a fix [Re: Zheka] #468489
10/07/17 06:31
10/07/17 06:31
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
I do not know of a Zorro problem with IB historical data. Nor that Zorro requests "5 hours" or "trades". Only prices and volume are requested, and only at session start. If you cannot download some specific IB data, please contact Support.

IB has strongly limited their historical data until 2016. That was mentioned in the manual, maybe that has confused you. The current limits are listed on the IB website. It's normally 1 year. Zorro, and any other software, can download all historical data within those limits.

Re: IB Plugin needs a fix [Re: jcl] #468492
10/07/17 14:36
10/07/17 14:36
Joined: Jul 2017
Posts: 783
Z
Zheka Offline OP
User
Zheka  Offline OP
User
Z

Joined: Jul 2017
Posts: 783
It would help immensely if instead of a blanket "there is no problem", we could have a fact-based, productive conversation.

I did not manually create those logs. Run a simple script:
Quote:
function run()
{
UpdateDays=5;
quit;
}

set logging level in IBGateway to "Detail" and see for yourself how exactly the api requests are formed, how often and which ones generate errors.

I really want to make it work and your any support would be greatly appreciated. Please post your log.

P.S:
- on "5hrs": I underlined the relevant fields in the quoted API requests above
- historical data availability from IB was not the point of my post.

Re: IB Plugin needs a fix [Re: Zheka] #468507
10/08/17 19:22
10/08/17 19:22
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online
Serious User
AndrewAMD  Online
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
jcl,

Is the IB plugin invoking a data request for every BrokerHistory2 call? I did the same thing with the Ally plugin because it was easier to implement it that way than to make my own tick buffer.

BrokerHistory2 has a 300-tick limit. Can we perhaps increase the limit or otherwise implement a BrokerHistory3 function where the plugin can return the number of ticks (say, 100,000 ticks) and point to a static/global array of ticks in the plugin as an output?

Re: IB Plugin needs a fix [Re: AndrewAMD] #468508
10/08/17 20:53
10/08/17 20:53
Joined: Jul 2017
Posts: 783
Z
Zheka Offline OP
User
Zheka  Offline OP
User
Z

Joined: Jul 2017
Posts: 783
Yes, it does look like a (legacy?) 300-tick limitation
Quote:
nTicks Input, maximum number of ticks to be filled; guaranteed to be 300 or less.
ticks Output, array of up to 300 T6 structs (defined in includetrading.h) to be filled with the ask price history

in filling-in t6 structures internally by Zorro is the reason for current IB plugin re-requesting data for each 5-hr step back in history.

Lifting this limit (and correcting plugins) would speed up loading historical data significantly.

Re: IB Plugin needs a fix [Re: Zheka] #468512
10/09/17 01:29
10/09/17 01:29
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online
Serious User
AndrewAMD  Online
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
Once again, this depends on how the IB plugin was written.

Re: IB Plugin needs a fix [Re: AndrewAMD] #468516
10/09/17 07:15
10/09/17 07:15
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Yes, all newer plugins use BrokerHistory2. The maximum number of ticks per call is 300, but it can be less dependent on the time period. This has no effect on speed.

- Zheka: Please read my response. I never say that there is no problem, only that I do not know your problem, and you should therefore contact Support. If you have an issue that the other users don't have, we must find out what's different - otherwise we cannot help. It's as simple as that. Your code produces no log, it just sets a variable and quits, so I do not see the problem with that. Downloading data from the broker is described in the manual. You can use the Download script or a simple function:

void main()
{
StartYear = 2017;
assetHistory("EUR/USD",1);
}

And if this does not work for you, just explain the problem to Support, and submit as many logs as you want. They can help only when they know the problem, and can reproduce it.

Re: IB Plugin needs a fix [Re: jcl] #468526
10/09/17 10:07
10/09/17 10:07
Joined: Jul 2017
Posts: 783
Z
Zheka Offline OP
User
Zheka  Offline OP
User
Z

Joined: Jul 2017
Posts: 783
Quote:
The maximum number of ticks per call is 300

Can this be changed? Its a bottleneck..

Quote:
this has no effect on speed.

It is obviously less efficient than requesting 1 day of data in 1 go, AND it leads to "pacing violations" on IB API much faster.

Quote:
Your code produces no log, it just sets a variable and quits

I have all the time referred to API messages logged by IB Gateway.

Quote:
I never say that there is no problem, only that I do not know your problem, and you should therefore contact Support

OK, will do. To be frank, I thought support reads this forum and there is no need to do the same thing twice.

Quote:
If you have an issue that the other users don't have

1. It's just that not many users know what's normal and/or know where to look and/or dare to ask.
2. It is all over the manual that getting IB historical data is slow and should not be relied upon.
This really puzzled me, since for the last 4 years I have been freely downloading historical data in MC quite swiftly.
IB's "1 year" limitation has been more of a 'disclaimer'; downloading 6-7 years of 1min data in one go has never been a problem. Yes, with occasional 'pacing violations' but all in all user experience had been smooth. So, I know it works, and so Zorro should be able to do it similarly well.

Re: IB Plugin needs a fix [Re: Zheka] #468527
10/09/17 10:17
10/09/17 10:17
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
You can easily see with the script above that the 1 year limitation and the slowness really exist. And it has nothing to do with 300 ticks. With 3000 ticks it would be not faster. When you claim you can download more years or faster with other software, then my first suspect is that you're simply downloading it in lower resolution.

Since it makes obviously not much sense to speculate here - if you think you have a technical issue, get to support. They do not read this forum. If it's a bug, help is free. And suggestions are free anyway.

Re: IB Plugin needs a fix [Re: jcl] #468529
10/09/17 10:23
10/09/17 10:23
Joined: Jul 2017
Posts: 783
Z
Zheka Offline OP
User
Zheka  Offline OP
User
Z

Joined: Jul 2017
Posts: 783

Yes, slowness is mostly because of other issues mentioned in my original post.

Still, can a 300 tick limitation be changed - to at least 2000?
This would make it compliant with IB API's recommendation (see above).

Page 1 of 5 1 2 3 4 5

Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1