Jcl,

It is stated numerous times all over the manual that
or that they are very slow and limited.

This proves true - with Zorro. But not with other platforms.
I have been using both Multicharts and Sierra with IB, and historical data download has always happened quite swiftly.

Just did a test, downloading 5 days of EUR.JPY 1 min data.
MultiCharts- <1min
Zorro - 8min - stopped after loading 1 day..

Looking at the IB Gateway logs, the reasons are as follows:

1. Zorro requests data in 36000s(=10hrs) chunks, and re-requsts every 5hrs (??!)


while MC requests data for 5d in one go.

From IB API guide(https://interactivebrokers.github.io/tws-api/historical_limitations.html#gsc.tab=0):

And so, it takes MC 15sec from the request to closing the historical data socket.

2. besides "Ask", Zorro also requests data for "Trades" - which do not exist for FX.

It should have been "Bid" (whattoshow=3), correct?

3. There is a huge lot of strange UnSet() messages generating errors.

There are also numerous ReqMrktData calls in the first 27 seconds of connection and ReqAccData in the very beginning (even though I pressed [Test], not [Trade]. Script is only "UpdateDays=5"). The first call for historical data is 27sec after pressing the "Test" button (2sec for MC).

Log files are attached.

Btw: MC opens 2 client connections in IB Gateway

I do not know of a Zorro problem with IB historical data. Nor that Zorro requests "5 hours" or "trades". Only prices and volume are requested, and only at session start. If you cannot download some specific IB data, please contact Support.

IB has strongly limited their historical data until 2016. That was mentioned in the manual, maybe that has confused you. The current limits are listed on the IB website. It's normally 1 year. Zorro, and any other software, can download all historical data within those limits.

It would help immensely if instead of a blanket "there is no problem", we could have a fact-based, productive conversation.

I did not manually create those logs. Run a simple script:

set logging level in IBGateway to "Detail" and see for yourself how exactly the api requests are formed, how often and which ones generate errors.

I really want to make it work and your any support would be greatly appreciated. Please post your log.

P.S:
- on "5hrs": I underlined the relevant fields in the quoted API requests above
- historical data availability from IB was not the point of my post.

jcl,

Is the IB plugin invoking a data request for every BrokerHistory2 call? I did the same thing with the Ally plugin because it was easier to implement it that way than to make my own tick buffer.

BrokerHistory2 has a 300-tick limit. Can we perhaps increase the limit or otherwise implement a BrokerHistory3 function where the plugin can return the number of ticks (say, 100,000 ticks) and point to a static/global array of ticks in the plugin as an output?

Yes, it does look like a (legacy?) 300-tick limitation

in filling-in t6 structures internally by Zorro is the reason for current IB plugin re-requesting data for each 5-hr step back in history.

Lifting this limit (and correcting plugins) would speed up loading historical data significantly.

Once again, this depends on how the IB plugin was written.

Yes, all newer plugins use BrokerHistory2. The maximum number of ticks per call is 300, but it can be less dependent on the time period. This has no effect on speed.

- Zheka: Please read my response. I never say that there is no problem, only that I do not know your problem, and you should therefore contact Support. If you have an issue that the other users don't have, we must find out what's different - otherwise we cannot help. It's as simple as that. Your code produces no log, it just sets a variable and quits, so I do not see the problem with that. Downloading data from the broker is described in the manual. You can use the Download script or a simple function:

void main()
{
StartYear = 2017;
assetHistory("EUR/USD",1);
}

And if this does not work for you, just explain the problem to Support, and submit as many logs as you want. They can help only when they know the problem, and can reproduce it.

Can this be changed? Its a bottleneck..


It is obviously less efficient than requesting 1 day of data in 1 go, AND it leads to "pacing violations" on IB API much faster.


I have all the time referred to API messages logged by IB Gateway.


OK, will do. To be frank, I thought support reads this forum and there is no need to do the same thing twice.


1. It's just that not many users know what's normal and/or know where to look and/or dare to ask.
2. It is all over the manual that getting IB historical data is slow and should not be relied upon.
This really puzzled me, since for the last 4 years I have been freely downloading historical data in MC quite swiftly.
IB's "1 year" limitation has been more of a 'disclaimer'; downloading 6-7 years of 1min data in one go has never been a problem. Yes, with occasional 'pacing violations' but all in all user experience had been smooth. So, I know it works, and so Zorro should be able to do it similarly well.

You can easily see with the script above that the 1 year limitation and the slowness really exist. And it has nothing to do with 300 ticks. With 3000 ticks it would be not faster. When you claim you can download more years or faster with other software, then my first suspect is that you're simply downloading it in lower resolution.

Since it makes obviously not much sense to speculate here - if you think you have a technical issue, get to support. They do not read this forum. If it's a bug, help is free. And suggestions are free anyway.

Yes, slowness is mostly because of other issues mentioned in my original post.

Still, can a 300 tick limitation be changed - to at least 2000?
This would make it compliant with IB API's recommendation (see above).

Yes, it can be changed, but all this was tested with the IB API. The 300 ticks have been most effective. What matters for speed is the total number of prices that you download, not the number of ticks per call.

Is 300 ticks an internal Zorro limitation (of saving to *.t8 structs) or is it specific/has grown from IB API's?

MC requests data in 1-day chunks(=1440 1min ticks) - which is in line with IB APIs recommendation and works well in practice. While Zorro makes 5 requests where 1 is really needed (Zorro also requests data in "36000s" (=600 tick) chunks but only processes 300).

This limitation might also be not optimal for other broker's APIs. Let's give Broker plug-in writers more freedom!


Say, we need to download 6 days (=8640 of 1min bars).
Zorro requests 2 types of data - Ask and Trade(should be Bid - to calculate spread?).
8640/300= 29 *2 sides=58 requests - which causes(because there are also other requests) a "pacing violation" of IB API.

300 ticks is the optimal size for most APIs, only a few need smaller sizes. This has something to with TCP/IP packets. Internet routers are most effective with packets below 5000 bytes, as normally generated by 300 ticks requests.

I do not know the IB data types, but you need the ask price and the volume for historical data, and that's what is requested.

And if you really get a "pacing violation" error, just contact support and describe how you did that. They'll look into it.

IB API sends volume through "TRADES" tick type - which is not available for FX at IB. Zorro's historical data request for "TRADES" (to obtain volume) generates errors.

TCP/IP tuning is definitely important for minimizing latency in RT trading, but much less so for downloading historical data.

Anyway, if you lift the limitation on Zorro's side, your IB plugin can still request data in 300 ticks.
But other people will be able to use it with more flexibility in their own implementations.

I reckon that it returns an error code when requested data is not available for a certain asset, but that should not be harmful.

And if you want to write a plugin and request 2000 ticks data, we'll certainly lift that limitation for you. Plugin authors get from us almost anything.

What takes MC <1min, takes Zorro forever. There must be reasons, and comparing logs give a hint.

I do not have time now to write a plug-in from scratch - I have no experience writing them in full; it would take me longer. It is surely easier to improve current code, than re-invent the wheel.

But AndrewAMD - author of AllyInvest plug-in - also asked to lift the limit.

Also: if we change from requesting Trades to Bid, we can fill in the actual Spread in historical data.

I will formally request it after I run a couple of API tests.

This feature would probably benefit me more than other plugin writers, though. For example, it might not benefit the IB plugin.

Ok. We'll lift that limit for the next release after 1.70.

Great!

What about requesting Bids to fill in actual spread values in historical data?

Jcl,
getting further into how IB Plugin currently works, I found that it actually requests data snapshots every 500ms. Is this by design? This guarantees significant differences between historical and live data.
IB streams data in 250ms snapshot at worst, and in 5ms snapshots for FX(if there is quote).
Can Zorro actually process all ticks, if fed by a plug-in?

Yes, but the default is that fast strategies request data faster than slow strategies. You can set this up with TickTime. For streaming data the plugin must trigger a callback function.

According to the manual, TickTime is for triggering tick() function.
Does it also influence frequency of data snapshot requests to the broker api?


Does it mean that in the current IB plugin it is not defined?
Is this then to be used in reqMktData() call?

The context for my questions is not that I want to execute an HFT strategy. It is rather to make sure that O-H-L-C are as correct (reflect full market activity and equal to historical bars) as possible.

Jcl,
please reply..
I am also not clear on why
How does a user-defined callback function relate to the Broker plugin subscribing (or not) to the streaming data updates?

The answer on both questions is yes - TickTime affects the price request frequency, and the current IB plugin uses no streams. If I remember right, streams cannot be used with portfolio strategies.

The mentioned callback function is BrokerProgress, to be called on incoming ticks. If the function is not called, the request frequency is determined by TickTime.

THANKS!

Dear JCL,

I have provided to support all the logs/evidence that IB plug-in requires a rework. They accepted it, but the timing is "next time" - which does not help things a bit.

In addition, the requested increase in the number of ticks Zorro stores/plugin requests came implemented as a read-only field to "GET" (what's the use of this??) - as confirmed by Support, - contrary to what's written in the manual and intention And IB Plugin does not seem to support it anyway.

Would it be possible to rework the IB plug-in sooner rather than later?
It is barely usable in production in its current form + IB API has undergone quite some changes over the last year.

Thank you very much in advance.

I do not really understand the objection against GET_MAXTICKS, but can confirm that the IB rework is planned for quite soon and will include several changes. For instance switching the price source from ask/bid to trades.

The objection against GET_MAXTICKS is that it is a READ-ONLY field, and so does not allow the end user to experiment with changing this value.
And the plug-in developer knows about this value anyway and would not access it via BrokerCommand(). What would be the use case for this command for the end-user?
Please let's make it also "settable".

Great news for the IB plug-in! I have 2 suggestions:
- for FX/metals let's report *mid-point*- as provided by IB - as Trades. There are no Trades per se for these assets.
- let's also implement IB's *5-sec True RT bars*.
Such 5-sec bar encompasses ALL ticks and comes with a 250-300ms delay - which is good enough for anybody trading at 10-15min+ resolution.
These bars come already cleaned from IB's Historical servers; are exactly equal to the "Historical bars" one can download from IB - and so can be saved to *.t6 structs already in RT (either as 5-sec or as 1min).
250-300ms delay - and you can trade exactly what you would have tested on!

For this to work properly in all cases, Zorro should ideally allow sending a real stop order to the broker (like it already allows with limit orders), not just simulate it.

GET_MAXTICKS is not for the end user, but instructs Zorro to download data in the returned packet size. The plugin developer, not the end user, is supposed to experiment with this value.

The price source switch command could also be used for switching to RT bars. It they are not a big hassle to implement, we can include them.

Zorro can send real stops to the broker, but the IB plugin does not support them. AFAIK the IB API has no stop order, but requires two orders instead.

So, can a plugin developer SET this value to experiment? Or you set it?
Let's set it to 2500-3000 for IB by default - as per IB's recommendation!

They should not be. Would be really great to save them to *.t6 structs in RT.

IB API offers a huge variety of orders. Depending on the exchange, some are native and some are simulated. https://interactivebrokers.github.io/tws-api/basic_orders.html#stop

Stops are mostly simulated, but they would definitely be effected much faster by IB, then by Zorro. So, please implement BrokerStop() for IB.

- Limit orders are native for most exchanges; but what Zorro actually implements instead is a "30-sec Stop-limit" order (via ORDERLIMIT flag+SET_LIMIT BrokerCommand).
Please add a BrokerLimit() command to allow placing resting limit orders (or a similar functionality to do that).

Overall, most limitations on orders mentioned in the manual re NFA are handled by IB internally; So, a Stop or TakeProfit can safely be sent as Stop and Limit order by the plug-in; without asking the user to separately call a BrokerStop() or BrokerLimit()/ ORDERLIMIT. It is less efficient/costlier to handle them on Zorro's side.

Some other wishes:

- OrderIDs - please return OrderId, instead of TradeIDs (=0 because they are not supplied by IB)

- please return Api Errors and Order State/ Rejection reasons

- please extend or allow to extend order(int type):
IB has a lot of great order algos, and there should be a way to use them from within Zorro. "Trails" and OCA, Combos have native IB implementations.

- please allow to receive IV/Greeks from IB

Thank you!

When you look in the API doc, you can see that you need two orders for a stop loss trade. This is NFA compliance and not at the broker`s liberty. Stop orders with NFA brokers are not planned at this point, but maybe in a future version.

JCL,

IB 1000% allows to submit Stop orders via their API regardless if those are entry or exit orders. Stops at IB will become market orders much faster then when Zorro does the same.

In none of the current IB Links is there any mention of "two orders" (or NFA):

https://interactivebrokers.github.io/tws-api/basic_orders.html#stop&gsc.tab=0
https://www.interactivebrokers.com/en/index.php?f=609
https://www.interactivebrokers.com/en/index.php?f=719

Where in IB API docs is there a reference to the need to place 2 orders? (and which ones?)

I trade with both MultiCharts and Sierra Charts, and both have been sending stop orders (entry and exit) to IB without any problem.

https://www.multicharts.com/trading-software/index.php/Order_Types
https://www.sierrachart.com/index.php?page=doc/InteractiveBrokers.php#IBStopOrderTriggerMethod

Where is the catch?

"Trades" and "Orders" are two different things.

A stop order is still only an order. Zorro was designed to manage "trades". Consider the implications for the plugin development process.

Correct. Opening 1 _Trade_ with a stop loss requires 2 _Orders_.

For a stop loss trade in the IB API, you first open a market order in the desired direction. Then you open a stop order in the opposite direction. IB also requires some "bracket" mechanism for preventing that the second order is opened when the first was not. The result is an open stop loss trade in NFA compliant way. Even though you used to click a single button for this in Multicharts or Sierracharts.

A "Stop-loss trade"?? That's really something new from the world of MT

IB API does not require any "bracket" mechanism - it works with raw orders. It is perfectly ok to submit a stop order with or without an existing position (or other orders to open a position).

I understand the roots of Zorro, but find it difficult to believe that it needs to continue to be confined to the Metatrader model. (with the consequence that all further broker integrations need to dance around these self-imposed restrictions).
If an API allows to place a stop or limit order - let the plug-in place it. It is probably simpler to just do it, than simulate internally.

Or implement BrokerStop() and BrokerLimit(), and/or extend the order() function to allow advanced user to implement any of the huge variety of order types offered by IB.

Freedom! Freedom!

I see my didactic limits, but think this brief discussion can now be concluded.

Without daring to dig again into the details already discussed here - but more generally I believe Zheka is right in the observation that the Zorro philosophy how to see/manage trades vs. orders/positions is (too) heavily tied to the MT approach.
Maybe it requires a rather strategic decision at some point if Zorro is supposed to support "official" markets (official exchanges such as CBOE, NYSE and the like, via brokers such as IB) or "proprietary" markets (market making MT brokers) or both, in full awareness that the requirements/approaches/methods can be significantly different for these different environments.
My personal view is that Zorro needs more adaption to IB type brokers if you want it to compete with alternatives such as Ninja Trader, Tradestation, etc. - even if (or just because?) Zorro's technical capabilities are already way superior. You recently made a major step into this direction by adding options and futures support. I think it would be more than consequent to do the natural next step with a more powerful IB plugin using the most common 80% of its wealth of additional features. I understand the additional efforts - and I could even understand if this would require a (reasonably priced) "ZorroS+" version...

I am not sure what an MT approach is, but if you mean the concept of trades, it has a simple reason - most strategy scripts are much easier to write with trades than with orders and positions. Even though 80% of Zorro strategy jobs at the moment use brokers with order/position APIs.

Yes, the concept of a "trade" as a logical entity is what I meant by "MT approach". Maybe there is more behind the scenes. Trades as logical entities do not exist in the world of official exchanges. The strategic question to answer is if Zorro internally needs this concept by all means (just for historical reasons or even better reasons?) or not. This is your decision as developers, of course. Just trying to explain from an IB trader's perspective where the conceptual challenge is. Eventually there is room for compromise...

Would a user want the ability to place explicit orders or set an absolute net position (long/short) for the current Asset?

That was exactly my perspective.
Zorro allows to easily research and prototype complex portfolio multi-asset strategies AND their execution with Options!
These can only be reliably traded in production with the likes of IB (CQG,TT,Rithmic) and the possibilities they offer.

I think the concept of a "trade" is needed internally - to keep separate track of positions for different algos/instruments.
Most probably nothing needs to be conceptually changed in user-exposed trade entry/exit functions.
But current limitation on order types submitted in a plugin seems to be driven by Zorro's self-implementation of NFA.

I wonder, what is the final result of the discussion on speeding up historical data download from IB? Has the download rate been increased? Is there some specific command an enduser has to invoke to speed up historical data download from IB?

It has been speeded up, and there is mention of this in the 'What's New'.

It still however downloads data appr.2x+ slower than how MC/SC do it. But now it is bearable.

One more thing that people might not recognize initially is that by default and by design Zorro does not capture all the ticks streamed by the datasource
(it does not 'subscribe' to data, but 'requests data updates' at specified intervals).

This can be regulated by TickTime, but the consequence of this is that range of bars in RT will be smaller/close of bars will be different from historical bars, and therefore there might be differences between signals/trades in RT and those obtained after downloading historical data and retesting (for verification).

I see! Very interesting.
One thing I noticed when downloading minute data is that different assets receive a different number of minutes even though the requested time period is exactly the same! (The difference is not just a few minutes, but thousands of minutes in some cases.) This seems weird.
Could one fix that by setting e.g.

TickTime=1;

for one minute bars, as you suggest?

No, TickTime regulates real-time(RT); it is frequency of updates is ms. Read the manual.

This difference you mention is with what plug-in and which assets?

I see, Thanks for the clarification!

I get this behavior when I connect to the IB API and run this script:

where "AssetsZ9.csv" contains the assets of Z9 strategy. As Zorro works on downloading the different assets, it reports the number of bars read in each case. (Each bar is one minute for this M1 setting.) And the number of bars read seems to differ for each asset, sometimes by thousands of minutes.

OK, it is now clear why the difference in minutes read comes up:



Apparently, the code assumes the data to be forex, and gets garbage values outside of market hours... Guess M1 is not a correct format for ETFs. Well, we obviously do get M1 data polluted by random garbage outside of market hours. Clearly, one can truncate that data stream to the desired hours in a day... it is a bit bothersome though.