Saving live data to history

Posted By: Zheka

Saving live data to history - 10/08/17 23:19

I stumbled across Zorro's behavior where the strategy downloads needed lookback data before going live and then trades, but none of the data obtained in real-time is saved to the *.t6 file.

Is this by design? (and if yes - why?
Why not save realtime data on the fly and then let users decide IF they want to re-download it running a separate script?)
Posted By: Dalla

Re: Saving live data to history - 10/09/17 06:48

Good questions, I've thought about this too
Posted By: jcl

Re: Saving live data to history - 10/09/17 07:31

This is by design. Modifying history files must not happen accidentally. But if you want to do it, just put some code for it in your strategy.
Posted By: Zheka

Re: Saving live data to history - 10/09/17 13:45

I would rather not put unneeded code in a live trading strategy. With the current history download speeds, if something happens after a week of trading, getting the system up again will take quite some time. And when testing pre-production, when you have to stop/start a strategy often - this becomes quite a waste of time.
And then there are issues when "retesting". Without recorded data it would be quite difficult to understand if/where things went wrong.
This becomes more pronounced if one uses recursively-calculated indicators like EMA: tiny price differences accumulate and the signals you get on "live" data will be quite different than those on 'history'. Reconciling the differences takes quite some time. And this is especially so with IB, which provides data 'snapshots' in realtime, but full history bars from the historical servers.

As a solution, I started using IB's "True 5-sec RT bars" - which come with a 250-300ms delay, but are clean data, representing all market ticks and coming from their historical servers. This way, there is no difference between "live" and "historical" and "what you trade is what you test/retest on".
https://interactivebrokers.github.io/tws-api/realtime_bars.html#gsc.tab=0
I convinced MC to implement it; Sierra implemented it long ago.

I will work with support to improve IB's history download speeds, but is it
possible to add a Flag to allow saving real time data to history files?
Posted By: jcl

Re: Saving live data to history - 10/09/17 14:39

I'm not convinced that this is a useful feature. Historical data should be generated with a dedicated function, not with live trading.

Aside from many other problems, live data is frequently subject to Internet interruptions and other issues - especially with IB. Data generated this way would be of poor quality and full of gaps. You had to fix it. For this you need code anyway, a lot more code than for collecting it while live trading.
Posted By: Zheka

Re: Saving live data to history - 10/09/17 15:11

It feels like we are talking about two different IBs:-)
I have not had a problem with IB's live data quality(nor order execution) over the last 3yrs. Minor discrepancies between live and historical had been resolved with True RT 5-sec.

People trading algorithmically with IB ($10K account min) from home -and not from a VPS 1-3ms away from IB's servers- probably deserve the trading experience they get.
Why manage to the lowest common denominator?
Give us options!:-)
© 2024 lite-C Forums