Hi, new Zorro user here. The software seems quite promising so far, but I'm struggling with how to get the data I want to work with in.

I have a few ideas for options trading strategies that I would like to try out. The historical data that I have already has some of the greeks and other parameters calculated by the source, plus, I've added some of my own calculations. This in total amounts to over a dozen of potential data columns (not necessarily all of which will be useful).

I realize that the Zorro's T8 (CONTRACT) format supports 2 extra fields except for the price data. Would it be possible to load the data consisting of "richer" records?
Could the system be extended with a custom data type used instead of the CONTRACT, so that the classical Zorro backtesting functionality could be used - but with more data available for the decisions (and machine learning)? If so, what would need to be extended?

The CONTRACT struct has a fixed structure. For more data, add datasets and store the data in them. But I would not store greeks. Better calculate the greeks directly.

If using datasets, are you suggesting simply loading the dataset, and (for the backtest) looping through it, and then retrieving individual values via dataVar/dataInt functions?

As for the greeks, my thought was that it may be beneficial to store greeks to speed up the backtest. Furthermore, I already have a dataset containing those, as well as some other pieces of information obtained from the broker, so I was considering testing a few things using those. Can you explain your thoughts why you would prefer to calculate the greeks on the fly?

What would be the appropriate way to backtest while using additional data contained in a data set?

Is it somehow possible to run the backtest directly off the dataset? That is, is it possible to use the contractUpdate on the dataset containing more data columns? In that case, is it correct to presume that the first few fields have to be in the same order as in the T8/CONTRACT struct definition?

If not, would the most appropriate way be to load the full dataset, and save its subset containing the data consisting of the fields from the CONTRACT struct to a T8 file? Then, each invocation of the run function during a backtest, one would need to look up the additional information from the backtest?

I have the exact same problem and questions! IB streams pre-calculated greeks, and I would guestimate that using those is more efficient than calculating locally for a number of strikes.

I too wanted to suggest to add 2-3 more fields for the greeks to the T8/Contract struct definition (or make it extendable by a user) and verify that IB plug-in can also populate those...Backtesting would be more efficient and RT too.

You have no limit for the data. When your options data is in a dataset options.t8, put the greeks and other data in a parallel dataset greeks.t99. To retrieve all greeks that belong to the current options chain, use dataFind() with the current date. It returns the row with the additional data for the first contract of the chain. The other contracts are in the next rows.

We will make this easier in a future version by storing the dataset row number of a found contract.

Thanks jcl, this clears it up somewhat.

Suppose I want to try a simple strategy using the additional data. Suppose I have options.t8 and options_with_additional.t99, where the latter is simply a superset of the former (i.e. the T99 also includes, beside additional data, the data that's already in T8).

I'd first load up both datasets, and update the contract chain with contractUpdate to the T8 dataset.

After that, I need to find the date for which I'm wanting to trade, and get the appropriate rows from the dataset using dataFind and the bar date (wdate?). Then, I'd loop thru the dataset, and figure out which contract(s) I want to trade. With this information, for each contract I want to trade, I'd create the CONTRACT struct, and try to find a matching contract within the chain (contract() function), and enter the appropriate positions.

Would this be a sensible approach?

Alternatively, from what I can figure from your answer, you were suggesting first going thru the contracts, and then use dataFind to use the appropriate greeks. In this approach, it's not clear how would I loop thru all the possible contracts to identify the ones that I want to trade.

I somehow missed on the might of free-form datasets! Got into details and here are the questions:

1. Would "99" in suggested "greeks.t99" typically be a Handle?
2.
What should be the format of records in dataset with a list of CONTRACT structs to be used in ContractUpdate()?

3. You seem to favor using big single files to store all of the options/ greeks data for an asset. Why? Would storing data in many files e.g. 'by expiry' be less efficient in practice?
Or is it because of the file name convention used?

4. Do i understand correctly that *.t8 files would update in real-time but a "dataset" would not?

IB streams greeks and IV together with prices in RT
https://interactivebrokers.github.io/tws-api/option_computations.html#gsc.tab=0

So, why not just add 4-5 fields to the *.t8 struct? (and avoid having to use several datasets, locating matching records,etc)

I also would like to clarify the example on the "Datasets" page in the manual:

The Format is different from the *.t8 struct. How would it map to it?

Hi, Jcl,

Would greatly appreciate your answers!

Best
Z.