Feeding data to AI

I am currently trying to figure out how to feed data from Zorro to an AI. I am wondering if the Zorro tools can perform the job or if a custom solution is necessary.

The AI requires whole data sets. Preferably D1 starting from year 2000 or earlier. The AI uses a genetic algorithm and has to train every day for different assets. Genetic populations are randomly generated beginning from the earliest timepoint in a data set. It makes no sense to train with short time spans. The AI must train 24/7 and thus needs access to the whole data set of each asset at all times, each day. For backtesting this obviously should be simplified to just train once against a data set, and then trade the AI predictions for any timepoint on the zorro backtester. In Live Trading, the AI needs new datasets each day, and then trade the last prediction for any asset on the new day.

So from what I have seen Zorro provides 3 tools for history data.
1. Series
2. assetHistory
3. dataDownload

Did I miss one?

Series seem to be of no good use for live trading, because the LookBack is capped at one year. In back testing it is somewhat usable, because with the set(PEEK) mode you can collect the price data at the start, pass to the AI, and then trade the predictions. This will not work for marketVol and marketVal data however, because data for BarPeriod > 1 cannot be correctly collected.
http://www.zorro-trader.com/manual/en/series.htm
http://www.zorro-trader.com/manual/en/lookback.htm
http://www.zorro-trader.com/manual/en/mode.htm

assetHistory() downloads data from the web or pulls it through the broker plugin. The manual says "MT4 servers usually have no long price history" which means broker data is not good enough for the AI. It also says "Price history is only downloaded in the INITRUN. For downloading data in the middle of a trading session or simulation, use dataDownload." This means it is not applicable for live trading. Also it appears to store all data in encrypted formats, so we cannot read the file manually. We probably still need it for back testing.
http://zorro-trader.com/manual/en/loadhistory.htm

dataDownload() downloads data from the web in CSV format. Can be done any time. This appears to be the zorro way to obtain a whole data set. Can be used in Live and Backtesting. Unfortunately there is no guarantee that it can provide what we need, nor is there a standard way to ask for what you need. Symbol names can differ between broker and online data provider, so it appears a symbol mapping has to be maintained. There is no guarantee that the downloaded data somewhat matches the broker data feed for live trading.
http://zorro-trader.com/manual/en/data.htm

I tried to use dataDownload() but haven't succeeded yet. GOOGLE yields in 404, AV yields missing apikey, QUANDL unrecognized symbol. I probably need to look up the asset names for quandl.

Given my lack of experience with dealing with asset data for trading, I wonder what an experienced trader approach to feed the AI would look like.

If you need a 17 years D1 data file in CSV format for training every day, then either use dataDownload, or download it directly with http functions.

There are not many data sources that provide 17 years history. Certainly not AV or STOOQ. You might have luck with Google or Quandl. And yes, for this you must know the name of your asset by the data source. - I wonder what AI algo would use 17 years data.

Can Zorro backtest with CSV files? If not, is there information on how to create zorro data set files? The files look encrypted or packed.

Yes. The file format is described under "File format" and the conversion of CSV to a dataset under "Dataset".

Ok excellent. I will have a go at this.

Btw I was wondering if the dataDownload() function really only supports the modes that are listed in the manual. assetHistory() has listed more options.

dataDownload:


assetHistory:


http://zorro-trader.com/manual/en/loadhistory.htm
http://zorro-trader.com/manual/en/data.htm

Is this correct?

Yes, but assetHistory does in fact also download a temporary csv file. It is History\history.csv.

Does Zorro.exe use curl and ever call curl_global_init / curl_global_cleanup ?

I cannot answer the last part of the question since I'm not familiar with the curl API, but yes, Zorro uses curl.