Gamestudio Links
Zorro Links
Newest Posts
Blobsculptor tools and objects download here
by NeoDumont. 03/28/24 03:01
Issue with Multi-Core WFO Training
by aliswee. 03/24/24 20:20
Why Zorro supports up to 72 cores?
by Edgar_Herrera. 03/23/24 21:41
Zorro Trader GPT
by TipmyPip. 03/06/24 09:27
VSCode instead of SED
by 3run. 03/01/24 19:06
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
3 registered members (AndrewAMD, Quad, TipmyPip), 865 guests, and 6 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
Feeding data to AI #469241
11/09/17 15:17
11/09/17 15:17
Joined: Aug 2017
Posts: 58
P
pascalx Offline OP
Junior Member
pascalx  Offline OP
Junior Member
P

Joined: Aug 2017
Posts: 58
I am currently trying to figure out how to feed data from Zorro to an AI. I am wondering if the Zorro tools can perform the job or if a custom solution is necessary.

The AI requires whole data sets. Preferably D1 starting from year 2000 or earlier. The AI uses a genetic algorithm and has to train every day for different assets. Genetic populations are randomly generated beginning from the earliest timepoint in a data set. It makes no sense to train with short time spans. The AI must train 24/7 and thus needs access to the whole data set of each asset at all times, each day. For backtesting this obviously should be simplified to just train once against a data set, and then trade the AI predictions for any timepoint on the zorro backtester. In Live Trading, the AI needs new datasets each day, and then trade the last prediction for any asset on the new day.

So from what I have seen Zorro provides 3 tools for history data.
1. Series
2. assetHistory
3. dataDownload

Did I miss one?

Series seem to be of no good use for live trading, because the LookBack is capped at one year. In back testing it is somewhat usable, because with the set(PEEK) mode you can collect the price data at the start, pass to the AI, and then trade the predictions. This will not work for marketVol and marketVal data however, because data for BarPeriod > 1 cannot be correctly collected.
http://www.zorro-trader.com/manual/en/series.htm
http://www.zorro-trader.com/manual/en/lookback.htm
http://www.zorro-trader.com/manual/en/mode.htm

assetHistory() downloads data from the web or pulls it through the broker plugin. The manual says "MT4 servers usually have no long price history" which means broker data is not good enough for the AI. It also says "Price history is only downloaded in the INITRUN. For downloading data in the middle of a trading session or simulation, use dataDownload." This means it is not applicable for live trading. Also it appears to store all data in encrypted formats, so we cannot read the file manually. We probably still need it for back testing.
http://zorro-trader.com/manual/en/loadhistory.htm

dataDownload() downloads data from the web in CSV format. Can be done any time. This appears to be the zorro way to obtain a whole data set. Can be used in Live and Backtesting. Unfortunately there is no guarantee that it can provide what we need, nor is there a standard way to ask for what you need. Symbol names can differ between broker and online data provider, so it appears a symbol mapping has to be maintained. There is no guarantee that the downloaded data somewhat matches the broker data feed for live trading.
http://zorro-trader.com/manual/en/data.htm

I tried to use dataDownload() but haven't succeeded yet. GOOGLE yields in 404, AV yields missing apikey, QUANDL unrecognized symbol. I probably need to look up the asset names for quandl.

Given my lack of experience with dealing with asset data for trading, I wonder what an experienced trader approach to feed the AI would look like.

Re: Feeding data to AI [Re: pascalx] #469246
11/09/17 16:14
11/09/17 16:14
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
If you need a 17 years D1 data file in CSV format for training every day, then either use dataDownload, or download it directly with http functions.

There are not many data sources that provide 17 years history. Certainly not AV or STOOQ. You might have luck with Google or Quandl. And yes, for this you must know the name of your asset by the data source. - I wonder what AI algo would use 17 years data.

Re: Feeding data to AI [Re: jcl] #469249
11/09/17 17:40
11/09/17 17:40
Joined: Aug 2017
Posts: 58
P
pascalx Offline OP
Junior Member
pascalx  Offline OP
Junior Member
P

Joined: Aug 2017
Posts: 58
Can Zorro backtest with CSV files? If not, is there information on how to create zorro data set files? The files look encrypted or packed.

Re: Feeding data to AI [Re: pascalx] #469278
11/10/17 08:26
11/10/17 08:26
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Yes. The file format is described under "File format" and the conversion of CSV to a dataset under "Dataset".

Re: Feeding data to AI [Re: jcl] #469289
11/10/17 11:17
11/10/17 11:17
Joined: Aug 2017
Posts: 58
P
pascalx Offline OP
Junior Member
pascalx  Offline OP
Junior Member
P

Joined: Aug 2017
Posts: 58
Ok excellent. I will have a go at this.

Btw I was wondering if the dataDownload() function really only supports the modes that are listed in the manual. assetHistory() has listed more options.

dataDownload:
Quote:
Mode
FROM_GOOGLE for downloading a time series from Google™
FROM_GOOGLE|1 for downloading only the last records, for live trading
FROM_QUANDL for downloading a time series from Quandl™
FROM_QUANDL|1 for downloading only the most recent record, for live trading
FROM_QTABLE for downloading a Quandl™ data table


assetHistory:
Quote:
Mode
FROM_GOOGLE - download daily (D1) price data from Google™ Finance.
FROM_QUANDL - download daily (D1) price data from Quandl™ (Zorro S required).
FROM_AV - download daily (D1) price data from AlphaVantage™.
FROM_STOOQ - download daily (D1) price data from Stooq™.
FROM_YAHOO - download daily (D1) price data from Yahoo™ Finance (currently out of service).
+UNADJUSTED - use unadjusted prices.
1 - download unadjusted one-minute (M1) price data from the selected broker.
0 - download unadjusted tick quote (T1) data from the selected broker (Zorro S required).


http://zorro-trader.com/manual/en/loadhistory.htm
http://zorro-trader.com/manual/en/data.htm

Is this correct?

Re: Feeding data to AI [Re: pascalx] #469290
11/10/17 11:35
11/10/17 11:35
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Yes, but assetHistory does in fact also download a temporary csv file. It is History\history.csv.

Re: Feeding data to AI [Re: jcl] #469295
11/10/17 15:17
11/10/17 15:17
Joined: Aug 2017
Posts: 58
P
pascalx Offline OP
Junior Member
pascalx  Offline OP
Junior Member
P

Joined: Aug 2017
Posts: 58
Does Zorro.exe use curl and ever call curl_global_init / curl_global_cleanup ?

Re: Feeding data to AI [Re: pascalx] #469297
11/10/17 16:19
11/10/17 16:19
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
I cannot answer the last part of the question since I'm not familiar with the curl API, but yes, Zorro uses curl.


Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1