Gamestudio Links
Zorro Links
Newest Posts
Blobsculptor tools and objects download here
by NeoDumont. 03/28/24 03:01
Issue with Multi-Core WFO Training
by aliswee. 03/24/24 20:20
Why Zorro supports up to 72 cores?
by Edgar_Herrera. 03/23/24 21:41
Zorro Trader GPT
by TipmyPip. 03/06/24 09:27
VSCode instead of SED
by 3run. 03/01/24 19:06
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
3 registered members (VoroneTZ, monk12, Quad), 829 guests, and 4 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rating: 5
Page 1 of 3 1 2 3
Backtesting with T1 data and variable spread #459580
05/26/16 14:00
05/26/16 14:00
Joined: May 2016
Posts: 180
Prague
pcz Offline OP
Member
pcz  Offline OP
Member

Joined: May 2016
Posts: 180
Prague
Hello everyone!

From the beginning I needed to backtest with tick data collected from my own broker that has variable spread. However I wasn't able to find how to do that. It took me several days but I think that I got it right finally. I will write the steps I took so it can serve as a tutorial for other people attempting to do something similar. I also hope that this will eventually help to discover bugs as I don't have much experience in C programming.

1. Obtaining data
For this demonstration I will use Dukascopy tick data downloaded using StrategyQuant Tick Downloader. First I downloaded the data for each currency pair and then I exported them to .csv files, each line in this format: YYYY-MM-DD hh:mm:ss.fff,bid,ask,bidVol,askVol

2. Splitting data
T1 conversion is more complicated than M1 conversion because there is no maximum size of the data set and the physical memory may not be sufficient to read the input file at once. Because I didn't want to end up wit too complicated Zorro conversion script I decided to split the data and reverse the line order using Python script and Linux shell command 'tac' (reversed 'cat' laugh ). The script can be found here: split.py. To use it in Windows you will need something like Cygwin.

3. Conversion
I had to modify the Zorro/Strategy/Convert.c script to accomodate for T1 structs ( ConvertT1.c ). You need to set 'base_path', 'smbl', 'start' and 'end' variables to use it. The input files should be in aforementioned CSV format (produced by split.py) and stored in base_path folder. Their names should be in this form: [SYMBOL]_[YEAR].csv. The important part is that if the SPREAD is defined using #define directive in the ConvertT1.c script, another .t1 file (besides the one with ask prices) is exported. Its name contains 's' at the end of symbol's name and it contains spread values for each tick.

4. Checking the converted data
To make sure that the conversion has been done properly you can use this modified export script: ExportT1.c

5. Simulating variable spread
The final step is using both .t1 files during backtest to simulate variable spread. One of the ways to do so is to include both SYMBOL and SYMBOLs in the backtest. SYMBOLs is an artificial asset used only to get the spread value for the actual asset. When the program loops through SYMBOLs, save the spread value from T1 struct to a global variable. When the program loops through the actual SYMBOL, assign the spread from the global variable to Zorro's Spread variable. And that's it laugh

Re: Backtesting with T1 data and variable spread [Re: pcz] #459625
05/29/16 08:17
05/29/16 08:17
Joined: May 2016
Posts: 180
Prague
pcz Offline OP
Member
pcz  Offline OP
Member

Joined: May 2016
Posts: 180
Prague
This is probably question for Zorro developers...

So...I am setting the spread as described above. However I'm using the tick() and priceClose() to assign it to a global variable first. I have TICKS flag set.

Code:
float spread;
function tick()
{
   // if current asset == artificial spread asset
   if(!strcmp(Asset, ast1s))
      spread = priceClose();
}



From Zorro documentation: In a TMF or tick function, priceClose() returns the last price quote, updated every tick when new price data becomes available.

The prices seem to be OK in general but the spread values are sometimes weird. I double-checked that the values I'm storing in my T1 conversion script are fine. Is there any kind of smoothing or something that could cause this? Here's what I get when exporting the T1 values. I aligned the corresponding times in original and converted series and highlited the discrepancies.





A link to download the spreadsheet: https://we.tl/0ikhO2QXAZ

Re: Backtesting with T1 data and variable spread [Re: pcz] #459696
05/31/16 16:50
05/31/16 16:50
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Zorro automatically fixes outliers. Probably that correction acts here since spread data always has lots of outliers. Set Detrend to NOPRICES for telling Zorro that this asset must not be fixed.

Re: Backtesting with T1 data and variable spread [Re: jcl] #459809
06/07/16 12:33
06/07/16 12:33
Joined: May 2016
Posts: 180
Prague
pcz Offline OP
Member
pcz  Offline OP
Member

Joined: May 2016
Posts: 180
Prague
Thank you for the reply. One last question - how does the selection of the next T1 price work? I mean - obviously you can save more than 10 prices per second to the .t1 file. But only one price per every 100 ms is considered during the backtest. If I export the .t1 data to .csv file I see that the millisecond portion of selected price slowly changes, for example:

hh:mm:ss.x40
hh:mm:ss.x40
hh:mm:ss.x40
hh:mm:ss.x40
hh:mm:ss.x41
hh:mm:ss.x41
hh:mm:ss.x41
hh:mm:ss.x42
hh:mm:ss.x42
hh:mm:ss.x42
etc... (but much slower)

The reason I'm asking this is that I want to trim the data before I convert them to .t1 files so the resulting file is as small as possible. It's especially important with some brokers that have hundreds of prices per second. But there seems to be no clear split (like lets say Zorro would take the last price from each whole hundred ms: .100, .200, ...) - rather it seems that the split shifts slowly, one millisecond at a time. Is that correct?

Re: Backtesting with T1 data and variable spread [Re: pcz] #459813
06/07/16 15:06
06/07/16 15:06
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
As to my knowledge, there is no limit to the number of ticks, only to the number of bars. You can have only 10 bars per second, but any number of ticks.

Re: Backtesting with T1 data and variable spread [Re: jcl] #459819
06/07/16 17:27
06/07/16 17:27
Joined: May 2016
Posts: 180
Prague
pcz Offline OP
Member
pcz  Offline OP
Member

Joined: May 2016
Posts: 180
Prague
Originally Posted By: jcl
As to my knowledge, there is no limit to the number of ticks, only to the number of bars. You can have only 10 bars per second, but any number of ticks.


You are absolutely right. I didn't realize that even with TICKS flag set run() is called on every bar and not every tick. Now, when outputting the prices in tick() function, I see them all with the exact timestamps +/- 1 ms. Thank you! laugh

Re: Backtesting with T1 data and variable spread [Re: pcz] #459859
06/09/16 00:06
06/09/16 00:06
Joined: Apr 2014
Posts: 482
Sydney, Australia
B
boatman Offline
Senior Member
boatman  Offline
Senior Member
B

Joined: Apr 2014
Posts: 482
Sydney, Australia
pcz, thank you for sharing this. Extremely useful!

Re: Backtesting with T1 data and variable spread [Re: pcz] #463654
12/19/16 23:14
12/19/16 23:14
Joined: Jan 2016
Posts: 12
New South Wales
C
CaptainChezza Offline
Newbie
CaptainChezza  Offline
Newbie
C

Joined: Jan 2016
Posts: 12
New South Wales
Hey guys,
is this currently the best way to import tick data for zorro, or have things changed?
If I test with FXCM on zorro it just uses 1m OHLC; if I subscribe and get Zorro S, do I have access to reliable tick data on FXCM, or is it a bit hit and miss?

Thanks

Re: Backtesting with T1 data and variable spread [Re: CaptainChezza] #463658
12/20/16 14:40
12/20/16 14:40
Joined: May 2016
Posts: 180
Prague
pcz Offline OP
Member
pcz  Offline OP
Member

Joined: May 2016
Posts: 180
Prague
You can download FXCM T1 data on Zorro Download page. The way I described is meant only for sources for which there is no possibility to download the data using Zorro or for situations in which it is necessary to backtest with variable spread. The variable spread backtesting can be achieved also with the new T6 struct as it allows for storing additional data besides OHLC. Related links:

http://zorro-trader.com/download.php

http://zorro-project.com/manual/en/export.htm

Last edited by pcz; 12/20/16 14:41.
Re: Backtesting with T1 data and variable spread [Re: pcz] #463666
12/21/16 00:36
12/21/16 00:36
Joined: Jan 2016
Posts: 12
New South Wales
C
CaptainChezza Offline
Newbie
CaptainChezza  Offline
Newbie
C

Joined: Jan 2016
Posts: 12
New South Wales
Is FXCM T1 data that reliable, or is it worth using duka.
Variable spread is handy though.



Futher, I'm definitely a noob with this, but where do you include in the your py script the csv file name?

Also, do you then just drag and drop the py script into the cygwin terminal?

Code:
$ /cygdrive/d/Trading/DukaData/QmYybyXg.py
/cygdrive/d/Trading/DukaData/QmYybyXg.py: line 1: from: command not found
/cygdrive/d/Trading/DukaData/QmYybyXg.py: line 2: import: command not found
/cygdrive/d/Trading/DukaData/QmYybyXg.py: line 3: $'r': command not found
/cygdrive/d/Trading/DukaData/QmYybyXg.py: line 4: syntax error near unexpected t                  oken `sys.argv'
'cygdrive/d/Trading/DukaData/QmYybyXg.py: line 4: `if len(sys.argv) != 2:



I also tried using the CSVtoT6 script as I noticed in that script there has information about converting trade station csv into t1.
So I put my csv file name into that line:
Code:
#ifdef TRADESTATION
#define TCK T1
string InName = "HistoryEURCHF_UTC_Ticks_Bid_2015.02.01_2016.12.19.csv";  // name of the CSV file
string OutName = "Historyticks.t1";



But that didn't work. Is it meant to?
I am using the normal zorro, not zorro s. Do I need to have zorro s for this to function properly?

If I wanted to test T1 data on EURCHF for example can I create a fxcm account, and download it from there?
Or what is a suitable way to pull it from a CSV file from dukascopy? (without worrying about variable spread at this point in time)

Do I simply create a back the front CSV file divided into years, into the format of:
Date, value

Or does your python script do that?

I do apologise for jumping off topic a little (since i'm now just talking about tick imports)
Thanks

Last edited by CaptainChezza; 12/21/16 08:51.
Page 1 of 3 1 2 3

Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1