iVolatility options csv file to .t8 Zorro file

Posted By: SBGuy

iVolatility options csv file to .t8 Zorro file - 06/12/18 22:24

So, I decided to buy some real options data from iVolatility for my options backtesting.

However, there's no Zorro script for converting CSV to .t8 file. Anyone care to share a few lines of code on how I might do this?

CSVtoHistory only produces .t6 files.

Thanks.
Posted By: AndrewAMD

Re: iVolatility options csv file to .t8 Zorro file - 06/13/18 00:32

The manual is your friend! laugh

You can parse CSV to t8:
http://zorro-project.com/manual/en/data.htm

CSVtoHistory is a great template. So modify it to suit your needs.

Other resources:
http://zorro-project.com/manual/en/file_.htm
http://zorro-project.com/manual/en/str_.htm
Posted By: SBGuy

Re: iVolatility options csv file to .t8 Zorro file - 06/13/18 23:05

Well, I have been going to my friendly manual before posting for help. However, it is not being cooperative, or I'm still very rusty with coding.

Anyhow, I successfully made a T8 file, verified with History.c, but only after manually changing the format of the Expiration Date field in the CSV file using Excel. Trying to be less of a hacker, I would like to massage that field in my Zorro script.

The problem is that the Expiration Date string appears as 3/24/2017, where the month and day is sometimes 1 or 2 chars. I need to transpose this into an 8 digit long - YYYYMMDD, which is what the t8 file format requires.

I'm been hacking with strdate, wdatef, ymd, strf, etc. all day and can't seem to get to YYYYMMDD.

How would you do this Master Andrew? I'm sure you probably have a 1 liner for this problem :-)

Many thanks!

Posted By: SBGuy

Re: iVolatility options csv file to .t8 Zorro file - 06/13/18 23:27

Wait.... Here is the solution!

ymd(wdatef("%m/%d/%Y",MyDate)

I'm going to post the full script when I'm done. No one should have to spend this much time writing a damn CSV import script, when whe should be spending time making money! :-)
Posted By: AndrewAMD

Re: iVolatility options csv file to .t8 Zorro file - 06/13/18 23:31

Yay! laugh
Posted By: SBGuy

Re: iVolatility options csv file to .t8 Zorro file - 06/15/18 16:07

I had some help from the guys at oP and they will post the final script on the manual. There were a lot of nuances that were not well documented when it comes to .t8 files.
Posted By: Phrendo

Re: iVolatility options csv file to .t8 Zorro file - 10/16/19 20:59

SBguy, I've looked all over for this script you worked on. I have the same conversion from IVol to tackle and can't seem to get it right. Did you end up posting this script anywhere?
Posted By: SBGuy

Re: iVolatility options csv file to .t8 Zorro file - 10/30/19 22:06

Sorry about the delay in response. Zorro posted it to their manual a while ago. It's at the bottom of the contract manual page.

https://zorro-project.com/manual/en/contract.htm

Code
// Example script for converting EOD options data to .t8:
// Format: underlying symbol, exchange, date MMDDYYYY, adj close, option symbol, expiry MMDDYYYY, strike, Call/Put, American/European, ask, bid, volume, open interest, close
// Sample: "TLT,NYSEArca,04/10/2015,129.62,TLT   150410C00112500,04/10/2015,112.5,C,A,17.3,16.2,0,0,129.62"
string Format = ",,%m/%d/%Y,,,i,f,s,s,f,f,f,f,f";

void main() 
{
// first step: parse the CSV file into a dataset
  int Records = dataParse(1,Format,FILENAME);
  printf("\n%d Records parsed",Records);
// second step: convert the raw data to the final CONTRACT format
  for(i=0; i<Records; i++,c++) 
  {
    CONTRACT* C = dataAppendRow(2,9);
    C->time = dataVar(1,i,0);
    string PC = dataStr(1,i,3);
    string EA = dataStr(1,i,4);
    C->Type = ifelse(*PC == 'P',PUT,CALL) + ifelse(*EA == 'E',EUROPEAN,0);
    int Expiry = dataInt(1,i,1); 
    C->Expiry = 10000*(Expiry%10000) + Expiry/10000; // MMDDYYYY -> YYYYMMDD
    C->fStrike = dataVar(1,i,2);
    C->fAsk = dataVar(1,i,5);
    C->fBid = dataVar(1,i,6);
    C->fVol = dataVar(1,i,7);
    C->fVal = dataVar(1,i,8); // open interest
    C->fUnl = dataVar(1,i,9);
    if(!progress(100*i/Records,0)) break; // show a progress bar
  }
  dataSort(2);
  dataSave(2,"History\\MyOptions.t8");
}


Posted By: interzonez

Re: iVolatility options csv file to .t8 Zorro file - 12/18/19 09:24

How to run this script on my CSV file of historical SPX options data?

Where to point to my CSV file or does it run in the same directory somehow?

Thank you.

laugh I see FILENAME - assuming I can swap out here for the CSV file?
Posted By: SBGuy

Re: iVolatility options csv file to .t8 Zorro file - 12/20/19 14:02

Yes. And output t8 is the last line in the script.

You might also need to play with Format string depending on the specific CSV format/order of your incoming data. See the manual on Format. Once you master that, you can import just about any CSV data into Zorro t1, t6, or t8 format.

I believe the latest distro of Zorro includes a variation of this script as an included script now
Posted By: interzonez

Re: iVolatility options csv file to .t8 Zorro file - 12/21/19 03:11

Thanks, yep using the script from the latest distro. Going around and around trying to match the format string to the CSV.

script and one line of my csv. probably I am missing something very simple and fundamental!

It parses 340648 SPX records. Then the error is: Error 013 invalid parameter 0

Date Expiry Under Strike Bid Ask OI Imp. vol fwd price Type
05/28/2019 12/20/2019 SPX C 100 2672.20 2675.80 533 1.696005 2810.23 E

Code

int Year = 2019;
string Ticker = "SPX"; // File name: SPX_2019.csv
string Format = "%m/%d/%Y,i,s,s,f,f,f,f,f,f,s"; // from the sample above

void main() 
{
// first step: parse the CSV file into a dataset
	int Records = dataParse(1,Format,strf("History\\%s_%i.csv",Ticker,Year));
	printf("\n%d %s Records parsed",Records,Ticker);
	if(!Records) return;
// second step: convert the raw data to the final CONTRACT format
	int i;
	for(i=0; i<Records; i++) {
		CONTRACT* O = dataAppendRow(2,9);
		O->time = dataVar(1,i,0);
		string PC = dataStr(1,i,3);
		string EA = dataStr(1,i,4);
		O->Type = ifelse(*PC == 'P',PUT,CALL) + ifelse(*EA == 'E',EUROPEAN,0);
		int Expiry = dataInt(1,i,1); 
		O->Expiry = 10000*(Expiry%10000) + Expiry/10000; // MMDDYYYY -> YYYYMMDD
		O->fStrike = dataVar(1,i,2);
		O->fAsk = dataVar(1,i,5);
		O->fBid = dataVar(1,i,6);
		O->fVol = dataVar(1,i,7);
		O->fVal = dataVar(1,i,8); // open interest
                O->fUnl = dataVar(1,i,9);
		if(!progress(100*i/Records,0)) break; // show a progress bar
	}
	dataSort(2);
	dataSave(2,strf("History\\%s_%1.t8",Ticker,Year));
}
Posted By: interzonez

Re: iVolatility options csv file to .t8 Zorro file - 12/21/19 06:00

Update - I changed the output code to hard code the export file name:

dataSave(2,strf("History\\SPX_2019.t8",Ticker,Year));

I now have the .t8 file though can't interrogate it with ZHistoryEditor

Will try further tests
Posted By: SBGuy

Re: iVolatility options csv file to .t8 Zorro file - 12/21/19 17:14

A few problem areas I can see right off the bat.

1. Your source data doesn't looke like a CSV. I don't see any commas.
2. Your Expiry is 12/20/2019. That's not an Integer expected by format "i"
3. Your 4th element in your data is supposed to be Strike. I see "C"

Date Expiry Under Strike Bid Ask OI Imp. vol fwd price Type
05/28/2019 12/20/2019 SPX C 100 2672.20 2675.80 533 1.696005 2810.23 E

So just go through the source data very carefully and match it up with the Format string.
You might need another step to pre-process your source file to match the Zorro Format string. Getting data into Zorro t8 file is a process, not automatic. The sample script is just a guide.

Or... get another data source with better initial data format.
© 2024 lite-C Forums