Posted By: boatman
Batch processing historical data in CSV format - 07/31/15 09:27
I have a set of historical price data from www.histdata.com which is aligned to the EST time zone (no adjustments for daylight savings). I would like to convert this to .bar files for use in Zorro using the code below, taken from the manual.
I believe that I first need to convert the time-stamps on the data to UTC time. Does anyone know a way to do this via script to prevent having to do it manually for hundreds of CSV files?
The data is semi-colon delimited of the format:
20150501 000200;0.954100;0.954330;0.954070;0.954110;0
So I guess I need some way to add 5 hours to each time stamp.
Thanks for any suggestions!
I believe that I first need to convert the time-stamps on the data to UTC time. Does anyone know a way to do this via script to prevent having to do it manually for hundreds of CSV files?
The data is semi-colon delimited of the format:
20150501 000200;0.954100;0.954330;0.954070;0.954110;0
So I guess I need some way to add 5 hours to each time stamp.
Thanks for any suggestions!
Code:
// import a historic data file from .csv string InName = "Data\\DAT_ASCII_EURUSD_M1_2013.csv"; // name of the CSV file string OutName = "History\\EURUSD_2010x.bar"; string readTick(string content,TICK* tick) { // tokenize a single line char* line = strtok(content,"\n"); if(!line) return 0; // read "20100103 170000;1.430100;1.430400;1.430100;1.430400;0" int Year, Month, Day, Hour, Minute, Second; if(10 != sscanf(line,"%4d%2d%2d %2d%2d%2d;%f;%f;%f;%f;", &Year, &Month, &Day, &Hour, &Minute, &Second, &tick->fOpen, &tick->fHigh, &tick->fLow, &tick->fClose)) return 0; // store the time in DATE format (ConvertTime is defined in Convert.c) // make sure that the time is UTC time!! tick->time = ConvertTime(Year,Month,Day,Hour,Minute,Second); // return pointer to next line return line+strlen(line)+1; } function main() { if(!file_length(InName)) quit("Data file not found!"); // allocate TICK array int maxticks = 60*24*365; // 1 year TICK* ticks = malloc(maxticks*sizeof(TICK)), tick = ticks+maxticks; // read ticks in reverse order string content = file_content(InName); while(content) content = readTick(content,--tick); // store the ticks int size = (int)(ticks+maxticks)-(int)(tick+1); file_write(OutName,tick+1,size); free(ticks); printf("\nRead %d ticks",size/sizeof(TICK));