I am struggeling to read a file line by line. Playing arround with all kind of funtions, some seem not to work?

the data looks like this:




First step would be to split the file into it's lines.
After that i want to split the values of each line line to copy them into a struct.

Now there are c funtions i would like to use e.g. getline() or strsep() but both seems to be missing in litec?
Can i add them somehow manually? Or there other function which will work for sure?
Often examples in c also use the term != EOF for loops regarded to reading files, but this also does not work in litec? (sometimes -1 or NULL seems to work instead.

Iam just very new to all of this and i am so confused, maybe i just do it wrong, but maybe it's all because of lite-c limitations? I would appreciate any help/hint!

Edit: just wondering if DataSet handlign would work for the data i got? With dataParse, ... what do you think? And if so how?

Kind Regards

Isaak

dataParse would be ok, unless you need the full strings in the struct. Then you must parse it by script. Clib functions like getline() are normally in the stdio header - if not, you can add them like the other functions in that header.

Thanks for the reply, does dataParse has a problem with ",,,," if there are no values sometimes?

What format string will i need for the example?

"+%Y/%m/%d,%H:%M,s,%c, ??? ,%i "

is it %s or s for the 3 character string (currency, aka market)?

iam unsure about the strings with different lenght and also containing characters like "[" or "/" or "%", only thing for sure is that ',' is the delimiter. so maybe %[^,] ? What about the lenght?

for the values after the name could i use f? would this automatic skip any characters like %,B,K (non numeric, no '.') like it behaves with i??



And after parsing...it stores the data somewhere, but how to access this data, e.g. if i want to find values from a certain day, oder by name, or by id (last value). No idea yet

I don't know how dataParse will react on irregular or no values - just try it. For generating the format string, follow the suggestions in the manual. You can always check the right format with Verbose = 7 or by exporting the resulting dataset.

To access data you use a Handle. It is just an int variable, any number convenient for you.
Using it in dataParse associates the whole dataset with this variable/number.
So, to access data later, just use this variable in functions like dataFind(),dataVar(),etc..

If format for each type of events is constant, then it might be easier to parse one big file into several datasets, per event, with parses only lines that contain the string Filter. The string is case senstitive and can cover several adjacent fields, including the delimiters. This way only lines with a certain asset name, year number, or other field content are parsed.

I personally did not try it, so I do not know the requirements for 'string Filter' and how it should look like. Experiment.

To search by date - use dataFind, but do dataSort before.
Searching by other fields will be like working with an array, calling dataVar(row,col), dataInt and dataStr. and comparing to the filter value you need.

Ok thanks for your input, i am going to try a bit more later.

Another thing i am confused right now:

i try to import price data with the CSVtoHistory script, (with an older version i think it worked)

The point is the Data is formated like this (and it's a huge file (500mb+):

1986.12.01,01:00,0.90110,0.90110,0.90110,0.90110,4
1986.12.01,01:01,0.90110,0.90110,0.90110,0.90110,4
1986.12.01,01:02,0.90110,0.90110,0.90110,0.90110,4
1986.12.01,01:03,0.90110,0.90110,0.90110,0.90110,4

i am using this format code: string Format = "+%Y.%m.%d,%H:%M,f3,f1,f2,f4,f6";

To notice is that there are no seconds, since it's M1 data

if i start the parsing script i get an error:

Error 058: Bad date 01:01,1.43
(1.43 is the start of the first price for the file i tried in this example)

That look like dataparse wants to have the seconds for sure? How to solve this? to Edit the files is not very easy for me as well, if it would be smaller i could just change format of time in excel, but it's to big file, would need to split it into 10+ files The Point is, it must have worked like this in version 1.72.x if i remember right, but not anymore.

The format string looks syntactically ok. Check if the file has really a header line. Set Verbose = 7 and check the log - there you see the first two parsed lines and can see what's wrong.

THe file has no header, with Verbose = 7 it shows only the same as always:

CSVtoHistory compiling...........

Parse HistoryGBPUSD_1_MT4.csv.......
Error 058: Bad date 01:02,1.43 - can't parse GBPUSD_1_MT4.csv
0 lines read

If it has no header, format begins with +0.

Yeah, noticed that and tested it (also before).


CSVtoHistory compiling...........

Parse HistoryAUDCAD_1_MT4.csv.......
Error 058: Bad date 01:00,0.90 - can't parse AUDCAD_1_MT4.csv
0 lines read


This time i used the file where the above Data sample comes from and added the 0 into the format string.

So it seems data parse really WANT to have seconds? Since it's trying to read it, even without putting it into the format string. Maybe if not in format string it should be 0 by default.

I think I see now the problem. The timestamp has a "," between date and time, but the "," is a delimiter. That's why the parser does not work.

It should be relatively easy to fix that on the parser side. When you can wait for the next beta release, we'll include the fix there.

Ok great, thanks a lot.

Ok it is working now, also the download from Oanda, thank you very much!

Coming to another problem, i just realised that my external data for forex (asirikuy) is using GMT+1/+2 with european dst.

What are my options to work with this data? Can i parse it somehow and store the UTC times? Or do i have to set an offset in my scripts using this data?

Both is possible, but the normal way is converting it. It's a bit tricky because it has not a constant distance to UTC. For determining if a time T is DST or not, use

Now = T;
if(dst(NOW)) ...

Just to understand it, does Zorro uses timestamps of the historic data always as UTC? Could i work with offsets in this case? (1 or 2h depending on dst) to get the "real" UTC time from my data? Or is it best to convert it?

The problem seems to be be that the data back in old history is shifted even more, so somewhere between there was a change.

How long is a trade week? Always 5*24h (in forex) or did it change over the year(s)? If so it would be possible to find the shift (by looking at first and last bar of a week)

Yes, timestamps are supposed to be UTC. So you must correct them if they are local time. It does not matter much for daily data, but it matters for intraday data.

Use the CME forex futures trading hours for reference.
The CFD from FXCM and other brokers align to the CME hours
Forex hours use DST.

But how to get the historic trading hours of the CME forex future?

The problem is that not everywhere in the World they have DST (Japan) or it's slightly different (US compared to EU). So if you need to be sure where in the timeline you are ist very complex and a big mess.

As a simple example .. market open times of all different markets at any time at least since 2002.

First there was a rule change back in 2007 for US DST. Then Japan has no DST. Australia is on the southern hemisphere. So many factors you have to think of. I never found a working indicator e.g. in mt4 who tought about all this and had no mistakes! It seems Zorro so far has the most advanced DST handling.

But stil i somehow have to get my CET backtest data into UTC, even though the data seems not to be 100% consistent with timezone, i think older data is different. My idea is to analyse it week by week, checking if there is data for 5*24h, if so shift itthat it starts 23:00 sunday, but sometimes there is 1h missing, have to see if it's always the first after the weekend then it would be ok (but still not nice that it's missing).

Biggest problem for me is that i am still a big noob in c litec, handling with files for me is very difficult, never did that before at all.

So you should use the Chicago DST.

That may not switch on and off in perfect sync with the CET DST. You are right here. That's the only point I have not verified.

To recap:
- if you want 'official' daily bars use the Futures FX opening time.

- best and simplest: download directly daily bars (already aggregated) from FXCM and use them as a separate time-series to work with your intraday bars.

- if you want to aggregate with Zorro, it's slightly more complex and the
results will not as perfect. Use CET 23H00 as the end of the day and aggregated with TimeFrame as shown in the example.

- usually around midnite CET volatility is low and differences are not huge.

Ok i figured out that my Data is 100% GMT+1/+2 (central european time)

This was made sure by checking for the regular NFP news, it's easy to find/visible and always at the same time.

So now knowing this i want to convert my data to UTC for Zorro.

How can i add/subtract 1 hour from a time? I also don't really understand Now and NOW why do i need this? can't find any examples how to use this.. (i need clear examples to understand/copy something i haven't done yet, even though its basic)

is this fixed now? didn't worked on this data last weeks

But others trying to load data from Tradestation have now same problem.. the missing seconds leading to an dataParse error