EOD data import using dataFromCSV()

I tested a very simple system using 2 external EOD data sets as "indicators":

#include <contract.c>

var ExtInd1() { return dataFromCSV(101,"+%m/%d/%Y,f","Data1.csv",1); }
var ExtInd2() { return dataFromCSV(102,"+%m/%d/%Y,f","Data2.csv",1); }

void run()
{
BarPeriod = 1440;

vars Diff = series(ExtInd1()-ExtInd2());

if(crossUnder(Diff,0))
enterLong();

if(crossOver(Diff,0))
enterShort();
}

Test results were looking too good to be true and I finally realized that Zorro is entering positions at the OPEN of the SAME daily bar the crossovers are signalled based on the external data. However, the external data is from END of day, so effectively Zorro was using indicator data that would not have been known yet in reality, hence the almost perfect results.

Now I wonder if this is just a conceptual misunderstanding of mine or eventually a Zorro bug? Or in other words, how to avoid this from happening? Actually Zorro should open the trade either at the close of the current bar or at the open of the next daily bar... Ok, I could shift the series Diff by 1 bar for the crossOver() condition but this looks like a dirty workaround to me.

Any recommendations how to do this the "correct" way?

If your external data is from end of day, then it has no business being used during that same day anyways. End of day data can possibly only be used for next day, which makes shifting your external data back by one day the only sensible thing to do, and not a "dirty workaround".

If you still like the results after testing it that way, all you'll have to do to reproduce the same behavior in live trading is to trade as the market opens. Pretty much a non-issue.

Thanks. Well, I was at least surprised of this behaviour since EOD data series should be rather common and I did not expect it required such explicit code in my script to avoid peeking into the future.
Maybe it relates to the more general question "which defaults apply when testing and trading with daily bars?". Trades always entering and exiting at the open? And how to change this behaviour? In fact I would still like to use my EOD signal to trade the very same day at the close (in live trading I would then use the indicator data e. g. 3 minutes before the close, of course - yes, it is available by that time). I believe I can achieve this in live trading using BarOffset but any chance to enter/exit trades at the daily bar close in test mode?
There is probably a section in the manual discussing this - I just haven't found it yet. Any hints appreciated.

It is here: http://manual.zorro-project.com/fill.htm

Normally it enters at close plus slippage, fill mode 3 enters at open of next day.

That's what I was looking for, thanks!

However, while playing with this I realized that I may have previously misinterpreted my test data. In fact it rather looks like Zorro (I'm using the latest release build 1.74.8) applies a 1-day lag between the actual data in the .t6 file (loaded from AV, verified with the Z History Editor and confirmed with Yahoo Finance).

Test script:

void run()
{
BarPeriod = 1440;
set(PRELOAD|LOGFILE);
if(is(FIRSTINITRUN)) {
assetList("AssetsIB");
assetHistory("SPY",FROM_AV);
}
printf("nTime: %s, Open: %f, Close: %f",strdate("%Y-%m-%d %H:%M:%S"),priceOpen(),priceClose());
}

logfile:

...
[1299: Fri 18-03-02 00:00] (267.70)
Time: 2018-03-02 00:00:00, Open: 271.410004, Close: 267.700012
[1300: Sat 18-03-03 00:00] (269.08)
Time: 2018-03-03 00:00:00, Open: 265.799988, Close: 269.079987

SPY.t6 in Z History Editor: (png image attached)

Is this a serious bug or am I doing something really stupid??

Well whats the price at midnight 00:00? Its not the future price of next day, its the last price of last day. So all is fine here, otherwise your strategy would be future peeking.

Ok, I could follow that logic if it applied the same way to dataFromCSV(). But there all data is apparently "known" the same day specified by the date in the same row. Referring back to my initial sample script: This script is effectively peeking 1 day into the future due to these two different ways the EOD data is processed. I find this quite counter-intuitive and I bet this is an unnoticed error in many scripts, at least by beginners. It just happened to be very relevant to my trading idea, otherwise I would have never noticed it (before trading it live). I suggest a related note in the dataFromCSV() documentation, at least.

You're right, one must be careful here. When EOD data contains a date, but no time, the timestamp is the begin of the day. But the data is from the end of the day. That's future peeking. Zorro corrects this automatically for EOD price history, but not for data imported from arbitrary files. This must be done by the user. I'll put a warning in the manual and maybe we'll provide a time offset feature on import in a future version.

Sounds good, many thanks.