|
2 registered members (TedMar, AndrewAMD),
1,033
guests, and 1
spider. |
|
Key:
Admin,
Global Mod,
Mod
|
|
|
Re: price() clarification
[Re: DdlV]
#435301
01/05/14 19:56
01/05/14 19:56
|
acidburn
Unregistered
|
acidburn
Unregistered
|
Thanks acidburn! True - Open is a bit more complicated - something like:
if((Market has been closed) or (Tick time == Bar start time)) Open = Bar's first Tick price;
else Open = last Bar Close;
???
Nope. I still don't like it. I think Open price should bear no connection to the last bar. One bar should be completely independent of the other. And bound only by time, as candlestick chart is time based. So quoting your rules, i think Open only need this one: "Open = Bar's first Tick price"
|
|
|
Re: price() clarification
[Re: ]
#435302
01/05/14 20:05
01/05/14 20:05
|
acidburn
Unregistered
|
acidburn
Unregistered
|
Ah, I think I see where you're leading with last bar's close. In a (very unlikely) situation when there was no trading at all in a 1 minute period, yes, if you want that slot filled you could copy the close price from the last bar to all of OHLC of the current bar. It's the best approximation.
Of course, situations like that should be very rare in todays forex market. But as it is fragmented (not centralized) some less popular brokers could see periods of inactivity. Although 1 minute is a long long time...
|
|
|
Re: price() clarification
[Re: pipclown]
#435305
01/05/14 20:34
01/05/14 20:34
|
acidburn
Unregistered
|
acidburn
Unregistered
|
I put together a Python script that parses and counts wicks in the .bar history files. A wick is defined as high price > open price or low price < close price.
Now, that I had time to run the script and compare results, all I can say is great detective work pipclown! I think there's no need to duplicate the effort, so I won't pursue the same analysis. But... I would like to ask you for a favor, if you can spare a bit more time on your excellent script. I'd do it myself, if I knew python.  I find your formula for a wick just a bit lacking. I think the existence of the upper wick would be better described as high > max(open, close) and analogous the lower wick as low < min(open, close). If you would be so kind to make this little adaptation and run it again, to see if it changes anything? Thanks!
|
|
|
Re: price() clarification
[Re: ]
#435308
01/06/14 02:27
01/06/14 02:27
|
Joined: Jun 2013
Posts: 1,609
DdlV
Serious User
|
Serious User
Joined: Jun 2013
Posts: 1,609
|
Thanks acidburn. I think it comes down to (again!  ) definition: If Bar is defined as the price action of the Ticks within the Bar's time period, then you are correct and a Bar's Open is the price of the first Tick. And there is always a gap from the prior Bar's close, as I understand Ticks only happen when price changes? If Bar is defined as the price action of the time period of the Bar, then: In continuous trading Open is the Close of the previous Bar, since that's the price at the start of the Bar (no gap), unless the Bar's first Tick is exactly at the Bar's start time (gap); and in discontinuous trading (Market was closed, halted, other?) by convention Open is the first Tick's price (gap)? Comments? Thanks!
|
|
|
Re: price() clarification
[Re: GPEngine]
#435310
01/06/14 04:12
01/06/14 04:12
|
Joined: Aug 2013
Posts: 22
pipclown
Newbie
|
Newbie
Joined: Aug 2013
Posts: 22
|
acidburn, yeah, you're right. Fixed it below. GPEngine, I modified the scripts a bit and added your suggestions. It now outputs some more tables with an overview for each asset. Also counts number of duplicate data it finds (bars with identical timestamps) Output from script:
Wick percentages
_Asset | 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
--------------------------------------------------------------------------------
UK100 | - - - - - - - 0.0 0.0 74.1 66.3 62.8
USDCHF | 56.1 64.4 63.3 74.2 78.9 0.0 0.0 0.0 0.0 86.3 84.4 82.4
GBPUSD | 46.7 61.9 68.8 77.9 81.1 0.0 0.0 0.0 0.0 86.4 84.8 84.9
EURCHF | - - - - - - - - - 89.1 61.9 76.2
XAUUSD | - - - - - - - - 0.0 86.1 86.3 87.6
GER30 | - - - - - - - 0.0 0.0 0.0 76.2 72.0
XAGUSD | - - - - - - - - 0.0 80.6 75.9 76.4
USDCAD | 35.8 61.8 65.5 75.3 76.0 54.8 0.0 0.0 0.0 82.4 79.7 69.9
AUDUSD | 31.9 58.8 63.2 73.9 74.4 56.2 0.0 0.0 0.0 86.9 87.2 83.0
US30 | - - - - - - - 0.0 0.0 75.0 70.0 67.2
EURUSD | 41.8 63.0 65.7 0.0 0.0 0.0 0.0 0.0 0.0 91.6 88.7 84.1
USDJPY | 47.1 61.3 61.9 76.3 76.8 0.0 0.0 0.0 0.0 78.0 76.3 87.5
NAS100 | - - - - - - - 0.0 0.0 51.9 48.6 47.6
NZDUSD | 35.2 59.1 66.0 73.6 75.4 57.4 66.0 72.0 74.3 - 83.5 -
SPX500 | - - - - - - - 0.0 0.0 60.5 53.4 51.7
USOil | - - - - - - - 0.0 0.0 68.7 68.4 64.3
Num gaps large than 3 days
_Asset | 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
--------------------------------------------------------------------------------
UK100 | - - - - - - - 5 5 3 3 4
USDCHF | 0 0 0 1 1 0 0 1 0 0 0 0
GBPUSD | 0 0 0 1 1 0 0 1 0 0 0 0
EURCHF | - - - - - - - - - 0 0 0
XAUUSD | - - - - - - - - 0 1 0 1
GER30 | - - - - - - - 3 2 1 2 1
XAGUSD | - - - - - - - - 0 1 0 1
USDCAD | 0 0 0 1 0 0 0 1 0 0 0 0
AUDUSD | 0 0 0 0 0 0 0 1 0 0 0 0
US30 | - - - - - - - 2 1 2 0 1
EURUSD | 0 0 0 0 1 0 0 1 0 0 0 0
USDJPY | 0 0 0 1 1 0 0 1 0 0 0 0
NAS100 | - - - - - - - 4 1 2 0 1
NZDUSD | 0 0 0 0 0 0 0 1 0 - 0 -
SPX500 | - - - - - - - 4 1 2 0 1
USOil | - - - - - - - 2 2 2 1 1
Num duplicates
_Asset | 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
--------------------------------------------------------------------------------
UK100 | - - - - - - - 577 618 629 598 485
USDCHF | 981 1132 1143 1225 1209 1052 1184 1194 1225 1231 1228 1011
GBPUSD | 825 1054 1163 1232 1214 1075 1165 1185 1214 1227 1225 1012
EURCHF | - - - - - - - - - 1229 1181 1089
XAUUSD | - - - - - - - - 1154 1153 1168 965
GER30 | - - - - - - - 629 680 624 679 556
XAGUSD | - - - - - - - - 1059 1136 1121 929
USDCAD | 619 1040 1082 1215 1171 931 1039 1069 1176 1219 1218 986
AUDUSD | 582 1001 1086 1211 1177 1013 1175 1187 1216 1231 1228 1019
US30 | - - - - - - - 821 1025 1076 1037 828
EURUSD | 840 1103 1133 1224 1191 1054 1209 1211 1226 1238 1231 1019
USDJPY | 945 1084 1157 1228 1195 1123 1208 1204 1219 1227 1219 1020
NAS100 | - - - - - - - 514 609 740 648 489
NZDUSD | 496 904 1071 1209 1167 990 1088 1081 1171 - 1229 -
SPX500 | - - - - - - - 746 849 930 834 638
USOil | - - - - - - - 794 1012 1075 1032 799
Start/end dates ok
_Asset | 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
--------------------------------------------------------------------------------
UK100 | - - - - - - - Y/Y Y/Y Y/Y Y/Y Y/N
USDCHF | Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/N
GBPUSD | Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/N
EURCHF | - - - - - - - - - Y/Y Y/Y Y/N
XAUUSD | - - - - - - - - Y/Y Y/Y Y/Y Y/N
GER30 | - - - - - - - Y/Y Y/Y Y/N Y/Y Y/N
XAGUSD | - - - - - - - - Y/Y Y/Y Y/Y Y/N
USDCAD | Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/N
AUDUSD | Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/N
US30 | - - - - - - - N/Y Y/Y Y/Y Y/Y Y/N
EURUSD | Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/N
USDJPY | Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/N
NAS100 | - - - - - - - Y/Y Y/Y Y/Y Y/Y Y/N
NZDUSD | Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y Y/Y - Y/Y -
SPX500 | - - - - - - - Y/Y Y/Y Y/Y Y/Y Y/N
USOil | - - - - - - - N/Y Y/Y Y/Y Y/Y Y/N
[Finished in 290.2s]
While these may be errors in the .bar files, please take it with a pinch of salt. What effect this have in the end I think jcl is the best one to answer. I merely found some discrepancies while fiddling with the .bar files. The start/end look wrong since I use .bar files from Oct 2013 and now it is 2014 Updated script:
import sys
import datetime
from struct import calcsize, unpack_from
from collections import namedtuple
Tick = namedtuple('Tick', ['open', 'close', 'high', 'low', 'time'])
AssetReport = namedtuple('AssetReport', ['year', 'asset', 'wick_percentage', 'start_date_ok', 'end_date_ok', 'num_gaps', 'num_duplicates'])
OLE_TIME_ZERO = datetime.datetime(1899, 12, 30, 0, 0, 0)
YEARS = range(2002, 2013 + 1)
ASSETS = [
'AUDUSD', 'EURUSD', 'EURCHF', 'GBPUSD', 'GER30', 'NAS100',
'NZDUSD', 'SPX500', 'UK100', 'US30', 'USDCAD', 'USDCHF',
'USDJPY', 'USOil', 'XAGUSD', 'XAUUSD'
]
def dt(oledt):
return OLE_TIME_ZERO + datetime.timedelta(days=float(oledt))
def parse_bar_file(filename):
ticks = open(filename, 'rb').read()
format = '<ffffd'
format_size = calcsize(format)
num_ticks = len(ticks) / format_size
ticks = [Tick(*unpack_from(format, ticks, format_size * i)) for i in xrange(num_ticks)]
ticks.reverse()
return ticks
def count_wicks(bars):
has_wick = lambda b: b.high > max(b.open, b.close) or b.low < min(b.open, b.close)
return len([b for b in bars if has_wick(b)])
def verify_starting_date(bars):
return dt(bars[0].time).day in [1, 2, 3, 4, 5]
def verify_end_date(bars):
date = dt(bars[-1].time)
if date.year == datetime.datetime.now().year:
return True
return date.day in [26, 27, 28, 29, 30, 31]
def count_gaps(bars):
num_gaps = 0
for index, bar in enumerate(bars):
if index == 0:
continue
gap = dt(bar.time) - dt(bars[index - 1].time)
if gap > datetime.timedelta(days=3):
num_gaps += 1
return num_gaps
def count_duplicates(bars):
unique_timestamps = set()
num_duplicates = 0
for bar in bars:
if bar.time not in unique_timestamps:
unique_timestamps.add(bar.time)
else:
num_duplicates += 1
return num_duplicates
def generate_reports():
reports = {}
for asset in ASSETS:
print "Generating for", asset, "..."
year_reports = []
for year in YEARS:
try:
bars = parse_bar_file("%s_%d.bar" % (asset, year))
if len(bars) == 0:
raise IOError("No bars in file")
except IOError:
# Some years are missing/empty
year_reports.append(None)
continue
num_wicks = count_wicks(bars)
wick_prc = num_wicks / float(len(bars))
start_date_ok = verify_starting_date(bars)
end_date_ok = verify_end_date(bars)
num_gaps = count_gaps(bars)
num_duplicates = count_duplicates(bars)
year_reports.append(AssetReport(year, asset, wick_prc, start_date_ok, end_date_ok, num_gaps, num_duplicates))
reports[asset] = year_reports
return reports
def print_table(reports, title, output_func):
print title
print "%6s | %s" % ("Asset", " ".join(map(str, YEARS)))
print "-" * 80
for asset, year_reports in reports.items():
sys.stdout.write("%6s | " % (asset,))
for report in year_reports:
if report is None:
sys.stdout.write(" - ")
else:
sys.stdout.write(output_func(report))
sys.stdout.write("\n")
print "\n\n"
def format_date_ok(r):
output = " "
output += "Y" if r.start_date_ok else "N"
output += "/"
output += "Y" if r.end_date_ok else "N"
output += " "
return output
reports = generate_reports()
print_table(reports, "Wick percentages", lambda r: "%4.1f " % (r.wick_percentage * 100, ))
print_table(reports, "Num gaps large than 3 days", lambda r: "%4d " % (r.num_gaps, ))
print_table(reports, "Num duplicates", lambda r: "%4d " % (r.num_duplicates, ))
print_table(reports, "Start/end dates ok", format_date_ok)
Takes about 5 mins to run on my i5.
|
|
|
Re: price() clarification
[Re: DdlV]
#435324
01/06/14 10:57
01/06/14 10:57
|
acidburn
Unregistered
|
acidburn
Unregistered
|
Thanks acidburn. I think it comes down to (again! ) definition: If Bar is defined as the price action of the Ticks within the Bar's time period, then you are correct and a Bar's Open is the price of the first Tick. And there is always a gap from the prior Bar's close, as I understand Ticks only happen when price changes? If Bar is defined as the price action of the time period of the Bar, then: In continuous trading Open is the Close of the previous Bar, since that's the price at the start of the Bar (no gap), unless the Bar's first Tick is exactly at the Bar's start time (gap); and in discontinuous trading (Market was closed, halted, other?) by convention Open is the first Tick's price (gap)? Comments? Thanks! In your first definition you're not right that there's always a gap (prices fluctuate and overlap a lot, 1 minute is a long time with many ticks, etc...). Your second definition is hard to parse, but if we concentrate on the "Open is the Close of the previous bar" part, then we would not have any gaps while the market is open? Yet, gaps do happen. I attach part of the M1 chart from this morning market open. Obviously you have no issue with the weekend gap, but you would have hard time explaining other gaps on the chart with your second definition.
|
|
|
Re: price() clarification
[Re: pipclown]
#435325
01/06/14 11:01
01/06/14 11:01
|
acidburn
Unregistered
|
acidburn
Unregistered
|
acidburn, yeah, you're right. Fixed it below.
Thanks pipclown! As i suspected it doesn't disprove your finding, although percentages should now be more correct. Your script is slowly becoming a standard to check history data.  I think it already deserves it's own topic on the forum and sticky status. Good job!
|
|
|
Re: price() clarification
[Re: jcl]
#435326
01/06/14 11:05
01/06/14 11:05
|
acidburn
Unregistered
|
acidburn
Unregistered
|
New data is indeed available by FXCM, and the problem only appeared in the data files downloaded 2011 or earlier with their old API. We'll download and replace the files in the next days. As to my knowledge, the issue has no practical effect on backtesting normal systems, so it is not a matter of urgency.
Probably the impact to 1h and 4h timeframes is negligible. But it is paramount on 1m and 5m. Don't know how many people look for market inefficiencies on such low timeframe, but I was guilty doing that in the past, and intend in the future. Please just leave a note when updated history files are in place, so we can download and replace the old ones. Thanks!
|
|
|
|
