Memory limit too low for tick data?

Hello, I'm getting Error 060: TIC memory limit exceeded. Citation from manual:

"Zorro can not allocate a too large memory resource - such as the ticks buffer for a simulation in TICKS mode, or the price buffer for a large simulation period and small bar period. The script can not be executed and Zorro must be restarted. This happens when the PC is either running low on memory (a Win32 process has 3 GB max.), when a too-large single block of memory is allocated, or when the memory is fragmented through many previous simulation or training runs. In the latter case the fragmented memory can be released simply by re-starting Zorro. Otherwise, memory can be saved by splitting the portfolio into separate strategies, reducing the simulation period (f.i. by setting a EndDate or MaxBars limit), not using the TICKS flag, and using M1 rather than T1 price data."

I'm trying to backtest only one year of tick data for one symbol with no optimization. I have Win7 64bit with 12GB in VirtualBox so the PC memory size is not a problem. The .t1 file has 328 MB. The original .csv file has 0.91 prices per second which is not that much for tick data.

The memory is not full or fragmented. I can't split the portfolio as it is only one symbol and one strategy. Setting MaxBars, EndDate or turning off TICKS flag doesn't help either - I guess it's because the whole .t1 file is loaded at once.

Using M1 instead of T1 is not an option for me. I've noticed that when doing so, the results can be quite different (I'm testing a strategy with thousands of trades per year - even small details matter).

The ability to test tick data was one of the main reasons I bought Zorro in the first place. So my question is: Is there any way to get around this problem? Is it maybe possible to increase the memory limit somehow? Thank you in advance for the reply. Cheers!

Exact same question here. I can't get lower than 2s Ticks until running into the memory issue. Limiting the backtest to one month doesn't help either.

By the way, what is

Error 057: not enough price ticks

?

Backtest is running after this Error without problems.

1 year of 1 price per second tick data is 60*60*24*252*16 = about 350 MB. So that should be normally not a problem. I suppose that the running out of memory has a different reason, like using more memory for something else. Can you post your script, or contact Support? They'll check.

The minimal example would be quite simple:



This is for the FXCM data (EUR/USD for 2015 is the largest file).

The exact error message is: "Error 060: TIC memory limit exceeded (788 MB)"

788 is twice the file size.

Indeed! That error message is in fact wrong, the memory did not run out. I suppose that this is a bug in the memory size calculation. I've notified the developer, and we'll upload later today a new beta version 1.45.2 where this is fixed.

I've downloaded the new beta version and it works fine, thank you!

Another similar problem - I get the same error when trying to backtest two years of data for a symbol if the sum of file sizes is cca 500 MB+.

Minimal example for FXCM data:



Again - trying in VirtualBox with Win7 64bit and 12+ GB RAM.

Yes, this error is indeed intentional for limiting .t1 data to one year. For two years you must run two separate tests. Reason is that the current version loads all ticks into memory. It's planned to cache the data in a future version and then there's no limit to .t1 data - but with the current version, one year is max.

I'm running again into the memory issue with my own collected data. If I want to store every price tick the .t1 file can have over 600 MB. In that case it's not even possible to backtest one year for one asset without getting Error 60 despite the fact that the system has 9 GB free memory. It's quite inconvenient to split the date ranges further. Is it possible to increase the limit somehow?

I guess you won't use it with a 32 bit app.

Still, in this case this shouldn't be a problem. A file causing the error has 599 MB (the error says TIC memory limit exceeded 799 MB). I can load slightly smaller file which has 572 MB. During the backtest Zorro uses only 781 MB of memory, so there's still lot of potential space.

A side note: Zorro is almost 5 years old. As I see it the market generated data volume has almost doubled since that time. It will probably grow in the future too. Something has to be done to keep up with this development.

The Zorro memory limit is 3 GB per instance, and 1.2 GB per single allocation, due to the 32 bit limit.

So why is it possible to perform backtest on 572 MB history file with Zorro taking up 781 MB of memory and 599 MB history file gives an error? It should be still in the limit, shouldn't it? And is there any chance there will be 64 bit version in the future?

599 MB is no problem. The limit is 3000 MB. So there must be a lot more allocated when you get an error message. You can see the current memory consumption in the Windows Task Manager.

A 64 bit version is possible, but would require a different compiler. The lite-C compiler generates 32 bit code.

It must be a bug then. Maybe again an error in memory size computation? See the screenshots with minimal example and file and memory information:

This runs OK:


This gives an error every time:

I see no screenshots, but a bug is always possible - however in this case you probably had to ask Bill Gates to fix it. Memory allocation is done by Windows, not by Zorro.

There's also another possibility: Memory is still free, but not contiguous, so huge arrays can not be allocated. That's the reason for the 1200 MB allocation limit. The limit can be smaller when your memory is heavily fragmented. I have not experienced any problems so far with large tick data files, but maybe I was just lucky.

That's weird, I can see them in different browsers. What about these links?

https://s29.postimg.org/h9tk578af/limit_ok.png
https://s29.postimg.org/qlv1ww0vb/limit_exceeded.png

With highest probability the memory not being contiguous is not the issue here. It happens every time, even after a fresh system start.

What I meant is whether it cannot be related to the issue described in the beginning of the thread - to quote you:



So you are indeed able to backtest on history data files larger than 600 MB with Zorro 1.52?

I can see the screenshots now, but they contain no info about the system memory allocation. There's a performance tab in the task manager where you can see that.

I can start your example script as often as I want and get no memory problem. However the 2015 EURUSD.t1 history is less than 600 MB. I have no >600 MB t1 history at this point - must download it first. If you're tight on memory, you can save some with the LEAN flag.

True. If I check the largest free block of memory available to Zorro it has 2 GB atm. Yet the script crashes with Error 060: TIC memory limit exceeded (799 MB). I uploaded the .t1 file if you want to give it a try: https://we.tl/aLaTBOnxeq

Thanks! I can confirm the problem with your .t1 file. It happens on my PC too and looks like a bad allocation order when Zorro loads the historical data - as if a new area is allocated before the old one is freed. This exceeds the limit every time. This can be probably fixed and we'll do that with high priority before the next release.

I've found another issue and I don't know whether it's related to the previous one. Hopefully you'll be able to reproduce. But description first:

I have a .t1 file which is only 1.05 MB in size. I can view the prices in ZHistoryEditor. But when I try to export the prices in Zorro it crashes with an Error 060: RES memory limit exceeded (1034 MB). It's the Error 060 as in the previous case but this time it's not TIC memory but RES memory. Then I change StartDate to a later date and I get Error 060: BAR memory limit exceeded (3345 MB). That's when I'm trying to use 100ms or 1s BarPeriod. If I change it to 1m the script runs without an error.

So here's the code:


And here's the file (don't mind it's not actually AUD/CAD prices in there ) https://we.tl/aCOlqt40zM

Maybe the last described problem could be also caused because there's one tick from different year than 2013. I didn't see it in ZHistoryEditor because it doesn't show all records in the file.

I haven't tried it, but don't think that a 32 bit program can handle one year of 0.1-second bars. Neither can probably a 64 bit program.

jcl: What I meant was that it was weird that this happened with 1 MB history file. Is it because Zorro pre-allocates memory based on StartDate and EndDate?

Yes. If you only have small history, set the start and end date to its start and end. Bad things happen when no history is available in parts of your test period.