How to keep track of Memory usage

Hi everyone,

I know the statistics panel has all the relevant information regarding memory usage but I need some clarifications on some things.

sys_memory : This is the value that shows what the engine has already allocated correct? This is allocated right at the beginning of the engine. This cannot be changed.

max_nexus : this is dependant on how much ram your target computer has correct? This will be a 32bit limit, correct?

d3d_texfree: this is the amount of texture memory available from your graphics card.


So in order to keep track of memory, I was thinking of making a visual bar, that decreases when memory is taken up. What statistics should I add up to make the upper limit value of this bar? I wanted something visual like a bar to show me how much memory Im consuming, as well as keeping it simple and having just a single bar.

So if Im working on a level, and I plop in a huge terrain that has a big texture, I want to be able to see that bar change due to the terrain using memory. I can figure out the bar.

Is the total consumed memory equal to all the stats in the Memory column added up?

also, Im noticing that using ent_purge on a terrain(.hmp) does not fully get rid of the texture memory used by the terrain. Is that normal behavior?

I am not sure. There is a bmap_purge(bmap); function in the manual, but it says it's for C-Script.

Under ent_purge(ENTITY*); it says it releases texture memory, so it probably should not retain texture memory. I am just starting to use these commands myself, and I'm having similar issues where memory continues to pile up as I load new entities...

, I just check task manager for ram usage. Your game should be below 1400/1500 mb iirc (depends a bit per pc).

New entities will always need memory, and if I know the budget, then I can work under it. If you can work under a budget, as well as purging memory, my understanding is that you shouldn't run into memory problems.

But of course it's not that easy. I can create a model using ent_create, see the d3d_texfree decrease. When I ent_purge and ent remove, that entity, I regain that d3d_texfree. This is good. But if I create a terrain, and ent_remove, and ent_purge, the d3d_texfree slowly increases....eventually giving me nexus too small (0) errors! So if I made a game and the player continually loads a new level, this could cause memory not to be cleaned...and eventually crash.

What should I do?

This is just a thought. I am wondering if it matters whether a texture is included in the model, or if it is external.

Also, does it matter whether a texture has been loaded with ent_morphskin?

I think, perhaps, that these external or loaded textures may not be removed by ent_purge.

If anyone knows about this, please inform...

Also, I was never quite sure about using ptr_remove vs ent_remove.

Since ent_remove mentions that it retains the resources in memory, and ptr_remove does not mention this, perhaps ptr_remove actually removes the resources too...

I have discovered something.

If you change an entities skin, use ent_setskin(); instead of ent_morphskin();

With ent_morphskin, it uses a string to load the bitmap. With ent_setskin, it uses a previously declared BMAP* pointer.

The difference is that you can only use bmap_purge on BMAP* pointers, not on strings. This has been my experience anyway.

Currently, I was able to load several entities, purge their entities and bitmaps and load more, and the "d3d_texskins" number stays fixed, it no longer creeps up as I add more entities...

I am still seeing overall RAM usage creeping up though, but not nearly as much as before. I believe there are some objects/entities that I have not yet purged properly, so I need to keep looking through my code...

Anyway, I hope some of this provides the info you're looking for. Let me know if you find out any more about this.

A limitation of ent_setskin is that when you set the skin to a model it will set that skin for every instance of that model, unless you use ent_clone.

ent_morphskin does not have this limitation, and it allows different instances of the model to have different skins.

I just noticed that using ent_create to make a Map Entity will increase the nexus a bit each time it is used, even when I use ent_purge.

By replacing my Map Entity with a Model, I have eliminated this problem. Perhaps there is a similar problem with terrain (.hmp) files? Could you use a model (.mdl) instead?

I can use models I guess. But why should that be a work around? Is every terrain considered a block of memory that cant be fully recovered?

After doing this test: I can create an .mdl file using ent_create, then ent_purge and remove, and i can regain that memory the entity used up. But if I ent_create a terrain, and ent_purge and ent_remove, the texture memory is never fully recovered.

A workaround here is just to have all your terrain placed within WED and never created.

UPDATE:

reloading a level (level_load), over and over again, with or without terrains is causing me to run out of memory

reloading the level is increasing my shadow map video memory consumption, the sha in the statistics panel.

Does this memory stat (sha) relfect the baked shadows in level geometry, or PSSM shadows?

I believe it's baked shadows.
"Amount of video memory currently used for wmb shadow maps." from manual

I am facing similar issues. It seems to be my "d3d_texbmaps" and d3d_texskins" that are stacking up with each level load.

I've created a smaller game which emulates the functions of my game in a hope to find out where the leak is coming from. So far no luck...

Each time I add a function to the smaller game to see how the memory gets handled, and so far the memory is reset on each level load. There is no stacking. However, if I keep adding functionality similar to my (big) game, eventually I'll either have a working game, or I'll discover exactly what's causing the problem.

Level_Load is "supposed" to clear out the memory, but there must be some limitation, or some loophole that allows memory to build up between loads. I will definitely follow up if I find the source of my leak.

UPDATE:
Actually, I can reproduce this memory drain by loading a .wmb file into my level. It increases the overall RAM usage each time I load it, even if I purge and remove it from the level, and even after I load a new level.

Hey Dooley!

I did the same thing. In my main level, I deleted more and more objects, and spammed the level_load on the same level. If i deleted everything except the bare minimum to run my level, no memory was being stacked. After narrowing it down, i found out it was map entities causing the problem!!!!

These map entities (.wmb) are placed in the level to prevent certain nodes from seeing each other. If i had none of these map entities in the level, memory isnt stacked/leaked. But once i placed these .wmbs in the level and reloaded constantly, the memory would stack. After going into the .wmb file itself in WED, i recompiled with no shadows, along with a few other options I dont remember off hand and resaved the .wmb.

Loading the new compiled .wmb into my main level, I was able to reload constnatly without memory stacking.

Question: Why would baked shadows in a map entity (.wmb) cause memory to be unreleased in level_load??

, wouldn't suprise me if it was a bug. But aren't map entities fairly slow compared to models? (I think Sivan tested this once with this map builder) Perhaps its better to just switch to models completely or perhaps bake in lower resolution shadows so the crash is more rare as a (crappy) quick fix for now .

I didnt know they were slower, though it does make sense that they are, being that they have bsp/collision/texture/shadow stuff in them.

Is there a way to see how much total memory an entity takes up? Outside of taking a before and after of the sys memory stat when its created?

Level_Load should be removing any extra memory used by these .wmb files. Is it possible for you to restart/reload the level on some occasions?

I was able to reload/restart the level. But the map_entities definitely stacked, in the shd column!! Which meant something was going on with the shadows. And once I changed the settings of the wmb file individually, the shd never stacked in level reload. I have no idea why

UPDATE:

spoke too soon. WMB map entities placed in the level are not really freed from memory when level_load is performed.

I have found another thing that seems to help. Instead of loading a level directly, and worrying about purging different entities with ent_purge, I've found that loading a blank level seems to clear the entities automatically. Then you can load the level you need next.

Something like this:



On my smaller game, this is not necessary. The engine seems to handle purging memory very well. I'm sure there is something in my code (on the bigger game) that is causing these problems, but I have yet to find it. In the meantime, these techniques seem to keep the memory from stacking...

Did you try spam loading the blank level then the new level and keeping track of sys_memory? That's how I noticed that some levels were stacking memory.

A few hints from the manual -
WMB - Map Entities
Shadow -
Some setting to look at..
++ http://www.conitec.net/beta/ad3d_texsmaps.htm
++ http://www.conitec.net/beta/amap_subents.htm
++ http://www.conitec.net/beta/preload_mode.htm
++ http://www.conitec.net/beta/aent_preload.htm
++ http://www.conitec.net/beta/aent_purge.htm
++ http://www.conitec.net/beta/aentity-dynamic.htm
++ http://www.conitec.net/beta/ashadow_lod.htm
++ other LOD factors.

Sounds like a bug, either level_load or in preload.
No worries... using ent_pugre with a little bit of logic and the CLIPPED flag could be a basic optimization. http://www.conitec.net/beta/ae_clipped.htm



The code and reply are just a guess. Also could be linked to a custom shader setting on map_ents.

I did not "spam load" my level. I'm not 100% sure what that means.

I don't know if you saw my comment about bmap_purge(), but it seems to be a good function that still works (in A7 anyway). It looks like it may have been replaced by ptr_remove, so that might work as well. But you will have to use these on the actual BMAP* image you want removed from memory. So, for instance, if your terrain has a BMAP* pointer for the texture you are using, you can bmap_purge it when you need to remove or change the terrain.

Also, if memory is an issue for you, consider switching to .dds textures if you are not already using them. In one afternoon I eliminated 264 mb of texture memory from my game. I plan to continue switching my textures over to .dds, but this has been a huge gain in my memory issue...

Hey dooley, by spam, I meant just quickly reloading the level constantly and watching if any memory stacks. I have switched over to DDS textures, and it definitely helps with keeping memory low. I wanted to eventually do this, but I guess its better to start now.

In a test level, Im getting a texture memory overflow crash. This is done by creating A LOT of map entities, which increases the nexus. I know I shouldnt create that many entities in a real game, but with a nexus of 500, I wanted to see if I could reach the entity limit, as well as keep track of memory usage. I would like to know why the game would crash this way:

I am avoiding map entities because they seem to stack the memory whenever you load them. I can't answer why.