Working with many entities

Hello, I have a large problem with my game.
I'm making a city building game. All buildings are built with cubes, these are models with a size of 32x32x32 quants. I don't use WED, 'cause I want to build the game area with my own editor ingame (it's a lot easier due to the big amount of cubes). So, here comes the problem. Because of the cube-building system, there are many entities in my game, and it uses an EXTREME amount of memory (the more cubes I put down, the more memory it uses, even more than 600 MB). When it reaches a certain value, the game crashes. And, on top of that, it is also slow because of the many entities.
So, my question is, how can I reduce the amount of memory the game uses? And how can I make it run faster? With segments, loading models only in a certain area around the player? I also noticed that textures with bigger resolution also takes a lot of memory, but I cannot reduce the texture resolution enough. My options are more limited because I don't make my maps in WED, but ingame with my own editor.
Thanks for the answers in advance.

I'll take Minecraft as example: Performance wise it's impossible to use one entity per cube.

Minecraft solves this problem by using chunks (1 chunk = 1 entity = (afaik) 16³ blocks). Also they only create the faces that you can actually see, which is another must-have in order to make the game fast enough to be playable.

But how should I make these chunks?
And how can I render only the faces I can actually see?

This was just an example and an explanation why your approach doesn't work. You'll need some experience and you have to figure out how you can achieve what you want to do.
Combining multiple blocks to a single entity is just a rough guideline.

Also you're not just hiding polygons that cannot be seen, you actually don't even create them.

the idea with chunking is more or less this :

by having a world with a lot of models/entities your
rendering time to render each separate model/entity
increases dramatically .

to try and reduce the rendering calls , you split your
world up in to volumes containing each a certain
amount of data , in this case that data would represent models/ entities , each volume can now
be referred to as a chunk ,if you ever looked at the gs
chunked terrain ,it works on the same principles those chunks store triangles of the large terrain
splitted up into separate pieces ,well its kind off the same concept. .

you need to create single meshes and entitys from a bunch of meshes and entitys , so that in total you reduce
the amount of entities while still having the required mesh geometry , your meshes could be procedural/ dynamic meshes wich you could remove /add faces to, based upon what faces is seen ..

These merged meshes /entities are then stored in chunks wich is a data array ,wich refers to these volumes of entities. .

then you can, based upon a view distance ,load and
unload these chunks of entities/models so that
enough is visible at a time to still keep your rendering time small enough while not looking to cheap .

basicly chunking reduces the amount of rendering calls by physically reducing the amount of entities and models ..

chunk sizes go a bit deeper because you cannot make your chunks to large or to small ..

you would have to research on it or you can take a look at the tust library for kubus wich is basically what I have described in bad english constructed sentences here..

chunking=joining splitting/slicing into chunks

edit: oh and by the way ,there is alot of memory management that goes into chunking a large area ,you need to unload data from memory you dont need
on screen and only load what you need ,usually you load a little a head of time so chunks dont just pop
up into view but seem to be there by the time you can view them , this loading is usually a backround threaded routine so you do not suddenly cause to much overhead on the main application thread .

jb

there is always the alternative that you could use models of larger areas like city blocks instead of single models for each building or you could make
the editor such that the single buildings are added together by the user
on a grid surface with the appropriate limits , and then joined into one piece and plugged into the world grid as larger single meshes ...chunks ,wich you could manage

then there's instancing also ..

here is a theoretical idea for you wich you could try
out and if it works okay it could possibly make things
simpler...

if you are placing these buildings on top of terrain
you could determine the chunk of terrain your entity
are placed upon , then you could alter that chunk
of terrain by merging the entity mesh with the terrain
chunk's mesh ,then get rid of the entity and only
keep track of where that entity is (what terrain chunk it is on)

this way the terrain chunks should manage the added
geometry on it's own ..

The only problem with this idea is ,I dont know how
you would keep your skins correct and you would
actually add more geometry to the terrain chunk
than was initially planned for each chunk upon
creation ..

enough from me ..

jb

I have to add that Acknex capabilities are pretty limited in this area, mainly due to the huge memory footprint of geometry. I already asked, looks like there's no way around this since you cannot modify the per-vertex data in Acknex.

Here's a little example:


8x8x5 chunks with 200 blocks per chunk added. Acknex can't take any more than this. If I add more blocks, random crashes appear, I'll get an oom exception or I end up with bugs that cause so much CPU usage that the framerate drops below 1fps.

welcome to acknex ;(

there is a 64k of indices or vertices limit per entity mesh ,think I read that somewhere in the manual. .

jb

I know but that's not the issue because I'm using chunks ;P

It's simply the used memory which goes up to ~1.8gb until acknex crashes.

Edit: and if my calculation is right, the worst case scenario, if the meshing is done properly, is 24576 vertices per chunk. The index limit cannot be the problem.

So how about this idea:

The Base level is out of a cell/region system, every region/cell leads to a file where all the Entitys are saved in - Position Action Behaviour stuff everything for this Region.

By leaving a Region the old one will be destroyed and the new one will be loaded.

I thought about this idea before, but the game still uses a lot of RAM even on small regions, because of the many entities. Thanks though
So, I'm a beginner in this part of game creation, how can I stick meshes together into one entity (i.e. chunk)?

Im a little bit confused... explain how your models work.

Are they big? Do they have high res textures? How many are they in a level?

They are small, about 32x32x32 quants size. They actually have high-res textures, with a size of 512x512. When I removed the texture completely from one of my cubes, it used a much smaller amount of memory, but removing the texture is not a solution
I tried one of my cubes at 16x16 texture size, but it didn't help much.
The problem is, the big amount of cubes I need in my level. If I build 3 houses, the game starts to lag, and uses about 600 MB of RAM. It's about a few hundred cubes, maybe 200-300 of them.

Have u tried using external Textures?

I have a Level with about 500 Modells in it wich are using 8 HiRes textures, thx to External textures i can add almost unlimited new models without losing fps or stronger memory.

Hires textures = 2048²

I haven't tried it yet.
Do you mean, to not apply the skin in the 3D modeller, but in source code? My models are pretty large because of the textures, my friend models them in 3ds Max.

Oh dear... U have to set this value in the Modeleditor, when creating a new skin and setting the Texture file, hitting the external Button.

I cannot explain that much also there is NO docu in the Handbook...

Hmm... I'll see about it. Thx!

to clear the texture thing up:

There is absolutely NO memory difference when using external textures. But usually it's better to use external textures, because they are easier to change and you can share the same texture between different models.

A texture that is stored internally requires more memory than the same texture externally when the external texture is compressed (pretty much every format other than .bmp or .dds)
Besides that, textures are never loaded uncompressed. Wether you use a 512² 84kb .png-texture or a 512² 1024kb .bmp-texture - it doesn't make any change in the memory usage because the texture is always converted to an uncompressed format (dds) when loaded by the engine.

The ONLY way to use compressed textures is using the .dds format with texture compression because dds is the same format as the textures that are stored in your vram. However, lossless compression is not possible.

"There is absolutely NO memory difference when using external textures. But usually it's better to use external textures, because they are easier to change and you can share the same texture between different models."

And thats essential, only with this u are able to load levels with much entitys fast enough. I think its also important for rendering, thinking about clonig rendering techniques.

I was talking about different models. If you create the same model over and over again it's still just loaded one time.

@ Kartoffel I only mean to mention the 64k limit because of the initial advice given ,it was clear that the op did not know about joining meshes ,therefore I mentioned it ..

@ robo I thought you had trouble rendering all of them, I missed the "out of memory" bit there in your
post ...