libAsset - A library for loading assets in the background

libAsset contains code for loading assets, like models, sounds etc, in a background thread without having you to fear about thread safety and such things.
The idea comes from this feature request from Jibb and his statement that A8 lacks of good asset preloading. The code has to purposes;
1) It saves you time waiting for this feature in A8.
2) It shows that one can easily build a bit more complex stuff in A8 and that it really isn't that limited as some people want it to be.


Usually you only need to call the wrapper functions for creating things. They all start with "la_" and then follows the usual Gamestudio name. So ent_create with background loading functionality would be la_ent_create.
Likewise, snd_create is la_snd_create.

If you want to load other assets or get some more info about the assets state, you can either use the high level int la_checkFile(STRING *file); function which returns 0 if the asset is currently being loaded, 1 if its loaded and -1 and -2 if there was a failure.
If this isn't enough for you, you have to dig a bit deeper, getting the la_asset struct from la_asset *la_assetForFile(char *file, char *filepath); and polling the la_asset struct repeatedly. You then also have to care about locking the loader, but usually, you never ever have to do this!

When you want the loader to reload a model, you first have to remove it. This is done via void la_assetRemove(char *file);. This function also takes care about assets that are currently being loaded in the background, so always use this function to remove assets!

The next useful function is int la_assetRemoveAll();. You MUST call this before calling add_new() because the loader works internally with add_buffer().
The drawback of this function is that it might does nothing when there are models loaded currently in the background. The function will then return 0, so you have to call it multiple times until it returns 0.
The simplified way is la_clear(), this will call la_assetRemoveAll() until everything is removed.


And finally, here is the download link with demo:
http://cl.ly/6wN1


Compatibility notes: Should work with A7 and A8, tested with A8.03 free.

It doesn't work.

Create three 4096x4096x24bit BMP textures a.bmp, b.bmp, c.bmp and assign the first two as new two skins to the marine.mdl and the third as new skin to the stone.mdl. Each texture is ~49MB big, so if they were streamed in the background, the boxes should appear first (since they are small in size), then the stone.mdl (approx. half as big as the marine.mdl then) and at last the marine.mdl because it's HUGE (~100 MB to load in one frame).

What happens if I run your example?

All entities are created ONCE in ONE frame. So, no multithreading, no streaming. It doesn't work as it should be.

Not really. I see why this happens for you, but your conclusion is wrong.
You get multithreading, the problem is the worker thread which owns the mutex for too long because of its priority. So the application flow is interrupted because the worker thread owns the mutex while in the meantime the model loading happens.
What you see is a loading in one frame.

I have uploaded a new version which replaces the old one. The only thing changed is the priority of the thread.

Btw, thinking that larger files are loaded slower than smaller files is also not true, assuming that you have a real spinning disk and not a SSD in your computer. Plus: The internal file reading scheduling of your OS might also alter the result.

Sid, I know that in a multithreaded environment some threads can work faster even if they have a greater workload than other threads due to the OS scheduler switching.

What I really want is that my Lite-C app runs regular for about x frames until a file is loaded. So, if it takes 2 seconds to load e.g. a model (because it has HUGE texture maps, like in the example I described above), I --DON'T-- want that the Lite-C app blocks for that amount of time, but that it runs e.g. still with 60fps for these 2 seconds and then I load the model, because it is finished streaming into memory.

So how am I supposed to run my game and load the model in the background without interrupting the main thread? I mean, should'nt that be the purpose of such a streaming feature? I assumed that your wrapper functions create the entity when it is finished, even if it is e.g. 120 frames ahead in time.

The wrapper functions create the entity when finished.. though you of course don't have to use the la_ent_create but use la_checkFile.

This will not directly create the object but only load the file in a seperate thread.

Sids code uses all the performance the thread gives, therefore the loading process will ~ takes the same amount of performance compared to opening med with the same model in the background...

Imo the most probable way to achieve what heelx wants is to read the file in blocks (e.g. not more than 64kb), and sleep the thread afterwards for a predefined amount of time. This will slow down the loading process and should normally keep the framerate to a maximum.

Okay, lets all bash HeelX for his stupidity:
So, our beloved HeelX did the following: He created three large BMP textures, 50 mb each, and added them to the model.
Then he loaded the model via la_ent_create() and wondered why it blocked the main thread. Turns out, he declared the textures as "extern" in MED and assumed that it would load the textures because, for whatever reason, add_buffer() or al_ent_create() would actually parse the mdl file.
NO! This is not how this works.

What happens is that my code works as expected, and then ent_create() kicks in, reads the model from the cached buffer and then sees: Yay, textures
And then happily loads 100 mb from the HDD.

So, how to avoid this? I'm not going to parse the files. Neither will add_buffer() parse the content (that would result in no longer thread safety). The solution: FIRST load the textures, and then load the model! The la_checkFile() function can be used to load the texture when it returns 1, its safe to load the model.
All in the background, all smooth.

How convenient ;-) then jcl should add engine functions for this. Everything else would un-ergonomic.

Thanks Sid! This is really useful And addresses the one aspect where I thought GS was un-modern.

Jibb

Okay, I gave it a go. I gave your contribution a go, and I tried my own really simple version under the assumption that file_load is thread safe. I included a model with a huge skin (not external -- the mdl file itself was > 100mb). In both cases (my implementation and yours) the application continued while the file was loaded invisibly in the background, but there was still quite a noticeable delay when the entity was actually created in the main thread.

I tried without any threading, and the delay was pretty much the same. Have you tested it with large files?

Jibb

EDIT: And I tried again with your code from the Future forum: same deal.

Can you send me such a model via PM or so? My MED in the virtual machine can only produce corrupted models when I add large textures to it.
When simulating a slow HDD, I get the marine.mdl loaded in ~3 frames while the cubes rotate happily, then it just pops up without any delay.

It's around 350 mb, but zips to around 43. If you don't mind, can you pm me your email address so I can email it to you?

Jibb

so, what's the current state of library?

It works great, but their is still some delay from loading textures into video memory, which is done through DirectX and apparently can't be done in the background.

I imagine other bigger games that do manage to stream in large textures without interrupting gameplay have the memory already allocated as one texture and load smaller textures into it gradually (stitched together, or alternating pixels, or something). That's a complete guess.

Jibb

Hey!
Thanks for releasing this. I tried to use the small version you posted in the other thread but it doesnt make any difference
Sadly this link here is down and I cant test it. Would someone be so kind and could upload a new version of this please?

It's in one of my github repositories: https://github.com/JustSid

Thanks!
Well, Im spawning 3 times the same model.
First time to load textures, One time with your function and one time with only ent_create.
Im using timer() to calcuulate the creation time:
Its near the same
The model has got many textures, bones and vertices, textures are intern for sure. IDK why it doesnt work

Try it 3 x 10.000 times...

Well, I just got a much better fix myself.
I used another model (CUBE_MDL) for the c_move etc and created my model with collision_mode = 0
There is no lag at all, and the best is theres no difference in code since Im using my own hitboxes