Preloading assets in C#

-=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- (c) WidthPadding Industries 1987 0\|347\|0 -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=-
Socoder -> C/C++/C#/Other -> Preloading assets in C#
Mon, 22 Oct 2012, 07:58
Afr0	Does anyone have any idea about this? I think I have an idea in my head about how to do it (load three or four random archives into a hashtable, then replace loaded entries with new entries when needed) - the question is if the loaded data will be deleted by the GC when not used. Basically I'm thinking that loading everything on the fly (which is what I'm currently doing) can't be a (the most?) effective way of doing it, even though it should be perfectly passable because of the GC. What's the common way of doing this in Java? -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Mon, 22 Oct 2012, 13:53
JL235	It depends on what you really mean by 'preloading'. If you mean whilst the level is loading, before the actual gameplay starts, then yes, that is normally the best approach. If you mean when the game starts up, that is the most ideal scenario, however it depends on how much memory your assets will take up. If it's 100mb, that's fine. If it's over 500mb, it will be too much. However plenty of games also load on the fly, whilst the game is playing. The 3D Elder Scrolls games are good examples of this, however it also happens on much smaller games too. Ideally you would want to avoid this, because of the complexity it brings to your game. It's not just a case of saying "I need x asset now", many games will try to predict what assets should be loaded/unloaded, seconds or whole minutes before they are needed. For example if you are a 30 second walk away from a new area, then it will be loaded in. Games often also take into account which direction the player is moving and looking, so they would only load in the areas 30 seconds in front of you (and 10 seconds in every other direction). For example John Carmack said at one QuakeCon keynote, that if you want to mess up the loading algorithms in Rage, walk through large sections of the game backwards. Ultimately, what you use depends on what you are building. There are different approaches, because there is no one size fits all. It also has very little to do with the language it's self; many JavaScript based games will have some kind of resource management built in (Play My Code too). As for the GC, the same rules apply as anything else. Barring a few corner cases (which practically speaking do not apply here), if you can get a reference to it, then it will be kept in memory. If you don't have a reference, then it won't be kept in memory. However in Java we do have 'weak references', and I'd expect C# has something similar. It's like a normal reference, however if the application runs out of memory, then the GC is allowed to reclaim it. This is useful for caching, such as pre-generating lots of images. However I've only ever had to use them in Java to work around the performance issues of Java2D, such as rotating and scaling (which are dog slow). After I moved to OpenGL, they just became unnecessary for saving memory (although I have used them since to help work around issues with GC pauses). Hope that helps!
Tue, 23 Oct 2012, 02:27
Afr0	Hm, well TSO is 1.5gig in size, so obviously I can't preload everything. The original game had a loading screen that loaded resources, but I've no idea what they were! But I know that initially I'll need the textures for the GUI. I'm also going to be needing the models and textures for the Create A Sim screen, but those are wayy to big to load all of them, so I'll have to load a random bunch of them. Initially one of the guys I was working with, Nicholas, implemented a loader that loaded everything! Literally, every texture and object in the entire fucking game! Needless to say, I've disabled the loading screen by now. But I figured it was time to do something about it, and do it properly this time... -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Wed, 24 Oct 2012, 02:55
Afr0	I discovered that Nicholas had already implemented what I was thinking of doing, only he was doing it with archives instead of files. Problem was that he wasn't ever deleting an archive after loading it, so I ripped out his code for now. I'm thinking 100 megs is probably too tiny though, I probably want the cache to be between 150 - 300 megs. The way I'm seeing it now, by keeping the cache at that size and filling it with files instead of archives, I'll minimize cacheswitching. -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Sat, 27 Oct 2012, 09:44
Afr0	Soo... does anyone have any idea how to make this more random? Preferably as random as possible. Any ideas and suggestions are welcome! Random Rnd = new Random(DateTime.Now.Millisecond); if (Rnd.Next() < 100) { //Head meshes. GetResourceFromLongID(0x2700000009); //uaasuit-clown-pins-head101-pelvis-suit-clown-pins-head101.mesh GetResourceFromLongID(0x8f00000009); //fahc402fa_bmbshl01-head-head.mesh GetResourceFromLongID(0x9200000009); //fahc404fa_prtygrl01-head-head.mesh GetResourceFromLongID(0x9200000009); //fahc401fa_femftl01-head-head.mesh GetResourceFromLongID(0xa700000009); //fahc200fa_punxz01-head-head.mesh GetResourceFromLongID(0xaf00000009); //fahc004fa_gma1-head-head.mesh GetResourceFromLongID(0xb100000009); //mahc007ma_gpa1-head-head.mesh GetResourceFromLongID(0xbc00000009); //fahc527fa_tsofemalien1-head-head.mesh GetResourceFromLongID(0xbd00000009); //mahc528ma_tsomalien1-head-head.mesh GetResourceFromLongID(0xc300000009); //fahc032fa_retro1-head-head.mesh GetResourceFromLongID(0xc700000009); //mahc033ma_boxer1-head-head.mesh GetResourceFromLongID(0xd700000009); //mahc406ma_loungeliz1-head-head.mesh GetResourceFromLongID(0x10e00000009); //fahc012fa_gma2-head-head.mesh GetResourceFromLongID(0x11200000009); //mahc475ma_blacklthr-head-head.mesh GetResourceFromLongID(0x11c00000009); //fahc041fa_mom2-head-head.mesh } else { //TODO: Load more meshes... } --v -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Sat, 27 Oct 2012, 14:58
JL235	Why would you want any randomness when loading assets?
Sat, 27 Oct 2012, 15:29
Afr0	Because I can't preload all the assets? The meshes alone are 100megs compressed! If the cache is at 250 megs I might be able to fit it all in, but decompressing all that data takes time. Anyhow, here's my current approach: //Load a bunch of random meshes... GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); GetResourceFromLongID(Get64BitRandom(4294967305, 9350143803401)); --v The problem with this approach is that I can't control the exact size of the data I'm loading in. Does anyone have an idea for a more precise way of doing this? Here's how I cache assets: public static byte[] GetResourceFromLongID(ulong id) { while (!initComplete) ; if (m_Resources.ContainsKey(id)) { //Resource hasn't already been loaded... if (!m_LoadedResources.ContainsKey(id)) { string path = m_Resources[id].Replace("./", ""); if (!File.Exists(path)) { string[] pathSections = path.Split(new char[] { '/' }); int directoryIdx = 0; if (path.Contains("/heads/") \|\| path.Contains("/hands/") \|\| path.Contains("/bodies/") \|\| path.Contains("/accessories/")) directoryIdx = Array.FindLastIndex<string>(pathSections, delegate(string it) { if (it.CompareTo("avatardata") == 0) { return true; } return false; }) + 2; else directoryIdx = Array.FindLastIndex<string>(pathSections, delegate(string it) { if (it.CompareTo("TSOClient") == 0) { return true; } return false; }) + 2; string directoryName = pathSections[directoryIdx]; path = path.Remove(path.LastIndexOf('/')); string archivePath = GlobalSettings.Default.StartupPath + '/' + path.Remove((path.LastIndexOf(pathSections[directoryIdx]))) + directoryName + '/' + directoryName + ".dat"; FAR3Archive archive = new FAR3Archive(archivePath); TryToStoreResource(id, archive[pathSections[pathSections.Length - 1]]); return archive[pathSections[pathSections.Length - 1]]; } else { byte[] Resource = File.ReadAllBytes(GlobalSettings.Default.StartupPath + path); TryToStoreResource(id, Resource); return Resource; } } else return m_LoadedResources[id]; } return new byte[0]; } /// <summary> /// Tries to store a resource in the internal cache. /// </summary> /// <param name="ID">The ID of the resource to store.</param> /// <param name="Resource">The resource to store.</param> private static void TryToStoreResource(ulong ID, byte[] Resource) { if (m_CurrentCacheSize < m_CACHESIZE) { m_LoadedResources.Add(ID, Resource); m_CurrentCacheSize += Resource.Length; } else { ulong LastKey = m_LoadedResources.Keys.Last(); m_CurrentCacheSize -= m_LoadedResources[LastKey].Length; m_LoadedResources.Remove(LastKey); m_LoadedResources.Add(ID, Resource); } } --v Edit: Whoah, logging reveals that I'm currently only loading 2mb! Is the below formula correct? Log.LogThis("Current cachesize: " + (m_CurrentCacheSize / 1048576) + "mb.", eloglevel.info); --v -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Sun, 28 Oct 2012, 06:30
Afr0	So I've run some tests and changed the code around a little, and somehow the cache ALWAYS ends up being EXACTLY the same size, down to the last byte. Can someone PLEASE explain what's going on? I'll paste the entire class, minus the methods that aren't being used. using System; using System.Collections.Generic; using System.Text; using System.Drawing; using System.Threading; using System.IO; using System.Linq; using SimsLib.FAR3; using DNA; using SimsLib.IFF; using Microsoft.Xna.Framework.Graphics; using SimsLib.FAR1; using TSOClient.LUI; using LogThis; namespace TSOClient { public delegate void OnLoadingUpdatedDelegate(string LoadingText); public class ContentManager { private const int m_CACHESIZE = 104857600; //100 megabytes. private static int m_CurrentCacheSize = 0; private static Dictionary<ulong, string> m_Resources; private static Dictionary<ulong, byte[]> m_LoadedResources; private static bool initComplete = false; private static Random m_Rand; private static List<Floor> m_Floors; private static List<Wall> m_Walls; private static Wall m_DefaultWall; private static Floor m_DefaultFloor; private static ManualResetEvent m_ResetEvent = new ManualResetEvent(false); public static event OnLoadingUpdatedDelegate OnLoadingUpdatedEvent; public static Wall DefaultWall { get { return m_DefaultWall; } } public static Floor DefaultFloor { get { return m_DefaultFloor; } } public static List<Floor> Floors { get { return m_Floors; } } public static List<Wall> Walls { get { return m_Walls; } } static ContentManager() { m_Rand = new Random(0); FAR3Archive packingSlips = new FAR3Archive(GlobalSettings.Default.StartupPath + "packingslips\\packingslips.dat"); List<KeyValuePair<uint, byte[]>> entries = packingSlips.GetAllEntries(); m_Resources = new Dictionary<ulong, string>(); m_LoadedResources = new Dictionary<ulong, byte[]>(); foreach (KeyValuePair<uint, byte[]> kvp in entries) { BinaryReader br = new BinaryReader(new MemoryStream(kvp.Value)); br.BaseStream.Position = 50; string path = br.ReadString(); br.BaseStream.Position += 8; ulong id = Endian.SwapUInt64(br.ReadUInt64()); string[] VersionElements = GlobalSettings.Default.ClientVersion.Split(".".ToCharArray()); //Hack to correct references to old files contained in packingslips.dat, //that exists from version 1.1097.1.0 onwards... if(int.Parse(VersionElements[0]) >= 1 && int.Parse(VersionElements[1]) >= 1097 && int.Parse(VersionElements[2]) >= 1 && int.Parse(VersionElements[3]) >= 0) { if (path.Contains("person_select_cityhousebtn.bmp")) path = path.Replace("person_select_cityhousebtn.bmp", "person_select_cityhouseiconalpha.tga"); else if (path.Contains("person_select_editbtn.bmp")) path = path.Replace("person_select_editbtn.bmp", "person_select_simcreatebtn.bmp"); else if (path.Contains("person_edit_exitbtn.bmp")) path = path.Replace("person_edit_exitbtn.bmp", "person_edit_closebtn.bmp"); else if (path.Contains("person_edit_skinblackbtn.bmp")) path = path.Replace("person_edit_skinblackbtn.bmp", "person_edit_skindarkbtn.bmp"); else if (path.Contains("person_edit_skinbrownbtn.bmp")) path = path.Replace("person_edit_skinbrownbtn.bmp", "person_edit_skinmediumbtn.bmp"); else if (path.Contains("person_edit_skinwhitebtn.bmp")) path = path.Replace("person_edit_skinwhitebtn.bmp", "person_edit_skinlightbtn.bmp"); } m_Resources.Add(id, path); } string s = new List<string>(m_Resources.Values)[35854]; initComplete = true; } public static byte[] GetResourceFromLongID(ulong id) { while (!initComplete) ; if (m_Resources.ContainsKey(id)) { //Resource hasn't already been loaded... if (!m_LoadedResources.ContainsKey(id)) { string path = m_Resources[id].Replace("./", ""); if (!File.Exists(path)) { string[] pathSections = path.Split(new char[] { '/' }); int directoryIdx = 0; if (path.Contains("/heads/") \|\| path.Contains("/hands/") \|\| path.Contains("/bodies/") \|\| path.Contains("/accessories/")) directoryIdx = Array.FindLastIndex<string>(pathSections, delegate(string it) { if (it.CompareTo("avatardata") == 0) { return true; } return false; }) + 2; else directoryIdx = Array.FindLastIndex<string>(pathSections, delegate(string it) { if (it.CompareTo("TSOClient") == 0) { return true; } return false; }) + 2; string directoryName = pathSections[directoryIdx]; path = path.Remove(path.LastIndexOf('/')); string archivePath = GlobalSettings.Default.StartupPath + '/' + path.Remove((path.LastIndexOf(pathSections[directoryIdx]))) + directoryName + '/' + directoryName + ".dat"; FAR3Archive archive = new FAR3Archive(archivePath); TryToStoreResource(id, archive[pathSections[pathSections.Length - 1]]); return archive[pathSections[pathSections.Length - 1]]; } else { byte[] Resource = File.ReadAllBytes(GlobalSettings.Default.StartupPath + path); TryToStoreResource(id, Resource); return Resource; } } else return m_LoadedResources[id]; } return new byte[0]; } public byte[] this[ulong id] { get { if (m_Resources.ContainsKey(id)) { //Resource hasn't already been loaded... if (!m_LoadedResources.ContainsKey(id)) { string path = m_Resources[id].Replace("./", ""); if (!File.Exists(path)) { string[] pathSections = path.Split(new char[] { '/' }); string directoryName = pathSections[pathSections.Length - 2]; string archivePath = GlobalSettings.Default.StartupPath + path.Remove(path.LastIndexOf('/') + 1) + directoryName + ".dat"; FAR3Archive archive = new FAR3Archive(archivePath); TryToStoreResource(id, archive[pathSections[pathSections.Length - 1]]); return archive[pathSections[pathSections.Length - 1]]; } else { byte[] Resource = File.ReadAllBytes(GlobalSettings.Default.StartupPath + path); TryToStoreResource(id, Resource); return Resource; } } else return m_LoadedResources[id]; } return new byte[0]; } } /// <summary> /// Tries to store a resource in the internal cache. /// </summary> /// <param name="ID">The ID of the resource to store.</param> /// <param name="Resource">The resource to store.</param> private static void TryToStoreResource(ulong ID, byte[] Resource) { if (m_CurrentCacheSize < m_CACHESIZE) { m_LoadedResources.Add(ID, Resource); m_CurrentCacheSize += Resource.Length; } else { ulong LastKey = m_LoadedResources.Keys.Last(); m_CurrentCacheSize -= m_LoadedResources[LastKey].Length; m_LoadedResources.Remove(LastKey); m_LoadedResources.Add(ID, Resource); m_CurrentCacheSize += Resource.Length; } } //From: https://social.msdn.microsoft.com/Forums/en/csharpgeneral/thread/cb9c7f4d-5f1e-4900-87d8-013205f27587 private static ulong Get64BitRandom(ulong minValue, ulong maxValue) { Random Rnd = new Random(DateTime.Now.Millisecond); // Get a random array of 8 bytes. byte[] buffer = new byte[sizeof(ulong)]; Rnd.NextBytes(buffer); return BitConverter.ToUInt64(buffer, 0) % (maxValue - minValue + 1) + minValue; } private static void LoadInitialTextures() { //LuaInterfaceManager.CallFunction("UpdateLoadingscreen"); //These textures are needed for the logindialog, so preload them. GetResourceFromLongID(0xe500000002); //Dialog. GetResourceFromLongID(0x1e700000001); //Button. GetResourceFromLongID(0x7a500000001); //Progressbar. //LuaInterfaceManager.CallFunction("UpdateLoadingscreen"); //Textures for the personselection screen. GetResourceFromLongID(0x3fa00000001); //person_select_background.bmp GetResourceFromLongID(0x3ff00000001); //person_select_exitbtn.bmp GetResourceFromLongID(0x3fe00000001); //person_select_simcreatebtn.bmp GetResourceFromLongID(0x3fc00000001); //person_select_cityhouseiconalpha.tga GetResourceFromLongID(0x3f800000001); //person_select_arrowdownbtn.bmp GetResourceFromLongID(0x3f900000001); //person_select_arrowupbtn.bmp //LuaInterfaceManager.CallFunction("UpdateLoadingscreen"); //Textures for the CAS screen. GetResourceFromLongID(0x3dc00000001); //person_edit_background.bmp //GetResourceFromLongID(0x3dd00000001); //person_edit_backtoselectbtn.bmp GetResourceFromLongID(0x3e000000001); //person_edit_cancelbtn.bmp GetResourceFromLongID(0x3e300000001); //person_edit_closebtn.bmp GetResourceFromLongID(0x3e400000001); //person_edit_femalebtn.bmp GetResourceFromLongID(0x3eb00000001); //person_edit_malebtn.bmp GetResourceFromLongID(0x3f300000001); //person_edit_skindarkbtn.bmp GetResourceFromLongID(0x3f400000001); //person_edit_skinmediumbtn.bmp GetResourceFromLongID(0x3f500000001); //person_edit_skinbrowserarrowleft.bmp GetResourceFromLongID(0x3f600000001); //person_edit_skinbrowserarrowright.bmp GetResourceFromLongID(0x3f700000001); //person_edit_skinlightbtn.bmp LuaInterfaceManager.CallFunction("UpdateLoadingscreen"); GetResourceFromLongID(0x89500000001); //cas-sas-creditsbtn.bmp GetResourceFromLongID(0x89600000001); //cas-sas-creditsindent.bmp //Textures for the credits screen. GetResourceFromLongID(0x8aa00000001); //creditscreen_backbtn.bmp GetResourceFromLongID(0x8ab00000001); //creditscreen_backbtnindent.bmp GetResourceFromLongID(0x8ac00000001); //creditscreen_background.bmp GetResourceFromLongID(0x8ad00000001); //creditscreen_exitbtn.bmp GetResourceFromLongID(0x8ae00000001); //creditscreen_maxisbtn.bmp GetResourceFromLongID(0x8af00000001); //creditscreen_tsologo_english.bmp GetResourceFromLongID(0x8a800000001); //cityselector_sortbtn.bmp GetResourceFromLongID(0x8a900000001); //cityselector_thumbnailbackground.bmp //LuaInterfaceManager.CallFunction("UpdateLoadingscreen"); List<ulong> RandomIDs = new List<ulong>(); for (int i = 0; i < 150; i++) { GenerateNewID: ulong ID = Get64BitRandom(4294967305, 9350143803401); if (!RandomIDs.Contains(ID)) RandomIDs.Add(ID); else goto GenerateNewID; } //Load a bunch of random meshes... foreach (ulong ID in RandomIDs) GetResourceFromLongID(ID); Log.LogThis("Current cachesize: " + (m_CurrentCacheSize) + "bytes", eloglevel.info); myLoadingScreenEWH.Set(); } private static EventWaitHandle myLoadingScreenEWH; /// <summary> /// Initializes loading of resources. /// </summary> /// <param name="ScreenMgr">A ScreenManager instance, used to access a GraphicsDevice.</param> public static void InitLoading() { myLoadingScreenEWH = new EventWaitHandle(false, EventResetMode.ManualReset, "Go_Away_Stupid_Loading_Screen_GO_U_HEARD_ME_DONT_MAKE_ME_GET_MY_STICK_OUT"); //ThreadPool.QueueUserWorkItem(new WaitCallback(LoadContent)); Thread T = new Thread(new ParameterizedThreadStart(LoadContent)); T.Start(); } /// <summary> /// Threading function that takes care of loading. /// </summary> private static void LoadContent(object ThreadObject) { //InitWalls(Manager.GraphicsDevice); //InitFloors(Manager.GraphicsDevice); LoadInitialTextures(); } } } --v -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Sun, 28 Oct 2012, 13:27
JL235	If you can't preload them all, then just preload the ones you need. Why pick at random? If you want to preload some you might need, then preload the most common assets, or preload the smallest. No point in just picking at random.
Sun, 28 Oct 2012, 14:29
Afr0	Yeah, that's the issue. There's no way to know which assets are used to create sims. But me and Andrew found out through a lengthy conversation on Facebook that packingslips.dat are used in conjunction with .xml files in the packingslips folder to map the assets. As it turns out, packingslips.txt contains a few bogus entries (that were removed before the game was shipped), and packingslips.dat a lot more (since it is 5+ megs, versus 2 megs for packingslips.txt). This should have been obvious from the start, as it is a totally maxoidian thing to do - ship an outdated mapping with the final game, and use it in conjunction with other maps that are up to date! \|edit\| This doesn't actually explain anything. Basically, I found out that generating random IDs didn't seem to work, and then I decided to try to load all the meshes. When I tried doing that, the process started crashing because of the bogus entries in packingslips.dat. So now I need to parse the entries in the .xml files as well... \|edit\| -=-=- Afr0 Games Project Dollhouse on Github - Please fork!