[video output=day543 member=cmuratori stream_platform=twitch stream_username=handmade_hero project=code title="Moving Unpacked Entities from the Sim Region to World" vod_platform=youtube id=kWXQTOPy_1o annotator=Miblo]
[0:01][Recap and set the stage for the day][:speech]
[1:10][Demo our new :memory viewer, with a view to working on the storage of our world's entity data][:"entity system" :run]
[6:05][Our :"entity system", not pre-designed][:research]
[8:10][Compare the :memory requirements of our :"entity system", with and without the ground_cover][:programming :run]
[8:55][Our concrete problem: visual effects only need to be stored while they are visible][:memory :speech]
[14:16][Alternative, lossy representations of entities][:"entity system" :speech]
[17:31][BeginSim() and the (un)packing of entities][:"entity system" :research]
[21:50][Changing the sim_region to contain a reference table of entity pointers contained in the world's cache][:"entity system" :research]
[24:22][Start to reacquaint ourselves with all uses of the sim_region's Entities][:"entity system" :research]
[26:40][Reacquaint ourselves with the sim_region's EntityHashOccupancy][:"entity system" :research]
[28:20][Continue to reacquaint ourselves with all uses of the sim_region's Entities][:"entity system" :research]
[29:29][Augment the world struct with UnpackedEntities, into which the sim_region may point][:"entity system"]
[37:04][The problem of determining which entities to pack][:"entity system" :research]
[40:32][Change all instances of entity array iteration in the sim region to operate as if on a reference table of pointers, introducing entity_iterator and IterateAllEntities()][:"entity system"]
[45:45][Begin to move all the existing entity storage, packing and unpacking code from the sim region to the world, introducing EnsureRegionIsUnpacked()][:"entity system"]
[56:51][Consider how to correctly set the origin of entities during unpacking][:"entity system" :research]
[1:02:42][Augment the world with a world_position UnpackOrigin, for EnsureRegionIsUnpacked() to use, and introduce RegisterEntity()][:"entity system"]
[1:10:55][Introduce a version in the world of CreateEntity() that takes a world *][:"entity system"]
[1:15:11][Introduce RepackEntitiesAsNecessary() in the world, for the sim region's EndWorldChange() to call][:"entity system"]
[1:24:57][Implement IterateAllEntities() and the entity_iterator version of Advance(), introducing FindNextEntity()][:"data structure" :"entity system"]
[1:30:58][Hit our assertion in GetHashFromID()][:"entity system" :run]
[1:31:30][Fix BeginWorldChange() to call EnsureRegionIsUnpacked()][:"entity system"]
[1:32:21][Hit our InvalidCodePath in EnsureRegionIsUnpacked()][:"entity system" :run]
[1:32:42][Remove Entities_ from the sim_region, and fix EnsureRegionIsUnpacked() to operate on the entities in the world][:"entity system"]
[1:33:28][Hit our InvalidCodePath in EnsureRegionIsUnpacked(), and investigate why][:"entity system" :run]
[1:36:15][Augment the world with an UnpackIsOpen to prevent multiple concurrent unpacks of the same sim region][:"entity system"]
[1:37:50][Hit our InvalidCodePath in EnsureRegionIsUnpacked(), noting that entity (un)packing succeeds during world generation, but not during simulation][:"entity system" :run]
[1:41:09][Enable EnsureRegionIsUnpacked() to detect infinite looping][:"entity system"]
[1:46:51][Hit our InvalidCodePath in EnsureRegionIsUnpacked(), assuming that it is not infinite looping][:"entity system" :run]
[1:47:17][Investigate why we are hitting the InvalidCodePath in EnsureRegionIsUnpacked()][:"entity system" :research]
[1:51:57][Step through many calls to RepackEntitiesAsNecessary(), watching the EntityP values][:"entity system" :run]
[1:56:48][Step through IterateAllEntities() to see that we do find stuff][:"entity system" :run]
[1:58:36][Step through EnsureRegionIsUnpacked(), watching which chunk is responsible for the majority of blocks][:"entity system" :run]
[2:01:33][Track our number of packs minus unpacks][:"entity system"]
[2:04:28][Find that the LastUsedEntityStorageIndex is 31 higher than our TotalEntityPacksMinusUnpacks][:"entity system" :run]
[2:06:32][Temporarily set a huge MAX_SIM_REGION_ENTITY_COUNT][:"entity system"]
[2:08:41][Step over BeginWorldChange() to find that we are trying to unpack 15383 entities, fewer than the LastUsedEntityStorageIndex of 21523][:"entity system" :run]
[2:09:22][Disable :lighting]
[2:18:08][Crash in LoadEntityReference()][:"entity system" :run]
[2:19:14][Make RepackEntitiesAsNecessary() take a SimRegion to pass to PackTraversableRefernce()][:"entity system"]
[2:20:08][Still crash in LoadEntityReference()][:"entity system" :run]
[2:20:43][Fix FindNextEntity() to only find an entity if its occupancy slot is not empty][:owl :programming]
[2:23:48][:Run without crashing, but see that RepackEntitiesAsNecessary() is happening too often][:"entity system" :owl]
[2:25:23][Step through BeginSim() to see a huge discrepancy between our TotalEntityPacksMinusUnpacks of 6109 and LastUsedEntityStorageIndex of 21523][:"entity system" :owl :run]
[2:27:58][Fix EnsureRegionIsUnpacked() to offset the Dest->P after actually setting the Dest][:"entity system" :owl :programming]
[2:28:13][Find that everything now works, noting the :performance of the game in debug mode without :lighting][:"entity system" :owl :run]
[2:29:29][Re-enable :lighting][:owl :programming]
[2:30:01][Find that we are fully back to normal][:owl :run]
[2:30:56][Q&A][:speech]
[2:32:00][@somebody_took_my_name][Q: Will the packed entity be a separate struct, and what is the plan of adding / removing data?][:"entity system"]
[2:34:07][@Brian][Q: I had a question regarding the Arena graph in the profiler, though it may have changed this episode, and you may not be able to display it in its current state. It was about why there were separations between the ground chunks (the first group, all were "green", but every X amount, there was a gray value that separated them). What was this separation?][:"entity system"]
[2:37:41][Remove the cruft from world_entity_block][:"entity system"]
[2:39:15][Check out our more closely packed entity blocks][:"entity system" :run]
[2:41:31][Change CreateWorld() to put the world in its own :memory arena, noting that GetWorldChunk() allocates space for the headers][:"entity system"]
[2:44:03][See that our entity blocks are packed the same][:"entity system" :run]
[2:44:24][Revert CreateWorld() to put the world in the GameState->ModeArena][:"entity system" :memory]
[2:44:47][Note the utility of :memory usage visualisation][:run]
[2:45:05][Change GetWorldChunk() to allocate space for chunks in bulk][:"entity system" :memory]
[2:49:29][Check out our perfectly packed entity blocks][:"entity system" :run]
[2:51:20][@sourdust][Q: Did you ever figure out MEM_LARGE_PAGES for VirtualAlloc? Is that what you're talking about?][:memory]
[2:51:35][@centhusiast][Q: How did people do :memory management in the era of Commodore 64 for writing games? It is so luxurious nowadays with all these amounts of memory we have on our machines]
