December Disaster #2 Random Game Crash in Tower

[RedKing]

As the title says, but here's the full details as far as I can gather:

Crashes are completely inconsistent, sometimes I finish the full run without any tech issue, sometimes I reach tower and it dies. Feels almost 50/50, flip a coin, and if it's tails, I get the boot on loading Tower.
Crashes only happen the moment Tower loads if the crash will happen at all, otherwise, it loads fine and I play through it. Tower load crashes seem to be while the screen is still black, but as the sound effect for warping in begins to play.
Have only ever played this quest in a party so far, don't much feel like suffering a solo instance of it just for testing sake.
Crashes with or without custom textures. A clean game seems to make no difference over one with IMGUI addons or texture mods.
Speed testing Tower areas in solo has not reproduced the crash a single time, by entering Tower quests for quick access.
Crashes have happened across multiple characters, but these same characters can also still win the coin flip and continue normally.

Also running it with expanded RAM from the launcher's DGVoodoo settings, (4GB mode)
Presentation model on Sequential
Phong on
ALT+Enter on
Fast Startup on
Bloom, Clean Capture, DOF, Tonemapping and Cel Shading all off
Disable skin SFX, No Mipmap Scale, IME and Low Performance modes all turned off
AA set to None
Direct3D 11 (this has been the only stable mode for me since the update introduced this)
Virtual Fullscreen at 1440p resolution
Rain/laser scale at default
Anisotropic filtering at 8x
High Res HUD enabled with HUD scale at 160%


System is quite modern, has been running otherwise fine with the usual compatibility and file ownership settings from an NVMe drive (not that it needs this speed)
I've played it on this system plenty long to know it isn't a hardware or firmware compatibility issue.
32GB dual channel DDR4 RAM at 3200MHz
12GB model 3080
Ryzen 9 5900x

 

[Ender]

The game loads all textures, models, and sound assets when entering an area. Those quests load more assets when you enter those areas. This is typically an issue caused by custom textures. You said you tried without using any--are you sure you're trying on a new install? If you simply uninstall the game, I think it will leave any custom files you may have put there, and because the game loads most things from data.gsl by default, custom files typically are new files added to the directory structure.

Can you check the crash details in Windows event viewer? Open Event Viewer, go to Windows Logs > Application on the left and find the red Error event for psobb.exe. Provide the exception code, fault offset, and fault module (everything after the final directory separator, such as psobb.exe--not the full path).

If the fault offset in Event Viewer is in the range of 0x388154 to 0x3882b9 and the fault module is psobb.exe, then it's a crash while decompressing assets (typically textures). Though this type of crash can sometimes have a fault offset in a few different functions.

Also you can test on your own much faster through sandbox and using /warpme to warp directly to an area. The quest uses the default floor number for each area, so you can load the quest and then try /warpme 17 to load into tower directly.

 

[RedKing]

Can you check the crash details in Windows event viewer? Open Event Viewer, go to Windows Logs > Application on the left and find the red Error event for psobb.exe. Provide the exception code, fault offset, and fault module (everything after the final directory separator, such as psobb.exe--not the full path).

If the fault offset in Event Viewer is in the range of 0x388154 to 0x3882b9 and the fault module is psobb.exe, then it's a crash while decompressing assets (typically textures). Though this type of crash can sometimes have a fault offset in a few different functions.

Also you can test on your own much faster through sandbox and using /warpme to warp directly to an area. The quest uses the default floor number for each area, so you can load the quest and then try /warpme 17 to load into tower directly.

After several long and painful scrolls through my application log, as requested:
I went through everything sorted by level and looked through all Error events, couldn't find PSO anywhere in there
Sorted the entire event log by source alphabetically and looked through P, still no sign.

I did find a .NET error with the application details showing online.exe as the culprit, but this was apparently from over a month ago, which has this in the exception details;

System.Threading.AbandonedMutexException at System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean) at System.Threading.WaitHandle.WaitOne(System.TimeSpan, Boolean) at ‬‭‮‪‎‪‎‫‪‍‏‭‏‬‮‏‫‫‏‍‏‏‮.‮‬‌‪‪‌‪‌‎‌‌‮‪‏‏‫‬‌‍‏‌‭‎‏‭‮‮(System.Threading.WaitHandle, System.TimeSpan, Boolean) at ‬‭‮‪‎‪‎‫‪‍‏‭‏‬‮‏‫‫‏‍‏‏‮.‭‏‪‪‌‪‫‭‏‮‫‎‮‍‏‎‬‮‌‎‍‮()

This probably isn't very helpful, though, and under keywords, it just has;

0x80000000000000

Sorting by date and tracing back to around the time of my last crash, I found an Application Error source event pointing to psobb.exe, but still nothing within that error log resembling what you've asked me to look for, just a bunch of temp file paths that lead to empty folders, and a WER report archive path that even given full ownership and control permissions this asinine system won't allow me to access. Event data looks like this;

-1918388164

1

APPCRASH

Not available

0

psobb.exe

0.0.0.0

00000000

psobb.exe

0.0.0.0

00000000

c0000005

0038819b

All the lines past this are the error log temp file paths that are empty or completely inaccessible now. But from the looks of this, it's spewing a bunch of nothing and not even deigning to give me a fault offset either in the general or details tab.

Older crashes I have seemed to have related to psobb.exe but the module is DInput8 so I guess are almost certainly not relevant to my issue.

Some other things to note, so far the only textures I have modified are character and some item textures, no location or enemy textures have been touched so far, and these were quite extensively tested for crashes... And I had a lot of those before I got that project moving properly. Nothing relating to the tower or any enemies spawned during that quest has anything custom at all to break, even when i run a full modified install.

As for how sure I am as to whether I'm definitely running a clean install - I have two completely separate installations on different drives, my main one being the one I'm working on the upscaled texture pack with, and the other being my clean fallback for things going nuclear.

I'll try sandbox rapid warp testing tomorrow, but it's 12:30am for me at the time of posting this reply, and I wasn't quite planning on still being awake right now.

EDIT: You mentioned the quest loading a more assets - I'm guessing your meaning is that these somehow use more assets than usual on a per area basis? Or is this an asset load on loading the quest? I know when I've had texture related crashes before, many of them happened the instant I hit P2, giving me no chance to try anything.

 

[Ender]

I'm not sure how or where you copied the info looking like that. Did you get this from Event Viewer or something else?
Anyway, I can tell c0000005 is the exception code meaning memory/access violation, the psobb.exe lines are application and module, and 0038819b is the fault offset, which occurs in one of the PRS decompression functions in the client. It's likely bad textures.

The quest enables loading of more assets for each of those areas. The standard client loads only specified assets when entering an area--that's one reason why enemies were restricted to only the areas Sega designated (probably originally done for memory constraints back on dreamcast/gamecube). So when you enter Tower normally in a quest like PW4, the game loads assets for only objects on the map, Ill Gill, Del Lily, Epsilon, CCA minibosses, and recoboxes. When you enter Tower in DD2, the game loads assets for every enemy in Episode 2 (and also a few additional ones for some objects).

Maybe I should add some debug mechanism into the client that will log each asset it's loading to some log file. Then if someone is crashing, they can enable that and figure out which asset it is and why it's bad.

Also I'm convinced there are compression issues inside Texture Manager at the end of compressed data streams because I've seen a few textures with a single bad byte at the end of their compressed data in their .bml. Maybe the client can find a way to handle it, or maybe I'll finish a tool to scan a directory for it and attempt to fix.

 

[RedKing]

I'm not sure how or where you copied the info looking like that. Did you get this from Event Viewer or something else?
Anyway, I can tell c0000005 is the exception code meaning memory/access violation, the psobb.exe lines are application and module, and 0038819b is the fault offset, which occurs in one of the PRS decompression functions in the client. It's likely bad textures.

The quest enables loading of more assets for each of those areas. The standard client loads only specified assets when entering an area--that's one reason why enemies were restricted to only the areas Sega designated (probably originally done for memory constraints back on dreamcast/gamecube). So when you enter Tower normally in a quest like PW4, the game loads assets for only objects on the map, Ill Gill, Del Lily, Epsilon, CCA minibosses, and recoboxes. When you enter Tower in DD2, the game loads assets for every enemy in Episode 2 (and also a few additional ones for some objects).

Maybe I should add some debug mechanism into the client that will log each asset it's loading to some log file. Then if someone is crashing, they can enable that and figure out which asset it is and why it's bad.

Also I'm convinced there are compression issues inside Texture Manager at the end of compressed data streams because I've seen a few textures with a single bad byte at the end of their compressed data in their .bml. Maybe the client can find a way to handle it, or maybe I'll finish a tool to scan a directory for it and attempt to fix.

The info I pasted was ALL from the event viewer, several different events - it just oddly copied as a table while not looking anything like one.

I'm not sure what textures could possibly just be bad, because I've been obsessively testing them while changing things around, and my last change was way back in maybe September without issues. The only textures I ever had crashes with were the earlier edits I made that had with Mipmaps and I solved that problem well before I got into the rest of them, which I individually made sure to test several times to be certain they were for sure edited and working.

Anyways, I'm doing some rapid warp testing now, so far I've not been crashing, custom texture setup and all. I'll keep going and see if it eventually just breaks. I might also have to completely unload and reload the quest several times for this, which will slow things down considerably.

You mentioned a PRS decompression being the operation that failed - are DDS files being converted to PRS during the packing to AFS process?
Also, if that's the issue, why are those same textures letting me go through the rest of the quest before Tower and then just dying at the end, and how is it only randomly triggering the crash? Can one bad byte really turn loading a specific area in a specific quest into a Russian Roulette? That just seems weird.

I did formulate a theory while sleeping on it that it could just be reshade slamming the VRAM limit of dgvoodoo, but while checking out my usage, it peaks out at just half of the allocation. So that's a dead-end.

 

[Usagi]

The info I pasted was ALL from the event viewer, several different events - it just oddly copied as a table while not looking anything like one.

You posted the info from white circle blue lowercase i symbol "Information", when asked for "on the left and find the red Error" (red circle white exclamation) which would give the asked for info with clear and concise labels.

Also, if that's the issue, why are those same textures letting me go through the rest of the quest before Tower and then just dying at the end, and how is it only randomly triggering the crash? Can one bad byte really turn loading a specific area in a specific quest into a Russian Roulette?

Oh very much so, in fact it can be such a huge problem an entirely different server that used it's own custom textures had random crashes for years for it's users randomly until I repacked and provided a fixed file.

Next time you crash, can you check the Event Viewer log at the time with the most recent and post it here again. I would just like to see if the information matches with what was previously posted, in case you got an unrelated error when trying to match the time going back and searching through all the logs.
Thank you

 

[RedKing]

You posted the info from white circle blue lowercase i symbol "Information", when asked for "on the left and find the red Error" (red circle white exclamation) which would give the asked for info with clear and concise labels.

No, it's actually from Application Error events in red.

Please, don't try to call me out as a liar. I'm giving information as best I'm able, with the very crap information being presented to me by this system.
When I post "this specific section is from an application error event" I do very much mean so.

As for the next time, I'm currently unsure when this wants to be, I've just tested several different characters with different inventories, rapidly warping through areas, skipping to tower, not skipping, full plays, using the terminal and then skipping, killing a few waves then skipping, and so far no crashes.
It's really being a pain in my ass, I want it to crash, and it refuses, but when I want to just play, I get crashed out almost 50% of the time.

Oh very much so, in fact it can be such a huge problem an entirely different server that used it's own custom textures had random crashes for years for it's users randomly until I repacked and provided a fixed file.

That sounds like it's going to create all kinds of pain in the ass given the inconsistency of symptoms. If there's a good way of detecting a problem in these converted and packed files, I'd love to know it and remove anything stupid caused by the texture manager, I've had to deal with enough of that program's stupidity, like copying blue over green and saving that to a dds to make one texture work because just trying to save blue on its own in a dds crashed the texture manager.


EDIT: I'll get back to testing through my normal play method once I've eaten food; I thought getting this crash would be quick so I didn't bother to launch via steam for my controller config and just endured the nightmare of keyboard. Because I use a Steam Controller to play.

2nd EDIT: If I go into the dgvoodoo config file and enable error logging under the debug list; would I be able to get hold of any crash info that way? I've noticed an error file in the log folder which remains completely empty so far.

3rd EDIT: Never mind on the dgvoodoo logging... the launcher just instantly reverted it to disabled on load, and I can't find the option to enable it in launcher. (Hopefully i'm just being blind with frustration and there's a way past this crap).

I feel like I'm just headbutting a wall trying to reproduce this crash deliberately...

 

[Usagi]

No, it's actually from Application Error events in red.
View attachment 21210
Please, don't try to call me out as a liar. I'm giving information as best I'm able,

I'm sorry something I said caused you to feel attacked, as that was not my intention. You can surely see that already in that snippet you posted, the general tab is formatted differently from anything you posted, which is from as stated (red circle white exclamation). I merely said that the reason your formatting and difficulty finding the labels: "exception code, fault offset and fault module", was because you searched and posted from "Information" (white circle blue lowercase i symbol).

There is no lying here on your part and no "trying to call you out", there is simply a bit of confusion. And I know you're giving info the best you're able; that's why I stepped in to point out the differences between the 2 locations to make it easier on you for future references.
And yes, I do know for a fact the first thing you posted was from Information and not Error in the Application Log, because "Error" doesn't show "Event Name: APPCRASH" nor does it have other stuff you specifically mentioned like "error log temp file paths that are empty or completely inaccessible now" when feeling like it was giving you none of the information asked.
Effectively both [Level: Error and Information] gave the same information, but I offered you the error in where you looked and the place you should look next time to avoid confusion and make it easier for your to post information.

Exception code: 0xc0000005
Fault offset: 0x0038819b
Faulting module: psobb.exe

 

[RedKing]

I'm sorry something I said caused you to feel attacked, as that was not my intention. You can surely see that already in that snippet you posted, the general tab is formatted differently from anything you posted, which is from as stated (red circle white exclamation). I merely said that the reason your formatting and difficulty finding the labels: "exception code, fault offset and fault module", was because you searched and posted from "Information" (white circle blue lowercase i symbol).

There is no lying here on your part and no "trying to call you out", there is simply a bit of confusion. And I know you're giving info the best you're able; that's why I stepped in to point out the differences between the 2 locations to make it easier on you for future references.
And yes, I do know for a fact the first thing you posted was from Information and not Error in the Application Log, because "Error" doesn't show "Event Name: APPCRASH" nor does it have other stuff you specifically mentioned like "error log temp file paths that are empty or completely inaccessible now" when feeling like it was giving you none of the information asked.
Effectively both [Level: Error and Information] gave the same information, but I offered you the error in where you looked and the place you should look next time to avoid confusion and make it easier for your to post information.

Exception code: 0xc0000005
Fault offset: 0x0038819b
Faulting module: psobb.exe

That's all fair, sorry for the misunderstanding, this error has me ripping my hair out and I suddenly can't replicate the crash at all after getting slapped with it by surprise several times. It's been getting on my nerves and I'm using all the exact same files I had at the time of my last crash, I really can't figure out what's different now.

As for the format, that's because I copied it from the details tab which has a couple expandable regions on it. It janks out and pastes it as a table (which I had to remove a bunch of empty cells from)

Just now looking at the code I was told to look for and the code you've quoted from my data set am I seeing that the offset as an 0x format piece is 6 digits past the x and should have honestly realized that window's stupid inconsistency of data presentation betrayed me into thinking the extra two 0s on the back end without an 0x meant it was completely different data. (simple ver: I'm just now noticing that the "0x388---" can also just be presented as "00388---" as shown in that stupid table I pasted...)

If you're able to help with a means of looking for precisely what data might have been made "bad" during the texture manager's conversions, it would be an honestly massive help, because my overall plan for this project of mine was to eventually have the ENTIRE game vanilla upscaled. And that's just never going to happen if a bare few scrambled bytes or flipped bits make completely inconsistent and random crashes appear out of nowhere in niche situations I'd struggle to account for. You'd said about repacking and fixing a bunch of textures for another server running full custom content, and I'd love to know how to process these fixes myself to not end up having to run around begging for people to throw their time at fixing crap created by texture manager.

 

[Usagi]

That's all fair, sorry for the misunderstanding, this error has me ripping my hair out and I suddenly can't replicate the crash at all after getting slapped with it by surprise several times. It's been getting on my nerves and I'm using all the exact same files I had at the time of my last crash, I really can't figure out what's different now.

As for the format, that's because I copied it from the details tab which has a couple expandable regions on it. It janks out and pastes it as a table (which I had to remove a bunch of empty cells from)

Just now looking at the code I was told to look for and the code you've quoted from my data set am I seeing that the offset as an 0x format piece is 6 digits past the x and should have honestly realized that window's stupid inconsistency of data presentation betrayed me into thinking the extra two 0s on the back end without an 0x meant it was completely different data. (simple ver: I'm just now noticing that the "0x388---" can also just be presented as "00388---" as shown in that stupid table I pasted...)

If you're able to help with a means of looking for precisely what data might have been made "bad" during the texture manager's conversions, it would be an honestly massive help, because my overall plan for this project of mine was to eventually have the ENTIRE game vanilla upscaled. And that's just never going to happen if a bare few scrambled bytes or flipped bits make completely inconsistent and random crashes appear out of nowhere in niche situations I'd struggle to account for. You'd said about repacking and fixing a bunch of textures for another server running full custom content, and I'd love to know how to process these fixes myself to not end up having to run around begging for people to throw their time at fixing crap created by texture manager.

No worries, I will not fault you for a misunderstanding as I know how frustrating this game can get haha.

Currently there is no means to instantly and easily have us say "this texture is bad", but Ender's message should give you hope. So I would not be discouraged or give up your upscale plan.

Maybe I should add some debug mechanism into the client that will log each asset it's loading to some log file. Then if someone is crashing, they can enable that and figure out which asset it is and why it's bad.

Also I'm convinced there are compression issues inside Texture Manager at the end of compressed data streams because I've seen a few textures with a single bad byte at the end of their compressed data in their .bml. Maybe the client can find a way to handle it, or maybe I'll finish a tool to scan a directory for it and attempt to fix.

Just play as normal for now and if you do crash again the soonest you can look at the Error Log and post it here again the better, I am sure it will most likely be the same error, but I just want to double check.
In the end if it's on your side with a texture or something else and you don't want to risk crashing, you can 2 EphineaPSO folders one with base graphics for stability and for testing your textures and hope for a future update where compression issues aren't a worry or easily found by the user.
If its on the server/quest side the goal is obviously not to crash people and I'm sure it will be worked on to fix it, but I make no promises obviously; Ender is the real brain behind it all, I just get brought along for testing, ideas and yell a lot when I find an overlooked bug LOL.

 

[RedKing]

I'll give it a go, though I'm not sure about braving DD in a proper play-through on ULT for the chance to get noped out again, thanks for the help on tracking this.

If its on the server/quest side the goal is obviously not to crash people and I'm sure it will be worked on to fix it, but I make no promises obviously; Ender is the real brain behind it all, I just get brought along for testing, ideas and yell a lot when I find an overlooked bug LOL.

I feel that, it used to be my day job for a while, and I always found the niche annoying shit that made programmers cry because the crash dumps made no sense to them. My crap superpower

 

[Ender]

You mentioned a PRS decompression being the operation that failed - are DDS files being converted to PRS during the packing to AFS process?
Also, if that's the issue, why are those same textures letting me go through the rest of the quest before Tower and then just dying at the end, and how is it only randomly triggering the crash? Can one bad byte really turn loading a specific area in a specific quest into a Russian Roulette? That just seems weird.

Yes, one bad byte can do that. If there is no 'end of file' marker when expected, reading the compressed data stream could go outside of the buffer. Or if a byte is bad and the client gets an invalid size or offset for the copy, it could be reading/writing outside of memory. The client could be lucky and the memory beyond that allocation is also mapped in the process--this wouldn't cause an access violation. But it could be overwriting memory intended for something else.

Contents within AFS are compressed. I'm not sure what files you've changed. Player model .afs files are loaded at game startup and not the culprit for that crash. If you're crashing when loading into an area, that's going to be an issue with something the map is loading. In this case, look at custom .bml files loaded by the map. In this case, all Episode 2 enemies.

If the crash has only ever been with Tower, these are additional .bml files loaded by Tower.
bm_n_delta_w_body.bml
bm_nctrunk_body.bml

.bml files are pretty much always compressed, though it seems like the client can support loading them if they are not compressed. Something for me to look into one day.

When warp testing, it's probably a good idea to restart the quest and client every few launches so that you're testing with different memory allocations. I did this when testing with the HD texture pack last week to find out that bm_ene_morphos.bml has an issue.