Random dynamic zone crashes

[fwaits]

I have a relatively new server setup and everything works fine, however, I get seemingly random dynamic zone crashes that take down zones with the following Debug popup error:

http://sdrv.ms/18XfDMB

If I click Retry on the prompts the dynamic zone that crashed will restart and come back up ok, but obviously if I'm not watching, these errors will eventually take a large number of the zones down over time.

Any help where I should be looking to figure out the problem?

Here is the dump from the last crash_zone_####.log file if that is relevant:

---------------------------------------------
[08.04. - 17:34:21] Starting Log: logs/crash_zone_1016.log
[08.04. - 17:34:21] EXCEPTION_BREAKPOINT
[08.04. - 17:34:21] SymInit: Symbol-SearchPath: '.;C:\EQ\EQEmuServer;C:\EQ\EQEmuServer;C:\Windows;C:\Windows\system32;SRV*C:\websymbols*http://msdl.microsoft.com/download/symbols;', symOptions: 530, UserName: 'fwwadmin' 
[08.04. - 17:34:21] OS-Version: 6.0.6002 (Service Pack 2) 0x110-0x3 
[08.04. - 17:34:21] C:\EQ\EQEmuServer\zone.exe:zone.exe (012D0000), size: 24543232 (result: 0), SymType: 'PDB', PDB: 'C:\EQ\EQEmuServer\zone.exe' 
[08.04. - 17:34:21] C:\Windows\system32\ntdll.dll:ntdll.dll (77180000), size: 1212416 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\ntdll.dll', fileVersion: 6.0.6002.18541 
[08.04. - 17:34:21] C:\Windows\system32\kernel32.dll:kernel32.dll (77010000), size: 901120 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\kernel32.dll', fileVersion: 6.0.6002.18740 
[08.04. - 17:34:21] C:\Windows\system32\SHELL32.dll:SHELL32.dll (764F0000), size: 11603968 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\SHELL32.dll', fileVersion: 6.0.6002.18795 
[08.04. - 17:34:21] C:\Windows\system32\msvcrt.dll:msvcrt.dll (76270000), size: 696320 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\msvcrt.dll', fileVersion: 7.0.6002.18551 
[08.04. - 17:34:21] C:\Windows\system32\GDI32.dll:GDI32.dll (759E0000), size: 307200 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\GDI32.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\USER32.dll:USER32.dll (76320000), size: 643072 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\USER32.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\ADVAPI32.dll:ADVAPI32.dll (75EB0000), size: 811008 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\ADVAPI32.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\RPCRT4.dll:RPCRT4.dll (75D80000), size: 798720 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\RPCRT4.dll', fileVersion: 6.0.6002.18024 
[08.04. - 17:34:21] C:\Windows\system32\SHLWAPI.dll:SHLWAPI.dll (763C0000), size: 364544 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\SHLWAPI.dll', fileVersion: 6.0.6002.18738 
[08.04. - 17:34:21] C:\Perl\bin\perl512.dll:perl512.dll (28000000), size: 1273856 (result: 0), SymType: '-exported-', PDB: 'C:\Perl\bin\perl512.dll', fileVersion: 5.12.3.1204 
[08.04. - 17:34:21] C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.6002.18305_none_88f3a38569c2c436\COMCTL32.dll:COMCTL32.dll (739F0000), size: 544768 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.6002.18305_none_88f3a38569c2c436\COMCTL32.dll', fileVersion: 5.82.6002.18305 
[08.04. - 17:34:21] C:\Windows\system32\WS2_32.dll:WS2_32.dll (75E50000), size: 184320 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\WS2_32.dll', fileVersion: 6.0.6001.18000 
[08.04. - 17:34:21] C:\Windows\system32\NSI.dll:NSI.dll (760F0000), size: 24576 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\NSI.dll', fileVersion: 6.0.6001.18000 
[08.04. - 17:34:21] C:\Windows\system32\VERSION.dll:VERSION.dll (74B30000), size: 32768 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\VERSION.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\ShimEng.dll:ShimEng.dll (73020000), size: 122880 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\ShimEng.dll', fileVersion: 6.0.6000.16386 
[08.04. - 17:34:21] C:\Windows\system32\apphelp.dll:apphelp.dll (75640000), size: 180224 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\apphelp.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\AppPatch\AcLayers.DLL:AcLayers.DLL (72F00000), size: 557056 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\AppPatch\AcLayers.DLL', fileVersion: 6.0.6002.18708 
[08.04. - 17:34:21] C:\Windows\system32\ole32.dll:ole32.dll (75FA0000), size: 1331200 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\ole32.dll', fileVersion: 6.0.6002.18277 
[08.04. - 17:34:21] C:\Windows\system32\OLEAUT32.dll:OLEAUT32.dll (770F0000), size: 577536 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\OLEAUT32.dll', fileVersion: 6.0.6002.18508 
[08.04. - 17:34:21] C:\Windows\system32\USERENV.dll:USERENV.dll (756F0000), size: 122880 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\USERENV.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\Secur32.dll:Secur32.dll (756D0000), size: 81920 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\Secur32.dll', fileVersion: 6.0.6002.18541 
[08.04. - 17:34:21] C:\Windows\system32\WINSPOOL.DRV:WINSPOOL.DRV (72010000), size: 270336 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\WINSPOOL.DRV', fileVersion: 6.0.6002.18392 
[08.04. - 17:34:21] C:\Windows\system32\MPR.dll:MPR.dll (75250000), size: 81920 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\MPR.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\IMM32.DLL:IMM32.DLL (75F80000), size: 122880 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\IMM32.DLL', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\MSCTF.dll:MSCTF.dll (76110000), size: 819200 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\MSCTF.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\LPK.DLL:LPK.DLL (772B0000), size: 36864 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\LPK.DLL', fileVersion: 6.0.6002.18051 
[08.04. - 17:34:21] C:\Windows\system32\USP10.dll:USP10.dll (76420000), size: 512000 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\USP10.dll', fileVersion: 1.626.6002.18244 
[08.04. - 17:34:21] C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18305_none_5cb72f2a088b0ed3\comctl32.dll:comctl32.dll (74540000), size: 1695744 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18305_none_5cb72f2a088b0ed3\comctl32.dll', fileVersion: 6.10.6002.18305 
[08.04. - 17:34:21] C:\Windows\system32\NLAapi.dll:NLAapi.dll (74A50000), size: 61440 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\NLAapi.dll', fileVersion: 6.0.6001.18000 
[08.04. - 17:34:21] C:\Windows\system32\IPHLPAPI.DLL:IPHLPAPI.DLL (751D0000), size: 102400 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\IPHLPAPI.DLL', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\dhcpcsvc.DLL:dhcpcsvc.DLL (75050000), size: 217088 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\dhcpcsvc.DLL', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\DNSAPI.dll:DNSAPI.dll (752E0000), size: 180224 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\DNSAPI.dll', fileVersion: 6.0.6002.18416 
[08.04. - 17:34:21] C:\Windows\system32\WINNSI.DLL:WINNSI.DLL (75290000), size: 28672 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\WINNSI.DLL', fileVersion: 6.0.6001.18000 
[08.04. - 17:34:21] C:\Windows\system32\dhcpcsvc6.DLL:dhcpcsvc6.DLL (75010000), size: 139264 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\dhcpcsvc6.DLL', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\napinsp.dll:napinsp.dll (734B0000), size: 61440 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\napinsp.dll', fileVersion: 6.0.6001.18000 
[08.04. - 17:34:21] C:\Windows\System32\mswsock.dll:mswsock.dll (74E10000), size: 241664 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\System32\mswsock.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\System32\winrnr.dll:winrnr.dll (73390000), size: 32768 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\System32\winrnr.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\WLDAP32.dll:WLDAP32.dll (764A0000), size: 299008 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\WLDAP32.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\PSAPI.DLL:PSAPI.DLL (757A0000), size: 28672 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\PSAPI.DLL', fileVersion: 6.0.6000.16386 
[08.04. - 17:34:21] C:\Windows\system32\rasadhlp.dll:rasadhlp.dll (732F0000), size: 24576 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\rasadhlp.dll', fileVersion: 6.0.6000.16386 
[08.04. - 17:34:21] C:\Windows\System32\wshtcpip.dll:wshtcpip.dll (74A60000), size: 20480 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\System32\wshtcpip.dll', fileVersion: 6.0.6001.18000 
[08.04. - 17:34:21] C:\Perl\lib\auto\Cwd\Cwd.dll:Cwd.dll (10000000), size: 20480 (result: 0), SymType: '-exported-', PDB: 'C:\Perl\lib\auto\Cwd\Cwd.dll' 
[08.04. - 17:34:21] C:\Perl\lib\auto\DBI\DBI.dll:DBI.dll (000B0000), size: 98304 (result: 0), SymType: '-exported-', PDB: 'C:\Perl\lib\auto\DBI\DBI.dll' 
[08.04. - 17:34:21] C:\Perl\lib\auto\List\Util\Util.dll:Util.dll (00090000), size: 28672 (result: 0), SymType: '-exported-', PDB: 'C:\Perl\lib\auto\List\Util\Util.dll' 
[08.04. - 17:34:21] C:\Perl\lib\auto\Filter\Util\Call\Call.dll:Call.dll (002B0000), size: 20480 (result: 0), SymType: '-exported-', PDB: 'C:\Perl\lib\auto\Filter\Util\Call\Call.dll' 
[08.04. - 17:34:21] C:\Perl\lib\auto\IO\IO.dll:IO.dll (00340000), size: 24576 (result: 0), SymType: '-exported-', PDB: 'C:\Perl\lib\auto\IO\IO.dll' 
[08.04. - 17:34:21] C:\Windows\system32\PROPSYS.dll:PROPSYS.dll (73C30000), size: 765952 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\PROPSYS.dll', fileVersion: 7.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\CLBCatQ.DLL:CLBCatQ.DLL (761E0000), size: 540672 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\CLBCatQ.DLL', fileVersion: 2001.12.6931.18000 
[08.04. - 17:34:21] C:\Windows\system32\rsaenh.dll:rsaenh.dll (74C00000), size: 241664 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\rsaenh.dll', fileVersion: 6.0.6002.18005 
[08.04. - 17:34:21] C:\Windows\system32\SXS.DLL:SXS.DLL (75580000), size: 389120 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\SXS.DLL', fileVersion: 6.0.6001.18000 
[08.04. - 17:34:21] C:\Windows\system32\dbghelp.dll:dbghelp.dll (6B370000), size: 901120 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\dbghelp.dll', fileVersion: 6.0.6001.18000 
[08.04. - 17:34:21] f:\dd\vctools\crt_bld\self_x86\crt\src\dbgdel.cpp (52): operator delete 
[08.04. - 17:34:21] c:\program files\microsoft visual studio 10.0\vc\include\xmemory (182): std::allocator<char>::deallocate 
[08.04. - 17:34:21] c:\program files\microsoft visual studio 10.0\vc\include\xstring (1997): std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Tidy 
[08.04. - 17:34:21] c:\program files\microsoft visual studio 10.0\vc\include\xstring (754): std::basic_string<char,std::char_traits<char>,std::allocator<char> >::~basic_string<char,std::char_traits<char>,std::allocator<char> > 
[08.04. - 17:34:21] ERROR: SymGetLineFromAddr64, GetLastError: 487 (Address: 0152ED1C) 
[08.04. - 17:34:21] 0152ED1C (zone): (filename not available): NPCType::~NPCType 
[08.04. - 17:34:21] ERROR: SymGetLineFromAddr64, GetLastError: 487 (Address: 0152ECC6) 
[08.04. - 17:34:21] 0152ECC6 (zone): (filename not available): NPCType::`scalar deleting destructor' 
[08.04. - 17:34:21] c:\eq\source\zone\zone.cpp (778): Zone::Shutdown 
[08.04. - 17:34:21] c:\eq\source\zone\net.cpp (419): main 
[08.04. - 17:34:21] f:\dd\vctools\crt_bld\self_x86\crt\src\crt0.c (278): __tmainCRTStartup 
[08.04. - 17:34:21] f:\dd\vctools\crt_bld\self_x86\crt\src\crt0.c (189): mainCRTStartup 
[08.04. - 17:34:21] ERROR: SymGetLineFromAddr64, GetLastError: 487 (Address: 7705D2E9) 
[08.04. - 17:34:21] 7705D2E9 (kernel32): (filename not available): BaseThreadInitThunk 
[08.04. - 17:34:21] ERROR: SymGetLineFromAddr64, GetLastError: 487 (Address: 771C1603) 
[08.04. - 17:34:21] 771C1603 (ntdll): (filename not available): RtlInitializeExceptionChain 
[08.04. - 17:34:21] ERROR: SymGetLineFromAddr64, GetLastError: 487 (Address: 771C15D6) 
[08.04. - 17:34:21] 771C15D6 (ntdll): (filename not available): RtlInitializeExceptionChain

[lerxst2112]

Zone is shutting down and a string in NPCType is getting deleted in the NPCType destructor, but the memory it points to is bogus. I'd look for someone using memset on NPCType which is a no no since it's not a POD type. Anywhere you're tempted to use memset, use a default constructor instead, so much safer.

Did you compile for Debug specifically or RelWithDebInfo? Asserts and debug memory validation should be turned off in a release build which would keep the error message from popping up. It might still crash on shutdown, but it would do so silently and not wedge the server with a bunch of stuck processes.

[fwaits]

Thanks for replying. I actually don't remember compiling with debug active to be honest, but I'm pretty new at this. At what step of building/compiling would that be defined and I'll go back through my steps and check for it? And for the record, this is a stock build of the emulator, not doing anything custom.

[fwaits]

I guess I mean is that defined during the CMake portion or the Visual Studio compile? Perhaps both? Any help is appreciated, thanks.

[KLS]

Thought this bug was fixed on the 18th.

[fwaits]

I think I built this a little before July 18th if that's the date you are referring to.

[KLS]

Aye that makes sense, this was a bug caused by the aforementioned memsets. I pushed a workaround to the bug to the source on the 18th that fixes the crash.

[fwaits]

So is the optimal standard to build the Source in VS with Debug or RelWithDebInfo? I notice it defaults to Debug.

[lerxst2112]

RelWithDebInfo is the best choice unless you're actually trying to use the debugger to debug something. It gives you the best performance and a stack trace to post if something crashes.

[fwaits]

Thanks for the help, I did go ahead and rebuild the source today under RelWithDebInfo mode and got the server running again. (Same DB and such since it was still the same rev) I have not seen any of these assertion errors yet, and I usually saw at least 1 or 2 within an hour or 2 of it running, so it looks like that may have done it. Thanks lerxst2112 and KLS for your input, it was very helpful.