Zone Crash (realloc)

Faydane · #1 08-10-2015, 08:51 AM

After years away, I'm trying to re-establish my own server. In the past, I used Windows XP, but since have migrated to GNU/Linux.

This build is on Mageia x86_64; both the client (RoF2) and virtualized server (built off the http://tinyurl.com/p458t64 guide). The SQL database is a separate VM (Maia), and both Everquest and Maia resolve to ip addresses via ping. At the moment, the server's firewall is totally disabled until I get the system working. The server is not wholly headless; it runs XFCE with a x11vnc over ssh backend for troubleshooting and server administration.

Possibly unrelated, but on the login screen, the character models are absent. Everyone looks like an Enchanter's animation (floating weapons and lanterns).

When I hit "Enter World", zone responds with the following (a form of segmentation fault):

Code:

*** Error in `./zone': realloc(): invalid next size: 0x0000000003b92250 ***

Zone log:

Code:

[33m[Zone Server] Loading server configuration..[0m
[33m[Zone Server] Connecting to MySQL...[0m
[33m[Status] Starting File Log 'logs/zone_18872.log'[0m
[33m[Zone Server] CURRENT_VERSION: 1.1.3[0m
[33m[Zone Server] Mapping Incoming Opcodes[0m
[33m[Zone Server] Loading Variables[0m
[33m[Zone Server] Loading zone names[0m
[33m[Zone Server] Loading items[0m
[33m[Zone Server] Loading npc faction lists[0m
[33m[Zone Server] Loading loot tables[0m
[33m[Zone Server] Loading skill caps[0m
[33m[Zone Server] Loading spells[0m
[33m[Zone Server] Loading base data[0m
[33m[Zone Server] Loading guilds[0m
[33m[Zone Server] Loading factions[0m
[33m[Zone Server] Loading titles[0m
[33m[Zone Server] Loading tributes[0m
[33m[Zone Server] Loading corpse timers[0m
[33m[Zone Server] Loading commands[0m
[33m[Zone Server] 250 commands loaded[0m
[33m[Zone Server] Loaded default rule set 'default'[0m
[33m[Zone Server] Loading quests[0m
[36m[Quests] Tying perl output to eqemu logs[0m
[36m[Quests] Creating EQEmuIO=HASH(0x2f5b568)[0m
[36m[Quests] Creating EQEmuIO=HASH(0x2f513d8)[0m
[36m[Quests] Loading perlemb plugins.[0m
[36m[Quests] Subroutine MakeTempPet redefined at (eval 43) line 1.[0m
[36m[Quests] Subroutine MakeTempPet redefined at (eval 43) line 1.[0m
[36m[Quests] Subroutine GetSkill redefined at (eval 43) line 1.[0m
[36m[Quests] Subroutine SignalClient redefined at (eval 43) line 1.[0m
[36m[Quests] Subroutine MakeTempPet redefined at (eval 43) line 1.[0m
[36m[Quests] Subroutine MakeTempPet redefined at (eval 43) line 1.[0m
[33m[Zone Server] Entering sleep mode[0m
[33m[Zone Server] Starting EQ Network server on port 7004[0m
[31m[Error] Failed to open port 7004[0m
[33m[Zone Server] Starting EQ Network server on port 7005[0m
[33m[Status] Booting tutorialb (189:0)[0m
[33m[Status] Loading spawn conditions...[0m
[33m[Status] Loading static zone points...[0m
[33m[Status] Loading spawn groups...[0m
[33m[Status] Loading spawn2 points...[0m
[33m[Status] Loading player corpses...[0m
[33m[Status] Loading traps...[0m
[33m[Status] Loading adventure flavor text...[0m
[33m[Status] Loading ground spawns...[0m
[33m[Status] Loading Ground Spawns from DB...[0m
[33m[Status] Loading World Objects from DB...[0m
[33m[Status] Loading Objects from DB...[0m
[33m[Status] Flushing old respawn timers...[0m
[33m[Status] Loading doors for tutorialb ...[0m
[33m[Status] Loading Doors from database...[0m
[33m[Status] Loading Alternate Advancement Data...[0m
[33m[Status] Loading Alternate Advancement Abilities...[0m
[33m[Status] Loaded 1567 Alternate Advancement Abilities[0m
[33m[Status] Loading Alternate Advancement Ability Ranks...[0m
[33m[Status] Loaded 6651 Alternate Advancement Ability Ranks[0m
[33m[Status] Loading Alternate Advancement Ability Rank Effects...[0m
[33m[Status] Loaded Alternate Advancement Ability Rank Effects[0m
[33m[Status] Loading Alternate Advancement Ability Rank Prereqs...[0m
[33m[Status] Loaded Alternate Advancement Ability Rank Prereqs[0m
[33m[Status] Processing Alternate Advancement Data...[0m
[33m[Status] Loaded Alternate Advancement Data[0m
[33m[Status] Loading Merchant Lists...[0m
[33m[Status] Loading Temporary Merchant Lists...[0m
[33m[Status] Successfully loaded Zone Config.[0m
[33m[Status] Loading timezone data...[0m
[33m[Status] Init Finished: ZoneID = 189, Time Offset = 0[0m
[33m[Status] Path File Header: Version 2, PathNodes 373[0m
[33m[Status] Path File ./Maps/tutorialb.path loaded.[0m
[33m[Normal] ---- Zone server tutorialb, listening on port:7005 ----[0m
[33m[Status] Zone Bootup: tutorialb (189: 0)[0m
[33m[Status] Starting File Log 'logs/tutorialb_version_0_inst_id_0_port_7005_18872.log'[0m
[33m[Zone Server] Time Broadcast Packet: EQTime [04:37 am][0m

world log:

Code:

[08-10-2015 :: 08:36:49] [World Server] HTTP world service disabled.
[08-10-2015 :: 08:36:49] [World Server] Checking Database Conversions..
[08-10-2015 :: 08:36:49] [World Server] Loading variables..
[08-10-2015 :: 08:36:49] [World Server] Loading zones..
[08-10-2015 :: 08:36:49] [World Server] Clearing groups..
[08-10-2015 :: 08:36:49] [World Server] Clearing raids..
[08-10-2015 :: 08:36:49] [World Server] Loading items..
[08-10-2015 :: 08:36:49] [World Server] Loading skill caps..
[08-10-2015 :: 08:36:49] [World Server] Loading guilds..
[08-10-2015 :: 08:36:49] [World Server] Loaded default rule set 'default'
[08-10-2015 :: 08:36:49] [World Server] Loading EQ time of day..
[08-10-2015 :: 08:36:49] [World Server] Loading launcher list..
[08-10-2015 :: 08:36:49] [World Server] Reboot zone modes OFF
[08-10-2015 :: 08:36:49] [World Server] Deleted 0 stale player corpses from database
[08-10-2015 :: 08:36:49] [World Server] Loading adventures...
[08-10-2015 :: 08:36:49] [World Server] Purging expired instances
[08-10-2015 :: 08:36:49] [World Server] Loading char create info...
[08-10-2015 :: 08:36:49] [World Server] Zone (TCP) listener started.
[08-10-2015 :: 08:36:49] [World Server] Client (UDP) listener started.
[08-10-2015 :: 08:36:51] [World Server] New TCP connection from 10.47.71.68:40909
[08-10-2015 :: 08:36:55] [World Server] New TCP connection from 10.47.71.68:40910
[08-10-2015 :: 08:37:31] [World Server] New TCP connection from 10.47.71.68:40911
[08-10-2015 :: 08:37:37] [World Server] New TCP connection from 10.47.71.68:40913
[08-10-2015 :: 08:37:56] [World Server] New TCP connection from 10.47.71.68:40915
[08-10-2015 :: 08:38:00] [World Server] New TCP connection from 10.47.71.68:40917
[08-10-2015 :: 08:38:02] [World Server] New TCP connection from 10.47.71.68:40919
[08-10-2015 :: 08:38:07] [World Server] New TCP connection from 10.47.71.68:40921
[08-10-2015 :: 08:38:17] [World Server] New TCP connection from 10.47.71.68:40922
[08-10-2015 :: 08:38:22] [World Server] New connection from 10.47.71.71:39667
[08-10-2015 :: 08:38:22] [World Server] Checking inbound connection 10.47.71.71 against BannedIPs table
[08-10-2015 :: 08:38:22] [World Server] Connection 10.47.71.71 PASSED banned IPs check. Processing connection.

eqemu_config.xml:

Code:

<?xml version="1.0"?>
<server>
	<world>
		<shortname>Campania</shortname>
		<longname>EverQuest Campania</longname>

		<!-- Only specify these two if you really think you need to.  (read: You don't) -->
		<!---<address>*********</address> --->
		<!---<localaddress>everquest</localaddress> --->

		<!-- Loginserver information.  Defaults shown -->
		<loginserver>
			<host>everquest</host>
			<port>5999</port>
			<account></account>
			<password></password>
		</loginserver>

		<!-- Server status.  Default is unlocked -->
		<!--<locked/>-->
		<!-- <unlocked/> -->

		<!-- Sets the ip/port for the tcp connections.  Both zones and console (if enabled).  Defaults are shown -->
		<tcp ip="everquest" port="9000" telnet="disable"/>

		<!-- Sets the shared key used by zone/launcher to connect to world -->
		<key>Latium_et_Imperium</key>

		<!-- Enable and set the port for the HTTP service.  Defaults are shown -->
		<http port="9080" enabled="false" mimefile="mime.types" />
	</world>

	<!-- Chatserver (channels) information.  Defaults shown -->
	<chatserver>
		<host>everquest</host>
		<port>7778</port>
	</chatserver>

	<!-- Mailserver (in-game mail) information.  Defaults shown -->
	<mailserver>
		<host>everquest</host>
		<port>7778</port>
	</mailserver>

	<zones>
		<defaultstatus>50</defaultstatus>

		<!-- Sets port range for world to use to auto configure zones -->
		<ports low="7000" high="7100"/>
	</zones>

	<!-- Database configuration, replaces db.ini.  Defaults shown -->
	<database>
		<host>Maia</host>
		<port>3306</port>
		<username>eqadmin</username>
		<password>*********</password>
		<db>peq</db>
	</database>

	<qsdatabase>
		<host>Maia</host>
		<port>3306</port>
		<username>eqadmin</username>
		<password>*********</password>
		<db>peq</db>
	</qsdatabase>

	<!-- Launcher Configuration -->
	<launcher>
		<!-- <logprefix>logs/zone-</logprefix> -->
		<!-- <logsuffix>.log</logsuffix> -->
		<!-- <exe>zone.exe or ./zone</exe> -->
		<!-- <timers restart="10000" reterminate="10000"> -->
	</launcher>

	<!-- File locations.  Defaults shown -->
	<files>
		<!-- <spells>spells_us.txt</spells> -->
		<!-- <opcodes>opcodes.conf</opcodes> -->
		<!-- <logsettings>log.ini</logsettings> -->
		<!-- <eqtime>eqtime.cfg</eqtime> -->
	</files>
	<!-- Directory locations.  Defaults shown -->
	<directories>
		<!-- <maps>Maps</maps> -->
		<!-- <quests>quests</quests> -->
		<!-- <plugins>plugins</plugins> -->
	</directories>
</server>

login.ini:

Code:

[database]
host = Maia
port = 3306
db = peq
user = eqadmin
password = *********
subsystem = MySQL

[options]
unregistered_allowed = TRUE
reject_duplicate_servers = FALSE
trace = TRUE
world_trace = FALSE
dump_packets_in = FALSE
dump_packets_out = FALSE
listen_port = 5999
local_network = 10.47.

[security]
plugin = EQEmuAuthCrypto
mode = 5

[Titanium]
port = 5998
opcodes = login_opcodes.conf

[SoD]
port = 5999
opcodes = login_opcodes_sod.conf

[schema]
account_table = tblLoginServerAccounts
world_registration_table = tblWorldServerRegistration
world_admin_registration_table = tblServerAdminRegistration
world_server_type_table = tblServerListType

GDB trace of zone (the ^C is manual; it hangs indefinitely without it):

Code:

[eqadmin@EverQuest server]$ gdb zone
GNU gdb (GDB) 7.8.1-7.mga5 (Mageia release 5)
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-mageia-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from zone...run
done.
(gdb) run
Starting program: /home/eqadmin/eqemu/server/zone 
Missing separate debuginfos, use: debuginfo-install glibc-2.20-20.mga5.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Zone Server] Loading server configuration..
[Zone Server] Connecting to MySQL...
[Status] Starting File Log 'logs/zone_19803.log'
[Zone Server] CURRENT_VERSION: 1.1.3
[Zone Server] Mapping Incoming Opcodes
[Zone Server] Loading Variables
[Zone Server] Loading zone names
[Zone Server] Loading items
[Zone Server] Loading npc faction lists
[Zone Server] Loading loot tables
[Zone Server] Loading skill caps
[Zone Server] Loading spells
[Zone Server] Loading base data
[Zone Server] Loading guilds
[Zone Server] Loading factions
[Zone Server] Loading titles
[Zone Server] Loading tributes
[Zone Server] Loading corpse timers
[Zone Server] Loading commands
[Zone Server] 250 commands loaded
[Zone Server] Loaded default rule set 'default'
[Zone Server] Loading quests
[Quests] Tying perl output to eqemu logs
[Quests] Creating EQEmuIO=HASH(0x121e380)
[Quests] Creating EQEmuIO=HASH(0x1230f88)
[Quests] Loading perlemb plugins.
[Quests] Subroutine MakeTempPet redefined at (eval 43) line 1.
[Quests] Subroutine MakeTempPet redefined at (eval 43) line 1.
[Quests] Subroutine GetSkill redefined at (eval 43) line 1.
[Quests] Subroutine SignalClient redefined at (eval 43) line 1.
[Quests] Subroutine MakeTempPet redefined at (eval 43) line 1.
[Quests] Subroutine MakeTempPet redefined at (eval 43) line 1.
[New Thread 0x7fffeb5ef700 (LWP 19807)]
[Zone Server] Entering sleep mode
[Zone Server] Starting EQ Network server on port 7006
[New Thread 0x7fffeac68700 (LWP 19808)]
[New Thread 0x7fffea467700 (LWP 19809)]
[Status] Booting tutorialb (189:0)
[Status] Loading spawn conditions...
[Status] Loading static zone points...
[Status] Loading spawn groups...
[Status] Loading spawn2 points...
[Status] Loading player corpses...
[Status] Loading traps...
[Status] Loading adventure flavor text...
[Status] Loading ground spawns...
[Status] Loading Ground Spawns from DB...
[Status] Loading World Objects from DB...
[Status] Loading Objects from DB...
[Status] Flushing old respawn timers...
[Status] Loading doors for tutorialb ...
[Status] Loading Doors from database...
[Status] Loading Alternate Advancement Data...
[Status] Loading Alternate Advancement Abilities...
[Status] Loaded 1567 Alternate Advancement Abilities
[Status] Loading Alternate Advancement Ability Ranks...
[Status] Loaded 6651 Alternate Advancement Ability Ranks
[Status] Loading Alternate Advancement Ability Rank Effects...
[Status] Loaded Alternate Advancement Ability Rank Effects
[Status] Loading Alternate Advancement Ability Rank Prereqs...
[Status] Loaded Alternate Advancement Ability Rank Prereqs
[Status] Processing Alternate Advancement Data...
[Status] Loaded Alternate Advancement Data
[Status] Loading Merchant Lists...
[Status] Loading Temporary Merchant Lists...
[Status] Successfully loaded Zone Config.
[Status] Loading timezone data...
[Status] Init Finished: ZoneID = 189, Time Offset = 0
[Status] Path File Header: Version 2, PathNodes 373
[Status] Path File ./Maps/tutorialb.path loaded.
[Normal] ---- Zone server tutorialb, listening on port:7006 ----
[Status] Zone Bootup: tutorialb (189: 0)
[Status] Starting File Log 'logs/tutorialb_version_0_inst_id_0_port_7006_19803.log'
[Zone Server] Time Broadcast Packet: EQTime [08:23 am]
*** Error in `/home/eqadmin/eqemu/server/zone': realloc(): invalid next size: 0x0000000001e17260 ***
^C
Program received signal SIGINT, Interrupt.
0x00007ffff5f6dbdb in __lll_lock_wait_private () from /lib64/libc.so.6
Missing separate debuginfos, use: 
debuginfo-install 
lib64lua5.2-5.2.3-6.mga5.x86_64 
lib64mariadb18-10.0.20-1.mga5.x86_64 
lib64ncurses5-5.9-21.mga5.x86_64 
lib64ncursesw5-5.9-21.mga5.x86_64 
lib64nss-mdns2-0.10-15.mga5.x86_64 
lib64openssl1.0.0-1.0.2d-1.mga5.x86_64 
lib64readline6-6.3-6.mga5.x86_64 
lib64zlib1-1.2.8-7.mga5.x86_64 
libgcc1-4.9.2-4.mga5.x86_64 
ibstdc++6-4.9.2-4.mga5.x86_64 
perl-base-5.20.1-8.mga5.x86_64
(gdb) bt
#0  0x00007ffff5f6dbdb in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007ffff5ee64e5 in malloc () from /lib64/libc.so.6
#2  0x00007ffff7dea2ae in _dl_map_object_deps () from /lib64/ld-linux-x86-64.so.2
#3  0x00007ffff7df0100 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff7dec124 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#5  0x00007ffff7defb73 in _dl_open () from /lib64/ld-linux-x86-64.so.2
#6  0x00007ffff5f9abcd in do_dlopen () from /lib64/libc.so.6
#7  0x00007ffff7dec124 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#8  0x00007ffff5f9ac84 in __libc_dlopen_mode () from /lib64/libc.so.6
#9  0x00007ffff5f6e475 in init () from /lib64/libc.so.6
#10 0x00007ffff6a52ae0 in pthread_once () from /lib64/libpthread.so.0
#11 0x00007ffff5f6e58c in backtrace () from /lib64/libc.so.6
#12 0x00007ffff5e88a9a in backtrace_and_maps () from /lib64/libc.so.6
#13 0x00007ffff5edb41e in __libc_message () from /lib64/libc.so.6
#14 0x00007ffff5ee5ba0 in _int_realloc () from /lib64/libc.so.6
#15 0x00007ffff5ee6a7e in realloc () from /lib64/libc.so.6
#16 0x00007ffff707dee2 in luaM_realloc_ () from /lib64/liblua.so.5.2
#17 0x00007ffff70793a9 in luaD_reallocstack () from /lib64/liblua.so.5.2
#18 0x00007ffff7079ac9 in luaD_precall () from /lib64/liblua.so.5.2
#19 0x00007ffff7079cd2 in luaD_call () from /lib64/liblua.so.5.2
#20 0x00007ffff707932c in luaD_rawrunprotected () from /lib64/liblua.so.5.2
#21 0x00007ffff7079f61 in luaD_pcall () from /lib64/liblua.so.5.2
#22 0x00007ffff70760a6 in lua_pcallk () from /lib64/liblua.so.5.2
---Type <return> to continue, or q <return> to quit---
#23 0x0000000000919d8d in LuaParser::LoadScript (this=this@entry=0x10f0240, filename="quests/global/global_player.lua", 
    package_name="global_player") at /home/eqadmin/eqemu/source/Server/zone/lua_parser.cpp:937
#24 0x000000000091a6a7 in LuaParser::LoadGlobalPlayerScript (this=0x10f0240, filename="quests/global/global_player.lua")
    at /home/eqadmin/eqemu/source/Server/zone/lua_parser.cpp:748
#25 0x0000000000a5359a in QuestParserCollection::EventPlayerGlobal (this=this@entry=0x1133a10, 
    evt=evt@entry=EVENT_UNHANDLED_OPCODE, client=client@entry=0x1e0a280, data="", extra_data=extra_data@entry=1, 
    extra_pointers=extra_pointers@entry=0x7fffffffd930)
    at /home/eqadmin/eqemu/source/Server/zone/quest_parser_collection.cpp:342
#26 0x0000000000a5381f in QuestParserCollection::EventPlayer (this=0x1133a10, evt=evt@entry=EVENT_UNHANDLED_OPCODE, 
    client=client@entry=0x1e0a280, data="", extra_data=extra_data@entry=1, 
    extra_pointers=extra_pointers@entry=0x7fffffffd930)
    at /home/eqadmin/eqemu/source/Server/zone/quest_parser_collection.cpp:302
#27 0x00000000006fcf44 in Client::HandlePacket (this=this@entry=0x1e0a280, app=app@entry=0x7fffdc003c00)
    at /home/eqadmin/eqemu/source/Server/zone/client_packet.cpp:437
#28 0x000000000071841c in Client::Process (this=0x1e0a280) at /home/eqadmin/eqemu/source/Server/zone/client_process.cpp:609
#29 0x000000000079541c in EntityList::MobProcess (this=0x7ffff6216cc0 <main_arena>, this@entry=0x1011f00 <entity_list>)
    at /home/eqadmin/eqemu/source/Server/zone/entity.cpp:481
#30 0x0000000000632598 in main (argc=<optimized out>, argv=<optimized out>)
    at /home/eqadmin/eqemu/source/Server/zone/net.cpp:466

Any insight at all into what's going awry would be greatly appreciated.

Thank you in advance,
-Faydane Grace
Long lost 60 Human Monk from Erollisi Marr

Faydane · #2 01-26-2016, 03:30 PM

The quest "quests/global/global_player.lua" was crashing the LUA parser. Removing it eliminated the crash.

The missing models were due to not having the old, original EQ files. I reinstalled from the 1999 discs, then copied my new install over the old, and the server now works as intended.

This line held the key to unraveling the fubar:

Code:

#23 0x0000000000919d8d in LuaParser::LoadScript (this=this@entry=0x10f0240, filename="quests/global/global_player.lua", 
    package_name="global_player") at /home/eqadmin/eqemu/source/Server/zone/lua_parser.cpp:937

tigurius · #3 02-24-2016, 09:19 PM

Thanks for posting the solution! Just moved my server to linux and ran into this same problem. If I figure out what the problem is with that lua file I'll post a fix.

tigurius · #4 02-25-2016, 09:52 PM

Took some time with this today. So its not the script itself but the lua parsing code that is throwing the error. I'm on OpenSuse 13.2 which came with Lua 5.2, so I manually installed Lua 5.1.4 (same ver as I was on for Win server) and swapped the symlinks so its used instead. Still no good, still crash on zones where lua is used. (Like I was fine removing global_player.lua and loading up in OT but soon as I zoned to POK i crashed) I'll see if i can dig into the parsing code a bit tomorrow to see why its throwing exceptions.

**Uleat** · #5 02-25-2016, 10:13 PM

If you don't mind, post your stack dump as well.

The more info, and eyes, we have looking at this, the quicker we may be able to fix it - or at least find the cause

provocating · #6 02-26-2016, 12:05 AM

If you are familiar with gdb, bt and such, post a dump.

I have some crappy notes I keep around...

Code:

./gdb --args zone
run

Code:

./gdb world
run

When you get a crash dump run bt after

tigurius · #7 02-26-2016, 05:22 PM

Ok found a solution-

Turns out the system originally had Lua 5.2 like I posted before. I did not however "re-build" with Lua 5.1 after installing it. So I built a new source dir and did a fresh git clone, did a new build with Lua 5.1 as the default (I also added Luajit to the system for measure). Build thankfully succeeded first try. Then just moved over the binaries with the new libluabind.a and libcommon.a to my server directory and everything is working good now. I can zone into PoK which has plenty of lua scripting and no crashes.

Tigurius

**Uleat** · #8 02-26-2016, 06:09 PM

Thanks for posting that solution!