Dev Blog: New Server Config Loader (Json)

[Akkadius]

Thanks to the work of Shin Noir (Xackery) from RebuildEQ, we've converted our server config loader from XML to JSON.

This is now live, whenever new binaries are used, world.exe will automatically pull down a config converter before attempting to load the config. This looks something like this:

https://i.imgur.com/yobyMNg.png

World will use the new eqemu_config.json effective immediately. All other utilities that were built around loading config variables from XML will have to be adjusted or updated to use the JSON format.

A backup of the old XML config will be located in serverdir/backups/

• The Perl MySQL plugin has been updated already: see here
• Linux Installer now uses the new .json format
• Windows Installer now uses the new .json format
• DiscordEQ OOC chat relay built by Shin should have a new release ready for this change: see here

If you find any tools or utilities tied to the old XML format that needs to be changed or adjusted, let us know in this thread and we will make sure that things get adjusted appropriately.

[The_Beast]

Sorry for an annoying question, lol. I was just curious about the future. If one was using old binaries, (for example, compiled last week),
is anything planned for the future that would interfere with older binaries patching into the public eqemu login server ?
This whole .json thing is something I need to learn about, so just checking for future reference

[The_Beast]

BUG ??
When I ran new binaries, it got this far and just hung there. But in my server directory, I noticed it kept deleting the
eqemu_server.pl file and replacing it. Just kept doing that continuously, (looped) It was just a test box anyways.
I also tried just running the script by itself (without binaries) and got the same thing.

Pulling down EQEmu Server Maintenance Script (eqemu_server.pl)...
[Update] Script has been updated, updating...
[Install] Installing :: eqemu_server.pl
[Update] No script update necessary...

EDIT - The "new" script works fine with old binaries, but I also noticed, from trying to run it before, it never did
update anything related to json conversion. (no config or anything). I tried both my own compiled new binaries
and the precompiled downloaded ones. Couldn't get past the "hanging" with any new binaries.

[The_Beast]

I grabbed the "fixed" eqemu_server.pl from gitpull today and works like a charm. Thank You

[Akkadius]

Originally Posted by The_Beast

I grabbed the "fixed" eqemu_server.pl from gitpull today and works like a charm. Thank You

Glad you got it sorted - however if that is happening to other folks I will need to iron out whatever was causing it

No worries

[The_Beast]

Smee again :P
Seems to have downloaded, yet, a newer script, but it's spitting out errors. It doesn't do the json conversion for xml
and oddly enough, it's setting a db version for bots, which I don't even have enabled, lol It set the version at 0 in DB.
This is all on a test box that I'm using to help work out bugs, so I can freely wipe the server and redo anytime.

[Update] Script has been updated, updating...
[Install] Installing :: eqemu_server.pl
[Update] No script update necessary...
cannot open file  at eqemu_server.pl line 326. <---
[Update] Bots database up to Date: Continuing World Bootup...
Invalid command 'start_from_world'
[Install] Done
cannot open file  at eqemu_server.pl line 326. <---
[Update] Bots database up to Date: Continuing World Bootup...

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>> EQEmu Server Main Menu >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

 [database]     Enter database management menu
 [assets]       Manage server assets
 [new_server]   New folder EQEmu/PEQ install - Assumes MySQL/Perl installed
 [setup_bots]   Enables bots on server - builds code and database requirements

 exit

Enter a command #> database

>>> Database Menu

 [backup_database]              Back up database to backups/ directory
 [backup_player_tables]         Back up player tables to backups/ directory
 [backup_database_compressed]   Back up database compressed to backups/ director
y

 [check_db_updates]             Checks for database updates manually
 [check_bot_db_updates]         Checks for bot database updates

 [aa_tables]                    Downloads and installs clean slate AA data from
PEQ
 [remove_duplicate_rules]       Removes duplicate rules from rule_values table
 [drop_bots_db_schema]          Removes bot database schema

> main - go back to main menu
Enter a command #> check_db_updates
[Database] Retrieving latest database manifest...
[Download] Saved: (db_update/db_update_manifest.txt) from https://raw.githubuser
content.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt
[Database] Reading manifest...

[Database] Retrieving latest database manifest...
[Download] Saved: (db_update/db_update_manifest.txt) from https://raw.githubuser
content.com/EQEmu/Server/master/utils/sql/db_update_manifest.txt

[Database] No updates need to be run...
[Database] Setting Database to Binary Version ((xmltojson.exe) from https://raw.
githubusercontent.com/EQEmu/Server/master/utils/xmltojson/xmltojson-windows-x86.
exe
Converting eqemu_config.xml to eqemu_config.json
Server element not found <----
Binary Database Version) if not already...

ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the ma
nual that corresponds to your MySQL server version for the right syntax to use n
ear 'from https://raw.githubusercontent.com/EQEmu/Server/master/utils/xmltojson/
xmlto' at line 1

[Akkadius]

I just ran a conversion using the latest script and latest binaries and it converted just fine, not sure how you're running into that issue.

Are you running your server binaries within the context of the server directory?

Are you doing something different?

[Darkscis]

I can confirm I also just ran the latest script on a virtual machine that hasn't even been turned on for 6 months. It converted the .xml to .json without issue.

[Shin Noir]

Originally Posted by The_Beast

Smee again :P
Seems to have downloaded, yet, a newer script, but it's spitting out errors. It doesn't do the json conversion for xml
and oddly enough, it's setting a db version for bots, which I don't even have enabled, lol It set the version at 0 in DB.
This is all on a test box that I'm using to help work out bugs, so I can freely wipe the server and redo anytime.

Look at your eqemu_config file, and check the top line.
It is saying it can't find the server element..
See if server is there. Is it by chance not all lowercase ? e.g. Server?

[The_Beast]

In reference my other post (see "works like a charm"), the whole server was working perfect without issues. Then tonight, all I did was
run the eqemu_server.pl script (in the server directory), and wanted to check_db_updates. But when I ran the script, it pulled an updated
version and got those errors shown. World won't even run now, so have to find out why. But as I said, I did nothing except run the script
for db updates, like I've been doing for a long time now.

Just to add: I've been trying to get it to convert the config over to json.(again). I have 3 different configs setup on this one box, for test.
One for local, one for lan, and one for public. I was simply trying to get them all (one by one) converted over to json. But deleting the json
config and putting one of those old configs in, it won't convert them now.

Originally Posted by Akkadius

whenever new binaries are used, world.exe will automatically pull down a config converter before attempting to load the config.

This is the reason I "assumed", if I deleted the json config, replace it with another older xml config, it would
convert it over when world was run. Guess I assumed wrong, lol

[The_Beast]

Here, just to simplify what I am talking about. I put new binaries in the folder. There is an eqemu_config.xml file in there.
I went into folder with a cmd window and run world and see what happens. Even though it "implies" that it is converting
the xml over to json, it doesn't do it. This is all that happens :

D:\eqemutest>world
Pulling down EQEmu Server Maintenance Script (eqemu_server.pl)...
[Download] Saved: (xmltojson.exe) from https://raw.githubusercontent.com/EQEmu/S
erver/master/utils/xmltojson/xmltojson-windows-x86.exe
Converting eqemu_config.xml to eqemu_config.json
Server element not found
cannot open file  at eqemu_server.pl line 326.
[World Server] Loading server configuration..
Error from reader: * Line 1, Column 1
  Syntax error: value, object or array expected.
[World Server] Loading server configuration failed.

For the moment, I can't figure it out yet. My coffee is getting cold, lol

[Akkadius]

The_Beast - see what Shin posted above

Looks like your XML is malformed or non-standard

If you can paste it to us via pastebin or otherwords with sensitive info redacted we could help point you in a direction

[The_Beast]

Hahaha, this funny. First of all Shin, sorry, when you said check top line, I thought you meant the json config. But just found out
if your xml config has "special characters" in the server name, the conversion won't work. I removed them and problem solved.

[Akkadius]

Originally Posted by The_Beast

Hahaha, this funny. First of all Shin, sorry, when you said check top line, I thought you meant the json config. But just found out
if your xml config has "special characters" in the server name, the conversion won't work. I removed them and problem solved.

Glad you got it sorted!

[Vexyl]

Interesting, but why not just use Lua for config since the server is already being linked against it?