Go Back   EQEmulator Home > EQEmulator Forums > Development > Development::Tools

Development::Tools 3rd Party Tools for EQEMu (DB management tools, front ends, etc...)

Reply
 
Thread Tools Display Modes
  #1  
Old 09-04-2006, 07:00 AM
zephyr325
Hill Giant
 
Join Date: Sep 2004
Posts: 117
Default New Tool: Migrate NPC's (spells/faction/loot/spawn/move/items) from PEQ <=> Cavedude

This post supersedes the one I posted here.

There are several enhancements to this program over the old one, but here's a full feature list. You can:

- Move NPC's from PEQ -> Cavedude, or Cavedude -> PEQ
- It will grab all pertinent info on the NPC, including spells, faction, spawn, loot tables, loot items and movement.
- You can move a single NPC, a list of NPC ID's, or a whole zone at a time.
- For a zone move, you can tell it to pull just the NPC's in the "From" database that the "To" database doesn't already have. You can also tell it to grab all of them.
- Output can be one file per NPC (i.e., 1001-Guard_Mezzt.sql) or one big file for all NPC's.
- If you tell it to, it will adjust the key fields for all the pertinent tables so the two databases will play nicely together. This should probably be used every time you run the program.
- I've done a lot of checking into what are safe values to increment key field ID's are (information is here). There is a section in the program where you can change these increments if you so desire.


Prerequisites:
- FIRST: THIS IS TEST CODE. BACKUP YOUR EXISTING EQ DATABASE.
- This is a perl program; you need perl installed.
- You will need to source both databases into mysql; this program will read from both of them. Make sure to keep them seperate (don't source them into the same database).


Installing it:
- Make sure you
__________________
"Like what you like, enjoy what you enjoy, and don't take crap from anybody."
Reply With Quote
  #2  
Old 09-04-2006, 07:06 AM
zephyr325
Hill Giant
 
Join Date: Sep 2004
Posts: 117
Default

(bumping this thread - the old post I had got on top of this when I edited it to point to this new post)
__________________
"Like what you like, enjoy what you enjoy, and don't take crap from anybody."
Reply With Quote
  #3  
Old 09-04-2006, 09:04 AM
Angelox
AX Classic Developer
 
Join Date: May 2006
Location: filler
Posts: 2,049
Default

The stuff I put out is probably only good if you have a new database, or are downloading my whole database.
If you already have a database you are working on and have added stuff, then there's always a chance things will get crossed and mess up your database.
If you want to add my stuff to your custom database - this tool is what you need; just download my database and merge what you want to yours. This is a great tool and long-waited for.
Thank you zephyr325 !!!
Reply With Quote
  #4  
Old 09-04-2006, 10:31 AM
zephyr325
Hill Giant
 
Join Date: Sep 2004
Posts: 117
Default

Actually, you can use it with other databases as well. The key is making sure that you're incrementing your key field ID's by an amount that isn't going to step on something you already have. Once you have those values, there is a section in this program that you can edit (its towards the top of the file, and labeled) that will allow you to put those amounts in, rather than the defaults I put in.

Check out the link above as well for my listing of what I did with each table; that will be a good starting point for folks to use.

The only one I've found so far that has a hard cap on it is npc_faction - make sure you don't increment your id to be more than 50000 or EQEmu won't start up. I've got a safety in the program that won't write that insert statement out if that value is too high.
__________________
"Like what you like, enjoy what you enjoy, and don't take crap from anybody."

Last edited by zephyr325; 09-04-2006 at 06:35 PM..
Reply With Quote
  #5  
Old 09-05-2006, 03:40 PM
zephyr325
Hill Giant
 
Join Date: Sep 2004
Posts: 117
Default

Small updates:

- Changed the -e option to not check at all for duplicate records in the target database

- put double quotes around all field values to keep the insert statement from generating an error if a field value is null
__________________
"Like what you like, enjoy what you enjoy, and don't take crap from anybody."
Reply With Quote
  #6  
Old 09-05-2006, 03:49 PM
vales
Discordant
 
Join Date: May 2006
Posts: 458
Default

Wow, I don't know how I missed it. :p

Thanks for the program, zephyr325!
Reply With Quote
  #7  
Old 09-06-2006, 04:07 AM
zephyr325
Hill Giant
 
Join Date: Sep 2004
Posts: 117
Default

np!

Next revision is going to work on a less verbose option both for screen and output file. Less screen time means the program will run faster, and a smaller output file means a faster load time as well as not chewing up those oh-so-expensive hard drive bytes. =)
__________________
"Like what you like, enjoy what you enjoy, and don't take crap from anybody."
Reply With Quote
  #8  
Old 09-06-2006, 11:10 AM
John Adams
Demi-God
 
Join Date: Jul 2006
Posts: 1,552
Thumbs down

Zephyr, you are truly amazing. Thank you for this effort. It immediately simplifies anyone wanting to work adhoc on their own data and providing it to the whole community.

Gold star for you!
Reply With Quote
  #9  
Old 09-06-2006, 03:05 PM
zephyr325
Hill Giant
 
Join Date: Sep 2004
Posts: 117
Default

I've put a quick little scripts out that will run the program against all the zones I know about and create a "zonename.sql" file of inserts.

You can get to it here.

I run things under linux/cygwin, so if you want to use it in a DOS format you'll need to replace the "./" in front of each line with "perl " so you get a line like:

Quote:
perl grabnpc.pl -k -z abysmal -o abysmal.sql
Also, if you want to get one big file of inserts, this is how I do it under linux:

Quote:
for i in *.sql; do cat $i >>biginsertfile; done
If one of you DOS gurus know the translation for that, someone can probably use it. Make sure you don't create your "biginsertfile" to end with ".sql", though, or you'll get into a recursive loop. You get really big files, really fast, doing something like that. =)
__________________
"Like what you like, enjoy what you enjoy, and don't take crap from anybody."

Last edited by zephyr325; 09-06-2006 at 11:26 PM..
Reply With Quote
  #10  
Old 09-06-2006, 03:39 PM
fathernitwit
Developer
 
Join Date: Jul 2004
Posts: 773
Default

to follow on to the previous thread, making items with arbitrarily high IDs like that is really not a good idea... we have a static length array to make item management fast and big gaps in item IDs waste a lot of memory (if it works at all)
Reply With Quote
  #11  
Old 09-06-2006, 03:57 PM
WildcardX
Developer
 
Join Date: Apr 2003
Posts: 589
Default

I can vouch for FNW on this. When I made a database that was a combination of the latest PEQ, Cavedude's latest and Angelox's latest work, I saw my server memory increase by 300% the normal for running just 10 dynamic zones. Atleast, this is what I think is causing it. I'm still trying to prove it, or I was, until I just read FNW's post above...

On the other hand, it sure is nice to have a database that is almost all popped up intil Omens.
__________________
Read my developer notes at my blog.

Quote:
If it's not on IRC, it ain't l33t!

Last edited by WildcardX; 09-06-2006 at 11:59 PM..
Reply With Quote
  #12  
Old 09-06-2006, 04:14 PM
zephyr325
Hill Giant
 
Join Date: Sep 2004
Posts: 117
Default

Thanks for the input, both of 'ya.

The basic premise that I worked off of was trying to find a safe number that I could add to each of the id's that would work for both databases. There was a lot of room for optimization, which I've incorporated below (and updated the web page that shows what I do for each table here):

npc_types.id: increment by 350,000 rather than 500,000
spawngroup.id: increment by 300,000 rather than 3,000,000
spawn2.id: incremebt by 125,000 rather than 500,000
npc_spells.id: increment by 300 rather than 5,000
npc_spells_entries.id: increment by 1,800 rather than 5,000
loottable.id: increment by 13,000 rather than 500,000
lootdrop.id: increment by 50,000 rather than 500,000
faction_list.id: increment by 1,000 rather than 5,000
items.id: increment by 100,000 rather than 500,000

As to the "will it work" thing, I've been doing some pretty extensive testing and its all worked so far. Thanks for the input.
__________________
"Like what you like, enjoy what you enjoy, and don't take crap from anybody."
Reply With Quote
  #13  
Old 09-06-2006, 04:25 PM
WildcardX
Developer
 
Join Date: Apr 2003
Posts: 589
Default

I'll give this a try. I am sure it will work, but for performance I really hope decreasing the amount of empty IDs between records will greatly reduce the server's consumption of memory.
__________________
Read my developer notes at my blog.

Quote:
If it's not on IRC, it ain't l33t!
Reply With Quote
  #14  
Old 09-06-2006, 04:47 PM
zephyr325
Hill Giant
 
Join Date: Sep 2004
Posts: 117
Default

Fathernitwit - is there some way to tell how much memory would be taken up by empty space between key id's vs. adding a bunch of new spawns/items/loot etc?
__________________
"Like what you like, enjoy what you enjoy, and don't take crap from anybody."
Reply With Quote
  #15  
Old 09-07-2006, 01:46 PM
soulshot
Hill Giant
 
Join Date: Jun 2006
Posts: 142
Default

If anyone wants to see the tool results in action check out the darktides server where this is being developed. Now we are going to attempt a PEQ base with cavedude added into it.
Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

   

All times are GMT -4. The time now is 03:35 AM.


 

Everquest is a registered trademark of Daybreak Game Company LLC.
EQEmulator is not associated or affiliated in any way with Daybreak Game Company LLC.
Except where otherwise noted, this site is licensed under a Creative Commons License.
       
Powered by vBulletin®, Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Template by Bluepearl Design and vBulletin Templates - Ver3.3