Sourcing Cave Dude's Database into PEQ
 

I would imagine this has been thought of already, but is there any way to source certain tables, like the npc_types table into PEQ's without messing everything up or no? Seems the tables have incredibly similiar structure, so with a little tweaking to make them the same, a mass "source everything that is in cavedude's that isn't in PEQ's" type function would insert all the missing npc info to at least have a basis to start from, but i notice the npcid's themselves are not necessarily the same for the same mobs in cave dudes vs. PEQ's ... is there any way around that or is that pretty much an insurmountable barrier?

you would need to go through the DB NPC by NPC.

Was giving this some thought, and here is a revised method...

would it work to maybe eliminate duplications where the mob 'names' are the same, thus eliminating the duplicate mobs from both tables (so i don't have 2 versions of the same gnoll from blackburrow)

After i've eliminated the duplicate entries to ensure that i'm only getting the newer npc's rather than duplications of the old, i could do a run through the tables, picking out when cavedude's npc and PEQ's npcs have the same npcid, then change cavedude's npcid to a unique one...

Finally, source info from the revised cavedude's npc_type table into the PEQ one giving me at least all the information that was lacking in terms of having at least the mobs availible to import

Furthermore i guess in theory you could change the npcid's for all the relevant spawn tables (npc_types, spawn2, spawngroup, spawnentry... am i missing some?) so that they would transfer over with their new unique id's and be able to spawn in the world, still without faction or loot but thats the next task...

So, as a newb programer anyone that could point out a hole to this theory so i could attempt to work on it s'more, it would be appreciated, right now i'm trying to figure out why i can't seem to successfully backup my database :)

They can be merged together. So basically you would probably want the zones that are populated in cavedudes DB after luclin (PEQ has luclin fully spawned already). You will have to go through and Dump and sort out lootdrop ,lootdrop_entries ,loottable,loottable_entries,npc_types,spawn2,spaw nentry,spawngroup,grid, and grid_entries table. You will have to remove all spawns and entries from luclin and prior from all those tables from cavedudes DB. Once you remove all those Entries you can just source in the left over info you have from the dumps you made of cavedudes DB into PEQ DB and you now have a merged PEQ/cavedudes DB. This is the way i did it and it is very tedious and may not be very future proof to when PEQ releases there next DB.

After searching the planet (and this site and affiliates) for that elusive "combined database", I've decided one of the best ways I can learn the Emulator DB structure was to write my own scripts for merging the two. So far, my eyes have started bleeding only a dozen times, because the auto_increments used to _fk some of the tables together are all over the place (I think that was mentioned elsewhere).

If I get my scripts done, I'll link them here. I work in ASP/VBScript and have no intention of writing other versions. Another thing I could do is post the sources my script generates once I am done - but I really think this has been done 100,000 times before "me", and I am just amazed it is unavailable anywhere.

I'll let you know how it turns out. Cuz right now, I have no idea if I am screwing up spawn and loot drops to all PEQ zones just by doing simple insert/replace. :) Probably. We shall see. Meanwhile, I am learning tons, and it's very exciting!

I am easily amused.


PS: I remember reading somewhere about a standard ID scheme of zone * 1000 for entries. This is what I was hoping to end up with by the time I am done renumbering everything. Ambitious, no?

Now, just out of curiosity as i have experienced in a brief run-through it looked like cavedude's database is lacking any faction or loot for anything beyond luclin anyway... and if mobs are factionized/itemized it's likely not enough mobs for me to particularly care about... so this means that 'in theory' all i'd need in order to have the mobs be created and spawn in front of me, would be to source over the npc_types, spawn2, spawnentry, spawngroup, grid, and grid_entries tables right? i don't really care about doors and whatnot either in terms of the 'general zone specifics' mentioned in another similiar post...

As a side note thanks to the great tool by Zephyr325 I've managed to already get the mobs that need to be sourced in listed (need to change all their id's which will be a pain with 2500+ mobs but oh well) and everything i need for spawn2, and spawngroup, are all ready to be sourced once their id's are re-written so things look plausible even though the id switching will likely make me go blind and take forever :)

Won't let me edit previous post so sorry for the double post...


I've managed to finish doing a patch to;

the id and loottable id (just in case) fields of npc_types
the npcid (to match npc_types id field) of spawnentry
the id and spawngroupid fields of spawn2

i tried to change the spawngroup id field and kept getting duplicate field error no matter what i tried to change it to, so i'm not too sure how to go about avoiding that problem, but thats another day

My real question is, i looked into changing id's in grid and grid_entries but i can't find any description of what their fields mean, the wikki is un-written and i didn't see any posts describing their function, so is there any id fields that i would need to change to make them unique against the peq fields?

btw i edited all the above by just multiplying their respective id fields by 1000, putting them theoretically well beyond the id reach of peq's future patches... so it should work fairly well unless i'm missing something that will crash it, which i probably did, and it probably will, but wheres the fun if it didn't right? :)

and i'm done... i got the spawn points in, and i even did the faction tables to see if it helped (not sure they sourced correctly, i got no errors but not sure the correct faction is to the correct mob) but anyway, if anyone wants the set of insert statements to at least populate all teh zones feel free to let me know and i'll try to get someone to host the files for me...

I for one would love to have them if you can find a host to serve them up. I'm assuming it would be too long to just insert as a code block in a post?

I think the problem may be bigger than we think...I did a pretty deep dive into the records.

There are 27,813 unique NPC ID's between the Cavedude and PEQ databases.
- Cave dude has 20,195 npc ID's; PEQ has 23,122 NPC ID's.

Of those:
- 8,482 of those NPC ID's and NPC names match up <- This is good
- 12,305 of those are unique (i.e., its in Cavedude but not in PEQ, or vice versa)
- 7,024 NPC ID's match up, but they have different NPC names. <- This is not good

In other words, there are over 7000 NPC's records that would fail if we tried to bring them over because their NPC_ID number is already being used by the other database.

Ok, so maybe we look at doing things by the NPC names, rather than their id:

- There are a total of 16,095 unique NPC names between the two databases.
- Cavedude has 12,043 unique npc names. PEQ has 13,349 unique names.
- There are 9,297 names that match each other between the two databases
- There are 6,798 names that are in one of the databases, but not the other.

So there are almost 6800 differences in the NPC names.

I'm really thinking that the NPC ID's is the way we should approach it. Otherwise, all the "a_decaying_skeleton" records would just show up as one NPC.

I'm ready to go after this to synchronize them, but my head is swimming trying to figure out the best way to do it.

Heres what i did...

I pulled up a list of NPC names in cavedude that wasn't in peq... pulled those out into a list

Then i ran a program to take every line of the insert statement from dumping cavedude's database where the name from the above list showed up, giving me several insert statements per name at times, all of which were not in the peq database...

Then i adjested the npcid's to make them larger than peq would ever use, so that there was no chance of a duplicate id entry forming when i sourced it over, nor would there be any chance of a future peq update overwriting the id that i assigned to these mobs... so it is (theoretically) 'update proof'

Thats how i got the npc_types table to work in terms of the npc's ids versus name...

I'm sure i missed some, it's not perfect, it's not pretty, but i gotta say i'm wandering around plane of valor right now on my peq database and looking at all the spawns that weren't there a minute ago ;)

I looked around and found a free hosting site for a temporary host until someone else can host it for themselves, here's the link;


http://upload.ohshare.com/v/8413635/...o_PEQ.zip.html

Note, you do NOT need any of cavedude's database on your machine to use this, they are straight insert statements designed to be run by mysql to insert into peq database... there are linebreaks every 80 or so characters in order to allow them to be straight copy/pasted into the mysql command line if you wish to do so (note some of these files are pretty long, i've found trying to paste in more than about 25,000 lines of code at a time can cause the command prompt to crash, so paste in sections)

Enjoy everyone, hope it helps some of you out, if nothing else at least it's a starting point :)

Thanks alot!

This adds 6648 spawns in 46 zones. :)

awk -F"'" '{print $2}' spawngroupnew.sql | awk -F"_|[0-9]" '{print $1}' | sort | uniq -c| sort -n
4 shadowrest
15 hohonorb
46 hohonora
49 mmcj
50 takd
51 pofire
53 tipt
54 poair
54 ruji
55 potorment
55 solrotower
60 qinimi
62 bothunder
64 natimbi
64 potranquility
74 povalor
82 potactics
89 veksar
104 codecay
112 postorms
117 harbingers
119 nedaria
125 hatesfury
128 powater
131 chardokb
132 riwwi
150 bloodfields
154 vxed
155 hateplaneb
156 qvic
187 barindu
189 poinnovation
200 potimea
209 podisease
218 draniksscar
229 pojustice
229 wallofslaughter
232 abysmal
233 ferubi
250 poeartha
255 poknowledge
258 provinggrounds
287 causeway
310 dranik
376 kodtaz
422 ponightmare


Just one question.

Why do formerly not populated zones do have spawngroup entrys like nedaria_63 and additions to PEQ zones entrys like podisease1640021? Is the _ missing or redundant?

Alot of the spawnentries and grid info i did as educated guesswork... like i said it's not perfect, just a point to jump from... if anyone feels so ambitious i'd recommend going through them more meticulously to find which ones specifically are needed and which ones aren't for these tables and clean up those two files a bit more than what i did...

Not quite sure if that answers your question or not, if not, pm me with more specifics

Hi, and thank you for your great work - this was very needed. If you need a space I can fix you up on my webserver, or I can post your stuff un my UGLY web page (ugly, but works). PM me with what you want.
I have a question; Are you using eight digit numbers? I'm asking, becuause I am using the eight digit numbers for my up dates to the zones, and don't want them to cross with yours.

a bunch of them are 8 digit numbers, so you'd have to check those... most of them are 9 digit numbers however i believe, you can always use a 'where id > 100000000' clause to source in most of them then peel through the rest...

all mine end in 000 though since it was a straight *1000 so that should make looking through the id's easier for you.

You might want to check out nine digit numbers - I had problems to where my EQEMU wouldn't load the database when they were nine digit.

How about generating the IDs with an fixed offset instead of an multiplier?
EDIT: On second though I dont think its that easy, you might have to discard the old IDs alltogether and give new IDs, counting upwards from the offset.

Be aware that I did not read through the whole documentation yet, so the below could be moot:
Maybe it would be nice to have some conventions what ID ranges to use for certain stuff. Eg. Half the usable range for PEQ itself. A well sized chunk for imports from Cavedudes and other databases(are there any?). And the rest for local server customizations.
I recon it would help server admins to make sure that their custom additions dont clash with further updates of the PEQ databases.

Where do I put the files I downloaded from the one post? And how do I get it to source in properly?

If I were you, I'd wait tell Zengez replies to my "nine digit" comment - You're database may not load after you do this.

I'll be testing it as is tonight because this is very exciting! but of course I have my current database saved off incase I need to roll back.

Yea, i had that problem originally because the database only supports some of the upper end ranges because of the size limitation of the field, but it won't tell you you've reached the max until you've already hit it, then try to go higher a second time...

basically if you multiply by 1000, then do so again, half yer id's turn out to be the same number which won't let anything load, but it won't error until you try to multiply/divide it 'again' once they are all the same number... which is frustrating...

Anyway i ran into this problem by accident, figured out what the problem was, then resolved it... so (in my case anyway) everything loads just fine and i didn't break anything... I can't imagine anyone else having this problem if i didn't because i haven't actually customized my server very much... but I make no promisses as i don't know other people's code... my recommendation; dump your database files before you source these in, then if it kills something do a kill on the table and resource in your dumped original version...


As to how to source it, i just did a straight copy/paste from the text files into the mysql prompt in order to put it in, but remember not to do any more than 20-25k lines of code at a time or you may crash the prompt (won't destroy your database or anything, just crashed the prompt)

This worked fine for me. No problems loading the database and all the zones previously empty have caveman's spawns in them.

Everyone has their own way of doing things, but starting out with the base of what the designers intended is always a good practice. Any work (or modifications to existing work) that I do will follow this basic format:

http://www.eqemulator.net/forums/sho...12&postcount=7

The only problem I really have with this is the merge confusion - which is not a problem to the PEQ folks, since that is not their focus - it's our, individual desire to have more content immediately that is driving this effort again. If you use REPLACE INTO, you run the risk of blowing up LOTS of PEQ data. If you use INSERT INTO, you'll bump into existing (unique) records and lose Cavedude's data. I see some tables with auto increment values, and others where there are none. The confusion for me lies in, say, Zone ID 202 having npc ID 202000, in spawn group <autoinc, so it could be anything> using loot table <auto, anything> with loot drops <auto, anything> etc... see where I'm going? Sure it cuts down redundancy to NOT tie npc loot, faction, or spawn id's to zone id's (* 1000) but at the same time, it becomes hell to manage.

As a learning exercise, this is where I am focusing my attention right now... learning the intertwinings of the database and seeing if I can make things make more sense to me. I doubt my effort will become an acepted norm, but at least I will understand what's going on when I am through.