EQBuilder 2.0

[trevius]

Let me know if you guys can get these to compile. If I am right and all that is needed is updated Struct/Opcodes, that should be considerably easy to do using ShowEQ source.

Since I am pretty busy with SoF still, I don't have much time to work on this just yet. But, if anyone wants to test out if my theory on how to fix them is correct, I would gladly spend a few minutes to update something easy like Doors so you guys could see if it then works to collect doors. If that doesn't work, then there is probably more to it than I am thinking. If it does work though, it should verify that we could have these great tools up and running again in no time. Really, using the ShowEQ source, almost anyone could do it. They keep opcodes that we would want for this in the /conf/zone_opcodes.xml file and the packet structures are in the /src/everquest.h file. ShowEQ and EQEmu use almost the exact same structure setup, so it will almost be just a copy and paste. Other than that, you may need to compare the Titanium struct/field names with the struct/field names from ShowEQ and make sure that they match. For the ones that don't match, you just need to use your best guess as to which Titanium field would be in place of the ShowEQ field. Most of those are very easy to tell though, because it is the difference of something like "Name" and "name" in most cases.

I will definitely take some time after SoF development settles down some and give these tools a shot. But, if anyone wants to have a try at it before then, feel free! I will definitely try to answer any questions anyone working on these might have. Though, I really can't help much with the actual compile. I suck at that stuff, lol. I would mostly just be able to help with Structs and Opcodes since I have been getting alot of experience with them lately :P

[Andrew80k]

I'm still having trouble getting the extractor to compile. There are some files missing from the cvs download that I can't even find in the svn. I tried to copy things around and comment stuff out but I'm still not having any luck. I may just skip it for now and see if I can just get the collector going. Once I have that going and collecting logs I can work on the extractor to get the data out of them. I'm working on it a bit tonight.

[trevius]

Hmm yeah, I can't get either of these even close to compiling either. If you can get either compiling, or if someone can't find a version that does actually compile, I will start trying out adjusting some structs and opcodes in them and running them to see if they might actually be able to get working with Live again. Once the compile works for both, it won't be hard for me to find out just how rough it will be to get them both updated for EQLive again. I imagine it won't be bad at all.

Really, all I need is something that can identify an opcode and then use a packet structure to parse the fields out and convert that to SQL. ShowEQ already has easily readable log files that could be parsed pretty easily. I know that is basically all these 2 programs are doing. If I knew enough about how to make something that could do that, I would make one myself lol. It would just need to be able to convert hex into strings, floats and int8, int16 and int32s. After that it would be all about putting them into a format that could make an SQL file or go directly into the database even.

I figure that if I have to, I may just use ShowEQ to collect the really hard stuff to get set perfectly like doors and manually break each field down from the packet and add them to my database manually. I could probably do doors in about 5 minutes each, which is much less time than trying to place them the hard way lol.

[trevius]

Yeah, just to test it, I just manually converted all of the doors in Crescent Reach into the database and they seem to work perfectly. Everything is in the right place and even lifts work great and are perfectly aligned. I just broke down each packet to fit the packet structure for doors, and then used hexvis to get all of the info from the hex that I needed for the table.

So, if needed, things like this could be done manually with some work, but it would be extremely nice to get the 2 tools working that were made to do this automatically

[Andrew80k]

Well, I was able to get the collector to compile under linux. It includes a bunch of stuff from the common files so I've been trying to make sure it was using the latest version of stuff. My next step is to take what I can of the ShowEQ stuff and make sure the structs and opcodes are correct for live and then try to collect to see what that does. Unfortunately, it's probably going to be slow for me for a bit as I just got into the teeth of my graduate classes. I will be trying to get this going over the next couple of weeks though. I'll post as I progress. I haven't gotten the extractor to compile yet, but I suspect that I know what is wrong and I think I have solved it but I haven't tried it yet. It's not much use without the collector working so my priority is on the collector. Once the collector is working again I can pass that to the PEQ guys and they can start collecting the heck out of live zones and I'll start working on the extractor. I don't think the builder will be hard once the others are working. It actually may work just the way it is, but we'll see.

[trevius]

Oh, you are working on eqcollector? I thought were were only talking about eqextractor and eqbuilder2. I think eqextractor pulls the data from the packet collect and puts it in a certain order or whatever and then eqbuilder takes the output from eqextractor and puts it into the database. I am not sure how eqcollector would fit into all of that, but I didn't think it would be useful to us anymore. Plus, IIRC, the structures and stuff work differently there, so it would be harder to get it going I think.

[trevius]

Ok, after re-reading the post that cavedude made earlier in this thread, it looks like eqextractor needs something before it that can actually convert the packets into something it can read to do the extractions from. I am guessing that is what eqcollector is, needed for then.

Really, it seems like quite a hassle to have to use 3 programs to accomplish this lol. The log files from ShowEQ should be pretty simple for a parser to be able to handle pulling the data from. They clearly show the opcodes for each packet and which direction they are being sent from. So, once a parser can identify those 2 things, it should just be a matter of running the data in the packet through a struct to have it converted. Then, once it is converted, it would just need something to tell the program which tables/fields in the database to put each bit of data into.

I am not really sure why these 3 programs are requiring so many extra files and stuff from the source code. Really, I would think that if they are just parsing something and converting it to something else, it should be pretty straight forward and not require a bunch of code to do it. It is too bad we don't have FNW here anymore to maybe give some advise, or at least to get the apps able to compile again After that point, I am sure someone can get them working with Live again.