If we're still looking for newer structs (after 2006), we can peek into the SEQ SVN. Specifically, SoF structs should be in Rev 686 (11/5/2007), Rev 688 (11/21/2007), or Rev 690 (11/27/2007). Since, according to Wikipedia, it was released on 11/13/2007, we'll probably want to look at Rev 688.

Ya, I have been using all of those to get as far as I have :P They have been very useful.

Even though SoF was released on 11/13/07, it was actually built on 9/7/07 (according to the EQ Debug Logs). So, anything from 9/7/07 to around the beginning of 2008 is probably useful. They may have patched in some of the SoF stuff to live before it actually came out, so the structures and stuff may have already been there for the most part. The main thing that would have changed alot are the opcodes. Unfortunately, most of the opcodes in SEQ weren't updated until after December, so many of them aren't correct.

I am thinking about trying a current version of SEQ and running a trial live account just to see if I can find 100% accurate structures that can be used to help getting SoF to work. I am betting that the current live structs are probably closer to SoF than the Titanium ones are.

If anyone has packet collects from right around the time that SoF was released (preferrably from SEQ if possible), I would love to get a copy of them. I think that would help alot once I knew what I was looking at. But, I am not holding my breath to get them, because I doubt anyone has still them.

I am not really sure what it is for, but a code obfuscater was added to SEQ around the time that SoF came out. From looking at the comments around the code for it, it appears to be used for pulling opcodes directly from the assembly code of the eqgame.exe. I have no idea how it is actually used though, or if I am just misunderstanding what it does. I have seen comments about it on the SEQ forums that seem to say something about opcodes changing from time to time when you zone or log on other characters/servers. I guess it is some kind of simple encryption or something. But, it seems that the new obfuscate can pull an opcode table from the exe file. If that is true, maybe we can use it on Titanium, and then on SoF and compare the 2 tables and compare the conf files for known opcodes and match them up. Here is a link to the obfuscate getting added to SEQ SVN:

http://seq.svn.sourceforge.net/viewv...85&pathrev=686

I moved this to the development section, because it seems more appropriate.

I figured out where it is currently breaking when it is trying to log in. It seems that the opcode OP_SendExpZonein=0x3703 is where the problem is. This is the last opcode that the client receives before it stops responding. I also verified that by removing this opcode from Titanium it will fail at the exact same point according to the EQ Debug Logs.

Actually, in Titanium, the log entry is slightly different, but it is the same thing just renamed:

My guess is that this opcode now needs to be encoded like many of the other important ones. And since it isn't being encoded, it isn't recognizing it, so it is failing. I am 99% sure that I have the correct opcode set for it.

Looking at the place where it is failing, here is the Assembly code for it:

And, from client_packet.cpp, here is the code that handles that opcode:
So, either the opcode needs to be encoded, or this handling code needs to be changed. I am guessing the opcode needs to be encoded, because the jump is looking for anything that isn't 0, so I would think that it is just making sure it got something for it. And, if it isn't encoded and is supposed to be, maybe it shows up as 0. But, if it doesn't need to be encoded, I don't know why it would be 0.

I will mess with it and see if I can figure out how to get that opcode encoded, but I don't really know how that will work, since the only opcodes I see currently getting encoded already have structures tied to them, but I don't see one for SendExpZonein. Unless maybe it is named differently.

At least I know where it is failing now, so I should be able to come up with something to move it to the next step. It should be getting pretty close now. I was able to find and verify more of the required opcodes for logging in over the past couple of days as well. Making some progress at least :)

I am still stuck at this same point, but I am not giving up that easily :P Last night, I started trying to figure out why character creation wasn't working. I have the correct Opcode, but I see that the server is expecting a struct to come in and the client is sending just the opcode with a size of 0. The client just hangs probably waiting for something back that we aren't sending. I am going to try filling in the needed character select stuff on the server side and then have it send the character create opcode back to the client. So, it would be working in reverse of how Titanium does it. If that works, then it means they might have been adjusting the order of server/client communications, maybe to optimize some stuff.

If so, maybe that is the reason that sendexpzonein is failing. It could be waiting for something else that we aren't sending. I will try forcing a few of the packets that normally follow the sendexpzonein and see if that makes any difference. I definitely see it is doing something extra in the assembly code of SoF that wasn't in Titanium, but there is no way to really tell what it is. It could either be waiting for an extra opcode that got added to the sendexpzonein stuff, or it could be waiting for a new structure that we don't have in Titanium. I am hoping it is the former, not the latter.

Other than that, the other big difference I notice is that some of the subs being called in area where the problem is happening have about 1000 set as the variables where Titanium has about 800 set. That makes me think that it is checking the size of a certain packet structure, but I don't know which one. If I am right, I think that one of the structures needed at this point has changed and we need to figure out which one and what it was changed to so it can be adjusted.

My last resort will be to setup the current showeq for Live and setup a trial account to watch the logs from SEQ and see what the current structures really are. I am sure that some of the SEQ structures are correct, but there is alot of info they don't need for SEQ to function so it probably gets ignored. Maybe I can find more details and get them filled in to get it working. Also, it would help to see if anything new is being sent. Unfortunately, I don't have any SEQ logs from when Titanium was running on Live, so I don't have anything to compare with. That will probably make feeding through this stuff considerably harder.

From looking at the MQ2 source, I found a few more item fields that may be needed for SoF to load items properly:

HeroicSvPoison
HeroicSvMagic
HeroicSvFire
HeroicSvDisease
HeroicSvCold
HeroicSvCorruption

MaxPower
Power

I imagine that the Herioc Resists are almost certainly required fields. The Power and MaxPower fields may not be required though, I don't really know much about them.

Maybe the MQ2 source can be useful after-all. After looking at it again, some of the stuff in there is starting to make more sense to me now that I am getting more used to looking at the assembly code from eqgame.exe. I found that MQ2 even has a version for the SoF retail eqgame.exe by looking at the debug from SoF:

And then at the MQ2 build for that date, it wants this exact version (seen in the MQ2Main\eqgame.h file):

So, that tells me that everything we needed to build EQ for SoF was actually already in place at that time.

This should also mean that SEQ structs and maybe some opcodes from that time might be useful as well. Unfortunately, SEQ wasn't updated between 3/25/2007 and 11/05/2007. It was just not functioning at all for about 6 months in between there. They finally got it working again in November right before the actual retail release of SoF. But, at least this confirms that I should use certain older structures over some that were changed after that.

After being stumped on this a while, I finally decided to take a step that would hopefully help me alot. I paid for my old EQ account so it could play on EQLive again. Then, I got the current version of ShowEQ working and put a hub in my network so I can sniff the packets to/from EQ. This is letting me watch the logs directly from EQLive that show pretty much everything I would need to get EQEmu working with EQLive. I am hoping that EQLive runs pretty close to how SoF did, and it should since there haven't been nearly as many changes to Live since SoF as there was from Titanium to SoF.

Already, I have found that the order of packets when logging in is pretty different from Titanium. I think I found the place that is stopping me from logging in all of the way. I just need to work on it some more to get it past that point. From what I can tell so far, it looks like this will be more helpful than anything I have tried. I am hoping to make more progress tonight now that I have this new information.

Here is an example of the logs of zoning in. I cut out a bunch of the actual data and stuff. I also made notes next to some of the packets.

SEQ isn't converting network to host byte order on the raw soe opcodes but here:

Connect Request
int crc length 0x00000002 = 2
int session id 0x6b0bee8f
max packet size 0x00000200 = 512

Connect Accept Reply
int session id 0x6b0bee8f
int encode key 0x45d7b502
char crc length 0x02
crypt options bitfield 0x0201
max packet size 0x00000200

Network Status Update - not really important.

[code]Dec 31 2008 05:43:05:722 [Raw] [Client->Server] [Size: 86]
[OPCode: 0x0300]
000 | 0a 00 09 00 00 94 35 00 00 00 00 4a 00 09 00 01 | ......5....J.... [code]
Combined packet

contains:
The first packet isn't a soe ack you can tell by the opcode and the fact that it doesn't include a sequence number. Likely some kind of I am ready for this info packet.

This is an ack packet, tho it looks wrong, they're usually 4 bytes total including the opcode 2 without this one is 7 byte with the opcode 5 without. The opcode is SOE_ACK tho.

These are standalone packets, encoded and compressed with a 2 byte sequence on the top probably 0x0000 since this is the first and a crc on the end. It's also possible to combine packets within an 0x0009 packet but you can't combine a raw packet and an eq packet inside the 0x0009 they both have to be EQ packets.

This is a fragmented packet like 0x0009 they are encoded and compressed they have a sequence number for recombining and a length byte. These are used when the data chunk being sent is larger than the max frame size sent on init: 512.

I know this doesn't help at all for figuring out what you need for the SoF stuff but maybe if you understand how the raw protocol works you'll understand it a bit better.

Ya, at least that clears a few things up that I wasn't quite sure about. I was ignoring the 0300 and 0900 because I figured they were doing something like that. I think 0d00 is also something like that, I am guessing raw packets that need to be encrypted? Either way, I ignore those as well :P

Just familiarizing myself with the packet logs, I started breaking down the structure for player profile to see if I could verify what it should look like currently for EQLive. So far I am almost done with it and will post the finished version when it is completed. I think it will help to have an accurate player profile with SoF once it is working at all.

Right now, I think I am really close, but I am doing the same thing live does and the client just stops at that first main loop every time. Maybe I have something messed up earlier on and it isn't figuring that out until that point. Or, maybe I have to have the AA stats opcode for it to continue, and I don't. I have almost all of the ones I need, but that one is really elusive to me lol. I can run Titanium fine without it, so I think it should be ok to run SoF without it, but no way to know for sure.

Basically, this is the order things happen on Titanium in the emu for the part I am having a problem with:

And this the order I am seeing from live for that same part:
So, I think SoF is supposed to be set the way Live is now. But for some reason, it still isn't working yet for me. I have tried both ways and more and the client never responds with the SendExpZonein as it should, or anything else for that matter. It is like it is either waiting for something that hasn't come yet, or something came but was messed up in format.

Well, I will keep plugging away at it. I learn more each day and it will help alot once I actually get to the point where I can start cleaning stuff up so people can actually use SoF to play.

Here is the example right from the logs from Live:


Here is the full 0x5821 that I am gonna check and see if I can find what it is:

I guess I am going to break down EQLive and see what each Opcode is so I can get an accurate order for what packets should be going when. Using SEQ, it shouldn't be too tough to get that all sorted out. Once I know the specific details, I am hoping to try them on SoF and see if it works.

Here is what I have so far:

I will edit this list as I can make more confirmations. It shouldn't take me long to figure out most of this list. I can tell for sure that some of the orders have changed, so I think if I can get that sorted out, it may work almost flawlessly for SoF.

I stumbled on a new struct last night. It doesn't really help this effort much, but it is probably worth noting for use later when we might be able to actually put it to use. There is a feature on Live that was in SoF as well, but not in Titanium. The new feature let's the client set certain spells to be blocked from hitting the client. This is for keeping buffs from overriding each other if you don't want them to.

The EQLive opcode for this is 0x0baa and I found the SoF opcode for it too. I built a struct for it with placeholders just to have something in place for it even though I don't know the values for it yet.

I also confirmed that EQLive now has 60 pages in the spell book as apposed to 50 in Titanium. This means that the MAX_PP_SPELLBOOK is now 480 in EQLive and probably in SoF as well. They had to add it when they upped player levels to 75, so I am sure it was in SoF, otherwise druids and maybe other classes could have overflowing spell books.

I am sure that most of my struct for SoF are correct or very close to it. I think I am just missing something minor that is keeping me from getting past this point I have been stuck at for the past couple of weeks. If I can just get pass that point, I feel that the rest will be much quicker and easier. My current guess is that I may not have all of the right opcodes being encoded on the way out that need to be for SoF. The client is expecting a 0 size opcode 0x1FA1, which is new since Titanium. Basically, it seems to be the same as SendExpZonein, accept instead of server sending that and getting it back from the client, it now sends this new opcode and waits for the SendExpZonein back from the client.

From looking at the EQLive logs, it looks like almost all packets are encoded now, at least while entering a zone. Maybe I have to set them all to be encoded for the client to get all of the info it needs for logging in. That is going to be a bit of a pain, because as far as I can tell, I have to create the handling stuff for each opcode that needs to be encoded. If there was a way for me to just set an opcode to be encoded in the Anniversary_ops.h and then only have to tell the Anniversary.cpp to use the structure for encoding, that would be pretty easy. But, it looks like I need to do something more like this for each one:

I'm not exactly sure what the OUT stuff means, but I am guessing that all of the stuff set to go OUT is the stuff that gets encoded. If so, then I wonder what happens to stuff in the structure that isn't set to go OUT in the ENCODE. Does that stuff just get ignored?

Long time lurker, 1st time caller. . .


This work you've been doing is very exciting. With the last ever boxed edition of EQ1 possibly being SoF, this is a big step. I have been looking through the wikis and the forums, and trying to find some up to dat info on if there is anything someone with an active eqlive account can do to collect information for the Devs.

Let me know if there is something that can be done with no experience besides setting up a mini login server in the basement.

Thanks,

Greldor
The Grand Creation

my god man...
 

i didnt believe in intelligent machines until i read this thread.

Trevius isnt human i tell you! he is the real life Bender!!!!


you're awesome man, good things are comming peoples!


on a nearly equally pointless side note. every time i type/read Trevius' name i think of Golan Trevise from the Foundation series. :) any relation by chance?

The ENCODE/DECODE is used when the struct is different between Client versions.

ENCODE 'encodes' packets on the way out, from the server to the client, and DECODE does the reverse.

The ENCODE is a translation from the struct in common/eq_packet_structs.h to the struct in the client specific struct in patches/<Client version>_structs.h.

To take your example for OP_ManaChange. The 'EMU' struct for this, in common/eq_packet_structs.h is:

while the Anniversary struct is:

As you can see, Anniversary has an extra field at the end of the struct.

Essentially, the ENCODE section in Anniversary.cpp is copying the fields from the Emulator version of the struct to the version that Anniversary edition needs. I think the outgoing packet is filled with zeroes before the ENCODE takes place, so unknown12 and unknown16 would be zero.

For each struct that has changed in SoF, you would need to map out the fields in the new structure and add it to Anniversary_structs.h, add the Opcode to Anniversary_ops.h and do an ENCODE in Anniversary.cpp.

You would also need to do the same for Client to Server opcodes where the struct has changed, but this time do a DECODE from the SoF client structure to the Emu struct.