Update

[Andrew80k]

I'm making some progress. I got all the zone stuff fixed last night, and started working on the player profile and AA's. I pulled out all the Spawn stuff that I did and have gone back to the drawing board on that. Going to work on some other stuff while I'm figuring out more of how it works.

There are 2 major things that the extractor does. It extracts data that it knows about, and "explores" data that it doesn't. The extraction part isn't too hard. Some of the newer variable length stuff is going to be a bit of a challenge, but nothing too tough. The explorer stuff is a little strange and is going to take a bit of work on my part to figure out.

So far the 2 collects that I made with the collector have been fine. It's getting the data out that is taking all the work.

[Akkadius]

This is very cool Andrew80k, I'm sure I will be using the crap out of this, other as well.

[trevius]

I think AA info will be useful and probably not to hard to extract. Though, I am not sure what we would need from the PP, unless you are just wanting to collect it to be able to identify stuff in it easier or something. I have actually started working a bit on a Live client for EQEmu, but haven't gotten too far just yet. I don't plan to put too much work into it like our other clients, but doing it more for fun and just to see how functional it would be without much work. But, when I am done, I should have the PP all lined up perfectly, or at least close, which if you still need it, you could use what I get on the SVN in the next day or 2.

[Andrew80k]

It doesn't seem to be. I don't think the PP is very useful except where it pertains to AA's to be honest, currently. Using the Explorers function it might be kind of useful for filling in the gaps at some point in the future. It's one of the easier things to get right in the extractor though so I'm starting at the easy stuff while I get a handle on how it works and the structs.

[Andrew80k]

I worked on the Collector most of the night last night as I don't seem to be getting AA's in the collections. I did a collection with my Ranger on live and I'm making progress with the spawns, they're still off some, but getting there. Player profile was just used in getting buffs, but there is a disconnect somewhere in the code for that. Despite me updating the PP struct, it was still not right and I'm not certain where it is off and I'm not going to spend many cycles fixing that at this point. More important stuff is needed. Not sure what the buff stuff is used for at this point anyway. Maybe someone else knows.

[trevius]

I am working on Live still and am now able to get to character select as of last night. I am hoping to make it in-game tonight if it isn't too much trouble. Once I get in-game, it should be easy for me to update all opcodes in the patch_Live.conf file to be 95%+ accurate pretty quickly.

Either way, I went through and updated the PP as best as I can to prepare for my attempts of getting in-game later. Here is how I have it set currently:

// Live May 5 2010 - Size 25312
struct PlayerProfile_Struct
{
/*00000*/ uint32  checksum;				//
//BEGIN SUB-STRUCT used for shrouding stuff...
/*00004*/ uint32  gender;				// Player Gender - 0 Male, 1 Female
/*00008*/ uint32  race;					// Player race
/*00012*/ uint32  class_;				// Player class
/*00016*/ uint8  unknown00016[40];		// #### uint32  unknown00016;   in Titanium ####uint8[40]
/*00056*/ uint8   level;				// Level of player
/*00057*/ uint8   level1;				// Level of player (again?)
/*00058*/ uint8   unknown00022[2];		// ***Placeholder
/*00060*/ BindStruct binds[5];			// Bind points (primary is first)
/*00160*/ uint32  deity;				// deity
/*00164*/ uint32  intoxication;			// Alcohol level (in ticks till sober?)
/*00168*/ uint32  spellSlotRefresh[MAX_PP_MEMSPELL]; // Refresh time (millis) - 4 Octets Each
/*00208*/ uint8   unknown00218[14];		// Seen 00 00 00 00 00 00 00 00 00 00 00 00 02 01
/*00222*/ uint32  abilitySlotRefresh;
/*00226*/ uint8   haircolor;			// Player hair color
/*00227*/ uint8   beardcolor;			// Player beard color
/*00228*/ uint8   eyecolor1;			// Player left eye color
/*00229*/ uint8   eyecolor2;			// Player right eye color
/*00230*/ uint8   hairstyle;			// Player hair style
/*00231*/ uint8   beard;				// Player beard type
/*00232*/ uint8	  unknown00236[4];		// was 14
/*00236*/ union
	 {
		struct
		{
		/*00236*/ EquipStruct equip_helmet; // Equiptment: Helmet visual
		/*00248*/ EquipStruct equip_chest; // Equiptment: Chest visual
		/*00260*/ EquipStruct equip_arms; // Equiptment: Arms visual
		/*00272*/ EquipStruct equip_bracers; // Equiptment: Wrist visual
		/*00284*/ EquipStruct equip_hands; // Equiptment: Hands visual
		/*00296*/ EquipStruct equip_legs; // Equiptment: Legs visual
		/*00308*/ EquipStruct equip_feet; // Equiptment: Boots visual
		/*00320*/ EquipStruct equip_primary; // Equiptment: Main visual
		/*00332*/ EquipStruct equip_secondary; // Equiptment: Off visual
		} equip;
		/*00236*/ EquipStruct equipment[9]; //Live Shows [108] for this part
	 };
/*00344*/ uint8 unknown00224[168];		// Live Shows [160]
/*00512*/ Color_Struct item_tint[9];	// RR GG BB 00
/*00548*/ AA_Array  aa_array[MAX_PP_AA_ARRAY];	// [3600] AAs 12 bytes each
/*04148*/ uint32  points;				// Unspent Practice points - RELOCATED???
/*04152*/ uint32  mana;					// Current mana
/*04156*/ uint32  cur_hp;				// Current HP without +HP equipment
/*04160*/ uint32  STR;					// Strength - 6e 00 00 00 - 110
/*04164*/ uint32  STA;					// Stamina - 73 00 00 00 - 115
/*04168*/ uint32  CHA;					// Charisma - 37 00 00 00 - 55
/*04172*/ uint32  DEX;					// Dexterity - 50 00 00 00 - 80
/*04176*/ uint32  INT;					// Intelligence - 3c 00 00 00 - 60
/*04180*/ uint32  AGI;					// Agility - 5f 00 00 00 - 95
/*04184*/ uint32  WIS;					// Wisdom - 46 00 00 00 - 70
/*04188*/ uint8   unknown04172[28];		// 
/*04216*/ uint8   face;					// Player face - Actually int32?
/*04217*/ uint8   unknown02264[147];		// was [175]
/*04364*/ int32   spell_book[MAX_PP_SPELLBOOK];	// List of the Spells in spellbook 720 = 90 pages [2880] was [1920]
/*06284*/ uint8   unknown06284[960];			// Spacer for the end of the book for now (pages 60 to 90)
/*07244*/ int32   mem_spells[MAX_PP_MEMSPELL]; // List of spells memorized
/*07284*/ uint8   unknown04396[28];		//#### uint8 unknown04396[32]; in Titanium ####[28]
/*07312*/ uint32  platinum;				// Platinum Pieces on player
/*07316*/ uint32  gold;					// Gold Pieces on player
/*07320*/ uint32  silver;				// Silver Pieces on player
/*07324*/ uint32  copper;				// Copper Pieces on player
/*07328*/ uint32  platinum_cursor;		// Platinum Pieces on cursor
/*07332*/ uint32  gold_cursor;			// Gold Pieces on cursor
/*07336*/ uint32  silver_cursor;		// Silver Pieces on cursor
/*07340*/ uint32  copper_cursor;		// Copper Pieces on cursor
/*07344*/ uint32  skills[MAX_PP_SKILL];	// [300] List of skills
/*07644*/ uint8   unknown04760[236];
/*07880*/ uint32  toxicity;				// Potion Toxicity (15=too toxic, each potion adds 3)
/*07884*/ uint32  thirst_level;			// Drink (ticks till next drink)
/*07888*/ uint32  hunger_level;			// Food (ticks till next eat)
/*07892*/ SpellBuff_Struct buffs[BUFF_COUNT];	// [960] Buffs currently on the player (60 Max) - was 600
/*08492*/ uint8   unknown08492[360];		// End of Buffs
/*08852*/ Disciplines_Struct  disciplines;	// [400] Known disciplines
/*09252*/ uint8   unknown09252[400];		// Discs?
/*09652*/ uint32  recastTimers[MAX_RECAST_TYPES]; // Timers (UNIX Time of last use)
/*09732*/ uint8   unknown08124[160];		// Some type of Timers
/*09892*/ uint32  endurance;			// Current endurance
/*09896*/ uint8   unknown09896[20];		// ?
/*09916*/ uint32  aapoints_spent;		// Number of spent AA points
/*09920*/ uint32  aapoints;				// Unspent AA points
/*09924*/ uint8 unknown06160[4];
/*09928*/ Bandolier_Struct bandoliers[MAX_PLAYER_BANDOLIER]; // [6400] bandolier contents
/*16328*/ PotionBelt_Struct  potionbelt;	// [360] potion belt 72 extra octets by adding 1 more belt slot
/*16688*/ uint8 unknown12852[8];
/*16696*/ uint32 available_slots;
/*16700*/ uint8 unknown12864[80];		//#### uint8 uint8 unknown12864[76]; in Titanium ####[80]
//END SUB-STRUCT used for shrouding.
/*16780*/ char    name[64];				// Name of player
/*16844*/ char    last_name[32];		// Last name of player
/*16876*/ uint8   unknown19588[8];  //#### Not In Titanium #### new to SoF[12]
/*16884*/ int32   guild_id;            // guildid
/*16888*/ uint32  birthday;       // character birthday
/*16892*/ uint32  lastlogin;       // character last save time
/*16896*/ uint32  account_startdate;       // Date the Account was started - New Field for Live***
/*16900*/ uint32  timePlayedMin;      // time character played
/*16904*/ uint8   pvp;                // 1=pvp, 0=not pvp
/*16905*/ uint8   anon;               // 2=roleplay, 1=anon, 0=not anon
/*16906*/ uint8   gm;                 // 0=no, 1=yes (guessing!)
/*16907*/ int8    guildrank;        // 0=member, 1=officer, 2=guildleader -1=no guild
/*16908*/ uint32  guildbanker;
/*16912*/ uint8 unknown13054[4];  //was [8]
/*16916*/ uint32  exp;                // Current Experience
/*16920*/ uint8 unknown13072[8];
/*16928*/ uint32  timeentitledonaccount;
/*16932*/ uint8   languages[MAX_PP_LANGUAGE]; // List of languages
/*16957*/ uint8   unknown13109[7];    //#### uint8   unknown13109[4]; in Titanium ####[7]
/*16964*/ float     y;                  // Players y position (NOT positive about this switch)
/*16968*/ float     x;                  // Players x position
/*16972*/ float     z;                  // Players z position
/*16976*/ float     heading;            // Players heading
/*16980*/ uint8   unknown13132[4];    // ***Placeholder
/*16984*/ uint32  platinum_bank;      // Platinum Pieces in Bank
/*16988*/ uint32  gold_bank;          // Gold Pieces in Bank
/*16992*/ uint32  silver_bank;        // Silver Pieces in Bank
/*16996*/ uint32  copper_bank;        // Copper Pieces in Bank
/*17000*/ uint32  platinum_shared;    // Shared platinum pieces
/*17004*/ uint8 unknown13156[1036];    // was [716]
/*18040*/ uint32  expansions;         // Bitmask for expansions ff 7f 00 00 - SoD
/*18044*/ uint8 unknown13244[12];
/*18056*/ uint32  autosplit;          // 0 = off, 1 = on
/*18060*/ uint8 unknown13260[16];
/*18076*/ uint16  zone_id;             // see zones.h
/*18074*/ uint16  zoneInstance;       // Instance id
/*18080*/ char      groupMembers[MAX_GROUP_MEMBERS][64];// 384 all the members in group, including self
/*18464*/ char      groupLeader[64];    // Leader of the group ?
/*18528*/ uint8 unknown13728[604];  // was [348]
/*19132*/ uint32  entityid;
/*19136*/ uint32  leadAAActive;       // 0 = leader AA off, 1 = leader AA on
/*19140*/ uint8 unknown14392[4];
/*19144*/ sint32  ldon_points_guk;    // Earned GUK points
/*19148*/ sint32  ldon_points_mir;    // Earned MIR points
/*19152*/ sint32  ldon_points_mmc;    // Earned MMC points
/*19156*/ sint32  ldon_points_ruj;    // Earned RUJ points
/*19160*/ sint32  ldon_points_tak;    // Earned TAK points
/*19164*/ sint32  ldon_points_available;  // Available LDON points
/*19168*/ uint8 unknown14420[68]; // was [136]
/*19236*/ float  tribute_time_remaining;        // Time remaining on tribute (millisecs)
/*19240*/ uint32  career_tribute_points;      // Total favor points for this char
/*19244*/ uint32  unknown7208;        // *** Placeholder
/*19248*/ uint32  tribute_points;     // Current tribute points
/*19252*/ uint32  unknown7216;        // *** Placeholder
/*19256*/ uint32  tribute_active;      // 0 = off, 1=on
/*19260*/ Tribute_Struct tributes[MAX_PLAYER_TRIBUTES]; // [40] Current tribute loadout
/*19300*/ uint8 unknown14616[4];
/*19304*/ double group_leadership_exp;     // Current group lead exp points
/*19312*/ double raid_leadership_exp;      // Current raid lead AA exp points
/*19320*/ uint32  group_leadership_points; // Unspent group lead AA points
/*19324*/ uint32  raid_leadership_points;  // Unspent raid lead AA points
/*19328*/ LeadershipAA_Struct leader_abilities; // [128]Leader AA ranks 19332
/*19456*/ uint8 unknown14772[132];		// was [128]
/*19588*/ uint32  air_remaining;       // Air supply (seconds)
/*19592*/ uint32  PVPKills;
/*19596*/ uint32  PVPDeaths;
/*19600*/ uint32  PVPCurrentPoints;
/*19604*/ uint32  PVPCareerPoints;
/*19608*/ uint32  PVPBestKillStreak;
/*19612*/ uint32  PVPWorstDeathStreak;
/*19616*/ uint32  PVPCurrentKillStreak;
/*19620*/ PVPStatsEntry_Struct PVPLastKill;		// size 88
/*19708*/ PVPStatsEntry_Struct PVPLastDeath;	// size 88
/*19796*/ uint32  PVPNumberOfKillsInLast24Hours;
/*19800*/ PVPStatsEntry_Struct PVPRecentKills[50];	// size 4400 - 88 each
/*24200*/ uint32 expAA;               // Exp earned in current AA point
/*24204*/ uint8 unknown19516[40];
/*24244*/ uint32 currentRadCrystals;  // Current count of radiant crystals
/*24248*/ uint32 careerRadCrystals;   // Total count of radiant crystals ever
/*24252*/ uint32 currentEbonCrystals; // Current count of ebon crystals
/*24256*/ uint32 careerEbonCrystals;  // Total count of ebon crystals ever
/*24260*/ uint8  groupAutoconsent;    // 0=off, 1=on
/*24261*/ uint8  raidAutoconsent;     // 0=off, 1=on
/*24262*/ uint8  guildAutoconsent;    // 0=off, 1=on
/*24263*/ uint8  unknown19575;     // ***Placeholder (6/29/2005)
/*24264*/ uint32 level3;		// SoF looks at the level here to determine how many leadership AA you can bank.
/*24268*/ uint32 showhelm;            // 0=no, 1=yes
/*24272*/ uint32 RestTimer;
/*24276*/ uint8   unknown19584[1036]; // ***Placeholder (2/13/2007) was[1028]or[940]or[1380] - END of Struct 
/*25312*/
};

Given the large number of fields, it is easy for the struct to get bumped off a bit. But, once I can start working on getting in-game, I should be able to iron out most of the kinks. And once I am all of the way in-game, I can probably get the PP looking pretty good if something is off a bit. Thought you might be able to make use of this if you do care to try messing with the PP at some point. If you give it a couple of days and work on spawns and such in the mean-time, I should have the PP looking pretty good by the time you come back to it again (if you ever feel like it).

[Andrew80k]

I played with it some, but it's giving me something weird. It keeps saying the size is wrong. I get the right size in the collect but it keeps saying the sizeof(PlayerProfile_Struct) is 19538. I can't seem to figure that one out just yet.

[Andrew80k]

Spent most of the night looking at packet dumps of spawns. Still not ready yet but I think I've figured out how to get there. Ugh.

[cavedude]

I could have sworn you had access to the collector's SVN, Andrew80k but it looks like not. I've added you so you can make changes as you have them. http://code.google.com/p/eqemutool/

[gaeorn]

Originally Posted by cavedude

I could have sworn you had access to the collector's SVN, Andrew80k but it looks like not. I've added you so you can make changes as you have them. http://code.google.com/p/eqemutool/

Yeah Andrew80k, it'd be awesome if you upload the code so we can get a group effort going on this. And there's nothing in this repo to harm so you don't have to worry about anything you changed in common.

[Andrew80k]

Thanks, CD.

So far there's not really anything to upload. I've spent a LOT of time in the extractor trying figure out the spawn structs. It's not far off but the coords are not right and I've been painfully adjusting the structure one character at a time over the last week or so. The collector seems to be pretty good but no way really to validate it without fixing the extractor first. Doors in both the collector and extractor seem pretty good. I haven't validated the zone points yet. I've noticed that Derision seems to be working a bit on the live stucts and that will definitely help.

[trevius]

They just patched Live again today, and at least a few things (like the PP) have changed for sure. Since the next patch isn't scheduled until June 9th (I believe), I am hoping to get most of the rest of the opcodes updated for Live pretty soon. Derision and I have been working on it a bit here and there. Once we have it pretty filled out, you may even be able to have the tools use the Live.conf and struct files directly from the EQEmuServer/Utils and EQEmuServer/common/patches/ folders. That way, as long as we somewhat keep up with each patch or so, the Live client and the Collector/Extractor tools should both remain functional without duplicate efforts (or at least not much).

As far as the SVN goes, what is on there currently isn't functional at all yet from what I have heard. So, any improvement is better than none. Since you have them both so they can at least compile, it wouldn't hurt to get what you have on the SVN. If you prefer to wait until you have it more functional, that is no problem. But, one of the nice things about using an SVN is that it lets others work together to finish things off so it doesn't all get put on a single person.

BTW, what compiler are you currently using to compile it? Last I heard, it was made for VS Pro or something.

[Andrew80k]

I'm using VS 2008. Not Express. As far as I know it will not compile with express though I haven't tried it since erde got it going. Straight out of the SVN it pretty much compiles and runs for me. I had to tweak the solution just a bit to point to the correct lib locations, but I think that was all.

The collector "works". It will collect packet information and some of it is solid. Some not so much. I do keep getting an unknown stream type in the collector and I haven't figured it out yet. I have put a bunch of debug info into the collector and have started doing more collects with it to help me figure it out. Mostly I'm just learning how it works.

What's working:

Zone info.
Doors
Player data
Some spawn info but some of it isn't right. I think that's more of an adjustment to the extractor. The collector just picks up packets and lays them down with OP_code "tags".

The extractor "works" for some things as well.

What is working:
Doors -- seems good
Zone info -- same here. seems good.
Partial spawn data

What's not working:

AA's -- not seeing this at all
Tasks -- all commented out in the extractor, but the data is there.
Spawn updates This seems to have 2 sized structures on live. One 40 bytes, the other 41. If I adjust the struct one way I get some, if I adjust it the other way I get more. I also consider this critical to being useful.
Objects -- ground spawns etc.

It's slow going but I'm working on it.