ruby extractor tool

[erde]

Hello, i am working on a ruby extension to read the packet logs. This is very experimental, currently

I am not sure what structs are important! to extract
What format should the tool output?

Here is some sample output
Log created: 2009-12-19 15:00:29 +0100

Opcode: 43ac OP_NewZone
Char Name: xxx
Long Name: The Plane of Knowledge
Short Name: PoKnowledge
zone: poknowledge
Type: 0
fog_red: [50, 50, 50, 50]
fog_green: [50, 50, 50, 50]
fog_blue: [155, 155, 150, 150]
fog_minclip: [400.0, 400.0, 400.0, 400.0]
fog_maxclip: [2000.0, 2000.0, 2000.0, 2000.0]
gravity: 0.400000005960464
time_type: 2
Sky: 0
zone_exp_multiplier: 1.0
safe_y: -148.0
safe_x: -285.0
safe_z: -159.0
max_z: 0.0
underworld: -5000.0
minclip: 50.0
maxclip: 2000.0
Short Name2: poknowledge
zone_id: 202
zone_instance: 0
fall_damage: 1

Opcode: 5c85 OP_GroundSpawn
Name: IT10645_ACTORDEF
Linked list [0, 142262936]
Drop Id 21
Zone Id 202
Zone Instance 0
Heading 255.0
Size 1.0
X 538.0
Y -405.0
Z -88.0
Object Type 1
Spawn Id 0
unknown096 [255, 255, 255, 255]

This ist the current source of the tool:

require 'ext/EQEmu'
require 'Live_Structs'

puts "GroundSpawn Test"

begin
	pfr = PacketFile::RbPacketFileReader.new 
rescue Exception => e
	puts "#{ e } (#{ e.class })!"
end

begin
	pfr.OpenFile( "packetlog-156738727.pf" )
rescue Exception => e
	puts "#{ e } (#{ e.class })!"
end

begin
	puts pfr.GetStamp()
rescue Exception => e
	puts "#{ e } (#{ e.class })!"
end

@op = OpcodeMgr::RegularOpcodeManager.new
@op.LoadOpcodes( "patch_Live.conf" )
begin
	while pfr.ReadPacket()
		opcode = @op.EQToEmu( pfr.getOpCode() )

		if opcode == EmuOpcodes::OP_GroundSpawn
			puts "Opcode: " + pfr.getOpCode().to_s(16) + " " + EmuOpcodes::OpcodeName(@op.EQToEmu(pfr.getOpCode()))
			p = pfr.getPacket()
			begin
			  a = Object_Struct.new
				a.read(p)
				puts "Name: #{a.object_name}"
				puts "Linked list #{a.linked_list_addr}"
				puts "Drop Id #{a.drop_id}"
				puts "Zone Id #{a.zone_id}"
				puts "Zone Instance #{a.zone_instance}"
				puts "Heading #{a.heading}"
				puts "Size #{a.osize}"
				puts "X #{a.x}"
				puts "Y #{a.y}"
				puts "Z #{a.z}"
				puts "Object Type #{a.object_type}"
				puts "Spawn Id #{a.spawn_id}"
				puts "unknown096 #{a.unknown096}"
				puts
			rescue Exception => e
				puts "#{ e } (#{ e.class })!"
			end
		end
		if opcode == EmuOpcodes::OP_NewZone
			puts "Opcode: " + pfr.getOpCode().to_s(16) + " " + EmuOpcodes::OpcodeName(@op.EQToEmu(pfr.getOpCode()))
			p = pfr.getPacket()
			begin
				a = NewZone_Struct.new
				a.read(p)
				puts "Char Name: #{a.char_name}"
				puts "Long Name: #{a.zone_long_name}"
				puts "Short Name: #{a.zone_short_name}"
				puts "zone: #{a.zone_short_name2}"
				puts "Type: #{a.ztype}"
				puts "fog_red: #{a.fog_red}"
				puts "fog_green: #{a.fog_green}"
				puts "fog_blue: #{a.fog_blue}"
				puts "fog_minclip: #{a.fog_minclip}"
				puts "fog_maxclip: #{a.fog_maxclip}"
				puts "gravity: #{a.gravity}"
				puts "time_type: #{a.time_type}"
				puts "Sky: #{a.sky}"
				puts "zone_exp_multiplier: #{a.zone_exp_multiplier}"
				puts "safe_y: #{a.safe_y}"
				puts "safe_x: #{a.safe_x}"
				puts "safe_z: #{a.safe_z}"
				puts "max_z: #{a.max_z}"
				puts "underworld: #{a.underworld}"
				puts "minclip: #{a.minclip}"
				puts "maxclip: #{a.maxclip}"
				puts "Short Name2: #{a.zone_short_name2}"
				puts "zone_id: #{a.zone_id}"
				puts "zone_instance: #{a.zone_instance}"
				puts "fall_damage: #{a.fall_damage}"
				puts
			rescue Exception => e
				puts "#{ e } (#{ e.class })!"
			end
		end
		if opcode == EmuOpcodes::OP_PlayerProfile
		  p = pfr.getPacket()
			begin
				a = CharProfileStruct.new
				a.read(p)
				puts

			rescue Exception => e
				puts "#{ e } (#{ e.class })!"
			end
		end
	end
rescue Exception => e
	puts "#{ e } (#{ e.class })!"
end

pfr.CloseFile()
puts "End"

[Derision]

Excellent work

I think some of the opcodes Cavedude would be most interested in are:

OP_NewSpawn
OP_SpawnDoor
OP_GroundSpawn
OP_NewZone

with a command line flag for the extractor tool to spit out SQL ready to populate the database.

Also:

OP_ClientUpdate

to create mob pathing grids. However this is more problematic to create automatically as they would really
need visual inspection to see where the mob completed a circuit of it's patrol route in order to avoid
redundant waypoints.

[trevius]

Yeah, that is some nice work, Erde

What are the 2 files you are requiring at the beginning there?

Have you considered using ShowEQ logs instead of the .pf log? ShowEQ log files might be a bit easier to read and I am pretty sure we can post accurate structures for them to update and they will update them so we can use their struct files with the tool. That might make upkeep a bit easier. Then we would have 2 projects helping to keep structs and opcodes up to date so the tool continues to work instead of getting outdated if people from here stop updating it.

I think Derision already hit most of the important packets we would want. You would just need to have it do MySQL queries to get them into the database. Though, until it was refined, it would probably be best to only update test databases and then copy over the good updates into the production database after verifying all went well.

I don't know how the old system handled pathing, but as Derision mentioned, it isn't an easy task to do. Ideally, you would want to be the only char in the zone while collecting pathing information or you risk people pulling mobs and throwing off your pathing info.

We probably also want zone points:
OP_SendZonepoints

And, I wouldn't mind having AAs to help get the SoF AA stuff worked out:
OP_SendAATable

Since for basic database population we really only require a few packet types, it probably wouldn't be too hard to keep this tool updated. As long as it can see Null Terminators and be able to do the proper conversions for 8bit, 16bit, 32bit, Strings, and floats, I don't think it would be too hard to add new fields or adjust structures and such when needed. Then, we would just need a config file for updating the few Opcodes we want to collect info from and another config file for updating the actual structures.

Just being able to collect doors, objects and spawns alone would be a huge step in the right direction

[erde]

Thanks Derision and Trevius

What are the 2 files you are requiring at the beginning there?

I have started to create a extension to ruby, i have used swig to wrap c++ to ruby. This tells ruby to use my extension.

But after getting all to work, i have decided to start from scratch To many problems, specialy type conversation uint16 to Ruby::Integer ...
The current implementation is completly written in ruby and seems to work very well.

Have you considered using ShowEQ logs instead of the .pf log?

I thought there is already a tool? The main reason was, i wanted to know if i am capable of writting such a tool in ruby.

This tool needs ruby 1.9 ! Sorry ruby 1.8 isn't supported yet!
You need to install the 'BinData' gem

gem install bindata

You could get the source here

http://www.eqdaemons.de/?page_id=6

Ok, this is a work in progress so dont expect too much
Usage:

EQEmulator Extractor Version 0.1.1
Usage: extractor [options] logfile
Options:
	--csv	Export to CSV-file

The export function generates: door.csv, object.csv, pos.csv, spawn.csv and zone.csv

Handled opcodes so far:
OP_ClientUpdate
OP_NewZone
OP_GroundSpawn

// Partially working ops, more work on the Spawn_Struct needed
OP_NewSpawn
OP_ZoneSpawns
OP_ZoneEntry

in utils/eqopcodes is a tool to update the patch_Live.conf from ShowEQ's worldopcodes.xml and zoneopcodes.xml
copy patch_Live.conf,worldopcodes.xml and zoneopcodes.xml into that directory

ruby merge.rb

will generate 3 files
patch_Live.conf.new = updated patch_Live.conf
showeq_ops.txt = opcodes from seq not found in patch_Live.conf
eqemu_ops.txt = opcodes from eqemu not found in seq files

This is collected data from the OP_ClientUpdate opcode, could this values be valid?

spawnId, padding0000, deltaX, padding0005, deltaHeading, deltaY, padding0006, y, animation, padding0010, heading, x, padding0014, z, deltaZ
0815, 0, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16386, 0, 28160, 0
0815, 48, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16386, 0, 28160, 0
0815, 1056, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16386, 0, 28160, 0
0815, 2448, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16386, 0, 28160, 0
0815, 3104, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 17698, 0, 28160, 0
0815, 3520, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16674, 0, 28160, 0
0815, 3952, 0, 96, 96, 117, 247, 43460, 340, 0, 316, 16386, 0, 28160, 524
2056, 0, 0, 126, 0, 64, 160, 40200, 8, 0, 2854, 7040, 1, 24322, 0
0815, 272, 128, 96, 96, 127, 196, 43204, 778, 3, 300, 20226, 0, 28160, 683
0815, 688, 128, 96, 96, 39, 352, 42948, 619, 4, 300, 16386, 0, 28160, 1621
0815, 1168, 128, 96, 96, 60, 254, 42436, 914, 5, 300, 16386, 0, 28160, 1206
0815, 1632, 128, 96, 96, 51, 416, 42180, 132, 5, 300, 16386, 0, 28160, 853
0815, 2912, 128, 96, 96, 88, 185, 41156, 92, 3, 268, 16386, 0, 28160, 853
0815, 48, 256, 96, 96, 101, 248, 40132, 54, 0, 252, 17986, 0, 28160, 1102
0815, 480, 256, 96, 96, 63, 416, 39876, 147, 4, 252, 16386, 0, 28160, 862
0815, 1776, 256, 96, 96, 73, 231, 38596, 463, 6, 252, 16386, 0, 28160, 1556
2056, 0, 0, 0, 264, 0, 0, 40200, 512, 2, 2853, 7040, 1, 24322, 0

[trevius]

No, looking at your client update collected data looks all wrong to me. Most likely, your packet structure is way off. Unfortunately, clientupdate is one of the packets that SOE likes to completely jumble around every patch or so to throw off ShowEQ people for a couple of days lol. So, most likely the structure you are using is for an older version of the client.

For testing purposes while creating your tool, you might be better off using structs from EQEmu to test it collecting data for either Titanium or SoF, whichever you chose. Both should be considerably more accurate packet structures than what ShowEQ has for them. Also, you would probably want to test on the SoF client, since it is by far closer to Live than Titanium is.

Then, once you know your tool is able to collect everything how you want it, it is just a matter of adjusting the packet structures and opcodes to match Live. For only a few packets like this, it should be very easy. I can figure out almost all of the packet structures we might need very quickly. Though, the clientupdate one is a bit harder for me, due to how they seem to throw in crap we don't care about as what we call padding. I think clientupdate is one of the only (or very few) packets that actually breaks int32s into separate sections, and only uses some of the bits. Without looking at the ISM output from IDA, it is hard to figure out what goes where. And, I don't know how to read ISM well enough to figure that stuff out yet. I had to lean on what info ShowEQ had and do some testing to finally get it working properly for SoF.

Here is the OP_ClientUpdate struct we use in SoF:

struct PlayerPositionUpdateServer_Struct
{
/*0000*/ uint16   spawn_id;			// Entity ID of the Spawn/Player
/*0002*/ signed   padding0000:12;	// ***Placeholder
		 signed   x_pos:19;			// x coord
		 signed   padding0290:1;	// ***Placeholder
/*0006*/ signed   delta_x:13;		// change in x
		 signed   delta_y:13;		// change in y
		 signed   padding0294:6;	// ***Placeholder
/*0010*/ signed   z_pos:19;			// z coord
		 signed   delta_heading:10;	// change in heading
		 signed   padding0298:3;	// ***Placeholder
/*0014*/ signed   y_pos:19;			// y coord
		 signed   delta_z:13;		// change in z
/*0022*/ signed   animation:10;		// animation
		 unsigned heading:12;		// heading
		 signed   padding0302:10;	// ***Placeholder
/*0026*/
};

I am sure that has been completely re-arranged by now on Live. ShowEQ does require at least some of these fields to be correct for their tool to work at all, so their structure should be at least partially correct if ShowEQ works with Live currently to show NPC movement.

I believe this is the current ShowEQ version for EQLive (as of 12/1 for OP_ClientUpdate:

struct playerSpawnPosStruct
{
/*0000*/ uint16_t spawnId;
/*0002*/ signed   padding0000:12; // ***Placeholder
         signed   deltaX:13;      // change in x
         signed   padding0005:7;  // ***Placeholder
/*0006*/ signed   deltaHeading:10;// change in heading
         signed   deltaY:13;      // change in y
         signed   padding0006:9;  // ***Placeholder
/*0010*/ signed   y:19;           // y coord
         signed   animation:10;   // animation
         signed   padding0010:3;  // ***Placeholder
/*0014*/ unsigned heading:12;     // heading
         signed   x:19;           // x coord
         signed   padding0014:1;  // ***Placeholder
/*0018*/ signed   z:19;           // z coord
         signed   deltaZ:13;      // change in z
/*0022*/
};

If you know the exact loc of an NPC, you can break down the packets using the suggested packet structs from ShowEQ and see if it matches the loc info. Delta is hard to figure out this way, but X, Y and Z aren't too bad. The only problem there is that there is no way to get a #loc of an NPC from Live. It is best to just have someone that is good with IDA to figure that particular struct out. The other structs are all pretty straight forward, and don't normally change much either.

[erde]

Hello,

i have updated the extractor, you could download it here: http://www.eqdaemons.de/?page_id=6

currently generated csv files:
door.csv = Doors (OP_SpawnDoor)
object.csv = Objects (OP_GroundSpawn)
spawn.csv = Spawns (OP_ZoneEntry)
zone.csv = Zone (OP_NewZone)
zonepoints.csv = Zone Points (OP_SendZonepoints)

Still in development: ( i am still fighting with this )
MobPos.csv = Mob position updates, not usefull yet !
SpawnPos.csv = Spawn position updates, not usefull yet !

This is still in development! I will try to add sql output like the old extractor had.

greetings Stefan

[erde]

Hi,

new release => http://www.eqdaemons.de/?page_id=6

Changes:
- new csv export (all fields are exported now)

[erde]

Hi,

new release => http://www.eqdaemons.de/?page_id=6

Changes:
- csv export fixes
- better spawn handling( OP_ZoneEntry, OP_ZoneSpawns )

[trevius]

I might have to pay to get my EQLive account running again so I can test this tool out. Looks like some very nice work

[sereal]

Does anyone have a copy of the source? The links appear to be dead.