Live Data Collection (Need Community Help)
 

I'm going to make this short and sweet.

We have made huge ground parsing Magelo's site for large amounts of data to catch up PEQ all the way through the current expansion. I spent time to write a web crawler that has gathered all of the following data types from all expansions to now:

• Merchantlists
• Loot
• Tradeskills
• Spawnpoints

I will be working on getting these in one by one, my life has been super busy and it has been hard to make time for things but I'm still trying to get these things knocked out because they are rather large. I also am not rushing because Cavedude is in the middle of moving, I am probably going to end up moving here shortly myself.

Asking the Community for Help:
Anyways, I need help from the community to collect logs from live. These are the logs that are generated from /log in-game from any and all time frames that have ever been played.

What will this enable us to do?
We will be able to collect quite a wide variety of data that just simply can't be automated through web sources. This will allow us to collect:

• NPC Relative Health
• NPC Spell Casts
• NPC Faction hits
• NPC Damage Data
• NPC Aggro Emotes
• NPC Death Emotes
• Other things I'm not thinking of right now

If we can get log data from PEQ's more recent expansions (GoD, LDON) all the way through the current expansion the more we will be able to fill in the gaps of all of the data that gets manually processed through the development procedures and standards that PEQ has put in place today.

My effort is going to be to take logs from players anonymously through a form and have a script go through and tear through the data existing through those logs. These logs will be kept privately and will only be used for the benefit of this community and will only be seen by trusted developers.

The UPLOAD form can be found here: http://eoc.akkadius.com/eoc/EQEmuDat...gCollector.php

Simply upload a eqlog file of your EverQuest live characters and once the data parsing process begins, I will try to make a way to have people see what NPC's have been collected in what zones and what hasn't collected so we can take an accurate hitlist against what we need to collect as a community.

This will take a while as we get people to dump their historic logs from the past, I encourage everyone to reach out to anyone they might know to help this process move along so we can get more data.

How/What:

• Typing /log in game in eqemu will create a logfile under EverQuest/Logs/eqlog_charname_servername.txt
• We will need preferably HUGE logs that contain data spanning several zones, old and big logs are completely fine
• The more combat spam the better, I don't care about /gu or /t messages, I won't be caring about any of that

If anyone has any questions about this please feel free to post.

Oh boy. Logs, if I can find the old hard drives that I used to play and guide on, shoot. We're talking a crazy amount. But, yeah.

Short and sweet answer, you need logs, from what expansion to which expansion?

what format would you like to receive them in ? what is the max size allowed ? are you talking about the log files an or are you asking for pcap files ? i have several live logs i started in 2003-present

we are talking about standard log files created by the /log on command.

any and all logs are helpful, but mostly from GoD+ as everything else is already "completed" more or less. Akka will have to respond to file size restrictions if any.

The default format in which they are generated in is required:

eqlog_charname_server.txt

I don't have any size restrictions, just load em up!

And yes all expansions GoD to now log data should be uploaded. Can use any and all, this needs to be a big community effort.

Would you like us to trim the logs to just those expansions, or can your parser sort it out. I have logs going back years.

Well, they did mention wanting to parse NPC Aggro/Death emotes and I know we are missing a lot of them from pretty much every expansion ... But stuff that isn't on PEQ yet would probably still be more useful.

Anything and everything, my parser will be able to differentiate the data.

where would i send them to akka?
, also anything on live that you need really bad?

still got active acounts ( 95sk/95ranger/95sham/95bard/90 rougue)

Read first post, there is a URL there and I think logs from pretty much everything is fine

yeahh im asking if theres something paticuller from a expansion that needs logging down cause i can go do that lol.

It says this in the original post. Pretty self explanatory to me.

Bowmaster's a pretty simple being... He's like an orc - point him in the direction you want him to smash, and let him have at it. With your team are you capable of HoT & VoA content Bow? That'd be the place to hit if you're capable. Hell I'd LOVE to see crystallos fully working, just to finally play through it rather than read about it.

Hey Akkadius your website rejected the log I attempted to upload tonight. It said it was not a log file but it is a legit log file. I tried to upload the file a second time thinking maybe it was just a fluke but it rejected it again. Any ideas?

ok i'll send you a log of our guild chat i did type /log on in guild chat

It's going to look for the eqlog in the name as well as .txt

Well it wasn't the name that caused the problem. Just a guess but I think 100% packet loss around hop 22 may have had something to do with my upload not working.

Try it again, I changed some max upload size settings on the webserver end.

How large was the file that you were trying to upload?

It uploaded successfully today and the file size 2.08MB.

Thanks for those who have uploaded logs, but so far this is looking pretty pathetic.

We really need to get a real density of logs before any parsing systems can really be developed around this.

If you know someone with a mother load of logs spanning lots of content please contact them and see if they are willing to contribute to this.

Haha I was just thinking about this a couple of weeks ago. Glad someone got around to writing a parser before me :)

I found 80 from a variety of characters + another 26 that I renamed as part of archiving back from 2002.

Oh wow, logs from 03/04. Now we're talking progress.

Once again if I can encourage anyone and everyone to try and get them to upload historic character log files this helps the project quite a bit.

If anyone has questions please let me know.

Thanks,
Akkadius

Have you thought about posting this on like the EQ sub-reddit? I'm sure there would be people at least interested in submitting some and there is a larger chance they have logs.

Any chance you can release some aggregated data once you're done with this project? I'd like to see how older mobs compare (hp/damage wise) to what we have in the peq DB.

Yeah I definitely will, I just need to collect a decent amount of logs before I start putting together the parser.

In the middle of working on some other rather large PEQ parsing projects to get loot, merchantlist, tradeskills from all expansions.

In the meantime I just need to start building the private repository so I have something to work with when the script is being created to parse through all of the text files.

After that I plan on having a web page that people can go to and see a hitlist of what has been collected and what hasn't

Question....Is test server data useful? I've got access to a multitude of accounts that can basically box everything up to VoA stuff.

Finished my first 80 using the web interface, found another 46 (in addition to the archived ones).

After sitting in front of my computer for an hour doing those, I decided I'd script it. If you have a linux box and lots of logs, try the following script:
Change or remove the '-F EQEmuForumName=CHANGEME' to your forum name if you want it to be included.

That's quite the collection you have there and that definitely will help.

I've updated the form to include how many logs we've collected so far and the size:

http://eoc.akkadius.com/eoc/EQEmuDat...gCollector.php

How are you storing the logs? Are you using the original filename? I have multiple logs which share the same filename (I would copy them elsewhere and/or rename them to make parsing of more recent stuff easier) which, if you keep them as the same filename, could get overwritten.

No, I thought that through beforehand. They never get saved as the original filename.

Update:

We have tons of 2002-2005 logs now, we need a high density of 2004-2012 logs now.

Need your help!

UPLOAD HERE:

http://eoc.akkadius.com/eoc/EQEmuDat...gCollector.php

I have all my EQ files since I started in 99. I'll start looking through them

I've not seen anything uploaded recently, the community needs more help with this than we are getting. People have no idea how much this is going to help the community.

You need to re-frame this to what the individual is going to get out of it. Give out some worthless points or some shit and work out a deal with some of the big servers to redeem them for trinkets, then you will be overwhelmed with logs.

well... although i quit playing live many years ago, I suppose I could login and ask some of the guilders who still play if they could share some of their logs. In the next few days I will attempt to get ahold of them.

I am out of town atm and can not collect logs for the time being. I should be back home in a week or so and will resume log collections.

Everyone will get punch and pie, and leprechaun points :D

I have 14 gigs of log files from 2004-2009 when I played whatever ridiculous amount of time I played EQ Live (something like 700+ days played 2001-2009). I started running logs some time during PoP, and logged pretty much all of my group/raid gameplay through the last boss of SoD. If something was in the game during that time, I probably killed it... a lot.

But, here's the thing. If I could parse out chat before sending it, and then send them zipped somehow, I'd prefer doing it that way. If your script strips out tells/guild/channels/raid/etc., I'd be willing to set up a server and run it locally, then zip everything up and ftp it somewhere.

Don't take it personally, but there's a lot of chat in there that doesn't have any business being out in the world. I was an officer/leader of a raiding guild for years, so there's a lot of private shit in there that I don't really feel comfortable sharing since the people involved really don't have a say. Then there's just the sheer file size of it all... zipping it down to 1/10 the size would help tons on my end.

Not sure if it's worth all the hassle, let me know ;p

Always surprises me how important MMO players think their drama is. Unless you guys were talking about murder, rape or molesting children... nobody gives a shit if you thought some app was gay.

Pretty sure the script rips out chat data and only parses the combat data.