Windows Server Installer :: Modular & Scripted

[kuahara]

I've had 13 existing toons parked in PoK since my last post over an hour ago. I created 11 new toons since then and have used the GM to summon some of them around. So far all their response times with the server are 30-45ms. 25 toons online and so far no sudden jumps to 5k, 10k, 20k or 30k ms.

Only new thing right now is that I can log in literally every toon except the reserve clerics. Trying to log just those toons in is causing them to autocrash for some reason, but it's probably just a bit of bad luck at this point.

If the issue recurs, I'll post back. Though I'm not sure what else can be done at this point.

I ran that continuous ping for a while, both before all the toons were online and after. Nearly all replies came in 11ms or less. One stray packet was responded to in 383ms. I'd expect that with the server running in a VM, the replies come in 1ms or less. I administer a small Windows domain at work that has about 60 machines in it spread across two buildings. As I write this message, I VPN'd into my work network, RDP'd into my work PC, then pinged a machine in the building across the street. The response times were 4ms, 3ms, 5ms, 3ms. When I ping the gateway (same building, but opposite side of the building), all replies are <1ms.

Here on my home PC, the server and clients are literally the same machine, one is just in a VM. Not sure why I can't get <1ms replies when I ping.

Pings to the gateway, which is physically separate, come back in <1ms.

If I fire up wireshark, is there some EQ specific traffic I can look at and see if that <localaddress> tag in eqeqmu_config.xml is doing what it's supposed to do or if the traffic is still being routed through the outside interface?



Edit: Every time I think I know something....
So I started having the issue with high response times again, but now I'm guessing it has something to do with what I'm doing for new toons. I have a hotkey setup on the GM to level it to 60, increase all the skills to max, hand it some plat, turn on GM speed (temporarily), give it a bunch of AA, etc... I'd moved the new 11 toons to PoK and hadn't had an issue. Unless I just needed to wait a minute for it to kick in after moving so many so quickly. As soon as I started smacking the hotkey to scribe all spells for each of the new toons, I started having this problem, so I popped open a cmd terminal and....




Also, I'm not convinced that it's the server. I left 1 toon in Neriak and his response time in game shows 31ms. I summoned him to PoK with the GM and upon zone in, he's showing 25.4k ms. Took him quite a while to zone in too.

[kuahara]

Is there any way to undo the update of those binaries? Because ever since I updated them, none of my characters can log in while MQ2 is running. If I shut it off, they can all log in fine. If it's on, they all crash trying to enter world now.

[kuahara]

ok, so I figured it out! I'm guessing my replies aren't being looked at in this thread anymore, but I'll update for the heck of it.

It probably seems obvious at this point, but I can reproduce the problem at will now. I didn't realize I was completely running out of memory. I thought I still had a pretty safe buffer. When I am nearly full and start another EQ instance, all the times shoot up from the 20-40ms range to 500+ and start climbing to nearly 30k.

I also want to point out that I created a new server using Windows Server 2008 R2 instead of Windows Server 2012 R2 and 2008 seems waaaayyyyyyyy more stable. I don't have half the problems I was having in 2012 and my setup/installation procedure was exactly the same. (only this time I took several VM snapshots in case anything breaks and I need to go back to a stable state).

Earlier, you mentioned that EQ shouldn't have this large of a memory footprint. Any chance you know how I might go about reducing the amount of memory each instance is using? Here's a screenshot of what my host machine performance looked like while everything was stable. And things were stable all day yesterday, all through the night while I slept, all morning. But at 31.5GB usage when I start a new instance and load up that last toon to push it over the edge. All in game response times shot as described above.



And here is a snapshot of the processes, showing how much memory each instance is using. How can I reduce this?


You also mentioned something to me about possibly switching to ROF2 client and I noticed you made that recommendation to several others as well. If possible, I'd like to keep the underfoot client. It has been very stable when playing on the existing EMU servers and I like the version of MQ that I have that runs with it. MQ is causing none of these problems by the way. eqgame.exe is using about that much memory even with MQ turned off. I'm afraid that if I switch clients, I might lose some of the useful plugins that I make regular use of and playing without MQ just doesn't work for me. I'd rather quit.

If you have any advice, I'd love to hear it. Going to google this problem for now.

Thanks.



Edit: That large CPU spike you see in the first screenshot is not normal CPU usage. I just happened to be loading up another instance when I took it. Idle, the CPU sits anywhere from 4-15% depending on what it's doing, even when I have 20+ toons logged in.

[Akkadius]

You're asking me how to reduce memory footprint of your client and heaps of other questions that are now outside of the scope of this thread. Some of which I've already answered

There are MQ2 builds for ROF2 they may not have your active hacks (warping etc.) but they are there at mqemulator.net

ROF2 is a much better client and far better at memory management. You can probably find it in a big red button on PEQ's website

Please start a new thread and we'll go from there - everyone on this thread chain gets notified via E-Mail

[kuahara]

If this isn't an appropriate place for me to follow up, just delete this. It's possible I'm the only idiot that would do something like this without realizing the consequences, but I wanted to follow up anyway in case someone else ran into it.

A huge part of my memory consumption wasn't just the eqgame.exe instances. Task manager on the host machine won't show it, but if you run a server in a VM and allot it 16GB of memory, it will (at least with this sql db running) consume the entire 16GB even when it doesn't actually need it. The guest taskmgr shows it only needed 2GB of memory, but with the VM powered up, I was losing 16GB.

Fortunately, the solution is simple. I just powered it off and lowered it's available memory to 4GB, and restarted. I was operating under the assumption that the VM would only use what it needed and was trying to play it safe.

Also, I tried ROF2. It does save me about 200MB on the first few instances, but the drawbacks weren't beneficial enough for me to switch away from UF.


Thanks for all the help and sorry for the mess.

[Akkadius]

Originally Posted by kuahara

If this isn't an appropriate place for me to follow up, just delete this. It's possible I'm the only idiot that would do something like this without realizing the consequences, but I wanted to follow up anyway in case someone else ran into it.

A huge part of my memory consumption wasn't just the eqgame.exe instances. Task manager on the host machine won't show it, but if you run a server in a VM and allot it 16GB of memory, it will (at least with this sql db running) consume the entire 16GB even when it doesn't actually need it. The guest taskmgr shows it only needed 2GB of memory, but with the VM powered up, I was losing 16GB.

Fortunately, the solution is simple. I just powered it off and lowered it's available memory to 4GB, and restarted. I was operating under the assumption that the VM would only use what it needed and was trying to play it safe.

Also, I tried ROF2. It does save me about 200MB on the first few instances, but the drawbacks weren't beneficial enough for me to switch away from UF.


Thanks for all the help and sorry for the mess.

So you set your MySQL configuration to use 16GB of your memory?

What drawbacks are you talking about

[kuahara]

No, when I configured the virtual machine in vmware, I told it that it could have 16GB of memory, 50GB of disk space, access to all 4 CPU cores, etc... When it comes to disk space, it only consumes what it actually needs. I assumed the same was true with memory, but it is not.

After a reboot of this machine, I popped open taskmgr and added up the amount of memory consumed by what little was running. Less than 2GB was in use. If I go over to the performance tab right next to processes, it shows about the same thing. Almost no memory is in use. When I fire up vmware and start up the server, things change. My processes tab shows vmware uses a few hundred MB of memory, but it still adds up to less than 2GB of total memory consumption. The performance tab, however, shows that I'm using 18.6GB of memory.

I go over to the VM (server 2008 R2) and open taskmgr there to check. The performance tab there shows the server is only using 2.7GB of memory.

If I shut down the VM, the host machine gets 16GB of memory back under the performance tab. It seems pretty obvious that the VM is reserving 16GB of memory from the host simply because it is allowed to use it, whether it needs it or not.

Another assumption I made that doesn't seem to hold true is that if the performance tab showed I was using X amount of memory, then I'd be able to go over to the processes tab, which lists every running process and how much memory it is using, and be able to add that up and get approximately the same total that the performance tab shows I am using. With vmware in play, this is not even remotely close to true.

I can provide screenshots of all of this in action.



Edit: Sorry, I didn't answer the question about the drawbacks of ROF2. I have an underfoot client with an MQ2 build for UF that includes the active hacks you referred to earlier. As a workaround, I could just make all my toons GMs and use #goto instead. I also couldn't make autologin work. I referred to your instructions in the wiki, but at the field where it says "server=" there's no instructions if you are running your own private server. I tried leaving it set to EZ, blank, and a number of other things that I thought might make it work with my personal server, but when I run the shortcut to get into the game, the result is always the same. It says there's an error in the GUI XML files and to check the log. The log only contains a timestamp and no other information. I noticed the maps were far from complete. I might just be able to copy the map files from my UF build and paste them in. There were a number of other things missing (a lot just from the UF MQ2 build) that I found I'd need to come up with alternate methods of doing. In the end, it just seemed like it'd make better sense to continue working on what I'd started before downloading ROF2. I might give it another shot later if there's a persuasive enough reason to go back.