Windows Server Installer :: Modular & Scripted

[kuahara]

Well... I'm back to where I started. I have no idea what was causing that problem. The bards have been afk twisting, took the wife out for about an hour, came back, and everyone's response time is at 90-135ms.

Was hoping to have something useful to reply back with. Sorry for all the fruitless chatter.

[Akkadius]

It sounds like you are experiencing packet loss. These symptoms occur when the server starts having to re-transmit packets to many clients at once and the ping/latency response from the client perspective rises.

If you're doing this on the same machine through VM etc. you may want to investigate potential bottlenecks or other things going on. Again we have folks who zone 50+ toons on our higher player-base servers and there are 0 issues unless their connection is experiencing packet loss, these sorts of symptoms occur where eventually they completely desync or things get sluggish.

18 clients - my suggestion as well is to use all the filters you can on your boxes, not MQ2 filter, but the client filters. These client filters are pushed up to the server and the server will not send you the packets for those messages, that can heavily reduce spam/resends in combat scenarios etc. If you were doing lots of combat and then you tried to zone not long after something like this that would cause your problem. Another client side thing that you should make sure you're doing as well is making sure that /log is disabled - this is a huge performance hit to the client and I wouldn't ever have it enabled unless you really need to log something

HTH

Akk

[kuahara]

I'll check to see if logs are on here in a bit and disable if needed.

I did finally do something that seems to have triggered it. I created 2 new toons while 21 other toons were online and in PoK. Both DE races. They each loaded into neriak with 30ms response times to the server, but all the toons in PoK had their response times shoot from 135ish to 5000-8000.

When I created one of the toons, I hit exit > logout and quit/closed the 2nd client so I could create only one at a time. I noticed that if I ever log a character out all the way to server select, then back to char select -> unti world, that instance of EQ will have severe artifacting issues (super stretched images that don't even closely resemble what the image is supposed to be). The artifacting is resolved by closing that one client completely and restarting it.

Am going to finish creating my toons. I expect to have around 30 when completely done and then see if giving everything a fresh start resolves the issue.

What response times are your 50+ toons that are all running on the same machine looking at?

I'd be real surprised if I was experiencing loss between 2 machines in the same network. Especially if it's not touching that outside interface anymore. I will do a continuous ping from client to server and check, though. Not sure if the results are relevant to the EQ client/server relationship or not. I assume it'd be indicative.

[Akkadius]

Originally Posted by kuahara

I'll check to see if logs are on here in a bit and disable if needed.

I did finally do something that seems to have triggered it. I created 2 new toons while 21 other toons were online and in PoK. Both DE races. They each loaded into neriak with 30ms response times to the server, but all the toons in PoK had their response times shoot from 135ish to 5000-8000.

When I created one of the toons, I hit exit > logout and quit/closed the 2nd client so I could create only one at a time. I noticed that if I ever log a character out all the way to server select, then back to char select -> unti world, that instance of EQ will have severe artifacting issues (super stretched images that don't even closely resemble what the image is supposed to be). The artifacting is resolved by closing that one client completely and restarting it.

Am going to finish creating my toons. I expect to have around 30 when completely done and then see if giving everything a fresh start resolves the issue.

What response times are your 50+ toons that are all running on the same machine looking at?

I'd be real surprised if I was experiencing loss between 2 machines in the same network. Especially if it's not touching that outside interface anymore. I will do a continuous ping from client to server and check, though. Not sure if the results are relevant to the EQ client/server relationship or not. I assume it'd be indicative.

One thing I want you to do is make sure you have the most up to date binaries. The installer doesn't keep the latest bleeding edge server binaries, only approved stable releases

Open up cmd

perl eqemu_server.pl windows_server_latest

Baseline that for troubleshooting and go from there

[kuahara]

Just completed that. I also forgot to mention that the artifacting thing I described, when logging out a character and logging in a different character on the same account.... or logging one out to server select, then back in again without closing the client. That issue happens even when I play on other people's servers. I'm not terribly worried about that one particular issue because it's easily fixed by just closing that one client and reloading it.

Screenshotted the output of the command you had me run.




Will post back with any changes. Going to go ahead and load up the other 20 toons for the heck of it right now, then create more and see if it causes their response times to shoot up.

[kuahara]

Not sure if it matters, but this happened when I started the server back up after the update. Not sure it effected anything as I was able to log my GM character in immediately after without any trouble. Just fyi.





Edit: Checked my eqclient.ini and Log=FALSE

[kuahara]

I've had 13 existing toons parked in PoK since my last post over an hour ago. I created 11 new toons since then and have used the GM to summon some of them around. So far all their response times with the server are 30-45ms. 25 toons online and so far no sudden jumps to 5k, 10k, 20k or 30k ms.

Only new thing right now is that I can log in literally every toon except the reserve clerics. Trying to log just those toons in is causing them to autocrash for some reason, but it's probably just a bit of bad luck at this point.

If the issue recurs, I'll post back. Though I'm not sure what else can be done at this point.

I ran that continuous ping for a while, both before all the toons were online and after. Nearly all replies came in 11ms or less. One stray packet was responded to in 383ms. I'd expect that with the server running in a VM, the replies come in 1ms or less. I administer a small Windows domain at work that has about 60 machines in it spread across two buildings. As I write this message, I VPN'd into my work network, RDP'd into my work PC, then pinged a machine in the building across the street. The response times were 4ms, 3ms, 5ms, 3ms. When I ping the gateway (same building, but opposite side of the building), all replies are <1ms.

Here on my home PC, the server and clients are literally the same machine, one is just in a VM. Not sure why I can't get <1ms replies when I ping.

Pings to the gateway, which is physically separate, come back in <1ms.

If I fire up wireshark, is there some EQ specific traffic I can look at and see if that <localaddress> tag in eqeqmu_config.xml is doing what it's supposed to do or if the traffic is still being routed through the outside interface?



Edit: Every time I think I know something....
So I started having the issue with high response times again, but now I'm guessing it has something to do with what I'm doing for new toons. I have a hotkey setup on the GM to level it to 60, increase all the skills to max, hand it some plat, turn on GM speed (temporarily), give it a bunch of AA, etc... I'd moved the new 11 toons to PoK and hadn't had an issue. Unless I just needed to wait a minute for it to kick in after moving so many so quickly. As soon as I started smacking the hotkey to scribe all spells for each of the new toons, I started having this problem, so I popped open a cmd terminal and....




Also, I'm not convinced that it's the server. I left 1 toon in Neriak and his response time in game shows 31ms. I summoned him to PoK with the GM and upon zone in, he's showing 25.4k ms. Took him quite a while to zone in too.

[kuahara]

Is there any way to undo the update of those binaries? Because ever since I updated them, none of my characters can log in while MQ2 is running. If I shut it off, they can all log in fine. If it's on, they all crash trying to enter world now.

[kuahara]

ok, so I figured it out! I'm guessing my replies aren't being looked at in this thread anymore, but I'll update for the heck of it.

It probably seems obvious at this point, but I can reproduce the problem at will now. I didn't realize I was completely running out of memory. I thought I still had a pretty safe buffer. When I am nearly full and start another EQ instance, all the times shoot up from the 20-40ms range to 500+ and start climbing to nearly 30k.

I also want to point out that I created a new server using Windows Server 2008 R2 instead of Windows Server 2012 R2 and 2008 seems waaaayyyyyyyy more stable. I don't have half the problems I was having in 2012 and my setup/installation procedure was exactly the same. (only this time I took several VM snapshots in case anything breaks and I need to go back to a stable state).

Earlier, you mentioned that EQ shouldn't have this large of a memory footprint. Any chance you know how I might go about reducing the amount of memory each instance is using? Here's a screenshot of what my host machine performance looked like while everything was stable. And things were stable all day yesterday, all through the night while I slept, all morning. But at 31.5GB usage when I start a new instance and load up that last toon to push it over the edge. All in game response times shot as described above.



And here is a snapshot of the processes, showing how much memory each instance is using. How can I reduce this?


You also mentioned something to me about possibly switching to ROF2 client and I noticed you made that recommendation to several others as well. If possible, I'd like to keep the underfoot client. It has been very stable when playing on the existing EMU servers and I like the version of MQ that I have that runs with it. MQ is causing none of these problems by the way. eqgame.exe is using about that much memory even with MQ turned off. I'm afraid that if I switch clients, I might lose some of the useful plugins that I make regular use of and playing without MQ just doesn't work for me. I'd rather quit.

If you have any advice, I'd love to hear it. Going to google this problem for now.

Thanks.



Edit: That large CPU spike you see in the first screenshot is not normal CPU usage. I just happened to be loading up another instance when I took it. Idle, the CPU sits anywhere from 4-15% depending on what it's doing, even when I have 20+ toons logged in.

[Akkadius]

You're asking me how to reduce memory footprint of your client and heaps of other questions that are now outside of the scope of this thread. Some of which I've already answered

There are MQ2 builds for ROF2 they may not have your active hacks (warping etc.) but they are there at mqemulator.net

ROF2 is a much better client and far better at memory management. You can probably find it in a big red button on PEQ's website

Please start a new thread and we'll go from there - everyone on this thread chain gets notified via E-Mail

[kuahara]

If this isn't an appropriate place for me to follow up, just delete this. It's possible I'm the only idiot that would do something like this without realizing the consequences, but I wanted to follow up anyway in case someone else ran into it.

A huge part of my memory consumption wasn't just the eqgame.exe instances. Task manager on the host machine won't show it, but if you run a server in a VM and allot it 16GB of memory, it will (at least with this sql db running) consume the entire 16GB even when it doesn't actually need it. The guest taskmgr shows it only needed 2GB of memory, but with the VM powered up, I was losing 16GB.

Fortunately, the solution is simple. I just powered it off and lowered it's available memory to 4GB, and restarted. I was operating under the assumption that the VM would only use what it needed and was trying to play it safe.

Also, I tried ROF2. It does save me about 200MB on the first few instances, but the drawbacks weren't beneficial enough for me to switch away from UF.


Thanks for all the help and sorry for the mess.

[Akkadius]

Originally Posted by kuahara

If this isn't an appropriate place for me to follow up, just delete this. It's possible I'm the only idiot that would do something like this without realizing the consequences, but I wanted to follow up anyway in case someone else ran into it.

A huge part of my memory consumption wasn't just the eqgame.exe instances. Task manager on the host machine won't show it, but if you run a server in a VM and allot it 16GB of memory, it will (at least with this sql db running) consume the entire 16GB even when it doesn't actually need it. The guest taskmgr shows it only needed 2GB of memory, but with the VM powered up, I was losing 16GB.

Fortunately, the solution is simple. I just powered it off and lowered it's available memory to 4GB, and restarted. I was operating under the assumption that the VM would only use what it needed and was trying to play it safe.

Also, I tried ROF2. It does save me about 200MB on the first few instances, but the drawbacks weren't beneficial enough for me to switch away from UF.


Thanks for all the help and sorry for the mess.

So you set your MySQL configuration to use 16GB of your memory?

What drawbacks are you talking about

[kuahara]

No, when I configured the virtual machine in vmware, I told it that it could have 16GB of memory, 50GB of disk space, access to all 4 CPU cores, etc... When it comes to disk space, it only consumes what it actually needs. I assumed the same was true with memory, but it is not.

After a reboot of this machine, I popped open taskmgr and added up the amount of memory consumed by what little was running. Less than 2GB was in use. If I go over to the performance tab right next to processes, it shows about the same thing. Almost no memory is in use. When I fire up vmware and start up the server, things change. My processes tab shows vmware uses a few hundred MB of memory, but it still adds up to less than 2GB of total memory consumption. The performance tab, however, shows that I'm using 18.6GB of memory.

I go over to the VM (server 2008 R2) and open taskmgr there to check. The performance tab there shows the server is only using 2.7GB of memory.

If I shut down the VM, the host machine gets 16GB of memory back under the performance tab. It seems pretty obvious that the VM is reserving 16GB of memory from the host simply because it is allowed to use it, whether it needs it or not.

Another assumption I made that doesn't seem to hold true is that if the performance tab showed I was using X amount of memory, then I'd be able to go over to the processes tab, which lists every running process and how much memory it is using, and be able to add that up and get approximately the same total that the performance tab shows I am using. With vmware in play, this is not even remotely close to true.

I can provide screenshots of all of this in action.



Edit: Sorry, I didn't answer the question about the drawbacks of ROF2. I have an underfoot client with an MQ2 build for UF that includes the active hacks you referred to earlier. As a workaround, I could just make all my toons GMs and use #goto instead. I also couldn't make autologin work. I referred to your instructions in the wiki, but at the field where it says "server=" there's no instructions if you are running your own private server. I tried leaving it set to EZ, blank, and a number of other things that I thought might make it work with my personal server, but when I run the shortcut to get into the game, the result is always the same. It says there's an error in the GUI XML files and to check the log. The log only contains a timestamp and no other information. I noticed the maps were far from complete. I might just be able to copy the map files from my UF build and paste them in. There were a number of other things missing (a lot just from the UF MQ2 build) that I found I'd need to come up with alternate methods of doing. In the end, it just seemed like it'd make better sense to continue working on what I'd started before downloading ROF2. I might give it another shot later if there's a persuasive enough reason to go back.

[Akkadius]

If I shut down the VM, the host machine gets 16GB of memory back under the performance tab. It seems pretty obvious that the VM is reserving 16GB of memory from the host simply because it is allowed to use it, whether it needs it or not.

You need merely < 4GB of memory for the guest if its just you.

Boot 30 dynamics in the launcher and call it a day.

Production servers as a comparison:

PEQ: http://peq.akkadius.com:19999/#menu_...pu;theme=slate

EZ: http://eznp.akkadius.com:19999/#menu...pu;theme=slate

Both servers are using roughly 5GB and they have quite a few processes running and many players on

Sorry, I didn't answer the question about the drawbacks of ROF2. I have an underfoot client with an MQ2 build for UF that includes the active hacks you referred to earlier. As a workaround, I could just make all my toons GMs and use #goto instead.

As far as #goto - go to your commands table and set the status for the command to '0' so any of your characters can use it without special gm flags.

. I tried leaving it set to EZ, blank, and a number of other things that I thought might make it work with my personal server, but when I run the shortcut to get into the game, the result is always the same.

AutoLogin on EZ's site works if you follow it verbatim - you have to use the build listed and you have to replace the file with the contents on the guide.

To find the server name you look in eqlsPlayerData.ini

I noticed the maps were far from complete.

Maps are simply downloaded from a 3rd party source, such as mapfiend

There were a number of other things missing (a lot just from the UF MQ2 build) that I found I'd need to come up with alternate methods of doing. In the end, it just seemed like it'd make better sense to continue working on what I'd started before downloading ROF2. I might give it another shot later if there's a persuasive enough reason to go back.

Ultimately whatever you choose to use client use is your preference, 98% of players on EZ/PEQ use ROF2 for a reason. The client is far better in so many ways, but they may not matter to you. I'm simply helping you with your issues

HTH

[kuahara]

I tried this with both server=2 and what you see in the screenshot. Both ways result in:



The log file that refers to contains only a timestamp and no other information.

My ini file contains exactly what the instructions at http://wiki.ezserver.online/MQ2_Autologin_Setup say it should. It was a direct copy/paste from your post.


As an aside, any chance you know how to get rid of the SC/EQ button in the ROF2 client?