lag problems

[flbren]

What would cause lag on my server once 25ish toons enter a zone? The lag goes up steady to 15000+ms. With a couple toons in zone I have no issues.For example I can have 24 toons loaded in guild lobby and lag out from anywhere to 500ms to 15000ms but have 3 toons in pok at 60ms. Lag doesn't seem to affect other zones.

[Nightrider84]

Now to clarify is this 25 toons in 1 zone while 3 toons are in a seperate zone with no issues all at the same time? Or do you mean if you have more than X players at anytime the server lags everywhere?

[flbren]

The lag is contained in the zone with the 25 toons and other zones with 3 or 4 players run fine. When loading 25ish toons in the zone when I first log in my ms is fine. But after 20ish toons its starts rising to 200 300 1000 etc.

[Nightrider84]

Well that could be a few things, Could be router related. Sometimes older routers can't handle the required lanes of traffic from multiple locations. Quick test for this would be to have like 30 toons logged in. But put them in about 10 different zones and see if the problem shows up again. If the problem doesn't show up than it might be something on the server itself thats causing the issue. But that is a weird issue to have honestly. Also are these 25 seperate people or are they all bots ect?

[Scorpious2k]

You might also want to check CPU usage, RAM usage (and swap space), and especially bandwidth. Sometimes ISP's talk about xx bps but they are really talking about downlink (incoming) and uplink (outgoing) is much MUCH lower, which would limit packets going to clients.

[flbren]

25 separate accounts that I loaded. What I can tell everything looked ok from the host.

[Nightrider84]

well test the 25 accounts at once in difference zones and see if you get that same latency spike, if you dont then have them zone into the same zone and see if the problem arrises

[ptarp]

I'm having this same problem. Running the code on Windows 10, there is no problem.. running on any windows server version, the problem is exactly the same.

Running on Windows Server, if you distribute the toons in different zones, most times the lag will go away.. Put them all in the same zone, ping and packet loss increase until it's unplayable. EQ reports up to 5k m.s. pings on a server within my local network. Server CPU stays around 6%, memory around 3%, and network around 250kbs sent on a gigabit connection.
If you do have toons in other zones, they aren't affected until that zone is loaded up with toons also. 22 or so in lobby and 6 in WoS.. the 6 in WoS can keep playing, the ones in guild lobby all lag out.


Running on top of windows 10 on the same machine with a copy of the same code, and copy of the same database, the problem goes away. Log in same toons, put them all in the same zone, and it stays playable.

[Akkadius]

Just for visibility guys, this is a known issue that we have been working on as of late.

I'll keep you posted when we have an update, I've not had time for my emulator backlog in the past few weeks

[Tom Cross]

Does anyone know if this is happening on linux boxes too?

[Akkadius]

Happens on Linux too yes, slightly less prevalent because of net efficiency of the linux core but don't both making a switching decision on that alone

We have a solution, we've been carefully testing it over the past few weeks because its a complex problem

I'll update when we have it in, we need to run it on PEQ for a while now that we've had it running at smaller scale

[Tom Cross]

So we get the whole community to get on the PEQ and we all have a party in the Nexus for a good test run?

[Akkadius]

Haha, not quite necessary, we have between 500-1000 toons on in a given day we should be able to give it a good run

[ptarp]

Originally Posted by Akkadius

Just for visibility guys, this is a known issue that we have been working on as of late.

I'll keep you posted when we have an update, I've not had time for my emulator backlog in the past few weeks

I've tinkered with this a bit.. I've gotten ping times to drop down to 350 with 47 in zone by dropping some of the extra database calls that aren't needed in Client::Save(uint8 iCommitNow) function.

Binds are a big one especially now with 5 binds. They don't need saved every loop through client::save, because they are already saved every time they change.

Pets are another problem.. Every time it goes through the loop, pets shouldn't be saved for every toon. Even warriors were getting hit by database.SavePetInfo(this); call. I moved that up 4 lines just above the } else {.

I have also added a Boolean to mine to prevent saving tribute if it hasn't changed, though that call to database.SaveCharacterTribute(this->CharacterID(), &m_pp); could probably be moved right into Client::Handle_OP_TributeUpdate function.

Still weeding some of it out..

[Akkadius]

These are all separate from the issue reported in the thread

There is a network problem that occurs under the condition where enough clients are in a zone and one or many have not so great connections our server network code will resend very aggressively which results in a server with a lower CPU frequency such as a Xeon 2.3Ghz to essentially choke itself from the rapid burst of resend packets.

The resend packets are choked primarily by compressing packets and sending them over the line at a rapid rate and the CPU will max out for that thread and can take several minutes to recover from

Correcting the resend logic is a complex issue because we use dynamic algorithms for resend rate and packet recovery that has to take in a lot of different factors.

We have some ideal settings figured out and are looking to get them pushed to mainline as soon as we feel confident that they don't produce any more regressions