lag problems

[Nightrider84]

well test the 25 accounts at once in difference zones and see if you get that same latency spike, if you dont then have them zone into the same zone and see if the problem arrises

[ptarp]

I'm having this same problem. Running the code on Windows 10, there is no problem.. running on any windows server version, the problem is exactly the same.

Running on Windows Server, if you distribute the toons in different zones, most times the lag will go away.. Put them all in the same zone, ping and packet loss increase until it's unplayable. EQ reports up to 5k m.s. pings on a server within my local network. Server CPU stays around 6%, memory around 3%, and network around 250kbs sent on a gigabit connection.
If you do have toons in other zones, they aren't affected until that zone is loaded up with toons also. 22 or so in lobby and 6 in WoS.. the 6 in WoS can keep playing, the ones in guild lobby all lag out.


Running on top of windows 10 on the same machine with a copy of the same code, and copy of the same database, the problem goes away. Log in same toons, put them all in the same zone, and it stays playable.

[Akkadius]

Just for visibility guys, this is a known issue that we have been working on as of late.

I'll keep you posted when we have an update, I've not had time for my emulator backlog in the past few weeks

[ptarp]

Originally Posted by Akkadius

Just for visibility guys, this is a known issue that we have been working on as of late.

I'll keep you posted when we have an update, I've not had time for my emulator backlog in the past few weeks

I've tinkered with this a bit.. I've gotten ping times to drop down to 350 with 47 in zone by dropping some of the extra database calls that aren't needed in Client::Save(uint8 iCommitNow) function.

Binds are a big one especially now with 5 binds. They don't need saved every loop through client::save, because they are already saved every time they change.

Pets are another problem.. Every time it goes through the loop, pets shouldn't be saved for every toon. Even warriors were getting hit by database.SavePetInfo(this); call. I moved that up 4 lines just above the } else {.

I have also added a Boolean to mine to prevent saving tribute if it hasn't changed, though that call to database.SaveCharacterTribute(this->CharacterID(), &m_pp); could probably be moved right into Client::Handle_OP_TributeUpdate function.

Still weeding some of it out..

[Akkadius]

These are all separate from the issue reported in the thread

There is a network problem that occurs under the condition where enough clients are in a zone and one or many have not so great connections our server network code will resend very aggressively which results in a server with a lower CPU frequency such as a Xeon 2.3Ghz to essentially choke itself from the rapid burst of resend packets.

The resend packets are choked primarily by compressing packets and sending them over the line at a rapid rate and the CPU will max out for that thread and can take several minutes to recover from

Correcting the resend logic is a complex issue because we use dynamic algorithms for resend rate and packet recovery that has to take in a lot of different factors.

We have some ideal settings figured out and are looking to get them pushed to mainline as soon as we feel confident that they don't produce any more regressions

[ptarp]

Yes. This is different. Server is running on an i5. CPU stays low. Around 6%. The issue with me seems that entity_list.Process() is taking too long. By the time you get through the whole list of clients, the first one is starving for packets. Each client added increases the ms reading reported by EQ (F11 to show it in top left corner)

This same thing will affect all operating systems. Win 10 seems better than server, but is still not working well.

I enabled MySQL logging to disk. Logging was on a secondary SSD drive. MySQL data files on drive C:, server folder on D:. that's how I saw how many times per second the MySQL was being accessed.. Some of it may not take that much time, but all together, it's a DoS bomb for the hard drive even if it's SSD like mine. Turn it on and look at a zone with over 24 or 25 in it, and you'll see what I mean. Look at the times for the first client going through client::save and compare it to the last.
Since I'm logging on a separate hard drive, performance doesn't change when I turn logging on/off.

I recommend you think about dealing with this before you worry about re-send logic.
Correct the issues I'm talking about, and your re-send issues may go away.
Hope this helps.

[Akkadius]

Originally Posted by ptarp

Yes. This is different. Server is running on an i5. CPU stays low. Around 6%. The issue with me seems that entity_list.Process() is taking too long. By the time you get through the whole list of clients, the first one is starving for packets. Each client added increases the ms reading reported by EQ (F11 to show it in top left corner)

This same thing will affect all operating systems. Win 10 seems better than server, but is still not working well.

I enabled MySQL logging to disk. Logging was on a secondary SSD drive. MySQL data files on drive C:, server folder on D:. that's how I saw how many times per second the MySQL was being accessed.. Some of it may not take that much time, but all together, it's a DoS bomb for the hard drive even if it's SSD like mine. Turn it on and look at a zone with over 24 or 25 in it, and you'll see what I mean. Look at the times for the first client going through client::save and compare it to the last.
Since I'm logging on a separate hard drive, performance doesn't change when I turn logging on/off.

I recommend you think about dealing with this before you worry about re-send logic.
Correct the issues I'm talking about, and your re-send issues may go away.
Hope this helps.

Again, these are completely unrelated.

Just because you saw a bunch of disk activity and a bunch of queries in a file doesn't mean that its the reason for lag. If you have an improperly tuned MySQL server along with something enabled that is pegging your MySQL server that is another thing and I'm happy to help diagnose those with you

I want you to contrast all of what you observed with PEQ's disk activity:

http://peq.akkadius.com:19999/#menu_...late;help=true

PEQ has over 800 players right now at maybe stays around 1MB/s writes if at all and occasional bursts, IO operations stay down at a very very low amount even for 800 players

Client::Save is a very light operation, there's maybe a handful of INSERT's or REPLACE into's that occur which are all sub 10ms inserts. We could use less Client::Save's in general but it really isn't the problem here

You don't need to turn on the MySQL general log when you can see exactly what a zone process is doing by enabling MySQL logging at the process level. Even if you pipe that to another drive it still is overhead to the MySQL process

https://github.com/EQEmu/Server/wiki...-System#gm-say

In the `logsys_categories` table you can shut off any category you are piping to file

Back to the Network Issue

We know exactly what's going on with the network issue because we've taken CPU snapshot profiles during the problem. It's just not a quick "Fix" and we typically chose to go through a very careful staged approach before reintroducing this into mainline because of the complex factors involved

The reason we've seen this far less on PEQ is because PEQ has a OC'ed 5Ghz core processor, DDR4 memory and NVME Datacenter SSD's. When the zone processes goes into resend storm logic, it can keep up with the very aggressive resend logic just enough until the client either disconnects from their own terrible connection or the client itself recovers.

There is still a breaking point with our hardware however, it just takes a lot more to get there. If we had over 100 toons in a zone PEQ and we had something produce enough resend logic (Like raid combat spam burning) it would trip the same inflection point that most folks are seeing on their Windows nodes at 20-40 people in a zone with 2.6Ghz ish processors and whatever else they're using on their boxes. Even with over 100 toons it is still very rare to see it just because of the very tight hardware that is being utilized

Regardless, you shouldn't need the above specs to run a server, that is not the point at all. The point is why we've not run into this issue up until this point because most of our code QA goes through PEQ and our hardware has been masking the problem. Before we released the new netcode overhaul to mainline we went through several several iterations of issues and actually drastically improved our overall netcode utilization massively which I am still super stoked about to this day, we just have this one issue plaguing people and we will have it resolved soon, so just stay tuned for updates