lag problems

**Akkadius** · #1 03-02-2019, 06:50 PM

We pushed changes last night that have been tested on PEQ for over a week with 800+ toons with no issues. Also tested on Legacy of Norrath before they shut down

https://ci.appveyor.com/api/projects...86-no-bots.zip

Give that a whirl

Drakiyth · #2 03-02-2019, 09:48 PM

Quote:

Originally Posted by Akkadius

We pushed changes last night that have been tested on PEQ for over a week with 800+ toons with no issues. Also tested on Legacy of Norrath before they shut down

https://ci.appveyor.com/api/projects...86-no-bots.zip

Give that a whirl

I plan to add this tomorrow morning to Varlyndria. We all thank you for this fix.

eldarian · #3 03-03-2019, 11:16 AM

Quote:

Originally Posted by Drakiyth

I plan to add this tomorrow morning to Varlyndria. We all thank you for this fix.

Let me know if this fix worked for you in any areas

Drakiyth · #4 03-03-2019, 05:08 PM

Quote:

Originally Posted by eldarian

Let me know if this fix worked for you in any areas

I added the source code from Akkadius' link above to Varlyndria early this morning, and then did a stress test with the server that didn't go so well. I even tried the pull method of the latest unstable source in the folder. The stress test in Nexus started bugging out with 18+ players when the spike came back. It does appear to be better than it was before, but not what I was expecting. Varlyndria is currently on a AWS Large T3 windows system. It has held over 118 clients online + pets just fine as long as they are in different zones/instances of the high-traffic hubs and under 11 in total. (on average). Now it seems like 16-18 or so, but the lag does come back full force and spikes the zone out badly -- eventually crashing it, or forcing me to shut it down.

I've heard from a source on my discord that Linux using developers are having more luck with it. I've been running windows since I started with EQemu and I've never seen an issue like this before, aside from not having enough connection speed to handle the player population.

At this point, I am hesitant/undecided to see if a stronger connection than T3 Large would produce better results with the change.

Any professional advice that can be given on the situation would be helpful.

ptarp · #5 03-04-2019, 09:38 AM

There just appears to be something in the windows compile that's making it "hiccup".. I'm wondering if I have to switch to Linux.

Maze_EQ · #6 03-04-2019, 11:02 AM

Quote:

Originally Posted by Akkadius

We pushed changes last night that have been tested on PEQ for over a week with 800+ toons with no issues. Also tested on Legacy of Norrath before they shut down

https://ci.appveyor.com/api/projects...86-no-bots.zip

Give that a whirl

It worked on our dev build with 80 clients in the same zone.

Our dev environment previously couldn't handle 20+.

ptarp · #7 03-04-2019, 12:36 PM

Quote:

Originally Posted by Maze_EQ

It worked on our dev build with 80 clients in the same zone.

Our dev environment previously couldn't handle 20+.

You're running windows?

**Akkadius** · #8 03-04-2019, 02:44 PM

So - we could be dealing with a few factors here, while the resend issue was a very valid issue that we took care of, I have a hunch that something is of influence in the windows realm here

I have another question for you guys, where have you guys been getting your binaries?

Have you been compiling them yourselves? In the past few months we switched our main source of windows binary updates from our CI system and I just want to rule out a bad or imperformant library or compilation setting

At the end of the day, Windows or Linux you should be able to run on either, we'll get it figured out

Maze_EQ · #9 03-04-2019, 04:01 PM

Quote:

Originally Posted by Akkadius

So - we could be dealing with a few factors here, while the resend issue was a very valid issue that we took care of, I have a hunch that something is of influence in the windows realm here

I have another question for you guys, where have you guys been getting your binaries?

Have you been compiling them yourselves? In the past few months we switched our main source of windows binary updates from our CI system and I just want to rule out a bad or imperformant library or compilation setting

At the end of the day, Windows or Linux you should be able to run on either, we'll get it figured out

I built these myself.

I'll see if i can repro with your installer.

**Akkadius** · #10 03-04-2019, 07:56 PM

For the sake of troubleshooting, can you guys try this build? No guarantees but it will tell us something if the symptoms go away

https://ci.appveyor.com/project/KimL...2988/artifacts

Drakiyth · #11 03-04-2019, 09:27 PM

Quote:

Originally Posted by Akkadius

So - we could be dealing with a few factors here, while the resend issue was a very valid issue that we took care of, I have a hunch that something is of influence in the windows realm here

I have another question for you guys, where have you guys been getting your binaries?

Have you been compiling them yourselves? In the past few months we switched our main source of windows binary updates from our CI system and I just want to rule out a bad or imperformant library or compilation setting

At the end of the day, Windows or Linux you should be able to run on either, we'll get it figured out

Lately I've been getting the updated source and binaries from the installer prompt in the main server folder: eqemu_server.pl

When I tried it the first time I got the files you put on this thread and also tried the latest unstable I pulled from there.

I'll load up those new source files you posted in a few here, stress test, and than share our experiences on here.

Drakiyth · #12 03-04-2019, 11:59 PM

The experiment with the binaries you posted above failed at 18 players with a few pets out. We spiraled out of control with lag in the Nexus.

**Akkadius** · #13 03-05-2019, 03:05 PM

Sounds good, thanks for trying that

Good News

Our core configurations are easy to tweak

Bad News

We don't know what settings to tweak yet

Good News Again

We are working on integrating stats tooling into the server code so that you guys can give us dumps on what is occurring in the zones at the time of this so we can see exactly what is happening at the network layer so we can give a proper prognosis

We are also working on a way to connect hundreds of headless clients to a server to simulate a stress condition so we can also debug this same issue without being reliant on someone with a player base and players logging in to replicate this issue, we can then replicate it on our own

That being said

The resend issue discussed prior in this thread was a real issue and we addressed the aggressive resend issues in weeks worth of testing on a few different servers

It seems that this network issue is common to Windows environments on a 2-2.6ghz Xeon chip and anytime there are 20+ clients in the zone

We are committed to getting it resolved, you shouldn't need to switch your OS and hardware to get around this issue so please be patient and we will get it sorted