Possible Login Server Issue

[trevius]

Here is a traceroute from Storm Haven:

traceroute eqemulator.org
traceroute to eqemulator.org (67.23.190.71), 30 hops max, 40 byte packets
1 DD-WRT (192.168.1.1) 0.575 ms 0.833 ms 1.692 ms
2 10.167.160.1 (10.167.160.1) 10.798 ms 11.767 ms 12.060 ms
3 96-34-52-44.static.unas.mo.charter.com (96.34.52.44) 12.246 ms 12.372 ms 12.472 ms
4 96-34-48-57.static.unas.mo.charter.com (96.34.48.57) 16.408 ms 17.219 ms 17.501 ms
5 96-34-2-194.static.unas.mo.charter.com (96.34.2.194) 17.683 ms 17.780 ms 17.877 ms
6 96-34-0-130.static.unas.mo.charter.com (96.34.0.130) 33.103 ms 27.883 ms 28.071 ms
7 96-34-2-123.static.unas.mo.charter.com (96.34.2.123) 31.412 ms 29.735 ms 32.252 ms
8 96-34-78-89.static.kgpt.tn.charter.com (96.34.78.89) 33.349 ms 33.782 ms 96-34-72-5.static.unas.mo.charter.com (96.34.72
.5) 33.636 ms
9 96-34-72-39.static.unas.mo.charter.com (96.34.72.39) 38.576 ms 96-34-72-37.static.unas.mo.charter.com (96.34.72.37) 38.2
32 ms 96-34-78-33.static.kgpt.tn.charter.com (96.34.78.33) 37.299 ms
10 dtr02spbgsc-tge-2-3.spbg.sc.charter.com (96.34.64.40) 38.731 ms 38.865 ms 38.969 ms
11 96-34-67-181.static.unas.sc.charter.com (96.34.67.181) 39.294 ms 39.414 ms 39.499 ms
12 96-34-67-179.static.unas.sc.charter.com (96.34.67.179) 47.360 ms 48.357 ms 48.467 ms
13 96-34-67-175.static.unas.sc.charter.com (96.34.67.175) 36.332 ms 34.701 ms 37.690 ms
14 68-115-195-86.static.spbg.sc.charter.com (68.115.195.86) 39.402 ms 39.257 ms 39.486 ms
15 te-5-5.rtr2.avl1.netriplex.com (67.23.161.254) 38.752 ms 39.037 ms 42.180 ms
16 eqemulator.net (67.23.190.71) 43.860 ms 43.474 ms 43.192 ms
XXXX@stormhaven:~$

Not sure that will be of any assistance though. I haven't noticed any packet loss during continuous pings while the problem is happening. I do agree that better LS connectivity logging might help to isolate problems like this. Heck, for all I know, it could be my server that started the eqemu website server list to start having status display issues. Though if that was the case, it could mean that any server could have potentially caused it for all which would be a good thing to resolve/prevent anyway.

My best guess so far is that the problem is somehow related to the fairly new (6 months old or so) code that allows servers to connect to multiple Login Servers at the same time. I know that when I removed the config to connect to LSs other than the EQEmu one, my server was suddenly able to stay connected to the EQEmu LS much better.

And, Rogean, I don't mean to seem like I am pushing this issue all on you or anything. It is just hard to figure out a connection issue only looking at a single side of the connection. If there didn't seem to be any other oddities going on, I would assume that the problem was on my end only and would work on it quietly by myself until it was resolved. Since it seems that it may not just be my server, it doesn't hurt to investigate the possibility of the issue being with the EQEmu LS or even one of the other LS like tsahosting.net or peqtgc.net.

I am curious if one of the other LSs was disconnecting servers, maybe they would try to reconnect to all LSs they are configured to connect to even if they are already connected? I should check with Gaeorn to see if he is seeing anything odd on tsahosting.net lately.

[Angelox]

Here's some web tests that may or may not be of help;
http://www.indeep76.com/eqemulator.net/
http://www.indeep76.com/checks/eqemulator.net/6851/
I reported the problems starting a few days ago because that's when they started. I mentioned it was after you did maintenance work, as an observation that might be of help to you; Prior to that date, everything was running fine, and I haven't changed a thing. I noticed an impressive amount of servers with the same situation, so I deducted the problem was LS server side.

My best guess so far is that the problem is somehow related to the fairly new (6 months old or so) code that allows servers to connect to multiple Login Servers at the same time. I know that when I removed the config to connect to LSs other than the EQEmu one, my server was suddenly able to stay connected to the EQEmu LS much better.

Not that it matters, but I can guarantee you this is new, as of a few days ago. currently, I'm only single connected (to AXCLassic LS) , because I can't keep a stable connection to EqEmu LS anymore. When I try to "Single LS" connect to EqEMu, it lasts a few mins and drops.

[Tabasco]

The host for my primary server (running 1771) reported packet loss issues from an over-utilized circuit, but are in the process of resolving the problem.
I've been connected for around 24 hours now with no problems so I'm inclined to think my issues were purely host related and just coincidentally timed, but I'm keeping a close eye on it.

I've updated my test server (completely different host) from 1771 to 1777 to see how it behaves.

[Astal]

Im also experiencing this problem

Ive changed nothing in my exes or anything, was working fine a few days ago.

[trevius]

Originally Posted by Angelox

Not that it matters, but I can guarantee you this is new, as of a few days ago. currently, I'm only single connected (to AXCLassic LS) , because I can't keep a stable connection to EqEmu LS anymore. When I try to "Single LS" connect to EqEMu, it lasts a few mins and drops.

Yeah, I agree that this is a very new issue. My theory was that with the addition of multiple Login Server connections, it might open up new possibilities where one Login Server could potentially bleed an issue into another Login Server. I don't know the details enough to really know if it is possible or not, but I think it could be something like this:

1. Login Server A is up and running perfectly fine without an issue.
2. Login Server B starts to experience an issue that is causing it to constantly drops world servers.
3. Multiple servers are configured to connect to both LS A and B.
4. As the servers start getting disconnected from Login Server B frequently, they start sending reconnects for both A and B at the same time even though A has remained stable.
5. Login Server A reacts badly to servers trying to register again that are already connected and registered. Somehow this causes all external servers (not on the same LAN as LS A) to experience connection issues with LS A.

Now, that example is probably not possible at all, but if nothing has changed to cause this problem, I am grasping at straws.

No servers have the info Rogean wants, because from a network perspective, everything looks perfect as far as I can tell. I never drop a ping to eqemulator.net and traceroutes and response times look great. I get about 35ms response time consistently, so that is not the problem.

I think part of the problem is that P1999 and PEQ both run on the same host as the LS, so they will never experience this issue if it is only from external networks, which is what it appears to be. And without P99 having this issue, I am sure it is harder for Rogean to troubleshoot from his end, or to even verify there is a problem at all. If P99 was experiencing this issue as well, it probably would have been resolved already.

Considering that so many servers are experiencing and voicing issues with the EQEmu LS only, I think it is pretty clear that the problem is there. Since PEQ and P99 don't have this problem at all, and all other servers seem to be having at least some sort of issue (some worse than others), I think that points to some type of issue with networking internal vs external. It could be a port issue, a routing issue, a DNS issue, or just about anything coming into the EQEmu hosting or anywhere up to that point. It could even be something related to the recent DDoS attack on P99 and related to how the hosting service assisted in mitigating the attack (assuming they did something).

Considering there was "hardware maintenance" on the 15th and a recent DDoS attack, it seems like one of those issues/changes could be to blame. Without us knowing exact details on issues/changes like that, it is hard to assist in possible suggestions for the cause and resolution of the issues.

[erik_llewellyn]

That makers a lot of sense actually since the easiest way to combat a DDoS attack is to only allow so many connection attempts from an address in a given amount of time before either banning the address or temp banning it. If the our external world servers loose sync and keep trying to reconnect, the LS may be seeing that as a DDoS attack and causing issues. Even more so if as you hypothisized about having multiple LS's syncing to one world server and if one goes out of sync it resend the world server login request back to all LS's in it's list.

[Akkadius]

I've mentioned these things to Rogean before this thread started. I've had all the same issues on two servers and I've seen them consistently. Not that they are ultimately keeping people from playing, but there still is a definitive difference AFTER whatever 'maintenance' happened. As far as what has changed specifically, that will have to be on Rogean's side. And I'm sure it's one of those problems that is very hard to reproduce or trace on his end.