LAG - connection issues.

[Lillu]

Just a few thoughts about connection issues and lag.

I'm monitoring server statistics/network usage on a daily basis and can safely say that 99% of the time when/if at all you would experience a connection issue it's usually not server related rather a connection issue between your client and the server. If a few others would also experience connectivity issue, it's still most probably an international routing/DNS/whatever issue and most of the cases it's temporary.
If the whole server lags out, login server and web services would be down now that is a server related issue and please report it asap (spamming ooc won't help).

This Sunday morning/afternoon was different because I noticed some network problem as well. I called our server hosting company today to get more information about what was going on. They informed me that another server in the datacenter got some serious dos attack and this is why other servers might have experienced "some" connectivity issue but it was "fixed" already.

As for recent lag complains, please try to provide us more information about your issue. What kind of network setup you have, when do you experience lag, is it permanent or casual, all clients or just specific ones, etc. We'll always check if a problem is server related but again most of the time it's a temporary server -client network issue and some patience will resolve the problem.

Thanks,
Lillu

[Reid]

I'm glad I was right in thinking it was data center related, as the server was stable for me before the hardware upgrades. It was frustrating over the last few days (week?) to have random disconnects or seemingly zone wide disconnects for multiple players. We called a few raids early because the clients were constantly getting out of sync, or you were zoning into 1k+ pings and relogs didn't always fix it.

It was fun though, theorizing in guild chat over what imagined lengths the other servers were going to to stop us from being more awesome than them.

Hopefully the data center got their house in order. I'm tired of raiding guild hall.

[vinstad]

I have no clue if it matters or not but everytime lately there is server wide lag i have noticed that the Akka's Funhouse server on the EQEMU site listing a population of -9000000 or more. Like i said no clue if they are connected in any way just something i noticed everytime i had relog or reconnect after being dropped.

[Taladome]

Currently unable to login getting timeouts and the dreaded an error occurred while joining the server. Just to be honest never had these kind of issues before the server move. It seems there are some major issues with one or more of the routes/routers headed to the server from my end pretty close to the server. Here is a text capture of the visual route I performed.

Report for login.thehiddenforest.org [195.5.177.28]

Analysis: IP packets are being lost past network "Docler Networks" at hop 16. The route has changed so the next network at hop 17 is unknown.

------------------------------------------------------------------------------------------------------------------------------------------------------------
| Hop | %Loss | IP Address | Node Name | Location | Tzone | ms | Graph | Network |
------------------------------------------------------------------------------------------------------------------------------------------------------------
| 0 | | 192.168.1.4 | PC-201105140039 | ... | | | | (private use) |
| 1 | | 192.168.1.1 | - | ... | | 0 | x | (private use) |
| 2 | | 59.58.222.1 | - | | | 49 | x- | 59.58.222.0 |
| 3 | | 218.6.30.222 | - | ?Beijing, China | +08:00 | 31 | x | CHINANET fujian province network |
| 4 | | 218.6.39.162 | - | ?Beijing, China | +08:00 | 40 | x | CHINANET fujian province network |
| 5 | | 202.97.66.149 | - | ?(China) | +08:00 | 53 | x | CHINANET backbone network |
| 6 | | 202.97.33.38 | - | ?(China) | +08:00 | 50 | x | CHINANET backbone network |
| 7 | | 202.97.60.38 | - | ?(China) | +08:00 | 48 | x | CHINANET backbone network |
| 8 | | 202.97.51.182 | - | ?(China) | +08:00 | 243 | x | CHINANET backbone network |
| 9 | | 218.30.54.182 | - | ?(China) | +08:00 | 219 | x- | Chinanet POP in American |
| 10 | | 80.91.253.69 | ash-bb1-link.telia.net | - | | 310 | -x | Telia International Carrier |
| 11 | | 80.91.246.59 | ffm-bb1-link.telia.net | - | | 500 | x | Telia International Carrier |
| 12 | | 213.155.133.51 | prag-bb1-link.telia.net | - | | 509 | x | TeliaSonera International Carrier |
| 13 | | 80.91.250.69 | - | ... | | 529 | x | Telia International Carrier |
| 14 | | 213.248.85.222 | doclerweb-ic-131274-bpt-b4.c.telia.net | - | | 375 | x | TeliaSonera International Carrier |
| 15 | | 80.77.112.193 | TE-3-3.core0.dhq.bud.doclernetworks.net | ?(Hungary) | +01:00 | 516 | x | Docler Networks |
| 16 | | 80.77.112.178 | GE-5-1.core3.dhq.bud.doclernetworks.net | ?(Hungary) | +01:00 | 536 | -x-- | Docler Networks |
| 17 | | ?80.77.112.178 | GE-5-1.core3.dhq.bud.doclernetworks.net | | | | | Docler Networks |
| ... | | | | | | | | |
| ? | | 195.5.177.28 | login.thehiddenforest.org | ?(Hungary) | +01:00 | | | EasyGo Kft. |
------------------------------------------------------------------------------------------------------------------------------------------------------------
Roundtrip time to 80.77.112.178, average = 536ms, min = 515ms, max = 688ms -- 2011t827?
H08:52:41

[Lillu]

Thanks Taladome. Info sent to the datacenter, we'll see what they'll reply. Tho 342 client connected to the server atm, so it's pretty weird. :/

[Wrinkly]

Hm, this is something I should be able to help with.

For one, make sure you have the ping meter up in EQ. I believe the shortcut key is F11. If it reads 100%, then you have a good connection to the server. It will also show the latency in milliseconds. Anything under 400ms (I'm sure this number is debatable and could even be higher, but 400ms with 100% has never given me trouble to the point of not being able to play well) is probably fine as long as you're still reading 100% meaning no lost packets.

This next part is a bit trickier. If you think there's something wrong with the server, you need to first rule out your own internet connection. Depending on where in the world you are, the best way to do this is to ping reliable node on the internet somewhere. For those of us in Canada and the US (maybe Mexico), you can type "ping 4.2.2.2 -t" from the command line. This is Level 3 Communications and you should always get a ping back from this node in under 60ms. Get to know what your reply times are typically from 4.2.2.2 and you now will be able to tell if you're having bad latency from your home internet. The tricky part comes in for the rest of the planet who is not in Canada or the US. If this is you, then I would recommend looking up some well known networks which are local to you, or maybe call your ISP for the IP address of a node you can run ping tests to. By the way, pressing ctrl+break will give you important summary information about your ping times. As embarassing as these results are, I will post mine as an example:

Ping statistics for 4.2.2.2:
Packets: Sent = 976, Received = 976, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 53ms, Maximum = 2369ms, Average = 425ms

I'm getting a new ISP tomorrow by the way, as a max ping time of 2.3 seconds is intolerable, and its not uncommon by any means with my connection....enough about my internet.

Now the piece of the puzzle thats really missing here, is allowing us to ping the actual EQ server so we can determine what the latency is *usually* like vs. what its like at the time we're all crying in OOC about the server having issues. I tried pinging login.thehiddenforest.org but there's no reply. And I'm also aware that this is just the login server, not necessarily the actual game server, so there's no telling if I'm even using the correct IP. If Lillu or anyone could provide it, I for one will run regular ping tests and be able to provide meaningful feedback about the connectivity between my Canadian based ISP and the server. Others will hopefully be able to use this information as well to cut down on the number of reports Lillu is receiving that are out of THF's control to fix.

[Taladome]

If you look at my post you will see that packets are being lost past hop 16. This happens when a router changes its route to try and compensate for a congested line or to shorten the amount of hops it takes to reach a destination. You can have a 500 or even 700 ping and still not see much of a difference in terms of lag in the game as long as the connection remains stable. TI in general has never been known for high quality backbones. Im constantly getting 600+ pings connecting to anything through them. Connections to the Middle US from China more hops and not a ping over 300. So you gotta consider how far into the Internet is considered "your" connection. I dunno what the options are over there but if the data center had a choice of switching to a different BP id certainly recommend it.

Tal

[Lillu]

At hop 16, ping is disabled, thats why all thepackate "loss" according to pathping. It's actually not a connection issue.

[Taladome]

Ping is open and udp packets are accepted on hops all the way to login.thehiddenforest.net Anytime you get packet loss because of changed routes its either a poisoned route, congestion over a lower bandwidth line such as a T1, or congestion created by a dos attack. Any other CCIE would tell you the same thing. You said its not a server problem and Im inclined to agree with you. Finger has to be pointed somewhere with so many people around the world getting the lag spikes at the exact same time. Server or connection?

Blame gnomes killem and find the real answer later.

Tal

[Wrinkly]

Ping is open and udp packets are accepted on hops all the way to login.thehiddenforest.net
Tal

This is incorrect. See - http://www.exit109.com/~jeremy/news/providers/traceroute.html#timeouts Specifically the first line

If your trace to a system ends in timeouts, and never completes, there could be a problem. (The other explanation is that a system is blocking traceroute attempts, either by filtering all ICMP messages or by other means.)

Another example of this can be found with IOS' default ICMP rate limiting.

I could operate a backbone router, and forward packets between routing peers while still refusing ping requests to any IP/subnet I wish and not have a "poisoned" route. How an individual router is configured to accept ping requests is entirely up to that routers admin.

When I said to run a ping test to a known reliable node its because you can often compare the results of a nearby node's packet loss or ping times with the results you see in ping times in the EQ client (F11). If you see a relationship between the known reliable node and your results in EQ, its likely your internet connection is the issue. For example, cranking the downloads while you game can create the effect of having lag on the server even though your local internet connection is just overloaded. This test will help reveal that.

Because there's so much network equipment out there and so many admins with differing levels of experience setting them up, we get inconclusive results a lot of the time from traceroute. This is why most multiplayer games including EQ include some method of displaying your ping time to the game server and no sign of a traceroute tool.