EVE Online: Dev Blog: Networking Changes on TQ

The latest dev blog out of CCP takes a look at the network behind EVE Online and how this series of tubes we call the interwebs works.

Well, we have done tons on upgrading PC machines with more memory faster hard drives, gone from floppy drivers to some RAMSAN thing, but while this was happening the network operations team has been looking at the most important thing (from the NOC standpoint), the network. Without a network connection to this interweb thing, EVE would be a great single player console game for some lonely guys in Telecity datacenter ;P

The Internet, how does it really work (yikes!!)

When I started working with Internet routing, coming from a rigid corporate environment, I was surprised how little control you have over your traffic and how flaky the Internet really is. I have read in books about all the bells and whistles of traffic engineering but I've never seen the real application until I met BGP and the Internet. As many of you know, the Internet started out as few universities talking to each other. BGP (Border Gateway protocol) is the main building block of the Internet. The Internet is built up by BGP speaking peers (routers). Each origin (like CCP) has a network slice and we announce our network to our neighbors and hope they announce them to other neighbors. We rely on our next hop (upstream) neighbors to accept our prefix and carry the traffic.

The major problem with the Internet is that BGP is not very deterministic; it relies heavily on something called an AS-PATH. Each entity on the Internet has an AS number (autonomous system). The path from client to CCP is usually the AS of client ISP, Tier1 SP (top of the Internet), CCP. There will often be other upstream providers of the client ISP. What is the problem here? Within one AS there could be 10 router hops (as seen via traceroute) Those hops could go to Europe, the US and then back to Europe. The best part is that there is a lot of meddling with the best paths. Some ISP's will have a good connection to some Tier1 providers, but may also have a cheaper connection with some lousy provider that they engineer the traffic to.

Read more here.