Sony's Death Star

[leane]

Saw this up on slashdot and though it was mighty interesting...

Engineering Everquest

Goes a bit into what kinda hardware Sony has running to keep EQ/EQ2 up and running.. Probably the situation is identical to any other MMPOG out there

"YOU'RE ABOUT TO GO into the Death Star," Joffe says. A balding, goateed guy with narrow, rectangular glasses, Joffe places his palm in a biometric scanner in the first-floor lobby of a building near Sony Online Entertainment's main offices, and a gray door unlocks noiselessly. The Death Star is a huge, warm, windowless room containing the rows and rows of servers that run Sony's online games. The whooshing of a massive air-conditioning system is so loud that conversation is almost impossible. A large steel cage surrounds more than 500 servers stacked 32 high in towering racks—and this is just one battalion, albeit the largest, in Sony's 1500-machine army of servers.

At any given time, Sony is hosting 150 000 gamers Other than the graphics and sound, which are loaded directly onto the players' machines from CD-ROMs when EverQuest is installed, everything else needed to run the game—including the players' characters—is stored on these machines. Sony calls them "world servers." It's more than just a catchy name. Remember that to prevent overcrowding, EverQuest is divided into dozens of parallel worlds. The worlds are reflected in the server farm as interconnected combinations, or clusters, of servers; the cluster size is based on how many users Sony expects to support simultaneously. In other words, each world is a cluster consisting of between 20 and 30 dual-processor computers. And within the clusters, individual processors are devoted to producing different pieces of geography—a town, a forest, a labyrinthine castle—in those worlds. When a player logs on to the game, the program, or client, being used connects to servers in the cluster the character was last playing in. Those servers then download data describing everything—the alter egos, locations, weapons, and other characteristics of other players who are logged on, plus all the relevant monsters and weapons nearby. A full EverQuest install, which requires six CD-ROMs, weighs in at about 3 gigabytes. To log on, players must have the latest version of the software, which is updated through downloadable patches every two to four weeks. When the patch is large—say, 25 or 30 megabytes—play stops and Sony takes the servers down for as long as several hours (but usually less than 30 minutes). During that time, Sony updates the code on the servers and sends the new software to the players. The patches are not merely fixes to problems. They can be brand-new content—a new city, dungeon, or continent to explore.

Each world can support about 2500 players at a time, although it can store data on up to 10 000. When EverQuest launched in 1999, there were just 12 worlds supporting 100 000 players. Last year there were 52 supporting half a million. As the game's audience expands and evolves, so does the architecture behind the scenes. The process begins every time Sony adds big new chunks of content to an existing world. New dual-processor servers are first added to the rack deep inside the Death Star. The more content required, the more machines are pressed into service to provide it. Once Joffe's team adds a server to the rack, he loads a Unix operating system onto it. Custom-built software then automatically loads all the game software and the programs the new machine uses to communicate with other servers, configures its storage systems, and starts it up. The server is then ready to be added to the cluster that supplies data about a particular world in EverQuest. The entire process of joining the server farm occurs within minutes, Joffe says. As characters populate the virtual locales that the new machine produces, their data are stored there. All those expansions make for a crowded server room. To be able to keep expanding its computing power without outgrowing the Death Star, Sony has begun using blade servers. Stacked like books on a shelf, the slim computers share certain hardware—such as power supplies, cooling fans, and network interfaces—among themselves, allowing more servers to be housed in a single rack [see "Blades Have the Edge," IEEE Spectrum, April 2005]. But even blades have their limitations. "Blades are so dense that they produce too much heat," Joffe says. "You can stuff them into small spaces, but you need space to cool them properly."

A FEW YEARS AGO, every time a player in a particular world entered a specific virtual dungeon, chances were the same central processing unit, dedicated to that place alone, served up the data seen on the display. Each CPU of each server in a world cluster was assigned a geographical task. But that way of dividing up computing resources turned out to be too inefficient to handle the new content the game's designers were pumping out. So a few years ago, Sony shifted to a new way of managing the system: just-in-time computing. As Joffe explains it, EverQuest's just-in-time system allocates computer resources based on user demand. "As they go into parts of the world that are dynamic, we launch a process that meets users' needs," Joffe says. For example, a player, Porslap the Dark Elf, is running through a corridor when he comes to a door. By opening that door, Porslap triggers actions on several machines. If, say, a dungeon lies behind the door, Sony's system looks up the data and software that describe that dungeon on one computer, finds some idle processing resources within the cluster, probably on a different machine, and runs the software on it. Sony's servers download the dungeon's data to Porslap's computer "just in time" to meet the inquiring elf.
WHEN PROBLEMS HIT EVERQUEST, as problems inevitably do, the first indication comes inside the Network Operations Center [see photo, "NOC, NOC"]. The NOC, a crowded room in Sony Online Entertainment's main office building, smells faintly of elementary-school glue. There are rows of flat monitors and lists of EverQuest world names projected on the wall: Morell Thule, Veeshan, The Nameless. There is a cushy couch against another wall, a couple of bottles of soda, and a few crumpled napkins. The room has a college dorm all-nighter feel, and for good reason. Twenty-four hours per day, 365 days a year, the NOC is ground zero for Sony Online Entertainment's technical support. Gamers game around the clock, so Sony has to be ready to handle problems whenever they happen, and developing and maintaining a system to overcome the challenges is essential to the game's success. For now, all is well, but at any moment the crew might have to save a dying CPU in the Death Star or resurrect software that has been killed by bugs. Three people per shift work in the NOC, and there are three shifts per day. During each shift, NOC staff monitor game activity, responding to players in remote locations and working with a custom suite of software tools to fix problems along the way. The center has shut down for just three days of work—all in 2003, when wildfires were closing in on the neighborhood. (Because all the tools run remotely, the staff members kept EverQuest going from their homes. A backup generator kept pumping power to the servers even as the fire threatened to black out parts of the city.) The biggest problem facing the NOC is bugs, which can have unusual consequences in EverQuest. The bugs that have to do with the EverQuest virtual economy are especially bothersome. Players receive virtual money, called platinum, through accomplishments such as defeating dragons. The platinum is then used to purchase weapons and other objects within the game. Therefore, bugs relating to items and platinum have the potential to dismantle the game's economy. For example, a shopkeeper in the game might hand over an expensive sword when a player had actually just paid for a less valuable dagger. Recall that EverQuest's economy extends into the real world, where the truly obsessed spend an estimated $150 million per year—20 percent of the total online gaming underground secondary market—purchasing everything from fully developed characters to rare weapons. Because of the overlap between the virtual and real economies, an incentive exists for cheating players to try to hack the system to duplicate platinum on the fly. Call them virtual counterfeiters. To hunt them down, Sony has a team, separate from the NOC, that combs game logs for suspicious activity. "We have ways of observing our world and what goes on," says Chris Kramer, Sony Online Entertainment's public relations director. "We keep checks on the economy, so when we find that something happened to create a spike in the economy, we go back and track it immediately." Hackers who are caught are expelled. However, because it's a virtual world, they can create new characters and return.

[Zhakrin]

Twenty-four hours per day, 365 days a year, the NOC is ground zero for Sony Online Entertainment's technical support. Gamers game around the clock, so Sony has to be ready to handle problems whenever they happen, and developing and maintaining a system to overcome the challenges is essential to the game's success. For now, all is well, but at any moment the crew might have to save a dying CPU in the Death Star or resurrect software that has been killed by bugs. Three people per shift work in the NOC, and there are three shifts per day. During each shift, NOC staff monitor game activity, responding to players in remote locations and working with a custom suite of software tools to fix problems along the way.

does that really say that there are a total of 3 GMs for EQ?

[leane]

Originally Posted by Zhakrin

does that really say that there are a total of 3 GMs for EQ?

No, those guys are NOC, they only handle Network issues, Servers dieing, zones crashing, players getting dropped because some yahoo cut through the fiber optic cable and took out the west coast... Worked for a few companies that had NOC centers involving like a dozen people, tech support(GMs) could call them but they weren't in the same area. Kinda like NOC is the GMs GMs

[Tuco]

I think that says there are 3 per shift, 3 shifts per day, so 9 people total in the NOC.

Whether these are GMs, or networking admins, I'm not sure. It appeared that they only handled hardware issues, and not, "I'm stuck." but I didn't read the article all that closely.

[Digo]

I thought it was obvious they weren't GMs? GMs handle customer support issues. Everything from harassment and basic game mechanic questions, to reporting issues to the developers. I'd wager most GMs in any given game haven't even seen the server room, let alone, handle networking issues.

[Kasonic]

There's no real need for a GM to even ever visit a company's HQ. All Caydiem and co. do is e-mail the devs and post on the forum; I'd imagine they work at home.

[Iannis]

telecommuting is awesome.

so...do I owe them a full 8 hours, or just the 2 PRODUCTIVE hours I'd actually put in at the office?

lol. "THIS IS THE BLADE, YOU PUT IT IN THE BLADE SERVER." +1 arbitrary and pointless Drew Cary point for Sony.

At the end...did they REALLY say that people spend platinum found on dragon corpses to buy a sword from a merchant? err...I see the columnist has never played EQ.

[Vorph]

Well, speaking as a software developer, the easiest way to make sure that your CS or QA dept. is completely ineffective would be to give them only email/IM access to devs. It's easy to blow someone off via email for months, but if they show up at your desk you usually end up agreeing to do what they want just to get them to go away. Having to deal with QA is a necessary evil... it's customers and the fucktard marketing dept. where you should draw the line.

[Kaxmax]

It's a cool article, and gives good insight on exactly how big of an operation an MMO is. I think a lot of people don't understand how much is involved in running one particular "world" server. I wonder how Blizzard's setup compares in size/magnitute.

[Nevergone]

Originally Posted by Kasonic

There's no real need for a GM to even ever visit a company's HQ. All Caydiem and co. do is e-mail the devs and post on the forum; I'd imagine they work at home.

Which is exactly why so many GM positions are outsourced.

You might be surprised how many big name companies don't handle their own customer service needs, they simply pay a different company to do it for them. When you call up Verizon, for example, to talk about changing your cell phone plan, you get to talk to people that are in no way associated with Verizon. Before anyone starts bashing Indians for taking American jobs, quite a few of these "customer interaction" firms are located within the US, and employ Americans.

[Khalikryst]

It's fun to pretend...

[leane]

Originally Posted by Nevergone

Which is exactly why so many GM positions are outsourced.

You might be surprised how many big name companies don't handle their own customer service needs, they simply pay a different company to do it for them. When you call up Verizon, for example, to talk about changing your cell phone plan, you get to talk to people that are in no way associated with Verizon. Before anyone starts bashing Indians for taking American jobs, quite a few of these "customer interaction" firms are located within the US, and employ Americans.

I used to work for one such firm, Sykes, and we handled like almost all of the tech support for SBC Global. Basically we were like Guides, we got all the shit so they didn't have to listen to morons. We could escalate to the higher tech support (GMs) who themselves could possibly go to the NOC guys.

Tech support isn't there to help people much as you would think otherwise.... actually helping people costs money and you try to do as little of that as possible, most of the time it became a support boundry and you could blame Microsoft and say we can't help you any further...