In December of 2009 MySpace launched new offering of streaming music video offerings in New Zealand. These new features included the ability to watch music videos, search for artist’s videos, create lists of favorites, and more. The anticipated load increase from a feature like this on a popular site like MySpace is huge, and they wanted to test these features before making them live.
If you manage the infrastructure that sits behind a high traffic application you don’t want any surprises. You want to understand your breaking points, define your capacity thresholds, and know how to react when those thresholds are exceeded. Testing the production infrastructure with actual anticipated load levels is the only way to understand how things will behave when peak traffic arrives.
For MySpace, the goal was to test an additional 1 million concurrent users on their live site stressing the new video features. The key word here is ‘concurrent’. Not over the course of an hour or day… 1 million users concurrently active on the site. It should be noted that 1 million virtual users are only a portion of what MySpace typically has on the site during its peaks. They wanted to supplement the live traffic with test traffic to get an idea of the overall performance impact of the new launch on the entire infrastructure. This requires a massive amount of load generation capability, which is where cloud computing comes into play. To do this testing, MySpace worked with SOASTA to use the cloud as a load generation platform.
Here are the details of the load that was generated during testing. All numbers relate to the test traffic from virtual users and do not include the metrics for live users:
- 1 million concurrent virtual users
- Test cases split between searching for and watching music videos, rating videos, adding videos to favorites, and viewing artist’s channel pages
- Transfer rate of 16 gigabits per second
- 6 terabytes of data transferred per hour
- Over 77,000 hits per second, not including live traffic
- 800 Amazon EC2 large instances used to generate load (3200 cloud computing cores)
Test Environment Architecture
SOASTA CloudTest™ manages calling out to cloud providers, in this case Amazon, and provisioning the servers for testing. The process for grabbing 800 EC2 instances took less than 20 minutes. Calls we made to the Amazon EC2 API and requests servers in chunks of 25. In this case, the team was requesting EC2 Large instances with the following specs to act as load generators and results collectors:
- 7.5 GB memory
- 4 EC2 Compute Units (2 virtual CPU cores with 2 EC2 Compute Units each)
- 850 GB instance storage (2×420 GB plus 10 GB root partition)
- 64-bit platform
- Fedora Core 8
- In addition, there were 2 EC2 Extra-Large instances to act as the test controller instance and the results database with the following specs:
- 15 GB memory
- 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
- 1,690 GB instance storage (4×420 GB plus 10 GB root partition)
- 64-bit platform
- Fedora Core 8
- PostgreSQL Database
Once it has all of the servers that it needs for testing it begins doing health checks on them to ensure that they are responding and stable. As it finds dead servers it discards them and requests additional servers to fill in the gaps. Provisioning the infrastructure was relatively easy. The diagram (figure 1.) below shows how the test cloud on EC2 was set up to push massive amounts of load into MySpace’s datacenters.
