Amazon EC2 Raw I/O Performance Benchmarks

Overview & Testing Methodology

We wanted to evaluate a combination of file systems supported by Oracle on Amazon Elastic Block Store (EBS) volumes. All tests were performed using Bonnie++ 1.96. We used EC2 Large instances running Oracle Enterprise Linux (OEL) 5.5 64-bit. Each test was performed 5 times in sequence, and the average of the tests is displayed. The labels indicate the file system, RAID type, and number of volumes. For example, EXT4 RAID0-4 designates a 4 volume RAID0 array on the EXT4 file system.

It's worth mentioning that I/O performance is highly variable, and changes from day-to-day. A more representative data set would undoubtedly be generated by repeating the tests on different days, and at different times of day. All our tests were performed in the US-EAST-1B Availability Zone.

From reading some of the very good analysis done by others on the subject, we thought we knew a couple of things:

• We'd probably need to use a RAID array to saturate the I/O channel.
• I/O channel throughput is affected by demand. If competition for the channel is abnormally high or low, then the results will be skewed. Because this test was done in a single Availability Zone and all the samples were collected in a short time window, you have to take them with a grain of salt.

Charts in blue (throughput) are higher-is-better. Charts in orange (latency) are less-is-better. Latency has a greater impact when reading or writing smaller amounts of data, but you can get a good feel for where the first-read and first-write penalties come from in the EC2 environment.

Block Read

Block Read Latency

Results indicate at least a gigabit I/O channel, and maximum read performance topping out at around 300MB/s. A few things really stand out:

• XFS consistently outperformed the other filesystems for latency, but stacked up about the same for throughput
• Adding more volumes to the RAID array negatively impacts the first-read penalty, but increases throughput
• Setting up software RAID (mdadm) improved performance considerably
• 8-volume RAID arrays had only moderately higher throughput than the 4-volume arrays

Block Write

Block Write latency

Block write operations were largely unaffected by RAID configurations, though again XFS stands out as the best performer.

Block Read-Change-Write (Rewrite)

Block Read-Change-Write (Rewrite) Latency

Equally important for database applications is the rewrite throughput. Interestingly, XFS is almost as fast on rewrite ops as it is on block write, but only when used in a RAID configuration.

Seek Latency

Here XFS didn't perform quite as well as the other filesystems against a single volume, but seek times across RAID arrays are more-or-less as expected, inversely proportional to the number of volumes in the array.

Summary

Our results indicate some good candidates for performance improvement:

• Maximum read throughput is approximately 300 MB/s
• Maximum write throughput is approximately 80 MB/s
• XFS outperforms EXT3 and EXT4 for most metrics
  (We'll be looking forward to XFS in Oracle Enterprise Linux 6, and hope it makes it into the default Amazon EC2 default kernels.)
• Read performance is considerably improved by setting up a quick mdadm array, and it looks like 4 volumes will get most of the performance out of the I/O channel
• Write performance is largely unaffected by the RAID configuration
• Rewrite gains a modest performance benefit from a RAID setup
• Seek times improve in proportion to the number of volumes in the RAID array, but the number of volumes in the array can have a negative impact on the first read/first write penalty

Feel free to leave a comment, or Contact Us if you have any questions. We'd love to hear from you!