Not Getting the EBS IOPS You Paid For?

In many cases, you may be surprised to see you don’t seem to be getting the IOPS you paid for on your Amazon Web Services (AWS) Elastic Block Store (EBS) volumes. Here are the reasons and what you can do to fix it.

EBS and IOPS Background

First, a bit of background.  IOPS means Input/Output Operations Per Second and is generally used as a measure of performance of computer storage such as disks or EBS volumes.  Higher IOPS values mean faster storage which can translate to better performance for your website or database.

AWS documentation tends to focus on IOPS for an EBS volume.  Your IOPS are directly tied to the size of your EBS volume.  For each 1 GB of the EBS volume, you get a baseline performance of 3 IOPS.  Volumes less than 1000 GB can occasionally support a burst speed above their baseline of up to 3000 IOPS.  Volumes larger than 1000 GB already have a baseline above 3000 IOPS and so do not need to burst to 3000 IOPS.

In many cases, people will buy a larger EBS volume to get better performance.  And, in general, this approach works.  But sometimes, it does not.

My Large EBS Volume is not Getting the Expected IOPS

Sometimes, you may create a large EBS volume with the expectation of getting a certain number of IOPS but find you are not getting that number.  The graph below shows a 3TB drive which is expected to get 9000 IOPS.  But as you can see, we are struggling to get 2500.

2016-03-09-IOPS-lg

 

Initially, we thought perhaps our application was not driving enough load to use all the IOPS, but we found our disk queues were approaching 15, which is longer than we would like.

2016-03-09-queue-length-lg

Our first clue came when we noticed the slight increase in IOPS occurred in parallel with a slight decrease in average read size.  The use case is a MongoDB database.  The reduced read size is due to MongoDB finding more data in the file cache over time and thus needing less from the file system.

2016-03-09-read-size-lg

We noticed that the IOPS multiplied by the average read size remained constant over time.  This value tended to stay right around 130MB/s.   The fact of this value holding constant, combined with the longer than desired disk queues, and the fact that more load on our application did not provide more throughput all led us to conclude we had an I/O bottleneck on this EBS volume.  But why is that?  We paid for a 3 TB volume which means 9000 IOPS.  Why are we not getting what we paid for?

IOPS are not the Only Limitation for EBS

As mentioned before, AWS documentation focuses heavily on IOPS when discussing EBS performance.  But there is more to it than that.  In addition to the IOPS limit that varies according to the size of your EBS volume, there is a hard limit on throughput per EBS volume.

If your I/O chunks are very large, you may experience a smaller number of IOPS than you provisioned because you are hitting the throughput limit of the volume.  — EBS IO Characteristics

The throughput limit of General Purpose EBS drives ranges from 128-160 MB/s depending on drive size.  While our drive should have received 160Kb, we started to suspect we were hitting a throughput limit.  Perhaps exacerbated by the fact we were not using an EBS optimized EC2 instance type.

The Read Bandwidth graph below shows we are not able to exceed a throughput of 130MB/s regardless of the load on the system.

2016-03-09-bandwidth-lg

What Can I Do?

Our options were to either spend more to get the better performance we desired, or accept the current performance level and save money by selecting a less expensive drive than the 3TB volume we were currently using.

Increase Throughput Limit with a Provisioned IOPS drive

According to the AWS Price Calculator we were paying $300/month for the large EBS volume that was not delivering the IOPS we had hoped for.  Because we were throughput bound, we needed to focus on improving throughput.  A simple option is to move to a Provisioned IOPS EBS volume.  Provisioned IOPS volumes offer double the throughput of General Purpose EBS volumes.  Instead of being limited to 160 MB/s, we could get 320 MB/s.  Based on our average read size, we are getting a bit under 2500 IOPS with the limit of 160MB/s.  We wanted to double that rate.  Provisioning 5000 IOPS should be enough to fully utilize the 320 MB/s limit and double our current throughput.  We really only needed 500GB of drive space and 5000 IOPS.  We could get a Provisioned IOPS drive that would double our performance for $387.50/month.  That would double our performance for only 30% increase in monthly cost.

If Performance is Acceptable, Rightsize the EBS Volume

Perhaps we would have liked better performance, but our budget would not allow for additional spending, and the performance we were getting was tolerable.  In this case, we need to stop spending extra money for performance we are not getting.  We provisioned a 3TB driver expecting better performance, but our large average read size and the EBS throughput limit meant we were only getting 2500 IOPS.  A smaller 835GB drive will still provide 2500 IOPS and will cost only $83.50/month.  This offers us a 72% savings for the current performance level.

Steps to Implement Our Solution

Regardless of whether we decide to spend more for a faster volume, or save money and downsize to a more cost effective volume that delivers the same performance, the general approach is the same.  We need to take the following steps:

  1. Provision a new EBS Volume
  2. Attach that volume to our EBS instance
  3. Mount the volume
  4. Copy the data
  5. Unmount the old volume
  6. Move the mount point of the new volume
  7. Detach and delete the old volume

 Provision the new EBS Volume

First, check the availability zone (AZ) of the EC2 instance you will be working with.  You must ensure you create the new EBS volume in the same AZ as the EC2 instance.  To check that, open your list of EC2 instances in your AWS Console and find the availability zone of your instance as shown below.

Screen Shot 2016-03-10 at 8.14.32 PM

Remain in the EC2 area of the AWS Console.  One the left hand column, click the link “Volumes” under “Elastic Block Store.”  Then click the large “Create Volume” button.

Fill out the form that appears based on whether you are upgrading to Provisioned IOPS or rightsizing to a smaller more cost effective volume.  The example below shows upgrading to Provisioned IOPS.  If your data is sensitive, select the option to encrypt the volume.

Make sure you choose the same availability zone as your EC2 instance!

create-ebs-volume

Attach the Volume

Select your new volume, and in the Actions menu, select “Attach”

attach-ebs-volume-1

Then select your EC2 instance from the drop down list.  You can accept the default “device name” but you may want to write it down.  It can be helpful when mounting the volume later.

attach-ebs-volume-2

Mount the Volume

To mount the volume, you have to log into your EC2 instance and issue various commands.  The exact commands will vary according to your operating system.   The links below will help you navigate this step.

Mount your EBS volume in Linux

Mount your EBS Volume in Windows

Copy the Data

The next step is to copy the data from your old drive to your new drive.  Be sure to stop any programs such as databases that might make changes to the files during the copying process.

Unmount the Old Volume

Using the appropriate tools for your OS, unmount the old volume.

On linux, this will be the unmount command.  On Windows, you will use the Disk Management tool you used to mount the new volume.

Move the Mount Point of the New Volume to Where the Old Volume Was Mounted

This step involves first unmounting your new volume from its mount point and then remounting it in the same place the old volume was mounted.  This step is necessary because various programs such as databases and web servers are looking for the data in the old location.

If you are using Linux, don’t forget to update fstab to ensure the drive is mounted at boot.

Detach and Delete the Old Volume

Once you have confirmed all your data was correctly transferred to the new volume and everything is working as expected, don’t forget to detach and terminate the old volume.  Until you terminate it, you will keep paying for it.  Those costs can build up quickly!

Go to the AWS Console, EC2 Section.  Select “Volumes” under “Elastic Block Store.” Select the old volume and on the action menu, choose “Detach Volume.”

detach-ebs-volume

Then on the action menu, choose “Delete Volume.”

Conclusion

IOPS are not the only factor limiting your EBS drive.  Overall throughput is also limited.  This can be a particular challenge if your average read or write size is large.  If you find yourself limited by throughput, you can upgrade to a provisioned IOPS drive to double your throughput.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s