July 6, 2024 Tommi Kannisto Cloud StorageHardware

How Storadera Overcame IOPS Limitation On Hard Disks

Blog cover image with text "Storadera Released Variable Block Size Technology" and images of HDD and SSD and text "IOPS?" and "TBW?"

At Storadera, we are obsessed with speed

Usually, (NVMe) SSDs are considered delivering great performance due to way higher IOPS.
*IOPS means Input / Output operations Per Second

We offer S3 compatible cloud storage (object storage) and obviously we need high performance storage system.
It may seem a bit controversial but we store customer objects on hard disks (HDD, also called hard drive or spinning rust).

How we can offer fast service on hard disks? To answer this, we need to understand performance difference hard disk vs. SSD or HDD vs. NVMe.

How much faster are SSDs compared to hard disks?

To answer this question, let’s go back to the reason why these days SSD is the go-to solution for anything that needs high performance.

The reason is that spinning hard disks have a dramatically lower number of IOPS.

Here is a quick comparison of one hard disk and two very different SSDs:

Storage TypeConnection protocolWrite IOPSIncrease over hard diskSource
Western Digital HC580 24TBHard diskSATA565
(c.a. 0.6k)
Samsung PM893 SATASSDSATA30kAbout 50xpm893
Micron 9400SSDNVMe800kAbout 1400x9400

Why these specific models:

  • WD HC580 24TB: we hope to start testing this model soon in our data center. It has amazing ArmorCache technology but this is a topic for another article.
  • Samsung PM893: we use this in our data center for object metadata storage. It has a SATA interface but it is more than enough for our use case.
  • Micron 9400: at the time of writing, it is one of fastest NVMe SSDs available on write IOPS.  These drives go to crazy capacity for SSDs (30TB) and price as well.

SSDs have 1400x IOPS avantage. Why choose hard disks?

Some obvious benefits of hard disks over SSDs:

  1. The price difference can be easily over 10x in favour of hard disks. If we can offer fast enough service on 10x less expensive hardware then it sounds like magic. Check out our pricing to confirm that Storadera has one of best prices in the industry.
  2. Reasonably priced SSDs are still much smaller in capacity and require more connectors. This in turn requires bigger and more costly server infrastructure around the drives.
  3. SSDs have a limited amount of terabytes written (TBW). After that, they just become read-only. Hard disks don’t have such strict limitations.

How typical storage systems operate with blocks

Typical storage systems divide incoming data to blocks. They can be very small, from 4KB to bigger, as big as 1MB to 32MB in size. There are many different flavours how storage systems are built, and how different components are called.

Still, often, there is some unit of data that has a fixed size that will be written to disk.

It is, of course, a huge oversimplification but let’s consider a hard disk having 500 IOPS maximum. On random write, IOPS can go down to 100. Now if we consider that writing one block takes 20 IOPS then we can have 100 / 20 = 5 block writes per second.

If the block size is 1MB then it means we can write 5 MB/s. Not a very huge number, considering hard disks can reach sequential write speeds close to 300 MB/s.

Some storage systems directly write customer files to the hard disk, without dividing or collection to blocks. In this case, small files (let’s say less than 100KB) are very problematic as they consume all the IOPS.

Additionally, there can be different factors like Erasure Coding and other reasons for overhead.

Trust me, even 5 MB/s is still a good number for such systems, many storage systems go below 1 MB/s when the written chunk is very small.

Such low speed is obviously a problem!

Storadera’s Variable Block Size Technology is the solution for IOPS limitation on hard disks

So, we concluded that writes and IOPS are very limited on hard disks.

We could increase block size to achieve better overall throughput. This would definitely solve this one thing but it has downside of longer latency.

For example, if you want to write 100MB block then you have to wait until customers have sent you this 100MB and then you can start writing it.
This causes higher latency, especially at the moments of lower incoming traffic. For background, we measure things in milliseconds. Huge block size on low incoming traffic moment would easily cause extra 1000 millisecond (1 second) or more on processing single upload request for the customer. So, this isn’t an acceptable solution.

Math tells us a really simple solution. We should use small block size when less data is coming in and bigger block size when more data is coming in. This is exactly what we did.

Storadera’s Variable Block Size Technology helps to overcome the IOPS limitation by changing block size according to current volume of incoming data. It balances block size with incoming bandwidth to always keep latency low and avoid being limited by hard disk IOPS. This optimization enables us continue offering well priced and high performance s3 compatible cloud storage.

The software engineering efforts to achieve this were quite high but we are happy with the results so far.

Storadera Variable Block Size Technology is available in our cloud locations (link).
Sign up to try using it for free

Did you know that you can have it at your location?

If you plan a Petabyte scale s3 compatible object storage, we can deploy it to your location and fully manage it for you. If it is a new hardware purchase then we can advise on the choice of hardware. And we can work on most existing hardware.

Read more to get started with your own Petabyte scale S3 storage

Leave a Reply

Your email address will not be published. Required fields are marked *

Comment *