At Storadera, we are obsessed with speed
Usually, (NVMe) SSDs are considered delivering great performance due to way higher IOPS.
*IOPS means Input / Output operations Per Second
We offer S3 compatible cloud storage (object storage) and obviously we need high performance storage system.
It may seem a bit controversial but we store customer objects on hard disks (HDD, also called hard drive or spinning rust).
How we can offer fast service on hard disks? To answer this, we need to understand performance difference hard disk vs. SSD or HDD vs. NVMe.
How much faster are SSDs compared to hard disks?
To answer this question, let’s go back to the reason why these days SSD is the go-to solution for anything that needs high performance.
The reason is that spinning hard disks have a dramatically lower number of IOPS.
Here is a quick comparison of one hard disk and two very different SSDs:
| Storage Type | Connection protocol | Write IOPS | Increase over hard disk | Source | |
| Western Digital HC580 24TB | Hard disk | SATA | 565 (c.a. 0.6k) | HC580 | |
| Samsung PM893 SATA | SSD | SATA | 30k | About 50x | pm893 |
| Micron 9400 | SSD | NVMe | 800k | About 1400x | 9400 |
Why these specific models:
- WD HC580 24TB: we hope to start testing this model soon in our data center. It has amazing ArmorCache technology but this is a topic for another article.
- Samsung PM893: we use this in our data center for object metadata storage. It has a SATA interface but it is more than enough for our use case.
- Micron 9400: at the time of writing, it is one of fastest NVMe SSDs available on write IOPS. These drives go to crazy capacity for SSDs (30TB) and price as well.
SSDs have 1400x IOPS avantage. Why choose hard disks?
Some obvious benefits of hard disks over SSDs:
- The price difference can be easily over 10x in favour of hard disks. If we can offer fast enough service on 10x less expensive hardware then it sounds like magic. Check out our pricing to confirm that Storadera has one of best prices in the industry.
- Reasonably priced SSDs are still much smaller in capacity and require more connectors. This in turn requires bigger and more costly server infrastructure around the drives.
- SSDs have a limited amount of terabytes written (TBW). After that, they just become read-only. Hard disks don’t have such strict limitations.
How typical storage systems operate with blocks
Typical storage systems divide incoming data to blocks. They can be very small, from 4KB to bigger, as big as 1MB to 32MB in size. There are many different flavours how storage systems are built, and how different components are called.
Still, often, there is some unit of data that has a fixed size that will be written to disk.
It is, of course, a huge oversimplification but let’s consider a hard disk having 500 IOPS maximum. On random write, IOPS can go down to 100. Now if we consider that writing one block takes 20 IOPS then we can have 100 / 20 = 5 block writes per second.
If the block size is 1MB then it means we can write 5 MB/s. Not a very huge number, considering hard disks can reach sequential write speeds close to 300 MB/s.
Some storage systems directly write customer files to the hard disk, without dividing or collection to blocks. In this case, small files (let’s say less than 100KB) are very problematic as they consume all the IOPS.
Additionally, there can be different factors like Erasure Coding and other reasons for overhead.
Trust me, even 5 MB/s is still a good number for such systems, many storage systems go below 1 MB/s when the written chunk is very small.
Such low speed is obviously a problem!
Storadera’s Variable Block Size Technology is the solution for IOPS limitation on hard disks
So, we concluded that writes and IOPS are very limited on hard disks.
We could increase block size to achieve better overall throughput. This would definitely solve this one thing but it has downside of longer latency.
For example, if you want to write 100MB block then you have to wait until customers have sent you this 100MB and then you can start writing it.
This causes higher latency, especially at the moments of lower incoming traffic. For background, we measure things in milliseconds. Huge block size on low incoming traffic moment would easily cause extra 1000 millisecond (1 second) or more on processing single upload request for the customer. So, this isn’t an acceptable solution.
Math tells us a really simple solution. We should use small block size when less data is coming in and bigger block size when more data is coming in. This is exactly what we did.
Storadera’s Variable Block Size Technology helps to overcome the IOPS limitation by changing block size according to current volume of incoming data. It balances block size with incoming bandwidth to always keep latency low and avoid being limited by hard disk IOPS. This optimization enables us continue offering well priced and high performance s3 compatible cloud storage.
The software engineering efforts to achieve this were quite high but we are happy with the results so far.
Storadera Variable Block Size Technology is available in our cloud locations (link).
Sign up to try using it for free
Did you know that you can have it at your location?
If you plan a Petabyte scale s3 compatible object storage, we can deploy it to your location and fully manage it for you. If it is a new hardware purchase then we can advise on the choice of hardware. And we can work on most existing hardware.
Read more to get started with your own Petabyte scale S3 storage