In the past, I’ve talked about ways you can learn about the IO patterns of your applications. Once you know the characteristics of the IO pattern of your applications, you can start to think about how to construct your storage solution from the perspective of performance and capacity requirements. In other words, you understand your application, so you know what kind of storage you need to feed it (what admins will call “feeding the machine”). But, your HPCC storage education is still not finished because before you put out a set of requirements for your storage system, you need to think a bit more about the architecture of the overall solution. In this blog I want to talk about the most fundamental architectural approach – tiering.

Do You Need it All on Spinning Media?

An interesting requirement that comes up in customer requests is the requirement for all of their data to be on high-speed spinning media (i.e. not tapes) all of the time. While I can certainly believe that there are cases where this is true, I also know from experience that this isn’t the case for all customers (actually most customers).

In reality a large percentage of customers only need some of the data to be on really high-speed storage at any one time. The rest of the time the data can be moved to low-performance, and presumably cheaper, storage. More over, some data can even be moved to tape using some sort of HSM. And finally, you could archive some data that you just need to have available, but not in a timely manner.

In general faster storage costs more. So the slower the storage, the cheaper it is, except if you get to an HSM which requires a reasonable initial investment but is reasonably cheap to operate (except for consumable media). Therefore if you can decide how much data you need on really fast storage, how much you need on slower storage, how much you can archive or dump to HSM, etc., then you can optimize your cost (i.e. save money!).

This general approach is called tier-ing. It means that you can create tiers of storage that are each configured for some level of performance and capacity. If data needs to be accessed faster then it should be in a tier that is faster. It the data is not needed then it is moved to a different tier. This is really the essence of HSM. But HSM is usually an expensive add-on and usually, but not always, deals with a single set of spinning media and tape. What I’m discussing is tier-ing between various spinning media.

Spinning storage (my generic term for hard drives, spinning and non-spinning) has really changed. We have 1TB and 1.5TB 7.2K rpm SATA drives with 2TB coming very quickly. We have 10K rpm SAS drives of various capacities (all below 1TB), and 15K rpm SAS drives of various capacities. And now you have SSD (Solid State Drives) of various capacities and connections (SATA and SAS). Notice that I haven’t mentioned FC drives (Fibre Channel). In general, unless there is an application that needs more IOPS (IO Operations per Second) than SAS can deliver and the number of drives is fixed, I don’t see them in HPCC very much anymore.

It is very tempting to use 1TB SATA drives when they are so cheap (and remember that the amount of storage you need never decreases!). But there are times when you need faster single disk performance or more IOPS than SATA, and for this we have SAS. For even more IOPS, we have SSD drives.

Each of these types of drives has a different capacity, a different price point, and a different level of performance for various metrics. You can also combine this with different RAID sets (e.g. RAID-1, RAID-5, RAID-10, RAID-50, etc.) to produce various levels of performance. To me this means that we have the ability to really tailor the storage to meet the specific needs and do it at a price point that meets the customer’s wallet. This is the fundamental underpinning to tier-ing.

Figure 1 below shows you a generic view of tiering. The width of the pyramid is the capacity and the height is performance (the higher the better the performance). Figure 1 - Tiering Pyramid At the bottom of the pyramid is tape which can be very abundant, but also very slow. Then you move up to SATA, 15K SAS, SSD, and finally ramdisks as the storage media.

Example of Tier-ing

We can create a high-performance RAID set such as RAID-10 or RAID-50 using SSD drives for the top-most tier (let’s call it Tier 1). This tier has very high throughput with lots of IOPS. But since SSD’s are the most expensive this tier is likely to be the smallest. Let’s call this Tier-1.

The next tier down uses cheaper storage such as 15K rpm SAS drives or even 10K rpm SAS drives. You can keep the same RAID level as the top-most tier or you can perhaps use a slightly slower RAID level that perhaps gets you more capacity. If Tier-1 uses RAID-10, then this tier, Tier-2, can use perhaps RAID-50. It’s reasonably fast due to the RAID-0 part, but uses a reasonable amount of capacity with RAID-5.

The next tier could focus on very large capacity but lower performance. So it will likely use larger capacity but slower drives. For so Tier-3, it’s fairly obvious to use 7.2K SATA drives. To maximize capacity, you can use a RAID set such as RAID-5.

Let’s summarize this sample tier-ing:

Tier-1: SSD drives in a RAID-10 configuration

Tier-2: 15K or 10K rpm SAS drives in a RAID-50 configuration

Tier-3: 7.2K SATA drives in a RAID-5 configuration

Tier-1 is going to be fastest in a number of respects – it has higher IOPS due to the SSD and the RAID-10 will give it excellent read and write performance. Tier-2 uses very fast spinning drives giving very good IOPS and good throughput but not quite at the same level as Tier-1. Finally Tier-3 gives you lots of storage using SATA drives, but the IOPS and throughput are much smaller than the other tiers.

I Need Hyperspace

Despite the high-speed nature of the Tier-1 storage there are times that people will need even higher-speed storage. What do you do? Can you get even faster than SSDs? The answer is that yes you can, but it will cost you :)
The way to get to faster storage is to use… ramdisks. Take a number of nodes and use about 70-80% of the memory and create a ramdisk. To the OS this appears like a storage device. You can then put a distributed file system across these ramdisks in the nodes, preferably one that can stripe across the ramdisks (don’t forget to connect the nodes with the ramdisks using a very high-speed network – preferably at least DDR IB using Mellanox ConnectX HCA’s).

Let’s call this Tier-0. If we add this tier to our sample set we have,

Tier-0: Ramdisks with striping

Tier-1: SSD drives in a RAID-10 configuration

Tier-2: 15K or 10K rpm SAS drives in a RAID-50 configuration

Tier-3: 7.2K SATA drives in a RAID-5 configuration

Now Tier-0 is the fastest tier in the entire file system. Ideally you would like the parallel file system across all of the tiers. If not, you create a file system on each tier then you just mount them on different mount points on the compute nodes.

Wait – There’s One More Step!

Before you grab these tiers and start building your storage system, let’s talk about a practical approach to tier-ing. It may be difficult to know which tier you need, what capacity you need for each tier, etc., until you deploy the storage and start production work. Also remember that you are likely to need more storage, so being able to expand existing tiers or deploy additional tiers is a HIGHLY recommended idea.

So, a good approach to deploying your storage is to start with Tier-3. It has the largest capacity, usually the lowest price, and the lowest performance. If you need more capacity you can expand it. But if you need more performance, then you deploy Tier-2 storage. It’s unlikely but possible that you will need Tier-2 when you first deploy the storage so first deploying Tier-1, watching the performance and the usage, gives you some time to determine what the Tier-2 storage should look like and when you are likely to need it.

If you deploy Tier-2 storage and discover that you need even more performance, then you can deploy Tier-1 storage. And if IO is still a bottleneck, then you can deploy Tier-0 storage.

What this “phased” approach to storage gives you a couple of interesting features. The first one is that it allows you to optimize your costs. If you don’t need faster storage, which is usually more expensive, then you don’t pay for it. You pay for the capacity and performance that you need.

The second feature that tier-ing gives you, is, well, cost. Remember that storage is one of the few aspects of society that gets cheaper over time. For a given level of performance and capacity storage gets cheaper over time. Alternatively, for the same amount of money, you can get faster storage and/or more capacity.

So phasing your storage into production using a tiered approach is a very smart idea because of a simple thing – money.

Final Exam

After reading this blog, I think you are ready for my simple final exam. Here’s your first and only question.

“When deploying storage what general approach should you take?” (in 5 words or last) (insert answer here)

For the impatient among you here’s the answer (no need for Cliff Notes): Tiered Storage If you had this answer written down, then you can graduate and move on to the next stage – choosing RAID sets and file systems (graduate courses).

But if you didn’t have this answer written down, that’s OK – don’t feel bad. I can count on 2 fingers the number of people that I know who truly understand the concept of tiered storage. So there are very few ahead of you in the class. But to make sure you can answer the question next time, let’s do some reviewing.

Tiered storage allows you to tailor the particular tier to the application. You can choose the storage devices, the RAID set, and the capacity for each tier. Then you design the lowest tier to be the slowest but with the largest capacity.

Then you deploy the lowest tier first, monitor the performance and the usage and make some decisions based on measuring code performance. If you need more capacity but not more performance, you deploy more capacity at the same tier. But if you need more performance, then you can deploy a new tier that is faster. You continue to do this as you need more and possibly faster storage. Ultimately you keep expanding up to Tier-0.

And finally, for extra credit, why is tier-ing such a good idea? The simple answer is – money. You can become the teacher’s pet by explaining why. The why is also simple – you tailor the capacity and performance to the demonstrated requirements so you don’t use too much hardware, hence saving money. In addition, since storage always gets cheaper over time, by delaying capacity and performance increases, you can save money.

Feel smarter? Believe me, if you understand these simple concepts and implement them, then you are light years ahead of a huge percentage of HPCC users. That means you are smarter. You’re Welcome.

Jeff