Beyond the buzzword: SaaS and storage management

It's easy to get lost in your own little world. But life is very good at bringing you back to reality on a regular basis.

I remember going to a family function a few years ago. A very well-educated friend of my parents' who happened to be a physician asked me what industry I'm in. Having been a new, eager employee to what was then called VERITAS Software Corporation, and having drunk the Kool-Aid of our mission to change the world with our "No Hardware Agenda" strategy, I answered proudly that I'm in the storage industry. The doctor responded excitedly that he was looking to move and needed to find cheap "storage" for his furniture during the transition. He asked for my business card.

The reality is that even in IT, storage is often an after-thought. The two things I hear from people that are not deep into this stuff are: (1) storage is so cheap and (2) we have TONS of storage. Storage, storage everywhere but not a spindle to use.

Unfortunately, in on-premise email archiving deployments, storage often ends up becoming an after-thought as well. Customers purchase email archiving solutions (whether SaaS or on-premise) for the mailbox management, E-Discovery, compliance and other benefits. Given that, in most cases, the team responsible for email and/or legal drives the decision. Buying a new, energy-hogging storage cabinet to keep the office warm is the last thing on their minds.

I can't count the number of visits I've had where the customer gets to the realization that they are going to be creating tons of data and need a place to store it. If they're lucky, they bring their "storage expert" into the room who often hasn't been informed about the project at all. In most cases, for small-to-mid-sized businesses, there is no such role, so they have to learn as they go.

Some of the challenges I've seen customers, particularly ones with limited IT staff run into:

"We have extra space on the SAN." In most IT departments, there is some extra storage "somewhere." Many small-to-mid-sized on-premise email archiving deployments often start by leveraging the available storage in-house. Unfortunately, most organizations quickly realize that archives keep growing and often very rapidly overwhelm available internal storage. In addition, as you'll see below, archive storage needs to be very finely-tuned and nine-times-out-of-ten, what the customer has in-house won't work for the archive.

"How much storage do I need?" Sounds like an easy question. I guess you can ask the storage hardware vendor and let him or her tell you, but that seems like letting the fox guard the hen-house. In reality, the answer is "a lot" and "more every day." Most on-premise vendors have tools and white papers - in many cases, ones that involve 50-100 pages of reading - to answer the question precisely. Obviously you need to assess your mail volume, message size, retention period and other factors. The sizing is surprisingly arduous and complex.

"It's an archive so I'll just use cheap disk." One of the basic principals of archiving is that old email will be accessed less frequently by users and therefore it can be stored on cheaper (lower-performance) storage. In geek-speak, this means things like using Serial-ATA drives for archival instead of Fibre-Channel for production email. Unfortunately, people take this to an extreme and often think the whole archive involves cheap storage. In reality, most archives have three pieces of data: (1) archived emails themselves, often written as files, (2) high-level metadata stored in some kind of relational or XML database and (3) search indices. The archived email indeed can be stored on relatively-cheap storage (though see below for more on this). But the database metadata needs to be stored on high-speed disk, like any database. Furthermore, the search indices are often even more temperamental about disk I/O times. As a rule of thumb, the database and index data can end up being 10% - 50% or more of the original content size. So you end up needing to buy a significant amount of significantly-expensive disk.

"This #!@# archive is slow!" Despite the best of software engineering in on-premise products, they are dependant on storage hardware and its proper configuration. Usually you only figure this out when you have a huge E-Discovery or compliance request and need to search through and export thousands or even millions of email messages. The searches can be slow due to poor performance or misconfiguration of the area where indices are stored. But the actual bulk export of messages itself can be a nightmare if the storage isn't configured correctly. Indeed, this mass export use case is why you can't just use cheap disk for archival without thinking it through at all. And of course, this all happens when you've got a lawyer camped out in your office waiting for the data.

"How do I move all of this data?" This one is perhaps the scariest of all. The math is simple. Storage arrays last 3-5 years. Archived data is often retained for 5-10 years or longer. So now you have this array with 100s of GBs or TBs of data. How do you migrate it all to a new array? How do you preserve compliance in the process? And make sure it all works? And how long will it take? This migration problem is a ticking time bomb for many customers and there is no easy answer.

Obviously with Software-as-a-Service, storage is still complex, but the customer doesn't deal with it, we do. And for whatever strange reason, I find storage fun. I've got issues...