There’s a sixth sense we in IT get when something’s up. Our technology nose detects a whiff, a change in atmosphere, and then the hairs go up on the back of our necks. It did with me one day recently. It all started when a small gathering of those in the operational group were peering at our real-time screens for operational data; I just made out the hushed sound of two words: “Oh no”.
So what happened? Well, some time back my Service Delivery manager had flagged that our SAN was running out of capacity and that our data center was nearing the end of its life. We checked our de-duplication software, and rather vainly we felt that it was working, but it was clear we were rapidly running out of capacity and we needed to get more storage.
One day we got caught off guard. We sat and watched as half the network was down, with the rest coming right behind it. We followed our process, but everything had happened at once before our designated solution had arrived.
I didn’t have to wait for the incident report to know this clearly couldn’t happen again. Mistakes happen, but a more flexible and agile solution would have allowed us to take some corrective action as soon as we needed to without watching a major incident unfold in front of us. Hopefully other enterprises can learn from our mistakes. Cloud based systems for smaller organizations can be very helpful in cases such as mine.
There are challenges:
The software doesn’t always works as intended, you have to watch for hidden costs, and it can be a lot of learning for some organizations and IT leaders. But I’m convinced this is the only way forward. I don’t want to invest in infrastructure that is oversized or has a backbone that can be scaled on the possibility of some future event.
Let’s face it; many businesses are struggling to offer meaningful strategies more than 18 months ahead of time, and we often plan on what we know rather than expectations. It’s a no-brainer then to build in a flexible approach to processing power and storage, and it even makes for easier conversations with the finance director.
Our plan at Oxfam now is to have a significant amount of our storage devolved off site and through a fully scalable solution. In the same way our website is now scaled to demand profiles on a cloud solution, I want our storage to have the same.
We rely too much on in-house capability, and it’s a challenge not only to maintain skills but also to the physical infrastructure. Our main data center is coming to the end of its life, so we now have an opportunity. The challenge I’ve given our team is to get us a solution that has flexibility, is easy to manage and keeps the finance director smiling.
Persuading the business to invest in storage that is just sitting there until it is needed is not a popular strategy nowadays, but this incident had a major impact on our office, and it’s clear that if we had the ability to scale up the extra storage required quickly, we could have avoided the outage.
We still have to manage what goes on and where, but that’s true no matter what technology we use. Having a flexible strategy would’ve helped to put my mind at ease and calm that IT sixth sense.
