This is the 276th article in the Spotlight on IT series. If you'd be interested in writing an article on the subject of backup, security, storage, virtualization, mobile, networking, wireless, cloud and SaaS, or MSPs for the series PM Eric to get started.
In our industry there’s a tendency to become a bit tunnel-visioned when it comes to Disaster Recovery. When it’s time to plan it out, we do our research, make some calls, send some emails, gather costs, write up our plans, get them approved, test them and file them away until the next audit. Then we do it all again. And, it all seems so elementary when we test, doesn’t it? Just like everything else in life, it all works perfectly when it doesn’t really need to.
Then something actually happens and a bit of panic starts to set in. Will our DR plan work? What if XYZ doesn’t work how it always has in the past? How much downtime should I expect today instead of yesterday because of this or that?
The technical world is full of variables. That’s what makes it so exciting.
But, what I want to talk about is the human side of Disaster Recovery. In the above scenario, we’ve taken every bit of tech that we can control, plugged it into a plan and gave ourselves permission to sleep at night. However, there’s an entire category of resource that generally goes unrepresented in all of our DR plans: users. What will they do? How will we handle their concerns? How do we turn their fears into some degree of comfort? Basically, how do we control the uncontrollable?
Through customer service skills: You know, those tools you were kinda taught the day before you took your first sweaty-palmed nerve-wracked tech support call? Yes, those. You have them... and chances are, you’re better at it than you think.
When I tell people what I do, they usually respond with, “Oh, a computer guy.” And, yes, I do fancy myself as a person who knows a thing or three about computers, but more importantly I’ve always seen myself as someone who helps people. Whether it’s helping with a computer or being tall enough to reach the whatever up on the tallest shelf, my job is to make sure that people get the help that they need to do their jobs. And, after a significant amount of time in this field, I’ve come to find that most of you reading this see yourselves the same way.
So what does this have to do with Disaster Recovery? Everything, my dear Watson!
In the case of a disaster, you’ll need to put your DR plan into place as soon as you can, but you’ll also need to communicate with your users. They’re going to be scared. They’re going to panic. They’re going to run up and down the hallway waiting for the sky to fall. Like a needy spouse, they’re going to need you to COMMUNICATE.
1. “Yes, it happened.”
The first thing you’re going to have to do is ACKNOWLEDGE THE PROBLEM. You know it happened; your users know it happened, but they haven’t heard from you yet that it happened. Whip it up into something clear, concise and understandable in non-technical terms: “Fileshare1 had a BSOD and upon reboot, CHKDSK found a bunch of disk errors it couldn’t fix and now the machine is unable to come back online. Can’t get the boot partition to load, so it’s going to be some time before we’re back up and running” is only going to get you a collective, “...the hell?”
Instead, try: “Our file server went offline, but we’re investigating and plan to have it up shortly”.
Short, sweet, to the point in language they can understand. Not only does it make the problem more easily understood, it makes your task more relatable. So get this out during or even before you start working on the fix.
2. “I’m sorry for any inconvenience this may cause.”
Step 2 is TAKING RESPONSIBILITY for the problem. The buck stops with you, admin.
Just recently my network was attacked by CryptoLocker (a feisty piece of ransomware that is currently making rounds), which finds its way into machines either directly through an email attachment or called in off the net by some other resident chunk of malicious code — most likely there because of a previous bad email attachment. Either way, the user had to take action in order to let this fox into the henhouse.
Does that make it the user’s responsibility? Nope, it’s all mine. Perhaps if I had trained them differently? Maybe I could’ve put stricter safeguards on my network? Should I not allow my users to map network drives (that’s how CryptoLocker gets around) and set up shortcuts to UNC paths? There’s a million things I might have done to prevent this (most of them completely impractical), but that doesn’t change the fact that it happened.
No use crying over spilled milk. Take responsibility for the problem because you’re certainly responsible for the fix.
3. “I want to let you know right away that your data is safe.”
Your users are going to have a lot of concerns when a disaster strikes, and your job is to address them. Here’s where you really get to save the day — to say, “Yes, it’s bad, but…”
In the case of our bum file server, we have offsite backups (so do you, right? RIGHT???) so I get to tell my users, “Regardless of what happened, we lost no data. It’s all safe.” Then I get to enjoy the sound of the collective sigh of relief traveling through the office like a stadium wave.
The cool part about this is that even though it might not be an immediate fix, your users are most likely already satisfied because you’ve been communicating. They know you know there’s a problem, they’re off the hook for what they might’ve done to cause it, they know you’re fixing it, AND they know it’s not going to hurt their workload. What user wouldn’t be happy with that?
4. “Please don’t hesitate to contact me with your questions!”
Now more than ever this is very important: ALWAYS leave the door open in situations such as these.
Your users will have questions: How do I get this back? Did I lose anything? What happens to changes made since the last backup? Why did this happen? You’ll be the only one that can answer these and other questions about the disaster. But when you do, remember to do so with a smile.
We’ve all seen it before and, in some of our cases, we’ve actually been on the other side of this. We know how dramatically these situations can be taken by users. But, like a child or a pet, they’ll only see as big a problem as you see. So, if you’re patient, understanding, reassuring and confident about the outcome, they will be too. Even if they have no idea what happened or what it’ll take to fix it, they’ll follow your lead.
5. Disaster recovery is about people.
At the end of the day, the problem may or may not be solved, but you’re well on the way. You’ve got your plan in hand, you’ve called home and told your spouse/partner/kids/guild/whatever that you’re gonna be home late, and you’re ready for a few hours in the server closet and some ninja-like command-line-fu over a pot of strong coffee.
But what’s special here — what makes you a rock star— is that before going to work on the technical problems that befell you, you’ve already taken steps to overcome the other disaster: the bigger disaster, the one that happened to your users.
Through communication and interaction, they may not be necessarily happy about what happened, but you’ve already made a world of difference by respecting and maintaining the trust they’ve placed in you. Remember, they hired you to be the master of all things tech, but the only reason you’re still there is because they trust you. And, that trust is earned — so feel good about that.
---
Got tips, thoughts or DR planning tales of your own? Chime in in the comments below!