What, when and why it’s worth every dime.
Having been in various forms of executive management for over 22 years, I can attest it is sometimes hard to fork over the cash for “what if” scenarios. Sure, we all have health insurance, car insurance, life insurance, and home or renter’s insurance, but what about Disaster insurance?
I am not talking about a policy with your local insurance agent; I am taking a continually tested, often updated, business continuity plan. One that will ensure your company will survive in the event of a disaster. We have all heard about this, but I wanted to chat about the nuts and bolts of what this means and why it is important.
Recently, our company, SmartSearch had to initiate its Disaster Recovery Plan. In December 2017, fires broke-out in Southern California. The Lilac Fire of Northern San Diego destroyed 185 structures, many of which were homes and some were businesses. The SmartSearch offices were in one of the voluntary evacuation zones of this fire. Strong Santa Ana winds fueled the flames that headed towards our Oceanside home office. We host personal information on over 35 million candidates, as well as hundreds of staffing firms and companies that rely on us to keep their data safe. Many of our employees were in evacuation zones as well, some voluntary, some mandatory, some in critical zones. So, our challenge was, “How do we allow our employees to attend to their family needs while keeping all systems operating, even if our building were to be destroyed?” This situation is what keeps a CTO like me up at night.
So, what did we do? In short, every aspect of our Disaster Recovery Plan was executed without a hitch. And how were we able to do this? We could execute due to diligent planning, planning, and more planning. I can attest, every second spent, every dime spent was well worth it. It allowed us to go through a potentially high-stress situation with calm, defined purpose and confidence. I can confidently say that this organized and prepared strategy kept stress at bay and us on course to execute the plan.
At this point, you may be thinking, “Yes, but what are the chances of a fire breaking out where I live? It won’t happen to me.” This should be the time to discuss what a “Disaster” is. It does not have to be a fire, or earthquake, or tornado, or flooding, or hurricane. It could be a three-day power-outage because a car drove into the power-box down the street. Disasters come in various forms such as your ISP closed their doors without warning and you have no Internet access, an upset employee steals or destroys your data, a hacker gets into your system with ransomware or a water-line break in your office. A disaster recovery plan needs to address all disasters, whether it be from nature, hackers, bad driving, or just bad luck. After all, you are not just protecting your clients, or your data, you are protecting people from financial and emotional harm. If your business goes under, people lose their jobs; other companies may lose their businesses, and lives can be, dramatically, impacted. As corporate leaders, we have a responsibility beyond just data and buildings, people trust and rely on us to plan accordingly, and take steps to protect their future.
When the fire broke out, we knew it was a dangerous situation. Before any evacuation orders came in, we immediately went to our Disaster Recovery Plan to review it. A good plan delegates tasks to key employees. Those employees should know their roles and practice them on a regular basis. A good plan will also have contingencies in case some employees can’t be there due to their personal-life disasters.
Our plan starts with identifying the threat. Before you can fix or prepare for anything, you must first, correctly, identify the danger. In this case, our threat was a catastrophic, fast-moving fire heading in our direction. As mentioned, it could have been that our network performance was brought to its knees by a “denial of service attack.”
We had multiple possible threats to consider. The power company was cutting power to areas of the fire and near the fire to reduce the possibility of electrical fires and accidents. So, we had a very real chance we would lose power and/or Internet access. We powered up our on-site generator, verified fuel was all capped off, and our backup-tank was also fueled. Redundancy is also important in a contingency plan. In our office, we created redundancy by bringing in multiple fiber lines from different providers. One uses the local phone carrier’s lines for, “the last mile,” it is above ground. Another is below ground, leaving the building in a different direction which goes to Cox Business Networks on their private network.
Additionally, our strategy requires a communications plan to all stakeholders. Nightly, we export an emergency list of contacts off-site in case we need to initiate a broadcast text or email from outside our offices. Our plan also utilizes cloud technology to store back-ups, as well as traditional tape backups stored off-site. We verified all was well with our off-site facility to act as a new IT hub for us, in the event our corporate offices, which host a data-center, became damaged or inaccessible.
As the fire grew, we officially initiated our full Disaster Recovery Plan. Backups were made of all systems, on an on-going basis, so if we did have to change locations, the data loss would be minimal, if any at all. All local backups were moved off-site immediately but close by in proximity. Our co-location center was prepped to take on the full load of all clients, and our various Cloud-based contingencies were initiated and prepped as well. Communications were sent to clients. Tasks were re-delegated based on the availability of staff, as many had to go to their own homes to begin evacuation.
As the voluntary evacuation orders came into our offices, we reduced our on-site staff to the minimum needed, and other off-site employees continued additional preparations as defined in our Disaster Recovery Plan.
Luckily, the fires did not reach our offices, we did not lose power for days, and we survived with zero downtime. (Though in the past we did survive a multi-day power-outage with zero interruption of service.)
But having a plan we are all familiar with, and that we test often, provides our employees and customers with invaluable peace of mind. During this event, our team operated in a professional manner that made me proud.
So, that is our story.
The question is, are you prepared?
Here are some questions to ask yourself:
Do you have a plan to communicate with all stakeholders?
Do you have clear guidelines as to how to evaluate and determine what threats you are facing?
Do you know who does what and when, and who takes over if a person is not able to carry out his assigned tasks?
Do you have different plans to deal with data breaches, flooding, fire, power-outages, employees who damage your company property? What happens if you can’t get to your offices? Do you have off-site backups for data (i.e., Cloud, tape, etc.)? Is there a plan to get that data to where it needs to be? (We keep accounts open with various Cloud providers that allow us to ramp-up quickly with virtual servers, etc. if needed… one of the benefits associated with the modern age.)
Does your business rely on the Internet? What if your Cloud provider is compromised and do you have a backup plan for that? For those on the Cloud, is your backup data stored separately from your Cloud provider (maybe Google Cloud stores your backups, while Azure hosts your primary data center).
If you have a hybrid system of hosted servers and clouds, is everything redundant? Do you have extra-parts on-site?
And what about the people? They keep your company running day-in and day-out. Do you have any contingency plans to assist employees? Something as simple as an internal Facebook page or Twitter account for employees to communicate back and forth can help during a disaster.
In closing, I hope disaster never strikes anyone, but if it does I assure you of this, the time it takes to plan and prepare ahead is well worth its weight in gold.