Protecting IT from Disasters

© 2010 by Nick B. Nicholaou, all rights reserved
President, Ministry Business Services, Inc.
Reprinted from CTI’s Your Church Magazine

“Only the flexible survive.”

It’s a phrase we’ve heard— and possibly said— many times.  Life is unpredictable, and challenges do come our way, so how should uncertainty affect the way we approach information technology (IT) in our churches and ministries?  We are dependent on our computers today in ways we might never have imagined.  We rely on them to store contact information, process financial transactions, communicate— many of the nuts and bolts that enable and hold ministry together.  Therefore we should do all that is reasonable and cost-effective to protect our organizations from being shut down by a disastrous event.

Planning now gives the flexibility needed for ministry survival.

And in the ever-changing legal and financial regulatory landscapes, IT disaster recovery plans are even more necessary.

Audits Force IT Planning
Because of the role IT plays in our ministries, certified public accountants are now required to test for good IT strategies.  Those familiar with CPA audits know that auditors are increasing their focus on IT issues.  Even if your church doesn’t get audited, these areas are wise to address:

• Management Oversight.  Who is responsible for your church’s IT decisions?  Chances are good you’re relying on a talented staff member, volunteer, or vendor— and that’s okay!  CPAs need to see that implemented strategies they may have recommended were overseen and approved by top management.

Those in top management may not have the expertise to make technical recommendations.  But they have perspective of the ministry’s direction, and hopefully, a sense of whether a recommendation is a good fit for the organization by way of integrity and stewardship.

• System Security.  This is an area of great importance and it must be addressed. Tackle such issues as:
• Firewalls for systems that have full-time Internet connections;
• server room security; and
• additional policies to keep out those who should not be on the system.

The most common mistake made in this area is the password policy.  The corporate standard in the U.S. is to have cryptic passwords of a significant length that cannot be easily cracked.  But that strategy doesn’t work well in church and ministry offices.  We have found many times that those kinds of passwords are so hard to remember that most church and ministry system users write them down on a note kept under their keyboard, in their desk calendar, and so on.

The better policy:  Require passwords to be at least 6 digits and include all of the following:

• At least one number,
• At least one punctuation, and
• At least one capital letter.

We recommend using an acronym of a favorite verse or worship song.  These are easily remembered, cryptic enough to not be easily hacked by an Internet program, and offer the added benefit of reminding folks of the reason they’re logging in to your system.  This strategy, coupled with using the network’s Invalid Login Attempts function (we set it to lock a user account for thirty minutes after three unsuccessful login attempts), is good and effective protection.

We also recommend not letting users change their passwords because most will not choose strong enough passwords, making the system vulnerable.  So don’t set passwords to expire; instead, change them for users if they’ve shared theirs with someone.  The policy should state that staff members are not to share their password, but that if they do, they should inform their supervisor and have the network administrator set a new one.

• Record Retention.  Another important area with legal ramifications involves record and data retention.  Attorneys who specialize in the IT field say there isn’t a simple remedy to this policy requirement.  Based on their comments, though, retention policies should address two areas:
  1. Email.  Email should be archived for two years unless your state requires something longer.  This would satisfy most legal challenges in federal and state courts, and the most important factor is that you have a policy in place that you adhere to.
  2. Files that may be necessary in pending litigation.  Any files that might be subpoenaed in litigation must be archived in their original format and held indefinitely.  That includes personnel-related files, church governance files (including minutes), negotiations, and so on.

Disasters Come in All Sizes
Churches could experience most any kind of disaster, and should one occur, it is good management and foresight— and the favor of the Lord— that save the day.  Each of the following disasters may affect a church’s IT strategy:

• Fire
• Burglary or theft
• Earthquake
• Storm damage
• Water damage from pipes
• Data theft
• Equipment failure

Reasonable protection against these disasters does not require a NASA-sized budget!  Some disasters can only be minimized by investments in hardware (like a backup system), but others can be minimized by simply having good policies in place.

Keeping Things Running
Having a solid backup strategy is essential, but it is not enough.  Though backups address disaster recovery, they don’t address business continuity.  Business continuity policies speak to how the church will survive a larger disaster, such as a fire, hurricane, tornado, or earthquake— and keep running.  How will the ministry survive if one of its most important and valuable assets— its data— is no longer available?

• Backup Strategy

Whenever possible, it’s best to back up the entire system often, using one of the following two strategies:

1. Nightly Tape Backup. Tape backups have improved their speed and capacity, keeping pace with the requirements of most churches and ministries.  It’s best to back up the entire system every weeknight and take one backup tape off-site each week.  Choose the backup tape from your heaviest processing day each week to take off-site.  For instance, most churches do most of their data processing on Monday because they’re modifying the database with new members, address changes, and contributions.  In those settings, it’s best to take Monday night’s tape off-site.  That means asking a trusted member of management to put the backup tape in their backpack or purse every Tuesday and bring back the previous week’s tape.  This inexpensive step can be very helpful for recovering data in case of a burglary or other disaster; and doing this weekly means the church never has to rebuild a system with data that is more than a week old.  One precaution to consider:  Because the backup information is sensitive, a church might want the person who takes the tape off-site bonded, in case something should happen to the tape and its contents while it’s in their care.

   There’s a lot of discussion among IT professionals about getting beyond tape technology.  Some alternative choices are external hard drives and online storage.  When I recently researched corporate best practices in this area, I found the majority still prefer tape because of its lower overall cost and its reliability.  Very few have changed to alternative media options.

   My firm’s tape backup recommendation involves an external LTO4 backup drive attached to the file server via SCSI cable. Dell (our favorite hardware vendor, quote #558200935) quotes that setup for $2,018, which includes 20 backup tapes and delivery but no state or local taxes.  For software, we prefer Symantec’s Backup Exec, and we’ve found the best place to buy it is Consistent Computer Bargains (800-342-4222).

   We like to configure the backups to run automatically Monday through Friday nights, to do a full system backup (excluding workstations since we always configure them to save their data to the file server), and to do a full comparison of the backup to the original files before ejecting the tape.  The LTO4 spec can backup at least 800 gigabytes of data (the manufacturer claims 800 gigabytes to 1.6 terabytes, but we’ve found it’s best to only rely on the lower number), and do it very fast.

2. Constant Data Replication. Larger organizations that need much larger server hard drives use a technology called Storage Area Networks (or SANs).  SANs are expensive devices with lots of storage capacity and redundancy (safety) built into them.  In organizations that use SANs, they often run at least two of them in different physical locations to protect against disasters.  They then synchronize the contents of both so that they are constantly backing up each other.  This makes their data available in an instant.

An example might be to have a SAN at each end of a larger building or larger campus, and possibly another off-site.  In my research of best practices, I found that corporations with larger needs than what tape can satisfy use SANs.

My firm’s SAN recommendation is an EqualLogic built to the storage capacity needs of the ministry. We prefer EqualLogic because it has the best combination of engineering (abilities and tools), redundancy, and price.  We’ve found the best provider to churches and ministries to be VR6 Systems since it discounts them steeply for churches and ministries. One caveat, though, is that to get VR6’s best price, they need to be your first contact regarding EqualLogic since the manufacturer will otherwise require VR6 to share the profit from your purchase with whichever other vendor has “registered” you first.  We recommend emailing Jason Powell at jason.powell@vr6systems.com. Contacting Jason first will save you thousands and will bring this better technology closer to your reach.

• Other Important Considerations

Whether yours is a small or large organization, there are a couple of strategic issues that will make your disaster recovery or survival through a disaster more likely:

• Data Location.  There are two possibilities regarding the location of your organization’s data:  local drives or server drives.  Server drives don’t fail as often because they are engineered for higher performance and reliability.  We recommend configuring your system workstations to focus their data storage on the server drives, which will help achieve a more predictable outcome when trying to survive a disaster.  It will also help your team because it will mean that a system user can log in at any workstation and get to their data if their workstation is unavailable due to theft, hardware failure, or something else.
• Avoid Quick and Cheap Fixes. Quick and cheap fixes have many more shortcomings, and they make achieving a full and timely recovery from disaster less likely.  In addition to not being able to include necessary system files and being less reliable (something usually not discovered until they are really needed), trying to recover through them often costs significantly more. 

A great rule of thumb when thinking of network technology is that if the hardware you want (desktop computer, notebook computer, switch, backup solution, and so on) is readily available in stores, it’s most likely the wrong option and you don’t want it.  Instead, establish a relationship with an IT firm you trust and consult with the people there.  Doing so will save you both time and money.  You will even find that the right solutions often don’t cost more than poor solutions, and sometimes even cost less.

Business Continuity
Recovering from a disaster is essential, but some disasters in recent history (such as Hurricane Katrina and 9/11) have emphasized the need to be able to survive and minister during the disaster and during the disaster recovery.  Though the difference between disaster recovery and business continuity is subtle, it is important.

There are certain types of data your team will need to access during a disaster and during the recovery, and the bigger the disaster, the longer it may take to recover.  In Hurricane Katrina, people were evacuated in all directions, yet still needed to communicate and process certain financial transactions, such as payroll.  In 9/11, organizations’ data structure was vaporized with no possibility of recovery.  These are rare but clear examples that should guide preparation.

Step 1.	Categorize the types of data in your organization.  Some categories might be congregational databases, financial systems, email, letters, various ministry and department files, pictures, videos, audio files, older files, and so on.
Step 2.	Meet with leadership and ask them to prioritize what categories of data need to be recovered, and within what timeframe they need those categories available to the team.  Let them know that the decisions they make will drive the strategy and expense of your disaster recovery and business continuity plan.  For instance, if everything needs to be recovered and running within two hours, you’ll need to have a SAN, but if not, then tape backup will do.  And if certain databases or email needs to be up and running within a couple of hours, then you may need to look for a vendor who can host them online when needed.
Step 3.	Research IT best practices to meet the requirements of your leadership’s decision, and present a budget and plan.  If they require you to make modifications, do so explaining the consequences to their original requirements.  When the budget and plan are finally approved, and they state the recoverable timelines they should deliver, have leadership formally approve it.
Step 4.	Implement the new plan.
Step 5.	Test the new plan.  This is the piece that is often not done, but it is essential to ensure compliance with the requirements of leadership.  Be sure to report the results of the test to leadership and get their approval.

By doing all five steps detailed above, you will have assurance that you have balanced the needs of the organization with the available funds to do what is best.  Further, in the case of a disaster, leadership will have clear expectations of how the systems will come back online, which will help relieve stress.  This will also make your CPA happy at the next audit.  And best of all, it will help you and your leadership to hear the words we all hope for at the end of the journey: “Well done, good and faithful manager.”