Posted on Behalf of Marianne Soucy, Manager, Client Education Services
One of the worst support calls I have taken in my IT career was for a client that had a new employee accidentally delete an entire drive of a virtualized SQL server.
Should he have had access to do so? Of course not, but it was done.
I was relieved to learn that they had a backup plan in place, with nightly backups happening regularly.
I was then informed that the backups were stored…on the same drive as the databases they were backing up. And nowhere else.
There were no backups.
I wanted to throw up, and I wasn’t even the one who had made any of the catastrophic mistakes in this chain of events.
It made me realize how some of the most “common sense” IT procedures are not necessarily common sense, until one has seen the chaos that can be created without following best practices. Here are some of the best practices we recommend around data backups:

  • Back up all critical data automatically, on a schedule that leaves the least exposure window for data recovery if needed.
    • Review what to back up with the business users. While databases might seem obvious, there may be data stored in folders or flat files locally or other areas that the IT staff may be less aware of. Understand the entire data flow when designing your backup plan.
    • Are your backups full backups, or incremental? What is the restore point? What data will be lost if called into play?
    • Verify periodically (we suggest quarterly) that any backup processes are actually taking place, and leaving usable backup files. It is not uncommon for a backup job to be set up, but something changes (like a database name or server move) along the way, leaving the original backup routine no longer actually capturing the data it is supposed to.
  • Is there an alerting mechanism on any failed backup routines? Who monitors those?
  • Is there an alerting mechanism to hard drive failures or “out of disk space” events? Even better, can it be set to warn staff BEFORE the system is fully down (low disk space)?
  • Storage of all backups should be on hardware OTHER THAN the hardware being backed up, in case of failure.
    • Ideally, with removable media or cloud copies for true disaster recovery (Offsite tape, CD, Cloud).
  • Conversely to the point above, how are you protecting access to the PHI stored on those backups?
    • Is any removable media treated with high security?
    • Do you have a destruction process for older backups, both physical and electronic, containing PHI?
    • Who has access to this data? Is there an audit process in place?
  • Know *and practice* your backup recovery process.
    • How many machines are involved in a particular data flow? How will the recovery process be different if machine X or machine Y is the one that fails?
    • Know who has access to where the backups are stored, and how to reach that staff at all hours
    • Know how the data will be restored or recovered, and how that process will affect the length of the downtime incurred.
      • For example, in the case of a SQL server, is there another SQL server already installed and licensed that may be used?
      • Is there hardware/a VM where one could install SQL to replace the down hardware? How long will that take?
        • During off hours, who does know how to install and configure SQL to the same specifications? Do they know what the license keys are?
      • Does the replacement hardware have enough space and RAM to replace the existing hardware, even temporarily?
      • If your recovery process is to roll back to a snapshot of an entire server, how will that affect any OTHER processes or data living on the same server? For example, to rescue a corrupt integration database, would you also be rolling back a database on the same server for your finance department?

We strongly suggest a written recovery plan, with a periodic exercise to mimic a failure event, to make sure the plans and processes work in reality the same way they were conceptualized to work. It is time very well spent to identify any potential holes in your plans before they are needed!
Please let us know if you need any assistance in designing your backup plan for your Summit products.