AAPM: Disaster recovery planning takes time, but worth it
PHILADELPHIA--When Hurricane Ike hit Galveston, Texas, in 2008, Collin D. Brack, MBA, was ready. Earlier in the year, he and his IT colleagues at the University of Texas Medical Branch at Galveston underwent disaster preparedness, which allowed him to change a few processes. Those changes enabled his crew to restore IT functionality within three to four days at a remote site after Ike's rains had knocked them out.

Brack, manager of clinical informatics in the department of radiation oncology, spoke last week at the annual meeting of the American Association of Physicists in Medicine (AAPM).

"The trend is for the enterprise IT departments to plan to have mission critical systems offsite and to make those redundant through high-speed interconnects between the main facility and the offsite disaster recovery facility," Brack said. "Our disaster recovery facility is in Dallas, five hours away. We've spent years ensuring there is dedicated and sufficient bandwidth so we can have a mirrored system five hours inland that affords those levels of protection."

Brack's disaster recovery scenario is called a "hot site," meaning it delivers a real-time transactional mirror of key resources that are also in Galveston. "That is cutting-edge mission critical disaster recovery," he said. "If you don't have the resources for a hot site, a 'cold' or 'warm' site is also available. These are merely scaled down versions of the hot site."

Disaster recovery planning, part of a larger process known as the business continuity plan, includes resumption of applications, data, hardware and communications such as networking and other IT infrastructure in the event of an emergency. The business continuity plan includes planning for non-IT related aspects such as key personnel, facilities and crisis communication.

The goal of any disaster recovery plan is to identify central information system weaknesses. The plan is a dynamic document, one that must be reviewed yearly, changed if necessary and signed off by higher level executives.

"This planning document has teeth because it has the stamp of approval from the administration," Brack said. "This document tells you exactly what to do in case of an emergency."

In order to do disaster recovery planning correctly, some foundational data management/best practices must already be in place. There must be some level of data and network security, some level of data redundancy and backups, and some level of determination of what data are important and what data are secondary.

Data can be categorized as source data or legal data. What is legally required to save can vary by state, specialty and practice setting. For radiation oncology, these might include consent for treatment forms, consultation forms, operative procedure reports and practice guidelines. These data generally form the legal business record and would be released upon request.

Source data are an adjunct component of the legal business record. They are often maintained in a separate location or database and provide the same level of confidentiality as the legal business record. In the absence of documentation, e.g., interpretations or summarization, the source data should be considered part of the legal health record, Brack said.

"It's important for facilities to decide how they will delineate between the source and legal data," he said.

Other types of data are administration, such as for payment purposes, and derived data, such as monthly Q/A reports.

When backing up network data, it's important to know if the data are corrupt. Hence, network security must be in place, including antivirus programs for physical and virtual networks. Brack noted that vendors are not responsible for antivirus protection and often discourage its installation. "But all network data must be protected from viruses and any shortcomings with the information systems must be addressed with leadership," he said.

In the radiology department, team players to work on disaster recover include medical physicists, PACS administrator, IT administrator responsible for the RIS and one or two key players for the enterprise IT team, who will help understand what resources are available at the department level.

Medical physicists in radiation oncology play a more integrated role in IT management and in disaster recovery. "They wear various hats, that of the PACS and RIS administrators, and they also are responsible for treatment planning system maintenance and the integrity of all the data image management," Brack said.

The disaster recovery plan requires dedicated resources, Brack said. It might take different forms, but every group in a hospital has one. As part of the business continuity plan, it should contain passwords and cell phone numbers and addresses of key personnel. "The disaster recovery process forces departments to take a close look at its processes and have all the needed information in one place in the event of a disaster," he said.

For example, with whom will the hospital contract to take care of its cancer patients in the event of an emergency? "Someone has to identify a few key facilities that can treat your cancer patients in the event of an emergency. You have to go through an entire legal process, draft a contract, identify staffing ratios and other important aspects of patient care. That's what saved us. It took two years of negotiations to find out what that arrangement looked like, but when Hurricane Ike hit, we were ready," Brack concluded.