When doctors need to make life and death decisions with images from state-of-the-art imaging equipment, network engineers know that good planning and redundancy can be essential. Doctors and nurses at San Antonio Community Hospital (SACH) must have access to as much information as possible to diagnose their patients' ailments whenever needed.
SACH is located in Upland, Calif., some 35 miles east of Los Angeles. The region represents one of the fastest-growing metropolitan areas in the United States.
Fulfilling its commitment to provide leading-edge medical care, SACH upgraded its radiology services with a picture archiving and communications system (PACS) three years ago. Because PACS images can be created, stored and shared anywhere, healthcare providers have come to value the immediate access to the diagnostic information. For that reason and others, a PACS network failure is unacceptable. The loss of radiology images would delay critical diagnoses. So, SAHC set out to create a PACS network with reliability equal to a telephone switch and with recovery time of less than 100 milliseconds.
THE CRITERIA
In order to replace x-ray film, SACH had two major requirements before installing the PACS. If doctors were going to depend on the digital images to make life-and-death decisions, the network that delivers the images must always be operating. The previous network infrastructure could not have handled the increase in network traffic, so the facility would have to install and manage a new network that never fails.
When completed, the SACH PACS network would need to process 11,000 exams per month from 30 image capture units of various types and present the images to 25 color and black-and-white diagnostic workstations. The images would be archived up to three terabytes on RAID5 disc storage and high-speed tape jukebox for historical data. Doctors would need to be able to access a PACS image in two seconds on-demand.
Currently, SACH stores and displays approximately 30,000 files per month from modalities that include, ultrasound, CT, MRI, nuclear medicine, computed radiography, digital radiography and fluoroscopy. Both the network and the wireless system must scale for bandwidth for the heavy traffic load.
The most important requirement is that the network must never fail. It must be operating 99.999 percent of the time. That prerequisite equals less than five minutes of cumulative downtime over a 12-month period. If the network were to fail, it must recover so quickly that the computers on both sides of the network would not know the network had failed and there would be no interruption of services.
BROADCAST STORMS
The possibility of failure using the network for PACS was clearly illustrated to the IT staff after a visit to a nearby hospital that recently installed a new PACS. In the first three months of operation, the PACS network suffered two so-called "broadcast storms." Each time the PACS network was down for six hours, leaving the hospital staff without radiology images for diagnosis. The hospital had been built to withstand an 8.5 scale earthquake, but a data network broadcast storm could bring down the entire PACS and leave doctors without access to diagnostic information. This PACS network was based on Layer 2 switches.
A broadcast storm occurs when a faulty computer talks too much. The bad computer transmits particles or packet fragments. Fragments are not legal packets; they have no source or destination address. Since there is no destination, the would-be destination never acknowledges receipt of the message. Without that acknowledgement, the source simply continues its attempts to transmit again, sending more fragments, filling the network with particles and generating a dust storm in the local network segment. This is called a jabbering node. Other computers become quiet, waiting for a break in communications before they can talk.
In a broadcast storm, the network can grind to a halt when the system is flooded with incomplete messages from damaged or intermittent computer network interfaces. The Ethernet media access uses carrier sense multiple access protocol (CSMA). Multiple computers access shared media, and each computer listens to see if other computers are talking (also known as carrier sense). If another computer is talking, the first computer listens passively. If no other computer is talking, the computer transmits its message.
LAYER TECHNOLOGY
Layer 2 networks use forwarding bridges. A bridge sees a data fragment and doesn't find the destination address, so it forwards the fragments to other network segments. Soon the broadcast storm propagates everywhere, so that no computers in the hospital can talk. There is no way to know the source or node, so segments of the network must be shut down until the bad computer is isolated. This search can require hours of system downtime.
In a Layer 3 network, with added intelligence to process switching and routing, the core uses an IP router and the edge of the network uses IP routing switches. Each edge routing switch only forwards legal packets, with a source and destination address. The fragments from a jabbering node encounter the first routing switch port, and without a legal destination address, the router picks it up and does not forward packets, but discards the fragments. A jabbering node can only affect two nodes - itself and the router port - thus isolating it from the rest of the hospital network.
This is a wonderful concept, but typical routers use software to process destination addresses, so they are much slower than Layer 2 bridges for PACS data traffic. SACH opted for Extreme Networks Inc.'s Layer 3 switching product. The IP router process is embedded in silicon hardware, so the processing of destination addresses is much faster. Routers provide another important benefit. Routers also permit multiple paths from source computer to destination. If one path fails, another will take over and the good packets can reach the PACS destination successfully.
The routing switches permitted SACH to use Layer 3 routing and block broadcast storms and provide alternate paths from PACS source to destination.
THE RIGHT PATH
With assistance from Santa Clara, Calif.-based Extreme Networks, SACH staff installed a Layer 3 network infrastructure. The PACS network uses open shortest path first (OSPF) protocol to automatically route PACS information around any network failure. OSPF is smart enough to route traffic to another path so fast that when a switch fails, the computer doesn't even know part of the network has gone down. The recovery is almost as fast as two milliseconds.
In addition to added intelligence with Layer 3 capabilities, the network is designed to be fully redundant. In all, SACH has installed two redundant Extreme Black Diamond 6808 core switches, seven Alpine 3808 intermediate switches and 30 Summit 48 edge switches across three campuses and eight buildings, connecting 850 end stations.
The PACS has been operational for some 38 months since July 2000 with no downtime, and SACH has exceeded its goal of less than five minutes of downtime over 12 months.
That is not to say there have been no problems at all. Some components fail, but redundant systems carry the network safely. Doctors can depend on the PACS network to deliver information reliably, so they can do their jobs effectively.
To keep it that way, and manage the entire system with a small staff, the hospital installed management products, such as Live Health from Concord Communications Inc., of Marlboro, Mass., and Extreme Networks' Epicenter software. Concord monitors every network segment 24x7, and forwards traps to Epicenter when part of the network slows down. Epicenter sends the staff an automatic page if something fails or slows down, notifying them of the changes. This is like having 20,000 pairs of eyes watching every segment of the network 24x7.
New wireless networks at SACH and mobile PACS workstations permit bedside patient data entry and provide portable x-ray previews in emergency and surgery departments through wireless networks.
Thanks to OSPF, SACH can use Layer 3 networks that route PACS images around network failures. And, thanks to new technology, OSPF routing switches are fast enough to route packets to scale with the demands of large PACS image files.
Based on broadband access and reliability of the network, SACH is in the enviable position to implement voice over IP wireless data networks for doctors and staff, saving time and cost.
Credit for SACH's success also must go to Irv Hoff, SACH's manager of converged networks and information services, for his support throughout this process.
Jan Snyder is Senior Telecommunications Consultant at San Antonio Community Hospital, Upland, California