MDinTouch Prevention & Recovery Plan

The first step towards assuring availability of our service is prevention. The MDinTouch service has been designed and implemented with many features to prevent system downtime resulting from certain types of failures. As such, the service is highly fault-tolerant allowing MDinTouch to recover from component failure. These types of measures, described below, comprise our Prevention Plan.

Our Recovery Plan addresses those failures that cannot be prevented through the design of the service. The events that would necessitate activation of the Recovery Plan are very severe in nature; such as a natural disaster. They are highly unlikely but possible; and as such, require a Recovery Plan.

The Prevention Plan implements redundancy within the service at the Primary Data Center, whereas the Recovery Plan primarily allows for a Secondary Data Center to take over the hosting of the service. There are five essential categories of the service and its environment that absolutely must be in place in order to run the MDinTouch service, both at the Primary Data Center as well as at the Secondary Data Center:

Power
Internet
Computing hardware
MDinTouch software
Data

The MDinTouch Prevention Plan provides redundancy across these five categories within the Primary Data Center. The MDinTouch Recovery Plan assures appropriate availability of these categories at a Secondary Data Center.

Prevention Plan

The Prevention Plan is our design and implementation effort to completely shield our clients from certain types of failure (where a normal purchased product would most likely not have the capability to actively recover as it is very expensive to implement).

The following aspects of the service, or running the service, have been designed and implemented explicitly according to the Preventive Plan:

Power

Every computer or hardware device in the Primary Data Center is protected by an Uninterruptible Power Supply (UPS), which provides continuous power to the MDinTouch service should commercial power become unavailable for a short period of time.
Every computer or hardware device in the Primary Data Center is provided power for up to 7 days through a diesel generator, permanently located at the Primary Data Center, should commercial power be unavailable for an extended period of time.

Networking

MDinTouch utilizes multiple, redundant Internet providers simultaneously with automatic fail over to eliminate or at least minimize the effect of network failures to users.

Hardware

Server hardware is configured with redundant power supplies, redundant network cards, and redundant hard disks in RAID sets. In most cases, multiple hardware failures will present no effect to users.

Software

Application software configurations use the following techniques to support high availability:

Active-Active mode - multiple physical servers service requests under normal operation. Connections to a failed server will be moved to a properly operating server in the event of a failure.
Active-Passive mode - one logical server (which may include more than one physical server) services all user requests. In the event of a failure, the passive standby is promoted to active mode.

Server Clusters

All servers, with which MDinTouch client software communicates with, are configured in Active-Active Mode. The software executed on these servers is self-disabling should it become unhealthy. The MDinTouch client software will transparently locate a healthy cluster node (e.g., server) in the event of a failure.

Fail Over Servers

Oracle Corporation, the world's largest database company, provides the database technology utilized by MDinTouch. The Oracle technology is configured in Active-Passive Mode where the passive database standby service is updated in real-time along with the active database server. In the event of a failure of the active database server, the passive database service is immediately promoted to active mode.

Data Backup procedures

In addition to the patient health information and other system configuration data being protected in the Active Recovery Plan, all data is also fully backed up on digital tapes once per day at night.

Recovery Plan

The Recovery Plan is our solution for assuring operation of the service should a major disruptive event take place that rendered the Primary Data Center non-functional.

Recovery Situations

There are two basic scenarios that would call for a Recovery Plan situation:

Multi-component failures - In this scenario several components have failed concurrently. For example, the Primary Database server and the live Secondary Database server have both failed, or all redundant Internet connectivity has failed.
Loss of Primary Data Center - The threat of a complete loss of the Primary Data Center would most likely result from either physical damage or unavailability of commercial power for an extended period of time.

For any event resulting in power failure or physical damage to the Primary Data Center, the Recovery Plan calls for the Secondary Data Center to be activated. The Secondary Data Center is a continuously live computing environment that is in daily operational use by MDinTouch as a staging environment. Before a new service or feature is brought live, the new software version is staged for a period of time to assure that it does not introduce any flaws. In a situation where the Primary Data Center becomes unavailable, or has the potential to become unavailable, the Secondary Data Center is prepared to assume the role of being the hosting site for the service.

In conclusion, the MDinTouch service is extremely functional and comprehensive. The core functions of the service run on dedicated computer clusters, spanning many computers.

In order to recover from a failure that has engaged the Recovery Plan, MDinTouch has established several operational policies and documents that explicitly state how to configure and startup every aspect of the MDinTouch service to the last detail. In addition, MDinTouch has established relationships with several commercial data centers that could act as alternate Secondary Data Centers, or become a new Primary Data Center, should it be impossible to re-establish the current Primary Data Center.