By deploying high availability (HA) technology, you can shrink disaster-induced system downtime to 20 minutes or less. You can also deliver uninterrupted system access while maintenance activities are being performed.
HA systems can sometimes appear inordinately abstract, though, especially when the chips are down and failover procedures are not working as they should. Printed HA documentation can take the mystery out of a system failure and is indispensable to anyone who wants to maintain control.
Good HA documentation has two parts. Both parts share the same objective, which is to help administrators and others avoid getting caught in the slipstream of confusion and quickly restore access to mission-critical business functionality.
The first part of an HA document clarifies procedural uncertainties and sheds light on HA priorities, government regulations, and recent changes to system settings, among other things that, if not duly noted, will only prolong the misery. And to ensure its accuracy and viability, it must undergo periodic review.
The second component is a log that details such things as occurrences of exceptional system behavior and the actions taken to correct these aberrations. Even the most automated and technologically advanced HA system relies on the success of its smallest components.
Document Number One: The HA Operations Piece
There are many ways to structure an HA operations document, but it should always contain the following elements.
Business Requirements
Include a couple of paragraphs on why your organization has HA in the first place. Are you using it to create a true HA environment or just a backup tool? This will help technicians in charge of recovery understand what is a priority and what is not. Most often, companies implement HA to stem financial losses associated with system inaccessibility, but the central mission of HA might more directly be to improve business continuity, ensure system availability, or protect data.
Performance Benchmarks
Your best shot at a speedy recovery is to have a backup system that has the same processor functionality, memory, and disk sizes as the primary system. Although HA systems can function competently in situations in which the primary and backup systems differ in performance, matching operating system release levels and application software programs can support a more complete recovery.
Also, performance of the entire HA system depends upon the communications link between the two systems. If you don't have good communications, you will not have high-performance HA.
That said, many organizations save money by deploying backup systems that have less horsepower. Sometimes, they'll use an old production system as their backup, and on other occasions, they'll acquire a system that has been specifically sized to handle a subset of the primary system's functionality. Most HA experts don't like this option and say it's a disaster in the making. In any case, the IT department should know what to expect in terms of performance when the user load on the backup machine reaches its mean and mode.
Performance and Service-Level Agreement Information
Performance agreements are now widely used to set acceptable levels of service for internal and external system users. The service-level agreements (SLAs) that exist within these performance agreements provide a definition of acceptable service, outlining performance metrics like system response time and uptime, ways to manage problems, methods of handling disasters, and more. If the terms of an SLA are ignored, penalties can be imposed or the agreement can be terminated. SLA compliance falls squarely on members of the IT department. These quantifiable parameters should be clearly posted in your HA operations document.
Regulatory Requirements
The body of law that addresses sustained high levels of readiness and system resiliency is growing. And, because questionable business practices that cost investors and employees billions of dollars slipped under the radar for years, protecting data from "unintentional" destruction is now mandated for publicly held companies. Specific industries are subject to other forms of legislation. Stiff penalties can be imposed for failure to comply with these regulations.
Succinct documentation on these regulations and how they specifically apply to HA can offer important guidance to those who manage or modify the system. None of this information will really matter in the dark moments after a crash, but once access to the system has been restored, measures can be taken to bring the system back into spec.
Failover and Failback Strategies
What if? What will we do in specific situations? How much coverage will we have? Volumes have been written on failover and failback strategies. Ultimately, specific circumstances should trigger fixed responses. You can choose not to failover (cold failover) after being offline for two hours at midday, but your users and business will suffer. A hot failover strategy would have system users logged back on in minutes—but at a higher price. Any policies pertaining to how technicians are expected to react should be clearly stated, reviewed, and understood.
Changed System Values
There is a set of roughly 150 security-related system values in i5/OS. HA solutions reset some of these values upon installation, and some vendors offer a tool that documents these values both before and after they are changed. In the absence of such a tool, you can print a listing of these values. If you choose to print them, you should produce both before and after copies so that you can compare the values on these two lists if you find yourself in a situation in which you or other users cannot access the backup system.
Threats
Make a list of possible threats and estimate the likelihood of their occurrence and the impact they will have on business. By documenting the various types of threats that your production computing environment is most likely to face and an appropriate response for each one, you can limit vulnerability and speed up recovery.
Most threats can be placed in either of two categories: internal or external. Internal threats are posed by failing systems components and by employees. You can minimize your vulnerability to internal threats through sound system maintenance and tight security policies. External threats that involve loss of power or communications are relatively easy to circumvent with backup power supplies and contingency satellite uplinks. Other events like fires, floods, earthquakes, and tornadoes can be much more severe and represent a greater challenge for the IT department.
Sometimes, events occur that are so severe and widespread that recovery, while possible in some limited way, does not make sense. Reestablishing online order entry, SOA-based credit card transaction systems, and warehousing systems is futile if the warehouse and your entire inventory are in splinters. Understand and document what this limit is.
Network
Include documents on how different devices are connected to your network and how the network itself is configured. Do you have separate communications links for these backup systems at the site? Are both the primary and target systems on the same subnet, or are they on different subnets, each with its own IP addressing scheme? Know which IP addresses are hard-coded into programs. These cause a problem and are increasingly used in Web-based applications.
Be sure to also document how the individual user connections are established. Are they handled within a DNS server? Do they have hard-coded client access sessions with the IP addresses coded into the connection?
Hardware
Relative to hardware concerns, most companies use several different servers to satisfy their business computing requirements these days. In a failover situation, it's important to know which of these systems are duplicated at the backup site. Do you, for instance, have a backup fax server or Web server available to switch over to? Keep track of the systems that are available to you so you don't overlook something important and don't waste time trying to restore something you don't have backup assets to cover.
Software
The last thing you'll want to have to deal with in the midst of a disaster is a software license key issue. Make sure you have one license key for each copy of software that you have on the backup machine, and reference these keys in your operations document. Vendors are protective of their license keys and require help desk technicians to go through strict procedures to issue or reissue them. You don't want to have to deal with that when your production system is down and the only thing keeping users from signing onto the backup system is a software license key.
Also, know whom to contact, from your technical support representative to the salesperson who handles your account. The ideal situation is to have 24x7 software maintenance contracts with all of your vendors if you are a shop that works around the clock. Take a few notes on who will be available after normal working hours and who will not.
Database
A common practice used to simplify the rollover process in HA is to make the machine-specific local relational database names match on both the primary and backup machines. That way, it's transparent to the ODBC or JDBC drivers. If your primary box fails and you have to change the DNS entry on the network to redirect users to the backup machine, the common relational database name is automatically picked up. If different relational database names are used, you'll have to recompile or change every entry that points to the old relational database name. Make sure the common local database name is present in your HA operational documentation. If any confusion arises after the DNS entry is changed, you'll have the relational database names right in front of you.
Testing the HA Environment
You'll want to test your HA capabilities with a fair degree of regularity. You'll also want to test them after small problems occur and are resolved, because changes to the HA environment could adversely impact your ability to switch users to the backup system if a serious event occurs.
One quick way to test HA readiness is to initiate a role swap, cross your fingers, and see what happens. A test role swap verifies the viability of the communications link, the backup hardware, and the software applications. The test role swap only verifies basic functionality, though. You also need to evaluate or test the integrity of the data that resides on the backup machine. Is it up-to-date? Are all of the key relationships intact? During test role swaps, users typically stay on the backup machine for only a short time, not long enough to make sure that the data is 100% viable. For this, a small group of test users from different departments should be assigned the task of periodically conducting extended tests on the data.
Who Is Your HA Vendor?
If you're in a jam and can't seem to figure out what to do next, your HA vendor can help isolate problems. Chances are, the person you speak with will have worked through hundreds of failover conditions. Have contact names, emergency telephone numbers, and email contact information readily available.
Document Number Two: The HA System Log
In addition to the operations guide detailed above, another useful document that should be created, maintained, and frequently reviewed is a log used to record several types of important events that pertain to the HA environment.
Failover Events
Whenever the production system fails, it's important to log as much information as possible about why and when. If system variables change in the failover state and things don't match once recovery measures have been taken, you can go through your log, determine when the system failed, and restore these settings to the values that existed just before the failure.
Corrective Actions
Keeping track of who took corrective actions, what they did, and when these actions were taken is valuable for auditing purposes and might be required by law. Granted, when you are scrambling to restore sign-on screens, you're not taking thoughtful, introspective notes. You can't ask a firefighter to take incident notes when he has an axe in one hand and a hose in the other. When the smoke clears, though, it's important to make notes about every corrective action that was taken and what did and did not work.
Computing Environment Changes
It's good to have historical information on environmental variables handy. You want your backup environment to match your primary production environment. Most of this information is stored in proprietary IBM libraries and doesn't get replicated over to the backup system because it is release-dependent.
Exceptional Activity
Creations, deletions, and other security-related activity is logged in the QAUDIT journal. All you have to do is turn on auditing. If you have configured your system's security mechanisms appropriately, information on who changed what and when will be automatically logged.
Many HA solutions also run a scheduled audit and automatically log exceptional events that are specific to HA, like journals that are out of sync between the primary and backup machine, data that failed to transfer to the target machine, and intermittent communication failures.
The Devil Is in the Details
The more detailed your HA documentation is, the more helpful it will be. If the information is organized logically, anyone who is at the helm at the time of failure can use it.
Robert Gast writes for Vision Solutions, a leading developer and integrator of iSeries high availability and continuous availability solutions. Vision Solutions recently joined with iTera, Inc. to form the number one HA company in the world. Together, they specialize in the development of software solutions to increase and improve the availability of IBM System i data management systems and streamline data management processes.
More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.
TRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now! Request Now.
Forms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.
Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:
Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.
IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn:
LATEST COMMENTS
MC Press Online