23
Tue, Apr
1 New Articles

Delivering High Availability

High Availability / Disaster Recovery
Typography
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

Globalization, the Internet, competitive pressures, and higher consumer expectations have significantly increased the need for dependable 24x7 information systems in businesses, as well as in not-for-profit and government organizations. Fortunately, a number of products, from a variety of vendors, provide the means to improve the availability of data and applications, but these products alone are not enough.

Information technology is complex. The failure of any one of thousands of hardware and software components or the failure of the link between those components may be sufficient to shut down an application—or possibly all of an organization's systems. Thus, ensuring that none of the pieces in the high availability puzzle are missed is a demanding task that requires thorough analysis, rigorous planning, effective education, and expert execution of the plans.

The preceding paragraph refers to only hardware and software failures, but there are many more potential causes of data and application downtime. The mistake that organizations often make when considering their high availability requirements is to equate reliability and availability. If the two were equivalent, organizations would not need to do much to ensure reasonably high availability. It would occur naturally. Reliability refers to the mean time between failure (MTBF) of a system or a piece of hardware or software. Because information technology, particularly hardware, is now so reliable, only organizations with the severest of availability requirements are likely to spend significant time worrying about its failure.

Unfortunately, reliability and availability are not synonyms. Hardware and software failure are only two of many causes of downtime, and they're far from the most serious. Disasters such as fires, floods, hurricanes, earthquakes, ice storms, and lightning strikes can also disrupt operations for hours or even days at a time, yet these too are infrequent events. Nevertheless, there is one cause of downtime that is frequent and unavoidable—planned maintenance. On a regular basis, organizations shut systems down to perform hardware and software upgrades, backup and reorganize databases, or undertake other vital maintenance. In the past, when systems ran only during "business hours," this wasn't a problem. Maintenance, along with the running of batch jobs, was undertaken during those off-hours. Today, the increased need for 24x7 operations has eliminated those "batch and maintenance windows" and, as a result, has sent organizations scrambling to find ways to perform necessary maintenance and run batch jobs without shutting down online operations.

Recognizing these differences between reliability and availability is the first step on the road to achieving higher availability. But avoid the urge to rush out and buy something labeled as high availability hardware or software immediately upon coming to that realization. Fulfilling your availability objectives requires considerably more work than that, including analyzing the requirements and designing, planning, implementing, validating, sustaining, and managing the solution.

Analysis

Availability requirements are not all created equal. They vary both in the organization's business needs and in the specifics of the technologies that must be protected.

Different companies have different business demands. A banking application that runs Internet-based services may process transactions that can each represent thousands or even millions of dollars—transactions that originate electronically and that, at least initially, have no reliable hardcopy backup. This sort of system must be available around the clock, and there is no margin for error when it comes to protecting the integrity of the data it processes. In contrast, a system that a single-shift manufacturer uses to record orders originally received on paper has far less stringent availability requirements.

Availability needs differ not only in magnitude, but also in their nature. Organizations must consider two generic types of requirements: data integrity and application availability. For some organizations, particularly those that process high-value transactions entirely electronically, data integrity is absolutely critical. They cannot afford to lose a single transaction or allow a single piece of data to become irrecoverably corrupted for any reason. For other companies, such as those that process low-value transactions with accompanying paper trails, some data loss may be acceptable (as long as the loss can be detected) because the errant data can be recreated.

There's also a range of application availability requirements. For some applications, such as those that support high transaction volumes for around-the-clock, global operations or Web-based retail sites, any downtime, whether planned or unplanned, is costly. For other applications, such as low-value, back-office administrative systems, off-hours downtime is not an issue, and even the occasional hour of downtime during business hours will not have a significant impact on the organization.

Recognizing these differences is important because investments in higher availability follow a law of diminishing returns. For an organization that, to date, has done nothing to address availability, the cost of eliminating the first hour of downtime is orders of magnitude less than the cost of eliminating the last hour of downtime. Likewise, the cost to protect against the loss of even a single transaction in the event of a disaster is typically somewhat higher than the availability costs that you will have to incur if you are willing to accept a few lost transactions in those rare circumstances. Almost all organizations can justify taking the most basic steps toward enhanced availability. On the other hand, only organizations that place a particularly high value on each hour of operations will achieve a return on the investment required to almost totally eliminate downtime and lost data.

Applying this requirements analysis at the enterprise level will generally lead to suboptimal returns on your availability investments. Consider a major online retailer that processes millions of dollars of orders an hour and that operates in a very competitive environment. Eliminating as little as one hour of Web site downtime per year might generate a net benefit of hundreds of thousands or possibly more than one million dollars. The company would, therefore, be justified in spending a considerable sum to protect the systems that support its online retail operations. On the other hand, the unavailability of its human resources system for an hour—particularly in the middle of the night or on a weekend—would result in little or no financial loss. If the company made the same investment in availability for all of its systems, it would either spend too much to protect its human resources application or too little to protect its Web retailing application. Thus, it's important to perform the analysis at a level of detail such that the availability requirements are consistent within the unit under investigation.

Don't restrict the analysis to solely business needs. Also inventory your information technology and the interconnections between individual pieces of hardware and software. Consider which applications will be affected by the unavailability of any of those technologies. At the same time, document hardware and software maintenance schedules and indicate how each type of maintenance will affect application and data availability. This information will be important when it comes to selecting availability products because you must be certain that they will support your hardware, software, and existing processes.

Design

The design phase of a high availability project uses the results of the planning phase to sketch out a blueprint for an architecture that will fulfill your objectives. Because this blueprint must be shaped by what is possible given today's technologies, at this stage you will also have to evaluate the availability products on offer by the various vendors and map them against both your hardware and software environment and your availability objectives. Because this evaluation must be completed as part of the design activities, the final selection of the products that will be used is typically made in this phase.

Planning

The result of the planning phase is a detailed map of how you will get from your current level of availability to the one identified in the analysis phase and scoped out in the design phase. The planning phase then draws on this and follows a rigorous project management methodology to produce a detailed implementation plan that describes the following:

  • Implementation objectives—Restate the objectives that were established in the analysis phase. These represent your desired availability endpoints. The success or failure of the implementation phase (or, more likely, phases) will be measured against these objectives.
  • Implementation approach—Identify the high availability products that will be employed and determine whether they will be installed and implemented using internal resources, external resources, or a combination of the two. If the availability solution will be implemented in multiple phases, map out those phases such that, wherever possible, the greatest benefits will be generated earliest.
  • Key implementation assumptions—Spell out any assumptions that are critical to the success of the project, such as the availability of key resources or technologies.
  • Project organization, with specific roles, responsibilities, and activities—Use your normal project management methodology to define all aspects of the high availability implementation project.
  • Key dependencies—Identify any tasks that are dependent on the completion of other tasks.
  • Project schedule, with key milestones—Set out a timeline for all activities with milestones against which the project schedule can be tested throughout the project, allowing corrective action to be taken as required should the schedule begin to slip.
  • Education requirements—Implementing and then managing a high availability infrastructure inevitably requires skills that didn't previously exist in the organization. Identify those skills and develop an education plan that will successfully build them.

 

Implementation

If your implementation phase is completed successfully, at the end you will have a high availability solution that protects your data and applications to the level specified in the analysis phase. Getting to that point involves installing the new availability technologies, configuring the software, integrating the administrative processes required by the new technologies into your other business and IT processes, and then testing the solution to ensure that it has been installed correctly and will perform according to the specifications.

During this phase, you should develop an Availability Operations Handbook that describes the ongoing availability management processes. It may not be necessary to develop the handbook from scratch. Instead, you may be able to start with a handbook provided by your high availability vendor and customize it for your environment. The finished handbook should include the following:

  • Reference information such as contacts and important telephone numbers, along with an inventory of the hardware and software protected by the high availability infrastructure
  • A high-level diagram of the computing environment and the data integration topology
  • Basic instructions for backing up data, starting and stopping the availability products, swapping the roles of the primary and backup systems, upgrading products, and applying fixes
  • An operations schedule of daily, weekly and monthly activities, along with any other activities triggered by specific events
  • An explanation of how problems will be supported internally and/or through the vendor, including a description of the escalation procedure to follow if a problem is not resolved in a timely fashion

The Availability Operations Manual should also include sections used to log the following on an ongoing basis:

  • Issues that arise and how they were resolved
  • Completed maintenance tasks

Education

During the education phase, classroom, onsite, and/or online training gives employees the skills they need to manage, administer, and maintain the implemented solution on a day-to-day basis. Where appropriate, the vendor's off-the-shelf education offerings should be customized to address issues specific to your organization.

Effective teaching is a difficult skill to master, but it is essential if the education programs are to achieve the desired results. Therefore, a vendor's training capabilities, including the experience of their instructors and the quality of their courses, should be one of the factors you consider when selecting a high availability vendor.

Validation

What works in theory doesn't necessarily succeed in practice. After implementation, the full high availability solution must be validated against the stated objectives. That means more than just checking that data is being replicated to a backup site or that you can successfully switch users from a primary server to a backup—although those are certainly components of the validation procedure. In addition to ensuring that the technology works as specified, you must also validate that your employees have the necessary skills and that the defined processes are being executed properly and are having the desired effects.

During the validation phase, revisit the Availability Operations Handbook and verify that it accurately and completely documents the necessary processes. Test employees' reactions to a simulated disaster to ensure that they can fully recover operations within the specified time constraints. Run diagnostic tests to ensure that backup data accurately reflects production data. And test the switchover from the primary server to the backup to ensure that process works as expected.

As you perform all of these validation processes, document lessons learned from anything that goes wrong.

If you use external resources, such as professional services from your availability vendor, to perform this and/or the preceding phases of the availability project, the service provider will turn the new availability infrastructure and responsibility for its ongoing maintenance over to you at the end of the validation phase.

Sustenance

Your computing environment will change over time. Hardware, applications, and systems software will be upgraded from time to time. New data fields will be added to databases. Security processes, hardware, and software will change. The geographic dispersal of your information technology may also change. Any of these alterations may affect your high availability infrastructure and processes. Therefore, it's essential to upgrade, modify, and, most importantly, test your high availability solution on a regular basis to ensure that it continues to provide an adequate level of data and application availability.

In an evolving environment, there's a tendency to focus on changes in the technologies while ignoring the relevant processes. That's a mistake. On a regular basis, review the Availability Operations Handbook to ensure that the procedures it documents are still appropriate in the current technology environment.

Management

It's not just the technology environment that changes over time; the business environment also changes. Evolving customer demands, the adoption of new lines of business, increasing competitive pressures, and other factors will inevitably alter your organization's availability requirements. Furthermore, high availability is not an all-or-nothing proposition. Your first availability project will likely not tackle all of your enterprise-wide needs. Thus, there will be an ongoing need to undertake new projects to improve availability. Consequently, the phased availability planning and implementation methodology described above will need to be repeated and managed on an ongoing basis.

Availability Doesn't Just Happen

If there is one lesson that should be taken from the preceding discussion, it is that availability doesn't just happen. You can't simply buy a product, plug it into your information technology architecture, and forget about it. Computing environments today are very complex, with thousands of points of failure and myriad maintenance requirements. Except in the very simplest of architectures, a single, unmanaged off-the-shelf product will not prevent all potential downtime.

Business environments are no less complex. Availability needs vary among organizations and even among departments and applications within a single organization. Consequently, to derive the highest possible returns from each availability investment, availability must be planned and managed in a way that takes into account these differing requirements.

Because of this technology and business environment complexity, protecting the availability of data and applications requires a comprehensive and rigorous methodology that includes meticulous analysis and planning, expert implementation, and thorough ongoing management.

Alan Arnold is President and COO of Vision Solutions. Prior to joining Vision in 2000, he was a senior technology executive and subject matter expert for IBM technology at Cap Gemini Ernst & Young U.S. LLC. Arnold is recognized as an expert in the field of managed availability technology. He has authored or co-authored five books on technology and business topics that have been published worldwide. He has also written numerous articles for some of the leading publications in the industry.

BLOG COMMENTS POWERED BY DISQUS

LATEST COMMENTS

Support MC Press Online

$0.00 Raised:
$

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: