07
Thu, Nov
10 New Articles

Falling Dominos: Planning a Backup Strategy

High Availability / Disaster Recovery
Typography
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

As a kid, I remember having Dominos around the house to play with. I don’t think I ever really “played” Dominos, but I would build things with them, and inevitably something would cause them to fall. As they came down, I knew before the last one fell exactly what I needed to do to rebuild them the next time.

Dominos isn’t just for kids anymore. In fact, Lotus Domino has become the buzz in the IBM AS/400 world, as it’s a great tool for business. To support Domino in the business world, administrators need a plan for rebuilding or recovering Domino’s components. In this article, I will discuss various mechanisms available to create an application backup strategy for Domino for AS/400.

Domino backup strategies should be defined to allow for recovering either parts of a Domino server, such as mail files or logs, or the entire server, in the event of a disaster. Domino is made up of programs and customization information that comprise the server, as well as databases that contain user data. On the AS/400, the programs and customization information reside either in libraries or in AS/400 Integrated File System (AS/400 IFS) directories, which contain more customization files, product information, and databases. Figure 1 contains a list of the libraries and AS/400 IFS directories that make up the Domino product.

When the server is initially created, it is important to create policies for safeguarding the location of files in the Domino environment. To have Domino files outside of the appropriate directories not only adds complexity to the backup strategy but also confuses the server. The Domino server understands from the set up what the name of the Domino data directory (where Domino user data files reside) will be. If, for example, /notes is the main directory for the server, this directory is available for the Domino server to work with. If a new file were created outside of this data directory, you could have security challenges and pointer problems within the server. Good policies for file placement are strongly recommended. If you are not sure about placement, Lotus reference materials, such as the one listed in the References and Related Materials section at the end of this article, or the Lotus Web site or support lines are good resources for answers.



Creating a Backup Strategy

Understanding what comprises Domino for AS/400 and where those library and AS/400 IFS directory components are located is the key to creating your backup strategy. When Domino is installed, you should immediately save a copy of the application (programs and customized files). Consider this your “golden” copy, at least until after the next release of Domino is loaded. The two main commands for saving Domino are the SAVLIB and SAV commands. Typically, Domino databases in the /notes/data directory change daily, and to capture those changes, the SAV command is used. Since Domino application library objects are not generally updated daily, the libraries can be saved less often using the SAVLIB command, with perhaps the only exception to this being the QUSRNOTES library. The Domino libraries and the AS/400 IFS *IBM directories will be captured during a full system save (GO SAVE Option 21), and should be saved individually prior to a Domino for AS/400 upgrade. Figure 2 (page 60) contains a list of the different methods for backing up Domino for AS/400.

Some AS/400 shops may have a very narrow window of opportunity for a daily save of AS/400 IFS directories, including Domino user directories. In this case, a partial or incremental backup should be considered. A partial backup may include saving only the mail and other critical databases on the server. Capturing only the mail files involves slightly modifying the backup string to include the mail directory and possibly the file extension:


SAV DEV('qsys.lib/tap03.devd') OBJ(('/notes/data/mail/*.nsf')

('/notes/data/mail.box')) SUBTREE(*ALL)


Saves all files in the /notes/data/mail directory which end with *.nsf


SAV DEV('/qsys.lib/tap03.devd') OBJ(('/notes/data))

SUBTREE(*ALL)


Saves all files in the /notes/data directory.

The MAIL.BOX file contains inbound mail, so if only saving mail files, it also may be included in the aforementioned backup string. Other examples of files to consider when taking this approach are shared mail files, discussion databases, or databases specific to certain applications. The obvious drawback to specifying individual files is that it introduces the possibility of missing or leaving out critical database files.

Another way to reduce backup time is by using incremental backups. The simpler of the two is the strategy of backing up any changes made since the last full backup. If, for example, you run a full backup on Sunday nights and all other nights you run an incremental backup, you need only restore the Sunday night backup and the latest incremental backups in the event of a crash. If you were to crash on a Thursday, you would restore Sunday night’s backup, then Wednesday night’s backup. With this strategy, backups grow larger throughout the week, but the recovery is fairly simple.



The following is an example of how to backup incremental changes since the last full backup and run each day between full saves:

SAV DEV('qsys.lib/tap03.devd') obj(('/notes/data'))

chgperiod(*LASTSAVE) subtree(*all)

If even this requires too much time, the incremental backup process can be taken a step further by only backing up changes since the last incremental backup, which requires fewer tapes and less time. The trade-off for saving time is recovery complexity: Now if you crash on a Thursday, you need to restore Sunday, then Monday, Tuesday, and Wednesday.

The following is an example of how to backup data from one incremental to the next. Notice that the start date on the change period parameter is the date after which objects that have been changed are to be saved.

SAV OBJ((‘/notes/data/*.*’)) CHGPERIOD(‘01/15/01’ 030000)

updhst(*yes)
SAV OBJ((‘/notes/data/*.*’)) CHGPERIOD(‘01/16/01’ 030000)

updhst(*yes)

This is run every night, updating the date and time each night until the next full Another consideration may be directory synchronization objects. Directory synchronization can be used to keep the Domino directory (also known as the name and address book) synchronized with the AS/400 System Distribution Directory (SDD). If using directory synchronization, consider backing up the AS/400 SDD files as well. These can be saved with the SAVOBJ command as follows:

SAVOBJ OBJ(QAOK*) LIB(QUSRSYS) OBJTYPE(*FILE) DEV(DEVNAME)

Remember that to prevent locks, directory synchronization should be stopped before backing up files. A call to the program supplied with Domino for AS/400 will take care of this for you. The call is run as follows: call QNOTESINT/QNNDIEND.

Timing Is Important

Now that you know what options are available, timing is of the essence. Most environments will require that the server be brought down to obtain clean Domino backups. There is an option for save-while-active in the SAV command; however, it is not bulletproof, and Domino generally doesn’t play nicely with it. The purpose of using the save-while-active function is to allow users to access Domino while it is being backed up; however, databases within Domino in use at the time of the save will be locked by the Domino application and therefore cannot be saved. And, no matter how many times you flush out the Domino server with the drop all and dbcache flush commands at the server console, the server is subject to users and tasks reinitiating their connections, creating more locks on files and objects.

Prior to ending the server before saves, there are maintenance jobs that run on the server that should be taken into consideration. Domino behavior is partially defined by a file in the user data directory called notes.ini. Some of the parameters in this file instruct the Domino server when to perform various types of maintenance. Once this maintenance is started, you should allow it to finish before bringing down the server for backups or other maintenance.

To look at this file, IBM initially provided a PTF to install the Edit File (EDTF) command. With later releases of Domino, this command was added as Option 13 on the Work with Domino Servers (WRKDOMSVR) command. In either case, you can look at



save.

the notes.ini file by issuing the following command from a command line: edtf ‘/notes/data/notes.ini’.

You are looking for statements similar to the following:

ServerTasksAt1=Catalog,Design,UpdAll
ServerTasksAt2=Object Collect mailobj.nsf
ServerTasksAt5=Statlog,Object Info -Full
ServerTasksAt6=Statlog

The lines above indicate that maintenance is being performed at 1:00 a.m., 2:00
a.m., 5:00 a.m., and 6:00 a.m. With this in mind, 3:00 a.m. might be a good time to schedule the server to come down for backups. In addition, you should time the backups to occur when the server has the least amount of user activity. You can manually check user activity by issuing the sh users command in the Domino console which will display users who are logged into the console and how many minutes since their last activity.

If agent managers are in use, you may want to ensure your backup doesn’t occur while a scheduled agent is running. To avoid such conflicts, you can use the Domino console command Tell amgr schedule to display all agents scheduled to run by the agent manager for the current day. Determine from the list of run times whether or not you will have any conflicts between the backups and the agent manager tasks.

If your environment dictates the highest availability possible, there are other alternatives. Clustering Domino servers lends itself nicely to a high-availability environment, allowing one server to be backed up while another services clients. Partitioning servers with replication also provides redundancy of data for backups. IBM’s Backup, Recovery and Media Services package is the first such application to support a true save-while-active backup, and Domino reportedly works well with it. If these options are out of your price range, backing up to save files on DASD is a last resort for saving time. Although the files must eventually be written to tape, write time to disk is generally faster than writing to tape; hence server downtime is lessened. Save files on disk can be written to tape at any time before the next backup. Before implementing this, just make sure the DASD for the save file is available.

Backup Tapes

Now that you have looked at backup types, schedules, and ways to limit the impact of backups to users, plan for how long the company needs to keep those backup tapes. Backup tape and file retention is something often planned for when running nightly, weekly, monthly, and yearly saves. You are most likely aware of the legal issues involving the retention of financial, medical, and other data that may be required to be produced or reproduced at a later date. The retention of email data is at the company’s discretion; however, you should take time to consider the importance of the data contained in each user’s mail file. Could there be a need to reproduce the content of an email message or one of the many database files sometime in the future? Consider the importance of any email that contains, for example, contract or medical information. Do you need to keep this information longer? Careful planning may prevent restore headaches in the future.

Executing the Backup

Once everything is in place, it is time to execute the backup. Figure 3 (page 61) shows a sample CL program for shutting down and backing up a typical Domino for AS/400 system. (Note, however, that you likely will need to modify this program to accommodate your own installation.)

The sample CL program outlines the basic steps for attempting a controlled shutdown, backup of user directory data, and restart in an unattended mode. There are lengthy delays built in to ensure that the server comes down clean; if, however, all else



fails, the server is brought down hard with an *IMMED. This forces an end to the Domino server job, tasks, and subsystem, but it should be used only as a last resort. With the server down, a backup of the /notes/data directory is executed, the subsystem description deallocated, and the Domino server restarted. Remember, to be successful, your program should be tested and retested to ensure it functions as you expect it to. Different systems may require longer or shorter delays, or additional shutdown steps. The goal is to bring the server down clean, get a good backup, and restart the server—with a minimal impact to users.

Although the information in this article may seem like a lot, with a little planning, backing up Domino for AS/400 is very straightforward. As you can see, Domino is not yet at a point where you can backup and restore individual documents within Domino databases from the AS/400 command prompts. Eventually, Domino may get to this level; until then, you can take satisfaction in knowing that you have taken all of the necessary steps to protect and recover all of the pieces in the event that Domino falls.

REFERENCES AND RELATED MATERIALS

• Lotus Domino for AS/400: Installation, Customization, Administration (SG24-5181-00)
• Lotus Domino on iSeries home page: www.as400.ibm.com/domino
• Lotus Support home page: www.support.lotus.com/lshome.nsf

Libraries and AS/400 IFS Directories 5769LNT Optional Part Backup Category Description

QNOTES *BASE *IBM Domino for AS/400 Product Library

QNOTESINT 1 *IBM Directory Synchronization Library

QNOTESAPI 3 *IBM C APIs Library

QNOTESCPP 4 *IBM C++ APIs Library

QNOTESLSKT 5 *IBM LotusScript Extensions Library

QNOESHTST 6 *IBM HiTest APIs Library

QUSRNOTES *ALLUSR Customization Information Library

/QIBM/ProdData/Lotus/Notes *IBM Product Information Directory

/QIBM/UserData/Lotus/Notes *ALLUSR Customization Information Directory

/notes/data *ALLUSR Directory for Databases on the Server

(created during server setup and named by the administrator)

Figure 1: Components of Domino for AS/400 include both libraries and AS/400 IFS directories and files and contain product information, as well as customization information.



Backup Method Description Recommended Use

GO SAVE, Option 21

GO SAVE, Option 22

SAVLIB *NONSYS

SAVLIB *IBM

SAV OBJ((‘*’) (qsys.lib *OMIT) (qdls *OMIT))

SAV OBJ((‘/notes’) (‘/QIBM/UserData/Lotus’))

SAVOBJ((‘/notes’) (‘/QIBM/ProdData/Lotus’) (‘/QIBM/UserData/Lotus’)

Restricted-state save. Saves the entire system. Save all QNOTESXXX libraries, QUSRNOTES library, and all AS/400 IFS directories (*IBM and *ALLUSR). This is a long- running save.

Restricted- state source saves *IBM libraries and directories on the system, no user data. This includes all QNOTESXXX libraries, as well as the /QIBM/ProdData/Lotus/Notes directory. No user data is captured with this save.

Restricted-state save. By itself, this save captures all libraries on the system (*IBM and *ALLUSR libraries). For full recovery of Domino, a SAV of IFS or specific Domino directories needs to be done as well.

Saves all *IBM libraries. A backup of QUSRNOTES, as well as SAV of AS/400 IFS directories, also needs to be done for Domino recovery strategy.

Captures all AS/400 IFS directories, except QDLS folders and QSYS.LIB folder. All Domino directories are saved; no Domino libraries are saved.

Weekly, monthly or quarterly, or before any major software application upgrades, hardware upgrades, hardware moves, or extended system outages.

Periodically as part of a full system save, or daily as stand-alone AS/400 IFS backup. This backup captures all AS/400 IFS directory and file data, except QDLS and QSYS.LIB information.

Captures Domino
user
AS/400 IFS directories. As part of a daily backup strategy for the Notes environment, this backup captures Domino user data.

Captures all Domino-related AS/400 IFS directories.

Before a Domino upgrade, in addition to a SAVLICPGM 5769LNT and a SAVLIB QUSRNOTES, this captures the IBM Domino directories, as well as user data.

Figure 2: Depending on the type of information to be saved, there are a number of backup options available.


Either before applying system PTFs or as part of a regular backup strategy, in addition to an *ALLUSR or a SAVLIB strategy.

Weekly, monthly, or quarterly as part of a full system save strategy. Use in conjunction with SAVSYS, SAV, and SAVDLO. Before any major application upgrades, hardware upgrades, hardware moves, or extended system outages.

Either before applying system PTFs or as part of a regular backup strategy, in addition to an *ALLUSR or a SAVLIB strategy.


5769PW1 V4R4M0 990521 SEU SOURCE LISTING 01/22/01 10:39:45
PAGE 1

SOURCE FILE . . . . . . . JULIE/QCLSRC

MEMBER . . . . . . . . . ENDDOM

SEQNBR*...+... 1 ...+... 2 ...+... 3 ...+... 4 ...+... 5 ...+... 6 ...+... 7 ...+... 8 ...+... 9 ...+...
0

100 PGM

200 CALL PGM(QNOTESINT/QNNDIEND) /* END Directory Synchronization */

300 MONMSG MSGID(CPF0000)

400

500 DOMINOCMD: SBMDOMCMD CMD('drop all') SERVER(DOMSVR1)

600 SBMDOMCMD CMD('dbcache flush') SERVER(DOMSVR1)

700

800 ENDDOMSVR: ENDDOMSVR SERVER(DOMSVR1) OPTION(*CNTRLD)

900 MONMSG MSGID(CPF0000)

1000 DLYJOB DLY(300)

1100 ENDSBS SBS(DOMINO01) OPTION(*CNTRLD)

1200 ALCSBS: ALCOBJ OBJ((QUSRNOTES/DOMINO01 *SBSD EXCL)) WAIT(300) /Check to see if the
Domino Subsystem is down */ 1300 MONMSG MSGID(CPF1002) EXEC(DO)

1400 ENDDOMSVR SERVER(HO3DM3) OPTION(*IMMED) /*END DOMSVR AND SBS */

1500 DLYJOB DLY(120) /*WAIT FOR FINISH */

1600 ENDDO

1700 DOMSAV: SAV DEV('qsys.lib/tap.devd') OBJ(('/notes/data')) +

1800 SUBTREE(*ALL) SEQNBR(1) ENDOPT(*LEAVE) INFTYPE(*ALL) UPDHST(*YES) +

1900 CLEAR(*ALL) /*SAVE NOTES DATA DIRECTORY AND ALL SUBDIRECTORIES */

2000 DLCOBJ OBJ((QUSRNOTES/DOMINO01 *SBSD *EXCL))

2100

2200

2300 STRDOM: STRDOMSVR SERVER(DOMSVR1)

2400 ENDPGM

* * * * E N D O F S O U R C E * * * *

Figure 3: Automate the nightly task of shutting down and backing up Domino for AS/400 with a simple CL program.



BLOG COMMENTS POWERED BY DISQUS

LATEST COMMENTS

Support MC Press Online

$

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: