03
Sun, Mar
0 New Articles

Hierarchical Storage Management-- Better Access to Your AS/400 Data

Typography
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

It’s the end of the week, and, just as you’re getting things cleaned up in preparation for the weekend, a coworker comes into your office and explains that she must have immediate access to a file that was saved to tape and deleted. After looking through copious save logs, you identify the tape containing the file, travel to the building containing the off-site backups, retrieve the desired tape, return to your machine room, and, finally, restore the file. Congratulations! You’ve just experienced the most basic example of hierarchical storage management (HSM).

In this article, I will discuss the concept of HSM and some of the available commercial products that implement it. I’ll also be discussing how you can use the tools that are available with OS/400 to implement your own HSM solution, including a way for your programs to access data stored offline without any modifications.

Storage on any computer comes in various layers. The fastest storage—in the registers of the CPU itself—is extremely expensive on a per-byte basis. The next step down in speed is cache memory, followed by RAM, fast hard disks, compressed hard disks, and, last but not least, removable forms of storage, such as optical, tape, and floppy disks. The price per byte of storage decreases as the access time increases. Thus, there is a huge difference between the cost of storing one byte in a CPU register and the cost of storing one byte on a tape, just as there is a huge difference in the time it takes to access those bytes.

In a perfect world, every computer would contain enough RAM to house every datum and program that would ever be needed. The speed of this fantasy computer would be incredible. In an imperfect world, the computer would have only enough disk and memory to process one job at a time. Such was the situation in the early days of computing when programs and data were stored on punched cards or the first floppy disks. Today’s IT manager is in charge of one or more computers that fall somewhere between these two extremes. For the remainder of this discussion, I’ll assume that the AS/400s you’re responsible for have a sufficient quantity of memory to address any performance problem due to a lack of memory.

Hierarchical storage management has been part of managing a computer system since the very first computer was brought online. The premise is quite simple: Maximize


system transaction throughput by storing the most frequently required data on the fastest (most expensive) devices, while storing less frequently accessed data on the slowest (cheapest) devices. The quantity of storage media is based, as always, on budgetary constraints and access requirements.

Planning for maximum transaction throughput includes not only the memory and disk requirements for the computer system but also the time of the animated overhead (the people) that service or use the computer.

What’s That Data Worth?

The reality-based anecdote that appeared at the beginning of this article demonstrated a large time-based cost for the retrieval of a file. Another way to look at this is from the viewpoint of a datum’s value. An open accounts-receivable entry is typically more valuable than a closed accounts-receivable entry, so I’m willing to store it on pricier disk real estate. As that entry moves from open accounts receivable to closed accounts receivable, from current month to previous month, from current quarter to previous quarter, from current fiscal year to previous fiscal year, and from pre-accounting audit to post-accounting audit, its value to the corporation decreases. As the immediacy of the data decreases, so does its value, which means it can be staged through its life cycle to increasingly slower and cheaper devices. HSM is all about deciding when the data should be devalued and ensuring that it remains at the appropriate level of accessibility throughout its life.

There are many options available for staging the data on a large AS/400 configuration. One of these typically would have the requisite system auxiliary storage pool (ASP), plus a number of user ASPs, and possibly a tape or optical library.

As an example, the aforementioned open accounts-receivable entry would be created on the system ASP, presumably on the fastest drives available. The entry would remain there until the account is paid. Once the debt is satisfied and the entry only of historical value, it could then be moved to a user ASP, perhaps on a compressed drive where access speed is traded for denser storage capabilities. As time progresses, the entry might migrate to a tape in a tape library or onto an optical medium in an optical library. At some point, it would find its way into the corporate archives or into the “bit bucket.”

HSM Products

IBM has several software products that will help you create an HSM solution, such as ADSTAR Distributed Storage Manager (ADSM) for the AS/400, a client/server solution that permits mixed clients to store data on one or more servers. For instance, one AS/400 could serve as a repository for backups of PC clients and other AS/400s. ADSM client software is available for all the popular PC operating systems, including the Windows variants, OS/2, and Linux. ADSM can be set up so that clients send their backup data to the AS/400 at their convenience, or they can be polled by the AS/400 when it is ready to accept their data. Once the data has made it to the AS/400, there are a number of options available, such as how many versions to save, how long to keep each version, and when to move the data to tape.

Another IBM software product that adds to the HSM toolkit is Backup Recovery and Media Services (BRMS) for AS/400. This program is a sophisticated tape librarian capable of automatically retrieving archived objects—that is, objects saved in a tape library with their storage freed.

Of course, there are many more storage and retrieval vendors out there besides IBM, each with its own unique products. For example, LXI Corp. has a product called Media Management System (MMS), which is a tape management storage and retrieval system for the AS/400. It also has a relatively newer product, Tape Management System (TMS/ix), which runs across both AS/400 and UNIX boxes simultaneously. There are many other vendors who provide similar services and products. Check the Online Yellow Pages on the Midrange Computing Web site at


www.midrangecomputing.com/yellowpages for more vendors providing competing products and services.

Build Your Own Solution

Even if you don’t have an AS/400 large enough to have tape libraries and user ASPs or a budget large enough to purchase all of the software mentioned, you can still build your own HSM solution. Don’t believe it? Then consider that the AS/400 already implements HSM. Let me give you an example. We, as programmers, do not need to concern ourselves with where OS/400 is storing our objects across the various storage layers. When we call a program, regardless of where it is located on the various storage layers, it gets transparently loaded into real memory and is then executed. The first call loads the program from disk into memory.

Subsequent calls may find the program already available in main memory or, perhaps, still in cache. We, as programmers, need only intercede when a requested object no longer exists online.

IBM has provided APIs to make the creation of a custom-built HSM solution possible. The APIs are Move Library to ASP (QHSMMOVL), Move Root Folder to ASP (QHSMMOVF), Move Spooled File (QSPMOVSP), and Save *STMF with Storage Free (Qp0lSaveStgFree).

The first two APIs, QHSMMOVL and QHSMMOVF, do exactly what their titles suggest: They move either a library or a root folder, respectively, from one auxiliary storage pool into another. Before performing the actual move, they ensure that sufficient space exists in the target ASP to permit a successful transfer. Neither of these APIs, however, are useful if there are no user ASPs available on your AS/400.

The third API, QSPMOVSP, moves spool files from one output queue to either the top of another output queue or after another spool file in the same queue. Should the target queue exist in a different ASP than that of the source, a test for adequate space is performed. The move is accomplished only if it passes the test, just as the first two APIs do. There are some restrictions to the use of the various APIs. It is not possible to move system libraries (those beginning with the letter Q) or libraries that contain objects that are not permitted in a user ASP or objects that can’t be renamed. The QSYSWRK subsystem must be active, since it handles migration jobs, and the objects to be moved may not be allocated by another job. In short, any limitation that normally applies to the location and use of an object is still enforced even when using one of these APIs.

Some of the functionality of these HSM APIs already can be found in CL commands. For example, the functionality of the QSPMOVSP API is available in the Change Spool File Attributes (CHGSPLFA) command. Figure 1 is a program that demonstrates the use of the QSPMOVSP API to move a spool file from one queue to another. You could just as easily use the CHGSPLFA command to perform the same task. But the APIs provide more flexibility since they can be called directly from a high-level language. The ability to save a stream file object with freed storage is not available in the SAV command as it is with the Qp0lSaveStgFree API, so, in this case, the API has the clear advantage.

Weighing Your Options

If you plan to work with these or any of the other OS/400 APIs, be sure to load licensed product 5769SS1 Option 13 of OS/400: System Openness Includes. You can find this on the OS/400 installation media and load it through the licensed program menu (GO LICPGM). Option 13 loads the library QSYSINC onto the AS/400. This library includes source files containing members you can copy into your programs. In turn, these source files contain predefined data structures that make accessing the APIs much easier. For example, member QUSEC in file QSYSINC/QRPGLESRC provides the ILE/RPG common data structure for the error code parameter used by many of the APIs. The


comments included in the members provides some good information about the APIs, so they’re worth studying.

Depending upon your current AS/400 configuration, your reaction to these APIs could fall anywhere between boredom and excitement. If you have one or more user ASPs, you can write software that takes advantage of the APIs by moving static information from the system ASP to a user ASP. This user ASP would need to be backed up only whenever additional information is added to it. Thus, you can easily improve your backup times with just a little creativity.

Archiving Solution

Even if you find yourself uninspired by the APIs mentioned in this article, there exists one exciting HSM opportunity available for AS/400s of all sizes. To take advantage of it, you need to load 5769SS1 Option 18 of OS/400: Media and Storage Extensions (MSE). This option provides a user exit program and OS/400 exit points that permit a user program to be called whenever an archived version of an object needs to be restored to the system to satisfy a system request.

An object is considered to be archived when you have saved it to an offline medium, such as tape, and have specified that the storage for the original object should be freed. This causes the object to be saved and then the contents of it to be deleted from the system. The description of the object, including the save information (such as the device type and volume ID of the save medium) remains on the system. Without MSE, whenever access to an archived object is attempted, OS/400 returns a variety of error messages to the caller that all amount to the same thing: The object is inaccessible. With MSE and one of your own programs registered to an MSE exit point, you can intercede. The program can then do something as simple as prompting the user to insert the appropriate tape, restore the file, and exit. OS/400 then attempts to access the object and finds it completely intact. Meanwhile, the program that attempted to access the archived object will never know what happened. Instead of having to code special logic into each program to ensure object accessibility, the function can be pushed down to the operating system level. This is akin to using database triggers to enforce business rules.

Option 18: Media and Storage Extensions is not free. It costs $995 regardless of machine size. It is provided on the OS/400 installation media and can be installed through the licensed program menu (GO LICPGM). As with all IBM V4R4 software, you can “try it before you buy it” for 70 days, so you do have some time to play before you pay.

Design Time

To really take advantage of HSM, you need to design it into your applications. The APIs discussed in this article work on entire libraries or root directory trees. Saving objects, such as files, causes them to be saved as a complete entity. So, in both cases, you may not have the granularity you need for HSM. By redesigning your systems and applications so that you can split things into more logical units, perhaps by their relative “value,” you’ll be able to keep your system throughput and human throughput optimized.

References and Related Materials

• Complementing AS/400 Storage Management Using Hierarchical Storage Management APIs, Redbook (SG24-4450-01)


* TO COMPILE:

*

* CRTBNDRPG PGM(xxx/MOVSPLFR) SRCFILE(xxx/QRPGLESRC)

*

*

** MOVSPLFR - This program demonstrates the QSPMOVSP and QHSNDPM APIs

*

* References: System API Reference Manual SC41-5801

* file: QSYSINC/QRPGLESRC,QUSEC

* file: QSYSINC/QRPGLESRC,QSPMOVSP

*

** MODIFICATION HISTORY

** 2000-04-24 Initial incarnation. Barry L. Kline

** Fields used as parameters to this program

D srcJobName s like( qspsjn01 )

D srcJobUser s like( qspsjun01 )

D srcJobNumber s like( qspsjnbr01 )

D srcSplFile s like( qspssfn )

D srcSplFileNbr s 5p 0

D targetQueue s like( qsptoqn )

D targetQueueLib s like( qsptoql )

* Fields used as QMHSNDPM parameters

D callStack s 10 inz('*')

D msgData s 100

D msgFname s 20 inz('QCPFMSG *LIBL ')

D msgId s 7

D msgKey s 4

D msgLen s 8b 0

D msgType s 10 inz('*STATUS')

D stackCounter s 8b 0 inz(2)

* Fields used as QSPMOVSP parameters

D fmtName s 8a

D fmtLength s 8b 0

* This data structure is for the error code parameter use in both APIs

* Data structure modified from QSYSINC/QRPGLESRC,QUSEC

Dqusec ds

D qusbprv 1 4B 0

D qusbavl 5 8B 0

D qusei 9 15

D quserved 16 16

D qused01 100

* Copy the API headers provided for QSPMOVSP. There are two

* different data structures provided in the /copy member which

* represent the two different functions available in QSPMOVSP.

* We're going to use QSPF0100 to move the file to a new output queue.

/copy qsysinc/qrpglesrc,qspmovsp

** Program entry.

*C *entry plist

C parm srcJobName

C parm srcJobUser

C parm srcJobNumber

C parm srcSplFile

C parm srcSplFileNbr

C parm targetQueue

C parm targetQueueLib

* Fill in QSPF0100 fields

C eval QSPSJN01 = srcJobName

C eval QSPSJUN01 = srcJobUser

C eval QSPSJNBR01 = srcJobNumber

C clear QSPSIJID01

C clear QSPISFID

C eval QSPSSFN = srcSplFile

C eval QSPSSFN00 = srcSplFileNbr

C eval QSPTOQN = targetQueue

C eval QSPTOQL = targetQueueLib

* Fill in the fields required for the call to QSPMOVSP

C eval fmtName = 'MSPF0100'

C eval fmtLength = %size(QSPF0100)

* Move the spool file.

C call 'QSPMOVSP'

C parm qspf0100

C parm fmtLength

C parm fmtName

C parm qusec


* If any errors occurred, pass them back to the caller of this program

C if qusei <> *blanks

C

C call 'QMHSNDPM'

C parm qusei msgId

C parm msgFname

C parm qused01 msgData

C parm qusbavl msgLen

C parm '*DIAG' msgType

C parm callStack

C parm stackCounter

C parm msgKey

C parm qusec

C

C endif

C eval *Inlr = *on

Figure 1: Here’s an example of using the HSM APIs to move a spool file.


BLOG COMMENTS POWERED BY DISQUS

LATEST COMMENTS

Support MC Press Online

$0.00 Raised:
$

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: