Learn about flexibly tracking field changes.
On the IT Knowledge Exchange, Luissimon recently posted, "I need to get which field changed in a file, instead of writing multiple (IF this field changed THEN DO…) for each field in each file. I was wondering if I could use some tool in RPG for doing that?" While I can't say that I'm familiar with an RPG built-in to accomplish this, I can see how using a few system APIs within an RPG application program can meet this need.
In this column, we'll look at how we can register a trigger program to be called whenever an update operation is performed on a file and how that trigger program can then determine, using the List Fields (QUSLFLD) API, which fields have been changed—and without the program having to know in advance what file or fields are being tracked.
Now before we start, I do want to point out one major item. This article shows how to provide this auditing function using one program. I'm using one program solely for the purpose of being able to demonstrate everything in one article. In a production environment, I would never use this implementation, though I would use the database triggers and system APIs being discussed in this article. Rather than one large program, I would use three distinct programs. After we've looked at the sample program, I'll overview how I would implement the various functions of the program in three different programs.
Here is the source for our example program AUDDTACHG (Audit Data Change).
h dftactgrp(*no)
d AudDtaChg pr
d TrgBfr likeds(QDBTB)
d LenTrgBfr 10i 0 const
d AudDtaChg pi
d TrgBfr likeds(QDBTB)
d LenTrgBfr 10i 0 const
****************************************************************
d CrtUsrSpc pr extpgm('QUSCRTUS')
d QualUsrSpcN 20a const
d XAttr 10a const
d IntSize 10i 0 const
d IntValue 1a const
d PubAut 10a const
d TxtDesc 50a const
d ReplaceOpt 10a const options(*nopass)
d ErrCde likeds(QUSEC) options(*nopass)
d Domain 10a const options(*nopass)
d TfrSize 10i 0 const options(*nopass)
d OptSpcAlgn 1a const options(*nopass)
d FindChgs pr
d LstFlds pr extpgm('QUSLFLD')
d UsrSpcName 20a const
d Format 8a const
d QualFilNam 20a const
d RcdFmt 10a const
d OvrPrc 1a const
d ErrCde likeds(QUSEC) options(*nopass)
d LstRcdFmts pr extpgm('QUSLRCD')
d UsrSpcName 20a const
d Format 8a const
d QualFilNam 20a const
d OvrPrc 1a const
d ErrCde likeds(QUSEC) options(*nopass)
d RtvUsrSpcPtr pr extpgm('QUSPTRUS')
d QualUsrSpcN 20a const
d UsrSpcPtr *
d ErrCde likeds(QUSEC) options(*nopass)
****************************************************************
d SpcPtr s *
d ListHdr ds likeds(QUSH0100)
d based(SpcPtr)
d FldEntryPtr s *
d FldEntry ds likeds(QUSL0100)
d based(FldEntryPtr)
d RcdEntryPtr s *
d RcdEntry ds likeds(QUSL010001)
d based(RcdEntryPtr)
d ErrCde ds qualified
d Hdr likeds(QUSEC)
d MsgDta 256a
****************************************************************
d AftImagePtr s *
d AftImage s 32766a based(AftImagePtr)
d AftNullPtr s *
d AftNullValue s 8000a based(AftNullPtr)
d B4ImagePtr s *
d B4Image s 32766a based(B4ImagePtr)
d B4NullPtr s *
d B4NullValue s 8000a based(B4NullPtr)
d GetOut s n
d RcdFmt s 10a
d SpcName s 20a inz('AUDDTACHG QTEMP')
d X s 10i 0
****************************************************************
/copy qsysinc/qrpglesrc,trgbuf
/copy qsysinc/qrpglesrc,qusec
/copy qsysinc/qrpglesrc,qusgen
/copy qsysinc/qrpglesrc,quslfld
/copy qsysinc/qrpglesrc,quslrcd
****************************************************************
/free
if (GetOut);
*inlr = *on;
return;
endif;
select;
when TrgBfr.QDBTE = '1';
// Insert operation -- not a concern today
when TrgBfr.QDBTE = '2';
// Delete operation -- not a concern today
when TrgBfr.QDBTE = '3';
// Update operations have before and after images
B4ImagePtr = %addr(TrgBfr) + TrgBfr.QDBORO;
B4NullPtr = %addr(TrgBfr) + TrgBfr.QDBORNBM;
AftImagePtr = %addr(TrgBfr) + TrgBfr.QDBNRO;
AftNullPtr = %addr(TrgBfr) + TrgBfr.QDBNRNBM;
FindChgs();
other;
// Basically Read operations which should never
// trigger this program...
endsl;
*inlr = *on;
return;
// *************************************************************
begsr *inzsr;
QUSBPrv = 0;
ErrCde.Hdr.QUSBPrv = %size(ErrCde);
RtvUsrSpcPtr(SpcName :SpcPtr :ErrCde);
select;
when ErrCde.Hdr.QUSBAvl = 0;
// User space previously created
when ErrCde.Hdr.QUSEI = 'CPF9801';
// UsrSpc not found, so create it
CrtUsrSpc(SpcName :' ' :4096 :x'00' :'*ALL' :' '
:'*YES' :ErrCde :'*DEFAULT' :0 :'1');
if ErrCde.Hdr.QUSBAvl <> 0;
dsply ('Unable to create *USRSPC: ' +
ErrCde.Hdr.QUSEI);
GetOut = *on;
else;
// Get accessibility to user space
RtvUsrSpcPtr(SpcName :SpcPtr :ErrCde);
if ErrCde.Hdr.QUSBAvl <> 0;
dsply ('Unable to access *USRSPC: ' +
ErrCde.Hdr.QUSEI);
GetOut = *on;
endif;
endif;
other;
// Something seriously wrong.
dsply ('Serious Error found: ' +
ErrCde.Hdr.QUSEI);
GetOut = *on;
endsl;
if (not GetOut);
LstRcdFmts(SpcName :'RCDL0100'
:(TrgBfr.QDBFILN02 + TrgBfr.QDBLIBN02)
:'0' :ErrCde);
select;
when ErrCde.Hdr.QUSBAvl <> 0;
dsply ('Unable to access record formats: ' +
ErrCde.Hdr.QUSEI);
GetOut = *on;
when ListHdr.QUSIS <> 'C';
dsply ('Unable to access all record formats: ' +
ErrCde.Hdr.QUSEI);
GetOut = *on;
when ListHdr.QUSNbrLE <> 1;
dsply ('More than one record format found');
GetOut = *on;
other;
RcdEntryPtr = SpcPtr + ListHdr.QUSOLD;
RcdFmt = RcdEntry.QUSFN05;
endsl;
endif;
if (not GetOut);
LstFlds(SpcName :'FLDL0100'
:(TrgBfr.QDBFILN02 + TrgBfr.QDBLIBN02)
:RcdFmt :'0' :ErrCde);
select;
when ErrCde.Hdr.QUSBAvl <> 0;
dsply ('Unable to access field info: ' +
ErrCde.Hdr.QUSEI);
GetOut = *on;
when ListHdr.QUSIS <> 'C';
dsply ('Unable to access all fields');
GetOut = *on;
other;
// Continue on
endsl;
endif;
endsr;
/end-free
*****************************************************************
p FindChgs b
d FindChgs pi
/free
for X = 1 to ListHdr.QUSNbrLE;
if X = 1;
FldEntryPtr = SpcPtr + ListHdr.QUSOLD;
else;
FldEntryPtr += ListHdr.QUSSEE;
endif;
select;
when ((FldEntry.QUSNVA = '0') or
(%subst(B4NullValue :X :1) = '0' and
%subst(AftNullValue :X :1) = '0')) and
%subst(B4Image :FldEntry.QUSOBP :FldEntry.QUSFLB) <>
%subst(AftImage :FldEntry.QUSOBP :FldEntry.QUSFLB);
dsply (%trimr(FldEntry.QUSFN02) +
' value changed');
when %subst(B4NullValue :X :1) = '1' and
%subst(AftNullValue :X :1) = '1';
// Both are null, so no change
when %subst(B4NullValue :X :1) = '1' and
%subst(AftNullValue :X :1) = '0';
dsply (%trimr(FldEntry.QUSFN02) +
' changed from null to not null');
when %subst(B4NullValue :X :1) = '0' and
%subst(AftNullValue :X :1) = '1';
dsply (%trimr(FldEntry.QUSFN02) +
' changed from not null to null');
endsl;
endfor;
/end-free
p FindChgs e
As AUDDTACHG will be used as a database trigger program, the system will call the program passing two parameters. As documented in the IBM Knowledge Center under How trigger programs work, the first parameter will contain information about the change operation and the second parameter the length of the first parameter. Documentation for the first parameter, often referred to as a trigger buffer, can be found here with the QSYSINC include for the parameter in member TRGBUF of QSYSINC/QRPGLESRC. TRGBUF defines the static portion of the parameter with data structure QDBTB, and AUDDTACHG prototypes the first parameter passed to it (TrgBfr) as being likeds(QDBTB).
Having received the two parameters, the *INZSR subroutine of AUDDTACHG runs. The *INZSR, after setting the Bytes provided field of the QUSEC API error code structure to 0 (send exceptions) and the Bytes provided field of the ErrCde API error code structure to a non-zero value (return exceptions in ErrCde), checks to see if the user space QTEMP/AUDDTACHG exists by calling the Retrieve Pointer to User Space (QUSPTRUS) API to set the pointer variable SpcPtr to the address of the user space. If this is the first time the trigger program AUDDTACHG has been called in the current job, the user space will not exist (ErrCde.Hdr.QUSBAvl will be non-zero and ErrCde.Hdr.QUSEI will be CPF9801 – Object in library not found) and AUDDTACHG will then create the user space with the Create User Space (QUSCRTUS) API and again call the QUSPTRUS API to set SpcPtr to the address of to the user space.
The SpcPtr variable is used by the program to base the common List API generic header (data structure QUSH0100 from QSYSINC/QRPGLESRC,QUSGEN) using the name ListHdr.
Our desire is to call the List Fields (QUSLFLD) API in order to determine what fields are defined for the record change that caused AUDDTACHG to be called, which is what the API will return to our user space. The QUSLFLD API, however, requires a record format name (in addition to the name of the file) and, unfortunately, the trigger buffer passed by the system to AUDDTACHG does not include the record format name. So the program calls the List Record Formats (QUSLRCD) API to get the record format name associated with the physical file TrgBfr.QBDFilN02 (the file name passed in the trigger buffer) in library TrgBfr.QDBLibN02 (the library name passed in the trigger buffer).
As physical files can have only one record format, and AUDDTACHG will be used only as a trigger for physical files, the program sets the variable RcdFmt to the value of the first, and only, record format name (RcdEntry.QUSFN05) returned by QUSLRCD. This "first" entry is accessed by setting the pointer variable RcdEntryPtr to the address of the user space (SpcPtr) plus the QUSLRCD List API generic header offset to list data (ListHdr.QUSOLD). The RcdEntryPtr variable is used by the program to base the QUSLRCD RCDL0100 returned record entry (data structure QUSL010001 from QSYSINC/QRPGLESRC,QUSLRCD) using the name RcdEntry.
Having obtained the necessary record format name, AUDDTACHG then calls the QUSLFLD API, reusing the same user space, as we have what we need from QUSLRCD, to access all of the field definitions associated with the file (using the same trigger buffer fields of TrgBfr.QDBFilN02 and TrgBfr.QDBLibN02 for file and library name, respectively) along with the record format name found in variable RcdFmt. The *INZSR subroutine then returns to the main line of AUDDTACHG.
In the main line of the program, a check is made to see what type of operation caused our trigger program to be called. This information can be found in TrgBfr.QDBTE (trigger event) with a value of '1' indicating a new record was written to the file, a value of '2' indicating a record was deleted from the file, a value of '3' indicating that an update of an existing record was done, and a value of '4' indicating that a record was read from the file. For today, all we care about are changes, a TrgBfr.QDBTE value of '3'. The other cases are basically a subset of a change operation in that an add adds all fields, a delete deletes all fields, and a read doesn't change anything.
Having determined that the current operation is an update, AUDDTACHG sets four pointers:
- B4ImagePtr to where the "prior to update" record value is found in the trigger buffer. The B4ImagePtr variable is used to base the variable B4Image, which is defined as a character field with a length of 32766 bytes (the maximum database file size). The value of B4ImagePtr is set using the address of the trigger buffer plus the trigger buffer–provided original record offset within the trigger buffer (TrgBfr.QDBORO). B4Image now represents the entire "before update" record contents.
- B4NullPtr to where the null byte map associated with the "prior to update" fields is found in the trigger buffer. The B4NullPtr variable is used to base the variable B4NullValue, which is defined as a character field with a length of 8000 bytes (the maximum number of fields in a record format). The value of B4NullPtr is set using the address of the trigger buffer plus the trigger buffer–provided original record null byte map offset (TrgBfr.QDBORNBM). B4NullValue now represents the entire "before update" null map contents.
- AftImagePtr to where the "after update" record value is found in the trigger buffer. The AftImagePtr variable is used to base the variable AftImage which, like B4Image, is defined with a length of 32766 bytes. The value of AftImagePtr is set using the address of the trigger buffer plus the trigger buffer–provided new record offset (TrgBfr.QDBNRO). AftImage now represents the entire "after update" record contents.
- AftNullPtr to where the null byte map associated with the "after update" fields is found in the trigger buffer. The AftNullPtr is used to base the variable AftNullValue which, like B4NullValue, is defined with a length of 8000 bytes. The value of AftNullPtr is set using the address of the trigger buffer plus the trigger buffer–provided new record null byte map offset (TrgBfr.QDBNRNBM). AftNullValue now represents the entire "after update" null map contents.
Null byte maps, if you're not familiar with them, allow you to determine if a given field is set to a NULL or non-NULL value where NULL means no value (which is different than, say, a field set to all blanks). For each field of the record, there will be a 1-byte character field in the null byte map. If the null byte map field is set to '0', then the field is not NULL. If the null byte map field is set to '1', then the field is set to NULL.
Having set the various pointers to address the before and after record values and null byte maps, AUDDTACHG calls procedure FindChgs().
FindChgs() loops thru all of the field definitions returned by the QUSLFLD API. The number of fields returned, the to value associated with the for statement, is found in the QUSLFLD List API generic header number of list entries (ListHdr.QUSNbrLE). As when we wanted to access the first RcdEntry in the *INZSR, FindChgs() sets the pointer variable FldEntryPtr when accessing the first field entry (X is equal to 1) by adding the List API header offset to list data (ListHdr.QUSOLD) to the address of the user space (SpcPtr). Subsequent field definitions (X is greater than 1) are accessed by adding the List API header size of each entry (ListHdr.QUSSEE) to the current value of FldEntryPtr. The FldEntryPtr variable is used by the program to base the QUSLFLD FLDL0100 returned field entry (data structure QUSL0100 from QSYSINC/QRPGLESRC,QUSLFLD) using the variable FldEntry.
Within the for loop, a select is run for each defined field. The first when is checking for the following conditions:
- Does this field even allow NULL values? This is determined by examining the QUSLFLD returned value FldEntry.QUSNVA (Null values allowed). If the value is '0', then NULL values are not allowed.
- If NULL values are allowed, are both the "before update" and "after update" fields set to a non-NULL value? This is determined by examining the trigger buffer null byte map fields associated with the field. If the Xth byte of B4NullValue is '0', then the "before update" field was not NULL, and if the Xth byte of AftNullValue is '0', then the "after update" field is not NULL
- If either of the two previous checks are true (the field is not NULL-capable or the field is NULL-capable and both the before and after fields are not NULL), then a check is made for the "before" field value not being equal to the "after" field value. This is done by comparing the substring of the B4Image and AftImage values. The %subst start position is determined using the QUSLFLD returned value FldEntry.QUSOBP (Output buffer position) and the length using the QUSLFLD returned value FldEntry.QUSFLB (Field length in bytes). If the B4Image field value is not equal to the AftImage field value, then a message is displayed using the dsply operation. The message provides the name of the field that was changed (using the QUSLFLD returned value FldEntry.QUSFN02 (Field name) with the text 'value changed').
The following when operations are pretty straightforward. If both "before" and "after" null byte map values are '1', indicating that both are NULL, we know there was no change.
Otherwise, there was a change, either changing from NULL to not NULL or from not NULL to NULL, and an appropriate message is displayed.
When the for loop completes and all appropriate messages related to field value changes have been displayed, FindChgs() returns to the main line of AUDDTACHG, *INLR is set on, and the program returns.
That's it. Per the original request of "…which field changed in a file…," we have identified each field that was changed and avoided any hard coding of field names with multiple if checks. I suspect, however, that a follow-on request might be "OK, I now know the fields that were changed, but I really wanted to know what the before and after values were when the field changed. Can this be done in the same manner?" The answer of course is yes, and all you have to do is patiently wait for next month's API Corner.
Assuming that you've stored the above program source in member AUDDTACHG of source file QRPGLESRC and that the library containing QRPGLESRC is in your current library list, then you can create the program with the following command:
CrtBndRPG Pgm(AUDDTACHG)
To test AUDDTACHG, we'll create a physical file with a variety of data types, sizes, and attributes. The file will be DTACHGPF (Data change physical file) with the following DDS definition.
R RECORD
TXTFLD 30A ALWNULL
VARTXTFLD 50A VARLEN
NBRFLD3P0 3P 0
NBRFLD10P2 10P 2
NBRFLD4B2 4B 2
NBRFLD9B0 9B 0
NBRFLD18B0 18B
NBRFLD10S0 10S 0 ALWNULL
NBRFLD15S2 15S 2
DATFLD L ALWNULL
TIMFLD T
TSFLD Z
Assuming you have stored the above DDS source in member DTACHGPF of source file QDDSSRC and that the library containing QDDSSRC is in your current library list, then you can create the file with this command:
CrtPF File(DTACHGPF) SrcFile(QDDSSRC)
And then you associate AUDDTACHG as an update trigger to DTACHGPF with this command:
AddPFTrg File(DTACHGPF) TrgTime(*After) TrgEvent(*Update) Pgm(AUDDTACHG)
Now add a record to DTACHGPF using whatever tool you are comfortable with (DFU, SQL, RPG program, etc.) and, after adding the record, change one or more field values again using the tool of your choice. You should then get the messages identifying only those fields that you changed.
I mentioned at the start of this article that "In a production environment I would never use this implementation" and that I would tend more toward using three distinct programs. The three programs I have in mind are these:
- The first program would be an interactive program that maintains a control file of those files and fields within the file that are to be audited for change (or insert and delete activity). The program would be passed a file name, use the logic found in the *INZSR subroutine of AUDDTACHG to access the fields defined for the file, merge this list with any existing file/field records in the control file, and present the user with a subfile showing those fields currently audited and those not being audited. The user makes whatever changes are desired (stop auditing this field, start auditing this field), and the program then writes/updates to the control file those fields that are to be audited. For each field being audited, the control file record would contain the essential pieces of information used by the AUDDTACHG FindChgs() procedure—the name of the file (TrgBfr.QDBFilN02), the name of the field (FldEntry.QUSFN02), whether or not the field can be set to NULL (FldEntry.QUSNVA), the offset to the field within the record (FldEntry.QUSOBP), and the length of the field in bytes (FldEntry.QUSFLB).
- The second program would be the trigger program associated with any file that has one or more records in the control file from program 1. The trigger program would verify that the file is audited (there is at least one record in the control file for the file being changed) and, if so, maintain three (or five) transaction files. The first transaction file would be a header file most likely indexed by the current %timestamp value. Each header record would also contain (at a minimum) a unique numeric value identifying the transaction, the file name (TrgBfr.QDBFilN02), the file library name (TrgBfr.QDBLibN02), the type of trigger event (TrgBfr.QDBTE), and the qualified name of the program performing the trigger event (to find this, you can refer to an earlier API Corner article, Retrieving Information, Part I, which uses the Retrieve Call Stack (QWVRCSTK) API). This program would then write to a second file (indexed by the same unique value used in the header file) that holds "before images" (B4Image) and the "before null byte map" (B4NullValue), with the third file (again indexed by the same header file index value) holding the "after images" (AftImage) and the "after null byte map" (AftNullValue). If the combination of Image and NullValue into one record/file exceeds the maximum size of a record, then the fourth and fifth transaction files would contain the B4NullValue and AftNullValue images (again indexed by the same unique value found in the header file). To support auditing of insert operations (TrgBfr.QDBTE = '1'), this program would write the header record, the AftImage value, and the AftNullValue value. To support auditing of delete operations (TrgBfr.QDBTE = '2'), this program would write the header record, the B4Image, and the B4NullValue value.
- The third program would read the header file (using the timestamp index) maintained by the second program and, for update events, access the before and after transaction files using the unique key value found in the header file record. Using the control file maintained by the first program, this third program would then perform the processing being done in the FindChgs() procedure of AUDDTACHG to identify any changed data values and record those changes to a printer file, a permanent audit file, or just dsply them (though this last option is rather unlikely). Having processed a given transaction the program would then delete the header record and any Image/NullValue records associated with the transaction
This three-program approach would have many advantages over an implementation such as AUDDTACHG. Three that readily come to mind are these:
- Most files contain at least some fields that do not need to be audited, so why process them every time?
- Most files are not going to change very often in terms of the fields defining the record, so why re-access the field definition values (that is, use QUSLFLD) every time? Just record the essential facts about fields that are to be audited once.
- Why do the audit analysis as part of the application transaction (basically slowing down interactive response time)? Record what's needed (the who, what, when of the header file) and do the field level analysis in a background batch program.
I will add that this three-program approach is not just theoretical. I successfully implemented this approach a few years back for a company needing to audit field-level changes involving several hundred files. They had absolutely no desire to change any of the application programs maintaining these files, and they wanted to minimize the impact to their users. I will point out, though, that my implementation included a few additional "bells and whistles"—items like having the transaction file records written by the second program be in the same commitment cycle as the initiating program in order to catch rollback situations, control level values in the transaction header file to accommodate changed record-level definitions (new fields, changed field definitions, etc), and the like.
Have Some System API Questions?
As usual, if you have any API questions, send them to me at This email address is being protected from spambots. You need JavaScript enabled to view it..
Bruce Vining is president and co-founder of Bruce Vining Services, LLC, a firm providing contract programming and consulting services to the System i community. He began his career in 1979 as an IBM Systems Engineer in St. Louis, Missouri, and then transferred to Rochester, Minnesota, in 1985, where he continues to reside. From 1992 until leaving IBM in 2007, Bruce was a member of the System Design Control Group responsible for OS/400 and i5/OS areas such as System APIs, Globalization, and Software Serviceability. He is also the designer of Control Language for Files (CLF).A frequent speaker and writer, Bruce can be reached at bvining@brucevining.com.
MC Press books written by Bruce Vining available now on the MC Press Bookstore.
 |
||
 |
|
IBM System i APIs at Work Leverage the power of APIs with this definitive resource. List Price $89.95 Now On Sale
|
LATEST COMMENTS
MC Press Online