What You Should Know About the Format of Records Saved to a Save File

RPG
Typography
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

When a file contains variable-length fields, the data of the fixed-length fields of its deleted records can still be retrieved.

 

A known technique to retrieve the records deleted from a file consists of saving the file to a save file and extracting the records from the save file. This technique is based on the fact that, in many cases, the records of the file are stored in the save file maintaining their format, apart from a status byte added by the system at the beginning of each record.

 

However, there are situations where the format of the records in the save file is different from the original format. This happens when one of the fields of the file is variable-length, nullable, or of type date-time (DDS types L, T, and Z). In these cases, we can still retrieve the data of the fixed-length fields of the deleted records and, what's more, the data of some variable-length fields for some of those records.

 

Some Terms

From now on, when we talk about the length of a field, we will be referring to the number of bytes (not characters) and, for variable-length fields, will not take into account the two initial bytes.

 

So the current length of a variable-length field will refer to the number of bytes taken up currently by the data of the field, its maximum length to the maximum number of bytes that its data can take up, and its allocated length to the number of bytes that the system allocates for its data.

 

For example, the maximum length of a single-byte variable-length field of 10 characters (10A VARLEN) is 10 (not 12), and its current length can be from 0 to 10. The maximum length of a graphic variable-length field of 10 characters (10G VARLEN) is 20 (not 22), and its current length can be 0, 2, 4...or 20.

 

We will say that a variable-length field is of type 1 if the number of bytes allocated by the system for the field is equal to its maximum length. If it is less than its maximum length, we will say that the field is of type 2.

 

The space allocated by the system for a variable-length field is calculated as shown in Figure 1:

             

130121 MaciasFigure1 

Figure 1: This is the space allocated for variable-length fields.

 

Note that the system can allocate some space for a variable-length field without the user's request. On the other hand, when the user requests the system to allocate a number of characters for a variable-length field, the number of characters eventually allocated by the system can be greater than the one requested. We will see later how to obtain the length of the space allocated.

 

Data Records Saved to a Save File

When a physical file is saved to a save file, the records of the physical file are stored in the save file with a format different from the original one. First of all, every record of the physical file is stored in the save file in a logical record A and, in some cases, in a logical record B.

 

Records A contain the data of the fixed-length fields, the variable-length fields of type 1, and the variable-length fields of type 2 whose current length is less than or equal to their allocated length. Records B contain the data of the variable-length fields of type 2 whose current length is greater than their allocated length.

 

To simplify the following description, let's suppose that only one of the members of the physical file is saved to the save file.

 

Records A (corresponding to the records of the saved member) are stored consecutively in positions 1 to 512 of a group of records of the save file. These positions of these records of the save file make up what we will call the area A of the save file. Records B are stored (in general, consecutively) in positions 1 to 512 of another group of records of the save file. We will call these positions the area B of the save file.

 

Records A and B contained in a save file are shown in Figure 2:

 

130121 MaciasFigure2 

Figure 2: Here's an example of a save file containing a physical file.

 

To find out where area A begins, we have to look for a record of the save file whose positions 15 to 24 contain the name of the saved member. This is the first record of the save file containing information about that member. Then, look for a record whose position 2 is equal to x03 and whose position 33 is equal to x80. This is the first record of area A and contains the first logical record A from position 33.

 

To identify the initial record of area B, we have to look for the last record of the save file belonging to area A (we will see how to in the section "Format of Records A") and then for a record whose position 2 is equal to x13. This is the first record of area B.

 

Areas A and B are divided into segments 16MB long (2^24 bytes). As each record of the save file contains 512 bytes, a segment is stored in 32K (32,768) records of the save file. The first 32 bytes of a segment are reserved for the segment itself.

 

A record A can be stored in two consecutive segments. In this case, the part of record A stored in the second segment will begin in position 33 of the segment (as its first 32 bytes are reserved).

 

A record B cannot be stored in two consecutive segments. If a record B does not fit into the remaining space of a segment of area B, the whole of it will be stored in the following segment from position 33.

 

Length of the Space Allocated for Variable-Length Fields

The length (in bytes) allocated by the system for each variable-length field of the physical file, as well as whether they are of type 1 or 2, is stored in the save file between the first record containing information about the saved member and the first record of area A (see section "Data Records Saved to a Save File").

 

We have to look for a record of the save file whose positions 47 to 56 contain the name of the member and then for a record whose positions 1 to 2 contain, in binary format, the number of fields of the file. An array with one item per field of the file is stored in this record from positions 33 to 512 (and, if necessary, in the following records from positions 1 to 512). Each item of this array is 32 bytes long. Its format is shown in Figure 3:

 

 130121 MaciasFigure3 new

Figure 3: This is the format of an item containing variable-length info for a field.

 

For an example, see section "Example of Records A and B for a File with Variable-Length Fields."

 

Format of Records A

RecordA = Status [VarLenInfo] [NullInfo] FieldA1 ... FieldAn [VarLenFiller]

 

where [ ] shows that an element can be present or not and n is the number of fields of the physical file.

 

Status

 

1 byte. Values x80 or xA0 show a valid record, xC0 or xE0 a deleted record, and x00 or x01 the end of the area A.

 

VarLenInfo = RecordBSegment  RecordBOffset  RecordBLen

 

Exists if any field of the physical file is variable-length. In this case, it is 8 bytes long.

 

RecordBSegment

 

2 bytes, binary format.

Number of segment of area B where record B lies. It is 0 when there is no record B associated.

 

RecordBOffset

 

4 bytes, binary format.

Offset of record B inside the segment. This offset ranges from x0000 0000 to x00FF FFFF (that is, byte 1 is always x00).

 

The absolute offset of record B in area B can be calculated as follows:

(RecordBSegment - 1) * 2^24 + RecordBOffset.

 

RecordBLen

 

2 bytes, binary format.

Length (in bytes) of the record B. Can be 0.

 

NullInfo

 

Exists if any field of the physical file allows the value null. In this case, 1 bit is used for every field of the file to store whether it is null (1) or not (0). Therefore, the number of bytes needed is n / 8 rounded up, where n is the number of fields of the file.

 

FieldAj = FixedField or VarField     for j = 1 to n

 

VarLenFiller

 

Exists if any field of the physical file is variable-length and the length (in bytes) of record A is not a multiple of 16.

It contains as many bytes x00 as necessary for the length of the record A to be a multiple of 16.

 

FixedField

 

Contains the data of a fixed-length field. This data has the same format as the physical file except for the types shown in Figure 4:

 

130121 MaciasFigure4

Figure 4: These are the types with different formats in a physical file and a record A.

 

Figure 5 shows how some date-time values are stored in a record A:

 

130121 MaciasFigure5

Figure 5: These are date, time, and timestamp values in record A.

 

As a field of type date is stored in record A as a number of days, we can obtain its equivalent in *LONGJUL format by taking any of the dates of Figure 5 as a reference and taking into account the leap years.

 

VarField = [VarFieldFiller]  VarFieldLen  [VarFieldAllocSpace]

 

VarFieldFiller

 

1 byte x00. Exists if the offset of VarField in record A is odd. This way, the offset of VarFieldLen will always be even.

 

VarFieldLen

 

2 bytes, binary format.

Contains the current length (in bytes) of this variable-length field.

Possible values: from 0 to 32740 (x7FE4)that is, up to 32740 single-byte characters or 16370 graphic charactersand -32768 (x8000). The value x8000 shows that the data of this variable-length field is not stored in record A but in its associated record B.

 

VarFieldAllocSpace

 

Space allocated (if any) in record A for the data of this variable-length field. Its first VarFieldLen bytes contain the data of the variable-length field when VarFieldLen is different from x8000.

Can exist even when VarFieldLen is equal to x8000.

See section "Length of the Space Allocated for Variable-Length Fields" to learn how to obtain the length of VarFieldAllocSpace.

 

Properties of Records A

All of the records A have the same length.

 

If the physical file contains a variable-length field, the length (in bytes) of record A is a multiple of 16.

 

The offset in record A of the length of a variable-length fieldthat is, the offset of VarFieldLenis always even.

 

The record number 1 of the physical file member is stored in record A number 2, the record number 2 in record A number 3, and so on. The record A number 1 does not correspond to any record of the physical file member.

 

A record A can be stored in two consecutive segments.

 

Format of Records B

RecordB = RRN1 AccumLenB1 ... AccumLenBm [DataB1] ... [DataBm] +

                RRN2 EndOfRecord

 

where [ ] shows that an element can be present or not and m is the number of variable-length fields of type 2 of the physical file. See section "Length of the Space Allocated for Variable-Length Fields" to learn how to obtain m.

 

RRN1

 

4 bytes, binary format.

Relative record number of the record of the physical file member whose data is stored in this record B. Can be 0.

 

AccumLenBk      for k = 1 to m

 

2 bytes, binary format.

Length (in bytes) occupied by the data of the k first fields of the record Bthat is, the sum of the bytes occupied by DataB1, DataB2...and DataBk.

 

DataBk                                for k = 1 to m

 

Contains the data (if any) for the kth variable-length field of type 2 of the physical file. This data occupies AccumLenBk - AccumLenB(k - 1) bytes, being AccumLenB0 equal to 0.

 

RRN2

 

4 bytes, binary format.

Relative record number of the record of the physical file member whose data is or was stored in this record B.

 

EndOfRecord

 

1 byte with value x73.

 

Properties of Records B

Records B can have different lengths.

 

The offset and length of record B are stored in its corresponding record A (in VarLenInfo).

 

When the offset of record B is different from 0, its length can still be 0 (meaning that record B contains no data right now).

 

A record B cannot be stored in two consecutive segments.

 

Deleted Records

For a record deleted from a physical file, all the data in its corresponding record A remains unchanged except Statuswhich is set to xC0 or xE0and RecordBSegment, RecordBOffset, and RecordBLenwhich are set to 0so the data stored in NullInfo, FieldA1...and FieldAn can be retrieved. On the other hand, and unfortunately, we have no direct reference to the location of its record B.

 

Example of Records A and B for a File with Variable-Length Fields

Physical file FILE1:

 

A          R RFILE1

A            FLD1           3A

A            FLD2          30A         VARLEN

A            FLD3          30A         VARLEN(10)

A            FLD4          20G         VARLEN(5) CCSID(13488)

A* UCS-2: Space=x0020 1=x0031 2=x0032 3=x0033

A            FLD5           3A         ALWNULL

 

CRTPF FILE(QTEMP/FILE1) MBR(MEMBER1)

 

insert into QTEMP/FILE1 values

('AAA', ''     , 'FOURTEEN BYTES',

   vargraphic('12345' , 5, 13488), 'END'),

('BBB', 'VARLEN', 'FOURTEEN BYTES',

   vargraphic('123456', 6, 13488), 'END'),

('CCC', ''     , 'OVER FOURTEEN BYTES',

   vargraphic('12345', 5, 13488), null )

 

 

Save file:

 

SAVOBJ OBJ(FILE1) LIB(QTEMP) OBJTYPE(*FILE)

       DEV(*SAVF) SAVF(QTEMP/SAVF1)

       FILEMBR((FILE1 (MEMBER1))) UPDHST(*NO)

 

The format of records A and B are shown in Figures 6 and 7, respectively:

 

130121 MaciasFigure6

Figure 6: This is the format of records A for FILE1.

 

130121 MaciasFigure7
Figure 7: This is the format of records B for FILE1.

 

Records of the save file:

 

First record containing information on MEMBER1:

 

                                                    RRN  From  To

*...+....1....+....2....+....3....+....4....+....5   33     1  50

­­­­    FILE1     MEMBER1              °   ­­­ø­­­­­­­­­­

FFFFCCDCF44444DCDCCDF44444444444440900070000000000

FFFF693510000045425910000000000000B000000000000000

 

Record with "MEMBER1" in position 47:

 

                                                    RRN  From  To

*...+....1....+....2....+....3....+....4....+....5   41     1  50

­­­­     m   o-¯L           Ø       a  °FILE1     MEMB

00000903296BD000000000008200F0008009CCDCF44444DCDC

01083404D60C3000010000000000F00010B069351000004542

 

....+....6                                                 51  60

ER1

CDF4444444

5910000000

 

Record with the number of fields in positions 1 and 2:

 

   < beginning of item   > end of item

   V Variable-length indicator   ww allocated length

   nn number of variable-length field of type 2

 

                                                    RRN  From  To

*...+....1....+....2....+....3....+....4....+....5   43     1  50

               Ë  §

­­­­000000000000000700B0000000000000000000000000000000

05030A100009000303500000000000000403000A0000000000

                                <        V  nnww

 

....+....6....+....7....+....8....+....9....+....0         51 100

                       Ø

00000000000000000100000800000000000000000000000001

00000000000000040E000E000001000000000000000000040E

             ><        V  nnww               ><

 

....+....1....+....2....+....3....+....4....+....5        101 150

     Ø                               µ

0001080000000000000000000000000200020A000000000000

00000000020E0000000000000000060800000000030A000000

     V  nnww               ><        V  nnww

 

....+....6....+....7....+....8....+....9....+....0        151 200

                                          Ø

00020000000000000200000000000000000000000080000000

00000000000403000C20000000000000000000000000000000

          ><       V  nnww               >

 

First (and only) record of the area A:

 

   Aj___ FieldAj   S Status   v VarLenInfo   n NullInfo

   f filler   ww length of variable-length field

 

                                                    RRN  From  To

*...+....1....+....2....+....3....+....4....+....5   49     1  50

­­­­     © u o-¯L                   Ø

00000B0A296BD0000000000000030000800000000044400000

03083404D60C30000100000000000100000000000800000000

                                          A1_A2_A3

                                Svvvvvvvvn   fwwww

 

....+....6....+....7....+....8....+....9....+....0         51 100

                              Ø         AAA     FO

0000000000000000000000000044408000000000CCC00000CD

00000000000000000000000000000000000000001110000E66

______________A4__________A5_           A1_A2_A3__

              ww             fSvvvvvvvvn   fwwww

 

....+....1....+....2....+....3....+....4....+....5       101 150

URTEEN BYTES   ? ? ? ? ?END Ø         BBB Ø   FOUR

EDECCD4CEECE000303030303CDC08000002020CCC08000CDED

4935550283520A0102030405554000100000102220000E6649

____________A4__________A5_           A1_A2_A3____

            ww             fSvvvvvvvvn   fwwww

 

....+....6....+....7....+....8....+....9....+....0       151 200

TEEN BYTESØ           END Ø         CCC   Ø

ECCD4CEECE800000000000CDC08000004020CCC00080000000

35550283520000000000005540001000102833300000000000

__________A4__________A5_           A1_A2_A3______

          ww             fSvvvvvvvvn   fwwww

 

....+....1....+....2....+....3....+....4....+....5       201 250

           ? ? ? ? ?

00000000000303030303444000000000000000000000000000

000000000A0102030405000000000000000000000000000000

________A4__________A5_

        ww             fS

 

First (and only) record of the area B:

 

   Bk___ DataBk   nnnn RRN   wk AccumLenk

   E EndOfRecord

                                                    RRN  From  To

*...+....1....+....2....+....3....+....4....+....5   57     1  50

     © u o-¯L                             VARLEN ?

01000B0A296BD00000000000000000000000000001ECDDCD03

03083404D60C30000100000001000000000206060251935501

                                          B1____B3

                                nnnnw1w2w3

 

....+....6....+....7....+....8....+....9....+....0         51 100

 ? ? ? ? ?    Ë          OVER FOURTEEN BYTES    Ë

0303030303000070000000101DECD4CDEDECCD4CEECE000070

02030405060002300030003036559066493555028352000330

__________               B2_________________

          nnnnEnnnnw1w2w3                   nnnnE

 

Appendix

 

Figure 8 contains the maximum values for database files that we have been implicitly dealing with:

 

130121 MaciasFigure8

Figure 8: These are the maximum values for database files (from V5R3 to V7R1).

 

BLOG COMMENTS POWERED BY DISQUS