20
Sat, Apr
5 New Articles

The API Corner: Performing CCSID Conversions Under Program Control

APIs
Typography
  • Smaller Small Medium Big Bigger
  • Default Helvetica Segoe Georgia Times

Learn how to use the iconv() API.

 

For the last two months, in the articles "In Search of System Values" and "Accessing System Values," we've been using the subprocedure ConvertBuffer() to convert the UTF-8 (CCSID 1208) encoded XML output of the Retrieve Command Definition (QCDRCMDD) API to the CCSID of the running job. In those articles, any discussion of the ConvertBuffer() subprocedure was deferred to a later article. This is that "later" article. The source for the subprocedure we'll be discussing is shown below.

pConvertBuffer   b                                            

dConvertBuffer   pi                                            

                                                                

dIconvOpen       pr           52   extproc('QtqIconvOpen')  

d ToCode                       32   const                    

d FromCode                     32   const                    

                                                                

dIconv           pr           10i 0 extproc('iconv')          

d iconv_t                       52   value                    

d InputPtr                       *                            

d BytesToCvt                   10i 0                          

d OutputPtr                       *                              

d BytesAvlForCvt               10i 0                          

                                                                

dIConvClose       pr           10i 0 extproc('iconv_close')    

d iconv_t                       52   value                    

                                                                

dInputPtr         s               *   inz(%addr(CmdD_UTF8))

dOutputPtr       s               *   inz(%addr(CmdD_Job))  

dBytAvl_CmdD     s             10i 0 inz(%size(CmdD_Job))  

dRtnVal           s             10i 0                      

                                                            

dFromCode         ds                 qualified            

d CCSID                         10i 0 inz(1208)              

d ConvAlt                       10i 0 inz(0)                

d SubstAlt                     10i 0 inz(0)                

d SSAlt                         10i 0 inz(0)                

d InpLenOpt                     10i 0 inz(0)                

d ErrOpt                       10i 0 inz(0)                

d                               8   inz(*ALLx'00')        

                                                            

dToCode           ds                 qualified            

d CCSID                         10i 0 inz(0)                

d ConvAlt                       10i 0 inz(0)                

d SubstAlt                     10i 0 inz(0)                        

d SSAlt                         10i 0 inz(0)                        

d InpLenOpt                     10i 0 inz(0)                        

d ErrOpt                       10i 0 inz(0)                        

d                               8   inz(*ALLx'00')                

                                                                    

diconv_t         ds                                                

d                               10i 0 dim(13)                      

                                                                    

/free                                                              

                                                                    

iconv_t = IconvOpen(ToCode :FromCode);                            

RtnVal = Iconv(iconv_t :InputPtr :BytRtn :OutputPtr :BytAvl_CmdD);

RtnVal = IconvClose(iconv_t);                                    

                                                                    

/end-free                                                          

                                                                    

pConvertBuffer   e                        

While the subprocedure contains quite a bit in terms of prototype and data definition, there are only three executable lines, which represent the use of three APIs in performing the CCSID conversion from UTF8 to the job CCSID. These APIs are Code Conversion Allocation (QtqIconvOpen), Code Conversion (iconv), and Code Conversion Deallocation (iconv_close).

The QtqIconvOpen API is used first and identifies the type of CCSID conversion to be performed. There are two input parameters, and the API returns a return value.

The first parameter identifies the CCSID to convert to, the second parameter the CCSID being converted from. In addition, the second parameter defines characteristics of the CCSID conversion, such as conversion alternatives and how the length of the data to be converted is determined. Both of the input parameters to the QtqIconvOpen API use a common structure which is formatted as…

  • A 4-byte integer identifying a CCSID value. The special value 0 indicates that the job CCSID is to be used or, if the job CCSID is 65535, the job default CCSID.
  • A 4-byte integer, used only with the second input parameter, identifying the conversion alternative to be used. A value of 0 represents a round-trip conversion, 57 an enforced subset match conversion, and 102 a best-fit conversion.
  • A 4-byte integer, used only with the second input parameter, indicating whether the number of substitution characters encountered (when the conversion alternative value is 57) is to be returned by the iconv API when CCSID conversions are performed.
  • A 4-byte integer, used only with the second input parameter, indicating how DBCS shift-states are to be set when calling the iconv API.
  • A 4-byte integer, used only with the second input parameter, specifying how the iconv API is to determine the length of data to convert. A value of 0 indicates that iconv will be explicitly provided with the length of data to convert by way of a parameter passed to the API, a value of 1 that the iconv API is to determine the length based on the presence of a null byte in the data stream to be converted.
  • A 4-byte integer, used only with the second parameter, indicating whether it is an error when DBCS data is encountered when converting from a mixed-byte CCSID to a SBCS CCSID. A value of 0 indicates that the iconv API is to use a substitution character when DBCS data is encountered, a value of 1 that the API should return an error.
  • An 8-byte character field that is currently reserved and must be set in its entirety to x'00's.

The return value of QtqIconvOpen is defined as an array of thirteen 4-byte integer values and is used in subsequent calls to the iconv and iconv_close APIs in order to identify the open CCSID conversion that is to be used by these APIs. With one exception, how the system uses this array is not documented. As a user of the API, you simply need to make sure you have allocated an array of thirteen 4-byte integer elements for this return value. The one exception is that if the first element of this array is returned with a value of -1, then an error was encountered when opening the CCSID conversion session. In this situation, errno can be examined for further information on the failure and subsequent calls to APIs such as iconv should not be done (using this returned descriptor).

Data structure ToCode is used in subprocedure ConvertBuffer() as the first parameter when calling the QtqIconvOpen API and data structure FromCode as the second parameter. The initialization of ToCode indicates that the XML data is to be converted to the job CCSID as subfield ToCode.CCSID is set to 0. The initialization of FromCode indicates that the data being converted is currently encoded as UTF8 (FromCode.CCSID is set to 1208), that a round-trip conversion is to be done (FromCode.ConvAlt is set to 0), and that the program will explicitly tell the iconv API how many bytes of data to convert (FromCode.InpLenOpt is set to 0).

While subprocedure ConvertBuffer()calls the QtqIconvOpen() API only once, a program (or job) can have multiple CCSID conversion sessions defined concurrently. Though perhaps a bit of overkill, you can have in excess of 100,000 types of CCSID conversions open at the same time.

The second API used is iconv and is used to perform the actual data conversion from one CCSID to another. There are five parameters passed to the iconv API, and the API returns a return value.

The first parameter to iconv is input-only and is a CCSID conversion session descriptor previously returned by the QtqIconvOpen API. This parameter is passed by value, as indicated in the iconv API prototype Iconv found in subprocedure ConvertBuffer.

The second parameter is a pointer that is used as both an input and an output parameter. On input to the iconv API, this pointer addresses the first character to be converted, and upon return by the API, this pointer addresses the first byte following the last byte used in the current conversion. In subprocedure ConvertBuffer, this parameter is prototyped as a pointer passed by reference. An alternative approach would be to prototype a pointer passed by value where that pointer value is set to the address of the updateable pointer variable.

The third parameter is a 4-byte integer value that can be used as both an input and output parameter. If the input length determination option (FromCode.InpLenOpt in subprocedure ConvertBuffer), when previously calling the QtqIconvOpen API, was set to 0, then on input to the iconv API, this parameter provides the number of bytes to be converted. Upon return from the API, the value of this parameter reflects the number of bytes left to convert. If an input length determination option (FromCode.InpLenOpt) value of 1 was set when calling QtqIconvOpen, then this parameter, when calling iconv, must be set to 0.

The fourth parameter to iconv is, like the second parameter, a pointer that is used for both input and output. On input, this pointer addresses the first byte to be used when writing the CCSID converted data. Upon return from the API, this pointer is updated to address the first byte following the last byte of the converted data that was written.

The fifth parameter is, like the third parameter, a 4-byte integer value that is used as both input and output. On input to iconv, this parameter contains the number of bytes available in the output buffer for the writing of converted data. Upon return from the API, this parameter reflects the number of bytes remaining (still available for use) in the output buffer after the writing of any converted data.

The return value of the iconv API is a 4-byte integer value providing status information related to the API. A return value of 0 or greater indicates that the conversion was successful while a value of -1 indicates failure (with errno providing additional information concerning the failure). A positive non-zero return value is possible if, when calling the QtqIconvOpen API, a count of the substitution characters encountered is requested (that is, if a value of 1 was used for FromCode.SubstAlt in subprocedure ConvertBuffer).

In subprocedure ConvertBuffer, the call to iconv is done with the following statement.

RtnVal = Iconv(iconv_t :InputPtr :BytRtn :OutputPtr :BytAvl_CmdD);

The parameter iconv_t is the CCSID conversion descriptor returned by the previous call to QtqIconvOpen and requests a CCSID conversion from UTF8 to the job CCSID. InputPtr is a pointer initialized to the address of our input buffer (subfield CmdD_UTF8 of data structure CmdD_RcvVar, which is the UTF8 encoded XML data returned by the Retrieve Command Definition (QCDRCMDD) API) and addresses the first byte of the XML data to be converted. BytRtn is the length of the XML data returned by the QCDRCMDD API—that is, how much UTF8 data is to be converted. OutPutPtr is a pointer initialized to the first byte of our output buffer (CmdD_Job) and addresses where the first byte of CCSID converted XML data is to be written. BytAvl_CmdD is the size of the CmdD_Job output buffer and defines how many bytes of converted XML data can be written in the job CCSID. The call to iconv by ConvertBuffer is a bit simplistic in that it assumes success and assumes that all of the XML data can be converted into an output buffer of 65527 bytes. But knowing the characteristics of the DSPSYSVAL command, this seems reasonable. In a future article, we can look at more-complex CCSID conversion environments where we may need to do a wee bit more checking on the returned results of the iconv API.

The third API is iconv_close and is used to close a previously opened CCSID conversion session. There is one input parameter, and the API returns a return value. The input parameter identifies the open CCSID conversion that is to be closed and is the conversion descriptor previously returned by the QtqIconvOpen API. The API return value can be used to determine whether iconv_close was successful or not in closing the CCSID conversion session. A value of 0 indicates success, and a value of -1 that an error was encountered. In an error situation, errno can be used to determine the type of failure.

While use of the iconv_close API is optional (all open CCSID conversion sessions will be closed implicitly when the job ends), each open CCSID conversion session does consume some system storage. So closing a session when you know you're done with it is recommended.

CCSID conversions, under program control, are that simple—or at least they are in the case of the LSTSYSVAL program.

Next month, we'll look at some additional considerations that come into play when doing CCSID conversions—in particular, conversions involving UTF8 as use of this encoding is rather pervasive when working in a network of systems. Though not difficult to accomplish, we'll also look at LSTSYSVAL support for displaying the QLOCALE system value (which I had originally intended for this month but am now deferring due to space reasons).

As usual, if you have any API questions, send them to me at This email address is being protected from spambots. You need JavaScript enabled to view it.. I'll see what I can do about answering your burning questions in future columns.

Bruce Vining

Bruce Vining is president and co-founder of Bruce Vining Services, LLC, a firm providing contract programming and consulting services to the System i community. He began his career in 1979 as an IBM Systems Engineer in St. Louis, Missouri, and then transferred to Rochester, Minnesota, in 1985, where he continues to reside. From 1992 until leaving IBM in 2007, Bruce was a member of the System Design Control Group responsible for OS/400 and i5/OS areas such as System APIs, Globalization, and Software Serviceability. He is also the designer of Control Language for Files (CLF).A frequent speaker and writer, Bruce can be reached at This email address is being protected from spambots. You need JavaScript enabled to view it.. 


MC Press books written by Bruce Vining available now on the MC Press Bookstore.

IBM System i APIs at Work IBM System i APIs at Work
Leverage the power of APIs with this definitive resource.
List Price $89.95

Now On Sale

BLOG COMMENTS POWERED BY DISQUS

LATEST COMMENTS

Support MC Press Online

$0.00 Raised:
$

Book Reviews

Resource Center

  • SB Profound WC 5536 Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application. You can find Part 1 here. In Part 2 of our free Node.js Webinar Series, Brian May teaches you the different tooling options available for writing code, debugging, and using Git for version control. Brian will briefly discuss the different tools available, and demonstrate his preferred setup for Node development on IBM i or any platform. Attend this webinar to learn:

  • SB Profound WP 5539More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.

  • SB HelpSystems ROBOT Generic IBM announced the E1080 servers using the latest Power10 processor in September 2021. The most powerful processor from IBM to date, Power10 is designed to handle the demands of doing business in today’s high-tech atmosphere, including running cloud applications, supporting big data, and managing AI workloads. But what does Power10 mean for your data center? In this recorded webinar, IBMers Dan Sundt and Dylan Boday join IBM Power Champion Tom Huntington for a discussion on why Power10 technology is the right strategic investment if you run IBM i, AIX, or Linux. In this action-packed hour, Tom will share trends from the IBM i and AIX user communities while Dan and Dylan dive into the tech specs for key hardware, including:

  • Magic MarkTRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now!  Request Now.

  • SB HelpSystems ROBOT GenericForms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.

  • SB HelpSystems ROBOT GenericIT security is a top priority for businesses around the world, but most IBM i pros don’t know where to begin—and most cybersecurity experts don’t know IBM i. In this session, Robin Tatam explores the business impact of lax IBM i security, the top vulnerabilities putting IBM i at risk, and the steps you can take to protect your organization. If you’re looking to avoid unexpected downtime or corrupted data, you don’t want to miss this session.

  • SB HelpSystems ROBOT GenericCan you trust all of your users all of the time? A typical end user receives 16 malicious emails each month, but only 17 percent of these phishing campaigns are reported to IT. Once an attack is underway, most organizations won’t discover the breach until six months later. A staggering amount of damage can occur in that time. Despite these risks, 93 percent of organizations are leaving their IBM i systems vulnerable to cybercrime. In this on-demand webinar, IBM i security experts Robin Tatam and Sandi Moore will reveal:

  • FORTRA Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:

  • FORTRAManaging messages on your IBM i can be more than a full-time job if you have to do it manually. Messages need a response and resources must be monitored—often over multiple systems and across platforms. How can you be sure you won’t miss important system events? Automate your message center with the Robot Message Management Solution. Key features include:

  • FORTRAThe thought of printing, distributing, and storing iSeries reports manually may reduce you to tears. Paper and labor costs associated with report generation can spiral out of control. Mountains of paper threaten to swamp your files. Robot automates report bursting, distribution, bundling, and archiving, and offers secure, selective online report viewing. Manage your reports with the Robot Report Management Solution. Key features include:

  • FORTRAFor over 30 years, Robot has been a leader in systems management for IBM i. With batch job creation and scheduling at its core, the Robot Job Scheduling Solution reduces the opportunity for human error and helps you maintain service levels, automating even the biggest, most complex runbooks. Manage your job schedule with the Robot Job Scheduling Solution. Key features include:

  • LANSA Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.

  • LANSAWhen it comes to creating your business applications, there are hundreds of coding platforms and programming languages to choose from. These options range from very complex traditional programming languages to Low-Code platforms where sometimes no traditional coding experience is needed. Download our whitepaper, The Power of Writing Code in a Low-Code Solution, and:

  • LANSASupply Chain is becoming increasingly complex and unpredictable. From raw materials for manufacturing to food supply chains, the journey from source to production to delivery to consumers is marred with inefficiencies, manual processes, shortages, recalls, counterfeits, and scandals. In this webinar, we discuss how:

  • The MC Resource Centers bring you the widest selection of white papers, trial software, and on-demand webcasts for you to choose from. >> Review the list of White Papers, Trial Software or On-Demand Webcast at the MC Press Resource Center. >> Add the items to yru Cart and complet he checkout process and submit

  • Profound Logic Have you been wondering about Node.js? Our free Node.js Webinar Series takes you from total beginner to creating a fully-functional IBM i Node.js business application.

  • SB Profound WC 5536Join us for this hour-long webcast that will explore:

  • Fortra IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn: