QLOAD: Using delimited files

The two most recent versions of QLOAD have added a feature to allow you to upload messages onto queues using QLOAD even when the file is not in the standard QLOAD format (i.e. it wasn’t from a QLOAD unload).

UploadDelimitedFileIn fact, the file can be in almost any format you want, so long as it is possible for you to describe where the messages are in the file.

QLOAD can now use files containing message data where each message is delimited from the next message in some way. This could mean that there are tags, or separator characters between the messages or even, the simplest example, that each message is on a new line.

Message text 1
Message text 2
Message text 3

msgs.txt: A file with one message on each line

To load the above file onto a queue, with one message taken from the text on each line, you would use the following QLOAD invocation.

qload -m MQG1 -o Q1 -f msgs.txt -Cd -nn

You haven’t specified any delimiters on this command, and that is because QLOAD assumes that new line is the delimiter you want if you don’t say otherwise. The -n switch is what says you want to parse a delimited file, and saying -nn tells QLOAD not to include the delimiters (in this case the new line character) as part of the message text. You could be explicit about the delimiter in use with the following command.

qload -m MQG1 -o Q1 -f msgs.txt -Cd -nn -ne:"\n"

In our next example, we have a file with some character separators.


data.txt: A file with single character separators

To load the above file onto a queue, with one message taken from the text on each line, you would use the following QLOAD invocation. Note that you must have the final character delimiter for all four messages to be found.

qload -m MQG1 -o Q1 -f data.txt -Cd -nn -ne:/

Helpfully, QLOAD will output a message to tell you how many messages it has found with a prompt before you continue, to help you spot when it hasn’t found all the messages you hoped it would find.

4 messages found in the file. Do you want to continue? (y/n)

If you have explicit start and end delimiters, then you can start to jump past data in the file that is not part of a message, which is handy for commentary. For example with the following file:-

Here are a few messages
and then here are a few more

info.txt: A file with start and end delimiters, and comments

To load the above file onto a queue, ignoring the comments and only pulling out the message data between the square brackets, you would use the following command.

qload -m MQG1 -o Q1 -f info.txt -Cd -nns:[ -nne:]

There is also a short-hand you can use when you have delimiters of the same length, as shown in the next command.

qload -m MQG1 -o Q1 -f info.txt -Cd -nnd:[]

Delimiters don’t have to be single characters, nor do they have to be excluded from the message. You may well have message content that is naturally delimited, for example XML messages. The outside tag of the structure of your XML message may well be all you need as a delimiter.

  <date>12 Jun 2019</date>
  <date>29 Aug 2019</date>

xml.txt: A file with XML message data

A file like this where the delimiter is part of the message data can be loaded with the following command.

qload -mMQG1 -oQ1 -f xml.txt -Cd -ni -ns:"<message>" -ne:"</message>"

In this case you cannot use the -nd flag because the delimiters are not of the same length, and you must specify each one individually using the start (-ns) and end (-ne) flags.

If you need to have some message properties added to your message when it is uploaded, this data can also be found are delimited text in your file. Looking again at the XML data example, here we have added some property data to the file.

  <importance dt='i4'>1</importance>
  <date>12 Jun 2019</date>
  <importance dt='i4'>2</importance>
  <date>29 Aug 2019</date>

xml2.txt: A file with XML message data and message properties

To load this file onto a queue, you now need to provide two sets of delimiters so that QLOAD can know where the message data is and where the property data is.

qload -m MQG1 -o Q1 -f xml2.txt -Cd -nis:"<message>" -nie:"</message>" -nip:"<usr>" -niq:"</usr>"

While it is likely that you will use this second set of delimiters for message properties, that is the <usr> folder in an MQRFH2 header, in fact you can use it to construct any MQRFH2 folder that you require since you provide the complete folder structure.

The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.


QLOAD version 9.1.1 is released

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

The main features of the release are as follows:-

  • Non-QLOAD file can contain property data
    Extending the feature introduced in QLOAD V9.1.0, this allows your delimited file to contain a second set of delimiters to mark where property data to be added to the message is to be found. Look out for a more detailed blog post coming soon.
  • Fully qualified queue name separator character
    If your queue names contain one of the default separator characters that QLOAD looks for, you can now specify exactly which character should be considered a separator to avoid confusion.
    For example to put messages to a queue called Q1/SALES on queue manager MQG2:-

    qload -m MQG1 -f data.qld -$# -oMQG2#Q1/SALES

    or to avoid QLOAD thinking that you want to put messages to a queue called SALES on queue manager Q1, when actually your queue is called Q1/SALES:-

    qload -m MQG1 -f data.qld  -$#-oQ1/SALES
  • Backout queue
    This version of QLOAD will pay attention to the backout threshold (BOTHRESH) and re-queue name (BOQNAME) you have defined on any input queues, and re-queue any messages with an MQMD.BackoutCount that meet or exceed the threshold.
  • Application Name
    If your IBM MQ version is appropriate, QLOAD will set an application name of ‘MQGem Software QLOAD’
  • Summary Display
    The summary display is extended in this version to show more age and size ranges in its output. For example:

    Message Ages
    < 2  secs < 10 secs < 1   min < 1  hour < 1   day < 1  week
    --------- --------- --------- --------- --------- ---------
            0         0         0         0         0         2
    < 2 weeks < 4 weeks < 3 month < 6 month  1  year
    --------- --------- --------- --------- --------- ---------
            2         2        13       110         0         0
    Youngest Message : 4 Days 2 Hrs 29 Mins 28 Secs
    Oldest   Message : 19 Weeks 3 Days 3 Hrs 26 Mins 7 Secs
    Message Sizes
    < 100b <  1Kb <  4Kb <100Kb <  1Mb < 10Mb  50Mb
    ------ ------ ------ ------ ------ ------ ------ ------
         1    128      0      0      0      0      0      0
    Smallest Message : 4 bytes
    Biggest  Message : 368 bytes
  • Counting mode
    Sometimes you might want to simply discover how many messages are on your queue that meet a certain criteria. For example, how many are over a certain age, or how many contain a certain string of text. Older versions of QLOAD could be used to do such a count with a little bit of manipulation using a command like the following:-

    qload -m MQG1 -i Q1 -T05:00:00 -f null

    New in this version, you can specifically request a count of the messages as follows:

    qload -m MQG1 -i Q1 -T05:00:00 -dk

    or if you literally only want the number of the count as output, you can use:

    qload -m MQG1 -i Q1 -T05:00:00 -dK
  • Update Put Date and Put Time
  • Normally QLOAD will fill in all the context fields in a message from the file or queue you are loading from. In this version however, if you wish to do that but with a current PutDate and PutTime, you can use the -Ct option.

The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

MQGem Monthly (July 2019)

MQGem Monthly Banner

Welcome to the July edition of the MQGem Monthly Newsletter, bringing you company news from MQGem Software, information about the MO71, MQSCX, MQEdit, Q and QLOAD products, our IBM MQ training modules, and interesting articles about IBM MQ.


At the beginning of July, a new version of MO71 was released with a number of customer requested features. Read more here.

IBM announced, and released IBM MQ V9.1.3 on the distributed platforms this month, with the z/OS version coming in the early days of August. Read more about this release here, and we’re collecting up posts about it here.

This month, IBM MQ has moved to a new Support site to replace the IBM Service Request Tool. More details in the link.

Interesting IBM MQ links

Here are a few links that caught our eye this month.

We share all the above links on our Twitter feed and Facebook page so if you don’t want to wait until the end-of-month newsletter, follow us on there.

Upcoming events

There are a number of conference events where IBM MQ will be featured in the coming months. Hopefully you’ll be able to sign up to one near you and obtain some MQ technical education.

Upcoming online events

If you’re unable to get to any “in-person” events, then these online webcasts are just what you need.

Product Info

This months highlighted product information is a video showing how to simplify the MO71 GUI interface.

IBM MQ Latest Maintenance

Make sure you’re on the latest maintenance level of your version of IBM MQ. These are the latest Fix Packs at the time of going to press. Check here for the latest versions now, and here for the planned dates for the next ones.

Handy IBM MQ Resources

If you have a question about IBM MQ that you can’t find the answer to, these resources are good places to ask your question.

Contact Information

MQGem hopes you are enjoying your summer holidays. We’d love to hear from you. Get in touch by email or follow us on one of our social media channels.

← Previous Newsletter

IBM MQ and MQ Appliance News – July 2019

On Tuesday July 9th, IBM Hursley announced the next in the series of Continuous Delivery releases for IBM MQ V9.1 and the MQ Appliance. IBM MQ V9.1.3 was made available on July 11th, with IBM MQ for z/OS V9.1.3 targeted for August 2nd.

Downloading IBM MQ Version 9.1.3 Continuous Delivery

Here are the announcement letters:-

Links of interest:-

We’ll collect up any other links about the new release as we find them and put them all here.

MO71 User Columns enhanced in V9.1.2

The status output from IBM MQ contains a number of fields which steadily increase as things happen. The table below contains a few examples of these. I’m sure you have used a number of these to determine whether, for example, “is this channel actually doing anything?”, by hitting refresh on a GUI screen or repeatedly issuing an MQSC command to see if the numbers are changing.

Status Field Name MQSC Keyword
Channel No. of messages MSGS
Bytes sent BYTSSENT
Bytes received BYTSRCVD
Buffers sent BUFSSENT
Buffers received BUFSRCVD

In the latest version of MO71, the user columns feature has been enhanced to allow you to turn these fields into a rate field. Here’s how…

I’m going to use the Bytes Sent and No. of Messages status fields on a channel, to determine the transfer rate (bytes/second) and the message rate (messages/second) of my channel. I’ll start with the No. of Messages field. My filter on a Channel Status List will look like this:-

usrcol("Message rate", MSGS, text+diff+rate);TRUE

Filters, as you may already know, are a boolean expression, so just in case my user column has a value of zero (which would result in the row not being displayed), I finish off my filter with “TRUE” just to ensure each row is always displayed, even if there is a zero value.

I have used two new options on my user column – the ‘diff’ option and the ‘rate’ option. These will often go hand-in-hand. The ‘diff’ option instructs MO71 to calculate the difference between this value and the previous value – handy when using these numbers that just steadily increase. If you combined the use of ‘diff’ with the auto-refresh option on list dialogs, you would have some semblance of a rate. However, also adding the ‘rate’ option instructs MO71 to calculate the rate for you using the time period since the last refresh of the dialog (so you don’t have to use auto-refresh).

Note: While most of the time you will likely use ‘diff’ and ‘rate’ together, there are some instances where you might only need one. For example, the output from RESET QSTATS is already showing you the difference from last time, so in that case all you would need is the ‘rate’ option.

I also want the transfer rate, so I’ll add another very similar user column to my filter. Here’s the addition:-

usrcol("Transfer rate", BYTSSENT, text+diff+rate+bytes)

This looks almost identical to my first user column. The one difference is the ‘bytes’ option. This marks the new user column to have a unit of Bytes, and the new Units feature of MO71 kicks in and labels it according to your preferences for units.

Here’s a screenshot of both user columns in action.

MO71 User Column Rates

Channel Status List Dialog with rates in user columns

Another set of status fields output by IBM MQ includes counts of things that, rather than steadily increase, tend to waver around. Some examples of these types of fields are shown in the table below.

Status Field Name MQSC Keyword
Queue Queue Depth CURDEPTH
Open Input Count IPPROCS
Open Output Count OPPROCS
Oldest Message Age MSGAGE

Fields like these, and of course also your rate fields from the first half of this post, can be averaged using new user column features in the latest version of MO71.

To demonstrate, here’s a filter to create a user column showing the average queue depth.

usrcol("Average Depth", CURDEPTH, text+avgl);TRUE

In the same way that IBM MQ indicator fields have a short term and a long term rolling average, MO71 user column averages use a similar concept. You can choose a short term average, ‘avgs’, or a long term average, ‘avgl’, depending on how you want spikes in the numbers to be shown – do you want anomalies to be “forgotten” by the average number fairly quickly, or to affect the average number shown for a longer period.

I’m also going to add another filter to calculate the average message age.

usrcol("Average Age", MSGAGE, text+secs+avgl);

Adding these to my queue status display gives me a way to tell that there had been a spike of messages on my queues, even though everything is all good now, the long term average takes a while to forget about the spike and allows me to see a past problem.

MO71 User Column Averages

Queue Status list dialog with some User Columns displaying averages

Of course, the beauty of these user columns is that you are only limited by your imagination. See what you can come up with.

The new version can be downloaded from the MO71 Download Page. Any current licensed users of MO71 can run the new version on their existing licence. If you don’t have a licence and would like to try out MO71 then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Field Units in MO71

As an MQ administrator you are no doubt painfully aware that not every time based field in MQ is in seconds, and that not every size based field in MQ is in bytes. How many times have you forgotten that expiry is measured in tenths of a second?

The latest version of MO71 has made a number of changes to aid you in your endeavours when reading or entering the many and varied numbers that form part of your queue manager configuration.

Unit Labels on Field and List Titles

The first, and perhaps simplest, aid to your memory is that all fields which have some sort of unit, have this unit appended to the end of the field and list titles. This screenshot shows full length unit names on the labels on the left of the screen, but you also have the choice of short names and abbreviations (more later).

MO71 Unit Field Labels

MO71 dialog showing units on field labels

You can also choose to have these labels on your list dialogs, next to the field titles in the column headings. Again, you can choose between full length unit names, short names or abbreviations. This screenshot shows the short form unit names in the titles.

MO71 Unit List Title Labels

MO71 list dialog with unit labels in the column titles

Unit Labels on Field and List Values

In addition to unit labels on titles, you can also (or instead) have the values annotated with units. There are various different patterns you can choose to display with your values, and you can choose a different pattern for values which represent a size, values which represent a time and percentage values. Each style can be set to “Auto” which allows MO71 to display them the best way. But you can choose other display types – experiment with them and see which suits you best.

The screenshot below shows an example of each:

  • Time fields
    • HB Int: showing the values as minutes which is easier to understand than having to think about 300 seconds
    • Comp Time: showing an indicator which works in microseconds
  • A size field, Bytes sent: showing some values in bytes (B) and some larger values as kilobytes (KB)
  • A percent field, Comp Rate: showing an indicator which works in percentages.
MO71 Unit values

An MO71 list dialog showing a selection of Unit values

Label choices

As noted earlier, you have some choices in the types of labels you can use. The table below shows some examples without providing the full set – the pattern should be clear from this subset.

Unit Type Full Short Abbreviation
Time Seconds Secs s
Size Kilobytes KB KB
The difference between Short and Abbreviation here is the space between the number and the unit
Percent Percent (only on titles) % (used everywhere else)

You can choose exactly which units you wish to be displayed in the various different places in the Units tab of the preferences dialog; including, if you wish, to turn them all off. There is different configuration for normal dialogs, exported reports and HTTP output, as with different uses you may, or may not want units added to these.

MO71 Unit Preferences

The Preferences tab in MO71 to control how units are used

Transient Status fields

When looking at configuration fields, it is probably important to see the full detail. So for example, if you have a queue with a Max Msg Length of 32,762 bytes that will be displayed as 31KB 1018B (that’s 6 bytes short of 32KB). However you could choose to display sizes with the “One Decimal” option, and then it would be shown as 31.99 KB. You’ve lost a little detail here with than choice of display.

However, there are some fields, that we’ve called transient status fields, that are output only, and regularly changing, where such a display would actually be completely appropriate. So, you can choose “Auto” or “Parts” for most of your fields, but set “One Decimal” for your transient fields. The list of fields we consider to be transient are marked in the tables here, which lists all the fields in MO71 that have units.

Entering values

Finally a word on entering values for fields with units. If you have a field that is a time field, like Heartbeat interval, you can type in “300” or “5 mins” and MO71 will interpret it for you and send the appropriate number to the command server. This could be very handy the next time you need to set MaxMsgLength to 100MB – no longer do you need to calculate what that is, just type in “100MB” in the entry field in MO71 and it will do it for you.

You also have a choice whether the entry field shows the formatted version of the value, e.g. “100MB” or the actual number that MQ provides. Select the “Units Entry Field” option in the preferences to choose between the two behaviours.

Here at MQGem, we hope these changes will help in those moments where you are unsure of the unit, and save you from interpreting MQ output in a way that is out by a factor of 1000 – “was that microseconds or milliseconds?”

The new version can be downloaded from the MO71 Download Page. Any current licensed users of MO71 can run the new version on their existing licence. If you don’t have a licence and would like to try out MO71 then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Working with MQ Dates and Times in MO71

The administrative commands in IBM MQ contain a number of date and time fields, for example when a channel was started, or when an object was altered. These are returned as a pair of fields, one for the date and one for the time.

Previously, if you wanted to sort a list of items into date order you could use MO71’s secondary sort to help with this. First click on the date column to sort by date first, and then hold down the Alt key and click on the time column to add that as a secondary sort. You’ll see a big arrow and a little arrow in the column headings to show the primary and secondary sorting fields. You can also see the sorting fields and change them through the “Alter List…” dialog.

MO71 Dates Sorted Two Fields

MO71 queue list dialog sorted into time order using the two MQ provided fields

However, this did mean that you couldn’t do any further sorting because you had used up the secondary sort field on getting the list in time order. The new version of MO71 provides a combined date and time field for each of these MQ field pairs, which means you can sort into time order with a single column heading click, and in addition you can sort on another attribute as well if you need to. Just use the “Alter List…” dialog to switch out the original two MQ date and time fields, for the combined one provided by MO71, and click on the column heading to sort by that new column.

MO71 Dates Sorted Single Field

MO71 queue list dialog sorted using a combined date/time field

There are some other advantages to this combined date/time field. One of these is the format. In the example above you can see the date is formatted a little differently to how MQ provides it. As the MO71 user, you can control the format of this field. You can use a format that isn’t even string sortable and MO71 will still ensure that it is sorted in date order for you (as illustrated by the above screenshot). This means that if your preference is to swap round the day and month, the column will still sort correctly into date order. The format of these combined date/time fields is controlled in MO71 Preferences, under the Time tab. You can choose from a number of provided formats from the “Time Sample” drop down, or edit the “Time Format” field to create your own using the various substitutions available. To make recent activities pop-out more from a list, you can have additional formats for things that happened today and yesterday, or in the last 24/48 hours, by adding formats to those fields too. For example, to omit the date or add the string “Today” instead of the date. Whatever catches your eye the best.

MO71 Dates Preferences Formatted Time

MO71 Preferences, Time Tab allows you control of the Formatted Time

Viewing multiple Queue Managers

We are all aware that the times stored in MQ messages are in UTC, but all the other time fields in the MQ administrative interfaces are returned in the local time of the queue manager. Some of you are lucky enough to work in the same time zone where your queue managers reside, but for the rest, you need to be aware of the time zone of the queue manager when you are viewing data that includes dates and times. Looking at the screenshot below which shows three international queue managers in different time zones, you might initially be forgiven for thinking that they had all been started at the same time.

MO71 Dates Queue Manager List Start Times

An MO71 Queue Manager List showing Start Times

However, this assumption does not take into account their timezones. In the new version of MO71, with the combined date/time field, you are able to see any of the date and time fields returned by the queue manager converted from the known timezone of the queue manager into your own local timezone. IBM MQ does not have a way for a remote administrator to know what timezone a queue manager is in, so on the MO71 Location Dialog you can select a timezone for your queue manager if it is different to your own timezone.

MO71 Dates Queue Manager Timezone

MO71 Location dialog – set the Queue Manager Timezone

Once you have set the timezone, you can see these new combined date/time fields taking into account their timezone, and the above display can now look as follows. In addition this screenshot illustrates that the chosen timezone from the location dialog can be added to this display which acts as a handy reminder of where this queue manager is, especially if your queue manager names don’t make it as obvious as my example ones do!

MO71 Dates Queue Manager List Timezone

An MO71 Queue Manager List showing the Timezone of the QMgr and start dates converted to local time.

With this mechanism it is now very simple to sort date/time fields across multiple queue managers so that they are displayed in the order that things happened, rather than just the time that it happened in that location.

The relative times that events happen can be important to notice when looking for cause and effect in problem diagnosis. This new version of MO71 now makes this easy to do, even across multiple time zones.

The new version can be downloaded from the MO71 Download Page. Any current licensed users of MO71 can run the new version on their existing licence. If you don’t have a licence and would like to try out MO71 then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.