QLOAD: Using delimited files

The two most recent versions of QLOAD have added a feature to allow you to upload messages onto queues using QLOAD even when the file is not in the standard QLOAD format (i.e. it wasn’t from a QLOAD unload).

UploadDelimitedFileIn fact, the file can be in almost any format you want, so long as it is possible for you to describe where the messages are in the file.

QLOAD can now use files containing message data where each message is delimited from the next message in some way. This could mean that there are tags, or separator characters between the messages or even, the simplest example, that each message is on a new line.

Message text 1
Message text 2
Message text 3

msgs.txt: A file with one message on each line

To load the above file onto a queue, with one message taken from the text on each line, you would use the following QLOAD invocation.

qload -m MQG1 -o Q1 -f msgs.txt -Cd -nn

You haven’t specified any delimiters on this command, and that is because QLOAD assumes that new line is the delimiter you want if you don’t say otherwise. The -n switch is what says you want to parse a delimited file, and saying -nn tells QLOAD not to include the delimiters (in this case the new line character) as part of the message text. You could be explicit about the delimiter in use with the following command.

qload -m MQG1 -o Q1 -f msgs.txt -Cd -nn -ne:"\n"

In our next example, we have a file with some character separators.

message1/message2/message3/message4/

data.txt: A file with single character separators

To load the above file onto a queue, with one message taken from the text on each line, you would use the following QLOAD invocation. Note that you must have the final character delimiter for all four messages to be found.

qload -m MQG1 -o Q1 -f data.txt -Cd -nn -ne:/

Helpfully, QLOAD will output a message to tell you how many messages it has found with a prompt before you continue, to help you spot when it hasn’t found all the messages you hoped it would find.

4 messages found in the file. Do you want to continue? (y/n)

If you have explicit start and end delimiters, then you can start to jump past data in the file that is not part of a message, which is handy for commentary. For example with the following file:-

Here are a few messages
[message1][message2]
and then here are a few more
[message3]
[message4]
[message5]

info.txt: A file with start and end delimiters, and comments

To load the above file onto a queue, ignoring the comments and only pulling out the message data between the square brackets, you would use the following command.

qload -m MQG1 -o Q1 -f info.txt -Cd -nns:[ -nne:]

There is also a short-hand you can use when you have delimiters of the same length, as shown in the next command.

qload -m MQG1 -o Q1 -f info.txt -Cd -nnd:[]

Delimiters don’t have to be single characters, nor do they have to be excluded from the message. You may well have message content that is naturally delimited, for example XML messages. The outside tag of the structure of your XML message may well be all you need as a delimiter.

<message>
  <date>12 Jun 2019</date>
  <priority>High</priority>
  <department>AG75</department>
  <code>HT7228</code>
</message>
<message>
  <date>29 Aug 2019</date>
  <priority>Medium</priority>
  <department>AL23</department>
  <code>LL3460</code>
</message>

xml.txt: A file with XML message data

A file like this where the delimiter is part of the message data can be loaded with the following command.

qload -mMQG1 -oQ1 -f xml.txt -Cd -ni -ns:"<message>" -ne:"</message>"

In this case you cannot use the -nd flag because the delimiters are not of the same length, and you must specify each one individually using the start (-ns) and end (-ne) flags.

If you need to have some message properties added to your message when it is uploaded, this data can also be found are delimited text in your file. Looking again at the XML data example, here we have added some property data to the file.

<usr>
  <importance dt='i4'>1</importance>
</usr>
<message>
  <date>12 Jun 2019</date>
  <priority>High</priority>
  <department>AG75</department>
  <code>HT7228</code>
</message>
<usr>
  <importance dt='i4'>2</importance>
</usr>
<message>
  <date>29 Aug 2019</date>
  <priority>Medium</priority>
  <department>AL23</department>
  <code>LL3460</code>
</message>

xml2.txt: A file with XML message data and message properties

To load this file onto a queue, you now need to provide two sets of delimiters so that QLOAD can know where the message data is and where the property data is.

qload -m MQG1 -o Q1 -f xml2.txt -Cd -nis:"<message>" -nie:"</message>" -nip:"<usr>" -niq:"</usr>"

While it is likely that you will use this second set of delimiters for message properties, that is the <usr> folder in an MQRFH2 header, in fact you can use it to construct any MQRFH2 folder that you require since you provide the complete folder structure.

The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

QLOAD version 9.1.1 is released

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

The main features of the release are as follows:-

  • Non-QLOAD file can contain property data
    Extending the feature introduced in QLOAD V9.1.0, this allows your delimited file to contain a second set of delimiters to mark where property data to be added to the message is to be found. Read all about this feature here.
  • Fully qualified queue name separator character
    If your queue names contain one of the default separator characters that QLOAD looks for, you can now specify exactly which character should be considered a separator to avoid confusion.
    For example to put messages to a queue called Q1/SALES on queue manager MQG2:-

    qload -m MQG1 -f data.qld -$# -oMQG2#Q1/SALES

    or to avoid QLOAD thinking that you want to put messages to a queue called SALES on queue manager Q1, when actually your queue is called Q1/SALES:-

    qload -m MQG1 -f data.qld  -$#-oQ1/SALES
  • Backout queue
    This version of QLOAD will pay attention to the backout threshold (BOTHRESH) and re-queue name (BOQNAME) you have defined on any input queues, and re-queue any messages with an MQMD.BackoutCount that meet or exceed the threshold.
  • Application Name
    If your IBM MQ version is appropriate, QLOAD will set an application name of ‘MQGem Software QLOAD’
  • Summary Display
    The summary display is extended in this version to show more age and size ranges in its output. For example:

    Message Ages
    ============
    
    < 2  secs < 10 secs < 1   min < 1  hour < 1   day < 1  week
    --------- --------- --------- --------- --------- ---------
            0         0         0         0         0         2
    
    < 2 weeks < 4 weeks < 3 month < 6 month  1  year
    --------- --------- --------- --------- --------- ---------
            2         2        13       110         0         0
    
    Youngest Message : 4 Days 2 Hrs 29 Mins 28 Secs
    
    Oldest   Message : 19 Weeks 3 Days 3 Hrs 26 Mins 7 Secs
    
    Message Sizes
    =============
    
    < 100b <  1Kb <  4Kb <100Kb <  1Mb < 10Mb  50Mb
    ------ ------ ------ ------ ------ ------ ------ ------
         1    128      0      0      0      0      0      0
    
    Smallest Message : 4 bytes
    
    Biggest  Message : 368 bytes
  • Counting mode
    Sometimes you might want to simply discover how many messages are on your queue that meet a certain criteria. For example, how many are over a certain age, or how many contain a certain string of text. Older versions of QLOAD could be used to do such a count with a little bit of manipulation using a command like the following:-

    qload -m MQG1 -i Q1 -T05:00:00 -f null

    New in this version, you can specifically request a count of the messages as follows:

    qload -m MQG1 -i Q1 -T05:00:00 -dk

    or if you literally only want the number of the count as output, you can use:

    qload -m MQG1 -i Q1 -T05:00:00 -dK
  • Update Put Date and Put Time
  • Normally QLOAD will fill in all the context fields in a message from the file or queue you are loading from. In this version however, if you wish to do that but with a current PutDate and PutTime, you can use the -Ct option.

The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

MQGem tools on Linux on Little Endian Power

MQGem is pleased to announce the availability of three of its products on the Linux on Little Endian Power platform.

We have had these products available on Linux on Power for a number of years, and now have added the Little Endian platform to the list.

Thanks to those customers who requested it.

You cannot run a Big Endian application on a Little Endian platform and vice versa. Ensure you download the appropriate version of the tool for the platform you are using. For example, if you try to run the Big Endian version on a Little Endian platform, you will see an error something like this:-

-bash: ./mqscx: cannot execute binary file

If you are not sure which platform you are using, try one of the following commands:-

lscpu

which will show you output similar to:

Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                ...

Or if you have an older version of Linux, which doesn’t provide that command, try:-

uname -a

which might give you hint.

Linux gemlnx4.mqgem.com 3.10.0-327.36.3.el7.ppc64le #1 SMP Thu Oct 20 04:58:48 EDT 2016 ppc64le ppc64le ppc64le GNU/Linux

Extra Reading Material


The new versions can be downloaded from the website from the links given above. Any current licensed users of these products can run them on the new platform using their existing licence. If you don’t have a licence and would like to try out any of our products then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

QLOAD version 9.0.3 – Reading messages from the log

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

This release was created for one specific customer requested feature, retrieving messages from your queue manager log.

IBM MQ comes with a utility, dmpmqlog, which can format your queue manager transactional log and dump out the contents of messages in a hex format (example below snipped for brevity).

LOG RECORD - LSN            
**********

HLG Header: lrecsize 720, version 1, rmid 0, eyecatcher HLRH

LogRecdType . . : AQM Put Message (257)
Eyecatcher  . . : ALRH                 Version . . . . : 1
LogRecdLen  . . : 700                  LogRecdOwnr . . : 256    (AQM)
XTranid . . . . : TranType: NULL
QueueName . . . : HELLO.WORLD.Q                               

Data  . . . . . : 
00000:  41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF    AQRH....ÿÿÿÿÿÿÿÿ
:
00160:  4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00    MD  ............
00176:  00 00 00 00 22 02 00 00 52 03 00 00 4D 51 53 54    ...."...R...MQST
00192:  52 20 20 20 00 00 00 00 01 00 00 00 20 20 20 20    R   ........    
:
00368:  20 20 20 20 20 20 20 20 0B 00 00 00 43 3A 5C 6D            ....C:\m
00384:  71 6D 38 30 30 34 5C 62 69 6E 36 34 5C 61 6D 71    qm8004\bin64\amq
00400:  73 70 75 74 2E 65 78 65 32 30 31 38 30 34 32 33    sput.exe20180423
00416:  30 34 33 34 33 34 35 36 20 20 20 20 00 00 00 00    04343456    ....
00432:  00 00 00 00 00 00 00 00 00 00 00 00 FF FF 00 00    ............ÿÿ..
00448:  48 65 6C 6C 6F 20 57 6F 72 6C 64 21                Hello World!

Log To QueueThis format is a little tricky to extract the data from to reinstate your message. As you can see above, you get the MQMD and the message data all in one block. It gets even more tricky when your message is broken up into several log records!

This version of QLOAD hopes to make the process of recreating message from your transactional log files much simpler to do. It can only work when the messages you want are in the log file though. There’s no magic!

Running QLOAD against the queue manager log

With the following invocation, QLOAD will run the dmpmqlog utility and parse through the output looking for the queue you requested, in this example HELLO.WORLD.Q, and write any messages it finds to the HELLO.WORLD.RCOV queue on queue manager MQG2.

qload -m MQG1 -j* -i HELLO.WORLD.Q -m MQG2 -o HELLO.WORLD.RCOV

Since you are reading the log files using dmpmqlog, the queue manager, MQG1, cannot be running when this command is issued and so the messages cannot be put onto a queue on that queue manager as part of the command. In this example I’ve chosen to put them onto a side queue on another queue manager but equally I could write them into a QLOAD file and then later put them onto the queue manager once it has started up again, as follows:-

qload -m MQG1 -j* -i HELLO.WORLD.Q -f HELLO_WORLD.qld
strmqm MQG1
qload -m MQG1 -f HELLO_WORLD.qld -o HELLO.WORLD.RCOV

Running QLOAD against output from dmpmqlog

If you have already run dmpmqlog, perhaps to determine whether the messages you wish to retrieve can be found in the log, then you may have already spent a few minutes waiting for the utility to complete. You can use the output files you generated from the utility as input into QLOAD.

dmpmqlog -m MQG1 -b > dmpmqlog.txt
qload -j dmpmqlog.txt -i HELLO.WORLD.Q -m MQG1 -o HELLO.WORLD.RCOV

Running QLOAD against copied off queue manager logs

A third alternative is to run QLOAD against queue manager log files that have been copied off. In this case you provide QLOAD with a directory containing these files.

qload -m MQG1 -j c:\mqmarch\log\MQG1 -i HELLO.WORLD.Q -m MQG1 -o HELLO.WORLD.RCOV

This command drives dmpmqlog with the -f parameter, and so the criteria for this directory must follow what is described in Knowledge Center for running dmpmqlog in that way. In addition, you must use the -m parameter to specific the queue manager name that these log files came from so that queue names can be found, and that queue manager cannot be running at the time.

The specified directory must contain the log header file (amqhlctl.lfh) and a subdirectory called active. The active subdirectory must contain the log files. By default, log files are assumed to be in the directories specified in the IBM MQ configuration information. If you use this option, queue names associated with queue identifiers are shown in the dump only if you use the -m option to name a queue manager name that has the object catalog file in its directory path.

QLOAD filtering

Whichever of the above methods you choose to use, you can add many of the QLOAD filtering options to your command.

For example, you can find all the messages containing a particular search string.

qload -m MQG1 -j* -i HELLO.WORLD.Q -s "World" -m MQG2 -o HELLO.WORLD.RCOV

Or perhaps you need to pluck out one specific message using its message id (or correlation id).

qload -m MQG1 -j* -i HELLO.WORLD.Q -gxm414D51204D51473120202020202020202604E15A21D49501 -m MQG2 -o HELLO.WORLD.RCOV

QLOAD also has time based filtering, so if you’re looking for a message from hours, days or weeks ago, you can go directly to that time period. This example is looking for messages that were put onto the queue between 18 and 19 hours ago.

qload -m MQG1 -j* -i HELLO.WORLD.Q -T0:18:00,0:19:00 -m MQG2 -o HELLO.WORLD.RCOV

Note: Filtering by Selection String (the -H parameter) is not available when your input source is a file (either a QLOAD file or an MQ log file) since it uses the Selector feature of MQOPEN.


The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Use MQGem tools with IBM MQ on IBM Cloud

You may have seen the recent announcement from IBM about the experimental IBM MQ service running on IBM Cloud.
IBM Cloud

You can learn more with these resources:-

When you read the description of this new service in the Bluemix catalog, you’ll see it says the following:-

Manage MQ your way

Manage your cloud-based queue managers with the tools you know and love – including MQ Explorer, the MQ Console, or via MQ Script Commands (MQSC).

This blog post is here to assure you that the tools you know and love from MQGem Software; MO71, MQSCX, MQEdit and QLOAD can all also be used with IBM MQ running on IBM Cloud.

Once you have created your MQ on IBM Cloud Service and Queue Manager, as shown in the above video, and your queue manager is up and running, you’ll have a view something like this.

IBM Cloud QM List

A list showing my Running queue manager

Click on the three vertical dots on the right of your queue manager to “Download Connection info”, or alternatively view the details of your queue manager and then there is a button there too which allows you to download the “Connection Information”. Either way, you’ll be presented with a pop-up which allows you to download a plain text file which contains the queue manager name, hostname, port number and a couple of channel names, one called an Application Channel and one called an Administration Channel.

IBM Cloud Download Connection info

IBM Cloud Download Connection info

In order to remotely connect to your IBM Cloud queue manager you will also need to be able to log in. As the MQ on IBM Cloud documentation describes here, you need the API key as your password to go with the user id ‘admin’. Follow the instructions on that page to obtain your API key.

Now you have all the pieces of information you need to set up any of the MQGem Software tools to administer your queue manager on IBM Cloud. As a reminder, these are the things you will need.

Item From where
Queue Manager Name You invented it when you created the queue manager. If you’ve forgotten it, it’s also in the text file you downloaded with the connection information.
Channel Name The “Administration channel name” can be found in the text file you downloaded with the connection information.
Connection Name This is built by concatenating the “Hostname” and “Listener port” details (with brackets round the port number) that can be found in the text file you downloaded with the connection information.
User ID This is ‘admin’
Password This is the API Key that you created by following the linked instructions.

On the pages that follow, we cover how to use the above information you have gathered in your table to configure each of our tools to connect to your IBM MQ in IBM Cloud Queue Manager. Go directly to the page for the tool you want to use, or page through each one in turn.

  1. MO71
  2. MQEdit
  3. MQSCX
  4. QLOAD

If you don’t have a licence and would like to try out any of our tools then send an email, noting which tool you’d like to try, to support@mqgem.com and a 1-month trial licence will be sent to you.

QLOAD version 9.0.2 – Required Rate

MQGem Software is pleased to announce that a new version of QLOAD, our queue load/unload tool for IBM MQ, is now available.

This release was created for one specific customer requested feature which we call Required Rate Processing.

Required Rate

QLOAD Source to TargetThis feature allows you to test the speed that your system, or specific applications in your system, can process messages. You may have a target rate you need to be able to achieve, and you can use QLOAD to send messages through the system at that rate to see whether the system (the channels and the applications) can keep up. You may for example have TLS channels; or message compression; or AMS message level encryption applied to the messages which are of course, changing the performance profile of the system. With QLOAD you can test the rate using exactly the same shape and size of messages that you will use in the production environment.

Starting with a queue that contains a representative selection of example messages, you can then use QLOAD to copy those messages and inject them into the system at the rate you require. In the example that follows, the required rate is 500 messages/second and the test is to run for 10 minutes.

qload -m MQG1 -i EXAMPLE.MSGS -o TARGET.QUEUE -R500:600

QLOAD will browse the EXAMPLE.MSGS queue and send copies of those messages to the TARGET.QUEUE. When it reaches the end of the EXAMPLE.MSGS then it will go back to the beginning of the queue and send them again, as often as required until the test ends. You can just have a single message on the queue and send it repeatedly, but to offset the cost of resetting the browse, it’s better to have 10 or more messages on the queue. In reality all your messages will not be the same, so a good selection of example messages is certainly worth collecting.

If you want the test to run until you tell it to stop, you can indicate that you want it to run indefinitely with an asterisk for the duration.

qload -m MQG1 -i EXAMPLE.MSGS -o TARGET.QUEUE -R500:*

There are two other ways you might use this feature.

Discover Achievable Rate

Instead of telling QLOAD to run at a certain rate, you might instead want to discover what your highest rate could be.

To run QLOAD in this way, you use a command like this:-

qload -m MQG1 -i EXAMPLE.MSGS -o TARGET.QUEUE -R*:*

And you will see output like the following, as QLOAD ramps up the rate.

Monitoring depth of queue 'TARGET.QUEUE'
Message Rate 802 msgs/sec
Message Rate 5200 msgs/sec
Message Rate 5548 msgs/sec
Message Rate 5516 msgs/sec
Message Rate 5409 msgs/sec
Message Rate 5566 msgs/sec
Message Rate 5503 msgs/sec

It does this by monitoring the depth of the target queue, which might be a transmission queue being read by a channel, or it might be an application queue being read either by the read application or by another instance of QLOAD acting as a sink.

qload -I TARGET.QUEUE -f null -w600

Trickle Feed the contents of a queue

So far you’ve seen QLOAD copy example messages from a source queue and inject them into the system over and over until the duration of the test is complete. Alternatively though, you could use this feature to move messages from a side queue (the contents of a batch run, or messages from a backout queue or a Dead-letter queue) back into the main system. If you have a large depth of these messages you may not want to load them back into the application queue all at once, since you know deep queues hurt performance. So instead you can trickle feed them into the system using QLOAD.

qload -m MQG1 -I SIDE.QUEUE -o TARGET.QUEUE -R50

The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.

Migrating a Queue Manager?

If you’re planning to migrate a queue manager, from one machine to another, or perhaps you’re consolidating some queue managers onto an MQ Appliance, you are probably aware of, and have even become familiar with, the steps required to export your object definitions and recreate them on the new queue manager. There are a number of tools available that can do this for you – including those from MQGem.

What about your messages though?

While it’s generally a good idea to reduce the load, and drain off as many messages as you can prior to migrating a queue manager. For example if it’s in a cluster, suspend it from the cluster before the move, to reduce the number of messages heading its way. Let all the applications drain the queues of messages and so on. However, it’s not always possible to drain every last message from all your queues.

QLOAD Offload all queuesIn the latest version of QLOAD, V9.0.1, there’s a new feature that will help out when migrating a queue manager. It allows you to unload the messages from all your queues, with one command. And then you can load the messages onto the queues on another queue manager, with one command (or piecemeal if you prefer). You would combine this with your favoured object definition export tool, and before loading the messages, you’d first recreate the queues on the new queue manager with the exported commands.

Here’s an example of QLOAD unloading all the queues on my queue manager.

qload -m MQG1 -i* -f*

The command will produce the following output to show what has been unloaded. Using the -i (lower case) flag means that the messages are only browsed on the queues and are not destructively removed.

APP1.INPUT                                           2  Done.
APP1.RESULT                                         10  Done.
APP2.INPUT                                           1  Done.
APP2.RESULT                                          4  Done.
Q1                                                  28  Done.
Q2                                                  42  Done.
SYSTEM.ADMIN.QMGR.EVENT                              7  Done.
SYSTEM.AUTH.DATA.QUEUE                             126  Done.
SYSTEM.CHANNEL.SYNCQ                                 3  Done.
SYSTEM.CHLAUTH.DATA.QUEUE                            5  Done.
SYSTEM.CLUSTER.REPOSITORY.QUEUE                      3  Done.
SYSTEM.DURABLE.SUBSCRIBER.QUEUE                      1  Done.
SYSTEM.HIERARCHY.STATE                               2  Done.
SYSTEM.INTER.QMGR.FANREQ                             1  Done.
SYSTEM.RETAINED.PUB.QUEUE                            2  Done.
WORK.REQUEST                                         7  Done.

Total : 7 Queues, 94 Messages
 plus : 9 System Queues, 150 Messages

Listing the directory where I ran the qload command, I can now see I have a file for each queue, with an extension .qld. If you prefer to have a different extension then you can alter the command accordingly. For example, use -f*.txt.

I can copy these files to another machine, or simply use a client connection to the other machine accordingly. Then I can run the following QLOAD command to load the messages onto the new queue manager.

qload -m MQG1 -o* -f*

QLOAD will make some checks when you run a load for multiple queues in this way. It will check that all the queues exist that it has files for in the directory matching the file pattern you specified (which assumes an extension of .qld if you just use ‘*’), and it will check that all those queues are empty. If it finds any problems it will report as follows:-

APP1.RESULT                            RC(2085) Unknown object name.
APP2.RESULT                            Not empty.

There are potential problems with these queues.
Are you sure you want to continue?

This lets you know to go and define the missing queues if you need them – there are messages to go onto them so the assumption is that you do need them. It also warns you of non-empty queues. Now if you’ve already started using this queue manager, you might be expecting this situation, but otherwise, you should rectify it, and then re-run QLOAD.

APP1.INPUT                                       2     Done.
APP1.RESULT                                      10    Done.
APP2.INPUT                                       1     Done.
APP2.RESULT                                      4     Done.
Q1                                               28    Done.
Q2                                               42    Done.
WORK.REQUEST                                     7     Done.

Total : 7 Queues, 94 Messages

You’ll notice that this output does not mention the SYSTEM queues that were offloaded. A generic upload will not upload most SYSTEM queues.
SYSTEM.CLUSTER.TRANSMIT.QUEUE and SYSTEM.DEAD.LETTER.QUEUE are the exceptions. The messages were unloaded to files though, so if you really need to load them you can do so the traditional way, by specifying the full queue name on the command and not using the generic upload.


The new version can be downloaded from the QLOAD Download Page. Any current licensed users of QLOAD can run the new version on their existing licence. If you don’t have a licence and would like to try out QLOAD then send an email to support@mqgem.com and a 1-month trial licence will be sent to you.