{"id":1007,"date":"2024-04-04T12:40:21","date_gmt":"2024-04-04T02:40:21","guid":{"rendered":"https:\/\/www.infrasoft.com.au\/?p=1007"},"modified":"2024-04-04T14:35:08","modified_gmt":"2024-04-04T04:35:08","slug":"ulingas-new-data-transformation-capabilities-including-iso8583-support","status":"publish","type":"post","link":"https:\/\/www.infrasoft.com.au\/?p=1007","title":{"rendered":"uLinga’s New Data Transformation Capabilities – Including ISO8583 Support"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Powerful new feature provides extensive data manipulation options<\/h3>

uLinga for Kafka, widely recognised as the best solution for integrating NonStop applications with Kafka, and uLinga Gateway, the newest product in the uLinga suite, have recently been enhanced to add powerful data transformation capabilities.\u00a0 This article gives an introductory overview to these capabilities, and then takes a deep dive into one possible use case \u2013 transforming the venerable ISO8583 standard into JSON.<\/p>

uLinga Gateway – Introduction<\/strong><\/p>

uLinga Gateway, recently added to the uLinga suite, provides a wide range of network protocol handling and message manipulation services that can be used in a number of applications and use cases.\u00a0 uLinga Gateway can handle any combination of TCP\/IP, TLS, HTTP, Pathway, Guardian IPC and more, allowing you to connect almost any NonStop application with any endpoint via simple configuration.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Now, with Data Transformation, uLinga product capabilities are extended to allow easily configured data transformation and message manipulation.\u00a0\u00a0 uLinga\u2019s Data Transformation can handle any combination of binary and JSON messages for any input or output messages, with additional message formats in development.<\/p>



<\/p>

\u00a0<\/p>

Use Case – ISO8583 Data Transformation<\/strong><\/p>

\u00a0<\/p>

The primary focus of this article is a very common use case in NonStop circles, involving messages using the ISO8583 standard.\u00a0 ISO8583 has been in very wide usage in the NonStop financial transaction processing space for decades, courtesy of its broad adoption between financial institutions, switches and card brands such as Visa and Mastercard.\u00a0 While ISO8583 is undoubtedly powerful and very capable in this space, it is not a simple protocol, and developing new applications to support this standard can be a challenge.<\/p>

uLinga\u2019s Data Transformation now supports ISO8583 messages, and can transform them into a format more easily supported by external entities, such as JSON, allowing REST endpoints to access traditional transaction processing mechanisms.<\/p>

ISO8583 messages have a number of aspects that can take some time to learn and understand, including bitmaps and variable length fields.\u00a0 A standard ISO8583 message looks something like this:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A common use case for this type of data streaming is to send application logs (such as a BASE24 Transaction Log File, or TLF) to Kafka.\u00a0 BASE24 usually has 2 or 3 TLFs active at any point in time; uLinga for Kafka supports this via FILEREADER configuration.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The Message Type Indicator (MTI) indicates what type of message this is, and bitmaps are used to indicated which fields are present in the message.\u00a0 The fields themselves can be fixed format, or variable with a two- or three-digit length (LLVAR and LLLVAR). The entire message can be in ASCII or in binary\/binary-coded decimal.\u00a0 There is a good primer on ISO8583 message formats available in Wikipedia<\/a>.<\/p>

An example use case would be where a NonStop application processes incoming ISO8583 requests to allow access to the financial transaction processing services it provides (for instance, balance enquiries).\u00a0 Traditional clients would access this service using ISO8583 requests sent directly into the NonStop, as depicted below.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The JSON request does not need bitmaps, and variable length fields are handled simply through the JSON formatting \u2013 no need for LLVAR and LLLVAR fields.\u00a0 uLinga Gateway adds these elements as part of building its ISO8583 request.\u00a0<\/p>

A similar transformation takes place with the response, allowing the client to once again deal with simple JSON.<\/p>

uLinga Gateway\u2019s Data Transformation GUI (built into the WebCon UI) allows you to easily build your data transformation, mapping fields from input to output via drag and drop.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Message Queuing<\/strong><\/p>

Kafka can be used for message queuing, and can provide a very high performance alternative to traditional message queuing solutions.\u00a0 On the NonStop, uLinga for Kafka can facilitate message queueing implementations, by virtue of the range of APIs it supports, as well as its \u201cproduce\u201d and \u201cconsume\u201d functionality.\u00a0<\/p>

One way message queuing, from NonStop to Kafka (and from their to other platforms and applications) can be achieved in much the same way as already outlined in \u201cData Streaming\u201d above.\u00a0 One way message queuing from other applications to Kafka (and from there to NonStop) via uLinga for Kafka requires ULK\u2019s consumer functionality.\u00a0 uLinga for Kafka can monitor one or more topics in a Kafka cluster, consuming (reading) new messages as they are posted to the topic.\u00a0 Once consumed, they can be presented to a NonStop application via a range of mechanisms \u2013 via an interim file, or directly presented via Guardian IPC, TCP\/IP or HTTP communications.\u00a0 ULK\u2019s consumer functionality looks like this:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t
Figure 4 - uLinga for Kafka Streaming EMS Events to Kafka<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

uLinga Gateway Data Transformation can also transform other types of binary messages into JSON, and even convert between formats \u2013 binary to binary or JSON to JSON for instance.\u00a0 When combined with uLinga Gateway\u2019s protocol transformation capabilities you have an extremely powerful tool to address virtually any application connectivity requirement.<\/p>

\u00a0<\/p>

\u00a0<\/p>

\u00a0<\/p>

uLinga for Kafka and Data Transformation<\/strong><\/p>

uLinga for Kafka, which is now in use at a number of HPE NonStop installations worldwide, has a broad range of options for integrating NonStop applications and data with Kafka.\u00a0 Some of the key features of uLinga for Kafka include:<\/p>