This article is a detailed tutorial on leveraging Boomi’s Integration Platform as a Service (iPaaS) to resequence out-of-order messages coming from Google Cloud Pub/Sub with the help of an In-Memory Data Grid. While there are many use cases of why you may want to integrate with the Google Cloud Platform, this article focuses on a particular use case that we run into often where you need guaranteed delivery of messages in the order in which they were generated. We leverage the Boomi platform with Hazelcast to achieve the reordering and then publish the messages to a JMS queue.
Now that you know what to expect from this article, let me give you some background before diving into the detailed steps to accomplish this tutorial’s objectives.
Background: Boomi Platform, Boomi Integration Process & Google Cloud Pub/Sub
Boomi Integration Platform
Boomi Integration Platform is a market leader in Gartner’s Enterprise iPaaS Magic Quadrant for seven years with end-to-end capabilities, including data discovery, application and data integration, API management, data quality governance, B2B integration, low code app development, and more. From an integration perspective, it provides connectors and tools to implement and access different phases of the integration lifecycle — building, deploying, testing, and managing processes with their dependent components.
Boomi Integration Process
A Boomi Process represents a business process or transaction-level interface between two endpoints/systems that may contain a series of shapes connected to illustrate the steps to transport, route, or manipulate data from source to destination. The structure and format of Boomi data are represented as Profile. Boomi supports a broad range of connectors, but in this use case, connectors that have been used are Google Cloud Pub/Sub, JMS, and HTTP for integration.
For Google customers, Boomi delivers an easily scalable integration platform to manage, move, orchestrate, and govern data across applications, whether it’s on-premise or in the cloud.
Google Cloud Pub/Sub
Google Cloud Pub/Sub is an asynchronous messaging service that decouples services that produce events with the services that process events. Google Pub/Sub can be used as messaging-oriented middleware or event ingestion and delivery for streaming analytics pipelines. It also offers durable message storage and real-time message delivery with high availability and consistent performance at scale.
However, there’s a limitation in distributed messaging systems: it does not guarantee in-order or FIFO (first-in-first-out) delivery. Yet, in many real-time business use cases, our customers need the sequential order messages as it originated from the data source. With the help of Boomi and Data Grid, we can resequence message order and can publish to any target endpoints. I came up with the following solution to handle the above scenario by considering a real-time use case.
Resequencer rearranges out-of-sequence messages back into the order. In this use case, incoming message streams (random order) are received from Pub/SUB, and then Boomi sends them to the target endpoints in an orderly manner.
Introducing Our Use Case: Handling Purchase Order Events From A Point Of Sale System
A point of sale (POS) is where customers make their payments for products or services at the store. Every time a customer makes a purchase, they’re completing a POS transaction.
Purchase Order events are published from several POS Systems to the POS Server. These events need to be combined so that the target user/service will receive a complete Purchase Order. Also, we need to note that events received by POS Server may not be in the same sequence as they were created due to system processing time or network delay.
PO events are published to Google Cloud Pub/Sub to simulate the above scenario. For each PO, a source sends a series of events (XML messages with EventType as STARTPO,ENDPO,LINEITEM). Also, the PO ID of all event messages belonging to a Purchase Order are the same, and ENDPO event also contains the total count of Line Items for that Purchase Order.
Boomi Platform provides Google Cloud Pub/Sub Connector to receive and send messages with PUBLISH/PULL/ACKNOWLEDGE operations. Boomi sends all messages to Data Grid via a REST API call and later publishes a complete Purchase Order into JMS with the same order sequence as they originated from the source. Here Resequencing of messages is based on PO ID and SEQUENCE ID of each line item of an event.
Without further ado, let us move on to the actual hands-on tutorial.
Tutorial: Using the Boomi Integration Platform to resequence out-of-order messages coming from Google Cloud Pub/Sub
Prerequisites
Before you start the Boomi integration, keep the following requirements ready:
- Boomi account
Create your Boomi Platform account https://platform.boomi.com/#login; - Google Cloud account
Create your Google Cloud account https://cloud.google.com/gcp, create a project, configure IAM, create a topic and subscriber in Pub/Sub - JMS
Install and Configure JMS - Data Grid
Install and configure any Data Grid
Note: ActiveMQ (as JMS) and HazelCast (as Data Grid) have been used in this use case. Visit http://activemq.apache.org/ and https://hazelcast.com/ for installation.
Data Grid
Several Data Grid tools are available like HazelCast, Infinispan, Ehcache, etc. For this use case, I used HazelCast, an open-source in-memory data grid (IMDG) based on Java. Right now, there is no connector available in Boomi Platform to access IMDG, hence a REST Layer has been created to access Data Grid. (Hopefully, we’ll have a Data Grid Connector available soon in the future.)
The following is a snapshot of REST APIs created that Boomi used to access Data Grid.
Create the Boomi Process
Create a new process in Boomi Platform and add a Connector Shape, choose Google Pub/Sub as Connector Type, PULL as Action Type, then create Connection and Operation as follows:
Provide the Google Cloud project name that you created in your Google Cloud Account.
Provide OAuth 2.0 information which includes Grant Type, Client ID, Client Secret, and Scope in OAuth2.0 tab.
Test your connection
Now, create an Operation in Boomi Connector Shape and Save it.
Save your connection details after adding Connection and Operation.
Your Boomi Process should now look like this:
Update your Boomi process as follows.
Add Process Route to identify the Type of inbound message and redirect it to corresponding Sub Processes. Creating a Sub Process is discussed later in this section.
Configure Process Route for STARTLINEITEM/LINEITEM/ENDLINEITEM
Add new Connector Shape and provide Type as Connection, Component Name, Folder and select Connector as JMS.
Set JMS Connection with the following details:
Create JMS Operation and set destination with the Queue Name prefix with “dynamicQueues/”.
Note: Here Test is the queue name that will be created in ActiveMQ dynamically.
Create a sub-process to process STARTPO Events.
To access hazelCast REST API, add HTTP Client Shape.
Note: HazelCast Connector is not available hence a RESTful web service has been developed with Spring boot framework, which is internally called HazelCast APIs.
Similarly, create two more Http Client connectors for Cache PO and Send Complete PO as mentioned in the Start Line Item SubProcess.
Cache PO will persist Events into HazelCast whereas Send Complete PO will publish complete PO from DataGrid into JMS.
Create 2 more Boomi sub-processes:
LineItem SubProcess (process LINEITEM Events)
EndLineItem SubProcess (process ENDPO Events)
configure these processes with Process Route.
Write to us if you want in detail implementation of the above Sub-Processes.
Test Process
Before testing our Boomi Process first publish some sample data into Google Cloud Pub/Sub.
You can view your data in Pub/Sub Subscriptions console.
Test Boomi process and verify documents are getting processed successfully.
Open Process Call and verify if the document is processed via sub-processes successfully.
Once Boomi Process sends messages into HazelCast, you can view PO IDs on HazelCast Dashboard.
Once all messages are received from POS Server for a PO ID, Boomi will process and publish them into ActiveMQ in the same order as they originated from the source. Open ActiveMQ web console and explore Queue to find your complete PO.
We are good with the testing so now let us deploy the process.
Deploy Process
Create a “Packaged Component” of Boomi Process, deploy it into the Atom Environment, and schedule it.
Your Boomi process is now deployed and ready to receive messages from GCP!
References:
Google Cloud : https://cloud.google.com/
HazelCast : https://hazelcast.com/
ActiveMQ : http://activemq.apache.org/
Boomi : https://boomi.com/
Google Cloud Pub/Sub : https://cloud.google.com/pubsub/docs/overview
More about JMS Configuration and GCP Certificate Creation and Import:
Configure JMS in Boomi:
JMS libraries must be placed in the Boomi platform Account Libraries, these libraries are shipped with the JMS provider. Required libraries (JAR files) are in the ActiveMQ installation folder (activemq-all-5.15.12.jar, geronimo-j2ee-management_1.1_spec-1.0.1.jar).
Add libraries into Account Libraries
Account Libraries are used to upload and manage custom JAR files for integration processes.
Now, you can create a new Custom Library component. This component is a collection of Java Archive files (JARs) you can use to support integration processes. Custom Libraries are reusable, which is why we can use them in several Boomi Processes.
Create a new Component and choose Type as Custom Library, provide Component Name and choose a project folder.
Add ActiveMQ jars in Custom Library and Save.
Create Package Component and Deploy the Custom Library into Atom Environment.
Note: Listeners cannot be tested in the test environment.
Create Certificate/Private key in Google Cloud Platform:
Some configurations are required in the GCP for integrating with Boomi.
Certain connectors might require the use of a certificate to ensure security when transmitting data across the communication protocol. Connectors such as FTPS, SFTP, HTTPS, AS2, and many others require the use of certificates to encrypt data and verify the digital signature of the source sending the data. You can create two types of certificates in Boomi (i.e. X.509 and PGP). Here X.509 Boomi certificate (Private) has been used.
Login to your Google Cloud Platform and create a service account.
Provide Service account details
Provide appropriate permissions
Grant users (optional)
Once the service account has been created, go to Google’s Credentials page, service account should be listed.
Now click on the service account highlighted above and create a new Key.
Select P12 and save the key in your system.
Certificate generation is done at GCP, this will be used in Boomi.
Import certificate in Boomi platform:
Login to your Boomi account and create a Certificate component and provide Component Name, Boomi project Folder, and Certificate Type as X.509.
Import the certificate (key) you downloaded from the GCP service account.
Save and Close
You imported the certificate successfully in the Boomi Platform!
Summary: Using the Boomi Integration Platform to resequence out-of-order messages coming from Google Cloud Pub/Sub
There you have it, a detailed tutorial on using Boomi’s platform to resequence messages received out-of-order from Google Cloud Pub/Sub, and resubmit them to a message queue to be processed further with the guarantee that the messages will be in chronological order.
If you have questions on how you can best leverage Boomi, need help with your Boomi implementation, or our Cloud Services, please reach out to us to get a conversation started.