The Ingest Kafka Stream in API template in Connect+ helps you consume data from a Kafka topic and transform it for use within the Capillary platform.

Use case

Scenario

An airline brand uses a Passenger Service System (PSS) to send post-flight passenger data to the Capillary platform. This data includes passenger details, flight completion status, and other relevant metadata. The PSS sends the data in bulk every hour, with peak rates reaching up to 1,000 requests per minute. To handle this high volume efficiently, the PSS streams the data into a Kafka queue.

The system needs a reliable way to:

• Continuously consume data from the Kafka queue
• Transform it into a format compatible with Capillary’s platform
• Forward the processed data for further actions, such as awarding loyalty points or updating customer profiles

Solution

Use the Ingest Kafka Stream in API template in Connect+ to build a Kafka consumer that retrieves and processes the data efficiently. This solution:

• Connects to the specified Kafka topic
• Consumes messages from the Kafka queue
• Use Neo to transform data to match Capillary’s format
• Forwards the transformed data to the relevant Capillary platform APIs for further processing

Prerequisites

The following are the prerequisites for using the Ingest Kafka Stream in API template:

• Name of the Kafka server
• Kafka topic
• Consumer group ID
• Username and password to access the Kafka server.

Configuring the template

To configure the Ingest Kafka stream in API template, the following are the steps:

1. In the Connect-to-source-kafka block, enter the Kafka details including the server address, kafka topic, username and password from where the data is imported. For the information on configuring this block, refer to Connect-to-source-kafka.
2. In the neo-Transformer block, enter the URL of the Neo dataflow created to transform the data required as per your requirement. For the information on configuring this block, refer to neo-Transformer.
3. In the Connect-to-destination enter the details of the API endpoint. For the information on configuring this block, refer to Connect to destination.