Connect+ has various blocks that are used to create a dataflow. The blocks in a dataflow is based on the selected templates and can differ from template to template.

Connect to source

This block enables you to define the source location of the data that needs to be ingested. The connect to source block has the following fields:

Defining column header for transform block

Adding the source file header information using a CSV file with the column headers

You can upload a CSV file with the column headers of the input source file or define the header data manually. This makes mapping fields in the transformation block easier. Once you define the headings, they are automatically retrieved in the transformation blocks and you can map corresponding API fields against them.

The maximum supported size of a CSV file is 5 MB. After attaching the file, you can delete it by clicking the delete icon. The system allows you to switch to the manual option after uploading the CSV file. The values should be separated by a comma only. Watch the video below for more information on uploading the CSV file and mapping the API fields.

Adding the header information manually

If you do not have a CSV file for mapping, you can select the Add manually option and define the headings manually in the transformation block. Watch the video below for more information.

If you do not have a CSV file for mapping and the source file is headerless you can select the Add manually option and choose the Is file headerless option to define the headings separated by a comma. These headings are retrieved in the Transformation block, and you can also add additional headings in the transformation block itself. Watch the video below for more information.

Decrypt data

This block enables the descryption of an encrypted source file. It is an optional feature that can be enabled if the source file is encrypted. All the fields within this block become mandatory once you enable the Decrypt data action block. You can use the Is Enabled drop-down to control the activation of this block.

Connect to destination

This block enables you to define the details of the v2 API that will be used for running the dataflow. The API details are auto-filled. You must enter the API client key and secret key to initiate the API.

Transform data

Data transformation in Connect+ is the process of converting the ingested data to JSON format. To achieve this, you need to map the fields in the source files to the appropriate API fields in this block. This helps connect+ to convert the data to JSON, call the APIs successfully and upload the data. These fields in this Connect+ block are parameters of the JSON payload of the API associated with the template that you have selected for the dataflow.

Perform the following to map the fields:

1. If the workspace is linked to multiple orgs, select the org ID from the Intouch Org drop-down.

2. In the File Delimiter text box, enter the delimiter used to separate the headings in the source file. By default, the delimiter is set as a comma.

3. Map the API fields with the appropriate columns in the source file. To map, perform the following:

   1. If the column header fields were not uploaded or defined, enter the column header name in the File header text box. This is the name of the column header in the source file that will have information for an API field. For example, a column named source can have values for the API field called source. You can also click on Add more headers and can add more headers or add headers using expression.

   2. Select the API field name from the API fields drop-down that corresponds to the column header specified in the File header text box. The number of mandatory fields that need to be mapped is displayed next to the parent API field.

   3. If required, in the Filter data text box, use expression language and add a filter condition to include or exclude data from the source file. Example: Consider a transaction dataflow that includes a Financial status header with values such as paid, refunded, or not paid. To exclusively process dataflow for transactions that are either paid or not paid, you can use the filter expression ${header_value:equals('paid'):or(${header_value:equals('not paid')})}.

Watch the videos in the Defining column headers section to learn how to map the API fields.

Adding a header using the expression

If the value of an API Field is the same for all the customers, you can use const{value}. For example, to categorize loyaltyType as loyalty for all customers, you can enter const{loyalty} against the API field loyaltyType. For more information, refer to the documentation on using expression.

To add a header using an expression,

1. Click Add header using expression.
2. Enter the expression in the text box.
3. From the API field drop-down, select the API field you want to map to the header.

Adding additional header

If you selected Add manually as the Column headers input method (with or without header names) for transforming data in the Connect to source block, you will have the option to add additional headers. You can add the additional header while mapping the fields or editing.

🚧

Note

If you added the header information through the CSV file method, you cannot add additional headers.

Join data

This block enables the merging of two or more files and adds the merged file to the platform. You need to enter the details of files that need to be identified, compared and merged. Enter the below details:

FTP file upload instructions

When uploading files to the FTP path:

• Ensure that all files that need to be merged are added simultaneously.
• If you are adding different sets of files that need to be merged, maintain a minimum gap of at least 10 minutes between each upload. For example, if you want to merge files A and B, and also C and D, do not add files A, B, C, and D together. Add files A and B first, and then add files C and D only after a minimum of 10 minutes.

If you need to re-add any files:

• Remove the existing file from the process folder first.
• Then, add the new files to the input folder for proper handling.

Rebuild headers /Define headers and transform data

CSV-To-XML-Converter

Enter the following details.

Push to S3

Enter the following details:

• S3 Object name/Filename - Defines the name that will be given to the file(s) uploaded to S3. Use the default value. You can also use the expression language and define the file name. For the "Audience reload from FTP template", enter the object name in the format <audience_Group_Name>/${filename}, where audience_Group_Name is the audience name used in the Engage+.

• S3 Bucket Name - Enter the S3 bucket name.

  

• Content Type - Use the default value application/octet-stream. This enables the copying of all file types.

• Access Key & Secret Key - Enter the S3 access key and secret key.

• AWS region - Enter the AWS region to which you are transferring the files.

  • US East 1 - For the bucket in the India region
  • EU West 1 - For the bucket in the EU region
  • AP Southeast 1 - For the bucket in the Singapore region
  • US East 2 - For the bucket in the US regions

Note: If you do not have information on the Access key, secret and AWS region, create a ticket to the Sustenance team.

Encrypt data

This block enables the encryption of an output file generated, using the PGP algorithm. All the fields within this block become mandatory once you enable the Decrypt data action block. You can use the Is Enabled drop-down to control the activation of this block.

Schedule trigger

This block enables you to schedule the time at which the dataflow should be triggered.

You can configure the dataflow activity to execute daily, hourly, or on a minute basis.

🚧

The minimum triggering interval for the dataflow is set at 5 minutes, meaning that even if a value less than that, such as 1 minute, is entered, the dataflow will still trigger at 5-minute intervals.

Schedule trigger at specified minutes

Select Minutes to schedule a trigger at specific minute intervals The minimum duration allowed is 5 minutes.

Schedule hourly trigger

• Select Every hour(s) to schedule the dataflow trigger at regular intervals, such as every 2 hours. Specify the frequency in the "Every ___ hour(s)" field.
• Select At to set a specific time for dataflow execution, such as 13:00. Configure the time using the provided hour and minute drop-down menus.

Schedule daily trigger

• Select Every day(s) to schedule the dataflow trigger at regular intervals, such as every 2 days. Specify the frequency in the "Every ___ day(s)" field.
• Select Every week day to set a specific time in a week day for dataflow execution, such as 13:00. Configure the time using the provided hour and minute drop-down menus.