File Transfer & Transformation

File transfer and transformation templates move, merge, reformat, or convert files between FTP, SFTP, and S3 locations. These templates do not send data to Capillary ingestion APIs. Select a template, configure the block details for your environment, and deploy the dataflow.

The following templates are available for file transfer and transformation.

SFTP to SFTP with 2 files join

Downloads two files from an SFTP server, unzips, joins them on matching fields, zips the merged output, and pushes it to an SFTP server.

Use case

A brand exports two related datasets to an SFTP location daily and needs them combined into a single file before the data is shared with a downstream system. This template joins the two files on a common field and delivers the merged output to the destination SFTP server in a single scheduled run.

Block configuration

The following table lists the blocks in the SFTP to SFTP with 2 Files Join template, describes what each block does, and provides the configured values for each field.

SFTP to FTP with 2 files join

Downloads two files from an SFTP server, unzips, joins them, zips the merged output, and pushes it to an FTP server.

Use case

A brand stores two separate data files on an SFTP server and needs to deliver a merged file to a partner system that accepts data only via FTP. This template handles the join and transfer in a single operation.

Block configuration

The following table lists the blocks in the SFTP to FTP with 2 files join template, describes what each block does, and provides the configured values for each field.

SFTP to FTP with 3 files join

Downloads three files from an SFTP server, unzips, joins them, remaps headers, zips the merged output, and pushes it to an FTP server.

Use case

A brand receives data across three separate files from different upstream systems and needs to consolidate them before sharing with a partner. This template joins all three files and remaps column headers to match the partner's expected format before delivering the output to the FTP destination.

Block configuration

The following table lists the blocks in the SFTP to FTP with 3 Files Join template, describes what each block does, and provides the configured values for each field.

SFTP to FTP with 4 files join

Downloads four files from an SFTP server, unzips, joins them, remaps headers, zips the merged output, and pushes it to an FTP server.

Use case

A brand consolidates data from four source files before pushing the merged result to a downstream FTP location. This template handles the multi-file join, header remapping, and delivery in one scheduled operation.

Block configuration

The following table lists the blocks in the SFTP to FTP with 4 Files Join template, describes what each block does, and provides the configured values for each field.

SFTP to FTP with 3 files join with encrypt and decrypt

Downloads three files from an SFTP server, decrypts them, unzips, joins, remaps headers, zips, encrypts the output, and pushes it to an FTP server.

Use case

A brand receives three encrypted source files on an SFTP server and needs to merge them before sending the combined result to a partner via FTP. The partner requires the output file to be encrypted. This template handles decryption on ingress, the join and header remap, and re-encryption on egress in a single automated flow.

Block configuration

The following table lists the blocks in the SFTP to FTP with 3 Files Join with Encrypt and Decrypt template, describes what each block does, and provides the configured values for each field.

SFTP to SFTP merge 3 files join with encrypt and decrypt

Downloads three files from an SFTP server, decrypts them, unzips, joins, remaps headers, encrypts the output, zips, and pushes it to an SFTP server.

Use case

A brand picks up encrypted customer data exports from an SFTP location, generated by the Data Export Framework in Insights+. The brand needs to merge three files, secure the output with encryption, and deliver it to a destination SFTP folder. This template was introduced to give brands a secure, generic way to migrate files between SFTP locations with both decryption and encryption support.

Block configuration

The following table lists the blocks in the SFTP to SFTP Merge 3 Files Join with Encrypt and Decrypt template, describes what each block does, and provides the configured values for each field.

SFTP to SFTP merge 4 files join with encrypt and decrypt

Downloads four files from an SFTP server, decrypts them, unzips, joins, remaps headers, encrypts the output, zips, and pushes it to an SFTP server.

Use case

A brand consolidates four encrypted source files into a single encrypted output and delivers it to a destination SFTP location. This template handles all steps from decryption through to re-encryption and transfer in one scheduled run.

Block configuration

The following table lists the blocks in the SFTP to SFTP Merge 4 Files Join with Encrypt and Decrypt template, describes what each block does, and provides the configured values for each field.

SFTP to SFTP transfer

Transfers files from one SFTP server to another. Supports optional encryption and decryption as part of the transfer.

Use case

A brand needs to move files generated by the Data Export Framework in Insights+ from a Capillary SFTP location to a destination SFTP server managed by a partner or internal team. When data security is required, the template can decrypt the source file and encrypt the output before delivery. Encryption and decryption are both optional and can be used independently or together.

Block configuration

The following table lists the blocks in the SFTP to SFTP Transfer template, describes what each block does, and provides the configured values for each field.

SFTP to SFTP with headers definition

Downloads a single file from an SFTP server, unzips it, remaps or defines column headers, zips the output, and pushes it to an SFTP server.

Use case 1

A file named abc.csv is being transferred from the Capillary server to a brand's server. The columns have headers named Transaction_value and Bill_ID. The brand requires the file to arrive with column names TAmount and Bill_No instead. This template renames the headers during transfer without changing the underlying data.

Use case 2

A file named abc.csv is being transferred from the Capillary server to a brand's server. The columns in the file do not have header names. The brand requires the output file to include column headers. This template adds the defined header names to the file during transfer.

Block configuration

The following table lists the blocks in the SFTP to SFTP with Headers Definition template, describes what each block does, and provides the configured values for each field.

SFTP to FTP with headers definition

Downloads a single file from an SFTP server, unzips it, remaps or defines column headers, zips the output, and pushes it to an FTP server.

Use case

A brand transfers a single file to a partner system that accepts data only via FTP and requires specific column header names. This template remaps or adds headers during the transfer to match the partner's expected format.

Block configuration

The following table lists the blocks in the SFTP to FTP with Headers Definition template, describes what each block does, and provides the configured values for each field.

FTP to S3

Transfers files from an FTP or SFTP server to an AWS S3 bucket.

Use case

A brand stores processed output files on an FTP server and needs to archive them in an S3 bucket for downstream processing or long-term storage. This template picks up the file from FTP and uploads it to the configured S3 bucket and path in a single scheduled operation.

Block configuration

The following table lists the blocks in the FTP to S3 template, describes what each block does, and provides the configured values for each field.

CSV to XML conversion

Converts CSV files to XML format and delivers the converted file to a destination SFTP server. Supports optional encryption of the output file.

Use case

A data analyst receives data from various departments as CSV files containing fields such as product ID, product name, and price. The downstream system ingests data only in XML format. XML's hierarchical structure also makes the data easier for both humans and automated systems to interpret. Using this template, the analyst converts all CSV files to XML on a schedule, removing the need for manual conversion before delivery.

Block configuration

The following table lists the blocks in the CSV to XML conversion template, describes what each block does, and provides the configured values for each field.