Connect to Source

This block enables you to define the source location of the data that needs to be ingested. The connect to source block has the following fields:

Field name	Description
Hostname	URL (address) of the SFTP/FTP server from where the source file is available. For example, data.capillarydata.com.
Username and Password	Credentials to access this SFTP/FTP server. This provides read/write access to the files on the server.
Source Directory	The directory path where the source file is available. For example, /APAC2Cluster/A_Connect Connect+ usually supports any text-based files with a delimiter, irrespective of extensions such as .txt, .csv, or .dat etc. In addition, _.ok _file format is also supported.
Filename Pattern	File names in the regex pattern. If you give a filename pattern file..csv _, the application takes any file that starts with the file name file*.
Processed Directory	The path to save the file to be processed. The data to process is captured from the processed file. Processing is the operation of formatting and transforming a given set of data to extract the required information in the appropriate format. For example, /APAC2Cluster/Process.
Unzip Files	If the files are compressed, you can select this option to unzip the file and then select the original file.
API Error Filepath	The SFTP/FTP path where you want to save the API error file. This file includes the error details that occurred during the API calls. This field is applicable only where API endpoints are required (data ingestion).
Search Directory Recursively	Select the checkbox, if you want to search for the defined file anywhere in the root folder of the server. For example, if there are multiple folders inside /APAC2-Cluster/A_Connect, it looks for files matching the pattern inside each folder under /APAC2-Cluster/A_Connect.
Port	The source SFTP/FTP port number. Generally, 22 for SFTP and 21 for FTP.

Source directory file path — Copy directory file path

Defining column header for transform block

Adding the source file header information using a CSV file with the column headers

You can upload a CSV file with the column headers of the input source file or define the header data manually. This makes mapping fields in the transformation block easier. Once you define the headings, they are automatically retrieved in the transformation blocks and you can map corresponding API fields against them.

The maximum supported size of a CSV file is 5 MB. After attaching the file, you can delete it by clicking the delete icon. The system allows you to switch to the manual option after uploading the CSV file. The values should be separated by a comma only. Watch the video below for more information on uploading the CSV file and mapping the API fields.

Adding the header information manually

If you do not have a CSV file for mapping, you can select the Add manually option and define the headings manually in the transformation block. Watch the video below for more information.

If you do not have a CSV file for mapping and the source file is headerless you can select the Add manually option and choose the Is file headerless option to define the headings separated by a comma. These headings are retrieved in the Transformation block, and you can also add additional headings in the transformation block itself. Watch the video below for more information.

S3 File Service Location as a Source

Field Name	Description
Bucket Name	The name of your S3 bucket. Provided by the access team.
Region	Region in which your bucket is located. Provided by the access team.
Access Key and Secret	These credentials are required to access the S3 bucket. Provided by the access team.
Input Path	The location within the S3 bucket is where the input data resides.
Output Path (Processed Path)	Defines where the processed data will be placed in the S3 bucket. The output path cannot be the same as the input path or a sub-path of the input path (e.g., you cannot have "input/" and "input/processed/"). Additionally, the output path cannot be the root ("/"); a folder must be created within the root for this purpose.
Error Path	Designates the location for storing error files.