AI/ML Models

Overview of AI dashboard

AI models dashboard is Capillary’s platform to run machine learning models. The dashboard offers self-serve personalization with several models available out-of-the box. It allows you to launch models for your org in just a few clicks and leverage valuable inferences from them.

The dashboard abstracts the complexity of data science and makes machine learning models easily accessible to analysts. You can make business decisions based on the insights provided by the model. Since, processing and making inferences on data present in large volumes cannot be handled manually, machine learning provides an intuitive solution to the problem. The dashboard closely works on the principles of machine learning operations (MLOps). MLOps streamlines the data science outreach for an organization.

Benefits of AI dashboard

The dashboard aims to address the following benefits for running and managing machine learning models efficiently.

Ready availability of machine learning models

The dashboard fosters model discovery with all existing and upcoming models available on the dashboard. To know the list of models that can be used for running personalization campaigns, for example, brand users need not rely on Capillary Customer Success (CS) teams.

Seamless integration with Capillary application

Once the models go live on the dashboard, the results (predictions) can be utilized in Engage+ as filters to run personalized campaigns.

512

Likewise, users of Insights+ can leverage live models to generate reports that depict vital statistics.

All model results can be visualized on Insights+ reports. However, the Insights+ team needs to setup KPIs for the reports to view results.

937

Improved productivity

The dashboard has been designed with the goal of making Artificial Intelligence (AI) self-serve. A marketer looking to run a personalized, churn killer campaign, for instance, need not rely on internal data science teams. Capillary’s Insights+ team intervention is also not required.

With the dashboard, the marketer can leverage the relevant model while utilizing predictions to run personalized campaigns. There is no productivity loss as there is no dependency on internal or external teams.

Easy management

The dashboard provides a timely management of the machine learning models that an organization deploys. The dashboard validates and trains the model regularly in periodic intervals to ensure that predictions are accurate and based on recent data.

To get started with the dashboard, see here.

Use Cases

Model display nameUse cases of the model
Transaction Propensity1. The high propensity customers can be encouraged to try products that have been newly launched. Orgs can identify the customers that are most likely to transact in the next 30 days using the transaction propensity model and target these customers with specific campaigns.

2. Orgs can also identify the customers that are least likely to transact in the next 30 days and exclude them from campaigns.

3. Product preferences of customers who are most likely to transact in the next 30 days can be analyzed using Insights+. Use results generated by models in Insights+ for reports.
Customer Churn Prediction1. Orgs can identify the customers who are likely to churn in the next 30 days and these customers can be targeted with aggressive churn killer campaigns.

2. Customers who are least likely to churn can be targeted with informational campaigns (instead of aggressive, personalized offers) to keep them engaged.
Campaign Response Prediction1. Orgs can identify customers who are most likely to respond to a campaign. For instance, customers not likely to respond to a festive campaign can be excluded and campaign costs can be saved.
CLTV Prediction1. Orgs can identify the most valuable customers for the brand using the CLTV Prediction model. Orgs can create an exclusive experience for such customers.
2. Orgs can identify the customers with low CLTV value and offer them loyalty services or additional discounts to upscale their value.
3. Orgs can use a combination of CLTV prediction and churn prediction models to arrest churn of high value customers.

New terminologies in AI/ML

TerminologyDescription
AccuracyIt is the number of correct predictions divided by the total number of predictions made. Accuracy highlights the rate of accurate model prediction.
Accuracy MetricsAccuracy metrics indicate numerical values that evaluate the performance of the model.
Data validationIt is to check the accuracy and quality of the source data before training. It ensures that the mistakes are addressed and not silently ignored.
DatasetA set of data available in a tabular format is called a dataset. It is the set of data processed in a machine to make the predictions.
F1 scoreIt is the harmonic mean of precision and recall, taking both metrics into account.
Machine Learning Operations (MLOps)MLOps is a set of best practices that seamlessly brings data science solutions into existing systems. Enterprises can set up efficient data science solutions without disturbing their current set up.
Machine learning modelA machine learning model is a mathematical representation of a set of data and is used to make predictions/inferences. There are several types of machine learning models available and the models offer unique advantages for different use cases.
Model trainingA data science model needs to be trained with available data to make effective predictions on newer data. A machine learning algorithm learns from a dataset and predicts results for a different set of data.
PrecisionThe ability of a classification model to identify only the relevant data points.
Re-trainingAny model will decay as time elapses and new data keeps coming in. Consequently, deployed models would need retraining at certain intervals. Re-training with newer data increases the prediction accuracy in the longer run.
RecallThe ability of a model to find all the relevant cases within a dataset.

Data models

The AI/ML dashboard uses machine learning (ML) and artificial intelligence (AI) algorithms in the background. The dashboard models are named as per their functionality, for instance, if a model predicts when a customer is likely to make a transaction, it is titled ‘Transaction prediction’.

The propensity models of dashboard are described in the following table -

ModelDescriptionMore Info.
Transaction predictionThe Transaction prediction predicts customers who all are likely to purchase in the future time period - 30 days.The Transaction prediction model uses a binary classifier composing a primary ensemble of Light Gradient Boosting Machine (LightGBM), and Random Forest.
Customer Churn PredictionThe Customer Churn Prediction predicts customers who have a high chance of churning.The Customer Churn Prediction model uses a binary classifier composing a primary ensemble of Light Gradient Boosting Machine (LightGBM), and Random Forest.
Campaign Response PredictionThe Campaign Response Prediction is very specific to a certain campaign period and given a campaign who would respond. This also takes the previous response rate of the person into account.The Campaign Response Prediction model uses a binary classifier composing a primary ensemble of Light Gradient Boosting Machine (LightGBM), and Random Forest.
CLTV PredictionThe CLTV Prediction predicts the customer lifetime value for a year and then extrapolates for the next 3 years.The CLTV Prediction model uses a combination of regression and bucketing based classification algorithms.
Product AffinityThe Product Affinity ranks the product in the order of choice of the customer.The Product Affinity model uses the Proximity Model and FIR score Model.

Get started with the Dashboard

  1. Log on to InTouch of your cluster (APAC, Europe, and India) and navigate to Menu > AIRA Dashboard.
  2. Select your organization (only for users who have access to multiple orgs). Else, the org is selected by default.
1000
  1. AI Models Dashboard
    This shows the snapshot of status of all models of the org.
552

When new models are available for the organization, a note is displayed with the number of models available as highlighted in the preceding screenshot.

The significance of each model status is as described in the following table.

StatusDescription
LiveThe model is active and running currently. Live models are ready to use by the organization.
In ProgressThis indicates two possibilities - when the source data is under validation or the model is under training.
Action NeededThis indicates two possibilities for models post data validation
the data validation is incomplete and need to be fixed or
the data validation is complete and awaiting training.
Not InitiatedIt indicates that the model has been selected for the org but not initiated.

Add model

The dashboard offers several models related to retail and you can leverage them for analysis. There are five models available currently on AIRA. You need to add models manually to your org according to the business requirement.

There are four steps involved for an org to use a model.

Initiate model

You can either select a new model offered by Capillary or work on pending models that have been initiated.

To select a new model, use the following steps.

  1. On the AIRA Dashboard page, click New model.
  2. Choose a relevant model and click on Initiate. The model is initiated and its status/sub-status changes to Not Initiated/Disabled. Then, proceed to Step 2: Validate Data.
1218 1067

Validate data

Once you initiate a model, you need to validate the data to ensure the data is proper without any errors.

To validate the data,

  • Navigate to the Validate tab, click Validate. You will see this only if the the validation is pending for the model.
942

Data validation triggers and the status/sub-status of the model changes to In Progress/Validating.

When the validation is completed, the status changes to User Action Required/Ready for Training. You can proceed to Step 3: Train Model.

1066
  • User Action Required/ Data Quality: Needs Attention: The status or sub-status of the model when the data validation has failed. You can see the reason why the data validation has failed along with the recommended solution.
1070

Fields in Data Validation tab

966
OptionDescription
Validation Status Disabled: The data validation for the model is not yet initiated. You will see an option to validate data.
In_progress: The data validation for the model is in progress (dataset is being ingested in the Customer Data Platform).
Success: Validation is complete and the model is ready for training.
Failure: The validation is failed due to insufficient data for other reasons
ObservationsThis shows the reason identified for validation failures. This shows N/A for successfully completed validations.
Last AttemptShows the recent date of data validation.
  • To download data validation report, click Download report. The report downloads in a HTML format.
606
  • Successful validation report: When the data is validated successfully, you will see the summary of the data quality as shown here.
880
  • When the data validation is not successful, you will see an exhaustive report that helps identify root causes of failure such as scarcity of historical data or insufficient number of campaigns.

Train model

You need to train the model after the dataset validation is successful. The training process usually takes less than a day.

  • To train a model, navigate to the Training tab and click Train. The status/sub-status of your model changes to In Progress/Training.
1064

If validation has failed for the model, you will see a note to fix validation issued (as shown in the screenshot)

568

When the model training has been successfully completed, you will see a screen in your Training tab similar to the below screenshot.

1064

You can view the descriptions of the fields of Training tab, see the following sections.

Check data accuracy

A model’s performance decays with time (with the availability of new data) and if the accuracy metrics downgrade, you must re-train the model.

444

If the model has not completed all the previous stages, then the Accuracy tab shows a message conveying that no data is available for accuracy.

When all the phases have been completed successfully, the model goes live! The status/sub-status of the model changes to Live/Monitoring Accuracy.

You can view the descriptions of the fields of Accuracy tab in the View model details section.

View model details

There are two ways by which you can search for a model (as depicted in the following screenshot).

  1. Search models by model name: Use the Search box to search for your preferred model. You will start seeing the models as you type.
  2. Filter models by status: By default the table shows the list of all models. You can filter models by a specific status. For example, see only Live models. Use the available drop-down box and select your preferred status to view.
1225

To work on an already initiated model, click on the model to view.

1214

The details of the model are available in the Overview tab.

509
  • Model name: Shows details of the model.
    • Status and sub-status: The significance of status and sub-status of a model are described in the following table.
Overall StatusSub StatusDescription
In ProgressValidatingData required for the model is currently under validation. As soon as the Validate button is clicked, it will go into this status/sub-status.
In ProgressTrainingModel is currently under training after training has been initiated by a user.
User Action RequiredData Quality: Needs AttentionData validation has been completed and the data required for moving to the model training phase has issues. Fixes need to be made in the data before attempting data validation again.
User Action RequiredReady for TrainingData validation is successful. User has to manually trigger the model training step.
LiveMonitoring AccuracyModel training has been completed successfully and it is now Live, predicting propensity scores. Accuracy metrics are currently being monitored for the model.
Not InitiatedDisabledThe model has been selected but not yet initiated.

Last status update: Displays the date on which the model status was updated.

About: Shows information about the model, explaining what the model predicts and the type of results it generates. The About section is followed by the status of each stage of the model, and a brief information about what is achieved in each stage. The status of each stage is highlighted by 4 symbols. The purpose of the symbols is described in the following table.

SymbolDescription
Green ticThe adjacent stage has been completed successfully.
Red cross iconThe adjacent stage has error(s) and user intervention is required.
Red refresh iconThe adjacent stage is in progress.
i iconThe adjacent stage has not yet been initiated/started.

View training status

The Training tab displays the fields as described in the following sections.

Training Status: There are three different statuses of training. See the following table for details.

StatusDescription
DisabledThe model has not yet reached the training stage or the training is yet to be triggered. You will see an option to train the model.
In_progressModel is currently under training if the training has been initiated by the user.
SuccessThe model has been trained successfully.
  • Last attempted on: This shows the date on which the model was last trained.
    • Avg training time: This field displays the time (in seconds) that was taken to complete the training phase.
    • Total training runs: This field displays the number of times the model has been trained successfully.
    • Previous Training Runs: This displays the history of training runs in a tabular format. The table has the columns labeled Training Attempt No, Status, Completed time and Time taken.

View accuracy status

The Accuracy tab displays the fields as described in the following section -

FieldDescription
Accuracy StatusSuccess is shown when all the stages have been completed successfully and the model is actively performing its predictions.
Last updated atThe date on which the accuracy status of the model was last updated.
RecallIt tells us what portion of actual positives was identified correctly.
PrecisionTell us what proportions of positive identifications were actually correct.
F1 scoreHarmonic Mean of precision and recall.

AI Models

EntityReference Link
Artificial Intelligence Retail Analytics (AIRA) OverviewArtificial Intelligence Retail Analytics (AIRA) Overview
Key Terminologies of AIRAKey Terminologies of AIRA
Use Cases of AIRA modelsUse Cases of AIRA models
AIRA data modelsAIRA data models
Get Started with AIRA dashboardGet Started with AIRA dashboard
Add ModelAdd Model
View details of a modelView details of a model