AML Customer Churn¶

1. The Data¶

Customer Churn Dataset¶

The Customer Churn dataset is designed to analyze and predict customer behavior related to discontinuing a subscription. Understanding and predicting customer churn helps businesses identify patterns, develop retention strategies, and improve customer satisfaction, which directly impacts revenue and growth.

Dataset Features¶

• CustomerID: unique identifier for each customer.
• Age: customer's age, between 18 and 65.
• Gender: customer's gender.
• Tenure: number of months the customer has been with the company, between 1 and 60.
• Usage Frequency: frequency of service usage, between 1 and 30 times.
• Support Calls: number of support calls made by the customer, between 1 and 10.
• Payment Delay: number of days payment is delayed, between 0 and 30.
• Subscription Type: type of subscription plan (Premium, Standard, Basic).
• Contract Length: duration of the subscription contract (Annual, Quarterly, Monthly).
• Total Spend: total amount spent by the customer.
• Last Interaction: days since the last interaction.
• Churn: Customer has churned (1) or not (0).

Motivation¶

With my experience in creating and optimizing web-apps to reduce customer churn, analyzing this dataset will enhance my understanding of customer behaviors, leading to improved UI/UX strategies. This understanding will help me as a web developer to create better features, ultimately reducing churn rates in my current role.

Source¶

The dataset is sourced from Kaggle and can be found here: https://www.kaggle.com/datasets/muhammadshahidazeem/customer-churn-dataset.

Descriptive statistics¶

Load the data¶

The dataset contains a wide range of data points with 64,374 entries and 12 columns (11 features and 1 target). No attribute has missing values.

In the initial evaluation, the CustomerID attribute, which is a unique identifier, doesn't add value to our model for churn prediction and will create unnecessary complexity for our initial data analysis and visualization. Therefore, it can be dropped. Removing unnecessary information is crucial, as it serves as noise for our model and should be eliminated to improve model performance.

Additionally, there are no no missing values, so there will no need to drop or replace null values during the data preparation phase.

In order to follow best practices, it is good to create a train/test split and set the test set aside, so the training set is manipulated. For the exploration phase, a copy of the training set will be made to analyze the data without affecting the original training set.

Statistical Summary¶

Summary of key observations¶

1. Customer Demographics and Behavior:

• Support Calls: mean of 5.04 calls, indicating higher level of customer support interaction. It might be beneficial to examine the correlation between support calls and churn rate.
• The standard deviation values suggest a diverse customer base in terms of age, tenure, and spending habits.

2. Churn Distribution

• Churn Rate: approximately 47.4% of customers have churned, indicating that almost half of the customers have discontinued their relationship with the company. Understanding and addressing this churn is critical for developing effective retention strategies.

Class Distribution¶

The class values are slightly imbalanced, with a slightly higher count of "No Churn" compared to "Churn." However, the imbalance is not significant, so special handling, such as resampling techniques, is not required at this stage. However, it's important to note that imbalanced classes might increase the risk of overfitting towards the dominant class in some models such as Decision Tree classifier (CART).

It is necessary to distinguish between categorical and numerical data, as only numerical data can be used for correlation and skewness analysis.

Correlations between Attributes¶

• Payment Delay and Churn: stronger positive correlation (0.557) between payment delays and churn. This shows that customers with higher payment delays are significantly more likely to churn, which may indicate potential financial issues or dissatisfaction with the service.
• Support Calls and Churn: moderate positive correlation (0.304) between support calls and churn, indicating that customers who make more support calls are more likely to churn. This suggests that customers who frequently contact support may be experiencing issues or dissatisfaction, leading them to leave.
• Usage Frequency and Churn: weak negative correlation (-0.118), indicating that higher usage frequency is slightly associated with lower churn.

Monitoring these correlations carefully is essential to ensure they do not significantly influence the model training process. Careful feature selection and potential transformations will be necessary to manage these correlations effectively.

Skew of Univariate Distributions¶

The skew values are close to 0, indicating that the distributions of the numerical features are nearly symmetric. Normalizing these distributions will still enhance performance of some models such as K-Nearest Neighbors by ensuring that all features are on a similar scale.

Visualization - numerical features¶

Univariate Histograms¶

Most features do not follow a Gaussian distribution. There is bias in support calls and payment delays towards higher.

Linear regression may not be ideal since it assumes a linear relationship and normally distributed (Gaussian) data, which is not present in our dataset. Non-parametric models that do not assume a specific relationship between variables, such as Decision Trees, Random Forests, or K-Nearest Neighbors (KNN), are preferred. These models are likely to perform better compared to linear regression models, as they can handle non-linear relationships and do not require the data to follow a Gaussian distribution.

No attributes are currently scaled, which will need to be addressed during the data processing phase. Many algorithms perform better after normalization, and it is essential to scale the features appropriately to improve model performance.

Density Plots¶

To observe the distribution of each attribute more clearly, density plot analysis will be used. This method will also confirm whether the features follow a normal Gaussian distribution.

Box and Whisker Plots¶

Box and whisker plots are another useful method to review the distribution of each attribute. The absence of dots outside the whiskers indicates that there are no outliers or anomalies in the features.

Correlation Matrix Plot¶

The correlation matrix plot visually represents the relationship between different features in the dataset:

• Positive Correlation: variables that increase together, such as payment delay and support calls with churn (indicating higher churn rates with increased payment delay and support calls).
• Negative Correlation: variables that move in opposite directions, like total spend and churn (indicating higher spenders are less likely to churn).

Implications:

• Feature Relationships: important for feature selection.
• Model Selection: prefer non-parametric models like Decision Trees, Random Forests, or KNN.

Visualization - categorical features¶

For categorical variables such as Gender, Subscription Type, and Contract Length, bar plots will be used.

Potential biases include a gender distribution skewed towards more females. Distribution of subscription type and contract length is nearly even, indicating no particular type dominates the dataset.

Summary from numerical and categorical feature analyses¶

The data analysis indicates that numerical features are not normally distributed, with some skewness observed in support calls and payment delays. Applying normalization will be an essential step in preprocessing. Log transformations can also be considered to improve skewness in support calls and payment delays if necessary after initial model evaluation. Given the distribution characteristics and correlations, non-parametric models like Decision Trees, Random Forests, and KNN are likely to perform better. The box and whisker plots show no dots, indicating the absence of outliers. Balanced categorical features and a slight gender imbalance do not require special handling.

Data preparation¶

A new copy of the training data is used again. During the final model evaluation, the pipeline will incorporate these data preparation steps.

1. Removing Unnecessary Columns, Handling Duplicate Rows, and Addressing Missing Values: this step is typically essential, but in this case, there are no missing values, so it does not need to be incorporated.

2. Convert categorical features into numerical formats.

One-hot encoding transforms each category into a separate binary column, ensuring that categories are treated equally without implying any order. LabelEncoder was not used, as it assigns integer values to categories, which could mislead algorithms into interpreting them as ordered. One-hot encoding avoids this issue, making it more suitable for this dataset. Additionally, drop='first' was used to avoid multicollinearity by dropping the first category of each feature. This means the first category is inferred when all other categories are 0, preventing redundancy and ensuring the model interprets the data correctly.

3. Rescaling the data - many machine learning algorithms benefit from having all attributes on the same scale.

After rescaling with MinMaxScaler, StandardScaler or any other standardization techniques will not be used. Standardization is typically useful for transforming attributes with Gaussian distributions and varying means and standard deviations into a standard Gaussian distribution. However, this step will be skipped as our data does not follow a Gaussian distribution.

Influence of Data transformation on ML models¶

Normalization significantly improves KNN accuracy, increasing it from 0.809 to 0.903. This is because KNN relies on distance metrics, which work better when features are on a similar scale. This highlights the importance of selecting appropriate preprocessing techniques based on the specific needs of each model.

2. Constructing and Selecting Features¶

In the next step, feature selection will be performed to identify the most important features. This process helps reduce overfitting, improve accuracy, and decrease training time. Filter-based methods such as Univariate Selection and Feature Extraction with Univariate Statistical Tests (Chi-squared for classification), along with wrapper methods like Recursive Feature Elimination (RFE), will be used to select the features that contribute most to the target variable.

VarianceThreshold¶

SelectKBest¶

Recursive Feature Elimination - RFE¶

Now, Recursive Feature Elimination (RFE) will be used to evaluate different numbers of features to determine the optimal set.

RFE will be run with a varying number of features, ranging from 4 to 9, and performance will be evaluated using cross-validation (cross_val_score). This approach will help identify the number of features that have the best performance based on the mean cross-validation score.

• Consistent Feature Importance: both Logistic Regression (LR) and Decision Tree (CART) models, when using RFE, select similar important features such as 'Age', 'Tenure', 'Usage Frequency', 'Support Calls','Payment Delay' and 'Gender_Male'. This indicates that these features are highly informative for predicting the target variable.
• LR: shows a consistent mean score of 0.83 when using 7 to 9 features.
• CART: Achieves perfect accuracy with 9 features, but this may indicate over-fitting as the model captures all details in the training data.

Visual representation of feature importance using CART¶

Conclusion¶

All feature selection methods (SelectKBest, Variance Threshold, and RFE) identified similar important features, confirming their significance for our predictive model.

Given the consistency across these methods, it can be concluded that these features are robust indicators for predicting the target variable. For the final model evaluation, RFE with Decision Trees will be used, selecting 6 or 7 features. This approach balances the model's complexity, helping to prevent overfitting while maintaining high accuracy.

Selected features: Payment Delay, Support Calls, Gender_Male, Usage Frequency, Tenure, Age.

3. Building ML algorithms¶

Model selection¶

Three machine learning algorithms were selected for our labeled classification task:

• Logistic Regression (Supervised)
• K-Nearest Neighbors (Supervised)
• Decision Tree Classifier (Supervised)

Selection Strategies¶

• Logistic Regression: good baseline, easy to implement and interpret. However, it assumes linear relationships and Gaussian data distribution which our dataset doesn't have.
• K-Nearest Neighbors (KNN): it does not assume specific data distributions (non-parametric model) and benefits from feature normalization. It uses a distance metric to find the k most similar instances in the training data for a new instance and takes the mean outcome of the neighbors as the prediction.
• Decision Tree Classifier: handles non-linear relationships, identifies feature importance, and works well with mixed data types. Decision-tree learners can create over-complex trees that don not generalise the data well so that's why they are prone to overfitting, manageable with techniques like pruning and feature selection. Pruning tree is implemented via updating the model parameters such as max_leaf_nodes, min_smaples and max_depth.

Now the final train_set will be used.

Create pipelines¶

Create util functions for model evaluation¶

Create base models for initial evaluation¶

Observations:

• LR: provides a good baseline with balanced performance across metrics. It shows a decent ROC AUC and stable learning curves.
• KNN: demonstrates significant improvement in accuracy and ROC AUC after normalization, highlighting the importance of preprocessing.
• CART: achieves near-perfect scores but shows signs of overfitting. It's crucial to prune or regularize the tree to prevent overfitting.

Model optimization - hyperparameter tuning and feature selection¶

After the initial model evaluation, the next step is to optimize our models through hyperparameter tuning and feature selection. Based on initial evaluation, models like K-Nearest Neighbors (KNN) and Decision Trees (CART) showed promising results but indicated overfitting as CART was showing almost perfect scores. To address this, feature selection and hyperparameter tuning will be incorporated in order to enhance performance and prevent overfitting. This will be part of our pipeline process that was already established in the previous step with normalization and category encoding.

Feature Selection Strategy:¶

Based on the initial analyses regarding feature selection in step 6, six features will be selected for the final models. By using FeatureUnion, SelectKBest and RFE are combined within the pipeline, ensuring relevant features are selected during cross-validation. This integration enhances the accuracy of the final models and prevents data leaks.

Hyperparameter Tuning:¶

RandomizedSearchCV will be used for hyperparameter tuning, suitable for large datasets, and it shows much faster execution times compared to more complex Grid search. This method uses a random search approach to parameter tuning that samples algorithm parameters from a random distribution for a fixed number of iterations and then models are constructed and evaluated for each combination of specified parameters.

Logistic Regression (LR):¶

• solver: 'liblinear' is good for smaller datasets; 'sag' and 'saga' are faster for large ones.
• penalty: both 'l1' and 'l2' were tested. C (Regularization Strength): Tested [0.01, 0.1, 1].

K-Nearest Neighbors (KNN):¶

• n_neighbors: as the model showed sign of overfitting during the initial base evaluation, higher values [50, 75, 100, 125, 200, 250] will be used to find optimal neighbors.
• weights: 'uniform' and 'distance' to assess performance improvement.
• algorithm: default 'auto' was used as it will attempt to decide the most appropriate algorithm based on the values passed to the fit method.

Decision Tree Classifier (CART):¶

• criterion: Both 'gini', 'entropy' for split quality.
• splitter: default 'best' will be used. This strategy chooses the best split at each node.
• max_depth: default is 'none', where nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split. An array [1, 2, 3, 4, 5] will be tested to control tree depth. For example, a depth of 1 might underfit the data as it may not capture complex patterns but is less likely to overfit, which is the current problem. Depths of 3 or 4 balance complexity and the ability to capture patterns and are often considered good starting points. Depths of 10 or more create a complex decision boundary and are prone to overfitting, which will be avoided as overfitting is already an issue.
• min_samples_split: controls the minimum number of samples required to split an internal node. Smaller samples create more complex trees and are more prone to overfitting. An array [80, 100, 120] will be used to ensure adequate samples for splits.
• min_sample_leaf: the minimum number of samples required to split and internal node.

• The best parameters for LR were selected saga, with l1 penalty and c of 0.01.

• The best parameters for KNN were selected as distance weights and n_neighbors set to 50. After further evaluation through trial and error, the model still showed overfitting in cross-validation scores, so the default weight of "uniform" was selected and n_neighbors was set to a higher value of 100.

• The best parameters for CART were selected by RandomizedSearchCV as min_samples_split of 120, min_samples_leaf of 50, max_depth of 5, and criterion entropy. High focus was on addressing model overfitting, as this was the main concern after the initial base model evaluation. The model showed a good match between training scores and cross-validation scores. Max_depth was further decreased to 4, as with a depth of 5, there was a high discrepancy between false positives and false negatives.

4. Evaluating models and analyzing the results¶

Final model evaluation is done on unseen test data that was extracted using the train-test split method in a 70/30 ratio, one of the first steps before the initial analyses. As part of the baseline and final model evaluations, a confusion matrix with predictions on the x-axis and actual outcomes on the y-axis was used. Our binary classification problem involved True Positives, False Negatives, False Positives, and True Negatives. A second report, "Area Under ROC Curve," was generated to evaluate the model's ability to differentiate between positive and negative classes. Accuracy is measured by the area under the ROC curve, with anything above 0.9 representing an excellent score. Since this is a classification problem, a classification report was also generated to gain quick insight into the model's accuracy using precision, recall, f1-score, and support for each class. Finally, a learning curve was generated to show the model's performance on the training data in comparison with the unseen test data, represented by the cross-validation score.

The best-performing model is CART, which initially showed significant overfitting. However, after hyperparameter tuning and feature selection, a mean accuracy of 94.43% was achieved. Precision, which focuses on the accuracy of positive predictions and minimizes false positives, is higher for "No Churn." Recall, which shows the model's ability to capture all actual positives and minimizes false negatives, is higher for "Churn." The F1 score indicates balanced performance for both classes, demonstrating good overall accuracy. The confusion matrix shows some discrepancies between false positives and false negatives but shows much better results compared to KNN. The ROC AUC of 0.986 demonstrates an excellent ability to distinguish between classes. The learning curve shows that the training and cross-validation scores are very close, indicating minimal overfitting and good generalization. Based on these observations, CART, after hyperparameter tuning and feature selection, is recommended due to its balanced performance.

KNN model shows a good overall performance with a test accuracy of 91.98% and a ROC AUC of 0.972, the confusion matrix reveals an imbalance in false positives (1257) and false negatives (291). This suggests that the model is more prone to misclassifying 'Churn' instances. Despite applying different values of n_neighbors, this imbalance persisted.

LR model, when evaluated on the unseen test data, showed a test accuracy of approximately 82.48%. The cross-validation score is quite consistent with the training score and the confusion matrix reveals balance between false positives and false negatives. However, as expected from the initial analysis, the LR model has the lowest accuracy among the models tested. This aligns with our prediction since the data does not follow a Gaussian distribution, and models like KNN and CART, which do not assume a specific data distribution, were anticipated to perform better.

Final resolution: CART and KNN are recommended models.

Overall, the accuracy of KNN and CART is much better than LR, which is why these models were primarily selected. KNN is a simple and easy-to-implement algorithm that showed good results for our classification problem. It is also easier to explain and understand, as it predicts the label of the test dataset by looking at the labels of its closest neighbors in the feature space of the training dataset. Even non-technical persons can quickly grasp the concept with the important parameter of n_neighbors.

Decision trees, on the other hand, showed even better results than KNN, but they are more demanding in terms of computation power and hyperparameter tuning, with a higher risk of overfitting and bias towards dominant classes. CART is relatively easy to interpret even for non-technical stakeholders as well. Another benefit is that CART does not make any assumptions about the distribution and can handle both categorical and numerical features without requiring feature scaling, which simplifies the data preprocessing phase. In contrast, KNN needs to be normalized. Additionally, CART can be used for feature importance, whereas KNN does not have this option, and other selection techniques, such as SelectKBest, need to be used.