Prediction on Customer Conversion (PySpark + ML)

One-line summary: Predict whether a customer will convert from a multi-channel digital marketing campaign, and identify the strongest conversion drivers.

---

Overview

This project uses a real-world digital marketing dataset to:

1. understand what drives customer conversion, and
2. predict conversion using two modeling approaches: Logistic Regression (interpretable baseline) and XGBoost (non-linear, higher performance).

---

Problem statement

Marketing teams run campaigns across channels (email, social, web, etc.), but often struggle to answer:

• Which customer behaviors and campaign attributes matter most for conversion?
• How accurately can we predict conversion using scalable modeling workflows?

---

Dataset

Local dataset stats (from the provided CSV)

• Rows: 8,000
• Columns: 20
• Target column detected: Conversion
• Class balance:
  • 0: 12.35%
  • 1: 87.65%

Columns:

• CustomerID
• Age
• Gender
• Income
• CampaignChannel
• CampaignType
• AdSpend
• ClickThroughRate
• ConversionRate
• WebsiteVisits
• PagesPerVisit
• TimeOnSite
• SocialShares
• EmailOpens
• EmailClicks
• PreviousPurchases
• LoyaltyPoints
• AdvertisingPlatform
• AdvertisingTool
• Conversion

Source: Kaggle – “Predict Conversion in Digital Marketing Dataset”

• Size: ~8,000 rows, ~20 variables
• Target: Conversion (1 = converted, 0 = not)

Put your dataset file here:

data/raw/digital_marketing_campaign_dataset.csv

Note: The original notebook used a local path; this repo structure makes it portable.

---

Tools and technologies

• Python
• PySpark (SparkSession, Spark DataFrames)
• Pandas / NumPy
• Matplotlib / Seaborn (EDA visuals)
• statsmodels (Logistic Regression with p-values)
• scikit-learn (split, metrics, cross-validation)
• XGBoost + GridSearchCV (tuning + feature importance)

---

Methods

1) Data preprocessing

• Null checks
• Dropped low-value identifiers (e.g., CustomerID)
• Encoded categorical features (CampaignChannel, CampaignType) via one-hot encoding
• Train/test split (80/20), stratified on the target

2) Exploratory analysis (EDA)

We compare key behaviors between converters vs non-converters.

Example plots you can include in images/:

• images/eda_previous_purchases.png
• images/eda_email_clicks.png
• images/eda_adspend.png
• images/eda_time_on_site.png

Sample (add your exported figures):

3) Modeling

A) Logistic Regression (interpretable baseline)

• Fit initial model with full feature set
• Backward elimination (drop insignificant variables step-by-step while monitoring test error)
• Evaluate with classification report, confusion matrix, and ROC-AUC

B) XGBoost (tuned)

• GridSearchCV for hyperparameter tuning
• Evaluate with classification report + ROC-AUC
• Feature importance for interpretability

---

Key insights

From EDA + modeling, the strongest signals of conversion are typically engagement and intent variables such as:

• EmailClicks / EmailOpens
• PreviousPurchases
• TimeOnSite
• PagesPerVisit
• ClickThroughRate / ConversionRate
• AdSpend
• Conversion-focused campaign type (vs awareness/retention)

Demographics (age, income) were comparatively less predictive in this dataset.

---

Dashboard / Model / Outputs

This repo is structured so you can store:

• Exported figures in images/
• Saved models in models/
• Final writeup or report in reports/

Recommended outputs:

• images/roc_logistic.png
• images/roc_xgboost.png
• images/xgb_feature_importance.png
• reports/final_report.pdf

---

How to run this project

Option A — Run the notebook

1. Create environment

python -m venv .venv
source .venv/bin/activate   # (Windows: .venv\Scripts\activate)
pip install -r requirements.txt

2. Place dataset

# Put the Kaggle CSV here:
data/raw/digital_marketing_campaign_dataset.csv

3. Open notebook

jupyter notebook notebooks/01_customer_conversion_pyspark.ipynb

Option B — Run as scripts (recommended for GitHub readers)

python src/run_pipeline.py --data data/raw/digital_marketing_campaign_dataset.csv

---

Results and conclusions

• Logistic Regression: strong interpretability, weaker detection of non-converters (class imbalance impact).
• XGBoost (tuned): better overall discrimination (higher ROC-AUC), stronger performance on imbalanced classification.

This project demonstrates a practical workflow to go from a multi-channel marketing dataset → EDA → interpretable baseline → tuned model → actionable insights.

---

Project structure

customer-conversion-prediction/
├─ data/
│  ├─ raw/                # original dataset (not committed)
│  └─ processed/          # cleaned/encoded outputs (optional)
├─ notebooks/             # readable notebooks for learning
├─ src/                   # reusable scripts (pipeline-style)
├─ images/                # figures used in README
├─ reports/               # final report / writeup
├─ models/                # saved models (optional)
├─ docs/                  # extra documentation
├─ requirements.txt
└─ README.md

---

Credits

Course project: BAN 5600 – Advanced Big Data Computing and Programming (Clark University)

Dataset credit goes to the Kaggle dataset author.