Spark Fraud Detection | Naim Makhlouta

The need

Fraud is extremely rare, which makes accuracy misleading: a model that always answers “not fraud” scores 99.8% and catches nothing. Real transaction volumes also call for tools that scale beyond one machine’s memory.

What I owned

The full pipeline, from Spark ingestion and analysis through to training and evaluation.

The solution

Ingest

Spark DataFrameClass distribution check

Explore

Correlation matricesBoxplots per featureMissing value check

Prepare

VectorAssemblerStandardScaler80/20 splitClass weights

Train and evaluate

Logistic regressionGradient-boosted treesPR AUC and fraud F1

Each model was trained on both the full data and a reduced, more balanced subset.

How it was built

On the full data, correlations with fraud are washed out by the majority class. Comparing against a moderately balanced subset of 2,982 rows revealed which features really separate the classes.

Correlation matrix on imbalanced data, nearly blank off the diagonal — Full data: the signal is hidden by the imbalance.

Correlation matrix on the balanced subset with strong patterns — Balanced subset: strong correlations with fraud appear.

Results

Model	Data	Fraud precision	Fraud recall	Fraud F1	PR AUC
Logistic regression	Full	0.11	0.89	0.20	0.63
Gradient-boosted trees	Full	0.90	0.76	0.82	0.82
Logistic regression	Reduced	0.90	0.89	0.90	0.95
Gradient-boosted trees	Reduced	0.94	0.89	0.92	0.96

On the realistic full data, GBT performed best, with 74 of 98 frauds caught and only 8 false alarms out of nearly 57,000 legitimate transactions.
Weighted logistic regression shows the trade-off, catching more fraud (87 of 98) but raising 681 false alarms.
The reduced-data scores are higher, but they come from a test set that does not reflect the real fraud rate, so they should not be read as production performance.