Audit ID: CBA-2026-002 · Model Risk and Compliance · April 2026

Mobile Money Fraud Detection Audit

An independent fairness and performance audit of an XGBoost fraud detection model trained on 6.3 million mobile money transactions, evaluated against BoG CISD 2026, NIST AI RMF 1.0, and the EU AI Act.

Headline Finding

The model achieves 99.27% overall fraud recall but provides significantly weaker protection to low-balance users. Low-Balance TPR = 0.6667 versus High-Balance TPR = 0.9972. Miss rate disparity: 33.3% vs 0.3%, a 110-fold gap. Equalized Odds Difference = 0.3305, exceeding the 0.10 regulatory threshold by three times. BoG CISD 2026 Annexure E §l(i) mandatory suspension triggered.

Audit Scope
  • Dataset: PaySim synthetic mobile money, 6,362,620 transactions.
  • Model: XGBoost, 300 trees, SMOTE oversampling.
  • Proxies: Transaction type (CASH_OUT vs OTHER) and account balance tier.
  • Frameworks: BoG CISD 2026, NIST AI RMF 1.0, EU AI Act 2024/1689.
Key Metrics
  • PR-AUC: 0.9357 — 725x better than random. PASS.
  • KS Statistic: 0.9916 — near-perfect score separation. PASS.
  • MCC: 0.4419 — high false positives, threshold calibration needed. FAIL.
  • EOD (balance tier): 0.3305 — three times over threshold. FAIL.

Phase 1

Pre-Training

Phase 2

Training

Phase 3

Post-Training
Phase 1 — Pre-Training
SectionDescriptionStandardOutcomeArtifact
Scope and Regulatory Alignment Audit objectives, proxy variable design, regulatory framework mapping BoG CISD 2026 Annexure E §e(i)(3) COMPLETE View
Environment Setup and Thresholds Library setup, pass/fail thresholds defined before any results seen BoG CISD 2026 §122(1) COMPLETE View
Data Audit and Integrity Checks Missing values, duplicates, leakage detection, column exclusions documented BoG CISD 2026 Annexure E §f(i) CLEAN View
Pre-Training Proxy Detection Chi-square and Cramer's V confirming structural fraud imbalance before training BoG CISD 2026 Annexure E §g(iii)(3) IMBALANCE CONFIRMED View
Exploratory Data Analysis Transaction type analysis, temporal patterns, amount distributions, zero-balance logic, correlation heatmap, Invisible 41 discovery NIST AI RMF MAP 1.1 41 LOW-BALANCE FRAUD CASES View
Feature Engineering Audit 13 features engineered and documented. Implicit wealth correlation flagged on balance-derived features. BoG CISD 2026 Annexure E §j(ii) RISK DOCUMENTED View
Phase 2 — Training
SectionDescriptionStandardOutcomeArtifact
Model Training Audit XGBoost 300 trees, SMOTE applied. 164:1 High-Balance to Low-Balance fraud ratio in training confirmed. SMOTE limitation documented. BoG CISD 2026 Annexure E §g(iii)(1) STRUCTURAL GAP CONFIRMED View
Phase 3 — Post-Training
SectionDescriptionStandardOutcomeArtifact
Performance Evaluation MCC 0.44 FAIL. PR-AUC 0.9357 PASS. KS 0.9916 PASS. Log Loss 0.0135 PASS. Per-group MCC confirms Low-Balance underperformance. NIST AI RMF MEASURE 2.3 MCC FAIL · 3 of 4 PASS View
Fairness and Bias Audit Transaction type PASS. Balance tier EOD = 0.3305, three times over threshold. Low-Balance TPR = 0.6667 vs High-Balance 0.9972. BoG CISD 2026 Annexure E §l(i) HIGH RISK · SUSPENSION TRIGGERED View
Error Analysis Low-Balance miss rate 33.3% vs High-Balance 0.3%. 110-fold disparity confirmed. NIST AI RMF MEASURE 2.11 STRUCTURAL PROTECTION GAP View
Explainability SHAP analysis. Top features: newbalanceOrig, oldbalanceOrg, amount. Wealth correlation in balance features confirmed. BoG CISD 2026 Annexure E §j(i) RISK DOCUMENTED View
Deployment Readiness Assessment Threshold sensitivity analysis. 8 of 8 deployment prerequisites unmet under BoG CISD 2026. BoG CISD 2026 §100 NOT DEPLOYMENT READY View
Monitoring and Drift Plan Per-group monitoring framework, retraining triggers, and drift detection requirements documented. BoG CISD 2026 §100(6) FRAMEWORK DOCUMENTED View
Final Scorecard and Risk Register 5 of 8 metrics pass. EOD balance tier FAIL. Model not deployment-ready. BoG notification required. BoG CISD 2026 §115(2)(b) DO NOT DEPLOY View