FAIRNESS AND BIAS

---

Goal: Check whether the model treats our two proxy groups equally. This is the core of the audit.

Metric 1: Statistical Parity Difference (SPD)

Does the model predict fraud at the same rate for both groups?

SPD = predicted fraud rate for Group A minus predicted fraud rate for Group B.

If SPD is 0, perfect equality. Our threshold: absolute value of SPD must be below 0.10.

Metric 2: Equalized Odds Difference (EOD)

Does the model make the same quality of errors for both groups?

This checks two things simultaneously. The True Positive Rate (what percentage of actual fraud does the model catch per group?) and the False Positive Rate (what percentage of legitimate transactions does the model wrongly flag per group?).

EOD is the maximum difference in these error rates across groups. Our threshold: absolute value of EOD must be below 0.10.

SPD tells us whether one group is disproportionately flagged. EOD tells us whether the model is equally good at its job across all groups. In this audit, EOD is the more important metric. We are not just worried about who gets flagged. We are worried about who gets left unprotected.

---

# ── SECTION 1: LOAD CHECKPOINT ───────────────────────────────────────────────

import pandas as pd
import numpy as np
from scipy import stats
from sklearn.preprocessing import LabelEncoder
from scipy.stats import ks_2samp
from fairlearn.metrics import demographic_parity_difference, equalized_odds_difference
from sklearn.metrics import matthews_corrcoef, log_loss, confusion_matrix, precision_recall_curve, auc
import matplotlib.pyplot as plt
import seaborn as sns
bins_proxy   = [-np.inf, 0.0, 50397.0, np.inf]
labels_proxy = ['Low-Balance', 'Mid-Balance', 'High-Balance']

df = pd.read_csv('checkpoint.csv')

print(f'Checkpoint loaded: {df.shape[0]:,} rows x {df.shape[1]} columns')
print(df.columns.tolist())

Checkpoint loaded: 6,362,620 rows x 11 columns
['step', 'type', 'amount', 'nameOrig', 'oldbalanceOrg', 'newbalanceOrig', 'nameDest', 'oldbalanceDest', 'newbalanceDest', 'isFraud', 'isFlaggedFraud']

import pandas as pd

results = pd.read_csv('model_results.csv')

y_test = results['y_test']
y_pred = results['y_pred']
y_prob = results['y_prob']


s_test = results[['balance_group', 'tx_type_group']].astype(str)

print("All audit variables (including s_test) are now defined.")

All audit variables (including s_test) are now defined.

# Thresholds from SETUP, THRESHOLDS AND ENVIRONMENT Notebook section
THRESHOLDS = {
    'SPD':      0.10,   # Fairness
    'EOD':      0.10,   # Fairness
    'MCC':      0.50,   # Performance
    'LOG_LOSS': 0.40,   # Performance
    'PR_AUC':   0.70,   # Performance
    'KS':       0.30,   # Performance
}

fairness_results = {}

for proxy, label in [
    ('tx_type_group', 'Transaction Type (Economic Role Proxy)'),
    ('balance_group',  'Account Balance Tier (Wealth Proxy)')
]:
    sf = s_test[proxy].astype(str).values
    spd = demographic_parity_difference(y_test, y_pred, sensitive_features=sf)
    eod = equalized_odds_difference(y_test, y_pred, sensitive_features=sf)

    spd_pass = abs(spd) < THRESHOLDS['SPD']
    eod_pass = abs(eod) < THRESHOLDS['EOD']

    print(f'\n{"=" * 60}')
    print(f'Fairness: {label}')
    print('=' * 60)

    groups = sorted(np.unique(sf))
    if proxy == 'balance_group':
        order  = ['Low-Balance', 'Mid-Balance', 'High-Balance']
        groups = [g for g in order if g in groups]

    group_stats = {}
    print(f'\n{"Group":<14} {"N":>8} {"Actual Fraud":>13} '
          f'{"Pred Fraud":>11} {"TPR":>8} {"FPR":>10} {"Pred Rate":>10}')
    print('-' * 78)

    for g in groups:
        mask = sf == g
        yt_g = y_test.values[mask]
        yp_g = y_pred[mask]
        tp_g = ((yt_g == 1) & (yp_g == 1)).sum()
        fn_g = ((yt_g == 1) & (yp_g == 0)).sum()
        fp_g = ((yt_g == 0) & (yp_g == 1)).sum()
        tn_g = ((yt_g == 0) & (yp_g == 0)).sum()
        tpr_g      = tp_g / (tp_g + fn_g) if (tp_g + fn_g) > 0 else 0
        fpr_g      = fp_g / (fp_g + tn_g) if (fp_g + tn_g) > 0 else 0
        pred_rate_g = yp_g.mean()
        group_stats[g] = {
            'n': mask.sum(), 'tp': tp_g, 'fn': fn_g,
            'fp': fp_g, 'tn': tn_g,
            'tpr': tpr_g, 'fpr': fpr_g, 'pred_rate': pred_rate_g
        }
        print(f'{g:<14} {mask.sum():>8,} {yt_g.sum():>13,} {yp_g.sum():>11,} '
              f'{tpr_g:>8.4f} {fpr_g:>10.6f} {pred_rate_g:>10.4%}')

    print(f'\nSPD = {spd:+.4f} | Threshold |SPD| < {THRESHOLDS["SPD"]} '
          f'| {"PASS" if spd_pass else "FAIL"}')
    print(f'EOD = {eod:+.4f} | Threshold |EOD| < {THRESHOLDS["EOD"]} '
          f'| {"PASS" if eod_pass else "FAIL"}')

    fairness_results[proxy] = {
        'spd': spd, 'eod': eod,
        'spd_pass': spd_pass, 'eod_pass': eod_pass,
        'group_stats': group_stats
    }

print('\nFairness metrics computed.')

============================================================
Fairness: Transaction Type (Economic Role Proxy)
============================================================

Group                 N  Actual Fraud  Pred Fraud      TPR        FPR  Pred Rate
------------------------------------------------------------------------------
CASH_OUT        447,193           798       3,656   0.9900   0.006420    0.8175%
OTHER           825,331           845       4,592   0.9953   0.004550    0.5564%

SPD = +0.0026 | Threshold |SPD| < 0.1 | PASS
EOD = +0.0053 | Threshold |EOD| < 0.1 | PASS

============================================================
Fairness: Account Balance Tier (Wealth Proxy)
============================================================

Group                 N  Actual Fraud  Pred Fraud      TPR        FPR  Pred Rate
------------------------------------------------------------------------------
Low-Balance     420,537             6         172   0.6667   0.000399    0.0409%
Mid-Balance     419,077           225       4,390   0.9733   0.009958    1.0475%
High-Balance    432,910         1,412       3,686   0.9972   0.005279    0.8514%

SPD = +0.0101 | Threshold |SPD| < 0.1 | PASS
EOD = +0.3305 | Threshold |EOD| < 0.1 | FAIL

Fairness metrics computed.

# ── SECTION 8: FAIRNESS VISUALIZATION ────────────────────────────────────────
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
fig.suptitle(
    'Section 8 — Fairness Metrics: SPD and EOD\n'
    'ClearBoxAI Audit CBA-2026-002',
    fontsize=13, fontweight='bold'
)

colors_map = {
    'CASH_OUT':    '#FF6B6B',
    'OTHER':       '#4ECDC4',
    'Low-Balance': '#FF6B6B',
    'Mid-Balance': '#FFC107',
    'High-Balance':'#4CAF50'
}

for row_idx, (proxy, res) in enumerate(fairness_results.items()):
    gs     = res['group_stats']
    groups = list(gs.keys())
    if proxy == 'balance_group':
        order  = ['Low-Balance', 'Mid-Balance', 'High-Balance']
        groups = [g for g in order if g in gs]
    bar_colors = [colors_map.get(g, '#90CAF9') for g in groups]

    tprs = [gs[g]['tpr'] for g in groups]
    axes[row_idx, 0].bar(groups, tprs, color=bar_colors, edgecolor='black', alpha=0.85)
    axes[row_idx, 0].set_title(
        f'{proxy}\nTrue Positive Rate per Group\n'
        f'EOD={res["eod"]:.4f} | {"PASS" if res["eod_pass"] else "FAIL"}',
        fontsize=9, fontweight='bold'
    )
    axes[row_idx, 0].set_ylabel('TPR (fraud caught per group)')
    axes[row_idx, 0].set_ylim(0, 1.2)
    for i, v in enumerate(tprs):
        label = f'{v:.4f}' if v > 0 else 'ZERO'
        color = '#C62828' if v == 0 else 'black'
        axes[row_idx, 0].text(i, v + 0.03, label,
                              ha='center', fontsize=9, fontweight='bold', color=color)

    pred_rates = [gs[g]['pred_rate'] for g in groups]
    axes[row_idx, 1].bar(groups, pred_rates, color=bar_colors, edgecolor='black', alpha=0.85)
    axes[row_idx, 1].set_title(
        f'{proxy}\nPredicted Fraud Rate per Group\n'
        f'SPD={res["spd"]:+.4f} | {"PASS" if res["spd_pass"] else "FAIL"}',
        fontsize=9, fontweight='bold'
    )
    axes[row_idx, 1].set_ylabel('Predicted fraud rate')
    for i, v in enumerate(pred_rates):
        axes[row_idx, 1].text(i, v + 0.0001, f'{v:.4%}',
                              ha='center', fontsize=9, fontweight='bold')

plt.tight_layout()
plt.savefig('fig_fairness_01.png', dpi=150, bbox_inches='tight')
plt.show()

===

Finding: Fairness and Bias Audit

Finding 2a — Transaction Type: CASH_OUT vs OTHER

Result: Both metrics pass.

Metric	Value	Threshold	Result
SPD	+0.0026	below 0.10	PASS
EOD	+0.0053	below 0.10	PASS

Per-Group Performance Breakdown (Transaction Type)

Group	N (Samples)	Actual Fraud	TPR	FPR	Predicted Rate
CASH_OUT	447,193	798	0.9900	0.006420	0.8175%
OTHER	825,331	845	0.9953	0.004550	0.5564%

The model treats informal economy users (CASH_OUT) and formal economy users (OTHER) almost identically. CASH_OUT users have a slightly lower TPR 99.00% vs 99.53% but the difference is 0.53 percentage points. That is well within acceptable limits and does not constitute a fairness failure.

CASH_OUT users are flagged at a slightly higher rate (0.8175% vs 0.5564%) which SPD captures at +0.0026 essentially zero difference in practical terms.

This is a meaningful finding in itself. The model does not discriminate between informal and formal economy users at the transaction type level. The fairness failure, when we find it, is not here. It is in the balance tier.

---

Finding 2b — Account Balance Tier: Low-Balance / Mid-Balance / High-Balance

Result: EOD fails. Critical finding.

Metric	Value	Threshold	Result
SPD	+0.0101	below 0.10	PASS
EOD	+0.3305	below 0.10	FAIL more than three times the threshold

Per-Group Performance Breakdown

Group	N (Samples)	Actual Fraud	TPR	FPR	Predicted Rate
Low-Balance	420,537	6	0.6667	0.000399	0.0409%
Mid-Balance	419,077	225	0.9733	0.009958	1.0475%
High-Balance	432,910	1,412	0.9972	0.005279	0.8514%

---

TPR Disparity Analysis

The True Positive Rate (TPR) gradient tells the story clearly:

• High-Balance: 99.72% fraud detection
• Mid-Balance: 97.33% fraud detection
• Low-Balance: 66.67% fraud detection

A TPR of 0.6667 means the model correctly identified 4 out of 6 fraud cases in the Low-Balance group.

At first glance, this may appear acceptable. However, two critical factors invalidate that interpretation:

1. Extremely small sample size

   • Only 6 fraud cases exist in the Low-Balance test set
   • The per-group MCC from the Performance Evaluation section is 0.1245, indicating performance barely above random
   • MCC provides a more reliable signal under class imbalance

2. Severe disparity across groups

   • Difference between best and worst group:
     99.72% – 66.67% = 33 percentage points
   • Equal Opportunity Difference (EOD): 0.3305
   • This exceeds the 0.10 fairness threshold by more than 3×

This constitutes a material fairness failure, independent of sample size limitations.

---

Interpretation

Consider two individuals who both experience fraud on their mobile money accounts:

• A customer with GHS 80,000:
  The model is highly likely to detect the fraud. Investigation begins. The fraudster is pursued.

• A customer with GHS 150:
  The model is significantly less reliable. The fraud is far more likely to go undetected. The fraudster may face no consequences.

This is not a marginal statistical fluctuation.
It is a systematic disparity in protection.

---

Root Cause Analysis

This outcome is driven by severe data imbalance:

• Total Low-Balance fraud cases in dataset: 41
• Training set: 35 cases
• Test set: 6 cases
• High-Balance fraud cases: 5,738

Imbalance ratio: ~164 : 1

Even with SMOTE applied:

• SMOTE generates synthetic samples by interpolating existing data
• With only 35 Low-Balance fraud cases, very few synthetic examples were created
• The imbalance was effectively preserved or amplified

As a result:

• The model learned High-Balance fraud patterns in detail
• It learned Low-Balance fraud patterns poorly and unreliably

The observed:

• TPR = 0.6667
• MCC = 0.1245

both reflect this thin representation problem.

This is a structural data coverage issue and will persist across retraining unless explicitly addressed.

---

Regulatory Status

Regulation	Provision	Status
BoG CISD 2026	Annexure E §l(i) – Material bias	TRIGGERED
BoG CISD 2026	§115(2)(b) – Notify regulator	REQUIRED
BoG CISD 2026	Annexure E §e(i)(3) – Fairness risk identified	CONFIRMED
NIST AI RMF 1.0	MEASURE 2.11 – Bias documented	Complete
NIST AI RMF 1.0	§3.7 – Harmful bias managed	Breached
EU AI Act 2024/1689	Article 6(3) – High-risk classification	Reference

---

Risk Assessment

Risk Level: HIGH

• EOD = 0.3305
• Breaches fairness threshold (0.10) by >3×
• Meets criteria for material bias

Under BoG CISD 2026 Annexure E §l(i):

Model deployment must be suspended until effective mitigation is implemented.

---

Auditor: Kwadwo Amponsah
Organization: ClearBoxAI
Date: April 2026

df.to_csv('checkpoint_v2.csv', index=False)
print("Checkpoint v2 saved.")

Checkpoint v2 saved.