Home / Projects / AI Support

AI Validation · Operations Analytics

AI-Augmented Customer Support Operations Analytics

Measured performance defines where AI can assist and where human review must remain.

View governance decision ↓ View GitHub ↗
Role AI Analyst · Data Analyst
Data 28,587 support tickets
Validation 300 successful classifications
Best result 63.3% category accuracy
Decision Restrict automation

My approach

Build a controlled evaluation sample

I reduced 28,587 source tickets to 16,338 English records and evaluated 300 successful classifications.

Compare predictions with ground truth

I measured accuracy, precision, recall, F1, and confusion matrices instead of trusting model confidence.

Set the automation boundary from evidence

I retained human review for priority and department routing and qualified only a limited category case.

Governance decision

Do not automate an unvalidated decision.

Model confidence does not replace measured accuracy.

Validation results

Performance changes the implementation.

The weakest outputs remain under human control.

63.3%

Category accuracy.

38.0%

Priority accuracy.

27.7%

Department accuracy.

Decision evidence

Where can AI assist safely?

The dashboard separates observed operations, measured model performance, and recommendation evidence.

01 · Operating baseline

Is the opportunity material?

Why this visual exists. Ticket volume is kept separate from a scenario model for possible time and cost savings.

Customer support operations dashboard with ticket volumes and assumption-based AI-assisted triage scenarios.
Finding

The dataset has 16,338 English tickets. A 50% coverage scenario estimates 544.6 hours and £9,802.80 saved.

Interpretation

The workload is observed. The savings are simulated from explicit time and cost assumptions.

Decision

Use the scenario to size interest, not to justify automation before validation.

02 · Validation and routing

Which decisions pass the evidence gate?

Why this visual exists. Accuracy, confusion, coverage, and estimated error risk are shown beside the routing policy.

AI validation dashboard with category, priority, and department accuracy, threshold risk, routing policy, and confusion matrix.
Finding

Category accuracy is 63.3%. Priority is 38.0%, and department is 27.7%. Only 89 Request cases reach 100% observed category accuracy.

Interpretation

Model confidence is not a control. Empirical performance limits automation to one narrow category case.

Decision

Allow limited Request category routing. Keep priority, department, and other categories under human review.

03 · Root cause

Where should the team reduce demand?

Why this visual exists. Recurring themes are linked to evidence-bounded self-service actions, while wording-quality issues remain visible.

Root cause and recommendations dashboard with recurring support themes and evidence-bounded actions.
Finding

Integration support is the largest named theme at 66 tickets, or 22.0% of the classified sample. Unmapped wording is 30.3%.

Interpretation

The theme supports a content opportunity, not a forecast of ticket deflection.

Decision

Improve integration setup and troubleshooting guidance, then measure adoption and ticket outcomes.

Source: original Power BI screenshots from the project repository. Savings are scenarios; model results are from a 300-ticket evaluation sample.

Related concepts

Explore the controls behind the AI decision.

Implementation proof · Python

Convert measured accuracy into a routing rule.

The policy uses empirical accuracy and sample size. Predictions that fail either condition remain under human review.

View implementation on GitHub ↗ 
recommendations = accuracy.copy()
eligible = (
    (recommendations["empirical_accuracy"] >= decision_threshold)
    & (recommendations["ticket_count"] >= minimum_sample_tickets)
)
recommendations["routing_recommendation"] = eligible.map(
    {True: "Auto-route category", False: "Human review"}
)