How to evaluate fraud detection and false positives

A fraud tool can flag more events while making the business worse off. To evaluate fraud detection false positives, compare known abuse, legitimate customers and missing results on the same action cohort. Report the denominators, action taken and label maturity; an overall accuracy number cannot answer all three questions.

Create labels outside the detector

Ground truth must come from evidence independent of the candidate score. Use confirmed operational outcomes, reviewed cases and controlled test identities where appropriate. Keep the label source, confidence and date. A high model score is not a confirmed fraud label.

Some observations remain unknown. A customer who retries after a decline may be legitimate, but a retry alone is not proof. Successful verification, an appeal investigation and final transaction evidence can contribute different strengths of support. Preserve uncertainty rather than forcing every observation into a clean binary dataset.

Decide what constitutes a false positive

Distinguish the model’s flag from the application’s restriction. A flagged legitimate session sent to a successful low-friction verification is different from a legitimate order permanently declined. Both can create cost, but they should not share a misleading denominator.

State whether an unknown case is excluded or counted separately. Otherwise two reports can use the same labels and produce different rates without an actual product difference.

A worked example with explicit units

Consider an illustrative cohort with 1,000 confirmed legitimate attempts and 100 confirmed abusive attempts. A candidate flags 80 abusive attempts and 20 legitimate attempts. Fraud recall is 80 out of 100, or 80 percent. If every flagged case has a usable label, alert precision is 80 out of 100, also 80 percent.

The legitimate flag rate is 20 out of 1,000, or 2 percent. If the application restricts only 10 of those legitimate attempts, the legitimate restriction rate is 1 percent. These are arithmetic examples, not measurements of a vendor. Reporting one of them as a generic “false-positive rate” hides the distinction.

Compare candidates on the same opportunity

Keep the protected event, available evidence and outcome window consistent. Record which inputs each candidate actually receives. A tool missing the account association cannot fairly be compared with another supplied a complete account history.

Start in an observation cohort or controlled rollout. Review latency, unavailable results and legitimate-user restrictions before expanding enforcement. The integration guide explains how result and event references prevent accidental joins between a browser visit and the wrong account action.

Separate the reasons a checkout failed

An issuer decline, merchant risk decision, failed authentication and customer abandonment have different causes. Keep them separate in the event record. A change in issuer approval can mask an improvement in merchant risk decisions, or vice versa.

Forter’s false-decline article discusses the relationship between identity assessment and legitimate approvals. Stripe’s Radar documentation describes controls within a payment workflow. Use those sources for product context; use your own comparable outcome set to measure impact.

Include delayed outcomes and segments

Payment disputes can arrive later than account events. Keep the cohort open long enough for the type of loss being studied and publish its maturity date. A recently approved order with no dispute yet is not the same as a mature legitimate label.

Segment by action, platform, region, new versus returning customer, privacy tools and shared networks. A good overall result can hide a weak browser or a high-friction customer segment. Also track collection availability: unavailable evidence is a product-operating outcome, not a legitimate negative.

Decide with cost and recovery in view

Estimate the cost of confirmed loss, legitimate restrictions, manual review and verification. Add the software and implementation cost. The cost guide supplies a budgeting model; the comparison identifies different product scopes.

For account abuse, tools such as ShieldLabs add device/network reasons and ready-made findings. Review the reasons and the customer’s recovery outcome together. A lower restriction count is valuable only when the business still controls the losses it set out to prevent.

FAQ