Validation of "Neural Networks based Wire Payment Fraud Detection Model" for strategic and regulatory submission for a large US based Financial Institution
Client : Large US Financial Institution
Objective
To validate a neural-network-based wire payment fraud detection model for regulatory submission and enhanced strategic decision-making. The goal was to validate the bank’s model for two large US portfolios and provide a decision on the model's use.
CRISIL's Solution
CRISIL GR&RS performed validation of the wire payment fraud detection model. This involved:
- Reviewing all documents submitted for validation and assessing whether more information was required from stakeholders to complete the thorough validation process;
- Holding regular discussions with stakeholders to ensure all ambiguities in their reports were removed before performing additional tests - at CRISIL’s suggestion - to evaluate the model's performance.
Validation Methodology
- Thorough study of the model development document and other documents and research articles relevant to understanding of the model;
- Review all tests done by the developers for evaluating the neural networks model performance;
- Prescribe and review results of performance tests other than those done by the model developer.
Validation Highlights
- The neural network model was found to be not performing well
Client Impact
- Model validation completed in short timeframe
- Significant shortcomings in model performance identified
- CRISIL provided insights and suggestions, including new performance metrics to evaluate fraud model and enhancements to the quality of model documentation
- Created a thorough validation document to be used by the client for CCAR submission
- Provided specific recommendations about future use of the model in the two target portfolios