Bayesian stress testing machine learning solutions – new paper

PUBLISHED ON

‘Bayesian stress testing of models in a classification hierarchy’ is our latest paper and has been selected to appear in the proceedings of the International Joint Conference of Neural Networks (IJCNN) 2020, part of the World Congress on Computational Intelligence (WCCI). This post summarises the contents and key points.

The paper was written by Kate Kelly and myself and you can read the full paper here. The code used to produce the results in this paper is also available here.

This is invaluable research that is applied directly to the automated AML investigation technology that we develop to help banks around the world to fight the money laundering and financial crime.

Why stress test machine learning solutions?

The reality of building machine learning driven products, involves building more than one model to solve the problem at hand. For example, to identify a relationship between two individuals in a text: first you need a model to identify the names in the text as entities (named entity recognition), followed by a classification model to predict the relationship between these two entities as expressed by the text.

When we start to rely on more than one model in our solution, where the output of one model becomes the input to another model(s), then propagation error becomes a real challenge. Other practical issues also arise:

  • What will be the impact of changes to the input data for one model on the overall performance of the solution?
  • What is the contribution of each model to the overall performance?
  • How to prioritise the improvement of the models in order to make the biggest impact on performance of the solution?

In the paper we present a Bayesian framework to stress test machine learning solutions. The framework is generic and agnostic to the modelling methods used.

For more details on inference and sampling please refer to the full paper here and for those interested, you can also access the code used to produce the results in the paper here.

Additional papers and information about the research work we do in partnership with academia is available on our Research page.