Modelling Uncertainty in Black-box Classification Systems

Autor/a

Mena Roldán, José

Director/a

Vitrià i Marca, Jordi

Pujol Vila, Oriol

Tutor/a

Torrent Moreno, Marc

Data de defensa

2020-12-15

Pàgines

148 p.



Departament/Institut

Universitat de Barcelona. Departament de Matemàtiques i Informàtica

Resum

Currently, thanks to the Big Data boom, the excellent results obtained by deep learning models and the strong digital transformation experienced over the last years, many companies have decided to incorporate machine learning models into their systems. Some companies have detected this opportunity and are making a portfolio of artificial intelligence services available to third parties in the form of application programming interfaces (APIs). Subsequently, developers include calls to these APIs to incorporate AI functionalities in their products. Although it is an option that saves time and resources, it is true that, in most cases, these APIs are displayed in the form of blackboxes, the details of which are unknown to their clients. The complexity of such products typically leads to a lack of control and knowledge of the internal components, which, in turn, can drive to potential uncontrolled risks. Therefore, it is necessary to develop methods capable of evaluating the performance of these black-boxes when applied to a specific application. In this work, we present a robust uncertainty-based method for evaluating the performance of both probabilistic and categorical classification black-box models, in particular APIs, that enriches the predictions obtained with an uncertainty score. This uncertainty score enables the detection of inputs with very confident but erroneous predictions while protecting against out of distribution data points when deploying the model in a productive setting. In the first part of the thesis, we develop a thorough revision of the concept of uncertainty, focusing on the uncertainty of classification systems. We review the existingrelated literature, describing the different approaches for modelling this uncertainty, its application to different use cases and some of its desirable properties. Next, we introduce the proposed method for modelling uncertainty in black-box settings. Moreover, in the last chapters of the thesis, we showcase the method applied to different domains, including NLP and computer vision problems. Finally, we include two reallife applications of the method: classification of overqualification in job descriptions and readability assessment of texts.


La tesis propone un método para el cálculo de la incertidumbre asociada a las predicciones de APIs o librerías externas de sistemas de clasificación.

Paraules clau

Sistemes classificadors (Intel·ligència artificial); Sistemas clasificadores; Learning classifier systems; Incertesa (Teoria de la informació); Incertidumbre (Teoría de la información); Uncertainty (Information theory); Aprenentatge automàtic; Aprendizaje automático; Machine learning

Matèries

004 - Informàtica

Àrea de coneixement

Ciències Experimentals i Matemàtiques

Nota

Programa de Doctorat en Matemàtica i Informàtica

Documents

JMR_PhD_THESIS.pdf

6.549Mb

 

Drets

L'accés als continguts d'aquesta tesi queda condicionat a l'acceptació de les condicions d'ús establertes per la següent llicència Creative Commons: http://creativecommons.org/licenses/by-nc/4.0/
L'accés als continguts d'aquesta tesi queda condicionat a l'acceptació de les condicions d'ús establertes per la següent llicència Creative Commons: http://creativecommons.org/licenses/by-nc/4.0/

Aquest element apareix en la col·lecció o col·leccions següent(s)