The Xai paradox: Systems that perform well for the wrong reasons

Research output: Contribution to conferencePaperAcademic


Many of the successful modern machine learning approaches can be described as ``black box'' systems; these systems perform well, but are unable to explain the reasoning behind their decisions. The emerging sub-field of Explainable Artificial Intelligence (XAI) aims to create systems that are able to explain to their users why they made a particular decision. Using artificial datasets whose internal structure is known beforehand, this study shows that the reasoning of systems that perform well is not necessarily sound. Furthermore, when multiple combined conditions define a dataset, systems can preform well on the combined problem and not learn each of the individual conditions. Instead, it often learns a confounding structure within the data that allows it to make the correct decisions. With regards to the goal of creating explainable systems, however, unsound rationales could create irrational explanations which would be problematic for the XAI movement.
Original languageEnglish
Publication statusPublished - 2019
EventBNAIC/Benelearn Conference - Brussels, Belgium
Duration: 6-Nov-20198-Nov-2019


ConferenceBNAIC/Benelearn Conference
Internet address


Dive into the research topics of 'The Xai paradox: Systems that perform well for the wrong reasons'. Together they form a unique fingerprint.

Cite this