What if the AI answers are wrong?

Cartoon Nr. 1838 from the xkcd [1] Webcomic by Randall MUNROE describes the state-of-the-art in AI/machine learning of today in an brilliantly sarcastic manner and brings the current main open problem straight to the end: You always will get out results from any of your machine learning models. The main question remains open: “What if the answers are wrong?”. This may be ignored in some domains, desired in other domains, but in a safety-critical domain, e.g. in the medical domain, this is crucial [3].

[1] https://xkcd.com

[2] https://en.wikipedia.org/wiki/Randall_Munroe

[3] https://arxiv.org/abs/1712.09923v1

There is also a discussion on this image: