Human-in-the-loop AI

Very interesting: in the recent April, 5, 2018, TWiML & AI (This Week in Machine Learning and Artificial Intelligence) podcast, Robert Munro reports on the newly branded Figure Eight [1] company, formerly known as CrowdFlower. Their Human-in-the-Loop AI platform supports data science & machine learning teams working on various topics, including autonomous vehicles, consumer product identification, natural language processing, search relevance, intelligent chatbots, and more, and most recently on disaster response and epidemiology. This is a further proof on the importance of the human-in-the-loop interactive machine Leanring (iML) approach! Awesome discussion led by Sam Charrington:

This fits well to the previous discussion with Jeff Dean – who emphasized the importance of health and the limits of automatic approaches including deep learning, listen directly at:



iML with the human-in-the-loop mentioned among 10 coolest applications of machine learning

Within the “Two Minute Papers” series, Karol Károly Zsolnai-Fehér from the Institute of Computer Graphics and Algorithms at the Vienna University of Technology mentions among “10 even cooler Deep Learning Applications” our human-in-the-loop paper:

Seid Muhie Yimam, Chris Biemann, Ljiljana Majnaric, Šefket Šabanović & Andreas Holzinger 2016. An adaptive annotation approach for biomedical entity and relation recognition. Springer/Nature: Brain Informatics, 3, (3), 157-168, doi:10.1007/s40708-016-0036-4

Watch the video here (iML is mentinoned from approx. 1:20):

Here the list of all 10 papers discussed within this 2-minutes-video

1. Geolocation –
2. Super-resolution –
3. Neural Network visualizer –…
4. Recurrent neural network for sentence completion:
5. Human-in-the-loop and Doctor-in-the-loop:
6. Emoji suggestions for images –
7. MNIST handwritten numbers in HD –
8. Deep Learning solution to the Netflix prize –
9. Curating works of art –
10. More robust neural networks against adversarial examples –…
The Keras library:

A) The basic principle of the iML human-in-the-loop approach:

Andreas Holzinger 2016. Interactive Machine Learning for Health Informatics: When do we need the human-in-the-loop? Brain Informatics, 3, (2), 119-131, doi:10.1007/s40708-016-0042-6

B) The entry in the GI Lexikon:

C) The experimental proof-of-concept:

Andreas Holzinger, Markus Plass, Katharina Holzinger, Gloria Cerasela Crisan, Camelia-M. Pintea & Vasile Palade 2017. A glass-box interactive machine learning approach for solving NP-hard problems with the human-in-the-loop. arXiv:1708.01104.

D) Outline and Survey of application possibilities:

Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis & Douglas B. Kell 2017. What do we need to build explainable AI systems for the medical domain? arXiv:1712.09923.

Andreas Holzinger, Bernd Malle, Peter Kieseberg, Peter M. Roth, Heimo Müller, Robert Reihs & Kurt Zatloukal 2017. Towards the Augmented Pathologist: Challenges of Explainable-AI in Digital Pathology. arXiv:1712.06657.


Obama on humans-in-the-loop

How artificial intelligence will affect jobs

In an discussion with Barack OBAMA [1] on how artificial intelligence will affect jobs, he emphasized how important human-in-the-loop machine learning will become in the future. Trust, transparency and explainabiltity will be THE driving factors of future AI solutions! The discussion interview was led by the Wired [2] Editor Scott DADICH, and MIT Media Lab [3] Director Joi ITO. I recommend my students to watch the full video. Barack Obama demonstrates a  good understanding of the field and indicates indirectly the importance of our research in the the human-in-the-loop approach [4], despite all progress towards fully automatic approaches and autonomous systems.

More information see:

[1] Barack Obama was the 44th President of the United States of America and was in office from January, 20, 2009 to January, 20, 2017. He was born August, 4, 1961 in Honolulu (Hawaii)

[2] Wired is a monthly tech magazine which reports since 1993 on how emerging technologies may affect culture, politics, economics. Very interesting to note is that Wired is known for coning the popular terms “long tail” and “crowdsourcing”.

[3] The MIT Media Lab is an interdisciplinary research lab at the Massachusetts Institute of Technology in Cambridge (MA), which is part of the Boston metropolitan area in the north, just across the Charles River – not far way from the Harvard Campus.

[4] Holzinger, A., Plass, M., Holzinger, K., Crisan, G.C., Pintea, C.-M. & Palade, V. 2017. A glass-box interactive machine learning approach for solving NP-hard problems with the human-in-the-loop. arXiv:1708.01104