Amuyla Yadav (Penn State)-Sidestepping the Black-Box: A New Paradigm for Explainable AI

Existing work in Explainable Artificial Intelligence (XAI) has been focused on developing techniques to interpret decisions made by pre-trained & black-box machine learning (ML) models. This black-box assumption is reasonable in a lot of settings, e.g., explaining Amazon’s recommender systems requires assuming a black-box model because it is infeasible to assume glass-box access to Amazon’s proprietary models, etc. However, I argue that in many real-world settings (especially those that pertain to low-resource domains), the black-box assumption is unnecessary, undesirable, and often, overly limiting. In this talk, I motivate the need to move away from the black-box assumption of XAI by discussing two deployed use cases of responsible AI research: (i) automated tele-triage for poor pregnant women in Kenya, and (ii) raising awareness of HIV among homeless youth in Los Angeles. Through my experiences with the deployment of AI in these domains, we will argue the need for a new paradigm in explainable AI. Next, I will discuss two new frameworks: (i) CounterNet, a novel end-to-end learning framework which integrates Machine Learning (ML) model training and the generation of corresponding counterfactual (CF) explanations into a single end-to-end pipeline; and (ii) RoCourseNet, a training framework that jointly optimizes predictions and recourses that are robust to future data shifts.