Plan for Failure to Succeed with Artificial Intelligence Healthcare – ICTworks

Recent advances in Artificial Intelligence (AI) suggest that AI applications could transform healthcare delivery in the Global South. However, as researchers and technology companies rush to develop AI applications that improve the health of marginalized communities, it is critical to consider the needs and perceptions of the community health workers (CHWs) who will have to integrate these AI applications into the essential healthcare services they provide to rural communities.

What do artificial intelligence designers and developers need to know to effectively create appropriate AI systems for community health workers? First and foremost, Community Health Worker Perceptions of AI-Enabled Mobile Health Applications in Rural India suggests an urgent need for AI developers to ensure they have a deep understanding of the context in which they plan to deploy an AI system.

While this is true of all ICT4D research, AI technologies present new societal risks and complexities (e.g., inequality, fairness, accountability, transparency, unintended con- sequences, etc.) that must be proactively studied before attempting deployment.

For example, this study suggests that AI developers would do well to plan for failure. Potential failures include both the possibility of the app delivering an incorrect diagnosis (i.e., misclassification) and the possibility of out-of-app failures due to infrastructural challenges (i.e., no connectivity, phone malfunction).

In the face of an error, most CHWs said they would simply repeat the procedure until they achieved the desired outcome, something that they assumed they would intuitively know. However, given CHWs low levels of AI knowledge and technology know-how, and their strong positive feelings towards the technology, a more likely and concerning outcome may be that they simply do not challenge the outcome delivered by the AI system.

Thus, rather than considering the potential for system failures as an afterthought or unlikely occurrence, it is important for AI developers to systematically and proactively identify, assess, and mitigate both the failures themselves and potential harms caused by such failures in AI-based products and services, especially when those failures may be invisible to users and have serious consequences for patients.

This study also suggests that it is crucial for organizations that plan to deploy AI mHealth systems to carefully consider and plan for sustainability, maintenance, and repair of these systems. As with any new technology, deployments in the wild will require constant technical support and maintenance.

CHWs frequently said that they would want to be able to call the company for assistance. Without such scaffolding, any AI intervention is bound to fail. While the need to plan for maintenance and repair is true of all technology deployments, especially in ICT4D, the complexity of troubleshooting and maintaining complex AI software may require continued involvement of highly-skilled AI designers and developers.

It is unlikely that local repair ecosystems, such as those that have emerged for mobile phone repair, will possess the tools or capabilities to appropriately troubleshoot complex AI systems.

Developers of AI systems will also need to pay close attention to the impact on CHWs work. CHWs are already burdened by heavy workloads and the introduction of new AI tools will inevitably increase this workload (even if the ultimate goal is to decrease it).

This study shows that deploying AI systems within CHWs workflows will likely result in additional work that is both visible (e.g., actually using the AI app) and invisible (e.g., explaining and justifying use of the AI app and its decisions/consequences to their communities). In addition, this work will be unevenly distributed across CHWs, with older, less tech-savvy CHWs likely spending more time doing invisible work as they struggle to operate the AI app.

AI developers need to account for this additional work and extra burdens shouldered by CHWs when weighing the benefits and harms of AI systems, and offer continued training and support.

A lightly edited synopsis of Community Health Worker Perceptions of AI-Enabled Mobile Health Applications in Rural India

Original post:

Plan for Failure to Succeed with Artificial Intelligence Healthcare - ICTworks