Scientists are urging caution before artificial intelligence (AI) models such as ChatGPT are used in healthcare for ethnic minority populations. Writing in the Journal of the Royal Society of Medicine, epidemiologists at the University of Leicester and University of Cambridge say that existing inequalities for ethnic minorities may become more entrenched due to systemic biases in the data used by healthcare AI tools.
AI models need to be ‘trained’ using data scraped from different sources such as healthcare websites and scientific research. However, evidence shows that ethnicity data are often missing from healthcare research. Ethnic minorities are also less represented in research trials.
Mohammad Ali, PhD Fellow in Epidemiology at the College of Life Sciences, University of Leicester, said: “This disproportionately lower representation of ethnic minorities in research has evidence of causing harm, for example by creating ineffective drug treatments or treatment guidelines which could be regarded as racist.”
“If the published literature already contains biases and less precision, it is logical that future AI models will maintain and further exacerbate them.”
The researchers are also concerned that health inequalities could worsen in low- and middle-income countries (LMICs). AI models are primarily developed in wealthier nations like the US and Europe and a significant disparity in research and development exists between high- and low-income countries.
The researchers point out that most published research does not prioritise the needs of those in the LMICs with their unique health challenges, particularly around healthcare provision. AI models, they say, may provide advised based on data on populations wholly different from those in LMICs.
While crucial to acknowledge these potential difficulties, say the researchers, it is equally important to focus on solutions. “We must exercise caution, acknowledging we cannot and should not stem the flow of progress,” commented Mr Ali.
The researchers suggest ways to overcome potentially exacerbating health inequalities, starting with the need for AI models to clearly describe the data used in their development. They also say work is needed to address ethnic health inequalities in research, including improving recruitment and recording of ethnicity information. Data used to train AI models should be adequately representative, with key actors such as ethnicity, age, sex and socioeconomic factors considered. Further research is also required to understand the use of AI models in the context of ethnically diverse populations.
By addressing these considerations, say the researchers, the power of AI models can be harnessed to drive positive change in healthcare while promoting fairness and inclusivity.
Read the full paper at: https://journals.sagepub.com/doi/full/10.1177/01410768231187734
