Professional conferences can be great places to get a lot of thinking done. You get together with a whole bunch of your colleagues, and you talk about all of the things that are important and exciting in your field. For me personally, conferences also tend to be lovely opportunities for reflection. The usual daily routines take a back seat to conference rhythms, and that creates a space for the mental momentum of conference discussions to spill over into areas beyond the topics on the meeting agenda. I’m going on my own anecdata a little bit, as I’ve recently written conference-inspired pieces here, here, and here.
I’ve just gotten home from the American Thoracic Society (ATS) international conference, my chance every year to get together with 14,000 of my closest friends to talk about lung diseases, ICU medicine, sleep medicine, environmental health, and a whole bunch of nerdy science. As I made my way home, I realized that I did less of that stuff this year than in any of the previous 15 years. This year I did a lot more advocacy and activism work, a lot more private sector facing stuff, and a LOT more thinking and talking about artificial intelligence.
The discussions about AI are a given these days, especially at scientific and medical conferences. Even as AI mania reaches an ever more feverish pitch, I find myself becoming increasingly disturbed by the incursion of AI into science and medicine. Not because the economics are largely unsupportable and likely to collapse like a dying star, though I think that’s true. And not because the C-suite decision makers in universities and health systems seem much more worried about not being left behind by the AI train than about actually understanding how AI can or cannot help real people, though I think that’s true too.
No, I worry most about two very particular things that are especially relevant to biomedical sciences and to the stewardship of the health of persons:
Neither science nor medicine are algorithmic. They are and *must be* fundamentally human endeavors.
Both science and medicine are supposed to have data as their cornerstone, yet we seem all too eager to waive that fundamental requirement when discussing and deploying AI-enabled tools.
Knowing with Our Hearts and Guts and Bones
The indispensable element of humanity for both science and medicine is something I’ve felt in my heart and guts and bones, but it became unmissably real for me in a way that hit like an emotional and spiritual thunderclap during the plenary talk from my treasured friend Dr. Rana Awdish at the ATS conference. Rana’s second book, “After Shock: Learning to Reinhabit My Body After Illness” is a beautiful, thoughtful, and deeply personal treatment of what we talk about (and, as importantly, what we too often *don’t* talk about) when we talk about healing. Like all powerful and important stories that reach us in our most human places, this one has been in the works for a while, and it made its debut in physical book form at the ATS conference. She literally first held a copy of the book that will be widely available in a couple weeks (order here!) at the conference, and she generously agreed to speak on some of the key themes in the book in her plenary talk.
I was in the audience to support my friend, because I love her and that’s what we do for people we love. It can be very daunting to stand on a big stage and say very vulnerable and true things. Rana has helped me be brave many times, and I wanted to help her too if she needed it. But, I really didn’t expect to hear anything new or all that surprising. I’ve had the incredible privilege of watching this book take shape. I had a hint as to how she was going to structure the talk. Like Fred Rogers could do, I know Rana has a way of telling the whole truth in clear and simple words that manage to bring tears. But I don’t cry easily. I know the author and speaker, I know the book, I know the ground we’re probably going to cover. So I didn’t even have tissues with me.
I cried at a place in the talk that I suspect very few other people did. She was talking about her first book, the story of her own journey surviving life-threatening illness and the beginnings of recovery, told from the perspectives of both physician and patient. “In Shock: My Journey from Death to Recovery and the Redemptive Power of Hope” has appropriately made all of the lists of the most important books in medicine, and it was recognized very quickly after its publication as being profoundly important. In her plenary talk, Rana was showing pictures of herself from that time, pictures from promotional events and recognition received for In Shock. In that moment, I really saw that earlier version of my friend. A survivor, but not yet healed. My heart swelled and broke simultaneously. I wanted so badly to invent a time machine and travel back in time to give my friend a huge hug, but the best I could do was to tell her about it later and cry again.
Rana’s talk started 20 minutes late because that’s how things go sometimes at conferences. The people ahead of you talk too long, or the planned transitions are too short, and that’s just the way of things. I was worried that they would cut her short, and I knew that was not okay (I had actually started drafting an angry email before she even started her talk). When I cried for that past version of my friend, the intensity and importance of that shared humanity had the effect of bolting me in place. I had built in exactly 20 minutes after the end of the plenary talk before I myself was to be giving a talk elsewhere at the conference, about artificial intelligence and the opportunities that exist for AI uses in pulmonary, critical care, and sleep medicine. To be on time, I would have to leave the plenary talk before it was over. If I stayed to the end, I would be late.
Of course, I was *supposed* to be late. I could do nothing other than be late, because really I was choosing to be present for learning something so very human from my friend. I was right where I needed to be, and really I wasn’t late at all but was right on time. This was a moment that carried the weight of thousands of moments yet to come. Choose humanity and the kinds of knowing that come from our shared human experience, or choose technology and the appearance of something truthful that it offers without shame or awareness. It was so easy. When I started my talk, I said, “I’m not sorry I’m late. I was listening to the plenary talk about healing, and about different ways of knowing, and about inhabiting our fully human, often imperfect, sometimes broken bodies. If we ever prioritize AI over our humanity, we are all truly lost.”
Apparently even the Pope thinks that’s true.
No algorithm could make a choice like that. Algorithms know nothing of intuition, or the true weight of the spirit, or what it means to feel emotions that don’t even have names and to be moved by them. Algorithms can’t experience the thrill of potential that the scientist can see in a table of numbers or squiggly lines on a piece of paper, can’t get goosebumps in that moment and decide in that moment that food and sleep and all the other things can wait because nature is whispering something really important and you MUST KNOW. Algorithms can’t know that the factually right answer might be the exact wrong answer for the person in front of you. Algorithms can’t sense when a patient needs to say more than they’re saying, and the thing they need to say, if said, will change their whole course of care. Algorithms can’t know with their hearts and guts and bones the way doctors and nurses and therapists can. And if you take these away, you’re not doing science or medicine anymore.
In God We Trust, All Others Bring Data
I had a chance to think and talk about AI a lot at the ATS conference. I gave two talks about AI, one on the regulatory landscape, the other on potentially exciting opportunities for AI to positively impact pulmonary, critical care, and sleep medicine and research. I also listened to a lot of talks and discussions about AI and how people are thinking about it and using it. In most of those discussions, there was something that gnawed at me… or more correctly, something that *wasn’t* present that gnawed at me.
If you make a claim about a diagnostic or a drug or a device, you’re legally required to have provided high quality data that has been evaluated by an independent regulator (usually the FDA in the United States) and found to support your claims. You need to do that BEFORE you make the claims. Indeed, you can get in big trouble if you don’t do that, and it’s a key raison d’etre for the FDA. “Do your own research” is tantamount to caveat emptor when we’re talking about medicines and medical devices, and we know from bitter historical experience that this approach hurts people.
In light of the combined weight of established best scientific and medical practices, regulation, law, history, and the principles of stewardship, it is shocking to me how uncritically AI-enabled tools and workflows are being deployed across the biomedical research and clinical practice enterprise. In exactly zero of the presentations or conversations I witnessed did anyone say, “Our widget does XYZ and makes PQR improvements, and here’s the clinical study that demonstrates it.” And apparently these questions aren’t being asked very often in ways that inform purchasing decisions.
Imagine if a company came to your health system and said, “This bedside diagnostic test will let you diagnose impending Holy Crap Syndrome faster than current standard of care approaches, resulting in lives saved and costs reduced year over year.” And then imagine that they showed you no data to support any of those claims. Would you recommend that your purchasing department buy hundreds of kits or set up a service contract with the company?
Making claims like these may not be illegal, but it doesn’t exactly feel ethical either. Friends and colleagues like Tamiko Eto have written far more eloquently than I can regarding how current law, regulation, and policy may be behind the times in terms of addressing the gray areas where some biomedical AI tech lands. But beyond law, regulation, and policy is a more fundamental question for me: When and why did we collectively decide that the principles of good science and good clinical care could be bypassed where AI is concerned? We’re well aware of some of the risks, and we’re learning about others. We’re hopeful about the benefits, though in most cases those benefits are not at all certain. And the thing is, we *already have* a way to assess potential benefits against risks. THAT’S WHAT CLINICAL TRIALS WERE LITERALLY INVENTED TO DO!
Thankfully, it appears that at least some folks are coming out of their AI fever. The editorial board at Nature Medicine recently published an editorial unambiguously titled “Show us the evidence for the value of medical AI” and I cheered aloud. Actual clinical trials are starting to happen, and they are informative indeed. The LungIMPACT trial was a “prospective, multicentre, randomized controlled trial [that] tested whether immediate AI prioritization of primary care-requested chest X-rays (CXR) influenced time to computed tomography (CT) and lung cancer diagnosis….” Surprisingly, AI in this context showed no benefit for the primary outcomes and had significant independent associated costs. A recent preprint reported a series of randomized trials assessing the effects of AI assistance on arithmetic reasoning and reading comprehension. The findings are quite intriguing. Using AI assistance does increase short-term performance. However, after just 10 or so minutes of AI assistance use followed by removal, both performance and persistence decline below those from participants who never had access to the AI assistance. Though not medically or scientifically specific outcomes, I’d argue that arithmetic reasoning and reading comprehension are fundamental to both science and medicine, and that decreased accuracy and persistence are potential risks relevant to both science and medicine.
What Can We Do?
I’m finding that asking questions is really helpful for shaping discussions and for shaping my own thinking and feeling about AI and my relationships to it.
“What’s the hurry?” – It has seemed to me that a LOT of AI-related discussions are propelled by FOMO (Fear Of Missing Out). I can easily see how this helps folks peddling AI services and solutions, but I don’t always see how this helps me or those whom I have promised to serve. If a compelling case cannot be made for hurrying things, I get suspicious.
“Who does this really help? Show me how.” – In biomedical research and clinical care, the immediate answer is always going to be, “This helps patients, of course!” It’s the equivalent of the R, S, T, L, N, and E in the bonus round in Wheel of Fortune. It’s a given. So, show me exactly how it helps patients! Not theoretical patients who, for the purposes of discussion, happen to have interests and goals that exactly align with the hospital CEO. Real people, with real lives constrained by real pressures, who need real healing.
“Very cool! What data do you have to support that claim?” – I’ve already addressed this one. Talk is cheap. Data will speak louder than any unsubstantiated claim, or at least it should.
“Tell me about the non-technical aspects.” – This sounds like a weird one, but it can be very revealing. Many AI vendors in the science and medicine spaces skimp on the scientific or clinical expertise in their leadership structure – e.g., they’ve got the business part (an MBA or someone like that) and the AI part (math and computer folks), but they don’t have enough actual scientist or clinician muscle to know the fields they’re trying to engage with. If they can only talk about the technical parts and clearly don’t understand the human aspects of doing science or medicine, it’ll show. More importantly, if they haven’t even thought about the human aspects, the ethical aspects, the ways in which their AI widget interacts with and impacts real people who know things in their hearts and guts and bones, they may not be ready.
For me, this isn’t about being a Luddite or being scared of AI. I’ve been talking about and working to establish the potential benefits of AI in medicine and science for years. But I am no more willing to sideline my scientific rigor, my medical responsibility, or my humanity for artificial intelligence than I have been for anything else in my life (including the best job I’ve had so far in my life). Science and medicine are done by humans and for humans. Neither endeavor is merely technical. Ethics matter. Human values and virtues matter. Authentic healing, spirituality, inspiration, passion, celebration, all of the colors and textures of human experience matter. To quote Westley from The Princess Bride, “Anyone who says differently is selling something.”
Infographic from the frickin Vatican, of all places! Used without permission, may God forgive me. I wonder if it’s AI-generated in any aspect….