Affiliations, Impact Factors, and Mental Laziness

Don't skip brain day at the mental gym

I’m going to get around to talking about why I think using affiliations as thinking shortcuts is a problem in the present moment. I’ll probably have a fair bit of negative stuff to say about how we use various flavors of affiliations – who you work for, what journal published your paper, where you went to school, that kind of thing. But I want to say two things up front.

First, I don’t think paying attention to affiliations is always a bad thing or that cognitive shortcuts are always a bad thing. For instance, if someone is strongly affiliated with a hate group, I’m going to pay attention to that. Affiliations can tell you something about someone’s behaviors and likely behaviors, and that’s important information. Is it certain that someone in a hate group has acted or will act in ways that are motivated by hate and intended to be harmful to the hated? No, it’s not certain, but it’s fair to say it’s much more likely than for someone not affiliated with that hate group.

Second, I think many of the problems with how we handle affiliations are closely linked to our tendency to equate affiliation with identity. Your professional affiliation is XYZ, therefore you *are* this identity that I associate with XYZ. Most of us know this isn’t true, and we would get the question right on the quiz. Does affiliation equal identity? No, of course not. But our behaviors often don’t match what we know to be true, and what I hope to do here is help pull apart affiliation from identity and help think about ways to be more intentional about doing so in our day to day.

Boxed In, Boxed Out

I have a new professional affiliation, and very soon I’ll be able to announce what it is. But I’ve been thinking a lot about professional affiliation over the last 10 months. When I left my position at NIH, it was vitally important to me to show up for the scientific community, to be present and tell the truth and stay involved. I looked for opportunities to go to conferences, to meetings, to workshops, to give talks, to be wherever scientists were going to gather. It was especially important to me since my colleagues still at NIH were not allowed to do these things.

I would be filling out the registration information for these gatherings, and there it would be – the “Affiliation” box on the registration form.

Sometimes it would be “Institutional Affiliation” or just “Institution” or sometimes “Employer.” Some version of “Who do you work for?” It’s essentially always a required field, so you can’t leave it blank. I struggled with what to write every time. None or N/A? Independent something-or-other, or just independent? Former NIH? Over the last 10 months, I’ve tried all of these either alone or in combinations, and no answer has felt exactly right. One dear friend suggested “Itinerant Scientist” and I confess that I haven’t yet tried that one, so it’s possible the answer has been in front of me all along.

It took me a while to realize what bothered me. By making “Affiliation” a required field – meaning you can’t leave it blank if you want to participate – it sends a message that if you don’t have an affiliation, you don’t count. You’re out. Doesn’t matter who you are, what you know, what you’ve done or what valuable ideas and energy you might bring to the event. It felt like the message was that if you’re unaffiliated, you’re unwelcome.

Recently, I learned that Google Scholar takes it even further. I went to update my Google Scholar profile, and I learned that a Gmail address isn’t accepted for verification of the account. So, until I have an email address through an employer or something like that, I am stuck with an unverified Google Scholar profile. I guess I’m not really a scholar until I have a job?

Unverified profiles apparently don’t show up in search results, which is kind of a bummer when one is searching for jobs where recruiters and potential employers might be interested in seeing my profile. Click “Add” and you get:

And then…

If this were the only example of how we reduce professional identity to a single feature, you wouldn’t be reading this. I’d complain over happy hour appetizers, but that would be it. We’re here, though, sooooo….

Impact Factor

You must’ve known we would have to talk about this at least a little bit. Journal impact factor is a shining example of affiliation trying to stand in for way more than it possibly can.

By way of quick orientation, journal impact factor is a metric calculated for any given scientific journal that takes into account the number of articles published by that journal and the number of times on average that an article published in that journal will be cited by other articles published anywhere. Bigger impact factor is supposed to be better. So, if a journal publishes very few articles and they get cited a lot, that journal is thought to have a big impact on the field, and thus a big impact factor.

Never mind the obvious ways that journals might game this system and the fairly apparent motivations for doing so. Let’s take a step back from that and look at what actually happens. Dr. Genius publishes a paper in the Journal of Awesome Science, a journal with a big journal impact factor. Someone almost inevitably says, “Oh! That’s a really high impact journal!” or “Wow, high impact paper!” or even says outright “Cool! Journal of Awesome Science has a really high impact factor!”

Did you catch the sleight of mind? Judgements have been made about Dr. Genius’s manuscript even though we haven’t actually read the manuscript! We have no idea what’s in it. We’ve judged the article, and judged Dr. Genius, based solely on an affiliation. You may be saying, “Hold on, don’t loop me into this. I didn’t make any judgements about Dr. Genius or their article precisely because I haven’t read it.” Are you sure? And even if you didn’t judge Dr. Genius and their article based on journal impact factor, there’s a decent chance that your university and/or Dr. Genius’s university did, for things as consequential as promotion and tenure. There’s research on this - https://elifesciences.org/articles/47338

Stated differently, my problem with journal impact factor is that it is used to try to define and to quantify a concept – impact – based on features that have little to nothing to do with the thing it’s trying to address. Indeed, journal impact factor wasn’t ever intended to be used this way. It was a metric that was supposed to help librarians working in the era of print-only journals decide which ones they wanted to allocate scarce subscription fees and scarce shelf space. The academic/scientific community co-opted the metric. The problem is that many folks in the scientific community pay close attention to journal impact factor despite knowing that it’s being misused and despite knowing that it doesn’t actually tell you anything about impact! Impact of any scientific finding can only be assessed over time by taking into account the actual things that change in a way that’s at least a little bit connected to the finding.

Do you even mentally lift, bro?

Oh yeah, I also think h-index and i10-index and all of those metrics are pretty much hot garbage too, for many of the same reasons. All too often these metrics are used as a stand-in for an individual scientist’s overall contributions and impact on the field. To show you just how stupid that is: Fred Sanger OM CH CBE FRS FAA, the two-time Nobel Laureate in Chemistry (one of only 5 two-timers in the history of the Nobels and one of only 3 two-timers in the same category) who first sequenced insulin and a bunch of other proteins and who then invented the method of DNA sequencing that still bears his name and that was foundational to the creation of molecular biology as a field, had an h-index of 43. According to my unverified/unverifiable Google Scholar account, my h-index is 42. So much for h-index. Judge Fred Sanger’s contributions or my contributions or anyone’s contributions based on a more holistic view, please.

It’s this feature – our apparent willingness to sacrifice truth and completeness for convenience and the feeling of certainty, even when we know damn well that we’re doing it inappropriately – that I think makes our relationship with affiliations potentially really toxic. This is at the heart of my beef with affiliation and journal impact factor and h-index and so many other strategies to try to reduce cognitive load. That relief that comes with the illusion of certainty and the convenience of metrics that are known to be deeply flawed can be enough to short-circuit and circumvent careful deliberation, the actual doing of the mental work.

One of the things that worries me the most about the current state of the art with generative artificial intelligence is its potential to plug directly into our natural tendencies to seek certainty and to try to lower our cognitive load, even at the expense of truth and completeness. Not only the potential to plug into that tendency but also to greatly accelerate and extend its effects. When discussing the use of AI for scientific or legal writing or analyses of scientific or legal documents, I’ve asked colleagues, “If you had a colleague or collaborator who was very confidently wrong sometimes or who just made stuff up occasionally, would you continue to work with them?” The answer appears to be yes more often than I would’ve guessed.

We simply must get better at recognizing when we cannot or should not take shortcuts, and then we must get better at doing the work. Beyond the generative AI problem, I see modern hiring processes as being a ready example of the harms of cognitive shortcuts and overly energetic reduction of complexity to a single variable. There are more highly trained, highly experienced superstars on the biomedical and scientific job market right now than ever before in my career. It’s a situation that hopefully will never happen again, but the current moment represents massive opportunity for potential employers. And yet, I see so many organizations completely squandering their chance to bring on global thought leaders, often because they aren’t expending the effort to really understand the opportunity in front of them. I had a conversation recently with someone whose work has had demonstrable global health impact. This person was looking for their next professional opportunity, and many organizations could not think of anything more impactful and creative to do with this person than to put them in a high volume patient care role. Though this person could do that, so could someone who has just completed their clinical training. There was no willingness or effort expended to keep the aperture of possibility open any wider than the typical, and every one of those organizations missed a one-time opportunity. This kind of cognitive laziness is a driver of failures of imagination (which I wrote about previously). Imagination takes work.

So what do we do?

How do we help ourselves avoid falling into the trap of how we naturally tend to deal with affiliations and other similar cognitive shortcuts?

We each start with ourselves. Develop the habit of pausing, however briefly, to ask if the conclusions you’re drawing are in fact supported by the data you have. What do you actually know about a candidate for the job you’re hiring for based on where they went to school? Do you have reason to believe that a scientific article is really good based on where it’s published? Does your reasoning stand up to external scrutiny (note that this is precisely why peer review exists – to see if our reasoning stands up to external scrutiny)? If not, and you’re a person who prioritizes good data and solid evidence, you know what you have to do. You have to look deeper.

We can help our colleagues. Again, this is what peer review is supposed to be. We’re supposed to help one another reach a sufficiently high standard of evidence for the purpose at hand. It’s not about reflexively demanding more, or about shooting down anything that you evaluate no matter what, or about pushing others to do the study you would’ve done or you wish they had done. It’s about conclusions supported by what we know at the time. If you’re positioned to help a colleague recognize that they might be taking cognitive shortcuts that could lead them astray, practice the art of gentle critique through inquiry. Very few of us like unsolicited advice, especially if that unsolicited advice is, “No, you’re wrong, here’s what you should do.” So this may indeed take practice and finesse. It’s important enough to get right, in my opinion. If you know someone who is especially good at this kind of gentle critique, seek out their mentorship.

Be intentional about focusing on what actually matters. This is easier said than done sometimes. It’s not always clear what actually matters, and even if there’s clarity, there’s not always agreement on what actually matters. But to return to the issue of institutional affiliation, consider this question: Does requiring someone to list an institutional affiliation contribute directly to what actually matters? If so, keep it. Maybe you really need to know how many people at your event are from Awesome University or Amazing Company, Inc or wherever. But also consider what cognitive shortcuts you might be taking, what assumptions you might be making based on data that aren’t really telling you what you think they’re telling you.

And consider that you communicate information about you and your organization and your thought processes when you use cognitive shortcuts – affiliation, impact factor, whatever – as a substitute for careful deliberative processes.

Prompt: “a cartoon of a human brain with muscular arms lifting dumbbells”. I’m taking a shortcut by not trying to fix those super weird hands and dumbbells.