As spending on genAI continues to grow, companies are eager to find a return on the investments they’ve made. There are signs that few have yet pulled that off. US tech stocks dipped in August when MIT Media Lab’s Project NANDA released a report stating that “the vast majority [of AI pilots] remain stuck with no measurable P&L impact.”
Meanwhile, some executives we’ve spoken with worry that the pressure to quickly prove return on investment (ROI) could prematurely stop valuable projects or force leaders to think too small.
“Imagine a 19th-century business leader debating the ROI of electricity,” Liran Belenzon, co-founder and CEO of BenchSci, which makes AI applications for pre-clinical research and drug discovery, recently wrote on LinkedIn. “‘Should we switch from gas lamps? What’s the ROI?’ The question seems absurd today because electricity was a transformative technology, not just a tool for marginal gain. AI is the same!”
We spoke with Belenzon about why measuring the ROI of AI initiatives can be so difficult and why focusing on it could cause companies to miss valuable opportunities. Here are highlights from that conversation, edited for length and clarity:
You recently wrote on LinkedIn that “focusing solely on quantifiable ROI can be a trap.” Why do you see it as a trap?
Because it’s very hard to quantify the impact of AI on the workforce. And because it’s hard, you might not be able to or you’re going to get it wrong—and that mistake or inability to do so might lead you to not invest in it. Because the impact can be so big, it makes it hard to quantify in a way that is attributable. This electricity quote is not mine—I heard it somewhere in a book—but let’s say [we] could compare AI to electricity. What’s the ROI of electricity? You can’t really quantify it or even imagine what it enables you to do because it enables you also to architect everything so differently.
There are different aspects of AI. There’s the generative [aspect], there’s the automation one, but also in the novelty part of it. How do you quantify these things? This comes from two worlds. One of working with our customers, where we deploy an AI copilot for the smartest scientists in the world and big pharma, where our value prop is better, faster, cheaper. You can quantify faster and cheaper more easily than the better. Did we discover something new? And what does that translate into? Is it a billion-dollar opportunity?
[Second], internally for us, we made a decision we’re going to give everybody who works here access to AI tooling. And we’ve had a conversation about our engineering team, which is roughly a hundred folks: Should we open these models in Cursor or those models in Cursor? What’s going to be the ROI? How are we going to measure it? How are we going to attribute [it]?
Our best assumption is that if they have access to these tools, they’re going to do a much, much better job. If that assumption holds, let’s take the simplest thing that we can measure—let’s say usage, even though it’s not going to be perfect and maybe it’s not even something we want to optimize for—but just as something that’s close enough to value attribution to tell us how we’re doing and let’s just go with it. It’s fine to make that assumption. So that’s where [that LinkedIn post] came from and why I decided to write that.
What is it about AI, specifically, that makes it so hard to measure ROI?
It depends on which applications of AI. Because the ones that automate a workflow, we can say, ‘Well, this took me 10 hours, now it takes me an hour.’ They can be] quantif[ied]. But everything that’s around AI and decision making, I would argue that it’s extremely hard to quantify because you’re trying to measure the impact of a better decision.
Then there’s two questions. You might have said, ‘Hey, would you have cracked this code or figured out how to do this without [the AI tool]?’ Which engineer wouldn’t want to say yes? That’s one. If you do say yes, would you have gotten there in six months? Would you have never gotten there?
For us, it’s very relevant because what we do is really act as a copilot for scientists, and that’s not around workflow automation. That’s mostly about getting them information and answers to questions that many of them just couldn’t have answered themselves because we were connecting millions of different data points together that are just impossible for the human mind to do. Did that lead to an acceleration of a program by five months? Did that lead to a discovery you would never have gotten to yourself? What’s the ROI on those things? Because something is so hard to measure, sometimes people say, ‘I’m not going to do it’ or ‘I’m not going to pay X amount’ because I can’t directly attribute it, which actually hinders innovation.
An executive reading this would likely respond that they need some way to know whether they’re getting anything from this technology and if it’s worth the investment. What advice would you give them?
I would say use common sense. Because the complexity of ROI models are endless. We can ask our engineering team, ‘Do you feel like this is making [you] more productive? Yes or no? If so, how?’ Take that information and make a decision.
In a blog post earlier this year about BenchSci becoming an AI-first company, you wrote, “Before adding new people or processes, we ask: ‘Could AI do this?’” Does that factor into decisions about whether or not to hire people at BenchSci?
Yeah, 100%. It’s like, ‘Why can’t AI do this?’ I’m not saying AI can 100% do something someone does. But maybe [it] can do 20%. But 20% more across five people is a person.
Another thing I’ve learned is that without constraints, there’s no creativity and innovation. There’s no forcing function to do that. Constraints are just so important for creativity and innovation.