IBM charts a new research path with MIT

In a few short years, generative AI has gone from being a novelty to a technology used by millions of people worldwide to boost productivity, creativity, or offload repetitive cognitive work.

David Cox watched this historic shift as IBM’s inaugural director of the MIT-IBM Watson AI Lab. When IBM committed funds in 2017 to launch the lab, AI ran mostly in the background of people’s lives, forecasting sales, detecting suspicious credit card transactions, and filtering spam.

Today, it dominates the conversation — and can even generate a conversation of your choosing, when prompted, in the form of a podcast or any other format. Whether generative AI turns out to be as transformative as the internet, it has already fundamentally changed everything from content creation to software development to scientific discovery.

Meanwhile, another revolution is brewing, and IBM and MIT are pivoting to embrace it. Under a new agreement signed last week, the lab will expand its scope to include quantum computing in addition to AI. The lab will be renamed the MIT-IBM Computing Research Lab to reflect its wider research agenda.

A decade ago, IBM put the first quantum computer in the cloud and has ticked off milestone after milestone since then. New advances in quantum-centric supercomputing were highlighted this week at Think, including protein simulations involving more than 12,000 atoms and research tied to drug discovery and fusion energy.

With quantum quickly maturing into a useful tool for scientific research, this moment feels eerily reminiscent of when IBM founded the lab with MIT, pre-ChatGPT. AI’s potential then could be sensed if not plainly seen, but the labor involved in data curation and training single-purpose models at the time limited its wider adoption.

The picture is entirely different today, of course, with AI embedded in virtually every aspect of daily life and a new paradigm shift in computing on the horizon. Cox, who now heads IBM’s language model research, will continue to lead the lab through this next iteration with his counterpart at MIT, Aude Oliva.

We sat down with him to talk about the lab’s next chapter, what a strong industry-academia collaboration looks like, and how for him, jazz piano is a creative outlet and a form of catharsis.

The lab is expanding into quantum computing. Why now?

The aperture is much wider. Quantum is on an arc to become very powerful in the next few years and I’m optimistic about the potential crossovers with AI. There is a common misconception that quantum computers are uniformly faster than regular classical computers, for every kind of computation. In reality, for most of the things we use computers for today, it's hard to beat a modern CPU or GPU. But for certain important classes of problems, quantum computers can provide solutions that would take an impossibly long time to compute with a classical computer. That means the payoff isn't just that quantum computers are 'faster,' it's that they open new realms of possibility. Solutions that would take lifetimes to compute classically are within reach.

When you started the lab, did you imagine AI and quantum would be on converging paths?

We knew there would be overlap and interaction, but the speed with which these technologies are each developing has exceeded our most optimistic predictions. When we started the lab, we had a few quantum projects that were mostly theoretical but now everything has become tangible. Quantum is moving into that ‘new’ territory, and its intersection with AI is exciting.

How has the culture of the lab changed with the rise of LLMs?

In the lab’s early days, more of the work we did felt speculative, like we were searching across a range of technologies—including generative AI—planting seeds that might pay off later. Now it’s clear which bets paid off. We still make exploratory and contrarian bets, but gen AI has changed the landscape and the research we do reflects that. Our ability to move from the lab to production has also radically changed. You can draw a straight line between our research and the technology that people and customers are using. It brings into focus the value of what we’re doing.

Not all collaborations with academia succeed to this extent. What’s the secret?

It's a few things. The most important one is that IBM grew a research presence here, on campus. IBM also brought in people who know their stuff. If you just throw money over the wall, it's not going to work — you have to have people on the ground, working shoulder to shoulder, and students coming over and hanging out. The 10-year commitment makes a big difference because we can develop relationships. One MIT faculty member said he likes working with us because we drive him toward interesting problems. Often, new ideas come from trying to solve problems that others haven’t tried to solve before. We can sometimes help our MIT colleagues discover new problems they simply hadn’t considered before, because they have a different frame of reference. The understanding that comes from solving those problems needn’t be overly application-specific; more often than not the insights that are gained are quite general, and intrinsically, scientifically interesting.

What made you leave your teaching and research at Harvard to come to IBM?

Even when I was professor, AI was becoming increasingly industry-dominated because of the resources required to do the research. It’s gotten a million times worse since then. Rather than leaving academia altogether, it was appealing to have a bridge. I was also drawn to the zero-to-one quality of the opportunity. I have co-founded a few companies over the years, and the blank canvas aspect of building something new is extremely gratifying. At the same time, while the Lab was new, much was familiar. I did my PhD at MIT, so it was also a bit like coming home.

Would you encourage students today to choose industry over academia?

I don’t think it's either-or. Some industry jobs are a one-way trip, but here at least we’re focused on advancing the state-of-the-art. People come out of here with an attractive portfolio and CV for getting a faculty job. It can be quite fluid. Even full-time professors will take sabbaticals to work in industry or engage in other ways.

You apparently have been coding since you learned to read. How did that happen?

My dad worked for Digital Equipment Corporation and brought home a computer back when that was not a normal thing. It was a DECmate and it had this glowing phosphorous screen which called to me. The programs you could write were very limited — it was all text-based, but the constraints made it fun because you had to figure out clever solutions to get it to do interesting things.

You’ve talked about an emerging new law for LLMs, akin to Moore's Law for computer chips. What does that mean?

There is a remarkably consistent trend that we and others have been noticing about language model size and capability. Basically, language models become more “capability dense” over time, at a predictable pace. So, yesterday's 70-billion-parameter model is today's 7-billion parameter model. You can build a 10 times smaller model a year later that’s as capable as the big model. At some point, it becomes foolish to play in the upper end of the scale game. It's expensive to win, and your lead is ephemeral; there will always be a pack of smaller models just behind you, doing the same thing for a radically lower price, after only a short delay.

Will AI eventually behave more like software?

AI models are amazing, but they can be frustratingly unpredictable in a way that regular software is not. We’d like to bring back some of what we gave up when we moved to LLMs in terms of computer science discipline. Software has principles — consistency, encapsulation, abstraction, and formal guarantees. When your ‘program’ is a natural language prompt you lose that. You can't even port between two different LLMs because no one can specify what differentiates them. We can write in natural language, which we couldn’t do before, but we think we can combine the reliability and abstraction we had with software while maintaining the magic that LLMs bring.

The best of both worlds — software’s reliability and AI’s improvisational ‘magic.’ Is that the idea behind IBM’s Mellea project?

Mellea is trying to be a systems programming-like layer for people to contribute ideas. You write code that calls a function, which could prompt the model to do something, or activate an adapter, and automatically parse the results. That’s what Granite Libraries are, basically. They're functions that you can call to ask if that answer that I'm about to send to my user is a hallucination. The function could call an LLM to judge whether it was wrong or activate an adapter that has the model itself decide. The user doesn't need to know how that function worked. They just want the right answer at the least cost.

You’re apparently a serious jazz pianist. What does your creative process look like?

If I'm not creating something that hasn't been done before, I’m not interested. I started playing piano in fourth grade and probably would have dropped it if it were just a matter of reading what’s on the page and playing it faithfully. With jazz, the sheet music only tells you the chord changes and melody — almost everything else is made up as you go. You usually play the melody — the head, they call it — once and then improvise the rest. Everyone trades off, and then you play the head again at the end and you're done.

A bit like an LLM, responding to an ambiguous prompt! Have you played professionally?

I was in a quartet through high school and two of our members became professional musicians. I've never auditioned for the New England Conservatory, but I accompanied our bass player during his audition. The other two members became scientists. I remember my parents asking, are you sure you want to go into science? Don’t you want to pursue your passion? And I said, oh, God, no! It’s a hard life and I had plenty of other interests. But it was fun and just recently I started playing a lot more.

To relax?

Honestly, some days it feels like an exorcism. It's something I have to do, and I don't know why — not 100%. It's more like a traumatic catharsis. But yeah, I do feel better afterwards. I can’t quite articulate why, but I do feel like my sudden return to playing music, and improvising, is somehow connected to the current moment in technology. There is something in the air, and it has to do with creation. I don’t know where any of this leads, but I’m enjoying the ride.