CATHY O'NEIL: I figured that people in finance really were experts, knew what they were doing. Once I was working inside the financial system when it was cratering, I was like, wow, I'm not so sure that the people that were kind of running the Fed, running the economy at the time, had a very simplistic and incorrect model. Predictive algorithms are optimized to the success defined by their owners. And it's all about the power dynamic.

DEE SMITH: Hello, I'm Dee Smith. Today I'm going to be speaking with Cathy O'Neil. Cathy has a PhD in mathematics from Harvard, worked as a quant in the hedge fund world, and wrote a book with the memorable title Weapons of Math Destruction. We're going to be talking about the algorithms that we have come to depend on and how they may not be as dependable as we think.

Hello, Cathy. Good to see you.

CATHY O'NEIL: Thanks for having me.

DEE SMITH: Thank you for coming to visit today. I'd like to talk a little bit first about your interesting background. Because you've got a very interesting pathway that led you to where you are today with some very interesting detours and in ways, and out ways, and byways. So tell me how you got interested in mathematics and what led you to where you are today.

CATHY O'NEIL: Wow. I think I started becoming interested in mathematics when I was five. My mom gave me a spirograph set. You know what those are? Like, you draw those little circles inside circles.

And I just remember seeing periods-- I'd, like, there was something that had a pattern of six and then something else that had a pattern of three. And so that's where I sort of figured out prime divisors. I figured out that there's a three and six, and there's also a two and six.

And then I started thinking about other numbers and how they had different numbers hiding inside them. I didn't know that was math exactly. But I was very fascinated by this idea of primes.

My mother is a mathematician, which is probably the biggest influence on me. She got her PhD in the '60s from Harvard in Applied Mathematics. And my father, as well, is a mathematician. So it was kind of the family business, to be honest. It wasn't that much of a stretch.

But I wasn't really, really into math because I was really into music, and I wanted to be a musician until the summer I turned 15. And then I had a choice to make between going to a music camp in Switzerland to practice piano five hours a day or to a math camp in Western Massachusetts.

And I went to the math camp. And that was kind of the last moment I had anything else in mind until I turned 35. Basically, that summer when I was 15, I decided to become a math professor.

DEE SMITH: So you have what I would call a very deep background in mathematics. It doesn't get much earlier than how early you started. And that led you on an interesting journey, and you ended up at the hedge fund DE Shaw right at the time that the financial crisis hit.

CATHY O'NEIL: Yeah, that's right.

DEE SMITH: As a quant.

CATHY O'NEIL: Yeah. I mean, I got to New York. I was an assistant professor at Barnard College. I had done all the things-- get my PhD, become a postdoc, et cetera. And once I became a professor, I was like, actually, you know, I'm not really sure this is suited to my personality.

I like people. I like feedback. And being a professor in math, in number theory-- slow field. You know, it wasn't exactly the most stimulating, exciting environment. So I wanted to-- but I loved New York.

So I got a job in 2006, early 2007, to work in the hedge fund DE Shaw. And I left academia and went there in June of that year. And almost immediately the financial crisis hit.

DEE SMITH: So what was it like to be there? That was an important time. What was it like to be there with the financial--

CATHY O'NEIL: Right. I mean, I was working with Larry Summers on his projects. So it was very central to the financial system with these fancy people. It was weird. It was really weird. I mean, it was disillusioning, I think is the right way of thinking about it.

I had not known much. I was very naive. Politically, I was sort of apolitical. I figured that people in finance really were experts, knew what they were doing. I was one of those people that was like Alan Greenspan is so cryptic that he must be a brilliant man, you know?

Once I was working inside the financial system when it was cratering, I was like, wow, I'm not so sure that they know what they're doing. I'm pretty sure that kind of nobody knows what anybody's doing. Or they know maybe locally what they're doing. But most people don't have a real sense of it and in the scale of it, especially.

DEE SMITH: What do you ascribe that to? A, the belief that they do believe they know what they're doing, but the fact that they really don't.

CATHY O'NEIL: It's a good question. I think it's telescoping. Its modeling and telescoping. OK, so let me unpack that a little bit.

Basically, the people that we thought knew what they were doing were all economists. And economists-- what they do is they simplify the world into toy universes, these models that they have, for the economy, or whatever people-- people themselves-- like, people are rational agents that know how to get what they want. They have utility functions, blah, blah, blah.

Everything is simplified. Everything is actually more complicated. And at some level, I think economists know that. But at a high level, they really do think in terms of models.

And I think the people that were kind of running the Fed, running the economy, at the time had a very simplistic and incorrect model of derivatives in particular, like what derivatives were doing. They just really modeled them as, like, spreading risk. And they thought of it as a diversification, which is a good thing. If we all have a little bit of risk, it's better than having risk concentrated in one place. That was their model.

What, in fact, was happening was that risk was proliferating because of the opacity of the instruments that were being traded so much, like the mortgage-backed securities with the credit default swaps. That stuff wasn't simply spreading risk. It was creating risk and spreading risk.

And yes, we did spread risk. That's one of the things that happened. And that's why when the crisis happened, it hit the world, rather than just a few banks, right? So yes, there was spreading. But there was so much proliferation.

There was so much of the expansion of the housing bubble itself because of the derivatives, because of the markets. It was like everything was greasing every other wheel at the same time, and nobody was really keeping track.

DEE SMITH: So spreading risk actually made risk worse.

CATHY O'NEIL: In this instance, absolutely. I mean, if you had a certain amount of risk and you just spread it thinner, and thinner, and thinner, that might be a good thing. But if you have a machine that builds risk and spreads it, then it's hard to keep track. And that's a closer analogy to what was going on.

DEE SMITH: So you know, the question of the models is a very interesting question. Because people make these models, and then they become convinced, for whatever reasons of human nature needing to feel like we are in control of things or we actually understand what's going on when we don't. We convince ourselves that the models do represent reality.

I mean, this is true, and not just in financial systems, but in companies, you know, where people make these complex spreadsheet models, and they think they reflect reality. And of course, they don't. And you know, that's another related kind of problem. How do you see the reality of modeling having evolved, if at all, since the financial crisis?

CATHY O'NEIL: I mean, certainly, models are used more if you include in that not just economic models, like we're discussing, but things like algorithms. And think about all the VC-funded big data, AI companies nowadays. They are all based on the assumption that predictive analytics, AI, or whatever you want to call it will be able to solve problems. So they're sort of inherently assuming that modeling works or works well enough for this company. Uber, all those companies that are sort of using the algorithm as a basic business model.

DEE SMITH: And what do you think? Do you think that-- I mean, give me your opinion of that.

CATHY O'NEIL: I mean, you know, full disclosure, I have an algorithmic auditing company. The point of my company is to poke at that assumption. And the very short answer is it's a very narrow view.

Most algorithms work narrowly in the way that the company that built them and is deploying them wants them to work, but probably fails in a lot of other ways. And they might be unimportant to that company. But it might be important to the targets of the algorithm or some other side effect of the algorithm.

So I mean, just going back to the way that economists thought about derivatives, like the way they talked about it, the way they thought about it-- it worked for them. Put that in quotes. Because that's what you'll find over and over again with models.

If models are working for the people that are using them, whether that's because the data looked good and that they weren't looking at other data, or because it worked for them politically, or because they kept on getting better and better jobs when they talked about how great these models were--

You could even think of it as a corruption in a certain sense. Because working politically for them is still working for them, right? I'm just saying that that is a very narrow view. And the real question isn't this work for you-- because yes, it is. You wouldn't be doing it if it didn't-- the real question is for whom does this fail? For whom does this fail?

And that's the question that isn't being asked, wasn't being asked then, still isn't being asked now.

DEE SMITH: But also, the definition of working for you can also be misleading. I mean, what does it mean to be working? It worked for you for the moment. Maybe you got a promotion or something. But if it brought down your company, is that really working for you?

CATHY O'NEIL: If you got another job. I mean, that's the thing about people don't quite understand how cynical the world of finance was at that time. I would talk to people about that. Like, oh, this model seems flawed, seems like as a business model, it's dangerous for the company. Oh, but I'm just going to skip ship when it fails, and I'll get another job. And that was the assumption.

So it's a very, very narrow perspective. And yeah, working means, in the case of many of those models that we saw fail during the crisis, simply meant short-term profit. I mean, it was very simple. It was very money-based.

The kinds of algorithms that I think about now-- let's talk about the Facebook news feed algorithm-- works for Facebook, again, and it ends up translating into money. But the short term, the sort of more direct definition, is engagement, like keeping people on Facebook. So we're going to privilege the news feed items that keep people on Facebook. We're going to demote the items that people tend to leave Facebook after reading or seeing.

And just that one thing-- of course, it is aligned with profit, because the longer people stay on Facebook, the more they click on ads, the more money Facebook makes. And so that's their narrow definition of working. They're like, this is working because we're making more money. It's very clear that that's their incentive.

But what we've seen in the last few years-- and it was pretty predictable, actually, looking back at it-- is that that also privileged things that we find outrageous. Because why do we stay on Facebook? To argue. Things that we find divisive. Why do we stay on Facebook? To get outraged, to fight with each other.

DEE SMITH: To be part of a group that excludes others.

CATHY O'NEIL: Yeah, or to even be radicalized, and to find your people, your new radicalized people. There's all sorts of stories we've heard. What it doesn't privilege is thoughtful discussions that makes you go and do your own research at a library. Like, we all know that, right? That's not happening.

So we've seen that when Facebook optimizes to its bottom line, its definition of success, which is profit, it's also optimizing directly away from our definition of success, which is in being informed and not fighting.

DEE SMITH: And there's an even further irony to it, which is that by optimizing to that narrow definition of working for them, they've put their company in the crosshairs. In other words, it may not work for them. It may be a very extremely corrosive thing for the company itself in the longer term, in the bigger picture. And yet they've been focused on this very short term-- short termism, of course, is one of the great problems of our ages.

But to back up for a minute. I want to ask you about what an algorithm is. Because it is a term that's thrown around all the time. And you know, I'm a music and math person too. And there is a kind of beauty and crystalline structure to mathematics that makes people think that it doesn't lie.

And mathematics indeed does have proofs, and mathematics itself doesn't lie. But the assumptions behind mathematics most certainly can lie. Walk me through a definition for people, who don't maybe understand when they throw around the word algorithm, what is an algorithm.

CATHY O'NEIL: OK. I'm just going to back up and just disagree with one thing you just said, which is I feel like axioms in mathematics, if stated as axioms, they're not lies. They're just assumptions. The thing that we're going to see in my explanation of what an algorithm is is that it's not mathematics at all, actually.

So what is an algorithm? When I say algorithm, I really mean predictive algorithm. Because if just taken straight up, an algorithm just means a series of instructions. Like, that's not what I mean. I mean a predictive algorithm.

So what I mean by that in almost every algorithm I will discuss is predicting. And not just predicting something, predicting a person. Most of the examples I talk about predict a person.

Are you going to pay back this loan? Are you going to have a car crash? How much should we charge you for car insurance? Are you going to get sick? How much should we charge you for health insurance? Are you going to do a good job at this job? Should we hire you? Are you going to get rearrested after leaving prison? That's your crime risk score.

It's a prediction. It's a scoring system. It's even more precise. It's a scoring system on humans. Like, if your score is above 77, you get the job. If it's below 77, you don't get the job, simply that kind of thing.

But more generally, a predictive algorithm is an algorithm that predicts success. Success is the thing I've just been mentioning in those examples. Like, are you going to click? Are you going to get in a car crash? Those are the definitions of success. Specific event.

And the reason you have to be so precise about that is because you train your algorithm on historical data. So go 10 year, 20 years. This is what I did when I was working as a quant in finance. You look for statistical patterns. You're looking in particular at initial conditions that later led to success. So people like you got raises. People like you got hired. People like you got promoted in this company.

So we're going to hire you because we think your chances of getting a raise, getting promoted, and staying at the company are good.

DEE SMITH: Because you match the pattern of people who have had that happen.

CATHY O'NEIL: Exactly. And so the inherent thing is things that happen in the past, we're predicting will happen again. But we have to define what that means. Like, what particular thing is going to happen. That's the definition of success.

So really, to build an algorithm, a predictive algorithm, you just need past data and this definition of success. And that's it. And then you can propagate into the future patterns from the past.

DEE SMITH: And so how does that play out into-- I've heard you give a wonderful real world example--

CATHY O'NEIL: Oh, yeah.

DEE SMITH: --of your--

CATHY O'NEIL: Of my kids.

DEE SMITH: Your kid.

CATHY O'NEIL: Yeah, yeah. So I talk about this. Because I mean, I really do think it's, like, a no-brainer. It's not complicated. It's something we do every day. Sometimes we give the example of getting dressed in the morning. Like, what am I going to wear. You have a lot of memories.

It doesn't have to be formal. It doesn't have to be in a database. It's just like memories in your head-- things I wore in the past. Was I comfortable? If that's the definition of success for you today, being comfortable, you have a lot of memories to decide what to wear if you want to be comfortable. If you want to look professional, then you have memories to help you look professional.

Another example I'd like to give, though, that shows more of the social structure of predictive algorithms and how things can go wrong is cooking dinner for my family. So I cook dinner for my three sons and my husband. And I want to know what to cook.

And so I think back to my memories of cooking for them. This guy likes carrots, but only when they're raw. This guy, you know, doesn't eat pasta, but he likes bread. And then I cook a meal. Of course, it depends on what ingredients are in my kitchen. So that's data I need to know. How much time do I have-- also data I need to know.

But at the end of the day, I cook something. We eat it together. And then I assess was this successful. And that's when you need to know what was my definition of success.