Gigi Hirsch, LEAPS: Thank you for jumping in on this today. We wanted to spend a little time thanking Kay and recognizing her for all of her contributions, as she transitions from being one of the core employees on LEAPS staff, to senior advisor. Also, we want to welcome Fotios on board and to let people know who he is. So this conversation is for the two of you to interact around LEAPS work and the vision that Kay has been working on so hard since she's been with us, and then having Fotios picking up the reins. A main set of activities we're going to be focused on in the coming year, and probably more than one year, is a joint pilot that involves the Predictive Outcomes Platform and Precision Reimbursement. So there is an opportunity to talk about this—about the importance of advancing real world evidence (RWE) and next generation payment models in a coordinated way, so they can evolve together. So, Kay, what would you say are the key message points around this pilot?

Kay Larholt: I've been thinking about that, and if you go back to first principles and what LEAPS was about, and if you go back to some of our very early schematics, it was about getting the right treatment to the right patient at the right time. It was also about being able to consistently marry the science with the needs of healthcare, which led us to a POP to try and understand which patients are likely to respond to which treatments. You know, a single biomarker is very rarely the answer—and where those fail, combining them with additional real-world features could potentially improve their predictive power in treatment decisions. The idea of a POP was compelling. And, at the same time, as the work of the NEWDIGS FoCUS team progressed, it was clear that we need to improve our ability to track outcomes in order to implement Value-based Perfomance payment models. We began to see the potential to connect these—a connected platforms that together could help us track outcomes and predict outcomes. And so, once we started working on that piece, it became clear that an outcomes tracking platform was important. But that's not enough, we need to be able to predict outcomes—that is, improve patient outcomes through more accurate therapy selection. And that’s how I’m thinking the predictive outcomes platform fits into what LEAPS is doing now.

Gigi: We’ve come a long way since you first joined the NEWDIGS team!

Kay: Five years today, I think! You wanted me to start at NEWDIGS in time for the upcoming Design Lab in February 2017—which took place in the middle of a snowstorm, remember that?

Gigi: Yes, that's right—the campus was closed for that one. It seems like predictive outcomes models will be very complex and nuanced, and involve different phases. How has your thinking evolved as our work on POP has advanced?

Kay: Oh, absolutely, I mean—not even that long ago, when we first were talking about predictive outcomes and integrating evidence, we assumed that meant a meta-analytic approach. That was my bias, because that's what I did for my thesis. But then suddenly, between Fiona [Cai] and Dave [Fritsche], they came up with federated learning and we started looking into that. We realized that actually federated learning is probably going to be the answer, it may be a mixture of the two. And that's why it's so critical that someone with Fotios' experience comes on board. Even if I were not changing roles, we would have needed Fotios because data science has changed so much. I'm a statistician. The last thing we want to do is use outdated methodology. We have to continue to think of bolder ideas and bigger ways of solving these problems, because they are so complicated. It is going to be critical that we don't get left behind by doing things the old way. Data scientists and statisticians look at the world in very different ways. Fotios, you have the advantage of being both, but I don't.

Gigi: I can't remember the history, did you just join us when the Next Wave Forum happened?

Kay: I had just joined beginning of that year.

Gigi: So, do you remember that event? Fotios, we had a kickoff event at the end of 2017—we opened up LEAPS with a splash. Janet Woodcock was a keynote speaker along with a number of other innovative thought leaders. We had some people talking about adaptive platform clinical trials and their experiences with that. I'm just curious Kay, what did you think of that event, when you look back on it?

Kay: I think it was very important. It was a great starting point, and that’s when some of our key collaborators began working with us. You made me moderate a panel, which was terrifying. but it worked, so I was glad about that. I remember the lunch, when Michael Cusumano gave the presentation on platforms—not platform trials, but platforms from other markets, like Apple and telecommunication platforms. People were fascinated and it introduced many of us to important new concepts, like network effects, that we now can begin to see emerging in our LEAPS design work.

Gigi: I was thinking the other day of the sheer number of people that Kay brought to NEWDIGS. They have helped us in so many ways—not just showing up for one Design Lab, but have run teams and co-authored papers. That's been a real bonus, on top of everything else—Kay and her Rolodex.

Kay: I just say to Gigi, it's because I've been around for so long… I've been in the pharma industry for 30 years, I mean—you know people when you've been around that long. The industry is not that big, especially not when you're talking about the statistics group. Obviously, I did more than statistics, but yes, it's been nice to be able to bring some friends along and have them so interested and get involved.

Gigi: Fotios, how about you, what do you think of all this?

Fotios Kokkotos: You know, this is a nice way to start—going back to Kay's five years of work here, and the principles that you mentioned, and how I can take that and add on the data science, machine learning, artificial intelligence component. We've seen some of our partners already emphasize this in conversations. Technology's changing constantly every day. As a teacher, there are always new things that we need to teach our students and, by the way, integrate them into our platform, too. The intersection of statistics and data science, machine learning—I think it's something that we need to work with. We need to get educated from our partners, but at the same time, we need to show our partners that we embrace this new world but also having our feet somehow grounded to our statistical principles. The big mistake that we statisticians make (and we did make, and we will make) is that we don't want to play with others. We are very proud of ourselves—we are nerds!—and we think that statistics is just something very unique. Nobody else can touch it, nobody else can talk unless you have a PhD in statistics. That's a big mistake, and I'm thankful actually that this group here embraces the fact that we need to work together with data scientists, computer engineers, and so on. All the key people from the different organizations.

Kay: Yeah, on my desk I have a plaque, "I'm a biostatistician. I solve problems you don't know you have, in ways that you can't possibly understand." That's true for a lot of different fields—that's true for a data scientist, too. But that's the nice thing about NEWDIGS, the consortium is a collaboration, it's truly breaking down silos so that no one silo thinks that they alone have the answer, because if they do, they're wrong.

Fotios: That's right.

Kay: I've been saying for a long time, when we first started working on challenges with predictive biomarker in LEAPS, I remember having this discussion: If one type of data was going to give you the answer, then someone would have done this by now, because plenty of groups have huge amounts of claims data, or huge amounts of EMR data, or huge amounts of molecular data, but they haven't solved it in most cases. Diseases are complicated, I mean the doctors don't need us to tell them that, but you know rheumatoid arthritis is a very heterogeneous disease, and probably multiple different diseases. Similarly, with cancer, they keep coming down to smaller and smaller sub-types, right? Now there's no such thing as breast cancer anymore, you have to go down to early breast cancer, late, the different types, etc, ultimately turning cancer into multiple orphan diseases? And that's a good thing, because then it is getting closer to precision medicine, it is getting closer to finding the right treatment. But unfortunately, it's still not happening enough that we're getting to the right treatment for the right patient at the right time. So we need more evidence, and that's what it is—it's not more data, it's more evidence. We had that discussion the other day, Fotios.

Fotios: Right.

Kay: Really, you can have as much data as you want, but if you haven't turned it into evidence, and fit for purpose evidence, then data scientists have just been playing with data—which you know we all like to do, but it hasn't actually created anything useful.

Fotios: Yeah and I must confess, this is a very difficult problem. I don't expect people to understand what we're trying to do here right off the bat. We just have to listen to their concerns and keep working. Keep redefining ourselves, reinventing stuff. It's a very, very difficult problem. Even if we get somewhere in small fractions, to me, it's a success, right? Even to make a little bit of progress here within the next year—to me that's a success story. And so you know I keep my bar low, if you wish, if we can exceed it great, but I also know that it's a super difficult problem to deal with.

Gigi: I'm just curious—of the things that you've heard that we're up to, and as you're getting to know our environment, what are you most excited about? And where do you think the greatest challenges might lie?

Fotios: So, the most exciting piece, in my opinion, is the federated learning approach with an integration of the appropriate AI machine learning skills and tools. And, of course, working with all these different partner enterprises who will share models that they develop behind their firewalls on their own data—this to me is very, very exciting. And it's so much fun. I always used to work with one data set with one client at a time. So I couldn't compare and contrast with the others. Now you're giving me the opportunity here, with this platform, to actually interact and work with all these different companies with a tremendous amount of success stories. On the other question, what's the challenging piece here—it's that we are dealing with all these different players. Each one of them has their own reasons for joining this group. Gigi, what you're trying to do is to find this common denominator, that all of them can work together. Even though you know when they join NEWDIGS they also have their own kind of personal projects. So that's something that's extremely difficult. I'm glad I don't have to do that—I'm just a technical person! But in your head, you always have to manage expectations. This is something that, it's not easy.

Gigi: You're most excited in some ways, on the technical front, correct? The greatest challenges are more around the alignment of incentives and agendas, on the human side.

Fotios: Correct. I'm not here to write a policy. I'm here to give you enough evidence for someone else to write the policy, right? Including reimbursement. We're talking about reimbursement and all that—there is a policy that someone has to work out the details. Thankfully I don't have to do that! So, what I'm providing, hopefully, is better evidence for the smart people to go out there and develop their reimbursement procedures and protocols and so on.

Kay: The more I think about it, and Fotios hasn't had the opportunity to see this yet—and it's been hard obviously over last two years, when we've been remote—but I will tell you that the impact of in-person Design Labs is unbelievable! The people who go in skeptical, and then become some of the most involved, they drink the NEWDIGS Kool-Aid because they realize, these are people who are making decisions, and we are all learning together. Exeriencing the full potential of Design Labs is more challenging when they are virtual. I mean, people say "oh, it's just a webinar" or "it's just a workshop"—No, it's far more than that. I really, really enjoy them, and I think we keep making progress. Sometimes it's big things that come out of it, and sometimes it's little things that come out of it, but it all moves things forward.

Gigi: Kay, going forward will you be coming to these events sometimes? Hopefully they'll be in person again soon. Will that be part of your role as a senior advisor?

Kay: I'm hopeful. I mean—you know I don't want to commit, but I have them on the calendar. But I also have a huge long list of things I want to do that have nothing to do with pharma or biostatistics or anything else, like all the things that I put off for an awfully long time that I really need to start taking care of now.

Gigi: Fotios, out of curiosity, I'm just trying to think of what else we would like the community to know about you. You must be pretty excited about what you've built at Merrimack.

Fotios: At Merrimack College, that's right. I joined Merrimack six years ago. I live in North Andover, MA and I'm a huge proponent of community service. My colleagues were interested in data science, but didn't know how to connect the dots and build a program, so I helped them with designing the new online graduate and undergraduate programs in data science. Starting from 2017, we now have over 100 graduate and undergraduate students in our programs. I enjoy the academic world but also continue to provide data science and statistical consulting services to healthcare companies. And the other thing that I don't think you know is that I'm very proud to be a member of the faculty team supporting the sustainability islands solutions program at the University of Aruba. The program trains students to tackle issues such as climate change, food shortages and waste management solutions on this small island in the Caribbean. I provide statistical services pro bono to the island, the government, as well as working with local professors and students. I have the pleasure of sharing and teaching data science to local students, so they can have better jobs, but also assist ongoing sustainability projects on the island. You can see the applications of data, science and statistics everywhere in this world—and I'm glad to be part of this team too.

Kay: Yes, I know I think about it, too. I do a lot of formal and informal mentoring. I do some with public health Masters students, helping them as they're thinking about careers and everything. But also somehow I became the mentor for all my friends' kids who are in, or considering STEM fields. And I just remember talking to one of the guys, I said to him, you could be a data scientist and that's great, and you can be a really good data scientist. But in the end, at some point you're going to want to care about the context. I cared about health care. You might care about retail or you may care about climate change or whatever else. Data science is more than just crunching numbers, or seeing patterns, or fitting models; if you don't actually care about why you're doing it, you won't be an outstanding data scientist. That's why, obviously, NEWDIGS is so meaningful to so many of us, because everything we do every day is about doing something that's going to make things better, I mean you truly feel that. Maybe we're not curing cancer, but we're curing something just as difficult, which is a completely messed up health care system in this country, where the incentives are so misaligned that doctors can't do the right thing for the patient. You do believe that people want to do the right thing and then you find all these things that are stopping them from doing the right thing. There's got to be a way to fix it, and that's why I came to work every day because I'm thinking, what can I do to help fix it? I don't know how to practice medicine, but maybe something I do can help.

Fotios: That’s the theme in NEWDIGS—patient-centricity. How many times in my first three weeks of working with you all have I heard it? And no matter what the slides are, we always go back to that theme.. Whether it’s Design Labs or any other meeting here, knowing that patients are why we are here is what brings us all together and keeps us focused and productive, including payers, pharma companies, FDA, clinicians, patient advocacy groups, VCs—who knows what's next? A related issue that I’ve heard you guys talk about is that patient-centricity is the centerpiece of the work, but our solutions must also ensure economic sustainability of the healthcare system. Sustainability is an interesting topic with many dimensions. For example, machine learning models waste a lot of energy—computing power, electricity, right? There is a paper I just came across about a new artificial intelligence network that is trying to cut the amount of energy by half, cut the consumption that this particular model required. So computational efficiency- this is another dimension of sustainability that is important to our environment, and something else we should be thinking about.

Gigi: This is interesting because one of the things that we had been talking about for a while was how many of the companies we work with are putting in so many resources into building their own data lakes. They license massive amounts of data, that require many resources to maintain it. Many companies are doing it in parallel, it's duplicative. And a big part of what we're trying to do in LEAPS is to define a new collaborative space where we can address important shared evidence gaps better, faster and cheaper together than alone, with no proprietary risk to any party.

Fotios: Right, and federated learning can do that—not only by helping us with evidence, but it's also addressing the issue of sustainability.

Gigi: And we've always talked about the importance of platform strategies for greater scalability and sustainability of RWE production. So far we had been thinking of economic sustainability but it would be interesting to explore the dimension of computational sustainability. It would be interesting to see if there is a way to quantify this dimension.....

Fotios: I think our partners can help us with that. I have a few ideas, but I think our partners can help us with that.

Gigi: I think that would be great because the sustainability—and the increasing emphasis in the investment world on ESG [Environment/Social/Governance investing]—it's another powerful dimension for funders to consider in terms of the public benefit of our approach in LEAPS.

Fotios: Precisely.

Gigi: Are you both optimistic about the potential for federated learning and data science to solve some of these intractable differences in incentives across the parties? And how would that happen?

Fotios: [Laughing] I'll let Kay go first!

Kay: I think the technical challenges are easier to address than the incentives. You know, I do think that COVID has shown us some important things. While it has illustrated shortcomings in our public health system for sure, it has also shown us some amazing accomplishments that are possible when incentives are aligned. Like the acceleration of development of life saving vaccines. But in general, the incentives are a major hurdle.

Fotios: I agree with Kay but we can't give up. Even if we only take baby steps, I think we're moving in the right direction, but we cannot give up, especially when we're talking about patient centricity.

Gigi: I would agree with that. I think one powerful incentives lies in making sure that the evidence we produce is valued by all the parties, which is why we spend so much time on the planning in the life cycle: plan, produce, and use evidence. That planning, I think, is key, and right now there isn't a place where that is orchestrated. There's no entity that does that. So making sure that the evidence that is produced is valuable to everyone. Secondly I think we're in a unique position where we can also >catalyze change in reimbursement

Kay: This kind of goes back to the work in NEWDIGS on adaptive licensing. The case studies we explored there highlighted the need for modernizing how we generate and use real-world evidence to progressively reduce uncertainties about safety and effectiveness of drug therapies, and in conjunction, develop more flexible reimbursement models that go either up or down depending on what that evidence reveals.-

Gigi: Well that’s a great note to end on. Kay—thanks again for the many ways you helped us build a strong foundation for the Land of LEAPS. We look forward to continuing to work with you as an advisor. And thanks too for finding yet another gem in your Rolodex at this critical transition point for us. Fotios—welcome aboard! We look forward to working with you, and together we can build on this powerful foundation to bring sustainable, patient-centered innovation to life.