A Visit with Michael Harhay

0:00Judith: Welcome to Berry's In the Interim podcast, where we explore the cutting edge of innovative clinical trial design for the pharmaceutical and medical industries, and so much more. Let's dive in. Scott Berry: All. right. Welcome everybody. Back to In the Interim. I am your host today Scott Berry and I have a guest with me today, and we've got a number of topics to get to. But first, let me introduce our guest for today, Michael Hoge, who is the As is an associate professor of Epidemiology Medicine and statistics and data.

0:35sciences. There's a lot of, and in there, and, we'll, we'll, we'll talk about that at, By the way, the university of Pennsylvania at Penn, uh, Mike Michael has an interesting history, at Penn. Uh, we'll go through that. he's also the founding director of the Center for Clinical Trials Innovation, which specializes in statistical design, analysis and interpretation of. Large randomized trials and cohort studies. We'll talk more about that. But Let's.

1:06first talk about how you got here, Michael. Uh, You have a PhD in epidemiology from Penn as well as a master's in statistics, uh, uh, as well. So, uh, you have multiple degrees from Penn. So let's talk a little bit about how you got here, a little bit about your history to this point, and welcome to in the interim. Michael Harhay: great Scott. Um, thanks you so much for having me. It's a real pleasure to be here. Um, a big fan of the podcast. Um, yeah, so I've written a lot of personal statements.

1:40I've told this story a lot of times and, um, I wish it was a little bit more romantic and straightforward, but. I've always kind of, uh, believed in this concept of a calculated drift, and I think my calculated drift towards Academia started, um, as freshman, sophomore year of college. I went to a small liberal arts school outside of Pennsylvania for those, um, not from the us It's has about 2000 students. The name was Berg. Um, and I just really enjoyed it.

2:08I like being in an academic setting. Um, I majored there in neuroscience and philosophy and very much wanted to stay. And do more academics. Um, and I like the multidisciplinary, um, opportunity of being in a university setting where I could dabble in the sciences and the humanities and philosophy and I didn't really know how to do that. Um, so some Googling I found Penn's bioethics program, um, which saw some bio related to neuro, and I saw some ethics related to philosophy, and it seemed like a good next step.

2:40Um, so I came to Penn, um, the year after graduation. Um, so this is 2006 at the time, and I quickly found out that you could pay for your graduate degrees if you were a full-time employee. So I got a job in Penn's pulmonary and critical care division. Um, basically working on patient registries for pulmonary hypertension. And I just started taking classes and learning how medical research worked. Clinical research worked, um, alongside a bioethics degree.

3:12I was really hooked on the production of data and how data was used. Um, and I didn't quite know how to transform that, so I just kind of kept on taking classes over multiple years until I met a young fellow who had transitioned onto faculty. And I joined, um, I started my PhD in 2013 in epidemiology. So during that time I picked up a couple degrees, um, in bioethics public health and demography, and then. Ultimately my PhD, but a lot of that I think was just me experimenting with different ways of doing human and human level and population level research and trying to figure out what I found attractive.

3:53Scott Berry: uh, interest. So, so at what point in there. Did the quantitative side of that, you had philosophy and bioethics and it sounds like initially maybe in the the pulmonary and critical care you were doing some data stuff, at what point did the quantitative side of this interest you? Michael Harhay: so I was, always handing off data sets and then they would come back from a PhD level statistician where collaborator and there are always gaps where I was kind of asked to do smaller things, um, in Excel.

4:23And I think I increasingly just started to teach myself how to code, um, first and formally. Then I started taking, um, classes in 2009 where I was first exposed to data. Um, I know some users and listeners may not be a big fan, but I just really enjoyed coding. Um, and then I joined an analytics group in 2011 where I started to really get to work with large data sets and. I loved it. I enjoyed nothing more than waking up, making coffee and just coding throughout the day.

4:52And, um, I purchased a bunch of books back then and I increasingly just got really interested in transforming data, extracting things from it. Um, but very aware that I needed some type of formal training to really translate into a set of tools, um, that I could actually extract things from. Scott Berry: Interesting. So, so if, if we're, if we're counting here, just to make sure I have this right. So you have four Master's degrees and a PhD. Do I have that right?

5:29Michael Harhay: Yes, that's correct. No more for now. Stopping. Scott Berry: Okay. Okay. Uh, alright. We'll never say never in that. All right. So I asked you how you got here and now. So let's talk about here. What, what, what is here. so you're a, an associate professor of Epidemiology and, and you founded, now, uh, upon reaching Tenure, Tenure, you founded. the Center for Clinical Trials Innovation. Uh, tell me a little bit about the center for Clinical Trials Innovation.

6:01Michael Harhay: Yeah, so it's very much. A culmination of about 10 years of being exposed to different topics and trials. Um, so during my PhD um, in epidemiology, I had a little bit more of an apprenticeship type where I worked under a physician scientist and MD PhD, um, who ran both behavioral trials but also comparative effectiveness trials. um, and uh, behavioral trials tended to be large multi-arm, uh, trials with multiple different con contrast.

6:31And then we had a couple. Critical care based trials in, um, our healthcare system. And both of them kind of brought me into a lot of different questions, um, that we can return to, but some of them being, um, just dealing with multiple intervention studies and multiple comparisons and thinking about what the risk of false answers was. And that always kind of bothered me when We were doing behavioral things. Um, and then on another side, uh, we were doing studies in our healthcare system and there's inherent clustering, um, inherent time factors.

7:03There's also fixed sample sizes, um, and you can't really control them. So you're trying to think about how to do things best. And then another area. of interrelated research that, um, I spend a lot of my time thinking About that's become a big driver in the center is how to deal with, um. Not fully observed outcomes where we call it informatively truncated outcomes. So in the ICU, we're very interested in questions like duration and mechanical ventilation or how long an individual is in the ICU, but if a percentage of those people die, you don't have kind of a clear treatment effect estimate comparison, because for some people time in the ICU is time to death.

7:40And for other ones it's time to discharge. And at the time I didn't really have the ICH nine framework for, um, the. Intercurrent events. So a lot of my dissertation was trying to make sense of that, um, in a more, um, immature way. But after that, I did a, um, K 99 or zero zero award, which w um, supported my postdoc and then became my um, early career faculty award. Um, it supported me my time at Penn and during that time I started to build out the work on, um, intercurrent events and the s demand framework became more popular.

8:15And I started to get involved in more formal ways, um, in trials increasingly as like an MPI in a trial where I would work, um, as a kind of methodology lead with a clinical lead. And that started to lead to a lot of research questions that we didn't really find answers for in the literature. And I can talk more about them, but over the last four or five years as an assistant professor, I started to feel like We were trying to answer questions where the methodologic guidance for how to answer those questions wasn't clear.

8:46And what I wanted to do with the center was to kind of create a way to answer those, but also, um, to support a training hub where we could train young academic trial methodologists. Um, it tends to Be mostly statisticians as far to these real world questions and to kind of advance their careers and, um, that's what I'm trying to put together at the moment. Scott Berry: Very nice. So I, I, want to come back to this question of measuring things, uh, uh, critical care.

9:13Uh, very interested in, in endpoints, in critical care, the, uh, about the center now. So you have Uh, dedicated uh, personnel to this. You have other individuals at Penn that are part of this center, so this is just a, a relatively new center, but you're looking to expand its impact and its role, it sounds like. Michael Harhay: yeah. Um. I think it's a little bit of, it kind of existed in a non-formal way on one hand. And the other hand is like, if you build it it will come.

9:43So, um, I have a couple, we have two methodology grants. One's an RO one from the NIH and one pcori, and that supports like formal statistical methodology. We mostly focused on this question of intercurrent events and composite outcomes. And then we have several trials that we support. Across the continuum. Some just being in consult, uh, consultative support and others where I'm the formal pi and we have one of someone on my team serving as a primary statistician.

10:13Um, and these are all health system based trials. So I've been using those different funding mechanisms to kind of build a base. So we have a formal PhD staff, statistician, two postdocs. And then a lot of collaborators who are really into the calls, um, and are helping me bring on mentees and also bringing on their own trainees in different stages. So it's starting to become a little bit of a critical mass and gets them inertia. Um, and it, it's kind of really cool to watch.

10:41I don't know exactly what it will become, but there's a lot of people who want to do more innovative health system trials, and that's become very much our focus and. Increasingly, there's just new opportunities and new, um, venues that We can expand ourselves. And I'm, I'm, very strategically trying to find cool internal partners. Um, and it's been a lot of fun to kind of create and just kind of be creative on a vision, um, like something like that.

11:08But it's also to calculate a drift concept. Scott Berry: Yeah. Yeah. Uh, very nice. So I I want to, I. I I am very interested. in this idea of healthcare based, uh, uh, health system based trials. Tell me a Little bit about what that means for Penn. Do you have a set of hospitals and you're doing comparative effectiveness within this? So it's part of caring for the patients, but also learning what, what, what what is available for Penn? Michael Harhay: Yeah, so we have several trials within Penn, um, and also we collaborate with several external partners.

11:42So let's start with Penn because it really, I think, highlights. Where I think the innovation kind of comes into what we do. So Penn Medicine, um, has six hospitals, uh, depending on how you count about 40 exclusive primary care practices and then a handful of other practices. There's obviously outpatient clinics, all different types of things in the greater Philadelphia region, and that is a remarkable. Opportunity for experimentation. Um, you can improve primary care practices, you can improve in hospital practices, you can improve ED delivery.

12:20And we have multiple different ways to experiment. We also spread across wide geographic areas with different, um, patient populations. We're serving different patient populations, so going out towards rural Pennsylvania, going into more dense, um. Northern New Jersey area. Um, and then obviously have different practice variation. So we team with a couple different groups. At the moment, it's principally critical care, primary care practices, and we have a specific center called the Center for Healthcare Innovation and Transformation.

12:51That also is collaborating across the health system, trying to help different service lines, um, like obstetrics or um, oncology improve. Or do things that they believe will improve the delivery of their care or just sometimes be, have a good return on investment. So I can give you a couple examples of some projects we've done that I think are kind of neat in that vein. Scott Berry: yeah, no. So that is that I, I think this is actually the future where if you think about it from a perspective, our who do we learn from about, right ways to give treatment, what treatments work or not.

13:25We learn from such a, small fraction of patients. Uh, in Clinical trials, randomized said, but yet we're, we're, these decisions are being made every day. and I think critical care is a really nice area and and I'm, I'm echoing what other clinicians tell me. I'm not an ICU doc, but largely care is random. There's, they, they don't know a great deal about this. They're individually perhaps learning at a patient level, but None of this, uh, there's a huge set of things that are done that we don't learn from that we could potentially make this a learning healthcare system.

14:01We could treat them better. We could learn, uh, extend those. I was involved with UPMC, which has a large extent of, of hospitals west of you, uh, Pittsburgh based. But even through that, Where even in elective surgery, there's a great deal of things that, That they really don't know, uh, about how to care for nausea and vomiting, how to, you know, what should you do before surgery? all of these things that could be learned as you treat in that. So I'm fascinated.

14:33You said you had a few examples of that. I'm fascinated by what some of those examples look like. Michael Harhay: Yeah, I think some of them are so elegant and clever and they have such profound changes. So, um, I'll I'll start with them and we can Go through them in a different depth if you find 'em interesting. So we just had one paper, um, published in JAMA Surgery where the idea was to change how many opioid pills the EHR, um, default ordered.

15:01So if you come into the ER and you have a major injury, the default number of opioids may just be 30. So we changed it to 10. And then asked how frequently do people come back for new medications? How often do people complain? Um, that it's insufficient. And we found that, generally speaking, very few physicians overrode the change of the default prescription number. And we didn't have a large number of complaints to the ability we were able to follow up patients.

15:28So opioid um, use is a big challenge in the Philadelphia region, and one of the goals is to keep opioids. Where they should be ideally, um, and to those who need them. So simple kind of default changes in our EHR are something we try to leverage a lot. We've used similar mechanisms to increase the delivery of palliative care. Also to make the delivery more targeted. So it is a finite resource and you need to, um, send the physicians out to where they can ideally be most, um, useful and have the largest impact.

16:01We've used it to kind of triage people. Um, in our primary care practices, we have used it to, if someone's come back several times and just has a high blood pressure that's not coming outta control, maybe we need to identify them as someone who's high risk for genetic disorder and should be screened for a different type of therapy. So we are trying to find ways to improve, and this is not really me thinking of the clinical ideas. I'm kind of helping with the design, but.

16:27These are the questions we get is like, this is something that we're not doing in a standardized fashion. It's kind of an easy win. If we could do it better, can I test it and create some evidence base around it? Um, and people are really clever clinicians, really see how the things work, and it's really neat to hear about the human computer interface with DHR and ways to modify it. And I'm helping kind of put a statistical paradigm around that to test it.

16:51Scott Berry: So interesting. So the, the question of how many pills is the initial order, the default order, um, within that, um, that's done at a cluster level or a hospital level, it sounds like not a Patient level. Yeah. Michael Harhay: Yeah. So for a lot of reasons, um, we tend to do the vast majority of our research at a cluster level. It tends to be a practice level or a hospital level. Um, so we have six hospitals in our system, or 40 primary care practices.

17:19So we tend to do cluster level randomization. We also do provider level randomization, though there's some, there can be challenges of that in different fields. Um, some just shift, um, specialties just have the way the patients Are exposed, um, can potentially shift. So we're tend to mostly randomize at the cluster level. We do do some individual level patient randomization, but that's not kind of our, um, It's a, it's a minority of what we get to do.

17:47Scott Berry: Yeah. So I'm, I'm, I'm interested as you do this in, in, a number of trials that have tried to be, embedded within the healthcare system and, and that there's a whole range of what embedded means. But in that setting, once you're addressing one question, you're collecting the data and you're looking at it, it, becomes. I don't know, intoxicating is a pejorative word, but it becomes intoxicating to say, we've got a captive set of patients and questions.

18:16let's investigate, two things rather than one. let's do factorial things. Let's do three things. You know, it's almost free to add a third thing once you're doing two and, and you know, four sort of thing. Has that. happened within these trials? Michael Harhay: Yes, because a lot of the interventions we're doing are not usually single component interventions. They tend to be multiple components. And there's a couple scientific questions in the multi-component question, just by default.

18:46So do all the components need to be there? Is it the existence of one? Can we get a little bit of a, an effect by adding one? And then if we add a second one, do we get kind of an additive or a multiplicative effect? Um, so we play a lot in cluster designs, cluster randomized trial designs. Just for listeners who may be unfamiliar, that means that we're randomizing. Groups of people. So groups of people based on, um, what primary care practice they come into, what week they get their critical care, what emergency department they come in, in our healthcare system.

19:18Um, so we are playing around with a couple different component designs, um, where we are trying to look at all these different questions. And so, yes, a large number of questions tend to be embedded in a lot of our trials. Um, and it is largely to try to decompose. Different, uh, contributions of different, um, strategies to improve an outcome, Scott Berry: Yeah. Very nice. Uh, well, what's come up in a number of hours as well is. Combining together cluster, randomized with individual randomized in a factorial way.

19:52Have, have you run into that. Michael Harhay: so we have our first trial, uh, under review at NIH where we have. Cluster level randomization of clinics and then, um, individual level randomization within patients. So we are just kind of this, um, two level cluster randomized cluster randomization. I Can't take credit for the design. Um, one of my primary collaborators and, um, my, uh, cluster trial, um, Oracle is fan, Lee is a biostatistician at Yale.

20:23Um, so he's a big. Thought leader in our group, um, and contributes very heavily in our program. So he's developed cool designs like that. So I'm very hopeful that when NIH funding kind of comes back online officially, that will be one of the, um, cool things we get to really roll out. Scott Berry: Hmm. Very. nice. Very nice. I I, you talked about endpoints, I imagine in all of these endpoints are, are interesting, but Let's go back to, the critical care scenario.

20:48And I bring this up because my call last night, with. we have a global call for. REMAP cap. The REMAP cap is a trial that is a community acquired pneumonia is the cap part of it. And so we generally, have hospitalized patients in, in ICUs. And we, we, for 10 years now, I've talked about what's the right endpoint. Um, and, and it it's continually the discussion. So it sounds like you spent a good bit of your dissertation on this question. Of, of this And still addressing this question.

21:22today. And it's Interesting. You brought up the role of E nine and, and, and, all of this, So let's talk about critical care. endpoints, um, within this. Michael Harhay: Yeah. I've spent, I think, uh, at this point a quarter of my life trying to solve this question. So Scott Berry: yep. Yep. We see a lot. of this question, and I, I've kind of fought hard that even they'll ask questions about what is the, what is the distribution, of length of mechanical, ventilation for those that don't die?

22:02So let's set up a scenario where it's intensive care. You can go back to. COVID and some individuals come in and they're on, me, uh, uh, life support, let's call it. And you're interested in the length of life support. You're doing interventions to them, and you're wondering that and some patients die. during that process, if you're really interested, in the length of organ support, what do you, uh, what do you do? with That patient, within the setting?

22:30and it's always struck me as hugely problematic when you sensor that. out or you remove patients that die, and you only look at the length of organ support and you've pulled people, out that in fact are the, the worst Outcomes Within that, It's, it's a really hard question without somehow including mortality. as part of that. Michael Harhay: Yes. I couldn't agree more. Um, you want me to respond Scott Berry: I, I want you to respond. Yes. Michael Harhay: Yeah.

23:02So, um, I this is, I think this question has, uh, very profoundly shaped me as a methodologist. I think it shapes the way I think about questions. It's been very transformative for me, and I, I can just give you some, uh, cliff notes about that and you can tell me which part you wanna jump in most. So, as a PhD student. At that time was calls inference. It was around of course, but I don't think it was as popular as it was today. And the Esti mans framework, especially in our current events, I, I, didn't have a formal framework, so at the time I was just thinking about it as we have a distribution in front of us, how do we model that distribution?

23:43And it's, you know, tends to be a little bit of a negative binomial distribution kind of gamma, where you have a bunch of people early who have, are discharging in a Long tail. But then you have people that die. So this really kind of complicates the question. So for people, as I mentioned before, people who die, it's not duration, mechanical novation, uh, mechanical ventilation, it's time to death. And for those it's, uh, who survive, it's their time to discharge.

24:11So I spend some time exploring what are options. And the big one at the time was competing risk models. And the challenge of competing risk models, just like survival models, is they kind of make this assumption. About what would've happened if people didn't have the event of interest. And this is counterfactual and obviously people are not going to not die. So it didn't really make conceptual sense and my PhD mentor at the time said, yeah, competing risk may be a modeling solution, but it's conceptually invalid.

24:39So then I moved on to this concept of joint modeling, where you would kind of model the, um, longitudinal probability being on where the daily likelihood of being on a ventilator. And then you can also model the, um, competing events separately and then pull them together. And I remember my other mentor said, but now we're thinking about a counterfactual probability distribution and being on a, uh, ventilator that would've been observed if people never died.

25:05And that was the first time I was really exposed to this concept of the hypothetical scenario for estimates. And I never found it conceptually, um, digestible. So then I spent a few years kind of getting into this estimate called the survival Average Causal Effect. And I find a lot of attraction here, but it has a limitation. Um, so I can talk about that for a Scott Berry: minute yeah. Yeah. So, so let's like de describe what that, is. Um. Michael Harhay: So it's a clever estimate, and the idea is that at the beginning of a trial, you have these latent strata of individuals.

25:40So the simplest scenario is you have Some people that would always survive to the end of the trial. And let's just say this is a 60 day follow up. You have some people who are very sick and are gonna die, and then you have these two middle strata, and these are people who, I'm sorry, that last strata is two people. They would die regardless of. They were exposed to the intervention where to control arm. Then you have these two middle strata and You have some strata that will only survive if they're exposed to the intervention.

26:07And then some people, they'll be harmed by their intervention. So they would only survive. They are, uh, ex um, given to control. And what you want to be able to do is identify these always survivors, this four strata. And these are these people that would always survive to end the trial regardless if they received intervention or control. And the concept there is that if you can isolate this strata people or stratum, um, then you can say this is the average causal effect among these individuals.

26:34And I think there's a lot of kind of conceptual attraction there. But the obvious kind of issue is that you're no longer looking at the full randomized population and that raises issues of the ITT. Um, it's also a much smaller population and critical care trials are already small. So I like that. And I spent a lot of my life developing statistical methods to estimate that ' quantity. cause I do think it's informative and I think certain clinicians like it and it's.

27:01Viable and attractive. Um, just as additional information, I think about it as like a subgroup type of analysis. Now that's informative, but for the last several years we've increasingly been working on composite outcomes, which is the kind of last third, uh, intercurrent strategy. And we spend a lot of time trying to think about What is a statistically valid, patient, relevant, and clinically interpretable. um, Uh, composite outcome and my colleague Chris Scott Berry: Yep.

27:31So, so let's let's pause here for a second. Uh, and then we'll go to, to, to, to Chris. So a composite. You talked about hypothetical, and so largely in the ES demand, we're trying to identify what would've happened if this, this intercurrent event of mortality wouldn't have happened. In order to model that, a composite outcome is saying, We're actually not gonna even make mortality a an intercurrent event anymore. It's We're gonna make it part of the endpoint.

28:00That Somehow we're gonna create a single endpoint where mortality is part of that endpoint and, and that's making it a composite, uh, uh, within that. And so that, that's what you mean by that. Okay. So, So, what might that look like in an, in the intensive care, what's a composite strategy of mortality? and organ support in the ICU? Michael Harhay: Yeah, so I think the most traditional one, um, in, I know you have used a variation of this in some of the remap cap platform and many trials do is what we call ventilator free days or a wider capture.

28:38Its organ support free days. And the concept here is that you have a distribution of days over a duration of period of time. So let's just take 30 days. And for everybody who survives, you have the number of days that those people didn't require. Mechanical ventilation, where some other modality of organ support. And then for those who die, they tend to get a fixed value. variations of it have treated that value as zero, like no organ support free days.

29:07And there's another derivation that uses it as negative one to kind of demarcate that it's a separate, um, level in this continuum of the survival, um, the, the observations that would observed among survival, those who survived. Is that a fair summary of it? Scott Berry: I I, think that's a fair summary. And so you've got, you've got, um, this minus one outcome that a patient died. Zero would then be that the patient survived, but They had zero days free of organ support for 28 days or 30 days.

29:41They might have been on organ support the entire time. Somebody who's a one. Survived. And they had One day, free of organ support, uh, and then two and then so on. Michael Harhay: correct. Scott Berry: And so now that endpoint is a, composite. it includes mortality. Uh, in the end point, it includes us. by the way. This shows up in all kinds of diseases. We do a lot of neurological, diseases. You started in, uh, your, your undergraduate in neurology, but you can think of Alzheimer's.

30:11where you look at a cognitive test of an individual, and then you have patients that die in that process. And what do you do with death? And cognitive decline is, is really the same sort of issue. We Have, this in acute stroke. we have, individuals and we have, their neurological status, and you have some that die. What? How do, how do you do this? so in many ways now you've created a composite endpoint. what I've been worrying out a a lot about is how do you analyze that thing?

30:42So Michael Harhay: Yep. Yeah. no, it's um, it's a numerical or statistical solution. That has, um, kind of just really to me, intractable conceptual challenges. Like how do you analyze that distribution? So then it brings back to something I mentioned a few minutes ago. You can look at it and you can say, okay, we're, is it left skewed? Right? Skewed normally, streed skewed. Then You can think about like, well, do you want to prioritize certain, um, shifts?

31:11Do you care about the shift between negative one to zero? Um, it's just a lot of questions that I have not really been able to fully set on. Um, and that has always brought me back to the estimate framework where I tend to look at these distributions and say, okay, well now we solved the issue of what to do with death, but what do we actually want to extract from this solution? Um, and I've never personally found the ventilator free day where to organ support free day to be kind of, um.

31:41that attractive. So over the last few years, um, we've been spending a lot of time thinking about this methodology of the win ratio, which also has limitations, but I'm happy to talk about some of that concept and what I like about it a little bit more. Not maybe tons more, but a little bit more. Scott Berry: Yep. So, um, uh, my issue interesting, the organ support free days. We use that in remap cap, and we had a number of results that came out, of it, and I think it worked very well.

32:10We did a proportional odds model across that, and it worked. very well. And I'll say that because The distribution of those on active or placebo. when the clinicians looked at it and the statisticians say, these are different, they looked at it, and they said, that's clinically meaningful. Uh, and they, they, they thought they could explain the effect, and largely. it's because in COVID, it was a very homogeneous disease. The effects. were largely proportion across the scale.

32:43They had the same effect on mortality. They also shortened organ support. There were some, exceptions to that. Uh, so I think it worked because at the end of the day, the clinicians thought it worked. Uh, in that, the huge thing I'm. running into now is almost every, the way everybody analyzes this. There is a statistical model that comes out, whether it's, the win ratio, whether it's the proportional odds that that creates a weighting of those values.

33:15Mortality is weighted relative to zero through a statistical assumption, and that bothers me. I don't think that, should be done. I think we need to go all the way. And this is very Bayesian perhaps. Maybe Bayesian is not the right word, but we need to clinically weight those states and analyze them that way. I think otherwise you're invariably gonna end up with results you might not actually believe are clinically relevant. Michael Harhay: Yeah.

33:41Um, first. I would like to just at least modify my last comment. I may have not spoken, so I don't, I have no fundamental problem of any approach to composite outcomes. I, and I agree with you. They give you. a net, they give you a summary of the net benefit of an intervention. I find that generally attractive. I've heard, I've heard you and other people talk about this weighting scheme and it's, it's a very provocative thing. 'cause sometimes I think we do think that we do other things like the win ratio and circumvent this issue, but we've just done it differently.

34:10So. Scott Berry: right, right. Michael Harhay: Your, point's well taken. Um, yeah, I, I, sorry. Just thinking about like, all these things get me really excited, so, Scott Berry: so, Yeah, no, it, it's Michael Harhay: way to jump. Scott Berry: exciting, right? I, yeah. Talk Michael Harhay: to talk about poor outcomes, but I can feel this is like, to me, one of the cooler and more challenging and like, um. Such essential problems that exist in clinical research today.

34:35And as you mentioned, it spans so many different disease areas and pathologies and better methods are so important. Um, and ways to, um, incorporate stakeholder perspectives and weigh that in so that we can figure out weighting schemes that are really meaningful to the stakeholder distribution involved is what Is such a kind of unicorn and holy grail that, um, we spend a lot of time thinking about how to do, but haven't really quite figured out how to do well.

35:01Scott Berry: Yeah, and I, and I feel like that's, such a, a, as you started this, does it get you out bed in the morning? And I I feel like it's. It's where we become, really scientists and we're using quantitative things to tie to clinical outcomes and that, that's exciting. It's not a, a mathematical thing that we do back in our lab, and it is really neat, but it doesn't really help patients. Right, right, right. I, I brought up the, the B word in a question.

35:28I ask a lot of people on here. Do you consider yourself a Bayesian. Michael Harhay: Yeah, so as I mentioned to you in our email, like I didn't get a lot of exposure to Bayesian thinking before I was a postdoc and really before COVID. And it was innovators like you, who really, I think brought Bayesian very much out of the shadows into the mainstream. And when I started first reading it, I was just, I found it so intellectually. Um. Appealing. It just makes sense to me to think that way.

35:59I like to think on a continuum. I don't like to be, I realize that there's a decision theoretical component where you have to decide whether or not something's efficacious or not. enough to move on it. But I I like the probability and the posterior distribution. Um, the other thing that I find very attractive about it, um, this is a long way a of saying, yeah, I'm a convert. Um, I still do a fair amount of like frequentist trials. And I, I appreciate a place for them and I, I, I've heard other speakers and I would like to think of myself as intellectually nimble enough to kind of choose among a set of tools for the task at hand.

36:34But I feel like Bayesian fits a lot of task at hand for me. So I'm increasingly trying to push people to do it, And I also feel like it makes a lot of sense, and this is where I was, um, going with my answer, if I can, um, tie it up, is when I was working on a lot of behavioral trials where we were trying to reduce. People from smoking by giving them, um, some type of financial incentive or even in our healthcare system where we're just trying to nudge someone.

36:59So you're looking at two or three different, to me, very low risk interventions. Like what is the harm of giving someone the wrong EHR Alert, We're giving them too much money to stop smoking. So I always thought about like, what Errors meant versus just wanting to know if something's effective. So I kind of like the Bayesian framework because I don't kind of like, I don't find an intrinsic penalty of a P value and minimizing P value in alpha spending to be a meaningful set of constructs to guide me when I'm doing inference in a lot of the questions I'm asked to help people with.

37:34So I think for a number of reasons, I have. circled back to a Bayesian framework, um, to answer questions. And I realize there's a lot of discussion that could go on some of what I just said, and there's a lot of details in the new FDA report. But globally, I find that lens to be an attractive one to think about the broad array of questions, um, in a lot of different areas that I'm asked to think about. Scott Berry: Yeah, and especially given all your work you've done.

38:02in comparative effectiveness, you know, we, we have these phase three trials for a new molecular entity and I get the 0.025 and a a p value Makes sense. But if you're comparing strategy one to strategy two to strategy three. The Whole. idea of hypothesis testing doesn't really work. And so within all of this setting, this makes a ton of sense. Yeah. Well, that's fabulous. The work you're doing sounds super exciting, um, and and stuff. that gets me outta bed in the morning and very interested in, so I'm, I'm I'm glad that that excites you as well.

38:36I hope as this develops and this center innovates, you'll come back on and tell us about all the cool stuff. You're, doing, Michael. Michael Harhay: it'd be my pleasure. Scott Berry: Wonderful. Um, And thank you for joining us here in the interim everybody, we'll be here next time on the interim. Thank you.