A Discussion on Virtual Controls

0:00Judith: Welcome to Berry's In the Interim podcast, where we explore the cutting edge of innovative clinical trial design for the pharmaceutical and medical industries, and so much more. Let's dive in. Well, welcome everybody back to In The Interim. I'm your host, Scott Berry. And as, as you know, or i- if you don't know, this is a podcast, uh, sponsored by Berry Consultants, where we talk about all things clinical trial science, medical science, d- uh, developments in these areas, and the, the intersection of these.

0:40Um, now, I may… We, we do this mostly weekly episodes and a lot of them, uh, recent topics that come up. We love to hear… I'd love to hear from you what topics you'd like to hear from, from, from me or a group here at Berry that spend a lot of time on adaptive trials, Bayesian. We, we think Bayesian. We believe in Bayesian. Uh, we, we, we practice Bayesian. Uh, by the way, Berry Consultants, uh, last year we looked at all the trials we designed, and 50% of them were Bayesian 50% of them were frequentist.

1:22We design frequentist trials, um, but we think Bayesian. So if there are things out there that you think would be good topics for a podcast you'd like us to talk about, that would be great. By the way, this week I got a, um, a, a request on LinkedIn to talk about multiplicities. What do Bayesians think about multiplicities? Uh, which by the way, is a really hard topic, but, but one that, uh, I do have one plans coming out. We have a really neat paper, uh, here on doing some things with multiplicities, and really, it, it touches on the view of, of, of a Bayesian's view of m- of multiplicities.

2:01So I'd love to hear what, what, what topics you might find interesting. Uh, another way to judge this out there is various conferences or invitations that we get to speak and, and what's going on in the, the world of conferences. I was actually at two conferences this last week. One of them was on statistical innovations in rare diseases, and, um, we've talked about a lot of the concepts that was discussed there. Very, very exciting, um, uh, conference that went on.

2:37The second one of them was a conference called… It was a regulatory science hackathon, and CSRC hosted this. Number of people from industry, academia, uh, from, from, uh, FDA were there. Now, what was the topic? Uh, what? Regulatory science hackathon. What, what's the topic of that? The title of it was Digital Twins and Synthetic Controls in Cardiovascular Clinical Trials. So this whole topic and question of digital twins, synthetic controls, and how do they work within clinical trials.

3:23So I was asked to give a little presentation there. I've, I've interacted on a number of things. I, I'm not gonna talk about anything that was said at the conference. Part of this hackathon is that what, what is said there stays, stays there and was thought to be more productive. But I'll talk about what, what I said. I can talk about what I said in this. But again, wanna go back. If there are certain topics you would like discussed, you know, things we may think… you know, uh, don't think of that you would think would be interesting from that perspective, would love to, to talk about those.

3:59So My discussion, and I, I, I'm not gonna go through and, and, and present exactly as I presented at the conference, but I thought I would go through the topics. So I started off, and how I wanna start off is, is thinking about what is a digital twin, virtual control, talk a little bit about these. Uh, I'll talk a little bit about things that have been done before we started talking about something called the digital twin, uh, virtual controls, controls, historical controls.

4:35How does this all fit in? What, what, what are these things? So a digital twin is a very, uh, specific term given to a digital model of an, of a real-world physical product. So i- if you go to Wikipedia, a digital twin is a digital model of an intended or actual real-world physical product, system, or process I, I'm gonna talk about this as something that is specifically around the mechanical parts of this. So for example, in cardiovascular disease, a digital twin would be a model of the human cardiovascular system that you could go in and alter something in this system.

5:35You could decrease the LDL in that digital model, and it would tell you the cardiovascular risk in the next 10 years. Now, we, we, we know something about this, um, w- in, in it, and you could imagine a digital version of a cardiovascular human system that we alter Lp, or we alter the weight of the individual. We alter the characteristics, and it would tell us something about the system. And you can imagine in clinical trials, we'd be very interested in understanding what is the cardiovascular risk in the next five years, say, uh, of that, of that, uh, system.

6:28Now, uh, an example of this, and I'll, I'll present-- I wanna differentiate that from other types of things, but a very mechanical, um… at, at, at a, at a small level, a micro level of a particular system. Now, ARPA-H came out, uh, and ARPA is the Advanced Research Projects Agency, and the H is for health. There's DARPA, which are defense-related ar- uh, Advanced Research Projects Agency. There's probably others, I'm sorry, I don't know. But ARPA-H is specifically around health, and this is-- these are projects that are, I'm, I'm sorry for the term, but moonshots, or they are somewhat risky, high risk, high reward type, uh, projects funded by the, the federal government.

7:32Uh, certainly, uh, could be funded by multiple organizations. The one I'm gonna talk about here is funded by the US Department of Health and Human Services So ARPA-H put out a call that they referred to as the CIRCLE program. Now, this project specifically is looking at digital twins for the treatment of patients in the intensive care unit, in the ICU. And they put out a call to build a digital twin of a patient in the ICU and test it for its ability to assign immune modulation therapies for these patients.

8:22The overall goal of this project, the moonshot, if you will, is can we reduce ICU stay by 25% or more? Tied to that, save lives, improve the outcome of patients in the ICU, and they specifically talk about a 25% threshold. So they, the, uh, and this, this-- They announced last week the rewards. They went through an RFI and an RFP, and they re- announced the rewards, uh, September 22nd. You can go on and find the, the, um, uh, the announcements of this.

9:03And they funded it up for $145 million total over five years. So this is a big project, a big advance project to, to see whether digital twins of patients' immune system Can these strategies improve outcomes? So what does that mean specifically? What they're looking to do in Circle is to build a digital twin of a patient's immune system, and then when a patient is in the ICU, they would ask the digital twin, would any of these, say, 15 therapies, steroids, tocilizumab, other immune modulation therapies, and IL-23, other things, most of these are already approved, and we understand what these therapies do to a human That the digital twin would have a particular human comes in and has biological characteristics that would be measured at baseline and then serially through their stay.

10:19And you would ask the digital twin, "Should we give a therapy? Should we change our therapy? How should we treat that patient?" You can imagine a patient, and I, I'm not an MD, so I'm a statistician, so I, I apologize if I'm getting things wrong. But you could imagine a patient has very high CRP levels or something within their system that puts them at risk for potentially advancing to sepsis or, or, or s- or bad outcomes. And the digital twin looks at the 15 different potential therapies and says, "Oh, if you give this therapy to that patient based on the digital twin, it thinks their CRP is gonna be lower, and they're much less at risk for bad outcomes, and they're going to have better clinical outcomes."

11:16And the digital twin says, "This patient will not, uh, uh, g- come down with sepsis," or, "They will not have downstream sequelae that mean they're gonna have worse outcomes." And that's all within this digital twin of the immune system. Now, they're gonna run a clinical trial to see whether that's true. First, they're gonna do in silico simulation, and they're gonna look at multiple patients and say, "Aha, if we would've intervened, could we have had this 25% reduction?"

11:52Entirely in silico, if the digital twin is correct, does it have the potential to reduce ICU stay by 25%? Now, they would then hope to test this in the last couple years of the project, where you could imagine a clinical trial that randomizes a patient to the digital twin care or standard clinical care of patients, uh, uh, w- the usual care. Then we can measure specifically the randomization to the digital twin. Does it actually change care by this 25%?

12:34So a lot of things going on there within that. But you can see within CIRCLE, that type of digital twin is a model of the system Now, you can see within clinical trials, I come at this as a clinical trialist. I am not an expert in creating a digital model of the s- uh, a cardiovascular system or an immune inflammation system for a patient in the ICU. Imagine a patient at risk at sepsis, a patient that has had an MI, a patient that has had other ARDS, o- other things that you might be interested in modeling their system within that.

13:26I don't have any expertise in that, but how does it go within clinical trials? So I gave you a, an instance of how that could be used in a clinical trial. Now, there are ma- many other types of digital twins. I and we at Berry, we use digital twins all the time, but it's a different kind of digital twin. It's a digital twin of a clinical trial. So we build, and that's what Berry's been doing for 26 plus years, when we build a trial design, we build a digital twin of that.

14:01It models the entire system. It models patients being enrolled in the trial. It models the outcomes being observed, adaptations in the trial, whether it's response adaptive randomization, sample size selection, sample size re-estimation, enrichment. Whatever's happening in that clinical trial, we build a digital twin of that, and then we apply things to that digital twin. What would happen if enrollment changes to that real system? And we measure what would be the outcome.

14:37What would happen if this drug works in half the patients? What happens to the real system when we apply these thi- these, these perturbations or, or effects to the digital twin that we've built of the trial? We see how frequently does statistical significance happen. How frequently does the trial stop early? How frequently does each particular thing… And we measure that. So we build digital twins. It's a real digital twin of a, of a, of a particular system.

15:15Now, there are many other of these. I, I wear a WHOOP band. Uh, it's a wearable that measures various aspects. It measures my sleep, measures my exercise, measures my heart rate, measures other things about me, and interestingly, it gives me a digital age. It tells me how-- what my biological age is, and every week, based on what I do during that week, it tells me how my biological age has gone. For example, it might say, "You aged at 1.0 during this week," which is the, the average age of a week is of course 1.0.

15:59Or you average-- you aged only at 0.5 Now, a lot goes into that where it believes by measuring my biological parts, it has a system of the body. It, it is a, a digital model that it can tell me how much I aged during that week based on the measurements that it took during this week. Now, I have no idea how good this is. I don't rely on this in some sense that now because I only aged 0.5 during that week, I'm gonna live 0.5 weeks longer. I don't believe that, uh, but it's a, it's a helpful thing to me.

16:37Now, we wanna build digital twins that do that, and how do they interact in clinical trials? Now The most pure digital twin doesn't need a clinical trial. By definition, it doesn't need a clinical trial. What do I mean by that? If we were to build a really good system, a cardiovascular digital twin, in the sense of what a digital twin is, and we say, "What happens if we give a drug to a particular patient, Jane Doe?" We can l- we, we can measure what that drug does to its lipids, to its LP little A, to the weight, to all these things that… A, and a much finer detail of the cardiovascular system.

17:32It'll tell us, is that a good thing? What is the cardiovascular risk of that patient now over the next five years? And if we give a, a, a GLP-1 to a patient for six months and their weight goes down by 20%, say, and the, that digital twin model will say… And other things happen to the body, other biometrics change, other characteristics that the digital twin knows might say, "You have lowered your cardiovascular risk in the next five years from 8% to 3%," I'm making up the numbers.

18:13But that's right. The answer, it's correct. That digital twin is so good, you don't need a clinical trial. Now, you, need a good digital twin, uh, to, to that extent, but that's the, the, the interesting aspect about a digital twin at the system level, don't need clinical trials. That's somewhat the whole point to the digital twin. Now, of course, we're gonna wanna show that this works. Like in Circle, we might do one-to-one randomization to the digital twin or a different form of care to show the digital twin works.

18:57Now, we might also do trials where we enroll somebody, and the digital twin says they have a 4% risk in the next five years, and we enroll a trial to see, do they have a 4% risk over the next five years? To validate the digital twin in some particular ways. You can also imagine this being a feedback loop to the digital twin. Now, these digital twins are going to be built very likely on observational data, where they look to see somebody that has this particular characteristic, and then they look to see what was their cardiovascular risk.

19:40You can imagine we've got incredible, we, uh, the community has incredible data out there on GLP-1 trials, other heart failure trials, uh, it- trials on liver disease, MASH, many, many trials out there that we could build pretty good digital twins based on the characteristics that are collected. There's a Framingham risk model that you can go online and ask for your risk, and you can change one of the things you inputted, like your weight or, uh, are, are you, are you, uh, what's your cholesterol levels?

20:20Do you have high blood pressure? All those things you could change and see what it does to your risk. If that thing's a really good digital twin, it'll tell you what your cardiovascular risk is Okay. Now this whole idea of feedback loop. I was at a conference a month or so ago on reinforcement learning. So reinforcement learning, and I'm not an expert, and many of you are gonna know much more, causal modeling approaches. Uh, really interesting and trying to read more and more about this.

20:56And largely, these are models that might be used to train an algorithm to play chess or train, uh, uh, automated, uh, self-driving cars to, to, to drive well, not to hit things, to behave well, that this reinforcement modeling is using… I- is used. Now they-- there's two really important parts to this, what's called offline learning, where you have a large set of data and you run your causal models to train something based on this, and that's offline learning.

21:33And then there's the online learning part, where you're actively changing something, and you're seeing the behavior of it really to learn causal pathways. If we change Lp, and yes, I'm aware that several weeks ago, a Novartis trial that looked at preventing cardiovascular outcomes for subjects with high Lp, a secondary prevention trial, does it reduce it? And they announced that the trial was not positive. I don't know the results of it. I was not involved in it in any way.

22:12But it brings to question, is altering Lpsomething that has a causal effect on cardiovascular events? Now, it still might. Might be the wrong patients, too late, all kinds of interesting things to it. Now, that online learning could be fed back into the digital twin, so it learns much more that not only i- there's characteristic A and now you come in and you have characteristic B, is that a better cardiovascular risk? But does changing it from A to B change their outcome?

22:52Which of course, is the critical thing. And by the way, in the ICU, it's gonna be the incredibly critical thing. I don't think the Circle project's gonna work. I think patients in the ICU are much too complex. Now cardiovascular, I think we probably have really good models of it, that we understand this a lot better than the ICU. Now, eventually we will have incredible digital twins of cardiovascular systems and ICU. When is eventually? Uh, I, I don't, I don't know.

23:27Now, that's not necessarily my point to this discussion. My point is more how do clinical trials w- do this? Uh, what, what impact do these things have on clinical trials? Now, that, that was focusing on the sort of system level. What is a synthetic control So I'm gonna call a synthetic cr- control not system modeling. You could imagine a digital twin of a system level having John Doe come in, and it gives a prediction of what's gonna happen to that patient.

24:14A digital twin could do that. But there are many, many other things that I don't think it's appropriate to call them a digital twin. I think we should reserve digital twins for system-level modeling. Something that's a mathematical prediction of a patient, I don't think is a digital twin. I don't think we should call the Framingham Risk Model a digital twin. I think it's a mathematical prediction. We have a lot of models that do mathematical prediction, and they're not system-level detailed modeling.

24:55Now, this is a s-- uh, uh, think about it as a statistical model, a mathematical statistical model based on a, a relatively few covariates, 20, 10. I think of a digital twin that might have hundreds if not thousands of characteristics it's modeling. So based on that, can we give a mathematical prediction of feels, functions, survives? Now, um, um, there's-- what could we do with such a thing? The, the EU, and you can go on and look for, uh, announcements, the EU, um, talked about and qualified a framework for AI-generated prognostic digital twins, and specifically, this is a company, Unlearn.

25:54I don't have anything to do with Unlearn. But they got this qualified by the EU. It is a mathematical prediction. Now, a lot of it is proprietary, and they don't tell us, and they make reference to a deep learning model. I think they largely have datasets of demographics, and they're making predictions of a patient's, whether that's an Alzheimer's patient, an ALS patient, a cardiovascular patient, and they generate a single prediction of that patient.

26:28Now, what do they do with it? How might we use that in a clinical trial? That has been qualified to use as a covariate So if I run a trial that is one-to-one randomized active against control, for every patient that comes in, you use it to get a prediction of what the 18-month, uh, CDR sum of box is going to be for that patient. You use that as a covariate. You get the actual outcome in the trial, and you estimate a treatment effect where you put in a, a coefficient for that covariate that you estimate.

27:09So you might… Your, your model might be off a little bit, but it's equally adjusted in the two groups. Now, this is a good thing to do. If that model's a good prediction of the outcome, it's gonna reduce the standard deviation. It is not being used as a control. It is not a virtual control. So it doesn't fit within that framework, but it's a way in which it's used in trials. Now, if we use other covariates like baseline CDR, maybe some biomarkers, other things, you might do just as well as reducing the variability.

27:49So this is not a topic on whether that's a good thing to do or a necessary thing to do. But in some sense, it's not a virtual control. Now, r- let's remember, why do we do clinical trials? Why do we enroll a control group in a clinical trial? We enroll 100 patients on a particular therapy. We enroll a control group to understand the counterfactual. What would happen to those 100 had we not given them the treatment? That's what we'd like to know. So what do we do?

28:26We enroll 100 patients, and we don't give them the treatment. And a, a lot of stuff we do in trials is to try to get them to be the counterfactual of the 100 that took the treatment. We do stratification of baseline factors, so they have similar age and severity and all things we think matter. We also randomize so that their, the counterfactual doesn't come with it some systematic bias, that it's a good 100-patient counterfactual So that, that's why we do a randomized control.

29:07Now, what happens if we have a very, very good prediction of every one of those 100 patients? We have a statistical model that says this patient is likely to, to glide-- decline nine points over nine months in an ALS trial on the ALS FRS. Or this patient is, has over the next two years, they have a 9% chance of having a MACE event Without the treatment on a reference treatment standard of care, placebo, whatever it is. If we have that, we don't need a control group.

29:57We know the counterfactual. We have evidence of the counterfactual. You don't need a control. So there's a huge potential in clinical trials that these mathematical models, virtual controls could have an enormous impact on clinical trials. Now, for years, we-- Berry Consultants has been around 26 years and started this Bayesian thing. We've been doing this for 26 years. Now, the mathematical models we had may not have been fantastic. They were not patient-level models.

30:42So not uncommon, there's a previous trial that ran with that reference control group, a placebo. Somebody ran an ALS trial with 12 months exposure on a placebo. Uh, we're doing cardiovascular. They ran a placebo group in a five-year cardiovascular risk trial and saw a 7% MACE event rate at, at two years. That's on my slide, so let's go with that. 7% MACE e-event rate on two years for the control group. We never had patient-level data. We had a publication that told us that.

31:29That's a mathematical model that in this new trial I'm going to run, I'm gonna use that mathematical model as a 7% control rate. So if my treatment in that new trial showed a 3.5% event rate, we'd say, "Aha, we have a 50% reduction-- uh, ha-hazard ratio of 0.5." We reduced events by 50% because we have a mathematical model of what would have happened without the treatment Now, that estimate, that 7%, doesn't change depending on the patients enrolled in that trial.

32:10We hope they're similar. Now, you can't separate the fact that it came five years later or seven years later, the temporal differences. But that's all we had. Had we had the patient-level data, we would do different things. So that is a mathematical model. Now, at the end of the day, we're gonna judge how, how, how good are these models? Are the models good? And by the way, are they good enough that you don't need a randomized control? That's largely the huge question here.

32:46Now, that question of good enough depends on a lot of things. I'll, I'll, I'll come back to that What's changing in the world now-- Uh, by the way, going back to that, you don't have to accept necessarily on face value that that 7% is right. We've been designing trials that have been unbalanced three to one or four to one between treatment and control. Now, that 7%, we can learn with newly enrolled reference group controls, placebos. Do they have a 7% event rate?

33:28If it starts to look like that they are reproducing that 7%, we wanna use the mathematical model really strongly to enfor- to reinforce the control arm and make the four to one look like four to four. And we set up statistical methods that allow that to happen. If we're see 3.5% on the new placebo, we wanna discount that 7% and don't really wanna use it much because it appears it's not a good mathematical model. So we build clinical trials that have Bayesian methods, Bayesian hierarchical models, MAP priors, other things that use that to the extent they're predictive and the, the, the-- it's adaptive or dynamic in learning if it's predictive.

34:26Now, we might even change the randomization ratio if it's not predictive. If we're enrolling four to one and we're not predicting the controls very well in that setting, we might need to make this more like four to four in the trial. These-- We've been doing these for years and how that fits in with this question of virtual controls. Now, that's something that's done at a marginal level. We've always been doing that at a marginal level. Why? Because we never had the datasets.

34:59Twenty years ago, you couldn't get data in ALS trials, in Duchenne trials, cardiovascular trials. Data was all cordoned off. It was, it was proprietary. You couldn't go and get the data from that publication. It's very rare that those datasets were available. It's been something in my time in clinical trials, the-- my 26 years, it-- there's been an explosion in the availability of data, whether it's EHR data, quote unquote real world data, clinical trial controls, clinical trial actives, registries.

35:37There's been an explosion in the availability of data and patient-level data. So what changes when we have patient-level data What it is, is we now have a mathematical model that's participant-specific. It's not predicting everybody in the trial is 7%. That's what the old models-- And by the way, we still do those, uh, now. Um, but that's what that methodology assumes, is that every patient is 7%. Now, what we have is by having patient-level data, two different patients that come in will have a different prediction.

36:22One patient, John Doe, might have a 9% prediction based on their, their, their variables, covariates, age, risk, where Jane Doe has a 5% or a 3% risk. So every patient that comes in, you attach a risk to them. Patient-level stuff. What, what are those? Propensity matching is exactly this. If we were to match the three closest patients in a large database to John Doe, those three patients form a prediction for John Doe if they were taking that reference that those three patients were taking.

37:07And it might say 9% is the risk for John Doe based on propensity matching. That's a mathematical model. We might do a, a large-scale Cox model with a great deal of covariates, the Framingham risk model, that gives a five-year, two-year risk for this patient based on their covariates. Hot-deck imputation, where it is largely like propensity matching that you, you match, and you might even match it based on changes over time of a patient where you go get the five patients that look the closest to John Doe, and you use that as a control group.

37:49That's a mathematical prediction. That's a statistical model. I don't think that's a digital twin, but you might use that as a virtual control. Now, there's this whole scheme of deep learning, machine learning, AI, neural networks, super intelligence. M- I'll call those black box models that when John Doe comes in, you plug in the black box, and it gives a prediction for John Doe. And, you know, it's a statistical model i-in, in, in my, in, in my understanding of this.

38:26Now, what's really interesting about this is how do we use these in clinical trials? It's a lot different than the 7%. If you've got this, I think we're going the wrong route with this a little bit. I-- When that comes in, I don't want a single forecast of John Doe of what's gonna happen. I want a distribution over that. I want… What you can do, if you've got a black box model, give me a probability distribution It could be 10,000 imputed values from that estimate.

39:07I don't want the w-- a, a propensity single match. I want propensity-weighted probability distributions on that. Now, it brings about an interesting thing in a clinical trial. Suppose we're investigating a novel therapy, and we were using this virtual control, and we've got patient-level predictions. So John Doe comes in, and I've got a distribution on how they're gonna change. And let, let me change it a little bit to ALS, a forecast of what's gonna happen on their functional rating score mortality.

39:47I've got a probability distribution. And I've got 100 patients that I've treated, and I've got 100 distributions. That's a little bit different than a clinical trial where our, our normal thing is give me a control, give me 100 controls and 100 treatments. Now, what I could have is 10,000 x 100 controls, or every patient is evaluated within the probability distribution. Where did they fall? Did they fall at the median? Did they fall-- Wow, that patient fell at ninety-nine percentile.

40:23We do disease modeling where we look at how much have we changed the forecast of the patient. And the estimand is a disease slowing or an endpoint slowing or an endpoint effect where every patient gives us an estimate of that, and we evaluate those parameters. Or we build a parametric model around it. Or we could simulate ten thousand patients from the probability distribution, and our control group now is a hundred times ten thousand, and we can do more traditional things where we estimate hazard ratios.

41:01It's a whole new thing in clinical trials. It's bringing about a mechanism that we haven't used much but is much more powerful than something that doesn't give us an, uh, uh, that uncertainty. Why does that matter? One patient may come in and we have very good forecast of that patient. And then when they have their outcome, we can evaluate it relative to that distribution. Jane Doe might have huge variability where their outcome ends up very far from the predicted outcome.

41:39But in terms of standard deviation, it's not very far. The Z-score from it is not very far. That's hugely valuable in understanding did the treatment we give have an effect on the outcome So really important in a whole new science of comparing to a probability distribution rather than 100 controls. Now, we are still using, in the clinical trial, Bayesian hierarchical modeling to evaluate are these predictions good predictions? And again, we might do four to one and the ability to enroll patients on that reference control to validate, verify, measure are my-- is my statistical model predicting well?

42:35Hugely valuable. First of all, for everybody to accept this. If you know the model is good, you don't need this. But as we're creating these models in the virtual controls, imagine going to a regulator and saying, "You just have to trust that this model works." Or saying, "We're gonna enroll prospective controls and we're gonna evaluate does the model work, and you're gonna be able to see that when we say seven percent, it's seven percent or two percent or, or that."

43:06Now, it also brings out a really interesting thing. Do we try to measure, suppose, a bias in that? That the model's pretty good, but it tends to be off by a hazard ratio of .8 in both groups and adjust by that, and it allows us to have a strong control that's been temporally adjusted in the trial. Hugely valuable stuff. It's a way, by the way, to combine together perhaps causal modeling with online randomized learning. This is where the world is going.

43:44The potential impact of this is absolutely enormous. Might cut every trial in half, and it does more than that because if you have 100 versus 100 in a clinical trial to power a treatment, and instead you know the o-- the-- a counterfactual prediction of those patients, you might only need 70 of those patients because you have a much more reinforced control arm. Hundred versus 100 when you have the variability of the 100 controls, but it might be 70 versus 10,000 times 100, which is, you know, uh, uh, a very large number.

44:1810,000-- I can do 10,000 times 100. I think that's a million. Maybe I can do that. Um, that it-- you can reduce the 100 to 70. Cutting trials by s-- uh, enormous fractions across that is hugely valuable, uh, in that setting. Now, digital twins at the system level, even more valuable that we don't even need clinical trials, and we're gonna get there Um, and again, I don't know if that's 15… By, by the way, we're gonna slowly get there in different diseases.

45:09We have some of them now. We're gonna get closer and closer to that. Um, uh, is it 10 years, 20 years, 50 years? Imagine the unbelievable modeling we're gonna have of a natural history patient in many, many diseases. It's gonna be so good that our clinical trials have to change. Uh, by the way, if the clinical trial ecosystem doesn't change, we're gonna be passed up, and we're gonna be discarded, and it's gonna be Google, Amazon, Apple, uh, y- you know, many, many other organizations that are gonna own the data and the models that are just gonna pass by.

45:55And so this is an incredible way to work with that within our clinical trial ecosystem. Now, makes the job of a regulator incredibly hard. I, I, I… You know, and it's gonna be harder. But also, the ability to partner on approving better treatments, better patient outcomes for regulators, developers alike is, is really, really exciting, uh, going forward. So I, I sort of stratified things by population marginal level statistical models, patient level statistical models, digital twin system level models, and how that does.

46:41What does that do to our clinical trial ecosystem? I hope you're as excited about this as I am. I think the clinical trial science is gonna keep up. It's gonna be super exciting in this. So I, I appreciate you joining me for this discussion. It was a great discussion with all the stakeholders involved. I think there'll be many more of these within different diseases within clinical trial science. So I appreciate you joining me here, and until next time, we'll be here in the interim