#296: Avoiding Major Oopsies: Twyman's Law, Intuition, and Valuing Accuracy Over Precision

0:00[Tim Wilson]: Welcome to the Analytics Power Hour. [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. [Tim Wilson]: Hi everyone, welcome to the Analytics Power Hour. [Tim Wilson]: This is episode number 290, oh wait, no, wait, let me check, this is episode number 296. [Tim Wilson]: That was a close one. [Tim Wilson]: I mean, how embarrassing would that have been? [Tim Wilson]: We're all set to have a discussion about data accuracy and getting business partners [Tim Wilson]: to trust the data and I almost whiffed on something as simple as the episode number.

0:40[Tim Wilson]: Already though, perhaps I've undermined your trust in me, Tim Wilson, sitting in the host [Tim Wilson]: chair that is usually occupied by Michael Helbling. [Tim Wilson]: Perhaps I have. [Tim Wilson]: Let's ask my co-hosts. [Tim Wilson]: Moe Kiss from Canva, welcome to the show. [Tim Wilson]: What do you think? [Tim Wilson]: Did my little gaff destroy my credibility? [Moe Kiss]: Yeah, trust is completely lost. [Moe Kiss]: No, I'm joking, you nailed it.

1:02[Tim Wilson]: It's been lost, it was lost years ago with you, so, and Julie Hoyer, from further, have [Tim Wilson]: I already destroyed our audience's confidence in me? [Julie Hoyer]: What's a mistake among friends, right? [Julie Hoyer]: I'm just your pre-read. [Tim Wilson]: Okay, I like it. [Tim Wilson]: Well, that's the topic for this episode. [Tim Wilson]: I mean, sort of. [Tim Wilson]: All data can be digitized and an overwhelming amount of it is and digitized data breaks [Tim Wilson]: down into like discrete little ones and zeros somewhere down the chain.

1:33[Tim Wilson]: It should be cold and objective and to use it effectively, we need to get it to be as [Tim Wilson]: accurate and precise as possible, right? [Tim Wilson]: I mean, well, maybe. [Tim Wilson]: So joining us for a discussion on just that topic is Arik Friedman. [Tim Wilson]: Arik started his career as a software engineer, then took a turn and did a PhD in computer [Tim Wilson]: science focused on privacy-preserving data mining, which is a tongue twister, then went [Tim Wilson]: on to work a few years as a program manager for Microsoft R&D, then he popped back over [Tim Wilson]: to research for a while, and now he's been at Atlassian for over 10 years where he is [Tim Wilson]: currently a senior principal data scientist focused on product analytics.

2:16[Tim Wilson]: And today he is our guest. [Tim Wilson]: So welcome to the show, Arik. [Arik Friedman]: Thank you very much. [Arik Friedman]: Long time listener. [Arik Friedman]: Happy to be here. [Arik Friedman]: And I assure you, like all of these career moves made a lot of sense, like it, it made [Arik Friedman]: a lot of sense and at the time, it was a logical progression at the time. [Arik Friedman]: Absolutely. [Tim Wilson]: I went architecture, technical writing, marcom analytics.

2:39[Tim Wilson]: So I, I sympathize and I actually, I don't know that any of those made any sense. [Tim Wilson]: But that'll be something for me to ponder on my deathbed, but so Arik, you presented [Tim Wilson]: at the last measure camp in Sydney and in that presentation, I think you had a, a fun [Tim Wilson]: or maybe it's relatable at least, or maybe it's a tragic story about an active users [Tim Wilson]: metric that looked like almost too good. [Tim Wilson]: Like the results were brilliant.

3:09[Tim Wilson]: Everyone was thrilled, but you had this like nagging feeling that something was off and [Tim Wilson]: you couldn't really nail down the specific problem. [Tim Wilson]: So maybe that's a good way to sort of start the show. [Tim Wilson]: Cause I think it gets to something really human about our relationship with data. [Tim Wilson]: So maybe we'll kick things off by having this, having you walk us through kind of what happened [Tim Wilson]: there.

3:29[Arik Friedman]: Sure. [Arik Friedman]: So like this is a story from a while back, but you know, at the time the product team [Arik Friedman]: was working on a product feature, a big change in the user interface and a lot of investment [Arik Friedman]: went into that. [Arik Friedman]: So we put a lot of work on actually testing things, you know, A, B testing. [Arik Friedman]: I worked with another data scientist to make sure that we got things right.

3:55[Arik Friedman]: And I remember back then, you know, us standing in the boardroom with all the PMS and it's [Arik Friedman]: looked like really a big change. [Arik Friedman]: Like it moved the needle quite significantly. [Arik Friedman]: So that was, that was a big deal. [Tim Wilson]: Everybody were happy. [Tim Wilson]: And then as we went on and rolled out this feature, we saw what we expected to see. [Tim Wilson]: Active users went up. [Arik Friedman]: But then, you know, over time, I'm starting to develop this, you know, old feeling, you [Arik Friedman]: know, you look at data over time, you get a sense of how it looks, how growth looks [Tim Wilson]: like.

4:36[Arik Friedman]: And I was looking at the curse and they looked a bit suspicious, right? [Arik Friedman]: Like it's not supposed to go like that. [Arik Friedman]: And I started looking into that, okay? [Arik Friedman]: Because it felt a bit odd. [Arik Friedman]: And I remember, you know, trying to go down to specific trace logs, trying to find what [Arik Friedman]: went wrong, I found like everything checked. [Arik Friedman]: So you know, after putting some time into that, I said everything was good.

5:04[Arik Friedman]: We actually have a reason to believe that, you know, active users goes up, all is fine, [Arik Friedman]: right? [Arik Friedman]: I cannot say anything about this. [Arik Friedman]: I don't have a smoking gun. [Arik Friedman]: Fast forward, sometime later, another data scientist and a software engineer, of course, [Arik Friedman]: we keep looking into this and they actually found a bug. [Tim Wilson]: It turned out that because of a bug, it caused inflation of active usage monitoring.

5:30[Arik Friedman]: And that was an unpleasant surprise for the product team. [Arik Friedman]: So for me, it definitely caused a lot of thinking, you know, like, how come I didn't find that? [Tim Wilson]: What could I have done different? [Arik Friedman]: And that, you know, that causes a lot of reflection. [Tim Wilson]: Well, did you go tell them, like, immediately, abruptly, did you ease it out? [Tim Wilson]: Like how did you actually communicate, like, what's the rest of the story on that?

6:01[Tim Wilson]: Yeah, so I think that's probably the biggest mistake I made at the time because, you know, [Arik Friedman]: I didn't find an issue. [Arik Friedman]: And you know, I, at least back then, I thought, you know, we are the data people, we are the [Arik Friedman]: evidence people. [Arik Friedman]: And if we don't have the data to back up what we say, we should shut up. [Arik Friedman]: And what I learned from that is actually, you know, our intuition are, you know, that's [Arik Friedman]: part of our expert opinion.

6:31[Tim Wilson]: And we should sometimes just go with it. [Arik Friedman]: And I think there are a lot of things we can do ahead of time, you know, to prevent mistakes [Arik Friedman]: or to check things. [Arik Friedman]: But at least for me, like, for this story, like, when I look, you know, what could I've [Arik Friedman]: done different, I actually knew to do all the checks. [Arik Friedman]: And eventually, when your intuition is your last line of defense, and sometimes you just [Arik Friedman]: have to go with that.

6:58[Moe Kiss]: So sometimes, I don't know, I, firstly, I just want to say thank you. [Moe Kiss]: I'm really grateful that you're sharing, you know, straight off the bat, an experience [Moe Kiss]: of where you made a mistake that's incredibly humble and wonderful for folks to be able to [Moe Kiss]: learn from. [Moe Kiss]: So a very big thank you. [Moe Kiss]: And I suppose I just, I'm curious to understand this intuition piece, right? [Moe Kiss]: Like I feel we all have it.

7:19[Moe Kiss]: And I know you and I have had conversations previously about like when we're making decisions, [Moe Kiss]: you know, we have data, we have intuition, we have, you know, maybe previous experience, [Moe Kiss]: we have all these different factors and part of our role is helping pull those things together. [Moe Kiss]: But I'm curious to understand. [Moe Kiss]: So how has this changed how you would show up? [Moe Kiss]: Like, let's say this happens next time, you can't find a smoking gun, you can't find [Moe Kiss]: a data point, but you had your, you know, your intuition in your gut that's like something's [Moe Kiss]: not right here.

7:46[Moe Kiss]: Like, do you think this time you would say something and how would you frame it? [Arik Friedman]: Yeah. [Tim Wilson]: I think, first of all, it's about being more opinionated. [Arik Friedman]: And in the, like over time, that's part of my growth journey. [Arik Friedman]: I think in the past I would, I tended to be a bit more, you know, impartial, right? [Arik Friedman]: We let the data speak. [Arik Friedman]: Like we're just there to give the data its voice.

8:12[Arik Friedman]: And I think over time I feel a bit more confident, you know, to be opinionated, like I have my [Arik Friedman]: own opinion of things. [Arik Friedman]: And I think we're actually expected to be opinionated. [Arik Friedman]: So I think that's, that's one thing, just like be more confident in our own expertise. [Arik Friedman]: And then I think it's probably, I mean, the opinion on its own is not enough, right? [Arik Friedman]: Like a probably, even if we have just this intuition, we will still be expected to, you [Arik Friedman]: know, okay, dig into that.

8:43[Arik Friedman]: Okay, can you actually find the evidence, but I think it's a start of a conversation, [Arik Friedman]: right? [Tim Wilson]: Like this is what I think. [Tim Wilson]: Maybe the data looks a bit odd and we maybe want to dig a bit more into that. [Arik Friedman]: And then the business can decide, you know what, no, like we did all our checks and balances. [Arik Friedman]: It's good. [Arik Friedman]: We did our due diligence. [Tim Wilson]: Let's go on.

9:06[Arik Friedman]: Or they could say, you know, let's, let's take more time and actually give it more space [Tim Wilson]: to dig into that. [Arik Friedman]: And at the time I basically made my own decision and said, you know, I looked enough into that [Arik Friedman]: and I kept it with myself. [Arik Friedman]: And this is something where we need to be more open and just share what we think with [Arik Friedman]: a business partners about that. [Tim Wilson]: So I will sometimes find myself not doing this necessarily with a ton of structure and rigor, [Tim Wilson]: but trying to think what I expect to see before I actually look at the data, or sometimes [Tim Wilson]: asking if I really don't think I have context and my business partner says, I think it'll [Tim Wilson]: go up.

9:47[Tim Wilson]: I'm like, what is, like, what does up mean for this metric or what is, what would that [Tim Wilson]: mean? [Tim Wilson]: Like I don't give me, use it as an opportunity to try to get a little more context about their [Tim Wilson]: intuition about their business, like there's, there's some psychological trick or something [Tim Wilson]: where you, you sort of force yourself to set what you think you're going to see as opposed [Tim Wilson]: to waiting until you see it and then you immediately rationalize why it makes sense.

10:15[Tim Wilson]: Like I had an example from years ago where a product marketing manager came racing in [Tim Wilson]: because he had already told the whole company that this tiny little change he'd made to [Tim Wilson]: a webpage had like drastically driven the track traffic up, which made no sense. [Tim Wilson]: And that was one where I dug in and found that it was, it was a bot, there was a company [Tim Wilson]: that was selling us software and they pointed literally at his page to gather some data for [Tim Wilson]: the pitch in, in a few days.

10:47[Tim Wilson]: But it was one where it was like, he saw it, he was a good news, kind of like the active [Tim Wilson]: users. [Tim Wilson]: But if I'd gone back, if I'd known he was making that change, I would say, what do you [Tim Wilson]: realistically think this is going to do? [Tim Wilson]: Because if it goes way past that, maybe we want to, you know, apply a little bit of [Tim Wilson]: climate law or, you know, something to it. [Tim Wilson]: But I think there's, I don't know, I feel like that's a whole, it's a challenge, [Tim Wilson]: especially for people just entering a field or the space to have that intuition and to [Tim Wilson]: have the confidence on top of it.

11:23[Tim Wilson]: Like you said that a few times that it's like, as you get more experience, the more [Tim Wilson]: faith you have in your intuition and your conviction. [Tim Wilson]: And that's like, how do you, how do you develop that if other than letting time pass? [Arik Friedman]: I think a lot of this is getting to live a bit in your business partner's world, like [Arik Friedman]: getting to speak the language and, you know, getting to know what they care about, what [Arik Friedman]: they're after.

11:53[Arik Friedman]: I think that adds a lot of context. [Arik Friedman]: And for example, I remember at some point, you know, I started hearing PMs talk all [Tim Wilson]: the time about like JTBD, JTBD this, JTBD that, like jobs to be done. [Arik Friedman]: I had no idea what it was all about. [Arik Friedman]: And, you know, I started digging into that, reading a bit about that. [Arik Friedman]: And, and, you know, what I found, like, I didn't really get what they were talking [Arik Friedman]: about. So getting into the world, learning about that, I think helps create this [Arik Friedman]: common language and thinking about it from their perspective.

12:29[Arik Friedman]: So I think that definitely helps us, you know, to get, so to speak, like in their head. [Tim Wilson]: When that was one where you actually, if I, I had this happen with OKRs once where I [Tim Wilson]: was working with someone who was, and I knew what OKRs were, I'd lived in kind of [Tim Wilson]: an OKRs training, but it was like, oh, this client is really into it. [Tim Wilson]: I'm going to go get a book. [Tim Wilson]: And didn't you do that?

12:51[Tim Wilson]: Didn't you wind up going and like read, like whatever the JTBD Bible is? [Tim Wilson]: Like, OK, if we're going to use this, I want to understand it. [Tim Wilson]: I mean, you probably found out that they actually weren't using it correctly. [Tim Wilson]: It's like Agile or any other number of things that use the acronym, but maybe [Tim Wilson]: aren't applying the process. [Arik Friedman]: Yeah. So I went on and I actually read about jobs to be done.

13:15[Arik Friedman]: And I, it was actually weird because like what I was reading and what I was [Arik Friedman]: hearing from the PMs were not exactly the same things. [Arik Friedman]: And at the time, like in our confluence, like internal wiki, I wrote a blog post. [Arik Friedman]: Hey, like, are we doing JTBD wrong? [Tim Wilson]: And it was a bit of a clickbait, but it started a conversation [Arik Friedman]: because I mean, then I started like through the conversation with them.

13:40[Arik Friedman]: I started to find out, oh, actually there are different kinds of definitions [Arik Friedman]: of JTBD out there and having this conversation and trying to understand, [Arik Friedman]: you know, what they really mean when they talk about that, that really helped. [Arik Friedman]: And I did it from the perspective of, you know, how it can be more useful [Arik Friedman]: as a data scientist and what is it that they actually after when they talk [Arik Friedman]: about understanding the JTBD.

14:03[Arik Friedman]: So I think it was first, you know, just to catch up with them, but also to see, [Arik Friedman]: you know, how can I answer their questions and how can I better [Arik Friedman]: understand the question and, you know, improve myself as a data scientist [Tim Wilson]: that helps them. Michael, I have news. [Tim Wilson]: The AI analyst is emerging. [Tim Wilson]: Oh, that's big coming from the quintessential analyst. [Tim Wilson]: What do you mean, like a cryptid?

14:32[Tim Wilson]: Well, I mean, more like a more like a job promotion, but, you know, [Tim Wilson]: with more existential dread, you know, how foundation models created the AI [Tim Wilson]: engineer role. Yeah. Developers got all these cool titles and analysts got. [Tim Wilson]: Can you pull this by in today? Exactly. [Tim Wilson]: But now we're watching the birth of the AI analyst, someone who uses LOMs [Tim Wilson]: to multiply their capabilities without, you know, multiplying their stress rash.

14:57[Tim Wilson]: Nice. So an analyst, but with superpowers and fewer open tabs. [Tim Wilson]: Exactly. And the tool for that is Prism by ask-y.ai. [Tim Wilson]: Yeah. Prism is basically the interface between what I mean and the 900 steps [Tim Wilson]: I don't want to do. You ask in plain English and it helps you get from [Tim Wilson]: question to analysis really fast. [Tim Wilson]: And it doesn't forget your world. Prism's jam memory as J.A.M. [Tim Wilson]: their jam memory remembers your definition like what your means by conversion.

15:31[Tim Wilson]: Which table is the source of truth? [Tim Wilson]: And that July data, don't forget, it's, you know, cursed. [Tim Wilson]: Yes. Thank you. I'm tired of explaining our metrics like it's the extended [Tim Wilson]: MCU universe. Plus, you can capture like repeatable workflows as skills, [Tim Wilson]: portable expertise, like, you know, clean UTMs or fixed GA for channel [Tim Wilson]: grouping or standardized campaign naming and reuse those across different data [Tim Wilson]: sets. I like that because I do a lot of copy paste.

16:06[Tim Wilson]: This is can't continue type of feeling. [Tim Wilson]: And I feel like it would be nice to be like run skill, look smart, drink water. [Tim Wilson]: Nice. Want to become the AI analyst before your coworker does? [Tim Wilson]: Go to ask-y.ai and join the wait list. [Tim Wilson]: Yeah. And use code APH and ask why it pushes you to the top of the wait list. [Tim Wilson]: That's ask-y, the letter Y.ai and use code APH. [Tim Wilson]: Yeah. The AI analyst is here.

16:38[Tim Wilson]: This product is in beta, but you can get in on the ground floor. [Tim Wilson]: And it's coming for your busy work, not your job. [Moe Kiss]: So, you know, relax, chill out. [Moe Kiss]: So, Arik, you and I have spent a lot of time talking about this whole like [Moe Kiss]: accuracy versus precision playoff. [Moe Kiss]: And it's something that has just really resonated with me because I would say [Moe Kiss]: I always lean to kind of like the best possible answer with the time [Moe Kiss]: that we have in the business to help make a decision.

17:07[Moe Kiss]: Like, I suppose I would say I'm like quite pragmatic, [Moe Kiss]: but a lot of what I hear coming from data scientists is this number is wrong. [Moe Kiss]: This one's right. We can't do it this way. [Moe Kiss]: This, you know, we have to do it this way because this is the right way. [Moe Kiss]: And I guess I just wanted to hear your framing about this like playoff [Moe Kiss]: between accuracy and precision. [Moe Kiss]: And like, I don't know, you have such an elevated way of thinking about it.

17:31[Arik Friedman]: Yes. So I think like straight from primary school, [Arik Friedman]: the way we're taught things is that accurate and precision are like [Arik Friedman]: being accurate and precise are the same things, right? [Arik Friedman]: So at school, you get like this big equation. [Arik Friedman]: And like in the data science world, you can think about a business question, [Tim Wilson]: like, you know, what, what kind of features correlate with user satisfaction [Arik Friedman]: or something like that?

17:56[Arik Friedman]: Or how can we predict those kind of things in parallel? [Tim Wilson]: Like, like at school, you might get a question, like, you know, [Arik Friedman]: how much is like 2,124 times 3,926? [Arik Friedman]: Like you get this big equation. [Arik Friedman]: And what you're taught is that you need to go through the methods, right? [Arik Friedman]: Like you have to multiply this digit by that digit, carry over. [Arik Friedman]: If you get anything off, like any single digit is off, you lose all your marks.

18:25[Arik Friedman]: So if your answer is not precise, it's not accurate. [Arik Friedman]: You lose all your points. [Arik Friedman]: And I think we kind of like carry on this mindset with us into our jobs. [Arik Friedman]: And in the business domain, actually precision and accuracy are not the same [Arik Friedman]: thing because, you know, because, oh, like these big numbers, it's about, [Arik Friedman]: you know, 2k times 4k, it's about 8 million. [Arik Friedman]: And like about 8 million is accurate.

18:50[Arik Friedman]: It's not precise, but it's accurate. [Arik Friedman]: And I think that also, like when we get, you know, business questions, [Arik Friedman]: there are many ways we can go and approach and solve them. [Arik Friedman]: You know, we can throw them, I don't know, like hidden Markov models [Arik Friedman]: or, you know, clustering algorithms. [Arik Friedman]: So there's all this arsenal of like all these methodologies that we learned.

19:13[Arik Friedman]: And like, you know, that's kind of like even part of our pride in the craft, right? [Arik Friedman]: Like we want to show that we know to do all those things. [Arik Friedman]: But sometimes you can get a quite a good answer just by writing [Arik Friedman]: like a very simple SQL query. [Arik Friedman]: And, you know, it's maybe not the best answer, but it's good enough and it's faster. [Arik Friedman]: And there is this quote from John Tukey from like his, he had like this paper [Arik Friedman]: about the future of data analysis.

19:43[Tim Wilson]: And like this is from like the sixties. [Arik Friedman]: And he says, like far better and approximate answer to the right question, [Arik Friedman]: which is often vague, than an exact answer to the wrong question, [Arik Friedman]: which can always be made precise. [Tim Wilson]: And I think it really captures well that, you know, it's first of all, [Arik Friedman]: it's better to answer the right questions. [Arik Friedman]: And I think that part of our job is to help get to those right questions.

20:09[Arik Friedman]: But also when we answer, it's not just about precision. [Arik Friedman]: It's, you know, the answer can be accurate without being precise. [Tim Wilson]: And I think that's a way for us to be more fast and effective [Arik Friedman]: and be focused on business impact. [Tim Wilson]: But there's there's a there's a challenge, right? [Tim Wilson]: That that our business partners accessing tools and data, [Tim Wilson]: they're working with spreadsheets, they're working with dashboards [Tim Wilson]: because they're in this digital world and processing and multiplication is is is cheap.

20:44[Tim Wilson]: Like they are conditioned to see things that are precise. [Tim Wilson]: I mean, I love the like the the accuracy versus precision. [Tim Wilson]: Like a dark archery thing that sort of shows accurate the grid, [Tim Wilson]: the two by two of accurate and precise, accurate, not precise, precise, not accurate, [Tim Wilson]: not accurate, not precise. [Tim Wilson]: And it shows like the spread around the target. [Tim Wilson]: And I think that's good for us to be aware of.

21:13[Tim Wilson]: And part of it is, I mean, can that be handled a little bit with the language [Tim Wilson]: of saying, try to get yourself out of the boat of providing. [Tim Wilson]: Precise answers, like get comfortable with even if you have a precise answer, [Tim Wilson]: better to kind of back off the precision of it a little bit [Tim Wilson]: so that somebody isn't looking at you're not arming them with ways to. [Tim Wilson]: To find another person, it may be too reasonably accurate, [Tim Wilson]: but the precision means that the the first three digits kind of differ.

21:49[Tim Wilson]: Like there's there's a challenge with us understanding that [Tim Wilson]: and our business partners thinking that way. [Tim Wilson]: And the levers we can pull to try to help them think in language of. [Tim Wilson]: Of this is this is accurate, you know, it's about it's about eight million, [Tim Wilson]: you know, or whatever the answer is. [Tim Wilson]: And they're probably not going to object to that. [Tim Wilson]: They're not going to come back and say, what do you mean about a million?

22:16[Tim Wilson]: I need that down to the to the first digit, right? [Tim Wilson]: I mean, it's the final line to walk. [Arik Friedman]: Yeah, and I think it really depends on the context and the question being cast [Arik Friedman]: because in some contexts, like if you deal with financial data, [Arik Friedman]: you definitely want to be precise. [Arik Friedman]: It's important, but most like a lot of the business questions [Arik Friedman]: are more direct directional in nature.

22:38[Arik Friedman]: And when you deal with directional questions, it doesn't matter as much. [Arik Friedman]: You know, if you're precise and actually precision [Arik Friedman]: can kind of draw the attention to the wrong thing [Arik Friedman]: because you don't care about the position necessarily in these kind of questions. [Arik Friedman]: So this is about getting to the bottom of what [Arik Friedman]: are the decisions that they are about to take [Tim Wilson]: and what really matters for answering those decisions.

23:02[Moe Kiss]: So one of the things I wanted to like challenge and have a discussion on. [Moe Kiss]: So I'm trying to find a specific article. [Moe Kiss]: And of course, I can't. [Moe Kiss]: But there was some research that was done with pricing and Uber. [Moe Kiss]: And I often quote it when I'm counselling people on buying engagement rings. [Moe Kiss]: Buying a two-carat diamond ring is more expensive than a one point nine nine nine [Moe Kiss]: carat ring, right?

23:24[Moe Kiss]: Like we have this bias and heuristic to think that the two carat, [Moe Kiss]: the rounded number is better. So hold on. [Tim Wilson]: Just wait, wait a minute. [Tim Wilson]: How many people are you counselling on buying rings? [Tim Wilson]: Like, what is your world? [Moe Kiss]: I my team are quite young. [Moe Kiss]: And so a lot of them are going through that life stage. [Tim Wilson]: Those are just one on ones. [Tim Wilson]: They're like, hey, what's going on?

23:47[Tim Wilson]: I could really do some help. [Moe Kiss]: Anyway, OK, carry on. [Moe Kiss]: So also, I don't know who's buying your two-carat diamond ring. [Tim Wilson]: It's a it's a anyway. [Moe Kiss]: Is that that's a that's a big one? [Moe Kiss]: Three carat is quite big. [Moe Kiss]: Yes, that that's not for America, but for Australia. [Moe Kiss]: Americans tend to buy much bigger rings. [Moe Kiss]: Anyway, back to the point of the story. [Moe Kiss]: Well, and now the lab, you should see these rings.

24:12[Julie Hoyer]: People are thrown around. [Tim Wilson]: Sorry. You guys are living in a different world. [Moe Kiss]: But what I was going to back to my point, [Moe Kiss]: one of my concerns is like, I hear you and I agree with everything that you're [Moe Kiss]: saying, right? But especially, OK, the same thing happens [Moe Kiss]: when you're trying to counsel people on like a day right. [Moe Kiss]: Sometimes like people will intentionally [Moe Kiss]: like make a number, not a round number, [Moe Kiss]: because people are more inclined to believe it, that you've put more work [Moe Kiss]: into it, because we have this bias that say if you say $1,600 a day, [Moe Kiss]: that like it's not it's you you've just picked a number from the sky [Moe Kiss]: versus having put thought into it.

24:52[Moe Kiss]: So like I'm just trying to think about how that plays off with our discussion [Moe Kiss]: here about not necessarily always wanting that precision. [Moe Kiss]: So you're saying if we don't go precise enough, [Julie Hoyer]: they won't trust the number. [Moe Kiss]: Yes. So if we say, oh, the number is eight million versus eight million [Moe Kiss]: two thousand and ninety seven, will people just be like, [Moe Kiss]: is that bias going to come to life?

25:18[Moe Kiss]: Where they're like, oh, maybe that number is not right. [Moe Kiss]: And like I can kind of disregard it because it's come from nowhere. [Moe Kiss]: Or like, do you see what I'm saying about like, does the precision [Moe Kiss]: give any extra credibility or build more trust? [Arik Friedman]: Right. So in those cases, like precision sometimes is kind of like the signal [Arik Friedman]: that you did the work. Right. [Arik Friedman]: And again, I think these kind of things are context dependent because [Tim Wilson]: ideally, if you have established a good like [Arik Friedman]: trust trust relationship with your business partner, then they trust your [Arik Friedman]: judgment and said you made the call the right call about what's a good methodology [Arik Friedman]: to approach this.

26:01[Arik Friedman]: And in some cases, yeah, maybe precision does matter. [Arik Friedman]: Like maybe it's a signal that's important to them. [Arik Friedman]: And in that case, yeah, maybe you do need to put a bit more work [Tim Wilson]: and and give that kind of signal. [Arik Friedman]: So that definitely is context dependent. [Julie Hoyer]: And part of the context, I'm just thinking like, [Julie Hoyer]: Tim, like back to some of our discussions we've had with clients [Julie Hoyer]: when they have really low volume, right?

26:27[Julie Hoyer]: The difference of one is greater than if you have a large volume. [Julie Hoyer]: The difference of 10 is like minuscule or, you know, like you can kind of ignore it. [Julie Hoyer]: So I do feel like, too, it's like, are you speaking of a really small end? [Julie Hoyer]: Because then you don't want to round compared to when you're speaking large [Julie Hoyer]: volumes, you can round to eight million and it's probably OK. [Julie Hoyer]: The other thing with precision that I think is interesting as we get sucked [Julie Hoyer]: into a lot is when the stakeholders want to compare systems, [Julie Hoyer]: numbers from systems, which we know are not going to give you the exact same one.

27:05[Julie Hoyer]: Like directionally, they should be close enough. [Julie Hoyer]: But I do feel like that is a scenario where they really get stuck on. [Julie Hoyer]: Like, why is this one percent different? [Julie Hoyer]: And it's like, well, it's OK. [Tim Wilson]: It's like eight million. [Julie Hoyer]: It's all right. [Julie Hoyer]: And we're not talking dollars and you know, which one's your source of truth. [Julie Hoyer]: So we should be OK to continue to use, you know, the other [Julie Hoyer]: number in our decision making.

27:30[Julie Hoyer]: But that's really hard, I think, for stakeholders and even for analysts [Tim Wilson]: to navigate. But I think we tend to make, I mean, if it's like, [Tim Wilson]: you're going to err on one side or the other. [Tim Wilson]: I mean, I look at somebody makes a line chart to show the results [Tim Wilson]: and they label every single data point down to the first value. [Tim Wilson]: And that's not so like there's there there are choices made [Tim Wilson]: in the presentation of the information where I think you can say [Tim Wilson]: there's precision precision there.

27:57[Tim Wilson]: I'm showing a dashboard, but this is rounded to the nearest million [Tim Wilson]: or the nearest thousand. [Tim Wilson]: It's not I'm not giving you eight zero zero zero zero zero zero. [Tim Wilson]: I'm saying eight million, you know, or eight point one million. [Tim Wilson]: You know, give them a give them a decimal place. [Tim Wilson]: So I think there are choices. [Tim Wilson]: I mean, certainly are to your point that like the context [Tim Wilson]: and actually Julia point as well, like the context does matter.

28:24[Tim Wilson]: When does it matter? [Tim Wilson]: But I I feel like there's like the trap we can fall into on the [Tim Wilson]: analytics side is we have the precision precision might as well show it. [Tim Wilson]: You know, our P value is not only is a less than point oh five. [Tim Wilson]: It's every model spits out. [Tim Wilson]: It's the P values point oh oh oh one three four seven two. [Tim Wilson]: Like please don't put that, you know, just kind of paste.

28:49[Tim Wilson]: Yeah. P values less than point oh five. [Arik Friedman]: And Julia, I think, by the way, it's a great point [Arik Friedman]: because it matters a lot like why the numbers are different. [Arik Friedman]: And, you know, scenario one, the numbers are different. [Arik Friedman]: We have no idea why. OK, that's that's not a good place to be. [Arik Friedman]: And that's that's a place where you probably do want to ask, you know, [Arik Friedman]: questions why why do these things don't match versus a different scenario [Arik Friedman]: where, you know, we actually expect those numbers not to be perfectly the same [Arik Friedman]: because maybe we measure slightly different things or, you know, [Arik Friedman]: there are known reasons why they should mismatch.

29:23[Arik Friedman]: So I think usually it's more about the confidence. [Arik Friedman]: Like we know what's going what's going on here and team to your point. [Arik Friedman]: Yeah, like maybe if we think that this will just distract and raise, [Arik Friedman]: you know, questions that are not relevant, then yeah, maybe it's better off [Arik Friedman]: to reduce the position to just avoid that altogether. [Tim Wilson]: So I think the other I think business our business partners [Tim Wilson]: and then data teams get sucked into it when it's the different systems aren't [Tim Wilson]: going to reconcile and I think to your point, like you should understand [Tim Wilson]: not down to the perfect we can net out everything, but say these are the [Tim Wilson]: the big movers of the differences.

30:05[Tim Wilson]: A lot of times there's the ability to say over time, look, these move in the same [Tim Wilson]: direction, I watch companies say, well, we just need to pick the system of record [Tim Wilson]: for that metric, which feels like my the hairs on the back of my neck go up. [Tim Wilson]: Like that's a cool idea that somebody in the C-suite or one level down said, [Tim Wilson]: we have the solution. [Tim Wilson]: We just pick our system of record.

30:31[Tim Wilson]: But the reality is all those systems exist for different reasons [Tim Wilson]: and it gets back to context. [Tim Wilson]: So I think there's a part that says we need to put this to rest at some point [Tim Wilson]: by doing a little bit of an analysis of saying these differ. [Tim Wilson]: These are the main reasons they differ. [Tim Wilson]: Let us show you that they generally move in the same direction over time. [Tim Wilson]: And now we're just going to have our standard little footnote that these, [Tim Wilson]: you know, move differently.

31:01[Tim Wilson]: But I think also we're going to go look at the system that gives us the most [Tim Wilson]: what we need because it's often kind of like upstream system in some process [Tim Wilson]: has this data, has this data, hands off to another system, which goes [Tim Wilson]: downstream from that. [Tim Wilson]: So you have to pick the system where you can get the slice. [Tim Wilson]: I mean, I'm thinking in a CRM digital analytics CRM sales world, [Tim Wilson]: you can necessarily track the marketing channel all the way through to a sale.

31:32[Tim Wilson]: And in the middle, you've got something like a lead that the downstream ability [Tim Wilson]: to follow it comes out of one system. [Tim Wilson]: The upstream to follow it comes from another system and heading off and [Tim Wilson]: understanding, and maybe this gets back to your point earlier. [Tim Wilson]: You're like, you really need to be figuring out is the question [Tim Wilson]: being asked very clear and precise and let that drive everything as to where you [Tim Wilson]: look, what level of precision, how you pull it, what the intuition is.

32:03[Tim Wilson]: And now it just went on a bit of a tirade. [Julie Hoyer]: Well, no, but you bring up a point that, you know, maybe this is me still like [Julie Hoyer]: wrapping my head totally around the accuracy versus precision. [Julie Hoyer]: But I do keep thinking of what you were calling out like the target. [Julie Hoyer]: So we were talking about people being most comfortable with precision of [Julie Hoyer]: systems matching, but we just discussed, you know, thoroughly, like why that's, [Julie Hoyer]: we know it doesn't happen.

32:30[Julie Hoyer]: But to your point, Tim, if they are, if we can explain in at least broad strokes, [Julie Hoyer]: Arik, as you said, like why these things are differing. [Julie Hoyer]: And if we then can say, well, as long as we're pointing this at the right [Julie Hoyer]: like target, the right question, the right problem to be focused on, then [Julie Hoyer]: can we be comfortable with accuracy without precision between the tools? [Julie Hoyer]: Like, am I taking this way off?

32:59[Julie Hoyer]: But you're saying, you know, like, if they're always about 1% off, they're [Tim Wilson]: all hitting the right target, maybe not closely bunched together, that would [Julie Hoyer]: make people happy with accuracy and precision. [Arik Friedman]: Yeah. [Arik Friedman]: So I think that data science teams usually have a good opportunity to [Arik Friedman]: collaborate and aim to standardize measurements. [Arik Friedman]: And, you know, one example, like I've seen in the past where, you know, [Arik Friedman]: growth teams and product teams had different measurements of impact.

33:31[Tim Wilson]: And that can be very confusing because, you know, when one team says, oh, [Arik Friedman]: like we had that impact, and then the other team kind of like interprets it [Arik Friedman]: in their own language. [Arik Friedman]: So I think definitely for data scientists, we can be in a role where [Arik Friedman]: we, you know, coordinate between our teams to see, you know, can we actually [Arik Friedman]: standardize and measure things the same way and ensure that we actually get [Arik Friedman]: the same numbers all over the place.

33:59[Arik Friedman]: But as you said, like sometimes it's not possible, right? [Arik Friedman]: Sometimes, you know, there are actually very good reasons why things [Arik Friedman]: should be different. [Tim Wilson]: And maybe then we should just use different names for them. [Arik Friedman]: So it's very clear that, you know, when this team talks about something, [Arik Friedman]: it's not the same thing that the other team talks about. [Arik Friedman]: So I think we definitely have a role to play with this.

34:20[Tim Wilson]: So can we shift a little bit, because some of this talking about kind of [Tim Wilson]: getting trust in the data, and it feels like there is a, there's a big one [Tim Wilson]: we covered, which is like, we'll get people to trust in the accuracy. [Tim Wilson]: We've covered two things. [Tim Wilson]: If I'm going to mid-show recap, which we don't really do recaps and I don't [Tim Wilson]: know what the hell I'm doing. [Tim Wilson]: But we've got kind of the develop and trust your intuition.

34:46[Tim Wilson]: We've got the think about accuracy and precision differently. [Tim Wilson]: We haven't actually covered like, how do you not push bad info out? [Tim Wilson]: And the intuition is kind of like one hook into that, but I like the [Tim Wilson]: frenetic pace everybody in modern business is working in. [Tim Wilson]: There is a drive to get the task completed off my desk over the wall and [Tim Wilson]: into the hands, which just like begs for mistakes to be made.

35:23[Tim Wilson]: And the cost of making a mistake with the query, with the report is of damaging [Tim Wilson]: like trust. [Tim Wilson]: So like how, what other like ways do we have? [Tim Wilson]: We, we, we briefly referenced like time is law, which really goes to the [Tim Wilson]: intuition and doing checks, but like, what are other ways to maintain trust in [Tim Wilson]: the data from our business partners? [Tim Wilson]: How do we prevent ourselves from undermining trust?

35:57[Tim Wilson]: It's like a long ramble with a broad question. [Arik Friedman]: Yeah. [Tim Wilson]: I think first thing and something that I definitely would like to see people do [Arik Friedman]: more is just stop and ask, does this make sense? [Arik Friedman]: And I don't think people do that enough. [Arik Friedman]: And it's like at every stage, right? [Arik Friedman]: Like, you know, you look at the raw data, does it make sense? [Arik Friedman]: Is there anything off there?

36:25[Arik Friedman]: If you apply methodology, does it make sense in this business context? [Arik Friedman]: You get a certain outcome. [Arik Friedman]: Does this outcome make sense? [Arik Friedman]: So I think this is kind of like a first line of defense. [Arik Friedman]: You know, does it make sense? [Tim Wilson]: But aren't there, there are times, there are times where it makes sense, but it's [Tim Wilson]: actually wrong. [Tim Wilson]: Like if you're like, oh, it made sense.

36:44[Tim Wilson]: Therefore, I'm going to go full steam ahead. [Tim Wilson]: And then you find out later that. [Tim Wilson]: Oops. [Tim Wilson]: Just because it passed that hurdle, it was plausible, but. [Moe Kiss]: But you're making the best decision you have with the information available at the time. [Moe Kiss]: I feel like the dust doesn't make sense. [Moe Kiss]: Actually, it's just a like, it's a spot check, right? [Moe Kiss]: I actually think the thing that we need to do better at is.

37:10[Moe Kiss]: Does it make sense with our stakeholders? [Moe Kiss]: Like actually bringing them into some of those checkpoints versus kind of like, [Moe Kiss]: does it make sense to me individually? [Arik Friedman]: Absolutely. [Arik Friedman]: I mean, I guess like this is the first check that you need to do with yourself [Arik Friedman]: before you engage with others. [Arik Friedman]: And I think that's already a checkpoint that I think can probably spot some of the issues.

37:35[Arik Friedman]: Beyond that, yeah, I mean. [Arik Friedman]: Involving other people in this thought process. [Arik Friedman]: And like in general, you want to do peer review process for everything. [Arik Friedman]: And by the way, I'm coming from a computer science background. [Arik Friedman]: My brain is wired as a software engineer. [Arik Friedman]: So basically any time that I do anything that involves statistical modeling or math, [Arik Friedman]: I will always pull someone else that has like, you know, strong.

38:04[Arik Friedman]: Math jobs and statistical modeling, you know, hey, check by work. [Arik Friedman]: Like, did I actually apply the methodology correctly? [Arik Friedman]: Because I know that they think about this in a different way than I do. [Arik Friedman]: And they can spot things that I want. [Tim Wilson]: Will the question come up in that context? [Tim Wilson]: Because you said like, does the raw data make sense? [Tim Wilson]: And from like the EDA of saying, I've made my initial query.

38:30[Tim Wilson]: Now I've got a table of data that has say 25 columns and 150,000 rows. [Tim Wilson]: Do have I gone through and checked that one is 150,000 rows seem like the right [Tim Wilson]: number of rows? [Tim Wilson]: Like that's like a saying to like, is the, does the size of the data make sense? [Tim Wilson]: But going kind of checking for gaps, distributions, the values, like checking [Tim Wilson]: the columns, doing kind of a, doing that step of saying, this should be the right [Tim Wilson]: data, but have I actually done, have I checked all the values, all the variables [Tim Wilson]: to see if the distributions and values are reasonable?

39:17[Tim Wilson]: Like not just trust the query, because that adds time and it's probably a judgment. [Tim Wilson]: It depends on how solid and simple the SQL is. [Tim Wilson]: Does that make sense? [Tim Wilson]: And would somebody, if you're having, if you're running it by, is this the right [Tim Wilson]: model, how likely are they to say, did you double check that the data that you're [Tim Wilson]: feeding into this model is the data that you think it is?

39:43[Arik Friedman]: Yeah. [Arik Friedman]: And I think it's probably both those things. [Arik Friedman]: And it depends also if it's like, is this the first time I'm answering [Arik Friedman]: this kind of question or not? [Arik Friedman]: And I'll give us an example. [Arik Friedman]: Like there were cases, you know, at some point we worked on, say, internal [Arik Friedman]: developer effectiveness metrics, and we tried to understand, you know, [Arik Friedman]: flow of work and things like that.

40:07[Arik Friedman]: And this was the first time we went to calculate these kind of things. [Arik Friedman]: So, you know, I worked on this, there was another data scientist [Tim Wilson]: calculating the same metric in parallel, and we got completely different results. [Arik Friedman]: And the thing is that none of us was wrong. [Arik Friedman]: I mean, technically, none of us, we didn't make any mistakes or technical [Arik Friedman]: mistakes in the process, but we started working through the calculations step by [Arik Friedman]: step and really as we realized that, oh, we actually made different assumptions [Arik Friedman]: about what the data meant, and we made different assumptions about what we [Arik Friedman]: wanted to measure.

40:44[Arik Friedman]: So just by working through this process until the numbers matched, it allowed [Arik Friedman]: us to align on the assumption and it gave us the confidence that we were [Arik Friedman]: actually measuring the right thing. [Arik Friedman]: And so, like, it's definitely not something you will do with any metric [Arik Friedman]: or any calculation that you do, but definitely, like, the first time [Arik Friedman]: that you do something, it's good to have, like, this cross check.

41:10[Arik Friedman]: And once you got these right ones, okay, the next time you already know [Arik Friedman]: what you're doing and you don't need the same level of, you know, due diligence. [Tim Wilson]: So Julie's, like, lighting up on the, like, oh, yeah, yeah, assumptions. [Tim Wilson]: Like, that's another thing we haven't, we've talked about it [Tim Wilson]: in the past, but the discipline of documenting the assumptions that you [Tim Wilson]: made, and it may be, like, the list of assumptions I made were these 20.

41:37[Tim Wilson]: If I'm having another analyst review it, I'm going to show them all 20. [Tim Wilson]: When I present this to my business partner, maybe I even pre-read it, I'm [Tim Wilson]: like, these are the three or four that I want to make sure they understood [Tim Wilson]: that I made these assumptions. [Tim Wilson]: Like, we've talked about that with, as a practice of analytics of you are [Tim Wilson]: making these decisions and just, like, making an assumption and moving [Tim Wilson]: ahead is, like, pretty dangerous actually having in the process a place [Tim Wilson]: to write down those assumptions, one from a repeatability, somebody [Tim Wilson]: checking your work, but also to actually go back to the business and actually [Tim Wilson]: include that in the results.

42:18[Tim Wilson]: Not the exhaustive, here's how we made the sausage, but, hey, I just [Tim Wilson]: want you to know, going into this, and if that's your opening slide and they [Tim Wilson]: say, well, that's a bad assumption, then you actually screwed up because you [Tim Wilson]: should have done a pre-read with, hey, before we present this, these are [Tim Wilson]: like the assumptions we made and here's why we made them. [Tim Wilson]: But how much, how much do you think the business do that?

42:41[Moe Kiss]: That they actually, like, I feel like assumptions are included, they're often, [Moe Kiss]: like, at the bottom of the slide, in the text somewhere, and, like, the [Moe Kiss]: business just glosses over them. [Moe Kiss]: Like, I would love a business stakeholder to be like, and I've had that [Moe Kiss]: at previous companies where, like, I've had very senior folks, like the [Moe Kiss]: founders or CEO, be like, I don't agree with this assumption, like, let's, [Moe Kiss]: let's debate this, let's talk it out.

43:02[Moe Kiss]: But I don't see, and I would love to see more of that. [Tim Wilson]: I mean, I find it as a way to, you know, on the building trust to not have [Tim Wilson]: that in the delivery, but it's like, I mean, say there's an executive you're [Tim Wilson]: presenting to, but there's this business partner that you're actually [Tim Wilson]: collaborating with, it's a great way to throw in slack, say, hey, is it safe to [Tim Wilson]: assume this and have them say, yes, is it safe to assume that?

43:29[Tim Wilson]: And then I, I sometimes will put those assumptions, like, front and center. [Tim Wilson]: Like, I want to be confident that, and, but, but I probably wouldn't say I made [Tim Wilson]: the assumption, I would say we made the assumptions. [Tim Wilson]: And hey, just so you know, we assumed, you know, that, that holiday has no [Tim Wilson]: impact on our bottom line yet. [Tim Wilson]: No, that's, that would be a bad assumption to make.

43:53[Tim Wilson]: But I mean, it's not, it's not just like a CYA of like, throw this in the, in the [Tim Wilson]: footnotes, it's, I think it's as much of a show that you're trying to understand [Tim Wilson]: the context and the operating environment, which is building trust upstream, [Tim Wilson]: which also means that there will be more trust in the output. [Tim Wilson]: No? [Julie Hoyer]: Honestly, though, I feel like sometimes that type of documentation [Julie Hoyer]: and assumptions are like, definition stuff too.

44:21[Julie Hoyer]: I'm with Yuma, like it's refreshing and encouraging when you are stakeholders [Julie Hoyer]: key in on it and understand the value of it or want to debate it and can [Julie Hoyer]: call it out if it's wrong. [Julie Hoyer]: But like, honestly, I think a lot of times it is more useful for yourself or [Julie Hoyer]: others to replicate the work later on, or if you have to build on the current work. [Tim Wilson]: Like, I think sometimes if you're not kind of obsessed with like documenting [Julie Hoyer]: all of those things, I mean, I've been the analyst that's been given a [Julie Hoyer]: spreadsheet with random numbers, supposedly where they pulled it.

44:56[Julie Hoyer]: And I have no other definitions or assumptions. [Julie Hoyer]: And they're like, oh, recreate this and do it again for the new time period. [Julie Hoyer]: And you're like, F my life, I'm never going to get this. [Julie Hoyer]: And the person that made it originally is gone. [Julie Hoyer]: So all that to say, like, sometimes if it's not building trust directly [Julie Hoyer]: with stakeholders, I do think it helps like maybe the analytics team [Julie Hoyer]: broadly or the data scientist team broadly, more for the replicability is how I see it.

45:26[Arik Friedman]: Yeah, I think definitely like documenting the analysis or the assumptions. [Arik Friedman]: And like, that's probably not the document that you're going to eventually [Arik Friedman]: show as your final result, but it's probably good to always have like, [Arik Friedman]: here's the technical document with all the details, all the assumptions. [Arik Friedman]: So it's available there. [Arik Friedman]: And so you can reproduce this.

45:46[Arik Friedman]: And yeah, but like referring to most point, I think you have this [Arik Friedman]: conversation with your business partner is part of the process, right? [Arik Friedman]: Like you get a question. [Arik Friedman]: Like, here is how I understand it. [Arik Friedman]: This is my interpretation. [Arik Friedman]: This is how it translates to the methodology. [Arik Friedman]: So you definitely want to be sure that, you know, you're on the same page [Arik Friedman]: and you're actually answering the same question.

46:10[Julie Hoyer]: The other thing I think of too, like if it does get put in the footnotes [Tim Wilson]: of the document or this the presentation, like sent to your business stakeholders [Julie Hoyer]: mode, like I have spent so much time talking to people and on my team [Julie Hoyer]: or being so worried about myself, what happens when I give this deck to my [Julie Hoyer]: stakeholders and then they pass it to their stakeholders and they pass [Julie Hoyer]: it to their stakeholders and then they start asking follow up questions [Julie Hoyer]: and nobody knows to ask me for the clarification.

46:37[Julie Hoyer]: So like, honestly, I think my safeguard and anxiety is what makes me put [Julie Hoyer]: a lot in the final documentation in the appendix or in the footnotes. [Julie Hoyer]: Because I'm like, hey, if this gets passed like three degrees away from me, [Julie Hoyer]: which again, that's a win if your work gets passed that far along. [Moe Kiss]: I see this happen all the time with experiment results [Moe Kiss]: where like it, you know, a data scientist has pulled something together [Moe Kiss]: and then someone like summarizes it and someone summarizes it for a slack message [Moe Kiss]: and like these steps further and further away, which look, do I want stakeholders [Moe Kiss]: to be able to interpret experiments and communicate it?

47:14[Moe Kiss]: Absolutely. Lutely. [Moe Kiss]: But I think sometimes that's like just the the the understanding of like the [Moe Kiss]: core tradeoffs kind of gets watered down or there's like different incentives [Moe Kiss]: and it just gets really tricky. [Moe Kiss]: Arik, I know like you've done a lot of thinking about this, about like once [Moe Kiss]: the analysis is out, the experiment result is out and you kind of lose control [Moe Kiss]: of the narrative, so to speak, like how do you approach that?

47:41[Arik Friedman]: Yeah, so it definitely happens like sometimes you you get a chart [Arik Friedman]: or something that gets copy pasted and then someone puts it on a slide deck [Arik Friedman]: and it's completely out of context and you lost that. [Arik Friedman]: And like one one thing that I try is again, like if you have a documentation [Tim Wilson]: of your analysis, like you should definitely have a link to that. [Arik Friedman]: So if it's copy pasted, at least the link to the document is still there.

48:11[Arik Friedman]: And, you know, there's the footnote in the chart where you can highlight [Arik Friedman]: the main details, so it's definitely a question of a balance, right? [Arik Friedman]: Like you don't want to overload your visuals with all the assumptions [Arik Friedman]: and all the details, it's distracting, but at least have this kind of, [Arik Friedman]: you know, selective context or pointer to where the information is. [Arik Friedman]: So it kind of like travels together with the visuals.

48:39[Arik Friedman]: And, you know, sometimes, you know, you just lose control [Arik Friedman]: and you have nothing to do about that. [Arik Friedman]: But at least you can take some mitigating steps. [Arik Friedman]: So, you know, the evidence is there. [Arik Friedman]: Like if anyone is really like will want to find this information, [Arik Friedman]: there is a way for them to get there. [Arik Friedman]: So that's the least we can do to at least control that.

48:59[Tim Wilson]: Yeah. I mean, I think like pointing to like having the footnote of like, [Tim Wilson]: what was the what was the right with the source? [Tim Wilson]: I mean, you're well stated that it's like you don't want to overload it [Tim Wilson]: with all the assumptions, but you want to provide the breadcrumb to say [Tim Wilson]: and the more you can put that proximate to the chart. [Tim Wilson]: So they're not likely, I mean, that's pretty gross misbehavior [Tim Wilson]: if somebody says, oh, there are footnotes on this slide, [Tim Wilson]: but I'm going to pull just the chart and drop it in an email.

49:28[Tim Wilson]: There's a little bit of that's on them if it gets its own legs. [Tim Wilson]: If you're giving it like, hey, this is probably important reference [Tim Wilson]: information that should go along with it. [Tim Wilson]: I think that's good. [Tim Wilson]: But this, I feel like we could go on for [Tim Wilson]: multiple hours, but unfortunately, we need to head to rap [Tim Wilson]: or we will lose trust with our audience by having a two hour episode, [Tim Wilson]: which when you're called the analytics power hour, [Tim Wilson]: that would be a disconnect between data sources.

50:05[Tim Wilson]: No, that's a that's a stretch. [Tim Wilson]: But before we leave, the last thing we like to do on the show always [Tim Wilson]: is go around the horn and share a last call, [Tim Wilson]: something that might be of interest to our users. [Tim Wilson]: And Arc, you're our first guest. [Tim Wilson]: Do you have a last call or maybe a couple last calls you'd like to share? [Arik Friedman]: The first one is, I guess, a classic, the ICLR, like the introduction [Arik Friedman]: to statistical learning book, which is actually a free resource.

50:35[Arik Friedman]: And today, there's also a version for Python introduction [Arik Friedman]: to statistical learning in Python. [Arik Friedman]: And, you know, at times when, you know, AI sucks all the attention, [Arik Friedman]: I think that actually going back to the basics, to the foundations [Arik Friedman]: is, you know, just as important as ever, if not more so. [Arik Friedman]: And I know that at least from my experience, even just going over [Arik Friedman]: the first chapters of the books, you know, linear regression, [Arik Friedman]: it's like a very practical oriented book.

51:06[Arik Friedman]: So, yeah, that's that's a good recommendation that I usually provide. [Arik Friedman]: Like it landed for me. [Tim Wilson]: They have an R version and a Python version with, like, examples on it. [Arik Friedman]: Is that right? Yeah. [Arik Friedman]: So there are two versions of the book. [Arik Friedman]: The original one was with R. [Arik Friedman]: Oh, it's ISLR and ISL. [Tim Wilson]: And ISLB. [Tim Wilson]: Introduction to stuff.

51:23[Tim Wilson]: OK, gotcha. [Arik Friedman]: And about the available at www.startlearning.com. [Arik Friedman]: And like I noted, at least for me, it landed a lot of concepts [Arik Friedman]: that I didn't really get before. [Arik Friedman]: So I really recommend that. [Tim Wilson]: And if I'm allowed a second and a second call, [Arik Friedman]: which is maybe more related. [Arik Friedman]: So I saw recently an article from Hamel Kussein [Arik Friedman]: called Revenge of the Data Scientist, and it's actually [Arik Friedman]: both available as a YouTube talk.

51:55[Arik Friedman]: It's a Pi AI talk from March. [Arik Friedman]: And he also posted it as a Twitter article. [Arik Friedman]: And he actually talks about, I think it's about, [Arik Friedman]: specifically about mindsets, you know, we have the mindset that you go [Arik Friedman]: with to, you know, in this agent-first world. [Arik Friedman]: And I think that actually his point is that the data scientists [Arik Friedman]: approach, their mindset, their core skills, like a exploratory [Arik Friedman]: data analysis, metric design, model evaluation, all these things [Arik Friedman]: are as critical as ever and try to translate really well to this world.

52:29[Arik Friedman]: So I definitely recommend giving this a read. [Arik Friedman]: And also Hamel Kussein and Trash and Carr had a terrific [Arik Friedman]: episode about AI evils in Lenny's podcast. [Arik Friedman]: So that's a great resource as well. [Tim Wilson]: Awesome. So that was three, really, sort of. [Tim Wilson]: But those all sound amazing. [Tim Wilson]: I feel like I've read the first three chapters of like more. [Tim Wilson]: Those are the books I'm most likely to abandon, but still get a lot of value [Tim Wilson]: because it's like chapter four, where I start to I'm like, [Tim Wilson]: what, we're at the area under the curve.

52:59[Tim Wilson]: I'm like, oh, boy, oh, boy, the equations are getting pretty in-depth here. [Tim Wilson]: So I kind of want to check those out. [Tim Wilson]: Julie, what's your last call? [Julie Hoyer]: Well, my last call is accurate and precise. [Julie Hoyer]: And I'm feeling like maybe I have called this one out before. [Julie Hoyer]: But you know what? [Julie Hoyer]: It's such a good one that I'm just going to say it again. [Julie Hoyer]: It's called the Moenarch app.

53:26[Julie Hoyer]: And I've actually been using it now for quite a while for my own [Julie Hoyer]: family budgeting and financial like tool for the family spenders. [Julie Hoyer]: And it's really nice because you can connect everything directly into it. [Julie Hoyer]: So on top of having all your different credit cards, bank accounts, [Julie Hoyer]: you get really nice cash flow visuals. [Julie Hoyer]: You can recategorize like any of your expenses that come through.

53:56[Julie Hoyer]: And it will notify you like, hey, this is a recurring one or hey, [Julie Hoyer]: this is one that's not categorized. [Julie Hoyer]: You want to come in and quickly do it. [Julie Hoyer]: But budgeting is not easy to do on your own. [Julie Hoyer]: And I feel like this is the first app and different thing I've tried [Julie Hoyer]: that I can actually keep up with it. [Julie Hoyer]: I can quickly get to it. [Julie Hoyer]: You know, there's no delay like I spent.

54:15[Julie Hoyer]: It shows there by paycheck hits. [Julie Hoyer]: It shows there. [Julie Hoyer]: So it's been awesome. [Julie Hoyer]: You can also set a lot of your budgets and goals in the app as well. [Julie Hoyer]: So like for all your different categories, you can customize categories or not. [Julie Hoyer]: And then you can say, you know, I'm looking to spend X amount [Julie Hoyer]: each month in each category. [Julie Hoyer]: And it just, I feel like makes it a lot easier to actually live out [Julie Hoyer]: the financial plan and budget that we're going for.

54:41[Julie Hoyer]: So check it out if you're needing a tool. [Tim Wilson]: Have you figured out a way to turn off the turn off the net worth? [Tim Wilson]: The words because because there are periods where I want to not look at that [Tim Wilson]: when the yeah, I'm a monarch user. [Tim Wilson]: So I'm a fan. [Julie Hoyer]: But you are. [Julie Hoyer]: So you guys, it must be good if Tim's into it. [Julie Hoyer]: If he approves of the visuals and the tools.

55:03[Tim Wilson]: Yeah, judge, judge your trust in the source. [Tim Wilson]: Moe, what's your last call? [Moe Kiss]: I know we talk about her a lot. [Moe Kiss]: But Cassie, Cassie has her cough has a really great new. [Moe Kiss]: I mean, it's just her regular newsletter, but she's doing a couple [Moe Kiss]: of back to back newsletters on why the vibe coding will bite you. [Moe Kiss]: And here is exactly where and she's talked through a few like [Moe Kiss]: text scenarios of where things have gone really wrong.

55:33[Moe Kiss]: Like prod systems being completely white, things like that. [Moe Kiss]: So definitely go have a listen to that. [Moe Kiss]: But I think the thing like the real takeaway is that the stories [Moe Kiss]: are all about the same thing, which is just like misplaced trust [Moe Kiss]: and the speed at which it computes. [Moe Kiss]: And so it's not like the nobody got hacked. [Moe Kiss]: AI didn't go rogue. [Moe Kiss]: It's just like people let their guard down.

55:57[Moe Kiss]: And I think the thing that's been on my mind the most, which [Moe Kiss]: she's so she's just so damn articulate, but it's like expertise [Moe Kiss]: won't save you guardrails might. [Moe Kiss]: And I think guardrails is the topic that just keeps looping on my brain [Moe Kiss]: at the moment. [Moe Kiss]: And so, yeah, I've really enjoyed that newsletter and she's got a [Moe Kiss]: couple more coming out on the same topic. [Moe Kiss]: So check it out.

56:17[Tim Wilson]: I think that series like motivated me to dive into some new vibe [Tim Wilson]: coding project specifically. [Tim Wilson]: So far, I'm safe. [Tim Wilson]: I haven't crashed the podcast because that's where most of them happen. [Tim Wilson]: Starting to wonder if Riverside, our podcast recording app might [Tim Wilson]: actually be leaning a little too much on vibe coding is it's having [Tim Wilson]: some of the joys it's been bringing us of late.

56:38[Tim Wilson]: But so I'm going to pander to mo here a little bit because with [Tim Wilson]: the new season of choiceology that Katie milkman came out with. [Tim Wilson]: Um, one of the things she has is this checklist, which is this mapping of [Tim Wilson]: now she said she had a couple of undergraduate students do it. [Tim Wilson]: And that means it's like in a PDF for some insane reason, but she's [Tim Wilson]: basically gone through and looked at like, what are the different topics [Tim Wilson]: like attribution bias or Dunning Kruger or left digit bias, like the [Tim Wilson]: stuff that her episodes cover, this kind of reverses it.

57:16[Tim Wilson]: And it's a guide to broken down by these are all the topics of kind of [Tim Wilson]: cognitive biases, um, then which are the episodes you can actually listen to. [Tim Wilson]: So if you're not a, um, Katie milkman choiceology completionist, like I am, [Tim Wilson]: but you're like, I wonder if she's ever had an episode about, um, you know, [Tim Wilson]: mean reversion neglect, then this little guide will pop you to it. [Tim Wilson]: It is comically in a PDF, which I'm like, this is great.

57:46[Tim Wilson]: You guys really work to format this thing to one page. [Tim Wilson]: And she's about to start a new season, which means this, this of all things [Tim Wilson]: that should be a vibe coded website where it gets updated and maintained. [Tim Wilson]: This is it, but you know what the undergrads, they'll learn. [Tim Wilson]: That's how it was scoped academia. [Tim Wilson]: They're going to do their little thing. [Tim Wilson]: So, um, with that, um, our thanks so much for coming on.

58:14[Tim Wilson]: I feel like this is a case where we actually have like the show prep [Tim Wilson]: documents that have like even more gold in it that we were not able to get to. [Tim Wilson]: So, so we will have a lot of fun with that content our own ourselves. [Tim Wilson]: We may figure out how to bring you back for more of that. [Tim Wilson]: So thanks so much for coming on. [Arik Friedman]: Thank you very much. [Arik Friedman]: Awesome. [Tim Wilson]: So if you, uh, listeners, um, we love to hear from you.

58:38[Tim Wilson]: So if you'd love to have you leave a review or rating on whatever platform [Tim Wilson]: you listen to us on, if you'd like a free sticker of the podcast, uh, you [Tim Wilson]: can go to analytics hour.io and request one. [Tim Wilson]: I will, if you've gotten this far in the episode and you think, wow, that [Tim Wilson]: was a smooth conversation and these guys are professional, I will just call [Tim Wilson]: out now that we have dealt with, uh, tornado warning.

59:04[Tim Wilson]: They led to a power outage and two young children and a dog sheltering [Tim Wilson]: in place and with one co-host, we've dealt with a busted internet, um, that [Tim Wilson]: has been busted for the entire episode. [Tim Wilson]: But of course the repair team showed up for that during the episode. [Tim Wilson]: Um, and we've dealt with various cases of people dropping off and returning [Tim Wilson]: and not even realizing that we were still recording the show.

59:30[Tim Wilson]: So I encourage you to stick around for the outtakes because there [Tim Wilson]: might be some real doozies in those. [Tim Wilson]: Um, I would like to Tony, please leave this in. [Tim Wilson]: Thank you so much. [Tim Wilson]: Cause if you pulled this thing together, you're a good on your mate. [Tim Wilson]: Uh, so it's been fun. [Tim Wilson]: It's been a fun discussion. [Tim Wilson]: Uh, we've been at this for four hours to get this one hour pulled together.

59:52[Tim Wilson]: No, it hadn't been quite that bad, but we would love to hear from you. [Tim Wilson]: We'd love to hear if you thought that the edit was pretty smooth. [Tim Wilson]: If you've got your own thoughts on how to build, maintain, recover, trust, uh, [Tim Wilson]: you can reach out to us on LinkedIn. [Tim Wilson]: You can reach out to us on the measure slack. [Tim Wilson]: You can just send us an email at contact at analyticshour.io. [Tim Wilson]: So for Julie, for Moe, for all of the conspiring mother nature and construction [Tim Wilson]: projects that tried to not allow us to record this show about trust and accuracy [Tim Wilson]: and precision, keep analyzing.

1:00:34[Announcer]: Thanks for listening. [Announcer]: Let's keep the conversation going with your comments, suggestions and questions [Announcer]: on Twitter at, at analytics hour on the web at analyticshour.io, our LinkedIn [Tim Wilson]: group and the measure chat slack group music for the podcast by Josh Crowhurst. [Announcer]: Those smart guys wanted to fit in. [Announcer]: So they made up a term called analytics. [Announcer]: Analytics don't work. [Charles Barkley]: Do the analytics say go for it?

1:01:00[Charles Barkley]: No matter who's going for it. [Charles Barkley]: So if you and I want to feel the analytics, they go for it. [Charles Barkley]: It's the stupidest, laziest, lamest thing I've ever heard for reasoning in competition. [Charles Barkley]: Fuck my life. [Moe Kiss]: So the work that's being done out the front of my house. [Moe Kiss]: Knocked the NBN cable out. [Tim Wilson]: I'm a rap now. [Moe Kiss]: Oh, shit. [Moe Kiss]: Were we still recording?

1:01:28[Moe Kiss]: Oh, fuck my life. [Tim Wilson]: I had to smoke this, but I was, I was trying to make so many notes that I, uh, on other [Tim Wilson]: aspects that I doubted. [Moe Kiss]: There's a lot going on today. [Moe Kiss]: Oh, my God. [Moe Kiss]: Well, you gave me so much. [Tim Wilson]: I just, there was no writing. [Julie Hoyer]: Nobody has the heart to tell you if I'm like, well, [Julie Hoyer]: then the second one, one is like, so can I wrap? [Julie Hoyer]: Oh, guys.

1:02:02[Julie Hoyer]: All right, guys. [Arik Friedman]: I definitely want to take Tony in advance because like, yeah, like, that's probably [Arik Friedman]: some work to do here. [Arik Friedman]: Yeah. [Julie Hoyer]: My hot mic was the least of our concerns. [Julie Hoyer]: I don't know if you would have thought, Jesus. [Tim Wilson]: Yeah, but I mean, I think it's, I think it's going to come together well. [Tim Wilson]: And my mid-show recap was like me organizing my thoughts because I was like, [Moe Kiss]: I think we're actually hitting on some, I actually feel like Arik, we need like [Moe Kiss]: another two hours with you because there's so much stuff here.

1:02:37[Moe Kiss]: It's such gold. [Julie Hoyer]: Seriously. [Julie Hoyer]: Yeah. [Julie Hoyer]: You had so much in the show prep doc that I was like, oh, I want to talk about that. [Moe Kiss]: Oh, and this is why I was like, I knew that the two of you would really like [Tim Wilson]: Arik because like you're the same with the very good at prep and [Moe Kiss]: organization and all those things. [Moe Kiss]: I literally bought a new microphone and it's still.

1:03:01[Moe Kiss]: Why have I still got this shitty one that doesn't even have a proper stand [Moe Kiss]: and it still works great and everyone keeps buying all these fancy ones. [Moe Kiss]: It sucks. [Julie Hoyer]: I got to return my hundred dollar one, I guess, and try a twenty dollar one. [Julie Hoyer]: Maybe I need it to be less sensitive. [Tim Wilson]: I think the show might need to buy you an audio interface. [Tim Wilson]: I think it's I don't don't don't don't make any moves.

1:03:24[Tim Wilson]: Don't do anything drastic. [Moe Kiss]: Yeah, the audio interfaces and changing microphones. [Tim Wilson]: It is because it's moving where the preamp is. [Tim Wilson]: It moves it into its separate. [Tim Wilson]: It moves it from a little thing, crappy thing in the microphone. [Tim Wilson]: It then takes it out and puts it in a dedicated box. [Tim Wilson]: OK. No, this is so above my head. [Julie Hoyer]: I just want to buy a microphone that works.

1:03:59[Tim Wilson]: Rock flag and accuracy versus precision.