The Role of AI in Testing with Debbie O'Brien

0:01Speaker 1:How'd you like to listen to dot NetRocks with no ads? Easy? Become a patron for just five dollars a month. You get access to a private RSS feed where all the shows have no ads. Twenty dollars a month, we'll get you that and a special dot net Rocks patron mug. Sign up now at Patreon dot dot NetRocks dot com. Hey, welcome back to dot net rocks. I'm Carl Franklin at Amrichard Cabell. We've been doing this now for many, many years, over twenty It's been a while, and this is episode twenty eleven, twenty and eleven. Can you blame it?

0:49Speaker 2:I can? I also know this history bit is only fifteen shows away from ending, because I'm not doing the future.

0:55Speaker 1:No. All right, what happened in twenty eleven, let's talk about it. It was kind of a sad year for world events. Well some happy, some sad. The Arab spring was in full tilt. Governments fell in Tunisia, Egypt, Libya, and Yemen, and unrest spread throughout the region.

1:16Speaker 2:Syria, in theory started but it turned into a civil war.

1:19Speaker 1:Turned into a civil war, right, so this movement, you know, reshaped Middle Eastern politics and ultimately contributed to conflicts that continue today like Syria. But it was a pro democracy movement protests. That's why they call it the Arab spring. Oh Man, the Tohoku earthquake and tsunami magnitude nine point zero earthquake.

1:45Speaker 2:Yeah, the first televised tsunami ever, right, because it happened in the morning. The traffic helicopters were up and they literally just streamed the water coming out. Like you always wondered what tsunami looked like. We've had all those paintings and things like that, but here it was, and you know what, it's just not preaty and it's just water keeps coming and it just keeps coming.

2:05Speaker 1:Of course, Fukushima this is the same event, right, Yes, it's a nuclear accident, was a result of that snami.

2:12Speaker 2:It's normal for reactors to be shut down after an earthquake in case pipes are broken in things, so they shut down all the reactors, reactors one through four because there's actually six there. I've done this as a geek out, as you know, one through four need active cooling, so the generators all lit up because there's no electricity being made anymore to run those pumps to keep those reactors cool. The cooling takes a month, so logically they would probably power back up after they've done the evaluations. And then the tsunami came over top the tsunami wall flooded the entire area, including the generator rooms, which were in the basement instead of on the roofs of the building where they were recommended to be put. At that point, there's no more cooling going into three other reactors, one of them, number four, was shut down.

3:00Speaker 1:And of course, oh Sama bin Laden was killed by Navy seals in a raid on Abadabada. I love that name, Obadabad Abadabad. So, of course, ten years after the September eleventh attacks which he masterminded, the first Libyan civil war in the far of Mamarga Dafi, So that was a kind of a big deal.

3:26Speaker 2:Starts in February, overthrown by August, dead by October.

3:29Speaker 1:Yeah, it's tough to be a dictator. It's tough. The European sovereign debt crisis intensifies. Greece, Ireland, Portugal, Italy and Spain struggled with severe debt bailouts, austerity measures and fears over the Euro dominated the news. Occupy Wall Street this was an interesting one and I've forgotten all about this. Protesters challenged income inequality and the influence of large financial institutions, and it popularized the phrase the ninety nine percent right or the top one percent, and it's.

4:02Speaker 2:All fall out of the subprime debt crisis.

4:05Speaker 1:Yeah, of course, of course you mentioned Syria. Syria and uprising peaceful demonstrations against President Bashar Ala Sad escalated into armed conflict and became one of the defining geopolitical crises of the twenty first century.

4:22Speaker 2:Only went for fourteen years too, which is really a very long time.

4:25Speaker 1:No, just bad and it was I remember finding it hard to find who's the bad guy in this situation?

4:33Speaker 2:Right, and well, it turns out people don't faction up neatly into good and bad. Oddly enough, people are people.

4:40Speaker 1:It's funny, yeah, yeah, but all right. So christ Church, New Zealand earthquake.

4:48Speaker 2:Two earthquakes, one in February, one in June Boom Boom pulverizes a bunch of the bedrock that the city is built on. They ultimately will take them three years to redline certain neighborhoods. And then you can't live there anymore. Yeah, pay off the folks that owned homes in those areas. I have friends who went through that haven't been back.

5:08Speaker 1:In England, they were rioting after the police shooting of Mark Duggan or Dugan, violence spread across London and several other English cities, prompting a national debate over policing and inequality. Kim Jong Ill bit the dust and Kim Jong un assumed power his son. There you go. The Super Outbreak of twenty eleven, one of the largest and deadliest tornado outbreaks in US history, struck the Southeast and hundreds of tornadoes killed more than three hundred people and caused billions in damage. Doesn't sound like a lot three hundred people, because tornadoes strike a very narrow band, don't they.

5:50Speaker 2:Yeah, But most of the time tornadoes don't kill anyone, right, So it was incredible.

5:55Speaker 1:Yeah, It's not like a flood or something like that where masses of people die, but still horrible. And finally, the royal wedding of Prince William and Catherine Middleton. There you go.

6:06Speaker 2:Yeah, Wales. I would also throw in the youngest country in the world, South Housana is formed a way split off from Sudan in July.

6:14Speaker 1:Wow.

6:14Speaker 2:Still that's the latest country.

6:16Speaker 1:Wow.

6:17Speaker 3:And in October in all of the one good thing of the whole year.

6:22Speaker 2:Yeah, I know it's bad. Yeah, good things are relative concept. I'm not going to get into Sydnanese geopolitics, although very aware, like they play this game where the South Sudanese have the oil, but then the sudan the northern part, actually has the transport to take the oil out of the country, so they have to make a deal and they're not, so the fight still goes on there too. And in October, the population of the world crosses seven billion people.

6:47Speaker 3:Hey, that's positive.

6:48Speaker 1:Good for us.

6:49Speaker 2:That's a lot of people ready for to space. Give us some space news, Richard. This is the last year of the Space Shuttle. There are three flights, one for each vehicle, so Discovery flies to the space station in February and Endeavor flies in May. Each you're bringing payloads to the space station, and finally in July Atlantis, and that will be it for US manned space flight for nine years, not until twenty twenty when crew Dragon finally flies. So for that time, the only way to and from the space station is via Russian SOYUS. It's also the reason the geek outs exist because of a half hour rant on the phone with you where I screened about everything it was wrong about the Space Shuttle, and you said, dude, we should record that, and I thought it was a really bad idea and I was wrong.

7:36Speaker 1:YEA.

7:36Speaker 2:Turns out that was a good idea. Go other space news. In March, the Messenger spacecraft goes into orbit around Mercury, the only thing that's ever done that. Besides Marin or tend doing fly buys at Mercury back in the seventies, Messager stands alone. It's a very very difficult planet to get to, and it will spend four years around Mercury before they'll de liberately de orbited sort of impacts on Mercury, and in that time one of the things that'll detect is evidence of ice in polar craters. On in July, the Dawn Probe enters orbit around Vesta, the first time that had ever been done, best to the second largest asteroid in the asteroid belt, probably a planetary core ripped apart by Jupiter, much like Series and possibly Psyche. It'll spend fourteen months in orbit around don before flying off to Series as well. In August, the Juno spacecraft is launched on an Atlas five towards Jupiter. It'll be the first of the solar powered spacecraft to go to Jupiter, but it'll be years before it'll get there. And in November, the Mars Science Laboratory is launched towards Mars. You know it as Curiosity, but it only gets to be called that.

8:48Speaker 1:Over on the.

8:48Speaker 2:Computing side, this is the year Intel presents the idea of the new unit of Compute, or knuck. They also help finance Ultra books as a sort of counter to the MacBook Air, and so various companies start making these nice, slim, trim laptops that people like work very well. We get I E nine in March. In May, Microsoft acquires Skype for eight and a half billion dollars. In June they launch Office three sixty five, and in September we get the very first build conference where they show pre release versions of Windows eight, Server twenty twelve, talk about Studio twenty twelve, and we walk out of that show, you and I saying what if dot net doesn't rock? Because there was no dot net sessions at build, and so we started the Tablet Show as a hedge, which we'll make one hundred and forty episodes of over three years, and after those three years, we'll have concluded the dot net still rocks and roll the Tablet Show into the main show. Twenty eleven also the year that Netflix becomes an online streaming service in September. It's also the year that the consent decree with the federal government at Microsoft

9:59Speaker 2:ends finally after ten years, and in December, silver Light five, the very last version of silver Light ships. Not that many people care at this point after the craziness of twenty ten, and that's the one that had sixty four bit support and also supported GPUs and I nine. This is also the year that Attachment acquires Novel, and during that acquisition, the new leaders look at what's going on over on the mono side and tell our friend Miguel Diakaza, hey, you should lay everybody off, and when you're done, lay yourself off too, because this is all a waste of money.

10:36Speaker 2:And Miguel will tell everybody they're laid off, but please don't get another job, because we're we're going to try and fix this. Calls up his friend Nat Friedman and forms a new company called Zamorin, and they'll ultimately make a deal to take control of the software they built and become that company. In October is also the year that the first Apple iPhone ships was Sirie, built in Seria, originally a separate product that was an app you could install, but Apple acquired it and added in. It's also the year that Steve Jobs resigns in August and passes away on.

11:09Speaker 1:Yeah. Also in twenty eleven, Apple broke sales records they built. They pulled in the biggest profits to date. Yeah, and it's ironic that they lost it.

11:18Speaker 2:Became the most valuable company in the world to that moment.

11:20Speaker 1:Yeah, they wrot. They lost Steve Jobs the same time.

11:25Speaker 2:And last but not Elise andres Helsberg announces that he's leaving the sea Sharp team, handing it over to Match Targetson, who still shepherds it to this day. Doesn't really talk about what he's doing next. We'll later know that it's typescript. But if you recall that was a year I said, you know, I'm not sure who loves c sharp anymore except for Miguelda Kaza.

11:47Speaker 1:Oh. The irony.

11:48Speaker 2:They had something, isn't it? And that's all I got?

11:50Speaker 1:Friend? Yeah, all right, well, I guess it's time for me to roll the crazy music and talk about better NO framework. Awesome?

12:06Speaker 2:What you got this weeklyady?

12:08Speaker 1:I got an email from our old friend Brent Rector. Remember him? Oh wow, that's a while ago. Yeah, he wrote the original dot net obfuskater called Demeanor. Yeah, he's got some new AI things in it. And I said, well, I'll tell you what I'll do this. I'll announce this is a better NO framework. But give me a paragraph to read so you know I can be intelligent because I haven't used an obfuskater in a long long time. Yeah, no kidding, So this is what he says. Demeanor is the original dot net obfuskater I first shipted in nineteen ninety nine. It's a complete standalone tool. The full feature set runs from the command line and in your CI pipeline, no AI required. It's Audit tells you exactly what would break if you obfuskate and how to fix it before it touches a bite new and version six. You can also drive it from your AI assistant, ask what breaks if I obfuscate this write in Claude code or co pilot, cloud based LMS or a local one. But AI use is an optional convenience, not a requirement. No other dot net obfuscator does this. It's at wizusoftware dot com. Just search for demeanor. Awesome. So Brent, thank you for that.

13:22Speaker 1:And it's good to hear from your old friend, and I wish you a lot of luck. That's what I got today. Cool. Who's talking to us?

13:28Speaker 2:Richard Gravity comment off the show nineteen seventeen which you're back in twenty twenty four with our friend Debbie O'Brien. Maybe you've heard. Yeah, we're talking a little bit about play right, which we've done more than once. And Rob King had this comment where he said, top episode chaps. I can answer a few the questions raised because we were talking about some issues around testing, except for CSS inspection and working out where something came from. You have a couple of options to make your life easier. Firstly, the dev tools are really good at pointing out where our rule is being applied. Secondly, using a preprocess like SCSS or SAS helps you organize r CSS into hierarchical structure, which helps negate having to hunt and pack where rules coming from. You also employed source maps for development. You can link them through from a rule into dev tools and see the exact source Fellow came from. In my humble opinion, writing's raw CSS without a preprocessor. It's like handwriting HTML without an autocomplete or hints there. Next thing you do were taken electrons and organize them with chromium.

14:21Speaker 2:Tip tweezers, That's sorry, that was me. And for snapshot testing, Blazer b Unit and Verify are one hundred percent certified. God sends for making testing as painless as possible. It goes so far as to say I enjoy writing tests now, don't talk crazy talk rop come on. And as mentioned before, once you've written a few tests, Copilot can pick up and write even more while I haven't gotten to doing. Image snapshots is available in the Verify framework, although the image Magic library works also. Yeah, and we ended up doing shows about Verified because it's that cool.

14:54Speaker 1:And it's also written by guess who Simon Crop, oddly enough one of the most pro.

15:00Speaker 2:Well thanks for your greats comments there, Rob, and a copy of music go by. It's on its way to you. And if you'd like a copy of music cob buy. I write a comment on the website at dot NetRocks dot com or on the facebooks. We publish every show there and if you comment there and I read it on the show, we'll send you a copy of music co buy.

15:12Speaker 1:And if you can't do that and you want to just get music to code buy, you can go to music too coode buy dot net. It's twenty two tracks that are twenty five minutes long of music specifically scientifically designed to keep you in a state of flow while you write code or do anything else that requires focus. And with that, let's bring back Debi O'Brien. Hi, Debbie, let me read your bio. This is a shortened bio. You can get the full one on the website. With over fifteen years experience in front end development, Debi O'Brien has worked as a tech lead and consultant for many important clients with various technologies and often with a strong focus on performance. She has led teams both in house and remotely, as well as giving workshops and training. Debbie as many years of experience as a mentor for online learning platforms and open classrooms and as a teacher at View School as well as jam Stack Explorers, and she's a writer for Ultimate Courses. Debbie is a platform engineer, Applied AI at Zephyr Cloud, Google Developer Expert in Web Technologies, Next Ambassador and also a former Microsoft Most Valuable Professional

16:23Speaker 1:and Developer Technologies Media Developer Expert and get Hubstar alumni. And I just want to say, why such a slacker, Richard? Why do we have to have slackers on.

16:35Speaker 3:The YouTube one? I meet the YouTube one to just like tick all the boxes and I'm just not there yet.

16:42Speaker 1:Well, welcome back, author of playwright, which I don't think was in that bio, but I.

16:48Speaker 3:Don't think I'm the author the face let's say the face.

16:51Speaker 1:That's right. Yeah, you didn't write it, but you have evangelized it into.

16:55Speaker 2:Yes, although I mean and you still are, but you're not at Microsoft anymore.

16:59Speaker 3:No, they let me go terrible. It's not with that, I know, thinking is that craziness?

17:07Speaker 2:So what are you doing?

17:08Speaker 3:Funnily enough, I'm trying to do something different and I'm instead I'm building AI agents for playwright specialists. So because at the end of the day, playwright is just very cool.

17:19Speaker 2:Yeah, Well, in the last show we did. We were talking about the role of agents with Playwright. So I think you came into something that was incredibly important. I'm not surprised that whoever you talked to is like, could you do that for us to please?

17:31Speaker 3:Yeah, And I think at that moment it was like it's still companies are still trying to figure out like should we be doing the AI kind of thing, Like it was a you know, scared of jumping down that channel because then you might be afraid that we're losing the product. And it's like, no, the product needs AI because we at the end of the day like everything is a little bit better at AI.

17:52Speaker 1:I have something to I have a story for a debbie. And that is after the last time we talked, Jeff Fritz from Microsoft and I started a YouTube podcast called Code It with AI and one of the first demos we did was use get a copilot with Playwright to create basically a user manual for this software for this app that we had built or he had built. Yeah, and he just let it go and it got snapshots with Playwright of you know, images, and put that in the documentation and created a user manual. It was that was mind blowing.

18:36Speaker 3:I think people don't understand it's not everyone. So people do like how many things you can do with Playwright, right, it's not just for tasting, no, And I know, like we evangelize testing because that was the focus of the product. But there's like so much more you can do. And now at AI browser automation, yeah, I mean anything you can do with the browser anyone to automate you can you can do And like I'm actually using Playwright Overchrome devtol's protocol to test a Tory application, so it can basically, you know, open the application, click around, do things, but it's actually testing the Playwright AI specialists that I created in the platform that I'm the we're working on. So I'm using Playwright to test itself. It's kind of cool.

19:21Speaker 2:That's cool, all that sort of recursive behavior, which seems to be really common now with these tools.

19:27Speaker 1:We have to use the tool to test.

19:28Speaker 2:The tool, and they constrain the tool and understand what the heck the tools are.

19:32Speaker 1:Well, And the ultimate recursion is the current version of Claud is writing the next version of Claud.

19:38Speaker 2:Yes, from what I understand, I think that's been true for a while, which is why they also amplify the stupid Yeah, but it works works as a relative concept.

19:50Speaker 3:You're absolutely right.

19:56Speaker 2:But it's also i mean, obviously it's driven through the browser, but you're able to test a lot of different things with play right, you actually do have to do a whole ton of things.

20:04Speaker 3:Well, this is what I was This is what I'm saying, Like we're testing a Tory app which is not a browser. Yeah, yeah, right, So there are things you can do if you figure out the ways and put the tools to practice, you can you can do crazy things, like you know, test applications that are not browser based, which works. It needs it needs a little bit of and you lose a little bit of things, but it works. But yeah, another couple of cool things we did. We have a new product called the AI Platform, and we were kind of soft launching, right, we don't want to tell anything about it because we're launching it because it's really cool, but there's do too many books so trying to be perfect.

20:43Speaker 3:So like we're going to launch it next week to a couple of clients get on board and test it out and stuff, but we have no documentation and like, oh, you know, like engineers don't write any documentation, So how do you write documentation for a whole new platform that works? And like, we have obviously the source code, but it and I was like brand new to the company, so I'm like, I know nothing about this, So where do we get all the knowledge from? How do we do this?

21:08Speaker 3:And I build a skill that uses Playwright that basically opens the app and starts going through it and using the app like a user word, documenting its user stories, going through the source code, and then putting together a whole documentation and in four days. Now it was a lot of work, right, It wasn't just like go and do this, Like there's a lot of iteration. But when in four days I had I can't remember many pages, maybe twenty five pages of documentation plus screenshots, I think is pretty amazing.

21:35Speaker 1:Yeah.

21:35Speaker 2:I think it's also a great point that these days, with these tools, you're able to learn about somebody software really efficiently, and so then and in the process generate things that are useful for the whole company. So your process of getting up to speed with the company and its products actually benefits the company because you ultimately end up with a deeper understanding of what actually exists. Totally yeah, very cool. Things have changed, Yes, they have versure. It used to take weeks to get someone up. Yeah, now it just doesn't have to be that way.

22:11Speaker 3:Now you're just turning the deep end and you just got to talk to an agent and figure it out yourself, and if you ask the right questions, you get there quicker.

22:17Speaker 1:Yeah. Well that's that's it, isn't it. I mean, do we spend time with AI creating user manuals and tools or do we just let the user kind of talk TODAYI and figured out what they want when they want it.

22:28Speaker 3:This is a thing, right because we created the documentation. And then I'm like, nobody's going to read this documentation because nobody read documentation in the past. But llms will read documentation.

22:39Speaker 1:Yeah.

22:39Speaker 3:So if you have a we're using rs press for the website and it has an LM text file as well, so you can just copy that page link, trow it into your agent and go read this and tell me how to do this, or go ahead and do it yourself. Right, But the documentation is there for the people who still are on that path of I need to read documentation. But really everything should be in the platform and you should just be able to use the agent that's in the platform to say, can you go and create a workflow for me? Can you create a list of tasks for me? Can you check that task off? And that's what I test using play right, I basically use Goose as my agent and using playwright or CDP basically get it to do all that stuff just so kind of like ensure it works. But everyone in the platform should be just using agents and never ever ever need to go outside and switch context and go to a documentation website.

23:34Speaker 2:Yeah right, Well, because the other piece here would be you know, the agent will go read the documentation for you, but then you also have a CLI and an MCP, or should at least, But how do you divvy up those roles?

23:46Speaker 1:Like what do those.

23:46Speaker 2:Pieces do in this equation rather than maybe using the documentation here?

23:52Speaker 3:Well, you create skills, right, or what we do is in air plat room we create specialists, which are just kind of like agents with specific skills. Okay, So like there's an onboarding specialist and onboarding basically has the knowledge of our docks built into it, So then that agent just can read our docks because it has already kind of saved into its memory, and so it can reference it and then just answer those questions. But it couldn't answer your question about the playwright docks because it doesn't know what playwright is, right, It only knows ours. So you've got to like create different kind of mini mini agents or specialists we call, and then like give them specific tasks because otherwise one super agent it's just too much, right, it doesn't know it.

24:29Speaker 2:Yes, yeah, so the constraints are really important. You know what it's knowledge based is, so you have some confidence that it's returning information based on that knowledge based rather than something else.

24:39Speaker 3:Yes, exactly, Yeah, and.

24:42Speaker 1:It does seem to be.

24:43Speaker 2:You know, we keep we keep hearing the mythology of the generalized intelligence, and every time we find success with this by being highly specialized.

24:52Speaker 1:Yes, do you do you know of or use squad?

24:58Speaker 3:I know of, but haven't used it.

25:00Speaker 1:Yes, it's the same kind of multiple specialist idea, the multi agent thing where you have one ring leader and it delegates tasks to the different specialists depending on what they do, which is.

25:13Speaker 3:Kind of like how you would have worked right in a company, like you know, you were the dotnet specialist, I was the front end specialist and like you just separate the task flow and everyone go and do this specific task that they're specialized in.

25:25Speaker 1:Right yep. I give you an interface, you go to it. You don't even need to know what I'm doing behind the effort.

25:34Speaker 2:But also this sort of adversarial configuration between the agents where one of them is writing code and the other one is testing it, and there's probably a PM in the equation making sure the feature set lines up, and then that that iterates within itself, like don't bother me until the code is complete enough that the PM agent sides off, and that it's accurate enough that the test agent sides off.

25:58Speaker 1:And then the scribe writes the doc documentation and there you go.

26:01Speaker 3:Yeah, yeah, And then where does the human come into the loop.

26:04Speaker 2:And the part of all this you're you're the supervisor and all the.

26:09Speaker 1:Tester make sure that they didn't write stupid code and stupid English or whatever the language is. Make sure it works, and then check it in. I never allow the agents to check in my work. I always tell them right up front, that's my job. Don't check in anything.

26:26Speaker 2:Yeah, interesting sets of loops.

26:29Speaker 3:Now with these tools, it's all loops. We've changed the terminology every every it was gonna say every year. It was like every couple of months we have like a new from a vibe to a prompt to now we're in looping. We're looping land, which is really just a workflow, and then like and then just well this bash loop was the big thing, and then bash loop. That's just loop, right, But yeah, loops are very important for ensuring that the agent doesn't have too much to do and go a bit crazy, which is probably like what happens when the human has too much to do. They get a bit tired, and then their work gets sloppy, so you've got to like give them some sleep and then loop. So the age is not sleeping, but it's kind of like, right, you've done that task and you can rest your brain and I can go to the next one.

27:16Speaker 2:Keep the context under control, right exactly. We should start using that term then, because I would always call that developers thrashing, Right, they've gotten a little too tired and they're hammering away at Covida not getting any results crashing. I agree with you that I definitely worked a problem yesterday on an automation I was working on where I realized we were actually getting worse, like, all right, I'm going to stop. This is close enough for now. I'll come back to this later. Go dump your context because reality is that agent was thrashing.

27:47Speaker 1:Yep, I've experienced that too.

27:49Speaker 3:We all have that.

27:50Speaker 1:When that happens, I usually hit control see and say, hey, do you want me to run this and set a brake point somewhere and tell you what the variable is because I'm here, I can help. I can help. Yeah.

28:03Speaker 2:Now you make me think in terms of now, do I want a monitoring agent for each of these agents. It's watching its progress and when it sees the curve decline, take action against it.

28:13Speaker 3:Yeah, because sometimes it goes off for one, doesn't it, And you've got to.

28:16Speaker 1:Like, I got to do something in this process though. I mean, I'm perfectly willing to be that person with the like you.

28:23Speaker 3:Know, right, But yeah, it's it's changing how we're working. But then, one question I have for you is how do you see teams working together? Because we're all working individually really at the moment. Now we've kind of separated away a bit from working together and we're just working with our agent, getting our agent to do everything. But it's just one person and one agent and then you're one person. Richard you're one agent, and Carl you'r one person.

28:53Speaker 3:You're an agent. Where does the team collaboration come into things?

28:56Speaker 1:For me? I do work with a team, but I'm usually assigned one task at a time, and the other people on the team are working on other aspects of the application or other applications. So it's just that we're not all working on the same thing because we have now just power to go down and into a particular domain and just make it right or add features ourselves, rather than with a team of people. So the teams are still there, but now they can be assigned different domains, I.

29:32Speaker 3:Think, which means they're kind of on their own right.

29:34Speaker 2:Yeah, But it strikes me that the change here is horizontal organization versus vertical organization. You used to be a back end developer or a front end developer, that sort of thing doesn't mean much anymore. Now you're going to own this segment of this application from front to bath. So you're now vertically oriented, supervising a set of tools to get to the deliverables for that piece, and there can be several of you working like that. All of those parts will ultimately have to interplay.

29:59Speaker 1:With each You might have a database guy who's the only database guy or gal, and so everybody needs to lean on them once in a while. But that's business as usual.

30:11Speaker 2:I think over on running as we're talking about a lot more automation for database administration and operations too. Because the tools are doing well.

30:19Speaker 1:They really are wish to take a break. Yeah, let's do that. We'll be right back after these very important messages.

30:28Speaker 2:And we're back. It's not a rock's amateur Campbell. Let's Carl Franklin, Yo, our friend w O'Brien about this weird new world where testing is still important and good tools are still important, but how we get the test together has changed.

30:42Speaker 3:Yeah, and do you know what as well, this is a big problem right now because I remember, like when I first started working for play right, the conversations and the talks I was giving, we're all about testing is hard, Testing takes time. Companies don't want to invest in testing, and now all that's out the window. Now testing is easy, Testing is cheap. Anyone can write tests, right, the agent could just write it. Doesn't mean they're good, but you know, everything can be done, so there's no excuses for not having tests. The problem that I find right now is that companies are moving at a speed so fast that the tests are slowing them down, almost so they can't keep up with the tests because they're vibe coding so much that they're getting a lot of stuff out the door.

31:23Speaker 3:And then like the bottleneck is there's too many tests now to run. There's you want to get something shipped, you want to get something past CI, and like you know, it's everything's too fast. And then the human is the person who's slowing everything down trying to manually review and before the merge and manually check that those tests are okay and everything's working. I think that's the hardest part.

31:47Speaker 2:Yeah, I've been working with organizations where the testing has to pass before it gets presented to the person. Anyway, the quality of tests is a really interesting question exactly. Generated We've seen the same thing with interns writing tests too, like that it's a test, yes, and it passes, yes, but does it actually validate anything important?

32:06Speaker 3:Yeah? Is it worth testing even Yeah? Because sometimes you could test something but maybe it's already covered in unit tests. You really need to put in your end to end tests as well, right, so it's a lot of there's a lot of things. That's why I'm working on this agent to try and like ensure you know, this is what you're responsible for. But you don't need to test everything. It's not important. It's not like we need to have tests that are good and that validates something very very important. Otherwise, we can just ask an agent to test everything in the in the app and it will come out with hundreds of tests and we're like, yeah, we have like one hundred percent coverage, right, is that what you want?

32:40Speaker 2:Well, and you can paralyze a lot of that. We did this extensively with large scale web development back in the day, where we would literally this is pre cloud, so that's how long time we'd harness multiple machines in automation, tests across them and run them all in parallel. And you can do it. I've done it in the cloud as well. It's just expensive and it's a little bit more complicated. But again, these tools will help you to paralyze tests if you want to do a ton of them.

33:03Speaker 2:The question is of the test particularly relevant, like are you really finding anything.

33:08Speaker 1:Out well, you know, And that's where I would be asking the tool to quantify the tests, and to give me a report about all the tests that are there. I mean, I can go look in the test harness and all that stuff, but you know, give me some documentation about this, and then look over the documentation, compare it to the application and see if there are any holes. Simple stuff.

33:28Speaker 2:All the best tests I've ever seen were the ones we wrote after a bug.

33:33Speaker 3:Yeah, yeah, of course, like.

33:34Speaker 2:After we had a failure in production or something like that. Like before you go fixing it, you write the test to catch it. Then you fix it so the test passes and everything works, and those all those tests became really valuable to the point where we tended to keep them separate from the unit tests and things like that, like these bug derived tests where we're much more robust, and there were things you didn't want to have happen again, so they were they're very protected class.

34:01Speaker 3:Speaking of bugs, I wrote a skill that uses player right obviously on CI to launch the application and use the application and find a bug. Once it finds when it creates an issue, nice files that issue. And then there's another skill that goes along and finds an open issue, reproduces over CDP Chrome devtos protocol takes a screenshot to show like, this is what it's not working, go to the heads and fix it as a screenshot of the application with the fix showing that it works, and the new test to show that it's now tested, and then like submits it. It's amazing. This is what I'm saying.

34:38Speaker 3:The bottleneck is the human, because the human's reading this and going is this really true? Like do we really you know, do we just press mergers this like really really a good fix? Is this correct? And it's like trying to persuade the human is the hardest part. But it's very very cool. So we only run it every hour because otherwise we'd be inundated with like so many books being found and fixed and one's sure of like, you know, merge one hundreds in confidence go. But yeah, it's cool.

35:03Speaker 2:Well, you talk about what's going on with security testing right now with mythos and such, where they're finding hundreds of vulnerabilities and recommending zero days as well potential zero days.

35:16Speaker 3:And how do they decide which ones to merge or when to merge or is it do you know what I mean, You've got to analyze that mergency. Is it really going to fix it or is it going to break something else?

35:24Speaker 1:Is this bug finding and fixing tool of yours available for people to use?

35:29Speaker 3:It's not actually because I don't. I should create something if people want it. I guess it's not difficult, of course not. It's just play right and a skill right, and that.

35:43Speaker 2:Tester mindset of when offered a.

35:46Speaker 3:Value, maybe the mindset, Yeah, do you.

35:49Speaker 2:Put a negative value in? Do you put a floating point value in? Do you put null in? Like just that you know, being evil user contrary? And what's all the values I could fire into the that would break things?

36:01Speaker 3:Yeah, it's totally all maybe like make sure you do not do that. You do not write things, do not fix the guardenails are very important.

36:09Speaker 2:Yeah, but you know, if you're going to make robust validation, you've got to try all the things that might end up in a field.

36:14Speaker 3:But it's never a silver bullet, right as in, like we run this every hour and ci it doesn't mean that we're catching the best books. It doesn't mean we're fixing the best books either. So it's all like a trial and error as you do things, and things are going to get better, and maybe we should just have a whole loop that just just run it for one day and it loops over itself and does everything and I don't know, merges itself and then like we'll see what staying is the application in is it bugless or not?

36:41Speaker 1:Oh?

36:41Speaker 2:Yeah, no, if you've got if you still got all you can eat account somewhere to just stick that thing off on a branch and hammer away and see where you get to. That be interesting, wouldn't it. But if you're paying by the token, I don't know if I'd turn that on.

36:56Speaker 1:I don't think anybody's anymore. Well, if you get that something like on GitHub, even if it's a read me or something, just let me know, and Fritz and I will have you on coded a I to give a demo, because that seems like a very valuable set up to me.

37:11Speaker 3:Yeah, we could test out and someone else's ce I and see what they're on their applications. He works, it works locally as well. And that just doesn't see I because we actually this is what we did as well. We were going to do a release and then like they said, right, let's get everybody together and try and find books a real typical manual kind of way, like let's go and and there's a lot of sections to complete, and I was like, this is not going to work.

37:34Speaker 3:You're never no one's going to get anything done in two hours a book bashing session of two hours. So I built the skill and then I showed everyone how to use the skill, and then I said, I'm not going to give you a prompt because Richard's prompt's going to be different to your prompt, Carl, My prom's going to be different. And I think that's okay. Because you're going to ask it for different things, you're going to get different results. And it was really cool because they all kind of found books in different ways and fix books very very quickly. That's when I ended up building the fixed Miedburg. But it worked really really well, and that's when we said, let's put this on CI. But you've got to test all these things locally in house, get a lot of team members on it, and get the feedback from it, and keep improving and iterating on it.

38:13Speaker 1:Nice. Awesome.

38:14Speaker 2:You mentioned earlier on about using Goose. Should we talk a little bit about this. It's an open source product, right.

38:19Speaker 3:Sure, Yeah, it's part of the Agentic Foundation now and it's it was originally by Block, but Goose is basically I don't know, I've fallen in love with it. Obviously. I worked at Block for a couple of weeks, so that's what was part of my job for a couple of weeks, was evangelizing Goose and then I got let go because they deleted some members of the company.

38:41Speaker 2:Yeah, it was like a layoff of four thousand. There's something crazy like that.

38:45Speaker 3:Just four thousands, just forty percent of the company. Yeah, so not many.

38:49Speaker 1:It was the excuse.

38:51Speaker 3:I mean, I mean, isn't it always the excuse these days? So what is anyway? So Goose is like an agent, just like cloud code, just like Codex kind of. But I think Goose is probably very underrated in the sense that Goose was able to do all the things that Cloud had recently did, like cloud cowork and you know, browser use and all these other tools that like became like a cloud made kind of like cool. Goose already had We were already using it and doing stuff with this for ages before, but we didn't they didn't have anyone to evangelize it enough or just show off because it was built for in house first, and they built it for at block, right, and then they kind of said, right, let's make this open source because it's a really cool tool.

39:37Speaker 3:So yeah, so Goose is free to use. You just bring your own provider whatever you know you want to use. It's really easy to switch between them. I find it one of the best agents for having these analysis conversations. When I want to build something, I'm not looking at code because there's no real code in there, right, there's no there's no editor built in or anything, so you're literally just talking analyzing things, and then you can create like, you know, recipes, so if something you want to run against your loops kind of thing like that was always there in Goose before skills obviously are in their session history is all in there, but you've just got a lot of like like they were the first to have like mcps, so they're the first to kind of test out and have everything. But yeah, Goose is amazing and everyone should use it a hell of a lot more.

40:27Speaker 3:But everyone uses the bigger names like they called in the codex because they're more famous.

40:32Speaker 2:Yeah, so the only difference here is more famous.

40:34Speaker 3:I still use codecs though in Claude, right, Yeah, it depends what you're doing.

40:40Speaker 1:It's even weird.

40:40Speaker 2:I mean, that's basically where React came from, like Big or Angular for that matter, Like internal libraries become open source libraries because they work. They were benefiting a company, now they can benefit more people.

40:53Speaker 1:M exactly.

40:55Speaker 2:That makes a lot of sense. I'm glad to know about that.

40:57Speaker 1:Actually I haven't seen it before.

40:58Speaker 2:So what's next?

41:02Speaker 3:Yeah, what's next in the world for you? I don't know. I mean I'm having fun being more like a step back into kind of like the engineer role as opposed to just just doing like the content creation side of things or or leading the product area. So I haven't actually done a video in a long time. I really need to more podcasts and more videos, but it's really hard to also find the time. When you go deep into the engineering part and start building things, it takes a lot of time, a lot of thought, a lot of mindset, and then you kind of like go, you build something. It's really cool and like you said, Richard, or maybe someone might want to see that scale, right, Yeah, I don't know. Yeah, maybe it would, but I don't even like share it because I'm thinking, like this is just for air company and why would anyone else want to do it because I think everyone else is already building that, right, Yeah.

41:51Speaker 1:I don't think so. I think it takes people like you too, you know, push the envelope and show people what's possible. There's a lot of people just waiting for that kind of thing, I think.

42:03Speaker 2:And just this view of a workflow, I mean, you have an advantage. You've worked through a lot of different products here, You've evaluated testing in a lot of ways. So I really appreciate your insights on this stuff just because you've seen so many versions of it.

42:17Speaker 3:Yeah, I guess so.

42:19Speaker 2:Yeah.

42:19Speaker 3:It doesn't mean I have all the answers there, no.

42:22Speaker 2:No, none of us do. But do you have a set of experiences that I think are really helpful, just as I. You know, we have the same way. We have no excuse for one hundredercent test coverage, Like I can never get the interns to do it, but I can get the bots to do it. We have no excuse for documentation anymore. It'll generate it.

42:40Speaker 1:So and.

42:44Speaker 2:Right, well, I think it's one of the ways now when I'm with different products that I'm working with.

42:47Speaker 1:When I when the when the.

42:50Speaker 2:Copilot of the agent struggles with it, you know, they're not internally, they're not using it for generate the documentation. Like I'm looking at you, Ubiquity. I use a lot of their networking here, but the documentation is always wrong. Yeah, because obviously they're not generating it, and so it cripples your ability to work with it. Often the tools guiding me to menu items that don't exist anymore, screens that don't exist anymore, like that stuff's wrong.

43:16Speaker 3:Well this happened at Playwright, sure, I bet well. Literally I couldn't keep up with, you know, the changes, and especially the screenshots and things. And that's why the team hated when I put screenshots and tried to make documentation look beautiful because they were like, it's going to be outdated in like a couple of weeks, and I was like, I know, I wanted to be beautiful. Yeah, I like, now it could just be beautiful every day, like you literally it costs money to run it, but you can just regenerate everything very easily.

43:42Speaker 2:Yeah, it should just be part of the pipeline. But you're right, you know, as our token economy becomes realistic bit by bit, you know, we're in this window now of now you're going to have to pay for this stuff. We're going to figure out if it's worth it. Yeah, I mean the reality is if your customers are more successful as a product, they'll probably buy more of it, and if they're not successful with it, they're going to buy less of it. And this is a part of making that successful. You know, documentation used to be another and it's just not anymore.

44:10Speaker 1:On a GitHub copilot anyway, I crack the code of AI credits and how much they're worth. So you notice if you're using copilot cli now you doesn't say how many tokens or requests. It says aiic AI credits, and they go up. But basically it's a dollar for one hundred AI credits. So if you pay one hundred dollars a month, you get you know, one hundred times one hundred AI credits. So that's that's good to know.

44:44Speaker 2:Then what work does that do?

44:46Speaker 1:What's that?

44:47Speaker 2:What work work does one hundred credits do?

44:49Speaker 1:Well? That's the question. Yeah. So I had a I don't know, an eight hour coding session a couple of days ago, and it cost me like forty bucks.

45:00Speaker 3:It's is not bad, right, It's not if it was good that eight hour session, because it's again like that human working for eight hours, did they produce good work or were they just sitting at a desk at a computer on right?

45:11Speaker 1:Yep?

45:11Speaker 2:Oh no, I've had a team of three working on an optimization plan for three months and at the end of it, the system was slower. That's the set of salaries due for me. You know, that's a lot of money.

45:21Speaker 3:Yeah, So I think that's going to be a problem overall for many companies. Sure, like we're going to eliminate people because AI is obviously faster at doing certain tasks, But then you end up with different problems, and who's going to solve those because AI is not ready to solve all the problems. We're not there yet.

45:41Speaker 2:Well, and I think it's full You know, it's way too soon to be talking about laying off people, even though folks are using that excuse to lay people off, and to be perfect a blunt because I cited all the time Jack Dorsey's excuse of laying of using AI's excuse to lay off those people was it's really because his crypto project failed. But if he emits his crypto failed, then it stock goes down. But if he says Ai has improved our efficiency. This stock goes up the fact that he's hurting people secondary to the point. A normal thing for companies when they get new tools like this is not to lay people off, but rather to take advantage of it, increase their productivity, and see if they can get more value with the folks they have. After all, they just gave them skills in the process of building these automations. It's only when you run out of ideas for growth that you might reduce your headcount. But the usual reason people reduce headcount is because they're not making money.

46:34Speaker 2:If you were as successful as tool, you are making money.

46:37Speaker 1:From the customers I've talked to. The backlog is still very full, huge, So yes, I've talked to two.

46:44Speaker 2:Teams now that ran through their feature sets like a year's worth of planned futures done in a few months, to the point where the customers don't want anything at the moment. They're still learning the new product, but now they're going into the backlog of things they never got to before.

47:00Speaker 1:And empty technical debt. Yeah, it's perfect time to do that. Yeah, yeah, wow. This has been a fascinating in inanating conversation it's always good to talk to you, Debbie. Yes, I don't know anything else you want to mention before we stop.

47:16Speaker 3:I guess we could talk just quickly a little bit about what we're working on when it comes to the AI platform that I mentioned, where I'm testing out and working on. Yes, So this is a new way of thinking that companies are going to need to work with. And we're not the only company doing something like this, and I think getub I've done something similar with Ace that they're going to be releasing, and there's a couple of different different companies out there. But it's the idea of making humans and agents working together in the same platform, right, And I know, like, yes, okay, now you can put claud in slacks. That's kind of the same process. Because the problem is we're all working individually and we're all like I'm on Goose, you're on Codex, you're on cloud code.

47:56Speaker 3:You're getting stuff done and everything's on GitHub and then buying its merge together. But there's no like the conversations of like what do you think about this, Richard? Is this a good way of going about things? So having the agents in the channel with you to be able to have those conversations, so you don't have to like have a conversation in Slack and then like do the work outside in somewhere else. You bring the agents, you cut and paste it into codex exactly, so you bring everyone into the same place. But not just like you know, Slack plus AI agents. We've also like put in the editor, so yes, code is right in there. The browser is in there as well. This is what makes it exciting because when I talked about playwright and I said, like, I'm you know, getting the playwright specialist. The playwright specialists can go ahead to a year, I'll go and write those tests. But it can also then open up the browser inside the platform and actually test out the you know, the like open the HTML report for example, of all your tests and even you can click or it can click for you and open the trade viewer and you've

48:59Speaker 3:got the trade sphere of all the tests as well.

49:01Speaker 1:Wow.

49:02Speaker 3:Right, So that's kind of in the platform. So it's like putting there's a lot of mini apps that can go in there, and then the workflows are there, the tasks are there, so it's kind of like a little bit almost like everything is in there, but it kind of is everything in there, but the idea of stopping you from context switching and jumping between that many apps and then losing that context, because I think that's the biggest problem we have these days, is we're losing context.

49:25Speaker 3:And I have, like all my conversations are in Goose of the moment. That's what I'm using. But we need to start bringing it into the platform when it's ready because nobody else can see what I've done. It's what's on my computer stays in my computer. And it's really like we're not we're not merging that into into the files, right, which I know there's another couple of companies thinking about ideas I've heard of entire they're doing some some cool stuff with that where you're like you put in the agent process in true into the commits, right, so it's like it's committed because then you can go back and forward and say, oh, this is done because the agent decided this, because et cetera, et cetera, et cetera. Mm hm, so's it's an interesting space, but yeah, a lot of cool things happening.

50:07Speaker 2:Yeah, and it's got to be a battle then demanded, you know. At the same time, we're talking about specializing with these agents too. So if they I appreciate the idea of give them access to the whole context, yeah, then also you now have to constrain them down to this is the area of your responsibility in this context exactly.

50:24Speaker 3:And that's why you have specialists and you know, this is your knowledge garden. This is your knowledge garden. This is your one And it's a lot of work, right, there's a lot of pieces put into the AI platform. You can all check it out the A platform. It's live, it's still got folks. But yeah, this is how we're going to work, I think in the future, having agents humans in together. Yeah, until the humans are deleted completely and it's only the agents. But I think we're miles away from that.

50:50Speaker 2:I don't know if that's actually gonna be the outcome, because they don't actually have reasoning, right, They've just the real decision making is still being done by people and they'll probably always will be yes.

51:01Speaker 1:See.

51:01Speaker 2:It reminds me of learning Agile, of rethinking the way software was being built, how the teams work together, what information need to be passed around, how it got managed. We have we're in the process of building tools again. To change the way our development cycles were. Yes, which it always feels like a waste of time. But at the same time it's like, when this works, well.

51:26Speaker 3:You move amazing. Yeah.

51:29Speaker 1:Yeah.

51:30Speaker 3:In the meantime, we're just going to burn tokens.

51:32Speaker 1:The right time.

51:33Speaker 2:We burn tokens so they make us pay for them.

51:37Speaker 1:Well, okay, we've come to another fantastic the end of another fantastic show. Debbie, thanks very much. It's been a pleasure to talk to you.

51:45Speaker 3:It always is, yes, and I hope I'm back again. And you know it's not an agent that's taken over my place. When I speaking, rocks.

51:55Speaker 1:Happen. All right, Thanks Debbie. Then we'll talk to you next time on dot net rocks. Dot net Rocks is brought to you by Franklin's Net and produced by Pop Studios, a full service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online at PWOP dot com.

52:37Speaker 4:Visit our website at d O T N E t R O c k S dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives. Going back to show number one, recorded in September two.

52:50Speaker 1:Thousand and two, and make sure you check out our sponsors. They keep us in business. Now go write some code, see you next time. Gotcha, daddy to see a summer time that pass hard than my Texas in red