LLMs in Software Testing: Start with Small, Bounded Tasks - Klaudia Dussa-Zieger

0:00Speaker 0: We are at the position that we need to decide how much liberty, how much autonomy do we want to give these systems. Speaker 0: So it's like going to the left or right? Speaker 0: Going in the middle of what do you want? Speaker 0: An autonomous agent who is able book appointments for me and can do some money transfer stuff. Speaker 0: I have given access all over the system. Speaker 0: from my experience with projects usability this first characteristic dropped.

0:30Speaker 0: if time is really an issue. Speaker 0: I think with the advent of more autonomous system, i think that the gap between using AI and testing AI is closing again because the more autonomy you give to a AI agent ,the better it has to be. Speaker 1: Welcome to Software Testing Unleashed! Speaker 1: Im Richie And im happy to share another episode from the TestCon V conference for Mauritius. Speaker 1: My guest today was Claudia Dusar-Zieger.

1:02Speaker 1: We talked about how to use language models in testing... ...in a very, very practical way. Speaker 1: why small functions like writing keyword documentation work so well Why reviews only got good with reasoning models and where AI is still no help at all. Speaker 1: That's all you hear on this episode. Speaker 1: And now enjoy! Speaker 1: It's the first time you're in my English podcast. Speaker 0: That's true! Speaker 0: We just Speaker 1: talked about it, two times at a German one and now for the first both from Germany, I'm from Austria but it's Speaker 0: still Speaker 1: counts.

1:54Speaker 1: But we are here in Mauritius the test conference and i like that. Speaker 1: the spirit is a very active community in software quality, software testing here at Mauritious. Speaker 0: yeah...I think you can feel there isn't rising community. Speaker 0: It hasn't started all that long ago six or seven years old, something around these lines. Speaker 0: It's their fifth conference so it still relatively long and you can see I think also the average age if i compare to conferences in Germany is way younger perhaps not as young as the Conference in Kenya because they were all very young.

2:30Speaker 0: but yeah lively energetic community for sure. Speaker 1: And you gave a speech yesterday, one of the first slots here about LLM in testing and that's I think a good point to start into look at it deeper because since we have LLMs now on the plate since few years often some speeches are only yes but they're not very specific or concrete. Speaker 1: And now we can maybe go a little bit deeper. Speaker 1: How can you really use the stuff in testing?

3:08Speaker 0: Yeah, okay! Speaker 0: So let me perhaps start further back... We started out actually trying different AI's already from last year but that was a visual classification algorithm to recognize moving objects on the GUI. Speaker 0: LLMs, no generative AI neural networks. Speaker 0: But for the last three-three and a half years we are looking into applications of language models how you could actually employ them in a smart way in software testing?

3:44Speaker 0: And given that fact is working with a lot of text, with lots of languages specifications test scripts reports yeah We thought it might be really worth trying then try to do little pieces. Speaker 0: so Our intention was not to come up with a fine-tuned, general testing LLM but really have small functionalities that would benefit the user. Speaker 0: And we started... With the test specification. Speaker 0: actually there is larger prototype where used an approach in additional project specific information.

4:19Speaker 0: so it's a rag approach retrieval augmented generation which you use as standard Generic model, and we started out with Lama three one. Speaker 0: And then we moved to clothes three five in. Speaker 0: just by the numbers you can see it's already a couple of years ago Speaker 1: like legacies off Speaker 0: It's You know what other side note? Speaker 0: One of the issues or I think that challenges I personally have with the language models, is they're evolving so fast.

4:49Speaker 0: The frequency's going down. Speaker 0: at the beginning it was about half a year and now i think every couple of weeks you have new model which is really tough. Speaker 0: but coming back to the language once. Speaker 0: yes we used this tool look at requirements in derive test specifications out of that after several iterations because at the begining had complete cycle without any human intervention. Speaker 0: that worked, but not too well.

5:18Speaker 0: I mean the intention was to actually create something so you wouldn't start on a white piece of paper... But it would start with a draft version in order to get around the writing block yeah? Speaker 0: So then we presented this to the testers and they felt a little bit left out. Speaker 0: We thought two birds with one stone improve the quality of the final result and get them into the boat so that they can see, have a say to it. Speaker 0: They can control at least little bit but had very strong opinion... ...that wanted share their experience.

5:56Speaker 0: for then was important one spot where could feed in additional information which is what we actually did. Speaker 0: So that worked well, and that was the starting point. Speaker 0: And then we decided... Now I'm speaking not as the ISDQB president but as a consultant coming from IMUs. Speaker 0: We do have a test management tool. Speaker 0: Then we decided to add little AI functionality in the Test Management Tool. Speaker 0: In our tool you can do test specification using keywords Not only for test automation But also manual testing And given the fact that keywords are basically code, right?

6:40Speaker 0: It's a script. Speaker 0: They're doing programming and they don't like to document most of their programmers! Speaker 0: The first tiny little bit we wrote within the tool was functionality. Speaker 0: where you press a button it reads through... Implementation of the keyword and it comes up with a header, so you get to short header. Speaker 0: You've got the description off the parameters that are longer description what he word is doing.

7:06Speaker 0: they just have to press a button read through it and then there done in their quite you know very easy very small test. Speaker 0: but they were quiet happy within and the amazing or at them. Speaker 0: The result now is that the descriptions are more unique and not uniform. Speaker 0: Everybody uses it, doesn't vary much so... That would work really well! Speaker 0: Then next thing was given a fact we have distributed development environment or distributed team.

7:35Speaker 0: The quality of the implementations for keywords were different. Speaker 0: We said why don't you use an LLM for reviews? Speaker 0: Given the fact rules, how keywords need to be implemented. Speaker 0: We have a glossary which terms you're allowed use and stuff like that. Speaker 0: so we played around. Speaker 0: try to feed in as additional information. Speaker 0: our rules are glossary and see if the LLM could do good review. Speaker 0: And there it really turned out.

8:02Speaker 0: You needed very specific about what LLMs were using. Speaker 0: So normal chat GPT versions didn't work but had one of their reasoning models. Speaker 0: So we ended up with O one and did it there. Speaker 0: And It worked actually quite well, so it catches most of the more obvious stuff. Speaker 0: but you get it very fast to get like this right within a second or two. Speaker 0: You do typically tour three cycles and then formally The keyword looks pretty good and Then they implement that test automation guys.

8:31Speaker 0: the engineers came in. Speaker 0: They said them We can look at Logic of the keyword if everything is fine. Speaker 0: So they also appreciated that. Speaker 0: so that were like, The first two little implementations we had in our tool and test bench And then next one was actually coming up when We decided that we needed a translation functionality. Speaker 0: That kind of sounds funny, but it works really well with language models... ...a translation functionality from technical defect description to a domain hooser could understand.

9:05Speaker 0: I mean, if you anticipate some erroneous behavior. Speaker 0: You typically have a very nice error message. Speaker 0: But if you really run into a technical problem where you get an defect message which is thrown by the system it's typically Very technical and since we are working with a lot of domain testers They don't understand what is in there? Speaker 0: Yeah And we figured well feed the defect message plus the test case that led to the defect into an LLM.

9:32Speaker 0: It could do the translation language, right? Speaker 0: Its language again and then also worked very nicely. Speaker 0: That Speaker 1: sounds great! Speaker 1: I like their approach too. Speaker 1: look at different spots in there. Speaker 1: And when you said about the review one question because When i looked at development teams who are now using AI for reviewing they're very happy To have a neutral instance more or less objective, not the senior deaf who is blaming them or another.

10:00Speaker 1: Do you have any feedback from people whose code gets reviewed by your AI? Speaker 0: No, not really. Speaker 0: We got feedback that they... Not with respect to this issue so you have somebody who's blaming someone else. Speaker 0: So it was not the issues but we were very happy They could rid of the tedious reviewing task where You can just come up With plain simple stuff and get rid Of all boring things And focus on interesting issues.

10:28Speaker 0: That is their feedback. Speaker 0: The other thing perhaps because The team knows each other despite the fact that they're in two different locations. Speaker 0: They know, which are there for more than ten years. Speaker 0: So their very well connected and I don't think that was Really? Speaker 0: The issue at least didn't tell me put it That way but they were happy to get the boring stuff out of the way in a Very fast manner And then that worked really Well.

10:54Speaker 0: so that Was actually the starting point. Speaker 0: and Then when we talked or when i talk too couple Of our colleagues of my colleague's um they Were using co-pilot for coding without even talking about it, right? Speaker 0: And then interesting enough when I talked to one of the guys just floor below me. Speaker 0: He said he's not only using for coding is also develop scripts for vagrants so that you can set up the environment automatically.

11:20Speaker 0: And I thought, That was actually quite interesting because that is the DevOps approach right? Speaker 0: You build-up to test environment automatically. Speaker 0: and he said yeah it's kind of a boilerplate thing. Speaker 0: He gets ninety five percent The last five percent has do by himself and he knows what the script is supposed to look like so we can easily check it. Speaker 0: And for him, its just also a question of efficiency. Speaker 0: He just puts in, gets within seconds.

11:45Speaker 0: It would take perhaps half an hour Perhaps fifteen minutes or longer but it's faster. Speaker 0: He just has to review it, and then he's fine. Speaker 0: So that also works. Speaker 1: Did you have more iterations to get it run or with reviewing? Speaker 1: Or is this one shot where... Speaker 0: The review thing took several iterations because we started out I think with ChatGPT-FORZERO And they didn't work well there. Speaker 0: If you look at language models You really need be careful about which year you're talking.

12:21Speaker 0: Because the reasoning models haven't been there from the beginning, so we started out with a normal chat model and then we saw coming looking at the results that they were not good enough in them. Speaker 0: We figured perhaps we need to move to the newer reasoning models And really try out the reasoning. Speaker 0: one on that did The trick. Speaker 0: yeah. Speaker 0: So for the documentation of the keywords? Speaker 0: I think there won't be the one generic solution that covers everything.

12:54Speaker 0: You need to be more specific what you really want and different skills for it. Speaker 1: I think what you say is so important that we have to be clear. Speaker 1: What's the problem? Speaker 1: We want to solve and use specified very, very clear problems or things you want to resolve by AI... Actually Speaker 0: for us it was an experiment! Speaker 0: Is this going to work? Speaker 0: Can we make it work or Speaker 0: not?! Speaker 0: The same approach also took as an experiment perhaps even more as a challenge when we try to look at the usage of lounge language models for their digital accessibility tests.

13:31Speaker 0: Yeah, because we have a group that just deals with Digital Accessibility testing and We always had a discussion how much can you automate? Speaker 0: And they were all forty percent. Speaker 0: but we can't do more. Speaker 0: Because if we looked at the tanks we need to Evaluate it then say It's okay or if its not ok. Speaker 0: so there is Always A human person needed and we said let's Just Try it. Speaker 0: It's like challenge accepted, let us just try.

13:56Speaker 0: and we also just tried out again as an experiment with this new agent technology the setup where you use NCP and stuff like that And We Just used it for The search of the alternative or For the generation Of the Alternative text for pictures Where You need to interpret the picture. Speaker 0: That from me was actually I thought that would be the Challenge And it worked. Speaker 0: It worked! Speaker 0: Again, it doesn't work perfectly.

14:26Speaker 0: so sometimes the system is proposing something where I would say that's okay but we're a specialist for digital accessibility said its duplication and does not make sense But still gives them again inefficiency. Speaker 0: So just doing faster goes through all picture like on-the-fly And then you get the results, again. Speaker 0: You get the result back and here as a human in the loop have to look at it and say It's okay. Speaker 1: Yeah I think this is so important To understand that we don't need The hundred percent solution out of the air So We can review it when we Have people there who see what they outcome Then Can reflect on That yeah?

15:07Speaker 0: Yeah i think This Is also Right now for me The interesting point because I think right now, given that MCP was developed and MCP the model context protocol where you connect different functionality for AI systems or large language ones. Speaker 0: That was just developed and released in November. Speaker 0: twenty-twenty four so a little bit over year not quite one and half. Speaker 0: So everything is very still very new And we don't have years and years of experience AI agents that run autonomously is increasing.

15:46Speaker 0: Yeah, now our first version there was nothing autonomous. Speaker 0: we told it every step what it's supposed to do but Now It Is Increasing and I think right? Speaker 0: We are at the position That we need To decide how much Liberty How Much autonomy Do we want to give these Systems? Speaker 0: so it's like Going to The Left going to the Right Going to the Middle What do we Want? Speaker 0: And then And again, we're back to some of the basics of testing.

16:12Speaker 0: It's a risk-based approach. Speaker 0: again Do I want to accept that if i have To give this the autonomous agent? Speaker 0: Yeah My banking information my health Information and stuff like That. Speaker 0: and you know even If i look at austria are You coming from austria open claw? Speaker 0: yeah From The austrian developer Scary Scary, an autonomous agent that is able to book appointments for me... ...that's able do some money transfer stuff and I have given him access all the systems.

16:49Speaker 0: Interesting if you want put it nicely! Speaker 1: The guy has now a good job at opening my eyes. Speaker 0: And i read in interview that he has the vision, we will have personalized AI agents and they should be as easy so his mother can work with them. Speaker 0: So for me it's a completely new level because I'm pretty sure his mother won't judge whether this thing is working correctly or not. Speaker 0: but the level of automation, access rights you need to give open claw.

17:32Speaker 1: Well, come back to testing. Speaker 1: Did you have some experiments where then said okay it doesn't work? Speaker 1: It is not the way we can go there with an AI? Speaker 0: Yes! Speaker 0: Experiments perhaps are a little too strong word... We had two projects that were actually doing a bit of reverse engineering and for older, existing embedded code doing unit tests because they need to do it due to regulatory reasons. Speaker 0: And that is very cumbersome.

18:05Speaker 0: It's like a little bit like arcology right? Speaker 0: You need to look into the details you need to figure out and see plus-plus Speaker 0: code.". Speaker 0: I was asking... Actually one of our AI experts working in these projects. Speaker 0: when i asked him if an AI could help your language model he said no No, because you need to look into the existing code. Speaker 0: So it's not about building code. Speaker 0: It's doing the unit test for these codes and they don't want to just cover it on The line coverage basis.

18:35Speaker 0: They really want to write Test cases that make sense. Speaker 0: And he says that level of expertise is Not there. Speaker 0: He doesn't see it. Speaker 0: It's not like writing a header or keyword? Speaker 0: concentrate, dig into the existing code. Speaker 0: Which of course is not well documented right? Speaker 0: So I don't want to say how good or bad the quality of the coded but definitely it's not documented. Speaker 0: there are no requirements so you have to do everything just by looking at the code.

19:08Speaker 0: and from his point of view for that kind... For that particular problem he says No! Speaker 0: I think would need to invest a lot of time to train the models, or somehow get them up to that level. Speaker 0: they could understand it. Speaker 0: And by then perhaps... That's his viewpoint in doing this for himself and actually team four. Speaker 0: but he said I was just asking that question last week because i was very curious. Speaker 0: He didn't see an opportunity.

19:40Speaker 0: They are not good enough yet. Speaker 1: Do you think when we look in the direction of test automation, at a higher level where there are possibilities to use LLMs or is it not really... a good part to use it there. Speaker 1: Use for reviewing the stuff, but are they in other ways? Speaker 0: Well I think test automation is for coding itself. Speaker 0: you can use co-pilot and that would work as well as normal programming. Speaker 0: Automation is nothing else than programming.

20:14Speaker 0: so yeah i think that will work. Speaker 0: if we're looking at the test automation on system level If your going through graphically user interface AI applications that do the recognition of objects, if they move or slightly named differently. Speaker 0: I think it would also work. Speaker 0: there have been applications where you use the word embeddings from LLMs in order to find different path identifiers if your first one breaks. Speaker 0: So that goes into the direction of self-healing test cases, there was a presentation by Manika Jihari at the Robocon twenty twenty five and I Was just asking this year The robo con is always in February so that was two weeks or two months ago That they're not pursuing that idea anymore.

21:08Speaker 0: So I need to dig a little deeper why there are no doing it. Speaker 0: when? Speaker 0: It came up last year. Speaker 0: We also had a discussion with our specialist and they said if They use the embeddings in order to do self-healings of course, they are covering Up real mistakes. Speaker 0: Yeah And it's hard to say whether was accidental or Whether There is A Real Mistake Where They Really Should Look at. Speaker 0: so there Are Some I mean, definitely for the normal coding and let me call it more advanced stuff we'll need to see.

21:40Speaker 1: Yeah okay... We cannot really predict a future up-to now with AI. Speaker 1: It's crazy! Speaker 1: Honestly it is Speaker 0: crazy. Speaker 0: But Speaker 1: do you have any wish for the AI? Speaker 1: To support us in testing process where there are now a lack or gap? Speaker 1: What can you imagine? Speaker 1: where the AI will help us in the future as a tester? Speaker 0: Where we have a gap. Speaker 1: Or something that you want to get rid of during testing process!

22:12Speaker 0: I think if the AI would really give us speed and efficiency, then all functional tests are done so that we could concentrate more on security issues or performance issues. Speaker 0: on usability, we were just discussing it yesterday at the conference from my experience in projects. Speaker 0: Usability is the first characteristic that has dropped if time really an issue because functionality must be there and then performance security.

22:43Speaker 0: so perhaps to speed up a process where you can take care of these characteristics which typically are neglected If You Don't Have To Do That. Speaker 0: Personally I would love more applications for doing the test reporting since writing the information, summarizing the information is something that I personally don't love to write. Speaker 0: If i have to write something it's an effort. Speaker 0: so if get support there getting stuff together and writing... And this a good application of her language model because its piece of text!

23:18Speaker 0: That would be nice. Speaker 0: more support here. Speaker 0: let me think what else? Speaker 0: Yeah, that's for I would say that for right now. Speaker 0: That would be it. Speaker 0: And another thing and perhaps let me just state that i think with the advent of more autonomous system I Think The gap between using AI and testing AI is closing again because the more autonomy you give to an AI agent the better It has to be. Speaker 0: so far we were talking about two different sides of this same metal But I have the feeling now there's a two different paths converge again because people are really looking at quality of output.

24:05Speaker 0: of the LLMs because it has to be good enough that you can automatically go into next processing step or at least allow the agent. Speaker 0: To continue doing it by himself and That was not necessary before, Because You had always The human in between. Speaker 0: so all these agents when we're talking about Agent farms for orchestrated teams I think the quality Of the Agents themselves become more important. Speaker 0: And from just using gen AI to support testing now were moving a little bit back to testing gen AI, which given the hype that we have around Gen AI wasn't it not much in focus before.

24:43Speaker 0: But I think were getting there now. Speaker 0: i have an impression where we're closing that gap. Speaker 1: Yeah! Speaker 1: We will see! Speaker 0: So Speaker 1: hopefully you know the Imbus test bench developer here this episode and now making them module for Test Reporting. Speaker 1: more AI for you. Speaker 1: Claudia, thank you very much that are here in the show. Speaker 1: That give us some insights on how to use AI throughout testing process in a concrete and specific way.

25:12Speaker 1: I really Speaker 0: appreciate it. Speaker 0: Yeah my pleasure! Speaker 0: My pleasure Speaker 1: And yeah i wish you great conference. Speaker 1: next week is IST-Guby meeting here at Mauritius too. Speaker 1: A lot of work groups. Speaker 0: We will release new AI testing syllabus. Speaker 0: The new gen AI version will be released. Speaker 0: Yes, a lot of AI related stuff too. Speaker 1: It's important It's important. Speaker 1: Thank you very Speaker 0: much!