AI-Assisted Testing: Testers Keep Control of the Process - Mitko Mitev

0:00Every let's say five years there is some buzzwords coming on the market. Like first was test automation, it will replace testosterone. Then it was what was it after that? Uh agile. Uh then it was uh DevOps. Uh now it's AI. Everybody's you know saying that uh we will get rid of QAs and testers but uh we're still here You know, AI has at this stage of development of AI has a bit of a um challenge with uh context and business logic and also b behaving as a user, you know, predicting what the user actions will be.

0:36that's not that easy for uh artificial intelligence. This has to be still in control of humans. People are thinking maybe w in the future we will not need um that kind of skills like knowing TypeScript or Python, you know, maybe that's not uh important anymore. Um I still think that's uh not going to happen because um even if it is possible for AI from playing language to generate scripts. Somebody has to make sure they're correct and that they're doing the the you know the job that they have to do.

1:15AI tools have better English than mine so welcome to Software Testing Unleashed. Podcast for testers, developers and test automation engineers and all the people in the software development process who want to create great amazing quality software. My today's episode was recorded at TestWarez. A great Testing conference in Poland. The days were full-packed with great quality content and I had a great time discussing and chatting with the Polish and the international software test community.

1:47If you haven't been here, you should definitely go to one of the Next Test Warriors conferences. You find the link to the conference in the show notes. My guest today is Mitko Mitev. With more than 30 years of experience in the software quality assurance industry, he's one of the leading software test experts. In Southeast Europe. I know him from the International Software Testing Qualification Board, ISTQB, where he is very active, participating in the efforts for setting, improving, and popularizing international standards.

2:19We talked about how How can AI enhance our testing business? Where can we use AI in the different activities of our test process? What are the benefits and what are the pitfalls? I'm happy that Mitko shared his experiences here in the podcast and now enjoy. Hi Mitko, great to have you here on the show. Thank you for inviting me. Uh thank you very much. Yeah, it's it's a pleasure for me. We know each other so long from the ISTQB stuff and now We are here in uh Wisla at the Test Warriors and uh meeting at this conference and now we have time to make the podcast episode.

3:07I I read your abstract in the for the in the agenda from the conference and uh you're dealing with a very popular uh topic nowadays, uh how AI impacts our testing uh Yes, um it's a hot topic right now. You know everybody's talking about uh artificial intelligence and uh um as you see in the last three, four years it's going into all directions in life people start using it for different purposes. Um in different uh professions and business areas And of course it's uh getting more and more integrated in the IT uh industry.

3:47Um my talk was um focused focused on uh how it can enhance testing especially because we are testing people you know and uh and I I wanted to address uh address it uh how we can use AI tools in different uh uh phases of software testing process and you know I'm uh I person so um I try to structure my um talk uh around the phases that we have in uh STQB foundation level let's say you know test planning test uh design analysis test design execution reporting um so I wanted to uh tell the people how uh easy is to use different tools in different phases.

4:36and um there are a lot of AI tools right now on the market and I think there will be even more in the future. Uh some of them are mature, some of them are not so uh very well developed yet but uh Things are changing fast. And um uh if you start using AI, you can um make your life very not very easy but easier. You can save probably th twenty, thirty, forty percent of the effort and time. Some uh labor-intensive tasks can be done quickly. And uh of course focuses on test automation because a lot of people are doing test automation as well.

5:19I can help there in that area. as well and uh nowadays uh with the coming uh agents on the market it's becoming even more interesting because uh they're more or less uh becoming uh autonomous. Mm-hmm. And they can you can just tell them what to do and they uh they're going and do it. Of course, um you cannot over rely on the AI tools, you have to have people to control it, train it and and verify the results in the end. But indeed the result can be uh satisfactory Yeah , uh it's still it's still uh crucial that we have our basic testing knowledge uh to to verify the the output.

6:00Yes, absolutely. I don't people are afraid that they're gonna replace us in all these tools, but I don't believe that. I think uh the scope of the work of the testers of the QAs will change, but um we will stay in control. Yeah and we can say that the process will be AI assisted but not completely replaced by AI. I think uh most of the uh mechanical tasks and out out uh automation tasks will be probably done by AI, but you know AI has at this stage of development of AI has a bit of a um challenge with uh context and business logic and also behaving as a user you know, predicting what the user actions will be, that's not that easy for uh artificial intelligence.

6:54This has to be still in control of in in real life uh we c uh we can now use it uh and to to support us. Um yeah I give some examples about uh let's say how to create a test plan. But My intention was to focus more on um on labor intensive tasks. Like in most cases these um areas where creating test cases for example and test data. You know, especially for the test data. It can generate a lot of test data very quickly by just simple explanation what exactly you need.

7:46For example, you can say give me two hundred different users with bank accounts and uh limits on transactions and all this stuff and then you can generate data very quickly. Also for the test cases, the responsibility, if you have documentation and requirements or use cases you can connect your confluence or some other tools and tell uh LLM uh can you generate test cases based on the requirements that I have already. So it's becoming really interesting time for uh people who like uh new technology, uh early adopters as I say.

8:25Um so those kind of heavy tasks, let's say if you have to generate 4,000 test cases, for example, for some complicated software. Um probably a human will need a a month. And with AI support you can probably do it in a week. So performance wise You know, it's it's helping a lot. Yeah, yeah. And is the AI nowadays in in charge of of using our test techniques like the accuration partitioning or and this stuff, or is it not that useful in this context?

9:01Uh it depends on the context of the model. Um usually um you know When you have to use um some kind of uh large uh language model, for example, you train it in advance with uh data and uh in your business domain and uh a lot of uh companies have already prepared uh AI uh AI models with uh test data and testing and of course if you use those kind of techniques that you're mentioning Um it's not a problem to use the techniques and you can just say do this and do that and uh use this uh test technique and that's okay.

9:45Of course you can also say choose y you choose uh which techniques to use and uh you know STQB is a standard, uh de facto a standard in the world. So even if you don't put a lot of context uh behind in in the model. Even uh by public information that is available, techniques are also very broadly uh known and people can use them. Usually no problem. Yeah, yeah. Especially with the exploratory testing, there this in this area AI can also help a lot.

10:19Um of course, you know, um people uh believe that uh with exploratory testing um mostly senior people should be uh should be uh busy but um You can instruct your AI to do uh to go different directions in unknown sequences of the process so he can simulate different activities that even people cannot imagine. Yeah, yeah, yeah. It c it can be a good combination to our critical thinking from the testing with the input from the from the AI stuff. Yes, uh that's the way I see it in the future.

11:00I s I s I see it like a combination of AI and uh people. Yeah. And uh of course as I said the role of the testers will shift Towards more creative tasks, more thinking and decision making, validation, verification and training of the model, because AI is all also has to be controlled trained and controlled and verified results um and more um Mechanical tasks, automated tasks will be shifted towards AI. You also mentioned before that uh the topic of Test automation uh and and uh what what do you think where where are we now in the AI context with test automation and what what what is the direction this automation stuff will go in the next years?

11:55Um So at the moment there are a lot of possibilities and um you know popular um tools for test automation uh integrating also AI uh support and and you know um there are popular concepts like MCP right now that you can connect uh LLM with a test automation tool and basically automate the whole process. Also with the coming on uh agents on the market, you can instruct um the AI to generate your uh native language s instructions like test cases in native language.

12:38So some people think the future of the scripting, like writing scripts is probably gone because if you can actually from normal language go directly to test execution without going through the scripting, that's an interesting concept. Of course it will go through some scripting, but it will be automatically generated. So people are thinking maybe in the future we will not need um that kind of skills like knowing TypeScript or Python, you know, maybe that's not uh important anymore.

13:16Um I still think that's uh not going to happen because um Even if it is possible for AI from plain language to generate scripts, somebody has to make sure the correct and that they're doing the the you know the job that they have to do. So um I still think people have to have some serious knowledge about uh But may maybe it can be a good uh a good interface for the business people to uh what we try with the local tools nowadays that they have and like in like in jet uh uh plain text they can give into an and tool and and then get a a draft for a for a script which can be used by the the test automation engineer or something.

14:01Exactly. That's that's actually the added value. of uh using AI because it widens the the uh group of people who can actually do this kind of uh activities. Um before um only testers and test automation specialists could do this part and now with the introduction uh introducing the plain language uh test combinations and instructions, business people can be included in this group. So this will definitely add value. Uh the projects will benefit from from this opportunity because with the plain language it can we can instruct the two what to do, that's really uh a beautiful opportunity.

14:48Yeah. Do you think uh that it's There's also a way to use it more in the in the test management stuff or in the reporting stuff uh future? Um yes, the test management part is interesting because um Um as I said you can actually plan. There are agents called planners, so you can actually do a good planning and based on a risk assessment uh which areas we have to test more and which areas we have to test Less also um AI has an influence in the defect analysis, then you can use it uh to cluster uh similar defects then put on uh the uh projectors you know the uh importance of the different areas of the software that generate more defects for example and then focus the testing in that area too So with clustering of defects and risk analysis and root cause analysis, also with AIs you can analyze the local files and this can give you a lot of information for the reasoning, the the cause of the um the effects and root cause will be easier if you use uh AI.

16:02And at the end with the reporting Uh it is uh if you use AI then you can easily generate a report and summarize it and make it easy for C level managers to read it and say, okay, this is what happened, this is why it happened, and this is what uh our recommendation is if we're going live or not. So the process of management the whole testing uh activities will be easier, I think. Yeah. And it uh well I can imagine what comes to my mind is that we we have all this by reporting to do a more uh specific to the different audiences we have in in in our company so the eye can help us to to go uh into the direction of the different stakeholders too.

16:48Yes absolutely you can You can specify different kontext avarising and reporting the information um intended for business people, intended for technical people, for DevOps, for operational people. So yes, this is really Jusfå and also as I'm not native speaker English um I can say uh AI tools have better English than mine. So for sure the reporting will and the planning will be uh in better English than than some of us. Yeah, yeah. So a huge support uh possibility for us in the testing industry in the in all the different phases, but not replacing us.

17:37very crucial to know because a lot of people I think have nowadays are afraid that their testing job is is gone but as you as I understand it's changing. It's not You know, I've been in uh long years in the industry and every let's say five years there is some buzzwords coming on the market. Like first was test automation, it will replace testosterone. Agile. Uh then it was uh DevOps, uh now it's AI, uh all the low-code stuff too. You know, s saying that uh we will get rid of KAs and Tesla's uh but uh we're still here.

18:14You know. And I don't believe it will happen again. It will shift the um skill set and the focus because Um as a friend of mine said, um people should stop asking on interviews what's the difference between class and object. When you hire uh testers, you should probably ask what is MCP? Do you know how LLM works? Do you have an idea how agents work? So really the skill set and the focus will be shifted but we will be needed. Um For me the future is uh AI assisted but not replaced.

18:54Yeah, yeah, yeah. And and maybe it's even more work for us to do in the next years when we think of the wipe coding stuff and all the stuff we're just generating software and code with unknown quality. Yeah. Somebody has to test it. Well AIF affects also software development, not only testing. I mean nowadays i it is possible for, you know uh software development teams to generate software very quickly and you can't really cope if if If the development is going that direction, you can't really cope with uh human testing only.

19:27You really need uh AI support in this uh activities as well. So um I see it as a combination and the whole teams are moving in that direction. And um You're right. The speed of generated software producing software is enormous and we have to deal with that somehow. Um some people say I will uh study AI but next year. I would recommend everybody to start now. Yeah. So, you know, if you uh Wait for three months you may be late already. Yeah, that's true.

20:09It's it's a a thing we have to deal now with. Yes, yeah. Yeah, you don't you don't have to postpone it. You have to have to start now. I'm a big believer, you know. Yeah, yeah. And you are also a a very uh integrated, respected uh foundation in the in this TQB area. So you're dealing with I ISTQB stuff over decades, I would say, now Uh more than two decades. Yeah. Yeah, yeah, and and you are worr one uh is very very deep inside uh this this stuff.

20:41And uh how when I want to learn testing in the AI area now, what what is the IST QB bringing? there to support me as a tester. Right. So I will recommend people to follow the process that ITQB has described. We do have two syllabi, different syllabi in the area of AI. One is testing AI , which is on the market probably. around two years. And the other one is brand new since July. It's called Gene AI In Testing. Which is very modern. Both of them are very interesting.

21:24One of them is how to test AI products. The other one is how to use AI products in the day-to-day work. Däfter you first have to have fundation. Obviously, because in för oss this is really the basis. of the software testing knowledge. Um I do believe uh the interest in this two syllabus in the in the next months and years will be enormous because it's really a key uh key area of knowledge that we need to support. And for the GNA especially because it's it's changing so fast.

22:05We um intending to ch to update it every three to six months to catch the new trends and and tendencies. So I would recommend to everybody um to take a look on the syllabi AI syllabi of ISTQB. Okay, I think it would be a good starting point to get absolutely and it we know the ISTQBS as a standardized body is uh uh doing a good stuff there is all it's solid work for the foundation body of knowledge is uh it's a good place um I said already we are de facto the standard in the industry.

22:41So and um I know the syllabi they're very well structured. uh knowledge is very well structured. So it's a good uh starting point for people to get into the testing with AI and yeah and the Mitko, thank you very much for being here in the show that you uh shared your insights, your experiences with the AI stuff. I think it's the a topic we have to deal as a tester and QA people for the next now and for the next months and years. Yes, it is it is a key uh key area that we have to put some efforts.

23:15Yes, yes, that's true. Thank you very much for Very happy that you were here and enjoy the rest of the conference.