Quality Hunting: Testing for Information Instead of Bugs - Rik Marselis

0:00Testing is not about finding bugs. Testing is about getting information about the quality level of your system. Teams take totally different directions. Even if they have the same assignments, still they are going to look for different things. My advice to such teams is to think about the structure of your report at the start. Most teams, they have sprints and then the result of several sprints into production and before that I think would be an ideal moment to do some quality hunting.

0:37Welcome to Software Testing Unleashed. My name is Richie and I’m happy to have another episode from the Istanbul Software Testing Conference 2026 for you. My guest was Rik Marselis. He joined for one of the first episodes of this podcast which was about test design techniques. But today we talked about quality hunting. Quality hunting is his alternative to bug hunting. Why testing is about information rather than of bugs, how a quality hunt works in practice, and what helps your stakeholder sleep well at night.

1:10All that you will find in this episode. And now enjoy! Hi Rik, great to have you in the show again. Yeah, good to be here again, Richie. Yeah, it’s fun. We’re here at the Istanbul Testing Conference. Yeah. Your third time, you initiated with Okan together this conference. Yes, indeed. So, for me it is the third Istanbul Software Testing Conference. Yeah. And it’s great to see it grow and, well, have new things all the time, like you coming here to do a podcast recording.

1:50So, that’s a great thing. Yeah, it’s nice and it’s a very professional conference, I think. So as a speaker today, I saw the briefing up front, the emails I got, I was so happy to have all this information in one place and really, really great organization here. Yeah, it is. Yeah. But we don’t want to talk about the conference too much. We want to talk about one part of your book, of your content you created. It’s quality hunting. Quality hunting, yes.

2:21And that’s... It started with my frustration about bug hunting. Yeah. Because some people think bug hunting is great, but bug hunting starts with the premise that you should find bugs. Yeah. And in my opinion, testing is not about finding bugs. Testing is about getting information about the quality level of your system. And if you come across bugs, well, then you know the quality is not what you expected. Then you may do something about it. But the main thing is that you want to know about quality.

2:59So that made me think. And a couple of years ago at the Eurostar conference, I was involved in the test lab and we wanted to organize some hands-on stuff. And we organized the first ever quality hunt. And that worked surprisingly well. And especially you saw there that some people still had the old idea of finding bugs. So at the end of the quality hunt, they came up with a big pile of problems they found. Yeah. But it didn’t say anything about the quality.

3:32It just said we found problems. Yeah, okay. But is it good enough? Yeah. Whereas another team only found, I think, like eight problems. But of course, they looked in how can it be used. and for normal audience it works good enough. Yeah. Also they found if you are a disabled person it doesn’t work good enough because some choices are not available for you. And so they had a very good evaluation of the quality. And that led to describing it more formal and we introduced it in two ways in the last couple of months.

4:14And the first way is we introduced it in the book, Amplified Quality Engineering, where basically the book is about using AI for testing, but also doing quality engineering for AI-based systems. Yeah. But one of the things with AI, and that’s also something you emphasized in your talk on this conference, is that it is still about the people making decisions and it’s not just about AI running the show. And quality hunting is a thing where people still need to think about what was the goal? Do we actually get what we need?

4:57And quality engineering, we described a lot about it in the previous book, by the way, that was in the previous podcast, "Quality for DevOps Teams." But based on that book, I was also invited to work together with a lot of other people on the new ISTQB syllabus, Quality in DevOps. And in that syllabus, we also introduced quality hunting as one of the approaches to make sure that also in a DevOps environment where people say automate everything, you still keep the human curiosity and a critical mind to make sure that you get a good quality.

5:43I like the topic now much more than before because the people are so involved. So that’s always my big topic. So I can think of bug hunting is very easy. So you’re searching for bugs. So it’s a very clear goal. Quality hunting sounds for me like a little bit fluffy. So let’s go into it. How can we make it that we go for quality there? Well, the idea of quality hunting is of course that you first determine what actually are we looking for? What kind of information do we need?

6:19So you need a little bit of preparation before and determine, okay, what do we want to know? And then you start looking for that. And just like bug hunting, quality hunting is also organized a little bit like a game. So people can win a prize and I always advise to do a small prize because it’s not about the winning, it’s about getting good information. But by having a few teams competing to get good information, they also stimulate each other to look further and really find the information that is needed for the stakeholders.

6:59And so typically we describe that quality hunting should be done by teams about half of the size of an agile team. As you know, average agile team is seven people. So let’s say four people in a quality hunting team. And then typically it should be no more than half a day or like three hours or something. But then if you have multiple teams, like four, five, six teams together, you get an awful lot of information in a very short time. And for stakeholders, it’s very nice to get all that information and very quickly learn about what the experience is with the system.

7:38So this might be a good addition to, for example, acceptance testing. And you can even involve your end users or managers or other people involved, so they also get a first-hand experience about the quality. And you mentioned that before we have to, what are we looking for in this session? So we have to define that. Can you give us some examples what could be that what we are looking then? Now well, one of the things of course is think of your quality characteristics.

8:11So it’s not just about does the system do what it’s supposed to do, which of course is important, but also how does it do it? So what non-functionals are we looking for? And some of course are more easy than others because to do a performance test in a quality hunting way, probably not the best thing. But usability testing is something that is quite hard to define beforehand. is quite easy to do in such a short period of time. So that’s an example of a thing where you could well organize a quality hunt.

8:51Okay, so then we have our goal and I don’t know, not everyone is aware of bug hunting and all the structure in there or how does the process go. So maybe we can go one time through this half day of a quality hunting session. How we have the teams with five, six, seven. So you have the teams, they determine, or maybe somebody who makes the assignment determines what kind of information are we looking for. And then the team together decides how are we going to approach this?

9:22What can we do in the time we have? And then they basically approach it as an exploratory test. So think of a test they can do, do the test, learn something, check if that helps them get the information they’re looking for. And that way progress in quite an exploring way of the system, because you don’t prepare much up front. But you do get triggered by what you see happening and then go on with that, but also sometimes take a step back and again think what was the end goal?

10:01The end goal was that we should supply this kind of information to our stakeholders. And then basically continue that. And don’t forget that in the end, you need to report the information. So it is also about gathering the information, structuring it, and making sure that your stakeholder gets true information and not just raw data about what you have done. And that’s also something we describe in the book, the hierarchy of data, information, knowledge and wisdom.

10:39And you see that we at least make... Oh, what a noise here. Somebody is running a cart through the hallway. But so, of course, when testing, you first find some data, but you assemble that to information. And what you want to convey to the stakeholders, actually knowledge, so they actually know about the quality. And then that’s up to them to have the wisdom to decide whether they go live or not. But the teams are totally free in how to approach this session.

11:19Yeah, it’s not very much bound to what they are allowed to do. But the main thing for them is to remember the goal is that we supply the best information possible in this time frame to our stakeholders. And then of course, a little bit of the joy is that multiple teams are trying and then the stakeholder decides who won this time. But often it is also about winning together because the teams together get a lot of information because what you probably will notice or what you will notice often is that teams take totally different directions.

12:04Even if they have the same assignments, still they are going to look for different things. Take for example, usability. Some teams will look for, does it look good or does it align with our company rules or guidelines or style? But other people will probably think about accessibility and think about people with disabilities, how can they use the system? So for the stakeholder, often it will be hard to determine who actually wins because there’s value in everything.

12:39Yeah, yeah. And that’s also why maybe you better not do a big prize, but maybe a few smaller prizes Wow. Yeah. And when I do a bug hunting session, the output is normally a list of bugs. Yeah. So in quality hunting, is it in your experience also more list format or more prose text or images or what’s your experience there? Or is it completely open? That depends on the team. My advice to such teams is to think about the structure of your report at the start.

13:16And by the way, that’s not just for quality hunting, but that’s my advice. Always in testing, first think about your report. What are you going to convey? What message do you want to have for your audience? And then you have basically the structure of your report and you only have to fill it in. Yeah. And that’s what you do during the quality hunting session. you check what kind of information do we already have, what kind of information do we need to add.

13:46And at the end, you have a nice overview of the things you learned, the information you have, so that your stakeholders can get the right knowledge. Yeah, yeah. When someone wants to start with this new form of gathering information and has no idea from bug hunting and quality hunting, Do you have any idea or tip with which goal they should start? Because quality characteristic is very broad and maybe it should be something to start small to get some experience with the methodology of quality hunting.

14:25Well, I think the main thing is to ask your stakeholders what kind of information would help them to sleep well at night. Because that’s often said that what is quality engineering and testing all about is make sure your stakeholders can sleep well and they don’t have to worry about the system. So it starts with what kind of information would they like to have and then try to gather the inputs to generate that information. So it starts with thinking about your stakeholders, your users or their management or whoever is the stakeholder, the product owner of course.

15:08Do you have any experience on what is a good time frame to repeat such a quality hunting session because if it’s every week it gets boring maybe or do you have any idea what would be a good time frame? I would say that it depends a bit on the situation. So one choice might be that you make sure that at least every time you want to release to production, you have done some quality hunting before that. Of course, if you truly do DevOps, you release every single user story to production right away, and well, then it wouldn’t work.

15:53But most teams, they have sprints and then the result of several sprints is put into production. And before that, I think, would be an ideal moment to do some quality hunting. So it’s also a bit of a choice to the teams. Okay, yeah, so I will take that with me and when I come to my client next Tuesday, because Monday is holiday. I will mention that, that we do such a quality hunting session. I’m really curious about how it works there. Yeah, well, if you want to know more, then you can read both the chapter in the book.

16:34It’s also on the TMAP website. And you can, of course, look in the new ISTQB syllabus, which is about to be released. I think it’s going to be released next week. So, yeah, that’s also so. Hopefully, very many people in the quality and testing world will soon learn about it. Yeah, that’s great. So, we will also link the books and the article in the show notes so that everyone can use this information too. Yeah, very good. Rik, thank you very much that you were here in the show again.

17:05Yeah, you’re welcome. With a new topic and a very interesting topic, a good inspiration for the community. Thank you very much and enjoy the rest of the conference. And see you in episode three with Rik. Oh yeah, definitely. Thank you. Bye.