Testing Complex Systems: Ship Small and Learn Fast - Jean Francois Riverin

0:00Speaker 0: Testing is not about preventing errors anymore in the complex world, it's to learn what works and doesn't work. Speaker 0: So learning isn't about having it right! Speaker 0: Have you ever seen on a Friday like today? Speaker 0: You put something into production... ...you did your best test that you can for development but suddenly you're called. Speaker 0: during the weekend I used ASDQB to get certified. Speaker 0: everything was looking at how-to tests Twenty years.

0:29Speaker 0: I said that what they do wrong because there's still a problem in production. Speaker 0: And not building bridges, we cannot talk to each other. Speaker 0: They are interacting but then software and no softwares. Speaker 0: also another software. Speaker 2: Welcome to Software Testing Unleashed. Speaker 2: My name is Richie and I have another episode from the Testcon V conference, From Mauritius! Speaker 2: My guest was Jean-François Rivery.

0:56Speaker 2: We talked about why software isn't just engineering Why our systems are complex like medicine or weather Not complicated Like a bridge While testing is learning And not being right. Speaker 2: What that means for next release. Speaker 2: That's all you hear in this episode and now enjoy. Speaker 1: It's great that you are here. Speaker 1: We're at the Testcon V in Mauritius on a great island and hopefully we can enjoy also the weather outside, the conference building Speaker 0: of course!

1:53Speaker 1: You had already your talk... ...here at the conference? Speaker 1: Yes Speaker 0: I have Speaker 1: And i saw your abstract translated it to German, I understand. Speaker 1: Oh you did? Speaker 1: It was in French that they abstracted! Speaker 1: And what is about...it's a new perspective for Agile or maybe Agile done right. Speaker 0: Exactly yeah..It's new because if you look at on the internet or chat GBT there's nothing about uh-what i'm talking about.

2:24Speaker 0: and something come into my mind saying are we right? Speaker 0: And we were testing and developing our software. Speaker 0: So the point was that you look at that, then you say why there's still problems with the software? Speaker 0: Why does it crash? Speaker 0: Why is the software still crashing? Speaker 0: Where are we? Speaker 0: always in chaotic mode trying to solve problem where we're supposed To know the answers and test properly?

2:50Speaker 0: We here have a test conference thing That if you should test well You should get right answer But it's not the case. Speaker 0: And I'm exploring. Speaker 0: why is that? Speaker 1: Yeah, so now we are curious! Speaker 0: It's where... I found out there was something you said were engineers but it looks wrong to me that they're engineers. Speaker 0: We start with a code. Speaker 0: Yes Engineering We build code and softwares. Speaker 0: But then Something happens Softwares get integrated into other software create an ecosystem.

3:27Speaker 0: He communicates with other softwares, you know? Speaker 0: And then suddenly maybe we're not in the engineering realm anymore Maybe were something more complex and it's why I use this framework to say that We are going from a complicated work where have best solution or best answer and switch to something that is complex More like ecosystem The medical, the living world. Speaker 0: Even finances like that or meteorology. Speaker 0: So if you switch to parenting then You don't have a right answer and maybe it's why we struggle.

4:06Speaker 0: We try not having the right answers but who are unable too? Speaker 0: Because in complex worlds where a doctor himself doesn't know everything he tries To see what your having give some good medication but sometimes it doesn't work. Speaker 0: And he has to understand why, exactly what we should do with software? Speaker 1: What are the typical solutions that we try to use in our complicated world for complex problems and why they're Speaker 0: not working... One of the things that ASQB says is you cannot test everything.

4:43Speaker 0: That's already something that should sound a bit better. Speaker 0: if you can know everything and you try to solve everything. Speaker 0: You're trying have the test at the end, but since it's not possible we are using our right paradigm. Speaker 0: so what I would do in a complicated world is that if i test with best way then all results will be answered. Speaker 0: Then we said that if I'm developing the software, architecture and everything And they give you the software to test.

5:19Speaker 0: You will be able to test it and should not have any surprise in production. Speaker 0: That's where were wrong. Speaker 0: then we're seeing errors in production. Speaker 0: It happens. Speaker 0: We don't understand why A lot of times testers are blamed. Speaker 0: Why haven't you tested this? Speaker 0: forecast everything that happens. Speaker 0: And when you look at the framework and saying that the problematic is complex, it's not possible... You cannot predict so it's unpredictable.

5:57Speaker 0: So its just afterwards That they add their right answer or no Or not? Speaker 0: So we need to test often Not one big development per year by month the smaller portion of code that you can put in production and test it, make sure it works. Speaker 0: The better it is because testing isn't about preventing errors anymore to learn what's working or not. Speaker 0: so learning doesn't mean having it right. Speaker 1: I think this is a huge shift.

6:32Speaker 1: we have to do pure engineers a lot. Speaker 1: We get all our DIN norm standards and all the stuff, we're optimized and automated everything... And if you are so deep in this complicated world that you have sophisticated solutions then it's hard to come there into complex worlds! Speaker 0: But do you have doctors? Speaker 0: Are they engineers? Speaker 0: No, don't call them engineers. Speaker 0: Like you're a farmer... You cannot predict what's going to happen.

7:11Speaker 0: You have go on the field every day and see if the weather or water or sun is growing. Speaker 0: in medicine It has to do with your doctor but it can not predict what will be happening for five years. Speaker 0: But as an engineer create a bridge and say this bridge is good for a flood, one hundred year flood but not a thousand years because I can calculate what's going to happen in one thousand years. Speaker 0: Because it's predictable you cannot do when it's unpredictable.

7:49Speaker 0: It's something that really different. Speaker 1: When we look at the complex solutions and paradigms you told about, it's more learning. Speaker 1: What are other perspectives that have to take into solve the complex problems? Speaker 0: The perspective of solving a complex problem is use what we're going with agile or devops. Speaker 0: It uses solution which says go in production often small do experiments try to fail too safe. Speaker 0: They call that an uncomplicated word.

8:24Speaker 0: in engineering, the failed-to-safe tool is designed with backup solutions to make sure it doesn't fail. Speaker 0: but for us its more safe to fail. Speaker 0: so uh... In our case It's another paradigm where we want to experiment. Speaker 0: We want this. Speaker 0: whether it works or not Its' not important! Speaker 0: We want to learn about it. Speaker 0: So if doesnt work If it small Doesn't create a big problem. Speaker 0: The important things in software.

8:51Speaker 0: now, it will be the complex word to learn about what's going to happen. Speaker 0: You don't know customer reactions so you need to go into production and say oh its working or not working... ...you have those big data with AI world that we're living in which can itself! Speaker 0: be able to change this is on cold and use the data that in a way, we don't know would never expected. Speaker 0: It means that were really complex word because uh... We cannot even predict what's going to happen.

9:23Speaker 0: so when you do it often see their results and adapt accordingly. Speaker 1: I think there are huge steps for people or teams old world or from the complicated, because there are still complicated problems maybe. Speaker 1: How can we support teams and how can we bring them to this new mindset? Speaker 1: This new perspective... Speaker 0: We need to talk about it I think is what i saw even myself. Speaker 0: It took me more than twenty years to realize that ...I'm not applying the right parenting Because I am kind of an analytic person.

10:06Speaker 0: I wanted to have the right answer, their right solution. Speaker 0: And when there was a bug and there's problem with software... ...I always ask myself what did i do wrong? Speaker 0: And use ASCQB to get certified in everything! Speaker 0: I was looking at how test for twenty years. Speaker 0: I said that.. What they'd do wrong because There is still problems. Speaker 0: production To figure out. Speaker 0: maybe I wasn't applying the right solutions taken another ways, just to think out of the box.

10:38Speaker 0: Not staying in despairing because you said that oh it's unpredictable. Speaker 0: and then look at what I developed. Speaker 0: five principles that i showed here today. Speaker 0: Use those principles of the complex world, saying that you need to adapt. Speaker 0: You have to save and fail safe. Speaker 0: Have the retroactions, feedback loops... ...you also get your metrics right in everything and monitoring. Speaker 0: So if you put monitoring metrics into data then you're able take decisions based on this data instead trying to solve all problems because it's unsolvable.

11:15Speaker 0: It is not something which can be predicted. Speaker 0: So you need to change your mindset, we need talk about it. Speaker 0: I don't see this kind of lectures in the ISTQB. Speaker 0: It just explains how do things and i think were on our own way. Speaker 0: if are trying continue that. Speaker 0: when we put the ICQB Foundation five years ago decided have a new foundation for zero And we put HIL and DevOps in it, but we didn't go and indeed.

11:47Speaker 0: We still have the engineering view of it into the STQB. Speaker 0: But we put some complex without explaining why DevOps is working. Speaker 0: So we said that use those techniques It will work like to only work if you had the right data and say that's just a great technique for the right problem. Speaker 1: I think when we look at teams who are doing their daily business And someone hears this episode Yeah, we should go there. Speaker 1: I think experience is one of the parts that a team can make some learnings and first steps into it.

12:25Speaker 1: When you look at testing what do you think would be good start when coming from an old world to say okay What could we get in order for us? Speaker 0: What really changed my mind while trying to use complex because i'm coming form those words What is a doctor can do? Speaker 0: So he will use all this knowledge to try and fix the problem, but just be humble saying that maybe it's not the right answer. Speaker 0: Even chat GPT says oh it's eighty nine or ninety five percent of the right answers... But never one hundred per cent are correct because they're in complex sport anyway.

13:05Speaker 0: so human relation. Speaker 0: So if it's not one hundred percent, we should be humble. Speaker 0: Look at what you're doing and make sure that your learning from your experience... ...and try to apply the best thing you can but knowing that maybe this may not be a solution in the end. Speaker 0: so go into production, try to convince yourself. Speaker 0: Sometimes the problem is the managers and business work where they say that you should predict, I have a budget.

13:37Speaker 0: You should go in the budget but... ...you cannot ask for it as a doctor or software. Speaker 0: We need to switch the paradigm. Speaker 0: we need talk about this so we can go into the business with our manager and say It's not working. Speaker 0: give them some... ...some way of working. Speaker 0: So give them freedom To learn. Speaker 0: Let your developers, let your software testers learn about what's going on. Speaker 0: They will come up with the best solution at that time but they only know afterwards.

14:10Speaker 0: so don't expect them to know before. Speaker 0: it is not how it works. Speaker 1: That reminds me of a discussion I heard last year in Germany where software engineer stated that we have this misunderstanding, the software is science and it's not science. Speaker 1: In Speaker 0: my term, what I said at the end of this conference is that in terms may come up with maybe not the best one. Speaker 0: It's a complex software world or not?

14:41Speaker 0: the bioengineers stuff. Speaker 0: but just want to highlight we're engineers because were creating code. Speaker 0: We are still engineering part it like a bioengineer software But a Bioengineer trying change function for cells and your body Then we doing that But we're doing more than that. Speaker 0: That's why the bio is really important, and it's complex. Speaker 0: so you should use engineering in a complex world. Speaker 0: Try to have your best solution but know this isn't maybe not the right answer.

15:12Speaker 0: You come up with the best one at the time. Speaker 0: like meteorologists They are never able predict five days upfront. Speaker 0: So... It was because of its complexity And even in finance if finance was engineering everybody would be rich. Speaker 0: Yeah, yeah. Speaker 1: When I'm totally clear with that do you have an argument for me which i can take and go to my CEO? Speaker 1: And say we have to change something because... Speaker 0: The argument is the chaos!

15:47Speaker 0: The argument what started at a conference was Have you ever seen on Friday like today You put some thing in production.. You did best test that you can ..the best development. Speaker 0: But suddenly your call during the weekend, there's a fire. Speaker 0: Your department. Speaker 0: they receive lots of calls and customer departments have lot of calls. Speaker 0: you need to fix things. Speaker 0: but even when you fix it creates another problem.

16:14Speaker 0: It is not predictable. Speaker 0: always in this rush because they don't want go into production anymore They say oh its not safe just wait for six months before going into production. Speaker 0: then theres big problems. Speaker 0: I roll back everything. Speaker 0: it costs a lot not using the right solution because when you're using complicated parenting in complex world, You always come back. Speaker 0: It's what the framework says.

16:40Speaker 0: and do the chaotic parenting. Speaker 0: so even chaos...you are on fire or extinguished fire to solve problem. Speaker 0: that is big problem and an urgent problem. Speaker 0: The argument is that if you don't do that, You will always have the same result. Speaker 0: Trying to apply the same solutions and saying that maybe I've not hired a right person Maybe Richard will be not the right person or someone else? Speaker 0: The problem would stay the same because it's fundamental.

17:15Speaker 0: It's not about knowledge Not your capacity for work. Speaker 0: And they say oh...it's unpredictable. Speaker 0: But yes! Speaker 0: So what can we do when it's unpredictable? Speaker 0: You look at it, you don't try to plan up front but not too much upfront. Speaker 0: We tried to plan just for the next release really small and do it. Speaker 0: so we need to convince him. Speaker 0: is that he's to say are you tired? Speaker 0: To work in a world where always have customers complaining fix-to-do The value of what were worked with?

17:48Speaker 0: we can Deliver his house What we're delivering delivering today because we'll have to fix both fixed problem. Speaker 0: go through customer support. Speaker 0: you really want to change that. Speaker 0: And changing it is to develop small, early and frequently... ...and then learn about what you can. Speaker 1: Okay, so now we have some arguments to go through our managers and see. Speaker 1: They Speaker 0: should listen to that! Speaker 1: Otherwise... We can send them this podcast episode And then maybe they Speaker 0: are going this way.

18:22Speaker 0: This podcast is. Speaker 0: there's no picture with him but I think That we can send a presentation Can show how it works. Speaker 0: be pragmatic, look at the world. Speaker 0: And it's called a Kinevin framework anyway so Gaelic word. Speaker 0: but this framework is something that exists from Dave Snowden since two thousand and was not really well-known in some industries. Speaker 0: But software isn't out. Speaker 0: It makes sense.

18:58Speaker 0: I'm just building bridges talk to each other. Speaker 0: They are not interacting, but I'm in software. Speaker 0: and now software talks. Speaker 0: another software. Speaker 0: there's AI inside of it. Speaker 0: so why not saying that oh what i can change how can i change my company? Speaker 0: To make sure the software will not break? Speaker 0: yeah we don't want that anyway. Speaker 0: That Speaker 1: is great. Speaker 1: maybe you could also put a link for your presentation into the show or something like this would be great for the community.

19:24Speaker 1: Jean-François Great that you were here on the show. Speaker 1: give us an insight from from the complicated world to complex worlds and give us some ideas or solutions for that. Speaker 1: Thank you very much, I hope you enjoy rest of our conference a lot on the island this weekend!