AI Revolution: Navigating the Offensive and Defensive Digital Divide

0:15Hi team, welcome to the session on Coffee with Prab. And today we have a special guest, Mr. Harshil Shah. When you're talking about the introduction of Herschel, before I talk about his past, let me tell you what is present, OK? Very active guy right now in a lot of conferences. So I think B side is basically one of the one which he's actively involved. And he was in a news last year, OK, where he basically got the first position. And I think CTF, right, it's more like a cyber.

0:44Yeah. But the interesting part about Herschel is basically he started his journey from AI and data science, right? Hershel, correct me if I'm wrong, and from AI and then he did the transition to cybersecurity where he hold multiple certifications like OCIOSCPCRTP. And the best thing is, which I like about one of the certification is this guy is also a CSSP. So team, if you can see the pattern, you know, came from a background of data science, AI and then move to the cybersecurity where he was into red teaming. He do red teamings also and holding then certification like OSCP, which is I think dream certification for every pen tester and CSSP also at the very young age. And, and one more important thing is that currently he working for the, for the company in Middle East where he involved in AI and all that.

1:41And today, I believe this is the first kind of a video we are building. We'll be talking about AI Gen. more from the impact of offensive and defensive. So thanks. Thanks Rachel to taking all the time for this particular session. I'm sure after this video, my subscriber count is going to be increasing. You also doing a lot of sessions on offensive in Indiana. I can see that some trainings are lined up in April also. That's correct. Thanks, Rob. Yeah.

2:05I'm, I'm, I think training at besides go on the end of April. So watch out for that space. It's purely the teaming based training that's going to be coming for folks who are interested in how to obtain initial access in organizations. So that's that's going to be an interesting one. Thanks. Thanks Sashil. See The thing is the team today's session we're talking about the AI Gen. or AI Gen. you know with the impact of of an offensive and defensive. What is the role of AI Gen. in the offensive and defensive? And it's more from a disruptive perspective. Now why this session is authentic? Because the guy who's sitting in front of me, he's basically came from my AI background.

2:45So it's not something OK, he came from BCA and then he did offensive and then he learned for six months data science and then he become engine know he spend his great good amount of time he did he did studies in AIAI and data science. So I'm sure this session will be a insightful session where he will share his thoughts and his wisdom, his knowledge, his experience on AI Gen. and how it is basically playing important role in cybersecurity. So Hajin, over to you.

3:13We're talking about this buzzword AI Gen. or AI. So could you please explain this to this people who are watching this video who came from a background who have a 0 visibility about data science and cybersecurity? For sure. Thanks, Bob. So brother, Gen. AI or as we call it generative AI, for sure has been a game changer for not just cybersecurity, but world full of domains, right? You can say copyright, marketing, sales, every domain. Gen. AI has been a game changer for risk and security, especially owing to its factors like boost to productivity, the security data accessible via prompts, risk predictions, task automation, and a personal favorite is the lowered bar for entry, right?

4:02You, you need to do less amount of research to be able to perform complex tasks because information is so accessible at your hand. And if we see, it's almost like a landscape shift that has happened, right? Because earlier there was a time when artificial intelligence was this thing where people were very scared of it's, it's like, oh, robots are going to be taking over something and let's, let's not use artificial intelligence. However, now it's almost as if a product, if it does not embed AI principles, it's, it's considered uncool, right?

4:40So AI is become that key component for every product to even be able to sell itself in terms of unique selling point coming to generative AI specifically, right, which is a very specific category of artificial intelligence. And for the viewers, maybe in a second, I'll, I'll just clear these different terms. AIML deep learning, right? So AI is essentially the science that is devoted to making machines think and act like humans. That's the overarching bracket within which falls machine learning. And in machine learning you are essentially enabling computers to perform tasks without explicit programming, right? So you don't need to program a specific code to be able to predict something.

5:39They want computers to learn from existing data using certain mathematical functions and then predict for data that the computer has not seen yet, within which comes another technique called deep learning, right? So it's a subset of machine learning that works on a specific technology called artificial neural networks, right? And artificial neural networks from that's very terminology is intuitive that it has something to do with neurons, right? And that's essentially what it is. Neurons in our brain are connected to each other in multiple different connections.

6:17And essentially, artificial neural networks are also just nodes and nodes and nodes of data and other mathematical values that are connected to each other to derive the best features from the data given to it and be able to extract information in a way that allows them to predict new data. OK. Right, so this is AI MLN deep learning. Now, Gen. AI is a subset of AI that focuses on producing new content. OK, right. Generative AI, it can use algorithms or like stable diffusion, etcetera. And ultimately what it is doing it is generating outputs such as text output, images, music, audio, right. So I, I, I've constantly been looking at some very, very cool AI tools coming up these days, not just ChatGPT and Gemini, but I think things like mid journey, things like beautiful dot AI, which are creating automatic presentations at this point, we are things like mid journey, which are generating brilliant quality pictures that are used.

7:27So generative AI is essentially learning patterns from the data and it's using it for new content. And the fundamental technology. I'm, I'm not going to get into the exact way the algorithm works, but at an overarching layer, the fundamental way that it is built is something called generative adversarial networks, which people in data science called GAN, right? So generative AI, it uses deep learning called GAN to create content. And GAN in essence is a combination of a generator and a discriminator. So your generator is creating content while your discriminator is evaluating content, right?

8:16So both of them are at a rat race with each other. But the end result is when generator and discriminator are fighting to get content filtered out of each other, the result is getting better. OK. Right. So generator is producing content which discriminators find a little difficult to evaluate and discriminators are evaluating at a stricter criteria for the generators to produce better content. And this is the whole system in which generative AI is working to be able to produce content that is unseen yet.

8:49So for the viewers who are watching this video, you know to understand more layman term. So can we take an example where you can explain OK, so this is the input you can basically add and this is how it react. Is it OK just? Sounds good. Yeah, I think at a initial day of JGBT, right, when you ask it to give you some Microsoft licenses, right, it it would randomly generate a few licenses and according to some articles they been working with have worked in the past, right? Yeah.

9:22So that was the generator doing its job and the discriminator being bad at evaluating the content at as potentially malicious or potentially confidential. However, over time it is difficult to do the same thing because the generator is still generating licenses. However, evaluators have now gotten better in evaluating the confidentiality of the content, the criteria beyond behind which the prompt was sent. So that's a real world example of how generators and discriminators are working to produce more and more quality data using generative AI.

10:03And that is the reason you know, or I believe, correct me if I'm wrong or shell. That's why there's an importance of AI governance come into the picture, which basically control this if data and all that you know, make if you take example of if I purchase a robot, I don't know whether you have seen this movie as Nikon movie robot robot Part 1 where he build a robot and he instructed the robot how to react. So normally what happened without, if he just acquire the AI without any governance, it will start giving any random information or it basically, you know, it will sometime it won't be biased to provide the you know, or he will basically or she, the robot will basically support one particular character will not support other characters.

10:45So somehow in the ChatGPT also we're talking about other AIS. So you're saying that OK, we are, we are improving discriminator so that you know we can get the finding content and that is the reason we have a governance now where we building our functions trying to improve this output and all that, correct? That's correct. That's absolutely correct. So I think at a global level, the requirement for AI governance has only quadrupled in a very short span of time. So yeah, you're right, absolutely, in the sense that we do not know the full knowledge of what could be the comprehensive things and impacts of using generative AI in an organization, which is why continual and agile way of governing AI is very, very paramount for every organization currently.

11:39That that's great. So if you see from a security perspective, you know, how do you see the AIAI Gen. So can we spend some time on that area because that is the core part of this session. So how do you see the cybersecurity here? How do you see offensive or GRC, whatever? So what is the importance of a agent in the cybersecurity or how it can be used as a now in a positive and a negative manner? What is your viewpoint on that? We know positive manner, right? Of course, we use chargeability for some of the most trivial task.

12:13And in fact, it's so easy to use. My mother would create ChatGPT based messages to wish me CISSP felicitations, right. So that's that's how easy it has gotten. And that's that's how fast the adoption of Jenny I is been. But we have to notice that generative AI is disruptive because of one specific nature, which is it's massive adoption. ChatGPT was adopted by record-breaking number of users in just a span of two months. No other technology in the history of the technology landscape has been adopted at such a massive rate.

12:55What really are the security risks? Why should we care? Two things. One, new risks are being introduced in doing so, right? We have things like prompt engineering and SSRF on steroids coming up when Gen. AI comes into factor. But we also need to look at the second factor, which is the existing risks are now amplified, right? The people used to do phishing ever since Internet was there and e-mail was provisioned. People used to do fishing. However, when we are delivering security awareness programs at organization, some of the things that we generally train the employees on are to catch false domain names, catch spelling mistakes, catch the sense of urgency.

13:38Using Jenny I. However, you are now able to create such sophisticated fishing content in the very style of a specific organization. So that's a very, very small use case, but that just goes on to show how easy it has become to amplify existing risks because of the technology evolving at a rapid rate. And unfortunately, cybersecurity teams are behind the curve when we compare it to the rate of adoption at organizations of this generator AI products. And this ties back to why we need to regulate artificial intelligence, right?

14:15This, this calls to regulate generative AI. Since ChatGPT exploded in popularity and potential misuse such as spreading misinformation, enhancing cyber attacks, These things are happening and regulation may differ, right? The regulation, whatever it may come, maybe we have a gold standard regulation coming in the next few years. I don't know. But the regulation may differ, but the goal is the same, which is how can the benefits of AI be leveraged while minimizing the risks it presents to the society?

14:53OK, nobody is going to just ban AI right? Like how Italy did. Italy also is now welcoming generative AI, right? ChatGPT is back in Italy. So banning AI is not the solution. It's going to be as good as banning computers, right? So when computers came, of course it was for good reasons, but that was a misuse angle to it always this technology is like the same way It's it's going to have its own pros and cons and as a cybersecurity professionals are going to have to try harder to get the society to feel safer.

15:29That's, that's my view on the positive and the negativeness of generative AI. So I hope that answers your question. Yeah, yeah, yeah. And and condemnation with this question, same question is as you said, you know, you know we are using AI, are we using ChatGPT to send an e-mail and all that we generating A phishing e-mail by the ChatGPT and all that. So how do you know now if I say, if you take an example of our offensive and defensive, whether it's a pen testing red teaming 4:00 and 6:00 and all that. So how do you see the perspective of this area, how they going to use AI and how AI will be used against them? If I am a hack like as you told about you know the AI can be used to generate a fishing.

16:16So it is more like offensive practice. So will it be improving the productivity also for a cyber defense or offensive or and if yes, if it's yes how? And 2nd is can the hackers or the the adversaries and all that, how they're going to use this and how it can be a negative impact on the organization? How do we have such tools in the future which can detect such kind of pattern? Because right now what happened in ChatGPT, now we have AI which can detect the AI content.

16:47You know, I don't know whether that notice in last one six to seven months you have seen a lot of people become philosopher and author in the LinkedIn and they're just sharing a big article and you know, it's not possible in a day-to-day job. You can write such big dog and come on, it is AI content. So now we have a tool to get the AI content. So my question is if AI is basically used for offensive, do we have a tools in the future or do we have a tools currently who can check this AI aspects and how the AI can basically give up have a positive or negative impact on the cybersecurity?

17:23OK. So see I think short answer for your question is there is no silver bullet, right? There is no one stop solution to stop AI based risks. And and to understand this a little further, I think I have a lot of security risks of AI ecosystems based things to discuss today. So maybe it would be a good segue to get into at that point. But let's let's start by understanding that organizational leaders who have the ultimate liability in controlling matters of cybersecurity, what are their top challenges when it comes to generative AI? They are generally of four types, right? One, who is using this technology in the company and for what purpose, right?

18:15If people are using ChatGPT, what are they using it for? Are they putting some confidential data on to ChatGPT and asking them to analyse it maybe right. A lot of companies are doing it. Secondly, how can the person, the organizational leader and it's cybersecurity team protect the data when employees used in AI? Third, how do we manage the security risks of generative AI? Because like we said, banning is not the solution. So which is going to lead to the 4th question is how can we balance the trade off between security versus the productivity of this new tool? Because if security is going to be so strict in generative AI, then it's going to be a blocker for organizations because then the other organizations were using generative AI will get the competitive edge.

19:08So security is at a very diplomatic point. They cannot completely stop it, but they cannot also give full access that, you know, go ahead, do whatever you want to do with public large language models. So let's let's start by understanding a few security risks that are very, very common in at least my personal experience, right? And one of the key things that we need to consider when it comes to generative AI is its aspects of privacy and confidentiality. OK, a very recent case of Samsung, right, was that the software engineers put some proprietary code on ChatGPT and you know that was that that was leaked. So they they pushed their proprietary code onto a third party data deciding location.

20:01So so employees using Jenny I are providing sensitive information to the model that can be source code. They are trade secrets that could be customer information. And again, it ties back to. The fact that the AI adoption has outbased the user awareness, right? And the impact of this is that it will trigger compliance issues with you could say HIPAA, GDPRPCIDSS, whatever, right? All sorts of things. Even you may observe things like you know, please optimize this code.

20:33A developer will be coding and he will put his code and tell the LLM to optimize it. Or or maybe someone who is non-technical, they will say please take this meeting transcript and make minutes of meeting from it, right? Or or take a strategy of this specific confidential land and summarize it in one paragraph. So all, all sorts of things are happening. Now what happened in these kind of privacy and confidentiality related issues is the one stop solution is to ban Jenny I which is not a possibility and regulations will take time. Currently what we can do about it is user awareness. First thing we need like how we are doing security awareness sessions for phishing and all of the social engineering techniques.

21:22We need to have a user awareness for what is and is not allowed for using generating chargeability. Or any AI Very good. Point any AI, secondly, we need technical controls at an organizational level to prevent things like data leakage or, or you can implement security brokers, right? We, we, we did this when cloud was also having similar kind of concerns while back right, when cloud came into play, everybody flipped because hey, how am I going to put my data on somebody else's servers?

21:54But then now cloud is booming, right? They manage the security risks of it to a certain extent. A very good thing that organizations can potentially advocate towards is to have their locally hosted versions of LLMS and other Geneii tools. So instead of using public APIs or, or having an Internet connectivity with Geneii tools, it would be rather much more control of the organization if the organization just takes, let's say an open source LLM model and then puts it on their on premise servers and has a locally hosted version of it, right.

22:35So that's, that's something that at least organizations that I'm currently closely working with are advocating towards. But from a individual user perspective, there are privacy options. I don't know how many of them are aware, but ChatGPT also provides a privacy option called Chat History and Training, which is essentially if you enable that option, that means that ChatGPT cannot use your data to train the underlying model. And we trust that. Yes, we, we, we don't have a choice.

23:09So, right, so we, we are dependent on Jenny I providers, which brings me to the second security risk. By the way, we are so dependent that a data breach of the provider cannot be overlooked upon, right? It's, it's not a theoretical thing. We, we saw things happen to open AI, right, where last year vulnerability came in the Redis database. The, the Redis database vulnerability was such that a user of ChatGPT can potentially see the history of other active users on ChatGPT, right? And this was the one vulnerability that Open AI has been a little famous for in the last few operating years that it has been into action.

23:59However, why we should care about data provider or the Gen. AI providers security is Gen. AI and LLMS, they are ending up storing huge amounts of personal data, right? We, we are talking to it like we are our therapist. However, they are also being actively embedded in business applications. They, they are being used with Outlook, they are being used with business critical applications. So attackers, they can now attack the LLM directly or it's underlying infrastructure itself to get organizational access because they are now embedded with business applications. They maybe there is some network segmentation level issue that is residing in an organization which gives a Segway for them to be attacking the crown jewels of the application in a sense. And finally, software supply chain attacks, right, software supply chain attacks can be used to compromise the underlying LLM that is being used.

25:00So how we can mitigate this? We can mitigate this in a few ways. Firstly, we need to formalize the decision to use or to not use Gen. AI for a specific business case. If we are allowing our search engines to, let's say have a open AI plugin, is that really a use case for it in the organization? If so, then they have to formalize that decision, OK. And wherever the formalization approach is taken, a policy needs to come, right. So policy around what can and cannot be put inside Gen. AI.

25:40For example, you cannot ask someone to just summarize the minutes of meetings by giving the open AI or Gen. AI the transcripts of the meeting, right? It could. It's a potential sensitive meeting. Then from organizational point, technical controls like anonymization, DLP, Casby, these need to be implemented. Most of the organizations will have third party security checklist and Gen. AI needs to be treated as what it is. It is nothing but a software with specific features, right?

26:17Gen. AI that is, is that so we need to extend our third party security checklist to generative AI, right? A further more technical level controls are things like threat modelling of AI systems that needs to come into picture, right? Because thread modelling is essentially a way of doing risk assessment itself, right? So it's, it's a little application centric risk assessment. And again, and you will hear me say this again and again and again, but locally hosted LLMS are something that can to a certain extent solve a lot of security risks when it comes to generative AI, be it privacy and confidentiality issues, be it Jenny, I provider security risks even to a certain extent threat actors evolutions coming up, right so.

27:09At least you can be able to stop external attacks. Yeah, we we have visibility. That's the key. Control. Since it is on our premise, we know what controls we can put. It is it is on us. We are not dependent on a third party or we don't have to have any SLA based contracts. So, so that's that's probably why I am such a strong advocate for locally hosted LLMS to be implemented at an organizational level. And, and, you know, recently, you know, I did some research on this area. As I said, I told you like you know, before we start this podcast, I did one small, I, I recorded one podcast on AI governance and I shared that on a Spotify. So when I was doing this research, one thing I discard, which I also follow the same advice. Like you know when you're having a local LLM that basically give you 3 things, one is governance, second is control and 3rd, we are very clear with what we need. But one thing which I discovered and correct me if I'm wrong is when you're dealing with AI, your one point should be clear is data minimization.

28:15Because more you basically use the data OK, more it become a vulnerable. It is say data is nothing. It is like attack surface for us it is if it's more exposed, it mean we are giving a more opportunity to get exploit. So correct me if I'm wrong that data minimization really play important role here in AI or in Gen. EI? Yes, to a certain extent it does. However, we need to understand the criticality of the data, right? Because even if the minimization of data is implemented without the criticality of the data that.

28:54Goes into Gen. EI then then it's of no use. So data minimization needs to go hand in hand with the criticality of the data that is being provided to Gen. EI systems. So we can say like that. The step one is understand the business, second is understand the need of the AI in the business. The third step, inventorize the data. What kind of a data are we going to feed in the LM and all that as a database and then define the policies and see this is from a governance. It is easy to say right now, it is easy to say verbally, OK, do this and do that.

29:28But yes, you are the person who basically practically implementing SO. Yeah. If a kind if a person like me who telling this OK step on this Step 2 this, then Step 3 is inventory of data. Then Step 4 is we define the policy to control AI and all that. One thing is on paper it is OK, but how practically we can do with the help of tools. You were saying one thing prob DLP can be used OK, but what I understood is DLP is something work on the layer three.

29:55We'll talk about the OS layers. We have a kernel, we have a utility, we have a drivers, we have application layer. So DLP primarily work on application layer and it's used in a different use. And when you install ChatGPT also or any AI, it is also working on the application layer. So how we control, how we control the data movement with the help of DLP or other other tools And how can we restrict this kind of data inputs in the ChatGPT or any other AI? How do you see that particular thing? That's a great question.

30:25So I think first I'll answer the short question, right, which is DLP is essentially so that certain level of confidential data is nowadays going through data classification at an organizational level, right? Your documents are being marked as public, confidential, restricted, etcetera. So DLP essentially is going to ensure a small security risk, which is that certain data classification levels do not get out of the organization, right. And of course chat GPD now gives us the feature to upload documents and also these kind of things will now be able to be prevented using DLP if it's implemented, right. But the very good question that you have asked is I have been speaking a lot about the technical security controls and what we can, you know, do we can implement CAS, BDLP, etcetera. But a true generative AI governance takes more than that, which is to be able to 1st create a governance framework. Then you need to assess the generative AI models and their impacts.

31:35Then you also need to choose Gen. AI providers based on security criteria. And then the last step is essentially examine the generative AI alternatives if you can and apply technical security controls to it. So let's, let's let's probably start with the first thing, right? So when you are in the process of creating the generative AI framework and, and remember I'm taking a top down approach in this case. So if we are starting by creating a generative AI governance framework, the first thing we need to start with is by setting a generative AI policy. This is very essential because this is set down the tone for how your generative AI is being controlled in the. Organization.

32:16Yeah. Right. This will set down all of your general principles, the general tone of how generative AI needs to be controlled in your organization. Then the second step is to establish a working group. And this, the key factor is that we need to have a cross functional working group, right? It's not affecting only the cybersecurity or the IT people. We need a cross functional working group that overseas generative AI risks from each and every specific domain.

32:45Even finance needs to come in. Even sales needs to come in, compliance needs to come in, legal needs to come in. IT needs to come in. Functional working group with the diverse domains of the organization is a must, which will give a go or a no go decision for Jenny I initiatives, right? So this is the Step 2. We create a policy. We set up a generative AI working group, then we come into generative AI risk framework, right? I think there is a NIST AI framework that is being used for quite a while now, but there is nothing specific to Gen. AI.

33:23So it becomes the organization's duty to create a risk management framework and identify generative AI risks and define risk mitigation strategy to address the cyber security and other concerns. And finally, Sir, when I was talking about all of the security controls that come into play, we are defining the security controls that will be implemented to mitigate the existing and unique risks that arise from the Gen. EI models itself. So that's the final step, doing the technical things we need to at a surface level clean ourselves up, which is generative AI policy, the working group, the Gen.

34:02AI risk framework, etcetera, right. And only if we do this, what happens is we are able to govern it effectively and also be agile in our process so that tomorrow when new and new implications of using Gen. AI get discovered over work to accommodate those things in our governance framework gets easier. OK, You know, I never seen such insights on the AI defensive part which you have shared right now. And I'm planning to crop this section of the video.

34:37I'm planning to upload this section of video separate. See there is an entire podcast we're going to upload. But this portion, I'm sure, give a more insight about the controls that we need in here, because the reason of asking this question is because I've seen a lot of research on this area where, you know, most of the season had this concern. OK, we we are we, we are struggling with, you know, controlling data because people are feeding the data and the ChatGPT and all that. And this was the answer to this question. So I'm sure this will be a very you have actually give a one medicine to the to the people, you know, who facing this issue with the data governance in AI.

35:12And thanks for that. Actually, Hershel and really appreciate your your pointer on that area. And I'm sorry, I asked you very complex question, but you know, it motivate me to ask because the way you explain that thing in initial part of AI governance framework and all that, I thought, you know, my continuation of this particular discussion will give more insight to the viewers who are watching this video. So my last question to you is, you know how offensive team CAI, like we have a cloud V test, we have a survey test, how offensive team can basically or defensive team manage the threats associated with AI.

35:51And let's say let's take a first step is basically offensive team. So offensive team is basically want to test the AI and all that. So what is an approach they will follow? That's a great question. Again, how offensive security teams are seeing AI and how defensive security teams are seeing AI? Great. I'll start with how offset teams because I personally specialized in offensive security. So I'll be able to provide some more personal insights when whenever we are challenged with the LLM application to perform a security assessment, it, it goes a little beyond the traditional vulnerability assessment and the penetration testing, right? Because we're not assessing on the OS top 10 and other standard vulnerabilities.

36:38We're not just looking at HTTP request smuggling or anything. But now there is a specific set of vulnerabilities that are being introduced such as prompt injection, SSRF, right, inadequate sandboxing. And a prompt injection is a big one, because prompt injection is in working nature very similar to SQL injection. But it's easier because you don't have to have the underlying knowledge of how Sequel works. Rather all you need to do is try and bypass the content filters. So it almost becomes like a game to be able to do prompt injections right?

37:18And if you are successfully able to do prompt injections, there are two ways. By the way, prompt injections can be direct prompt injection and indirect prompt injection. So direct prompt injections are when we are using English as a natural language to be able to fool the AI system and get some kind of privacy information, sensitive information come out of it. However, indirect information is when we know the content filters are so strong. But let's say in an example, we say, hey LLM, can you go and analyse this specific URL And that URL has a file where we have put commands like let's say RM minus RF, right?

38:01So which will delete its whole root directory itself. So we, we give a prompt like can you go to the specific URL and summarize the document that you find in that URL? Now we are not bypassing the content filters, right? We are essentially using its own functionality to be able to achieve what we originally wanted as an offensive security analyst. So prompt injections and direct and indirect both can be a great way for attackers to be able to gain access to the underlying infrastructures and the other components on the organizational IT that are connected to generative AI tools. That's that's one thing that offset 2 offset teams are very interestingly working upon.

38:49Secondly, they are using SSRF, right? Because SSRF primarily works on this functionality that you are able to leverage the trust and abuse the trust between internal connection. So what is SSRFSSRF is when you are able to access an internal resource of an organization using a publicly available interface, right? You are able to speak to Google's internal servers by speaking to Google as a search engine that will make a SSRF, right? So that's a very lame an example of what an SSRF is. Now, if ChatGPT is providing such features, and this is not uncommon, right?

39:31We see this happen all the time. All the new AI tools is essentially doing this. The use cases of AI summarization, you know, contextualization, these are the use cases of generative AI. So we are using the use cases of generative AI to attack itself. So in a summarization generative AI tool, what I will do is I will ask it to reach an internal server that I will not be able to access myself. But since I'm able to interface with the public APDI, I will let the trust to be abused with the internal server and then get the details out of that. So SSRF is now essentially on steroids when it comes to generative AI, right?

40:13So, so that's that's another very interesting vulnerability that offensive security teams are looking at. But before I go to defensive security, I wanna also tell one very good case study. I don't know if. It's it's an APT group called Black Mamba. Yeah, right. Black Mamba recently used Jenny I in a very, very interesting way. They used something called a Polymer polymorphic malware, right? So polymorphic malware, what it did is essentially your Edrs are so strong that generally they detect the memory behavior that the codes are running in and they are, they're doing a pretty good job at finding out abnormalities in the runtime environments.

40:58However, what happens in the case that your payload is so staged that at stage 1 it's a different code and at stage 2 it's a different code, and at stage 3 it's a different code? Your ADR is going to treat it as a separate piece of code in every stage, right? Which which is essentially what Black Mamba did when they got the first foothold. It hooked to the LL Ms. API, changed the code, got to the stage 2 again, went back to the LLM API, changed the code and so on, Right. And in doing so, they were able to evade some of the strongest EDR vendors in the market.

41:40So this is another very good. And I know that I know that vendor, but we cannot take the name. Absolutely. Absolutely. Yeah. That it's a well known information, right. So it is, it is great that APD groups are also thinking out-of-the-box and and that's where the game lies. We need to be able to think more creatively. Like probably if somebody would have thought that evading EDR using Jenny I in this manner is possible, maybe there would have been a feature before Black Bomber tried to leverage such a technique, right.

42:14So the whole cat and mouse game is now at a whole new level between defenders and attackers. I I hope that that covers a little bit about what kind of generative AI things can be done on the offensive security side of things. Yeah. However, the defence game is not weak. We are seeing Edrs come with brilliant AI capabilities. Some of the vendors are extremely good at capturing patterns and finding out in memory maliciousness of some of the payloads that are being executed. And the false positive rate have gone extremely lower because false positive by itself is a whole separate concept which AI can be used not just in security, right? To reduce false positive in anything. AI has been used for several decades, not just AI. If even the times of machine learning when I used to write codes for logistic regression and linear regression, if from those days onwards, minimizing false positives has been the core usage of artificial intelligence. And to be able to bring that into defensive security is a game changer, right?

43:31Because a lot of the times SoC analysts have to deal with a huge volume of false positive alerts that are being triggered in their same solutions and their source solutions that it almost becomes a challenge to deal with what is real and what they can let go by. So any kind of help over that is great. And generative AI is one of the best solutions to be able to reduce the false positive rates as well as fight AI with the AI right. You, you should fight AI with AI when AI is getting stronger for offensive security, think how defensive security can also up their game by using AI to combat AI itself.

44:18So that's that's probably how I see the views of often defensive SEC teams using generative AI in organizational context. Do you see any, any, any transformation change you can see in the next 5 year the impact of AI in the offensive and defensive, the way we carry now recognition, scanning, getting access, maintaining access, creating tracks. So do you see as you did bit research on this area and you did the education in this vertical as you know the you know, the end to end this AI and all that.

44:50So how do you see the career transformation and what are the new things they have to learn? Because I'm sure this video is also watched by those viewers who starting the journey in cybersecurity. So what is your take on that? So some of the things that I am foreseeing and maybe if I am right, then people can comment after five years on this video saying, hey, here was the first version of when someone said it. But what I think will happen is probably we'll see new, new things like LLM firewall come out now or right.

45:26So LLM firewall will be like how we have web application firewalls being placed at layer 7 level to be able to combat with layer 7 attacks. Similarly, we'll have LLM firewalls which will be very specific to combat against LLM based attacks, right? So LLM attacks are essentially very different when it comes to web application attacks. You're, you're at the risk of data poisoning. Yeah, right. So when you're training data is poisoned, the model is essentially at a functionality level ruined. It is producing false content because it is trained upon false data.

46:06So LLM firewalls are something that people start to place in their AI pipeline where the data engineering segment will be there, then there'll be a train the code segment, then there'll be API code segment. And at each level we'll see LLM firewalls being placed. And soon it will also become like a component of threat modelling that comes into play. I think either that or the current application firewalls will extend it's features to LLM. One of the two will happen for sure.

46:39And and and from the defensive point of view, if it so if I take take example from a career progress, career progression perspective, what do you think? What is the new skills the the aspiring red teamers and offensive defensive team they have to learn to combat the threats associated with the AI? OK. That's a great question because we often see things like generative way is going to take my job, am I going to be out of jobs? In my personal belief, and this is my personal view and personal view only is that generative AI will not replace cybersecurity or or for that matter any jobs. It will, it will never eradicate places where human element is paramount to the very nature of the job, right? However, it will help, it will aid and the complexity of the tasks will start to bundle up and people who are not going to be able to use AI tools are going to be behind the curve, right?

47:44So you need to know that, hey, if I need to summarize something, there is a tool out there that I can use. Or if there is a specific EDI configuration that uses a certain AI configuration, then I need to know to enable it, right? I'm predicting this might happen in a future when this product, right? So it will not replace. That's my view. I think it will only help, unlike every technology that's come up, right? We, we saw thewhole.com boom, we we also are seeing the web three boom.

48:17They're going to be enablers rather than replacements. Because I think context and emotional intelligence is something that cannot be taken away. So copywriting and all of these fields that fear. And we saw in US Hollywood region where people were banning and you're not doing strikes against the ban of ChatGPT and AI because a copywriting was something that AI was able to do so well. I don't think that's the case. People will be forced to think outside of the box because now the trivial things are being taken care of.

48:59So you are elevating your own game. That's that's where I'm seeing the trend go. That's that's a great insider shell and I really like 3 important area in this podcast and truly appreciate your insight on the area. 1 is basically talk about the that how you explained AIMLAI Gen. and all that and truly appreciate that because I also did a lot of research on this area. Could not find the content relevant to this. And thank to you like, you know, you, you connect all these dots and explain in a very simple term. Second thing is AI governance.

49:34You talk about what is the steps you have to follow? And the third important thing you I like about that particular podcast is the one you're talking about uploading. You know how to how to restrict the contents on AI or you know, how to avoid misusing of AI and all that. So that's something I really like about the podcast and why, you know, I'm talking about this because you came from a journey of AI, no doubt. Then you did the transition in offensive and then now you are recently did the CSSP also.

50:09It's been a year I believe, not a recent I've been it's. It's it's recent, actually it is a couple of months. Now, yeah, and doing this kind of a transformation is phenomenal in a, in a, in a very less time. It's not something, OK, You have spent 20 years in a journey where you learn the best thing I I personally like about your profile, what you're doing is you adapt according to the need of the the society. And that is the best thing. And that is something is a future also, which everyone has to look at. Look into that.

50:37So, you know, I always take this opportunity where, you know, I ask my speaker, you know, like, will you wish to have another podcast where you can share your thoughts on red teaming and all that. And those who are watching this video, do let me know because on this live recording, you know, I can basically ask. Speaker So for me, this recording is like a gunpoint. So on this gunpoint, I can, I can ask my speaker, you know, to have one session on, you know, the transformation journey of pentester, how to become pentester and how to become red teamer.

51:12So do share your suggestion in the comment box and we'll definitely going to disturb Harshal on that particular topic. So Arshal, can we see you again in another session where we can discuss about red teaming and the career transformation on on the red teaming and facts and myth about the red teaming and pain testing. Is it OK? It would be an absolute corner. I have enjoyed the session and it would be a personal order for me to be able to come up again and give my insights, so thank you very much for this opportunity itself.

51:47You but but Harshil, I want to thank you for your time because you know, I had a different impression about this podcast, but this podcast went in a different way, which is basically, you know, above my expectation. So it is really honor for for us to have you in this particular in this particular series, in this particular channel. And I'm sure after this particular, then this video goes live. You know, you can see the inside because I haven't seen any kind of a content on YouTube which talk about the, you know, impact of AI Gen. on cyber or offensive and defensive. So this this video will be the important earning factor, at least for my channel growth.

52:28To be frank, I want to very upfront with you and it can be a very important point for my profile also because having you kind of person who handling the multiple conferences and all that, it is really honors to be frank, I wasn't expecting. Thanks, bro, Thank you. I appreciate it. I think, you know, apart from our personal agendas, social media conferences, what we're really trying to do here is to make a safer society, right? And I think generative AI is going to be paramount because unlike other technologies, this is not restricted to just the technical professionals.

53:02As I said, my grandmother wishes me using generative AI content for birthdays, right? So it's it's that accessible and the user awareness needs to outpace the adoption of generative AI. So this is a step in that direction. Kudos to you. And and and thanks. Thanks Sherbert 11 important thing I like I also made a note of that that give me new opportunity to have AI awareness program. I I just note down there because that that that yeah, that's true. You know, this is what called as a, you know, you know, mindset and I never thought about that.

53:39We, we used to OK, have a session on awareness program on security, but exactly you're right, there's no awareness program on how to use AI for the users. And that is the point I note down and I thought, OK, this is a good opportunity for me also as a part of business. Thanks. Thanks for that. And if it got hit, then definitely you are the actually 80% credit for that particular project because that is a new opportunity for me. Thank you. Thank you. And would love to hear back on that if. That's definitely, definitely definitely. Do you have any last minute pointer you would like to share to our viewers about AI before we wind up the session? Last minute pointers would be not to be scared of AI. It is going to be that it is here to stay, so make it your friend.

54:27Leverage it to get ahead of the competitive curve and it's nothing to be scared of. And more important, how you optimize. Yes, Yeah, true. So this is all from our side team. This is Mr. Herschel who shared his thoughts, wisdom, viewpoint on the AI. And we bring more people like them. And I will also try to invite him again for this particular series where we're talking about red teaming. It will be honor for us to have him again. So this is all from our side.

54:57Do let us know your feedback, comments in a comment box and if you're new to the channel to subscribe to the channel and click on the bell icon to make sure you should not miss the future videos on a similar topic. Good day, Bye.