"My line that I use all the time is it's AI until you trust it. And then it's just automation"
Eran Leib (00:04.388)
So hello everyone. thank you for joining us for another episode of Backline Unfiltered. my name is Aaron Leib, one of the co-founders of Backline AI, and together with me is Aviad, my VP engineering and co-founder, and Hagai, our lead data and AI. And we’re very happy to have with us Bartley Richardson, our new advisory board member. And I’m gonna hand it over to you, Bartley, to introduce yourself.
Bartley Richardson (00:30.589)
Hey everyone, thanks for having me, guys. Like it’s great to great to be here with you all. I’ve known the back line people for a while now. so happy to happy to finally chat. yeah, like your mentioned, Barley Richardson, chief AI officer at CrowdStrike now. prior to that, spent spent around nine years at NVIDIA leading engineering for AgenTech AI, Agentic Toolkits, Cybersecurity AI. come from a CS theoretical computer science background, although
Haggai (00:57.887)
Okay.
Bartley Richardson (01:00.031)
You know, they don’t let me do any of that fun stuff anymore. right. but otherwise, yeah, hap happy to be here with you all.
Aviad (01:03.422)
Thank you.
Aviad (01:08.993)
Yeah, thank you so much for joining. and we have a lot of questions but and a lot of topics that we wanted to discuss. But first to continue exactly what you just said, we’re a little curious about the transition. you had a leading role, kind of a cybersecurity AI role in an AI company as Nvidia, and you switched to a
Haggai (01:18.718)
.
Aviad (01:35.764)
an AI role in a security company, which is CrowdStrike. can you explain a bit about that move, the current position and what happened during the transition?
Bartley Richardson (01:47.379)
Yeah, I you know, the the great the great thing, I I I think the great thing about the the transition is that at least from and this is just Bartley talking, right? Like but put from my from my point of view, having been at both companies, they’re they’re similar in a lot of very good ways. Right. Like, you know, I I think all of us know about know about NVIDIA and know about the culture there or know
Haggai (02:08.126)
you
Bartley Richardson (02:11.305)
about about the leadership and the strong leadership and the very technical minded nature, right, like of of NVIDIA, the company. there’s a lot of there’s a lot of ownership that, you know, is is passed around, right? Like at Nvidia, you know, that you that the thing that wins is the best idea, right? Like the right idea. And then we rally around and and then you rally around that thing. And I could say identical things about about CrowdStrike, right? Like and the the
the strong leadership and then George, you know, founder CEO is has done a really great job there. But the very technical, very rally around the mission, best idea wins, like culture, like, you know, it I think all of this when we’re we’re trying something new, right? Have this like, am I gonna fit here? Is this gonna have to relearn everything that I know about working with am I what’s gonna happen, right? Like you know a little bit going in, right? Like, but you don’t know all of it. I think it was it was really very seamless.
I think the the for me the rationale for the transition was was kind of twofold. One, prior to NVIDIA, I’d done work in the cybersecurity space, right? Like and so I was familiar with it. I didn’t come from a classical cybersecurity background, but was familiar with it from the from the essentially pure data analytics side of it. And and I I always wanted to
To get back to or always was trying to find a way, like how do I get back to solving this like or helping the community solve this very hard problem? I think I’m I’m talking to a bunch of cyber people now, right? Like, but you have one of the hardest problems. We have one of the hardest problems, right? Like I I can’t think of a problem set that that combines, you know, the the the heterogeneity of the data with the volumes that we see.
With the need to be right all the time, with the thankless job to be right all the time, right? Like I bet I bet no one calls you all and be like, hey, thanks guys, nothing happened today, right? Like no, nobody makes that call, right? So I think this problem is very hard to begin with. And then if you look at, you know, kind of the first part of this year and everything that’s happening, it’s a very exciting, terrifying, and exciting time to be in security, to be in cybersecurity. And so
Aviad (04:06.322)
Yeah. Yeah.
Haggai (04:08.507)
Thank
Bartley Richardson (04:24.434)
with the you know, the the the obviously the cup company cultures match and and then it was a lot of I I wanna focus, you know, focus is a superpower. I wanna focus on cybersecurity now and really helping the industry at this at this like w whatever this inflection point we are now with AI.
Haggai (04:30.842)
Okay.
Aviad (04:43.474)
Yeah, exactly. and to your point, bringing deterministic flows into AI agents because we have to be right in cybersecurity, I think that’s a very big challenge that all of us face. so a topic I would thought I think will be interesting is how can we trust an AI agent for security product? ‘Cause right, we don’t want it to do the wrong things.
Haggai (04:49.714)
you
Aviad (05:13.063)
what changes when agents have to be right about security specifically? When I chat with an agent, it’s fine if it will make a mistake, it makes no effect. so where exactly the do we draw the line? And also who draws the line? Is it the engineer, the legal, the product, the customer? So like I would say what what helps us to bring deterministic flows into AI agents for security, how do we make them
Right all the time. that’s an exaggeration, but still. Yeah.
Bartley Richardson (05:46.248)
Yeah, that’s the that’s the that’s the billion dollar question, right? Like if you could like wave your magic wand and like turn this probabilistic, non deterministic system, right? Like into into a a tree, right? Like that has a nice graph. Like I I think the answer you’re you know, we’re not gonna we only have we have a limited amount of time. We’re not gonna answer everything about the the question. I think you know, to your latter point about who makes that decision, I think that’s a that’s a share of responsibility model. You know.
Aviad (05:52.592)
Yeah.
Haggai (06:00.919)
Thank
Thank
Bartley Richardson (06:14.514)
different customers in different verticals, right? Like regulated versus unregulated, like different regionalities, different size, scope, mandates. They they’re gonna have different risk tolerances, which which we’re familiar with, right? Like that’s a a security one one. I think part then the rest of the onus is going to go to who are the trusted partners and trusted cybersecurity operators in the environment, right? There’s a reason that companies hire all of us, right? Like is because I I kind I kind of
When when when coding agents and all that kind of stuff first came out, there was this notion for a hot second that’s like, cyber’s dead because everyone’s just gonna roll their own cybersecurity, right? Like protection. Like, could you imagine? Could you imagine every company trying to implement their own? It it would be it would it’s it’s it’s an interesting thought experiment, right? Like, but like, you know, it’s it’s but anyway, so so who are those trusted partners? They’re gonna have to make some of that some of that risk calculus for them. There are there are what I tell people is is there are
Aviad (06:56.527)
Yeah.
Bartley Richardson (07:13.218)
there are ways to make systems more trustworthy and more deterministic without you know needing needing full level five or whatever autonomy, right, like that we have today. And I think part of that does start like you really like you pointed out, part of that starts with like we give people that continue to give people a chat experience, but gradually and gradually introduce more autonomy, right, like into these systems. and so I mean you know, an example of that is
you can have what what a lot of companies do, like what what we do is when we’re doing triage, right? Like something about stock triage, we can assign it a level of complexity, a level of of of criticality. There’s a multidimensional metric that you can assign it. And then if you’re able to self-remediate and close out the ones that you are are under this barrier, right? Like that that gets you part of the way there. I think the the way that we then move beyond that is we give the operators
visibility and trust and kind of pull back what exactly is happening like in this system. Because my my line that I use all the time is it’s AI until you trust it. And then it’s just the time there’s just automation. As soon as you trust the AI, right? Like then you stop even thinking about it like AI. You just think about this automation. So there’s the there’s the there’s the UX part of it. And then finally there’s the there’s like the technical part of it that you mentioned, right?
which which I think there’s a lot of ways that you can we can do this. Certainly we’ve been already experimenting and seeing success with even early LLMS judge, SLMS judge, all of these types of things. think about other ways that we could we can institute councils right on top of like more latency more more situations that are less latency sensitive. When you get into super high super high latency scenarios.
that’s where you have to, you know, you’re gonna have this stratification, right? Like of of capabilities. You’re gonna have inline capabilities that that can happen that just SLMs and LLMs and even some machine learning models just can’t operate at that speed right now. Right. Like and and the the ability to then dynamically change that content that you’re operating at speed, I think is going to be very important. and then for the computer science part, continuing to work on
Bartley Richardson (09:37.342)
How do we distill, quantize, or some new technique that we haven’t even thought about, like mixture of experts plus plus plus, where we solve some more of the activation parameter issues, push that down as far, as close to the agent as humanly possible.
Aviad (09:52.521)
Yeah exactly. and that’s maybe Hagai we talked about it, but that brings me to my next I think theory or question about guardrails when it comes to security with agents. Yeah. so ha ha what’s your take on this one?
Haggai (10:06.148)
Yeah.
Bartley Richardson (10:11.272)
With guardrails with the security agents? Yeah. I look, there’s there’s a like everything in security, there’s a there there’s a layered approach, right? The the onion or the parfait, right? Like of security, however you feel about it, right? guard guardrails will get you will get you so far, right? Like they’ll definitely the they’ll you know, you should you should guardrail around your prompts, right? Like you should do all these like types of things.
Aviad (10:13.46)
Yeah.
Bartley Richardson (10:40.668)
The agent, the agent becomes a little bit trickier because the the very things that give an agent power, that that give it, make it useful, are the things that present security issues with it, right? Like its ability to adapt, its ability to change its tactics, right? Like and do things, right? Like, you know, You ask your agent to go and accomplish a task for you, if it gets blocked.
it will try another way, right? Like to accomplish it. I like to describe these agents as like the ultimate goal seekers. They’re like the ultimate golden retriever. They’re the ultimate golden retriever. They they want to do everything they can to make you happy, right? Like they’re going to get the piece of information. And so and so once you once you start to restrict their ability to do these things, you you’re you’re kind of diminishing their their capability, right? Like so I think there are there are things we do with agents that go,
Aviad (11:14.086)
Exactly.
Aviad (11:21.224)
Yes.
Haggai (11:22.404)
You
Bartley Richardson (11:36.959)
kind of like beyond guardrails. There’s architectural things, right? Like we make sure that we make sure that the system prompt, right, that the agent was was spawned with or that its very core existence is not writable in the same space as the agent is in agent space itself. Injecting secrets and keys only at the agent boundary, right? Like is another thing that we do. doing things like where we’re you know th th things like what CrowStrike does with Guardian where we’re
Aviad (11:40.67)
Mm-hmm.
Bartley Richardson (12:05.758)
Combining process level data with it with prompt level data, right? So we can get a fuller picture of what the agent is doing, even if it spawns something, you know, even if maybe it spawns something outside of its sandbox where it’s not the runaround. I I think that’s the that’s really the the key, is you have to have this dynamic nature to it, right? Like you can’t put these like, you know, static, like the static garbage work really well for like, hey, work kind of well for.
Hey, I I don’t want my chatbot to like curse at me or I don’t want my chat bot to talk about my competitor products or I don’t want I don’t want to allow any type of this type of like prompt injection like into it, right? Or that kind of thing. But with agents, just the dynamic nature of them, but we have to find a more dynamic way of doing this garbage.
Aviad (12:53.82)
Yeah, it it’s the balance, I think, the balance between the sandbox, right, and what the agent can or cannot do. So it might think that it can do everything, but with the sandbox we can actually limit that. So like I I don’t think it’s r resolved yet. It’s a matter of trial and error, I will say.
Eran Leib (12:53.956)
I got a thing.
Bartley Richardson (13:11.868)
Yeah, a hundred percent.
Eran Leib (13:12.866)
Think I think there’s by the way very interesting nuance in in Aviad’s previous question that I I just realized. We’re always talking about agents and we’re saying how do you or it it actually depends from which side you’re coming, right? When you’re the customer or y when you’re the user, so to speak, then you’re saying, How do I know that it’s not doing something wrong? And what we’re trying to do is to make sure that it makes the right thing. So it’s kind of, you know, like they’re afraid, you know, like in many cases. And by the way, you can see that it’s it’s it’s shifting, you know, like in the last year it shifted
Aviad (13:41.435)
Shifting.
Eran Leib (13:42.623)
dramatically. Organizations that would never think to to walk into a w to into AI are, you know, like bracing it, you know, like insanely. And and you see that, you know, like it it the the discussion is shifting from how do I know it’s not doing something wrong to how do I make sure that it’s doing the right thing. Which is it’s a very interesting shift.
Haggai (14:02.381)
Yeah. yeah. So I would add to it, I think it’s not just a matter of doing the right thing or doing the wrong thing. And I think what we need to embrace is agents saying, I don’t know, right? So I can’t really resolve this problem. I don’t know if it’s true or if it’s an intrusion or it’s not, I’m not uncertain. And this is where we need the agent to be able to respond that.
Aviad (14:02.798)
I I think Hagai Yeah, go ahead. Sorry.
Haggai (14:31.852)
And when we do that, we need to optimize for force negatives, right? We can’t really accept agents to be mistaken saying something that is really bad happening, but missing that. So we need to be able the agents to always be correct when they say true, but being also be able to say, I’m not sure this is something that I haven’t been trained on or is, I don’t know, out of my…
train data set or I’m just uncertain of that. And I think the products, the security product that we’re building needs to be acknowledged to that. So we need to be able to deflect things to human. We need to be able to explain when the agent really is unable to respond to question.
And I think it’s not just one last thing. I think it’s not just about the autonomy. It’s also about the scale, right? So we can now detect, we can classify exposure, we can say it’s exposed or not exposed, we can fix at scale, we can do a lot of things at scale using agentic flows that we couldn’t have done before.
On top of that, there’s the autonomy layer. And I think we’re going to get into a moment where there’s just not going to be enough human to being able to classify and to say everything that is right or wrong. And what it leads me to, we need to really be able to evaluate the quality of what agents are producing, being able to really say the agent should or.
A critic agent should be able to say, okay, this is right, this is wrong, and we need the product to align on top of it.
Aviad (16:33.687)
So that yeah, go ahead about yeah.
Bartley Richardson (16:34.386)
Yeah, I that a hundred percent. A hundred percent. Like I I yeah, exactly. I think that and that’s the that’s the good that’s actually good news though, right? Like because like I I talk to people it, you know, it it’s it’s it’s even better because then to provide value, you don’t have to be right all the time. You don’t have to you know what I mean? Like even even if you’re even if you’re sure that you’re right, right, like and you can automatically you you know you’re right when you’re right, even if that’s
thirty percent of the time, even if that’s forty percent of the time, right? Like, you know, that’s that’s forty percent more than maybe than what you were that’s maybe twenty percent, thirty percent more than what you were doing right now. And the abilities of the rest to say, no, I don’t know, and then systems we pay to put in place to like co evolve and and improve that, right? Like so you’re constantly like moving this needle, I think you’re spot on.
Haggai (17:27.729)
Yep.
Aviad (17:28.149)
So how do you know that the agent has got better in its decisions? And I’m always thinking about it in the scope like of security issues. So, you know, you have those tri to you can try edge and you can suggest, but when it actually act and you know, we cannot ask it, did you stop the breach? Right? So how do you evaluate that to see that you go from thirty to forty to fifty seven?
Bartley Richardson (17:57.426)
Yeah, how how do you evaluate the breach that you stopped that never happened? Right? Like, you know, how how do you right? Like how do you yeah, right? It’s this and I so so
Aviad (18:01.693)
Exactly. Exactly.
Haggai (18:03.045)
Resume.
Eran Leib (18:06.735)
That you produce value and you can never prove
Bartley Richardson (18:09.374)
Exactly. That’s like cybersecurity in a nutshell. They’re like, well, yeah, now people will ask, Well, like, well, how are you protecting me? And they’re like, well, d did you get breached, right? Like or whatever. Yeah, there’s like yeah. And and it’s only an issue till it’s an issue. No, I get it. so so I think there’s you know, there’s there’s a few things there. Like one is I I’m not gonna say we don’t use them, but we should make use of the open benchmarks when necessary, right? Like to prove out tests.
Aviad (18:09.887)
Yeah.
Bartley Richardson (18:39.292)
I I have a few issues with with open benchmarks. I I think they should exist. I’m glad they exist, right? Like I think that you know there’s a place for them. they can really easily get get benchmaxed. I don’t think a lot of them are measuring the right things, right? Like, you know, because you can you can win a leaderboard and not stop the breach, right? Like to your earlier point. You can win a leaderboard and not stop the breach.
Rob, we’re also focusing a lot of the benchmarks are focusing on on coding vulnerabilities and coding exploitabilities. And I personally think for the customer, if we go back to the customer, the the real proof for them is like, well, how well did the agent, yeah, yeah, yeah. The agent did you your your thing did fine on this benchmark. Okay, great, like whatever. How well does your thing operate in my environment, in Sitio? Right? Like that’s the that’s the test really that we have to.
Aviad (19:30.384)
Exactly.
Bartley Richardson (19:31.218)
contrive, right? Like as a not contrive, but we have to come up with right, like as a as an
Haggai (19:35.167)
No benchmark will address that, how it works within your organization.
Bartley Richardson (19:38.515)
Nope. It can’t, right? Like it can’t because everyone’s a everyone’s a unique snowflake and everybody has like all these things in it. And every business cares about a different part of it. And it’s just the dynamics of it are so high that that the real test, and and I think we we even said this at Falcon, one example of the real test, if we think about think about defensive posture, is if I give the attacker the entire architecture, if I give it
all of the information if I give it how I hardened the environment against the attacker I hand it everything like that and that attack still cannot succeed right like I’m like that’s the that’s the panacea right like I gave you every piece I gave you everything that you needed to be successful and you couldn’t do it and then so that’s what you know at if I go back to my previous job that’s you know at NVIDIA we have this saying called speed of light we operate at speed of light
And and it can mean fast, but it also means like what’s the best you can do, right? Like what’s the best you can possibly measure? And then how close can you get to that? And I love this as a philosophy, right? So like speed of light in this case is like I give you every single thing about my environment, I tell you everything about it, and this attacker, however sophisticated, still can’t get in. Like that’s the that’s the speed of light, right? Like that’s the top. And so then I’m always measuring, I always want to try to measure.
Aviad (20:49.082)
Mm-hmm.
Bartley Richardson (21:05.936)
Some’s quantifiable, some qualitative. How fast, how close can I get to that speed of light, right? Like and how fast can I get to it? but that I think is the real test. It’s like if we’re measuring, it’s a measure based on, you know, a a factor of the the customer’s environment and what are they seeing to themselves? What what are they seeing?
Aviad (21:29.562)
Yeah, and having the agent being adaptive enough to connect to everything, to get the right context, because there’s a lot when you come to you know to very large enterprises, and to understand exactly what they need to do, that’s the challenge I think of agents when they come to fix security issues, you know, in scale, to Haggai’s point. I I I think also about the scale issue, specifically maybe in your current position.
Bartley Richardson (21:51.454)
Yeah.
Aviad (21:58.692)
You have a lot of propriety data that’s coming from the security tools. so how do you bring that information and leverage that to make the agents better? Is it like a pre-trained model? Is it a knowledge base? How how do you leverage that?
Bartley Richardson (22:15.442)
Yeah, so so we talk a lot about that. You know, we we Crowstrike is the the CrowdStrike platform sees sees a a lot of a lot of you know just a giant amount of data. It’s enough it’s enough data. We have enough data in the platform daily to stream two billion hours of four K video. Right. Like so that’s how much gets gets ingested, right? Like in into the platform. you know, we we see, you know.
Haggai (22:28.981)
Thank you.
Bartley Richardson (22:45.054)
trillions and trillions of events per per day. And so it that produces value, it produces challenges, right? Like data, data without data without the ability to operate the data or data without the ability to make use of it is is just data, right? Like it it doesn’t, it doesn’t mean anything. but we we see it across not just customers, but we see it across adversary groups. It it lets us do a few things. One, we we we announced at Falcon we have a series of
we have two classes of models one for offense one for defense so red tempest and blue solano it lets us take those models and and rl align them sft them however far it however far I said those backwards but however far down the staff we want to get into them the the trick there is not just having the data but having the data in a a way that is labeled appropriately for the technique you want to do.
Right. Like if you’re trying to like SFT on pair data, then you need the pair data, right? Like with it. Some we can label autonomously, but then some we also we have five hundred plus threat researchers, right? Like that are just working on this. So we can use as as the threat researchers are going about their day, what they do is they generate, they generate more of this data. They generate how they’re doing their their work, how they’re trying to find the adversary, what pieces of information they’re looking at.
This was something, this wasn’t something, it turns into this whole case, right? That then we can go back, ingest RL align against right, and even use it for some of our harness development, right, as well. And the same thing happens on the on the remediation side, right? Like where we have full end-to-end remediation ticket visibility that we can kind of look at again at at scale. that and you have to be careful with that, you know. you you want the right data blend.
Aviad (24:22.998)
Mm-hmm. Mm-hmm.
Bartley Richardson (24:39.432)
for these models that you’re that you’re fine-tuning in your post-training. yeah, you wanna, it’s this, there’s a little, there’s some science and then there’s art to this. There’s art to pre-training, there’s art to post-training, there’s art to alignment, and there’s this data blend strength.
Aviad (24:54.419)
Yeah. we we see that with remediation, I have to say. When you get all the data and then you create a knowledge base of remediation playbooks, where you see things that are happening, you know, with customers, with open source, etcetera. And then you need to understand exactly one you want what you want to keep, what you want to train, so the agent won’t think that, you know, one size fits them all and how they can be dynamic enough. Hagai, you want to add on top of that?
Haggai (24:54.718)
So.
Bartley Richardson (25:17.181)
Yeah.
Haggai (25:21.188)
Yeah, I want to say something about being right and evaluation. so, as you said, the agent is kind of a goal tree. He will try to satisfy, he will try to find the path to whatever you’re asking. And I think what we are embracing are just…
Classical benchmarks that are internal. So metrics that are coming from machine learning world, being able to really classify the correctness of our agent to really understand whether the plan was working as expected or not as expected, generating a golden data set to make sure that we really know what is the right answer.
taking the best of machine learning classic data science into the world of AI agent. And I think that’s what… So I’m not a big fan always of the big benchmark because the LLMs just learn these benchmarks and they are just being very good at them. Not because… Just because they learn what they should do. And the internal data and the internal…
Bartley Richardson (26:34.994)
Exactly.
Haggai (26:37.889)
assets that you have, these are the ones that really you want to leverage when it comes to two layers. One on the, as you talked about correctly, the remediation and the processes and bringing the right context into the time. So you’ve got, I don’t know if you’ve got the same data as a CrowdStrike, but we do need to fetch and to feed the agent with the best knowledge to do the task right.
But we also use our own proprietary data to evaluate, to know that our agents are not overfitted on our specific task to evaluate the correctness of our remediation and our detection problems. So bringing machine learning classic, data science classic metrics into the agentic world, being able to measure this way, these are proven stats.
that we’re embracing and that’s what gives us confidence that our remediations are really correct and we can automate a lot of things. So scale is getting tremendous and as scale grows, we need to really be certain that our solutions are correct and the way we do it, we pre-evaluate to make sure that our agents are correctly doing and we are in
in process evaluating to make sure that our results are really correct.
Bartley Richardson (28:12.221)
Yeah.
Aviad (28:12.43)
So to to challenge that, and also to Bartley’s point, detection and response got automated, triaging got automated. When we say, like you said, Bartley, it’s AI, but now it’s automated. Now r remediation, right, is coming to the world of AI agents, but it’s not automated. So what are we missing? What is the technical problem or is it like a conceptual problem of an organization?
Bartley Richardson (28:26.108)
Ha ha.
Aviad (28:40.748)
And how can we move forward from AI agents trying to remediate to make it a full autonomous flow?
Bartley Richardson (28:48.466)
Yeah, that’s the and and I think it’s hit, you know, we we’ve hit a little bit on it, you know, already is we’ve we’ve been we’ve been advantaged in these other areas because we’ve had the sum total of the internet, right? That that all the great frontier open source, closed source models have been trained on. So it turns out a lot of the a lot of the patching and vulnerability and exploitability writing, it turns out that’s software, that’s coding.
Right, like and and it turns out we have models that are very good at like coding and things like that. When you get into like we’ve been discussing, when you get into so like so like so so when you get into like automated the finding of the vulnerability, automating the exploit writing of it, right? Like of that, I think we understand that. You can you can control that in a very safe way. You can explode the exploit, right? Like inside of a container, but we’ve been doing that for a long time, right? Like with with Mauers.
and then when you get to the triage part, like we mentioned before, that makes sense too, because especially if you can bias it towards, you know, I I only want you to, I only want you basically, basically no false negatives, right? Like or whatever, like in this thing, and and make sure the make sure the things that that you are closing out that you are triaging away are actually not things, right? Like or that you’ve remediated it. we don’t have to get all the way there, like we can have parts of automation.
But but to get to the full like remediation of it, one, I think structurally we’re missing trust, right? Like as just a not us, but as a society, right? Like as a larger. Because you you all know and I know this was before we got involved in cyber, but like, you know, cyber c companies have been promised AI by cyber for decades before AI was it w back when the only papers existed were from the sixties, right? Like and it was like
Aviad (30:22.245)
Mm-hmm.
Aviad (30:34.435)
Yeah.
Bartley Richardson (30:38.898)
All this ML stuff and look at all this magic that we have. So we have to be very cautious against that. Because now I think for the first time, for the first time, we really do have the capability to go down this like automation like route and really be autonomous. but we’re dealing with, you know, some very valid trust and observability and kind of transparency issues that that we so that’s on on that part of it. I think the other the other part of it i that there’s two other parts. One is education.
And I say this for all the analysts, right? Like that are listening. It I see this a lot where like you’ll get a system and it’ll be like, it you know, it it it told me it didn’t know. and it gave me it gave me the wrong answer. I hate it. I’m never gonna use it again. Right. Like and what I try to convince anal analysts and these people that are doing this work, I’m like, What you saw that as like a false positive or whatever, or whatever it was being contextualized as. I’m like, but think about in the course of you doing your investigations, how many paths do you go down?
that turn out not to be the right path. And then you come back up and you have to do this like thing. I’m like, if we think about it in in the same type of term, like, well, it’s exploring this path, right? It’s doing all these things. It said this thing I didn’t I don’t like. I don’t agree with. I’m like, yeah, but but could you imagine doing the triage in that way? So the second is education. And the third it is technical, right? Like we we need a we we need ways and whether these are whether these are like better versions of consult models that we have.
Whether this is, I think it’s an ensemble type approach where we like not only just assemble ensemble SLMs and LLMs, but we ensemble more deterministic kind of features on top of models or aside models. there there’s voting capabilities, right, that happen that can happen. there’s a lot of like things like this in the environment. There’s
There’s new research into different types of models, models that go past the transformer architecture, right? I think those are those are slightly interesting. but we need to pay attention to those. There’s models that are existing today that don’t output in human readable format. And so these are also going to be kind of kind of issues that we come up with. But but I think I think if we can come up with and I don’t have a magic bullet for you, but but a a a metric for how well
Aviad (32:47.85)
Mm-hmm.
Bartley Richardson (33:02.334)
What’s the whatever you want to say? ROI, KPI, right? Whatever you want say. that that our customers are seeing in their environment as we as we deploy AI, right? Like inside of it. And it must be commiserate with the cost. I think part of it will be we just we we put so much trust in the system that that that will come naturally and that will come through through visibility and these other technical deterministic parts that we put on to on top of.
Eran Leib (33:30.795)
Yeah.
Aviad (33:33.472)
Couldn’t agree more. Yeah.
Bartley Richardson (33:34.524)
It’s a layered it’s a layered strategy. I I wish I had a magic bullet for the for the technical part of it.
Aviad (33:39.008)
It’s it’s not. It’s layered. That’s what we see. That’s what we do. Yeah.
Eran Leib (33:40.709)
I have one last question, which is kind of you know like I’m gonna say I’m gonna ask Bartley, not necessarily Bartley and CrowdStrike. and it’s more of you know like and think about it not as a prediction in a way. Where do you see
Bartley Richardson (33:58.558)
Yeah.
Eran Leib (34:08.257)
AI insecurity, where do you want to see it in like I don’t know, like a few years? Where do you see the future going?
Bartley Richardson (34:15.24)
my god, a few years? Right, like how how you how you gonna get past six months? just
Eran Leib (34:18.622)
I don’t know, not a few. Not a few years. Yeah.
Aviad (34:19.262)
Le let’s let’s start with twelve months. Yeah.
Bartley Richardson (34:25.928)
Yeah, right. I I think, you know, what one of the things, and this is gonna this is gonna sound terrible. I think I feel I feel like I’m gonna do it injustice. one thing is the the ability for and I hate this word, I hate it, but the the there’s this digital twin aspect that I I I like to call them these like complex complex simul complex on demand simulations, right? And
And I look at this as critical for us being able to trust and allow an agent to go out and do the work that it needs to do while enabling while enabling the trust to do full remediation, right? Like if I’m able to not just simulate, but able to actually carry it out on like live VM at scale 10,000 plus, right? Like infrastructure network.
in in a way that’s cost effective, in a way that that makes sense, I can I can essentially replay a lot of this. Now I hate the word digital twin because it makes it sound like it needs to be exact. And what we’re finding is that it it doesn’t. If you can if you can emulate or simulate the right parts of it, you can instrument the the desired effect. And now with these long, let’s think long horizon, long, long running agents,
We have this ability to harness all of these traces that are coming out of our environment, feed those back into different scenarios that go back into a different ways to remediate, right? Like testing and scale out of testing. So I think we’re gonna we’re gonna see renewed interest in whatever the world model is of the cybersecurity network, right? Like if I borrow from from from Jan Lacoon’s like vision for the world model, right? Like
What’s the version of that that we’ve never been able to achieve that we’ve we’ve wanted to achieve for so and we’ve done it in parts, right? Like we’ve emulated this part, network we can emulate a little bit, right? Like all these types of things. But to have true behavioral, adversarial, like this type of thing as is this again, that’s speed of light, right? Like how how close can we go? I I think it’s that. I think models, models capabilities will continue to be commoditized, right? Like we’ll continue to see.
Bartley Richardson (36:41.788)
better and better and better open weight and open source models like come out. We’ll continue to see better and better frontier closed models come out, right? Like with their own capabilities. It won’t be a it won’t be a closed versus open. It will be a closed with open strategy, right? I believe go I I really believe that going forward. and then and then more and more of the the work more and more of the work will be done. We’ve seen this throughout time. You know, we had all of these
Agent frameworks and what did they do? They patched out all the problems we had with the models, right? Like, and then we see as models get better, we need lighter and lighter and lighter frameworks around them, right? Like we’re doing harnesses with markdown files now, right? Like and all these types of things. We’ll see that progress, right? Like, and then the real kind of like work then becomes on how how are companies engineering their harnesses to best take, best utilize, and best.
Aviad (37:23.492)
Mm-hmm.
Mm-hmm.
Bartley Richardson (37:40.13)
implement models that they maybe they’re fine tuned or models that they’re pulling off of the off of the shelf. So that that part will be will be continually will be continually important, but we’ll see parts of that get consumed by the model.
Eran Leib (37:57.631)
Amazing. that’s a that’s a very good answer. thank you very much and thank you Hagai and Aviad for joining as well. And we’ll see you in the next episode.
Aviad (38:11.257)
Thank you.
Bartley Richardson (38:11.325)
Yeah.
