Software Security Gurus Episode #20: Larry Maccherone

Matias Madou:
Welcome to today's Software Security Gurus webcasts with Larry Maccherone. Welcome, Larry.

Larry Maccherone:
Thank you for having me. It's a pleasure to be here, Matias.

Matias Madou:
Absolutely. Hey Larry, do you mind sharing a few words about yourself?

Larry Maccherone:
Sure. I currently serve as the DevSecOps transformation lead at Comcast, but my background includes product line director at Rally software. I've been a serial entrepreneur most of my career as well. Did a stint in academia at Carnegie Mellon because I thought I could change the world with a position there and it didn't necessarily pan out. I'm also an active developer, I write code just about every day. I'm the primary author of a dozen open source projects, one of which gets 800,000 downloads a month and I pretty much try out all my tools and ideas on my collaborators and my own code basis before I bring them to Comcast and other places that I help out.

Matias Madou:
Very nice. I like it when people have real world experience with open source and even day-to-day coding experience, fantastic.

Larry Maccherone:
I don't think you can tell developers you're doing it wrong with any credibility unless you're actually doing it yourself and so that's why-

Matias Madou:
Exactly right. By the way, I think application security, the way security is moving is application security people, they have to be able to code. They have to be able to show what it looks like and what it should look like so a hundred percent points.

Larry Maccherone:
A lot of them are hired based on those four letter, five letter security certification acronyms. It's funny, I had to train my HR people that I will accept someone with zero security background if they had interest in it and they've been developing products where it matters whether they are secure or not. For instance, I didn't have a security background and I graduated with an electrical and computer engineering degree. My first startup that I launched while I was still an undergrad did process controls, but we controlled 60% of the world's power generation with our software via our primary customer which was GE power generation.

Larry Maccherone:
If we had vulnerabilities, it was the death of our business so we basically did security in the course of doing a good software engineering and those are the people that I want as folks that think of security as an aspect of doing good software engineering.

Matias Madou:
I could not agree more. What sparked this conversation was actually, I read an article about you in Business Chief, while at the same time I had written an article that you picked up in SC magazine and I have to say, we seem to have a lot in common. By the way, we will share the links afterwards in the description of the webcast here. Let's go through a couple of things if you don't mind.

Larry Maccherone:
Sure.

Matias Madou:
In these articles, we can both agree that moving through DevSecOps is on the one side, it's hard, but at the same time, if you just DevOps, right, well, you automatically include security. I see that with two aspects and I would like to touch on the two topics here, the first one is people and culture and then the second one is tools, but let's start with people and culture. Without people, without having the culture of writing secure code, it's not going to happen. You have a keen interest in social psychology and you're an engineer, tell me more about that. I'm super interested in that.

Larry Maccherone:
I mentioned my first startup and that we basically were developing vulnerability free software. We actually thought of vulnerabilities as defects and so we were essentially producing defect-free software. There's a high safety element in addition to a security element with power generation, so bugs that cause it to maybe not be attacked by a bad guy, but still not provide power to an operating room are pretty bad, plus, the things can vibrate themselves as death and blow up. Any bug was considered a problem and so I launched another company, a spin out company called Qualtrax that that was targeting this framework, marketing this framework. Carnegie Mellon got wind of that, invited me to be on a few panels and eventually said, "Come on, let's launch an effort inside of Carnegie Mellon to take your framework and change the world with it."

Larry Maccherone:
We did that, I went to Carnegie Mellon and we launched this thing called the CyLab. I launched this thing called Build Security In and we fundamentally failed to change the world. Out of the frustration of that failure, I basically said, "What was it missing?" At the time, the agile movement was taking off and they were having a lot of success in hooking the minds of developers and development teams. I actually changed gears and went to go work for an agile company rally that's where I learned how you actually get people to change their behavior. I became a student of The Prosci's ADKAR model. I started reading everything I could about behavior change and I think I took the ideas further than generic behavior change to behavior change that's very specific to develop our mindset and develop our team social interaction kind of thing.

Larry Maccherone:
The pull request for instance, is the best hook you could possibly ask for and most security tool vendors and security people think of that as an afterthought. "Oh yeah. We have a pull request something or rather somewhere here." That's my primary path into their workflow.

Matias Madou:
Why is that so important?

Larry Maccherone:
Well, think about it from the developer's perspective, right. Their new heartthrob is the code that they wrote this morning. They're in love with the code that they wrote this morning and they want nothing more than their friends, the peers on their team to say that their new heartthrob is handsome or beautiful. They will jump through lots of hoops to defend their choices that they made in that code they wrote this morning. If you give them instant feedback in the pull request that shows that they have a security vulnerability, they're going to read up on what that means. They're going to follow all the links to watch the video, they're going to make sure that that word gets removed from their new heartthrob before they present them to their friends.

Larry Maccherone:
Then, they get the endorphin rush of the pull request getting merged into the next higher level branch and it just reinforces that going forward. Compare that to the way it's typically done, at the best case scenario, it's usually three weeks later when a security person sends them an email, it's out of their workflow. That was my old heartthrob three weeks ago, I'm on to a new heartthrob, right. If it's code somebody else wrote three years ago, they could care less about it. You really have to find a way even to sneak in those old vulnerabilities into the pull request cycle and we've got a tricky way of doing that ourselves. If you touch on any code that, even if it's not in the lines you wrote, but it's calling code that has a latent vulnerability, then it will show up in your pull request as something you need to fix before that pull request get accepted.

Matias Madou:
Oh, okay. You make it their problem, essentially.

Larry Maccherone:
We make it their problem, even the old problems we make them-

Matias Madou:
You make the old problems the new problems. Excellent. No, it's interesting that a lot of people miss that because essentially what you're saying is it always has to be current. What they're working on, what they're doing right now, what's on top of their mind right now is what interests them and if you can weave your security... Whatever you want to tell the developer, if you can weave that in when it's current, when he's in the flow, then you have a lot more success than when it's just outside of his normal day to day flow.

Larry Maccherone:
Right, and the mindset's different. If my team is doing its job right, we're essentially invisible to the average developer. Sure, we have to engage with the team to get them to add a few lines of YAML to their pipeline so the scans start running, but my people never pester them about individual findings. They basically get the feedback directly from the tool.

Matias Madou:
Can we agree that people are the most important factor in getting security right?

Larry Maccherone:
Yeah, but it all ties together, right? It's a triangle, it's so hard to say that. It's the one that has gotten the least direct attention and needs more direct attention. When you say most important, it's the one we probably should work on the most.

Matias Madou:
I agree. If you look at vendors in this space, it's all about tools, tools, tools, and there's not a lot of people that think about the human aspect. It's still a person that is doing the thing, it's not a robot, although they are quite often controlled by machines.

Larry Maccherone:
They are thinking about a human when they're designing their security tools. They're thinking about the guy who's going to write the check that's in the security specialist group at the enterprise that they're trying to sell to and anything that appeals to that person is what they're targeting. A true Dev-First security tool looks fundamentally different and behaves fundamentally different, is built with a fundamentally different mindset. You can't just take an existing traditional mindset tool that was built with that mindset and slap on DevOps like lipstick on a traditional security pig, it doesn't result in a true Dev-First experience for security so it's very hard.

Larry Maccherone:
I think that's a big dilemma for these vendors, is that the group that has the money is not the people that really have the biggest influence over the security of the products and developers are notoriously stingy about spending money on tools so it's very hard dilemma for a tool vendor. I don't envy them, I want them to succeed the Dev-First few guys out there, but they're struggling I think because of this, the money is all on the scary side. That was one of the fundamental reasons my Build Security In initiative failed, is that all the energy and money and attention on security was done by security specialists and I was trying to appeal to developers 17 years ago when I launched Build Security In, it was just way too early to appeal directly to developers for that.

Matias Madou:
Yep. Couldn't agree more. The money is not with the developers and security team, different budget and so on and so forth. By the way, I saw your lipstick on a pig picture, you actually posting that on LinkedIn, I liked it. I liked it a lot when people gave some comments, you said, "Oh, is it like this?" And, "Yeah, it's like that."

Larry Maccherone:
Feel free to use that, it's a good little graphic. My daughter did that by the way.

Matias Madou:
People on the one side, let's talk a little bit about tools, let's touch on tools. More tools will not resolve the problem so what can we do then? If it's not more tools, what should we do about tools?

Larry Maccherone:
Here's the dilemma I see all the time. You have someone with budget, right, security specialists group that has budget, some vendor approaches them and says, "Hey, you're missing all of this really bad stuff and we can surface it to you." My reaction to those people is, "I already know about a bunch of bad stuff that I can't get resolved rapidly. Enough until you help me with the resolution side of it, I don't care about finding new stuff. In fact, finding new stuff actually increases my risk." I'm I'm now out actively selling that and I had data to back it up. My talk at RSA had a little bit of touching on that, but I've been talking about it more since the RSA conference. This was the 2020 RSA conference, the last one I was at in person physically before COVID hit that...

Larry Maccherone:
That isn't going to help me to find new stuff and the risk is higher if I know about it and I don't resolve it, than if I never knew about it, because either court is going to say, "Well..." Like Equifax, right. "You knew that you had a major vulnerability in the spring boot framework five months before the attack, why didn't you do anything about it?" We'll, that's a really bad position to be in court.

Matias Madou:
Good. Back in the day when I was at a static analysis vendor, we knew about the company in Germany that actually stopped scanning their code because if they knew about the problems, they had to remediate the problems and their solution was to just stop scanning. It was not going to figure out how to fix it, but they just stopped scanning.

Larry Maccherone:
Here's our thing, we spoonfeed our internal clients a little bit. We launch with one tool and we have the dial turn down really low and we drive resolution. If their sister team says, "I want the tool too." I'm like, "Well, get your sister team to resolve all of theirs and then we'll expand inside of your organization." They're like, "Why am I dependent on them?" Well, because we're investing in tools not because we want to find things, we don't want to find things for the sake of finding things, we want them resolved, and that group is not resolving them and so we wasted money on your sister team. We're wasting licenses on them, and we're actually increasing our liability risk. Until we do that, we're unwilling to expand the footprint inside of your organization."

Larry Maccherone:
The same thing with a single team is we give them one tool with the dial turned down really low, we will turn up the dial. We want the dial to a certain level before we say, "Okay. Well, here's the second tool that could help you find more things and here's the third tool that can help you find more things." They want all three tools day one, and they're like, "Well, let's start scanning and then we'll worry about resolution later." That's completely the wrong strategy.

Matias Madou:
I love it because all that is like you want to measure towards your end goal and your end goal is shipping code reliably in a fast way, that's your end goal, that is your business goal. Your business goal is not how many problems can I find because that doesn't line up with ultimately the business, what you're doing is you're putting metrics in place. I love the suggestions that you're giving over here, you're putting metrics in place that aligned with that end goal, which is shipping reliable codes.

Larry Maccherone:
There's a guy I work with who says, "The use of metrics management by metrics has been the most dangerous thing in history." The Enron was essentially that... That Enron scandal. I'm showing my age, most people aren't even familiar with the Enron scandal. A little more recently, but it's still probably showing my age a little bit was Wells Fargo, right. They had fake accounts being created because the account managers were measured on how many accounts were being created, not on how successful they were making money on those accounts because there was a dollar in each of those accounts or something that they personally put in, but they got rewarded based on that.

Matias Madou:
The metrics have to align towards the business goals.

Larry Maccherone:
Yeah.

Matias Madou:
Scanning is slow, how do you tackle that one? What is your solution resolution, because ultimately again, the same thing you do not want to stop developers. We're quite often seen as the blockers, the stoppers and scanning is there and it takes long, lots of false positives. Were you able to overcome that hurdle?

Larry Maccherone:
Yeah. Let's tackle two of those things you said separately. The false positive one we can tackle separately, let's hold that one for a second, let's let's talk about scanning speed. This is not a problem for us. I'm not really sure why it's such a big problem for everybody else, but I have some ideas. Let me describe how we do it and then I'll suggest why I think people may be going wrong with this a little bit. We set up all our teams to have a baseline scan run for SAST, a baseline scan run every night and then during the day their pipeline based scans are incremental just on the files that were changed by that particular PR. That runs fast and there's no risk.

Larry Maccherone:
Now, if somebody runs a tool that changes the header of every file and it ends up in a pull request, that scan's going to be like a baseline scan, but that's rare. That doesn't happen all that often and so 95, 99% of the time they're running an incremental scan and for that 1% to 5%, it's not a big deal. Now, SCA, software composition analysis, is actually a bigger bang for the buck for us, I estimated 10 to 20 times better ROI than SAST. Even though there's a 40, 50 year history of running SAST tools and everybody's trained as a security person, get a SAS tool, you're really much better off getting an SCA tool. We mentioned Equifax earlier, but Equifax was essentially SCA output and those tools run much faster because all they're doing is looking at a manifest of the dependencies you're using, and then looking up in a database whether they have publicly reported vulnerabilities. Again, they run instantly almost and so they're not slow at all and that's what we start people out on with that.

Larry Maccherone:
The third tool we use is IAST and IAST, basically adds a 10% or so runtime penalty to your integration tests. If you're running integration tests, which you should be as part of your pull request, anyway, that 10% delta is usually something they don't even notice is happening and so we don't have a problem with scan times. Now, DAST is a completely different animal. We don't have a DAST offering in our program because of what you're talk... Or, a fuzzing offering in our program, we now are exploring a true DevOps oriented DAST fuzzing tool called Mayhem. It's out by a group called ForAllSecure. It actually won the DARPA cyber grand challenge award, the million dollar award, as the best. Everyone thought a SAST tool was going to win that and it turns out this DAST tool won the competition.

Larry Maccherone:
It has a really slick way of combining nightly, continuous oriented background scanning with an in pipeline element. That is really the first of its kind and I think it can fundamentally change the way people think about DAST tools not being very conducive to DevSecOps. I think it will be very conducive so we're exploring that, we're hoping to get a pilot going later this year on that.

Matias Madou:
Nice. Interesting. False positives was the second one.

Larry Maccherone:
Yeah. We track false positive rates for every team and we pay very close attention to their resolution curve and their finding curve and their false positive curve. I could show you a visualization that shows it, and what we see is that the finding curve starts very steep, goes up very fast, when we first launched the tool with them. The resolution curve takes a big step function after we coach them and motivate them to actually start resolving the legacy things so we get them to a point where they start... What happens maybe 30% of the time, is the false positive curve will jump up to 20, 30% of the resolution curve and when that happens, we immediately schedule a meeting. We're watching these, we have automated sensor that detect this, and we schedule a meeting with the team and we analyze those false positives with them.

Larry Maccherone:
The traditional security mindset is that, "Oh, they're just marking them as false positives and get them out of the way. They're being bad and we should go police them." But, we have never found that to be the case. They are always had valid reasons for marking them false positives and the really great news, the best news of all of this, is that the vast majority of time, like 75% of the time, we can make a small tweak to the rule set. Now, we only buy tools that allow us to tweak the rule sets, right. We can make a small tweak to the rule set that specifies that this sanitizer is a valid sanitizer and the false positives go away and so those curves stop growing over time for those false positive curves for that team.

Larry Maccherone:
You have to do this tuning for every team and we have to do it because the one we have right now is Checkmarx based tool and it's really a great static analysis tool, but their rule customization licensing and their tool usage is really, you have to go through a three-day class to even be able to use it, but there's a culture changing. Code QL, which was acquired by GitHub from Sammle, they acquired all of Sammle, is essentially crowdsourcing the rule set, you can define your own. They're open source to all the rules, so you can... Developers, it's now code to them. They can then go modify the code and the rule set and so I'm hoping that we get to a place where development teams feel comfortable modifying their own security rules, just like they feel comfortable modifying and creating their own tests right now.

Larry Maccherone:
Remember when there was a QA department and it had automated tests for you and then it shifted the responsibility of the development team. I am hoping that things like code QL and Appirio is another great example of a company that's exposing the rule set to folks in a way that makes it really easy for them to tweak them and really match their context very well.

Matias Madou:
Actually, I couldn't agree more with that. I always liked tools that are customizable and they are essentially a framework, they come with a lot of things that-

Larry Maccherone:
Appirio actually markets themselves as a framework.

Matias Madou:
Yeah, it is. It comes with a lot of rule sets and recipes and whatever, but ultimately it comes down to your code base, what you want to get out of it and it needs some tweaking. People that still think, "Well, I'm going to spend some money, drop it on the teams and it's going to run, then it's going to find stuff and it's going to be the good thing." You still need to manage it, you still need to tweak it. If you make that investment going to get so much more out of these tools, especially in the DevOps way.

Larry Maccherone:
Yeah. Imagine a developer where we had a meeting with them, where we showed them how you could customize the rule set and we did that in 20 minutes, quick little part of an existing meeting thing. Imagine them then getting a pull request the next day, where there were three findings, two of which were real, but one of which was a false positive, and they want all three of them to go away because they want their pull requests to get accepted. "Well, yesterday they told me I could tweak the rule. Wait a second, let me go pull up that link and let me tweak the rule set. Oh my gosh, I'm done. Now, it's got a clean bill of health." That's the virtuous cycle I'm trying to get teams to where they own, not just their code and the quality of their code, but they own the security of their code and the tools that they use to validate that security.

Matias Madou:
That's exactly how you scale yourself and that's exactly how you make it everybody's problem and that's how you get everybody rallying around your initiative. Love it, I actually love it. Larry, I actually have one final question and you already alluded to it. I had a look at CyLab and more specifically, the Build Security In the content catalog have to say, "Oh my God. I think you were super lucky fortunate to work with some great people in that area." I saw Gary McGraw on there, I saw Ken-

Larry Maccherone:
Gary McGraw, Noopur Davis, who's the CSO of Comcast and brought me into Comcast and I were the three people who launched the Build Security In initiative. Now, Gary went off to make a lot of money on it and he created BSIMM out of it and Noopur and I went on to the agile movement actually.

Matias Madou:
That was going to be... My question was really, what is the unique experience like? How was that and what kind of thing came out of it, and what did you learn during that whole journey?

Larry Maccherone:
Well, this has been true of my career before and since then, I'm always ahead of the market. Sometimes I'm a year ahead, sometimes I'm three years ahead. In the case of Build security In, we were probably 20 years ahead of the market. It was a message that developers should own the security of their products when they didn't even own the quality of their products back then, right. That didn't happen until the agile movement and it was a QA department that owned the quality of the product. It was just a really tough message to do that so I basically have worked harder at not just defining what the future should look like and building it one example of it, which is what I did with my own startups, but I'm actually trying to figure out how do you get from where you are to where you're going.

Larry Maccherone:
Interestingly enough, Bill Gates was the master of this. He didn't just have a great vision, that envision included all the steps we get people from where they are today to where I want them to get. I've started to do that more than I used to and that's why I'm more active speaker on public circuits and author and that sort of stuff.

Matias Madou:
Fantastic. Larry, thank you very, very much for accepting this invitation to be the 20th guru on our Software Security Gurus, webcast.

Larry Maccherone:
I'm the 20th.

Matias Madou:
You're the 20th.

Larry Maccherone:
Is there a special star for that?

Matias Madou:
I'll send you some special slack, yes. How about that?

Larry Maccherone:
Okay. Matias, it was great, thank you. Let's keep complimenting each other on LinkedIn and Twitter posts too, I think it was really funny.

Matias Madou:
Absolutely.

Larry Maccherone:
You found an article about me and I found an article posted about you and that's how we connected, it was just a interesting coincidence.

Matias Madou:
Absolutely. Really appreciate the chat, thank you very much.

Larry Maccherone:
Thank you. Take care.