Software Security Gurus Webcast Episode #6: Steve Lipner

Matias Madou:

Welcome to the Software Security Gurus Webcast. I'm your host, Matias Madou, CTO and co-founder of Secure Code Warrior. This webcast is co-sponsored by Secure Code Warrior. For more information, see www.softwaresecuritygurus.com. This is the sixth in a series of interviews with security gurus, and I'm super pleased to have with me today, Steve Lipner. Welcome, Steve.

Steve Lipner:

Thank you, Matias. Glad to be here.

Matias Madou:

Thanks. Thanks for accepting it. Do you mind saying a few words about yourself, Steve?

Introduction

Steve Lipner:

Not at all. I'll try to be brief. So I actually started working in computer security, what we now call cybersecurity, 50 years ago this fall, working as a defense contractor for the US government in the Boston area. Spent the first part of my career worrying about trying to build provably secure operating systems. Some work on firewalls with a little company called Trusted Information Systems.

Steve Lipner:

Joined Microsoft as the leader of the security response team that dealt with discovered product vulnerabilities in 1999, and was at Microsoft from '99 until 2015, when I semi-retired to become the executive director at SAFECode, which is nonprofit that works in the software security space. My big claim to fame, the thing that I did that actually still lives on is the work on the Microsoft Security Development Lifecycle and the various successors and approaches that have come out of that initiative.

Matias Madou:

Absolutely So first of all, it seems to be a theme on the Security Gurus Webcast that people that retire do not really stop. We had Gary McGraw in the first episode and he was terrible at retiring and seems that you're in the same boat.

Steve Lipner:

Yeah. A friend of mine from my government contractor days said he had heard through the grapevine that I had flunked retirement.

Trustworthy Computing, Security Development Lifecycle, and the Code Red worm

Matias Madou:

Yes. So you've just mentioned it, it's almost 20 years ago when actually Bill Gates first wrote the Trustworthy Computing memo, two years after that in 2004, you and Michael Howard published a book called Security Development Lifecycle. My best guess is that in 2001, the computer worm called Code Red was one of the triggers for the Gates memo, which then triggered the SDL work. Is that correct?

Steve Lipner:

Pretty correct. 2001 was just a terrible year with Code Red, something called Nimda, which was another big worm that impacted the IIS web server, and another vulnerability called UPnP that affected XP clients wasn't actually ever exploited to my knowledge but got a lot of publicity. And so, in 2001, a couple of things went on. One of them was that Craig Mundy, who was one of our chief technology officers and Howard Schmidt, who you probably know of-

Matias Madou:

Yes.

Steve Lipner:

... late Howard Schmidt, like great Howard Schmidt, were working to try to influence the company and convince Bill that we needed to step up and make a commitment. That was the initiative that came to be Trustworthy Computing. Then the other thing is that after Code Red and Nimda, the small team that I led at that time was trying to figure out what we ought to do about security.

Steve Lipner:

We proposed initially to my manager in the Windows Division that the thing to do was to stop all development, train everybody on security and take some time and actually go through the Windows code base with all the developers to try to find and eliminate places where we had not done security right.

Matias Madou:

Really?

Steve Lipner:

The initial response to that was a little on the skeptical side, as you can imagine, but we just kept going back and answering questions. And so, in mid-January of 2002, both the Trustworthy Computing email and the announcement that we were going to stop Windows development and focus on security, came pretty coincidentally.

Steve Lipner:

I think that was very important, it was a combination of Bill's commitment and the terrible publicity from Code Red and Nimda and the like, together really influenced all the developers to say, "This is important. We're going to take this seriously." So we did what's now called the Windows Security Push, stopped all development for two months as it turned out. That was the release that became Windows Server 2003.

Matias Madou:

Two-

Steve Lipner:

Then ... Go ahead.

Matias Madou:

Two months? So I'm not aware of any other organization that just stopped development for two months and focused on security. That's pretty drastic.

Steve Lipner:

We did that with essentially all the major projects, all of the major products in the period 2002, early 2003. It was a brutal but practical way to sort of address the sins of the past. Then as we learned lessons from that series of security pushes, and when the executive management, we decided that we had to have a permanent process and basically bake security in forever, the lessons learned from those security pushes were the things that we picked up almost without modification for the first version of the SDL. And so, we launched the SDL in July 1st of 2004, and Michael's, and my book came out in '06. And so, it's actually based on one or two revs of the SDL after the very initial one.

Matias Madou:

Oh, so it was 2006 that the book came out? I thought it was 2004. Okay.

Steve Lipner:

The SDL was launched in 2004, it became mandatory for all products, July 1st of '04, and then the book came out a couple of years later.

15 years of changes in the SDL

Matias Madou:

So in these 15 years that it has been like a lot of evolution and change in the SDL?

Steve Lipner:

Mm-hmm (affirmative).

Matias Madou:

I was reading through a couple of the interviews that you gave over the course of 15 years. I think you pointed out three significant changes since that SDL book was written. I would love to dive into it a little bit deeper. So the first one was on diversity in programming languages.

Matias Madou:

The second was on speed, meaning like the SDL moving from Waterfall to DevOps or Agile. The last one was the supply chain or third-party code being used. So maybe let's start with the first one, like diversity and programming languages. As far as I know, the goal for the SDL was like single programming language, but that's no longer the truth.

Steve Lipner:

Well, so at the time we created the SDL basically it was for Microsoft, and basically Microsoft, we wrote code in C and C++ and some C#. Okay? And so, if you had rules for those languages, then you were pretty much done. The SDL, one of the things that made it work is that it was very specific for developers. Run this tool if you get this error message by specific number, then do this to fix it.

Steve Lipner:

The combination of the tools and their training enabled them to do that. So you didn't have to have the developers be security experts. They had to know what to do, which was a much more inclusive bar. SDL, a secure development process still has to have that character.

Steve Lipner:

But today you've got to have rules in training that address C and C++, and Ruby and JavaScript and Haskell, Lord knows whatever language people are using, you can't say, "We're only going to let you use C and C#." And so, the reality is that today organizations have maybe a harder time creating a secure development process, or at least a more open time in the sense that they just have to address a lot of different considerations.

Matias Madou:

Is it too loose these days? Should we try and converge again to using a couple of language or is that impossible you think?

Steve Lipner:

I'd go for impossible. I mean, you can't stuff the genie back in the bottle or something, but organizations, companies that are consulting, companies that are building tools, companies that are doing training, organizations like SAFECode, can band together and identify best practices for popular languages and then disseminate those. And so, there's a community that can tackle those problems.

Matias Madou:

Okay. So essentially developers have more power in this one than security?

Steve Lipner:

Well, it always had more power than security. What you have to do is get the developers motivated and equipped to use their power to write secure software. Because you're always going to have fewer security people than developers. And so, if the security people are running around chasing after the developers trying to fix what the developers did, that's just a guaranteed losing battle.

Matias Madou:

Yeah. Look, I think that's actually the second thing that you mentioned, right? Which is the speed of development, the SDL as they are moving from, from Waterfall to DevOps and Agile, that's another big change in the last 15 years that was not captured in the book, if I can say that, or was not relevant at that time.

Steve Lipner:

Well, there's actually a chapter in the book about SDL for Agile, I believe. I think Michael Howard wrote that chapter, and it's a pretty naive approach to SDL for Agile, which is a euphemism for saying it won't work. But by 2008, 2009, we had enough work going on using Agile at Microsoft so that we said, "Okay, we really have to figure out a way to do security for Agile that will work."

Steve Lipner:

A fellow named Brian Sullivan, who was part of the team at Microsoft, sort of looked at what Agile was about, looked at the SDL and said, "Okay, if we just take some requirements and say, we're going to do those every time we do anything, some that you have to do once per project, some that you can do periodically without undue risk," and basically parsed things out.

Steve Lipner:

That turned out to be a sufficient approach to enable us to convince ourselves that it was still SDL and to convince the development teams that it was still Agile. And so, that was what we did. That's sort of the basis for integrating SDL into Agile or DevOps, or whatever you want to do that. That's actually, as far as I'm concerned, something that people, if you understand that basic approach, you can do that in a pretty straightforward way.

Matias Madou:

Yeah. So adopting security towards the way developers want to work and figure out ways on how to do not break them and not stop them.

Steve Lipner:

Yeah. I mean, what you just said, adopting security to the way developers work, that's sort of the fundamental of the SDL. I mean, that's what it's all about, doing something that they'll consume and integrate into the way they work.

Matias Madou:

Mm-hmm (affirmative). And so, the last thing that you mentioned in that article that I read was the supply chain has drastically changed in the last 15 years. Like third-party code being used and addressing essentially the risk that comes from embedding third-party components and third-party pieces of code into your software. How drastic is that change in the last 15 years?

Steve Lipner:

It's a significant change. I mean, even back as long ago as Windows Server 2003, and before, we were shipping with Windows display device, display drivers, for example, that we didn't write, that we got basically in binary form from the display hardware manufacturers.

Steve Lipner:

So what do you do? Well, there's no magic, the fundamental is that the developer, the organization that ships, has to know what all is in the product. People call that software bill of materials, sometimes, but whatever you call it, the important thing is to know what's there, know what components you're relying on.

Steve Lipner:

Then from that, you may be able to do black-box testing. If it's open source, you may be able to do white-box testing. You may be able to do contract requirements upstream to make sure that developers are doing their own SDL so you're getting software that you can rely on.

Steve Lipner:

And so, that's something that you have to adapt, but it's just a consideration that you can't ignore in today's world. SAFECode published a report a couple of years ago on security for third-party components. That's a good resource for organizations to look at because if there's a vulnerability in a third-party component all your customer knows is that your product's affected.

Matias Madou:

Yeah. So I think with Microsoft, you essentially have more power to accept the third-party code into your organization. I think with other parties, more organizations, you just take it or you do not take it. They have less power over the third-party code, or is that a misconception?

Steve Lipner:

I think it may be. Microsoft has opened up to third-party and open source a lot, even in the five years since I left. So I think it may be a misconception, but a misconception overstating that anybody doesn't have to face the problems of third-party components. And so, what you do is just a combination of trying to assure yourself, trying to work with your upstream suppliers, and being prepared to manage the risk.

Lessons learned in a rapidly evolving cybersecurity landscape

Matias Madou:

Yeah. Okay. So let's switch gears for a bit. I would assume that in the 15 years since that book came out you have a lot of lessons learned. I actually noted down a couple of them. I'm not sure if you want to start off the top of your head, what the lessons learned are, or I can go through a couple of topics or things that I found on the web where you said, "Well, that's the lesson learned." After you.

Steve Lipner:

Go ahead. Go ahead.

Matias Madou:

So the first one is you said something along the lines of, "Hey, developers want to do the right thing." Definitely, there's like a SAFECode angle in there too. They want to be part of that. So it was that not the case in the early days or how do you see that? I would assume that even in the beginning the developers wanted to do the right thing but they didn't know how.

Steve Lipner:

There's a combination. In the eighties and nineties security was mostly sort of a theoretical concern. This could go wrong, but there's no evidence that is going wrong. By 2001, it was evident on the front page of The Seattle Times, for example, that security was a real problem that was affecting customer satisfaction and the reputation of our products. That was an attention-getter. So the developers definitely didn't like having that bad reputation.

Steve Lipner:

Then the question is, okay, what do you tell them to do? Okay? What can they do? What you don't do is tell them, "Okay, become a security expert, become a security penetration, tester." Not bad if they can do that, but most people have other jobs and don't want to do that. Maybe don't have the sort of talent or intuition to do it.

Steve Lipner:

So then you give them stuff that they can do with training and with tools, that's the other thing. If you tell them, "Okay, this is the way to eliminate this customer satisfaction problem. You'll find these kinds of errors. This is what to do about them." They can do that, and they will.

Cybersecurity tools vs. people: Which comes first?

Matias Madou:

Yeah. You said tools and training. I need to pick up on that one. So tools and training, what comes first? What comes second? Do you need both? What's your stance on tools and training to help developers?

Steve Lipner:

I think you've got to have both. We started the SDL in 2004 with in-person training, people standing up in front of rooms of a hundred or 200 people, security experts, talking to folks, showing slides, "Here's what to do." We had some tools, but a lot of it was, look for this and do that. Over time we increased our reliance on tools because they're more reliable. The barrier to entry is lower that they miss things less often.

Steve Lipner:

At the same time, we also went much more to online training, preferably just in time training so that the developers ... I remember when I was at Microsoft there was 40,000 developers. And so, you can't do individual person to person training with that kind of an audience. So really, it's just a scaling problem.

Matias Madou:

Wow.

Steve Lipner:

If it's a small organization with 10 people, I mean has the opposite problem, they could certainly do in-person training for those 10 people, but where are they going to get a trainer? How are they going to afford it? So it winds up that online training or packaged training of some form is really a critical resource.

Matias Madou:

It's interesting that like a lot of the things that you mentioned right now, it's really clear that Microsoft is really ahead of the curve in a lot of things. Somehow it doesn't show, or maybe I'm wrong here, it's not that that Microsoft is seen as the ultimate security company.

Matias Madou:

I know they do a lot of big efforts, but what you're mentioning over here, like trying to figure out how to do training for 40,000 people is something that companies are trying to figure out today. Other big and large organizations are trying to figure that out today. So I would say, that is definitely the top percentile of companies that I know of that is doing that.

Steve Lipner:

Well, I mean, a lot of the SAFECode folks, and of course SAFECode is an entity that enables companies to share what they do and how they do it, a lot of them are committed to training the tools to tackling open source and third-party code. So that's a sort of a common problem across the industry.

Steve Lipner:

I think that getting things wider through the community is important. If we could get the education system to do more about ensuring that people who learn to design software, build software, understood that security was part of the quality of what they were doing. I think that would be a big payoff if we could manage to do that. That's been an ongoing challenge.

Matias Madou:

Yeah. It has to be the same thing, like quality, it's something that we do on a day to day basis. We bake quality in, we bake security in, it's not an afterthought. It's just part of the process.

Steve Lipner:

Yeah, very much so.

Matias Madou:

Maybe a last question, statement, I don't know, but did you know that the word lifecycle is one word, and if you use that in alphabet products, it gets a squiggly line underneath, like, "Hey, it's wrong"? And in Microsoft, it's really one word, but in all the other things it's two words. So I'm wondering, why SDL and not as SDLC because everybody refers to it as SDLC? So I was wondering where it comes from?

Steve Lipner:

So there wasn't a lot of thought given to why we called it the Security Development Lifecycle. People still sometimes refer to Secure Development Life Cycle. We wanted to be close to, but avoid Software Development Life Cycle. And so, I think of people referring to Software Development Life Cycle as SDLC. And so, SDL was something you didn't run into, and so, I think that's why we chose that.

Matias Madou:

Oh, interesting. Interesting. So interesting that you on purpose wanted to avoid, "Software."

Steve Lipner:

Well, we just wanted to avoid the confusion. Just wanted to avoid the confusion. People know what an SDLC is, Software Development Life Cycle, whether you're doing Spiral or Waterfall, or Agile, or what have you. And so, we wanted something that wasn't the same acronym as that, but close enough to say, "It's something you ought to be thinking about."

Matias Madou:

Absolutely.

Steve Lipner:

As I said, there wasn't a huge amount of thought about in consideration of that.

Matias Madou:

Yeah, and I think everybody mixes them up these days, SDL and SDLC.

Matias Madou:

Steve, thank you very, very much for accepting to be the sixth software security guru on the Software Security Gurus Webcast. Thank you very, very much for accepting the chat. It was fantastic.

Steve Lipner:

Thank you, Matias. Thank you for inviting me. Looking forward to talking to you again. Bye-bye.

Matias Madou:

Thank you.