Software Security Gurus Webcast Episode #9: John Stewart

Matias Madou:
Welcome to the Software Security Gurus Webcasts. I'm your host Matias Madou, CTO and co-founder of Secure Code Warrior. This webcast is sponsored by Secure Code Warrior, and for more information, see www.software securitygurus.com. This is the ninth in a series of interviews with security gurus, and today I'm super pleased to have with me today, John Stewart. Hey John.

John Stewart:
Hey, how are you?

Matias Madou:
I'm all good. John, do you mind sharing a few words about yourself?

‍

Introduction

‍

John Stewart:
Not at all. So I've been an executive in large companies, the most recent one being Cisco and been an entrepreneur for most of my life all the way back to my first company at 14. And all total, my career has been completely about cybersecurity for about 30 years now. Hi, from California. I know you're in another country that I can't get to I don't think anymore, and you can't get to mine. So thank God for Skype and Zoom and all the WebEx technology of doing collaboration, otherwise we wouldn't be able to do this.

Matias Madou:
That's true. Very great to have you today. For today's webcast I have three topics in mind, if you're okay with that. And the first I would like to talk a little bit about software security at the SVP level, because let's start. You were at Cisco for 17 plus years, and your latest title was, let me read it out SVP chief security and trust officer, and quite often people in that position do not notice there's something called software security. However, if I look at the Cisco blog profile, which is still online, by the way.

John Stewart:
Good to know.

‍

A cybersecurity business leader who cares about code.

‍

Matias Madou:
For now. It explicitly says you were responsible for the Cisco security development life cycle. So I was blown away when I saw that. So you really care about the code then, and my question is a little bit like, why did you care at that level and so many other SVPs, CSOs in the world are like, "Software security. What on Earth is that?"

John Stewart:
There are a couple of epiphanies I had over time. The first was being a programmer, I remember creating problems just as much as solving them half the time, because it wasn't as if I went to college and in computer science had this defensive training and what I would call the opposite side of doing just programming and learning about security inside of software at all. I didn't get that experience. But when being at Cisco especially, one of the gravity of that job is recognizing that you're literally powering the internet, and consequentially with the transition of companies. And I remember this, you were stunned when networks worked at one point in time. You're just pleased that they were even up and operational and they went down regularly.

John Stewart:
But more and more, it became, "Hey, we have to have the internet." It's the equivalent of having power grids where just the whole world without it would be very disturbing. And consequentially, we sat at Cisco and went, "Wait a minute, we're building everything that's running it." So big surprise, it's dead center of something that's super important to the planet. Super important to companies where many, many companies even 17 years ago had said, "Look, we don't know how to run our company without a network and systems [inaudible 00:03:24]." And here we are talking to each other going, "Hey, does anybody actually check this stuff before we ship it? Is anybody validating code? Is anybody looking for the commonly known weaknesses?" And we didn't have a systematized process to do that.

John Stewart:
So when I thought about, "Hey, I'm going to be the public spokesperson if something goes wrong," and John Chambers and I were talking about it, we're like, "Yeah, we got to up our game." And it was somewhat self defense. I'm going to be the one that has to answer these questions because I've got this title and John's going to have to answer those questions because he's CEO. So let's try and get ahead of it. Final piece of this and then we can shift it a little bit. Final piece is you can't just start it and then go, "Okay, that's it." You actually have to build it over time because a big company with this many products and that much code can't just flip on a dime and do it well. So this is a constant persevering need.

John Stewart:
And that's why it's the secure development life cycle, we made it public and said, "We are doing it. Here's what it looks like." Because then it's beyond me and it's beyond Chambers, both of which, by the way, have transitioned out of Cisco, and Chuck and the team that are over at Cisco now carry the torch onward. And it's because you do it that way that you can ensure that even if you're passionate, I was passionate, to make sure we did the right thing by shareholders and customers, that it survives me and it survives people over time. So that's why it was all made public on the Trust Center and trust.cisco.com is because then it's something that you can't go back from. And truthfully, and everybody kind of knows this, it's like, what are you going to do? "We actually stopped doing the secure development life cycle. Just wanted to let you know publicly, we're out." Not going to happen. It's got to be there forever.

Matias Madou:
Is the fact that you guys were building networks and you were already really good at that, that you said, "Well what's the next thing?" Was it a natural progression then to go to the bits and the bytes that actually run these networks?

John Stewart:
I don't know that it was a natural progression. I think it was just that gravity and you ground yourself going, "Okay, if the internet has a security problem, we probably created it." From a code standpoint of the infrastructure. So I started articulating this a little differently inside the company and no one was going, "Oh no, we don't need to do that." It was, "Oh no, we really better get doing that." So yeah, there's always choices and it's difficult because you could fix all the bugs in the world of all the code you've ever written and you could turn the lights off afterwards because you'd spend so much time doing that, you wouldn't do feature velocity because honestly the market didn't... And this was what Scott Charney, by the way, told me, the market wasn't demanding we do it when we started this.

John Stewart:
It was around 2007. Very, very interesting, because then you've got to choose to do something that your customers aren't telling you you need to do. Whereas they're spending plenty of time telling you all sorts of features they need. They weren't demanding software security inside of code. So I was told by Scott Charney, who was a mentor of mine at the time, still is a good friend, and actually asked Microsoft to take their team and Scott's Trustworthy team to come down and teach us how they did it because their market was demanding it at the time. And Scott said, "You're going to have a heck of a time trying to do this without your customers demanding it of you." And I said, "I got it, but I have a feeling when they demand it, it's going to be really ugly if we're not already doing it, so we're goin to take a shot at doing it ahead of it." So that's how it all started.

Matias Madou:
And good to learn from people that already did stuff before. Microsoft was very, very early on. I had Steve Lipner on a previous episode and that was 2004 when they started doing the secure development life cycle. So it's good to learn from people that have already done it to avoid the mistakes they've done.

John Stewart:
Absolutely. And that's how I met Steve Lipner, actually, was back when we started this and Scott brought him down and that's how we first got to know each other and I'm fortunate to still call him a friend. And he is one of the foremost experts just like Scott was, and all of what they had learned in their company, I could translate something or some piece of it into the team we were building and I still give them plenty of credit. And the good news was we were able to pass it along. We've had multiple customers at Cisco that say, "Hey, how did you do it?" So we've passed on the good will that Scott and his team gave to us on to other teams that had started their security development life cycles in the last 17 or really the last 12 years since we did ours.

‍

Measuring the outcome of tools and training.

‍

Matias Madou:
So maybe a good second topic to go on with what you just mentioned is that when you were at that level, you said, "Hey, you know what? I really would to start measuring outcomes." And you wanted to really have a diverse workforce, because when I was reading up on the things that you did within Cisco, one thing that stuck me out is that you were always very focused on results, on outcomes. So I think within your department, correct me if I'm wrong, they just didn't buy a tool to tick a box. You wanted to know what the outcome was going to be. And at the same time, I think measuring security is always super, super hard and you can definitely measure the wrong thing. So can you give a little bit of a download of how you measured stuff? How did you look at tools and how did you measure the outcome of the tools instead of just giving money and say yes, tick in the box?

John Stewart:
So, for sake of argument, I'll focus on the software security part of this, because we also did all the InfoSec part and we did some customer engagement in revenue and all sorts of other things, but for just software security, we actually measured a ton of wrong things in our timeframe. So part of it was trial and error. What actually got us the difference we were looking for? For example, an outcome could be, and this is a wrong measurement, did all the code that is being compiled and sent to our customers go through a scanner, so a code scanner, so that way we could say, "Yeah, we actually ran Fortify or somebody's product on it." And the answer was that's actually one of the measurements we initially did. Well come to find out, and you're smiling, laughing, because you know darn good what happened, which is everybody did it. They just ignored the output of the scanner. So we actually got the outcome we wanted originally, but not really the outcome we intended. So yeah, that's a classic example of trial and error. And then we had to go back and go, "Okay, that's not actually what we meant."

Matias Madou:
The other one that I also heard was you just use a scanner and it wasn't supposed to be there for that language. So they use a scanner and it supports Java and .NET and you're scanning C code.

John Stewart:
I don't know that we didn't have that problem too. I don't remember it actually being our problem, but we had to just an equally silly one, which was you run the scanner and you don't do anything with the output, it's not really what we were going for. So we had to evolve this a little bit and we still, actually, even when I was leaving, there was a new set of discussions about evolving it. And what we ended up getting to was this. We measured things like median time or mean time to repair. So how long did it take when a problem was found, on how long it took us to fix it, a security problem in particular? So that was one. We also did, how many of them were created by us and how many were created by third party integrations? Not that you pass along the blame, because it's still Cisco's logo on this thing. But moreover, that you had to find out how much code was being generated by our own internal teams that was creating security bugs we didn't catch.

John Stewart:
And by the way, all of these relate to escapes. They all relate to things you didn't catch before they went out. By the way, then had to measure the efficacy of the controls. So with all the security bugs that were discovered after the fact, what controls should have caught them or could have caught them? So you measured efficacy on the total number of security bugs sent out, which controls should have stopped it? Maybe the controls actually... There's a common control that seems to be not strong enough to catch these things. And so the efficacy of all the controls started being measured too. By the way, useful thing about the efficacy of controls is you suddenly find controls that are actually meaningless. You start to reduce the number of controls because it turns out they're not catching anything. So you just go, "Okay, well that's pointless. So we just eliminate it."

John Stewart:
We also did things like... Excuse me. Were you supposed to catch them in alpha beta code or just in production code? Good question. That was another thing we actually goofed up, was we started articulating that it's not all code, it's code at certain points in its life. Alpha beta code is buggy as heck. It's supposed to be. You're testing and breaking things all the time. So we didn't want to over-engineer a set of controls that were supposed to be on that alpha beta code. We absolutely did want to make sure it was on production code that we shipped out. We had this start, and this was tough, this was a tough call. It bothered me a lot, but I truly respected that we had to do it. We had a brownfield greenfield discussion, which is, "Hey, do we start from all the code that we're shipping today forward? Or do we have to look backwards on all the codes supported at Cisco?" That was a hard call.

John Stewart:
I was the one that, at least I can count myself as accountable, and saying we have to go to greenfield and then gently, over time, take the rest of the code in history and just replace it with new versions. And that was an open discussion. We had a lot of debates about it. But we just looked at the technical debt of trying to do all the code backwards and went... I don't think we're ever going to catch up. So why not just do it over time? So we did things like that. We had discussions on third party. How long does the third party? Open source? What's the ingredients list? Do we know everything that's in the code that we're shipping? Believe it or not. That was a hard question to answer initially. Turns out we didn't.

John Stewart:
So all of those things are actually ones that continue to be ones that are reported all the way up to the board. And that was the next thing. I talked about the trust.cisco.com saying here's what the CSTL life cycle looks like, and we will always do it forever. But we also put board metrics in, frankly not only to inform the board of what they needed to know, but also to make sure that it was beyond me too. So that was something that was reported, that everyone from top to bottom knew at the company, it basically would be something you have as a matter of record [inaudible 00:14:21].

Matias Madou:
I'm doing quite often a talk on moving to DevOps and how to put security into DevOps. And a lot of the things that you're mentioning are actually... It's stuff that I'm referring to, like mean time to failure, trying to lengthen the time in between failures is not a good metric. You should not do that, because people will start hiding. What you should start measure is mean time to recovery. And that's exactly. So it must be that you guys were ahead of the time and it's good that you guys published a lot of the information because other people can, again, learn from your experience and not make the same mistake.

John Stewart:
I'll tell you what's interesting about using measurements though. And we learned, again, through a couple of mistakes. I'll give you a simple example. At one point, we started calibrating with our largest customers, commercial, service provider, "Hey, intellectually, what would you like?" And of course naturally what they want is when you find a security bug, you fix it quickly and you send us code. It turns out, though, that in our world, in the Cisco world, we have different responses based on which product we're talking about. I'll give you a classic example of where this can go wrong.

John Stewart:
So we start studying all the router code and the switch code, the big infrastructure devices, both in service providers, core networking, wireless, et cetera. Of course, we start finding security bugs we hadn't found before. We created a testing team that was... Frankly, their whole mission was to break stuff. And then naturally we fix it and we start producing code. Unfortunately, we started producing so much in so many versions of code that we're fixing all these problems. Our customers literally said, "Stop it, because every time you announced a new version, my auditors demand that we respond and have to put it in place. And you're putting out so much so fast that we can't keep up with you." And it was an interesting counter reaction, because originally it started off, "When you find it, fix it, and then give us new code." Then it's like, "Wait, you're in the core. You're actually upgrading our nervous system and our blood supply system. So heads up, you got to knock it off. So can you go back to quarterly, and then give it in a combined version?"

John Stewart:
Very interesting. I didn't see that coming. But intellectually I get it because you're overwhelming IT teams, and of course, if they're out of compliance, then they have a know a whole 'nother problem, which is we're creating a problem for them. So we had to negotiate back and forth. Now, if it's something like WebEx, that's a totally different thing. If we can do it as a service DevOps world, fixed and they don't even notice it? They'd be fine with it. Fix it as fast as you can. Just didn't see that the bifurcation coming. So we had to start applying metrics differently to different teams because of customer response. And that is one of the key learnings there, I think in the end, I figured out a little late, which is have your customers work with you on what you're supposed to ultimately get to, because intellectually, I want to just fix it and get it to them. And then thank goodness they've got it.

John Stewart:
But if it turns out they're not going to use it and I've sped up everything and then we're overwhelming them, we've made an unintended mistake, which is we only did our part for getting that there's they're apart. And if we put them in harm's way by trying to fix something with good intentions to get them out of harm's way... We fixed one problem, create another one, then we really haven't solved the problem that they ultimately want, which is to be safe.

‍

The business impact of women in cybersecurity, and diverse teams.

‍

Matias Madou:
Yeah, working closely with customers is always super important. We had a very similar story where we were pushing stuff too fast, and they were able to explain to the people on the platform that new things were coming at rapid pace, so we had a very similar thing. And then you start with feature flags and all the fun stuff. So you're also a big advocate of diversity in your team. Can you tell a little bit more, why should we have diverse teams and what was your approach, because my approach is, I'm trying with engineering to get as many females and diversity as possible, but ultimately, if I go back to university and I looked around me, it was 95% white males. So the inflow was simply not there. So how did you tackle that particular problem and what was the outcome? What was the outcome of having a diverse team?

John Stewart:
So there's a very personal story to this, which is my mom immigrated from Scotland pretty young after she lost both her parents, and then I have two older sisters, one of which is a Stanford PhD and the other of which is a senior official slash operator in the medical system in New York. My immense respect for the way that they thought differently than me has always been an influence in me as a person. And as a result, I saw up close, growing up, the fact that we didn't think the same. We actually argued intentionally in our family. It was not arguing in a negative way, but point counterpoint, bring information, bring data. And candidly, I think that the fundamental thing my dad was trying to do is get us to have critical thinking. So was my mom.

John Stewart:
Unfortunately, what oftentimes would happen, I was the youngest. I would get emotionally frustrated and just lose it in the middle of the argument and go... And then I'd lose. It was just, you lose. So the point of that story is to give you a sense of how far back I had a beginning if not deep appreciation for the difference of thinking, especially with men versus women. And then over time, what I started to notice in my professional career, my first manager was, was Michelle Gell. One of my most recent managers was Rebecca Jacoby. Two people I respect and call friends to this day. Was the way that they approach things would be differently than the way I approach things. So I started to look around all ways and go, there's got to be a reason that that can be useful versus detrimental, because I'm learning. So why not actually use it as a strength?

John Stewart:
So then I started looking at team outcomes and I began to put together that in fact, when I was working on at least male female mixed teams, it always seemed we got to better places than if I just was working on all female, aside from me, or all male. So that started my journey down in multiple phases towards the fact that I was advocating. Now, the second thing is that I have a daughter who's now 21, and as she was growing up, I became very aware of how it is not an equal playing field for men and women in business. And that bothers me. I don't understand why that would be the case. It's not as if there's an advantage one way or the other in business. It's just who you show up as a person and what you bring to the table, and if if you look at it equally, it should be valued equally.

John Stewart:
So that began to bother me a little bit. So it became a huge champion for a second reason, which is I don't want that world for my daughter. I want the level playing field for my daughter. And if we can't fix it tomorrow, we should at least start to fix it, so I started essentially advocating in that respect, too. Last but not least, and you talked about outcomes, which is definitely something I measure. I actually found myself in being in multiple professional situations, asking myself who are the best people that do X or Y, and come to find out, and I've run into this time and time again, there are jobs and roles, especially, where in all levels you'll find that the thinking process being different because of a man versus woman will actually really make a very different business outcome, because the thinking actually causes it to be better.

John Stewart:
Whereas traditionally you'd just say, "Okay, if I'm double blind testing, then I'm just going to put whoever's there." Actually, the diversity and debate and techniques of using different people and their different strengths. Absolutely. It creates better outcomes. I will give you a classic example. Very oftentimes I like to go fast and then iterate crazy. What I discovered in working with Rebecca Jacoby for example, is she's very methodical and strategic and thinks a couple of moves out. I do too, but we just go completely different techniques in order to get to a different place. So her role and what she brought to the table absolutely ensured we didn't go straight off a cliff, because she just thought differently. And it's not because, by the way, she's anything other than smart and capable and experienced and and and. So I was watching this all in action all the time, so it just became something I became very passionate about.

John Stewart:
There's another part of this though, which is it's an undertapped community. In fact, there's multiple undertapped communities. We're always saying, "Hey, we've got so many jobs. There's so many people. We have to hire more." Well, why not actually enable a community that's underrepresented already. If you argue that 50% of the population is male and 50% of the population is female, then if it's 95% of the population that you're working with are male, then clearly there's a whole community we could actually go tap into and actually hire because I'm looking for people all the time. So last but not least Michelle and a woman by the name of Cyprian Palma started creating a community in Cisco, Women in Cybersecurity, and then they started advocating in job fairs because it turns out you to see people that look like you when you want to work in a company and ask them like, "Hey, what's it like? Is it great to be in Cisco? Is a great to be a X, Y whatever you want to call them in Cisco? Is a good to be female in Cisco? Is it supported?" This and that and that.

John Stewart:
So it builds on itself because you start to get more and more people going, "Oh, that's a great group to work for because they really value the diversity of it all." So it yields longterm results as well as short term results if you do it right. And don't get me wrong, I needed a lot of advice. And the underrepresented communities that are still underrepresented are black African American, Hispanic. I mean, there's whole sets of other things that have to get better that I could only activate so fast on so many things, but in the end, they're now taking that mantle and they're jumping on board and doing other things and you can see in Cisco's commentaries publicly, especially at times we've had recently, that Chuck is a firm believer and he's activating on it. I love it. It's rewarding to me, and as you can tell, I'm slightly passionate about it, which is the reason I took over half the podcast to talk about it.

Matias Madou:
No worries. So, one thing I see is that sometimes... I'm in Belgium, and that was the case when I was at Ghent University. When I lived in the Bay Area and I was at Fortify, the talent pool was very different. Suddenly I was working... The majority of my team engineers was female, and it was just location wise there was a very different talent pool than where I was originally from. Somewhere in my mind, I feel that that is also the success of Silicon Valley, where you have this very diverse group of people that always work together on very, very innovative things, instead of other parts of the world where it's very strict, you have engineers and they're all white males.

John Stewart:
Yeah. We had to look at it in different populations. And by the way, that became its own set of things you had to monitor, because in different countries, simple example, and different cultures where I might start as an American and go, "Okay, equality." That's not where the world is. Some parts of the world don't see it that way yet. Some of them are emerging into it. Even frankly, in America, you got to constantly monitor it to make sure we don't go backwards and take steps back. It just can't be universally applied and go, "This is what we're doing." Because there are social pressures. There's a ton of social pressure in certain countries that you get to a certain age as a female, you get married and then you become the head of the household and that's it. You exit, if you will, you off ramp your professional career.

John Stewart:
And there's a ton of social pressure on women in certain countries, and that's very different than I think I experienced. So I've got to be mindful of that and use people from all walks of the company to educate me and then remind me that just because I say it, doesn't mean it just works that way, because I've got to take it from their perspective too. It's not as if it's a trivial thing to say, "Hey, we're doing it." That's the easy part. Then there's how many women actually want to work in different countries, different cultures. How many are staying at work beyond a certain period? How many actually are heard and the leaders that they work for actually agree with what I'm trying to say. That's the other part, is you bring a person in, and I think this is true for you, me, and everybody else, you want to work with somebody that is aligned to your value system on a lot. And I want my leadership to be the best leadership for the people that they have working for them. It's called servant leadership.

John Stewart:
If I've got a person that doesn't agree with my ideas on diversity and using talent and male and female mix and the like, they can put essentially some lipstick on it, if you will, and say, "Hey, I'm trying to do it." But if they're not really bought in, then any person working for them is going to notice. So I've got to suss out my leadership community too, because they've got to buy into it completely, or they're not going to be able to work for me. And that was part of the other learning, is it's just not about numbers. It's about the culture, especially, that you've actually got in a team.

Matias Madou:
I think we can all play a role in there, and I'll try to contribute in that aspect. Especially if you have leadership and underneath you, if you put them through the same values as you're trying to work towards, that's a starting point.

John Stewart:
I'll give you a kind of a funny story on this one too, because this was something that I'm still struggling with to this day, and I've been struggling with for ever, is I have a habit of saying, "Hey guys, let's..." Whatever. And it's a verbal tic that is so designed into my communicating style, that it is taking a relentless amount of energy for me to stop it. And I do it in all hands meetings. I would do it in settings where I was doing speeches, in one-on-ones or one advise, that kind of thing. And of course, what does that tell you? It tells you that even though I'm saying, "Hey, diversity matters, everybody's equal," I'm using a verbal cue which isn't aligning with that point. And I don't mean it that way, and yet I totally understand it's accepted and shouldn't be used because it's maligning, essentially, something I firmly believe in. Even as we're talking today, it drives me crazy that I say it. And I catch myself a lot of the times now after I say it, but I'm having a heck of a time preventing myself from saying it.

Matias Madou:
It's exactly the same for me. Because even during this webcast, a couple of times I used, "You guys in Cisco." And when I was saying it, I remembered, but I couldn't correct myself, unfortunately. So sorry about that.

John Stewart:
Yep. It drives me crazy.

Matias Madou:
So maybe a last fun question to finish off the webcast. I have a very unique name. If you look up Matias Madou, the way it is spelled, you find only one person on planet earth. If you look up your name, even within Cisco, there is a second person that has exactly the same name as yours. Is that a curse or is it a blessing?

John Stewart:
I consider it a blessing to have my name. It's a combination of family names. John Stewart's been in generations in my family. My middle name is Nickel, which is my mom's maiden name, and I am actually the youngest and pretty much the last standing one that has that name. My daughter's name is Nicole for that reason, it's an extension to pass along my mom. And of course my dad is Ron, but his brother was John and et cetera, et cetera, et cetera. So I consider it always a blessing to have the name that I have. It is however complicated to have that name, because there's a trillion John Stewart's in the world, one of which of course was a very famous in television for example and he's now very famous in the political arena.

John Stewart:
I will tell you a kind of humorous story though, which is that's not his name. If I've got it right, and I might be pronouncing it wrong, I think his real name is John Leibowitz, and he transitioned it for television, so I always thought to myself that I'm a victim of identity theft with regards to that Jon Stewart. But I've not been able to talk to him about it yet, and I think I'm probably a better off considering how smart he is to not challenge him on this thing. But I consider it more a blessing.

John Stewart:
I will tell you another part though, John Stewart as a full name wasn't the problem in Cisco for me. John was. And it's because there's only one John at Cisco when I was working there initially, and that was John Chambers, and the rest of us needed to use a different name. Sometimes, however, it was useful to have a conversation that I would say, "Hey, just say that John said." Because no one would know which of us actually said it. So that proved a little useful at times. But I actually started going by my initials a lot. A lot of people would just call me JNS because it was just easier to separate out who it was. Just the first name became the more complicated piece than my whole one.

Matias Madou:
JNS is a good one because that's how I found you on Twitter too.

John Stewart:
Yeah, exactly. Right. And I've used that as well quite a bit, just for email addresses and everything else, it's been something I've been using... Jeepers, it's probably 20 years, 25 years now.

Matias Madou:
John, thank you very, very much for accepting to be the ninth guru on the Software Securities Webcast. It was a fantastic chat. Thank you very much.

John Stewart:
Thanks for having me on, it was great to talk to you.