Software Security Gurus Episode #18: Brad Senetza

Matias Madou:
Welcome to today's Software Security Gurus webcast with Brad Senetza. Welcome, Brad.

Brad Senetza:
Hi. Thanks for inviting me.

Matias Madou:
Of course. Hey Brad, do you mind sharing a few words about yourself?

Brad Senetza:
So I work at Oracle, Global Product Security is our group. It's under Mary Ann Davidson, who's the chief security officer and we're responsible for kind of what the name says, product security at Oracle.

Matias Madou:
That can mean a lot of things and actually today I would like to dive a little bit deeper into two subjects around product security in Oracle if you don't mind, I hope they are near and dear to your hearts.

Brad Senetza:
Yeah, yeah. Sure.

Matias Madou:
So the first topic I would like to tackle is around distributed culture. I read a blog where you say, "Hey, you know what in Oracle the way we create systems, develop systems, we have some sort of a small centralized group and then we rely on a decentralized number of security enthusiast." You called them. Maybe let's start with the beginning. What is the central group like? What does it do? How big is that group?

Brad Senetza:
Yeah. So centralized groups are... Its kind of an evolution of what we've seen. Usually it was a security person and then they would assign a couple of security champions and then what happened was the security champions they get, I don't know, inundated with a bunch of stuff, right?

Matias Madou:
Yep.

Brad Senetza:
So then they started going to centralize security teams and the centralized security teams are in different groups. So not that one giant centralized security team, but more at the line of business level. So one product will have a centralized security team, like our cloud centralized security team and we'll have a SAS centralized security team, that kind of thing.

Matias Madou:
Okay.

Brad Senetza:
And then what we saw was, they would have issues, right? Because the security champion couldn't handle all the stuff. So naturally it goes to these security teams and they try to grow themselves. So you end up with these giant security teams and they can't handle it either. So what we found works best is when you take a small specialized security team and you use a distributed and I call them security enthusiasts because there's a million names. I don't know, everybody has a different name for them. Because well, there, we have a bunch of groups, right? We have a tens of thousands of developers and tens of thousands of consultants. And so, they're all divided into their own business units, under different VPs and stuff like that. So they have all different names. So I call them enthusiast, but enthusiast is kind of an interesting term because a lot of times we'll see that groups try to assign a security person. So if you make... At one time we had this requirement that you have to have a security person on your stuff. Right?

Matias Madou:
That's good.

Brad Senetza:
And then they would pick, the guy out of college. Okay. You're the security guy. Right? And he's like, okay, what do I do? And he's like, it doesn't matter. But we found that... Now sometimes you have to do that right. To make sure that you have coverage. But we found as time goes by, if you get the guy or the girl or the person that wants to do security, that it's way more effective. Right there... They're more interested, then they come up with great ideas and there's feedback and all that kind of stuff. So that distributed centralized team is usually smallish and they're excellent. So experts in a sense that they have more security experience than say other people. Right? You get typically in a development organization like Oracle, you get a ton of developers, but you don't get a ton of security people. And oddly enough, developers don't want to be security people.

Matias Madou:
Well, so no, and that's quite interesting. Right? So I was actually going to ask, how do you find these security enthusiast? Because in this huge company, Oracle. How can you identify the people that have an interest? Because normally developers, yes. I get what you're saying, not all of them are interested in security. How can you find these couple of people that really wants to take this thing on that drive this thing forward? I call them a satellite in the groups that are not central.

Brad Senetza:
Yeah. So we try to do things like meetups and stuff like that.

Matias Madou:
Okay.

Brad Senetza:
To drive an interest in security, we have different... We've had meetups for consulting groups. We have a dev sec ops meetup and we invite anybody that wants to learn. And you get really quite a diverse group of people there. And then we have different speakers with different topics, it's kind of cool. And then the groups kind of, I don't know, they sort of evolve on their own. So there's this guy knowing he's in PR and he's really excited about security stuff. And originally you think that's why would you want a PR person as a security enthusiast. But if you think of that kind of influence that they would have with the people that they deal with. That's kind of a cool thing. Right? Because they deal with guys that would talk to researchers that would give interviews on TV. And if the PR person that thinks that security is important, then I think we're doing a good job with security culture in that. [crosstalk 00:05:52]

Matias Madou:
Absolutely. So if you do these meet up somewhere, I read that, is it correct that you have like 2,700 people in a particular group? I think it's the consulting group that distributed consulting group is 2,700 people. How do you do a meter? How do you get alignment?

Brad Senetza:
Yeah, so that... Yeah, so those, like I said, the consulting is tens of thousands of groups or people. And so what we did was we created specialized training for our security enthusiasts, which includes, threat modeling and that's kind of cool, basic code stuff, stuff like don't hard code password. So there's, it goes from don't hard code passwords to security principles, threat modeling, that kind of stuff. So it's very broad based. And then we give them badges or whatever, and then that encourages them to take the training. We don't make the training mandatory or they take it. And then that's how we measure the number of people that... and then we don't have... So the meetup that we have global product security, we're a very small group, we hold meetups. And typically we get, I don't know, anywhere from... Well they usually start small, but around 200 people, but that's not 2,700 like-

Matias Madou:
That's a good size group, 200 people.

Brad Senetza:
And then what happens is that we encourage the smaller security. So you get a lot of those securities experts come to those meetups and then we encourage those guys, those groups to have their own meetups. And so it's very important. We invite everybody that wants to show, but we don't get 2,700 people. They don't show up to our meetup.

Matias Madou:
Actually I want to go. So my second topic was something that you already touched on, which is feedback loops. And essentially you're saying, Hey, you know what, we're doing training around threat modeling and I know that in DevOps, you try to automate as much as possible, but there's stuff that we can not automate, like, Hey, doing threat modeling training people is something that is hard to optimize doing code reviews. It's all very hard to automate nearly impossible. So how do you scale that? How do you optimize something like that in a huge organization?

Brad Senetza:
Yeah, so we push a thing we call iterative security design. And so that's... Because agile and the backlogs and all that kind of stuff, if you don't have security built in and you don't have the culture, then you're not going to add it to your backlog. And then, so what we do is we created kind of, I don't know if you'd call it a checklist, it's a set of guidelines and principles. You want to think of, like trust boundaries. Are there any trust boundaries being crossed here? And eventually, and its actually worked pretty well because we found places where in their design, they go, Oh, by the way, maybe we shouldn't be storing this sensitive information in these fields. Right? And that's kind of a cool thing that you would... Originally, we've had issues where they have people have done coding and they get all to the end and that's when they find, Oh, we have sensitive information here.

Brad Senetza:
Well, if you can find that at the very end, it takes weeks to go back and do the redesign. So if you can find it as part of your iterative secure design backlog kind of thing, then that reduces the rework. And that's really one of the most expensive things.

Matias Madou:
Yep.

Brad Senetza:
And if you look at... We have this centralized security team. Right? And they kind of said, well, we only want to... There's so many groups doing coding that they said, we only want to look at the important security issues or what's an important security issue. Is it something that has crypto in it? Or is it something that where we cross the trust boundary? So that's why we want the distributed guys to look and if they find something that looks, Hey, this could be an important thing. Then they can escalate it and get the guys the security experts to help them review the stuff. Because you can't review, can't do 500 code reviews a week. It's not possible. Well, maybe it is possible. I mean, it's not going to be great. I don't think.

Matias Madou:
So one thing you just mentioned is also like, what is obvious for certain people is not obvious for other people, like people starting in the field. Well... And even sometimes senior people, if they haven't come across something similar, well what is obvious for all that for one person is quite often non-trivial for another person.

Brad Senetza:
Yeah. That is one... Yeah. That's kind of the thing that we noticed. Right? Because as security people, you develop a certain, I don't know what it is, certain spider sense for things. Right? But other developers, they're great coders, but they don't necessarily can spot an obvious security issue when obvious to us, not obvious to them. So we run some... We help teams run vulnerability assessments. Right? And then at the end we run and say, some tool and then we go, okay, what do you think? And they go, it looks good.

Brad Senetza:
They don't overlooks good or bad. Right? It looks great, and then you go, well, what about this and this? And then it all, okay. So then you have to sort of train people that are new, what to look for, as an experienced person. If you don't have the training, it's not going to help. Right? So that's why we have those feedback loops. We have continuous training. And then they tell us what, with the feedback loop, it goes the other way. Right? We had a group that was a bunch of security guys, great guys. They were awesome dudes and they can code and do all sorts of stuff. So they automated the heck out of everything. They're like, Oh, we're looking at all this amazing stuff we have automated. Our DevOps is bang on it's the best ever. And then, so I talked to the QA group, I said, "How's the security testing."

Brad Senetza:
They're doing all this stuff. And they go, yeah. They test stuff that doesn't matter and stuff that doesn't work. And if you test... If you create your security testing in a vacuum, it's going to be awesome. Right? But if it doesn't provide value to the development teams, then it's kind of a waste. Right? So that's what that feedback loop helps with. And same with the... We rolled it out, we did this distributed thing to consultant and one of the things that we sort of didn't really take into account was the sales folks. So we never really made security training a requirement for salespeople back in the day. And then what would happen was the consultants. They're all, Oh yeah. Security, security in the sales guys. They didn't really... Wasn't on their radar.

Brad Senetza:
It's not that they didn't care. It wasn't on their radar. And so they would sign these contracts and then we... It get to the consultant, they'd go, what about all the security stuff? Like a what security stuff. So, there was a gap there, feedback loop. Hey, the security guys are not putting... Or the sales guys aren't putting security stuff into the contracts. So we change the model of our training and we added the sales folks and it's really been beneficial. Sales guys are, yeah, yeah security. They've actually been able to sell security services. Oracle being concerned about security is now a plus because other vendors are not so inclined to do the security kind of stuff that we are.

Matias Madou:
Yeah. So one thing I noticed is that security only works if the entire organization, first of all carries it. Like everybody needs to be involved, everybody has to stand behind it and say, yes, we would take this serious and we want to do this. And another point is absolutely. One thing that you mentioned is, it's a continuous process. It's not a one-time thing that you can do right now and everything is secure and you don't have to care about it anymore. So I really like her iterative process on, for example, threat modeling or training, and especially around threat modeling. I got some questions lately around like, Hey, how do you actually do that in the new world with rapid development and DevOps and all of that? So we'd love to hear your ex... Like frequency wise. how often do you look at applications? Is that monthly, yearly? A lot of people are still struggling with that, how you really do threat modeling in the new world.

Brad Senetza:
I want you looking at it on every one of your [sprint 00:15:08] reviews. Right? That's what I really want. We've seen in the past where they do, architecture reviews and all this stuff, they come up with the diagrams and they build all the threat factors and all that stuff. But the product's already out the door.

Matias Madou:
Yep.

Brad Senetza:
It's gone through 25 iterations by the time they do this gigantic architect review. Sure. They might find a few things, the product's not the same anymore.

Matias Madou:
Correct.

Brad Senetza:
If we do threat modeling, while we're doing development, we can find that, Hey, we're going to put sensitive information in these fields that really we shouldn't. Oh, by the way, we don't really want everybody being able to log into the admin possible. So maybe we should fix that now, before we get to the end of the product, that kind of thing.

Brad Senetza:
So yeah. So the iterative design stuff it's... And it's not the threat modeling, it's not really... It's not a formal kind of thing. It's more of a... When you do design, these are the things you want to worry about. Are you going to import third-party stuff? Right? Is that third-party stuff actually going to be supported once you get your application built? How are you going to patch it? I mean, that sounds really weird, but I talked to a team recently and I said, okay, you need to change these values and they go, Oh, we don't have a way to patch it.

Matias Madou:
Yeah. Yeah. But scale is different as well for you and your organization. It's not something that will be used by five people somewhere. it's quite often at massive scale that people are using these products. So if you do design and if you build something, you need to think about that.

Brad Senetza:
And I mean, yeah, the biggest products that are matured, the database and stuff like that, they don't have a lot of changes. I mean, when they go to the cloud that become on changes to that. There's still a bunch of development, there's still a bunch of new innovation, things like that. And the consulting groups, when they get into... When they go to the customer for their contract. That's always new kind of stuff. Right? A development organization. They typically, they'll be there for years. Right? So they can afford to put in these mature processes and everything. But if you're a consultant and you have to get in there and deliver a product in a year or six months, you don't want to come out at the end and have an insecure product and be in the news. Right? So you have to make sure from the get-go that what you're doing is secure. And that's kind of what we're trying to instill with that security culture.

Brad Senetza:
And I think we're doing a good job. I mean, the PR person is a great example, the sales guys. I talked to a team and they had like a PM. I'm like, why do we want program managers as security guys. Right? Who wants that? But then in the end, the PM came up with all these great ideas on how to improve the product, because they knew more about the product, the way the customers used it than the development staff did. So it's kind of beneficial. So if you can look at... That's why I say security enthusiastic instead of pointing somebody, okay, you're the guy. Sure. You're like a super smart security dude. Right? But that's your lane. If you get people outside your lane, you might get more ideas and it might actually help improve the product.

Matias Madou:
Absolutely. So I really liked that. The more people... Diversity is also an important one in there, essentially the more diverse people you get into the more ideas, the different angles you get in there. So it shows that truly Oracle carries about security and it's from the ground up. But also top down, everybody cares about security. So maybe one final question, Brad, I'm looking on Google, and Google is not helping me out. You're flying under the radar, my friends, luckily, some of your friends quite often post about you. So if I've done my research, correct, you like wine a lot. And one of the blocks was about the California Cabernet. So is there anything that you can recommend wine wise? what do you like? What can you recommend?

Brad Senetza:
Ah, see your research isn't that great. I haven't... I've gotten out of wine probably years ago.

Matias Madou:
Yeah. It was 2008. I'm saying you're flying under the radar.

Brad Senetza:
Yeah, yeah. I don't want to be on the radar. Right?

Matias Madou:
Okay.

Brad Senetza:
That's what I say to you. I was going to do that two finger thing for the whole video.

Matias Madou:
So then I'm going to ask another question. If you're no longer in wine, I see all these guitars. What is your favorite? Very favorite band.

Brad Senetza:
My favorite band. Oh, ah, Slipknot. Have you heard of that?

Matias Madou:
Oh, nice. Absolutely. Yeah. I love that. I love that stuff.

Brad Senetza:
Yeah, it's a real shame that we can't go to concerts anymore. That's one of the things I like doing [inaudible 00:20:39].

Matias Madou:
I've seen Slipknot in Ghent, when it was like less than a hundred people. They were not famous before they even became famous. Love that group, love that group.

Brad Senetza:
Always fun in concerts.

Matias Madou:
Yeah, absolutely. Hey Brad, thanks for accepting to be the 18th Software Security Guru and coming on the webcast. Thank you very, very much for this fine chat.

Brad Senetza:
Thank you. It was fun.