Software Security Gurus Webcast Episode #14: John Melton

Matias Madou:
Welcome to the Software Security Gurus webcast with John Melton. Welcome to the show, John.

John Melton:
Hey, thanks. I appreciate it.

Matias Madou:
Of course. Anytime, John. Hey John, do you mind sharing a few words about yourself?

John Melton:
Sure. So I'm the director of product security at NetSuite, which is an Oracle company. And yeah, I went through school, was going to be a developer, mostly excited by development, and was coming up on graduation around the time of the bubble burst and decided it was a good time to spend a little bit more time in school. So I went through grad school to do security, and then I've just threaded that needle for the rest of my career, bouncing back and forth between development and security. The last maybe five years, I've been pure security and building security programs, AppSec programs predominantly.

Matias Madou:
You've done way more than that, and let's dive right into it. Yeah, you've done a ton more. So I actually have three topics in mind for today's webcasts, and hopefully they are near and dear to your heart, and the first one I would like to start with, with the tail end, which is today, when I looked at your resume, and actually, I noticed it immediately that you're jumping back and forth between development and security. The second thing that I noticed is the fact that you moved the ladder very quickly within NetSuite. In one year, you moved from a senior to be a manager to be a director. And I'll be honest, John, that means you must do something right. And I'm actually very curious, what is this secret? What are you doing today or what did you do in the last five years that really made an impact within NetSuite and got you where you are right now?

John Melton:
Sure. So I think there's probably a personal side to that and an actual, more technical side that's more interesting to the audience. I think on the personal side, I jumped into technical leadership and management early in my career, and that was a mistake for me personally. I wasn't ready. I didn't have the empathy to lead people. I was more interested in the tech and really selfish at that point, and I pushed people a little too hard and wasn't empathetic, and so burned out a little bit, honestly, and went back to pure tech. And I was a little gun shy on that for a while. And then the opportunity came up to come to NetSuite. And prior to NetSuite, I had been at a product company, which was really beneficial for me to go between being part of a program, I had come from Wells Fargo, to being a vendor for a little while and understanding more what a broader scope of companies are trying to deal with. There are the big enterprises, there are the small startups, and I could see everything in between, and that was really, really helpful for me.

John Melton:
So when I landed at NetSuite, I was actually brought in to deal with a company they had acquired, which was a real DevOps focused, very modern development shop. And I had the opportunity to build an AppSec program from the ground up, which was really, really fun. And they were doing some good security practices but they didn't have a dedicated security person, if that makes sense. They had some practices in place that were great, but they didn't have anybody with the mindset necessarily of security or the technical backing. So I was able to jump in and we made some really quick effective gains within that first year. And so they took that and replicated me over to NetSuite more globally, so I got the chance to run the AppSec program there. And I think moving to manage product security, which covers both AppSec but also what you would typically call security architecture, the assurance or pen testing teams, and then we have another tooling squad in there that's a support layer, really boils down to trying to have a vision, I think, an overall program scope and laying that plan out.

John Melton:
And fortunately people saw the value in that. And I think that only worked for me because number one, I try to learn from lots, and lots, and lots of people, I try to focus a lot on what people fail at, which I think is really, really helpful. This is my quick plug for something I've been trying to get going for a long time, so maybe people who listen to this will build a failure con I would attend and I would be happy to speak at, or build failure tracks into your cons. That's more interesting to me than anything else because a lot of times you hear people coming to these cons and they'll say, "Look, I did this thing and I'm so awesome." And you go, "Well yeah, you are awesome, but I am not awesome apparently, because I couldn't get that thing to work." It'd be really helpful if people would have an eye towards this is the 18 ways that I failed.

John Melton:
And so really I think it was just banging my head against the wall for a long time technically, and seeing what worked and what didn't work, and then having more confidence in the programmatics around things. And also, again, personally, just I had built over time that empathy for seeing other people and how they had developed, and just basically maturing and experience, but also really getting excited about the people management portion. Being able to celebrate other people, see them succeed, trying to build plans for them, and help them be effective in their careers and their personal life, and trying to really just care for people. In that intervening time, I became a parent, which all of us experience that change and we go, I think naturally, more towards caring for others. And so that was certainly the case for me.

Matias Madou:
Yeah. It's an interesting experience. If you're young, you think about yourself, and you're not going fast or you're not climbing that ladder fast enough, but then when you're older and you're trying to make more other people successful, you become even more successful in what you're doing. So fantastic to hear that what you've done at Bronto, you were able to copy or replicate, but also in a scalable way then, because there's only one of you, so you must have figured something out that you were able to either replicate yourself or scale that, because you said also architectural analysis and that is manual, labor intensive. So how did you do that?

John Melton:
Yeah, great question. And certainly, I should give credit to everybody that works for me because they're the ones doing all the real work. But I think for the programmatics of the thing, every program matures over time. So you always start with ad hoc and you're doing a thing here and a thing there, and you're kind of throwing darts at the board to see what sticks at the beginning. And then, I'm a big believer in OWASP Top 10 at scale. At scale, OWASP Top 10 makes sense, and that makes sense because they're surveying thousands of companies, and the data in aggregate is going to make sense. OWASP Top 10 is probably going to prove to be true in aggregate, but statistics don't matter to the individual. So for your company, you don't have an OWASP Top 10, you have those one, two, or three that you should focus on, and your business risk, and those kinds of things, and your environment and all those kinds of things.

John Melton:
So as an example at NetSuite, we run a platform where we allow our customers, partners, our employees themselves to write code that executes on our platform. So we have RCE as a service. That's a thing that not every company does and so that's a unique thing that we have to contend with, and those are worries and areas of enormous risk for us that not necessarily everybody else is going to have to address. We also host websites for our customers. Again, something not everybody's going to have to address. And so we've got a lot of institutional knowledge that we worked from, but also where we are seeing the most lift, I think, is certainly everybody talks about automation. I mean, at this point, you have to do that. Trying to be intentional about training our people in the right way to do things.

John Melton:
We're big believers in the notion that I know is near and dear to you of invariants, that's a great thing about Sensei, which we use and like, as opposed to saying, "You're doing a bad thing," we want to say, "Here's the good thing, the right thing, the thing we've chosen." Other people refer to this as guard rails, the paved path. This is an idea that's been around for a while. And actually I think that's secret sauce behind safe companies. The more safe, more secure companies have figured out that they can channel their development teams into these paved paths. I always pick on Netflix in this regard because they talk about it a lot. Netflix has a stellar security team, and I know several people there, I believe in them a lot, they're a fantastic team, but I don't think they would be as effective as a security team if it weren't for their engineering model, which is you pick one or two things and the stack is fully well-supported. Everything's there laid out for you, and it's hard to make a mistake.

John Melton:
If they were you can pick anything in the world and we're going to just try to secure it all, you can't pull that off. And so that's really the model that we choose and that bleeds over into everything. So we've got a architecture team which we call security consulting internally, because we want to consult with our teams. And they are really focused on understanding what people are trying to do at scale, because they talk to lots of people, and then building patterns, standards and patterns, that are paved paths. So if you need to do IAM, this is the way we do IAM. If you need to do key management, this is the way we do key management. And here's the full stack for you. Here's the technology stack, here's the tooling, here's the everything. And then feeding out of that, our engineering staff builds rules that ensure we go down that path and looks for aberrations from that, our assurance team tests that we're on that path, and then our tooling team builds tools to automate all of that stuff.

John Melton:
So that's kind of how ... I think that's what has been most effective for us. That's the secret sauces that I've seen other groups talk about and try to infer from successful teams. They always seem to have ... Successful security teams always seem to be attached to really great engineering teams and really great engineering teams usually seem to be pretty narrow in scope of what they allow. You can't do the same thing 67 different ways, you pick one or two ways.

Matias Madou:
I can see why you're successful at what you're doing, because I couldn't agree more. If you can bake as many things as possible into your platform, just do it, then especially focus on the areas that you need to focus on, and that's different for every company. And then yes, security by making sure that the developers know what the standards are or that they know what the right way of coding is, the secure way of coding, which needlessly flows into my second topic, which is 10 years ago, you actually coauthored a paper called ASIDE. It's an IDE Support for Web Application Security, and that was published in 2011. I have to admit, it's super inspirational. I think it's very visionary because you were talking about an idea which was at that time not commercially available into the market. Do you mind sharing a little bit more about what you've done over there, and how that evolved, and how that affected you?

John Melton:
Sure. So that was paper I coauthored with some folks from my Alma Mater, UNC Charlotte, and UNC Charlotte is one of the organizations where I graduated from, and they were one of the first, I don't know, dozen or so schools that was rubber stamped by the NSA as the scholarship for service producing schools. So for those unfamiliar, the US government discovered in maybe the mid 2000s that the private market was paying security people a lot more than the public market was. And so they figured out that a lot of people were exiting, particularly their younger staff, and a lot of their older staff was getting close to retirement, so they set up a scholarship program and that's been ongoing. I was a beneficiary of that in one of the first rounds that came out, but they've coordinated with a lot of universities across the country to help build curriculums, and research programs, and those kind of things that really focus on software security at its core.

John Melton:
And our university had spent a lot of effort in the static analysis space initially, but then it mutated a little bit over to I guess what's classically called HCI, Human Computing Interface. Most people nowadays call it user experience or developer experience. And so the idea was okay, we have these classical models based on static analysis, based on compilers that we can give some information about something that is secure or not. Now, how do we get people to actually do that? Your background in Fortify. Fortify or any of the static analyzers will build you a big old pile of problems. And what do I do with that big old pile of problems? Well, I could go cut a half million tickets for a dev team that may or may not be effective. So you always end up in the classic answer for static analysis is how do I scale triage? How do I scale bug fixing? All that kind of thing. And some of the thinking there has been well, we can solve this with safe library.

John Melton:
So Google's a good example here. Instead of solving SQL injection 100,000 times, they went and built out a safe library for SQL. Christoph Kern has a really good paper where he and others, he's the lead on that paper and he's given a couple talks about it, it's really excellent, where they went after SQL injection and cross-site scripting, but for SQL injection, they built an API and a library wrapping that, and just made the problem go away. So if you've got 10,000 findings but you can convert the code over via automation and whatnot, that's a great solution, but that's a pretty mature solution, and you can't necessarily do that unless you happen to own the database like Google did. When you build that database driver, you can solve that problem. But for most mortals, that's not an option.

John Melton:
And so what do we do in normal situations? Well, we have to get where the developer is, and that tool at the time was meant to help train and change the workflow at the time. So most of the research was focused on can we turn a developer who is uneducated about these issues into a trained understanding developer via the IDE without going the separate training route and specific training. This is obviously something you care about with Secure Code Warrior and Sensei. That research was obviously basic research and it was really effective at proving out that it's possible, it's not always effective, but it's possible to boil the ocean. You can make people better by targeting them where they are at that moment.

John Melton:
And so all the things I talked about at NetSuite were informed certainly by that research. How do we change it at that moment? How do we get people at the right point in time? How do we build safe libraries in those cases where we can to give them a paved path, and they go in and they type some Hibernate library, and we're going to save that data off? Well, there may be nothing wrong, nothing insecure with that, but we as an organization have chosen not to use that library. We're going to choose this one API that we chose as an organization, and now we've solved that problem without a bunch of chagrin on the developers part, without a bunch of retraining, without a bunch of bug fixes, and tickets, and whatnot. And so we can inform that dev path through the IDE, which is where the devs live.

Matias Madou:
The last piece of the puzzle is indeed making sure they are aware, and you give that information at the right time in the right moment when they're there, when they need that particular API. Yes, you're still a big believer, I'm still a big believer, too. Fantastic.

John Melton:
Yeah. And a quick side note there, a lot of the basic research that we started with there was based on training, and not specifically InfoSec training, but just training in general, which was saying things like if you do yearly training, which every corporation does, most people forget 80 to 90% of that in a matter of two days. That's kind of the answer. So hopefully a few things stick, but you can't expect them to stick with everything. And so that contextual time sensitive training is really, really helpful, and it helps lock that thing in their brain, and obviously does enforcement.

Matias Madou:
Couldn't agree more. Last topic, if you don't mind, you are one of the authors, founders of the OWASP AppSensor. And I'm not sure if you've seen it, but congratulations, it's 10 years old, my friend, which means it's ancient.

John Melton:
Wow, I am old. Holy cow.

Matias Madou:
Yes, it is. One thing that I've noticed is it's a decade old and I would say it has been given a lot of attention for the first nine years, but for some reason, the last year, either you were on a break, or the world has moved on, or today's coding is totally different, but can you give a little bit more insight on the project itself, and where you think we are today with that project and the need for that project?

John Melton:
Sure, absolutely. So I'll try to keep it tight, but I will ramble here for a minute.

Matias Madou:
Go for it.

John Melton:
So the history is funny. So at the time, when I started working on AppSensor, I was at a bank and I was DevOps before there was a word for DevOps, I was dev, but I was also supporting this, and I got really frustrated at having to fix bugs, or find and fix bugs that I'm like my code should be handling this for me, and most of the exceptional conditions were just business logic errors. And the intuition that got me was this is not the first, it's not a one-off, it wasn't a null pointer exception. In other words, it wasn't just a plain coding exception, it was a business logic flow exception where the user had done a particular activity. And most of the time they were doing that more than once, so I could have detected it if I was looking for it. And so I started hacking together some basic Bash automation to parse log files and this, that, and the other. And then the light bulb went off that oh, I can do this in the application and have the application solve that problem for me.

John Melton:
I thought I was a genius, and I went out and I was like I'm going to do research because this idea is going to change the world. And lo and behold, there's, at this point, decades of research in fraud detection that I was unfamiliar with from academia. And I discovered that, and I was like oh, maybe I'm not so clever. And, also unbeknown to me, about six months before that Michael Coates, who runs Altitude Networks now, at the time he was working at-

Matias Madou:
Twitter?

John Melton:
Forget their name. Nope, it was before Twitter. The consultancy up in Maryland. I forget the name of it, but it's an AppSec or it's a security consultancy. I'll think of it in a minute. It was Jeff Williams and-

Matias Madou:
Aspect.

John Melton:
Aspect, thank you. Dave's old company. So he was working at Aspect and through the OWASP summer of code, he had written a paper on AppSensor. He had already had the name together and this, that, and the other. And so I discovered it, I went out and they had a little bit of code there, and I went out and filed a bunch of bugs. And Michael responded to the mailing list. I joined the mailing list. He responded to the mailing list and said something to the effect of, "Hey, everybody, I want introduce our new dev lead, John Melton." And so I back channel messaged him and I said, "Hey, man, we don't know each other that well. What are you talking about?" And his response was something along the lines of, "Well, I'm not the right person to lead dev on this, and it looks like you're interested. So do you want to do it?" And so I kind of got guilted into it, but I'm super happy that I did. It's opened a lot of doors and I really do believe in the project.

John Melton:
Fast forward a little, Colin Watson out of the UK, he's the real hero on AppSensor because he wrote the vast majority of that paper book, it's a couple hundred pages long, very detailed, it stands the test of time. If you're interested in this topic, you should definitely go read it, skim it. Great, great work from Colin. And I led up development. For those who are not familiar, AppSensor, the idea is that you can detect attackers, not necessarily attacks, and the intuition there is that if you look for what an attacker is trying to do, oftentimes that will stand out. So your classic [WAF 00:22:08] is going to detect XSS and [SQLi 00:22:09], but if you see somebody start poking at certain things, oftentimes you can tell that's just bad behavior, and this comes out of fraud detection and all of that work as well. But oftentimes, you can see bad behavior starting to raise its head and you can say, "Well, this person's done in number of these things, we can now classify them as being dangerous.

John Melton:
So two things I talk about there, one is Ryan Barnett, who used to work with Trustwave, he did a research project with ModSecurity WAF where he invited all the best SQLi hackers around the world, he set up a standard ModSecurity security and asked them to break it. And they broke it. They broke SQLi. But the minimum number of tries that it took to break was 80 and change, 81, 82. And the intuition there was if I was looking for numbers 1 through 80, I could have tripped an alarm at 10 or 15. If you've tried 10 or 15 SQLi attacks, you're not a normal user. Let me just shut you down. And the other thing there is more of an intuition that helps cement the idea of AppSensor, which is if you imagine a banking account, and you log into your online banking, and you've got your list of accounts, you've got a savings, and a checking, maybe two savings and a checking, and you click that link and it takes you to the detail page about that.

John Melton:
Well, if you imagine what's happening behind the scenes, you've got some kind of an identifier from the database, at [GUID 00:23:39], a auto increment, whatever. And you click on that and it takes you to that detail page. Somehow that click communicated to the backend this is what I want to see. And there was an authorization check that should have happened there to ensure that I'm looking at the things that I can access. Well, if I get back accounts five, six, and seven, and I send a request for seven, that's a normal flow. If I send a request for 255 and it wasn't presented to me in the screen, that's an immediate alarm bell going off. And that means one of two things has happened. Either your app is broken and it's presenting me things I shouldn't see, or two, I'm being a bad person. I'm doing some nefarious activity. And so the intuition behind AppSensor is that I want to look for those conditions, typically things that you might find in threat modeling or talking to developers about what are the non-happy path flows, and then put sensors everywhere in your code.

John Melton:
And there's AppSensor the idea, which is what I've just described, and then AppSensor the toolkit, which is the code base. And the toolkit itself was the primary thing I worked on, and it essentially provides you a backend that allows you to write rules, and collect these data points, and do analysis, and that kind of thing. And so the notion is that you would send over those signals, you'd have a backend running, and then that would give you a mechanism for responding in some way, logging, paging people, blocking accounts, whatever response you want to take. So why has it not changed a whole lot here recently? So the primary reasons are John is lazy. That's a big one. And the second there is-

Matias Madou:
John is busy. That's what you said, right?

John Melton:
John is lazy busy, yup. And the real reason is actually, for the most part, it just works. It's real simple. There's not much to it. I'm not trying to solve the world's problems, I'm trying to give a simple tool kit. And so usually when I talk to people about AppSensor, there's the three phases, is what I would describe. First is you start understanding the idea. The idea is the trick for AppSensor really. It's not the code, it's not the toolkit, it's the idea. If that intuition makes sense to you, you can now solve lots of problems. And it's in a accretive effect, because the first sensor that I trip or that I embed in my code, I start getting good data. The second, third, and fourth give me additional good data. The 100th still accretes my value, and over time, just like an MVP in development, I'm able to get value all along the way. And so that's really helpful.

John Melton:
The second phase would be looking for something like an AppSensor toolkit, where I need something that'll help me get going at whatever scale I'm starting with. Google, Facebook, Twitter, they're not going to pull AppSensor in and have it work. They have a different level of scale problems. So at some point you're going to hit a scale threshold where AppSensor toolkit is not your right solution. It'll work fine for, I'd say, small to medium scale. Once you hit high scale, you're going to want to build your own implementation. So Michael Coates had built an implementation at Mozilla that they used. It was based on Python, which was their dev language, and they did some really interesting things. A lot of financials look at AppSensor because it helps them solve some compliance. Financials and healthcare, that's a lot of the people that talk to me. And then there's been some really interesting work more recently, and I'm going to butcher the name of the company. I can't remember. I just talked to a guy.

John Melton:
I tweeted it out. Maybe we can link it later after the talk, but they're doing some really interesting work around connecting the operations security side to the development side and being able to look at that data, pulling it out of WAFs and gluing all that stuff together using the ... I forget what's the popular framework for security operations, Red Teaming anymore. There's a NIST ... Or maybe not a NIST framework. Anyway, there's a framework that people are operating out of on that side of the house. And so they're looking at purple teaming and solving a lot of these problems. So they go in, they pen test, they find a set of problems, they work with the blue team to build detections for those sets of problems, and then they cycle. And all the time, they're creating this data. Now, they're putting that data, I think, in [ELK 00:28:19] at the time, or maybe [Splunk 00:28:19], but feeding it into a sim and working in that way. So there's lots of options there, and they put out some really novel research here recently. It was good work.

Matias Madou:
So bottom line, it works, it keeps working, and it's going to be here 10 years from now?

John Melton:
That's a great question. I would love if the answer is it goes away. One thing I should mention, there are some vendors in this space who are coming around, so some of your [RASP 00:28:49] type companies, and that is the way that again, I love open source-

Matias Madou:
Professionally supported then.

John Melton:
Yeah, exactly. There's a high bar of confidence you have to have in a system that might be blocking attacks, and having professional support's a great thing. I think the real value in AppSensor, even though I worked on the code for a lot, I think the real value is in that idea. If you can get hold of that idea, and start putting it in your environment, and executing on it, and saying it's a natural outflow of threat modeling with your developers and with your architects, and saying if this is the happy path, just like we talked about with development, Hibernate may be completely safe, but we don't use that because it's not on our happy path.

John Melton:
If you've got a happy path of a logical business flow through the application, then you see aberrations from that, why let the person just do that all day. Eventually, after 80 tries, they're going to get ... they're going to pop SQLi. Just stop them before that. And so I think that mind shift, that light bulb going off is the real value in AppSensor. And then use the toolkit if you want, and certainly bug me if you need things fixed.

Matias Madou:
I hope it's still here 10 years from now, although you disagree, but hey. Last question.

John Melton:
I hope the idea is here. I hope somebody else has built something better by then.

Matias Madou:
Last question. You love code, and I know that you've been involved in teaching people how to code, like you were involved in [CodeMash 00:30:21] and [KidzMash 00:30:22]. What do you recommend your kids to code on? Or you don't care and they play Fortnite?

John Melton:
So we're not a big games family. We're not a huge screen time family. My life goal, it seems increasingly I see this because of COVID, my life goal is to throw my computers away the day that I retire and just be on the farm. I like farming, and gardening, and that kind of thing. So my kids have hatchets, and they go out and play, and cut down trees, and have a good time. That being said, obviously there's a lot of benefit to tech, and so I do teach them coding. We started with Scratch. We started with some basic books. We worked into Python. It is not the thing they're going to pick on their day off, if that makes sense. So I'm intentional about it, but I have taken the approach of I want them to be cognizant of coding, but I have the same belief when I'm teaching anybody to code, coding, you have to have a desire to do it, like any other job.

John Melton:
But you can pick up the semantics pretty easily. And really what I want them to do is have the mindset of thinking about how to break down problems, the engineering mindset. And so we spend a lot of time on logic, and basic engineering, and physics, and that kind of thing. Fascinating seeing things with new eyes through kids, so that's really fun. But yeah, Python, Scratch is great. I think they did the Hour of Code thing at school. That's when they were five or six. That was the way we got introduced to it. And five-year-olds don't really grow up coding, but we can try.

Matias Madou:
Yeah, okay. So 10 years from now, they are actually going to ... they will be the developers that maintain AppSensor. That's correct, right?

John Melton:
That's the plan, yes.

Matias Madou:
And you can retire.

John Melton:
That's the plan. Anybody who wants to take that work, yes. Yeah, I really want them to be able to ... I think of coding as a super power, and I talk to all our security people. We actually hire for programming and can teach security better than the opposite way. And programming is great in that you can take something from an idea to a thing. That's always fascinated me. The first day I saw that I was like whoa. It's kind of the same idea with this 3D printing that we got here recently, which is wait, I can have an idea, I can draw it, and then it becomes a physical thing. And so that's what I want them to appreciate, and anybody that comes in our field. I want people to work on security problems, or dev problems, or whatever they are, and be fascinated by them, and have a passion for it. I get a little bit passionate when I talk about these things.

Matias Madou:
Yep. Great note to finish this off. John, thank you very, very much for accepting to be the 14th guru on the Software Security Gurus webcast. It was a fantastic chat. Thank you very much.

John Melton:
Thanks so much. I appreciate it.