Software Security Gurus Webcast Episode #4: Aaron Bedra

Matias Madou:

Welcome to the Software Security Gurus Webcast, I'm your host Matias Madou. CTO and co-founder of Secure Code Warrior. This webcast is cosponsored by Secure Code Warrior. For more information, see www.softwaresecuritygurus.com. This is the fourth in a series of interviews with security gurus, and today I'm super pleased to have with me, Aaron Bedra.

Aaron Bedra:

Hey, thanks for having me.

Matias Madou:

Hey, Aaron, do you mind sharing a few words about yourself?

Aaron Bedra:

Sure. My name is Aaron Bedra. I've been working in software/security for my entire career, actually. It's kind of been a pattern of rotating in and out of strictly security roles, and strictly software roles, and combinations of the two. I found it to be kind of my happy place in terms of what I do.

Aaron Bedra:

I spent time as a CSO, as a CTO, as a principal engineer/architect, kind of held I think most of the roles in technology throughout my career. It's been fun to kind of just see what all the different roles mean and how they function and kind of collect that into a series of things to share.

Matias Madou:

Nice. So fantastic. So for today, I actually have two topics in mind if you don't mind, and I hope they are near and dear to your heart. So the first one is around more of the regulation part, and the second one is around a holistic approach to software security. So let's start with the first one. So I think you've worked now for a decade or even more, if I'm not mistaken, into payment processing, health insurance treating. So I think by now you've had a lot of insights on how regulators work.

Aaron Bedra:

Definitely.

Matias Madou:

And sometimes how overzealous they are with attempting to weaken encryption, for example, in certain processes.

Aaron Bedra:

Yes.

Cybersecurity compliance: Europe vs. USA vs. APAC:

Matias Madou:

So how do they stack up against each other? And let's start from the top. Let's start with Europe versus US versus Asia.

Aaron Bedra:

Certainly, yeah. So in terms of regulation, Europe seems to lead the charge in terms of privacy, advocating for citizens rights to protect their data. Europe seems to be driving a lot of the personal protections, which is great. Unfortunately, application of this has been bespoke at best, I think, around I think GDPR being one of the more recent pushes, where it was a great idea, but as it's unfolded, it's been hit or miss in terms of support around the world. And also really even what the legislation been able to do across the world in terms of protecting people's data.

Aaron Bedra:

It's made people think, which is great, and it's made companies rethink how they're storing data. I've seen and been a part of this for quite a while now. It's been nice to have folks reevaluate, be able to as a person request that you be forgotten, request what data people have on you. We've always known that a lot of data about us floating around, but it's been interesting to see really how much it is now that things have surface. I think it helps people kind of cement the idea that digital privacy is really not much of a thing. There was a lot of expectations, a lot of assumptions, but I think this kind of now cemented that it doesn't really exist.

Matias Madou:

Yeah. And so you're saying Europe. Do you mean really Europe? Or do you mean Germany? Because even within Europe, there's quite a difference, right?

Aaron Bedra:

There is, yeah. You know, I'm not as familiar with every single country's restrictions. Usually more for me, it's the broad perspective of how I approach this when doing business or operating business in the EU. And now the EU and the UK has a separation, because even though there is still kind of an agreement in place there, things could possibly change. I'm thinking about that as something that I think everybody needs to be tackling, not lightly, before that time bomb goes off.

Matias Madou:

Yeah, yeah, yeah. And in the US, how do you look at privacy? And especially, I think states over there, there's a big difference state by state. If you look at California, I think they're maybe leading the pack in that area.

Aaron Bedra:

Yeah, there's quite a few states, and it depends on what they're focusing on. California and Nevada and Massachusetts, there's several different States that have different types of rules.

Matias Madou:

Mm-hmm (affirmative).

Aaron Bedra:

But the rules in the US seem to be more focused around corporate. You'll definitely be finding things with healthcare. The US has some pretty strict healthcare rules, I wouldn't say the strictest, but definitely seems to be the most impactful set of rules. With HIPAA, unfortunately they're a little bit gray there. I did spend quite a bit of time working in healthcare, and while HIPAAs rules are meant to be kind of short and sweets, they can lead a lot to interpretation, and there's no formal certification program. There are things like HITRUST and ANAC and other certification bodies that are private. And while HIPAA could fine you if you have a breach, they don't really have any way to formally certify. I think the payment card industry has PCI certification, and it's a formal process that everybody has to go through.

Aaron Bedra:

So HIPAA is a little gray there, but on the other side of the country, you've got the State of New York, and the New York Department of Financial Services being one of the first bodies that really, really showed its teeth in terms of financial regulation and data reduction. And they made a very extensive and very exhaustive list of controls based on the FFIEC guidelines. And I've obviously been a part of that working in finance. I've seen that from multiple angles now. And I think that probably has the most direct guidance of anybody I've seen so far in terms of maturity, risk measurements, completeness. PCI does have good guiding principles and an array of interpretation left open to auditors. But I really think the NYDFS rules are more complete because they think more holistically about a lot more broad topics than PCI does.

Matias Madou:

As a technologist, is that a curse or a blessing? Because I can look at this in two ways. On the one hand you can say, well, it's really a curse, all this regulation, it's a curse, it's stuff that we need to do on top of things. But at the same time it can also be a blessing where you can potentially get more funding for what you're trying to achieve, and you have some things to back you up. So how do you look at that?

Aaron Bedra:

So holistically I think it's a blessing.

Matias Madou:

Okay.

Aaron Bedra:

In a lot of ways. If you're a startup company tackling healthcare and finance, it's rough. If you're a startup in the United States and you're in one of those two places, you must secure extra funding to be able to get through this, right. Because it's not just about building this, you also have to pay for the auditing. And auditors are not cheap, right? It costs a lot of money to go through this. And when you're doing this, especially with a small team, it's almost everything you're doing. The amount of effort required to get through HIPAA or HITRUST, ISO certification, a SOC 2 type 2 certification, the NY DMS certifications, or following those rules.

Matias Madou:

Yeah.

Aaron Bedra:

It's a lot of work, and I think people underestimate how much work it truly is. And for me, when I think about that, it's not how do I get through the audit, it's how do I make my company live this?

Matias Madou:

Yep.

Building a culture of security compliance:

Aaron Bedra:

If you really go all in and make it part of your company, the audits are much easier. And not just year one, but in year five and in year 10, it's just part of your culture, right? And if you make it part of your culture, it's much, much easier to just have it fall out naturally. It's when you check the boxes and you're constantly fighting it, that it becomes difficult. That's when you have gaps, that's when you lose some momentum there. But for larger organizations, I think it is a way to put a little bit of structure around the security aspects of the business, especially businesses that really haven't had to focus on that much in the past.

Matias Madou:

Yep.

Aaron Bedra:

And I think that's really important to be able to set a little bit of foundation. And I've worked in some of those fields as well, where it wasn't natural to have information security as part of it, yet here the company is kind of in a new landscape trying to mature, and being able to set a budget and name a CSO, and have the report to the board. Those kinds of natural lines of communication when opened, I think can be healthy as long as they have the right people in place.

Matias Madou:

Yeah, I actually, I fully agree with that. I'd much rather work with an organization where they really want to do that, they didn't want to check the box, but they really want to do that. And yes, there is a regulation that backs up what they're doing, but it's much better over time. It's much better over time to do it in a decent way.

Aaron Bedra:

Yeah. And I do have an experience when I was working as a CSO at Eligible. The CEO there, Kaitlin, we basically stopped new feature development for the better part of six months, just to pursue culture shifting certification, to really dig in and re-engineer and re-architect part of the platform around the culture of security. And it was a all in team effort to make it work. And I mean, it's seriously kudos, right. To have a leader who realized this is really crucial to move forward as a business, as this matures, and make that kind of sacrifice, that kind of choice. That's really, really interesting, and a lot of respect there to kind of push off the short term opportunity in favor of a longer term kind of culture and mission. And I think it really showed throughout the last few years.

Matias Madou:

Wow, that's very radical. I haven't heard about that quite often where you really stop feature development to make sure that you have this thing right from the start, essentially.

Aaron Bedra:

Yeah. It was the primary effort. The best engineers in the team were involved. It was really a go all in and don't come back out until it' something that will last, which is really cool.

How developers can be more rigorous, using flexible systems to tackle vulnerabilities:

Matias Madou:

Very nice. Let's shift gears and let's tackle the second topic if you don't mind. So another thing that you've been focusing on from a security perspective recently is how development teams can be more rigorous and use principled methods to create systems that can easily change.

Aaron Bedra:

Yeah.

Matias Madou:

That includes the stability, resistance to coupling, and defining essentially an algebra of execution that makes verification of change more tenable. So what we can best learn on how to not create vulnerable software.... we will always have vulnerable software. We have to live with that, there's always going to be problems. So then it actually takes time to fix the vulnerability, but then you risk introducing new failure and essentially you go down the track where you can suffer greater loss by fixing the problem then leaving it in the first place. So how do you look at the holistic approach of drastically reducing bugs, which ultimately will reduce vulnerabilities, but you take a holistic approach in reducing bugs altogether.

Aaron Bedra:

Yeah. I mean, really, if you think about vulnerabilities, they're bugs in software. And if you can do your part to reduce the amount of bugs, you'll naturally reduce the amount of vulnerabilities. But for me it's always been, and will always be, a systems thinking and systems theory exercise. It's about how do you treat the system as a whole, not an individual component. Emerging properties sometimes are surprising, right. But rigor and discipline are the foundation of what made this work. Yes, sometimes you have to make something fast. Yes, IT is here to serve the business, to create function and form for the business. IT is not driving the business, and it can be selfish at times to go down too much of a radical. At the same time, discipline and rigor, I think, sometimes get sacrificed in favor of speed of execution.

Matias Madou:

Yep.

Aaron Bedra:

Things like always managing your dependencies and always updating them, there's probably a few companies who might have been bitten by that in the last couple of years that are very, very well known, where it's not about having that vulnerability. Yes, we can all do our best to not have them, but they're going to still happen. There's nothing we can do about that. Software will change, every day something new comes out and despite your best efforts, there's always going to be a vulnerability.

Matias Madou:

Yep.

Aaron Bedra:

And I think the most dangerous thing is not having vulnerabilities, because we all have them. It's the rate at which we can fix them. Once we identify them, can we turn around, upgrade that dependency, get in the code and fix that thing once, in one place and have it naturally cascade? Or does it take us months of testing to upgrade because we're 15 versions behind or because that one cross inscription vulnerability exists in 50 different places, or maybe you have 200 different places because we maybe chose the wrong library, or we built our own thing, or because we mismanaged encryption, things that could be fundamental in the system that are just broken.

Aaron Bedra:

And the other part is the coupling, right? Some parts of our system are so intertwined and so tangled that one small change trickles and you have that kind of butterfly effect where everything falls apart because one thing changed. And despite our best effort, every large system, every [inaudible 00:15:19] system has that one piece that everybody's scared to touch. And if you think about it, and you look at things, that's probably where all the bugs are sitting. It's the least tested code, it's the highest churn, if you look at the chart of files versus changes in git, it'll be the file that has changed the most often, it has the most lines of code, and the worst psychometric complexity, and the lowest test coverage. All the things that scream red flag. That's the area where you have that.

Aaron Bedra:

But you know, I think the discipline of breaking it apart, of writing tests, and you mentioned defining algebra. I think that defining algebra is very important. What you're trying to do, being able to describe what you're trying to do. I have recently been in favor of letting the play system kind of help guide you. I don't mean to say the types are the answer for everyone, they're not. But in some ways, when you have a type system that supports it, being able to let the types drive you towards a mechanism or an algebra where you can't put the wrong thing in the wrong place, it's nice, right. It helps you kind of be a little more principled. But really it's about that testing. It's about being able to put things in the right places and verify you've done so, as many ways as possible. And I think that always gets sacrificed in favor of, we need to get it done now. And that sloppiness is rough.

Matias Madou:

Yeah. So I checked out a video of you, a presentation that you gave a couple of years back, at I think Go To Conference, and what you're just mentioning is like, hey, how can people do better? Like what kind of steps do they need to take? But over there, you also pitch hey, we all have this monolith, and now we have to break it up into microservices. So you've now pitched a couple of ways on how to do better, how to be better in software development. But if you have a monolith today, there's still plenty of companies out there that just have the monolith view, you know.

Aaron Bedra:

Yes, yes.

Matias Madou:

Where do you start? Where on earth do you start? Do you start with testing? Do you start with breaking out one component? I saw this really interesting graph from you where you just say, you know what, building microservices is not just take the component, put it on a different server, and just connect to it. That is not building microservices, but where do you start?

Aaron Bedra:

Yeah. You know, monoliths aren't all that bad. I think it's okay to have a monolith. If your teams are struggling with delivery, if the rate of change is too slow, if you've put yourself into a place where it's too hard to move forward, then maybe it is more reasonable to break things up. Or organizational structure could be dictating this, team structure could be dictating this, size of teams could be dictating this. The

switch to microservices, for me, is part and parcel about agility, right? Can we do this, and can we increase our ability to change and be flexible?

Matias Madou:

Yep.

Aaron Bedra:

One of the most dangerous mistakes I see people make over and over and over again though, is breaking that thing into multiple processes and forgetting about the substrate. The protocols, the latency, how things are communicated, how things fail. A method of call is very different from a protocol exchange between two processes and two different machines. There's so much to think about in terms of failure and retry and traceability. You exponentially increase the amount of things that go wrong when you make a microservice.

Matias Madou:

Yep.

Aaron Bedra:

And that gets left in the wayside way too often. Just because HDP and Jason are simple and there's a thousand libraries out there that make them work, does not mean it's the right choice. And on top of that, because HDP is stateless, you have so much overhead every time requests are made. And I see that mistake being made over and over again, and it's two years down the road and all of a sudden it's like, okay, this is too slow. And it's because it became so bloated with the exchange of information.

Matias Madou:

So is it correct to say that you're more advocating for first get your engineering practices up to date and make sure you you can make quick changes, if it's a monolith or if it's microservice, it's irrelevant, as long as you can write features fast that are tested in a proper way and are deployed in a reliable way. Is that correct?

Aaron Bedra:

Yeah, yeah. Discipline and rigor wins for me every time. It doesn't matter what language you're using. It doesn't matter what framework. It doesn't matter whether the microservice is a monolith, all that stuff is a matter of how your team prefers to function and how they're optimally performing. That's more of a who wants to do what, right? Maybe you have a team of people who some like Java, and some like C#, and some like Haskell, and some like Closure, that's fine. It doesn't matter. You let people be as productive in whatever environment they can be productive in.

Matias Madou:

Yeah.

Aaron Bedra:

But, I think it's really about how disciplined can everybody be in the process and how confident are they going to be the delivery. I currently work in an industry in trading where mistakes, incorrectness can cost a lot of money. And so discipline and rigor is really, really important to me and my team as we're working on things.

Matias Madou:

Yeah.

Aaron Bedra:

And so that's part of culture. And I think that's the other thing, this stuff can't just be a poster on a wall, a motivational statement. It has to be something that the team lives and breathes every single day.

Matias Madou:

Yeah, especially in your area, I remember that a QA server that they didn't update one of your, I think a former competitors, where they didn't update one of their servers, one out of eight servers wasn't updated and they did random trades, and ultimately that company went under.

Aaron Bedra:

Oh, I think you're probably speaking of the night capital incident.

Matias Madou:

Can't remember, it's a couple of years ago.

Aaron Bedra:

Yeah. I mean very costly mistakes can happen. Not something I'll comment too much on, in that particular case. However, there's a lot of information out there that's publicly available and yeah, the cost of mistakes can be high if left unchecked.

Matias Madou:

Yes. Last question for you, Aaron.

Aaron Bedra:

Yeah.

Matias Madou:

Maybe a personal one, but I know you're into barbecuing.

Aaron Bedra:

I am.

Matias Madou:

And is barbecuing without a big green egg, is that the real barbecuing or is that not really barbecuing?

Aaron Bedra:

I think anything can be real barbecuing. I liked the egg, I think it's pretty great. It's kind of like cheating. I do quite a bit of that. But it's definitely, definitely fun, especially when we're all at home and nowhere to go. You need something to keep you busy.

Matias Madou:

Yeah. I saw a couple of pictures from you in the last couple of weeks with that big green egg. So yes, clearly it keeps you busy.

Aaron Bedra:

Yes. It keeps me full too.

Matias Madou:

Aaron, thank you very, very much to be the fourth guru on the software security webcast. It was absolutely a fantastic chat. Thank you very much.

Aaron Bedra:

Thank you.

Matias Madou:

Thank you man.