Software Security Gurus Episode #24: John Heldreth

Matias Madou:
Welcome to the Software Security Gurus webcast, I'm Matias Madou, the CTO and co-founder of Secure Code Warrior. And with me today, I have John Heldreth. Welcome John.

John Heldreth:
Thank you, Matias. Happy to be here.

Matias Madou:
Fantastic. John, would you mind introducing yourself?

John Heldreth:
Of course. For those that might not have heard my name before, my name is John Heldreth, I'm the CEO and founder of a group called ASRG or Automotive Security Research Group. We focus on building up the competencies in the automotive security area. This is what I do during nights, so in my free time so to say. During the day I'm also working for Porsche Engineering. Currently, I am the leader of product security at Porsche Engineering.

Matias Madou:
Excellent. I'm especially interested in your night job, to be honest. [crosstalk 00:01:07]-

John Heldreth:
It doesn't involve a cape or anything, I promise.

Matias Madou:
Let's see, John, ASRG, it sounds very interesting. I've done a little bit of research where you bring people together, but can you shed a little bit more light on why it was founded ultimately?

John Heldreth:
Of course. Actually in 2016, I was trying to build up the product security competencies of Porsche, and we were looking for talent. We were trying to figure out what was going on in the market, trying to understand what's going to make sense. How are we going to build up this topic within, not only Porsche, but Porsche Engineering? And then we started looking at the market trying to understand what is needed. And one of the things we saw is that in IT security, the community is just a huge part of not only self-development, but a huge information source. People, they develop their opinions. Twitter is a huge thing in the IT security community.

John Heldreth:
So we started looking for where are there opportunities? Where can people come together and talk about automotive, which is not like any other product. Not only from a technical standpoint, but process and market and so on. So we had really a new thing going on in the industry. There was no kind of community supporting this and we said, well, since we didn't find anything, let's just try building something ourselves and see if anybody else is interested to join, learn, collaborate, and network together.

Matias Madou:
So ultimately you broke down barriers between companies and instead of silos in their own organization trying to do something, what you're trying to do is bring people together and chat how to do better as a group. Is that correct?

John Heldreth:
That's correct. You have to understand that this is not a problem for a company. It's a problem for us as users of these products, which are vehicles, right? And if every company makes the same mistakes, actually we, the users, are going to reap all the, not awards, but all the issues, the failures, and so on. And to ensure that we do get better products, more secure products in the field, we said, well, why not come together and we talk about the issues. We keep IP aside, right? We don't talk about what we're developing as products, but we're talking about our experiences, our knowledge. Making sure people know when a new vulnerability is found and it affected our products maybe. We just give the hint, hey, listen, watch out. And this way we can build better, more secure products together.

Matias Madou:
I'm actually super happy that you created something like that because this industry is going through a dramatic change or a big change, the car industry. And I'll be honest, I'm surprised it's only five years old, the ASRG, because the transformation is going on for a while now. Back in the day, I drove a MINI Cooper and I don't think it had a lot of lines of code in there. Can you shed a light on, hey, what are we talking about today? What is a car in terms of software? How many pieces of software are in there? How many lines of codes? Is there any rough estimate?

John Heldreth:
Well, yeah. So actually there's some really nice statistics that are floating around the industry right now and I think it's something like, a fighter jet has around one million lines of code, but the newest vehicles with the newest features and functions actually have somewhere around 10 million lines of code in them. So we're talking factors of 10 here and we're just starting, right? So we're just starting to get into these new features and functions that are requiring such software intensive ECUs to be used. This is just the beginning, right? As we move into autonomous driving, stage four, five. Who knows? Five plus, six, whatever, you're just going to see it explode.

John Heldreth:
And we also changed the strategy a little bit too. Before it was we developed one ECU to do one function or a few functions, right? And now we're getting to the point where no longer do we develop something that's hardware and software specific, right? We develop a hardware that can be expanded, continually updated, features and functions can be shifted in and out and even between the cloud. So you're not actually making decisions in the ECUs anymore. And this is really fascinating because we're actually being more efficient with decision-making, but ultimately this requires more software in the end, of course.

Matias Madou:
It always interested me cars and connectivity and how the components were talking to each other. And one thing I did back in the day, 10, 15 years ago, was hooking up a CAN bus reader into my car and essentially encompasses some sort of an open bus system, but did simple record and replay, not attacks, but the blinkers went on, I recorded that, and then you redo that. Is that still the thing? Are we still talking, hey, the CAN bus is the central thing where all the components are on or has that evolved to something completely different these days?

John Heldreth:
The CAN bus is still there, definitely. It really depends on what type of automobile we're talking about. If we're talking about the very lower level type of basic, I won't name any names here, but just simple vehicles that are of value. Yeah? There's a lot of CAN bus used, still used, LIN bus and so on. And as you know, as you've seen and also practiced probably, CAN bus is not a secure networking-

Matias Madou:
It's very open.

John Heldreth:
... protocol. Yeah. And well, that was never the intention, right? It was never intended, it was never designed to be a secure communication channel. It was there actually to provide a communication channel between ECUs in an environment where it's very difficult, very noisy, actually from EMC perspective. So it served its purpose at the time, it's still serving its purpose. It's still a great thing. It's still a great network protocol interface, or network protocol, but it doesn't support security. You can on top of it, on top of the protocol, introduce some type of security, of course, but it's not exactly what's...

Matias Madou:
So what do modern cars use these days? If it's not the CAN bus, do they have their own proprietary system where components communicate with each other or is there a new thing in place that everybody seems to use?

John Heldreth:
Well, there's not really proprietary physical layers, right? So mostly we are seeing a lot of CAN bus, right? This is still a thing. And actually from a tech model perspective, you must have access to the CAN lines to make an attack, right? So we still assume that the vehicle itself is protecting the CAN bus and in some instances say that's acceptable. There's also LIN interfaces, which is a single line communication, very low. I think 19.2K is the most and it's time synchronized. That's one possibility for lower level. We also have stuff like FlexRay, which is supposed to be like kind of the middle step between something like CAN bus and going towards this Ethernet type of networking a physical level or layer.

John Heldreth:
So we have FlexRay. There's also another kind of intermediary physical protocol level layer is the CAN FD protocol, which is also possible. This is extended CAN frame addition with some additional features and functions built into it. But ultimately we are, so at least the automotive industry in general, if we need high bandwidth or high transfer rates situations, we'd go in the direction of automotive Ethernet.

Matias Madou:
Okay. And so one thing that came to my mind when you were talking about that was, I know like in airplanes, well, you have the avionics systems. They are just simply not connected to the cabin system. So they are ultimately air-gapped. Critical things are simply air-gapped. Is that similar in a car or is really everything connected and even the most critical components are connected and you can influence the car, the driving of the car on these bus systems?

John Heldreth:
Yeah. So actually it's not so easy to answer. I would say, of course, it depends on the vehicle architecture itself. But if you do look at some of the basic kind of like structures, the way that people have set up their networks, usually they're either domain-based or they have kind of features and functions separated. So safety related functions or driving related functions would be coupled together and those would be segregated on its own bus, right? So you try and couple these together and then you have like, just like in networking, either a hub or a switch, that's actually managing the communication between these different branches, these different domains. So at least that's the strategy. You can even go one step further and you start looking at virtual networks and stuff like this when you get into Ethernet, but from a basic standpoint, at least we'd use the domain-based where we separate things and kind of control the communication between each of the segments.

Matias Madou:
So I would imagine like with, there's a transformation in software, first of all, like it's much more connected. People expect more from the software in the car itself, but then there's a second stream that is going on, which is the move from combustion engine to electric engine. Which is, I would assume again, more ICS in the car, more connectivity. So it makes it even more complex. These two drastic changes that are going on collide at the same time, essentially.

John Heldreth:
Yeah. But when we start going about physical mobility, how are we actually moving the vehicle? Actually, we have the same challenges that we did before, just a different solution. So before we even had the internal combustion engine, we had to control it. We had an ECU, which was managing all of that. We had huge application tables to control how the motor is behaving and to safeguard against itself. So I would say that that's just changing or transforming from the internal combustion engine over to now [inaudible 00:13:25] electronics. Sorry, I don't know what it is in English, but these ECUs that are controlling now motors instead of engines. And it's the same also for steering. For steering, it's just the same as well. We have steering assisted systems with motors there. So it's a big change for the industry of course, but when we talk about the development cycle, it's kind of the same.

Matias Madou:
Okay. So then on top of that, we have even a third one that I can think of, which is self-driving cars. And I'm not sure where you are, but if you're in the car industry you must like cars and you must like driving cars. So I'm not sure where you are with, hey, you know what? Are going to be an early adopter or a laggard with self-driving cars?

John Heldreth:
[inaudible 00:14:21]. I mean, just from my point of view, you have to be an early adopter, right? You have to take the risk because the investments, the learning curve, all of these types of things, is so big that actually if you don't get it right and be on the first part of this curve here, you're going to lose your business, the trust in your clients and stuff like this. I think this is much more important. We're not going to get everything right the first time. It's not going to be perfect. However, we do need to take the theory and put it into application. And if no one takes that risk, if no one takes the first step, then everything will stay in the research domain and never make it really into the consumer market.

Matias Madou:
Yeah. Safety is the most critical in that area, right? I would say for the people that are creating the cars, everybody is looking at, hey, how are they going to develop that in a safe environment? But I would assume there's more to that. There's also the user of the car that still needs to handle it with care. What are your thoughts on safety aspect in terms of self-driving cars and vehicles in general?

John Heldreth:
So I have to first say that I'm not an expert and I'm not into the whole discussion, kill the driver or kill the person outside or whatever it is. This probably is not the direction we want to go. But when it comes to autonomous driving and stuff like this, the safety of the vehicle is actually pretty well developed. We have standards, we have vehicles being developed. Safety standards, [ISO 26262 00:16:15], and these types of standards are there to help protect the vehicle from itself, right? But now we're trying to protect the humans or the users, the drivers, the passengers, from themselves.

John Heldreth:
And now we get into a big discussion, okay, what is really driving? And if you ask every person what driving really is, you'll get a different answer. So how can you, or how can a company, an OEM, understand specifically what driving really means to their clientele, their customers. We have to follow the rules. We have to follow the [inaudible 00:17:00]. We have the [inaudible 00:17:02]. We have to follow all these laws, regulations, but at the end of the day there's going to be a driving style that's also kind of downloaded into each of these vehicles and you're going to get a different feel. Everything is not going to drive the same way.

Matias Madou:
That is interesting. So essentially you have profiles that can download on your driving style. That is super interesting. Maybe last topic on ASRG, actually, no, I actually still have two. One is, well, now that we have that car, I had a quick look at my car and I could identify already four ways of connecting with the car. I could identify like, hey, the software can be uploaded or downloaded over 4G, 5G network. I'm able to go over WiFi to my car. There's a Bluetooth connection to my car. That's number three. And number four was my key itself was making a connection, a wireless connection to the car. Are there more points of interaction with the outside world? Is it four or is it, no, it's way more?

John Heldreth:
So this is one of these activities you do during a risk assessment engagement, right? And you start looking, okay, how could I approach this? If I was an attacker, or if I wanted to get access to an asset, how would I get in there? And I have to admit, it's almost like an art, right? Because some of these guys are really smart. Yeah, just like you said, any of these interfaces, like Bluetooth, Near-Field communication. Think about your tire sensors, which are sometimes wireless, depending on your car. All of these wireless interfaces, WiFi, even hotspots for WiFi, 4G, 5G, 3G, 2G. Don't forget 2G is probably a backup system for some of these car companies that are thinking cars are going to be in an area where they actually don't get good 3G or 4G service. And so they can go down to 2G as a backup, which is ultimately not very secure and can be very easily manipulated and faked, so to say.

John Heldreth:
But before we get into something else here, think about all of the other ways you can get contact to the car. So this is the wireless stuff, right? This is stuff that you can do from being away from the vehicle-

Matias Madou:
Yeah, not touching the car.

John Heldreth:
... but once you start to touch the car, once you get closer without being inside the vehicle itself, not having any access, where are these CAN interfaces going to? Where is these LIN interfaces? All of these communication points are accessible. Sometimes you can even just cut a hole in the sheet metal and get access to it, right?

Matias Madou:
That's true. I actually thought about that. It's interesting.

John Heldreth:
Even headlights these days. I think about just how easy a headlight is to exchange, right? You just kind of rip it out of there. Or a taillight, which is smart these days. So they're actually CAN connected depending on the vehicle.

Matias Madou:
I was going to pick up on your 2G. You said, hey, it's insecure and we still need to protect against that. I would like to dive into problems. There will be problems with cars, and there will always be problems with cars. Personally, I think it is super important that people do not exploit it, but if they do exploit it that they are telling the car manufacturers or the ASRG group in general, what is the best approach for people that find problems in cars?

John Heldreth:
So, first of all, before we go into disclosure or something like that, let's talk just two seconds about our responsibility as drivers, as passengers, as users of these products. Our vehicles, our automotive infrastructure, is not going to be perfect. Okay? The OEMs are actually doing a great job of trying to get everything a hundred percent, but there is no 100% secure solution. If something is secure, secure today, tomorrow it's completely unsecure. So we always have to keep in mind that listen, OEMs, they're working hard. Even suppliers are working hard to make sure everything is secure, but it's our responsibility as well. Like when we purchase a product, we shouldn't just trust that everything works as it is, that it's secure or safe. It's also our responsibility to look into this, right? To test it, to challenge it, to tear it apart if you want.

John Heldreth:
And this is just my viewpoint, but something I reiterate at ASRG very often is listen, it's our duty as users of these products to challenge, test, make sure that what is happening on the OEM side, which is ultimately budget-driven, right? They're trying to make a profit. That what they've delivered is really secure and they're not going to be perfect. So there's always going to be something to find. And I just want to bring that across that, please, go ahead and test things. Look into things. Hack it up. Look into everything. Have fun with it. But then at the end, if you do find something, it's really important that we disclose things appropriately, responsible disclosure. And I think this is where you were going, Matias.

John Heldreth:
Responsible disclosure is really important because we drive all these vehicles, right? And if we find something it's really important that we talk to the OEM, to the people that have produced these products. Because actually, even if you say, well, I don't drive this car, I'm just going to go to the magazine or the news article or put this on Twitter or whatever, just to get a little bit of fame, you're actually putting a lot of people at risk. Or you're putting your families, your friends, maybe whoever is driving these cars, but don't forget these cars are actually driving next to pedestrians as well. And so it's just really important to make sure that we are disclosing things responsibly.

Matias Madou:
In the car industry, I was just wondering, back in the day companies weren't really set up to accept responsible disclosures. They're like, well, what are you doing with our products? You're taking them apart? We don't really like that. And a lot of these companies already went through that transformation. I would assume that the car industry also went through that transformation and now they are set up to accept responsible disclosures and they see it as a very good thing I would assume.

John Heldreth:
Right. Definitely most OEMs have some type of responsible disclosure program. Even a lot of them are working with companies like HackerOne or Bugcrowd to actually even promote it in a more controlled atmosphere. This helps them be able to get some other viewpoints. People are testing their products. They're also getting paid for it, which is great. But most companies do have now a way to get in touch with the people that can make change. Right? I do have to admit it's not so easy sometimes and ASRG is also in that position. So if you do find something, you can contact us at security@asrg.io. We can get in a discussion with you and help mitigate. We usually have the right contacts and then you can stay anonymous if you want. Or if you're looking for some kind of retribution, you can also go into discussion with them as well.

Matias Madou:
Excellent. That is very good to know. Let's switch topics. I would like to go to modeling of cars. And the way I think about it is like this car, this is always the same thing. It doesn't really change. If you have the car, the car is a car. It doesn't make movements or whatsoever. Yes, the tires are rolling and you can steer it, but ultimately the car stays a car. But the entire environment around that car is changing. If the car is driving in a rocky terrain, if the car is driving at five miles an hour or 200 miles an hour, that's a very different thing. So I know it's a thing that you really like, modeling cars. Can you tell a little bit more like, hey, how do you do that? How do you model the car and then ultimately the environment?

John Heldreth:
Yeah. So modeling in automotive industry is a huge topic. There's many different use cases. From motor optimization or application, simulating, okay, how and when [inaudible 00:27:06] and at which angle and everything. How should it be done? Okay, where do we set the diagnostic limits [inaudible 00:27:13]. Simulation is used almost everywhere. In the drive terrain. In the shift points for the transmission. For, oh my goodness, how much pressure might be created in the washer fluid? There're so many use cases here that the modeling really helps us get a starting point of where we need to understand like the application, right? Because ultimately we're going to go drive the vehicle. We're going to see how it responds, behaves, and then make adaptations based on our basis simulation model, right? But in the future, we're going to have simulation in a whole different perspective. And for security, we talk about digital twins. Have you heard about digital twins, Matias?

Matias Madou:
No. I have not.

John Heldreth:
So digital twins are going to enable us to simulate what the expected car behavior was. Right? So based on all of the sensor data, the vehicle is going to continue driving, make its own decisions, but also the sensor data is going to be sent to the cloud, and this sensor data is going to be put through the same models based on all of, hopefully, the same version and everything. And based on what happens on the road decision and based on what happens on the digital twin side, we'll be able to see or identify potentially possible issues, not only on the security side, but also on the safety side.

Matias Madou:
Interesting. And so who's making the decision then. Is it still the car or is it the digital twin?

John Heldreth:
Of course, ultimately the user, him or herself is of course always, especially in the autonomous driving topic, the driver always has the highest priority. The vehicle itself actually has more than one decision path. Usually there's a priority set up and based on every organization's different viewpoint of risk, they'll make different decisions. So it's not so easy say, well, it's always the car, it's always the person, it's always the cloud. But I would have to say that definitely it's not the cloud at this point in time. And the decisions are always made where the information source is generated, right? Sensors, [crosstalk 00:30:01]-

Matias Madou:
The reason I ask is I can see initially why it is the car, but I can see over time why it could potentially be the cloud because the cloud has more information on other people that drove on that same road and experienced similar things. If you sprinkle AI on top of it and the cloud learns, I would assume that the cloud has ultimately or hopefully more insights and can potentially make a better decision than the car with the limited CPU power and the limited environment information that it has. But maybe that's a far-fetched future, I don't know.

John Heldreth:
We start to get into predictive driving, right? So we're getting information from some other source that's ahead of you on the road. Maybe the vehicle determines a slip, maybe an oil spill or something on the road, and of course then it's sending this data either to the cloud or directly vehicle to vehicle and saying, please, I noticed that resistance between car and road has changed, be careful. But ultimately it's the decision of the car what to do with that information, right?

Matias Madou:
Yeah, that is true. So, John, I have a final question for you.

John Heldreth:
Yes, sir.

Matias Madou:
You're in the car industry, and if you go into the car industry you must love the sound of a car. So my question to you, are you driving an electric vehicle or when are you going to drive an electric vehicle?

John Heldreth:
So if you look at being friendly to the environment, yes, I do want to drive an electric vehicle. I do believe in the idea of electric vehicles and the ability to help save the environment a little bit. I do have one concern though. I'm really worried about the battery waste and usage. Where do we put all these batteries? What kind of environmentally saving model are we going to have in the future so that we just don't have stacks of batteries laying everywhere? Or I don't know. So definitely I would drive an electric. Right now I think I would be more interested in driving the hybrid. I love the technology, their marriage between the internal combustion engine and electric motor and those technologies altogether in one. So really great [inaudible 00:32:40] that's been done.

Matias Madou:
I actually couldn't agree more on the electric. I think there is a lot of stuff that we still need to figure out. And this is going super fast, maybe a little bit too fast. Batteries is one, but also the infrastructure on how to distribute electricity. Can the infrastructure handle all these electric vehicles that are going to come on the roads. So I do think it's going to be a bumpy road ahead until we are fully electric.

John Heldreth:
But don't forget that actually electric brings in a performance topic as well. We have instant torque in an electric motor, right? And some people say, oh, well, it's not going to have the same performance as an internal combustion engine. I would disagree. You see a lot of hypercar companies coming out with electric motor strategies or support for electric turbos and all this stuff, just because it makes more sense. You can get more performance out of the vehicle. So I think we're going to see a change over time from the internal combustion engine to the electric motor or electric vehicles. And it's not going to be so much environmental, save the environment, it's going to be, hey, listen, electric, this is the performance. This is the race topics. I think there's even E Formula 1 now.

Matias Madou:
We will see more of that. John, thank you very, very much for coming on onto the show. This was a fantastic chat. Thank you very much.

John Heldreth:
Yeah. Matias, thank you very much. I always appreciate talking about stuff like this. So thank you for having me. Looking forward to discussing further with you.

Matias Madou:
Absolutely. Thank you.