EPF Office Hours AMA - Fredrik

Anyway, um welcome everyone. Uh we have another office hours today and after a brief pause, uh we are back in the EPF 7 office hours uh with uh AMAs with a special guest. So, we have the series of calls where we invite uh some senior core developers, people from Ethereum to um uh tell you about uh their background, their work, and um inspire you to work on the core development. So, uh it's my honor to welcome here today Frederick. Uh Frederick is uh the uh one of the leaders of the protocol cluster in Ethereum Foundation.

Um uh before he worked on the security team uh making sure that the not just the protocol, but a wider Ethereum ecosystem uh has proper security practices. So, um I think he's a very uh important person and very interesting person to talk to and to meet. I hope you prepared some interesting questions for him. Um and uh welcome Frederick. Uh so, good to have you here.

And um Yeah, uh

Yeah, thank you, Martin.

Of course. Uh and before we get from to the questions from the fellows, I uh just wanted to get started uh with a couple of my own questions just to uh just uh so they can get to know you better. Maybe a bit about your background like uh I was wondering if you can tell us uh more about yourself, about uh your background before Ethereum like uh what did you do uh uh before your Ethereum journey, uh what did you work on before, uh what did you study, and uh what led you what what what what ultimately led you to to work on Ethereum?

Yeah, of course. So, yeah, I um I I I've been around for quite some time now. Uh I Um I see So in high school I spent a lot of time studying uh like corporate economics and stuff. Um I figured that that wasn't really my thing because I was very interested in computers and IT and especially security. So after high school I I started working at this uh uh so like IT support company where I spent a few times uh talking to people, helping them how to reset their internet and install their printers and a bunch of other stuff uh which uh wasn't I guess the most interesting thing to do, but I think it was very interesting like it was very useful in terms of like learning how to speak with people from a lot of different um tech skill levels, which I think was very valuable um afterwards.

But uh yeah, after a few months I I left and I went to work for Blizzard Entertainment instead and uh we worked on uh uh a game called World of Warcraft that I'm not sure how many of you have played, but yeah, we we worked on that. I spent spent most of my time analyzing like large-scale databases of what was happening in game uh trying to prevent people from losing their money, uh stopping people that were abusing the game in various ways such as automating it through bots and yeah, basically detecting and and banning and uh yeah, getting funds back to people, so I guess in a way like, yeah, tracking what what happened, just like you can do on chain today, um, which a lot of companies are spending time doing. Uh, so we did that, but it was, yeah, in the game, basically. I spent four five years there, four and a half five years. And, uh, then I decided I wanted to focus more on, let's say, actual IT security, so I moved to another company, uh, which was later acquired by Orange, which is this French company that has I guess, uh, um, services across the world with internet and stuff like that, but what I worked on was the European market, and we were focusing on delivering service security services to critical infrastructure entities such as governmental agencies, uh, central banks, airports, uh, um, like ship, like boats, shipping entities, uh, national health care system, police, and those kind of things.

And, that was pretty fun. The thing is that it's usually very slow moving. Like, if you're a central bank, you probably don't want to use the leading tech, you want to use things that has been proven to be very, uh, stable and and such. So, I I learned about this thing called I guess, Ethereum in 20 16 2017 and that was really cool. I I had heard about Bitcoin many years before um but it it didn't really strike me as super interesting.

Ethereum on the on the other hand was way more interesting because I could immediately see how it could be used outside just being say a currency because you could utilize like at the time things like Amazon Web Services Azure and Google Cloud and such was very like prominently used in in IT. And when I when I saw and read about Ethereum I was like this is actually something that could be utilized in similar fashions or I could at least map the use cases to that but it's more decentralized, it can provide censorship resistance and another type of security. So I got really intrigued in that and I spent a few years looking into it. Um and also all clear ones and other types of projects. And then Justin Drake ended up sending out pulse of X.

Uh I think this well this was in 20 2020 and he said that the EF was looking for security people to help with the merge basically. So at that point I I reached out cuz this had become like my true passion. I spent some time obviously at work, but then my full spare time was basically spent on blockchain. So, this was super interesting. It still felt a bit scary to like take the plunge and move over.

Um So, I spoke with Justin many many times, like thought about this, but then then I ended up joining and uh Yeah, I've been super happy I did. This has been the most amazing thing in my life and uh super grateful to be part of it. Um Yeah, I've been leading the I started by leading the protocol security team. And then I spent a lot of time uh doing things like the trillion-dollar security initiative. And uh Yeah, like Mario said, right now I'm uh I'm leading the Let's say the right side of uh of the protocol cluster.

And the right side consists of more of the engineering teams. Uh which means uh security uh geth networking testing specifications. And uh DevOps or panel ops, basically. Um acting as I guess you can call it a facilitator or whatever to to those to those things.

Yeah, I mean Yeah, it was very big shoes to fill for sure. Martin has been around for ages and has a deep like extra he had an extremely deep knowledge about Ethereum. So and then Ryan obviously as well and then many others. So coming in without all the context that they had was obviously quite scary. But I mean everyone I think this is something that I have noticed is that everyone in in this space is extremely willing and happy to collaborate and to share.

There is no one who is like oh, I'm not going to tell you this or look down on you before you ask me because you're asking questions and stuff like that. This is something that people actually see as a positive thing. They want you to ask questions. They want you to learn, they want you to to excel and advance. So, yeah, I it was scary at first and it's very like it's extremely different, I would say, from working at a traditional web two or normal I say normal, but a normal blockchain type of entity.

There's a lot more, you know, you need to do things on your own accord. You can't just sit around and wait. You need to figure out what's most impactful and then end up doing it, basically. And this can be a bit challenging for people in the beginning, especially if they're used to more of a traditional entity, but yeah, I think everyone that I've talked to, and this I mean, this is also true for me as well, has said that this is they don't want to go back to working at a let's say normal type of entity. They want to continue working in open source, they want to continue working with public good items because it is so much more more rewarding, basically.

Yeah. Yeah, I think the some of the most um well, exciting things for me right now is that we have made this kind of shifts into I guess narrowing down what we are focusing on. So, with the mandate, we said we're going to focus we're going to go all in and focus on censorship resistance, openness, privacy, and security. I think this has been it's been very good cuz now we have a mm big better understanding of like these are the things that that matter, and these are the things that we need to do in order to make sure that Ethereum can continue to thrive for, you know, 100 years or whatever. And um with regards to the constraints or yeah, constraints there are Well, let's start start with the I guess challenges.

And And one of the challenges that we have a lot of things that we want to do. Um but there's only so much time to do it. And I think if you look at things like post-quantum, which we don't really know when it's going to come, but from the sounds of it right now, Q-day is going to come in 2029. And what that means is that, you know, if you're a blockchain or anything that's using cryptography and you don't have you haven't prepared yourself or added um or fixed your your your protocol so that you're not affected by this then at Q-Day it's possible that you know someone could end up compromising the encryption and stealing funds as an example. Um and this would obviously be very bad.

So this is a big constraint uh which means that we are constrained in time and a challenge there is then to decide okay based on this constraint that we do have we have 2029 right now who knows if that's going to advance or not based on like AI advancements which has been quite crazy in the last 2 years or even 1 year um and if you think about what's going to happen in another year. So we need to figure out okay what does that mean for future hard forks as an example. Um how much can we fit into them what kind of things should we fit into them when it comes to upgrading. Uh so that's that is a concern. Um I think that's what's mainly on my mind right right at this point at least.

We have just problems to solve. I hope that um fellows can get some some inspiration from what I'm working on related stuff already. Uh and then now it's my question for you folks like what are your concerns what are your questions? Guys I hope you uh have some some questions for Frederick anything you want to chat about with him. I wanted to just do these first couple of questions to have some introduction and some overview so you understand who Frederick is, what's his what's his role and his background, but now I want to hear some of your questions.

Please raise your hand and and um uh uh feel free to ask anything. Um yeah, Connor has a question. Go ahead.

Hello. Uh I'm just wondering as a secure like somebody focused on security, if you think of like proof of stake security and a proof of stake attack um in the context of like all of the other stuff that can go wrong with Ethereum on the social layer, how much would you say that like the cost of a proof of stake attack, like how many billions it is, how would you describe like how much risk is there in terms of protocol security versus actually like ACD, like the ACD layer and the ability for other people to actually change the protocol or influence core development. Where would you say the bigger risks are?

Um I would guess well, my gut feeling would be ACD as in that is some potentially easier attack vector because like you say it requires a lot of money to do it and even if you do it, you can do social recovery afterwards which means that, you know, it is you know, someone could have bought all of this ETH, spent it and then the ecosystem decides to do a social social slashing or social recovery uh to get get past this basically. Um while ACD is you know, you could do some I don't know. When I think about security, I always think in terms of social engineering being the easiest way to basically breach a system. Um when we were doing pen tests and such, if we couldn't get in through any vulnerability in the system, we could always get in through a social engineering attack. Like that was always possible.

And I think this is something that I'm not saying like doing this on ACDs easy because it's not. There are so many entities uh keeping track on this and there are so many people and it's it's open for everyone to listen to. So, there are so many ways that this is being protected that I don't view it as a as a major risk, but it is of course something that we need to keep an eye on to make sure ACD is not captured in some way by a an entity that is not looking on for Ethereum long-term and be that let's say no clever social engineering type of attack is implemented in the protocol. So, that's yeah, that's how I view

Yeah, I can I I guess you can also say that I I have spent quite a lot of time working on let's say well, protocol security obviously, but also things on the layers above. So, I've been collaborating a lot with the the security alliance and uh the red guild and other parts that are uh yeah, working on security on the layers above. Um In case anyone has any questions about that.

Hey, yeah, I have a question. Um so, what do you think protocol engineers should do more if anything to make sure that their changes and what they work on is not affecting the security of the protocol? Like, are there places that you recommend more focus like spend more time digging into specs or read other clients' code or focus more on testing like fuzzing invariants? Any any practice that you would like to see more?

Uh yeah, I think the tests are probably the some of the most critical stuff. Um They are the ones that catch the vast majority of issues that are introduced cross-clients. And this is super important. We find divergences between clients quite frequently through tests. And yeah, this is the say the first uh wall of defense that we do have.

And then we do more specialized testing after that or like reviews and audits after that. And that could be fuzzing for example. Um as well as manual reviews and using LLM technology and such to review it. But yeah, the the tests is something that we we put a lot of value in and importance on especially when uh when something is introduced as some EIP. Like making sure that there is a proof of concept, there are tests that clients that can implement.

Uh this can be as a newcomer, it can be pretty hard perhaps to do this and think about all the edge cases and so forth. But like to go back to what I said before, if someone do reach out to the specifications team or testing team, uh they are from what I've experienced myself and from what I heard from others, they are very happy to to help guide how to do it and and um yeah, come with come with feedback basically on like this engineering idea should have these types of tests. You maybe say I have these 10, they say well, here are five more ideas that you can add etc. And and also speaking with people, getting getting their input tends to be very valuable in terms of thinking or like coming up with edge cases that it's hard because it's very hard to think about every single thing by yourself. So yeah, getting more people involved in something is usually also a key to success to success.

Um yeah, Frederick, thank you very much for your time. I wanted to ask about the security and your thoughts on machine checked to approve selling for the security to purpose, especially considering some potential compiler bugs. As like recently I saw a guy kind of he proved that one does not equal zero, for example, because of there's some compiler bug.

Yeah. Um we haven't really started to look into it that much from a security point of view, from at least protocol security side. I think maybe Cody from the research side has started to look into it, but I'm not entirely sure. Um I don't know, I think well in general, this is I this is probably how we would apply it to this as well, is that like you mentioned, we do this type of walls, or you can call them layers. Um you know, in traditional security you would call it the security onion, where you have these different types of ways of ensuring that things are secure.

And so if you only depend on I don't know, formal verification to verify that something is secure, you're probably going to run run into issues. Um because it's very hard to make sure that everything is completely specified and everything can be fully formally verified, and you can uh you might even with the specification, there might be issues with that. Like, the specification could have bugs in it, which has been the case um with with the Ethereum specifications that at one point we we found a a bug there. Um So it's Yeah, like I said, it's about using these different types of security layers and um not depending on a single one, um but also using the everything in the toolbox because there is no silver bullet, basically. So, anyone who says, "Yeah, this formal verifying, that's fine."

I wouldn't buy that. Um there's a lot of stuff that people need to do to ensure that the the client or whatever is is secured.

Yeah, I am I mean the landscape has completely shifted and uh like if you if you go back I guess two years from now we started using LLMs. I mean this was before there was Cloud Code or Codex or anything like that. Uh back then you either had the webpage uh or the web GUI or you had the API. And uh we ended up building tools that used the API to for example review PRs and things like that to find issues. And this was something that um yeah, proved quite successful.

We found issues uh where if if it had been merged then it would have been a mismatch with the other clients. So there would have been a uh fork basically between the clients. And and this was quite useful. And then you know, things became more advanced. Cloud Code came out.

Um Codex came out and a bunch of other stuff and I mean the vast majority of things that we find today are through the use of AI tools in some type of form. We still do just manual reviewing from time to time but and especially when there's a hard fork. But the vast majority of stuff is found through LLMs and the reason is that they can just hammer on for hours looking at things that might have been overlooked 4 years ago, 8 years ago, and that no one is really looking at now, but the LLM can just hammer at this and try to try to find the issues. Um they're not time constrained like humans are. Um so they you can just launch a thousand of them, have them just run your idea and do it for 24 hours until until yeah and and see if you find something or not.

I think this um I this is going to become more and more advanced. Um I do think it's possible that we will be at a stage where vast majority of vulnerabilities will have been found. Um Maybe in a a year from now. And that's I'm I'm not saying this for like Ethereum, but but for every piece of software basically that you would be able to basically reduce the amount of potential vulnerabilities down to almost zero. Still going to be some issues obviously, but not not as it is today.

And I think there are two reasons for this. A, the cybersecurity capabilities of the models are going to improve, but maybe more importantly, more people are going to start using models to code and those models will have the security awareness uh or security mindset built into it. Um There's going to be a progress, obviously. Like if you if you take a year if you go back a year from now and you decided to code an application just by coding it it would have had vulnerabilities in it. Um but I today it's better.

It might still have some issues in it. Maybe it adds an API key to the front end or whatever, but things are improving quite a bit and I think a year from now it's going to be even more. So I I do think kind of similar to cars, how AI-driven cars the concept or idea is that there will be fewer accidents because there's going to be fewer humans actually driving and the the machines or LLMs or AI cars will be able to to drive more safely, more coordinated, etc. And I I believe we're going to be in a similar position with regards to to code as well. Yeah, for sure.

Like I my main concern today is probably bytecode and software and especially on on it like protocol critical software being bytecode and like that is what keeps me up at night in a way because I don't trust it. Um but there's probably going to be a time when we can. I'm not sure when that is going to be, but for now, I I definitely don't trust it. And I think another thing I should probably mention is like today the the best models are like from a capability point of view, most likely closed source uh frontier models, but I do think there's going to be a shift there at some point. I think we're pretty close, but I do think we're going to get a shift at a point where these open weight models that you can run locally and you can, you know, you don't have to worry about censorship or privacy or other parts, those will become more efficient and uh yeah, I think that would also be extremely extremely useful.

Yeah, I let me uh re- read that there are no stupid questions whatsoever.

Uh hi everyone. Uh I would like to know that um what is the security layers that you check on the Ethereum. And um like uh in a uh in in details if if it is it if it doesn't take too much time for you. Yeah, that's the question. The the layers of security.

So, one layer from the tech so far I get that you check um with the uh AI. And uh I I would like to know more about the other layers that you uh layers, tools, and stuff that you use and for a security. Thank you.

Yeah, of course. Uh so Yeah, like like you mentioned AI. I mean, that's a big layer today that we use um or we spend a lot of time on because it's it's very efficient. But uh yeah, we do spend a lot of time on other stuff, too. Um One thing in particular is uh fuzzing.

And um Fuzzing is actually very good on Ethereum because of because of the way Mhm. It was promoted to have diverse diversity basically. So, I guess to explain that a bit better. Um When you do the fuzzing, what you do normally is that you you look at you take an application, you check what it can do, and you say let's just toss throw a lot of stuff in as input into the application and see if we can make it crash for example. Or behave in a way that it shouldn't behave.

Um And this is something that you can do with practically any application. With Ethereum, you can do something else as well. You can do something called differential fuzzing. And differential fuzzing means that you throw something as input into application A. You look at the output.

Then you throw the same thing into application B. And you store the output. And then you compare the the both both outputs. And if there's something different there, then something is strange. Um Because they should behave the in the same way.

They should have the same up If they have the same input, they should have the same output basically. So, this is something that we can do with the with Ethereum. We have the the EVM. Uh uh the Ethereum Virtual Machine. That one handles the uh transactions that come in.

And if you send a transaction to one of the clients, it should behave the same way on all of them. So, what we do there is that we we do this thing called differential fuzzing. Um we have this tool called Go EVM Lab. It's got a public version that's kind of not been kept up to date. And then we have a private version that we have started to I guess use utilize a bit more.

But, the the public version is still available, and it is possible to run it against all the clients, etc. Um So, that is That is something that we do quite a bit. And you can do this on on on the EVM. You can also do it on on the consensus layer clients, like check how consensus is behaving between the clients, like do they have would do the clients behave the same way in every situation, etc. Yeah, the idea is that you just throw in a lot of randomness and make sure that they behave the same way.

This has been super effective at finding flaws, um because we do have so many different clients. It's like 11 clients total. And um you know, each client implements specification, but they also implement their own optimizations for the specification. And um when they do these optimizations, sometimes things behave a bit differently. So, yeah, that's that's what this is for.

Um we also do um manual reviews. So, um that means that you many times take specification, you look at a client, and you verify that they have implemented it according to the specification. Um on top of that, other stuff that's being done is like providing grants to external entities or people to do security stuff. Um we do security coordination, which is also very important, like sharing um best practices between the client teams or L2s or whatnot. Um and um Oh, yeah, we have the bug bounty program.

That is also one of the layers. Um Basically, the idea the idea behind the bug bounty program is that if all the other layers fail, like we missed to find something, then an external person can find it. And instead of them exploiting it, the vulnerability, they can report it to us, and they receive um some rewards for it. I think the highest reward we have right now is uh let me I should check so I don't uh say say the wrong number, but I know we increased it just recently. So, now it's Yeah, it's a million dollars for a critical vulnerability.

So, if someone finds a critical security vulnerability on Ethereum, then they and report it to us, they would receive a million dollars. So, that's like the um the backup plan, you can call it, uh if all the other layers fail.

Hey, thank you so much. Uh I would like to know about the the the security that you check. A two question, like the security that you check on layer twos, first of all, and also how you check uh the security on the network uh level. I know that you mentioned the fast testing and the um consensus layer. Uh but I mean the network layer just um the the network um itself.

Yeah.

Thank you.

So, yeah. So, we have for the network itself, uh uh I'm yeah, I'm if I um misunderstand the question, please uh let me know, but uh like if you look at the network as in the mainnet, then we have we have the DevOps team or PanOps team, they run a bunch of sentry nodes, and um and monitoring. So, if there is a let's say all of a sudden finality stops, or there is uh a drop in attestations, or there is uh an increase of missed slots, or all of a sudden like more than 10 validators are slashed, then um we do have monitoring in place and we have sentry nodes, so we can detect these kind of things. And when that happens, I mean, usually that would happen around a hard fork, like all of a sudden attestations or proposals um drop and it could be for because people haven't upgraded or it could be that there is a bug in one of the clients um because of that you know, there that client stops proposing or at attesting or it could be that yeah, basically those kind of things. So, that's kind of what you do from a monitoring point of view there.

Um for layer twos, um we don't particularly look into the layer twos themselves. Um Most of the time they have their own security teams, so we focus on the Ethereum mainnet, but we find a lot of we find issues in clients, like Geth or Reth or other like Nethermind or others clients that L2s are using. So, when mainnet has been secured, um we try to tell the L2s that you guys should be aware that there is a security issue with this client, you should do a release and get that out to your nodes, basically. Um but this is a bit of a sensitive thing because you can't tell the specific vulnerability to too many people because then there is a risk that someone malicious sees that vulnerability and exploits it on mainnet or an L2 or whatnot. So, there's a lot of coordination efforts being taken place there.

And going back to AI, AI has made this a lot harder, like significantly harder. In the past, we used to hide vulnerab- security fixes in um in releases. So, you know, you would have a release, you would have a bunch of code changes in them, and somewhere in there there would be a security fix. But now, this is a problem because in many cases some AI can figure out like has to been a security fix in a release, for example, or what is it, and that has shifted the way we have to do releases and announcements about vulnerability fixes, basically. So, now we try to uh like pre-announce in many cases saying, you know, at this time there's going to be an upgrade available.

We urge everyone to upgrade as soon as possible um to make sure that yeah, the network is secured, basically.

Thank you so much.

Oh, you just mentioned that fasting test is very useful for like identify the the issue in the protocol and some consensus client stuff. I'm curious about if you know about a simulation called like deterministic simulation test. So, it is uh it is a test you can simulate all the states in your distributed system. I'm curious if you know about that and how would this uh maybe maybe there's some useful case in Ethereum. And what do you think of that?

Uh yeah, so um I believe we have been using this system called Anthesis. If I recall correctly, this is what that was basically doing. Um I don't know if you heard about that, Jeff.

Okay. No worries.

Okay, I don't know if you heard about that, but yeah, basically we have been using that. And um we have it's been useful, but from our experience, it's been extremely time consuming. Um And that is why we haven't really Well, we we we ended up stop using Anthesis for that reason, that it just took too much time and but uh I think with AI now it's probably easier to automate this in a way that wasn't really possible before. Um But, yeah, in the past this was basically the case. But, we also do things like I don't know if you if you've seen Hive, for example, and and other stuff that's uh yeah, basically also using these deterministic tests.

Okay, thanks.

Yeah, thank you very much. Always a pleasure to be here and