Post-Merge Ethereum Client Architecture by Adrian Sutton | Devcon Bogotá

foreign [Music] what an awesome stage to give my first Devcon talk on uh it's fantastic to be here thank you for taking the time to come and listen uh so I've got a fair bit to cover I've had to kind of keep cutting slides all week because I realize how small 25 minutes is so let's get into it my name is Adrian Sutton I'm a lead blockchain protocol engineer with consensus I started out four and a half years ago actually working on hyperledger Basu before it was first released back when it was Pantheon and more recently I've been focused on techu the consensus client so I've had some experience with both the consensus side and the execution side building new clients and bringing them and getting them ready for production and then going through and uh yeah probably the highlight of my life now will be having having seen the merge complete and go through so very proud of that and very excited for for ethereum what we're going to talk about today is some of the things that we didn't actually kind of get done and some of the consequences of the way we did the merge which was fairly minimal deliberately so we could get it done so we'd get rid of all that burning proof of work so uh I wanted to tell you everything everything you ever need to know about ethereum clients I'd love to talk to you for a couple of hours about that about consensus and execution layers but I've got 25 minutes that's a bit of a problem so we're going to cut the scope down a bit and we're going to skip over some of the finer details we're focusing on the more high level stuff here the architecture how things fit together this isn't the kind of talk where I'm going to tell you about the nitty-gritty details of how the engine API Works a little bit of assumed knowledge that you know ethereum here but I think that's pretty safe for this audience I think you should all be able to follow along pretty well to give you some background there's three scene three key things that I want you to know about ethereum clients and that'll form a real grounding for the things we're going to talk about post merge and what they mean and the impacts of that so the first thing is there is a lot less difference between consensus clients and execution clients than most people believe yes they do quite different jobs yes at every opportunity when we were designing the beacon chain we said nah mate we can do that better let's pick a different technology so there's basically no shared code between the two but there are a lot of shared concerns and Concepts and there is a lot that we can learn by talking to The Other Side by consensus client developers talking to execution client developers and vice versa networking is probably a really good example of that in that the core concept of it really has three parts the first is Discovery we need to find peers second is Gossip so that new information flows around the network quickly and the third is request response which is the ability to say hey I missed some data can you give it to me or I'm syncing how do I catch up each of those are quite different Technologies for Discovery consensus uses Discovery V5 executionally uses Discovery V4 for the other two it's lib P2P versus Dev P to p and in terms of the formatting on The Wire it's ssz versus rlp as well so completely different Technologies underneath it but still the same concern of how do we find peers we can trust because the world is out to get us in client development we don't trust anyone and there's this sea of peers how do we find the good ones how do we find the honest ones and how do we get a good view of the network and know what's going on things like civil resistance are common across clients dos prevention and those kind of peer scoring algorithms the trade-offs between reducing latency when we're gossiping data by broadcasting it to more peers and then the the duplication of that data and the increase in bandwidth and then ultimately they're both blockchain clients they're both dealing with this tree of blocks and this concept of reorgs which is fairly unfamiliar for people who are new to client development this idea that the world is is you know fixed here it's not like a single database suddenly we we realize hey there's a better chain over here and we reorg how do we pull out say transactions or attestations and other operations from the chain we were on and check that they all made it onto this new chain lots of code and lots of logic flows around that and lots of impact throughout the client in the way it's designed the way it works and the kinds of things we have to think about when dealing with algorithms so on the consensus side probably the big thing that we keep coming back to and drives a huge amount of our decisions and has impacts everywhere is the fact that proof of stake is not just a block driven blockchain it's also time driven so even if no blocks are produced the world State continues to change now that we're in proof of stake every 12 seconds is a slot whether it's empty or not we apply rewards and penalties and and concepts for for validators and as a result of that there's a fairly limited amount of time for blocks to be produced gossiped around the network imported into other clients and then out of stations produced and that's that first four seconds of a slot uh it's pretty tight and particularly post merge we're seeing more and more that sometimes we don't quite make it that blocks are coming out late that they're taking too long to import and then attestations are being produced before that block is actually ready and before and so we get a wrong head vote which reduces the validators rewards and you know slight impact on the the performance of the network so if you ever propose something to a consensus client Dev and it's going to slow down that process of getting a block out and then brought into the client ready to be attested to we're going to have a hard time and we're going to push back on the execution side it's a bit different time is still important obviously in performance but the big thing is this world State a beacon nodes database is in the order of 50 gig to 100 Gig for the total database on an execution client it's more like 500 gig to a terabyte very rough numbers I told you would be high level in hand wavy um but you know this order of magnitude difference in size now has all kinds of implications for performance and it means that there are all kinds of different approaches to how you handle sync to how you prune that data and manage it as it changes how you index into that data that impact then flows through the rest of the execution client so every time we're touching World State everything that has has operations to do whether it's transaction gossip and so on all comes back to this world State and getting access to it and that's what drives a lot of big things for execution clients and we want to give them scope to keep inventing new ways of dealing with things okay post merge world now that we've set the scene um the deployment model you've probably gotten the hang of these days we have two separate clients you need to run both a consensus client and an execution client and they both connect to their own PDP networks for gossiping data and retrieving stuff the blocks come through the consensus client pass through the engine API to the execution client and it's it's that engine API that allows the consensus client to to control the execution client and this is one of the first things people notice as something we probably should clean up post merge of why do I have to run two pieces of software for a single ethereum node well there's a couple of solutions we can look at for that the first is embedding a light consensus client into an X into each execution client I really like this idea it's simple for users to run a node it reduces the system requirements like client consensus is a lot less impact and a lot less bandwidth and CPU than the full consensus protocol but it does come with some trade-offs you will always be one slot behind head because you've got to wait for the sync committee to do its job you can't run a validator node this way and areas it's a reduction in security guarantees for most nodes like if you're the type of person who really just wants to run one thing because you're running a home node and you just want to send some transactions it's probably a really great solution but it's not for everyone so often people suggest that we should combine clients we should take a consensus client and mash it into the same process with an execution client I'm a lot more skeptical of this it does make it simple for users to start a node in that they've got one process But ultimately it's really bad for client diversity we've got something like five consensus clients and four execution clients to choose from there's 20 different combinations are we really going to maintain 20 different combined clients so you can run them as a single process and who's going to do that and even if we were prepared to do it there's all kinds of dependency conflicts that we're going to get into because all of a sudden two separate development teams work is being pushed into a single process and to resolve that means development teams working much closer together and being restricted by what they're doing so we get a lot more cognitive load for core devs a lot more coupling and we lose this nice separation that we've somewhat accidentally got between consensus and execution layers um and that's that's a real disadvantage for the ethereum ecosystem because that separation and the ability to have people who specialize in consensus and people who specialize in execution uh allows us to build stuff a lot faster and build stuff much more in parallel the final thing is it encourages client scope creep that might be a little unexpected here so let's dig into it a bit more this is what a real ethereum deployment often looks like and it's not as simple as the first picture um because we learned ages ago even before the merge right we let ages ago that putting your wallet inside your execution client is just dumb it's really bad for security and so we stopped doing it you know I if you've managed to load your private wallet key into yes still and you're using those Key Management apis please stop um use a real wallet and so you're already using two processes you're already using two different things to access ethereum then not every consensus client supports running the validator client in the beacon node at the same time in the same process some do some don't very common requirement for people wanting to set it up to have a separate validated client for various reasons and you might even take it a step further and want to secure your keys more and so you use an external signer that possibly has an actual database for slashing protection you probably should be monitoring your 32 East investment at minimum so you get something like Prometheus Everyone likes flashy graphs so now you've got provider and then you start adding on things like MAV boost so all of a sudden we've got a lot of stuff going on uh it's you know you can't imagine all of this being put into one process it's not viable it's not what we want to do and it you know and we don't want to lock people into saying hey get started with ethereum here's a single process oh you want to do something well ah yeah throw out everything you learned do it a different way doesn't make a lot of sense so what I think is the real solution here is eat OS important note please don't make it an actual OS right we've got good operating systems but this idea of moving up a layer of coordination is going to be really important so that we can maintain this decoupling of consensus and execution clients they don't need to know too much about each other they use the standard apis to communicate but we can start with here's a single process you run and it will spin up your execution and consensus client and have them hooked together ready for you then if you want to add a validator it'll add in a validated client for you it's just another config option without having to start again same for Mev same for metrics and so on the really good news about this it already exists um in in fact it exists in multiple forms things like East Docker if wizard dap node sterium all have some version of this or that that makes it easier to run a client kind of hides the details of of the fact that there are multiple processes and coordination required of course the downside is exactly that we're hiding the complexity underneath the hood lots of different things going on lots of moving parts and we need to be very careful to ensure that we have good definitions between each of those parts so things like building execution and the engine API well and making sure that all those little details get fleshed out and documented and so on and that over time we keep making it more robust okay beyond the initial deployment what are some of the other problems and opportunities that we now have because of the merge and and the way that these two clients work the first one is a pretty big one we got a bunch of data duplication so the consensus client blocks include the execution payload which is what your execution client sees as a block kind of by default both clients wind up storing that data so we've duplicated the execution payload which is what contains all the transactions it's pretty big and that increases our disk space requirements overall so you will probably see if you're running a beacon node post merge its database will grow in size faster than it did pre-merge because of these extra transaction data there are a couple of ways we can address this the first one is that there's a proposal that's been around since pre-merge but just execution climates didn't have time to deal with it basically and I'm not sure consensus clients did either uh for get payload bodies now what this is is it's a fairly efficient way for a consensus client to say hey I need the payload for this range of blocks and that way the consensus client can simply stop storing it and instead it can rely on the fact that the execution client will be storing those blocks anyway the consensus client does need that data and it needs it fairly quickly when it gets it because things like queries to its rest API if you ask for a block it's got to give you the full block so it's got to go and get it and the same for networking when peers request a block it's the full block that we need to send back and that in particular we want to be fairly fast and we really want it to be efficient that's probably the Big Challenge we have with this deduplication right now is that we're asking the execution client to store data and give us access to it on our terms if you remember the big thing for execution clients was managing data they have a lot already their databases are under a lot of pressure and that's the performance bottleneck so Lighthouse has a feature where they will kind of emulate this get bodies this forget payload bodies through the existing Json RPC Cool Tech well done um and it turns out it puts a lot of pressure on the execution clients sometimes it works well if you've got enough I O sometimes it hasn't and they've added an option to be able to turn it off and just go back to duplicating the data which for some people has worked out uh better the other issue that floats around here is that we get an increased coupling between the two clients again um not so much in terms of The Logical side of it in terms of you know what their responsibilities are but more in that if your execution client is resyncing your consensus client just lost a bunch of block data it thought it had and there's a lot of complexity around how to deal with that it's it's reasonable you'll probably just start getting errors from the rest API and say oops don't have that block um all of a sudden same for peers it's quite possible to go and have that block anyway um all these details can be worked out but it's kind of part of cleaning up the mess we left behind when we got the merge done the other solution which is a bit easier and kind of cheating is let's just not store all the blocks so we've talked about this with eip44s on the execution client side for quite a while and the consensus client learning from that actually put in the spec that we don't have to keep all the blocks all the way back to Genesis it's never been spec that way uh it it's about five months that we need to store so if we just started pruning those older blocks and deleting from the database even with this data duplication we wind up with a pretty small database for our Beacon node it's probably enough it certainly would be for a lot of people and we can do D the deduplication as well to make it even smaller the really nice thing about that is that uh for for consensus clients that aren't trying to form an archive node that would make their disk usage almost static it would grow but only really slowly as the number of validators grow and the the beacon state grows which is it's relatively small that way we're kind of avoiding this this ever-growing creep of disk space usage that we've kind of seen with ethereum because of the growing World state it's nice to the consensus client doesn't necessarily have to have that problem drawback is obviously if you're running an archive node then you want to store all the blocks you want to store all the data and you're really going to want to lean on the deduplication to save this device instead of this approach and the other part which things like the portal Network are aiming to solve uh that those older blocks are harder to find you can't just request them from any peer on the network and they potentially become unavailable that's technically okay but it doesn't feel good for a blockchain to kind of lose old data so it's not really something we want to encourage and we want to have systems in place so that we do hold on to them there's a bunch of research going on there uh for example era files are defined and and we can use a number of things like that okay so the other place where we see an interesting interaction excuse me between the consensus and execution clients oh that water didn't help yeah so this interaction between the two clients into in terms of non-canonical blocks so blocks that we receive that we don't believe are part of the canonical chain maybe we receive them really late um or whatever for an execution client particularly ones that only store one version of the world State rather than the full tree of all the world States they're really tempted to want to just not bother executing the pay the transactions because they don't want to store that world state they're only storing one anyway um and so the the engine API allows that and allows them to say this isn't worth executing I will store it and it's here if it becomes canonical sure I'll execute it but I'm not doing it now and they can return accepted the problem is on the consensus client side we have to track that that block isn't fully valid so the execution client didn't execute it because you know World State's big that makes a performance problem that kind of thing on the consensus client though that delay in execution might become a problem if we have to reorg over to that block because it's going to add delay right at the point where we're likely to not want it that time sensitivity thing so we're about to create a block and we've realized hey we're on the wrong chain we're going to reorg over to this one all of a sudden we've got a block we don't know is valid that we think is now the canonical chain we have to wait for the execution client to execute it before we can safely perform any validator duties that are involving that block that might delay the valid duties best case but it may just cause them to be missed um so it's going to be really important to to have some understanding of these different pressures between clients because for short reorgs it's actually really important for the execution client to go ahead and and spend a bit of CPU and disk IO to execute all those blocks to avoid potentially causing the consensus client to be in a time pressure scenario but then for long reorgs that's probably okay it's less common uh and much less likely that we'll need to reorg to something that that would cause a long reorg Danny Ryan's talk and the opening session particularly showed that where we're just not seeing many reals at all which is fantastic okay so summing this all up there's there's a bunch of other problems that that are there and and clean up that we need to do these Loose Ends but they all Loose Ends they're not going to require more hard Forks to fix they're not going to require massive amounts of effort it's just the normal day-to-day plugging away of engineering so keep up with the latest versions of your clients and life will keep getting better I think we want to continue to learn from the other side right consensus and execution clients through the merge process have really started talking to each other a lot more than they did back before we were really focused on the merge we want to keep that going we don't though need execution client devs to be an expert in consensus layer that split is powerful and we want to really leverage it and look for other places where that kind of split can be useful and that's part of embracing this kind of multi-component future of ethereum that you don't need to know everything about every component of ethereum you can know your little part even just within the the layer one that's all I've really got time for um so uh you know thank you for listening and I've I do believe we have some time for questions which would be awesome hello yes we do we have time for questions uh do we have any question from the audience please raise your hand hey um uh cool talk thanks uh so you mentioned multi-component and embracing that um aspect and I think for the most part you know we're on our way towards that I think one area where that is not quite um the case is between like the beacon node and the validator across clients right for like prism or Lighthouse whatever um do you know um is there a road map with like Basu for example or sorry taku um for that to allow like validator I take a validator talk to like a prism Beacon node and um like what are your thoughts there what are the challenges to be able to accomplish that yeah absolutely I think it is really useful to be able to have you know I take without a letter client talking to a lighthouse Beacon note the good news is that the standards are there for that um so from right back before the beacon change launch this is theoretically possible in practice it's been a bit hit and miss uh we've actually seen really good compatibility with everyone else except prism so prism uses grpc uh they kind of got in first and did it and we didn't listen to them and did a rest API because we liked it more and kind of outvoted them so we put them in a hard place there um so their validated current uses grpc they do expose the standard rest apis so most clients can their validated clients can connect to prism but prisms valid a client generally can't connect to anything else otherwise they generally interrupt pretty well and particularly for teku because infuro was exposing the beacon node apis which they've deprecated and removed now because people kept running validator clients against inferior which is kind of terrible um so it is possible it is something that we need to to ramp up testing on more every so often we find a little corner case that didn't work as well as it should and those kind of things but yeah a lot of the that the base layer is there for it mostly it works and I think particularly with post merge it's going to become more and more common because you so often use a validated client that failed over between two Beacon nodes now there is another question another hey Adrian thanks for the talk uh I want to ask what are your thoughts on current design did the consensus client is tightly coupled with the execution Cloud I mean one consensus cloud has to talk to One execution client that sound like what's the reasoning behind it another thing like what do you think about DVT distributor validator Tech and how will it influence the the future of clients yeah so on the first part there it is pretty much um and I probably should have mentioned this that that one consensus client generally has a tight coupling to one execution client the key reason for that is that the consensus client is is writing and updating and controlling the execution client there are some solutions and I think this is part of the multi-component stuff that you can actually run a middleware layer that will take the option the the calls from the consensus client and spread them across multiple execution clients I love that I love that it's built as a separate thing and not something that is a client Dev I have to do myself because it's hard and there's a lot of different policy scope there in terms of do I trust this is my primary execution client you know what if they disagree um do I want two out of three to agree all these kind of details we can build that really well with a dedicated product rather than um trying to build it you know five times once in each consensus client uh in terms of distributed validated technology I'm not a huge expert but again that's pushing towards becoming a middleware layer and I think that's again really powerful and really good it's certainly uh is a great technology to help de-risk some awesome operations with ventilators and have unique deployment models that are that are around there so it's something I'm really Keen to see keep evolving and keep uh kind of finding new ways of doing it and and ideally that we find ways of adjusting pretty much the the validator apis that we use those rest apis I talked about before adjusting them so they're more and more conducive to an external piece of software just being able to handle it all and handle it well there might be some trade-offs there we might hit some limits I don't know but yeah definitely a technology to keep investigating thank you oh thank you thank you very much Adrian with some an amazing conference and please a big Applause for him thank you everyone thank you [Applause]