Killing ETH - Finding Consensus Issues on Layer 1

foreign [Music] so next up we have Mario's Van Der viden who is on the Geth team and works on fuzzing and testing and some other things I know Danny Ryan likes to take all the credit for the merge but these guys played a crucial role as well and he's going to talk about some of the bugs that they found and maybe some of the things that could have gotten horribly wrong didn't thank you yeah I I was actually surprised that so little got wrong with the merge and I'm really happy to do it to to stand here today and and say that we didn't kill eth um but we tried to and this presentation is a bit about some of the strategies that we use to kill eth and it's a big part of the presentation is about some of the bugs that we found some of the interesting issues that have have popped up during the merge testing and also uh before that during our normal testing so today I'm going to talk a lot about consensus issues consensus issues are differences between the implementation and the spec I think that's the big thing about ethereum we have a specification and we have multiple clients implementing the specification so for the execution layer we have four different clients that Implement exactly the same thing and we need to make sure that they do exactly the same thing because if if one implementation um uh rejects a transaction that they that the other implementations accept then the chain will split into two and uh we want to prevent this and so we um over the years we developed a bunch of testing strategies uh for exactly these problems and um one the newest one that we developed for the merge are Shadow Forks basically we take a copy of the blockchain we configure some node some of our nodes with the new rules and at the merge transition the new the new nodes create their own chain which is in parallel to the main to the main chain uh which what is really interesting about it both chains share the same state so all of the transactions that are valid on one chain are also valid on the other this means we can we can run tests with mainnet load which is sometimes very important to test for performance and also later on we will see a bunch of issues that came up during Shadow forking that we wouldn't have found otherwise another big part of what we do is differential testing basically because we have these different implementations it has a we have a we have a really nice uh we have this really nice property that we can verify them against each other so we create an input we give it to the different implementations and then we collect the outputs of these different implementations and verify that everyone does exactly the same thing and there are different strategies to generate these inputs we have for example the ethereum tests which are just static test cases where we know the answer for for the test case but we can we also created fuzzers uh for creating just random mesh transactions and and contracts to test the different implementations um something new that we also did during merge testing is create malicious notes um malicious notes are basically folks of a client software that does that changes some of the rules and so malicious notes can insert bad transactions change header fields and really big values or no values and can try to break the other nodes on the network these malicious notes can we have them for the consensus layer where you have them for the execution layer and the malicious consensus layer nodes can for example double vote so vote on two different uh to two conflicting blocks which would mean that they can be slashed um or use fake signatures send something some weird Network packages and just in general try to cause Mayhem on the chain and this is a non-conclusive list of some of the testing tools that we have built over the years um for example go evm lab is a toolkit for evm testing that we use to um create a test case executed on on a client and collect the the output from the client we have Hive which is a really nice continuous integration regression testing Suite where we have we run these ethereum tests which is a set of 48 000 test cases every night against the different implementations and verify that nothing has gone wrong when they update something and then we have these malicious nodes both both on the CL and the El node the CL and the El layer and then we have some companies that we work with kurtosis and antithesis for fuzzing and and test Nets these are like bigger test Nets that we that we set up run once one time through the transition and then then verify that the transition was correctly and then we scrapped the test net and and start from start fresh and then we have some some kind of one-off tools uh like TX first which is send some interesting transactions it's not really for finding bugs but more for creating load on the network so if we create a test net uh one of the first things that we do is use TX first to basically Hammer the nodes and see if they if something breaks there uh with much fuss was a one-off tool that we used to First the engine API and we have the beacon first which is a tool that is uh has been around very long and fusses the different Beacon clients and found a bunch of interesting bugs okay so the rest of the talks is just talking about some of the interesting bugs that we found over the years um this bug was found by Martin right before the merge so whenever we have a whenever we have a consensus upgrade whenever we have a fork we actually we ramp up testing we we take out all the all tools that we have and and try to break it um here the Visa client had a had a bug in the gas calculation as you can see there the gas goes down and at some point there's an underflow and the gas goes extremely high this is a consensus issue because other clients will compute this correctly and on networks where biso is the only client this would actually be a Dos Vector on the network because the nodes will keep running the transaction and never stop well they stop once this amount of gases is run out which is basically never um then we have the death of kinsugi so we set up the kinsugi test net which was the the real big test net uh that we had where we also invited the community to participate in in testing and on this uh Kentucky test net we set up the bad block generator the the bad block generator from from the execution layer and it changed the blocks so for example it created a block with invalid extra data and this it actually I think it actually set the extra data and Nimbus was unable to sync because of that and then we had the TX further running the x-fuzzer created a transaction that had the revert op code in it with a and and this triggered a consensus an issue in ecmjs and so they couldn't uh couldn't there wasn't consensus issue there and then we had a the three-way consensus split which actually broke the network and this was a split between Geth uh the tiku guest combination and piso another mind um yeah so the three-way consensus split basically um we had the father replaced a block hash with its parent hash this block should be rejected because the hash doesn't match what we what we give it within the engine API um this check was actually in this in the spec but some of the clients didn't implement it correctly so bisu did not have this check another man actually had the check but it also cached the payloads by block hash and so we looked up the the block hash the wrong block hash so that it was in the cache and then the the implementation just assumed that that the payload was valid so this split the network into Geth and another might be so with another might be so being wrong I guess being actually the the correct client and then we kind of thought okay we're going to fix it and we're going to keep the the network running and and try to to get the network back uh back again and but during this time the bet the the bad block generator actually created another issue it created the block where the block number was set to one and in Geth we have a cache to check whether we need to sync and one issue is with tiku they executed all Forks of the blockchain there were like 30 different competing Forks uh executed them simultaneously which flushed the cache and so we queried the database by parent hash and block number minus one block number minus one was Zero which which failed and triggered another sync cycle and because we were trying to sync to the Genesis block that actually violates some preconditions so get panicked and that the tickle get node shutdown so after we had this three-way consensus split we decided at some point okay it's just too hard we're going to deprecate um deprecate the con the kinsuki test net um I already talked a bit about Hive Hive um we use Hive to execute a bunch of tests and most of them are spec tests so we for the specification we create test cases and I've found an incredible amount of bugs so shout out to Mario if he's I think he's sitting here yeah and so for example we had a division by zero in the exchange transition configuration call some some rules around the time step didn't really work and um so I've really found a lot of issues that we have between guest and the spec and then we had the testing the merge effort uh which was really nice over 400 people got involved with testing the merge they sent transactions on their test Nets they set up notes reported issues and the most important thing they create a documentation so we should be better at this we should be better at creating a documentation but we are unfortunately not so we kind of rely on the community to educate other people in the community and they also found some some interesting issues in in go ethereum that were kind of related on on the usage of it and I think they also found a bunch of other bugs so the community found a bunch of other bugs in different implementations and then we had the shadowfox the cool thing about the shadow Fox is we are actually testing with the real Network and the real data and so one thing that on the on the first year of work we saw that the gas limit was quickly dropping and the issue there was the default gas limit in gas is actually was actually 8 million and we never caught that and we caught it for the first time doing this this Shadow Forks because on mainnet the the miners back then voted to to increase the gas limit um another issue was we had a memory blow up during reworks there was some weird issue where it would re-orc the note 600 000 blocks and this like increased the memory until the the node crashed um re-orging 600 000 blocks is not something that we are able to do but finding this issue actually uh showed us that the rear procedure was taken to up to a too much memory and so we changed that and then we had the base the engineers um basically the engineers between the execution layer and the consensus layer is different so what consensus layer clients need to do is change the endginess of the fields and prism used the wrong Engineers um thus it created bad blocks when the base fee was over 255 255 because one byte you know uh yeah I would really like to get you guys involved uh testing the merch was a was a was a great way for the community to be part of this um and we have a bunch of new upgrades a bunch of interesting stuff um that we also want to test so uh test the search test The Verge and eventually we also want to test the purge and this community efforts makes me really happy that so many people are interested in this are interested in contributing their time so if you um if you want to become part of this I want to give a special thanks to all of these clients all of these client teams and people involved in testing and testing the merge and I want to give a special shout out to Martin I hope he's watching right now for keeping us and ethereum stable we're also currently hiring within the ethereum foundation to increase the the testing efforts the testing team so if you're interested in getting a job at the foundation and and help us test the new updates contact Mario and yeah so for when I inevitably have too much time on my hands I have another eight minutes I have a bunch of backup slides with more cool bugs and issues that we found uh you might remember this one the bug that took down infuro basically the memory returned by the return data copy of code was shallow copied and so when we modified it broke something and we actually found the bug rather quickly and fixed it but the fix wasn't announced we created a new version of get and we kind of forgot about it and someone actually found that bug and triggered it on mainnet because they saw that only one percent of the nodes on mainnet actually still have this this version that that's broken uh the problem with was that these one percent of nodes were actually inferior uh included in fewer notes so uh for three or four hours in Bureau it went down that's why you shouldn't use a centralized RPC provider but that's a that's a that's another issue go run your own notes um and uh yeah we kind of had to take the blame for this um this one was also a really interesting one so we have fuzzles that run on OS as fast continuously and at some point we got an email hey we have a panic and whenever we get a get an email about a panic that's usually pretty bad because then someone could just send a transaction to the network and all of the guest nodes would go down and that's not great so the issue was that you can see the comments that says choose shift equals B minus one but the shift was actually B not B minus one and so that's the fixed body for the issue and we triggered this bug within the mod x pre-compare when the modulo was extremely large yeah then we have the some bug that we've got via the Bounty program so we have the Bounty program if you find something and instead of triggering on it on mainnet send us an email we're especially pissed if we if you send us an email and then go trigger the background mainnet um so send us an email first talk to us and you will actually get a bounty um The Bounty size was increased to 250k and we actually quadrupled it for the merge so there's a lot of money to be had if in finding in funding consensus bugs um here when when we input in 256 min 2 to the power of 255 um it would the the implementation of nethermine for the for the modular operator would actually negate the result-wise resulting in 2 instead of -2 so all other nodes computed -2 another my nodes computer 2 and so they would spit on a split on a different Fork and this one is a Dos via militia snap request that was found by by Gary n Martin from the guest team and here the issue was that someone could craft a weird get try note packet um that's that's a network packet that requested a missing try note and you could crash a the guest mode with it these kind of issues are not that bad because you only crash one node if we have crashes in if we have crashes within the evm those are usually worse because you can crash a bunch of notes and actually during investigating of this we created a fuzzle for it and the fuzzle found a second panic here we have one I actually wrote my master thesis on fuzzing evm stuff um and created this fuzzy VM program and the fuzzy VM found an issue in the copy op codes called Data copy code copy x copy return data copy and the issue was these op codes consume three items from the stack destination source and length of the data to copy and never mind actually halt the execution if length was zero so you can see this is one of the tools we have where we can where we have two different executions we have two different traces of the execution and we can compare them and you can see that here the sorry here the the execution just holds and that's that's the consensus issue um then we had an uh another issue found by fuzzy VM was a denial of service in in bisu basically when you called the mod x pre-compile uh piece of wood reads all of the parameters even if base length and modulus links with this zero and so you could put a really really big exponent and this could crash piece of nodes and this is the newest one that we've found uh right the the 110 22 bug was actually uh one I guess 122 was actually the version that we wanted to use for mainnet for the merge and it contained a regression that could corrupt the local state basically whenever we we have so we have these try notes this is where all the stuff from the other contract State and the counts are stored and when we modify them they need to be flushed to the disk in the same order as they are inserted into what we call the dirty cache and so you want to you want to insert if you modify d b and a you want to insert D first B second and then a third and what we did was actually insert B first and then insert a and then we saw that we should insert D but we saw that b was already inserted so the code assumed that everything was correct and we never inserted D so we ended up with some dangling dry nodes that's it thank you very much for coming and thank you for having testing the merch and I hope to see uh more people test the merch in the future thank you [Applause]