Tales from interop by Parithosh Jayanthi | Devcon SEA

[Music] [Music] So today we're going to talk about Tales from interrup the main point is how do we get corev so about 120 people working on roughly Five Forks to not make everything super chaotic so before we start I'm going to set the stage a little bit what is pectra so Petra is the fork um after Denon Denon is the one that we shipped in um March of 2024 that's with 4844 uh the scoping discussions kind of started earlier this year compared to the previous times so we already had a general idea of what's going to go in Petra in uh Jan of 2024 so Cent teams kind of started working and kind of started bike sheding what's going to go in uh naturally since we started we over promised or over committed what we wanted to do and by may we had EIP 374 which is a version of account abstraction uh max effective balance eof pure us as well as a lot of other miscellaneous eips that were supposed to go into Petra and that's the stage in which we are going to be talking about for most of the talk and then I'll continue with what Petra looks like today um ssz epbs veral as well as veral transitions were also features that we wanted to think about and wanted to test um over the course of the year right there are different teams working on things at the same time specifications are getting updated at the same time so you're essentially talking um you're looking for a moving Target or you're implementing a moving Target so how do we actually ship this thing uh enter Nea interrupt so this is an event that was held in Kenya for every client team to participate in so these are the people that are building the Petra Fork um just a side note we also had a Frontiers event in Kenya where we got to meet a lot of local Builders and that was extremely nice and I hope we continue that type of format in the future um the aim was to work on Petra as well as all the features that I spoke about earlier and the idea was to try and figure out all the hard problems see what we needed to do and how do we ship it it's an inperson event and since most of the client teams are spread around the world it's invaluable to actually be in the same room you can figure out a problem within a matter of minutes instead of waiting for the person who lives in Seattle to wake up and spend the next two days figuring it out um there's about 120 people who are invited and we split it into five work streams so Petra EF pias verle and vocal transition and like I mentioned there's 120 people uh um we're also a very opinionated Bunch so it's not like 120 people have the same machine there are people with Arch Linux with disabled kernel modules over there and we need to make sure that the tools work for them there are people with Windows machines we need to make sure that the tools work for them um the five work streams make it hard to keep up with what's going on as well um there's not enough dedicated devops people for each client team um it's impossible to keep up with updates imagine 120 people messaging you hey I pushed this commit can you actually deploy it for me um it's hard to test if a feature works as expected because as mentioned earlier the feature itself is being defined so we don't know how it's going to look so we don't know how to test it and uh a lot of this is also tying into what Guru spoke in the previous talk we're hoping that the execution spec tests Etc help in the future but at the time of the entrup event we didn't have that um discussion rooms are happening all day so it's not like we're put in one location we can actually work on the feature there's also meetings to attend all day while all of this is happening um and fun fact if you ask a client Dev to compile another client chances are they'll fail at it or they will do roughly as badly as you will and if you're compiling Lighthouse then you'll spend 30 minutes failing at it and Chaos really [Music] rules um so the first thing we wanted to Target was um how do you take care of all of these work streams right uh we added a BAS basic process so we had one document per workstream the workstream had an implementation tracker so you know who was doing what what EIP is fixed what specification they're targeting um what version or Branch should you look at if you had questions and sometimes also who to go to if you had questions um we had an inter interrupt shared landing page so you just had to remember one URL you go there you follow the URL based on what work stream you're on and you end up on a page that's actually useful for you um this really helped at least reduce the chaos for the initial Discovery process um but the next part was say I am a client and I want to use uh I'm a I'm a te client and I want to figure out how to run G because we need an El as well as CL that's working right um and we got some help from the ethereum foundation devops team so they um helped write a tiny bot and the bot was added to the chat and this bot essentially made API calls to a GitHub action CI that the e panda Ops team was already maintaining so we have the ability to build each and every single client out there almost all the tooling that we do it's it's a really nice workflow that already existed uh we just didn't want to give 120 people access to our core workflow so we wrapped it inside a nice telegram bot and uh put it in a channel so you could just do/ build follow that format at and as long as you did that it'll spit out a Docker image that you could also access via local cache uh fun fact 120 people bad internet bad experience for everyone so having a Docker cach really saved the day there um so this sort of um removed a lot of dependencies for us by using Docker we don't have to explain to someone which version of JavaScript they need to use you don't need to explain anything um they were able to at least deal with that level of abstraction and continue on focusing on their fork in itself the next part was we wanted to abstract the clients away from the client teams as well um same thing is compiling a client you probably don't know what flags are required for the other client either you could follow a really verbos document but yeah it's a new feature it's going to be hard so instead we relied heavily on curtosis so curtosis is a tool we've been using to switch from remote testing to local testing if you run that command so curtosis run um the ethereum package you will have a local ethereum network running with whatever specification that you give it and that specification is a simple yaml file um this oneliner sort of makes it easy for client teams to spin up a network and I'll SP speak about what that file looks like and what other features the file has but for now all you need to know is that it's a yaml file and this yaml file is able to um give us a network right this means youd never have to know what another client team is doing you can focus or deep dive or go into the breakout rooms and actually spend your time doing that um and the kosis uh tool also uses the docker buil cache which means we save a bunch of Downstream bandwidth at the event the next question came in uh how do you actually inspect this chain uh logs are great wa if you use debug or Trace logs good luck finding out what's actually happening in a meaningful amount of time um so enter Dora the Explorer it's a custom Explorer built by the eth panda op team mostly philli um and this also uses Tools in libraries that already existed and are mostly supported by client teams or client adjacent teams or testing teams so it's very much within the ecosystem which means whatever new Fork that exists will be updating said libraries which means Dora would continue to get updated and the benefit is um since it's custom built for Fox we can add whatever features we want uh the database is easily accessible so if you think up of I want to know how many of the blocks had this very specific thing um you can just run a SQL query against the database and figure out what's happening and we actually use this for figuring out some Mev statistics early on uh it's extensible so whenever we want to integrate other tools that the team has built uh you can actually integrate it into the same Explorer it's a nice friendly interface uh but one thing we do want to mention it's custom built for Dev Nets or test Nets we don't really plan on supporting it for main net main net requires optimization larger States main net requires you to build for users we're building for devs and an example of a tool that was integrated is um this is a view of how a test net looks so you can see who is peering with whom and you can look that there's one peer on the left that has a lot of other peers chances are that's a boot nodee and that's who a lot of external people are connecting into um and you can sometimes use this view to quickly see if networking is broken in some specific manner if you see two clusters that are really clear chances are someone has broken networking somewhere and we they we've also added support for consolidations over the last week uh so you can this is I guess one of the first uis where you can actually make a consolidation and I'll talk about where the consolidation will go soon um we're hoping that Launchpad and uh Beacon chain and a lot of the other tools that you guys are used to uh will have this support soon but again this is for pioneering Forks this is for trying stuff out ASAP Uh custom built for for the devs I guess the next question is how do you validate that you can actually interrupt with other clients right say I've done my Fork I've run my local test net I know that the node is working how do I actually know that it's doing something um introducing asserter for that so asserter essentially asserts that you've uh it does an action and it asserts the outcome of that action so we've written some basic uh um two a tests for the client teams and I'll go into what those tests are and the tests can be dynamically fetched from GitHub so we can keep updating these tests in the background as the clients continue to work on their stuff so an example of how this looks like is uh you would make a 2,000 eth deposit and then assert that the validator was activated and it would check that it went through the entire deposit workflow using the new EIP and you can find more about this um in the GitHub page there and this is pretty much how it looks like uh you get a nice UI with a green for pass X for fail as well as how long it took and what sequence it went by and if you click on a specific test you can see all the individual steps that that test took as well as which Step it actually failed at and we've organized all of this in into a lot of tasks so each thing is a task so example check consensus uh stability or check uh or general R transaction or whatever and the idea is that you can combine these tasks into arbitrary tests so every time you want to write a new test chances are there's already a task out there and if a task already exists then you can just create a file that creates a new test for you the bad part uh more test Frameworks you just had to talk about one test framework we already have Legacy test framework now I'm talking about another test framework it's kind of getting absurd at some point uh as a result uh asserter also just integrates the other test Frameworks into it so the idea is you can continue to write your execution um specification you can continue to write execution specification tests and asserter just pulls them in and executes them for you against the devet so one place to write the test multiple places to consume it multiple places to actually see what's happening and we're all hopefully a bit more happy with the workflow now the question comes in I've spoken about like three four different tools we maybe made it easier to run the client itself but now we're making it harder to run other tools right well the benefit of curtosis is that you can integrate all of these tools into curtosis itself so this file for example specifies uh re as well as teu node with a specific branch that a client might have worked on it specifies some Fork parameters that I want to change it spe specifies what tools I want spin spun up so Dora as well as asserter and it tells asserter um that it wants you to r that you wanted to R to run a test from GitHub and the test from GitHub tests everything about Electra so what we typically did at interop is we pinned this entire config as well as whichever clients we knew were working and people could copy this config modify just one parameter with what you what tool you or what client you are working on locally and compare it against another client that was validated to be working so that makes it a really smooth workflow for you and side note uh you can also do l2s on this so for example there's a simple exam uh there's a simple uh yaml there that spins up uh not one but actually two optimism l2s so in case you want to do L2 interrupt testing you can also do it within the same workflow now I'm quickly going to talk about where pectra is right now um how close are we to ship pectra kind of close um Petra as a fork since May was split into mostly two to three smaller Forks um so we have uh Max CB 7702 and all the staking uh workflow related eips going into pectra itself eoff as well as pias will be moving to the next F and a lot of it was was the outcome of NE interop as well as a lot of time spent by the devs to prototype these eips figure out how they run and then understand that okay maybe running all of these eips in one FK is going to be too much and the idea is that splitting it into smaller chunks makes it easier to deliver these eips to main net as well as to make sure that um we're delivering it safely um and securely there still might be some minor changes coming in but as far as we're concerned all the major changes have been finalized for Petra uh and client teams are slowly preparing to merge their code bases in which is a very good sign that's typically one of the last things that happen along with the test net uh before we actually decide to ship the fork and what's left the community is actually the hardest part and the biggest part right now uh We've written um in order to help the community we have a test net it's called meong um after the river that flows through southeast Asia this contains all the eips that uh go into pectra and the main part that we want is that people deposit a validator try out consolidations try out exits update your staking guides understand how EIP 7702 Works prototype wallets um basically try everything that you would want to try on mainnet right now and kind of the worst case scenario is we ship this Fork on mainnet and a wallet developer comes to us and is like oh I can't use 7702 for a reason XY Z there's still time to make changes but that window is closing extremely quickly and we want people to actually use this stuff and actually try this stuff out um yeah as a wallet developer please add support as a valid data um look into El triggered exits if you're someone who runs a staking pool or run a saking pool on behalf of friends try and figure out how you can do El triggered exits um as a node operator run a node we have really nice guides out there on how you can run a node as a tooling Dev please use max EB or support the features as a user try just sending a transaction and feeling good that your workflow is not broken I guess um and help us break it if you're uh well all of you have a Devcon ticket which actually means you have funds on the uh test net as well you can go to this URL and you will be able to use Zupas to claim some test net funds that means you can use that funds to deposit a validator you can run a node you can go through the entire workflow and we're really asking anyone who has experience with helping users in the past or helping node operators in the past to update your guides we tried writing the FAQ for pectra there's a lot of changes and a lot of edge cases so we need the communication engine of ethereum to really spin up now because we can't ship the fork without that sort of feedback and that sort of community engagement as well and at the risk of not committing to timelines I guess uh the meong test net is live right now uh people are trying it out time to get involved as the community The Next Step would be the hsky and seoa test Nets um we're hoping that once the last changes go in and one or two more Dev Nets have passed that we can slowly start committing to shipping these as well and the final step would be mainnet so looks like we're Rel relatively straightforward to shipping Petra um right now at least and in case you're interested in other topics that the team is handling we have a talk about uh data as well as how we're collecting data about ethereum upgrades how the Network's doing how blobs are performing Etc with a tool we've built called zatu um the Talk's happening later this week so please have a look uh I think it's in classroom D and it's going to be an interactive session so if you've ever wondered or wanted to dabble in data science that's your uh that's your that's the talk to go to and a simple example of how this sort of uh data looks like uh we this is for makong testnet so where um pushing information from all of these nodes into one place so there's roughly 95 sources of of data for this um and this tells us when blocks are coming in when attestations are coming in and when different events are happening on the network and the reason that this is important is if we're breaking something on the test net it's definitely going to break on the main net so you want to have this deep Network level introspection to know what's going on as well as for blobs we're still considering a blob increase to uh pectra and the long tail of the blob increase is ensuring that we have enough data for for shipping it and zatu is going to be one of the ways that we can produce the burden of proof for increasing blobs on ethereum and these are the kind of graphs that uh come out of this tool so you can actually see when blobs are arriving across these multiple nodes and yeah that's it for me thank you guys for attending and um yeah if you have questions Now's the Time thank you thank you we're going to jump right into question questions um so would you say that eth Panda Ops could be a potential single point of failure for ethereum's development um I don't think we're a single point of failure ideally the tools are self-sufficient uh We've written them following mostly standardized languages mostly standardized practices I think the only weird choice we have is most of curosis is written in starlark but it's python is so that's mostly fine so the idea there is that any devops engineer could could take that over if required and yeah if you want to help decentralize that we're happy less work for us thank you frog verse Panda question mark I think we need more swag we ran out of t-shirts already um we might need more swag and probably a similar frog style event for pandas how much do these does these infrastructure cost we're actually an extremely cheap team uh we run if you ever thought uh we overspend just spend dinner with barabus from the team and he will make sure that you are saving as much money as you can and most of us are kind of of that Vibe we've moved to lowcost um Cloud offering so most of our stuff is in digital ocean and uh we have some infrastructure in physical locations like Berlin or Boulder so we we've tried to reduce stuff as much as we can um of course there's going to be a tradeoff I think the most expensive thing that the team used to do was Shadow Fox and I also happy to share the numbers I think if we did a shadow for for about a week it used to cost us like $ 15 to $20,000 and during pre-merge we might have spent like 50 to $100,000 just on Shadow Fox and I think we had like 14 of them back then um and earlier this year we switched to a snapshot based approach for shadow Fox which means we've tanked that cost by a factor of 10 or by a factor of 15 so some of the time the team spends is also looking at the bill that we see and where we're spending money and how we can optimize that away so it it's a it's a hard trade-off sometimes the trade-off space is we got to ship the fork earlier we got to spend that money sometimes there's a downtime when we're bike shedding what goes into Petra and at that time we spend we spend the team time into figuring out how to reduce costs thank you thank you is there a public dashboard to Showcase these testing results that is actually something we've been asked and we want to uh help fix so um we want to integrate The Hive dashboard a bit more closely into this landing page um asserter actually has a public view as well so anyone can open up asserter and see what tests have passed and failed but it is relatively High burden in terms of understanding what actually went wrong um I think typically the easiest way is to listen into all C devs and usually the beginning of allord devs is either Barnabas or me giving an update on test Nets and test net progress for the week um and that's probably the best way to not spend all of your day trying to understand what's going on yeah wonderful well thank you thank you let's give it up for potty thank you guys for having me and we will be back shortly