Merge Testing Call

okay we are recording uh welcome to this merch testing call um i guess two goals but at least i have for this hour first of all i'll get on the same page with regards to what we actually need to do to test the merge um and ideally second roughly try to see if we can assign some groups or some folks like to kind of own some some of these tasks um yeah because i think in the coming months and two uh this is kind of gonna become the main thing uh we we end up spending time on so uh yeah we wanna make sure it happens um mikael has put together a great dock outlining i guess his view of everything that needs to be done so that's probably a good place to start um maybe it makes sense to start with like your dark miguel but uh kind of go a bit high level right like not necessarily into every specific test but just make sure that we kind of all agree about like the categories of tests that we needed if that makes sense and and to see if there's anything that's like missing um i know there were also some stuff we had on the readiness checklist and uh i'm not i haven't like cross-checked to see if like everything that's in your dock is there and vice versa um but yeah maybe we you can start by just kind of walking us through like the the buckets of like different testing that you imagine will need yeah sure so i can share my screen it helps yeah okay regarding the yeah do you see it can you see it i see yeah cool um regarding the nurture redness checklist yes this document is probably not like they are not very well um linked to each other so we need to update probably the merge readiness checklist with what we end up with okay so yeah regarding this test plan um this this is a draft first of all yeah there is the spec documents so this is like one of the most important sections here so what we what do we actually want to test um so yep yeah main categories like unit tests we have the unit test and consensus facts so i've just listed methods here that must be covered with deaths then we have these two eips and we have block tests and we have retest very test eth2 that is used to test client implementations execution layer client implementations that they satisfy uh the requirements of this entire tip i think this yeah this kind of test could be implemented with this tool um regression testing here i guess that we have those block tests for previous hard forks and we just want to be sure and i think this this pretty standard procedure to to run all tests before the fourth block and then in our case it will be before transition block just to to be sure that the proof-of-work part of the client works as expected and hasn't been affected by the changes introduced in this eighties we have fathers fathers i i think it's like a separate topic um not sure if we want to cover it today then this is more or less certain things at least for me probably i'm missing something here so feel free to add um yeah feel free to interrupt me at any point and add something so these two standalone testing is pretty much understood now we're um the things uh becomes complicated when we go to the integration testing when we need to test cl and el in different combinations and yeah i think that hive this is my assumption that hive will cover all or almost all of these things and this probably is very strong assumption i don't know because i don't have like rich experience with hive so we can test your engine api um eip implementations wire engine api interface so literally just take an el and run some and try to do some transition process testing just using the engine api implement interface it should be doable and i think it's easier than like the next step where we're testing the cl and el um fully featured clients and different combinations of these clients um so like yeah like here so it's like just taken two took two clients and uh probably several instances of uh to client implementations and just playing around with them in in hive and in those test scenarios yeah there is like a huge list of best scenarios and this is not like you know it's not the end list um each test case should be um yeah the work on writing the test cases in the detailed test cases should be done yet that's all we have system tests which are test nets and i don't think they should be covered in these documents i'd rather prefer them to see in other documents so that's yeah and some useful resources are also listed here this is all that came up to my mind with respect to testing and other end links like test factories for 43.99 for engine api uh made by marius and merge mark and so forth also here so this what was around recently that's basically it so um any questions so far any suggestions i'm hey guys i'm galen i'm curious if there's anyone on the call with like expertise in hive who can like validate that assumption or like talk about how ready it is to handle these these cases you mentioned maybe there's not yeah that'd be very good to be justified and to understand what do we need to extend the hive with or is it even possible to implement all this kind of stuff um i'm not like i'm not a huge expert in hive but uh i think all of this can be done uh we have uh right now we have like uh uh hive runs uh static test cases they run like we run the general state tests and we also run uh sync tests so that like we have two different clients thinking from each other and uh also like graphql tests and stuff like this so uh hive should be able to handle all of this pretty easily but yes it has to be implemented and then yeah that'd be great um as i said there was a pr to add sorry i was just gonna say i think there was a pr to add east to client simulators to hive that was merged a couple weeks ago yes but we'll still need like a whole simulator yeah i think that there still needs to be implementation of like simulators that connect the el and cl clients yes i think that just spins up the genesis and i think we need like simulator that connects to cl simulator that is a cl mock and this is in el moc if we want so did you write these did you use any of the notes from the interop to write this yeah okay yeah because i know yeah yeah yeah it's based on partially based on those notes okay i might have missed something while moving those notes from from from pictures to the text and i will check them so you you think that we can take yeah i was also thinking about like looking at this pr um also the question can we use merge mark anyhow to write simulator so i think merge mock will be so like an image merge mock va simulator basically i think it can be part of a similar so like right now if you want to run the consensus test against the client there's like a simulator that runs and it reads the test from this ethereum test repo and converts them into like a genesis file and then takes the block rlps and directly gives them to the clients and then at the end it checks to see if the head block is correct and so after the merge you won't be able to just simply import block rlps you also need to take engine api directives so i see merge mock is something that would sort of help replace that situation who has an experience in writing simulators four hive i i wrote the transaction simulator but uh yeah i can like i can take a stab at it and and and see but if someone else wants to uh uh wants to write a simulator um yeah so i guess i can add more on to this um for prism we have this end to end test case it does it basically what is basically it's basically what hive is doing from for instance is their client he launches the deposit contract and then he starts speaking nodes with bunch of validators it starts multiple beta nodes with validators and it does do all these state transitions epoch transitions and then it simulates slashing conditions and say blah blah blah but um right now he only supports prism so we would like to stephanie support more clients implementations we've been talking about that and i think that's essentially what hive wants to achieve is on site so yeah i'll be happy to like point you guys um to this repo and uh maybe we can get it to be compatible and even and even pour some of the work into hive i think that'll be great because our end goal is definitely do more multi-client end-to-end testing and now with this multi institution there client we have to hook up it sounds like a good time to to basically combine the effort together yeah it seems so mario has also uh wrote a bit uh about that i wrote a program to execute static test cases against the execution layer and this may be also yes this one yeah yeah basically that's not as full-fledged as high it's just very very simple it's just like um it sends um engine directives uh one after the other and just verify the the response from the from the client but yeah i'm i'm sure that hive is much more powerful and capable i think maybe if we can if i could somehow port uh the testage the space cases that i have there into hives into something that i can process that would be i i think even better i guess i think the way i've been staying is that we would take a lot of the code that mergemock is and just create kind of a simulator out of it and then that could read test fixtures like you've written would something separate to hide or what what do you mean sorry um i would i was in i was thinking that the the merge mock tool we would just take a lot of the code from that and add some code to read the static fixtures like you've written and just create a simulator out of it rather than having it being like a separate repository just have it kind of become a simulator [Music] that's that's the best way to do it have uh have the static test cases and then feed them to the simulator i guess the only kind of weird thing is is that merge mock is currently sending blocks over devp2p before the merge happens so it's like very much a real world test scenario i don't know if that's what we want that simulator to be um before the merge so it won't need to yeah i see right because it's pro like it it if you're running an execution layer client it is almost as if it's really going through the merge because it is receiving the things through the same types of code pass that it would in the real thing so uh there is a simulator that uses def but to be currently right it was just sense basically this should be yeah yeah but it seems like the simulators are pretty focused so there's like some that focus specifically on some dev p2p tests and there's like the consensus tests and those are done via like the import subcommand of clients and my understanding is that it's just like focusing specifically on we're testing the correctness of the consensus uh implementation rather than also having to potentially deal with networking things so the the transaction tests for example are also run via devp2p and the idea is to create like large transactions or large blocks or something that that should also like be dropped and stuff but yeah i think uh it makes sense to to to to to do it via the code paths that will actually be used okay so it means uh yeah if we do it with the code passes that actually used means that before them like pre-merge um all blocks stand while they fit the p then basically we switch for example to engine api to initiate the transition to finalize it this is what you mean i don't know if that was to me but i think that's what i mean yeah yeah okay yeah so do we need two simulators for this goal or do we need one simulator that can down both i think we need we need multiple anyway we need we need one for like static test cases and for sync um synchronizing different clients from each other should should also be a separate simulator under the under static use cases you you mean deterministic use cases or yeah like like test cases yeah so like the the right exact exactly these uh sequence of actions which lead to exactly this result so while with the scene it's like it's it's not deterministic but yeah like we use both i think we use static like static test cases for almost everything even in sync so what like sync does is you feed one chain you feed a couple of like predetermined blocks and then you'll synchronize the other uh thing from it and it's i think it makes sense to have like the stuff in hive be deterministic um because otherwise you will get like a lot of like weird stuff yeah i don't know and like non-deterministic testing fuzzing everything else that should not live in hype in my opinion yeah i completely agree we are not going to like test software to prevent race conditions why a hive it's not made for this i think i have a question regarding a marshmallow does at the moment merge mock support engine directives yes okay so i guess we just need to like pick uh regarding the hype when it just like probably the engine api is the easiest one to be to start with like create a simulator to test engine api implementation which will be then used by uh to test the these eaps implementations and just try to create a simulator right test why hive this is my naive thoughts towards like the step towards the first step of integration testing on the match makes sense okay cool then the next question is who is willing to start looking into yeah i guess and one maybe way to frame this is like does somebody have the bandwidth to look at this now and like not distract from basically working on the merger itself right like so obviously if like marines or parents spend their time on this they're like not working on the guests or prismatic implementation for the merge and that's like not ideal um yeah so i guess i'm wondering does anybody like not working on a client have the bandwidth and like the skill sets to add these uh simulators to hive i can take a look awesome great who said that oh alex okay sorry this is on my plan of things to do i just have been doing other things as well and slowly approaching that same but this is important matt this is important i would like to try to import the uh the test cases that i prefer from yeah to yeah that's yeah i can look at it awesome um okay i'm like my yeah okay so cool hype should be so resolved now right okay so my another question like that i have is um we should probably have a kind of like list of test scenarios somewhere at some place right yeah and i feel like proceed okay go ahead i was gonna say i feel like maybe we should link something in the readiness checklist where like we can uh basically you know say like this is like the kind of master testing document and then kind of track who's working on it um ideally just so we kind of have it all in one place yeah sure and yeah how come the format is not does not suit well for this kind of purpose i guess right because nobody can edit like or everybody can edit it or something yeah that should be good yeah sorry what github issues just yeah find some repo you can track it there yeah but these tests are like there's some different parts of the software and the specs are located in different repos and so on and so forth so we can make like some kind of meta spec thing although i i kind of feel like that was a little confusing personally will we use google sheets or any kind of this how about a google xml sheet with links to implementation of the test cases yeah i i feel like i um i saw this kind of test sheets some long time ago i mean you're you're like listing the test cases in in this way right yeah usually okay dimitri do you want to work on destiny aps um if that includes working on height no i don't have that much availability now i mean it's not it's it's it's not yeah i'm not speaking about hive now i'm speaking about standalone tests like regular block tests that you write that you're usually great with test eve yeah for the difficulty opcode we are going to make tests or for the random the random op code it looks quite straightforward once the get uh implements the merge we just make those tests in our usual format i thought that perhaps we would need the hive upgrade for all existing tests that we have because consensus change and the script importing blocks would need to execute this node simulator confirmation that this block is legitimate something like this hello so um i i did implement when i implemented 439 i implemented a way to pass the random field into a state test and execute the state test with with the post merge rules i'm not sure if that's the right way to do it or if we should rather have a new import command that says okay those like import those tests as if they were uh with uh post match rules uh i thought we can just use the blockchain test format as the block header is not changed we just treat uh nonsense hash field according to et and tests random op conject that way yes yes really need status format for that but you will need to switch the software to the proof-of-stake consensus tension yeah we need like we need to we need to uh detect that but like i guess that it's not a big of a problem but we have to think about how to do it uh who are we which client yes i guess you can use like some value for terminal total difficulty that is from genesis i don't know for the get um i use transition tool to generate the test it accepts the fork flag through which i tell on which rules to execute evm commands and when running when running on hive is that script uses command line there might something might change that what i was talking about that is because um get execute all the tests through hive what do you mean oh yeah all the existing tests the block channels that we have are executed on high on get through the execution script we trans cat command line and i think there is like genesis config file which tells which fork to activate so i don't know if that's gonna be enough of the transition because of the second client which should supposed to talk get and authorize blocks right as a validator that's why i think we need to have a simulator that can send pre-merge box and then also send box via the engine api yeah immediately there is an engine api as the engine api is the protocol that is used by cl to communicate to consider by consistent clients to communicate with execution their clients and this is going to be the natural um path to drop post merge blocks on execution engine execution their clients yeah currently scripts on get on hive do not use it right they would need to be upgraded and i looked at the command i think i can implement the same way and retest it the same way hivescript is doing to get client it's a slow way of executing the test we have because each test will require a client restart and might take longer longer time i mean i'm talking right now about executing the existing test that we have after pos merge so that they will work on hive also you've been saying that 43 like that random of code is straightforward what do you think about 36.75 like it's more um i mean it's i know the test for it is easy to implement because random is just read it from block header so and the block header format doesn't change so test format doesn't change and we just create a test where we put a byte code of random and just check that the value is taken from block header for this what is i mean because in status format we would need to change this format again for for this random field yeah and uh yeah unless someone really needs it i mean we can do it but maybe we just do it in blockchain yeah yeah i don't think it's like state test valuable here would you mind just to create this document with test cases yeah for the random upcode here we are going to create this um i'm thinking if it's a good idea to use the same document for all tests like probably use different tabs if it's a google sheet for for hive and for other stuff what do you think i mean like a list of implemented test cases right yeah and yeah a list of all test cases with the statuses like it's implemented it's in progress this is not started yeah i see so can we use the mainnet checklist for basically like the high level categories like for example say you know like 30 75 tests 43.99 tests uh you know adding like an engine api simulator to hive and then just link those to whatever repos or issues make sense so that we can use the readiness checklist as like that the kind of high level view which kind of links backs out to like you know potentially a set of issues or just like a tractor issue or whatnot in in those different reposts because yeah it's hard to like create a new place where everybody kind of goes to and we already have one um so i think there's yeah there's a lot of value in trying to um yeah i i yeah i think just trying to to move as much stuff as we can there and then have each repo kind of organized on its own yeah i think it's it's like a good idea and reasonable so they will be kind of separate like independent from each other yeah exactly but at least yeah in their readiness checklist we can like link to all of them and people like because clearly it's like a different skill set like do the hive tests versus just the actual blockchain tests and whatnot so i think just linking to wherever those live is is probably the way to go yeah cool i think it's a good idea the issue with this might be that i i don't know um like yeah right i don't know yet because i'm not that much familiar with five which simulators uh i would like in to implement this this hive stuff so need some help here yeah i think that makes sense i think we can just list like the basically you know the high level ones that we we put out like for the engine api for uh a consensus layer and then kind of yeah or have the people who are doing it in hive just make a pr to the readiness checklist with like better details right for instance i'm not sure that mocking el makes a lot of sense makes a lot of value but some of the things like this one could be verified only if er is marked with some but i don't know how to do this actually um so yeah let's not discuss it this is just one of the examples of like uncertainty in my case so how to implement this is this possible and yeah what to do does anybody have any questions with respect to this document um i have a question about the stay alone part uh so i saw you uh listed some uh helpers from the vegeta guy of the consensus fix so i wonder uh so we now we only test this cases uh unit test in pi spec we did them provide any test factors of this test i mean for the regulator guide test so um would it be useful to create hash factors for the radiator guide uh we didn't provide it then because uh this is not really part of the ancient consensus so the client can implement them in a way yeah just anything yeah yeah that's a good question i think we just need them to be tested and we just need to test this pack itself without exposing any vectors for validator guide so i have just parsed all the functions that i have met in the consensus packs merge folder so this is this is how this section like appeared so yeah okay got got it thanks i have a question regarding the integration um engine api statistics um so do they have to be implemented on top of hype but because from whatever support is impact mesh mock is more suitable to run static test vectors for this well i think the idea with hive is that it's just like this encapsulated piece of software that runs all of the tests we care about against ethereum clients and so it feels like it would like like right now we're kind of manually doing a simulator by running the el client locally and then running merge mock and then it sending requests to the el but i think that could all just live in the hive simulator and then it would like spin up like a docker file for each client and then do all of the different um tests against each one of them okay that makes sense okay and yeah i think also hive is yeah the um one of the upsides of hive is like it can be integrated in ci right so it can be run on ci and run all these tests um i don't know if if client implementations integrated it i guess yes but i'm not sure yeah for instance that does get run hive on ci well i've is uh hive pulls the new releases from all the clients and then tests them so i'm not sure if we triggered that by ci or if this is just done by hive itself oh you mean that it's cons like if there is anything new in the repository it's yeah it's almost constantly running yeah oh yeah cool yeah that's great also yeah so we have this using hive we have this kind of automation for and we can have this kind of automation for consistently clients as well like we will have to have this kind of automation for consensus tech lines because um yeah because some of the tests are involved involve consistently as well okay cool so regarding fuzzing i implemented a fuzzer for the engine api a while back that of course became stair with the new specs once the specs are like pretty stable i'll i'll take another step at it and let it run like uh um on on on an ef machine or something so if anyone is interested in fuzzing or fuzzing on the execution layer client side or maybe also fuzzing on the consensus layer client side they can they can talk to me and then we can we can figure stuff out cool so and it's just yeah the basic usage is just to drop keep dropping messages and see if client responds right if it's not crashed or yeah so um it like we i have like four different modes i think so some some just do random tests that should be like really random tested that should never work some do things like create a payload execute the payload uh set the fork choice to this payload um and the fuzzer itself can this uh can like choose between those different strategies and it already found an issue in gas with where it like it kept like it basically like it created blocks and then it kept um setting the head back and and like re-orgging the chain until genesis and then then it crashed something so it's like a really really nice way for the execution layer clients uh to test their stuff because it does a really interesting interaction yeah that's great and do we have do do we do like evm fuzzing stuff yes like we do we do evm fuzzing i'm also doing that um and uh perry and i i'm not sure yeah he's on the call um we we plan to uh um like we're doing differential fuzzing evm fuzzing so testing all the different like creating creating a state test and then executing it on all of the different clients and looking at the traces and see that they execute the same op codes that they have to stay same state route in the end and stuff like this and we plan to [Music] basically increase this a bit because right now it's only one machine running running those tests and uh we would like to like further this a bit um are you running them like to test all the client implementations or are running them in pairs or what no no i'm like all the client implementations uh so it's not all the client implementations it's uh uh geth arrigon bisu nethermind open ethereum those are the the the clients that are currently run there um if any one of the clients are listening in and want to be added to it you need to implement something like eap3155 which is a way to [Music] trace transactions and then contact me and we'll figure it out how to add your client to this yeah fasting is very important thing one of the tools um please uh give me the link also so have a look at it give me the document or the phone uh yeah for the test fuzzing um also i should i should also say that um bringing consensus to their clients to hive means that part of resources of consensus their client implemented implementers of the teams part of resources of the teams that implement the stack client will need to be not not of course not full-time dedicated but dedicated to to look at hive and if something happens yeah some stuff so it will need some attention to the fix i think i maybe this is naive ideally it's like if we can get people that are not on the core team they'll like do the implementation and then just reach out to those teams if there's like a problem with the client itself or whatnot it's kind of like an 80 20 thing where most of the work is not done by the client devs but then obviously if there's an issue with like adding lighthouse or adding prism like that might be quite team specific um and and uh and like we're gonna need their help but ideally it's not like the core thing they work on um and and we can have people that are not on those teams trying to help set that up i'm not sure how realistic this is but we can give it a try i'm planning to focus on working on testing related things for the next couple months um i'm just trying to finish up some unrelated things to hive right now so if people want to immediately jump in we could set up like a weekly sync or something and just talk about it yeah i don't know if anybody was interested in jumping in now i would be really interested like i'm still focusing on the implementation for most of my time but uh um i think that will will also shift in the next couple of weeks okay when the implementation is like getting stable and so i i'm also shifting to testing cool also do people in general agree with the high level outline of the test plan anything missed here um i i think we we have to test some some more edge cases things like uh what i did with the devnet like mining during the transition and and stuff like this yeah this is i agree not not like it's neither integration tests nor nor like static test cases so it's more like it's more defended as system dense yeah yeah yeah it's almost like qa in a way right like we need people like to stand up these networks make sure that they work and try to break them yeah and and like also like document all of the findings so for example i don't know i like i sometimes i don't get peers and because of that or like it i i i i i observe that it matters in which way i shut down the consensus lay and execution layer client if i shut down the execution layer client first then the consensus layer client will go into some kind of panic mode and like try to like send payloads and like not not receive them and then once i like restart the execution layer client the consensus layer client still doesn't 100 work or stuff like this so um that was something that i i saw with the lighthouse but um yeah i think it's important for everyone that runs this this kind of software that runs on the test nets to just uh like document every uh weird issue that's that's happening and send it to the teams i'm i like i got like three or four reports already and i was a bit like slow in in taking care of that but they should be taken care of sorry yeah comment from perry and chad yeah i think it's yeah it just should be done i think yeah just again look for like i know clarity if people want to like write this stuff down in like random places like wherever is most convenient for you but then just open a pr against the merge readiness checklist to like add a link there and that can ensure that like you know we have one place where we're keeping track of it but it's i think it's it's unrealistic to get people to write all these things down in the same place but hopefully we can just aggregate the links okay i'm collecting links you already have the link i think yeah so if these categories the high level categories are good so we can use we can start with them or to update the merge readiness checklist i think it could be done this week and and even though if we don't have this like um issue under or any other track and document under each of the categories i think it's also fine it can be added like later once created yeah yeah yeah we don't need everything now and i guess yeah just we only have like three minutes left but one thing worth noting is it feels like if we get all of this done it's like we've done like the basically the base cases um and then i think it's maybe a reasonable strategy to like start with this and like depending on how well this goes then see stuff like you know we talked about like stress testing networks with like uh larger blocks or stuff like that it feels like all those things can kind of come after but that just ensuring we have like a really solid foundation in the next couple months is like if we only get that that's already a really good valuable thing um hey guys by the way i wanted to add one thing um perry mentioned it in the chat so by the way hi i'm rafael um i'm working on getting like some bigger networks running like with all the client combinations um and this thing can either run like a private test net like in a local area network or even like exposing the nodes uh to the public internet and then you could like connect to all of them and the way i'm doing this currently is using my kubernetes so with that we should be able like to then uh create kind of big test nets uh like on the go it spawns up like quickly fast um so we can have like a full-fledged new network like in a couple of minutes yeah awesome cool well thanks everyone and yeah let's uh let's keep this conversation going on the discord um and yeah i feel like we have a lot of next steps here definitely thanks a lot everyone thank you thank you thanks everybody bye thank you