Simulating an Ethereum network at scale | Devcon SEA

[Music] hi my name is pop I'm a teacher at eum foundation and is Daniel our itum protocol uh fellow uh today we are we are going to talk about Network simulation uh before talking about the neigh explation we have to know first like why we do fation at all right uh from our experience uh uh we we usually have like two common use case like the first one is like for example if you if you have like the the protocol idea and uh you are not sure like uh if it's good or not right and uh maybe you you you you want to to take this first you you so you have to run the simulation uh yeah the the second one is like foro if you if you have the have the hypothesis that okay maybe you you have the that is good already but you are not quite sure yet maybe you you want to to run a simulation like to to confirm your idea uh when you do the usually the input usually be like the you have to have the protocol implementation first right otherwise you cannot stimulate it the the second one is like you you have to to have the network topology in mind like uh how much band withd uh each link have in in the in the network right or like uh how much is the ly and like how much is a packet L late right and uh you have to to to make the network phy like as similar to the L as possible otherwise like the simulation doesn't make sense at all right and the output of the cation uh what you usually want from the simulation is like you you want the metc like the number of reos or or message probation time and it in fact it can be anything that you want that that you are not sure that uh what it you be like before theion uh okay uh before before talking about like how we doation uh I want to talk about like how we do respiration like previously and uh firstly uh what we usually do is like we we learn the the client uh inside like the physical machine and then we use something like net M to uh to emulate the the the link like uh maybe uh you want some some like band withd on your link like you just use net M or maybe you you you have the latency that you want you just use net M and uh you just run it in the physical machine and that doesn't work right because it doesn't scale really well like when when you learn like a lot of node inside a skill machine uh they will compete for the CPU right and uh that is bad because uh the timing will be like inaccurate right yeah and like I think I think you can just learn a few note like inside a single machine so what would what what you will do instead is like you have to to have like a lot of machines to simulate right and that that is like a problem because it doesn't scale uh there is something called like this event simulation the discret eventation is like uh instead of running the the Cent inside the machine like directly you uh you run it inside a simulator and the simulator will will will will uh po the post and create the event like for example when when you send a message to network right and uh that that that's kind of an event and when you receive the message from the network that so the event is like instead of running it the in the machine it it will just capture that and post the event uh it's good because the clock in the simulation will be independent from the the the Le clock so uh yeah I I tell how it works like later yeah because because the the the the not don't have to compete for the for the CPU anymore so you can Lear like any number of machine inside a sec machine and and you will get a a quite good simulation result uh and uh we have a problem with the exting this event simulator because uh uh yeah like NST or or omnet uh uh when you do the you have to call the liary of the simulator and that's is a problem because uh because when you have to call the library you cannot implement the the protocol like in in in any language right you have to implement it in a supported language that that the simulator support right because you have to call the liy like yeah like like call calling the Liber to send the message and like receive the message something like that yeah but but uh we we found that there is a great simulator called shadow that that doesn't have this problem uh the shadow is uh instead of uh uh forcing you to to to to call the LI uh uh what it does is like it will intercept the system call that that the Cent make right and like so by doing this you can Implement your protocol like in any language because and then Le inside like Shadow uh yeah so uh the example of the uh the system that uh it will intercept is like the the Sensi call or receive uh receive s call yeah and that that's as I mentioned earlier it's a kind of an event and uh another important thing is like the get time of the day uh because because we intercept the the the get time the get time system call as well so uh so you can tell your client like what time it is right so that is why the the the the clock inside the simulation is independent from the the real clock because Shadow will will will tell Yokai and like what time it is so that's quite great uh yeah so in short Shadow can simulate like a lot of node right with the real software because you don't have to implement like in some specific language you can it in any language and then you just Lear Shadow with your Cent and then if in fact you don't you don't have to modify the c or you just use your Cen and and and let the shadow intercept the system call and we Implement something called like each other uh because Shadow uh work with like any kind of software but but but not specific to the eum Cent so we create something called each Shadow to to to let you Lear the simulation like easily you just like tell uh what Cent you want to use and and like how many note how many uh like yeah and then you just Lear it easily yeah the next is uh we are going to talk about like each Shadow let Daniel uh thanks po hi I'm Daniel and over the past few months we've worked on East Shadow uh which makes it really easy for anyone uh to run shadow simulations um including ethereum CLS um so in one sentence it is basically a configuration generator that makes it easy to simulate network uh ethereum networks in shadow um it has a couple of features for first of all it has a rather simple yam configuration format so you don't have to learn anything fancy um it allows you to simulate realistic GE location and network reliability it supports gu Lighthouse T and rest right now with more planned uh hopefully soon uh or at some point uh supporting all clients uh it also to capture client metrics which is nice for analyzing the simulations after they run and it allows you to run your own programs as well if you want to simulate some uh malicious nodes or some attack scenario finally it scales well so you can run simulations with over 100 noes on your laptop so ethereum as you know is a huge distributed Network and it's globally distributed yeah there are like a lot of nodes in Europe and North America but also few in Australia and East Asia and South America Etc um and for such simulating such a network to figure out future um changes to the ethereum protocol it is important that we can simulate the fact that we are distributed across the globe so here you can see a configuration sample uh of e shadow um on the top uh you can see uh we Define the first node uh which is located in East Asia it also uh sets a real ility which I'll will get into soon and on the bottom you see that we um Define 10 more notes in Europe and NA East and also North America East and it will distribute those 10 nodes um across these locations yeah uh home stakers are very important to ethereum like we want to be decentralized so um we want people to run actually nodes and validators at their homes um but the problem or the fact with that is that those uh nodes often have not as good of a network connection like um something running in a server farm and we can reflect this in the simulation by specifying a reliability the top note is marked as reliable which has a gigaby gigabit of bandwidth no added latency and no packet loss but the bottom 10 nodes are home nodes and they have restricted um bandwidth uh a bit of added latency on top of the geographic latency and also a tiny bit of packet loss yeah you can Define your own liabilities and locations if you really need them or just use the predefined ones for convenience clients uh as mentioned right now we support gu and Lighthouse which is like first class support right now and there's also uh support for re and which is currently in testing and it's pretty easy to run mixed client Stacks again uh we extended the configuration uh on top you can now see we have re and teu and on the bottom we use G Lighthouse and the lighthouse validator client and that's all you need to do for E Shadow to configure these clients and distribute the uh configured validators across uh the valid clients in the simulation uh we also support some uh other software for example in the bottom you can see Prometheus uh which will automatically be configured to uh receive all the metrics from the consensus layer clients uh we have a purpose build blob spammer because I did some simulations on peas and for that we needed a BL spammer so yeah it's also pretty easy to use but IIT it on the right here and also you can configure your own program so if you have a modified version of lighthouse for example which does something malicious just to test uh you can configure it as well supporting more clients well to be a really useful tool it would ideally support all the ethereum clients however this is not currently the reality because unsupported clients uh use an unsupported Linux system call like unsupported by shadow um or gets stuck in a spin log which Shadow can't really cope well with or it uses a supported link system call but the implantation in Shadow is a bit different uh than the actual implementation on Linux and Shadow supports currently 186 of approximately 375 Linux system calls that's a lot of system calls and um only four of approximately 80 socket options that you can set when you are opening a TCP socket or UDP socket and to support all programs would have to implement all these system calls and socket options perfectly accurately which is really hard to do and not really uh feasible to do on short notice um so we need to kind of pick what we need we could also just modify the clients that are running in the simulation to avoid those system calls or do not set this um socket option but that is also not kind of nice because we don't want to maintain Forks of clients indefinitely to support them in Shadow so yeah we have to implement these missing features in Shadow instead which uh will take some time simulation size so our claim is that we can run simulations with over 100 node on your Linux laptop but before we get to that let's uh look at small simulations small simulations with under 10 notes for example run pretty quickly like on my system they run faster than real time because as Pop uh mentioned earlier Shadow can also manipulate like the get time of day calls or something like that to pretend that time is running more quickly than it is in real life and as your simulations uh get larger of course the CPU time gets congested and uh simulations will run smaller here two examples uh on my system I usually run simulations with 38 notes which nicely fit into my 28 GB of uh main memory with res Lighthouse and a lighthouse Val cined per note and for me it took like 45 minutes per 30 minutes of simulated time so 1.5 uh times uh slowdown which is rather nice on pop system we tested 100 notes with G Lighthouse and Lighthouse Val client 16 GB of memory and to simulate that many nodes you will need swap space so we used uh 32 GB of swap there um this took far longer uh 11 hours for 10 minutes of simulat time so it is possible to run those but you should probably start with a small simulation to confirm that everything as well and then just let your laptop run overnight if you really want to have a larger simulation um you can also run really really large simulations with over 1,000 notes as uh we have retested and for that I rented a server and that uh had one terabyte of memory and that took for 45 minutes of simulated Time 4 and a half hours approximately it kind of varied a lot at that point um and also at that point what I want to note is that at that point memor speed becomes the bottleneck so we can just throw more uh virtual cores at it and hope it gets faster uh no as we as Shadow has to switch between these processes a lot we just exhaust the memory band with our main memory and we can't really um get faster um by yeah having more course um finally I want to point you to some uh another presentation that I held yesterday uh where I also present some experiments uh that I ran uh with e Shadow so if you want to see what if Shadow is capable of in practice how it can enhance your uh testing workflow uh feel free to scan the Q code I'm going to leave it on there for a few more moments and after that uh pop also has something to say yeah uh yeah thank you and actually uh the EF research is like uh looking for a P2P Network expert so if you are interested uh to apply you you can like go to the to our EF job website over there and uh yeah for the each Shadow you can you can also pay with each Shadow like at eum yeah that's it thank [Applause] you yeah okay yeah um similarly a 100 notes on a single laptop also in a single yamamo file um that's pretty cool right um I like that next time you think about simulating ethereum Network check out as you see before GitHub each shadow um next up Q&A uh please scan the QR code open MCAT right go to the bottom right you'll see the live Q&A there so click in there you can create an up questions here um so we'll go with the first question O Okay um the first question is can you scale Shadow horizontally by adding more workers uh I guess the workers mean machine right like yeah currently you have to run Shadow inside like a single machine yeah like yeah it theoretically it's possible that you can run it inside many machine but it's an open research problem and yeah and we cannot do that now okay um the next question is what is the difference between each Shadow and curtosis uh I think curtosis is like a do compos right and like you have to to like when you run many notes inside a single machine it will compete like it will it will share the CPU and if you run 2 minut note it will it it will like yeah it will be like too slow and like the the simulation will be like inaccurate like yeah I think you can just run like a few like less than 10 notes like inside a machine I guess okay um how do you handle the client's memory and storage requirements when running a lot of them uh on the same machine uh the St the storage is not a problem because because we don't have like the the E state so so so we don't have to have like a lot of stage but in in term of the memory yeah as as as Daniel uh talk about it like we use like a one tte memory machine to run like a th000 notes yeah so so you have to have a lot of memory uh if I may add to that um we also start with a fresh gen uh state so with a new defet and notes running on an basically empty stage use way less memory than a mainnet node so like that we can like if you think about it 1,000 nodes on one terabyte is 1 Gigabyte per node which is very very small but it's possible uh because we have this empty State okay um the next question is is simulation able to to capture some Edge case flaw in the protocol I'm not sure what it Cas mean like yeah uh I'm not sure like what Cas maybe uh well yeah I think it is because I um just mentioned the my other presentation you can also uh find it in the defon uh schedule um I used I used it actively to to find um issues well with the issues with the protocol it would be also uh possible to find it there if shadow in my opinion is more suited for Network related uh things be because yeah uh it it it it really puts its focus on on on the network layer uh but it's sure it's possible that you can configure some attackers that find some Edge case in the protocol through um their attack for example yeah oh go ahead go ahead no no next the next question is do sh does East Shadow support logging capture yeah yeah I would like to answer this question so much yeah uh yeah I think this is this is an important thing that I didn't mention in the in the in the presentation like yeah you can see every SD out like of every note every proed and like you can also see like SD out SD ever and you can see every file that that the that the cents produce like in every note yeah that that's quite cool okay cool we have one one more minute for Q&A so uh please keep the questions flowing in the next question is do you support DVT such as SSV or o o OBO I guess uh and what about IG layer ABS not yet I mean theoretically you could configure it kind of yourself however it's likely that those clients also won't work within Shadow Without some uh added work in shadow um yeah okay 30 seconds um is it complicated to add other clients yes that's why we hope that we can um like get the core developer Community interested in this because it's a lot of work especially in Shadow so uh for example te uh support uh an Anton Nur from consensus helped us a lot with this um if you are interested in adding your client or some other piece of software to e Shadow uh approaches we'll try to get you started all right so um unfortunately time is up uh there's a lot of upvotes there so okay okay one one more question then how do you define the simulation to run just submit the list of transaction to the network uh yeah if yeah if if you want to send uh a list of transaction you can do that you you can just uh you can just spin up a new a special note and then run the software to submit the transaction to to some note and the the transaction will be include on chain and everything and you you can look at everything yeah all right uh so unfortunately time is up I think Daniel and pop will be around hanging around so if you have more questions feel free to reach out to them probably down the stage um for this session thank you so much Daniel thank you so much pop for for the session yeah thank you so much thanks