Securing Grandine's Performance

All right, let's hear it for Zarathustra and Boma. Hi, guys. So we worked on consensus client performance profiling. Both of us had the same interest, starting off with Grandin We worked on consensus client performance profiling. Both of us had the same interests, starting off with Grandin and branching out a bit, as you do during work on a project.

There's a couple of repos listed there. Two of them are on GitHub, because that's where everybody else is. And I myself prefer GitLab, so I threw that in. I'm going to hand it to Boma, who's going to take the first bit. Hello.

Hello. Okay. Sorry. My name is Mercy, and I'm here to make a presentation. Although our project shifted a little bit, the initial goal was to work on security and testing.

But due to some, I underestimated some things. And so I had to focus on performance analysis, the comparative analysis between the construction client. Okay. So, okay. Sorry, so the initial goal was to do a security and reliability of grinding, and then there was a shift in focus which transitioned to profiling multiple clients due to technical challenges which I faced.

Oh, okay. Sorry. So first of all, I used so many performance tools. The ones that stood out to me was the flame graph and then implementing a timing matrix on grinding. This is an example of what a flame graph looks like.

Although it's not quite readable because you have to click in and then out to really view. But, okay. So looking at these stacks, grinding is actually performing very well from the stack overview, although there is a drop down from the Tokyo runtime, which is at 61.79%, which had a minus 35% drop down. So the key observation here is there is a high trend utilization on 97% in system trends and initial setup, significant drop down to 62% in Tokyo runtime operations, and then a consistent performance across Tokyo tax management.

Also, multi-trained worker execution showing similar patterns in different narratives. Okay, sorry. So this is the 35 percent. Looking at the system trade level, which is 97 percent, then the current level, which is 62 percent, and then the – which identifies it as 5% drop down. So the cliff there is kind of huge and the high performance areas are trade initialization, system level operations and then co-trend management which is pretty handled very well and performed very well in that aspect.

So areas of investigation a disclaimer, this is a research project I will call it a research because so many things can change and then I might be wrong in some cases. So areas of investigation for me is the secure ties scheduling, which is at 62%, and then the synchro time overhead and the tax profiling efficiency. And talking about the timing metrics, which I had to implement, for now, okay, Silo suggested that I had to do a comparative analysis between grinding and then Lighthouse. For now, I only have the data on grinding, and I'm yet to produce the data for Lighthouse to properly do a comparative analysis, but this is basically how the data looks like, and I'm hoping to use this to do more investigation on formal—to do more analysis on formal verification and fuzzing, creating a targeted fuzzing and—what's it called? Formative analysis.

Sorry. And— hold on. So the reason why I had to use tools like the frame graph and the timing matrix was to help me understand more about the conscientious client, because I really underestimated what it is to force and build a fuzzer for a conscientious client. Because coming from a smart contract security background, I thought it was the same perspective, but I was proven wrong, and the complexity was overwhelming. But using this analysis and using this data, I think I'll be able to produce and to come up with a strategy in order to build a fuzzer, which has to do with making it a targeted FOSA and also for formal verification.

Because I understood that there is not much work on formal verification on Ethereum consensus clients. Thank you. I can hand it over to you. So small premise to the second part. So as I mentioned, we initially started working on Grandin.

It was a slight shift in focus. Transitioned to doing this profiling of multiple clients. At least that's what I focused on. DevOps emphasis. I decided to go there since I have no experience doing any sort of DevOps work.

And I figured having a couple of months seemed like a lot of time. Here, that's a good thing to pick up. So that was fun. Built my own server from secondhand part. First time doing that, running Proxmox.

Pretty interesting experience. Learned a lot. Challenges, hardware constraints. I thought I had sufficient amount, but it turned out that maybe the minimum required specs listed are not entirely enough to run all the nodes on a single server. At least not the one that I decided to assemble.

And regarding the outcome, there's limited data. The results are modest modest but valuable insight into running Ethereum nodes was acquired so let me tell you a bit about the setup that I used so I listed on the left side for you guys the consensus clients that are there and the versions of those that I used two of them don't have a version specified that's because I didn't manage to run them so they were not included I would like to of them don't have a version specified and that's because I didn't manage to run them so they were not included. I would like to include them at a later stage when I have the capacity to do so. In terms of hardware I have dedicated eight cores to each of these. RAM 32 gigabytes and the disk space is 2 terabytes and VME drive which is sufficient for all these clients to not be limited there.

In the setup, I make sure that each of them running on a virtual machine is limited to this, basically. So there's no way that if one of the VMs isn't using all of its RAM that one of the others could use it because it's available. This is normally something that Proxmox supports, but that, of course, doesn't allow for a proper comparative analysis. Then the services that I also got to work with and got to learn about, and never mind, of course, I needed an execution client, and a whole bunch of other services that anybody who has done anything in DevOps probably knows. So Prometheus, obviously, for metrics.

I'm not going to mention them all, but I will just say that these were a lot of tools that were new to me. Quite interesting to see how many services you actually need to run such a such a node in a useful way, so that you actually can monitor what's going on. Then I'm going to show a couple of figures on some of the data. It's a bit older data, but so be it. So this is the CPU usage that I measured over the course of about a week, a little longer, for four of the consensus clients that I mentioned.

And now the question, obviously, is what does this mean? And I don't have a crystal clear answer to you. I do notice some things, which you'll probably also notice straight away, which is that Grandin seems, on the average, to have the lowest CPU usage. But there is somewhere a spike in the middle. I see that in a later figure.

In the next figure, I'll show as well. And honestly, I'm curious what happened there, but I simply don't know. I probably need more data. From the metrics alone, it's not so clear. It could be various things.

It might be something network related, could be something client specific. Simply don't have an answer for you there. Here I looked at the memory usage of the different clients. Also, you see the same spike in the same period, but it's for a different client. So yeah, there's definitely something going on there, and I would say that my suspicion is that it has something to do with my server at that point, since I see this weird pattern.

But like I said, I have no straight answers for you. And then the last figure I will show, I think this is actually one of the things that is highly relevant. I only show Grandin and Lighthouse here, because the initial focus was more or less on Grandin, and for comparison also taking into consideration Lighthouse. So the number of pairs that you have is obviously very important and you can see that there are actually quite large spikes to the downside. I have the limit set at a hundred so that's why it looks kind of artificially kept there.

But yeah if you lose a lot of pairs that that's obviously not a good sign. And well, it seems to be fairly, maybe not stable enough or not as stable as you might want it. So in terms of outlook, what I would like to do is refine and scale this node management. So I now have almost an automated setup for the provisioning of my VMs. I would still need to switch to NixOS or Telos to have it fairly immutable and item potent.

Then I want to integrate ETH Docker. This is something that I simply didn't have enough time to properly look into. So so far what I've been using is Sedge. Well, I started with simple scripts and running local binaries and eventually went on to use Docker Compose. But yeah, I would like to move to Kubernetes and I would like to also explore ETH Docker because I think it offers some of the things that I was lacking or missing in Sedge.

And then I only very recently, that is to say yesterday, learned about the secret shared validator. So I'm going to look into this. This is something in the direction that I was thinking I want to use this setup. I want to use this setup myself with an agentic system, because there I have a background. So I think it might actually partly at least overlap with the secret shared validator setup, because I was thinking in that same direction.

And lastly, I want to optimize the data queries, because the idea of this agentic system would be to integrate those metrics so that the agent Can proactively take the necessary measures for example manager node if you see that your peers are getting too low maybe you need to well do something to To fix that Yeah, that is all I have for you. So then I only have a Thank you slide left. Obviously, Mario and Josh, it was amazing. Thanks so much for all the time and effort that you guys spent in granting us this opportunity to be part of the EPF. Yeah, I think that's it.

Are there any questions? Thank you. Hey, thank you for an amazing talk. I was wondering if the notes that you ran were validators or just notes? No, also validators.

Also validators. Yeah. Okay, so it might be nice to see the difference between, like, for each client, what's different if they're a validator and if they're not. Yeah, I agree. There's lots of experiments you could do.

Really, really lots more. I would have liked the pieces and sort of working. And then you need to learn PromQL, okay? And then you learn a bit of PromQL and then you see all the metrics that are there. For example, Grafana, which I also used a little bit at somebody, I just decided, okay, I'm just going to do it.

So, yeah, I agree. There's lots more. And those things would be interesting to look at, yeah. Amazing, thank you. Any other questions?

Mario? Yeah, I was wondering, because you mentioned the setup with Proxmox and you want to automatize it, setup with Next. It's really cool. I'm just, you didn't really have in slides like some more specs about it. Like for example, what virtualization are you using with Proxmox?

Like KVM or LXC or, yeah. KVM is what I'm using, but I could explore maybe alternatives. I haven't explored any alternatives there. So you think it's worth exploring? Just the Linux containers are good for servers as well.

And I was wondering, like, because Proxmox supports both, and you just, yeah, you didn't have the details there, so I was wondering, like, which one. No, this would say. Yeah, cool. Thanks so much. Yeah.

All right. Thank you, guys.