"The Missing Punk" by Sebastian Bürgel x Gnosis & HOPR // ECC#2 - Buenos Aires 2025

[applause] Welcome. My name is Sebastian Seber VP Tech Hypnosis and founder of Hopper. And uh I want to talk about something that started as a discussion of the uh cipher punk retreat about two months ago in Berlin. And uh there was a conversation that we started there and had an ETH research forum afterwards and has been a collaboration since then mostly of Rich aka the wandering editor as a summer of protocols attendee at myself. So yeah just wanted to say thanks for these organizations which support not just the technical developments but also kind of the meta level growing up that we as an ecosystem have to do.

All right and with that um yeah I want to get started. I promise this will be a positive talk, but I want to get some unfortunate uh news out of the way. Like this was from the BBC. Usually when the BBC is involved, that's bad news for some reason. And yeah, as most of you have seen, there was some uh attack on by bit that unfortunately allowed North Korea to extract over a billion dollar from our ecosystem.

And yeah, um only a bit over a week ago, uh we've all been uh probably facing this balance of v2 exploit. I would like to highlight attention to the second line here. How many chains and environments found themselves soft and hard forking and having painful discussions around that. And yeah, there's more issues. This one might seem a bit obscure.

Um this is basically a issue that we were facing at Hopper. We're building a mixet on top of which we're building a VPN right now. And yeah, we have been very fundamental issues with processing events from Ethereum. ETH get locks is broken. And another issue that we've been seeing recently, I don't know if you can see it on this beautiful screen.

Um, there was an outage earlier this year of AWS um that took out not just the web two but also the better parts of the web 3, which is a bit of pity as left there is nicely highlighted here. So in this talk I want to be focusing on what is maybe not the worst economically speaking but you know the one that was closest to me and that I spent most of my time on over the past 6 months. Um yeah and I think shows just how much there's something broken in this ecosystem and that is the issue around ETH locks that we're processing at Hopper and what it means. This sounds very technical but I want to give a bit of context to take us on a journey. These things might seem random and disconnected, but I will show some connection that I hope you see at least later.

All right, so let's dive in and start by zooming out. What are the most fundamental features that you expect any blockchain to really have? You can write to it and you can read from it, right? So the right part is kind of obvious and is what gets most of the attention typically by sending transactions. Lots of discussions around how making that credibility neutral but actually really a blockchain is just a database and you only write to it because you have the expectation that you can later on read from it also right so we kind of take that for granted but I want to I will spend some more time on you know reading from a blockchain and why that is actually important that we just take for granted.

All right. So, how does reading actually work? Well, reading um a general purpose ledger like Ethereum is actually a bit more complicated, right? So, we're not talking about Bitcoin. We're talking about a general purpose ledger.

And um that one you basically can process all your past transactions and you know accumulate the current state, right? And that means things like uh your Ethereum balance, ERC20 balances and so on, right? So that is something what basically an Ethereum execution layer node does but that is extremely computationally expensive and comes with hundreds of gigabytes of uh data overhead. So that's painful to establish. So there's another way that people introduced which is processing of logs, right?

So instead of processing your entire history of data, you're basically just listening and every now and then a smart contract just emits an event and it can much more cheaply and easily like process this event in real time. Right? This is the endpoint what's called ETHgad logs and these logs are being processed and you know you use that in your wallet for example for your ERC20 transfers uh which you can then see in your wallet shows that you know your accounting software so that shows that and so on. So that's a meant it's a much more easy way to go ahead or so we thought. Um how we use that at hopper is actually hopper is a mixnet of hundreds of peers around the net which are sending information to one another and we have payment channels for an incentive layer for that right and these payment channels actually need to establish the state of the network.

you need to know who am I connected to, right? And it turns out for us that there was an issue. And after literally weird weeks of debugging really obscure issues, we found that this radio station that we connected to, which is basically an Ethereum client that is sending data or an RPC provider that you connected to, doesn't actually do a very good job in that. And yeah so what it actually found what we actually found is this this is an actual error we will see it later in a bit more detail is like it tells our hopper node which is running on this that hey um reopen this uh payment channel and hopper not what the hell are you talking about like this payment channel is closed and is not closed but open that's weird we have something that went missing here so uh what's actually happening this is the actual issue and you know somewhere on here which doesn't fit in the resolution. You saw exactly the message from the previous screen that basically we failed to process events you know and failing to process events is really a very fundamental feature that Ethereum provides for us right it's a very fundamental step in processing events and you know just getting to the latest state if that is broken the reading from the blockchain is kind of broken right that is a very very sad state in which we are in right now all All right, next one please.

So the problem is if we look at ETHG logs, well again a little bit poor resolution, but so what you should see on this screen here is actually the specification of ETG locks and there we're getting towards the issue that I would like to talk about and that is we say we have a standard but what we actually have here is just a couple of lines of an example. So it's basically something like a swagger example like request and response which has been helpful for airing out you know some very basic operational you know making sure that your type is correct and so on but otherwise that is all you know that is all that tells you how an Ethereum execution error client is to uh process events and you know when we see some much more nuanced edge cases like the one that I just presented where we just you know process a bunch of events and these events are just coming in out of order. Well, that is not captured in this very minimal standards. So, I will talk a bit more about standards here. So, first of all, these issues that I just mentioned, um, we're not the only ones to see them, right?

So, ETHLocks is actually something that's difficult to process and a bunch of clients have realized, okay, processing events on Ethereum is actually so difficult. So, we make some trade-offs here, right? So, we make some trade-offs for you, such as, you know, we only allow you to uh process the past, I don't know, 5,000 blocks or something, and we we basically cut corners and that is understandable, for example, for an RPC provider because otherwise you have kind of an unbounded cost, right? But what that means is that every application layer developer like us is kind of facing a variety of kind of unhappy paths and failure cases that you just, you know, have to figure out how to deal with. And also for client teams, they're confronted.

So this is this these are examples of every single Ethereum execution layer client um issues with ESGlogs from the last 12 months, right? So if you just go on GitHub and search for ECG logs, you'll find a bunch of kind of very very messy things here. So um even if you don't care about hopper right even if this you know payment channel payment channel network that we're working on kind of means nothing to you this is something that Ethereum should support right so this is something which should be a standard it's something that Ethereum should support but it's broken all over the place and for a very long time right and that is a problem right and you know there might be people hey Sebastian like Ethereum is working fine for me. Like what are you actually talking about? Right?

I I get it. Right? And this is part of the problem. So why is there no perception even of things being so broken as I just showed? So I think there's a number of reasons.

And the first one is that teams don't actually talk about it. And even issues I showed on GitHub is just the tip of the iceberg. You are just not going to GitHub and file an issue every single time you run into something. It's also a bit of a dick move to just go, "Hey, your stuff is broken. Just DM me, right?

we can we can sort it out. So there's some weird social pressure on this. The second one is for many applications you do get away working with Alchemy and Fura or you know whoever your RPC provider is. In many cases that is okay but that again basically provides even less incentives for people to fix issues. Right?

And the third one is that in um in in many of these cases it's not just um it's not just people dealing with this. It's also that you know uh people are questioning themselves right you look at this and they're like really is Ethereum that broken? I literally after after we tweeted about it, I had people reach out to me and were like, "Hey, Sebastian, I was just debugging something, you know, from major projects in the space and I had a corrupt database." And if you debug a corrupt database, there's been layers of software stacks on top of that before you get there. So, people are really going crazy and challenging themselves before they realize, well, actually, an Ethereum execution layer client is really that broken.

So, yeah, how do we get there? Um a few things I heard is Sebastian there really is no issue and I would say well you know lol no as the previous slide showed there's a bunch of issues for a bunch of teams um that do show that there is kind of broken issues. Um the second kind of cynical explanation is dubdubs don't know what they're doing. Um I would say that is a little bit too narrow view like eth logs is kind of hard to deal with. It's kind of messy and very nuanced in what you do with it.

And there's many different ways on how these logs are actually being processed, right? So, you know, you subscribe, you just get events, you subscribe to an event, you have filters for it. It's not easy, granted, right? The third one is RPC providers are greedy. Well, you know, guess what?

RPC providers are companies. Companies make money. That's what companies do. Shout out to everyone who works on DAO, alternative revenue, stream generations and technology like that. That is why that is important, right?

Because fundamentally these companies aren't really exactly aligned with what we set out to do here. So I mean yes and RPC providers by cutting their cost, you know, are looking what's the most expensive stuff that we're processing and many times that is things like events, right? So that's that's where they cut corners. But again, breaking compatibility in absence of standards. And finally, I heard well client devs don't know what they're doing.

Uh that's the one that I most strongly disagree with. Um some of the smartest uh folks that I know in the space are working on Ethereum client layer implementations. So that is certainly not the case. But it seems to be hard, right? So why is it that hard?

You know we have basically exe some of the smartest people who are actually getting together in the space who try to build something to to do the good thing and you know they have to deal with this and um well the actual situation is because their work environment looks more like this. They are trying to build clients and we are having unanswered questions that we don't have consensus on. For example, let's stick to the basic example of, you know, ETH get locks example. Again, it's something that a smart contract emits. For example, when you transfer an ERC20 token, right?

An ERC20 token, you know, let's say you want to find out all wrapped ETH that Vitalic ever, you know, transferred. So, you basically look from Genesis block to the tip of the chain and want to find out, you know, give me all transfers of wrapped ETH of, you know, where Vitalik, the address of Vitalic. was either the sender or recipient. Is that a valid use case? Is that something that clients should support?

Because you basically need to search through the entire logs from genesis to tip. And well, that's computationally expensive. So people cut corners there. We don't have consensus on that. And why do I bring this up and want to talk about it?

Well, because you know, my job is also VP technosis where I get to uh talk to RPC providers and find myself in discussions. Well, can we please get this to work? And like, hold on, Sebastian. What do you describe is not a valid use case? I'm like, why?

You know, show me this back. And guess what? There's none. We don't have consensus on how an Ethereum client is supposed to work. So, I did engage then in uh discussions on the uh all core devs testing call and turned out we don't even have consensus on if we should get to standardizing clients.

There's arguments such that I want to optimize my Ethereum client for ZK provability. That is an entirely different use case than me using it as an application layer developer, which again is an entirely different use case from somebody who just wants to run a validator to make money, right? The Lido people and I have very different, you know, goals in how they use Ethereum. Should we use the same client? Like Ethereum talks about client diversity, right?

Client diversity is something that we uphold very strongly. But like how does that get where does that get us? If we have one client that's made for ZK provability, another one that's made for kind of staking and another one that's made for application layer. Then basically we're in a situation where I am in right now with my DUP node right now. You know, I don't have many clients that I can actually run, right?

Because disk footprint is the other dimension to optimize for. So yeah, all of this is basically just saying that we have a staggering lack of standards in Ethereum land. If we don't get together to talk about this, we're going to end up in a tough spot. And to come, you know, to to get a bit more positive here. So what you should see here, these are the hive tests, right?

So for the ones who don't know what Hive is, it's basically a testing suite for execution layer clients of Ethereum. What we see here like in September when I prepared this E3 research post like there was like a number it basically shows how many tests are failing for every of the seven execution layer clients. Um and what we see on a higher resolution screen is uh that these uh passing test is going up over time. So that's great right? So thanks you know execution layer client teams for working on this taking this seriously.

this has not been moving for at least half a year that I've been watching it for the past three months since we brought some public attention to this you know things started improving and that's great thanks for that um but if we're honest about it you know I said this is also about Ethereum growing up right so what you see here is something on the order of 190 RPC compatibility tests right so that includes the trouble that I talked about on ETH get logs 190 tests and like roughly 50%ish are green passing which is not fantastic where should we be I have no idea but let's look at you know let's look at another example let's look at the web platform test right so let's look at what the other folks did who built a web back in the days and got eventually around of saying hey we need to standardize that because everything is utterly broken I have so many browser forks in my JavaScript code they just can't make any progress anymore and Well, they came out with a bunch of tests which are used for every major browser, right? So, every major browser is using these things to check if their stuff works. And what I want to highlight is the number on the bottom. Every major browser has a, you know, mostly passing number of tests, which is 2 million. There's two million tests.

And I challenge everyone who tells me, well, a browser is more complex than Ethereum. It's not. Ethereum can do so much, right? and you know us having basically uh how many orders of magnitude is that like four orders of magnitude like less tests for all of Ethereum's application layer interface which is the RPC layer has well that's ridiculous we need to level up and we needed to level up by orders of magnitude right and that's basically my my point here of getting you know getting that message out and getting us committed to get towards something like that I'm not saying let's let's build the web but let's get there So let's zoom out again. Right.

So that was very technical. We're diving into ECU logs. And I hope you know my my point here is that Ethereum clients uh well really need to get together to improve the interface of people to work on Ethereum. But there's more issues than that, right? There's more issues than that as we've seen in this very sad example.

So what actually happened here? I would say how do standards help us here? Well, actually, you know, I mean, how could we have prevented this this hack? I think there's no single or simple answer to that because a whole range of things went wrong here. But two areas of lack of standards that I would want to point out is a the lack of security standards of wallets was obvious.

Everybody who has kind of yolo signed a hash on a ledger knows what I'm talking about. People call call it like clear signing initiatives. I'm against that. There's no such thing as clear signing. Either I'm signing something that I understand or I'm yolo signing, right?

So I'm against the notion of clear signing. This is normal signing, right? You understand what you're signing, not just some random hash. So well, truth is most retail hardware wallets don't support any of that. They crop payload.

And yeah, we're we're no nowhere really there in rolling out useful standards in practice today. And the second one is this happened in safe infrastructure, right? Safe infrastructure is centrally hosted by the safe team. Why right wasn't by bit running their own? Well, because we don't have a standards for how this kind of secure DevOps infrastructure could be operated and run.

This is kind of the second domain that I would like people to spend much more time on thinking how can we standardize that stuff to make Ethereum actually secure because you know what actually secure is not just about your wallet. It's also about all the infrastructure stack around it. Let's look at the next one. So, um the balancer hack, right? So, uh this this looks like Barachchain.

Everybody likes to make fun of Barachchine. Me too. Um but many other chains also were finding themselves not just in a soft and hard fork discussion including Nosis. You can imagine you know I find that very contentious. So, but you find yourself basically discussing a soft and a hard fork in a war room setting where there's an active exploit and the attacker can literally move millions of dollars.

You're finding yourself in that setting. And in that setting, you now have to discuss the ethical considerations of a soft and a hard fork. I can tell you I have had many discussions in my life. That is not a setting in which I want to have these discussions. I would rather not talk about soft and hard forks because I would like there to be some standard of how we handle these things, right?

But certainly not in a war room. All right, so especially this last one um is contentious. I mentioned it, right? So for many people it's like this. It's kind of getting us back to a corporout environment.

And that's not the culture. That's not what I'm suggesting, right? Because I know how you feel about it. And I know you feel more about it like this, right? And I understand it.

So my point here is just um standards are not this right. So standards are basically about trying to get things right on our terms because if we don't do it on our terms, somebody's going to impose standards on us. And in the previous panel, we heard something about, you know, GDPR and things. Yes, I also think that GDPR is a standard. A B it has the right intention and it's rolled out in a horrible fashion.

Like cookie notices and me having to pay a lawyer to protect people's privacy is not helpful, right? It's just bad. Why does it happen? Frankly, because all of us, that includes myself, we didn't get our together to protect people's privacy adequately. So somebody else stepped in which for God's sake is the European Union in that case to try to take a stance.

So if we step up to bring up standards we can improve things on our terms. And secondly, what I would like to think about is well, I bring this up because otherwise things things keep happening, right? It's just not good enough to say we stand for privacy, right? Well, why do things still happen? Because large companies in the space know that everybody will be angry when Ledger was doxing their entire database for the 12th time on the dark web and two weeks afterwards it will be back to business as usual.

We need to set better standards for this industry. The next one is, you know, if if I think about privacy, like things do go wrong all the time, right? And if things go wrong all the time, um, yeah, we we kind of get less attentive to what is actually going on. And I would kind of challenge you and say, okay, well, if you just talk about this in your value statement, but you're not willing to take any actual steps towards privacy, right? Something tangible.

Are you actually privacy oriented or are you just a performance artist? Right? We need to set these standards so that we can actually get there. We need to really take these actional steps to not just have the European Union like march in and impose stuff on us. And finally, so taking a stance on these standards is a punk thing to do.

That's my point here, right? So taking action to improve something on our terms for a better outcome is punk. Caring is punk and Ethereum needs standards punk. Thank you for that and yeah happy to take the discussion afterwards. [applause]

[groaning]