Supporting the Ethereum Roadmap with a Decentralized Blockchain Data Supply Chain

foreign [Music] my name is Brandon Ramirez I'm a co-founder at the graph as well as a one of its core developers Edge node and today I'm going to be talking to you about the importance of decentralizing the blockchain data supply chain and specifically its importance to the ethereum vision as well as the ethereum roadmap moving forward so before we get into that let's first things first like what do we mean when we say the blockchain data supply chain simply put it's everything that happens between you as a user sending a transaction to modify the state of the blockchain and another user reading that modified State based on the transaction that you sent and at a high level we can break that up into three kind of stages you know on the left we have writing state so it's everything that happens with you sending a transaction getting it included in a block on the canonical chain which is through consensus and then Downstream of consensus you have everything that's required for that state to be read for the various applications that you might use that state for for example on a decentralized application um and I think most of us are probably more familiar with the right side of the supply chain because it's gotten a lot of attention in the last few years with things like Mev some of the proposals like Builder proposer separation and Meb boost so this is like an example of what the right side of the blockchain data supply chain uh will look like in ethereum in the future you know where you have a transaction going to a private or a public mempool you have people bundling that into larger transaction bundles you have block builders that might be these specialized actors trying to kind of extract as much Mev as possible and then you know eventually it makes it to a proposer and validated as part of consensus Downstream of consensus then we have everything that's required to get that data into a useful place for you to do something with it as a user broadly speaking we can break that up into extract transform and load and you don't always need to go through all those steps like for example you know Json RPC is a form of extraction there's many use cases that just consume Json RPC directly many people today consume blockchain data through subgraphs in which case they're using subgraph mappings as a transform step and then loading into a database or a store that's optimized for graphql queries uh later in this talk I'll be touching briefly upon two technologies fire hose and substreams which are Next Generation uh versions of the fire hose and extract uh logic and and the uh the supply chain got some autonomous lights over here at a high level we can sort of categorize the different approaches to decentralizing the blockchain data supply chain the first approach and I would say for a long time this was kind of the conventional thinking was what we call Web 2.5 and in web 2.5 basically what you're saying is we do all the things that we've been talking about you know the last few years to decentralize the right side so you know protecting against Mev keeping good stake decentralization and keeping good validator decentralization but then Downstream of that we let the reads be you know through proprietary apis centralized servers Etc um and I think the reason that uh there's been so much more attention to the web 2.5 vision is because failures on the right side of the supply chain tend to be a little bit more visible or at least a little bit more intuitive so it's like you're getting your transaction censored from being included in uh you know consensus that's something that's going to be very apparent and you can point to um similarly if there's actually like a determinism issue Upstream of consensus like some kind of data or execution inconsistencies that's actually going to lead to a consensus failure like a potentially like a unplanned chain split and that's a very visible type of failure and so I think you know rightfully so there's been a lot of attention to everything sort of Upstream of consensus but you can also have data inconsistencies for example Downstream of consensus that some things that uh actually our indexers in the graph have like come across and helped debug but they just don't get as much attention because uh you know inconsistencies in the way that you read data Downstream of consensus impacts you as a user but it doesn't show up in for example a consensus failure so the other approach is the what we call the web 3 approach web 3 simply put is full stack decentralization in the context of the blockchain data supply chain what that means is decentralized rights like what we just talked about but also decentralized reads A lot of my thinking on this is obviously influenced through my work on the graph but I want to note that there's other projects in this space like portal Network pocket Network true blocks that are all taking varied approaches I definitely recommend you checking out sort of you know this landscape of you know the decentralized reads so why do decentralized reads matter to the ethereum vision or to the web 3 vision I think we can start by just reasoning about it from first principles and like what it looks like to access a decentralized system through a centralized intermediary by a quick show of hands does anyone see something wrong with the diagram on the right side of the screen I'm glad I can educate you today so um you know what we're showing on the right is basically a decentralized system kind of on the top right being accessed through a centralized you know intermediary so like so much of the work that you guys have heard about in the ethereum ecosystem from ethereum researchers is really about maintaining decentralization at the blockchain layer at you know uh you know at the consensus layer and then what we've effectively done in many instances is taking this you know decentralized system with all its benefits all its advantages put it inside of a box and then accessing that box through a centralized gatekeeper and at that point what's inside the Box really becomes irrelevant right all the work that uh uh that's been done to like actually build this beautiful decentralized system is sort of opaque to the end user into the applications that are built on top um offeree I think put it really uh concisely he's one of the you know early prominent ethereum core developers that just you know if dapps are still accessing you know uh the blockchain through centralized you know hosting you know then the ethereum vision has effectively failed is the way that he the way that he put it I was working on an analogy to describe kind of the web 2 dilemma and hopefully this isn't too soon but one analogy that kind of made this stick for me a little bit is that if web2 is someone uh sneezing on you during the pandemic and web3 is a properly fitted n95 mask then web 2.5 is kind of like a chin diaper right and like I know there's a lot of different uh opinions and you know ideas on the efficacy of you know mask mandates Etc but I think we can all agree that the chin diaper really didn't make a lot of sense and that's kind of web 2.

5 right it just doesn't make sense it's sort of uh you know it's incongruous with the goals of of you know blockchain and web3 uh specifically it undermines a lot of the value propositions that you know the ethereum founders uh and a lot of the early you know builders in web 3 sort of uh you know United around in the first place so you know being able to build Unstoppable applications with composability when you start building on centralized infra run by you know single service providers well then your app is no longer Unstoppable it stops as soon as you know that platform gets shut down um once you stop building on Open Standards the composability of your applications and of the stuff that you've built goes down quite a bit and even if there was a standardization on these centralized servers there's a limit to how much composability you can get when you're building on centralized building blocks because the more centralized building blocks you compose the more brittle your system becomes because now you're basically you know if one centralized building block in your giant compose system goes out well now that ripples to breaking your entire system as opposed to you know Unstoppable smart contracts running on ethereum the way those work where you know a new D5 protocol can come along integrate with an existing Unstoppable D5 protocol and that composition uh can keep you know playing out until you get this you know this really rich tapestry of you know decentralized applications um it definitely hurts for usability you know if you're just doing one-offs all the time and then I think you know on ownership censorship resistance fairness I'm gonna try and paint this a little bit in a few examples uh the first is um how many people saw the the article that uh Moxie Marlin Spike from signal wrote earlier this year uh okay like let's say 30 so he he wrote I recommend you look it up he wrote a very well-intentioned well thought out critique of web3 from someone actually trying to make a good faith attempt at building uh some decentralized applications in the space and so one of the things that he did was build an nft app and to sort of uh kick the tires on like the guarantees that we think we have when we're building on nfts he basically built a single nft but depending on where you viewed it whether you viewed it on openc uh wearable or in your own wallet you got one of the three images shown on the screen so like when you're actually buying the nft or looking at it in a collection you're seeing like this cool piece of artwork and then by the time it makes it to your wallet you get a poop emoji and that's possible because the nft was not hosted on decentralized uh data and using a decentralized data supply chain it was using centralized servers to serve that that nft you know and and for a lot of us who are sort of inspired by the ownership vision of web3 you know that web3 is enabling digital ownership you know how can we own things when the actual information artifacts that we're owning are controlled by centralized you know intermediaries so this experiment also highlighted a a way in which we have censorship in quote-unquote web 3 today because once uh openc realized that Moxie had done this they actually de-listed uh his nft from openc which if openc was doing that as a individual application and doing that for you know sort of their own purposes and I think that's you know fine like we believe in choice you know at the at the level of projects and individuals um but the problem was that so many other projects in the ecosystem we're building on the open C API that him getting delisted from the openc application also made it so that the nft stops showing up in wallets and all these other places where you would expect to see an nft that you had purchased right so this is a real example of censorship um that's happened just this year in in the context of you know the blockchain data supply chain for a use case that we all I think care a lot about another one that this is a little bit more speculative but I think it's an interesting one just as a thought experiment is um index or extractable value this was an idea that was put forth by someone in our community you can think of this as being a little bit analogous to like Mev or also payment for order flow which is which got a lot of attention this past year or two when people realized that Robin Hood's business model for offering free um you know stocks trading to retail investors was that they sell the order flow to Giant hedge funds like Citadel and then Citadel with its privileged access to these to this order flow can then like extract all this value at the expense of the retail investor you could imagine a world where if we were all accessing the blockchain where we expect more of our financial lives to exist over time and we're all accessing it through a centralized intermediary uh that that centralized intermediary might gain a lot of Alpha from you know being the one that's seeing the sort of Google Trends style uh you know data around accessing and querying you know our financial lives as it exists on the blockchain okay so that's kind of where the web 3 vision is at today and sort of the importance of you know the uh keeping the blockchain supply chain decentralized let's talk about where ethereum is going and the ways in which a decentralized blockchain data supply chain supports that Future Vision so for those that saw vitalik's ECC talk earlier this year you might have come across these new um let's call them work streams that are all happening in parallel so you know they're not sequential Milestones but they're all all uh parallelized work streams that are happening in the ethereum ecosystem the merge The Surge The Verge and the splurge and the diagram on the left can look a little intimidating we're just going to focus uh on a few aspects of this roadmap specifically parts of the Verge uh The Purge and um parts of sort of the light client vision of ethereum so a brief overview um The Verge one of its sub goals is around supporting stateless clients stateless clients allow validators to sort of be small and light while having like heavier uh specialized Builders uh in the you know the right side of the blockchain data supply chain there's different approaches to this weak statelessness strong statelessness uh weak statelessness is where Builders sort of provide the um the witness of state that's being used by a transaction and strong statelessness the end user actually when they submit a transaction they would also have to submit the state that that transaction is going to access as a witness to the network The Purge is about history expiry as well as state expiry so eip44s covers the history expiry side of things which is the idea that after a certain amount of time in contrast to the way that ethereum works today full nodes and validators would be allowed to drop they'd allowed to be allowed to prune historical blocks um uh basically keeping the the storage footprint of these nodes lighter and then State expert expiry basically um uh attacks the unbounded state growth of ethereum where certain state if it's not touched frequently enough again would be allowed to expire and then like clients uh we're going to touch on a little bit in the next screen but I think they're interesting to bring up here because they share a lot of the same requirements as sort of those previous two milestones and they're also part of the solution space right because a lot of this stuff boils down to being able to access State and and data uh but the problem with like clients is that today at least like clients rely on altruism from the full nodes in the ethereum network so if those full nodes decide to support the light client protocol and serve data to these like clients then it works out great uh but the reality is that because of that lack of incentives uh as you know Piper one of the researchers working on this noted uh basically the the light client protocol in ethereum is a vast desert of starved clients desperate for data it's not working right it works nice in theory but in practice you need incentives or some way to make that data available to support these light Clan protocols which mind you is very important to the vision you know ethereum's founding vision of you know decentralization um so the common the common threads you know without going too deep into the uh the weeds of any of these sort of proposals in the road map there's some common threads that we can kind of uh paint here uh in terms of requirements and it's basically that all of these proposals are going to need to uh depend on some way of reliably uh efficiently and verifiably accessing either historical blocks or pruned blocks from the you know the blockchain or expired or uncached state so in the case of State expiry it's expired state in the case of stateless clients it might be just state that the Builder or the user needs that's not you know cached locally and the solution space breakdown here I think you know can be broken into two larger categories uh one is financially incentivized approaches you we mentioned those like uh those you know like clients needing cooperation from the full nodes which right now don't have an incentive um so things like the graph things like pocket Network could fit into that category um the other category here is using more like Tit for Tat incentives which are not Financial incentives but this is kind of how like the BitTorrent network works for example where you know if you're being a good citizen in that Network uploading data as much as you're downloading then uh there's sort of this reciprocity you know that you get and that's the approach uh being explored by uh projects like portal network using things like gossip protocols and dhts which again are techniques from BitTorrent and now ipfs as well uh and so why are we doing all this like why are we getting rid of all this state why are we getting you know what are we doing it for the fun of it no we're getting rid of this state in this history for a purpose and that purpose is scaling ethereum in a decentralized way so simply put what that means is being able to fit more transactions into a block being able to have bigger blocks while maintaining a small footprint for validating clients and and like clients and as an example of this one of the eips that's out there right now is called eip4488 it proposes a uh approximately 5x cheaper uh 5x reduction in gas costs for call data this is about supporting like the ethereum like roll up uh you know Centric future um but the reason that they feel comfortable doing this which was inevitably going to lead to bigger blocks or more full blocks uh is because it's intended to be paired with eip44s which is one of the proposals in The Purge which uh specifies dropping this requirement for full nodes and validator nodes to keep around all this historical blocks past a certain point so that's what all this really boils down to is decentralized scalability of ethereum okay so this last one is kind of a bonus section I think this is really interesting from uh both a couple standpoints one is that the fire hose which we we briefly noted earlier is a Next Generation extraction technology I think it's interesting a because I think it it should impact the way that ethereum clients get built in the future um but I also think it's interesting because it's an example of sort of this uh positive externality coming from an ecosystem that's tackling uh the blockchain data supply chain in a decentralized open source way right so this is fire hoses technology originally created by streaming fast which is another core developer in the graph ecosystem before we get into how it works it's worth um calling out some of the problems with Json RPC and the way that it works today so you know most of you I think are show of hands people that are familiar with Json RPC okay most people in the room so this is how you access you know data from like a Geth or an Aragon client today it basically depends on a running program to read data so if you look at this diagram on the left you kind of have this fan in architecture where all the users are sort of hitting a running process that process consumes CPU that process becomes the bottleneck for getting data out of the node and then because you need to have heavier nodes to serve all the data that people want like archive nodes potentially running Purity Trace it's also incredibly intensive on memory and solid state disks because in order to actually efficiently access that data you need to provision a lot of of both of the latter it's difficult to get query intermediate States so Json RPC really only lets you query data as of a block you can get a little bit more by using the parity Trace API but it's still incomplete also very cumbersome it can be difficult to debug so if you're using you know a subscription to like if get logs for example via inferior like we've encountered this you know in our own stack that like sometimes you know in the past messages will just get dropped you know due to you know transient Network events or partitions uh and there's just no way to debug that like you just miss the message and like uh it's very very difficult to figure out that like hey a message you should have gotten never made it to you um and there's a pretty incomplete verifiability story some of the some of the data you you want to get out you can get a Merkel proof for but other data uh like if you're using like the eth get logs uh interface for example there's not a really easy way to get a compact proof that says you didn't miss any logs you know in within some range of uh within some range of blocks um so what's the solution uh we think the solution is the fire hose approach uh it's streaming first so if you look at the diagram on the left uh data is being on like as soon as it's available it is being broadcast out in a fan out approach rather than this fan in it's being distributed uh as flat protobuf files so these can be distributed across uh you know commodity Hardware like and using things like you know uh Google Cloud object storage or Amazon S3 um because you know now we have these distributed flat files now we can parallelized workloads on those files you know all without even touching the blockchain node so this is much more akin to like what you would see in like a Hadoop uh you know big data architecture where you have all these flat files distributed across commodity hardware and then you have you know these compute clusters that can be scaled independently and spin up concurrent and parallelized access for doing this sort of transform steps you know on that data and so so when you do that stuff you get you know compared to the approach of using Json RPC today you you know you get a one to two uh order of magnitude you know increase in read performance uh based on the use case uh there's a couple integration strategies that I'm going to move through here real quick um the first is just integrating this as a drop in replacement for Json RPC basically have it be something that you know runs locally on the Node when you're running you know your node instead of accessing via Json or PC you would be using sort of like the fire hose stack and that's kind of like a really basic integration but in the future you you also might want to actually improve the verifiability guarantees that you get from firehose so you know you could imagine sort of the fire hose instrumentation logic running as a wasm process that is implemented as a read oriented roll-up so like an optimistic roll-up or even a ZK roll-up that actually uh gives you some guarantees on the validity of the sort of uh data that is extracted through that fire hose instrumentation process and then the Final Approach that you know and again these are still very early and US thinking about this but the Final Approach is you could even potentially integrate fire hose files into the consensus process itself now you probably wouldn't want to do that um initially because initially there's a strong benefit to having what's called like data agility being able to evolve the schema evolve the instrumentation logic based on sort of a changing understanding of what's needed but as those Integrations stabilize there might come a time and point where blockchain core developers decide to say hey we'll you know we'll reference you know a maybe the fire hose files from the previous Epoch you know at some future block right so that they're actually you have some guarantees natively in consensus I'm happy to announce that uh actually we have our first example recently of a blockchain core development team um self-serving their fire hose integration and I think this is going to be really important for every blockchain core development team especially including the you know ethereum ecosystem to eventually own their fire host deployment so that they can sort of maintain that for their own use cases and then that also has secondary benefits like basically automatic integration with protocols like the graph um there's another uh integration story here uh related to eip44 called shim clients but we're out of time so if you're interested in that uh you know feel free to come up to me and find me afterwards highly recommend checking out Alex B from streaming fast talk this week on substreams and then also Vincent from asari has a talk on standardized subgraphs but they're heavily using fire hose and substreams as part of their as part of their dog footing So yeah thank you guys very much that's my talk do we have time for questions or I guess we have time for a couple questions we have a mic in the back is there a published specification for the phone host fine there's there's pretty detailed documentation um I wouldn't say it's at the level of like an IEEE spec or anything like that but there's very detailed documentation on the architecture how it works the repos are open source um so if you just search like fire hose the graph docs uh uh you'll find it real quickly yeah Dave in the front so for oh thank you for the um the stateless clients in like eip444 is there any way to estimate or predict like how like basically what I'm trying to understand like can the graph Network itself as a whole essentially like earn income from providing a service of like you know basically the the current ethereum nodes retiring the state but then graph notes serving up that and then as a whole like the network getting some profitability from that yeah it's a really good question so this is um these are sort of emerging ideas in our ecosystem but like increasingly my view of the graph um uh is that it is going to be like a multi-service uh ecosystem so like today the primary service is you know you do ETL into a subgraph and query it via graphql but there might be a range of services for example accessing fire hose data directly or Json RPC data directly or data in a key value store or a SQL store and I think all of those could potentially be supported side by side in different like name spaces if you will within the graph ecosystem and in that context then you could imagine yes graph indexers and service providers being the um the endpoint that like light clients use to to get their data it could also be what blockchain nodes themselves use to hydrate uh you know data and some of these other you know related to these other Milestones so yeah I think I think it will be a source of uh query volume you know in the future on the graph yeah great question cool I think that's it thank you guys so much for your time and attention [Applause]