Markets for Public Goods Funding | Devansh Mehta - Ethereum Foundation

Hello. Hello. Now we have Devanch from Ethereum Foundation. Let's talk about public goods. So, apparently you guys use a you new method to distribute $350,000 of fund to public goods last year, right?

So he's going to talk about the methods, the new way of evaluating. Take the stage, give us a lesson.

Yep. Thanks for the quick intro. Um, so yeah, I have about 10 minutes to speak. So I'm going to talk about 10 different points. I think like a lot of us in the public goods space who divide money between projects have been searching for that holy grail where we can introduce markets into the equation where you can actually make like win or lose money based on how accurate you are and this is actually a fully endto-end completed pilot showing how we can actually trade on the value of a particular open-source repository.

Um so I'll give a quick overview. So a lot of us are looking at AI now and all of its different applications. So Vitalik had a blog post which came out last year that talked about the different possible uses. I think a lot of us are on the left hand side where we all want to make a single AI model give it our preferences and hope that it produces a good result. I think this is like not compatible with the crypto ethos or with the blockchain ethos and we're better off on the second or the third one like futarchi where we are trying to predict the an objective event and AI is helping us do that or in the middle where which I'll talk about a little more where humans give humans are the steering wheel and AI is the engine.

Um and what this concretely means is that humans make humans are good at making a few decisions while AI can make many decisions. So how do we use the few human decisions to select the winning AI model? Um so how do we create a market like this is the cool part like how do we start a public goods market. So the the initial point is defining your dependency graph. So if any of you have a project define who your project depends on.

This could be your employees. This could be other projects which you depend on. It can be anything. So once you've defined your like this is the Ethereum dependency graph which has like 10,000 different edges. So this is an unweighted graph and we want to now use AI to give a weight to this graph somehow so that we can put money in and it gets distributed over here.

So this has wide application for co-ops a new way of paying your employees uh helping support different projects that you depend on. Um so we'll start with like a quick or what we tried doing with this idea of like okay we've got a dependency graph of Ethereum how do we now get weights on it so that it's a weighted graph. So we started with like a Jura tree like as the originator of the idea Vitalik submitted 30 spot checks on the graph. So he looked at two random he looked at two random paths and he's and and he gave a judgment on which is more important. um he gave about 30 of those and he nominated two people who also gave 30 who nominated two who gave 30.

So this is like spot checking. We don't require humans to actually do the entire graph but what we do require humans for is to select the most aligned AI model. Um so like we created a market from all of these different edges in the graph. So um all of these markets have a value and if your AI model is saying that hey I think that this particular edge is underweight as defined by how these jurors are going to be evaluating it um then it's actually an opportunity for me to make money. So here we saw that solidity was worth 11% around and um and if you think it's more or less you can buy solidity shares and it goes up or sell them and they go down.

Um so this is how the market evolved over time like we actually saw a lot of activity in terms of people trading it and making predictions and uploading their predictions and it goes up and down. So it was a really fun time for the model builders. If anyone of your if anyone of you want a cool AI challenge like this is like a great weekend project to build a model get values and start trading on it. Um so these are the results that we got from it. like we had a bunch of repos and we see the results um where on the left hand side what the results were the which the market told us the value was and on the right hand side we have the value which the jurors said was the value so that's how the market resolved so for example anyone who bought the first repo EIP you could buy it at 10 cents and at the end of the market you get 30 cents from it because that's what the average of the jura ratings was um so we ran this initial pred like prediction market and this was the profit and loss that we got for like 20 builders who traded in the market.

Um, so we like sent them a subsidy in order to trade in the market and uh after that we like a majority of them did make a profit. A few of them made a loss. I think $160 was the highest loss that someone made but um here we can see that someone actually made twice the amount of money putting in $1,800 and then finally getting $2,000 in profits. Um so and like how did we get all of these? And here this is where pawn comes in.

Dylan, you can put your hand up. That's Dylan. Um, so to like we also wanted to create a zerocost method for AI developers to test their predictions. And that's where we used pond where the model builders could submit their model and see what their error score was. How close are their predictions to the ground truth data.

So you have a public data set which we test it with and what's called a hold out set which we only reveal at the end of the competition and to resolve the market. So this is like the broad overview where you collect data from a committee. You run a market to predict the results of that committee and you also run an AI competition where they can upload their results and then once the committee results actually come out, people get profit and loss and you can see what the winning model is. So I guess they like the main question is what is the utility of like taking all of these extra steps like why should we even do these two extra steps of a contest and of um and of running a market on it. So like in the middle column you can see jura votes.

So this is like a typical structure like a committee where we had a big committee making judgments and then we average those values and then we know which project should get how much money. Um but but but but because we ran the other two processes the contest for AI builders and the market we can actually question the jury like like look at the repo called TEU where the juror average is 17% should go to them in the money but the market is saying it's only 9.27 and 7.62. So we went back to the jurors and said hey there are some like big differences with this particular repo like what do you think is the right answer?

And some of the jurors actually said yeah the average is actually wrong of a committee like the other two are more correct. So I think the biggest advantage is that if you as a funer have to give money you have to rely on a monopoly which is the committee and if you don't like their liberation you have to tell them to go back on the drawing table and come up with new results or abolish the committee. But here the neat structure is you can get three lists from like with with very little extra work where the market is trying to predict the results of the committee and the winning model is chosen based on the least error score to the committee results. So you end up getting three different lists to choose from as a funer rather than being stuck to sticking with the committee recommendations. So this is why anyone who's running a funding round highly recommend you engage the AI community.

We have a great community of about 30 AI builders who who take part in all of our competitions. So either join the so either join our community if you're interested in building and hacking or um you can uh run one of your own funding rounds or even like civil detection anything which requires some human judgment rather than thinking that this human judgment will become training data in the future. Ask AI to predict your uh to predict your judgment and then the AI can learn oh I predicted this and I was wrong. Okay, this is how I should correct myself. So like remove that mental model of my data today is going to be training data in the future and instead try to set up situations where AI is trying to predict your decisions and then when you actually make your decisions you can see how close the AI was.

I think that's a more accurate and and a better way of actually training uh AI and on your data. Um so um so yeah this is the last slide. Um I'll talk very briefly about like uh some of the help we need from you. Anyone who's an open-source evaluator who can actually evaluate repos. We want your help in uh you can go to deepfundingjury.

com and with an ENS login you can come make judgments between repos and these judgments are used uh in the market to resolve the market for AIS and determine profit and loss. It is also used for the competition to determine the winning model with the least error score and um and like or if if if you're an AI builder then you can just like build a model and you can upload it on deep.co.pm. We have some markets live already and pond is going to be starting the market as well soon.

So if you don't want to bet your actual money on it, you can take part on pond. You can get a good score and then we'll give you some trading subsidy to take part in the market as well. So these are the two concrete ways that we need your help right now. Either be an evaluator or be a model builder and submit the uh and submit your predictions of how important different open source repos are. This round has about 5,000 open source repos, way more than any human can evaluate.

So we have no option but to rely on AI to get weights on all of these different edges. Um yeah and like I'll talk about maybe briefly about like what is any of how many of you are builders over here like who like building. Yeah. Okay. So like a lot of builders.

So um I have two and a half minutes. So I'll quickly go through some of the things that we actually need built as infrastructure. So a lot of these futuri prediction markets are using a version of unis swaps range limited constant amm which actually isn't the best method because the ultimate price depends on how much liquidity you inject. If you put too much liquidity, the price doesn't change at all. And if you put too little, it swings wildly.

So that's why when we showed Vitalic this uh like this like our implementation, he said you you've used the wrong liquidity method and you should be using something called LS LMSR where uh it's only based on the trades. It's not based on the amount of liquidity that you put in. So but we haven't found a good implementation of this. So that's something we want to do. The other part is every time we do these uh contests, we require going back to the AI builders and telling them hey could you please run your model again and upload a CSV file again like for the new contest where we have new open source repos.

But if they could somehow just directly submit their agent and we can invoke their agent for every new competition that can actually create a moat for a business where the moat is like why should I run this with you and not with someone else. It's because we already have many AI agents that can be invoked. So we haven't found a good uh system where people can submit an AI and it gets executed all without the model builder having to do any work. Um and yeah the last part is like the eligibility. Vitalik again suggested a unisoft style eligibility mechanism but we haven't got that built out either.

Um so yeah this is b this is broadly the five areas we need help. Hope some of you found that cool and like like this is incredibly useful for like like for a lot of people. These are all the Ethereum consensus and execution clients and dividing between and dividing money between them is incredibly hard especially because everyone tends to do it equally and I don't believe in like equality. I believe in like if you contribute more you should get more. Um so yeah this is a way of actually doing it in a way that people don't fight with each other.

Similar to how I might not like who the president is, but I respect the process by which they got elected. So similarly, I don't like how much money I got. I always think I should get more. But I respect the process by which it was arrived at how much I should get paid. So that's broadly the area that I work in at EF and happy to talk anytime.

Yep. And you can message me here on X or Telegram.