Using The Graph and Dune to Power on Chain Actions | Devcon Bogotá

foreign [Music] the workshop that we're doing is using the graph in Dune to power on chain actions my name is Christian this is Matthias and we're from component you can find us at component.fi and we'll quickly go through the agenda so you can get an idea of what we're doing here um so first things first I just wanted to mention that this will not be a code along we wanted it to be but some of the dependencies required uh you need pretty good Wi-Fi to download them so you'd have to run to the basement come back up uh we don't want that so instead we'll just discuss uh some of the things that you would have uh that you would have coded so one we want to get to know each other I think that's uh you know why a lot of us come here is to find like like-minded individuals uh two we're gonna do an overview of the web3 data stack um so give you an idea of like the different layers of the web3 data stack and how you can use them three um we're gonna do essentially an example that goes through uh that you would have coded along but instead we'll just kind of show you how we did it uh and can give you the information so you can do it yourself later um and that'll be data fetching so the data fetching process data processing and modeling so what you do with that data after you fetch it and then transaction execution so essentially a way to grab data process it model it and then have that do an on-chain action a transaction for you and then at the end we'll do breakout sessions just at your tables or whoever you're standing next to um instead of the instead of the coding element we'll just essentially design your own automation speaking with your table but we'll walk you through it we'll do some engineering instead of coding cool so uh this is actually uh my email um I don't know if you're familiar with Quest trade that's a Canadian uh company for uh for trading stocks and using derivatives I got margin called a lot uh as you can see um it wasn't that much money it's okay but um I'm not a good Trader basically um and that's kind of my motivation for for what we're doing um because I got margin called so much I know I'm a bad Trader you know maybe automating things and having uh you know logic do it for me is probably a better idea um so that's specific to D5 but of course there's many other ways that you can uh take what we're going to show you and use it you know across anything that's built on the blockchain okay so we want to start with everyone just kind of talking to each other at their tables um you know you could get to know each other maybe introduce each other but here's a prompt based on what I just showed you uh you know from myself uh and that's a time when things didn't really go your way in web 3 so that could be you know you lost money trading derivatives uh you know not in web 3 you can talk about that too a time when maybe you suffered a lot of impermanent loss or you can you can say good things too I mean um or a time when you should have sold the nft at the top you know that kind of thing um so yeah take five minutes and chat with uh the folks around you so we're gonna give a little overview of the web 3 data stack uh there's a lot here and uh we're not going to have everything um so absolutely if you're a you know a data protocol please reach out maybe we can add it to our slides um so we'll start with this nice little visual from Andrew Hong on Twitter um this was made about eight months ago uh and so here's a breakdown of a lot of the different sort of layers of of the web3 data stack you know mostly mostly relating to to ethereum and evm chains so obviously you have your your sort of base layer uh where you're actually running clients uh interacting with uh you know the ethereum blockchain state directly reading logs uh call data Etc uh you have nodes as a service you don't do that yourself but you're still operating on those sort of raw data streams and you have a layer on top of that you can think of that as you know the query and data mapping layer there's a little more structure there often these things are indexed um you know they're they're um potentially you know doing analytics takes uh it takes your abis and does some contract decoding so you can actually read the the parameters and the function names instead of looking at a bunch of hex codes um there's flip side and the graph and everybody knows those um there's uh then there's sort of that layer on top and this obviously encompasses a lot of things but they're higher touch more much more customized apis um the the data might be a lot more contextualized and a lot more useful for specific applications but potentially a little more obtuse um some apis you know are that data processing is happening in the background and you don't necessarily know um so things those are things like you know covalent etherscan some you know manual tagging systems like through nanson uh and then we have a few that we use pretty regularly so we wanted to make sure they made it on there um so and a lot has changed in the last eight months eight months so people know crypto Stacks uh David mihai um they run like crypto fees and L2 fees and all of those different front ends they built a little you know data processing um stack to make it easier to build front ends with uh with subgraph data uh the Dune is releasing their API we mentioned the Dune but API changes things pretty significantly uh we use the massari standardized D5 subgraphs uh pretty much on a daily basis um they've created you know an open standard for different kinds of uh of uh D5 Primitives across chains uh if you want a little more control ethl and true blocks are are pretty great they're both you know open tools allowing you to index your own index from your own node and even the ectl data is available in uh in a bigquery database you don't actually have to do your own uh your own indexing if you don't want to uh D5 llama uh you know they pump out a ton of new apis and new protocols uh their adapters are a pretty cool system and then we have another one spec um they're uh they do allow you to sync your own postgres database uh with uh you know typescript Transformers uh bringing in call data or sorry bringing in um uh you know transaction data uh indexing it and then bringing it into your own postgres database to live beside your own data cool so this is the process as I mentioned that we're going to be going through um so first thing we're going to do is we're going to do data fetching and for each of these sections we're going to talk about um the alternatives to what we're doing so to give you like a little bit of flavor um so you're not stuck with with with uh one thing uh and uh then we'll give we'll run through the example that we actually did um but yeah just an overview of data fetching processing modeling transaction execution so our example is using python to collect data and write a simple prediction model um so yeah the three steps are pulling each price data from the graph in Dune uh just to have kind of two data sets to to compare and to show you you know two different ways that you can do it and then using a auto regressive moving average uh model to predict price movement um I wouldn't use that uh Approach at home uh but you know for the for this presentation it works um basically if price go up uh swap usdc to eth on unit swap that's the transaction that will automatically happen obviously this was built originally as a code along so we tried to keep it as simple as possible uh a lot of this can be expanded on pretty pretty thoroughly definition so obviously we went over that data stack and there's there's so much there um so many different choices to be made and like any other choice you're making when building a product it's mostly about trade-offs uh engineering choices so we we wanted to talk about some considerations to keep in mind uh there are definitely others but some big ones are available Fields so they actually have the information that you need to use uh the data format are going to have to Wrangle all this data to to get it have it usable in your automations um scope does it cover the chains and protocols that you want to use are you going to have to you know merge different data streams in order to get information from Cross chain cross cross protocol um is it event driven or is it pulled uh you know flexibility can you extend the model uh stability are they going to change their interface on you while you're running your automations and break everything because for some reason we still don't Version Control apis uh everywhere latency if your your application requires is real time you can't be operating an hour behind blocks uh trust reliability and redundancy open openness of the actual transformation functions so you know understanding how that data was uh was transformed from you know raw inputs to the data you're looking at now and then openness of the data itself uh you know if you want to be able to inspect the internal State can you just get a dump of the database and check it out um those are all some pretty important things so we are in our example we are just fetching each USD price data from both graph and the and dune um of course this could be done through most of the apis that we mentioned um what we like working with both tools and so this was just a nice little example of using those it'll Doon first yeah we'll start with Dune so very simple query here um we're actually just using a dune created or user created table which gives you prices of different uh erc20s on layer one ethereum and here you can see we're basically just pulling in um the the time um which we're changing to hour because we want hourly hourly price and then we're doing the average of the price of eth um over that hour and then we're doing the max price and the Min price during that hour uh for the last 30 days and you'll see this query as a query ID here that's going to be relevant in a second cool so yeah as we said we don't have access to the uh Dune API right now um if someone from Dune is here please uh let us know we'd love to use it um but uh so instead we're using this this uh open source SDK that grabs essentially the the data from a query uh it's limiting in the sense that you can't actually run the query using this so you basically you know every time you want to do this you'd have to go back to do and run the query and then come back here and it'll update after you run this function um but yeah basically it's just grabbing kind of that that static data from the query the last time it was run then uh yeah if you just scroll up a little bit to data data one two three yeah uh just quickly mention this it's not that significant but we're basically just changing the uh Json format that we get in the response and we're turning it into a panda's data frame uh are people here familiar with pandas data frame a lot of nods okay cool rows and columns cool um and then yeah yeah there's a little bit of processing here to get you know the data wrangled so it's consistent across both Dune and the graph uh this one is you know the the graph data pull um so we're specifying the endpoint uh we're just using the uh chain link prices a sub graph here um in fact this data isn't actually hourly data um so we first you know we were using graphql to query the graph uh we're just looking for the eth USD asset pair we're ordering by Tam timestamp so we get the newest data uh we're ordering descending and we're grabbing the first thousand prices as well as the timestamps associated with that uh then we do the exact same thing here except we do one thing which is resampling so that we only end up with you know one timestamp every hour or one price every hour instead of instead of multiple okay and uh here's just an example of the outputs you'll notice that there's actually inconsistency between price and price here um that's because uh the Dune table actually does this in decimal format while you know the graph uh here this subgraph specifically uh looks like uh yeah eight decimal points um so uh the the fact is we're running this model separately so it doesn't really matter for this example but it's just one of those things to pay attention to obviously your data needs to be consistent yeah and one thing I'll I'll mention about that as well is um it's actually great to use more than one data source um just so you can actually check these things uh you know it's hard to know what the uh the source of Truth is so you know using different data sources check them against each other you know maybe before you run something is always a good idea and another thing is uh we didn't really mention this but um with the data we're pulling you know you know this is a simple example but the really cool thing is that you know anything on chain is fair game in terms of data that you can use so think you know lping lending protocols uh you know on-chain derivatives protocols nfts you know doesn't have to be defy related um you know you can do some pretty sophisticated stuff uh with different types of data so yeah moving into this section now so data processing and modeling um you know now we're at the point where we have uh the the Dune query and the graph query uh pulled in uh using Python and now we want to uh model it essentially so before we jump uh in back into the code um there's just a few considerations uh kind of like just what I said with like there's so much composability when it comes to data there's also infinite composability when it comes to modeling um you know you could use instead of using a Time series prediction you could do conditional logic like um you know maybe there's someone you follow that trades really well and anytime that they make a trade you want to mimic it uh right away you know stuff like that or if you know someone like me is not a good Trader you can take the opposite position um then of course there's there's so many types of machine learning algorithms time series predictions uh the list goes on and on for the types you can do and again I know we're focused on on D5 for that presentation this presentation because that's mostly what we interact with but you know if you're really into stable diffusion and you want to see text generated images uh you could always grab you know uh text from your lens protocol interactions and generate an image based on it and mint and nft on chain if you wanted to something like that you know that's really up to you because all this computation is done off chain the uh the sky is the limit foreign cool so uh quickly just on the uh I guess the modeling uh we didn't we already did the processing I guess earlier when we uh just reformatted stuff into into a panda's data frame um so here's it's it's quite simple um if you're familiar with like a little bit of data science I'm not a data scientist myself um but what we do is we basically split uh the data that we pulled in so the Dune data and the graph data and we split it into a train and test set um just basically 80 trained 20 tests not that sophisticated um and then what we plot it and we'll show um you know we'll show the results after and then we run it through a Arma model so uh Auto regressive moving average model to essentially predict um so we're doing this by hour and the goal here is to predict um whether the uh the price of eth in USD is going to go up in the next hour and so the logic here is basically just uh if the next hour price is greater than the last price then we want want to buy you if it's less then we want to sell or short or what have you it's that easy folks not Financial advice cool yeah these are the results so as you can see the so the red here is the test set um and we are basically you know the prediction is basically saying it's trading sideways uh maybe I think it probably goes down about 10 uh ten dollars roughly uh if anyone knows uh the crab strategy on and squeeze this would be a good time to use it I guess um yeah these are just the results yeah so the most web three relevant probably the most complicated um Step uh in automation here is transaction execution so there's a number of ways to do it so bringing these insights back on chain um so there's there's tons of considerations uh to think about uh which chains you're operating on is the environment adversarial so are you you know if you're working in D5 then you're probably looking out for Mev and sandwich attacks anybody trying to exploit your model um if you're you know minting nfts for art purposes and not for financial purposes it's probably less of a big deal that someone front runs you uh what kind of timeline are you operating on and you know some things have much larger delays than others a key management do you want to manage your own Keys what kind of permissions do those keys have should your logic be transparent so you know if you're running a proprietary training algorithm um you probably want that to be executed off chain and just have the results posted to the chain but if you're running something that you want to be transparent uh you you would likely want the the you know computation to be visible on chain uh you know what happened of a transaction fails if it's or it's congested these are things that are annoying for manual transactions but if you're not taking them in account when you're automating transactions you can get something stuck in the mempool forever you can potentially end up not being able to to transact if you're not you know replacing that transaction and replacing that nonce um so there's there's a bunch of options um there are almost certainly others as well and these are not necessarily mutually exclusive they're just kind of ideas of part of it so you know libraries that wrap on top of uh you know the wallet apis that translate Json RPC ethers web3js web3pi which we're using in this example uh if you're running something like that it's probably in some sort of function that you can either be running you know locally or on a cloud um you can you know use traditional cloud service providers or you know tenderly has their new web three actions which is effectively lambdas for for for web 3 interactions to be triggered by events um keeper networks uh so you can some keep your networks support like more complex off-chain logic um some only support uh triggering you know transactions with a function signature um uh there are transaction execution Services um so something like uh open Zeppelin uh Defender relay and uh inferior transaction Services they handle a lot of these questions that are kind of annoying for for a small development team to deal with so things like Key Management they allow you to hook into a KMS system um they also handle you know gas uh figuring out gas fees so if a transaction fails and the first or a transaction is is um waiting too long in the mempool because it doesn't have a high enough gas price assigned then uh it will actually ramp those up aggressively and it tries to figure out you know uh how much gas do we need to be able to execute this transaction in a certain amount of time that being the variable you set uh you can you need to make the decision you know if you're not actually doing uh execution and you're just posting data to the chain then you might want to actually more like an oracle model so just posting data you're not necessarily thinking about execution then you're more thinking about launching execution that is you're more thinking about how do I get a reliable uh like reliable posting of results to the chain um you can go through a multi-sig if you don't trust your automation you could have them as one of the signers of a you know one of two multisig where they you know basically are queuing up that transaction send yourself a notification and then accept or reject that might be the most thing that most people are more comfortable with you could potentially run things run the actual uh run some of this Logic on an L2 um so what's the name of the uh yeah uh there's a company called uh modulus Labs uh they're doing some pretty cool stuff uh so they created this thing called Rocky bot um that basically does what Matthias was saying uh where uh they're running uh it's on it's on start net uh and they're leveraging that and just basically uh posting the result to the chain uh so it's like a neural network on chain it's it's it's it's very cool um definitely check it out yeah and then I'm not very familiar with Mev or you know like uh being a Searcher so I'm sure there's other people who can expand on this but in that case you're probably using something more dedicated to your building bundles using the the Builder API I believe uh and for this example we're just using uh you know web3 dot Pi um this is probably the thing that people are actually the most familiar with uh were fully we're just you know setting some uh some variables we're selecting our chain being girly we're grabbing an RPC provider uh there's some middleware that's required for for Gurley here we're grabbing a private key from environment variables which of course means we have to manage that ourselves and then we are you know executing transaction in this case we are swapping on universal v2's router um so we're grabbing the ABI uh using that to to encode our function signature and uh you know setting some parameters and and posting that to the chain cool so before we go into this um just wanted three caps so we used a subgraph to pull um typical ethusd price data we use Doom to pull eth USD price data and we processed it we using python pandas and then we used web3 dot pi to actually send the transaction um it's actually what we were supposed to show just put putting it all together yeah it's again you know three steps here really oh grab graph and dune um you know formatted data uh check with the model whether we should buy graph buy based on the graph model whether we should buy based on the Dune model and you know if both are true uh we are buying yeah and one more thing to say here is um so this makes it look easy um and that's actually what we're working on at component so working on something called blueprint that abstracts away all the stuff we just had to do um although it's really fun to learn how it's done and you can definitely do it that way um and you know to see how it's done under the hood but what we do with that product is essentially uh try and take you to this step without having to do you know all the other stuff okay I guess it's me so uh you guys saw what we came up with uh now it's your turn um this might be a little long with the 10 minutes but um I'm sure everyone who came here has something in webview that they're passionate about uh that thing probably is generating quite a bit of data um and that's probably available through one of the many services we we mentioned um so we'd like to take a little bit of time uh you know explore some data sets um think about what kind of automation you could build using those as you know your input you can work as individuals or as a group to start uh after that we're going to sort of have a little 30 second pitches uh to everybody else in your on your table and you'll decide on one to sort of design as a group yeah so just to you know give you an idea of what you could do so anything in defy um where you know you're triggering you know based on a prediction model or based on some conditional logic it could be to sweep the floor of an nft collection um you know the really anything you want and any data that comes to mind that's particular to you and what you're doing thinking about those three steps where I'm getting the data from how I'm going to process it and then how I'm going to actually generate something actionable from it yeah go ahead all right folks uh we're happy if a few people came up with an idea and like thought about the process and I think if you did anyone come up with an idea raise your hand if you did you wanna you wanna talk about it so um we think that kind of defy investing has been quite risky forever and so the idea is kind of a generalized pressure release valve so if a condition is met whatever investment you have if it crosses over that threshold and pull pull out you know pull out to wallet essentially so you can think of a liquidity pool you're invested in there but if the two assets start deviating Beyond a certain range and you you exit that position thank you Greg anyone else you look like you have an idea okay Quick Pitch here let's hear it um I'll try to make it very quick yeah what I've learned throughout a lot of uh oh worry Parable loss experiences is that you have to optimize your buys uh that's a strategy for centralized Finance I mean centralized trading but I it it does work for defy and helping and it's basically spotting when the price hits it hits the threshold and it when when it's oversold within the Bollinger Bands like I mean like in the in the lower part of the Bollinger Bands and you you want to do that you want to identify that buy signal throughout uh different uh temporarily these temporality um like the weekly time frame the daily time frame and maybe not no shorter term like the time frames maybe the three day time frame I don't know but they are identifying buy signals throughout different temporalities using the build mirror pens those the that's that's basically oh it's working yeah you look like you have an idea no okay you have an idea okay um so we discussed an idea of Auto rebalancing crypto index so let's say you want to build a simple index that's like 50 wrapped VTC 50 wrapped if over time that balance is gonna be different 50 50 if the price of Bitcoin increases for example so maybe using some some kind of feed like that to calculate the balance of your portfolio and rebalance every hour week whatever cool yeah rebalancing I think is a is a very good use case like in whether it's you know just a simple spot trading portfolio or it's you know something a little more complicated like lping or Landing or borrowing that kind of stuff um do we have time for one more maybe we have a little time okay it's okay anyone else cool so um like um none of us were actually in device space but we actually uh thought about some other use cases um so um I'm working on um a reputation on computation protocol um based upon Paradise trust like a whole trust to him by how much and then I'm like in my output would be a bit basically the number crunching would be um like after all that because of the calculation like um each peer in my network is um assigned a trust score right and like he's actually from the dial space and we talked about how they can actually use this um this type of algorithm in like a voting uh versus like a simple prevention typically um my algorithm is pretty good at like I'm identifying the Civil clusters who actually trust each other but like none of the other real people actually trust any of them so maybe we can actually ingest this data and then um do that number crunching identify those civil culture and then um his Dao could actually um emit something onto the contract so I sort of like those um um accounts as potential maybe like avoid them you know things like that cool thank you maybe one more if anyone has one if not it's okay anyone Keen to share Adrian you you look like you want to share you're writing your idea right on your phone okay later okay um cool yeah that's it um so we'll just wrap things up here uh yeah we don't have time for that I don't think so we'll just go to the last last slide I guess yeah if you want to run with any of these just remember to think about like we have the basic idea right now but there's a lot more to it I think adversarially where are the attack vectors um how could your strategy be be manipulated um you know uh be defensive yeah and then just to close things out um to mention what we're doing again uh so at component we're building something called blueprint and all of your ideas except for maybe the last one that sounded really cool but it was a little over my head um but very very cool um but for the defy related stuff at least and would love to talk to you about what you're doing to see if we can incorporate that in the stack so if you have any ideas that you actually want to execute on or you have you know friends that like want to build these kind of data driven strategies um we're building tools that basically make the entire process easy so you only have to worry about the logic of what you're building um so yeah that wraps it up thank you everyone foreign thanks a lot I know it was an unconventional Workshop but I appreciate that everybody was talkative and you know engaging instead of just hanging out on their phones and things like that so it's great