Parameter Optimization and Emergent Behavior in DeFi

foreign [Music] this is a workshop will be talking about using agent-based models for analyzing D5 protocol and the examples will be we'll be using is through the Vega Market simulator we do a quick introduction and then I'm going to jump to a slide which will show you where to go on GitHub so that you can hack along I really assuming it's a workshop right so I'm expecting everyone to have a computer in front of them and start hacking but if not then hey you know you can just stay along for the chit chat so my name is David I've been with Vega for the last four years I mainly look after the kind of Financial Risk and economic design game theory analysis and together with me is Tom who is coming from traditional Finance he's a Quant with Vega help working on the working on the research team and one of the speakers Mark isn't isn't here but he's done a lot of the a lot of the RL stuff so he's the main brain behind that so I'm going to jump to I think here so yeah we'll go back to introducing what Vega is in a minute but given I don't know how fast the Wi-Fi is going to be in here um just running through a few firsts that set up steps that we can get going with while we're presenting uh for anyone who hasn't noticed there is Wi-Fi specifically for the workshops uh called Defcon workshop and the password is was it built build it 22. so give that a go if the Wi-Fi's being slow uh so this uh repo at the top here Vega Market Sim should be open if you go over there clone that that's your your first step you're also going to need go uh 19 I think it's necessary python ideally 310 39 might work and poetry just because it's probably the better uh package manager probably there's a requirements.txt if you just need that as well we can go through actually setting it up in a bit uh then if you want to be able to view our sort of front-end console you'll also need some some UI framework yarn and NVM for node management and if you want to try some RL at the end then Pi torch is a good starting point to download so broadly if everyone can make a note of that URL before we switch back and then we will start on actually presenting is everyone good cool we'll have an actual break for setting everything up in a bit but yeah so we're back to the introduction but it's it's sort of a workshop and if you need help the person who isn't speaking can actually come and help you with your you know whatever it is you're trying to do within this framework and if you're not having luck then we can see if one of us will have better luck and there are not so many of us and we've got I think two hours so we've got plenty of time to actually get everyone going and hacking so in terms of what we'll cover we'll quickly introduce Vega protocol then I will try to make sense for agent-based simulations in in the context of analyzing D5 protocols and then you know we will introduce what you already started setting up which is the Vega markets in which is relying on something called vegan all chain will um we'll explain what that is and then once we have that we will sort of explain how to use the market same how it's set up you know what what environment is there uh what agents are there and what you can do and at the end we're gonna show you how you can you know train reinforcement learning agents on top of this and why you might possibly want to do that so that's that's the plan so we start with what is Vega protocol so Vega protocol is a layer one blockchain for derivatives trading it's uh set up from the ground up for trading margins products and the price Discovery is either through limit order books or through auctions where do the auctions come in typically when you're setting up a new market you don't know what the price is so initially there it's running in an auction the auction determines what the opening price should be and then if everything goes well it's it's running as a limit order book if there are big price moves according to the risk model it might switch to a protective auction to make sure that the big price move is really you know what's happening economically it's not someone with a fat finger you know for getting a zero on an order and or adding a zero on an order and causing causing Mayhem uh anyone can create a market what it means is you submit a market proposal to the blockchain and then it goes through the governance and it's and it's voted on so that's what's meant by permissionless here and what kind of markets at the moment it's it's a supports cash settled futures there is roadmap for supporting any kind of derivative through a derivative payoff language which which will be probably basm based but that's that's just on the roadmap for now it's it's cash settled Futures and so basically if you can find an oracle for the market then you can then you can propose the market and it will run um it's got a bespoke liquidity provision mechanism because it's it's not based on now ever so popular constant product markets but it's based on a limit order book so you have to do something slightly different to bond liquidity to a limit order book so it does that and at the moment the the assets that have value that you would use a settlement asset on Vega they're bridged from ethereum though in the future that there will be other Bridges but uh ethereum bridge is there you can on Vega before proposing a market if you wanted to settle in some other asset that's not there yet you can propose the asset the asset will then be bridged from ethereum so you can imagine that there will be usdc tether ethereum whatever people want to do is that if you're designing something like a trading platform some of the goals that you might have in the design really sort of run at Cross purposes between a general purpose blockchain and uh you know what do you what you would ideally like for a trading platform so the things that I picked out are you know a general purpose blockchain you you need something like a gas mechanism for people to pay for the cost that whatever transaction the chain is processing for them you know the cost has to be covered by the gas and if you're trading this is really you know really sub-optimal in in the sense that there are transactions that you want to include and you don't want to penalize them by charging for them you know if someone wants to post a limit order that's good they're providing liquidity you really don't want to discourage them if someone cancels an order that's good they're providing price information you probably don't want to discourage that either so you know that's in products they're effectively promises right if someone doesn't look like they're going to keep their promise you want to close them out immediately ideally you don't want it farmed out to some Bots that are running somewhere because then you're opening yourself up to Mev and on a whole whole host of other problems so you can have Atomic closeouts again for margin you want to do potentially quiet Maps heavy risk of computations and you don't want to be paying for gas for those on a general purpose blockchain you can try to do Fair ordering or as Fair as your definition of fairness allows Fair ordering of orders and you know you you have one specific application in mind so you can optimize for latency for instance rather than optimizing for you know data throughput so you're not fighting at Cross purposes with other applications that would be running on top of a general purpose blockchain which would have other design goals in mind and one one thing that I didn't put on the slide but that's also a very good reason is that sharding is actually much easier in the context of trading because something happening on this market and something happening on that market can be completely separate apart from Talking together through a treasury chart so you have a very simple use case in the sense of what needs to know about what you can design charting relatively easily okay so that was that was Vega Now sort of more General case for doing just trying to get rid of the mouse more General case for doing agent-based simulations in in the context of D5 so you know we've all seen various D5 protocols and uh we've seen you know when things go well it's great and then when things don't go well it's called a hack or an exploit but quite often it's not an exploit of a code bug though that can happen it's it's simply an exploit of of the mechanism design you know you you go you want to design a protocol you start you you think what incentives should I create so that I've got the desired behaviors and you do it in isolation you really try to think it through right you you take the smartest people you can find there are you know dozens hundreds of incredibly smart people in defy they do their best they do what makes sense they think it through as much as they can but uh it's difficult it's really really difficult to think through all the consequences especially if you have one incentive mechanism here and other Hair Design maybe in isolation and now they're interacting right yeah oh vendor is a name of Claus's fairness pre-protocol for transaction ordering he can tell you all about that he's there the the you can just go and show him no yeah yeah so you you put two incentives mechanisms together if they were designed in isolation it's it's not you know completely clear that the combined effect is actually what you've desired and then of course D5 is all about composability they love that but once you introduce composability you're introducing another disproof stuff you set up a hypothesis which is testable you will never prove that it's true but you can set it up so that you run the experiments that should disprove it if you try their best and you've not disproved it then you'll then you let the hypothesis stand still they'll prove an otherwise and if you're wondering what this is you've probably most of you have probably seen Game of Life it's it's the idea that you know you you can take a bunch of very very simple rules and they can lead to very very complex behavior and sort of agent-based models are one thing that will allow you to analyze this complex Behavior so yeah uh definition of what an agent-based model is from from Wikipedia and everyone's favorite agent and quite often you want a lot of Agents or a lot of similar agents so uh agent Smith is good I've used this I've used this picture actually twice I I was when you talk about civil attacks I like to talk I like to use that picture but here it's not about civil attacks here they're good agents who are stressing your protocol so in terms of you know what type of Agents you you would find in agent-based modeling there are the the simplest ones are what I call zero intelligence which is a bit insulting but uh what it what it means is that they're not learning and they're not optimizing you just say this is the state of the world and if this is the state then they're gonna do this and if the state is something different they're going to do that so you know they're agents that have hard-coded actions and that's what they do they're the simplest to set up they're the simplest to understand and of course if you hard code the optimal actions then they're perfectly good if you don't then they're only as useful as the actions that you've that you've hard-coded and so the next level you you can think okay you know each agent I'm not going to hard code the actions a priori but I'm gonna tell the agent exactly what the environment what the rules of the environment that they're in are exactly what the rewards are for interacting with this environment according to the rules and I'm going to optimize in other words I'm going to solve for the optimal actions and that's something that comes under you know the name either mark of decision processes if you're doing discrete time steps or control theory stochastic control theory it's it's a reasonably well established field and you know there are a bunch of algorithms I could spend hours talking about algorithms that they have for solving problems there is there are a few problems that you can actually solve by hand one of the most popular is the linear quadratic regulator which you know um together with the Kalman filter was one of the great successes of of signs of the 60s you know Blended on the moon and all that so you know good stuff and then sort of the the fancy new stuff is is the reinforcement learning where you actually don't have to tell the agents what their environment is they just repeatedly interact with it observe the outcomes of their actions observe the rewards and update their behaviors so these are these are the type of Agents you might want to run you know on top of your environment to see what is happening and of course what is what is appropriate and what you need will very much depend on what you're trying to simulate what you're trying to test but basically this is your zoo and then the typical setup for for these things sort of you you understand you you have the agent agent takes action it changes the environment in the sense that it moves it from the current state to the next state so sort of new state comes out a reward comes out and the agent takes in the reward and the state and again chooses the next action and and you repeat and of course you can have you know these setups that you repeat sort of Forever Until Infinity maybe with some discounting or maybe you assume that there is some steady state or there will be a terminal time when the Game Stops or whatever problem finishes then you evaluate how everything then whether it will end well or badly and then one thing I wanted to point out here is because it will be relevant if you're looking at any of the code that we're providing is that the agent as it learning it sort of it's in a state it takes an action collects the reward sees the new state and decides on the new action and if you record this that that's sort of all the data that you need to train the agent sort of offline separate from the environment if you collect these sarsa sequences then you can train the agent separate from your from your environment okay so that's that's not so important now it will be more useful information later right and I think I'll hand over to Tom now who will evoke you through the you know the Vega Market Sim um it's a workshop so if you have questions I think the gentleman there already said yeah you're setting a perfect example keep asking questions oh do we need a microphone you think thank you hello hello um do state space approaches have any role to play in parameter optimization like a control control theory based approach for optimizing a ffected yeah so so you would set up you know your environment which in you know in in the case which we will show would be you know the whole Market how the trades happen how the orders are matched and then you run that whole thing through the simulation see what happens and then you want my want to do the optimization on top of that right but that's a agent-based approach right there's you could you could use a uh like a analytical solution so that's let's call it an agent-based approach a numerical approach but a control theory approach yes if you could if you could if you have an environment which is sort of sufficiently simple that you can really write down you know all that state transition equations then you can go and solve it through you know either pen and paper if you're supremely lucky or from through something like I mean you can policy iteration value iteration something like Matlab rather well that's actually numerical but you could use um software obviously to yes yeah and there is software out there yeah uh does this work for creating bots trading Bots yeah I mean like the whole framework you have here yeah so we're I mean Tom will talk through that so so I'll jump a bit but basically you know the environment on its own if it's just the market you know it's an empty Market it hasn't even gone out of the opening auction if no one proposes anything so so you automatically need bots so we will talk a bit about the Bots that are provided as part of the environment and of course you can write your own Bots uh that can do you know well whatever they want to do we've got uh we've got some basic Market making Bots we've got some basic you know price taking Bots we've got some basic um momentum Traders but Don will talk more about that so I'll hand over to him just a quick question so if this type of systems do you take the code from what you are testing for instance for the Vega protocol I just quickly check that it's it's all go code right okay so do you take this code and run it inside your simulation or do you create a model of that okay and do you think that that's that scalable if you want to if you want to simulate thousands of millions of Agents because perhaps you need you need that or do you think it it should be more in some cases you need a model of what you are actually testing the microphone so so we're running the whole Vega stack again Tom will show that but you're absolutely right and in some situations especially we want to do reinforcement learning having a simplified model on which you can train before you actually go on to the full stack which will be inevitably slower than you would like because things are never as fast as you want would be useful but we don't have that at the moment we're learning on the whole thing that's why it's important that we use an algorithm which collects the sarsa sequences and so then I can learn offline but we don't have model there are reinforcement learning techniques which sort of build up model from observations and you can do a lot of clever stuff but but I'm far from being an expert on that okay thank you so so one thing that I didn't say is one reason why D5 is so good for agent-based modeling is you know you can try and people have been talking about agent-based modeling in the in the real economy and economists would love to do that but they don't have the environment on which to execute it and in D5 we do have that because things are typically open source you know I can go and take the smart contract and run things through the smart contract I don't have to create the environment I just say okay I care about how this smart contract interacts with this one I take whatever ganache hard head I put it there I set it up as my environment and I pump thousands and thousands of simulations through that I can't do it in the real economy but here we can so it's awesome okay more questions [Music] okay are you really looking at things like protocol solvency or profitability of certain agents that are performing trades or kind of what are the reward metrics so I'm assuming they're not like bugs but more financial kind of outcomes so so for us it's very problem problem specific so for example if we want to test you know what some of the network parameters should be we set up an environment and and then we look at things like you know if if I've got a rationally behaving Market maker and the parameter is X what is their return on Capital and I'm thinking well you know maybe this is letting them earn too much money it doesn't make sense but in terms of the reinforcement learning guys you you would typically look at Financial rewards and then yeah you can either run it thinking of it you know maybe you're thinking they're trading perpetuals so it's just forever or you're thinking that they're trading a settlement market and so the reward comes at settlement but it's it's very much something that you have to design for the specific purpose that you have in mind this is sort of more the toolbox and I'm gonna you know wrap up the questions hand over to Tom and then you can ask afterwards because I have a field at with the questions we're sort of jumping ahead a bit so I I want to hand over thank you okay so yeah discussing the market simulator will get to doing so ourselves in a minute but first I think it's useful to go through and yeah to cover again slightly more technically some of the some of the questions how we are how we run these simulations um and how how we get interesting results out so the vegan market simulator broadly the way that Vega itself is set up we run with a tenement consensus layer for those who don't know tenement is it's a cosmos based consensus the problem with that is that it's generally tied to our block time which is about a second so if you run the whole thing you can watch the market going second by second and you can interact with it but you can't run as the question in the back said thousands of scenarios run it through a year do all these things so what we do we strip that out we run what we call a null chain entirely on on one PC and this just broadly accepts any transactions it's sent it will it still checks they're valid it still runs through the whole Vega core logic but um just adds into a block allows you to control time forwarding so you can it'll just sit and do nothing until you tell it to run forwards or fill up a block and so you can inspect at any point in time for however long you want what's going on then on top of this once we've got that running we have in Python an API layer which lets you express your trading actions your agents behaviors in much more Market Primitives without having to worry about the fact that you're running on a blockchain occasionally it still Creeps in because that's the way well but you can broadly talk about an agent trading on a market without having to worry about what Block it's in whether it's been confirmed and this kind of thing once we've got that we then build out our scenarios and our tests so I'll cover possibly next slide exactly what a scenario is but we can with a range of composable Agents composable setups build out scenarios to investigate and test either certain parameters test that the market doesn't explode test this the core itself doesn't explode or if you just wanted to build an agent you can try and have a realistic market and throw yours in there how this all works is your python layer manages we've got two different kinds of interfaces at the moment a Vega service null which spins up this as I mentioned a service entirely on your machine or what we've recently created which is Vegas Network and that allows you to connect to a running external network and run the same agents that you've just been running locally but on the real Network that's going on so that can be useful in some cases too it allows us for example on our test net we can run agents and simulations that other real people can interact with and see how that's working um we'll demo all that in a bit um so yeah back to setting things up and viewing I'll go through the viewing in a second but yeah yeah I was gonna gonna break here anyway so yeah so um we're back here I don't know how many people have managed to get to the end of this but I think we'll probably take a sort of 10 minute break to catch up anyone who has them we can wander around and see whether everyone will manage and then 10 15 minutes we can come back and continue with doing something more interesting with it if you think you've got it set up try and run these uh scripts at the bottom we can also tell you something more interesting to run if if you're stuck um I think I had a question yes so I can then more uh in a more real way basically test my uh my ideas it's actually it's something we're pretty much currently working on yeah is being able to better snapshot an existing running chain and be able to totally restore from that weird there is there's the capability currently to you can run your simulation and save that and rerun the same thing so you can have a an example that you run and then run your things after we don't currently have a way to pull from an existing Network okay thank you very much that people should be using in their protocol and I tend to tell them that they can more easily solve for tokenomics problems by including a market mechanism to optimize and let you know the market do those kinds of things like example uniswap V3 solves the hard problem of how to set the curve by allowing concentrated liquidity so my question is do you guys have an intuition for which kind of simulation like which kind of systems require a simulation and which kind don't oh good question I I guess I'd go back to as David was saying earlier it depends on how well you can if you can solve it mathematically that's probably always going to be optimal but when it reaches a certain point of complexity um yeah I don't know how you would do it without either an agent-based simulation or casting modeling or that kind of thing is there an upper limit to the kind of complexity that you can model in this because I'd imagine like in this example of using a market mechanism that adds a lot of additional complexity is that like would that be an issue in the Vega simulation in terms of complexity well for the simulation yeah I mean it multiplies the number of agents in the in the system um by like orders of magnitude if you have to use a market to um Source lots more participants ladies I guess is there is there an upper limit to the kind of complex system that the Vega Market simulation can handle uh well practically I guess there is enough limit we are the the system itself we're hoping we'll be able to scale to many thousands of people so that should also scale to to the number of Agents so I guess it's it's a a goal of both the Vega protocol itself and the market Sim thankfully have similar things because ultimately what we want to do is build a realistic market so if you can build as long as we can run a realistic Market on Vega protocol itself which should be able to do in the simulation cool thank you guys something they're exactly going back to the going back to the points that we were making I absolutely agree that if you uh if you're designing tokenomics or protocol novel then either you can prove that it works or you can use this agent-based simulation to sort of not prove that it works but show that it doesn't and go back to the drawing board yeah so in in terms of training though there are two questions I I can have the environment sort of I can have some agents that are better you know the agents take their actions quickly and get on with it but of course the moment you have more than one optimizing and learning agent then it's the whole you know Game Theory and and you know that they're probably and and then it's also very difficult to set up or very difficult you have to think carefully about how you set it up you know because is the is your is Agent a also trying to learn the optimal response of Agent B so that it can anticipate Etc and then you know they can end up chasing themselves in circles and all sorts of weird and wonderful things can happen you know it's uh game game theory and NRL comes in and uh it's it's very problem specific but you can you can end up with situations that they don't learn where you can end up in situations then they take a lot longer to learn because there is more than one learning can I have a show of hands of the people who are actually trying to set it up on their computer okay okay okay so so we're we're sort of now is the time to take the break and we're going to have a have a look around Okay so I think a few people have it running at least uh we're gonna I'm gonna crack on and then we'll have another break in a minute for doing the next bit so we can keep going around for anyone who's who's stuck on this bit but I want to give more interesting things to do on top of that so a quick aside once once the the null chain is up and running the micro SIM is up and running you see anyone who got it working would have seen something like on the right here which it is it's a printed output log of some stuff that's going on inside markets him exposes various apis for this but often you don't want to just have to manually call python apis for every single thing that you might want to experience so we've got a couple of useful portals portals on your data you can if you've set up the UI components as well change a flag uh this run with console equals true in generally wherever you happen to be running your Vega uh null chain that will bring up in a web browser the nice console that you see at the front there that is the same console that people use on Vega protocol itself so that has theoretically pretty much everything you need you can even trade manually through that as well if you want you can log in as your agents that you've set up uh we'll probably demo it a bit in a bit you also can is actually always launched if you take the logs there will be this graphql port and if you go there and you know graphql you can query the entire system with a fairly hopefully clear graphql queries there's on um what is it docs.vega dot XYZ there's a link in a bit the entire sort of structure of your graphql queries you want as well so for anyone who has good at running for more interesting scenario that's something you can try out running I'll leave it up um actually let me skip to the next thing this is what I'm going to talk through with this app so anyone can type it up so the components that Vega itself runs on a on a validated node that we're also going to be running locally that the null service spins up there are a few different ones we've got Vega which is the core process that's the um go blockchain client that does all of your transaction processing and outputs your States that is obviously it's a point in time proceeding thing so we also want the data node which is listens to external events output by the note core node itself and saves those into a a postgres database and allows you to query all of that historic data and that's generally what you'll be using when you're querying any data from Vega storage layer additionally there's a couple of extra things there's a console as I mentioned previously that isn't necessary but it's very useful front end there's also the Vega wallet the wallet is basically size transactions and ensures that any sort of custom signing we have some proof of work to stop spam attacks and stuff and that handles all that it's optional here because for the null chain what we actually do is Skip all of our validation so that it will blindly accept if you send a transaction and say it's from whoever it believes it's from whoever so you don't necessarily need that what it will spin up is if you spin up a console the two interact and a lot together so we use a full wallet for that so once we've got our our Market Sim set up how do we actually run our scenarios there set up as ideally Plug and Play components the overall thing we set up is we call a scenario that consists of an environment which is your background environment and your Vegas setup so that will include things like how many steps you want to run your simulation for uh how your Vega is set up and sort of what kind of logging you want that kind of background detail we then plug in the various agents some of whom will be setting prices they could be acting randomly and each of them will have something they do you'll see when you build one yourself in a bit something to do with startup often they'll force it themselves some tokens that kind of thing and then they've just got a function and each step call that function and it does whatever it wants to do it pulls in data from the market it trades they do whatever so scenario environment you'll see that when we go in a minute to the agents themselves so yeah what's an agent it's it's fairly simple ultimately it's an initialized step where they generally force it sometimes they will um do things so we'll have an agent that sets up all the assets because you start with a blank blockchain you don't have any assets any markets that kind of thing so there's you may see if you look through a market manager who does all this setup for you then they'll have a function that's called every scenario step however many steps you do generally will sort of randomly shuffle them so that you're not always getting the first agent called First in any step and then a finalized method that most mostly you can ignore but uh often you'll want one to sort of settle the market and things so that's how you you want to set up your agent there's one more thing I mentioned earlier that sometimes the blockchain does have to bleed back in you'll see scattered throughout occasionally you'll wait for total catch up function basically what this ensures is that you're getting a good view of the market currently you haven't um thrown way too many uh transactions at it everything's in sync that kind of thing so it just ensures that your more real world modeling rather than trying to ensure that you can get everything in one nanosecond through so um I'm gonna now talk through building one of these agents a really simple one it's just gonna do a bit of training I guess and then we'll break again and hopefully you can put together your own a bit where you're thank you Simmons Okay so for anyone who has it set up we've got this run simple agent in exam reinforcement run symbol agent I'll put up on the screen after this to pass to these files and then in agents we have our simple agent so what run simple agent does is it sets up a scenario um I can discuss the actual arguments with people but which we call curve Market maker and that a H jump into marketing basically that contains an agent who ensures that there's a nice shape of the market and a few who just sort of randomly buy and sell so they're very dumb theoretically you should be able to agent which is what hopefully we're going to build that runs this simple agent that we've got together here so what I've put together is a few nice template functions that is without having to worry too much about any of this works you should be able to cut and paste these into our step function which is down here we've got the simple agent and you can see it's got the structure as described earlier we've got our initialize where it's finding the market finding the asset and minting itself an asset and then say submit the market order from that check her positions is it almost gonna buy it is currently always going to buy so 50 50 chance of making money Okay so uh they're down to the relay just the hardcore Workshop participants you know you're the ones who prevailed with us despite Python and its foibles and the slow internet so I'm going to crack on if you're if you're coding keep coding uh you know if you have questions there will be time to ask questions and what we went to what we want to get to now is just to show you a little bit more of you know what the agents that that you have there you know the market maker Etc that you're interacting with a little bit the ideas that they're based on and then move on to the reinforcement learning part so you know as part of the environment as Tom was saying if you just set it up there is nothing it's it's the empty Vega blockchain as it would as it would be when it launches live you know no assets no markets everything has to be created through governance and you know once the asset is created once the market is voted through which you know the python sort of expedites for you you will still have just an empty market right no liquidity on it nothing no activity so effectively the code provides various versions of of market makers that are based on some stochastic control problems that are pre-solved coded up and basically running and then some sort of hacks where you know the optimal solution of the stochastic control problem is used as a heuristic and then you put something on top of that and you know there are you can you can have liquidity Trader you can have liquidity takers it just means they Place random Market buy and sell orders you can have informed Traders what does it mean well you just tell them what the price will be at the next step right you know maybe they they have inside information or whatever and um there are the momentum Traders through a library so you you can uh you can trade on on all the patterns that these chartist people have you know there are these vomiting camels and I don't know what else you know you you can code up agents that trade on that