Notable security incidents since Devcon V | Devcon Bogotá

foreign [Music] we have with us Tom we have Aaron and there's one more already stuck in traffic you may have experienced this as well this week and I'd like to give the word to Tom to kick off the first talk about the issues since Defcon 5. hi I'm Ryan Lackey I've been a cyberpunk since the early 1990s uh started the world's first offshore data Haven company in havenco on Sealand in the North Sea uh did defense Contracting building satellite and cellular networks in Conflict zones for a while and then I started as CSO at a bunch of crypto projects I'm currently CSO a chief security officer at evertas Insurance a crypto asset insurance company where I see a bunch of interesting risk events happen thanks Ryan uh we have an hour wow I mean there have been a lot of security incidents so I guess we have plenty to talk about but I just realized we have a full hour uh hi guys I'm Lane very excited to be here um I have been kind of steeped in hacker culture and hacker ethos since a pretty young age uh kind of High School 14 15 years old um specifically kind of penetration testing white hat hacking ethical hacking this kind of stuff um I sort of kind of fast forwarding to the present I was an ethereum core developer with the ethereum foundation for a couple of years and for the past three years have been working on a project which is a new layer one blockchain called space mesh based on a novel consensus mechanism so I currently lead r d at that project and just generally kind of follow the security ecosystem very closely contributes to you know kind of security response here and there and just think deeply about it in the context of my work at space mesh nice to meet you all my name is Arun uh I've been a full stack developer at the bug Bounty Hunter for a few years and just generally like a very active D5 DJ I was previously lead developer at a web 3 security uh like related startup called anti-dot finance and I'm currently head of research at Marana Ventures and before crypto uh I was a scientist in systems biology where I did research in high throughput protein sequencing and Tom Howard who's currently stuck in traffic as the founder of uh D5 payments wallet monsendo uh crypto options exchange power trade he's an unreformed D5 degenerate angel investor and he's now got a newly formed seat fund Network zero uh he's working on building tools and infrastructure to support the network State concept that Bellagio developed as sort of a successor to the nation state so there's a whole bunch of security incidents that have happened in the last three years the world was very different in 2019 and I'm sure we've all we can all remember how things have changed uh I think there's some some broad categories that we're going to discuss of things that have happened broken down by category and then hopefully for 10 or 20 minutes toward the end our views of how things are going to get better because the current state is terrifying like I I wonder how people who have very large fractions of their net worth in these in projects locked on the on chain sleep sadly every night I certainly don't and it's probably something that uh that that is not sustainable in its current level of vulnerability over time um so I think to start with one of the things I've noticed over the past like at least 12 years in the overall crypto industry is a lot of the vulnerabilities that we're seeing in crypto are actually old vulnerabilities that were either core internet infrastructure or fundamental to human nature they're not things that are really novel and new to any of these protocols they're a whole class of things they're like um Insider threats Personnel issues Key Management leaky abstractions bugs versioning things like that so there's a whole range of things that we've seen that that have really affected people and I think some of the incidents that we can talk about uh clearly fall into that I mean there's a bunch that I can think of but if either of you have any that you can think of that come off off the top of your head specific incidents and then the kind of vulnerability it was be interesting if you want to go this direction to kind of do sort of a taxonomy because there are so many different buckets I mean at least kind of three or four different buckets and you touched upon a couple of them right now some of them are social engineering related some of them are bugs in the code another bucket is um multi-cigs where kind of like your your vanilla Bridge exploit where you know one or more kind of keys were compromised um from where I sit the most interesting of these is kind of low level um bugs in sort of low-level libraries that are that that live there that lurk there for years and and they're not discovered until the whole world blows up so the most obvious and Salient example of this is the the binance hack that just happened a few days ago right that was like a very low level library and it's kind of in retrospect incredible that it was there for years and that code is used not only in the binance ecosystem but also in the cosmos ecosystem and Cosmos kind of got off like like very luckily in this case because the the vulnerability wasn't present there in the same fashion um I don't know we could dig into one of these or we could do a taxonomy in whatever Direction you think is the best way to go that's my favorite category of vulnerability yeah actually just to like jump on that point before I move on uh yeah it's really important to to realize that this code is very complicated and it's very new actually vitalica have a great talk a couple of days ago talking about bringing Roll-Ups online uh so I believe it was the uh the optimism uh fraud proof I think it was a get note actually it's about 34 000 lines of new code any and any small bug in that code could lead to actually billions of dollars being lost so how do we bring these like very complex pieces of software online there's a carry bill of dollars in kind of a safe and sustainable way and uh yeah definitely looking forward to exploring that question uh Jerry yeah I'm happy you mentioned this so like I'll try to share a little bit about what the world looks like from the perspective of a core developer right who worked on ethereum previously you know this is relevant in the context of my work on Space mesh and I think just any core developer we have this notion of a TCB TCB means trusted compute base and what it means is that there's some amount of code which is 100 critical to the functioning of the network and any bugs or vulnerabilities or exploits in this inside the trusted compute base are kind of apocalyptic right they're kind of like like stop the world scenarios like like many of the exploits we're talking about here and this is not new to crypto it's not new to web3 right this concept has existed for ever it's present in operating systems and things like this but as a core developer touching upon what you said a moment ago about kind of the diff you know looking at the changes that optimism has had to make to existing code like go ethereum or you know any anyone building um let's say modifying or tweaking or adding on to existing code like this is this is what keeps us up at night as core developers we want those changes to be as small as possible those diffs ideally to be on the order of like tens of lines rather than hundreds or thousands of lines because every one of those represents a very real possibility of the introduction of a bug right another principle of computer science is that there's no such thing as bug free code doesn't matter how good of an engineer you are it doesn't matter how good your software engineering practices are there's some number you can reduce it right you can reduce the incidence of bugs by following best practices which again have existed for decades things like code reviews pair programming there's various techniques but it can only go so low I think Microsoft or someone did research on this a decade ago and found that the most senior most experienced programmers are still introducing one bug for every thousand lines of code period one bug for every thousand lines of code go ethereum is probably on the order of 100 000 lines of code right and let's say some of those bugs have been found but there's still bugs there for sure so anyway it's just as a as a comment as a core developer like this is literally what keeps us up at night we want to minimize the number of changes and that's very very difficult when we want to add features and bells and whistles you know add new OP codes to our VM um what's another good example uh a lot of blockchains are using webassembly as a VM now this is actually what I worked on in ethereum this thing called ewasum which was a candidate to replace evm after after the merge um webassembly is a whole complex ecosystem and if we start putting webassembly-based smart contracts on chain then all of a sudden the compilers and interpreters for webassembly are now inside the trusted compute base so you've added 100 000 lines of code or something to your trusted compute base so one I'm jumping ahead here but one potential mitigation strategy is to like really minimize the number of changes the number of things you add minimize the size of your trusted compute base uh on that front I mean the concept of a TCB is something that I read about in the rainbow books that were the the nsa's security guides back in like the they were written in the 80s so this is a very well understood Concept in theoretical computer security uh unfortunately it's kind of theoretical because a lot of systems there's two classes of errors here one there's bugs in your GCB two there's not identifying what really is the scope of your TCB correctly there's a lot of the the craziest things I've seen like uh the Yuga Yuga hack where a moderators uh account recovery credentials for a Discord server were essentially um used to cause like 200 like crazy like multi multi-billion or multi-million dollar loss uh was like 200 million dollars or something like that um so yeah yes there was I mean these things happen you know in in all companies in all Industries but it just just your point about the credentials being stolen this is what happened with that massive I think it was Uber right about about a year ago or something someone got credentials of an administrator by a social engineering and then got inside the slack and got inside you know yeah so we have to be careful this stuff too yeah this is actually happened a good a crypto quite a bit uh now mostly with Discord and Twitter um actually exact xpt a couple weeks ago had like a really interest Twitter I was talking about someone having access to a admin Twitter panel and basically that was what allowed them to take over all these prominent Twitter accounts yeah I think one of the main issues is we're using essentially consumer tools like Discord telegram Twitter things like that for large Financial applications and um the people supporting the applications that we're using are not building them for the use case they're being used for they were built for gamers talking to each other about um progress in a game and things like that so expecting them to provide the level of security needed for a trillion dollar ecosystem for security just doesn't make sense and it's unfair to them to to really blame them in any way and we need better tools we need better tools it's interesting I think I agree very strongly with what Ryan just said the situation has gotten a little bit better pretty much everything has two-factor authentication now and if you ask any CSO any security expert in the world what's the one thing you can and should and must do immediately to increase security it's not only enable but Force two-factor authentication on everyone um so it's a little better than it was like five ten years ago like when the Discord didn't previously have two factors authentication now they do and I think it's I think it may even be on by default um but I guess I'm kind of I'm curious I don't know the answer to this do we think that those tools that we use to coordinate communicate Etc need to be custom built or do we think that we can adapt existing tools to those purposes yeah I'm a big fan of not reventing the wheel if we could take from the let's say 50 plus years of security research in web 2 and adapt it to web 3. um yeah there's no brainer we definitely should do that yeah I mean I think most of web3 security is actually web 2 security so these lessons have already been learned by Enterprise users uh I would go a bit further than just saying two-factor I think we should be using single sign-on Hardware keys that are not fishable like the whole concept of getting hacked because one of your emails clicks on an email and goes to the wrong URL or is message something unsolicited on telegram instead of signal or telegram instead of Discord where they routinely come with someone go to a URL that's a phishing URL click on something and then boom you're out 500 million dollars is crazy like that's the vulnerability of the system not of the individual that clicks on a link and the the idea of mandatory phishing training for users is woefully inadequate for protecting these systems if it's possible to have a button that you push right next to another button that you push every single day and you push the wrong button and it destroys the world like the problem is not the person pushing the buttons the problem is the person who designed the buttons and put them there so you really need to have a single sign-on so you can revoke credentials for an employee if an employee here like it's very common to have a laptop lost in a bar or have a cell phone stolen which might be unlocked and things like that you want to be able to immediately revoke everybody's admin credentials so I think essentially if you have administrative credentials for any project which includes public publishing to social media doing any sort of action on Treasury anything like that and you don't have the ability to revoke those credentials remotely on loss in a single like in a like five minutes after the event is reported you are making a big mistake and if it's possible for a user to go to the wrong URL by accident or by phishing and lose their keys or use their keys incorrectly you're also making a big mistake so single sign-on and a hardware enforced key credentialing are the way to go but luckily the the broader security industry has largely accepted that and we now have things like pass keys on iOS and we have uh like the the 502 ecosystem and everything else so the key management in the traditional Enterprise security world is getting a lot better here we also have great open source SSO tools as well as commercial tools so so that world is getting better but the the problem one of the biggest problems with security is the the attacker can find any vulnerability the defender needs to be protected against everything and you get a lot of cases where I mean everyone knows as a team like you have a certain number of hours in a week for your people and you want to spend your resources on building the product because if your company fails if your project fails no one uses it because you don't have any features that's that's also very bad so if you have one hour to spend you probably want to spend that on working on your product and not locking down security so there's a lot of cases where the uh the floor of security is not a uniformly high enough floor there's a vulnerability and things like that so I think making tools better making things secure by default is really the solution there so one thing I want to touch on with adapting web 2 security practices web 3 is that one thing we have in web 3 theorem web 2 is these systems need to be uh sensors persistent and that's like a very important factor and that's one reason why metamask doesn't have 2fa I mean part of the reason is also because of the way aoas work and you can have multi-cigs with 2fa but if you want like an uncustodial wallet built on top of the ethereum platform there is no real way to do a 2fa on neo8 you have to have a spark contract based wallet which I think people are kind of moving towards in the future but like one example would be for instance the mango Market hack last night um people always say after these D5 hacks well why don't you have let's say let's look at a circuit breaker something that says okay if you're trying to withdraw 50 million dollars from the smart contract maybe you have a human being involved in that process and one reason why we might not want to do that and I I say with an asterisk because in the end if any smart contract which has the ability for an admin deposit is not decentralized so most smart contracts and most deeper protocols don't really we have a true decentralization as of yet and I think that's probably a good thing at this stage but I guess like assuming that that wasn't the case and the smart contract was truly decentralized um you couldn't put a human being in that Loop because that introduces a point of attack for regulation and a point of attack for uh for other kinds of compromises like you mentioned so how do we do this in a way which is which is programmatic and sensitive persistent so I think there are going to be need to be some new design patterns in in defy and you know there's things you could do like for instance you could uh for instance maybe even to help like an open Zeppelin kind of like smart contract library that that kind of like meters and says okay like don't let you you shouldn't be able to withdraw more than like one million dollars worth of usdc every like hour or so and I think that's fairly reasonable although maybe for example cases it might not be and it also might be difficult to do in a generalizable way given that if you have for instance Oracle attacks the smart contract might necessarily know what the value of the assets being withdrawn are so you need to be careful about about that code yeah I agree with that stuff I mean we're kind of jumping ahead a little bit to like mitigations and solutions um I'll just add one more comment on that and then Ryan you can steer us in one direction or another but uh so you know the title of this talk is notable security and since since Devcon five so obviously if we're talking past three years A lot has changed in the past three years you know three years ago I would say best practices along the lines that you're describing were just emerging they didn't really exist we were still at that point struggling with basic things like re-entrancy and I think now we have a much better what's that we still are yeah we start this is true this will be a thing forever um but you know we have libraries that you can include now from people like open Zeppelin that protect against common classes of attacks like re-entrancy that's just one example um you know basic things like not having ever a single point of failure which is to say like a single key pair or something right you and and as as he pointed out you know uh externally owned accounts which is to say just simple basic key pair based accounts and ethereum are a single point of failure you lose the key your host there's there's no getting it back so having multi-signatures um having cool off periods or delays right both for governance as well as any time you're moving funds right having tears so that you can kind of move a certain amount of funds instantly but something over whatever that amount is you know requires a delay of two days three days um I don't know if people are familiar with this but the reason the Dow hack the Dao of 2016 wasn't worse than it was the reason that the the white hat hackers were able to recover the majority of the eth was because there was a delay actually multiple delays baked into the protocol right so it could have been far far far worse yeah in general you have the the frequency of an attack the potential value at risk the amount that is actually stolen which might be limited by all these other countermeasures and then any sort of recovery procedures that are possible and we've definitely seen a lot of progress in the last three years on both chain analysis and a lot of you ah cool cool great okay uh so we've seen a lot of um uh positive uh changes in the overall ecosystem where we have uh now some chain analysis tools we have a huge ecosystem a security audit and infrastructure existing in the world we have probably hundreds of audit firms both coming from the web 2 world and dedicated to web3 Smart contract audits so a lot of these vulnerabilities are are bugs in code and I'd much rather find a bug in code in an audit that I've just paid a couple hundred thousand dollars for rather than in the wild we also have more of a culture of bug bounties and Reporting so if there is a bug found I can legally make a couple million dollars as a bug Bounty and be a good person and also build my own reputation and do it publicly everything else rather than just exploiting it and probably taking 20 30 million dollars plus in dirty money and being a horrible person and having to hide it and everything else and no one learning from it so there's a lot a lot of positive changes that have happened the counter argument to all this is just the whole ecosystem has continued to grow so if a vulnerability affects one percent of the ecosystem or vulnerabilities affect one percent of the ecosystem the total dollar amount and the total number of users affected just goes up as growth goes oh um yeah just a couple quick thoughts to add to what Ryan said the tooling that Ryan was alluding to has gotten really powerful and and a lot of it is open source right so like folks like trail of bits have multiple libraries multiple tools that you can use to do inspection and Analysis at various layers of the stack and I would actually say I wonder if you guys agree with this um simply running these open source tools today will probably get you more than you would have gotten if you paid six or seven figures for an audit three years ago uh so there's really powerful stuff I mean it's it's depends on your your threat model like it may not be sufficient you should still do audits copy item tour but there are very powerful tools out there yes Slither is quite nice um one thing I want to say about the existing models is I do think they're they're fairly insufficient like I I don't think that uh aside from a few auditing firms have very style reputations like let's say trail of bits or or daub most times when I see an audit I'm not necessarily thinking like oh this code is safe because we've seen many times the audits are are faulty like they're humans are being human code people can miss bugs either way another thing about bug bounties is actually that uh the system is is just not really working very well um as a bookbinder I can tell you ironically it's often you make more money if you if you hack the smart contract and then hold the fund for ransom and then keep 10 versus just reporting the bug which is just insane but so that's an argument for possibly increasing the amount like there's a whole calculus of if you offer people too much for um reporting bugs then you end up with uh bugs get reported through that channel and possibly introduced and you have available as I used to do security Contracting in Iraq and Afghanistan and there was a concept of like paying people if U.S forces caused a injury to someone or death and there was a crazy cat like I would obviously say you should pay as large amount of money as possible but if you make that amount too much there was an argument that people would intentionally cause innocence just to get the county which is terrible but you have the same risk with insiders like one of the whole categories that crypto companies do not address really at the level that traditional finance and everything else addresses is Insider threat you can have a team if a team is a few people that have started a project together and have worked together for a long time that's one level of trust but if you've got a team that's hundreds of people you're hiring people using like LinkedIn and whatever else somebody joins you don't really know what people's motivations are they could be incompetent they could be nation state Affiliated they could be criminal Affiliated everything else they may introduce bugs intentionally and then exploit them they may discover something in the course of their work and then pass it off to a friend externally who exploits it there's all sorts of stuff like that traditional Finance solves this by having separation of Duties so no single person would ever be able to have a vulnerability that would ideally that would be able to exploit things but that isn't really compatible with an ecosystem where stuff is being built as you're as you're going like you have a trade-off basically you can build new stuff or you can prioritize for extreme safety and resilience against this kind of Insider threat and I don't think it would be good for the ecosystem if we became so paranoid that we didn't allow Innovation um so yeah right our final speaker has arrived for your panel yeah come on come on down welcome welcome sorry about that learning about Bogota traffic this morning getting ddosed on the road um cool yeah I'm Tom uh I founded a D5 product called mosendo a few years ago focused on D5 payments I also co-founded a crypto options exchange which is a centralized exchange so dealt a lot with custodial risk and then I've been an investor in the space for a while and just started a seed fund called Network zero so uh I've actually been following the mango markets hack uh even on the right over here there's some interesting developments will be interesting to get to at some point and so one other point just before we move on was the the interesting argument about decentralization the binance hack that happened a couple days ago they were able to mitigate their loss because they were able to just stop their chain which is crazy from a centralization perspective but did save them 900 million dollars so that's the counter-argument against uh proper decentralization also the hacker Brady some of that money into usdc which immediately got frozen which is a centralized stable coin so yes that's cool um so yeah so actually for that binance side specifically uh I would argue that was certainly change sovereignty is an important factor and social expense is important factor of mitigating loss for sure and that's something that for instance protocols which have strong Bridges to ethereum don't have because if that theorem bridge is hacked basically they have no ability to roll back changes on ethereum to recover that money but with the buying stack in particular the hacker had about three hours to wreak havoc and he didn't so that was almost like an issue of hacker incompetence I think that binance could have lost quite a bit more if the hacker was a bit more competent in that situation yeah one of the things that I think Tom's a particular expert in is a lot of the D5 vulnerabilities which in a weird way are some combination of software bugs and economic vulnerabilities and figuring out exactly what's what in that space is difficult the main value of having a taxonomy is because you can then identify Trends and underlying risks and assess them so I don't know if it's worth trying to figure out which of these assets which of these risks are more economic versus more software bugs and how to mitigate those but if you want to talk about some of the defy issues that you've seen over the last three years yeah we've gone over flash loan attacks yet okay not yet oh excellent uh so I find that very interesting because it's like this novel attack that no one imagined uh this and this has not not possible in any other ecosystem right it's not yeah uh it's uh and it's new since the last Devcon right uh flash loans only came into existence in late 2019 early 2020. and then we had the first uh flash loan attack happened in 8th that number 2020 in February I remember the bzx team being quite excited about their products and then that day they got flash loan attacks and I'd like to not assume a ton of prior knowledge it might be worth explaining what a flash loan is just as a starting point okay yeah so flash loans um flash loans are interesting and actually you probably only hear about them in negative context but they're actually a net positive for the ecosystem they're actually very useful so flash loan basically allows you to borrow a nearly unlimited sum of money without having any money yourself uh but you have to do it within a single transaction so at the end of the transaction say there's multiple lines that make up a transaction you have to repay that loan at the end of it so people use it a lot for Arbitrage Bots or keeper Bots liquidation Bots these keep financial markets in order because uh coder who has the skills to run these things but not the capital to can use Flash loans to say borrow a million dollars to Arbitrage a position across markets into place so I like to say that ethereum's superpower is composability permissionless composability right the fact that you can build a smart contract that calls into a smart contract that calls into a smart contract kind of ad nauseum actually quite quite a few layers deep and yeah flash loans are just a form of Leverage right so they allow you to pay a very small amount of Interest I mean it probably amounts to thousands or tens of thousands of apy on an annualized basis but because it's for the span of a single block time a tiny amount of interest and just it's just just just jack up to a very high level a very high amount of Leverage the amount of funds you're moving around and using in that single transaction and that can flow down some of the most sophisticated flash loans they touched dozens and dozens and dozens of different contracts they go you know 20 30 layers deep it's kind of fascinating to unpick them as a developer very I just want to distinct make a distinction between composability and atomicity uh because you can have composability over multiple transactions but the atomicity of a transaction is is what is the key distinction here with flash loans Arun you wanted to say something yeah yeah just to kind of like add a bit to that point um what's about flashlights for protocol perspective is basically riskless I can lend out any money for my my protocol and get repaid back by any transaction it could roll back to already paid basically there's no down to the protocol and it makes markets more efficient but yeah I think the issue is it it provides like asymmetric ability for for hackers to kind of wreak havoc it's definitely like a very powerful tool I remember at one point last year Dai was considering having the ability for both a mint arbitrary amount to die in a transaction and then repay a data transaction and you can imagine just like what people can be capable of they could just like mint Billy and die and then repaint another transaction that would be kind of insane at least with the current flash loans if the guesses aren't the smart contract you can't use them so there's only 100 million dollars of usdcfr contract I can only use 100.

3cc but if I could literally create like a trillion dollars of die out of thin air and then repay it that's a lot more dangerous yeah yeah one of the the issues is you could certainly do this with most of these exploits would be possible without the flash lines you'd be able to if you had a large pool of assets uh exploit something that requires a large bowl of assets to exploit it but the set of people who have a large pool of assets lying around is much smaller and it's very hard to move 100 million dollars on chain anonymously and then the whole chain of moving stuff to and from and everything else so this opens up to potentially any attacker in the world can can exploit one of these vulnerabilities and not simply the people who have very large pools of capital lying around so the the key to these flash loan attacks is that you basically have one smart contract system which is doing imperfect accounting on some other smart contract system and when they're coding it they think oh nobody could ever possibly have that amount of money to manipulate that imperfection they make assumptions but those assumptions are not checked basically right so after flash loans happen I kind of view it as like the immune system of defy like if coders are taking shortcuts and they're they're taking these assumptions and being imprecise in their calculations flash loan attacker will eventually find it and exploit it so it actually encourages people to be more responsible with their coding practices yeah and we were talking earlier about how much more mature things have gotten in the past years since Deb con four basically got five sorry this is Defcon six um but yeah I I think it's like a fascinating category of of attacks because uh they're not exactly bugs like code was kind of working as intended uh but like yeah they were able to get like insane amounts of Leverage uh the other way that the flashlight attacks are commonly used is for Oracle manipulation so uh the most common attack is uh say you've got some protocol that uses some asset as a collateral uh and say that collateral actually has very illiquid markets and maybe there it's only one unit swap Market uh so they can use the flash loan to just wind up that market to pump up the value of that collateral uh deposit that collateral into say some sort of lending market and then withdraw um assets against the value of that collateral and then they unwind that because they have to repay their flash loan and so that um collateral becomes worthless again um so that's been another common Vector of attack but that's been uh there's ways to mitigate that um with instead of using a you know just a uniswap pool as an oracle there's some proper Oracle practices to fix that but that's been quite interesting as well I I just want to point out these this this class of attack if you want to call an attack is really interesting and really fascinating but I think it's very important to make a distinction between this and other forms of security incidents because I'm inclined to actually put these sorts of attacks and again I'm using scare quotes um into the Mev bucket right it's as you said a moment ago it's code working as it was intended and it's this is not new I have a background in high frequency trading right we just call this Arbitrage right it's people finding mispricings or um yeah it's exactly like you said when there's some sort of a mispricing or an assumption that's baked in at the interface it tends to be the interface of two different kind of systems or something and and taking advantage of that um I'm not sure I'd call that a security incident like an actual bug in core code or something so it's important to make that distinction and by the way we should talk more about Meb because Meb is also really interesting this is why I've been using the word attack instead of hack let's talk about it okay cool yeah uh I just want to make a point that uh like more generally speaking because smart contracts are immutable um when you write smart Contour code it's it's very unreasonable to imagine like I have to write this code so it's a cure for the next hundred years against any future Innovation that could possibly happen in D5 like like a lot of these archives written before flash loans lots of came along you'll kind of support these smart contracts uh so how do we get around this well it's like really difficult right because uh as we've seen protocol upgrades um sure the devs can be responsible and kind of upgraded after protocol in response to like uh new developments but they're still like money and assets in the old protocol that can't be withdrawn and a huge class of attacks comes from people just like not being up to date on what's Happening and not withdrawing their money from from still protocol so I mean one very classic example was when when the whole Terra Luna blob happened there were so many missed pricings everywhere because everyone had some assumptions about the price of Luna on many different chains because everywhere and then people just reaching Havoc I remember Venus Marcus got here pretty bad by that one yeah I think we see a large number of things where in the traditional security world you you hear about like the zero day versus the the well-known exploit that's been out there uh things like the profanity key generator that yes if we were the first person like one you should have reviewed it two if uh you were the first person to fall prey to that yeah that sucks but everyone else who suffered from that and hundreds of millions of dollars lost after the well-publicized exploit existed and clear way to mitigate it existed uh that is a failure to stay up to date on on threat intelligence and and act quickly which is a problem of not knowing all the components of your ecosystem like your asset inventory and um having a procedure in place for handling this kind of crisis I don't know if we want to get into that more because the Luna uh X not even exploit the Luna threat was known for four years publicly and people chose to ignore that and some people weren't informed on that but I don't know if I don't know if it's worth exploring that but I mean a lot of very prominent folks among them like investors security researchers had were very vocal from you know months or even years prior uh about about this class of of economic attacks we're talking about economic attack now uh and in fact there were warning signs right because there was um there was a I don't remember the details but there was an attack six months prior or something against um one of the other coins or assets or something in the Terra ecosystem and the price kind of It kind of depeged and then went back so there were like warning signs ahead of time as well yeah what I found interesting because I've talked to a lot of people about this incident because it affected a lot of people um so can we can we recap the inside yeah sure so it is a notable security it is a notable security incident so um it basically uh this Terra Luna um uh design protocol was basically it had a mechanism to mince a a so-called algorithmic stable coin uh where you burn the The Sovereign asset uh Luna in order to Mint uh UST which is supposed to be packed to a dollar and the idea there is that you'll always be able to redeem it for a dollar's worth of luno which then gets minted um if this sounds like circular logic it's because it is it is uh so many including myself have been vocal publicly about the death spiral uh inevitability of this design and I don't think anyone expected it to get as big as it did but what I find quite interesting is that um not only did I would say people who are less informed in the space who it is not their job to maybe know better you know okay it's understandable that they didn't understand that would happen uh um and they were kind of sold a risk-free thing that wasn't risk-free um there was very professional Traders um and other people in the industry who also thought that this would not result in a death spiral they thought it would work and they were in it um it's like it's like looking for a perpetual motion machine you know it kind of keeps moving until it stops yeah so this was I think another this was actually I think more example of like the the hubris of success of the bull market where everyone was like oh we've like solved like financial problems we've solved the perpetual motion machine it's going to work it's working and it got just like really big and it was It was kind of just destined to not work but um so so something else that's new since the last Devcon which and this is actually the the aspect of the Terra Luna situation that I find the most interesting this goes back to composability which we talked about before it's the intertwined nature right it's that we have now assets built on top of assets built on top of assets and so when one failure occurs right three four years ago if something broke it didn't cause this Cascade effect throughout the ecosystem and actually in the case of Terra Luna it was across multiple ecosystems right I mean Bitcoin began crashing and there's a whole reason for that they had this Terra Luna Atari Terra Foundation guard um uh treasury that held Bitcoin they had to start selling Bitcoin to defend it but this um uh what's the word for it this uh uh contagion yeah the contagion that spreads throughout like this is a new class of economic risk that didn't exist before I think we need to understand it better because it will happen again but I think contagions existed in in tradify for quite a while people are well aware of of contagion and but you have something like like Central Bank to kind of like perhaps contain it and and uh and limit it which arguably is not good for the long-term economic uh viability of traffic markets and perhaps it's better uh with D5 for these to kind of play out and just kind of like it's almost like a forest fire Just Let the Fire Burn everything and then regrow everything versus just kind of keep these like sick trees alive forever like we have it in web 2. um so one thing that I want to kind of touch on with the hotel incident is we got very touched on it this idea of like the illusion of like security uh which was actually very dangerous we had the lfd reserve we had people like jump and Delphi and many prominent people kind of say hey we know about dead spiral we have a handle we have six billion dollars of BTC we're going to buy everything and it's going to be fine Ed uh I think many people looked at and said okay well jump has a lot of really good quantitative researchers I assume they did that kind of modeling I assume they they rent numbers and we're like okay this is probably work but it turns out no one knew what they were doing so how exactly do we do we like mitigate this particular uh class of like issues I think it's very dangerous when you have a situation where people know an issue can occur and then people also think that other people who are very smart also know this quicker and we see all types of spark contracts like when Alameda jumps into a smart contract I think people tend to assume that because they're in that spark contract they've audited it and they they know what's going on they put 100 million dollars into it they probably does happen in reality there's like several examples contract for yield and there's been critical bugs in that bar contract which is just kind of insane uh and it just kind of goes to show you that like yeah you really cannot rely on like reputation Authority in this space to make your decision making and that makes things very difficult I think for the average person working in D5 because people who are not experts in these systems rely on Experts and those experts have been showing time to come and again to not be doing proper due diligence on their own there's a range of attacks that have happened where the system as designed was a good design and even as deployed initially was a good deployment there was uh forget which Bridge it was that had nine separate validators that they collapsed uh Ronin sorry running yes yes they collapsed it down to smaller than the the blast radius number and well they so there was four four validators controlled by the foundation and five were independent and then there was basically it was a usability problem so uh withdrawals were happening too slowly so one of the by the way the four run by the foundation were like all in the same AWS machine so they were basically one and then uh they convinced a fifth one to give them like temporary permission to like sign for them so basically five of nine were running on one machine which got compromised um yeah so that was like okay we started with good practices and then like we the the I think the term is the normal normalization of deviance it's a concept from like aerospace engineering so one of the interesting things about crypto is and a lot of people I talk with uh come up with the same thing is uh if you're deploying an immutable smart contract and there's reasons why you want to be immutable um protection from Forced upgrades uh malicious Insider of everything else but if you're deploying in a mutable system you're really in the same safety regime as Aerospace nuclear engineering everything else not something similar to normal software engineering and you need to have procedures in place for for that kind of ecosystem where traditionally you have more way more design work than you have implementation lots of review you have review of your review process feedback on that and it's very different than normal software engineering and that's not really how stuff is built in the broader ecosystem super briefly as a developer to speak to this this is really hard right most of us I'm going to say us I mean core developers also application developers we don't have this background or this training um these practices have begun to spread a little bit through the industry and we do now have folks contributing to projects like ethereum who have backgrounds and things like Aerospace but we need way better education around this stuff I really strongly agree with that to that point and to an earlier point about Trad Phi uh in the tradify world there's something called ratings agencies this doesn't exist in crypto in in Trad fi you pay professionals to evaluate the risk and give it a rating of a financial asset not even a security not even security wise there's also security ratings and certifications as well um but so we're missing that in crypto there's no professionals that are actually actively monitoring even the economic risks of these various systems the thing that defy has that tradify doesn't have is real-time transparency and auditability so if you look to the 2008 housing crisis that was basically you know a series of packaged and repackaged and repackaged debt instruments that eight layers deep nobody could see through to what the underlying was and they just trusted that the ratings agency that gave it an A plus that everything was a plus turned out there was a bunch of you know F through a uh debt within that package which caused people to basically buy risk that they didn't know they were buying with D5 we have that real-time transparency but what we were lacking right now is uh basically professionals that evaluate that risk and give ratings on it or give metrics on it so I'm seeing um the beginnings of that happening I'm seeing a couple of projects that are focused on becoming this risk Oracle risk engine um so I'm excited for that in the future because you know we will combine this transparency and D5 with uh the the various ratings mechanisms or risk evaluation mechanisms that are coming out massive massive massive opportunity seriously anyone out there who's interested in this stuff who has a background in finance who's inclined this stuff is sorely needed and there's huge lack of the stuff in the market today it's a huge opportunity like work on it talk to us we'll we'll help you make sure you're talking to the right people yeah actually funnily enough uh like talk to him yeah yeah so so Auntie which was the project I was briefly Devon is essentially trying to do this in a in a decentralized way um I I think that one of the issues with having uh kind of like centralized risk Auditors um is the fact that they are foul we're not saying decentralized earthquers are perhaps better it's just the entire system as a whole I think uh is fallible to some level of human error and any kind of appeal to Authority I think is kind of dangerous I mean just because we have Moody's this period doesn't mean that tried fly is is immune to like contagion stuff like that like it'd be it it it definitely isn't like like these companies uh I guess like the issue the thing you haven't tried fight of in D5 sorry I'm kind of rambling at here is you have this kind of like Fail-Safe mechanism the Central Bank who's able to kind of rescue everyone in case everyone gets a bit out of control uh Ed I am not sure if if like Moody's makes things like that much safer uh it's probably better than having nothing at all but it definitely is not like ideal solution and it probably is is not going to work uh that well for for D5 where we have no safety net we probably need to be a bit more robust so the big problems with rating agencies is that they don't have any skin in the game so what I'm really bullish on is Insurance uh and other types of systems where they're underwriting the risk rating that they're giving um so there's lots of different ways you can do that whether it's with a regular insurance company which is what ever toss does or various D5 mechanism designs this is something that I've researched very deeply and seriously if anybody wants to work on this come talk to me because uh yeah they're I'm aware of some very interesting things there yeah we do underwriting for a lot of problems for technical risk and projects for practical risk and things like that and the problem with how much can you trust a software audit or a software Security review Auditors actually hate the term audit when applied to their their work product but how do you trust different firms and honestly at every firm I've talked to can do great work the the thing from an insurance and a Reliance perspective is you really need to look at the the like the the worst work that they'll pass and make sure that that is high quality because if you're going to just blanket trust the security review performed by a firm you really care about their internal Quality Control process and knowing that if somebody's it's a new system where they're not already expert in it or where perhaps they don't have their they're very busy one of the biggest problems in the security review industry is it takes three to 18 months to get queued up for a security review at a lot of these firms and as a result there's a huge pressure to ship and then get your review done while it's in production or get your review scheduled before you're finished building it in which case you haven't reviewed the system as it's deployed so one of the critical things that we absolutely as an industry need to improve is better tooling but also more Security Professionals in the industry so that we can do more of this stuff in parallel because every project out there needs to have this work done and it's a contended resource and you you absolutely want to be one of the ones that gets a Security review and not one of the ones that's waiting while you get popped I don't know if we want to touch on something that we've kind of uncovered which is that technical Auditors are basically expected to take on this role of doing an economic audit as well and they're actually not the best people to do that right they've they've categorized the known economic attacks but um for instance the mango Market attack which just happened yesterday was a blatant Financial Oracle manipulation attack they manipulated if you're not aware mango got drained for about 100 million uh the attacker increased the price of the mango asset by manipulating the price of mango across various markets and then borrowed better assets than mango against that unrealized position um and like the a technical auditor could have looked through the mango code and the mango team is super smart uh and and you know everyone involved like is you know knows what they're doing um but like this was a uh this is a very uh you know weird Edge case that somebody with a Financial Risk perspective would have been the only one that would have caught that just to be clear there are folks who specialize in the financial side of things so shout out to Gauntlet shout out to block science right they have tools uh they're experts in this but those are the only two I know of whereas there's dozens of Auditors who are like you're describing technical Auditors who I think would not have the sophistication to catch this category of bugs so again yeah so so one thing touching on the economic audit I do know subology programs that now do this kind of auditing separately and actually there are like for instance compound and Ave will will retain people to do ongoing economic Audits and I think it's very necessary to be able to adjust the parameters of your protocol uh it's subconscious way of course to to adjust for the changing kind of parameters of like what's actually in that protocol because otherwise you end up situations like for instance mango markets um if you're not aware this is a very Capital attentive attack it took I think around five or ten million dollars of the user on Capital due to that attack and the thing is like I think most people looking at Mega markets are like oh this book has 10 million dollars in it was audited is Oracle attack is going to be very expensive pull up it's not worth it once that's 100 million dollars in it so in this oracles are very viable so you actually need like a real-time update of these parameters over time to maintain the safety to protocol and that's something that's kind of missing except with like kind of like a tier wood or S tier protocols in D5 right now yeah I'd say another interesting thing that's happened over the last three years is the rise of the consumer nft space which is bringing a lot of new people into the space that that really aren't they're they're from an art background they're not people who are software engineers in a lot of cases who think about systems failures even if they don't think about it from a security perspective they think about software bugs so there's a lot of people who really don't have the um the background or the the experience to to know what to be afraid of just sort of like naturally and we haven't built as an industry that the right tools to make it safe for people to do things without understanding all the details of how it works like there's a lot of people who drive cars who don't know that much about how airbags work and seat belts and everything else they just know if you do it you're probably okay within certain parameters we are nowhere near that level of uh professionalism and reliability in the in the space and we'll see all these people that are that are new to crypto because of they've come in through the nft world and they fall prey to the same issue issues that that happened to early regular crypto users 10 years ago and are largely mitigated by Community knowledge and tooling there so it's kind of scary that there's the the high end of these D5 vulnerabilities we're showing new classes of attacks but some of the most basic things are happening to a new class of users right now as well so I think a notable security incident along these lines is like owners of board Apes revealing their private keys to social Engineers on their Discord and losing their Apes [Music] um you know I think you're right Ryan I think like it's incumbent upon us as designers and builders in this space to like build better tools and and install some of those safety rails or airbags like you're talking about but I just want to highlight that as someone who thinks deeply about this and has been working on this for years there's a tension here right because the easiest way to install safety rails is centralization right is to say yeah you know for my grandma like maybe she's not gonna really effectively manage her private Keys maybe she's not really going to understand a ledger or a multi-sig so maybe the best thing for her to do is keep her crypto on coinbase and and Outsource custody to someone else and that in some respects is antithetical to like the whole reason we're here so there's no easy solution to this but I just want to highlight that tension and that's where I approach this as a protocol engineer as a developer yeah I think kind of a good balance between this is that the base protocols really stay true to like the tenets of crypto it should be decentralized such a persistent uh permissionless all this stuff so Bill can build it on top of that and I think that it's kind of on us to educate people on the right right on board themselves like Grandma should not be probably creating a metamask wallet I think that's generally something that is is uh is difficult to secure I think for the vast majority of people uh and in the current situation as we see with the Mt space that uh like there's no way we can onboard a billion people into crypto with the current state like people uh like it's just like it's just going to create like a Feeding Frenzy for the Piranhas that are constantly looking for ways to steal people's money so we need to figure out these more robust wallet systems for these people uh and I think that's probably a better compromise we onboard people through like let's say a kind of abstraction wallets or or some kind of like non-custodial wallet solution with the hypervisor protocol layer which manages signing and policy networks and then we keep kind of like the more bare metal like quote-unquote riskier but more permissionless wallets for people who need them but one of the things that scares me is while these these losses might be relatively a small dollar values relative to the larger G5 particle hacks and things like that uh the number of victims is very high and politically if you have a lot of victims that individually like lost event of any kind and it makes it very easy for Regulators to point it at like a grandmother who's lost something or it otherwise unsophisticated user who felt prey to some some system or possibly was scammed and then pushed for regulation so we have the centralization risk as a as a the potential to use centralization as a countermeasure to a lot of this stuff is a risk of centralization but also acquiring a very very bad public relations nightmare as well as regulatory nightmare is also an existential threat to the the overall industry cool do we want to open up for Q a how are you excellent call when you talked a little bit about account instruction uh so I want to know if you could talk a little bit about that because I think that it's like their solution to have like easier Coastal systems without depending on I mean self-costal systems without depending on centralized entities so what do you think about that and how we are advancing and when do you think we will be able to have that for the General Public so I think there's like two classes of transcription um so just to define the term very generally kind of abstraction is essentially when you make no distinction between smart contracts and editor while it's on ethereum right now individual wallets are fundamentally different than smart contracts because individuals are our public private key pair smart contracts are are derived very differently and and honestly I have time to go into that right now but in a kind of abstraction they're basically the same smart contracts individual Arts Bar contracts and we have the ability to do that right now with things like like multi-cigs which obviously are somewhat valuable and in the future we may eventually end up doing up doing away with the distinction entirely so people with Roll-Ups were trying to do that but eventually ended up deciding not to because uh you know making these kind of protocol level changes just made it very difficult for ethereum developer to kind of compose and build on them um but the nice thing about kind of Attraction the design space is is very wide people could do things like multi-cigs they can do things that are much more complicated like have like off-chain oracles that you know involve human beings process and do basically kind of the sky's the limit in terms of what people are kind of building gear one interesting project that's kind of uh actually never mind up yeah sorry so we built this for space mesh okay we built the content abstraction I think we may be the first layer one maybe the second near has something similar uh it was really really really really really hard to your point it had very deep profound implications and ramifications on the protocol but it's a very powerful primitive and it means that the really simple way to think about this is every account by default is a multi-sig and there's always multiple Keys associated with it again this is also the case in near so shout out to near for kind of like pioneering some of this work it's a very important primitive and I think it will go a long way towards addressing some of the kind of classes of attacks we discussed here so I would like to ask which security incidents in your opinion was the most impactful not in terms of value lost but in terms of sending cripples across the ecosystem okay the first one that came to my mind this actually touches upon a point that Ryan made a moment ago which is that it's not necessarily as you said as well about the total amount of funds but it has more to do with the the number of people impacted and the sophistication of those people um there was a hack in a Solana ecosystem wallet about four or five months ago does anyone remember the name of the wallet slope wallet thank you uh and what happened here was a bug in the wallet that exposed the private keys because of some Telemetry kind of tooling uh very common because called Sentry or something I think we actually use it in in space mesh as well and the reason that I find this one so impactful is because this was the actual total number of funds was very small and I think it was eight or ten million dollars worth or something so it was like maybe even less than that but it was thousands of users who were impacted and each of them had a very small amount of money in their wallets and they woke up one day and their funds were gone that I don't think has ever happened before that particular class of attack and everyone was panicking and no one knew yeah it was a very scary attack even scarier about that is no one knew which wallet it was because a lot of people were reusing seed phrases across multiple wallets so they thought it was a Solana base level vulnerability or something it wasn't even localized to a wallet it was very scary watching that in real time just PSA because apparently it needs to be said do not log private Keys into your logger and also use Hardware wallets okay this this classic attack is not possible with a hardware wallet that too um I think probably the most impactful on the space has been the combination of the UST collapse which led to the three arrows collapse which led to an entire uh collapse of the crypto credit markets and has really raised a lot of red flags with Regulators like we've seen a lot of regulatory pushes because of that combination of UST and three arrows um and that has impacted a lot of uh you know non-crypto native folks who have sold a risk-free USD interest rate um so it did happen to be a large dollar value but it's having heavy political ramifications for the final remark Aaron uh plus one for UST plus one for Solana thank you this was really I learned a lot I hope you guys did too thank you very much to everyone else on the panel