Lessons from Web3 Security — ​engn33r | twyne

I'll pay you your commission later for the compliment okay so hello everyone today we're going to talk a little bit about web 3 security I'm going to keep it light so if you don't have deep technical background don't worry this is supposed to be a fun engaging informative and uh we'll dive more into the security side than perhaps you've seen before but hopefully in a relaxing way so uh world's fastest intro I'm engineer and I do or maybe did security at y audit and Y Academy so what we're talking about today is basically security a bit about the landscaped landscape today some of the misconceptions that a lot of people have with security in the space and also why it's hard because it's not always immediately obvious what makes this topic hard so Security in the space I mean if I was to ask any one of you what you've seen about Security in web 3 maybe it's something like this um just hypothetically um what's this it yeah uh it's not so great it's like when you see a headline that's like a 100 million gone in a single incident that's uh you know uh we'd prefer not to see these headlines um and then you see other ones like oh it's actually 167 million on this this other one and oh we got a few more 195 million and oh only only 8.5 million that's chump change like you can't even live off this I don't know if I can buy a hamburger um and then uh 6 million 182 million and you know what I'm I'm not even getting creative here because every single one of these headlines is with the exact same attack Vector it's like I'm not even looking about General attacks I'm just looking at flash loan attacks alone so this is how bad things are where a single type of attack you can have I don't know are we looking at like half half a billion dollars gone basically yeah so half half a billion with a single attack vector uh yeah it's probably more I mean now it's about like three billion just in the time since we started so uh anyway this is probably what you see you just see like everyone's doing a really bad job because you see the headlines but this is not the full story and we'll talk a bit more about the story today but this is probably how you feel when you hear security I'm sorry it's just how it is today uh so first let's get into a bit of the history because Security in the space has changed over time and even though you still see all these headlines of hacks happening I want to explain sort of the the maturity of the space now as opposed to uh four years ago 5 years ago when it was still very early so in 2020 we have everyone's uh favorite website for hearing about hacks wed. news uh if you are not familiar with this website I would recommend looking it up because they have uh very entertaining humorous writeups of hacks uh when they happen and defi summer begins defi summer is when a lot more defi protocols started getting deployed on chain um and that also meant a lot more money coming on chain and therefore more incentives for people to try and hack these projects and of course when there's only five projects on Chain versus 500 projects on chain uh 500 projects is a much easier Target to find the weak link to try and take money so uh really 2020 is you can say when things started getting uh bit more important for security just because there's more projects on chain in 2021 people started taking some action on the security side when they started seeing these hacks uh code Arena which is a uh platform for crowdsourcing uh security vulnerabilities uh some people call it a crowdsourcing audit platform whatever you want to call it uh that started in 2021 and one very famous security researcher Sam CZ Sun started publishing a lot more in 2021 uh again if you're in security and you haven't read all his articles I would highly recommend it uh even if some of the writeups are a bit old now there are still very valuable lessons in there um so 2021 it's it's after all these projects started deploying on chain and people start realizing this is important hacks are happening maybe your your money in the protocols is not safe if you're not paying attention to security um in 2022 there was a lot more f focus on security um I would say between 2021 2022 is when I entered the space and 2022 it was uh definitely a growth year a lot more protocols started realizing they need to do an audit this was not really a given before 2022 uh doing a security audit was maybe a bit optional a bit of a bonus and 2022 people started catching on if they don't do an audit or have some security people give their input uh there can be some dangers there so this is the year when if you wanted an audit you had to wait maybe six months maybe 9 months which is actually very ridiculous because crypto moves so fast no one is going to wait nine months you'd rather just ship it and then do the audit later and that led to more problems but anyway um 2022 I would say is when people started needing security but there wasn't enough supply of security people um but things changed because crypto changes and crypto moves very fast so in 2023 more security people started popping up um because Security in this space is so new if you had like two months experience you could already get a job um which is probably a bit dangerous but that's just how things were at the time um so 2023 I'd say a lot more supply of security knowledge got onboarded to the space uh which is great for the space as a whole because the more people who know how to stop the hacks the better for everyone users developers um everything and then 2024 where we are today I would almost say that there's like so many people that decided to shift towards security that maybe it's actually getting a bit crowded especially because we don't have quite the defi summer bull market as we did back in 2021 um so now the audit backlog is very short even at some of the most popular firms less than a month uh for the backlog uh there's many audit firms I actually often get asked do you know about this audit firm and I'm said I often reply I've never heard of that name like when did they appear on the scene so there's just a lot more security people is all I can say um I can't speak to the quality of all of them because clearly I don't know about some of them uh but this is just a bit of a a timeline we don't know what's happening next year I don't even want to guess um but I just want to share that things started from the perspective of we don't even know we need security some people came along and said security is kind of important might want to look at this and then security people started getting onboarded into the space and now it seems like we have a better balance so hopefully going forward we see fewer headlines um or at least headlines with let's say smaller figures ideally uh whether that's going to happen I'm not placing my bets but but let's see so when you hear security you might have a vague notion of like a paranoid guy in a basement wearing a tin foil hat um but there's a few different characters that might work in web 3 security and I just want to introduce the cast so that you have a bit better understanding of of if someone says they're in security which bucket you can place them um so first perhaps the most common is let's say a smart contract Auditor in a company or a dow uh their job is just to read code basically uh if you think of a book author as writing a book a uh an auditor is the book editor so they are reading the code they are circling the mistakes and they make sure the developer fixes those mistakes um before the book or the code is published now a security engineer is a little bit different uh there actually are not too many of these yet in the space but some large protocols I would say mostly layer one layer Two Chains uh have an internal security person and their job is to provide guidance on the architecture design to keep everything secure so they're like internal to a project the Auditors are usually an external person that you hire um solo Auditors if you're on crypto Twitter you've seen these people because they are way too loud just someone make them shut up um but the reason they're loud is because they have to do their own marketing because they're just like a single person so so the reason you see them shouting on Twitter all the time is because they have to to do this business model um and then the ones you don't hear about almost ever are the bounty hunters they just sort of lurk in code bases looking for bugs and then report it to a bug Bounty platform um so they actually don't want to share their methods usually because they're trying to race and beat everyone else to finding a bug and some code base um so anyway these are the different characters you can meet in the space and just wanted to share this in casee any of you are interested in the security side perhaps want to try and uh fall into one of these buckets in the future and let's do an honorable mention something that hopefully everyone here will learn about or or pay attention to because you might need it in the future uh seal 911 can we see a raise of hands who has heard of seal 911 okay great I'm glad I am bringing this up um this is basically a team of o volunteers like very smart security people in the space and completely voluntarily they have created let's say a helpline so if you have a security vulnerability in your protocol you can just reach out to them and say like help uh I don't even know what I'm doing but someone's hacking my protocol or like we just learned about a crazy bug and we don't know how to fix it so these people being experts are sort of running around everywhere in the space trying to put out all the fires um it sounds like it's a very hard life for them because it sounds like there's a lot of fires but if you have a problem this is a resource to know about so that's why I wanted to share this uh if you want to volunteer putting out fires in the space maybe you can also contact them um they're very busy okay why is security needed I don't even know if I needed this slide but I thought you know let's assume knowledge uh so every line of code should get some review because like even changing one line can make the entire protocol and secure uh it might sound hard to Fathom until you understand like the flow of logic always goes through certain paths and like if you just change the way it works in one place everything can be bad um and obviously the stakes are high this is why we have these sad headlines of nine figure hacks happening now and then uh and when Stakes are high then there's many black hats that would love to grab the free money out there so we need to try and prevent against this so some misconceptions of security if you hear solo Auditors on crypto Twitter they're going to pitch that their audits are bulletproof uh I'm going to tell you that's definitely not true and there's some protocols out there who have done like 10 audits or more for the same exact code base and the different audits reveal different issues so there's no such thing as a bulletproof audit uh all that you're getting is like some comments from someone who knows about the subject so uh but anyway that's the best we have today hopefully we'll get better in the future so I'm going to go through these misconceptions and I want you to also think if you have any questions or misconceptions or even comments about Security in the space because afterwards we'll do a Q&A or ask me anything and and uh you can pitch some question or prompt and you can get someone to give an opinion on it if if you want so anyway I'm going through these one by one uh first uh security is so hard to learn like it's it's so complicated there's there's crazy hacks and bugs in every direction and I'm not going to tell you it's easy uh but I'm going to tell you there's there's Pathways to get there with incentives and learning from a book is pretty dry and boring I don't know any of us want to go back to school anyone no no hands okay um but what if you were getting paid while you were learning and it's not from a book it's actual code and you can actually help people while getting paid so this is what we have in this security space is we have these contests so I'm I don't know if any of you in the back can see but I'll just read out some of the numbers here the prize pools we have 20K 70k 60k 200k 12K 15K half a million uh 89k uh another half a million 150,000 so yeah with numbers like this it's a little bit of an incentive to study um if you can get I mean you probably won't get all the half million but let's be optimistic like maybe you will uh so if you can learn about Security apply your knowledge and get paid on the way then really there is almost no barrier to entry because usually the barrier to entry is need to study for a long time and then you get paid but in security you can get paid from day one like your first day of studying so I would highly recommend if you are interested in this topic it's worth considering the contest route as a way to study apply your knowledge and even in the contest results you see how you stack up against other people so uh this is very cool and the fact that even these contests usually you don't need to kyc it's like literally no barrier to you may as well try if you're interested um it does take time you're probably not going to get much money in the beginning um but it's still entertaining uh as long as you're okay with some early failures uh I will say there's also different uh groups that have trainings I will show why Academy which I've been part of and secum have these basically free trainings that you can enroll in and learn more about how to find these books so may be hard to learn but you have incentives and there's so much open- Source free knowledge on the internet for this subject that uh it's very doable okay second misconception you have to know all the hacks ever like encyclopedia Knowledge full recall of everything and I don't think anyone has that in the space um I will even tell a little funny story uh from one audit I did we just onboarded uh a new guy and we had this new guy on the audit and uh the new guy wrote up an issue and the other person who is not a new guy on the audit with me was like have you ever seen this type of issue like what did he write up is it a real issue and I started researching him like oh yeah I saw this on Twitter like three months ago I didn't think it would be like a real thing you would find so this new guy who just like joined the team was already finding stuff that we I I had at least seen somewhere but I never looked into it deeply um so yeah no one no one knows all the hacks ever and everyone brings their own knowledge to the team and this is why it's very important to have more than just one person look at code um because for example pick a hypothetical person they might have really good knowledge of like economic attack vectors but then someone else is really good at math and someone else just knows about crazy re-entrancy attacks all these people have different skill sets and it's like any team you want a a team that balances itself out with good knowledge so uh yeah the other points here you can even just focus on a very specific type of bug and succeed you could do this with the bug Bounty approach to finding bugs uh or with the contest approach because both of those you don't need to find all the bugs just some um now Auditors working in a company do want to find all the bugs ideally uh yeah this is actually what I just said so I won't repeat myself um and many hacks are just known issues so you can actually look at a giant list of past hacks and go through them one by one this GitHub repository is excellent because it even has a proof of concept of how to execute the exact Hack That Stole money so if you were to have a time machine this repository would be worth like billions um basically so uh anyway would highly recommend it in fact uh I would say if I was starting from scratch today I would heavily rely on this this uh as a learning resource it's a stressful job yeah well the 21y olds would disagree um but I would say this isn't completely true like a lot of these misconceptions there is definitely some truth to it um not not completely true but I'll say uh it's different types of stress for each of these security characters um so for the Auditors who have to find all the bugs and like say this code is secure you're always in fear of like missing something because if you missed one thing and then some other audit company finds it some hacker finds it it's a really bad feeling and uh there's a lot of Auditors out there who have this feeling because they thought they were finding everything they missed something um so yeah that's one type of stress now the security engineer is probably being on the inside of a project Their Fear is to get a message at midnight like hey you up like we found a bug you have to like you know quickly rescue everything within minutes before the hackers do it so that's a different type of stress um the solo Auditors they're always on Twitter shouting because they want more customers um yeah so they're wondering where their next clients are and the bounty hunters just wonder if they'll ever find anything because the hit rate is so low um that yeah you can have months with no luck whatsoever uh but that's that's where the stress comes in so it's not the same stress for each person uh but you get to pick your favorite stress basically and I guess this jumps into a little bit of house security is a bit hard uh because you might expect bugs in certain ways certain formats certain uh types of attacks but there's always a greater series of possibilities than you might imagine so the real tricky part is expanding your expectations of where the bugs are to grow to the full space where they can actually exist so I would say that's sort of uh what every person working in security is striving to do to like increase their knowledge and understanding to have the two circles match okay another misconception North Korea they're just taking everyone's money no one can stop them they're like these these crazy hackers you see in movies with their hoodies on in the dark room and uh at least a lot of news stories would make you think this because like every time there's a lot of money stolen it's always North Korea it's like you can just predict it it's like oh I I don't even read need to read this news story it was North Korea like I I just know um I'm not going to comment on whether all those claims are true uh I don't think we've gotten many interviews with the North Korean hacker were like oh yeah that was me yeah I was I was so good at that one um but anyway uh what I will say about a lot of these attacks is very few of them are like really clever things a lot of them when you read the article when you read the postmortem it's like oh they didn't do this like well of course they got hacked it's almost like they I don't want to say they deserved it but like were you guys even checking with the security person like this is a very avoidable thing thing that you could have prevented um but the other difficulty is even if it looks simple after the hack happens the question is like did it look simple beforehand because there's so many ways that you can have some missing piece of security in developing something that uh it's hard to have all your defensive Shields always active um and another thing is that web 3 teams do forget about web 2 security sometimes and and I don't know if anyone here has figured out how to build web 3 without web 2 I think we're all still using the internet last I checked so there's some dependencies on web 2 that uh do need to be considered um I won't elaborate on this because this is not a talk on web 2 security but honestly we need to start having talks like that because a lot of hacks are getting back to this stuff so if if we can get web 3 without to let me know I I really want to be using that Tech stack and just to demonstrate uh why I would argue North Korea is not Unstoppable is a lot of these hacks happen again and again and again and this is why when I read some reports I roll my eyes like ah it's another one of those and this is a list this list I think is actually like one or two years years old I need to get the updated version from the Creator uh but you can see the most common attack we have 29 instances of this price Oracle manipulation attack 29 and probably every single one of those resulted in like I don't know probably average of a million dollars and more lost so I I think after maybe three of those we could have learned the lesson of what not to do but no it's 29 and Counting because I can guarantee you there's more coming um so is this a problem with developer education is this a problem with user education I really don't know exactly who we want to point the finger at but we don't need 29 of the same issue to learn this is an issue so uh anyway you can look at this list here I will point out number four stolen private Keys um how many of you uh have your private key on you and want to share it now anyone no no takers okay um this is like not even really a web 3 thing this is like do you have a password that you store securely um and I will say in web 2 security bad passwords is a common attack Vector but uh having these stolen it's like you wrote it on a Post-It note I don't I don't even understand like what's going on here so yeah but if you can avoid these issues you're doing well because these are the most common this is another misconception if you get hacked you're an idiot I'm not going to argue completely against it but um it depends on the type of vulnerability and again sometimes it's easy to say after like this was so obvious like no one no one checked this um but then other times it's like well they added it at the last minute and actually no security person even looked at that line of code so it's like I don't know you can point finger where you will but uh I wouldn't argue this is always true and there are always some extremely novel vulnerabilities that come about they are rare um but I will point out that like some attacks it's like some crazy series of six steps and no one's dreamed of these six steps before in that order and it's only something that could be done if the code is working in a certain way so like somehow the hacker found the right code the right six steps and put it all together and in those cases like I'm not here to congratulate the hacker but like that took some serious effort like that was not a straightforward thing um okay last misconception of course this one this is not a misconception this is just reality no just kidding um security jobs are the best um I will say it's also very good if you're already a paranoid person um speaking from experience it's a great fit for those type of people um if you're not paranoid you'll get there don't worry it just it's part of the evolution um now I will say one thing to keep in mind the devs you work with the developers who write the code because you're basically telling them that they're wrong they're going to argue that they're not wrong actually they're they're quite smart um this is just part of the game uh but at the end of the day you want users to be happy and safe that's the name of the game doesn't matter who's right um I will say one thing I learned after quite a while of doing this is that uh you get to a point where when you see code you just expect there's bugs and this can be a little bit uh sad at some point because you realize that this industry we're trying to build great things we're trying to like change the world it should be full of like rainbows unicorns Lambos all this stuff um but now when I look at the code I'm just like you know there's so many problems here all I have to do is like prove it like it's going to be easy just give me give me a few days um and that that's not necessarily a healthy positive view of like we're making progress so what I would suggest is having like some balance of looking on the developer side of what cool things are happening in the space uh just so you're not too negative because I I definitely got to a point where it was getting a bit too negative of what was happening so uh yeah I think this is a healthy thing to be aware of but yeah don't worry when you're in security you get to be the rain the Unicorn that comes and stamps approved on all the code and you sound so cool so this is basically what security people look like when they're having a good day yeah speaking from experience so why is security hard anyone see the flaw here this is a security test if you see the flaw raise your hand Compound bad uh no comp it's compound V2 code compound V2 Works anyone else so this is this there is no flaw that you can spot here but here's a hilarious thing you can take compound code compound Finance is like a working protocol it's been out there for a while you can copy and paste the code and then you can get hacked and it's like what did I even do I'm using the same exact code and I just got hacked and the reason that this is possible and actually it's not just possible it's here is all of the incidents of this happening it's not like one person did it and they got hacked it's like what two three four five six yeah I'm not even going to count but about a dozen um so all these times they just copied pasted the code maybe they changed it a little and in those cases I would have said like slap on the wrist or whatever like at least get an audit um but there's a lot of assumptions baked into the coat and if you're doing a parameter change it's not actually changing the code it's changing the value stored in the code it's like a a a setting that you're changing maybe the governance is changing something and if you don't know exactly what you're doing in those cases you can completely destroy what you've built and clearly that's not a good way to go go out um so anyway if you're copying and pasting code you need to also understand the code as well as the original developers and that's super hard to do because the developers that wrote the code they ran into bugs they had to add fixes to work around these bugs they understand like all the edge cases from writing the tests and most people that copy and paste they're not the type of person to put effort in sadly so am I saying you should avoid copy pasted code maybe it depends if you trust the developers to actually do the work to understand what they're doing properly um but this is just one example of how security is hard cuz you can look at the code and they say we just copied and pasted and you can verify that and say yep you copied and pasted but there can still be problems this is uh an example actually from a real audit um using compound V2 code so we'll have a real world example uh if you've heard of a it's a rather large uh protocol in the space uh you basically can deposit a token and get some yield and the way that the yield percentage is calculated is with like a two-piece defined curve and here we actually have the curves for D on I think this is on mainnet uh for a for compound and for an unnamed protocol and there's some Divergence here I'm not going to comment whether this Divergence is good or bad but there was no explanation from the developers about this Divergence it was just like well we picked some numbers and if you don't understand what you're doing with important parameter changes I would argue that's bad I don't know if it's like something that can get hacked tomorrow bad but it's not good so uh anyway just one example and also you can see that compound and a basically overlap until a certain point and then they they both split off at the same point so uh anyway just another question that the developers didn't really have a great answer to uh anyone spot a security issue here potentially it's not an immediate but okay that this is really hard I didn't really have a specific answer in mind you censorship that's not not a security issue it can be okay it can be yeah so we could say denial of service basically would be the security equivalent uh all that I did here was take a screenshot on ether scan of different blocks being mined and you can see that it's the same recipient uh or the same block builder for many blocks in a row and one possible issue here uh well I guess there's quite a few is if you have the same block Builder many blocks in a row this person can potentially do some Mev some minor extractable value to move transactions around in these blocks to extract profit and one thing that I think so far has been rather hypothetical is manipulating uh certain onchain oracles uh especially the decentralized kind um so if you're using Unis swap for Price data even their twap which is like supposedly a secure Oracle they have if you can do some funny business for many blocks you can actually alter some of the assumptions and the values of these oracles and that basically lets you change the price of an asset which is pretty dangerous sorry was there a question yeah yeah I just wanted to ask which chain is this is this this is M net yeah yeah and actually I think there was one place I found seven in a row and this was on the first page of searching I mean yeah I I would have thought that that like proof of steak would not allow this to happen it actually allows it to happen more sadly oh I see yeah so uh yeah uh I hope everyone sleeps well tonight you have a comment technically be is not a single BL buer it's a bundler so there's going to be multiple blog builders that send bundles to Beaver and beaver will EV them Bas on how much extract and then include the highest extract I am making this look a little more scary than it might be but uh there's still some scary stuff here that is worth paying attention to like quite high so they can definitely do attacks I think it's uh getting close to 50% if it's not there already so yeah um sleep well uh okay another uh just random uh point of why security is hard um does anyone know this EIP 155 9 sound familiar to anyone you might have heard of it some years ago um way back in prehistoric days if you're on main net uh so this EIP was uh at the time considered a pretty big deal and it was changing something related to the gas fees and burning eth uh but what I want to point out is a lot of people are aware of these L2 chains and these L2 chains are trying to provide something like the same experience as ethereum mainnet but cheaply and a lot of people put their code on mainnet and also on these L2 chains and what you have to realize is these chains are not actually the same so if you're putting the code the same code on different chains you have to know the differences of the chains and even though this EIP made it to main at ethereum in 2021 it didn't go to polygon until the next year took another year to get the optimism B&B took longer um so basically any EIP that exists that arrives on mainnet you cannot assume that it's going to arrive on the l2s until some later point and this actually became a slight issue recently with a new OP code push zero so if you compiled the code it actually wouldn't run on some chains because it's using something that doesn't exist there yet so so anyway ethereum itself as a blockchain changes over time and you have to factor that in if you're using multiple chains so anyway just another explanation of how security can get a bit uh annoyingly difficult so what does a future hold um I haven't shared anything like this before so you're getting some unique Alpha perhaps uh I think there's actually a lot more room for security work in the space that is not yet tapped into um and actually some of these I already know there is some demand people have been pinging me asking who can do this stuff and I just don't have a good answer yet um so I'm not going to I guess I can read these out very quickly uh but people who are specializing in wallet security there's a lot of wallet issues out there um but not many people are specialized in this uh front ends uh the front end for the DAP can have security issues uh even DNS can have security issues how you register the domain um there aren't people that I know that are specializing in the networking side like how the actual nodes are communicating um defi protocol Specialists there's there's so many Protocols of a single type now that you can just specialize in one um post deployment monitoring so just because you put your code on chain doesn't mean your job is done with the code you have to do some monitoring and especially when you're making updates or governance changes um blockchain specific specialization some blockchains are just different they're like really different so having a specialization in one can actually be a a full-time specialty now uh there are some tools that are just hard to use and having someone specialize in that can be good and uh actually this one there is a direct web 2 security analogy to this one where like if you get hacked then what in web 2 security there are literally companies you call like hey I got hacked bring in your team and then like the team just shows up and we don't have anything like that for web 3 yet but I think this is uh unfortunately something we need because these hacks still happen so uh anyway okay so a quick summary of what I talked about so I gave you an an overview of the landscape and how it's changed in the last 5 years um I shared some conceptions maybe you learned a little bit more about web 3 security uh or at least got some laughs and then even though security is hard in a number of ways uh I think there's more and more avenues that can exist if you're interested in getting into this space and uh also if you are are worried about using the technology in the space I think that more and more people on board into these niches to help let's say solve some of the issues we are seeing today so hopefully overall it's a relatively positive message as opposed to the scary headlines that I started the talk with and that's all thank you engineer ask session starts now so yes I uh will be open to any questions uh at all so go for it yeah how deep does the security engineer need to go because if you're looking at the code just for example in solidity that's AB exraction upon extraction upon extraction there is a compiler does he need to know the bite code and how it works and what is the limit that you need to go how deep you can say this is secure well I I want to just mention one security thing quickly for those of you who did pull out your phones to scan the code no doubt you found you've been rickrolled serves you right this is a test you just didn't know it [Laughter] yet okay I had my fun uh this was actually a suggestion the last time I gave a talk with the QR code so it's not for me um so I'll just repeat the question now that I've distracted everyone um you can get Rick R if you want to by the way QR code will stay so the question was how deep do you need to go because uh it's it's almost like diving down rabbit holes when you look at code you can look at the solidity code you can look at the bik code you could look at the compiler you could look at like literally every piece of the Tex stack and it's a really tall stack um so I don't have a perfect answer I can of course tell you how it works today um but I'm guessing uh one of the reasons that you asked this question is there have been uh vulnerabilities due to compiler bugs in the past I think the most well-known one was a Viper bug uh was that roughly a year ago I think uh I may have seen perhaps only the second bug of that type in the last couple months it was a I think fuel chain or something something gas I don't know um but it was like a compiler I'd never used myself I just saw on Twitter someone found a bug in this so I think more and more this will be a concern and uh a compiler bug for a language like C it can sometimes lead to say a Linux vulnerability a Windows vulnerability but if it lead to a D5 vulnerability it's like immediate monetary loss um so it's a lot more uh obvious what the impact is uh everyone can see it on chain everyone sees the stories so I think at the end of the day any any question about what the security people should do has to factor in like the effort and cost as well you can get an extremely comprehensive audit of your code it's going to take maybe a year and hire a team of 10 people and uh I don't know if anyone really wants to pay for that cost um so the the real problem is right now we have all these like cost tradeoffs and if you choose something that's suboptimal then you get hacked and it's like I didn't realize I was going to get hacked if if I knew I was getting hacked I would have paid a bit more up front like that's that's a typical approach I think uh so yeah there really is no good answer um I think what could be done is looking at what I what I always do in this web 3 security space is I try and find analogies to web 2 security where I was before because web 2 security has existed for much longer than web 3 security and if I can steal some ideas of what what evolved over there and then I look at web 3 where it's going and I'm like oh so the next step is this it's like I'm just looking at what happened in web 2 at that point of evolution and we can see potentially the exact same in web 3 so I think that's one way to approach things um I will say if you use that approach there actually aren't many good resources in web 2 security because no one actually reads all the code in web 2 unless you're talking about nuclear power plants or NASA and even NASA's had a bug that crashed a Rover where I think they didn't convert like feet to meters or something ridiculous it's like come on even you guys um so it's it's interesting how even though we have all these hacks we might already be in many ways better than web 2 security uh which it doesn't look like it but I will say one reason that you might think it looks like web 3 is worse is because a lot of web 2 hacks you never even hear about like how many of you have actually heard about a big web 2 hack with a bank anyone does anyone think that all banks are actually fully secure no uh if if you're also asking questions about what you've heard with web two hacks I there's one point I love to bring up which is how many of you have seen a headline of like North Korea hacking something China Iran Russia it's The Usual Suspects you've seen these headlines of course now how many of you have seen a Western Government hacking a headline anyone no h no comment uh so yeah I think web 2 can serve as an analogy for some extent but we are forging a lot of new ground so yeah no good answer yet thank you yes uh how effective are AI tools getting at code review and do you think they're going to replace some aspects of security auditing yes this question um so I've been asked by non-tech people if they're worried that AI is going to steal my job and I laugh I'm like do you know how many bugs we find in this code like if if the AI do it I'll just look for bugs in the AI code and have a new job like um so yeah I'm not particularly worried about uh that aspect but how useful they are for a smart contract code um what I often tell people is that AIS are trained on existing data from the web and some of these vulnerabilities there's like two blog posts that write about them so unless the AIS can somehow pull these right correct two blog posts at the exact right moment they're needed I think think the lack of data online is still a bit of like a uh problem for the AIS to to leverage in this domain uh I think they will improve and of course training an AI specifically on let's say the right uh information sources could help um but for me personally at least to the extent I've I've seen public research there could be private research that goes well beyond what I've seen uh I think the AI tools still lack some of the Nuance of the space uh for common vulnerabilities they definitely can spot some issues which is great um I think that actually means there's even less excuse for uh the most common bugs which like on this list so if if you're going to get hacked by one of these bugs at this point with some AI tools like come on like really really it there's 29 other people that had the same exact lesson AI can probably pick it up now and you did nothing so I think from this perspective hopefully it will be reduced on the common issues but the more rare issues are are difficult and I will also give one puzzle of like some of the security issues uh trail of bits wrote a report that some of the issues it would take uh AGI so uh generalized artificial intelligence to actually find them uh so I think it was like yeah half the issues so even if we had really good AI tools it couldn't find half of them and I'll give one example of like how how you'd even like try to get an AI to detect a bug so let's say you put uh $100 into uh some code and then after one week you pull out $101 is there a bug cuz you just got out $11 you only put in 100 where did the $1 come from so this is a sort of thing where it's like maybe AI has enough logic to like rationalize about what the code should do and say well actually you were only supposed to get $100 half but taking out1 is too much but that's really hard you have to like teach the AI exactly what the code should do and then it matches what it should do with what it is doing and I think at this point we're getting closer to formal verification where you write a specification exactly what the code should do and then match it against the code and I have to tell you from looking at many code bases no one writes that documentation so well it's like it's there was one project I looked at which had really good documentation and it was like a a friend of mine who knows my annoyance at documentation and I think I kind of like pushed him to uh do better so yeah um I think there's room for these tools they are still early and if we start seeing people train the AIS on the correct data I think there could be some more interest interesting stuff that happens um I don't know if we've gotten there yet but hopefully soon yeah a question about Tooling in general so for someone coming into the space do you think there is some like tooling or framew Frameworks that people should learn how to use it they want to be better at security because like in web 2 it's very common to use a lot of tools for quality control and for finding bugs and stuff like that but we three it's not that common so if you want to be a security expert should you just focus on learning how to do things and do it slowly and carefully and be correct or should you focus on like removing human error and speeding things up and stuff like that just in your experience yeah I think uh I can answer this from two perspectives and it it depends on your role so if you're a developer you shouldn't be wasting too much time on learning Security in full detail like from the developer perspective you want to have some continuous integration some automated scanning to detect the obvious bugs uh so from this perspective the tooling can be useful um and there there are some or at least one I think GitHub action from TR of bits to run Slither uh on your codes so I'd recommend doing this uh but from the security perspective if you're actually trying to find bugs I think I would personally recommend especially while you're learning the full manual approach um I will even say from the web 2 security side uh some people who are on these bug bounty platforms where you basically are just clicking around a website and then looking at the code and trying to find a bug and they actually live stream this when you watch their process they do have like they know about all the tools but the majority of what the experts and web 2 security do is just like manual like they just know exactly what to look for and the training for that only comes from manual effort so if you're letting the tools do the thinking you don't always understand the logic so if you want to be fast be a developer use the tools if you want to really learn security deeply Go full manual it's like uh someone driving a car if you're lazy go with automatic if you really want to be a race car driver get the manual great do thank you so much for for talking I work to the same field as a company called n group crypto Services yes I felt exactly the same uh about many things you you said so I guess my question uh uh is about code Arena um do you think it's a better strategy to learn solidity and focus on solidity or ignore solidity and just focus on other types of projects yeah um that's a good point in fact if we go to the contest uh page I will say for anyone who wants to know the website uh I think I tried to include the URL so if if you want to do contests that website Aggregates different contests uh so I'm just looking here if they specify which language on this screenshot I guess not um I will say just uh to add to the the history of security in the space like the The Last 5 Years uh let's say code Arena 3 years ago it was only solidity like uh only ethereum blockchain code smart contracts now they have basically anything uh there can be rust code there can be uh just normal web 2 code um it goes all over the place so I think from the current perspective you really have to judge based on how many contests there are in different languages and what your skill set is um I think there is a lot of opportunity doing non-s solidity stuff if you already have some skill if you don't if you only learn solidity like that's the first coding language you learned then maybe it's not recommended to Branch out because there's a lot of fundamentals you need to learn um but yes I think today the landscape has changed from even a year ago and that's definitely a very valid thing to think about if you have coding skill in other languages um I will say speaking from experience it does seem like there's much less attention paid to rust or other contest languages yeah good observation this is just a demonstration of how quickly the space changes because a year ago it was completely different like a year ago there was maybe two or three contests that were not in solidity but now uh the game has changed and even some of the rules on the contest websites has changed so there's a lot of like uh optimization to be done let's say yes yeah I heard um recently that a lot of big named Big Brand security companies uh are actually one of the worst at doing security Audits and then the smaller lesser known companies are much more effective how accurate is that uh yeah this is ALS a great question because it depends on the people working there and at one point I knew that a lot of people had changed from different companies and so people were asking me like who should I get an audit with and I just had like a list that I copied and pasted to different people in telegram like uh this one's good but like two of their top people just left so like maybe wouldn't recommend it and this is where having some of the insid or knowledge can be a bit helpful because uh let's say if you're on crypto Twitter but only following security people and you see like ah this guy was really good he just moved to this company oh and this person from a different company also moved there so maybe this new company is one to watch um I would say this is like very volatile um I think at one point when there were huge backlogs when was this I think I said 2022 there was a lot of hype over we need the audit done by this big name company and they had really long backlogs I will even name a name uh trail of bits I think specifically had like four to six month backlog at some point um but I think since then and especially with shorter audit backlogs now it demonstrates that like the market is saying that there isn't a single audit firm that's considered the best anymore um there's probably uh I mean I guess I should be promoting uh ones that I think are good but I'm not going to um you can DM me if you want suggestions anyone uh although I guess my knowledge is not as accurate as it was like a year or two ago when I was much more locked into where everyone was uh I think now it is like just recommended to do more than one is maybe one of the changes cuz before it was like do one make it a really big name and you're done and now it's a bit more of like a diversification approach I think famously uh Oiler Finance did over 10 audits on their code base which is uh from the cost perspective absolutely insane because it was like $2 to4 million spent for their code uh but due to their history they wanted to be secure so and they they are doing security very seriously is there a way to kind of evaluate these security companies if you don't have Insider knowledge uh no yeah uh I will say what I do uh from the outward facing side so like if I don't know where people have moved to different companies you can look at their audit reports and judge based on that but that also requires like a security person's knowledge to judge like oh this company the findings and their reports are just generic stuff uh whereas this audit company I read their report and it's like wow those were really hard to find issues so I don't know if you consider that Insider knowledge but at least like security knowledge to judge the reports um I will say at one point I was trying to do actually when what year was that was that last year last I was trying to do an initiative to judge audit quality um but it didn't get too far because it's really hard to create metrics around this so the closest I could get was how many findings per report uh any audit company would publish and then average it across many reports so if one audit company publishes like an average of three High findings per report that might be better than one high finding per report but then you have to see if anyone is gaming the system by elevating findings you have to think about the quality of the code that they're getting maybe they only get really good code at this company and so there's less findings so I've thought a lot about this but um and I actually made some graphs for this attempt and then those graphs got circulated around like oh this is this is how it is these are the best companies it's like no no you have to consider all the other variables but no one did that because it's Twitter so um yeah it's hard is all I can say okay any other questions if not you can catch me after the break and I'm uh often around so just ask if you have any security questions okay that's it engineer thank you