Lessons from the Nomad Hack

foreign [Music] I'm super happy to be here or not so happy um I would prefer to talk about something else but you know we do what we can right so um first we have to start with the basics I'm not here on behalf of illusory which is the company behind nomad so the views Express are my own and do not reflect those of the company as it's an active uh investigation right now an active incident we will be not doing any q a unfortunately um my name is Odysseus I'm a protocol engineer in Nomad I've worked in web3 as a consultant and also been working as a devops in iot so today we'll be talking about Nomad what is Nomad how the protocol works we will need that to be able to talk about bridges how they work on the incident and finally what are the learnings we have what did we learn from losing about having a hug has resulted in a 190 million dollars in tokens being evaporated Nomad is not a bridge Nomad has a bridge right but if no one is not a bridge and has a bridge that's namad is an optimistic protocol first of all which means that it has an optimistic security model it's a it's a protocol for interpretability which means that it allows applications to be able to meaningfully react to an event that happens in another blockchain right meaningfully that's super important we don't Define how your application will react to some event basically The Nomad protocol will just send arbitrary bytes from one domain to another so it's you the developer to interpret those bytes so Nomad is an optimistic protocol for interpretability that supports arbitrary messages between domains how Nomad works the first thing you have to know about Nomad and probably the last um it's that it's super simple on the sending chain all the messages that are being sent they're added to a metal tree right so and why we do that is because it's very easy with a medical tree to prove whether a methods belongs to the tree or not the information of inclusion in theory and in practice is included is compressed into the root so the protocol only it only has to do is to send that route from the sending chain the receiving chain if we do that securely then anyone can go to the receiving chain and prove that the methods indeed belongs to the tree that has that root so all that the protocol does really is that's passing this route from the sending name domain to the receiving domain domain is a chain right let's see the life cycle of a message denomad first we go to the home contract uh if you see here the home contract on the sending chain and we send the message right then a new route is generated then that route is relayed to the receiving chain and you will find itself in a contract called replica then we must wait for them to be some optimistic window to pass afterwards we can go to replica to the receiving domain and say hey here's the other proof of inclusion here is the message I want to prove that this message was indeed sent and after we prove a message we can process it and we will process a message basically uh the replica contract will take the message metadata call the contract uh destination contract and pass the message payload super simple now that we have in a very basic sense how understand how Nomad Works let's talk about bridges what's a bridge a bridge is a super simple application built on top of any interoperability protocol basically you go to uh the sending chain right let's say ethereum you go to the contract and say hey bridge I want to send my native tokens with for example it's a native token to ethereum it has intrinsic value and I want to send it to Paul to evmos right then the bridge will hold that width and it will send a message to the receiving chain to The Nomad Bridge receiving end and say hey you should mint you should create a new mud with a representation Dragon which doesn't have any any value in itself the value is derived from the fact that we can do the opposite that the user that holds the mod with can go to the bridge and say Hey I want to burn that token and I want to transfer my mod with back into ethereum so the bridge will send a message back and say hey unlock and your wave hug is when the all this locked collateral in the bridge is stolen right so now all these representations that are flying around they're essentially worthless because they can't be redeemed for the original asset for the asset has intrinsic value and I think that's why we have we've seen so many hearts and bridges not only because they are indeed complex systems and they are but because also they have so much collateral locked inside them they're very juicy targets they make good targets for hikers to pry and test for the vulnerabilities foreign let's talk about the incident what how was the number of the bridge possible to be hacked we'll talk about two mappings in the replica contract two pumpings was all it took the first mapping connects a route to the timestamp that says that after that time stop that route is indeed valid you can start proving messages against the root the root of the medical tree right so a new route comes and the new timestamp is generated so when now I can go to the replica and if the time time has passed I can say hey here's the message here's the proof I want to prove it against a valid route and if that happens then the message is proven now the message and uh when proving a message you connect the uh the message which is about 32 the route which was proven under solidity and the default values of marketings all the default value for byte72 is zero and for a number is zero as well so in a way if you look at the second mapping here in a way all messages are proven under the zero root proven but of course in the code we uh you know we we set an authentication flow that said that of course this is not a valid uh of course this is zero and if the uh the time step is zero then it's not valid because also the number zero is the default value the problem is that when we deploy the contract we set that to one non-zero value so what this results is that the root zero is active After Time step one and all messages are proven under the root zero so all messages are proven against an active truth so what users did they forged messages that were meant to the bridge that told the bridge hay and all that collateral and see any arbitrary methods that has not already been proven is proven under the root zero it could prove and process whatever they wanted and that's right there 190 million bug so why did it come up now right The Nomad protocol has been active for months we had an upgrade and during the upgrade we changed the semantics of the second mapping so we used to store an enum here like numbers numbers so if the number is one it's proven if number is 2 is processed so we didn't connect messages to the roots under the under which they are proven right so even if the route was active uh we you know through the authentication flow we made sure that we didn't authenticate that but we changed the semantics and that is why the bug was so hard to find our testing didn't find it our Auditors didn't find it because it needed the old state with a new code it's an old state with old code secure new state with new code secure all state with new code not secure Nomad now the protocol is closed will restart the bridge that's the hard part how do we restart an uncollateralized Bridge uh we'll be sharing more information soon uh the tldr is that users will be able to access to collect some of the recovered funds as the recovered funds and continuously flow into the bridge and they will do so fairly uh you'll be able to read more about it in our coming weeks in our blog posts and Twitter accounts so what did we learn what did we learn from this um I like history Bismarck says that stupid people learn from their own mistakes wise people learn from the mistakes of others so be wise we'll talk about not improvising that and overcome but we'll talk about test observe engage and communicate uh you can think about as different layers in the defenses of a castle right hopefully uh these layers of defense will stop the attackers from reaching the Citadel the king or queen it's like the treats his analogy uh security I'm sure most of your your researchers will know about that security um analogy but I think castles are way cooler than cheese that's why I prefer to do castles yes the bread and butter of every developer right um there are although there weren't any best practices I think the industry is now uh slowly aligning on these um on some you know best practice on this so we're having the unit tests property-based tests integration tests forking tests and Environ tests and I want to go a quick rundown through them first of all concrete tests super simple I want to make sure that the function if I give it five I will get 25 we also name it concrete function Concord test property-based tests are more advanced they force you to think about the properties of your code so basically to verify that this function will give the input multiplied by 5 always then we have the integration test where we want to test bigger picture uh features user flows um working test which is like a web free specialty all the other tests you can find them in other paradigms a forking test basically could be an integration test or a unit test but we test against the on-chain state and this is very important because as you saw a bug can Surface itself only using the Unchained state and finally warranties my personal favorite environments are these equations phrases that should always hold for your protocol right if that phrase doesn't hold at some point the protocol should be posted the protocol has broken in some way so this is a big project uh you'll do it in two phases first phase you will Define the invariance so it's a very theoretical phase not easy definitely and then it's testing the variance using any tools there are a lot of tools out there for example in Nomad the environment that broke was that all messages that are processed right received must have been sent people were processing messages that were never sent this environment broke and finally I would like to mention static analysis tools this could be very useful especially for neural developers as they can find some simple vulnerabilities and storage layout analysis basically you can use tools to verify that the storage layout of your upgradable contracts will not change uh Without You noticing and this is a very common source of vulnerabilities in my view you should prioritize unit test obviously a property based test um on forking tests this should be your primary focus with tools like Foundry boundary uh it's very I think it's easier than it used to be of course you should always audit you know use a do an audit although it's not a silver bullet so don't just do audits and of course always take the source layout always uh uh observe the second pillar now we have tested alerting if you receive a hero passage and you aren't already up it's too late like your alerting has failed you shouldn't wait from a certain Twitter account to tell you that there's a problem with your protocol this is a solved problem in web 2 web 3 would like to invent things again and again but it's a salt problem in wealth two so go and read the Google series handbook you'll get a ton of input also talk with your devops engineers if they have worked in web 2 before they will have a lot of insight for you uh usually start with an object uh you know the business objective and then Define actionable alerts actionable that's a keyword actionable that means that if an alert a is activated then you should do B it should be a very simple if that if then you know close you should have a playbook for every alert so if alert a happens you should do that and that's the way you should do it and here's the script you should run super simple if you don't do that you will get alert fatigue because you'll start not paying attention to alerts because you know it's not that important and by not paying attention you will miss that alert that will erect your protocol um a nice mental model for Alert in web free I think is uh heuristics and environment-based alerts characteristics are rules uh they require human intervention to make to understand whether that's a false positive or not are environment-based alerts where you have enough chain agent running and continues taking the variance of the protocol more complex but can be automated because environments shouldn't produce false positive like if the environment alert is on that means that either you didn't Define the environment properly or be protocol is broken another way probably it's a good thing to pause the protocol now engage testing has failed alerting maybe so now we're in the engaging we're in the first minutes of the of the incident what do we do uh apart our hello who's told us that would not rise to the level of our expectations we fall to the lever for training so if you don't have don't have an incident um Playbook if you haven't gone through that you will not be able to uh you know act appropriately and you will get wrecked even if your alerting was good so a good uh Incident Management means explicit ownership very specific persons should have very specific ownership of the management of the incident outcomes every person should know what should be the outcome of their work during an incident Game Day game day game day do that again and again go through simulations the entire organization should create game days for this incident because they will happen they will happen and you shouldn't be uh you know during the incident you shouldn't read the incident playbook for the first time uh yearn internally has a very nice blog post about it I highly suggest you look at it and you adapt it to your organization within an engaging proper in proper time we lost money right so it's now time to face the music what do we do I'm sure everyone would like yeah let's talk to the users you know we have to be transparent we have to tell them what happened we have to be no uh tell them before the region wrecked no nope you shouldn't talk to anyone to talk to your legal team you don't do a commit you don't do a tweet you don't to your mother if you're done with your lawyers you should think about what the users want but you should talk with your lawyers first when you do communicate you should be honest no sugar coating you know you just insult them you should have pre-approved messaging because that's easier like during a crisis after the first hours you're shaking you haven't slept you're high on caffeine you can't properly create you know communication or PR output you should tell them what you do what are you doing right now and what you plan to do let's see a quick timeline of the first days after an incident first of all we talk with our lawyers and we form them of the situation so they can start talking with uh law enforcement agencies for asset recovery we do the first batch of Public Communication we tell them what happened what do we want to do what are we planning to do we do we talk with our partners ideally through more privileged communication group like telegram maybe we can share it more than we share with the public because of ndas and all that uh because as you are losing money they are losing as well then you should talk with a chain analytics firm that is very important I will tell you why because apparently people they suddenly have a change of heart when legal enforcement agencies are zeroing in on your the real identities right suddenly they just want to return their funds back and the only reason uh and the only way for uh to recover assets is using a chain analytics firm as we go on and analyze all these transactions and find correlations and they would be able to point uh the law enforcement agencies to centralizing changes so they can freeze funds and requests of data personal data so if you want to recover your assets you will need a chain analytics firm then you will set up a recovery address where you can you will start collect the funds and align with your team on the Bounty and of course the bank has many legal externalities so again your lawyers will be your best friends