Challenges Developing and Maintaining Open Source Software in Web3

Hi, I'm Everett Hildenbrandt, as was introduced, and I'm the CEO of Runtime Verification, and I'm here to talk to you about the challenges of developing open source software and maintaining it sustainably. Not necessarily in Web 3.0. It doesn't really matter, Web 3.0, Web 2.

0. I mean, there's some different challenges in Web 3.0, but a lot of them end up being the same. So why are we qualified to talk about this at all? We have, you know, a huge GitHub presence, 170 public repositories, depending on how you count, 20, 12 to 37 of them are active at any given time.

It's spanning 15 different programming languages, lots of external collaborators, obviously the internal developers at the team as well. You know, we have dependency chains within that software that run six deep in some cases, and we have different teams that range in size from one to ten. We've grown teams. We've shrank teams. We've, you know, done all sorts of different projects over the course of the last six years as I've been working at runtime verification.

And we have super active and super awesome automation with CI and CD. So what are the challenges? I'm going to start from the easiest challenge to solve, which is the technical one. And a lot of people probably think, oh, this is the only challenge, the technical one. And, you know, with the technical challenge with managing open source software is automation.

You need to automate everything. So you want to enforce as much as possible. We use GitHub for that. You can use GitLab or whatever other version control and publishing software you want to use. We use CI for testing.

So every release is tested. That just keeps us sane, basically. Automated releases and updates. And so we actually have it. When we have those six dependency chains going on, six deep dependency chains, if there's an update to one of our pieces of software, it automatically pushes down the chain to the next piece of software for us.

And one of the kind of challenges that you might notice is, you know, the software goes through different life cycles, right? When you first are developing a piece of software, you're in rapid development mode. You're prototyping. You're trying to move as quickly as possible, you don't want to be slowed down. And that looks a lot different than when you're in kind of the later stages of a project where you're just evolving the software over time, and you have a lot more tests there to guide you.

So you need to accept that there are different life cycles to software development. And for example, keep GitHub enforcement turned off at the beginning of the development, and then, you know, a couple months in or a couple weeks in, turn on that GitHub enforcement. The second challenge is personnel, and this, maybe it even deserves to be the third challenge, because this is, you know, the medium difficulty challenge, but what I like to tell new hires at runtime verification is that learning to program is easy. Learning to program in a group is hard, right? And the key thing to remember is that you have to be respectful of other people's time, right?

And, you know, one thing to keep in mind is it's always harder to read code than to write code. And so you're submitting code changes or something like that, and it was easy for you to read it because you have the intent in your mind and you translated it into code, it's much harder to go from that code and translate it back into the intent as the PR reviewer. So as the person who's authoring code, you need to make sure that your code is, you know, small changes, isolated changes, easy to read, well-tested, so that the PR reviewer has an easy time pressing the approve button. You know, remote work obviously makes this a lot harder. You know, when you're all in the same office together, you just turn around, hey, like, what do you think of this change?

And you work on the same screen. But now we're talking about different time zones, and there's delays associated with that. So you have to keep all that in mind and actually keep it in your mental model for your development. And so here, process is key. Example, at runtime verification, I let everyone know, you know, beginning of your day, you clear your Slack messages first, then you do your pull request reviews, and then you do bulk code development on new code, and then you open new pull requests at the end of the day.

You do your pull request reviews first because that's blocking someone, right? That is causing a synchronization point between you and someone else. They might be on the other side of the world, and suddenly, if you don't do that pull request review, you're delaying them by a day. And then that maybe delays you by a day. And so suddenly, you know, some small change gets delayed two days, maybe three days.

If we're talking about an eight-week project and you have a handful of delays like that, you've lost two weeks on that eight-week project, right? That's not good. Anyway, next slide. The third challenge is money. And, you know, if you know how to solve this, please let me know.

The solution I found is to beg. And, you know, the problem with begging is that everyone starts begging, and you need to have, like, a begging process and have a little structure to it. And so you have, like, grants and retrofunding and side quests where you get people to fund one thing, but then you develop another thing as well. And the problem with structured begging is that it becomes a game. And big players are better at games than small players.

And so then the small players resort back to unstructured begging. And so, I don't know, we're back at square one. So anyway, that's my talk. You know, solve the technical challenges first. Prioritize those.

That's the easiest part. Make sure as you onboard people that you make them aware of the personnel challenges, and then come let me know when you figure out the money thing. Thanks. Thank you. Any questions?

Oh, there's one here. Great presentation. I just want to know more about runtime verification. I was checking the website, but it's much better for you to introduce what you do. Why are you working in open source?

Yeah, so we do mostly formal verification and formal verification tooling. So we have this project called the K Framework. That's kind of our biggest project that is a tool that enables us to do formal verification for any programming language basically And so we have formal verification tools for solidity for web assembly for rust We also have for some other blockchain like Cardano and Tezos and stuff like that so that require That's a big formal verification is hard software to develop, and it's used in a bunch of other contexts as well. And so we just end up having a lot of different repositories targeting different ecosystems and different programming languages that we do and most of that is done for security audits, basically. So.

Thank you. There's a question over there. Thanks. Hi. Because you already mentioned the workflow, especially the GitHub pull requests, so I had a question about that.

Do you have any advice in terms of leading the team and structuring the PR reviews? Because in my opinion, when there is an update or is a bug fix, it's quite straightforward. It doesn't really matter the number of commits, because you look at the change, you look at the description, you know kind of what's happening. But perhaps you have an advice on how to structure the commits when it comes to a new change. And when you said about like the method, like it's hard to construct the intent by looking at the changes.

So perhaps having more commits and following them one by one would help with that? Or you know what I'm asking? Yeah, yeah. Perhaps have it up there. I mean, it's challenging and it's challenging not because it's a non-solve problem.

It's challenging because people just don't do it, right? But I mean, have good commit hygiene. Try to make it so, I try to make it so every commit builds, if not passing tests. It might be that you're going through refactoring and tests have to be failing in the middle there, but every commit should build, in my opinion. And so that should make it so every commit you can review in isolation if that's necessary, which is not the ideal way to do it because that takes a little longer to review.

But then the second piece of advice I'd give on that is, as a pull request reviewer, just be really picky. You know, if someone sends you a pull request that is changing four different things and it's hard to review the whole thing and keep it all in your head, don't bother reviewing it. Just send it back to them and say, hey, you weren't respectful of my time. Break this into four pull requests and I'll review each one independently. Because ideally, as a pull request reviewer, it's easy for you to look at a pull request and say, this is a good change.

I'm going to hit approve and move on, right? But if you can't do that, and you have to now dissect what is this change doing, you're not the expert on it. You didn't write that code. It's much easier for the person who authored the pull request to go and split that up into multiple different pull requests than it is for you to dissect what they meant, right? So don't, as a pull request reviewer, don't be shy.

Just be like, look, be more respectful of my time, please. Go break this up into multiple pull requests so I can review each independently. That's my opinion on it. All right. Thank you very much.

Thanks. Thanks for the tips for PR reviews.