From PeerDAS to FullDAS: towards massive scalability with 32MB blocks and beyond | Devcon SEA

[music] Hi. Hello. So yes, I have a long title. Uh so I will be speaking about data varity sampling and uh it's a long title and I didn't want to make it longer but actually I have to make a small demalic. It's not even blocks.

it's data variability space that we are speaking of. So let's see who we are first. Uh I'm Chaba. I'm from the research team from Codex. Uh we are working on uh the centralized storage network providing high dutability guarantees and our research team is also working with DF on data varity sampling and also uh the client teams.

Okay. So I think you are all familiar with this chart. You have seen it like 10 times during this Defvcon I suppose uh we had the IP 4844 and the transaction fees on L2 went down right and the right side we s see those little little ones uh that's very good that's very nice of uh 4844 but I hear that we have a [laughter] problem uh and uh yeah to see that problem I have to zoom in sorry so if we zoom in we see this problem uh So there is a slight increase in L2 transaction fees and actually there's a problem. Uh L2 transaction fees are uh driven by uh blob space by datability space and that's running out. It is a problem and if that happens prices can rise fast.

There are other data solutions uh in codex we are also building one. Uh but here I am to speak of the Ethereum datability solution. So we should scale that Ethereum data with this solution. So uh yeah let's look a little bit behind these numbers. What you see here uh maybe you have already seen this one as well.

This is the daily average size of blocks uh on the network and you see that it was rising and then uh after the merge it started to rise and then uh with 4844 it uh reduced. So this looks fine right now. You see this less often. Uh this is the problem. So this is what our preparators are seeing.

This is the blobs. Uh so this is the blocks and the blobs commumulative plot uh together. So this is the data mount that they are seeing and now we can starting right uh there is more data to handle. But we are scientists we can look at this different way. So let's see it on a log scale and then it's not that bad actually like it's like just like more slow it's actually a different exponent uh but we are better than those CPU guys so uh why would worry okay we just have to do that little technology evolution which keeps this running and then we have a solution right we can zace and Uh and that's a plan and here's a plan.

We need to build us and then food us and we just have to be a bit lousy about the scale. So it's an unknown scale here and no screen there and then uh we solve the problem. What we want is more B space and what we want is less node traffic for that operators. Uh yeah we just have to work out these little networking details. So uh let's see those little networking details that we have to work out.

Uh but first let me uh have a quick detour networking. Uh but this is I was speaking of average box size till now. Uh but average is is a was a daily average and in the day many things things happen right every 12 seconds something is happening. So what you see here is the tail distribution of block size uh during a long period. You don't even see the small blocks you just see the large ones.

uh and this was before 4844 uh and what you see that there were blocks beyond two megabytes and uh our nodes have to handle those what if two blocks of such come one after another and you have to handle that. So you need our technology have to handle these things as well and we were measuring uh whether you can handle that and the good news is that uh you can handle that. So the network was was running and live. And what you see here is uh for different block sizes. These are the bins the smallest up to 32 kilobyte blocks and then between one megabyte and 2 megabyte.

How much time nodes need to get the blocks and the bigger the block the more time is needed. This is a distribution. So the nodes which are receiving it only here and there are many nodes which are receiving it before everyone almost everyone is receiving it before 4 seconds. So things are fine. Uh but clearly doesn't scale very well as you seen if we want to increase the block size then it would go to the right.

So we arrived to a point how to scale layer one from the networking side and we can do different things. Uh first of all we can change the slot time the slot timing window and the internal timers. uh we can change resource requirements. Uh it's a good discussion. Uh it's a bit controversial.

We have small gains. We have a lot to lose. So let me not go into that one. Then we can change uh the gossip sub the protocol uh the networking protocol. We can do improvements on gossip sub for large messages and this is in the making.

Uh it's important but this is not my talk today. Uh and it's relatively small gains compared to what we need. Still important. Then we can do sharding and data with sampling and this is my talk today. Uh this is large gains I hope and then there's the first thing we can do distributed block building.

Uh this is kind of intertwined with many things. Uh but promises large gains but basically that would mean reyncing the whole data flow of the block transaction from submission of the the transaction until the end. Okay. So let's focus on point three the sharding and data sampling over the network. So uh data sampling uh let's just key concepts.

So you need to prepare the data. We were speaking of blobs. Uh there are blobs. There's a package of blobs. Uh and then we need an encoding.

We need an encoding because we want to sample from it. And for the sampling to be effective, we need a coding. For the record coding, we need segmentation. So we need to segment it, chop it to pieces. We need to extend it with the recording.

Then we have to commit to it so that it cannot be uh [gasps] nodes cannot send four pieces basically. And once we done this we are ready with the data preparation. Okay. And here we have different structures. uh we have what we call the one-dimensional per structure where you have blobs which are extended and to and committed to or we have the two-dimensional structure where you have blobs which are extended in two dimension.

Uh if you look carefully you see that actually this is also two dimensional. The difference is that it's not extended by an additional code in this dimension and that has consequences. Okay. And then you just have to sample. It sounds uh easy, right?

uh the question is where you are simpling from and how you do this simple sampling so I wasn't saying anything about that uh so let's just see uh that okay so we will typically speaking of das das uh but it's actually a da plus an s so that is the da part which is a property of the of the of the network it's a global property the data is available the data is not available and to have the data available in the system I have to disperse it in the system. So I have to send it out different pieces to different nodes which are custodying it and only then I can start sampling. So this is the DA part. It is ensuring data ability. It is doing the sharding.

Nodes are only receiving pieces. And then there is the sampling part. Once the data is in custody at different nodes uh you can start test sampling asking for pieces and getting pieces little uh graphical representation. So uh you have a builder it does here in the example I was doing it with the 2D encoding. So it does the 2D encoding and then you have those beacon nodes in the system and what you are doing you are sending out these things uh pieces of this actually lows and columns to different nodes.

Someone is getting only two lows and two columns. Someone is getting a bit more. Someone is getting the whole thing because he's running 100 validators and he's securing so much money but it's better for him to get the whole thing. Plus doing that he can support the system. That's one side.

That's just the data vility. Now you have to sample and that's an individual node and actually every individual node has to sample. That's his own view. So he's picking a few pieces and then asking for those from custody. So getting this piece from here that from there that from there and if it manages to sample then things are good.

Okay. So we have this kind of two sides of things and you can see that on the network these are a little bit different and we need them to behave differently. So we need them to be secure, robust, fast and cheap. And we need sampling to be secure, robust, fast and cheap. Um so we need the same properties for the two things.

It's just uh different protocols. Okay. So I was saying that there is puras and there is fdas and there is difference in the data structure. Actually there are a number of differences. Uh if you scan that QR code we have a light upon on fdas which is starting from uh from puras and uh some of these techniques apply to puras some of them to fus.

So what you see here is podas and in piraas you have one dimensional extension and that also means that when you are sampling you cannot just get a small piece because it would not tell you probabilistically enough. You have to get a full column. So your sample is a column which is more data than a cell that you can sample if you have this two dimensional extension. uh but to make this work you need changes on the networking, changes in the coding, changes in many things. Okay.

[snorts] So let me just give you a comparison of Pas and FAS actually let me go back and and just go through this list. So we have different techniques here. Uh the first category is about sampling is about how I'm selecting what to sample. How I'm selecting those small pieces that I will be looking for. Then the second category is about uh al coding and the using of the coding because what I can do is if the data is coded I can use that as part of the transmission.

So when I'm sending the data, I can already use the Azure coding to recover the data and then send it on to others. And that's a very important property. We are always using aure coding when we are doing transmission. Actually take your mobile phone, take whatever. Uh here we can use a coding uh very efficiently to send data and when we are sending a row, we can take a piece and we can send it on a column.

And when we we have when we have received half of a low we can extend it uh to uh to the whole row for example uh and then there are techniques which are protocol changes and I'm not going through all of them just showing you the comparison. So peel us and fool us. Uh so one is cheap, robust, fast and secure and the other is cheaper, more robust, faster and more secure. And I was inclined to stop here. Uh but let's see a little bit more in detail.

Okay. So food is cheaper. Uh it's cheaper because it's as I shown it's sampling at the cell level and the cell is 512 bytes. actually it's 590 bytes because it's the data plus the proof uh which is still much smaller than than a whole column. So whenever we are set uh so the sampling uh becomes cheaper it's more robust uh because it can do a coding which is allowing uh what I was calling local repair but uh what local means uh is important to understand local means in the sense of the code in the sense of let me go back here in the sense of this square so I only have a few data I only have half of a column and I can repair something.

I can generate these. Well, I have these and I can generate these. Well, I have a half of the pieces here and I can generate this one. I cannot do that up there. And that's a huge difference.

That's a huge difference because that means that every single node can do repair not just big nodes who are having old data. Uh but every single node actually in that construct we don't need big nodes. uh which by the way we have in Ethereum so it's not like we don't have them uh but we are not relying on them doing the repair in the in the data structure and it's faster why it's faster uh because we are working on new protocols and uh both for the popsup part and both for the sampling part. So and yes uh is it more secure? Uh so the light size is still in research.

Uh should it be more secure actually when this is secure enough uh that we are changing to this? Actually it doesn't. Uh it shouldn't be less secure. Uh that's still in research. This is still in specification and changing.

So uh let me not do more comparisons because this will change and actually this is still in changing. Okay. Sorry. Uh last slide. As I said, it is not just pas and food.

Uh but we can go beyond. And uh to go beyond, we have to see the data flow. So how is this data flow? What is happening? What I was speaking about is what's happening in the consensus layer.

So when a block is generated, how that's distributed? But actually these blobs are coming as transactions from the world. Okay. So let's say one blob is coming in and that goes to the execution layer and the execution layer has the meool and the meool is the distributing this sharing this blob between nodes. Actually when it's blobs it's not shared like when it's normal small transactions.

When it's normal small transactions, these transactions are pushed to others. When these blobs is just information that I have this blob and then the others are pulling it over effect, these blobs are splitting in that network. Let's see there is another blob that's splitting also in the network and then there is one node which is selected as block proposal. takes the blobs, packages them, does the encoding and then the encoded version is getting spread in the consensus layer. Everyone is getting his please.

I think you start to see the problem, right? It's not the problem. It's an inefficiency. We have an inefficiency in the system in which we are distributing the data here and then we are actually redistributing the data there and is the same data. So we can optimize it a lot which is good right we need these optimizations for moose row to happen.

So uh we found another place where we can save a lot and have that curve go up. So what you can do in first instance is that we do the distribution up there but whenever the node is realizing that he has a piece he would need this and this but he has the blobs he has the data he can just pop it up from the execution client because the data is also there. Now, of course, this plot is optimistic because I was putting this white blob almost everywhere and I was putting this red blob almost everywhere in the network. In reality, you don't have every blob everywhere uh in the execution layer because if we we would we could do that like now you actually have but when we are scaling we will not be able to do that because if we can do that then we don't have a problem. every node can handle all the data which is there.

We want to scale beyond that. So in reality you would have some blobs here, some blobs there, some blobs there. But then when you do that distribution data can pop up and uh that's one of the optimizations. Of course that's a simple one. We can do much more complex ones where we have interaction between the two and we are avoiding some of these duplications.

uh but for that we have to reync uh the networking stack here which is that P2P here which is using P2P we are doing different kinds of gossiping distribution uh this is using gossip sub this is using other protocols so this is all uh this can all be the same uh but that's just starting okay thank you [applause] Okay, we are now here to take some Q& A. Um, so starting from the top, there are blob fees that play blah blah blah blah blah blah. There are blobies that [laughter] pay for blob storage, but is this verified at all? What happens if I get the fee but don't actually store the blob data? Um, okay.

Block fees that pay for blob storage. I don't know. Uh yeah, I'm not sure what the question is about. No worries. [laughter] Question answer, question asker.

If you have the question still, feel free to clarify. Clarify. Um but I'll mark this as answered. What are standing problems that hinder current adoption? Okay.

So um it's just in implementation. So the podas is in implementation. Uh it's still uh compatibility problems. build will be here uh very soon. It will not be in pect but it will be in the next work.

Uh for fas we still have to figure out uh many things. So uh there was that question mark on the on the on the security uh and we have to work on that to make sure that there are no issues with that. Cool. Why do we need 2D eraser coding? Can't we put all of the points on one polomial and extend it?

So putting them all on one point and extending it would be uh one huge uh code uh one dimensional and that would be very bad for repair in the sense that then you can only repair if you have all the data and that just uh means that uh you have difficulties repairing. The 2D has this advantage of having this called local repair property so that the code distance is small and you can uh you can repair pieces. So you can repair a single column, you can repair a single row and that gives nodes the possibility to rep start repairing and contributing even before getting the full data. If full DOS is way better than pure DOS, why not choose to implement it in the first place? Uh so as I said uh pudas itself is changing we have we were thinking of techniques for food then we realized that they are actually applying to pudas and then they are part of pudas uh we still changing that uh fas is still much more research uh it's a simple fact of life that uh there's too much pieces still moving in fas uh to implement that.

Cool. To cap us off, it seems full approach are much much better than pure das. Should we skip pure and implement the full dosas? Is there any other tradeoff? I so fas is using many things that pas uh does.

So skipping is is not a a dialect skip. It's not fully different. Uh it's it's giving it to the code. So it it has advantages but it has also complexities. uh whether we skip or not is uh might save something on the implementation.

Uh when we are introducing it, it might also skip something but we are losing also the graduality. So overall I'm not sure it's good to skip uh but we can uh we can think of it once food is fully evaluated. What are open research questions? So uh the open research questions is uh so the last slide was was was fully open uh this question or everything around that uh the on on food does uh there are several questions around the sampling how we can do the sampling safely in the sense that civil nodes cannot attack the system for example. So uh there are still things to clarify there.

Uh especially attacks, network partitioning and other things how they affect security. There's some fighting and uploading going on. This is fun. If 2D is better, would there [laughter] be any benefits of 3D? Absolutely.

Uh multi-dimensional encoding is fun. Uh [laughter] but no, actually not. So we just need the uh the local repability property and we have enough of that in 2D. 3D would not contribute to that uh too much. Uh can full be implemented progressively?

I think so. But we didn't define plugless. So uh it's something to think of but uh but we didn't define it yet. Cool. And then I think this question was already answered.

I think we already had that right. Yeah. Unless someone's aggressively or I just forgot to mark his answer. Um, cool. So, we have a minute and 34 seconds left.

I was spending time. Yeah. No, but so I guess there's one piece to this question. You can give him feedback on the QR code. So, we'll give you some time to tell them how great he did.

Collect a card. Um but other than that I think our next session um well I