Open + Decentralized AI

[Music] unmute there we go hello everyone I'm sorry I couldn't make it today um but it seems like everyone's having a great time in Thailand uh bit of introduction to myself I'm aad mustac uh previously a CEO founder of stability AI where we had about 300 million downloads of our open source models from stable video stable audio but most famously stable Fusion most popular image model in the world at one point we were running about 10,000 A1 100s which is one of the top 10 supercomputers and it was quite a ride and interesting but the original intention of stability was to be decentralized it was actually orig me to be a dow of dows um and earlier this year I had a big think about that and the future of AI and where we're going so in this talk today I'm going to touch on a few elements of that and try to frame it and uh how we can impact that now with my new organization shelling um we've kind of honed down to we're moving to a post labor economy so let's try and build um AI money to stack gpus and increase the intelligence by Building open source models and supporting others so the demand side of dpin that's a story for another day perhaps I think that there are three types of AI model and this is where it becomes very interesting uh the one that's been most famous has been these private models these chat gpts and anthropic claes and others these are like expert systems trying to achieve AGI kind of do everything and the knowledge that they have in the data set they have are private um that's interesting but I don't think it's that exciting honestly uh in the middle we have a different type of knowledge which is public knowledge that is licensed so that's Entertainment Innovation licensed data and weights and I put metal L model under this uh you know open weights but you don't know what's inside it on the left hand side I think is the most interesting which is common knowledge and we recently wrote a piece how to think about AI when we like models are like graduates and uh you know they're going to come out as doctors lawyers Etc every industry that's regulated is likely to require open source and open data models um and the stuff that's regulated the stuff that we need for living which I think is the most interesting part there was a very interesting paper by anthropic um at the start of the year called sleeper agents where they showed why it's important to know the data that goes in uh as and what are the ingredients how the sausage is made with just a few thousand words and trillions of words of data going into models you can make it so it turns provably evil with a single indicator and you can't tune that out and you can't identify it beforehand so I think rather than having black boxes the models that we have to run our lives will be open source open data the question is who will build those and who will coordinate those because the reality is models are reaching a bit of a plateau because you can't have that much higher on language understanding unless you get to uh full AGI so this is closed versus open weight models but models ultimately are a function of their data and the data sets are getting better and better and again this Gap is closing the output of models as well is getting more and more structured and more and more deterministic so we designed stable diffusion to be deterministic you put the same seed in you get the same output roughly now you're getting towards 100% on structured outputs so these little models which is a few gigabytes can transform any incoming data in a reliable deterministic way when combined with large context windows that makes a lot of the problems that we've been facing as a kind of community a lot easier and you should really look into where structured outputs are so open AI gp4 is now pretty much 100% on structured outputs so you have increasingly intelligent models that are getting more and more structured more deterministic um and they getting a lot cheaper so Frontier AI is at a 100 times drop in price um and that's amazing and that's a combination of faster compute but then also algorithmic optimization but I think what's more interesting is AI equi intelligence so this is a chart from Andreas and Horowitz of the cheapest LM with the minimum m u score uh indication of language understanding again a above 83 is that graduate level above 42 is you know it's okay it's not that smart um and I think this is a log on the left hand side so it's getting increasingly cheap and what does that really mean well for a start that equi intelligence level is getting above 100 in IQ so you have 120 IQ on open ai's 01 model um and that's quite something this is why they're looking for data sets um for example XI those of you that are IMO medalists they're hiring to train their model to generate synthetic and human-made data and that's quite something because o1 is probably a model that's maybe 30 gbes big and yet it has an IQ over 100 to put that in context this is average global IQ by country so where's Thailand on there you know you're in the 89 or so um the pairing of humans with AIS that have IQs above 100 will be incredibly interesting and I think this is what we are going to kind of look at as we think about how do we position for an open and decentralized future this chart however isn't because anyone's stupider than others it's a question of infrastructure so this is a very interesting chart it's energy versus wealth so on the left hand side there's energy consumption by capita and then there's GDP per capita if you don't have the energy then it's very difficult to build the institutions and Frameworks that you need to have that level of intelligence to have the schooling to have the electricity to have the power I mean leenu in Singapore said one of the most biggest advances in there was introducing air conditioning because it's hard to think when you're hot you know but this connection between labor productivity and energy productivity is being broken by these models because to remind you of kind of that drop in intelligence that purple line to run llama 3.2 3B requires 5 watts of electricity the original gpt3 required 1,000 wats of electricity so that's a massive drop and this comes at the same time as this chart here which is the solar PV cost per dollar it's now less than a dollar per watt of solar PV and a gpt3 plus to GPT 4 level intelligence only takes 5 wats of electricity to run the compute required to create it is in the order of magnitude of 10,000 A1 100s and I don't think that will exceed 100,000 A1 100s to give you an idea of how 100,000 A1 100s or actually h100s shall we say Elon musk's super compute cluster uh Colossus that's the total emissions of bit tensor over the next year you know it's about week of ETF inflows into crypto so I think that the compute side of things becomes interesting the energy side of things equi intelligence and again let's bring intelligence of 100 IQ to everyone so we can lift everyone to 100 IQ is interesting and it's within reach to decentralize this technology as in have it completely offline a solar power plant a solar power panel sorry plus a chip gets anyone a GPT 40 level AI in a few years and that's really interesting to think about because when it comes to you know cryptography and what we do and blockchains I think those are amazing for verification coordination and resilience and when we think about that leftand side open source open data models for the stuff that we need for living you know the stuff I think many of us actually join the industry for then we need to have verification of inputs and outputs and now again we can have deterministic ones we need coordination to create the data sets and to bring this technology widely you know the this incentivization we want it to be resilient we don't want it to be censored you know we don't want it to go down when there's a server error asual Etc and interestingly with a few gigabytes and a few solar panels we now have the ability to distribute intelligence to the edge with minimal interaction between them so I think that we're moving from this era of scale and again I think 100,000 maybe a million h100s are all you need quite on that in a few years um I think that the scaling hypothesis is showing minimal gains versus the data underlying and the most interesting thing now is the spread of this technology because you're moving from a research artifact that was interesting to stuff that's genuinely useful AI models that outperform humans on diagnosis and empathy in medicine AI tutors that are showing massive impacts 50% of certain industries are now using Ai and looking at this spread and the feedback loops from that that's how you build better data and better models so when we think about open and decentralized I think you know you got not your models not your mind I like to think that that's the equivalent not your keys not your crypto I think open models will run the world it will be the infrastructure upon which our societies run and I think you know let's make them awesome and let's think about leveraging all this technology and Innovation we're thinking about to do that because I think open source and decentralized AI has a massive advantage over these centralized ones because you need that spread you need that reach and as it goes into everyday life and we Leverage and the systems infra we've built we're in Prime position to help build the infrastructure of the future to make a difference in people's lives by increasing the intelligence of every person of our institutions and ultimately Humanity so thank you for that and hope to see you all soon take care