Summarize the following in 5 bullet points.
Title: “Leaked Internal Google AI Memo About “Google and OpenAI””
Transcript: “Google has no mode and neither does open AI this is from a leaked document from inside Google talking about how open source large language models have completely changed the game and put Google and open AI on their heels let’s review the document and see what it says let’s get into it now the gist of this document is open source models are proliferating so quickly and iterating so fast that it’s going to be impossible to keep the existing mode that Google and really open AI have and they talk ab out a bunch of different reasons for this let’s read a few lines from it we’ve done a lot of looking over our shoulders at open AI who will cross the next Milestone what will the next move be but the uncomfortable truth is we aren’t positioned to win this arms race and neither is open AI while we’ve been squabbling a third faction has quietly been eating our lunch the third faction that they’re talking about is open source models GPT for all alpaca llama Mosaic stable LM the list goes on and on and these models are moving so quickly compared to open Ai and Bard here they talk about being able to use these llms on any device these llms have gotten so small four gigabytes really you can put them on any device you can power them with a CPU you don’t even need a GPU and you can fine tune and personalize these models really easily and the cost to do so continues to decrease here it says llms on a phone people are running Foundation models on a pixel 6 at 5 tokens per second slow still it’s a pixel 6. it’s a mobile phone that anybody can get their hands on scalable personal AI you can fine tune a personalized AI on your laptop in an evening here they talk about generative art responsible release this one isn’t solved so much as obviated there are entire websites full of art models with no restrictions whatsoever and text is not far behind and then last multi-modality the current multimodal science QA Sota was trained in an hour they go on to talk about how their models still have a slight Edge in quality but that’s not going to be around for long and they actually show a graphic showing how quickly these open source models are catching up that the closed Source models that Google and openai have open source models are faster more customizable more private and pound for pound more capable they are doing things with a hundred dollars and 13 billion params that we struggle with at 10 million dollars and 540 billion parameters and they’re doing it in weeks not months because there’s no internal red tape to go through layers of approval considerations while being a public company these are just Engineers sitting in a room together iterating and releasing the data sets releasing the models as quickly as they can build them it is truly incredible we have no secret sauce our best hope is to learn from and collaborate with what others are doing outside of Google we should prioritize enabling 3p Integrations people will not pay for a restricted model when free unrestricte d alternatives are comparable in quality we should consider where our value add really is giant models are slowing us down in the long run the best models are the ones which can be iterated upon quickly we should make small variants more than an afterthought now that we know that it is possible in the sub 20 billion parameter regime so check this out here’s chat GPT in terms of quality chat EPT is 100 barred is 93 whether that’s true or not fine so the first model Lama 13B was released 68 of the quality of Chachi BT two weeks later alpaca 13B so alpaca if you remember is an instruction tuned version of llama so alpaca 13B 76 of Chachi PT two weeks later then one week later vicuna 13B which is 92 percent of Chachi PT and essentially equivalent to Bard here’s a little bit of the history at the beginning of March the open source Community got their hands on their first really capable Foundation model as meta’s llama was leaked to the public remember meta did not get it out there it was le aked it had no instruction or conversation tuning no rlhf reinforcement learning through human feedback nonetheless the community immediately understood the significance of what they had been given a tremendous outpouring of innovation followed with just days between major developments here we are barely a month later and there are variants with instruction tuning quantization which allows you to run it on a CPU easily quality improvements human evals multimodality rlhf and more and a lot of the se models are starting to build rlhf into their workflows so if you look at GPT for all’s recent releases they have a beautiful interface now and you can opt in to giving them your conversation history which is rlhf and really helps train the future models and here’s something that I love most importantly they have solved the scaling problem many of the new ideas are from Ordinary People the barrier to entry for training and experimentation has dropped from the total output of a major research o rganization to one person an evening and a beefy laptop why we could have seen it coming so here this person whoever it is starts to reflect on what they could have done and how they could have seen it in many ways this shouldn’t be a surprise the current Renaissance and open source llms comes hot on the heels of a Renaissance and image generation so they are talking about stable diffusion the similarities are not lost in the community with many calling this the stable diffusion moment for llms now generative art was already out there and then stable diffusion from stability. ai was dropped completely open source generative art model that other companies have built incredible products on like mid-journey in both cases low-cost public involvement was enabled by a vastly cheaper mechanism for fine-tuning called low rank adaptation or Laura and combining that with significant breakthrough in scale whether you’re talking about latent diffusion chinchilla and in both cases access to a suffi ciently high quality model kicked off a flurry of ideals and iteration from individuals and institutions around the world and in both cases they quickly outpaced the larger players now they make a comparison between stable diffusion and dolly dolly is a generative art model from open Ai and stable diffusion was released which is a completely open source version and stable diffusion is incredible in fact it’s better than Dolly now here it is the effect was palpable rapid domination in terms of cu ltural impact versus the open AI solution which became increasingly irrelevant whether the same thing will happen for llms remains to be seen But the broad structural elements are the same here they talk about Laura low rank adaptation which basically allows models to be trained and fine-tuned at a fraction of the cost Laura works by representing model updates as low rank factorizations which reduces the size of the update matrices by a factor of up to several thousand a factor of up to several thousand this allows model fine tuning at a fraction of the cost in time being able to personalize a language model in a few hours on consumer Hardware is a big deal particularly for aspirations that involve incorporating new and diverse knowledge in near real time so now we’re getting a little bit into memory rather than always having to provide the full context in a prompt you can get faster and faster fine tuning based on Laura and then here they talk about retraining models so when you have enormous models retraining them takes a ton of time and a ton of resources and the barrier to doing that over and over again is really high and when you have these smaller models paired with the Laura technology you’re able to train and iterate on these models over and over again at a really low cost rather than it costing millions of dollars to iterate on a huge model you can do it with thousands or even hundreds of dollars and normal consumer Hardware this means that as new and better data set s and tasks become available the model can be cheaply kept up to date without ever having to pay the cost of a full run by contrast training giant models from scratch not only throws away the pre-training but also any iterative improvements that have been made on top in the open source World it doesn’t take longer for these improvements dominate making a full retrain extremely costly and here’s a key statement large models aren’t more capable in the long run if we can iterate faster on small mod els the fact that you’re able to test so many new ideas so many new technologies so many new data sets and approaches on small models means you’re going to get to the best solution much more quickly even if the large models are better today and here’s something that the founder of GPT for all told me and here it is again data quality scales better than data size so when you have better data quality that is the key ingredient for having the best model it’s not having the most data it’s having the best data the best quality fortunately these high quality data sets are open source so they are free to use a lot of the data sets that open Ai and Bard are based on are completely free and open now now I think that’s going to change I’ve seen reports that Reddit is looking to charge tens of millions of dollars to access their data set and I bet core is going to do the same Twitter is going to do the same they’re going to be charging a ton of money but right now there are completely free and op en data sets out there that you can use directly competing with open source is a losing proposition who would pay for a Google product with usage restrictions if there is a free high quality alternative without them and we should not be able to catch up the modern internet runs on open source for a reason open source has some significant advantages that we cannot replicate now here they talk about brain drain we need them more than they need us keeping our technology secret was always a tenuous proposition Google researchers are leaving for other companies on a regular Cadence so we can assume they know everything we know and will continue to for as long as that pipeline is open so they’re trying to keep all their secrets inside the company but the researchers are leaving to go to other companies and bringing that research with them obviously they can’t bring any actual paperwork with them or hard research but it’s all in their mind they understand how it works and they can go recreate it somewhere else individuals are not constrained by licenses to the same degree as corporations so another reason that these open source models are able to move so much more quickly much of this Innovation is happening on top of the leaked model weights from meta so basically individuals are not really caring that’s leaked whereas a Google or an open AI can’t go and use meta’s model because it was leaked and it would be illegal for them and they don’t want the legal implications of doing that now here’s something super interesting paradoxically the one clear winner and all of this is meta because the leaked model was theirs they have effectively garnered an entire planet’s worth of Free Labor so their model was leaked and now everybody’s building on top of them and de facto they kind of own it since most of the open source Innovation is happening on top of their architecture there is nothing stopping them from directly incorporating it into their products that is their base data they own it so anything built on top of it they can use without even having to Blink the value of owning the ecosystem cannot be overstated now this person mentions Chrome and Android two incredibly popular Open Source Products from Google Android open source a ton of Engineers all over the world built on top of Android and really Google gains so much benefit from that a lot of browsers are built on top of chromium which is owned by Google even though it’s open source they get so much benefit from t he fact that they get to decide they get to steer the ship on these open source projects and to conclude they talk about what about open AI now why would Google go open source if open AI is closed Source why do we have to share if they won’t but the fact of the matter is we are already sharing everything with them in the form of a steady flow of poached senior engineers and until we stem the tide secrecy is a moot point so all these Google researchers is doing work are just getting pulled into t he open AI company and bringing their mind with them let’s look at this timeline February 24th 2023 llama is launched a mere week or so later the inevitable happens llama is leaked March 12th another week and a half later language models on a toaster so basically they took this llm and now they’re starting to put it on every single device a day later fine-tuning on a laptop Stanford releases alpaca which is an instruction tuned version of llama and they’re able to fine tune it on a laptop then l ess than a week later 4-bit quantization happens on llama and we can run it now on a MacBook CPU a GPU is not even necessary March 19th a day later cross University collaboration releases vicuna which achieves parody with Bard March 25th GPT for all is released you get to choose your own model it works incredibly well and it’s just rapidly getting better March 28 cerebrus’s release another open source model March 28th the same day puffed parameter efficient fine tuning technique is released Apri ating it’s only a matter of time if you like this video please like And subscribe I’ll see you in the next one”

– A leaked internal memo from Google discusses the impact of open-source large language models on the field.
– The memo highlights how quickly open-source models are iterating and how difficult it is to keep up.
– The memo mentions that closed-source models still have a slight edge in quality, but this gap is closing quickly.
– Open-source models are faster, more customizable, and pound for pound more capable than closed-source models.
– The memo suggests that Google should prioritize enabling third-party integrations and focus on making small variants of models more accessible.