AI Roundup 077: A good week for open-source
July 26, 2024.
A good week for open-source
Meta released Llama 3.1 405B, the first open model competitive with GPT-4. A day later, Mistral released Mistral Large 2, which had similar performance despite having three times fewer parameters.
Between the lines:
In addition to the upgraded performance, the new model family also sports a 128K context window and multilingual support, with multimodal inputs coming soon.
The release is a win for open-source AI, especially since Llama allows developers to use its outputs to train other LLMs.
But there are still some reasons to be skeptical: the Mistral model is only "open" for non-commercial use, and Llama requires certain naming conventions for downstream models.
Elsewhere in frontier models:
Stability AI unveiled Stable Video 4D, a model that generates videos from eight new perspectives based on video input.
Google Research detailed NeuralGCM, a model that can rapidly, efficiently, and accurately simulate Earth’s atmosphere.
And OpenAI published Rule-Based Rewards, a method for automating model fine-tuning to ensure safe behavior.
AI IMO
Google DeepMind has solved International Math Olympiad problems at a "silver medal level" thanks to two new models, AlphaProof and AlphaGeometry 2.
Why it matters:
"Silver medal level" means better than roughly 75% of competitors - a very impressive bar, though the AI did get more time than a normal human.
AlphaProof uses Lean, a programming language that can create verifiable proofs. It generates ideas for solutions and then only uses ones that are possible in the proof language.
There was also meaningful improvement over the first AlphaGeometry, which used a synthetic training approach to solving geometric problems.
Elsewhere in the FAANG free-for-all:
Microsoft unveiled a Bing generative search that shows LLM-generated answers with sources available to a small subset of users.
Google updated its Gemini chatbot with Gemini 1.5 Flash, featuring a 32K-token context window in its free tier and links to related content.
Meta plans to roll out its AI assistant to the Quest in August, in experimental mode for US and Canadian users.
OpenAI teased SearchGPT, a Perplexity/Google competitor that can organize links and summarize findings (though it is currently only available via a waitlist).
People are worried about training data
Since ChatGPT's launch, many have protested the use of public Internet content to train models. Now, we have a measurement of the Internet's response: 28% of the web's "most actively maintained, critical sources" block AI scrapers.
The bottom line:
Even if you believe it's okay to scrape YouTube videos or other data without consent, this is a tragedy of the commons playing out in real-time.
That's not to mention that AI companies can be outright disrespectful with how often they're hitting servers to grab content that's fair game.
The logical outcome seems to be a balkanization of the Internet; walled gardens that are only searchable by companies that have paid licensing fees.
Elsewhere in training data:
A new study found that AI models collapse when trained on recursively generated data.
Meta received an EU request to pause future AI model training on data in the EU.
And academic publisher Taylor & Francis sold authors' research to Microsoft AI without informing them or allowing opt-outs.
Elsewhere in AI anxiety:
Anthropic came out against California's AI safety bill but suggested several amendments that could change its position.
SAG-AFTRA announces a strike against major video game companies, including EA, over AI protections in contract negotiations.
Meta's Oversight Board wants to refine the company's policies around AI-generated explicit images.
And AOC's Deepfake AI Porn Bill unanimously passes the Senate.
Things happen
Airtable launches Cobuilder, an AI tool that generates apps from text prompts. EA used AI to create 3D avatars of 11,000 college football players for its game. VCs poured billions into generative AI startups in H1 2024. Malaysia's Johor state is attracting tech giants for data center construction. California Senator Scott Wiener discusses his AI safety bill. The grimy residue of the AI bubble. Google reportedly wants to put Gemini in smart glasses. Sam Altman calls for a US-led global coalition to ensure a democratic vision for AI. Kamala Harris takes a leading White House role on AI. Scientists hunt for clues about China's supercomputing progress. Nvidia is reportedly working on a new AI chip for China. Some Activision Blizzard artists were allegedly forced to use AI in their work. Japan lures AI companies with copyright laws that alarm creators. Taylor & Francis sold access to authors' research to Microsoft without informing them. AI paid for by ads – the GPT-4o mini inflection point. Big tech wants to make AI cost nothing. AI method rapidly speeds predictions of materials' thermal properties. Japanese chain uses AI to gauge staff smiles. 77% of employees report AI has increased workloads. $1T rout hits Nasdaq 100 over AI jitters. OpenAI removes AI safety leader Mądry. Americans admit they can't tell what's real anymore due to AI scams. TSMC CEO predicts AI chip shortage through 2025. The AI job interviewer will see you now. AI is quietly being used to pick your pocket.
