Last Week in AI

#130 - Llama 2, Elon Musk’s xAI, WormGPT, LongLLaMA, AI apocalypse, actors on strike

Tue Jul 25 2023

PodcastingAI ToolsLanguage ModelsArt ProtectionAI SafetyProtein GenerationBrain MappingRisks in AIRegulationsSAG-AFTRA Strike

Description

This episode covers a wide range of topics including podcasting, AI-assisted note-taking apps, language models for commercial use, advancements in AI research, mapping the brain, risks in AI development, regulations, and the SAG-AFTRA strike. The hosts discuss their own experiences with podcasting and interview John Crone, host of the Super Data Science podcast. They also explore various AI tools such as Notebook LM and GPT-4. The episode delves into the challenges and advancements in language models like Lama2 and Lomitoo. It highlights the importance of protecting art from AI with tools like Glaze. The hosts discuss the impact of AI on job markets, particularly in San Francisco, and the launch of Elon Musk's AI company, XAI. They also address concerns and challenges in AI safety, including alignment issues and OpenAI's partnerships with media companies. The episode explores advancements in AI applications and research, such as protein generation and 3D shape generation. It covers the use of AI in neuroscience, specifically mapping the brain of a fruit fly. The hosts discuss the connectome of the brain and debates surrounding artificial neural networks. They also delve into the risks and forecasts in AI development, including catastrophic events and super-engineered pathogens. The episode touches on regulations in China and the impact of AI on jobs. It concludes with a discussion on the SAG-AFTRA strike and concerns about AI-generated extras.

Insights

Advancements in Language Models

Language models like Lama2 and Lomitoo are revolutionizing commercial applications by offering powerful capabilities and improved performance.

Protecting Art from AI

Tools like Glaze provide artists with a solution to protect their art online by adding imperceptible alterations to images that confuse AI models.

Challenges in AI Safety

The alignment problem remains a core issue in defining safe goals for intelligent systems, and there are concerns about politicizing AI safety.

Advancements in Protein Generation

AI tools are designing custom proteins that have never been produced by evolution, opening up possibilities for vaccines and therapeutics.

Mapping the Brain with AI

AI has enabled the mapping of the fruit fly connectome, providing insights into neural pathways and advancing neuroscience research.

Risks in AI Development

AI experts estimate a higher risk of catastrophic events and extinction due to AI compared to super forecasters, highlighting the need for careful development and safety measures.

Regulations and Impact of AI

China has implemented rules governing generative AI services, while the US FTC investigates OpenAI. The SAG-AFTRA strike raises concerns about AI-generated extras.

Chapters

Introduction
Super Data Science Podcast and AI Tools
Hallucinations and Protection in AI Systems
Commercial Use of Language Models
New Language Model Lomitoo
AI Job Market and Elon Musk's XAI
Concerns and Challenges in AI Safety
Partnerships and AI Misuse
Improving Language Models and AI Applications
Challenges and Advancements in AI Research
Advancements in AI Applications and Research
Mapping the Brain and AI in Neuroscience
Connectome and AI in Neuroscience
Risks and Forecasts in AI Development
Regulations and AI Impact
SAG-AFTRA Strike and AI Misuse

Summary

Transcript

Introduction

00:00 - 06:38

The hosts introduce themselves and their guest, John Crone, who is the host of the Super Data Science podcast.
They discuss how podcast hosts often don't listen to podcasts, but they all enjoy each other's shows.
John talks about his work at Nebula and his book 'Deep Learning Illustrated'.
Jeremy has been a guest on the Super Data Science podcast and they talk about their previous episodes together.
John praises Last Week in AI for covering all the big news in a concise format that helps him stay up-to-date as a data scientist.
They joke about writing an Apple review for Last Week in AI.

Super Data Science Podcast and AI Tools

06:09 - 13:22

Super Data Science podcast receives positive feedback on being a deep and expansive show
Jeremy was a guest on the podcast before becoming a co-host
Listener comments and corrections include positive reviews and requests to continue discussing topics outside of expertise
Google launches AI-assisted note-taking app called Notebook LM for students to organize lecture notes and answer questions about documents
Notebook LM is an experiment aimed at reducing hallucination and false information
The app could potentially integrate with other tools in the AI stack for value accretion
Similar tools exist for analyzing PDFs and answering questions about them
GPT-4 is a powerful AI tool that allows for quick creation of tutorials and code demos
GPT-4 has reduced the need for double-checking but may still occasionally hallucinate
The hosts rely on listeners to correct any false information in their bullet points

Hallucinations and Protection in AI Systems

12:56 - 19:51

Hallucinations in chat GPT systems are becoming less of a problem thanks to tools like Arthur AI's firewall for LLMs.
The University of Chicago has developed Glaze, a program that can protect art from AI by cloaking images with slight alterations.
Glaze has been downloaded over 890,000 times since its release in March.
Artists are concerned about AI and Glaze provides a solution to protect their art online.
Adding tiny changes to RGB values can confuse AI models without being perceptible to humans.
Meta and Microsoft have released Lama 2, an open-source language model for commercial use.
Lama was leaked after being released to the research community, but now it is available commercially.
Meta has put effort into red teaming and testing the performance of Lama with external partners.

Commercial Use of Language Models

19:33 - 26:43

Lama2 is an open source language model that can be used for commercial applications.
The license for Lama2 has some restrictions, such as not using it for illegal activities or nuclear weapon development.
Companies with over 700 million monthly active users are not allowed to use Lama2 for commercial applications.
Lama2 is commercially licensed and can be accessed through a quick form on the official Meta AI Llama2 page.
Lama2 offers models with different parameter sizes, ranging from 7 billion to 70 billion parameters.
Benchmark results show that the Loma-2 models perform well compared to other open source models like Falcon.
The instruction fine-tuning and training data of Lama-2 have improved its performance and capabilities.

New Language Model Lomitoo

26:21 - 33:07

The new model, Lomitoo, is more performant and trained on two trillion tokens.
Lomitoo has double the context length, allowing for 4,000 tokens.
It was trained on 40% more data and has a Dabba 2 chat variant with over 1 million new human annotations.
Deciding to switch to a new model depends on technical considerations like context window size and commercial licensing options.
Falcon was not suitable due to its parameters, while Lomitoo performed well in comparison to other models.
Meta announced an open innovation AI research community for academic researchers and the Snama Impact Challenge to address important challenges using Lomitoo.
Lomitoo focuses on natural language benchmarks but doesn't perform well in code tests or math tests.
The paper discusses the use of reinforcement learning from human feedback and a new method called ghost attention for multi-turn consistency.

AI Job Market and Elon Musk's XAI

32:53 - 39:55

US companies are on a hiring spree for AI jobs, paying an average of $146,000 per year.
There are approximately 7.6 million open jobs in June related to AI.
San Francisco-based companies received nearly half of the world's AI funding in the first quarter of 2023, with OpenAI receiving $10 billion.
A neighborhood in San Francisco called Cerebral Valley has become a hub for AI startups.
65% of top AI companies in the US were founded or co-founded by immigrants.
Elon Musk has launched his AI company called XAI, aiming to create an AI that is maximally truth-seeking and curious about the universe.
XAI has a team of researchers from notable companies like OpenAI, DeepMind, Google Research, and Tesla.
XAI is considered a serious competitor in the race due to their significant investment in compute resources.
Concerns arise as XAI lacks expertise in AI alignment and may oversimplify the alignment problem.

Concerns and Challenges in AI Safety

39:29 - 46:51

There are deep problems related to inner alignment and power seeking that are not resolved by this initiative.
The Center for AI Safety, led by Dan Hendrix, has some overlap with the AI safety community but needs to make more moves in that direction.
The suggestion is to train the language model on truths from Truth Social for a truth-filled platform.
It is unclear what the plan is, but it seems they aim to be an open AI and deliver top-notch chat models.
Elon Musk's credibility among AI experts is questioned due to unfulfilled promises and recent contradictory actions.
There may be attempts at competition with OpenAI from this new initiative.
The concern is whether politicizing AI safety is the best approach.
The alignment strategy of this initiative remains vague, despite Musk's warnings about potential risks.
The core issue lies in defining a safe goal for an intelligent system and solving the inner alignment problem before determining objectives for the system.
OpenAI has struck a deal with Associated Press to pay for access to its news archive for training its AI chatbot. This could set a precedent for similar deals with other media companies in the future.
Shutterstock has expanded its deal with OpenAI, allowing them to license data including images, videos, and music for training purposes. In return, Shutterstock gains priority access to OpenAI's latest technology and editing capability.

Partnerships and AI Misuse

46:22 - 53:55

Charter Stock and OpenAI have partnered to provide priority access to OpenAI's latest technology and editing capability for stock images.
Adobe is indemnifying users of their image generation model by claiming ownership of the training data.
A tool called Warm GPT, based on GPTJ from Ellifer, is being used by cybercriminals for business email compromise attacks.
The tool is not open source and its training process remains undisclosed.
Language models like ChatGPT can be manipulated to generate phishing emails despite ethical guidelines.
Long Llama is a large language model capable of handling long context windows of 256 tokens, addressing the distraction issue caused by irrelevant information in inputs.
Long Llama uses contrastive learning and focus transformers to improve relevance in predictions.
The effective context window length of Long Llama was tested using the pass key retrieval task.

Improving Language Models and AI Applications

53:34 - 1:01:16

Researchers conducted a test called the 'pass key retrieval' task to determine the effective context window length of a language model.
The model was able to effectively retrieve the pass key from a large amount of irrelevant text, up to 256,000 tokens worth of prompt.
Using retrieval mechanisms during training allows for improving long context without changing the initial model architecture.
The approach is cost-effective as it doesn't require training everything from scratch.
Mixture of experts combined with instruction tuning outperforms dense models in chat-based tasks.
This combination allows for scaling to larger models while reducing inference costs.
Pairing mixture of experts with instruction tuning is where the field is heading.
There are trade-offs with mixture of experts models, such as overfitting and fragility in individual experts.

Challenges and Advancements in AI Research

1:00:55 - 1:08:22

Individual experts in the mixture of experts models are trained on less data, leading to a risk of overfitting.
The individual experts may memorize facts rather than learning generalizable facts about the world.
Instruction fine-tuning benefits mixture of experts models more than dense nets.
Instruction fine-tuning provides generality to each individual expert.
Mixture of experts models perform worse in dealing with multiple languages compared to dense versions.
The findings are evaluated on large models with billions of parameters.
AI tools are designing custom proteins that have never been produced by evolution, which can be used for vaccines and therapeutics.
Narrow AI applications are capable of doing superhuman things in bioengineering materials at a fraction of the cost and time compared to human research.
Techniques developed in computer vision, such as in painting, are being applied to protein generation and design with impressive results.
RF diffusion is a model used for protein generation and design that shows significant efficiency boosts compared to previous methods.

Advancements in AI Applications and Research

1:08:01 - 1:15:01

New software released in March has had a significant impact on the field.
A new paper called 'Sketch a Shape' explores zero-shot learning for 3D shape generation.
Hugging Face website hosts papers, including 'Mixture of Experts'.
AI-generated 3D models are improving rapidly, impacting CG artists and others.
The ability to generate 3D shapes from sketches eliminates the need for CAD tools.
Data limitations remain a challenge in generating accurate representations of sketches.
A workaround involves using embeddings from computer vision models to create representations of sketches.
This technique can be applied to designing proteins and predicting their structures.
'Patch and Pack' is a technical advancement that allows computer vision models to handle any aspect ratio and resolution without resizing images.
Scientists have used AI to map the brain of a fruit fly, which is an important animal model in neuroscience.

Mapping the Brain and AI in Neuroscience

1:14:45 - 1:21:56

Fruit flies, specifically Drosophila melanogaster, are a useful animal model in neuroscience due to their breeding frequency and lack of ethical concerns.
AI has been used to create a 3D image of a fruit fly brain by combining 2D images of brain slices.
The AI model has made the process of tracing neural pathways faster by 50 to 100 times.
Humans still had to do significant work over several years to complete the mapping of the fruit fly connectome.
AI is now being used as a tool in scientific research, enabling advancements in various fields.
Artificial neural networks were originally designed for studying neuroscience and have now come full circle by being used in neuroscience research again.
The fruit fly connectome map provides information about the connections between individual neurons, similar to knowing all the weights in a deep learning model.

Connectome and AI in Neuroscience

1:21:28 - 1:28:36

The connectome of the brain reveals how brain cells are connected to each other.
The number of neurons in a fruit fly's brain is around 120,000, but it can perform various functions like flying, sensing, and learning.
A simple flatworm called synorhabditis elegons has only 300 neurons, and their connections can be mapped as they learn.
In artificial neural networks, the number of parameters does not determine human-level cognition because there are other factors at play.
Deep learning systems and AI systems are a new kind of intelligence that is different from human intelligence.
Neural networks in AI are loosely modeled after biological neurons but have little similarity to them.
Debates exist regarding the level of abstraction in neural networks for effective learning and powerful systems.
It is challenging to compare the mapping of gradients in artificial systems with biological systems due to differences in calculation methods.
Policy and safety concerns related to AI apocalypse are discussed based on research showing over-training in individual sub-models.
AI experts estimate a higher risk of catastrophic events due to AI compared to super forecasters who have a track record of accurate predictions.

Risks and Forecasts in AI Development

1:28:11 - 1:35:28

AI experts define catastrophic risk of AI as an event where 10% of humans are impacted and extinction as a scenario where fewer than 5,000 people remain on the planet.
AI experts estimate a 12% chance of catastrophe and a 3% chance of extinction due to AI by 2100.
Super forecasters estimate a 2% chance for catastrophic risk and less than a 1% chance for extinction risk due to AI by 2100.
AI researchers at OpenAI, DeepMind, and Anthropic have higher estimates of AI risk compared to the study's median estimate.
The study also covers other risks such as nuclear risks, engineered pathogens, natural pathogens, and non-anthropogenic causes of catastrophe extinction.
Across both super forecasters and domain experts, the biggest extinction risk is from super-engineered pathogens.

Regulations and AI Impact

1:34:58 - 1:43:26

China has finalized rules governing generative AI services like JChad GPT, requiring providers to take measures to stop generating illegal content and report it to the relevant authorities.
These regulations apply to private companies but not to the government building its own AI systems.
Generative AI services must adhere to the core values of socialism and obtain a license to operate in China.
A survey by OECD suggests that 27% of jobs are at high risk of automation, with Eastern European countries being most exposed.
US senators have received a classified White House briefing on AI, organized by Chuck Schumer who is pushing for comprehensive legislation on AI.
The US FTC has opened an investigation into OpenAI based on claims of violating consumer protection laws and generating false statements about individuals.
UK universities have dropped guiding principles on generative AI, focusing on supporting students and staff in using AI tools appropriately and incorporating ethical use of AI in teaching and assessment.
The SAG-AFTRA union of actors is on strike due to concerns about synthetic media and the use of AI-generated extras.

SAG-AFTRA Strike and AI Misuse

1:42:57 - 1:45:22

SAG-AFTRA union of actors is on strike
AI proposal for actors sounds like a nightmare
Labor union SAG after has joined the writers Guild of America on strike
Negotiation about AI component and using people's likeness
Story will have more coverage in the coming weeks