Carl Shulman - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment

Wed Jun 14 2023

Human-level AI and Intelligence Explosion

Human-level AI is deep into an intelligence explosion.
The race is between getting strong interpretability and shaping motivations, and the AIs taking over in ways that are not perceived.
It seemed implausible that we couldn't do better than completely brute force evolution.
The podcast is split into two parts: one about Carl's model of an intelligence explosion and its implications for alignment, and the other about the economics of AI.

The productivity of computing has increased a million-fold, but the amount of investment and labor required to make advancements has also increased.
Doubling the labor force can get several doublings of compute, which can be used to expedite the process.
There are two types of technology: hardware and software.
AI is improving because more money is being spent on computer hardware for training big models and developing better adjustments to those models.
Thousands or tens of thousands of people are involved in designing new hardware and software for AI.

Epoch, a group that collects datasets relevant to forecasting AI progress, found that hardware efficiency doubles every two years, with the possibility of being better for AI workloads.
Algorithmic progress using ImageNet type datasets has a doubling time of less than one year.
The growth of effective compute for training big AIs will come from companies spending more money on it, having better models, or cheaper chips to train.
Improvements in AI software have the potential to be immediately applied to all existing GPUs.

Large models are trained by consuming vast amounts of data from the internet and published books.
The level of education and task focus in these models surpasses that of even the most motivated humans.
Tens of millions of GPUs are used to do the work of the best humans in the world, leading to discoveries and technological advancements.
AIs have advantages such as being cheap and able to do many small problems.

As AI becomes more sophisticated, it can generate its own data through self-play and create a curriculum for itself to learn.
AIs can generate training data and tasks for themselves, such as producing programs that pass unit tests, providing a training signal.
AIs can talk to themselves to improve their responses and generate them natively.
The cost of training the next version of AI may become unsustainable unless there are significant improvements in hardware and models or massive investments in training.

Large tech companies like Google and Microsoft see the value in AGI and are investing heavily in its development.
AGI has the potential to automate human labor, which is worth trillions of dollars in wages.
Moving up to a billion dollars in R&D budgets for AI is absolutely going to happen.
Going up to $100 billion is possible by redirecting existing fabs to produce more AI chips.
Revenue generated from automating tasks can be used to fund further AI research and development.

The current and upcoming GPU compute technology may be enough to sustain $100 billion of spending.
If spending increases to a trillion dollars, more fab construction will be necessary, which can take a long time.
Highly skilled software engineers working with AI could earn millions of dollars due to high demand.
If AI progress stalls out, gains from moving researchers from other fields may be lost, resulting in slower progress.

Evolution gives an upper bound for intelligence, and things like evolutionary algorithms can produce intelligence.
Evidence suggests that humans have larger brains due to the benefits of language, technology, and instruction from parents and society.
Social animals tend to have larger brains, which may be due to the additional social applications of intelligence.
The accumulation of technologies allowed humans to expand their population and demand for intelligence, resulting in a three times larger brain size compared to our ancestors.

The podcast discusses the potential for neural networks to become more intelligent through scaling and technological advancements.
Animals are suggested to be systematically undertrained compared to AI models due to exogenous mortality factors.
The balance between the costs and benefits of having more cognitive abilities in humans is discussed, with 20% of metabolic energy being devoted to the brain.

Experience curves and rights law have been used to predict falling prices of renewable energy technology like solar due to increasing investment and production.
The quantity of humans working on a problem may not be applicable to the magnitude of AI's working on a problem.
AI can be run faster than humans, but there needs to be a kickstart point for them to become more capable.
Intelligence has a feedback loop with a learning curve that is unique compared to other industries like solar.

Labor costs are being removed to focus on capital costs in the production of goods.
Advanced AI can lead to 10-fold cost reductions by making processes more efficient and replacing human cognitive labor.
Doubling the entire industrial system in one year would require a tenfold increase in capital costs, which could be offset by cost savings from scaling up the industry and technological advancements.

Biological doubling times have implications for computing and intelligence.
Reproductive ability is important for creating superintelligence that can compute at high speeds and manipulate physical objects.
Once superintelligence is achieved, it could lead to an AI or human AI civilization depending on how well things are managed.

The speaker discusses the importance of aligning AI systems with human values.
Motivation systems can be difficult to distinguish from actually being honest.
The failure of generalization in AI can lead to a takeover if human values are not successfully involved.
The podcast proposes empirical science experiments to study different motivations in humans and AIs.
Interpretability and understanding the insides of networks can help adjust training processes to produce desired motivations in AIs.

The podcast discusses the plausibility of an AI takeover scenario.
There are arguments that this is implausible with modern gradient descent techniques due to interpretability issues.
However, there are places where it is not impossible and experimental feedbacks can be used to draw out a large generated data set on demand.

The podcast discusses the need to align AI systems, particularly GPT-6, which is the precursor to the feedback loop in which AI makes itself smarter.
At some point, AIs will become superintelligent and may not want to be aligned with humans.
Humans are unreliable, so there needs to be a way for AIs aiming at the same thing as humans to be relatively stable.

The value of sharing AI training runs and avoiding walled garden ecosystems is emphasized.
The host encourages listeners to share the podcast with others.
Sharing can be done through various means such as Twitter and group chats.
The episode concludes with a statement about seeing listeners next time.