You have 4 summaries left

The Gradient: Perspectives on AI

David Thorstad: Bounded Rationality and the Case Against Longtermism

Thu May 02 2024
bounded rationalityinterdisciplinary researchBayesian modelsepistemic normsheuristicsreason responsivenessvindictory epistemologyoverthinkingtransparency in beliefsrational inquiryfriendship normslong-termismexistential risksepistemic humilityonline communitiespublishing challengestheory of change

Description

The episode explores topics such as bounded rationality, interdisciplinary research, empirical violations of standard Bayesian models, rational belief, epistemic norms, heuristics, reason responsiveness, metacognition, singularity hypothesis, long-termism, existential risks, epistemic humility, online communities, publishing challenges, and theory of change. The discussions highlight the importance of integrating psychology into formal models of cognition, addressing challenges in interdisciplinary research, and considering the impact of biases and limited information on decision-making processes. The episode also delves into the potential risks and benefits associated with long-term thinking and explores different ethical theories in relation to friendship and truth-telling.

Insights

The work being done aims to bridge the gap between formal models of cognition and human behavior by integrating aspects like limited memory, incoherence, time costs, and accuracy.

This integration involves blending old formal models with cognitive science to capture human elements like heuristics and limited evidence.

There is a movement towards interdisciplinarity in academia, especially among younger faculty members.

Grant funders play a role in promoting interdisciplinary research by focusing on problems rather than disciplines.

Critics of the standard Bayesian picture have struggled to provide a systematic alternative model, which is necessary for a valid criticism.

Bayesian models are successful in various areas of theorizing, including epistemology and philosophy of science.

Epistemic norms for belief involve a deeper normativity beyond just categorizing types of reasons, with criticism for violating these norms.

Reason responsiveness bridges the gap between what one ought to do and what one is rationally required to do, providing a coherent explanation.

Bounded rationality is a central concept, emphasizing the importance of cognitive limits and attention costs.

Heuristics are often used due to limited memory and information, involving search rules, stopping rules, and decision rules.

Vindictory epistemology challenges the idea of irrational thought and action, suggesting patterns of deviation from the standard picture can be rational.

Information sensitive account of deontic modals provides a middle ground between objectivism and subjectivism in bounded rationality.

Reason responsiveness in rationality involves responding correctly to the reasons one has, doing what one has most reason to do.

Epistemic norms should be considered in relation to everyday actions and decisions.

Overthinking is irrational and leads to wasted time, unhappiness, worse decisions, and decision paralysis.

Transparency in beliefs and deliberation can be complex and challenging to define

The speaker evaluates their view against competing views of rational inquiry, including those focusing on knowledge, coherence, and pragmatic goals.

The speaker's work is influenced by philosophers like Stephen Stitch and aims to offer a consequential account of rational inquiry with normative advantages.

Norms of friendship play a role in the argument, highlighting the conflict between rational belief and irrational inquiry.

There is a distinction made between rational deliberation and rational belief, showing how they can sometimes come apart in consequentialist ethics.

Chapters

  1. Bounded Rationality and Long-Termism
  2. Interdisciplinary Research and Challenges
  3. Empirical Violations of Standard Bayesian Picture
  4. Implicatures, Rational Belief, and Epistemic Norms
  5. Rationality, Bounded Rationality, and Heuristics
  6. Vindictory Epistemology and Information Sensitivity
  7. Reason Responsiveness and Epistemic Norms
  8. Transparency in Beliefs and Deliberation
  9. Overthinking, Regress Problem, and Rational Deliberation
  10. Constructing a Theory and Challenging Established Doctrines
  11. Epistemology, Ethics, and Complex Ideas
  12. Friendship, Rational Belief, and Rational Deliberation
  13. Improving Decision-Making and Rational Inquiry
  14. Metacognition, Bounded Rationality, and Indirect Normative Assessment
  15. Processes in Decision-Making and Singularity Hypothesis
  16. The Singularity Hypothesis and Hyperbolic Growth
  17. Long-Termism, Rationality, and Evaluating Existential Risks
  18. Epistemic Humility and Online Communities
  19. Publishing Challenges and Swamping Axiological Strong Long Termism
  20. Symmetric Beliefs, Long-Term Forecasting, and Theory of Change
Summary
Transcript

Bounded Rationality and Long-Termism

00:00 - 07:45

  • The conversation in the podcast revolves around bounded rationality and long-termism.
  • Guest David Thorstad discusses his journey into studying concepts like rationality and bounded rationality.
  • Thorstad reflects on his crisis of confidence in formal models of rationality lacking psychology and abstracting human behavior.
  • The work being done aims to bridge the gap between formal models of cognition and human behavior by integrating aspects like limited memory, incoherence, time costs, and accuracy.
  • There is a trend towards blending old formal models with cognitive science to capture human elements like heuristics and limited evidence.
  • Efforts are being made to bring different academic communities together, such as Bayesians incorporating psychology into their models.

Interdisciplinary Research and Challenges

07:20 - 15:15

  • Interdisciplinary research requires collaboration and recognition from academic institutions.
  • Challenges in interdisciplinary research include evaluation by one discipline, lack of understanding of other disciplines, and strategic publication choices.
  • There is a movement towards interdisciplinarity in academia, especially among younger faculty members.
  • Grant funders play a role in promoting interdisciplinary research by focusing on problems rather than disciplines.
  • The standard picture of rationality in economics was solidified through decision theory and coherence axioms.
  • The standard picture has underwritten successful normative programs like Bayesian epistemology but lacks psychology in its models.

Empirical Violations of Standard Bayesian Picture

14:52 - 22:00

  • Bayesian models are successful in various areas of theorizing, including epistemology and philosophy of science.
  • There have been observed empirical violations of the standard Bayesian picture, leading to a crisis of confidence in the normative theory.
  • Critics of the standard Bayesian picture have struggled to provide a systematic alternative model, which is necessary for a valid criticism.

Implicatures, Rational Belief, and Epistemic Norms

21:34 - 29:05

  • Implicatures and experimental instructions can lead to unexpected outcomes in studies.
  • Different views on rational belief exist, with evidentialism being a popular one.
  • Epistemology shifted from descriptive to normative in the 1950s, focusing on what is rational to believe.
  • Belief should be governed by epistemic norms that respond to truth, evidence, and intellectual considerations.
  • Epistemic requirements expanded beyond belief to include testimony, deliberation, and inquiry.
  • Arguments about belief may not apply outside the belief case.

Rationality, Bounded Rationality, and Heuristics

28:39 - 36:00

  • Rationality can be criticized without obvious circularity, leading to self-improvement through self-critique.
  • Community engagement and responsiveness play a significant role in improving rationality processes.
  • Bounded rationality is a central concept, emphasizing the importance of cognitive limits and attention costs.
  • Heuristics are often used due to limited memory and information, involving search rules, stopping rules, and decision rules.
  • Conscious and unconscious application of heuristics differ, with the need for formal theorizing in bounded rationality models.

Vindictory Epistemology and Information Sensitivity

35:34 - 42:09

  • Vindictory epistemology aims to explain when and why people deviate from theories due to the theories being wrong.
  • Reason responsive consequentialist view in rational inquiry focuses on doing what is best for making the world go as well as possible.
  • Information sensitive account of deontic modals provides a middle ground between objectivism and subjectivism in bounded rationality.
  • Subjectivism faces challenges in providing verdicts for agents who lack beliefs about everything, while information sensitivity offers a more natural approach based on evidence.

Reason Responsiveness and Epistemic Norms

41:40 - 48:52

  • Reason responsiveness in rationality involves responding correctly to the reasons one has, doing what one has most reason to do.
  • Reason responsiveness bridges the gap between what one ought to do and what one is rationally required to do, providing a coherent explanation.
  • Being rational according to reason responsiveness entails doing what one has most reason to do, which explains the value of rationality.
  • Epistemic norms for belief involve a deeper normativity beyond just categorizing types of reasons, with criticism for violating these norms.
  • Arguments by the New Evidentialists challenge the existence of non-epistemic reasons for belief, suggesting practical reasons lack the ability to directly motivate belief.
  • Transparency in deliberation can lead from questioning whether to believe something to questioning its factual truth, potentially leading to self-delusion and challenges in being fully rational.

Transparency in Beliefs and Deliberation

48:23 - 55:18

  • Transparency in beliefs and deliberation can be complex and challenging to define
  • Practical reasons for belief may not always influence one's actual beliefs
  • Epistemic norms should be considered in relation to everyday actions and decisions
  • Inquiry plays a crucial role in shaping our understanding, decisions, and overall well-being
  • Overthinking is not rational as it can lead to negative outcomes and decision paralysis

Overthinking, Regress Problem, and Rational Deliberation

54:56 - 1:01:41

  • Overthinking is irrational and leads to wasted time, unhappiness, worse decisions, and decision paralysis.
  • The regress problem of deciding how to decide involves inquiries about how to inquire, creating multiple levels of inquiry.
  • There is a distinction between what one should believe and how one should deliberate or inquire.
  • Epistemic norms for inquiry involve considering all relevant factors without overthinking.
  • The reason-responsive consequentialist view inquires into minimal criteria including trade-offs, stakes of inquiry, and rational permissibility of inferences made by stereotyping.

Constructing a Theory and Challenging Established Doctrines

1:01:14 - 1:08:22

  • The speaker wanted to include points in their theory that would be difficult to argue against and could challenge other theories.
  • Trade-offs are essential in rationality according to the speaker.
  • Stereotyping is a topic of concern for the speaker, who delved into literature to ensure their theory did not support harmful stereotypes.
  • The speaker spent time researching traditional criteria like accuracy and evidence before adopting a consequentialist approach.
  • There has been a rebellion in the stereotyping literature advocating for considering stakes and consequences.
  • The methodology behind constructing a theory like this one is discussed, drawing parallels with philosophical debates on consistency and intuition.
  • Younger epistemologists are challenging established doctrines in the field, advocating for theories that address issues of race, gender, and oppression.

Epistemology, Ethics, and Complex Ideas

1:07:54 - 1:15:16

  • Some individuals in certain communities of knowledge prioritize theories that address issues of race, gender, and oppression.
  • The speaker faced challenges in categorizing their work as either epistemology or ethics, leading to a crisis of confidence.
  • The speaker wrote a book to provide a platform for expressing complex ideas that may be difficult for referees to grasp quickly.
  • The speaker evaluates their view against competing views of rational inquiry, including those focusing on knowledge, coherence, and pragmatic goals.
  • The speaker's work is influenced by philosophers like Stephen Stitch and aims to offer a consequential account of rational inquiry with normative advantages.

Friendship, Rational Belief, and Rational Deliberation

1:14:57 - 1:22:00

  • The podcast discusses the reason-responsive consequentialist view and its ability to account for a wide range of normative claims about inquiry.
  • Norms of friendship play a role in the argument, highlighting the conflict between rational belief and irrational inquiry.
  • Consequentialism instructs prioritizing important goods like friendship even if it leads to potentially irrational beliefs.
  • There is a distinction made between rational deliberation and rational belief, showing how they can sometimes come apart in consequentialist ethics.
  • The discussion includes examples illustrating how different ethical theories handle dilemmas involving friendship and truth-telling.

Improving Decision-Making and Rational Inquiry

1:21:37 - 1:28:31

  • It is important to form beliefs and conduct inquiry in a way that can help friends become better people.
  • Vindictory epistemology challenges the idea of irrational thought and action, suggesting patterns of deviation from the standard picture can be rational.
  • Panglassian and meliorative epistemology are key concepts in vindictory theory, emphasizing the importance of acknowledging irrationality and improving decision-making processes.
  • Changing how information is presented can help individuals make more rational decisions, as seen in examples like doctors making fewer base rate fallacies when information is presented differently.
  • The recognition heuristic suggests that recognizing something can influence perceived quality or likelihood, impacting decision-making processes, especially in low-stakes situations like consumer choices.
  • Metacognitive processing and strategy selection are tools available to individuals to reflect on their decision-making processes and potentially improve rational inquiry.

Metacognition, Bounded Rationality, and Indirect Normative Assessment

1:28:04 - 1:35:09

  • Metacognition plays a crucial role in decision-making, especially in relation to the stakes involved.
  • There are two theories of metacognition: Bayesian and noetic feelings.
  • The podcast discusses bounded rationality and the standard picture, exploring a third way between blaming the theory or the agent for violations.
  • The speaker distinguishes between rationality of belief/action and rationality of deliberation, emphasizing the importance of inquiry in bounded rationality.
  • Indirect normative assessment challenges traditional views on rational beliefs and attitudes, suggesting that beliefs are rational if they result from a rational process of deliberation.

Processes in Decision-Making and Singularity Hypothesis

1:34:44 - 1:41:41

  • The discussion revolves around the fundamental nature of attitudes and processes in decision-making.
  • There is a shift towards studying processes rather than beliefs in bounded rationality.
  • The conversation delves into the singularity hypothesis and its similarities to ontological arguments for the existence of God.
  • The sensitivity of discussing certain topics, like the singularity hypothesis, especially within the effective altruism community, is acknowledged.

The Singularity Hypothesis and Hyperbolic Growth

1:47:20 - 1:54:21

  • The singularity hypothesis is supported by a brief argument that may not be sufficient for its magnitude.
  • Diminishing research productivity is a significant concern across various fields like technology, agriculture, and medicine.
  • There is a need to address diminishing research productivity to potentially impact the trajectory towards hyperbolic growth or the singularity hypothesis.
  • Progress studies are gaining attention as there seems to be a decline in concrete scientific progress compared to previous centuries.

Long-Termism, Rationality, and Evaluating Existential Risks

1:53:57 - 2:01:04

  • The speaker engages with arguments on long-termism and highlights potential mistakes in evaluating existential risks
  • One mistake discussed is the difference between absolute and relative risk reduction, emphasizing the importance of understanding cumulative versus percentory risk reduction
  • The conversation around long-termism includes discussions on adapting to challenges posed by the speaker's arguments, particularly regarding the magnitude of risk reduction needed over time
  • Background risks play a crucial role in assessing total existential risk, as interventions on specific risks may not significantly reduce overall risk without addressing all background risks

Epistemic Humility and Online Communities

2:00:41 - 2:07:34

  • Existential risk mitigation requires a structured approach to minimize risks effectively.
  • Belief in long-termism can lead to inflated views on the world's risks and potential impact.
  • Epistemic humility is crucial when making claims related to long-term impact.
  • Reliance on non-peer-reviewed sources like forums and blogs can raise concerns about credibility and expertise.
  • Insularity in online communities can drive polarization, group biases, and extremism.
  • Group deliberation dynamics may influence swings in opinions without strong supporting arguments.

Publishing Challenges and Swamping Axiological Strong Long Termism

2:07:04 - 2:13:31

  • The speaker discusses the challenges faced in getting their work published due to rejection and consensus pressure.
  • Shared disciplinary background provides support for overcoming publishing obstacles.
  • Long-termism community's focus on funding and incentives can impact beliefs and decision-making processes.
  • The concept of swamping axiological strong long termism is explained as prioritizing actions with significantly better outcomes for the long-term future.
  • Arguments are presented for the rarity of swamping long-termist options, including evidence-based reasoning and forecasting pessimism.

Symmetric Beliefs, Long-Term Forecasting, and Theory of Change

2:13:04 - 2:19:02

  • Prior beliefs should be symmetric when evaluating future outcomes, even with limited evidence.
  • Long-term forecasting beyond 25 years is highly controversial and predicting global values is complex.
  • Predicting the impact of interventions on various phenomena is challenging due to moving targets and interconnected effects.
  • Theory of change in effective altruism involves influencing the community's donation behavior and fostering collaboration in academia.
1