You have 4 summaries left

The Gradient: Perspectives on AI

Seth Lazar: Normative Philosophy of Computing

Thu May 23 2024
AI ethicsAI safetyPhilosophy of computingAttention allocationIntuitions in moral philosophyMetaphysical economyOpen source AICognitive dissonanceAlgorithmic systemsLLMsFeasibility horizonsExplainable AIManipulationPublic interest researchCollaboration

Description

This episode explores various topics related to AI ethics, safety, and philosophy. It covers anormative philosophy of computing, attention allocation, intuitions in moral philosophy, metaphysical economy, AI safety efforts, open source AI, cognitive dissonance in big AI companies, ethics and safety in AI development, algorithmic systems and societal impacts, LLMs and feasibility horizons, explainable AI, manipulation, public interest research, collaboration between tech companies and public interest researchers, algorithmic transparency, safe harbor for research, academic ecosystems, interdisciplinary work, conferences, and the importance of public input and expertise.

Insights

Attention allocation is a moral skill

Attention allocation is considered a moral skill with ethical implications in influencing others' attention.

Building robustness in AI systems

Building robustness and resilience in AI systems is crucial to address potential catastrophic risks in the future.

Empirical evidence is valuable

Empirical evidence is considered more valuable than a priori arguments when it comes to understanding and preparing for technological advancements.

Open source and open weights

Open source and open weights are important for mitigating current and near-term risks from AI systems.

Ethics and safety in AI development

Companies like Open AI and Antropic prioritize ethics and safety research in AI development.

Safety measures for algorithmic systems

Decisions about new technologies should not be left solely in the hands of unaccountable individuals to prevent significant societal impacts.

Focusing on empirical risks

AI safety efforts could be reoriented towards addressing risks from technologies that are well understood and empirical.

Balancing safety guarantees with advancements

Importance of balancing safety guarantees with advancements in capabilities in AI development.

Integration of technical AI safety work

Integration of technical AI safety work with a broader system safety perspective is essential for real progress in AI safety.

Collaboration between tech companies and public interest researchers

Collaboration between tech companies and public interest researchers, supported by regulation, is essential for ensuring ethical AI development.

Chapters

  1. Anormative Philosophy of Computing and Attention Allocation
  2. Intuitions, Ethical Theories, and Metaphysical Economy
  3. AI Safety, Attention Allocation, and Catastrophic Risks
  4. Superintelligence, Existential Risks, and Empirical Evidence
  5. Reckoning within the Effective Altruism Movement and Open Source AI
  6. Debates on Open Weights, Cognitive Dissonance, and Ethical AI Companies
  7. Ethics and Safety in AI Development and Agentic AI Systems
  8. Safety Measures, Algorithmic Systems, and Societal Impacts
  9. LLMs, Feasibility Horizons, and AI Safety Efforts
  10. Understanding Risks, Socio-Technical Approach, and Costly Policies
  11. Balancing Safety and Advancements, Explainable AI, and Trust Relationships
  12. Evolution of Explainable AI, Manipulation, and Public Interest Research
  13. Radicalization, Double Agents, and Ethical AI Development
  14. Ethics and Safety in AI Development, Public Interest Research, and Collaboration
  15. Algorithmic Transparency, Safe Harbor for Research, and Academic Ecosystems
  16. Interdisciplinary Work, Conferences, and Incentives
  17. Public Input, Expertise, and Collaboration
Summary
Transcript

Anormative Philosophy of Computing and Attention Allocation

00:01 - 07:38

  • Seth Lazar focuses on 'anormative philosophy of computing' which brings political philosophy to the AI world.
  • Lazar discusses catastrophic risk and how AI relates to it in a broader context.
  • Attention allocation is considered a moral skill with ethical implications in influencing others' attention.
  • The allocation of attention is not a zero-sum game, especially in the context of AI's impact on various areas.
  • Attention allocation can be independently good or bad, separate from other norms like epistemic considerations.
  • There is intrinsic significance in the way attention is allocated, beyond its instrumental importance.

Intuitions, Ethical Theories, and Metaphysical Economy

07:09 - 14:23

  • Using technology like headphones can impact the quality of attention given to children, leading to potential misunderstandings in communication.
  • In moral philosophy, there is a discussion on the origin of intuitions and the grounding of ethical theories.
  • Coherentist justification is important in developing ethical theories by integrating various judgments across different cases.
  • Comparing and contrasting competing moral theories, such as deontological and consequentialist approaches, helps in understanding different perspectives.
  • The importance of building compelling arguments on normative bedrock in moral philosophy to explain judgments effectively.
  • Considerations about metaphysical economy and parsimony when evaluating theories in philosophy.
  • Being open to skepticism and questioning societal influences on intuitions is crucial, especially in fields like AI where novel phenomena may challenge existing beliefs.

AI Safety, Attention Allocation, and Catastrophic Risks

13:59 - 21:17

  • Intuitions about forming meaningful relationships with AI companions may be biased and should be critically examined.
  • The evolution of the AI safety community has shifted towards a more grounded approach that focuses on existing technologies.
  • Advancements in AI, such as LLMs, have changed what is considered feasible in terms of catastrophic AI risks.
  • Building robustness and resilience in AI systems is crucial to address potential catastrophic risks in the future.

Superintelligence, Existential Risks, and Empirical Evidence

20:49 - 27:21

  • Advancements in AI have expanded the scope of philosophical engagement, particularly around superintelligence and existential risks.
  • There are similarities between arguments for superintelligence and ontological arguments for the existence of God.
  • Effective altruists have gained significant influence in addressing existential risks from AI.
  • Critiques exist regarding the effectiveness of purely philosophical approaches in dealing with future technological advancements.
  • Overreliance on present capabilities when considering future technology may lead to misjudgments and overgeneralizations.
  • Empirical evidence is considered more valuable than a priori arguments when it comes to understanding and preparing for technological advancements.

Reckoning within the Effective Altruism Movement and Open Source AI

26:52 - 34:41

  • There has been a lot of reckoning and discussion within the Effective Altruism (EA) movement.
  • Some philosophers from Oxford have had a big impact on the world through their work in EA.
  • Attention on current harms from AI systems does not undermine efforts to reduce risks from future AI systems.
  • Policy proposals to address current AI harms can also advance the cause of reducing risks from future AI systems.
  • Open source and open weights are important for mitigating current and near-term risks from AI systems.

Debates on Open Weights, Cognitive Dissonance, and Ethical AI Companies

34:12 - 41:23

  • Open weights in AI are crucial for reducing concentration of power and advancing safety research.
  • Debates around releasing model weights have evolved since GPT-2, with considerations on risks and benefits.
  • Keeping AI models closed can limit scientific progress but is understandable due to significant investment in training them.
  • Cognitive dissonance exists in big AI companies regarding concerns about future AI systems while pursuing AGI development.
  • Companies like Enthropic are exploring unique approaches like a publicly sourced constitution amidst complex interrelations within the industry.

Ethics and Safety in AI Development and Agentic AI Systems

41:00 - 48:19

  • Companies like Open AI and Antropic prioritize ethics and safety research in AI development.
  • Concerns raised about new AI startups not focusing on ethics and safety in their development process.
  • Some founders of early stage AI companies actively refuse funding from VCs associated with statements about AI safety.
  • Trend of building agentic AI systems with a focus on doing good for humanity observed among some developers.
  • Recognition of companies like Inbute for prioritizing safety and policy in their AI development.

Safety Measures, Algorithmic Systems, and Societal Impacts

47:55 - 55:06

  • Safety and alignment research is crucial in preventing toxic behavior in AI models like chat GPT.
  • Neglecting safety measures can lead to scandals and regulatory issues, as seen in social media platforms.
  • It's important to consider who has the authority to make decisions about algorithmic systems impacting society.
  • The debate between model capabilities and alignment/safety is ongoing, with some arguing that capabilities inherently include safety.
  • Decisions about new technologies should not be left solely in the hands of unaccountable individuals to prevent significant societal impacts.

LLMs, Feasibility Horizons, and AI Safety Efforts

54:39 - 1:01:58

  • LLMs demonstrate good moral judgment when asked to consider morally relevant considerations.
  • Feasibility horizons are used to anticipate outcomes based on existing systems and normal science improvements.
  • Scientific advances beyond the current feasibility horizon are unpredictable, leading to challenges in influencing future technologies.
  • Focusing on building robust research communities, regulations, norms, and practices is seen as a key strategy for preparing for risks from future technologies.
  • AI safety efforts could be reoriented towards addressing risks from technologies that are well understood and empirical.

Understanding Risks, Socio-Technical Approach, and Costly Policies

1:01:28 - 1:08:36

  • Focus on understanding risks from technologies that are empirical and testable
  • AI safety field needs to mature by incorporating a socio-technical approach
  • Feasibility horizons in AI involve considering potential capabilities beyond current scientific paradigms
  • Scaling may lead to emergent capabilities that render prior safety measures inadequate
  • Prioritize understanding and ensuring the safety of current AI systems before advancing to more speculative work
  • Costly policies should be based on solid arguments rather than speculation about future technology

Balancing Safety and Advancements, Explainable AI, and Trust Relationships

1:08:22 - 1:15:16

  • Discussion on the need for better arguments when implementing costly decisions
  • Concerns about the lack of full understanding of advanced AI systems and their workings
  • Importance of balancing safety guarantees with advancements in capabilities in AI development
  • Different perspectives on the level of understanding needed for societal impacts and moral considerations in AI technology
  • Considerations around explainability and transparency in AI decision-making processes

Evolution of Explainable AI, Manipulation, and Public Interest Research

1:14:48 - 1:22:18

  • Explainable AI has evolved from a purely mathematical concept to a more user-centered and socio-technical perspective over the years.
  • There is a debate between the need for faster, more user-centered systems and the benefits of slower development to consider consequences.
  • Approaches like privacy by design, fairness by design, and safety by design in AI may not fully address complex socio-technical problems without broader perspectives.
  • Integration of technical AI safety work with a broader system safety perspective is essential for real progress in AI safety.
  • Real manipulation and persuasion often stem from relationships of trust rather than just advanced language capabilities.

Radicalization, Double Agents, and Ethical AI Development

1:21:56 - 1:28:37

  • Radicalization can occur through forming bonds with individuals who manipulate and reward praise.
  • AI systems need to work across multiple modalities to be effective manipulators.
  • Proposed experiment 'Double Agents' aims to build trust with users, exfiltrate data, and study human-AI relationships.
  • Using LLMs for simulating human behavior in studies like building double agents is considered ethical.

Ethics and Safety in AI Development, Public Interest Research, and Collaboration

1:28:22 - 1:35:32

  • Debut of GPT-4 on Bing was engaging and eye-opening, highlighting the importance of ethics and safety in AI development
  • Public interest research is crucial to understanding the risks associated with developing AI systems like double agents and companions
  • There are concerns about companies not taking ethics and safety seriously in building AI systems, emphasizing the need for public interest-driven research
  • Collaboration between tech companies and public interest researchers, supported by regulation, is essential for ensuring ethical AI development

Algorithmic Transparency, Safe Harbor for Research, and Academic Ecosystems

1:35:13 - 1:42:26

  • Collaboration between tech companies and public interest researchers is valuable and may require regulation.
  • Algorithmic transparency efforts for social media companies should also apply to tech companies.
  • Support for safe harbor for public interest academic research on frontier AI is crucial.
  • Establishing ecosystems for independent academic research is necessary, such as the National AI Research Resource.
  • Importance of having publication venues that bridge different disciplines for academic research in AI ethics and safety.
  • Challenges in academia include retaining talent interested in public interest work due to industry opportunities.

Interdisciplinary Work, Conferences, and Incentives

1:42:01 - 1:49:21

  • The importance of working within existing structures to build legitimacy in interdisciplinary work
  • Challenges in making cross-disciplinary work valued and legitimate
  • The significance of conferences like Fact in promoting interdisciplinary collaboration
  • Incentives and challenges faced by academics in pursuing interdisciplinary research
  • Encouragement for PhD students and engineers to engage in existing interdisciplinary communities and networks

Public Input, Expertise, and Collaboration

1:48:53 - 1:50:17

  • Consider incorporating public input to respond to real needs
  • Evaluate the need for hiring someone with relevant expertise
  • Connect with individuals in your network who can help operationalize ideas
  • Ensure that those engaged in the work are supported and have necessary connections
1