Syntax - Tasty Web Development Treats

Supper Club × Messaging Queues and Workers with Armin Ronacher

Fri Jun 30 2023

Messaging QueuesMicroservicesTask HandlingSystem DesignEvent IngestionDevelopment ChallengesJavaScript EcosystemInsights

Description

This episode covers the fundamentals of messaging queues, their applications in microservices, task handling and scheduling, system design considerations, event ingestion architecture, challenges faced in queuing systems, development challenges in Python and Rust, complexities in the JavaScript ecosystem, and insights on maintaining personal health in open source projects.

Insights

Queues are essential for microservices

Microservices can communicate effectively using queues, which abstract away complexities and handle tasks like sending emails and processing payments.

Queues enable task handling and scheduling

Queues allow for retries, scheduling tasks at specific times, and storing tasks until services are back online. Different systems offer various ways to implement task queues.

System design considerations for queues

Different types of problems require different types of queues. Back pressure management is crucial, and having multiple independent queues can help manage workload.

Event ingestion architecture and challenges

Event ingestion systems like Sentry handle a large number of requests per second, with back pressure management and routing based on event types. Fetching source maps and handling mobile app SDKs pose challenges.

Development challenges in Python and Rust

Mobile development involves multiple layers and dependencies, while Rust offers better developer experience than C++. Python is widely used in AI but faces packaging and developer experience challenges. JavaScript on the server can be complex.

Challenges in the JavaScript ecosystem

The JavaScript ecosystem is complex and constantly evolving, with frequent updates to SDKs. Next.js has high churn. The guest shares their development setup and staying motivated in open source.

Insights and recommendations

Maintaining personal health in open source involves avoiding pushy projects, muting certain creators, and focusing on specific topics. The guest recommends an Austrian band called Buildable and a Python package manager called Pyenv.

Summary

Transcript

Introduction to Messaging Queues

00:00 - 07:41

Armin Rona, principal architect at Sentry, joins the podcast to discuss messaging queues.
Armin has a background in Python and open source.
He has built frameworks and utilities for developers, including Flask and Ginger.
Armin is currently interested in Rust and fixing Python packaging.
Sentry's teams in Vienna handle data ingestion into the system.
Messaging queues are used when an app needs to pretend work is done but it's not yet completed.
The idea is to distribute work to workers who will pick it up from the queue and eventually produce a result.
Using queues can free up HTTP request handlers to handle incoming data more efficiently.
Queues can be run on separate servers or as part of the same infrastructure depending on the use case.

Using Queues for Microservices

07:16 - 14:00

Building tiny microservices has become popular, but it can lead to a messy situation of different languages and technologies.
Queues are a good way to make microservices communicate with each other.
Different queue systems have different qualities and behaviors, such as RabbitMQ and Kafka.
Frameworks like Celery in Python help manage queues by abstracting away the complexities.
Common examples of tasks put into queues include sending emails and processing payments.
Queues are useful for handling external services that may be fire-and-forget or have unpredictable response times.
Queues can also be helpful in avoiding downtime by storing tasks until the service is back online.

Handling Tasks and Scheduling

13:34 - 20:06

Queues are a good way to handle tasks that may fail and require retries.
Payment transactions often need multiple retries, especially for credit cards that may be maxed out at the end of the month.
Certain queues allow you to schedule tasks at specific times, like running a task again in a day.
Postgres can be used as a queue for tasks that require strong persistence and long execution times.
Python's Celery framework has components like celery beat for scheduling tasks onto the queue.
Different systems offer various ways to implement task queues, depending on your needs.
Cron jobs can be added to queues to perform scheduled tasks.
Queues can be stored in databases as well.
Understanding queuing theory is important when building scalable systems with queues.
Idempotent tasks are those that won't create different results if run multiple times, while non-idempotent tasks may have issues if run twice.
Deciding what to do with a large backlog of items in the queue depends on your specific needs and goals.
The architecture of a task queue system can vary depending on the problem it aims to solve. It could involve state machines, code, or diagrams.

Types of Queues and System Design

19:47 - 26:26

Different types of problems require different types of queues
Scaling up certain systems like Rabbit is easier than others like Kafka
Back pressure management is crucial in queuing systems
Without back pressure, the system can become overwhelmed
Designing a system with infinite queues can lead to problems
Having multiple independent queues can help manage workload
Adding another queue may be necessary when resources are strained

System Architecture and Event Ingestion

26:03 - 33:15

Keeping independent systems on a code configuration level allows for splitting them onto dependent queues.
Kafka requires separating systems from the beginning and careful consideration of scaling due to its lack of fair distribution of work.
Sentry handles around 300,000 requests per second for event ingestion, including errors, session replays, performance metrics, and session data.
Not all events that reach the ingestion system are kept, such as those from customers over quota or unpaid.
Back pressure management is implemented in client SDKs to communicate with the injection system and prevent excessive traffic.
Events are split into different types and routed accordingly within the system.
Kafka provides disk buffering capabilities in case of extended downtime between systems.
Revit operates purely out of memory and has limitations on the number of events it can handle.
A processing pipeline handles events with varying processing times, depending on factors like downloading debug files or fetching source maps.

Challenges and Considerations

32:47 - 39:38

Fetching source maps for unified JavaScript files can be time-consuming and unpredictable
Visualizing a complex queuing system at scale is challenging
Losing knowledge and visibility in the system due to pre-aggregation and batch processing
Complexity increases when scaling, leading to edge cases being overlooked
Example of a mobile app SDK not upholding the contract of discarding old data
Customers' devices with local buffers can impact the queuing system
Dependence on customers to update SDKs introduces uncertainty and potential issues

Python, Rust, and Development Challenges

39:14 - 46:24

Mobile development is challenging because it involves multiple layers and dependencies.
Fixing bugs in mobile apps takes longer due to the approval process in app markets.
Rust is being used at Century for client-side CLI tools, core ingestion systems, and native crash reporting.
Rust's developer experience is better than C++ when dealing with binary data and compiler ecosystems.
Reviving Python projects in Rust, such as PyO3 and Maturin, allow writing Python extension modules in Rust.
Python is widely used in AI due to existing libraries and lack of strong competitors like Julia.
Packaging and developer experience around Python can be frustrating for AI developers.
JavaScript on the server can be complex and confusing for developers accustomed to traditional application development practices.

Challenges in the JavaScript Ecosystem

46:02 - 53:11

The growing complexity of the JavaScript ecosystem is high and there is a lot of churn.
Updates to SDKs are frequent and it takes effort to keep up with them.
Next.js may have the most churn due to its large user base.
The guest uses a MacBook Pro, a self-built Windows computer, and a Filco keyboard.
They use Vim or Helix as their text editor and Mona Lisa font.
For terminal and shell, they use ZSH on Mac and Windows Terminal on Windows.
They don't use Linux because the Mac ecosystem of apps is not compatible.
Staying motivated in open source is challenging, but adoption helps drive motivation.
The guest uses Twitter and Reddit to stay up to date but maintains mute lists for certain topics.

Insights and Recommendations

52:45 - 59:00

Avoiding pushy and marketing-y open source projects to maintain personal health
Muting certain creators and companies with an in-your-face marketing vibe
Staying up to date by focusing on specific topics that go in and out of favor over time
Sick pick: Austrian band called Buildable, known for their summer music
Shameless plug: Package manager for Python called Pyenv