Syntax - Tasty Web Development Treats
Supper Club × Messaging Queues and Workers with Armin Ronacher
Queues are essential for microservices
Microservices can communicate effectively using queues, which abstract away complexities and handle tasks like sending emails and processing payments.
Queues enable task handling and scheduling
Queues allow for retries, scheduling tasks at specific times, and storing tasks until services are back online. Different systems offer various ways to implement task queues.
System design considerations for queues
Different types of problems require different types of queues. Back pressure management is crucial, and having multiple independent queues can help manage workload.
Event ingestion architecture and challenges
Event ingestion systems like Sentry handle a large number of requests per second, with back pressure management and routing based on event types. Fetching source maps and handling mobile app SDKs pose challenges.
Development challenges in Python and Rust
Insights and recommendations
Maintaining personal health in open source involves avoiding pushy projects, muting certain creators, and focusing on specific topics. The guest recommends an Austrian band called Buildable and a Python package manager called Pyenv.
- Introduction to Messaging Queues
- Using Queues for Microservices
- Handling Tasks and Scheduling
- Types of Queues and System Design
- System Architecture and Event Ingestion
- Challenges and Considerations
- Python, Rust, and Development Challenges
- Insights and Recommendations
Introduction to Messaging Queues
00:00 - 07:41
- Armin Rona, principal architect at Sentry, joins the podcast to discuss messaging queues.
- Armin has a background in Python and open source.
- He has built frameworks and utilities for developers, including Flask and Ginger.
- Armin is currently interested in Rust and fixing Python packaging.
- Sentry's teams in Vienna handle data ingestion into the system.
- Messaging queues are used when an app needs to pretend work is done but it's not yet completed.
- The idea is to distribute work to workers who will pick it up from the queue and eventually produce a result.
- Using queues can free up HTTP request handlers to handle incoming data more efficiently.
- Queues can be run on separate servers or as part of the same infrastructure depending on the use case.
Using Queues for Microservices
07:16 - 14:00
- Building tiny microservices has become popular, but it can lead to a messy situation of different languages and technologies.
- Queues are a good way to make microservices communicate with each other.
- Different queue systems have different qualities and behaviors, such as RabbitMQ and Kafka.
- Frameworks like Celery in Python help manage queues by abstracting away the complexities.
- Common examples of tasks put into queues include sending emails and processing payments.
- Queues are useful for handling external services that may be fire-and-forget or have unpredictable response times.
- Queues can also be helpful in avoiding downtime by storing tasks until the service is back online.
Handling Tasks and Scheduling
13:34 - 20:06
- Queues are a good way to handle tasks that may fail and require retries.
- Payment transactions often need multiple retries, especially for credit cards that may be maxed out at the end of the month.
- Certain queues allow you to schedule tasks at specific times, like running a task again in a day.
- Postgres can be used as a queue for tasks that require strong persistence and long execution times.
- Python's Celery framework has components like celery beat for scheduling tasks onto the queue.
- Different systems offer various ways to implement task queues, depending on your needs.
- Cron jobs can be added to queues to perform scheduled tasks.
- Queues can be stored in databases as well.
- Understanding queuing theory is important when building scalable systems with queues.
- Idempotent tasks are those that won't create different results if run multiple times, while non-idempotent tasks may have issues if run twice.
- Deciding what to do with a large backlog of items in the queue depends on your specific needs and goals.
- The architecture of a task queue system can vary depending on the problem it aims to solve. It could involve state machines, code, or diagrams.
Types of Queues and System Design
19:47 - 26:26
- Different types of problems require different types of queues
- Scaling up certain systems like Rabbit is easier than others like Kafka
- Back pressure management is crucial in queuing systems
- Without back pressure, the system can become overwhelmed
- Designing a system with infinite queues can lead to problems
- Having multiple independent queues can help manage workload
- Adding another queue may be necessary when resources are strained
System Architecture and Event Ingestion
26:03 - 33:15
- Keeping independent systems on a code configuration level allows for splitting them onto dependent queues.
- Kafka requires separating systems from the beginning and careful consideration of scaling due to its lack of fair distribution of work.
- Sentry handles around 300,000 requests per second for event ingestion, including errors, session replays, performance metrics, and session data.
- Not all events that reach the ingestion system are kept, such as those from customers over quota or unpaid.
- Back pressure management is implemented in client SDKs to communicate with the injection system and prevent excessive traffic.
- Events are split into different types and routed accordingly within the system.
- Kafka provides disk buffering capabilities in case of extended downtime between systems.
- Revit operates purely out of memory and has limitations on the number of events it can handle.
- A processing pipeline handles events with varying processing times, depending on factors like downloading debug files or fetching source maps.
Challenges and Considerations
32:47 - 39:38
- Visualizing a complex queuing system at scale is challenging
- Losing knowledge and visibility in the system due to pre-aggregation and batch processing
- Complexity increases when scaling, leading to edge cases being overlooked
- Example of a mobile app SDK not upholding the contract of discarding old data
- Customers' devices with local buffers can impact the queuing system
- Dependence on customers to update SDKs introduces uncertainty and potential issues
Python, Rust, and Development Challenges
39:14 - 46:24
- Mobile development is challenging because it involves multiple layers and dependencies.
- Fixing bugs in mobile apps takes longer due to the approval process in app markets.
- Rust is being used at Century for client-side CLI tools, core ingestion systems, and native crash reporting.
- Rust's developer experience is better than C++ when dealing with binary data and compiler ecosystems.
- Reviving Python projects in Rust, such as PyO3 and Maturin, allow writing Python extension modules in Rust.
- Python is widely used in AI due to existing libraries and lack of strong competitors like Julia.
- Packaging and developer experience around Python can be frustrating for AI developers.
46:02 - 53:11
- Updates to SDKs are frequent and it takes effort to keep up with them.
- Next.js may have the most churn due to its large user base.
- The guest uses a MacBook Pro, a self-built Windows computer, and a Filco keyboard.
- They use Vim or Helix as their text editor and Mona Lisa font.
- For terminal and shell, they use ZSH on Mac and Windows Terminal on Windows.
- They don't use Linux because the Mac ecosystem of apps is not compatible.
- Staying motivated in open source is challenging, but adoption helps drive motivation.
- The guest uses Twitter and Reddit to stay up to date but maintains mute lists for certain topics.
Insights and Recommendations
52:45 - 59:00
- Avoiding pushy and marketing-y open source projects to maintain personal health
- Muting certain creators and companies with an in-your-face marketing vibe
- Staying up to date by focusing on specific topics that go in and out of favor over time
- Sick pick: Austrian band called Buildable, known for their summer music
- Shameless plug: Package manager for Python called Pyenv