Software Engineering Daily

Cloud-native Search with Paul Masurel

Tue Jul 18 2023

Search EngineData ManagementCluster ManagementLatency OptimizationRust Development

Description

This episode explores the challenges faced by existing search engines and how QuickWit addresses them. It focuses on reducing costs, improving reliability, and managing clusters for large amounts of data. The architecture involves ingesting data, crunching it using CPU, writing to disk, uploading to a store, and communicating with the metastore. QuickWit excels at maximizing CPU usage for indexing throughputs and reducing search latency. It also discusses the benefits of using Rust for development and recent updates in QuickWit's features and future plans.

Insights

QuickWit specializes in searching append-only cloud data

By focusing on specific types of data, QuickWit provides faster search times and lower costs compared to Elasticsearch.

Efficient cluster management is crucial for reducing costs

QuickWit's core architectural decisions aim to minimize expenses and simplify cluster management while providing efficient search capabilities for large amounts of data.

SSG improves latency and throughput for search on Amazon S3

Spinning Storage Gateway (SSG) allows multiple reads at the same time, reducing the number of hops and improving search performance.

Optimizing search performance involves splitting indexes and using time pruning

By splitting the index into smaller pieces and reducing the amount of data that needs to be searched, QuickWit improves search speed and efficiency.

QUIQ offers a cost-effective solution for incremental data

QUIQ is 10 times cheaper and faster than Elastic for append-only logs, providing consistency and easy outcome determination for ledger situations.

Data retention and deletion are handled by Quick-wits

Quick-wits offers automatic deletion after a set period of time and handles large requests like GDPR or churn-related data removal.

Rust provides safety and performance optimization for development

Rust is a simpler alternative to C++ with a higher level of safety when writing multi-threaded code. It allows for easy performance optimization by examining assembly code.

QuickWit's recent updates include support for graphaners and an Elastic Software compatible API

The main objective is to finish ongoing projects and put them into production. QuickWit is actively seeking design partners and customers for larger use cases.

Growing demand for solutions like QuickWit in various industries

The exponential growth of data in finance, medical, and government sectors creates a need for efficient search engines like QuickWit.

Chapters

Introduction
Reducing Costs and Cluster Management
Scalability and Object Storage
Improving Latency and Throughput with SSG
Optimizing Search Performance
Use Cases and Product Comparison
Data Retention and Deletion
Benefits of Rust for Development
Recent Updates and Future Plans

Summary

Transcript

Introduction

00:00 - 08:46

Elasticsearch is the most established solution for searching and analyzing large volumes of logs, but it can be costly and complex to manage.
QuickWit is a search engine that specializes in searching append-only cloud data like logs and ledgers, providing faster search times and lower costs compared to Elasticsearch.
Paul Masserow, the creator of Tantivi and founder of QuickWit, discusses the challenges faced by existing search engines and why QuickWit focuses on specific types of data.
The main challenges include decreasing costs, improving reliability, and managing clusters for large amounts of data.
Append-only data presents a particular problem because modifying or deleting documents requires making changes to every piece of the index that has been produced before. QuickWit addresses this by processing delete requests asynchronously in batches.
QuickWit's core architectural decisions focus on append-only data, reducing costs, and improving cluster management.
The goal is to provide efficient search capabilities for large amounts of data while minimizing expenses and simplifying cluster management.

Reducing Costs and Cluster Management

08:22 - 16:14

QuickWit addresses the core architectural decision of reducing cost and making it easier to manage clusters.
In the realm of security, people want to index everything and keep it for three years, but only one person searches the data when there is a security issue.
The index is not just for searching, but also for providing peace of mind.
The challenge in this case is how to index data quickly and efficiently.
QuickWit excels at maximizing CPU usage for indexing throughputs.
The architecture involves ingesting data, crunching it using CPU, writing to disk, uploading to a store, and communicating with the metastore.
Each step in the process consumes specific resources of the system.
A streamlined pipeline is necessary to optimize the process and maximize efficiency.
The interconnectedness of architectural components in QuickWit's architecture resembles driving a standard vehicle with precise timing and coordination.
Reducing search latency while using maximum CPU capacity is a challenge that can be solved by launching indexing and search on different hardware.

Scalability and Object Storage

16:00 - 24:04

Kafka 2 is used to ingest a massive stream of logs for the whole company
Indexing this massive amount of data involves a set of indexers working on their own
After finishing one batch, the indexers publish their splits to Amazon S3 via Metastore
The searchers in the separate world are stateless and coordinate work in a distributed way
Searchers connect to Metastore and dispatch work amongst themselves using gossip
The architecture separates searcher and indexer, allowing for rapid scaling and manageability
All data is stored on object storage like Amazon S3, which is cost-effective compared to local SSDs
Amazon S3 has slow latency, making traditional search methods impossible
Dealing with latency was a challenge that had to be overcome

Improving Latency and Throughput with SSG

23:37 - 32:13

Search on Amazon S3 is slow and has poor throughput
SSG (Spinning Storage Gateway) is a solution to improve latency and throughput
SSG allows multiple reads at the same time, increasing throughput
SSG reduces the number of hops to improve latency
SSG can handle large amounts of data but is slower for smaller amounts

Optimizing Search Performance

31:53 - 40:11

The search engine is fast with larger amounts of data but slow with smaller amounts
Indexing involves splitting the index into smaller pieces to improve search performance
Time pruning reduces the amount of data that needs to be searched
Inverted indexes are a common data structure used in search engines
Some systems choose not to build an index and rely on pruning and distribution instead
ClickHouse is a fast solution for business analytics, but searching can be slow
Using ClickHouse as a secondary index can speed up searches by leveraging its strengths
ClickHouse is a cost-effective option for faster results compared to traditional indexing

Use Cases and Product Comparison

39:43 - 47:39

Indexes in their product work well and offer cost savings and faster results
Use cases discussed include logs, traces, and complex setups
Startup playbook involves solving a specific problem and finding the right positioning
Competing with Elastic posed challenges in finding product-market fit
Elastic is a Swiss Army knife product that does many things but is not good for up-and-down data
Example of Elastic's inefficiency is document replication, which leads to unnecessary workload
QUIQ offers a solution that is 10 times cheaper and faster for incremental data
One use case for QUIQ is append-only logs for ledger situations
Append-only logs provide consistency and easy outcome determination
Ledgers in multinational banks face challenges with order of transactions and account balance checks

Data Retention and Deletion

47:21 - 55:14

Deleting documents in indexing can be tricky, especially with GDPR and permanent removal
Quick-wits offers data retention and automatic deletion after a set period of time
Quick-wits also handles large requests like GDPR or churn-related data removal
The founder's background is in search engine development
Algolia was spun out of ExaLid, a French search engine company
The founder worked at Atiku and Daikou before moving to Japan and joining Google
Tantivy, an open-source library developed by the founder, is popular among search engineers
Tantivy was created as an experiment to learn Rust and understand machine internals

Benefits of Rust for Development

54:53 - 1:02:48

The speaker never restarted their first project in Rust and found it to be a good language for refactoring.
Many people believe that Rust has a steep learning curve, but the speaker found it more efficient and productive than C++ after just two weeks of learning.
The speaker suggests that those familiar with C++ should try learning Rust as it is simpler than they think.
Creating a project that helps understand the inner workings of the system is better than doing basic tutorials.
While Java is a good language, Rust provides a higher level of safety when writing multi-threaded code.
Rust allows for easy performance optimization by examining assembly code, which is difficult to do in Java.

Recent Updates and Future Plans

1:02:18 - 1:08:46

QuickWit 0.6.6 has been released with new features including support for graphaners and an Elastic Software compatible API.
The main objective is to finish ongoing projects and put them into production.
Customers typically start working with QuickWit through small projects or big efforts.
A pipeline of potential customers and design partners is being developed through various channels, including podcasts.
Tests are conducted with a few petabytes of data to evaluate outcomes and identify larger use cases or potential contracts.
Some users have been indexing large amounts of data without informing QuickWit.
There is a growing demand for solutions like QuickWit due to the exponential growth of data in various industries such as finance, medical, and government sectors.