· 5 min
Designing Data-Intensive Applications
Reading and summarizing.
Preface
Q. Why data-intensive? A. In modern applications, processing large and varied data consistently and without slowing down is generally identified as the bottleneck. Applications have to be designed with this in mind.
Chapter 1. Reliable, Scalable, and Maintainable Applications
- Reliability: emphasized through various cases of service failures.
- Scalability: load can be expressed as a function. You need to calculate the load the system is asked to handle, e.g. n tweets written per second and m tweets displayed. If response time is the performance metric, the median is seen as p50 and outliers as p95–99. These percentiles often appear in SLO and SLA contracts. For handling load, stateless services scale out easily, but traditional stateful services (databases) are tricky.
- Maintainability: to minimize the pain of legacy systems, build a simple system (code) that is easy to operate and easy to change. The key is abstraction.
Chapter 2. Data Models and Query Languages
This chapter explains the following for the relational, document, and graph data models.
- Historical background
- How well each data model fits, depending on locality and the complexity of relationships
- Imperative and declarative queries
- Examples of databases and the query languages they use
The important point is that no data model is capable enough to fully replace another. So they have to be used together as needed. In my view, choosing the data model that fits the service's domain model comes first, but deciding not to increase the number of data models in use, out of concern for the complexity of the whole system, can be reasonable. For example, as long as performance is acceptable, you can store a JSONB column in a relational database such as PostgreSQL to cut down complex joins on reads. It is a trade-off that depends on the situation.
Chapter 3. Storage and Retrieval
Log-structured
- The log family works like appending a log as a new line at the end of a file. It is simple, with good write performance and less impressive read performance.
- Modifying only a single file would make it too large, so segments are split from the memtable down through level 1 on disk to the final level 6. At each stage, compaction sorts the data and stores it as segments in the level below. The newest information is stored last, so lookups go from the top down.
- For read speed, the keys can simply be indexed with an in-memory hash map, Bitcask-style. Alternatively, the keys stored in each segment can be passed through hash functions and the resulting values used to estimate the probability that a key exists, which speeds up lookups.
- The lower-numbered levels have small buffers; when a buffer is nearly full, the data from the next level is brought in and brought up to date, and segments are stored at that level. Think of it as the key range and the number of segments growing, and the data getting tidier, the further down the levels you go.
B-tree
| Comparison | Page-based storage (B-Tree family) | Log-structured storage (LSM-Tree family) |
|---|---|---|
| Examples | MySQL (InnoDB), PostgreSQL, Oracle | RocksDB, Apache Cassandra, LevelDB |
| Core structure | Fixed-size pages (8–16KB), B-Tree / B+Tree | MemTable (in-memory), SSTable (disk) |
| How data changes | In-place update | Append-only / out-of-place update |
| Write performance | Relatively low (mostly random writes) | Very high (mostly sequential writes) |
| Point read | High (fast access by traversing the B-Tree) | Medium to low (searches the MemTable and multiple SSTables) |
| Range read | Very high (sorted pages and linked leaf nodes) | Medium (merge-sorts results from several SSTables) |
| Write amplification (WAF) | High (a small change rewrites the whole page) | Low at first (rises when compaction runs) |
| Space efficiency | Medium (fragmentation and empty space within pages) | High (high compression thanks to immutable structures) |
| Background load | Low (mostly the checkpointer) | High (continuous compaction I/O) |
| Suitable workload | Read-heavy OLTP environments where consistency matters | Write-heavy environments that ingest large volumes of data |
Transaction Processing and Analytics
| Comparison | OLTP (online transaction processing) | OLAP (online analytical processing) |
|---|---|---|
| Main purpose | Fast, accurate processing of everyday business transactions | Multidimensional analysis for data analysis, reporting, and decision-making |
| Main users | General users, business staff, customers (many concurrent users) | Data analysts, data engineers, executives (a small number of users) |
| Unit of processing | CRUD on single records (Create, Read, Update, Delete) | Reading and aggregating large numbers of records (SUM, AVG, COUNT, etc.) |
| Transaction characteristics | Short, fast, small transactions; ACID guarantees required | Long, complex, large queries; read performance matters more than transactional guarantees |
| Response time | Very fast, in milliseconds (ms) | Seconds to minutes (proportional to query complexity) |
| Data structure | Normalization-centered (minimal duplication) | Denormalization-centered (Star/Snowflake schema) |
| Data freshness | Real-time (current data) | Periodically extracted and transformed historical data |
| Storage characteristics | Prefers row-oriented stores | Prefers columnar stores |
| Representative tech/DB | MySQL, PostgreSQL, Oracle, SQL Server | Snowflake, BigQuery, ClickHouse, Amazon Redshift |
Comments 0
Be the first to comment.