Part 1/11:

# How Uber's Engineers Handle Billions of Payments: The Architectural Journey of a Payment System

In today's digital age, managing payments for a vast user base is a daunting challenge that many tech companies face. Uber, renowned for its ride-hailing convenience, is no stranger to this complexity. With over 100 million users and billions of rides, the need for an efficient and reliable payment system became paramount as the service scaled. This article delves into the evolution of Uber's payment system and the innovative solutions its engineering team implemented to ensure seamless transactions.

## A Closer Look at Uber's Payment Model

Part 2/11:

At its core, the Uber model is simple: users request a ride, get picked up by a driver, and make a payment that is then distributed appropriately between them and Uber. However, as the platform grew, this straightforward process became more complicated. The sheer volume of transactions, collectively totaling hundreds of billions of dollars, brought with it instability, fragmentation, and severe transaction delays.

Imagine being an Uber driver waiting for payments after days of completed rides only to face a cyberattack that disrupted the entire payment system. Such scenarios triggered the urgent need for Uber to overhaul its payment infrastructure.

## Transition from Monolith to Microservices

Part 3/11:

Initially, Uber operated on a monolithic application architecture. However, as the demand surged, the engineering team transitioned to a microservices architecture comprised of thousands of individual services. They replaced previous database systems with Postgres and MySQL while continuously building out their technology stack. Despite these changes, they encountered significant issues with transaction processing, leading to the decision to rebuild their payment system from the ground up.

## Introducing the Job Order System

Part 4/11:

To address these challenges, Uber adopted a job order system—an innovative approach that handles transactions as jobs. Each job represents a specific trip or order, which can accommodate multiple associated requests. For instance, a user might request a drop-off at home with a stop at a fast-food restaurant along the way.

Within these jobs, monetary entries are structured to reflect the flow of funds from the payer to Uber. This maintained a critical concept: the sum must always equal zero, mirroring the principles of double-entry accounting—there can be neither creation nor destruction of money.

## Reinvigorating Financial Integrity with Immutable Ledgers

Part 5/11:

One of the vital considerations in developing the payment infrastructure was ensuring data integrity. Traditional databases tend to be mutable, allowing for changes that can compromise the accuracy of financial records. In contrast, Uber initiated the use of an immutable ledger system—where corrections are appended rather than altering previous entries.

This approach guarantees that if an error occurs during a transaction, such as a billing dispute, the issue is resolved by recording an additional transaction instead of overwriting the previous record. This rigid structure enhances the reliability of the financial data, ensuring clarity in the face of potential discrepancies.

## Development of Ledger Store

Part 6/11:

Taking their ledger strategy further, the team created Ledger Store, an application layer designed to work with NoSQL databases. Ledger Store introduces three pivotal concepts:

- **Sealing**: This involves closing a time span of transactions, making them read-only to prevent alterations after verification.

- **Manifest**: Files generated during the sealing process enhance security by documenting which transactions can be accessed and modified.

- **Revision**: This concept allows for documenting corrections while keeping the original data searchable.

To facilitate real-time processing of billions of transactions, Uber integrated Apache Kafka, a robust event streaming tool that helped manage the continuous flow of data.

## Overcoming Migration Challenges

Part 7/11:

As Uber contemplated moving to a new database, they faced the formidable task of migrating over 250 billion records—amounting to approximately 300 terabytes of data. They ultimately decided to process data in smaller batches rather than migrating everything at once, minimizing the risk of failure during such a monumental transition.

Their ingenious solution involved booting up workers that would process data from a checkpoint in cold storage, eliminating the risk of repeatedly processing already completed sections. This strategy not only saved time but also reduced costs during the migration.

## Shadow Rider: Managing Dual Databases

Part 8/11:

With the new database systems in the pipeline, Uber introduced the Shadow Rider, an intermediary service that allowed for syncing data between their current and new databases. It ensured that while data was being pushed to the main database synchronously, the new system faced no performance issues as it saved data asynchronously.

However, they had to ensure reliability during this asynchronous process, prompting the development of a spot repair job to identify and fix issues in real-time. A dual read and merge system was also implemented to enable the process of reconciling and validating data from both databases seamlessly.

## Indexing: A Key Component of Efficiency

Part 9/11:

With operations reliant upon trillions of transactions, efficient indexing became critical. Uber engineers designed a hierarchical indexing system, creating multiple indexes tailored for different functions, including strongly consistent indexes for sensitive transactions like credit card authorizations and eventually consistent indexes for less critical data.

The approach involved timely management of old data by utilizing time range indexes, allowing inefficient older records to migrate to cheaper, cold storage solutions.

## Conclusion: The Complexity of Upscaling Payment Solutions

Part 10/11:

Uber's engineers have demonstrated an incredible level of innovation and resilience in building a payment system that can handle billions of transactions without a hitch. The journey from a rudimentary payment model to a sophisticated multi-tiered architecture underscores the vastness of engineering challenges faced by a rapidly expanding tech enterprise.

The insights shared from Uber's engineering experience not only reflect the intricacies involved in financial technology but also highlight the need for continuous adaptation and creativity in addressing ever-scaling user demands.

Part 11/11:

In a world where digital payment systems are becoming increasingly complex, it is essential for tech giants like Uber to share their knowledge openly—applauding their engineering teams for their hard work, perseverance, and ability to embrace challenges that come with success.