Azure Cosmos DB - Melhores Práticas | Big Data Masters

Summary

In this video, Victor highlights critical mistakes data engineers often make when using Azure Cosmos DB. He explains the risks of not enabling multi-region replication, which can lead to downtime if a data center fails, and stresses the importance of selecting an effective partition key to ensure optimal performance. Additionally, he discusses the dangers of under-provisioning throughput resources, which can result in request throttling. By addressing these issues, engineers can ensure their Cosmos DB setup is reliable, scalable, and performs efficiently, avoiding costly disruptions and downtimes.

Takeaways

• 😀 Enabling multi-region replication is crucial to ensure high availability and disaster recovery for Cosmos DB.
• 😀 Not enabling automatic failover for the primary region puts the entire system at risk in case of failures.
• 😀 It’s important to create at least one writeable replica to avoid downtime if the primary region fails.
• 😀 Cosmos DB offers extremely low-latency data access, making it ideal for real-time data architectures in big data and analytics.
• 😀 Partition key selection is essential for Cosmos DB performance. Incorrect or random partitioning can lead to significant performance issues.
• 😀 If a poorly chosen partition key leads to an unbalanced distribution of data, Cosmos DB may experience resource limitations, leading to failures.
• 😀 A partition in Cosmos DB can hold only 20 GB of data. Once this limit is exceeded, no more data can be written until the size is reduced.
• 😀 A bad partition strategy might require recreating a collection and migrating data, which is time-consuming and inefficient.
• 😀 Be sure to plan partitioning ahead of time to ensure data is distributed evenly across all partitions, optimizing read and write performance.
• 😀 Under-provisioned resources can cause issues with Cosmos DB’s physical partitions, limiting the ability to handle increased load efficiently.

Q & A

What is the main topic of the video?

The main topic of the video is discussing common mistakes made by data engineers when using Microsoft Azure's Cosmos DB and providing advice on how to avoid these issues.

Why is it important to enable multi-region support in Cosmos DB?

Enabling multi-region support ensures that if a failure occurs in the primary region, the application can still function by redirecting requests to a secondary region. This helps in maintaining high availability and data reliability.

What could happen if multi-region support is not enabled?

If multi-region support is not enabled, your application could become unavailable if the primary data center experiences a failure, leaving the organization without access to the data and a lack of a disaster recovery plan.

What is the role of partition keys in Cosmos DB?

Partition keys in Cosmos DB determine how data is distributed across physical partitions. They are essential for optimizing performance by ensuring that queries scan only relevant data, rather than the entire database.

What are the consequences of choosing a poor partition key?

A poor partition key can lead to uneven data distribution, where certain partitions become overloaded (hot), resulting in performance bottlenecks, throttling, and inefficient queries. This can eventually necessitate data migration and reconfiguration.

What is meant by a 'hot partition' in Cosmos DB?

A hot partition occurs when one partition receives significantly more read or write requests than others, leading to resource exhaustion, throttling, or failures. This can happen if the partition key is poorly chosen and data is not evenly distributed.

What should data engineers consider when selecting a partition key?

Data engineers should consider how evenly data will be distributed across partitions. The partition key should be chosen to avoid overloading specific partitions and ensure that queries are efficient by narrowing the scope of data scans.

What is the potential issue with underprovisioning throughput in Cosmos DB?

Underprovisioning throughput can lead to performance issues because resources (such as request units per second) are insufficient to handle the load. This may result in throttling, failed requests, or slower response times, particularly in heavily accessed partitions.

How can underprovisioning throughput be addressed in Cosmos DB?

To address underprovisioning, engineers should monitor throughput usage and adjust the provisioned throughput to ensure it meets the needs of the application. Additionally, distributing requests evenly across partitions can help balance resource usage.

What is a physical partition in Cosmos DB, and why is it important?

A physical partition in Cosmos DB refers to a storage unit where multiple logical partitions reside. It is important because the throughput is divided among these physical partitions, and unevenly distributed data can lead to resource allocation problems, especially if some partitions are more heavily accessed than others.

What should be done if a partition becomes overloaded in Cosmos DB?

If a partition becomes overloaded, it may be necessary to rethink the partition key strategy, increase throughput provisioning, or split the collection into multiple collections with better partitioning to balance the load across physical partitions.