Deconstructing the Monolith (Shopify Unite Track 2019)

Summary

Kirsten West Ein, a senior developer at Shopify, shares her experience of the company's evolution from a small monolith to a modular monolith. She highlights the challenges Shopify faced with a monolithic codebase, including slow developer onboarding and test failures. After rejecting microservices, Shopify embarked on a componentization approach to organize code into well-defined components with clear boundaries. This modular monolith approach improved developer productivity, reduced complexity, and facilitated easier updates and scalability. West Ein emphasizes the importance of software architecture in enhancing team productivity and adapting to growth.

Takeaways

• 😀 Shopify's growth: From 200 employees serving 70,000 merchants to over 4000 employees serving 800,000 merchants worldwide.
• 😀 The importance of starting with a monolith for new companies: Focus on getting the product to market quickly without worrying about perfect design upfront.
• 😀 Crossing the design payoff line: When incremental functionality becomes slower, it's time to invest in architectural changes.
• 😀 Symptoms of design payoff line: Cascading test failures, high context switching for developers, and slow tests that impact productivity and happiness.
• 😀 Challenges of a monolithic architecture: Global accessibility of code led to tight coupling and inefficiencies in the Shopify codebase.
• 😀 Microservices were considered but ruled out due to high overhead, complex dependencies, and difficulties transitioning from the existing monolith.
• 😀 The solution: Componentization—a modular monolith that preserves the benefits of a monolith while introducing clear boundaries between components.
• 😀 Reorganization of code: Shopify restructured its application based on real-world concepts like billing, orders, and taxes rather than typical Rails structures like models and controllers.
• 😀 Decoupling components: The goal was to isolate dependencies, define clear APIs for each component, and ensure ownership of associated data within each component.
• 😀 Wedge tool: An in-house tool to track progress towards component isolation and enforce domain boundaries by measuring violations of dependencies across components.
• 😀 Future plans: Enforcing component boundaries programmatically and visualizing dependency graphs to reduce accidental or circular dependencies in the system.

Q & A

What key challenges did Shopify face with its monolithic architecture?

Shopify faced several challenges with its monolithic architecture, including difficulties in making changes without causing unrelated failures, long onboarding times for new developers, slow tests, and entangled objects. These issues were impacting developer productivity and overall system reliability.

Why does Kirsten West still recommend starting with a monolith for new products?

Kirsten recommends starting with a monolith for new products because in the early stages, the main priority is getting the product to market quickly. A monolith allows for rapid development, and there isn't enough information about the product and its domain to make design decisions early on.

What was the turning point for Shopify to rethink its architecture?

The turning point came when the incremental functionality became slower and slower to add, and the challenges of the monolithic system outweighed its benefits. This marked the point at which they recognized the need to rearchitect their system.

Why did Shopify rule out microservices as a solution?

Shopify ruled out microservices due to the infrastructural overhead, complexity of managing multiple deployment pipelines, and challenges with inter-service communication. The existing monolith was too intertwined for a simple switch to microservices, and transitioning would likely have meant starting from scratch.

What is the concept of a modular monolith introduced by Shopify?

A modular monolith is a system where all the code for an application resides in a single codebase and is deployed together, but with strict boundaries enforced between different domains. This approach allows for the benefits of both monoliths and microservices without many of the downsides.

What were the three main steps in Shopify’s componentization process?

The three main steps in Shopify's componentization process were: 1) reorganizing the code based on real-world concepts, 2) isolating dependencies by defining clear interfaces for each component, and 3) enforcing boundaries between components to ensure modularity.

How did Shopify reorganize its code to implement componentization?

Shopify reorganized its code by splitting it into components, each structured as a mini Rails application. They based this organization on real-world concepts like billing, orders, and taxes, rather than traditional Rails structures like models, views, and controllers.

What tool did Shopify use to track the progress of componentization?

Shopify used a tool called 'wedge' to track the progress of componentization. Wedge helped track violations of domain boundaries and data coupling across components, allowing Shopify to enforce standards and create healthy competition among teams.

How did componentization improve Shopify’s ability to make changes to the codebase?

Componentization made it easier to extract pieces from the core codebase when needed, as well as swap out old components for new ones with different technical requirements. It also improved the ability to manage dependencies and isolate issues, ultimately speeding up development and reducing risk.

What are the future plans for enforcing boundaries programmatically in Shopify’s system?

In the future, Shopify plans to enforce component boundaries programmatically by ensuring that each component only loads the other components it explicitly depends on. This would result in runtime errors if a component tries to access code outside of its defined dependencies, further ensuring modularity and isolating components.