Innovative Linkedin's Deep-leaning based CTR Modeling: The Deep, Wide, and Shallow Towers Explained

Summary

This video delves into a deep learning-based Click-Through Rate (CTR) prediction model used in LinkedIn's advertising system. The model uses a three-tower architecture: the Deep Tower for feature interaction, the Wide Tower for memorizing ad IDs and ensuring freshness, and the Shallow Tower for calibrating CTR predictions. By combining these elements, the model optimizes CTR accuracy, improving ROI for advertisers. Key points include handling over-prediction, the importance of freshness, and the trade-off between using pre-trained embeddings and learning embeddings during the prediction process. Overall, this model revolutionizes ad targeting through advanced deep learning techniques.

Takeaways

• 😀 **CTR Prediction is Crucial for Ads**: Click-through rate (CTR) is the key metric that determines how many people click on an ad after seeing it. It's essential for optimizing ad performance and maximizing ROI for advertisers.
• 😀 **Three-Tower Architecture**: The model architecture is composed of three towers: a deep tower for complex feature interactions, a wide tower for memorizing ad ID properties, and a cellot tower for calibration and improving prediction accuracy.
• 😀 **Deep Tower Focuses on Feature Interactions**: The deep tower uses a multi-layer neural network to process generalization features like user, ad, and contextual data, and learns embeddings optimized for conversion.
• 😀 **Wide Tower Provides Freshness**: The wide tower processes sparse ID features, memorizing specific ad ID properties and learning new ad IDs every hour to maintain freshness and relevancy in predictions.
• 😀 **Cellot Tower for Calibration**: The cellot tower introduces a simple linear layer to calibrate CTR predictions, ensuring that the probabilities are more representative of true conversion rates and reducing over-prediction.
• 😀 **Trade-off Between Embedding Approaches**: Two approaches for handling embeddings were compared: using pre-trained embeddings vs. learning embeddings directly in the deep model. The latter was found to provide better optimization for conversions.
• 😀 **Exposure Bias in CTR Models**: Exposure bias occurs when only a few ads are shown to users, leading to over-prediction of CTR. Calibration on the current model's exposed data helps mitigate this bias.
• 😀 **Calibration for Accurate CTR**: Calibration improves the accuracy of CTR predictions by adjusting the predicted probabilities, making them more aligned with actual conversion rates.
• 😀 **Hourly Updates for Model Freshness**: The wide tower is updated hourly, which allows the model to adapt quickly to new ads and maintain high relevance, ensuring that CTR predictions are based on up-to-date data.
• 😀 **Reducing Overfitting with the Cellot Tower**: Features prone to overfitting, such as position features, are passed only to the cellot tower, which helps in regularizing the model and improving prediction stability.
• 😀 **Final Model Conclusion**: The deep, wide, and cellot towers work together to improve CTR predictions by handling feature interactions, memorizing ad-specific data, and providing stable calibration, all of which contribute to better ad performance and higher ROI.

Q & A

What does CTR stand for and what is its significance in ad prediction?

CTR stands for Click-Through Rate, which refers to the probability of how many clicks an ad will receive based on the number of views it has been exposed to. It plays a crucial role in ad prediction because it helps determine the effectiveness and relevance of ads for the users, ultimately influencing the return on investment (ROI) for advertisers.

What is the role of the three towers in the deep learning CTR model?

The three towers in the CTR prediction model are: 1) The Deep Tower, which learns deep feature interactions and handles nonlinearity. 2) The Wide Tower, which memorizes sparse ID features like ad IDs and updates frequently to ensure freshness. 3) The Shallow (Cellot) Tower, which aids in calibration by providing a linear model to stabilize predictions and make them more accurate.

Why is calibration important in the CTR model, and how is it achieved?

Calibration is important because it ensures the predicted CTR probabilities align with actual conversion rates, improving the model's reliability. Calibration is achieved by using the Shallow (Cellot) Tower, which introduces a linear layer to adjust the probabilities. This helps reduce over-prediction errors, ensuring the CTR prediction is more accurate.

What are the advantages of training embeddings within a deep learning model versus using pre-trained embeddings?

Training embeddings within a deep learning model allows the embeddings to be optimized for the specific conversion goal, which is more effective than using pre-trained embeddings. While pre-trained embeddings can reduce engineering costs, they don't optimize directly for conversion, making the deep model with learned embeddings a better approach for CTR prediction.

How does the White Tower help improve the CTR prediction model?

The White Tower helps with fast memorization of new ad IDs and ensures freshness by being trained on an hourly basis. Unlike the other towers, which are retrained at a fixed interval, the White Tower can adapt quickly to new information, which is crucial for keeping ads relevant, particularly in fast-changing environments like LinkedIn.

What are generalization features in the context of the CTR model?

Generalization features in the CTR model include user features, ad features, and context features. These features help the model make predictions by considering various factors such as user behavior, ad characteristics, and environmental factors like time of day or current events.

What is the significance of the trade-off between relevance and revenue in CTR prediction?

The trade-off between relevance (CTR) and revenue (CPC - cost per click) is significant because advertisers want to ensure their ads are both relevant to the users and profitable. In the model, the goal is to optimize for CTR to ensure relevance while balancing this with the CPC to maximize the revenue generated from the ads.

How does the White Tower address the issue of freshness in CTR prediction?

The White Tower addresses the issue of freshness by being trained every hour, allowing it to quickly adapt to new trends, such as recently posted jobs or current events. This frequent retraining ensures that the model stays up-to-date with the latest information, which is important for keeping predictions relevant in dynamic environments like LinkedIn.

What is the problem of over-prediction in CTR models, and how is it mitigated?

Over-prediction occurs when the CTR model predicts higher conversion rates than what actually happens, leading to inaccurate ROI calculations. This issue is mitigated by using the Shallow (Cellot) Tower for calibration, which helps reduce over-prediction by introducing a linear layer to make predictions more stable and representative of true conversion rates.

What does 'warm start' training in the White Tower refer to, and how does it help improve the model?

Warm start training in the White Tower refers to initializing the model with embeddings learned from past data. It allows the model to quickly adapt to new information by updating these embeddings for new ad IDs while retaining the knowledge from previous ads. This approach speeds up training and ensures the model stays fresh while minimizing the impact of cold-start problems.