Replay Market Data

Summary

The video explains the process of sharing market data in binary format for various financial instruments, like Nifty and Sensex, through a replay server. The binary files are compressed and shared, and a custom wrapper mimics a live market data server. The speaker demonstrates how users can subscribe to this data, replaying market events programmatically, allowing them to simulate real-time trading. They also showcase setting up a virtual environment, running a replay server, and consuming the market data for backtesting or strategy validation, highlighting the system's speed and efficiency.

Takeaways

• 📈 The script discusses sharing market data in a binary format for instruments like Nifty Bank, Nifty midcap, and Sensex bankx index.
• 💾 The data is saved in a sequential fashion throughout the day and compressed using the Z standard, resulting in a .zstd file extension.
• 🔄 A wrapper has been created to consume the replay market data, simulating the market data web socket service provided by Zerodha.
• 🛠️ The replay server mimics the live market data feed but replays the saved data at a much faster rate, making it suitable for backtesting strategies.
• 🌐 The server is set up to connect to 'WS LocalHost', bypassing the need for an API key, and can be accessed by multiple strategies simultaneously.
• 📅 Users must specify the date for which they want to replay the market data, and the data files should be available locally in the 'tick data' folder.
• 🔌 The script provides an example of how to set up the replay server and consume the data using callbacks for events like on connect, on tick, etc.
• 🔧 The user can use any programming language to consume the data from the local server, as long as they point to 'WS LocalHost'.
• 📊 The replay server is useful not only for backtesting but also for execution error testing, as it can help identify issues in real-time data processing.
• 📝 The script includes a demonstration of how to set up the environment, install the required module, and run the replay server and usage example.

Q & A

What is the purpose of sharing the market data in binary format?

The purpose is to save the market data as it arises in a binary format for the instruments subscribed to in a sequential fashion, resulting in a large binary file containing all the day's data.

What does the '.zstd' extension in the file name indicate?

The '.zstd' extension indicates that the binary file is compressed with the Z standard compression algorithm.

What is the significance of the '.doblin' file extension?

The '.doblin' extension signifies that the data is in binary format and not in plain text, making it machine-readable rather than human-readable.

What is the motivation behind creating a local WebSocket server for market data replay?

The motivation is to save the market data and then programmatically replay it, allowing the data to be read individually and processed in real-time, similar to how live market data is consumed.

How does the replay server mimic the actual market data WebSocket service?

The replay server creates a local WebSocket server that mimics the market data WebSocket service provided by zerodha, allowing users to connect to it locally instead of the actual server.

Why is it necessary to override the kite ticker root URL?

It is necessary to override the kite ticker root URL to connect to the local server (WS Local Host) instead of the default zerodha server, facilitating the use of the replay server.

How fast can the replay server consume and transmit the data?

The replay server can consume and transmit data very quickly, capable of processing a week's worth of data in minutes, as it reads every packet and throws it out all at once.

What is the benefit of using the replay server for backtesting trading strategies?

Using the replay server for backtesting allows users to test their strategies on historical market data without waiting for events to occur in real-time, providing a fast and efficient way to evaluate strategy performance.

How can multiple strategies listen to the same replay server data?

Multiple strategies can listen to the same replay server data by initiating the same replay server process in different terminals, allowing each strategy to receive the data in parallel.

What is the process for setting up and using the replay server as described in the script?

The process involves installing the necessary module, setting up a local server that replays market data, and then using that server to stream data to a strategy or application that connects to it.

How can the replay server be used for execution error testing?

The replay server can be used for execution error testing by replaying market data and observing the strategy's performance and calculations in real-time, allowing for immediate detection of any mishaps or errors.