Tick-Level Market Replay
Tick level market data storage and replay services are powerful tools for historical insight and model back-testing. But present options market data rates are very high and official forecasts indicate that volumes are nearly doubling each year. This makes it increasingly difficult and costly to store tick data as it necessitates the parsing, indexing and storing of enormous volumes of raw data measuring in petabytes.
dxFeed solution utilizes the power of cloud computing to address the complexity of this problem and brings virtually unlimited capacity and scalability for the millisecond time-stamped tick data storage.
We eliminate the need for hefty investments in infrastructure, such as co-location space and power, and provide the necessary elastic storage and computing power from day one. Our solution is fully scalable for future growth (including the number of simultaneous client connections).
Our solution provides:
- Direct feed capture, storage and instant access to hundreds of terabytes of historical market data at the millisecond-time-stamped tick level.
- Unlimited capacity for data storage.
- Unlimited full order book depth storage.
- High redundancy of storage ensuring service availability and resiliency.
- Data security and privacy – no client data is stored in or transferred to the cloud.
- Elastic scalability for end-client connectivity.
- Seamless technology integration.
- No client-side hardware installation is required and no client resources are used to store data.
- Instantaneous access times – under 1 second full day extraction time, under 400 microsecond full book snapshot access time.
- Various means of data access:
- Time machine-like market replay for the full market or in part by subscription.
- Data extraction
- Data exploration
- API access for streaming or snapshot analytics
Instead of building vulnerable and expensive storage infrastructures locally, dxFeed provides high system availability. We ensure a robust operation through the use of inherent cloud storage redundancy and independent availability zones (with geographical distribution of content storage).
Data access security is ensured via a gateway service, which processes the data requests, secures the communication protocol and the encryption. Authorization takes place on the customer side authorization servers. As such, no client data is transferred to or stored in the cloud; only time-limited identity-agnostic security tokens are used.
Scalability and Performance
Client application connectivity end-points are hosted in the cloud and can be easily scaled depending on the volume of data requests. State-of-the-art end-client components seamlessly integrate into the client applications and provide a tick data request, download and replay directly from the cloud – anywhere, anytime. Replays can be cached locally, eliminating the need to download the data that has been already accessed.
Historical Data APIs
The following APIs are available for the service:
- dxFeed Java API: gives control over the full functionality of dxFeed realtime, historical and aggregated data (including market replay).
- REST web services and APIs:
- Instrument profile web service provides essential symbol attributes and symbol discovery
- onDemand tick data extraction web service extracts tick data for given symbols, time and event types from our cloud storage.
- onDemand tick data audit web service extracts data from onDemand cloud storage for given symbols, events and timeframes keeping the order. Used for trade audit purposes primarily, it currently provides access only to 1 month of data on all symbols. Limitations are configurable.