Alibaba Cloud Lindorm and Zhinian Technology release quantitative analysis and processing solutions for financial high-frequency trading data

About a one-stop high-performance one-stop solution for high-frequency data for banks, insurance, brokerage and private equity.

The L1/L2 quotation and transaction data of the financial market are very important data for quantitative transaction research. With the rapid evolution of digital services, transaction data with time-series characteristics has increased sharply, placing higher requirements on the underlying database and quantitative analysis system. Traditional relational databases support this level of data. Even if the database is divided into tables, the query performance is far from reaching the requirements. Commonly used columnar NoSQL databases can solve this level of data storage, but this kind of general-purpose storage engine lacks friendly support for time series data, has serious deficiencies in query and calculation, and cannot support real-time business calculations for quantitative financial scenarios. , Flow batch analysis, multi-source data fusion analysis.

Alibaba Cloud's native multi-mode database Lindorm and Zhejiang Zhinian DolphinDB released a quantitative analysis and processing solution for financial high-frequency transaction data, integrating DolphinDB's real-time and efficient data processing capabilities and Lindorm's multi-mode massive data fusion storage and analysis capabilities through cloud native methods, and integrated functions A powerful programming language and a high-capacity and high-speed streaming data analysis system provide a one-stop solution for the quantitative analysis and calculation of massive time series data in financial scenarios. The solution is simple to operate, highly scalable, has good fault tolerance and excellent concurrent access capabilities for multiple users.

Program advantage capability

database storage

• In-line memory engine with high throughput and low latency.
• The columnar hybrid engine (based on memory and disk) provides superior performance for data warehouses that store massive amounts of data.
• Flexible partitioning scheme: support value partition, range partition, list partition, hash partition and combined partition.
• Supports the number of partitions at the level of one million in a single table, greatly reducing the retrieval response time for massive data.
• Analysis in the library: Complicated programming and calculations can be performed in the database to avoid time-consuming data migration.
• Provides a variety of SQL function extensions, including non-simultaneous connections, window connections, pivot tables, composite columns, etc.
• Supports fast connection of multiple tables in the same partitioned database.
• data compression.
• Supports concurrent access by multiple users. Each user works in a separate session with a given authority.
• Metadata is highly available: Multiple control nodes use the Raft protocol to achieve strong consistency.
• High availability of partitioned data: A database can contain millions of partitions, and an improved two-phase commit protocol is used between multiple copies of partitions to achieve strong consistency of partition copies.
• High availability of operation and maintenance: online increase of server nodes, online balance of data between nodes, and online addition of fields for partition data tables.
• Incremental backup mechanism of the database: When the number of partition copies is N, in the case of N-1 nodes down, it is guaranteed that the system can still continue to write and read.
• Use the embedded distributed file system to automatically manage partition data and its copies, providing load balancing and fault tolerance for distributed computing.

Data analysis in the database

• The programming language is powerful and expressive. Support imperative programming, functional programming, vector programming, SQL programming and RPC (remote function call) programming.
• The syntax of the programming language is very similar to SQL and Python, easy to learn and use.
• There are more than 1,000 built-in functions, covering most of the commonly used data processing, data analysis, machine learning and other functions, as well as file calling and database management functions.
• High-speed distributed computing is realized through memory engine, data localization, fine-grained data partitioning and parallel computing.
• Provides just-in-time compiled version, which greatly accelerates the execution speed of for-loop, while-loop and if-else statements.
• Support multiple computing models, including pipeline, map-reduce and iterative computing.
• Provide snapshot isolation for dynamic data distributed computing.
• Increase system throughput by sharing data copies of memory in multitasking.
• Distributed data can be easily analyzed. After the script is written on a single node, it can be executed on the entire cluster without compilation and deployment.

Streaming data

• Seamlessly integrate streaming data and database tables. You can use SQL to query local streaming data or distributed streaming data.
• Built-in multiple streaming data aggregation engines such as time series, cross-section, anomaly detection, and responsive state engine.
• You can use user-defined functions in DolphinDB to process information.
• Sub-millisecond information delay.
• There is only a sub-second delay in updating the historical data warehouse with real-time data.
• Historical information can be reproduced from any offset.
• Provides configurable options (such as partitions, worker threads, queues) for flow control and performance tuning.

ecological

• Provides a variety of programming APIs, including C++, Python, Java, C#, Go, Excel, etc.
• Existing pandas programs can be run in DolphinDB through the pandas API (orca) with only a few changes.
• Provide a variety of plug-ins, including MySQL, ODBC, HDF5, Parquet, etc.
• Built-in web server for cluster management, performance monitoring and data access.
• Provide IDE (Integrated Development Environment) such as DolphinDB GUI and VS Code plug-in for data analysis.
• Realize system monitoring through built-in functions, web interface or Prometheus.

Copyright Statement: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.