The glory of real-time technology, Microsoft releases real-time big data analysis products!

The new programming model of cross-cloud smart devices and parallel technology is a key aspect of digital transformation. One of the key data types generated from these new application paradigms is telemetry data. Telemetry data is everywhere: IoT sensors, application logs, web logs, infrastructure logs, security logs, metrics, clickstreams, time series, etc. The powerful insights unleashed from these data are driving the advancement of connected devices that consumers and businesses rely on every day.

The use of telemetry data requires a flexible and adaptable platform, which must be able to handle large amounts of data and provide users with real-time insights to improve their operations and innovation. Traditionally, these data were stored and managed in the shaft system, lacking real-time visibility, limited scale, and high maintenance costs. In addition, it is very complicated to popularize and correlate these data with enterprise business.

What is Azure Synapse Data Explorer?

In order to allow customers to make full use of logs and telemetry data, Microsoft released a public preview of Azure Synapse Data Explorer. In order to supplement the existing SQL pool and Apache Spark engine, Microsoft optimized the new data browser runtime engine from the product level, using powerful indexing technology to automatically index free text and semi-structured data, making it nearly real-time The speed of querying a large amount of structured, semi-structured and free text telemetry and time series data, the following are some of the key features that make this possible:

• Powerful distributed query engine, index all data, including free text and semi-structured data. Data is automatically compressed, indexed, automatically optimized, cached on SSD, and persisted on storage. Computing and storage are separated, which gives users complete flexibility to automatically scale without downtime.
• The intuitive Kusto Query Language (KQL) uses the best text index of the Synapse Data Browser to explore raw telemetry and time series data for efficient free text search, regular expressions, and analysis of tracking\text data.
• Comprehensive JSON parsing function for querying semi-structured data, including arrays and nested structures.
• Native and advanced time series support the creation, manipulation and analysis of multiple time series, and Python and R execution in the engine support model scoring.

What is the architecture of Azure Synapse Data Browser?

The data resource manager cluster realizes a horizontally scalable architecture by separating computing resources and storage resources. In this way, users can expand each resource independently, for example, to run multiple read-only calculations on the same data. The data resource manager cluster contains a set of computing engines that are responsible for automatic indexing, compression, caching, and distributed query services. In addition, the data resource manager cluster also has a set of computing engines for data management services, which are responsible for background system operations, as well as hosting and queued data import. All data is stored on the hosted Blob storage account in a compressed, columnar format.

The Data Explorer cluster supports a rich ecosystem, and data can be introduced using connectors, SDK, REST API, and other hosting functions. Users can use data from temporary queries, reports, dashboards, alerts, REST APIs, and SDKs in a variety of ways.

What are the innovations and features of Azure Synapse Data Browser?

Unlimited Streaming Data -Data Explorer provides built-in integration for implementing no-code/less-code, high-throughput data import and caching data from real-time sources. Data can be imported from sources such as Event hub, Kafka, Azure Data Lake, open source agents such as Fluentd/Fluent Bit, and various cross-cloud and local data sources.

Unbounded Data Modeling -If you use Data Explorer, you do not need to generate a complex data model, and you do not need to write complex scripts to transform the data before using the data.

infinite data scale -Data Explorer is a distributed system, its calculation and storage can be independently scaled, can easily achieve data analysis above the PB level.

not require index maintenance -data can be optimized to maintain query performance without maintenance tasks, and there is no need to maintain indexes. When using Data Explorer, all raw data is immediately available, so you can run high-performance, high-concurrency queries against streaming data and permanent data. These queries can be used to generate quasi real-time dashboards and alerts, and connect operational analysis data to the rest of the data analysis platform.

Low latency, high performance, high concurrency -Data Explorer indexes semi-structured data (JSON) and unstructured data (free text), so it can run queries on such data very efficiently. By default, each field will be indexed during the data import period, and low-level coding strategies can be used through the corresponding options to fine-tune or disable the indexing of specific fields. The index range is a single data slice.

standard data analysis -Data Explorer standardizes self-service big data analysis through intuitive Kusto Query Language (KQL). KQL combines the expressiveness and powerful functions of SQL, as well as the simplicity of Excel. KQL is highly optimized and can use the first-class text index technology of Data Explorer to explore original telemetry data and time series data, realize efficient free text and regular expression search, and provide comprehensive analysis functions for query tracking\text data and JSON Semi-structured data (including arrays and nested structures). KQL provides advanced time series support for creating, operating and analyzing multiple time series, and provides engine internal Python execution support for model scoring.

Multi-ecosystem integration -Azure Synapse Analytics provides interoperability for data between Data Explorer, Apache Spark, and SQL engine, enabling data engineers, data scientists, and data analysts to easily and securely access the same data in the data lake Data and collaborate on it.

What are the digital business scenarios supported by Azure Synapse Data Browser?

Precise real-time behavior optimization

Azure Synapse Data Browser works flexibly between customers' Azure hybrid cloud solutions. For example, a railway network company can trust the Azure Synapse data browser to replace its local log management solution. For the transportation industry, safety is the primary consideration, because people's lives depend on real-time telemetry data. With the expansion of large-scale infrastructure across the country, railway management companies need a platform that can quickly obtain a large amount of time series and log data, and then create powerful insights and data visualization in Power BI. The Azure Synapse data browser allows the railway company to effectively identify behavior patterns or irregularities in its vast transportation network, thereby making the railway system safer.

Real-time supply chain insights

Azure Synapse data browser can build real-time big data analysis on customized event and log data, thereby saving time and resources for the enterprise and focusing on the core value of the business. For example, if an Internet food delivery company wants to improve their processes and business to provide a consistent and first-class customer experience, they may be hindered by slow, complex, and expensive log management technology solutions. However, using the Azure Synapse data browser engine, Internet food delivery companies can immediately benefit from faster data ingestion, higher concurrency, and greater flexibility. This will enable them to focus on their core mission: to provide people with delicious takeaways and consistent customer service.

Complex security incident handling

In the face of digital security threats, every second is important. Client online delays, network failures, and query timeouts can be devastating, but these issues can plague network security and log management service providers. Their existing technological solutions may hinder their ability to realize their core value propositions of accessibility and transparency. In this case, network security vendors can use Azure Synapse Data Explorer, which will provide them with a data platform that will provide their customers with valuable insights on threat detection, intelligence alerts, and security trends. Therefore, network security providers can build stronger relationships and more trust with their users.

---

In summary, Azure Synapse Data Explorer can create meaningful connections across various data sources and databases. Today, various digital businesses are overwhelmed by massive amounts of time series, logs, and telemetry data, which come from IoT devices, applications, websites, and other sources. This real-time continuous data flow can be overwhelming and slow for IT infrastructure. Using the distributed query engine of Azure Synapse Data Explorer, customers can gain powerful insights, allowing them to focus on their core business, whether it is creating a safer world or delivering the best food.

(Azure Synapse Analytics operated by 21st Century Internet is now available, click to read the original text to learn more.)