Real-Time Azure Data Processing Services

While the previously described set of services can be used in a variety of data processing tasks, it is important to note that there are other Azure data services that are used for niche data processing use cases. For example, Azure Stream Analytics and Azure Data Explorer are almost exclusively used in stream processing workflows. These services are out of scope for the DP-900 exam and will only be covered briefly in the following sections.

Azure Stream Analytics

Azure Stream Analytics is a PaaS stream processing engine that can be used to process high volumes of streaming data from multiple sources. Users can create Azure Stream Analytics jobs through the Azure Portal, Azure CLI, Azure PowerShell, or an Infrastructure as Code template like ARM. Jobs consist of three core components: one or more inputs, a query, and one or more outputs.

Inputs can include real-time message ingestion services like Azure Event Hubs and Azure IoT Hub as well as persistent data stores like Azure Blob Storage and Azure SQL Database. This enables developers to combine streaming data with historical data or with reference data for lookup operations.

Developers can use the Stream Analytics query language to filter, sort, aggregate, or join data from different sources. This language is a subset of standard T-SQL with additional functionality to apply computations over specific time windows. The language can also be extended with JavaScript and C# user-defined functions.

Jobs deliver processed information to one or more outputs. Azure Stream Analytics allows you to customize what happens based on the results of the data that was processed. Here are some common outputs:

• Services like Azure Event Hubs, Azure Service Bus, or Azure Functions to trigger alerts or custom workflows
• Power BI dashboards for real-time dashboarding
• Persistent data stores like Azure Blob Storage, ADLS, Azure SQL Database, or Azure Synapse Analytics dedicated SQL pools for long-term storage or batch processing

If you would like to learn more about Azure Stream Analytics, visit https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-introduction.

Azure Data Explorer

Azure Data Explorer is a near real-time processing engine that analyzes structured, semi-structured, and unstructured data across time windows. It uses the Kusto Query Language (KQL) to analyze data and is capable of ingesting and analyzing petabytes of data. Typical use cases for Azure Data Explorer include interactively analyzing logs and conducting time series analytics on metric data from IoT sensors.

If you would like to learn more about Azure Data Explorer, visit https://docs.microsoft.com/en-us/azure/data-explorer/data-explorer-overview.