Getting started with Stream. Insight - SQLServer. Central. Introduction. This article is the second in a series of articles describing Microsoft SQL Server 2. R2 Stream. Insight. While the first article described the background and application areas, this article describes how to can get started with Stream. Insight, with an example of running queries against financial data. You can see the entire Stairway to Stream. Insight series summary here. Background. Stream. Insight is a platform for Complex Event Processing (CEP) that enables the processing of high speed data streams from multiple data sources. It is one of the new components of SQL Server 2. R2. Example application areas for CEP include financial applications, fraud detection, manufacturing, surveillance of internet traffic and real- time business intelligence. Complex Event Processing is quite different than relational database queries. Instead of one- time queries against finite stored data sets you have continuous standing queries against continuous data streams. Also you may have a requirement of a near- zero latency. Challenges that are faced within the field of stream processing and complex event processing include: Performance and Scalability - including techniques such as clustering and distributed processing. Out- of- order data arrival. Data quality problems - duplicate data, retroactive corrections. Time discontinuities such as daylight savings and leap seconds. Combining real- time and historical data. A CEP Engine takes care of at least some of these problems, making building robust stream processing applications much easier. Stream. Insight architecture. Events. Events represent data that change over time. It can for example be stock prices, temperatures, Twitter messages or credit card transactions. Generally Stream. Insight is used to process low- level Primitive Events into higher- level Complex Events, but it can also be used for simple operations such as filtering. Events in Stream. Insight are of two kinds: INSERT Events, that are used to submit data. CTI (Current Time Increment) Events, a kind of system Event that provides the current time of the source and indicates the completeness of existing Events (that there will be no more Events with timestamps older than the current time). An INSERT Event in Stream. Insight can be of three different temporal types: Point Events, that has no duration. Interval Events, that has a duration. Oracle Complex Event Processing Type 4 JDBC Drivers iii. Using SSL with Microsoft SQL Server. also install the examples. SQL Server 2008 R2 Free Book from Microsoft Press. CHAPTER 8 Complex Event Processing with. I want Sql Server 2008 EBook and related examples also. Edge Events, which usually represent cases where the duration is still unknown at Event start time. The INSERT Events consist of a header, which is dependent on the temporal type, and a body which is called a Payload. The Payload is a user- defined class which contains the actual data. Only a limited number of simple data types are supported for the members of the Payload. Streams. All data in Stream. Insight is organized into streams. The streams represent a continuous and possibly endless flow of events. A single stream can only contain one kind of event - so if you have different kinds of events you need multiple streams. Input Adapters. Input Adapters are responsible for receiving Events from external sources and putting them into the Stream in a format that is consumable by the queries. An Input Adapter that always emits the same type of Event is called a typed Input Adapter. There are also untyped Input Adapters that can produce different kinds of Events. However, at run time each instance of an untyped Input Adapter can only produce one kind of Event. Output Adapters. Output Adapters are responsible for outputting the Events to external destinations (sinks). They are in many ways similar to Input Adapters, except for the direction of data flow. In the November CTP of Stream. Insight, only one output adapter can be connected to a single query, but there are indications that they will allow multiple output adapters per query in the future. Query Templates. Query Templates define the business logics of your CEP Application. They are defined declaratively using LINQ expressions. A number of computational operations are supported: Filtering (WHERE- clause)Projecting (SELECT- clause)Joins (Cross Joins and Full Inner Joins)Unions. Aggregations over a time window (SUM, MAX, MIN, COUNT, ..)Group and Apply (computations in separate groups)Top K within a time window (with ORDERBY)Temporal alterations. User- defined scalar functions (Filters and Projects)User- defined aggregates (Scalar)User- defined operators (Set based)Query Templates are only used in the explicit development model. Query Instances. Binding a query template with specific input and output adapters together with configuration information results in a query instance in the CEP server. A Query Instance can be started, stopped and monitored through Diagnostic Views. In the implicit and IObservable/IObserver development models, Query Instances are created when you bind you input, output and LINQ query together. Diagnostic Views. Diagnostic Views provide information about objects on the CEP server or statistics on the CEP server itself. They give information about the state of Query Instances, latency times, number of Events and memory consumption. Diagnostic Views are acquired by calling the Get. Diagnostic. View method. To get updated information you need to call that function repeatedly. Example - some queries against financial data. The following section will describe an example of running queries against stock quotes. I've chosen the IObservable/IObserver development model, because it is the simplest to start with. Also I've created some example files that contain real historical data from 2. You could easily create your own historical data files or replace the input by another adapter that provides live data (for example my Yahoo. Finance adapter article). The data provided in this example is: Ericsson Stock Quotes from New York 2. US Dollars (USD)Ericsson Stock Quotes from Stockholm 2. Swedish Krona (SEK)Nokia Stock Quotes from New York 2. US Dollars (USD)NASDAQ Index from 2. US Dollars (USD)Exchange Rate between US Dollars and Swedish Krona for 2. Note that the stock quotes are only daily values. In a real- world situation you would probably work with a much finer time- scale such as seconds or milliseconds. The example also contains exchange rates for US Dollar/Swedish Krona. The purpose is to show an example where the same stock is available on multiple markets, in multiple currencies, where there may be a possibility of an arbitrage (the practice of making a risk- free profit through a combination of matching deals to take advantage of the pricing difference). However, for a true comparison you would need to take the stock quotes and exchange rate at exactly the same time (or nearly), which they are not in my example. Prerequisites. To run this example you will need the following: Microsoft SQL Server 2. R2 Stream. Insight (Version 1 or later)Visual Studio 2. Visual Studio 2. 01. Stream. Insight can be downloaded from the link at the end of this article. There you can also find a link to the complete source code of this example. Step 1 - Creating your Visual Studio project. Now let's get started writing a Stream. Insight application. You can create any kind of . NET application, but for simplicity let's choose a Console Application (Visual C#): Note: In the current version of Stream. Insight (April 2. NET 4. 0 is not supported so you must choose . NET 3. 5. The next step is to add references to the Stream. Insight assemblies. You should add references to: Microsoft. Complex. Event. Processing. Microsoft. Complex. Event. Processing. Adapters. Microsoft. Complex. Event. Processing. Diagnostics. By default they can be found in the folder "C: \Program Files\Microsoft Stream. Insight 1. 0\Bin". You should then see them listed under References as below: In your program code you also need to add a few using- statements: Now we have all the prerequisites in place for starting to write our code. Step 2 - Defining the Event Payload.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. Archives
November 2017
Categories |