I love Sunday nights. Nice breeze flowing through the open windows, crickets chirping, pups resting, social networks are slower paced and email is still fairly quiet while Redmond is enjoying the end of their weekend. After starting a new healthy diet and exercise plan, I am craving sinful food and need to keep myself distracted. Playing with data and continuing on with the Exploring Oceans of Data series should keep me from going near that refrigerator in my kitchen tonight.
This week I will share how to design a real-time analytics solution using Azure Event Hubs, Stream Analytics and Power BI. This combination is one of few different real-time cloud analytics options with Azure and Power BI. These services can be purchased individually or in the new Cortana Analytics Suite or Azure IoT Suite bundles. You can find them by hovering over Data + Analytics or Internet of Things options in the Azure portal menu.
Azure Event Hubs is an impressive, highly scalable service for ingesting Internet of Things (IoT) event processing data sources. It enables the collection of event streams at high throughput from a diverse set of devices and services. Event Hubs can massively parallel intake millions of events per second via HTTP(S) or AMQP protocols. Once data is brought into Event Hub partitions, you can apply transformations and/or store the event data using any real-time analytics provider or with batching storage adapters. A peer of mine at Microsoft created an excellent overview video on Azure Event Hubs that showcases just how totally amazing this specific technology is for bringing in massive amounts of data in a big data world. When I saw this, I had to try it to see for myself how it really works. I could not believe how easy it was to set up. It took me a few minutes to set up my demo Event Hub with an Endpoint URL jendemo-ns.servicebus.windows.net to collect event messages.
Stream Analytics is great for querying live data streams like Twitter. It is often used to detect anomalies, trigger alerts when errors occur, or feed real-time dashboards like we will be doing with the Twitter Sentiment sample. Stream Analytics provides super simple out-of-the-box integration with Event Hubs and Power BI for developing end-to-end, highly scalable, Internet of Things (IoT) analytics solutions. If you’d like to learn more about Stream Analytics, check out the awesome learning path. Admittedly, I have not been through that training. I found Stream Analytics to be quite intuitive by just navigating through the menu options (Dashboard, Monitor, Input, Query, Output, Scale and Configure) in Azure.
Power BI is a cloud-based business analytics service from Microsoft that empowers anyone to experience any data – structured or unstructured – via simple drag-and-drop ease. Unlike many other dashboard solutions, Power BI can render live dashboards with moving charts and continuously updated visualizations for monitoring real-time streams from supported data sources.
You can use the Power BI REST API with any data source or Azure Streaming Analytics to render live Power BI Dashboards automatically. Alternatively, you can get near real-time analytics using simple “direct connect” data sources such as Analysis Services, Azure SQL Database, Azure SQL Data Warehouse or Spark with Power BI Reports. Power BI Reports issue live queries and return the most up-to-date data that is available in data sources as users browse them.
Other on-premises, SaaS and cloud data sources use data refresh schedules to keep data updated.
- On-premises: Access, File (CSV, XML, Text, Excel, Folder), IBM DB2, MySQL, Oracle, PostgreSQL, SharePoint Lists, SQL Server Sybase, Teradata, Hadoop HDFS and ODBC with more added weekly
- Cloud or SaaS: Web Pages, Salesforce, Google Analytics, Azure Blob, Azure HD Insight, Dynamics CRM and NAV Online, Facebook, GitHub, OData Feed, SharePoint Online, QuickBooks, Zendesk and more added weekly
Putting it Together
The Azure Stream Analytics team has a lovely public Twitter Sentiment Sample that showcases Event Hubs, Stream Analytics and Power BI live dashboards. I will highlight a few steps rather than duplicate their step-by-step tutorial here. By the way, this is not the only way you can get Twitter data into Power BI. A peer from the Azure Logic Apps team showed me their Twitter connector after initially tweeting about this demo. Ideally you would use a premium Twitter data service like GNIP to avoid running into the free API limits.
The first thing I did was set up an Event Hub and got my Event Hub namespace and URL connection string. You can find Event Hub under Service Bus in Azure Portal when you click the + NEW menu option at the bottom of the screen.
Then I downloaded the GitHub console app project and merely added my Twitter OAuth keys, desired hashtags to report on and my Event Hub connection string to the C# project App.config file. That was all I had to do on the coding part = no code, just add settings.
Now I did read through the app modules and methods to see what it was doing. Essentially this app monitors Twitter, calls the open source sentiment API when it finds a matching hashtag tweet, gets a score for the tweet and then sends that scored tweet JSON to my Event Hub.
To feed the Event Hub ingested data to Stream Analytics for querying and Power BI reporting, I then configured an input for my Stream Analytics job, added my query and added Power BI as an output destination.
Now in Power BI, I can design my Report and Dashboard using the live Twitter data set to showcase the live Twitter Sentiment data that is collected when my little C# console app runs. Thanks to my Twitter followers, I was able to confirm that it actually works by looking at each component throughput monitoring dashboard in the Azure portal and of course seeing Power BI Reports and Dashboards update. Soooooo cool, fun and way too easy for the power and value this combo offers!
Another Cool Idea
I hope you enjoyed diving deeper into real-time analytics using Azure Event Hubs, Stream Analytics and Power BI with the Twitter API. Another cool idea that I saw mentioned by Chris Webb in social media channels is using Bing Pulse to capture real-time audience feedback along with the Twitter chatter. Chris is going to do this at the upcoming PASS Summit keynote. If you want to participate, check out his blog instructions. I am sure looking forward to seeing that analysis and hearing about his real-time analytics experience.