Photo by John Adams / Unsplash

When it comes to innovation, leading companies throughout the world are openly communicating about it. Variables differ, but they all have an innovative culture, as well as agility and speed in common. The last two, in particular, are critical for any organization's ability to innovate. But where do you begin? We are fortunate to have new technologies, such as the cloud, that encourage innovation. In this article, we will look at how Microsoft Azure may help your organization innovate by using technologies, data, AI algorithms and machine learning.

Challenges in starting your data journey

Data has become a critical component of the decision-making process. But how can an organization become more data-driven, enabling it to be more agile and innovative? There is no way around the cloud as the foundation for a secure and governed data platform. We often see that businesses are frequently faced with a dilemma: whether to move fast or remain secure. The ability to do both is provided by establishing a cloud foundation.

Despite the fact that analytics and AI are the most important investments for company leaders, they face these challenges in becoming more data-driven:

• How to get the best return on investment;
• Cultural difficulties;
• How to become experienced data users;
• And how to get rid of data silos.

Read the Harvard Business Review Analytic Services report, Understanding Why Analytics Strategies Fall Short for Some, but Not Others.

Two critical analytics systems

Another challenge is that businesses are forced to maintain two critical, yet independent analytics systems: a data warehouse and a data lake. Every well-known company has a data warehouse with structured relational data. But do you want to experiment with big data and apply data science? Then a data lake is essential for exploring unstructured data and using algorithms to search for relationships. In Azure Synapse you find both a data warehouse and data lake to start your data journey.

What is Azure Synapse?

Azure Synapse, Microsoft’s PaaS (Platform as a Service) offering, integrates all your data (on-premises, cloud and streaming data) in one single environment.

In the platform you will find three unique components:

1. Business-critical data and dashboards, supported by data warehousing;
2. Data lakes to clean and explore (un)structured data;
3. Power BI (seamless integration) to create interactive data representations.

If you are using Synapse and have a proper dataset, you can start using Azure machine learning. So, to make data-driven decisions, you do not have to make a lot of custom developments or use lots of different tools. It all comes together in Azure Synapse.

Advantages of data analytics in Azure

When you have that data-foundation on Microsoft Azure, you can start your data analytics. 85% of those who had already done so, agreed or strongly agreed that well-integrated analytics databases and storage, a data management stack and business intelligence tools, were beneficial to their organization. Furthermore, customer satisfaction increased by 60% and they experienced 27% faster time to insights by using data. Finally, data can support businesses in cutting costs by 26% (see this study).

Do you already know about the Microsoft Cloud Adoption Framework? It is a proven guidance designed to assist you in developing and implementing the business and technology strategies needed for your company to be successful in the cloud. The framework offers best practices, data and resources that cloud developers, IT experts and business decision makers need to achieve their desired business outcomes. This standardized approach also helps to ensure greater coordination between company and technological strategies. Here are five reasons why you definitely should use a Cloud Adoption Framework.

1. Governance: budgetary control and access rights

One of the most critical aspects of your IT environment is keeping a tight grasp on the budget. In the cloud this may be even more relevant. The Cloud Adoption Framework enables you to apply governance at scale to your cloud environment in a controlled manner, preventing unwelcome surprises on your monthly statement.

When you grant application teams complete freedom, you may end up with very expensive cloud resources. While that is not always necessary. The framework helps clarify the cost flows, establishes budgets, directs access rights and restricts the use of certain resources when they are not in line with your policy. This way, you can keep a close eye on costs and access management within your cloud platform.

2. Cloud Adoption Framework ensures security and compliance

The more you work in the cloud, the more important it is to work safely and in accordance with GDPR. The framework gives you guidance in setting up (policy driven) guard rails on your cloud environment. By doing this, all your cloud solutions are secure and compliant by default.

In ‘old fashioned’ IT, environment security was implemented at the highest level. It was difficult to apply different levels of security for different solutions. This led to (unnecessary) high costs and probably a degraded level of functionality and usability. The Cloud Adoption Framework guides you in setting up an application centric approach. Now, you can work with a standard level of security that applies to all applications and then add additional security measures for specific components. This is referred to as layered defence. Within that framework, you can give the application team complete freedom to do their work.

3. A holistic approach with the Cloud Adoption Framework

Cloud adoption starts well before the selection of a cloud application provider. It begins when business and IT decision makers understand that the cloud will assist them in accomplishing a particular business transformation goal. Therefore, the platform is a full lifecycle framework for cloud adoption which goes beyond technology. From defining your strategy and objectives to actually setting up your cloud environment and providing code templates: the Cloud Adoption Framework assists organizations in aligning their business, culture and technological transformation in order to successfully achieve short-term and long-term objectives.

4. Applicable for large and small companies

The overall Cloud Adoption Framework is an extensive system with many processes. As an enterprise organization, you can use the entire framework in your organization and cloud environment from start to finish. However, even as a small business, you can easily use an aspect of the framework and extend it to more elements. We also use the Cloud Adoption Framework at a large retailer to set up the Azure platform, but also at smaller businesses, for example, to organize security. The framework has been demonstrated to be appropriate for any type of organization of any scale.

5. Quick results using the Cloud Adoption Framework

When you use the framework, you do not have to reinvent the wheel when it comes to configuring your Azure platform. Furthermore, you can be sure that you are using the cloud in the way that Microsoft intended and that best practices are being followed. We also use the Cloud Adoption Framework templates and specify them for the customer, which saves a lot of time. As a result, you have a cloud infrastructure that is optimally built and that allows you to get the most out of it. You are capable of being innovative with a scalable and future-oriented platform without compromising security, costs and privacy.

In this post I want to show how easy it is to setup a live stream of sensor data on the Azure platform and visualize this in Power BI.
I used a MXChip as sensor and let it send temperature and humidity data, but you’re free to use any other device to send the data.

Prerequisites
• Active Azure Subscription
• Power BI account

IoT Hub
First create an IoT Hub. For running this example, you can use the Free tier (F1). After the IoT Hub is created add a device (on the menu go to Explorers >> IoT Devices). Use the connection string of your added device to connect to the IoT Hub.

If you’re using the MXChip you can follow the getstarted tutorial (https://microsoft.github.io/azure-iot-developer-kit/docs/get-started/) to setup your device and connect it to the IoT Hub.

Stream Analytics Job
The next step is to setup a Stream Analytics Job. This job will send the data from the IoT Hub to Power BI.
From the menu in de Stream Analytics Job use the Job topology items.

1. Create a stream input (choose IoT Hub) and select the IoT Hub which receives the sensor data.
2. Create a stream output (choose Power BI). Fill the required field and use the ‘Authorize’ button to connect to your Power BI account.
3. Create a query to read the data from the input into the output.
   For example:

SELECT
    messageId, EventEnqueuedUtcTime, humidity, temperature
INTO
    [OUTPUT ALIAS]
FROM
    [INPUT ALIAS]
WHERE humidity IS NOT NULL AND temperature IS NOT NULL

4. If your device is connected and sending data to the IoT Hub, start the Stream Analytics Job.

Stream Analytics query language is a subset of standard T-SQL syntax for doing Streaming computations. For a full list of the support syntax features see:
https://docs.microsoft.com/en-us/stream-analytics-query/stream-analytics-query-language-reference

Power BI
Open Power BI online (https://powerbi.microsoft.com) using the same account as provided by the stream output in the previous steps. Navigate to the Datasets. When the Stream Analytics Job is set up correctly and your sensor is sending data the streaming dataset should be visible.

You can create a report using the icon on the right side of the dataset.
Create a line chart. Depending on the fields your datasets contains you can create a chart like this:

This blog post will describe the way to convert an Epoch time (Unix time) to a DateTime using a C# function and how you can use this in Azure Data Lake Analytics.
Unix time is defined as the number of seconds since midnight (UTC) on 1st January 1970.

The easiest way to work with Data Lake assemblies from Visual Studio is by installing the Data Lake Tools (see: https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-data-lake-tools-install).

Prerequisites
This blog assumes that you have a valid Azure subscription with at least the following resources running:
• Data Lake Storage
• Data Lake Analytics

.NET Class Library
First, we create a .NET class library which we later can use in Azure Data Lake Analytics. To do this follow the next steps.

1. Create a new U-SQL Class Library (assuming Data Lake Tools are installed)
   

2. Create a class and add the following function:

    public static DateTime UnixTimeStampToDateTime(long unixTimeStamp)
    {
        DateTime dateTimeResult = 
            new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc);
        return dateTimeResult.AddSeconds(unixTimeStamp).ToLocalTime();
    }
   

Date Lake Analytics
The next step is to register the assembly in Data Lake Analytics.

1. Create a database to save the created .NET assembly. To do this run a job with the following script:

CREATE DATABASE IF NOT EXISTS ReferenceDB;

2. Deploying the library to Data Lake Analytics is easy using Visual Studio. Right click the project and select ‘Register’. Select your Data Lake Analytics account and select the database you created in the previous step.

3. Submit the registration. After a successful registration your assembly is visible in the data explorer.
   

At this point you’re ready to use the created function in a U-SQL job.
Create a new job and use the following query:

REFERENCE ASSEMBLY ReferenceDB.EpochConvert;
 
@result = 
SELECT *
FROM (VALUES 
     (1530003614,
     EpochConvert.EpochConvert.UnixTimeStampToDateTime(1530003614))) AS vt(ts, dt);

OUTPUT @result
TO "/Output/epochConversion.csv"
USING Outputters.Csv();

This query generates two columns. One with the timestamp value and the other one with the converted value. The result is saves to a CSV file in the Data Lake Storage. You can browse the output by the Data Explorer.

Want to use your Azure Information Protection labels in SharePoint Online? This blogpost describes the way to do this.

Azure
Let’s start with some Azure Information Protection configuration within the Azure Portal.
Navigate to your AIP policy and select Advanced Properties as show in the image below.

Add the following properties (name/ value):

• SyncPropertyState : TwoWay
• SyncPropertyName : Classification

The value of the last property needs to be the same as the value for the title field in the global properties of the policy (see image below).

SharePoint Online
The next part is to configure SharePoint Online. Navigate to the library of your choice and add a new choice column. The name of the column must again be the same as the title field (see the image above). In the list of choices enter the exact label names as defined in Azure.

That’s it! Now we have everything in place. Create a document and add it to the library. You will see the classification label both in Word and SharePoint Online.

Remarks
This solution isn’t perfect there are some pitfalls:

• Setting the label from SharePoint Online will set the correct label, but it will not trigger the defined AIP protection options.
• A document created directly in SharePoint Online will not set the column. If you open the document and change the label it will set the column correct. Also, if you save the document locally first and then upload it to SharePoint, the column is set.

The integration between AIP and Office 365 will become better in the future, but for now this can surely be an useful solution.