Data Analysis on Digi Hunch

A taste of IoT device tracking

Fri, 03 Mar 2023 10:53:00 -0400

Last fall I worked on a demo project for IoT device tracking on AWS so I had some reading.

Overview

From a strategic level, as servers move away to the cloud, AWS envisions that the footprint left on premise will mainly be IoT devices. The role of the cloud therefore becomes a central point of management for IoT devices. As a result, most of the AWS services for IoT are managed services. The best resources are two white papers: IoT Lens – AWS Well-Architected Framework and Securing Internet of Things (IoT) with AWS. The most important services are:

IoT Analytics: Makes it easy to run sophisticated analytics on volumes on IoT data. It connects to the underlying IoT data store and allow you to build your own analytical queries and Jupiter notebooks.
IoT Events
IoT Core: Core features for IoT.
IoT SiteWise
IoT Device Management
IoT Greengrass
FreeRTOS: IoT devices usually need to be small and power efficient. The software and OS running on the device is important. FreeRTOS is a real-time operating system for micro-controllers supported by AWS. FreeRTOS provides kernel, OS and libraries to securely connect your edge device to the cloud in no time.

In this post I will explore IoT architecture at high level. In real life, you program your device with AWS IoT Device SDK and AWS IoT API in different languages. In this post, I use a script to simulate GPS data, and push it to AWS IoT using SDK. Then I render the location using sample AWS code for Amazon Location.

IoT Architecture

When creating IoT services we consider registration and telemetry capturing flow. The Well architected white paper proposes registration flow as such:

Registration Flow

Device Registry to keep track of devices (aka Things). You can find where your devices are, and filter by a common feature (e.g. ModelX device only). Registration flow usually involves a testing of communication between device and server. The authentication must be 2-way where server needs to validate device identity, and device needs to validate server identity. You can use a unique X.509 certificate per device to adhere to security best practices on AWS. This way, if one device gets hacked, the entire fleet of devices is not affected by one certificate being compromised. An alternative authentication method is Cognito. With Cognito you can sign your users into a mobile application, so you use IAM policies to authenticate them into viewing different dashboards or viewing the data that pertains to them specifically. IoT Core policies can help manage authorization.

The white paper also proposes a few options for capturing telemetry:

Options for capturing telemetry

These options presents a common pub-sub pattern, where the device streams message by topic to the Message Broker in IoT core. The IoT core also involves policy and rules. A rule may involve a subscriber to consume the messages. A more comprehensive architecture from AWS IoT device connectivity workshop looks like this:

The message path from device to IoT core remains the same. When building an IoT solution, we first address the messaging path.

IoT Protocols

In IoT core, Device Gateway is the entry point for IoT devices connecting to AWS.It supports MQTT, WebSockets and HTTP 1.1 protocols, on top of TLS. Registration flow uses HTTP/REST protocol for provisioning, and with MQTT protocol for a message test. For pushing telemetry, we can use both but prefer MQTT (topic based) because of its advantages in IoT messaging. Here is a good article on the differences.

AWS has a white paper on designing MQTT topics, with a few communication patterns and best practices. The SDK documentation also explained the communication protocols, including:

HTTPS: publish only
MQTT: publish and subscribe
MQTT over WebSocket: publish and subscribe. Device Gateway will maintain long lived, bi-directional connections, enabling devices to send and receive messages at any time with low latency.

Pay attention to the authentication mechanism. As to what protocol is used in a communication, they can be dynamically negotiated using the ALPN protocol. ALPN (Application-Layer Protocol Negotiation) is a TLS protocol extension that allows the application layer to negotiate which protocol should be performed over a secure connection in a manner that avoids additional round trips and which is independent of the application-layer protocols.

GPS data simulator

I don’t have a GPS chip. To get sample GPS data, I used geojson.io website, use a pen to paint the points and collect the result in JSON format. The data looks like this:

{
 "type": "FeatureCollection",
 "features": [
 {
 "type": "Feature",
 "properties": {},
 "geometry": {
 "coordinates": [
 [
 -119.4966304331144,
 49.88901098598916
 ],
 [
 -119.4966304331144,
 49.889903059931726
 ],
 [
 -119.49658503509582,
 49.89066350338888
 ],
 [
 -119.49658503509582,
 49.891365440561145
 ],
 [
 -119.4966304331144,
 49.89219897769374
 ],
 [
 -119.4966304331144,
 49.88735841200943
 ],
 [
 -119.49660773410511,
 49.8880603972496
 ],
 [
 -119.4966304331144,
 49.88885011844141
 ]
 ],
 "type": "LineString"
 }
 }
 ]
}

Save this file as map.geojson to later feed it to device simulation script.

Rendering location data

In AWS location samples project, the sample project maplibre-js-react-iot-asset-tracking is a good demo of IT. The readme document contains a walk through, using AWS amplify services. The steps includes creating certificates, configuring lambda function to add location data to tracker. The project directory also includes the device simulation script, as index.js. I slightly modified the content to this:

const awsIot = require("aws-iot-device-sdk");

// Replace with your AWS IoT endpoint
const THING_ENDPOINT = "safdsa-ats.iot.us-east-1.amazonaws.com"; // get from console
const CLIENT_ID = "trackThing01";
const IOT_TOPIC = "iot/trackedAssets";
const DEVICE_ID = "thing123";
const GEOJSON_FILEPATH="geojson/map.geojson";
const fs = require('fs')
const file_raw = fs.readFileSync(GEOJSON_FILEPATH).toString();
const positions_raw = JSON.parse(file_raw).features[0].geometry.coordinates
const POINTS_ON_MAP=[]
for (var i=0;i<positions_raw.length;i++){
 POINTS_ON_MAP.push({lat:positions_raw[i][1],long:positions_raw[i][0]})
}

const device = awsIot.device({
 host: THING_ENDPOINT,
 keyPath: `${__dirname}/certs/private.pem.key`,
 certPath: `${__dirname}/certs/certificate.pem.crt`,
 caPath: `${__dirname}/certs/root-CA.pem`,
 clientId: CLIENT_ID,
 keepalive: 60000,
});

console.log("Connecting to %s with client ID %s", THING_ENDPOINT, CLIENT_ID);

device.on("connect", async function () {
 console.log("Connected to device %s", CLIENT_ID);

 for (const point of POINTS_ON_MAP) {
 const message = {
 payload: {
 deviceId: DEVICE_ID,
 timestamp: new Date().getTime(),
 location: point,
 },
 };
 console.log(
 "Publishing message to topic %s: %s",
 IOT_TOPIC,
 JSON.stringify(message)
 );
 device.publish(IOT_TOPIC, JSON.stringify(message), { qos: 1 });

 // Set timeout to sleep
 await new Promise((resolve) => setTimeout(resolve, 10000));
 }

 device.end();
});

We can obtain certificate ID from AWS console or by CLI command:

aws iot list-certificates --output text --query 'reverse(sort_by(certificates,&creationDate))[:1].[certificateId]' | cat

When running the script, it pushes data to IoT core service. The AWS Amplify project creates Lambda function that is subscribed to the topic and trigger actions. The data are used to render points on a map, which is available on the front end.

Summary

This is an overly simplified use case but it covers the basics. IoT solution will use a lot managed service and familiar technologies (e.g. TLS, certificate). Creating an IoT solution is mostly about address the onboarding services and make use of the MQTT based workflow. AWS managed services makes these easier.

More IoT workshops are available on AWS workshops.

Intro to Data Analytics Platform on Azure

Sun, 21 Mar 2021 21:57:00 -0400

Having been in transactional data world for almost the entire career, recently I have to pick up quite a few things to catch up on the analytical workload. The main purpose of data analytics project is to build analysis services models and manage deployed databases. Later in this post I’ll discuss some useful Azure resources for data analytics.

Data Model

Data are typically organized in relational model for better transactional performance, following the normalization forms. The relational model, however, might not be the most appropriate schema for analytics. In this case, it is better to use a separate non-relational repositories that can store information in a format that better aligns with its semantics, and hence more friendly to analytical applications.

A model consists of: data sources, tables, relationships, measures, KPIs, roles, etc. The model can be deployed to analysis database (e.g. SSAS). When deploying, queries (from source) and calculations are done.

Data modelling is the process of determining how your tables are related to each other. This process is done by defining and creating relationships between the tables. From that point, you can enhance the model by defining metrics and adding custom calculations to enrich your data. Creating an effective and proper data model is a critical step in helping organizations understand and gain valuable insights into the data. The model is another critical component that has a direct effect on the performance of your report and overall data analysis. The process of preparing data and modelling data is an iterative process.

Data Warehouse

Moreover, organizations have multiple data stores, with varying formats and different structures such as live stream, sensor, etc. They all need to be combined to generate insights. The process of combining all of the local data source is known as data warehousing. The process of analyzing streaming data and data from the Internet is known as Big Data Analytics.

A data warehouse gathers data from many sources within an organizations. This data is then used as the source for analysis, reporting and OLAP. The focus of a data warehouse is to provide answers to complex queries. A modern data warehouse might contain a mixture of relational and non-relational data, including files, social media streams, IoT sensor data.

The diagram above is platform neutral. If you take Azure for example, a typical data warehouse platform involves the following components:

Azure Data Factory: ingestion of data (integration service)
Azure Data Lake Storage: store large quantity of data before analyzing
Azure Databricks: other forms of data preparation (transformation, cleaning) by Spark
Azure Synapse Analytics: store cleansed data, for Azure Analysis Service to consume
Azure Analysis Service: query Synapse Analytics for detailed analysis and generate insights
Power BI: Generate graphs, charts and reports by using information from Azure Analysis service

This is the diagram of those components as seen on Azure documentation:

Modern Data Warehouse

Let’s discuss each components.

Azure Data Factory (ADF)

Big data requires a service that can orchestrate and operationalize process to refine the enormous stores of raw data into actionable business insights. ADF is managed cloud service built for complex hybrid ETL, ELT and data integration projects. ADF retrieves data from one or more data sources, and convert it into a format you can process. The data sources might present data in different ways, and contain noises that need to be discarded. For example, the source data may contain dates with bad format. ADF can unify the data structure. In ADF, you define the work performed as a pipeline of operations. A pipeline can run continuously, or triggered by schedule.
A linked service provides the information needed for ADF to connect to a source or destination. A pipeline is a logical grouping of activities that together perform a task.

The ADF UX (user interface experience) lets you visually author and deploy resources for your data factory without having to write any code. You can drag activities to a pipeline canvas, perform test runs, debug iteratively, and deploy and monitor your pipeline runs.

Azure Data Lake Storage (ADLS)

A data lake is a repository for large quantities of raw data. Because the data is raw and unprocessed, it’s very fast to load and update, but the data hasn’t been put into a structure for efficient analysis. You can think of a data lake as a staging point for your ingested data, before it’s massaged and converted into a format suitable for performing analytics. Note that a data warehouse also stores large quantities of data, but the data in a warehouse has been converted into a format for efficient analysis. Data lake holds raw data, whereas data warehouse holds structured information.

Azure Data Lake Storage is essentially an extension of Azure Blob storage, organized as a near-infinite file system. It supports POSIX file and directory structure for storage and RBAC. ADLS is also compatible with HDFS, a popular open-source solution to store large quantities of data.

Azure Databricks

Apache Spark is in-memory cluster computing technology, much faster than disk-based applications, and works with multiple programming languages to let you manipulate distributed data sets (DDS). There is no need to structure everything as map and reduce operations. Databricks develops a web-based platform for working with Spark cluster. It provides automated cluster management and IPython-style notebooks.

Azure Databricks is a managed Apache Spark environment running on Azure to provide big data processing, streaming, and machine learning. Apache Spark is a highly efficient data processing engine, with rich selections of libraries, that can consume and process large amounts of data very quickly. Azure Databricks also supports structured stream processing.

Delta Lake is an open-source storage layer in Azure Databricks that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs.

Azure Synapse Analytics

Azure Synapse is an end-to-end solution. It unifies data analysis, integration and orchestration, Data Lake, Data Warehouse, ELT/ELT, ML capabilities and visualization. With Synapse, you can process large amounts of data very quickly. You can ingest data from external sources (e.g. flat file, ADLS, other DBMS) and then transform and aggregate the data into a format suitable for analytics processing. You can also use this data as input to further analytical processing using Azure Analysis Services. Azure Synapse is a comprehensive service with the following components:

Synapse Analytics – a successor of SQL DW technology. Synapse analytics has inherited its MPP capability.
Data Exploration – Synapse Studio makes data exploration in Data lakes, SQL engine and Spark very easy.
Data Integration – inherited ADF’s data movement and transformation components, which allows building complex ETL pipelines without code
Development – supports Spark, Python, Scala, Spark notebooks, SQL
Data visualization – Synapse Studio allows user to connect to Power BI workspace and get the same report development experience

The most critical component, Synapse Analytics is analytics engine, designed to process large amounts of data very quickly. Synapse Analytics leverages a MPP (massively parallel processing) architecture, including a control node and a pool of compute nodes. When you submit a processing request, the control node transforms it into smaller requests and send them to compute nodes. Each compute node runs the queries over the portion of data that they each hold. When each node has finished its processing, the results are sent back to the control node where they’re combined into an overall result.

Synapse Analytics supports two computational models: SQL pools and Spark pools. In a SQL pool, each compute node uses an Azure SQL Database and Azure Storage to handle a portion of the data. You can submit queries in the form of T-SQL statement. Synapse Analytics uses a technology named PolyBase to retrieve data from a wide variety of sources (e.g. Blob, ADSL, CSV). You can save the data read in as SQL tables in Synapse Analytics service. In a Spark pool, the nodes are replaced with Spark cluster. You run Spark jobs comprising code written in Notebooks (in Python, Scala, or Spark SQL). The Spark cluster splits the work out into a series of parallel tasks that can be performed concurrently. You can save data generated by your notebooks in Azure Storage or ADLS. To scale Spark pool, you can specify the cluster size, or turn on autoscaling.

Azure Analysis Service (AAS)

AAS is a fully managed PaaS that enables you to build tabular models to support OLAP queries. You can combine data from multiple sources (e.g. Azure SQL Database, ADLS, Cosmos DB, etc). You use those data sources to build models that incorporate your business knowledge. A model is essentially a set of queries and expressions that retrieve data from various data sources and generate results. The results can be cached in-memory for later use, or they can be calculated dynamically, directly from underlying data sources. AAS has significant functional overlap with Synapse Analytics, but it’s more suited for processing on a smaller scale. The comparison below outlines the difference:

Synapse Analytics	Azure Analysis Service (AAS)
– very high volumes of data (multi-terabyte to petabyte sized datasets) – very complex queries and aggregations – data minding, and data exploration – complex ETL operations – low to mid concurrency (127 users or fewer)	– smaller volumes of data (a few terabytes) – multiple resources that can be correlated – high read concurrency – detailed analysis, and drilling into data, using functions in Power BI – rapid dashboard development from tabular data

Many scenarios can benefit from using Synapse Analytics and Analysis Services together. If you have large amounts of ingested data that requires preprocessing, you can use Synapse Analytics to read this data and manipulate it into a model that contains business information rather than a large amount of raw data. The scalability of Synapse Analytics gives it the ability to process and reduce many terabytes of data down into a smaller, succinct dataset that summarizes and aggregates much of this data. You can then use AAS to perform detailed interrogation of this information, and visualize the results of these inquiries with Power BI.

Azure HDInsight

Azure HDInsight is a managed analytics service based on Apache Hadoop, a collection of open-source tools and utilities that enable you to run processing tasks over large amounts of data. HDInsight uses a clustered model, similar to that of Synapse Analytics. HDInsight stores data using ADLS. You can use HDInsight to analyze data using frameworks such as Hadoop Map/Reduce, Apache Spark, Apache Hive, Apache Kafka, Apache Storm and more.

Power BI

Microsoft PowerBI is a collection of software services, apps and connectors. It consists of a Microsoft Windows Desktop application Power BI Desktop, an online SaaS service Power BI service, and mobile Power BI apps available on any device. These three elements are designed to let people create, share and consume business insights. A common workflow with Power BI can be outlined as:

Bring data into Power BI Desktop, and create a report
Publish to the Power BI service, where you can create new visualizations or build dashboards
Share dashboard with others, especially people who are on the go
View and interact with shared dashboards and reports in Power BI mobile apps.

Basic building blocks in Power BI include:

visualizations: chart, colour-coded map, etc
datasets;
reports: a collection of visualizations that appear together on one or more pages;
dashboards: when you’re ready to share a report, or a collection of visualizations, you create a dashboard, much like the dashboard in a car, a Power BI dashboard is a collection of visuals from a single page that you can share with others. Often, it’s a selected group of visuals that provide uick insight into the data or story you’re trying to present.

Census Data from Statistics Canada

Thu, 25 Feb 2021 21:18:09 -0400

Statistics Canada carries census every 5 years, with 2016 being the last run. The census data by Statistics Canada provides a wealth of insights but are published in raw format. Post-processing work is needed to extrapolate information, such as median income of a neighbourhood, age distribution of a city, etc. For someone like myself without any background in geographical informatics, it took a bit of learning to see how these work together.

The following information are typically included in the Census data:

Population
Population density
Age
Structural type of dewellings
Family size
Marital status
Language
Income
Place of birth
Level of education
Occupation

We will start with level of Geographics. The level of geographics may change slightly between census programs in different years. The most recent 2016 census uses the following diagram to depict levels of geographics:

Geographic Levels

This diagram reflects a number of different hierarchies of geographic units. The best resource to understand each block, is the illustrated glossary and the chapter Census Geography in comprehensive Guide to the Census Population. For example, the chain on the far left of the diagram runs across these levels:

In this hierarchy, the level of Geographical Region of Canada is standardized in Standard Geographic Classification (SGC), in which the provinces and territories are also encoded. Note that each Census include a dictionary where all sorts of codes are kept. The dictionary also includes definition of the rest two levels: FSA (forward sortation area as the first three digits of postal code) and postal code (all six digits). Note that postal code is a mark of Canada Post Corporation, and you may translate postal code into other levels in standard geographic areas, such as CD. This is not straightforward though. You will need a product called Postal Code Conversion File (PCCF) for the conversion. Statistics Canada does not directly distribute this product. It works with its Data Liberation Initiative (DLI) partners to deliver this product.

On the diagram there are also other path to run down the hierarchy. For example, from Canada down to federal electoral district (aka ridings). However, the census is not carried out by either election ridings or postal code. Instead, it is carried out by its own collection of levels dedicated for census purpose. When using census data, we need to be familiar with these units.

Census metropolitan area (CMA) and census agglomeration (CA): formed by one or more adjacent municipalities centred on a population centre (known as the core), such as Chatham-Kent CA, Kitchener-Cambridge-Waterloo CMA. Note that CMA and CA can expand across provincial borders, such as Ottawa – Gatineau CMA. So CMA or CA is not a unit under province or territory.

Census Division (CD, essentially a region or county): general term for provincially legislated areas (such as county, municipalité régionale de comté and regional district) or their equivalents.

Census Subdivision (CSD, essentially a city): the general term for municipalities or areas treated as municipal equivalents for statistical purposes.

Census Tract (CT): small, relatively stable geographic areas that usually have a population of less than 10,000 persons, based on data from the previous Census of Population Program.

Dissemination Area (DA): is a small, relatively stable geographic unit composed of one or more adjacent dissemination blocks with an average population of 400 to 700 persons based on data from the previous Census of Population Program. It is the smallest standard geographic area for which all census data are disseminated.

Dissemination Block (DB): an area bounded on all sides by roads and/or boundaries of standard geographic areas. The dissemination block is the smallest geographic area for which population and dwelling counts are disseminated.

With these in mind, we can build two hierarchies closely related to census data:

Now we download census profile data from Statistics Canada. In the dropdown you can pick from the many of the aforementioned geographic levels.

If you pick Census tracts (CT), there is one data file. The CSV file is about 160M. Also note that under geographic level column, it indicates two levels: CA/CMA and CT, which is important to keep in mind as we go through the data. In the content of the CSV, under the GEO_LEVEL column, value 1 stands for CA/CMA and value 2 stands for CT. Therefore, when GEO_LEVEL=1, the GEO_CODE value is a CA/CMA code based on Statistical Area Classification; when GEO_LEVEL=2, the GEO_CODE value is a CT numerical name (preceded by CMA/CA code). What CT numerical name represents what geographic area, is all defined in Census Tract Reference Map. There is no textual name for each census tract.

To take another example, select Dissemination areas (DAs) from the dropdown. Now the size of the CSV becomes 1.6G, but smaller data files are provided by province and territories. Select the data file for Ontario only.

Note that there are five geographic levels as indicated: Canada, provinces/territories, CDs, CSDs and DAs. This suggests we will see 5 different values under the GEO_LEVEL column in the data file:

0 – Canada
1 – Provinces and Territories
2 – CDs
3 – CSDs
4 – DAs

Read the SGC documentation to understand the code from level Canada to level CSD. DA is similar to CT because the code is defined in reference map here.

Apart from DA and CT, there are other levels (such as ridings) with reference maps, as outlined in the Census geography page.

With all the above information, we can parse the data programmatically. Of course, the schema and coding information applies to Canada. Outside of Canada, pretty much all states have a counterpart government agency that manages census and statistics, just with different formats to understand from ground up. A lot of census geography concepts applies to other countries as well. For example:

Census Bureau of United States
Australian Bureau of Statistics
Office for National Statistics (UK)
Eurostat (European Union)

Welcome to the world of data.