Press "Enter" to skip to content

Category: Cloud

An Overview of Azure IoT Central

Published 2022-04-04 by Kevin Feasel

James Serra looks at IoT Central:

This is a short blog to give you a high-level overview on a product called Azure IoT Central. I saw this fairly new Azure product (GA Sept 2018) in use for the first time at a large manufacturing company who was using it at their manufacturing facility (see Grupo Bimbo takes a bite out of production costs with Azure IoT throughout factories). They have thousands of sensors that are collecting data for all the machines used in producing their products. In short, think of it as an “Application Platform as a Service (aPaas)” for quickly building IoT solutions. It’s boxing up IoT hub, Device Provisioning Service (DPS), Stream Analytics, Data Explorer, SQL Database, Time Series Intelligence and Cosmos DB to make it easy to quickly build a solution and get value out of the IoT data. To get an idea of the what this solution would look like, check out the IoT Central sample for calculating Overall Equipment Effectiveness (OEE) of industrial equipment.

I haven’t seen much use of this service, as generally any use case I’ve seen around IoT quickly turns into using IoT Hub and IoT Edge to develop custom code.

Comments closed

An Overview of the Microsoft Defender Ecosystem

Published 2022-04-01 by Kevin Feasel

Alan La Pietra looks at all the Defenders you can get your hands on:

Microsoft Defender Antivirus is available in Windows 10 and Windows 11, and in versions of Windows Server

Microsoft Defender Antivirus is a major component of your next-generation protection in Microsoft “Defender for Endpoint”

Microsoft Defender Antivirus is built into Windows, and it works with Microsoft Defender for Endpoint to provide protection on your device and in the cloud

I see the hand of marketing in this. Which means they’ll probably all have different names nine months from now.

Comments closed

KQL Series

Published 2022-03-31 by Kevin Feasel

Hamish Watson does a document dump:

So what did we do here?

It searched our stored security events in the SecurityEvent table for all Accounts that had a successful login in the last 3 hours and we chose to display only the Account and number of log off events per Account in numerical order with the highest at the top.

So far I’ve introduced some new operators and things – but what is a really quick way to learn KQL?

Start with this post and just keep navigating forward. Hamish has ten posts in total.

Comments closed

Zero-Rename Writes in ElasticMapReduce Hive

Published 2022-03-30 by Kevin Feasel

Suthan Phillips, et al, show off some updates to the way Hive transactions commit in AWS’s ElasticMapReduce:

Our customers use Apache Hive on Amazon EMR for large-scale data analytics and extract, transform, and load (ETL) jobs. Amazon EMR Hive uses Apache Tez as the default job execution engine, which creates Directed Acyclic Graphs (DAGs) to process data. Each DAG can contain multiple vertices from which tasks are created to run the application in parallel. Their final output is written to Amazon Simple Storage Service (Amazon S3).

Hive initially writes data to staging directories and then move it to the final location after a series of rename operations. This design of Hive renames supports task failure recovery, such as rescheduling the failed task with another attempt, running speculative execution, and recovering from a failed job attempt. These move and rename operations don’t have a significant performance impact in HDFS because it’s only a metadata operation when compared to Amazon S3 where the performance can degrade significantly based on the number of files written.

This post discusses the new optimized committer for Hive in Amazon EMR and also highlights its impressive performance by running a TPCx-BB performance benchmark and comparing it with the Hive default commit logic.

Read on for a description of how commit operations work in general and how the updated Hive committer can help with certain types of queries.

Comments closed

Right to Be Forgotten in Delta Lake

Published 2022-03-30 by Kevin Feasel

Milos Colic, et al, tackle a tricky problem:

With Delta, we have one more tool at our disposal to address GDPR compliance and, in particular, “the right to be forgotten” – VACUUM. Vacuum operation removes the files that are no longer needed and that are older than a predefined retention period. The default retention period is 30 days to align with GDPR definition of undue delay. Our earlier blog on a similar topic explains in detail how you can find and delete personal information related to a consumer by running two commands:

The part I’m finding tricky here is, how does this handle “time travel” scenarios in which you’re looking at prior iterations of data? I haven’t run through all of the scenarios so this is just speculation, but it seems that even with all of these changes, you’d still have to worry about historical data containing that sensitive information.

Comments closed

Migrating SSIS On-Prem Workloads into Azure

Published 2022-03-30 by Kevin Feasel

Jitendra Yadeo has put together a how-to guide:

– There can be scenario where organization wants to migrate there existing SSIS ETL process on cloud so instead of rewriting SSIS package using Cloud specific ETL tools like Azure Data Factory we can directly migrate SSIS packages and call it through Azure Data Factory.

– Goal of this blog is to show how SSIS packages hosted on on-premise can be migrated to Azure Data Factory (ADF) using Azure-SSIS Integration Runtime (IR).

Read on for a step-by-step guide.

Comments closed

Configuring FIDO2 for Azure Active Directory

Published 2022-03-29 by Kevin Feasel

Joey D’Antoni takes us through a process:

If this sounds scary, and it does to me, who is by far not an expert in all things security, but knows a little bit, you may ask, what are some alternative solutions? The answer to that question is Fido2, a different protocol for MFA and auth. Remember all of that stuff Microsoft talks about with passwordless login? That’s all based around Fido2. I configured this for DCAC’s Azure Active Directory yesterday, and I wanted to walk you through the steps.

Click through to learn how.

Comments closed

Azure ML Well-Architected Framework Review

Published 2022-03-28 by Kevin Feasel

Ben Brauer has good news:

Microsoft offers prescriptive guidance called the Well-Architected Framework that optimizes workloads implemented and deployed on Azure. This guidance has been generalized for most workloads and creates a basis for reliable and secure applications that are cost optimized.

We have begun to build on this base content set to include more precise guidance for specific workload types, such as machine learning, data services and analytics, IoT, SAP, mission critical apps, and web apps. Machine Learning was the first branch from the base content, which came into fruition in the Fall of 2021.

In case you have never used the Azure Well-Architected Review assessment tool, it’s really useful. It can take hours (or days) to go through the review but if you take it seriously and have the right people in the room giving answers, you’ll get concrete guidance on how to optimize your Azure-based solutions.

Comments closed