Press "Enter" to skip to content

Day: August 29, 2022

Ingestion from S3 into Azure Data Explorer

Published 2022-08-29 by Kevin Feasel

Anshul Sharma announces another source for Azure Data Explorer:

Today we are excited to launch the ability to ingest data from Amazon Simple Storage Service (S3)  into Azure Data Explorer (ADX) natively. 

Amazon S3 is one of the most popular object storage services. AWS Customers use Amazon S3 to store data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, applications, IoT devices, log analytics and big data analytics. 

Azure Data Explorer (ADX) is a fully managed, high-performance, big data analytics platform that makes it easy to analyze high volumes of data in near real time.  ADX supports ingesting data from a wide variety of sources such as Azure Blob, ADLS gen2, Azure Event Hub, Azure IoT Hub, and with popular open-source technologies such as Kafka, Logstash, Telegraph. With the new S3 support, customers can bring data from S3 natively without relying on complex ETL pipelines. 

Between this, ADF/Synapse pipelines, and SQL Server 2022, it seems that Microsoft got the message that people do use multiple clouds and do want to read AWS data in Azure. Which is good because that directly benefits me…

Comments closed

Apache Flink Updates

Published 2022-08-29 by Kevin Feasel

Danny Cranmer announces Flink 1.15.2:

The Apache Flink Community is pleased to announce the second bug fix release of the Flink 1.15 series.

This release includes 30 bug fixes, vulnerability fixes, and minor improvements for Flink 1.15. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA.

We highly recommend all users upgrade to Flink 1.15.2.

In addition to that, Jingsong Lee announces Flink Table Store 0.2.0:

Flink Table Store is a data lake storage for streaming updates/deletes changelog ingestion and high-performance queries in real time.

As a new type of updatable data lake, Flink Table Store has the following features:

– Large throughput data ingestion while offering good query performance.

– High performance query with primary key filters, as fast as 100ms.

– Streaming reads are available on Lake Storage, lake storage can also be integrated with Kafka to provide second-level streaming reads.

Read on for the changes in both platforms.

Comments closed

Overly Large Executors in ElasticMapReduce

Published 2022-08-29 by Kevin Feasel

Dmitry Tolpeko notes a change to Amazon ElasticMapReduce:

So 50 executors were initially requested with the required memory 22528 and 4 vcores as expected, but actually 9 executors were created with 112640 memory and 20 cores that is 5x larger. It should have created 10 executors but my cluster does not have resources to run more containers.

Note: The second log row specifies allocated vCores:5, it is because of using DefaultResourceCalculator in my YARN cluster that ignores CPU and uses memory resource only. Do not pay attention to this, the Spark executor will still use 20 cores as it reported in the third log record above.

Click through for the reason.

Comments closed

Raw String Literals in C# 11

Published 2022-08-29 by Kevin Feasel

Patrick Smacchia shows off raw string literals:

C# 11 introduces Raw String Literals. Undoubtedly this feature will become very popular because it represents an elegant way to solve some issues with actual string literal.

Let’s have a look at such raw string literal with interpolation. Notice that a raw string literal necessarily starts and ends with at least 3x double quote characters """.

This is similar to how several other languages, including F# and Python, do it.

Comments closed

Multiple Rows of Tabs in SSMS

Published 2022-08-29 by Kevin Feasel

Kevin Hill needs more than one row:

If you spend much time at all in SQL Server Management Studio (SSMS), you’ve no doubt had trouble figuring out which query tab is the one you need.

Many DBAs and SQL Devs find themselves (despite best intentions and New Years Resolutions) with dozens of tabs open, and many of them unsaved.

Kevin shows the built-in solution. My solution is Tabs Studio. I paid $50 for it and have zero qualms with the purchase.

Comments closed

SysInternals EULA Acceptance in Powershell

Published 2022-08-29 by Kevin Feasel

Jeff Hicks finds your terms acceptable:

I just saw a very, very handy thing on Twitter where you can set a registry key that will automatically accept all EULA prompts for the SysInternals tools. I know there is a command-line switch I can use, but I never remember to use it. Setting the registry key appears to accept EULAs globally. Naturally, I wanted a PowerShell script I could run.

Read on for that script.

Comments closed

Log Truncation Improvements in SQL Managed Instance Backup Restorations

Published 2022-08-29 by Kevin Feasel

Niko Neugebauer reports on some performance improvements:

There are multiple phases of the SQL MI database restore process, which, aside from the usual SQL Server restore operations, includes registering databases as an Azure asset for visibility & management within the Azure Portal. Additionally, the restore process forces a number of specific internal options, and some property changes such as forcing the switch to the FULL recovery model and forcing the database option PAGE_VERIFY to CHECKSUM, as well as eventually performing a full backup to start the log chain and provide full point-in-time- restore options through the combination of full and log database backups.

The restore operation on SQL MI also includes log truncation, and the execution time for the truncation has been vastly improved, which means that customers can expect their entire database restore process to become faster on both service tiers.

Click through to see what kind of performance gains you can expect.

Comments closed

Allow Azure AD B2B Users to Access Power BI Reports

Published 2022-08-29 by Kevin Feasel

Gilbert Quevauvilliers provides access:

I was recently doing some testing and when I went to share a Power BI report with an external (B2B) user I could not initially find out how to do this.

NOTE: I had already created my External (B2B) users, you can follow this guide below if you still need to create the external (B2B) users: Distribute content to external guest users with Azure AD B2B – Power BI | Microsoft Docs

I thought it would be good to share the steps on how I achieved this below.

Read on to see how. Also check out the insightful question in the comments section.

Comments closed