A Fine Slice Of SQL Server
Azmat Hasan shows us two approaches for customizing log4j properties on Databricks:
The goal of this blog is to define the processes to make the databricks log4j configuration file configurable for debugging purpose
Using the below approaches we can easily change the log level(ERROR, INFO or DEBUG) or change the appender.
Read on for those techniques.
Leave a CommentNiko Neugebauer has some advice on how to clean up backups which live in Azure Blob Storage:
Continuing the topic of the Backups to Azure Blob Storage that I have kind of kicked off with the post Striping Backups to Azure Blob Storage, I want to touch on the important aspect of “keeping it clean” – thus deleting the old backups.
On the regular Windows Server this is a rather easy task, and if you are using a standard maintenance solution, such as Ola Hallengren’s Maintenance Solution or any other one. You can also use the regular SSMS maintenance (*cough* for whatever reason that is unknown to me, that you might wish to *cough*), or you can easily set up a regular Windows Scheduler with Command Line Batch or Powershell or whatever tool/script/language you like.The situation is quite different with the Backup To URL functionality, the one that is available since more than 6 years (and the good old SQL Server 2012 has even got a support for it in a certain Cumulative Update – SQL Server 2012 Service Pack 1 CU 2, to be more prices)
Niko goes through five different methods you can use, so check it out.
Leave a CommentHasan Savran takes us through optimistic concurrency with Cosmos DB:
To handle this problem, usually developers use a column like LastUpdateDt. You bring this column to Frontend and post it back to database with updated model. If LastUpdateDt hasn’t changed updates goes into database. If LastUpdateDt is changed, that means somebody else updated this model and your code rejects changes.
In this post, I will try to answer questions like “ How do we do this in Azure Cosmos DB?”“Do I need to do all that logic manually by using Cosmos DB SDK? “ I will use Cosmos DB’s REST API to demo how Cosmos DB handles Optimistic Concurrency Control automatically. If you have experience developing any REST API, you might be familiar with headers like If-Match or If-None-Match. These HTTP headers controls what should be updated or not. Also, you can use them for caching too since they check which item is updated or not, you may need to cache items until they are changed.
Hasan has a demo for us as well, so check it out.
Leave a CommentRayis Imayev shows how to use the Flatten task with Azure Data Factory wrangling data flows:
Last week I blogged about using Mapping Data Flows to flatten sourcing JSON file into a flat CSV dataset:
Part 1: Transforming JSON to CSV with the help of Flatten task in Azure Data FactoryToday I would like to explore the capabilities of the Wrangling Data Flows in ADF to flatten the very same sourcing JSON dataset.
Click through to see what’s different.
Leave a CommentMark Broadbent solves a problem for us:
I have been experiencing a problem for quite a while now in my current environment, in that some of our old builds cannot be deleted. When you attempt to do so it results in the following error:
One or more builds associated with the requested pipeline(s) are retained by a release. The pipeline(s) and builds will not be deleted.Many of our pipelines have undergone a lot of change over time to the degree it is not even obvious anymore why (or indeed where) these builds are being prevented from being dropped. The only thing that is clear is that until they can be, the old build definitions will remain.
Regardless of the reason why, Mark has the answer for how.
Leave a CommentThe main reason I wanted access to Synapse is to play around with Spark.Net via the Synapse workspace Notebooks. Currently if deploying Synapse via the public Azure portal you only get the option to create a SQL compute pool, formally known as an Azure SQLDW. While this is good, it gives us none of the exciting things that we were shown about Synapse back in November last year during the Microsoft Ignite conference.
To get the good stuff in Azure Synapse Analytics you need access to the full developer UI and Synapse Workspace.
Click through to learn more about the experience.
Leave a CommentJason Bonello shows how easy it is to integrate Azure Key Vault into Azure Databricks:
In Azure Key Vault we will be adding secrets that we will be calling through Azure Databricks within the notebooks. First and foremost, this is for security purposes. It will ensure usernames and passwords are not hardcoded within the notebook cells and offer some type of control over access in case it needs to be reverted later on (assuming it is controlled by a different administrator). In addition to this, it will offer a better way of maintaining a solution, since if a password ever needs to be changed, it will only be changed in the Azure Key Vault without the need to go through any notebooks or logic.
If you don’t use Key Vault, Databricks does include its own secrets storage, so there’s really no reason to keep them in plaintext.
Leave a CommentSQL Server team has been working diligently in adding support for current Linux distributions. To this end, the team announced support for SQL Server 2019 on RHEL 8.0, Ubuntu 18.04 and SLES 12 SP5 within last quarter.
The team is glad to announce that the Azure marketplace PAYG (Pay As You Go) images for SQL Server 2019 on RHEL 8.0, Ubuntu 18.04 and SLES 12 SP5 have been made generally available. You can deploy these images to get the latest of both SQL Server 2019 functionality and operating system improvements.
With Ubuntu 20.04 coming out soon, it’ll be interesting to see when that officially becomes supported.
Leave a CommentJesse Gorter takes us through writing streaming data from Event Hubs into Azure Data Lake Storage:
In my previous blog I showed how you can stream Twitter data to an Event Hub and stream the data to a Power BI live dashboard. In this post, I am going to show you how to store this data for long term storage. An Event Hub stores your events temporarily. That means it does not store them for later analysis. Say you want to analyze whether negative or positive tweets have an impact on your sales, you would need to store tweets for a historical view.
The question is where to store this data: directly to the datawarehouse, or store it to a data lake? This really depends on the architecture that you want to have. A data lake is often used to store the raw data historically. Is is especially interesting because it allows to store any kind of data, structured or unstructured and it is quite cheap compared to Azure SQL database or Azure SQL datawarehouse. So for that reason, we are going to store it in a data lake.
Jesse walks us through data lake creation and data migration from Event Hubs into a Data Lake Storage container.
Leave a CommentJosh Darnell looks at the options for running SQL Server in Azure:
Things are always changing the cloud, but as it stands in early 2020, these are the main options for SQL Server in Azure:
– SQL Virtual Machines
– Managed Instances
– Databases (this one really has a number of sub-options, like serverless, hyperscale, and elastic pools)I’m going to touch on each of these options, why you might choose them, and some of my personal takeaways from the talk as well.
Read on for the summary of each. Each one fills a different set of needs and has a different set of available options.
Leave a Comment