Working With The Databricks API Via Powershell

Gerhard Brueckl has a Powershell module for interacting with Databricks, either Azure or AWS:

As most of our deployments use PowerShell I wrote some cmdlets to easily work with the Databricks API in my scripts. These included managing clusters (create, start, stop, …), deploying content/notebooks, adding secrets, executing jobs/notebooks, etc. After some time I ended up having 20+ single scripts which was not really maintainable any more. So I packed them into a PowerShell module and also published it to the PowerShell Gallery (https://www.powershellgallery.com/packages/DatabricksPS) for everyone to use!

This looks like a pretty good module if you work with Databricks.

Managing Powershell Core On Non-Windows Machines

Max Trinidad shows us how to grab the latest version of Powershell Core if you aren’t using Windows:

So, if PowerShell Core isn’t available in the package repository, with a few steps you can download and install PowerShell. But, the first thing I do is to remove it before installing.

Ubuntu

## - When PowerShell Core isn't available in their repository: (download and execute install)
cd Downloads
wget https://github.com/PowerShell/PowerShell/releases/download/v6.1.1/powershell_6.1.1-1.ubuntu.18.04_amd64.deb
sudo dpkg -i powershell_6.1.1-1.ubuntu.18.04_amd64.deb
## - When available in Apt/Apt-Get repository:
sudo apt install -y powershell #-> Or, powershell-preview

Click through for demos of CentOS (or any other yum-based system) and MacOS X.

Breaking Changes Coming To dbatools

Chrissy LeMaire warns us about breaking changes coming to dbatools with release 1.0:

Sometime in the next month, I’ll also be updating Start-DbaMigration to more closely match the parameters of Export-DbaInstance. Parameters like NoDatabases and NoLogins will be replaced by -Exclude Databases, Logins.

So the functionality won’t necessarily change, but if you have scheduled tasks or scripts that perform migrations, you will need to update your parameters once you update dbatools once these changes are made.

Keep an eye out for all of these changes if you’re a regular dbatools user or have processes scripted.

Removing The Azure Module

Max Trinidad has built a function to remove older versions of the Azure module:

As you probably know by now, “Azure RM” modules has been renamed to “Az” Module. Microsoft want you to start using this module moving forward. Currently, this new release is on version 0.5.0, and you’ll need to remove the any previous module(s) installed. Information about Azure PowerShell can be found on the following link.

Now, there’s always been a tedious task when manually removing module dependencies, and there’s no exception with the “Az” module.  So, we can all take advantage to PowerShell and create a script to work around this limitation.

And, below is a few options.

Max also provides us a couple of other options as well.

Useful Powershell Aliases For Docker

Elton Stoneman shares a few useful aliases in Powershell for managing Docker containers:

Docker PowerShell Alias #2 – drmf

Removes all containers, whether they’re running or not. Useful when you want to reset your running containers and get back to zero:

function Remove-AllContainers { docker container rm -f $(docker container ls -aq)
}
Set-Alias drmf Remove-AllContainers 

Use with caution

Elton shares several more at the link and also includes a link to a Github gist with them all.

Automating E-mail Of Windows Event Log Alerts Via Powershell

Dave Bermingham shows us how to automatically fire off e-mails for specific Windows Event Log event IDs:

The first thing that you need to do is write a Powershell script that when run can send an email. While researching this I discovered many ways to accomplish this task, so what I’m about to show you is just one way, but feel free to experiment and use what is right for your environment.

In my lab I do not run my own SMTP server, so I had to write a script that could leverage my Gmail account. You will see in my Powershell script the password to the email account that authenticates to the SMTP server is in plain text. If you are concerned that someone may have access to your script and discover your password then you will want to encrypt your credentials. Gmail requires and SSL connection so your password should be safe on the wire, just like any other email client.

Here is an example of a Powershell script that when used in conjunction with Task Scheduler which will send an email alert automatically when any specified Event is logged in the Windows Event Log.

Read on for the script as well as some bonus troubleshooting.

Approved Powershell Verbs

Richard Siddaway on approved verbs in Powershell:

The other very useful set information are the synonyms for verbs that you shouldn’t use. For instance don’t use Append, Attach, Concatenate or Insert – use Add. Some of this information is contextual though as you shouldn’t use Pop or Out as a synonym for Exit BUT Pop is perfectly valid when removing an item off a stack (Pop-Location is the only cmdlet I know of that works in that way).

Read on for a link to the approved verbs list.

Going In-Depth On Powershell Arrays

Kevin Marquette has a tour de force on Powershell arrays:

When your array is a collection of string or integers (value types), sometimes you will want to update the values in the array as you enumerate them. Most of the iteration loops above use a variable in the loop that holds the value. If you update that variable, the original value in the array is not updated.

The exception to that statement is the for loop. If you are wanting to walk an array and update values inside it, then the for loop is what you are looking for.

 for ( $index = 0; $index -lt $data.count; $index++ ) { $data[$index] = "Item: [{0}]" -f $data[$index] }

This examples takes a value by index, makes a few changes, and then uses that same index to assign it back.

This is a book chapter-length blog post full of good information.

Creating Fancy HTML Reports With Powershell

Jeffery Hicks shares several tips on creating fancy HTML reports using Powershell:

Usage is pretty straightforward. You specify one or more computers and off you go. There is a default value for the resulting HTML file, but you’ll likely want to specify your own.   Because the function is generating custom HTML on the fly, I also provided options for you to provide pre and post content HTML material, just as you might with ConvertTo-HTML. I also give you an option to specify a graphics file which is display like a logo at the top of the report. The graphics file will be embedded in the HTML file. The CSS is also embedded in the HTML making the entire file completely self-contained.

The one knock I have is the gradient color scheme:  people with Protanopia or Deuteranopia will have trouble reading the free space indicator, as the colors blur into one another.  Otherwise, this looks great.

Reuse Versus Learning

Fred Weinmann argues for the value in learning modules over built-it-yourself solutions for expanding knowledge:

When you start with a new technology, you don’t start on a green field. You’ve got lots of luggage you carry around with you (previous experience) and a perception where you want to go (project conditions, goals). However the technology you interact with may not be limited to just that.
Using, consuming and discovering tools written for a technology by someone who has been busy in that particular field for years can guide you in your own comprehension of the technology.
For example, if you were to shift to Database Administration of MSSQL servers, there is this community module called “dbatools” which covers most of the tools you will need (seriously, we spent lots of time on it to make that true). Now, no module can replace your own mastery of the topic. You will need to know how backup and restore works. How to design a new database and how to troubleshoot inefficient queries.
No tool can save you from needing to understand the concepts and the procedures.
However by looking up the commands, what they do and how they do it, you can benefit from the experience from some seriously senior dbas. For free.
The key point here is that going completely your own path may result in a bad solution, in piled up technological debt which you didn’t see coming because you didn’t have the context yet, because you tried to map new information into your previous context, whether that fit or not.

Click through for the full argument.  I’d go a step further:  these modules are capital.  They are the sum of knowledge built up over time and eschewing this so you can traverse the same ground and try to solve the same problems is a waste of time (unless you can do it better).  Build from what is there and use that precious time you have solving other problems.

Categories

November 2018
MTWTFSS
« Oct  
 1234
567891011
12131415161718
19202122232425
2627282930