Press "Enter" to skip to content

Category: Dates and Numbers

Adding the Current Date and Time to a PySpark Data Frame

Gilbert Quevauvilliers wants to know what time it is:

How to add current DateTime to existing PySpark data frame in a Fabric Notebook

In the blog post below, I am going to describe how to add the current Date Time to your existing Spark data frame.

This is really useful when I am inserting data into a Fabric Lakehouse table, and I want to know when the data got inserted.

Read on for the answer.

Comments closed

Migrating DATETIME Data to DATETIMEOFFSET

William Assaf adds some time zones:

I recently reviewed, worked on, and added a similar example to the DATETIMEOFFSET Microsoft Learn Docs article at the recommendation of my colleague Randolph West, who guessed (accurately) I would enjoy such a task. It was a nice pre-Build diversion. 

This topic is one that I have co-presented on in the past and hounded project capstone review presentations about. If you’re not storing time zone offset in your date/time data, you’re setting yourself up for future pain. That future pain is not what this blog post is about.

My preference is not to store time zone offset but instead store everything in UTC and perform any time zone switcharoos in the UI. But if you are storing local dates and times, I completely agree that you should keep track of the time zone. I worked for an east coast US company that bought a west coast US company, and both stored local dates and times in their SQL Server databases, making data consolidation a real challenge.

Comments closed

Working with Date Sequences in R

Steven Sanderson isn’t satisfied with a single date:

In the world of data analysis and manipulation, working with dates is a common and crucial task. Whether you’re analyzing financial data, tracking trends over time, or forecasting future events, understanding how to generate date sequences efficiently is essential. In this blog post, we’ll explore three powerful R packages—lubridate, timetk, and base R—that make working with dates a breeze. By the end of this guide, you’ll be equipped with the knowledge to generate date sequences effortlessly and efficiently in R.

Click through for several ways to generate date sequences, including weekly sequences.

Comments closed

Checking for Date Columns in R

Steven Sanderson is looking for a date:

As an R programmer, you may often encounter datasets where you need to determine whether a column contains date values. This task is crucial for data cleaning, manipulation, and analysis. In this blog post, we’ll explore various methods to check if a column is a date in R, with a focus on using the lubridate package and the ts_is_date_class() function from the healthyR.ts package.

Click through to see how, using lubridate and healthyR.

Comments closed

Finding the Week Number in R

Steven Sanderson checks the week:

When working with dates in R, you may need to extract the week number for any given date. This can be useful for doing time series analysis or visualizations by week.

In this post, I’ll demonstrate how to get the week number from dates in R using both base R and the lubridate package. I’ll provide simple examples so you can try it yourself.

Steven also makes a good point about ISO weeks (which are common in Europe) versus calendar weeks.

Comments closed

Translating Excel Date Values into R Dates

Steven Sanderson reads an Excel file:

Have you ever battled with Excel’s quirky date formats in your R projects? If so, you’re not alone! Those cryptic numbers can be a real headache, but fear not, fellow R warriors! Today, we’ll conquer this challenge and transform those numbers into beautiful, usable dates.

This is a common pain point in a lot of libraries and Steven shows how to solve it in R using a pair of functions.

Comments closed

Extracting the Month from a Date with R

Steven Sanderson asks what month it is:

Greetings fellow R enthusiasts! Today, we’re diving into a fundamental task: extracting the month from a date in R. Whether you’re new to R or a seasoned pro, understanding how to manipulate dates is essential. We’ll explore two popular methods: using base R and the powerful lubridate package. So, let’s roll up our sleeves and get started!

Read on for several examples across two solution spaces.

Comments closed

Finding the Earliest Date in R

Steven Sanderson puts on the archaeologist’s fedora and bullwhip:

Greetings, fellow data enthusiasts! Today, we embark on a quest to uncover the earliest date lurking within a column of dates using the power of R. Whether you’re a seasoned R programmer or a curious newcomer, fear not, for we shall navigate through this journey step by step, unraveling the mysteries of date manipulation along the way.

Imagine you have a dataset filled with dates, and you’re tasked with finding the earliest one among them. How would you tackle this challenge? Fear not, for R comes to our rescue with its arsenal of functions and packages.

Click through to see how, keeping those pernicious missing values in mind.

Comments closed

Calculating Date Differences in Month with R

Steven Sanderson has ways to track months:

Greetings fellow R enthusiasts! Today, let’s dive into the fascinating world of date calculations. Whether you’re a data scientist, analyst, or just someone who loves coding in R, understanding how to calculate the number of months between dates is a valuable skill. In this blog post, we’ll explore two approaches using both base R and the lubridate package, ensuring you have the tools to tackle any date-related challenge that comes your way.

Read on to see how to do this in base R as well as the lubridate package.

Comments closed

2024 Data Professional Salary Survey Results

Brent Ozar counts the cash:

This is the 8th year now that we’ve been running our annual Data Professional Salary Survey, and I was really curious to see what the results would hold this year. How would inflation and layoffs impact the database world? Download the raw data here and slice & dice it to see what’s important to you. Here’s what I found.

Read on for the results and Brent’s analysis.

Comments closed