A Fine Slice Of SQL Server
Sebastian Sauer makes a list and checks it twice:
In this post, I want to show you how to work with list columns in R. List columns are a powerful feature of the
tidyversethat allow you to store multiple objects in a single column of a data frame. This can be useful when you have a list of objects that you want to keep together, such as a list of data frames or a list of models.
Click through for a demo.
Comments closedIvan Svetunkov puts on the forecasting hat:
I have been asked recently by a colleague of mine how to extract the variance from a model estimated using adam() function from the smooth package in R. The problem was that that person started reading the source code of the forecast.adam() and got lost between the lines (this happens to me as well sometimes). Well, there is an easier solution, and in this post I want to summarise several methods that I have implemented in the smooth package for forecasting functions. In this post I will focus on the adam() function, although all of them work for es() and msarima() as well, and some of them work for other functions (at least as for now, for smooth v4.1.0). Also, some of them are mentioned in the Cheat sheet for adam() function of my monograph (available online).
Read on to learn more. H/T R-Bloggers.
Comments closedSteven Sanderson does a bit of merging:
Combining data frames is a fundamental task in data analysis, especially when dealing with datasets that have different structures. In R, there are several ways to achieve this, using base R functions, the
dplyrpackage, and thedata.tablepackage. This guide will walk you through each method, providing examples and explanations suitable for beginner R programmers. This article will explore three primary methods in R: base R functions,dplyr, anddata.table. Each method has its advantages, and understanding them will enhance your data manipulation skills.
There are quite a few examples here, depending on whether you intend to join the datasets or perform a set operation such as union or intersect.
Comments closedStephen Turner reads some data:
In these slides I also learned about the nanoparquet package — a zero dependency package for reading and writing parquet files in R. Besides all the benefits noted above, parquet is much faster to read and write. And, as opposed to saving as .rds, parquet can easily be passed back and forth between R, Python, and other frameworks.
Let’s take a look at how reading and writing parquet files compares with CSV, either with base R or readr.
Stephen shows one of the best-case scenarios for Parquet: lots of data (100 million rows), relatively few columns, no long strings, etc. That leads to a massive improvement over using CSVs, even if you ignore the metadata and formatting benefits. I wouldn’t expect the benefits to be nearly as significant with wide text columns and very little value overlap, but that’s also pretty uncommon for the type of dataset we’re analyzing in R.
Comments closedSteven Sanderson needs a GROUP BY clause:
Combining rows with the same column values is a fundamental task in data analysis and manipulation, especially when handling large datasets. This guide is tailored for beginner R programmers looking to efficiently merge rows using Base R, the
dplyrpackage, and thedata.tablepackage. By the end of this guide, you will be able to seamlessly aggregate data in R, enhancing your data analysis capabilities.
Click through for several code examples.
Comments closedSteven Sanderson splits out some data:
As a beginner R programmer, you’ll often encounter situations where you need to divide your data into equal-sized groups. This process is crucial for various data analysis tasks, including cross-validation, creating balanced datasets, and performing group-wise operations. In this comprehensive guide, we’ll explore multiple methods to split data into equal-sized groups using different R packages and approaches.
Click through for methods and examples.
Comments closedSteven Sanderson tips the box plot over:
A boxplot, also known as a whisker plot, displays the distribution of data based on a five-number summary: minimum, first quartile, median, third quartile, and maximum. It highlights the data’s central tendency and variability, making it easier to identify outliers.
Read on for examples in base R and ggplot2.
Comments closedSteven Sanderson looks for oddities:
Outliers can significantly skew your data analysis results, leading to inaccurate conclusions. For R programmers, effectively identifying and removing outliers is crucial for maintaining data integrity. This guide will walk you through various methods to handle outliers in R, focusing on multiple columns, using a synthetic dataset for demonstration.
The techniques Steven uses are perfectly reasonable (though I like to use MAD from the median rather than standard deviations from the mean because MAD from the median doesn’t suffer from the sorts of endogeneity problems standard deviation does in a dynamic process). My primary warning would be to keep outliers in a dataset unless you know why you’re removing them. If you know the values were impossible or wrong—for example, a person who works 500 hours a week—that’s one thing. But sometimes, you get exceptional values out of an ordinary process, and those values are just as real as any other. I might have had a sequence in which I flipped a fair coin and it landed on heads 10 times in a row. It’s statistically very uncommon, but that doesn’t mean you can ignore it as a possibility or pretend it didn’t happen.
Comments closedSteven Sanderson is going for a walk (not the after-dinner kind):
Welcome to the world of ‘RandomWalker’, an innovative R package designed to simplify the creation of various types of random walks. Developed by myself and my co-author, Antti Rask, this package is in its experimental phase but promises to be a powerful tool for statisticians, data scientists, and financial analysts alike. With a focus on Tidyverse compatibility, ‘RandomWalker’ aims to integrate seamlessly into your data analysis workflows, offering both automatic and customizable random walk generation.
Read on to learn more about the package, including why you might want to use it and the functionality you can get out of it.
Comments closedSteven Sanderson builds a table:
Tables are an essential part of data analysis, serving as a powerful tool to summarize and interpret data. In R, the
table()function is a versatile tool for creating frequency and contingency tables. This guide will walk you through the basics and some advanced applications of thetable()function, helping you understand its usage with clear examples.
Click through for more information and several examples.
Comments closed