A Fine Slice Of SQL Server
I would like to illustrate a way which omitted variables interfere in logistic regression inference (or coefficient estimation). These effects are different than what is seen in linear regression, and possibly different than some expectations or intuitions.
This is an interesting article and there’s a really good comment helping to explain this effect in epidemiology.
Comments closedThere is a tool (or you can consider it as an editor) in Fabric for data scientists. As a data scientist, you must work with the data, clean it, group it, aggerate it, and do other data preparation work. This might be needed to understand the data or be part of the process you do to prepare the data and load it into a table for further analysis. Data Wrangler is a tool that gives you such ability. You can use it to transform data and prepare and even generate Python code to make this process part of a bigger data analytics project.
Data Wrangler has a simple-to-use graphical user interface that makes the job of a data scientist easier.
Read on for a video as well as a demo in written format.
Comments closedChristian Lorentzen has released a new package:
Version 1.0.0 of the new Python package for model-diagnostics was just released on PyPI. If you use (machine learning or statistical or other) models to predict a mean, median, quantile or expectile, this library offers tools to assess the calibration of your models and to compare and decompose predictive model performance scores.
This looks like a really useful package, so check it out.
Comments closedShivansh Mundra, et al, report on some research:
With the rise of AI-generated synthetic media and text-to-image generated media, fake profiles have grown more sophisticated. And we’ve found that most members are generally unable to visually distinguish real from synthetically-generated faces, and future iterations of synthetic media are likely to contain fewer obvious artifacts, which might show up as slightly distorted facial features. To protect members from inauthentic interactions online, it is important that the forensic community develop reliable techniques to distinguish real from synthetic faces that can operate on large networks with hundreds of millions of daily users, like LinkedIn.
There are some interesting findings here.
Comments closedMichael Mayer runs an analysis:
This is the next article in our series “Lost in Translation between R and Python”. The aim of this series is to provide high-quality R and Python code to achieve some non-trivial tasks. If you are to learn R, check out the R tab below. Similarly, if you are to learn Python, the Python tab will be your friend.
This post is heavily based on the new {shapviz} vignette.
I appreciate the effort to include both R and Python code in this analysis, and recommend you peruse both sets of code listings.
Comments closedJi Shen shows off how to perform discrete time series in SAS:
The HMM procedure in SAS Viya supports hidden Markov models (HMMs) and other models embedded with HMM. PROC HMM supports finite HMM, Poisson HMM, Gaussian HMM, Gaussian mixture HMM, the regime-switching regression model, and the regime-switching autoregression model. This post introduces Poisson HMM, the latest addition to PROC HMM in the SAS Viya 2023.03 release.
Count time series is ill-suited for most traditional time series analysis techniques, which assume that the time series values are continuously distributed. This can present unique challenges for organizations that need to model and forecast them. As a popular discrete probability distribution to handle the count time series, the Poisson distribution or the mixed Poisson distribution might not always be suitable. This is because both assume that the events occur independently of each other and at a constant rate. In time series data, however, the occurrence of an event at one point in time might be related to the occurrence of an event at another point in time, and the rates at which events occur might vary over time.
HMM is a valuable tool that can handle overdispersion and serial dependence in the data. This makes it an effective solution for modeling and forecasting count time series. We will explain how the Poisson HMM can handle count time series by modeling different states by using distinct Poisson distributions while considering the probability of transitioning between them.
Read on for an overview of Hidden Markov Models (in general and the Poisson variation in particular) and some of the challenges you can run into when performing this test.
Comments closedI recently came across an insightful research paper titled “Moving Fast With Broken Data” by Shreya Shankar, Labib Fawaz, Karl Gyllstrom, and Aditya G. Parameswaran from UC Berkeley and Meta. The paper addresses the significant issue of data corruption in machine learning (ML) pipelines, which often leads to decreased model accuracy. The authors present an automatic data validation system implemented at Meta that aims to solve this problem.
Sounds like I have some beach reading.
Ed. Note: He’s kidding, right?
Ed. 2 Note: About going to the beach maybe.
Ed. & Ed. 2 Note: HAHAHAHAHAH.
Yeah, I hired Statler and Waldorf as my editors. Worst Best decision of my life.
Steven Sanderson wraps up a series on shiny and tinyAML. Part 3 extends options for regression:
As data science continues to be a sought-after field, creating a reliable and accurate model is essential. While there are various machine learning algorithms available, the process of selecting the correct algorithm can be complex. The
{tidyAML}package, part of thetidymodelssuite, offers an easy-to-use, consistent interface for building machine learning models. In this post, we will explore a Shiny application that utilizestidyAMLto build a machine learning model.Today I have updated the
tidyAMLshiny app to include the ability to set the parameter of thefast_regression()function.parsnip_fnsand this is things likelinear_reg.
And part 4 includes classification:
This is a Shiny app for building models using the
{tidyAML}which is based on thetidymodelspackage in R. The app allows you to upload your own data or choose from one of two built-in datasets (mtcars or iris) and select the type of model you want to build (regression or classification).Let’s take a closer look at the code.
This was an interesting series, for sure.
Comments closedAyodeji Ogunlami mixes approaches:
In developing this hybrid system, sets of rules are required as well as a machine learning model. I would be making use of a vehicle insurance dataset from Kaggle in this demonstration.
The dataset can be downloaded from this link: https://www.kaggle.com/datasets/shivamb/vehicle-claim-fraud-detection
The ML model would be built using a random forest classifier on Azure Databricks using Pyspark.
This seems to be the most sensible approach, especially given how rare actual fraud incidents are and what that imbalance does to classification algorithms.
Comments closedJohn Mount is not a fan of hyperparamters:
In my opinion one can see this scam of hiding some debt in with an asset spreading.
Earliest modeling systems, such as linear regression, had no hyper-parameters. An under specified algorithm was not considered a fully specified method.
Click through for John’s thoughts on the matter. I’m sympathetic to this argument and want to bring in an extra point John didn’t make. With hyperparameter tuning, you also introduce the risk of spurious correlation between the label and input features. This is particularly relevant if changing the seed or making hyperparameter tweaks results in a major change in model effectiveness.
Comments closed