The Big Data in Real World team creates a Spark user-defined function in Scala:
In this post we are going to create a Spark UDF which converts temperature from Fahrenheit to Celsius.
Here is our data. We have day and temperature in Fahrenheit.
And, of course, it’s roughly the same in PySpark. Also, note that user-defined function performance will take a hit, and that answers are fairly consistent through the years, so save these for when you need them.