A couple of weeks ago, at my work place, I wrote a metadata-driven data validation framework for Spark. After the initial euphoria of having created the framework in Scala/Spark and Python/Spark, I started reviewing the framework. During the review, I noted that the User Defined Functions (UDF) I had written were prone to throw an error in certain situations.
I then explored various options to make the UDFs fail-safe.
These are like any other code: you want it to be as robust to failure as you can get it (or at least robust enough at the margin).