Selecting a List of Columns in Spark

Unmesha SreeVeni shares how to get a specific set of columns by name from a DataFrame in Spark:

Create a List[Column] with column names.

scala> var selectExpr : List[Column] = List("Type","Item","Price")

<console>:25: error: not found: type Column var selectExpr : List[Column] = List("Type","Item","Price") ^

If you are getting the same error Please take a look into this page .

Read on for the demo.

Related Posts

Database-First or Kafka-First for Event Streaming

Gwen Shapiro takes us through a scenario where database-first writes for event streaming makes the most sense: Note that the DB does quite a lot for you: it enforces serializability, locks, your logical constraints, etc. If the DB is distributed (Vitesse, Cockroach, Spanner, Yugabyte), it does even more. If you were to go Kafka-first… well, […]

Read More

Handling Errors in Kafka Connect

Robin Moffatt shows us some techniques for handling errors in your Kafka topics: We’ve seen how setting errors.tolerance = all will enable Kafka Connect to just ignore bad messages. When it does, by default it won’t log the fact that messages are being dropped. If you do set errors.tolerance = all, make sure you’ve carefully thought through […]

Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.


March 2019
« Feb