Li Jin and Kevin Rasmussen cover the concepts of Flint, a time-series library built on Apache Spark:
Time series analysis has two components: time series manipulation and time series modeling.
Time series manipulation is the process of manipulating and transforming data into features for training a model. Time series manipulation is used for tasks like data cleaning and feature engineering. Typical functions in time series manipulation include:
- Joining: joining two time-series datasets, usually by the time
- Windowing: feature transformation based on a time window
- Resampling: changing the frequency of the data
- Filling in missing values or removing NA rows.
Time series modeling is the process of identifying patterns in time-series data and training models for prediction. It is a complex topic; it includes specific techniques such as ARIMA and autocorrelation, as well as all manner of general machine learning techniques (e.g., linear regression) applied to time series data.
Flint focuses on time series manipulation. In this blog post, we demonstrate Flint functionalities in time series manipulation and how it works with other libraries, e.g., Spark ML, for a simple time series modeling task.
Basho went all-in on a time-series product for Riak and it did not work out well for them. I’ll be curious to see if Flint has more staying power.