Chris Marshall shows how to perform anomaly detection using AWS Kinesis Analytics:
The RANDOM_CUT_FOREST function greatly simplifies the programming required for anomaly detection. However, understanding your data domain is paramount when performing data analytics. The RANDOM_CUT_FOREST function is a tool for data scientists, not a replacement for them. Knowing whether your data is logarithmic, circadian rhythmic, linear, etc. will provide the insights necessary to select the right parameters for RANDOM_CUT_FOREST. For more information about parameters, see the RANDOM_CUT_FOREST Function.
Fortunately, the default values work in a wide variety of cases. In this case, use the default values for all but the subSampleSize parameter. Typically, you would use a larger sample size to increase the pool of random samples used to calculate the anomaly score; for this post, use 12 samples so as to start evaluating the anomaly scores sooner.
Your SQL query outputs one record every ten seconds from the tumbling window so you’ll have enough evaluation values after two minutes to start calculating the anomaly score. You are also using a cutoff value where records are only output to “DESTINATION_SQL_STREAM” if the anomaly score from the function is greater than 2 using the WHERE clause. To help visualize the cutoff point, here are the data points from a few runs through the pipeline using the sample Python script:
This kind of scenario is pretty cool—you could also do things like detecting service outages in streams (fewer than X events in a window, where X is some very small number relative to your overall data) or changes in advertising campaigns.