Existing platform based Hadoop architectures make several implicit assumptions on how users interact with the platform such as developmental research versus production applications. While this was perfectly good in a research mode, as we move to a modern data application architecture we need to bring back modern application concepts to the Hadoop ecosystem. For example, existing Hadoop architectures tightly couple the user interface with the source of data. This is done for good reasons that apply in a data discovery research context, but cause significant issues in developing and maintaining a production application. We see this in some of the popular user interfaces such as Kibana, Banana, Grafana, etc. Each user interface is directly tied to a specific type of data lake and imposes schema choices on that data.
Read the whole thing. Also, “Securing the data plane” sounds like a terrible ’90s action film.