There are many “Known Unknowns” in the data and analytics ecosystem, but there is definitely only one “Known Known” — your stack is ever-evolving.
What I mean by this is that there are always shiny new tools, platforms, or frameworks your team can use. The lakehouse, as an example, is the most recent evolution of the modern data stack. A whole suite of tools will pop up to help your team manage your lakehouse. The issue with these new tools, though, is that they inherently present brand new challenges that you subsequently need another set of tools to solve. It’s a recurring nightmare!
Why do I highlight this issue? Because it has become increasingly clear to me that being data-driven and using technology are often mutually exclusive.
Quick side-story: I recently read The Tycoons, by Charles R. Morris and was excited to learn something about Andrew Carnegie. He had a very large team of bookkeepers and accountants create monthly reports for him on sales and operations. He would anxiously await these reports, and viciously analyze costs to identify inefficiencies. Andrew Carnegie seems to have been the father of the modern data analyst!
If Carnegie was able to become one of the most successful businessmen of all time using pen and paper instead of a Tableau dashboard, then we need to rethink what the actual challenges of data teams are.
This issue is crystallized in a recent piece from MIT, Why Culture is the Greatest Barrier to Data Sucess:
“The advent of big-data solutions and a next generation of data management capabilities — Hadoop, data lakes, DataOps, and modern data architectures — have been helpful but have not assured successful business adoption or outcomes. Technology does not appear to be a barrier or the problem. Only 9.1% of executives pointed to technology as the principal challenge to becoming data-driven.”
Enterprises seem to understand that technology does not solve all of their problems.
So, here’s my bold prediction for 2021: companies will realize that their data team challenges are human problems.
Data teams are inherently diverse: scientists, analysts, engineers. They all have different processes, different goals, and different DNA. Data leaders need to rethink the way data teams collaborate. They don’t necessarily need to implement a new technology framework, they need to implement a new framework for how their teams work together.
Predictions for 2021 from around the ecosystem:
The Year in Data and Analytics by James Densmore
Unpacking the Data Hype by Sarah Nöckel
2021 Trends in Data Management from G2
DataOps is more than DevOps for Data from The New Stack
8 Predictions for 2021 from the CTO of Amazon
That’s all folks! Enjoy the time off and see you next year!
One Bold Data Team Prediction for 2021
Great post, thank you Andrew. Could not agree more! I think Data Literacy training will be one small component, tools like Atlan are really neat, and then finally I think the importance of Leadership training can not be understated. I think we see that pure tech teams can get away with substandard leadership because the teams still produce "something" at the end of the day, but the bar is higher for data projects with the results a little more bar-belled (way good or way bad).