Hadoop in the Enterprise — When and Why
Data & Analytics
Hadoop was the right tool for a specific era and a specific type of problem. Understanding when it is still the right choice — and when it is not — prevents over-engineering and avoids expensive migrations.
Key Patterns & Steps
- Hadoop fits when: data volume exceeds what a relational database can process cost-effectively, batch processing windows are acceptable, and data is semi-structured or unstructured
- HDFS + MapReduce for batch ETL pipelines; Hive for SQL-like queries over HDFS; Spark on YARN for faster in-memory processing
- Do not choose Hadoop for real-time analytics — Kafka + Spark Streaming or Azure Event Hubs is a better fit
- For new greenfield data platforms in Azure, evaluate Azure Data Lake Storage + Databricks before committing to Hadoop
- Operational burden is high: HDFS replication, NameNode HA, YARN resource management, and Kerberos security all require specialist knowledge
- PoC evaluation: run the same batch job on Hadoop vs Azure Data Lake + Databricks and compare cost, speed, and operational overhead
Lessons Learned
Most of the Hadoop clusters I saw in enterprise were underutilised and over-resourced. The data teams that chose cloud-native equivalents in 2018 avoided a Hadoop migration project in 2022.
Technologies
← Back to Architecture Notes