Getting Started with Data Engineering

_{written by Richard Taylor}

This is an excellent overview article that covers essential knowledge areas related to data engineering. Specifically the author digs into the following topics:

• A brief history of the big data sector and why it exists.

  • What is big data?
  • Terminology guide

• What a data engineer is and how and why the role came to be.

  • What is a data engineer?
  • Fundamentals and ongoing trends
  • Roles and responsibilities as a data engineer

• Big picture overview of the technologies data engineers have been using thus far.

  • The big data landscape
  • Hadoop, Hive, Zookeeper
  • Spark
  • The CAP Theorem
  • Non-relational (noSQL) databases
  • Relational databases

• What the future holds.

  • Continued HDFS reliance
  • Stream processing

View post

_{*Source: Medium}