You will get acquainted with the data engineering workflow, data product stages, the latest trends in data engineering, as well as key data tools and applications. Additionally, you will learn about successful data solutions and general architecture, gain knowledge in data governance and security issues, and see the difference between the major providers of cloud services.

You will be introduced to the world of Hadoop, the leading choice for storing and processing big data. In the upcoming lessons, you will explore why this platform is so popular today, its benefits for businesses, and how it handles high-speed processing and reliable data storage. Additionally, you will learn about its ecosystem and key features.

You will delve into Hive’s features and prepare for real-time big data projects. You will learn about user-defined functions, which are powerful tools for extending the Hive Query language. You will also learn about using transactions with ACID semantics and how they work in Hive. Moreover, you will explore Hive statistics and their importance. Lastly, you will gain insight into Hive optimization techniques that enhance its performance.

You will study the basics of the open-source distributed processing system for big data workloads, known as Spark. Alongside familiarizing yourself with its key components, architecture, and various applications, you will explore the extensive range of operations that Spark offers. This includes learning about the extract, transform, load (ETL) processes and understanding the three sets of APIs available in Spark. The following lessons will deepen your understanding of the Catalyst optimizer and introduce you to Project Tungsten for enhancing your ability to optimize Spark applications’ efficiency. Additionally, you will gain insights into Spark Streaming and how to use it for real-time analysis.

You will gain basic knowledge about Apache Kafka: its core capabilities, advantages, and drawbacks. You will explore the history of its transformation from a messaging system into a complex data product. What is more, you will get acquainted with the Kafka Connect framework and Kafka Streams library, delving into their respective roles within the Kafka architecture. In addition, you will learn how to optimize Kafka to meet your project’s goals and become familiar with monitoring processes and key metrics for analyzing Kafka’s performance.

You will build a clearer understanding of streaming data processing and how such data can be used for analytics purposes. You will gain further insights into operating Spark Streaming. You will see how Kafka Streams library is used in building applications and microservices that store data in Kafka clusters, both for input and output.

You will become familiar with flow-based programming and its main concepts, as well as the terms “dataflow” and “data pipeline”. Then, you will learn about Apache NiFi framework with special focus on its main features and Web UI. In addition, you will get acquainted with StreamSets Data Collector engine and figure out how it can be used for effective Data Movement.

You will learn about workflow orchestration and grasp the tools used for managing big data workflows.

You will discover NoSQL databases: their advantages, disadvantages, and peculiarities. You will also study different types of NoSQL databases: document, key-value, column, and graph. Then, you will examine the most popular NoSQL databases: MongoDB, HBase, and Cassandra. Moreover, you will find out how these databases work and how their data models and architecture are organized. Finally, you will get tips on which NoSQL database is the most suitable for different types of projects.

You will continue exploring engines which are commonly used in real-world big data projects. You will be introduced to Elasticsearch — a tool for searching.

You will dive into the challenges of handling data on a larger scale. You will learn how to process and analyze big data through the power of cloud computing — an approach which can revolutionize your organization.

You will gain hands-on experience by completing a project that simulates a real-life situation. You will be able to showcase the result in your portfolio.

You have the option to attend career services webinars to help you create a compelling resume and acquire effective job search techniques. You will get access to resources that will help you successfully land your first job as a big data engineer. Take advantage of one-on-one career advisory sessions to ask any questions and gain support.