Duke University
3 weeks to complete at 10 hours a week
Gain the skills for building efficient and scalable data pipelines. Explore essential data engineering platforms (Hadoop, Spark, and Snowflake) as well as learn how to optimize and manage them.
Experience in working with Python, Git for version control, Docker for containerization and Kubernetes for deployment and scaling.
In this course, you will learn how to work with different data engineering platforms, such as Hadoop and Spark, and apply their concepts to real-world scenarios. You will explore the fundamentals of Hadoop to store and process big data, delve into Spark concepts, distributed computing, deferred execution, and Spark SQL, and gain hands-on experience with PySpark DataFrames, DataFrame methods, and deferred execution strategies. Additionally, you will explore the Snowflake platform, gaining insights into its architecture and key concepts, and learn to create tables, manage warehouses, and use the Snowflake Python Connector to interact with tables. Finally, you will practice the essential skills for seamlessly managing machine learning workflows using Databricks and MLFlow, including creating a Databricks workspace, configuring a cluster, loading datasets, and orchestrating the entire machine learning lifecycle.
Duke University has about 13,000 undergraduate and graduate students and a world-class faculty helping to expand the frontiers of knowledge. The university has a strong commitment to applying knowledge in service to society, both near its North Carolina campus and around the world.
This course is designed to help learners pursue or advance their career in data science or data engineering. By mastering the technologies and methodologies covered in this course, you will be well-equipped to tackle the challenges of today's data-driven world and take on roles such as Data Engineer, Data Architect, or Data Scientist.
Course fee
Start date
You will be redirected to the training provider to enquire directly. Workinitiatives is not the course or training provider. We aggregate and list courses only.