Catalyst Optimizer: Predicate Pushdown in Apache Spark
Vishal Jadhav if in your code/query if you are filterring the data at the end, Catalyst optimizer (in prediction pushdown) will apply filtering on input or source and then do the other…
Data Engineering
PySpark cache() Memory Management – What Happens When Data Exceeds Memory?
both cache() and persist() store data in memory to speed up the retrieval of intermediate data used for computation. However, persist() is more flexible and allows users to specify storage…
SCD Types Explained with Examples
Slowly Changing Dimensions (SCD) are used in data warehousing to manage historical changes in dimension tables. There are several types of SCDs, each handling data changes differently. Types of SCDs…
PySpark with Databricks – Complete Data Engineering Guide
PySpark Interview Questions (lec 6) How to read files in spark? You are going to see how to read different file formats in pyspark. first, you need to create the…
How Would You Handle Skewed Data in PySpark? (Part 2 – Alternative Approaches)
📌 Updated Guide Available: Check out our comprehensive updated guide on this topic: How to Handle Skewed Data in PySpark – Complete Guide Handling skewed data in PySpark is crucial…
zepto global data engineer interview questions
Programming/Algorithm Question:You are acting as a bank. You are given a list of customers’ transactions.The transactions can be either a deposit (positive value) or awithdrawal(negative value). The initial balance in…
LeetCode Data Engineer Interview Questions
Ace Your Data Engineering Interview: Essential LeetCode SQL, Python, and ETL Questions
Data Engineer Interview Guide – Questions, Tips & Preparation
CAP theorem Lambda vs kapp architecture Star vs snowflake schema Data warehouse , data lake, delta lake, dataware house Scd types SQL How would you write a query to calculate…
Surrogate key
Surrogate keys are artificially generated primary keys, typically integers, used to uniquely identify records. In data warehouse you may have same record with multiple entries(this records columns value may change…
Pyspark Zero to Expert Tutorial
Pyspak complete tutorial from scratch Shuffle: when data persists and is shared with other executors.whenever shuffle happens job is get divided into stages Driver Executor in the following diagram, the…