Generated by All in One SEO v5.0.1.1, this is an llms.txt file, used by LLMs to index the site.

# DataEngineerBlog.com

Design Engineer Build

## Sitemaps

- [XML Sitemap](https://dataengineerblog.com/sitemap.xml): Contains all public & indexable URLs for this website.

## Posts

- [Essential debugging with a Local Spark Cluster on Windows](https://dataengineerblog.com/essential-debugging-with-a-local-spark-cluster-on-windows/) - Learn how to set up a local Spark master/worker cluster on Windows to debug production issues without Docker. hi
- [Top 10 Data Engineering Skills Every Company Needs in 2026](https://dataengineerblog.com/top-10-data-engineering-skills-every-company-needs-in-2026/) - Most in demand skills for data engineers in 2026
- [What Is Data Engineering? A Complete Guide for 2026?](https://dataengineerblog.com/what-is-data-engineering-a-complete-guide-for-2026/) - Data Engineering has become one of the most essential disciplines in today’s data-driven world. Whether you’re a student, a beginner exploring tech, techie or someone interested in how companies use data for decision-making, understanding data engineering is a major advantage. This guide explains the fundamentals of data engineering in a clear, professional, and approachable way.
- [PySpark interview question [2026]](https://dataengineerblog.com/pyspark-interview-questions/) - Top 50 recently asked Pyspark Interview Questions Big Data
- [How to find duplicates in a table](https://dataengineerblog.com/how-to-find-duplicates-in-a-table/) - Identify duplicates criteria The first step is to define your criteria for a duplicate row. Do you need a combination of two columns to be unique together, or are you simply searching for duplicates in a single column? In this example, we are searching for duplicates across two columns in our Users table: username and
- [Difference Between WHERE and HAVING Clause in sql](https://dataengineerblog.com/difference-between-where-and-having-clause-in-sql/) - HAVING Clause is used to filter the records from the groups based on the given condition. HAVING clause alwaysed used after
- [Time travel - BigQuery](https://dataengineerblog.com/time-travel-bigquery/) - what is time travel in bigquery? Using big-query's time travel concept we can access any modified(updated or delete) data. we can access this data till last 7 days. This concept in Bigquery help for creating backup and disaster recovery. You can access data from any point within the time travel window, which covers the past
- [LeetCode Data Engineer Interview Questions](https://dataengineerblog.com/leetcode-data-engineer-interview-question/) - Ace Your Data Engineering Interview: Essential LeetCode SQL, Python, and ETL Questions
- [How would you handle skewed data in PySpark?](https://dataengineerblog.com/how-would-you-handle-skewed-data-in-pyspark/) - Handling skewed data in PySpark is crucial for optimizing performance and ensuring efficient processing. Skewed data occurs when some partitions have significantly more data than others, leading to uneven workload distribution and potential bottlenecks. Here are several strategies to handle skewed data in PySpark: What is Skewed Data? Skewed data occurs when certain keys or
- [sql to pyspark query converstion](https://dataengineerblog.com/sql-to-pyspark-query-converstion/) - In this article, we are converting the SQL queries to Pyspark code String operation in Pyspark Date/Time Operators: Window functions
- [SCD Types Explained with Examples](https://dataengineerblog.com/scd-types-explained-with-examples/) - Slowly Changing Dimensions (SCD) are used in data warehousing to manage historical changes in dimension tables. There are several types of SCDs, each handling data changes differently. Types of SCDs with Examples and Implementations SCD Type 0 (Fixed Dimension) No changes are allowed once the data is inserted or No changes are allowed after the
- [How to handle skewed data in PySpark?](https://dataengineerblog.com/how-to-handle-skewed-data-in-pyspark/) - 𝗛𝗼𝘄 𝗗𝗼 𝗬𝗼𝘂 𝗛𝗮𝗻𝗱𝗹𝗲 𝗦𝗸𝗲𝘄𝗲𝗱 𝗗𝗮𝘁𝗮 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸?This is a critical PySpark interview question! Here’s the breakdown: ✅ 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗦𝗸𝗲𝘄𝗲𝗱 𝗗𝗮𝘁𝗮? A skewed partition in Spark occurs when the data is distributed unevenly across the available partitions, causing some partitions to be significantly larger than others. This imbalance creates performance bottlenecks because the tasks processing the large
- [How Catalyst Optimizer Works in PySpark](https://dataengineerblog.com/catalyst-optimizer-in-pyspark/) - 𝗪𝗵𝗮𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗖𝗮𝘁𝗮𝗹𝘆𝘀𝘁 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗿, 𝗮𝗻𝗱 𝗛𝗼𝘄 𝗗𝗼𝗲𝘀 𝗜𝘁 𝗪𝗼𝗿𝗸? This is a must-know PySpark interview question! Here’s the breakdown: 𝗪𝗵𝗮𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗖𝗮𝘁𝗮𝗹𝘆𝘀𝘁 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗿? Catalyst is Spark SQL’s 𝗾𝘂𝗲𝗿𝘆 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸. It transforms your DataFrame/Dataset operations into an 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 𝗽𝗹𝗮𝗻. 𝗛𝗼𝘄 𝗗𝗼𝗲𝘀 𝗜𝘁 𝗪𝗼𝗿𝗸? 𝗟𝗼𝗴𝗶𝗰𝗮𝗹 𝗣𝗹𝗮𝗻: Converts your code into an abstract logical
- [GCP BigQuery](https://dataengineerblog.com/gcp-bigquery/) - Following are the most important topics in bigquery. This is also important topics in a perspective of GCP Profession Data Engineer exam. BigQuery basic concepts wildcard tables _table_suffix External Table vs Internal Table partition and clustering:-https://youtu.be/wapi0aR4BZE?si=iwkrC9fUY2ConUgB shad tables Bigquery optimization Bigquery history-based optimization Anonymous Dataset - what is the Anonymous dataset in big query Time
- [Data Modeling - How to design it?](https://dataengineerblog.com/data-modeling-how-to-design-it/) - Data modeling is a structured approach to designing and organizing data for a database or system. Here are the key steps:1. Identify Business RequirementsUnderstand the purpose of the data model.Gather requirements from stakeholders.Identify key entities, relationships, and data needs.2. Define Entities and AttributesIdentify the main objects (entities) in the system.Define attributes (fields) for each entity.Determine
- [What is a federated query in BigQuery?](https://dataengineerblog.com/what-is-a-federated-query-in-bigquery/) - Federated queries are used to query the data on external sources like Cloud Storage, BigTable, CloudSql, Spreadsheet in google Drive. Federated queries or have queries that access data that isn't directly stored in bigquery, but instead its store in places like Cloud storage (files in Parquet, ORC format files) BigTable and CloudSQL Spreadsheets in Google
- [partition and clustering - BigQuery](https://dataengineerblog.com/partition-and-clustering-bigquery/) - Limitations maximum 8000 partitions possible on a single table and upto 4 columns can be used for clustering Partition can be done on Date, DATETIME, TIMESTAMP OR INTERGER RANGE COLUMN. when you use clustering and partitioning together, the data can be partitioned by a date, date time or timestamp column, and then clustered on a
- [Data Engineer interview.](https://dataengineerblog.com/data-engineer-interview/) - CAP theorem Lambda vs kapp architecture Star vs snowflake schema Data warehouse , data lake, delta lake, dataware house Scd types SQLHow would you write a query to calculate a cumulative sum or running total within a specific partition in SQL?How do window functions differ from aggregate functions, and when would you use them?- How
- [SQL](https://dataengineerblog.com/sql-interview-question-100-asked-questions/) - sql Interview coding/Query questions Difference between WHERE and HAVING clause in SQL LIMIT and OFFSET - The most Important concept union and union all Distinct - to get unique/diffent rows. it eliminates duplicate and show only unique records Normalization and denormalization Joins anti join self join inner outer left,right,full join Aggregates function – count(), min(),
- [What Are Accumulators, and How Do They Work?](https://dataengineerblog.com/what-are-accumulators-and-how-do-they-work/) - What Are Accumulators, and How Do They Work? This is a most frequently asked PySpark interview question! Here’s the breakdown: What Are Accumulators? Accumulators are shared variables used to aggregate values across tasks in a distributed system. They are primarily used for counters or sums (e.g., counting errors or tracking metrics). How Do They Work? Only the driver program can read the accumulator’s value. Worker
- [Given a code - How Catalyst Optimizer Works in This Code?](https://dataengineerblog.com/how-catalyst-optimizer-works-in-spark/) - You have the following code. Explain how the catalyst optimizer works in the code? Explain in detail from pyspark.sql import SparkSession # Initialize Spark session spark = SparkSession.builder \ .appName("SalesCustomerJoin") \ .getOrCreate() # Read sales data into df1 df1 = spark.read.csv("path/to/sales_data.csv", header=True, inferSchema=True) # Read customer data into df2 df2 = spark.read.csv("path/to/customer_data.csv", header=True, inferSchema=True) #
- [Catalist optimizer : Prediction pushdown](https://dataengineerblog.com/prediction-pushdown/) - if in your code/query if you are filterring the data at the end, Catalyst optimizer (in prediction pushdown) will apply filtering on input or source and then do the other operations. Predicate pushdown means that the optimizer tries to move filter operations as close to the data source as possible, even if the filter is
- [What happens when you enable cache() in PySpark and the dataset exceeds the available memory? How does Spark handle this situation, and what potential issues might arise?](https://dataengineerblog.com/what-happens-when-you-enable-cache-in-pyspark-and-the-dataset-exceeds-the-available-memory-how-does-spark-handle-this-situation-and-what-potential-issues-might-arise/) - both cache() and persist() store data in memory to speed up the retrieval of intermediate data used for computation. However, persist() is more flexible and allows users to specify storage levels, including disk storage. when you enable cache() in PySpark and the dataset exceeds the available memory? How does Spark handle this situation, and what potential issues
- [Pypark - Databriks + spark Rajas data engineering](https://dataengineerblog.com/pypark-databriks-spark-rajas-data-engineering/) - Rdd, dataframe, dataset on-heap vs off-heap memory (lecture 4) Data bricks cluster (lecture 5) In DataBricks first, we need to launch the cluster and attach it to the notebook to execute the code cluster: cluster is computing infrastructure in the DataBricks environment. It is a set of computation resources and configurations. Cluster Types: All Purpose/Interactive
- [Pyspark Zero to Expert Tutorial](https://dataengineerblog.com/pyspark-zero-to-expert-tutorial/) - Pyspak complete tutorial from scratch Shuffle: when data persists and is shared with other executors.whenever shuffle happens job is get divided into stages Driver Heart of the Spark Application Manages the information and state of executors Analyzes, distributes, and Schedules the work on executor Executor Execute the code Report the status of execution to the
- [Surrogate key](https://dataengineerblog.com/surrogate-key/) - Surrogate keys are artificially generated primary keys, typically integers, used to uniquely identify records. In data warehouse you may have same record with multiple entries(this records columns value may change like country column is updated ) to identify this records you may need to use surrogate key Natural Key (also called Business Key or Composite
- [zepto global data engineer interview questions](https://dataengineerblog.com/zepto-global-data-engineer-interview-questions/) - Programming/Algorithm Question:You are acting as a bank. You are given a list of customers' transactions.The transactions can be either a deposit (positive value) or awithdrawal(negative value). The initial balance in the bank is 0.The bank can choose any customer to start serving from the list.Once a customer is picked, the bank will continue serving the
- [PWC Data Engineer Interview questions](https://dataengineerblog.com/pwc-data-engineer-interview-questions-bangalore-2024/) - So, overall the interview process had 3 rounds, lasting 1 hour each. 📍 𝗥𝗼𝘂𝗻𝗱 𝟭: This round began with a detailed discussion about my current projects, like: ->Optimisation techniques used in projects → Issues faced in the project → Hadoop questions And then coding questions on SQL and Python. 📍 𝗥𝗼𝘂𝗻𝗱 𝟮: This round focused
- [SQL Aggregates function - count(), min(), max(), avg(), sum()](https://dataengineerblog.com/sql-aggregates-function-count-min-max-avg-sum/) - SQL also supports the use of aggregate expressions (or functions) that allow you to summarize information about a group of rows of data. SELECT AGG_FUNC(column_or_expression) AS aggregate_description, … FROM mytable WHERE constraint_expression; Without a specified grouping, each aggregate function is going to run on the whole set of result rows and return a single value.
- [SQL Joins (anti, self, natural, lnner, left, right, full join)](https://dataengineerblog.com/sql-joins-antiself-natural-lnner-left-right-full-join/) - A JOIN clause is used to query and access data from multiple tables by using logical relationships between them. Join syntax SELECT column, another_column, … FROM mytable INNER/LEFT/RIGHT/FULL JOIN another_table ON mytable.id = another_table.matching_id WHERE condition(s) ORDER BY column, … ASC/DESC LIMIT num_limit OFFSET num_offset; The INNER JOIN is a process that matches rows from the first
- [How to delete duplicates from a table](https://dataengineerblog.com/how-to-delete-duplicates-from-a-table/) - To delete duplicates from a table, you'll need to identify the duplicate rows first and then delete them using SQL. The approach varies slightly depending on whether you're working with SQL, a specific database management system (e.g., MySQL, PostgreSQL, SQL Server), or a tool like Excel or Python. 1. SQL - Deleting Duplicates The general
- [SQL LIMIT & OFFSET - SELECT ROW](https://dataengineerblog.com/sql-limit-offset-select-row/) - SQL LIMIT and OFFSET LIMIT: Limits the number of rows returned by a query OFFSET: Specifies how many rows to skip before returning the results Here's an example of using LIMIT and OFFSET in SQL SELECT * FROM table LIMIT 5 OFFSET [Number of rows to skip]; In the above SQL query, it will skip the first x number of rows mentioned in OFFSET and then
- [GCP Bucket - 100% All topics cover in single post](https://dataengineerblog.com/gcp-bucket-100-all-topics-cover-in-single-post/) - following are the most important topics for the gcp bucket which is really helpful in real life as well as gcp exam perspective Bucket object Retention Policy Versionning Lifecycle Manegment Signed url ACL access - object level access Storage class Storage classes Standar Nearline Coldline Archive
- [Understanding and Mitigating Hotspotting in Google Cloud Spanner](https://dataengineerblog.com/understanding-mitigating-hotspotting-google-cloud-spanner/) - Learn how to prevent and mitigate hotspotting in Google Cloud Spanner to maintain optimal performance and scalability. This guide covers causes, signs, and strategies including schema design, query optimization, sharding, and load balancing. Get insights from real-world case studies and best practices for effective data distribution and resource utilization.
- [Short and wide tables in Google Cloud Bigtable.](https://dataengineerblog.com/short-and-wide-tables-in-google-cloud-bigtable/) - Certainly! Let’s explore the difference between short and wide tables in Google Cloud Bigtable. Short Tables: In a short table, each row typically contains a small number of columns. These tables are well-suited for scenarios where you have a single key (the row key) and need to store data associated with that key. Short tables
- [Amazon data engineer interview questions - 2024](https://dataengineerblog.com/amazon-data-engineer-interview-questions-2024/) - Technical phone screen Assess your technical abilities Coding - Data engineer coding round question Return the numbers with the maximum count from a list Flatten all objects to a single key value dictionary from a nested json object Find the sum of any two numbers in an array that could be equal to x SQL
- [BigTable - Tall tables vs narro Tables](https://dataengineerblog.com/gcp-bigtable-tall-tables-vs-narro-tables/) - Bigtable is a NoSQL database service provided by Google Cloud Platform (GCP), designed to handle massive-scale, high-throughput workloads. While it's commonly associated with storing and processing large amounts of data, including wide and sparse tables, Bigtable can also be used to store tall, narrow tables efficiently. In the context of databases, "tall" and "narrow" refer
- [Ultimate Guide to BigQuery Slots: Boosting Performance and Managing Costs Efficiently](https://dataengineerblog.com/bigquery-slots/) - BigQuery slots are a fundamental aspect of Google Cloud's BigQuery service, which is a fully managed, serverless, highly scalable, and cost-effective multi-cloud data warehouse. Slots represent the computational power required to execute SQL queries and perform other data operations within BigQuery. Here's an overview of BigQuery slots: What are BigQuery Slots? Definition: A slot is
- [GCP Dataflow - Apache beam](https://dataengineerblog.com/gcp-dataflow-apache-beam/) - Most important concept Basics ParDo PCollections Ptransform pipeline Windowing function Watermark Triggers Tumbling/Fixed window - https://youtu.be/0sUJA3tDFiY?si=fC_p2SpFoDZbhKyk Hopping/sliding window - https://youtu.be/0sUJA3tDFiY?si=fC_p2SpFoDZbhKyk Session window - https://youtu.be/vqiHXdrn3sU Global Window - https://youtu.be/vqiHXdrn3sU (Batch Data) In the context of data stream processing and windowing, these terms—tumbling windows, hopping windows, session windows, and global windows—describe different ways to group and analyze
- [Comprehensive Guide to Windowing Functions in Data Stream Processing: Tumbling, Hopping, Sliding, Session, and Global Windows Explained](https://dataengineerblog.com/windowing-in-stream-processing/) - windowing in stream processing
- [View vs materialized view - BigQuery](https://dataengineerblog.com/view-vs-materialized-view/) - difference between view and Materialized view
- [BigQuery - Anonymous dataset](https://dataengineerblog.com/bigquery-anonymous-dataset/) - BigQuery writes all query results to a table. The table is either explicitly identified by the user (a destination table), or it is a temporary, cached results table. Temporary, cached results tables are maintained per-user, per-project. There are no storage costs for temporary tables, but if you write query results to a permanent table, you
- [problem with to many partition in database or storage systems](https://dataengineerblog.com/problem-with-to-many-partition-in-database-or-storage-systems/) - Having too many partitions in a system, especially in a data storage context like a database or a distributed file system, can lead to several issues.Here’s a breakdown of the problems that can arise and potential solutions to manage and mitigate these problems: ### Problems with Too Many Partitions 1. **Increased Overhead**: - **Metadata Management**:
- [GCP IAM (identity access management)](https://dataengineerblog.com/gcp-iam-identity-access-management/) - who can do what on which resource who (identity) - its user (userid or email-id) what(action) - create, update, delete which(resources)- cloud storage, bigquery. in simple word user/employee(who) having given a permission of creating storage bucket, deleting storage bucket. A employee given Creator role i.e. Storage Object Creator (roles/storage.objectCreator) then he can create objects in
- [[CloudAcademy] Google cloud engineer certification - original Exam question, Preparation](https://dataengineerblog.com/cloudacademy-google-cloud-engineer-certification-original-exam-question-preparation/) - Google cloud sdk - command and usages we are going to see the different google cloud sdk commands and usage installing the cloud SDK https://cloud.google.com/sdk/docs/install-sdk Google Cloud Storage Google Cloud Storage Important topics Storage Classes Standard: Best for short-term storage and frequently accessed data Nearline: Best for backups and data accessed less than once a
- [[Coursera]Google Cloud Associate certification Preparation - Crack it now](https://dataengineerblog.com/google-cloud-associate-certification-preparation-crack-it-now/) - This article is more than sufficient preparation and crack the google cloud associate certification exam. this post covers topics, exam question and dumps. Difference between IAAS vs PASS Google Cloud Network Location > region >> zone Each locations is divided into several different region and zones. In the above example Location is London, having Region
- [walmart data engineer interview questions](https://dataengineerblog.com/walmart-data-engineer-interview-questions/) - Walmart data engineer interview questions and answers
- [World of Prompt Engineering: Jobs, Salary, and Courses 2024](https://dataengineerblog.com/prompt-engineering-jobs-salary-and-courses/) - All about prompt engineering job salary and courses.
- [Big data engineer interview question and answer - 2024](https://dataengineerblog.com/data-engineer-interview-question-and-answer/) - Data engineering interview question and answer

## Pages

- [Contact us](https://dataengineerblog.com/contact-us/) - Let's Connect Hi! Thank you for visiting DataEngineerBlog.com. Whether you are a fellow engineer stuck on a Spark exception, a recruiter looking for a senior resource, or a business interested in a partnership, I’d love to hear from you. How to Reach Me The best way to contact me is via email. I aim to
- [Disclaimer](https://dataengineerblog.com/disclaimer/) - Last Updated: December 7, 2025 1. General Disclaimer The information provided on DataEngineerBlog.com ("the Site") is for general informational and educational purposes only. All information on the Site is provided in good faith, however, we make no representation or warranty of any kind, express or implied, regarding the accuracy, adequacy, validity, reliability, availability, or completeness
- [Privacy Policy](https://dataengineerblog.com/privacy-policy/) - Last Updated: December 7, 2025 1. Who We Are Our website address is: https://dataengineerblog.com. We are committed to protecting your personal information and your right to privacy. If you have any questions or concerns about this policy or our practices with regards to your personal information, please contact us at contact@dataengineerblog.com. 2. What Personal Data
- [About Us](https://dataengineerblog.com/about-us/) - About Me – The Engineer Behind the Code Real-World Data Engineering, Beyond the Documentation Hi, I’m Vishal. Welcome to DataEngineerBlog.com. I am a Senior Data Engineer with over 8+ years of experience in designing, building, and optimizing scalable data infrastructures. I created this site with one goal: to bridge the gap between theory and production.
- [System Design for Data Engineers](https://dataengineerblog.com/system-design-for-data-engineers/) - The system design round for data engineers is slightly different from traditional system design interviews. In this article, you will explore various interview questions along with their answers. Design storage system How to choose file format or when to choose parque data format vs avro file format Different types of system Batch processing Realtime/Streaming (most
- [Data Engineer Interview Questions](https://dataengineerblog.com/data-engineer-interview-questions/) - Zeta global My interview experiences and Interview questions. Python interview questions asked in the interview SQL questions asked in the interview PySpark question asked in interview Data Engineering concepts ETL vs ELT Data warehouse, Data Mart, Data lake, data lakehouse, delta lake, Data Mesh Data Modeling and Concepts - Best Resource Star schema vs Snowflake
- [Python](https://dataengineerblog.com/python/) - Try except interview question can have multiple except blocks in a try statement in Python? Yes List vs tuple - which situation you use which one Dictionary Can we use list as key in dictionary Can we use tuples in the dictionary key Can we use list inside tuple as a dictionary key

## Categories

- [Google Cloud](https://dataengineerblog.com/category/cloud/google-cloud/)
- [SQL](https://dataengineerblog.com/category/sql-for-data-engineers/)
- [Data Engineering](https://dataengineerblog.com/category/data-engineer-interview-questions/)
- [Data Engineer Interview Questions](https://dataengineerblog.com/category/data-engineer-interview-questions/data-engineer-interview-question/)
- [Prompt engineering](https://dataengineerblog.com/category/prompt-engineering/) - prompt engineering job, salary and courses
- [BigTable](https://dataengineerblog.com/category/bigtable/)
- [Cloud Spanner](https://dataengineerblog.com/category/cloud/google-cloud/cloud-spanner/)

## Tags

- [bigquery](https://dataengineerblog.com/tag/bigquery/)
- [bigtable](https://dataengineerblog.com/tag/bigtable/)
- [short table](https://dataengineerblog.com/tag/short-table/)
- [wide table](https://dataengineerblog.com/tag/wide-table/)
- [hotspot](https://dataengineerblog.com/tag/hotspot/)
- [spanner](https://dataengineerblog.com/tag/spanner/)
- [sql](https://dataengineerblog.com/tag/sql/)
- [LIMIT](https://dataengineerblog.com/tag/limit/)
- [OFFSET](https://dataengineerblog.com/tag/offset/)
- [spark](https://dataengineerblog.com/tag/spark/)
- [catalyst optimizer](https://dataengineerblog.com/tag/catalyst-optimizer/)
- [pyspark](https://dataengineerblog.com/tag/pyspark/)
- [dataengineer](https://dataengineerblog.com/tag/dataengineer/)