Catalist optimizer : Prediction pushdown

if in your code/query if you are filterring the data at the end, Catalyst optimizer (in prediction pushdown) will apply filtering on input or source and then do the other operations.

Predicate pushdown means that the optimizer tries to move filter operations as close to the data source as possible, even if the filter is applied later in the query.

For instance, if you have a query that filters the data after a join operation, the catalyst optimizer will attempt to push the filter down to the individual data sources before performing the join. This reduces the amount of data that needs to be processed during the join operation, leading to more efficient query execution.

Before Optimization (Initial Logical Plan)

# Original query with filter applied after join 
SELECT * 
FROM sales_data s 
JOIN customer_data c 
ON s.customer_id = c.customer_id
WHERE s.sales_amount > 1000

After Optimization (Logical Plan with Predicate Pushdown)

# Optimized query with filter pushed down 
SELECT * 
FROM (
        SELECT * FROM sales_data WHERE sales_amount > 1000  -- Pushdown Filter
      ) s
JOIN customer_data c 
ON s.customer_id = c.customer_id

here it is scanning sales_data at the beginning it self before optimization it in bringing all the data into memory.

Predicate pushdown is a powerful optimization technique that helps in reducing I/O and improving query execution times by minimizing the amount of data read and processed at each stage.