Spark

From a sales question to a checked Spark result

01 / 60

Why Spark?

Spark is a distributed processing engine. It can read data from HDFS, object storage and other sources.

Spark plans a connected sequence of distributed operations, from reading data to writing results.

It can pipeline compatible work and reuse cached results across queries.

Shuffles, spills and durable writes still use network or disk. Performance depends on the workload.

Chained MapReduce jobsJob 1map → reduceStored resultwrite + readJob 2map → reduceSpark: connected operationsRead + filterGroup + sumWrite resultCache for reuse

Spark offers flexible pipelines and reuse; it does not make every operation in-memory or automatically faster.

02 / 60

Sales by region

A retailer asks for the total sales amount in each region. Each row is one sale; region is the grouping key.

A retailer has three sales: North 10, South 20 and North 5. What total belongs to each region?

One input row is one sale. One output row is one region, so North combines two sales.

The checked answer is North 15 and South 20; the total amount remains 35.

Input: individual salesOutput: regional totalsNorth10South20North5North1510 + 5South20203 sales rows2 regional totals

North = 10 + 5 = 15; South = 20. The grand total is 35 before and after aggregation.

03 / 60

Define the regional totals

Define the accepted sales and the meaning of each output row.

04 / 60

The Spark workflow

Start with what one sale row means. Then describe the result, ask Spark to calculate it, and check the saved answer.

The required answer stays North 15 and South 20 through every step; names and types, operations, execution and checks must agree.

Define inputOne row = one saleregion + amount typesDescribe resultKeep positive salesGroup by regionSum amountsRun queryshow() or writeSpark calculates resultCheck answerNorth 15 · South 202 rows · total 35
05 / 60

RDDs and DataFrames

RDD (Resilient Distributed Dataset) holds distributed records. A DataFrame holds distributed rows with named, typed columns.

Both represent the same sales. Named DataFrame columns suit this regional query; RDD functions offer a lower-level record view.

RDDTuples in this exampleDataFrameNamed columns + schema("North", 10)("South", 20)("North", 5)region: STRINGamount: DOUBLENorth10South20North5Functions transform recordsExpressions refer to named columns
06 / 60

Schema

A schema states the name and data type of each column, so Spark can interpret CSV text as typed rows. In the notebook, spark is the SparkSession used to read it.

North,10 becomes region = North and amount = 10.0. North,? needs a data-quality check; declared types alone do not prove every value is valid.

CSV textRead with a schemaRead outcomeregion,amountNorth,10South,20North,?region: STRINGamount: DOUBLENames + data typesNorth | 10.0South | 20.0North,? → inspectspark.read.schema("region STRING, amount DOUBLE").csv("sales.csv", header=True)printSchema() reports types; inspect actual rows for invalid amounts.
07 / 60

Transformations and actions

A transformation such as filter, select or groupBy describes a new distributed result. Spark records the work to do; the original DataFrame stays available.

An action such as show(), count() or write needs a result. Spark then executes the required plan.

Existing dataTransformation → new planAction → computed rowssalesNorth | 10South | 20North | 5valid = sales.filter( "amount > 0")Describes rows to keepNo result requested yetvalid.show()North | 10South | 20North | 5
08 / 60

Regional aggregation

Start with three sale rows: North 10, South 20 and North 5. Filtering amount > 0 keeps all three.

Grouping brings the two North amounts together; summing produces one row per region. The code beneath the diagram expresses these same steps.

Three input salesGroup by regionTwo regional totalsNorth | 10South | 20North | 5North: 10 + 5South: 20Filter amount > 0: all passNorth | 15South | 202 groups · total 35filtersumfrom pyspark.sql import functions as Fvalid = sales.filter(F.col("amount") > 0)totals = valid.groupBy("region").agg(F.sum("amount").alias("total"))totals.show()
09 / 60

Projection and filtering

Projection with select keeps chosen columns; filtering keeps rows that pass a condition. In this temporary variation, South −4 is filtered out.

Projection changes columns; filtering changes rows. This variation leaves North 10 and North 5.

select("region", "amount") → filter("amount > 0"). N = North; S = South.

InputN | 10 | Oct 01S | −4 | Oct 02N | 5 | Oct 02selectN | 10S | −4N | 5filterN | 10N | 5
10 / 60

NULL values

NULL means an amount is missing, not that it equals zero. In this variation, test 10, NULL and −4 with amount > 0.

The positive-sales rule keeps 10; NULL and −4 do not pass. This variation does not change the original three-sale result.

amountamount > 0Filter result10TrueKeep 10NULLUnknown / NULLExclude from this filtered result−4FalseExclude −4Surviving sum10Does not prove missing values were harmless
11 / 60

Actions

An action asks Spark to execute a DataFrame plan. show, count, collect and write return different kinds of result.

Choose the destination deliberately: collect gathers all rows at the driver; write stores a distributed dataset. A small final answer can still need a large scan or shuffle.

OperationDestination / resultStudy distinctionshow(5)Display up to five rowsA sample does not prove full correctness.count()One row-count valueCounts rows, not the sum of amount.collect()All result rows in driver memoryCan exhaust driver memory.write.parquet(path)Dataset files in storageSeveral part-files can form one dataset.limit(5)New DataFrame planTransformation; not a collected Python list.
12 / 60

SQL and DataFrame API

SQL and the DataFrame API can express the same Spark query. A temporary view lets SQL refer to the existing sales DataFrame during this session; it does not save a durable table.

This SQL uses the same positive filter, region grouping and amount sum as the DataFrame query. Changing SUM(amount) to COUNT(*) would answer a different question.

sales.createOrReplaceTempView("sales")
totals_sql = spark.sql("""
    SELECT region, SUM(amount) AS total
    FROM sales
    WHERE amount > 0
    GROUP BY region
""")
totals_sql.show()

Same intended result

North | 15.0

South | 20.0

13 / 60

Which expressions run, and what does count return?

Use the three original sales rows. Predict each variable’s type and the final answer.

Distinguish a grouped count expression from the action that counts DataFrame rows.

a = sales.filter("amount > 0")
b = a.limit(2)
c = a.groupBy("region").count()
d = c.count()

Questions

1. Which lines build plans?

2. Which line requests the result?

3. What are the two rows in c?

4. What is d? Are b’s rows fixed?

14 / 60

Grouped counts

a, b and c are DataFrames; d is the integer 2 returned by the final action.

a.groupBy("region").count() builds a lazy aggregation; c.count() is an action. The final action does not use b.

ExpressionMeaningResulta = sales.filter("amount > 0")Keep positive salesPlan for 3 rowsb = a.limit(2)Bound the result, without orderingPlan for up to 2 unspecified rowsc = a.groupBy("region").count()Count sales inside each regionNorth: 2; South: 1d = c.count()Count rows in the grouped result2 groups → integer 2
15 / 60

Action: Running the query

An action requests computation of a DataFrame plan. Here totals.show() computes the regional sums for North 10, South 20 and North 5.

The driver plans and schedules the work; executors process data partitions. The cluster manager supplies executor resources.

1. Actiontotals.show()Request the result2. DriverPlan calculationSchedule tasks3. ExecutorsRead + filterGroup + sum4. NotebookReturn via driverNorth | 15South | 20
16 / 60

Driver, manager, executors

An action asks the driver to schedule work. The cluster manager allocates resources; executor processes read and process data partitions.

One task processes one partition in a stage. An executor is a process that can run tasks, not a name for an entire machine.

DriverPlans the workSchedules tasksCluster managerAllocates resources for executor processesResourcesExecutor APartition 0North 10 · South 20Executor BPartition 1North 5TasksStorageExecutors read input and write output
17 / 60

Notebook to executors

In the notebook, spark is a SparkSession used to express the query. The driver coordinates its execution; executors read storage directly.

The notebook submits work through the Spark application. Sales rows need not pass through the notebook or driver.

Cluster manager allocates resources; executors access storage directly.

Notebookspark: SparkSessionSubmit an actionDriverPlan + scheduleCoordinate tasksExecutorsRead partitionsCompute / write
18 / 60

Partitions

A partition is a slice of distributed data. The three original sales occupy two input partitions, so this filter stage has one task for each partition.

Three rows in two partitions produce two tasks in this stage. Row count, partition count and machine count differ.

Input partitionsFilter stageOutput partitionsPartition 0North10South20Task 0Keep amount > 0Partition 0North10South20Partition 1North5Task 1Keep amount > 0Partition 1North5
19 / 60

Narrow transformations

A narrow transformation can process each input partition without moving rows to another partition. Here filter and select work locally before totals are merged.

The positive filter keeps all three sales locally. North 10 and North 5 still need to meet for the regional sum.

Positive filter · independent lanesP0: North 10, South 20filterP0: North 10, South 20P1: North 5filterP1: North 5North total · cross-partition dependencyP0: North 10P1: North 5must meetNorth: 10 + 5 = 15
20 / 60

Shuffle

A shuffle redistributes intermediate data between partitions so rows with matching keys can be combined. North 10 and North 5 begin apart.

The North contributions can meet after redistribution and sum to 15. Shuffle moves data and can add network or disk cost.

Local partial sumsGroup matching regionsMergePartitionRegionSum0North100South201North5North10 + 515South2020
21 / 60

Execution hierarchy

An application contains work requested by actions. Jobs consist of stages; each stage runs tasks over partitions.

A shuffle can separate stages. Count the level being asked about: rows, partitions, tasks, stages and machines are different quantities.

Application = driver + executorsAction requests computation → one or more jobsStage A · before shuffleStage B · after shuffleTask 0 → partition 0Task 1 → partition 1Task 0 → output partition 0Task 1 → output partition 1
22 / 60

Stage boundaries

A stage groups tasks that can run without a shuffle between them. The next stage needs the preceding stage’s shuffle output before it can merge the results.

Here two input tasks run independently. The shuffle brings North 10 and North 5 together; the next stage merges them into North 15.

Stage 1 · local workStage 2 · mergeRead → filter → partial sumCombine received valuesShuffleRoute by regionTask 0 · input partition 0North 10South 20Task 1 · input partition 1North 5Task · output partition ANorth: 10 + 5Total = 15Task · output partition BSouth: 20Total = 20
23 / 60

Local combining

Local combining makes partial totals inside each input partition before data moves. In this four-sale variation, add North 7 to the original sales.

P0 can send North 15 and South 20 instead of three raw sales. Combined with P1’s North 7, North becomes 22.

Four raw records → three partial records. N = North; S = South.

Input partitionsP0: N10, N5, S20P1: N7Local sumsP0: N15, S20P1: N7Final sumsNorth: 15 + 7 = 22South: 20
24 / 60

Lineage

Lineage records how Spark derived a computed partition. If it is lost, Spark can rerun the steps using available input and resources.

Recomputation cannot replace a missing source or correct bad logic. A cached partition is not a durable backup.

Lost computed partition → recompute from source → restored resultAvailable source + lineageInput partition 1North | 5Operation: keep North rowsOriginal executorExecutor failedComputed partition lostReplacement executorFilter completedNorth | 5 ✓
25 / 60

Physical plans

A physical plan describes the operations Spark chooses to produce the requested result. For positive sales summed by region, it may scan, filter, combine locally, exchange and finish the sum.

explain("formatted") reveals the plan selected before execution. Runtime evidence in the Spark UI can show costs and adaptive changes.

Conceptual plan · read bottom to top

Final aggregate: sum by region
  Exchange: partition by region
    Partial aggregate: local sums
      Filter: amount > 0
        Scan: region, amount

What to identify

Scan → source and required columns

Filter → which rows qualify

Exchange → redistribution boundary

Aggregate → local and final work

26 / 60

Early filtering

If the question changes to North-only sales, filtering South before grouping can reduce downstream work.

Early row and column reduction can reduce later work; source-level read savings depend on support.

Preserve required columns and query meaning when moving or adding reductions.

Scan inputAll regionsMany columnsKeep needed dataregion = Northregion + amountAggregateFewer rows to combineFewer columns to carry
27 / 60

Keys and joins

Keys combine values in an aggregation and match rows in a join.

28 / 60

RDD word records

In an RDD, flatMap splits each input line into individual word records. map then pairs each word with 1.

Each pair represents one occurrence; repeated words will be combined by key.

Lines“red blue”“red”flatMapred · blueredmap(red, 1) · (blue, 1)(red, 1)
29 / 60

Word count with two APIs

RDD reduceByKey adds values for equal word keys. collect() requests the small RDD result and brings it to the driver.

Lab 07 uses DataFrame split, explode and groupBy("word").count() to obtain the same grouped counts.

lines = sc.parallelize(["red blue", "red"])
words = lines.flatMap(lambda s: s.split())
pairs = words.map(lambda w: (w, 1))
counts = pairs.reduceByKey(lambda a, b: a + b)
counts.collect()

Lab 07 DataFrame route

split lines → explode words

groupBy("word").count()

red | 2; blue | 1

show() requests the result

30 / 60

Duplicate join keys

A join matches rows by a key. A sale with two matching lookup rows appears twice; here North maps to both A and B.

Define key uniqueness and join type before trusting totals. A left join preserves unmatched left rows but still duplicates repeated matches.

SaleMatches in lookupInner-join rowsNorth | 10A and BNorth | 10 | A; North | 10 | BNorth | 5A and BNorth | 5 | A; North | 5 | BSouth | 20No matching lookup rowNo output rowBefore: 3 rows; sum 35After: 4 rows; sum 30Duplication and loss happen together.
31 / 60

Left joins

A left join keeps each sale even when no lookup row matches. With this duplicate North lookup, matching sales still appear twice.

Left joins preserve unmatched left rows, but repeated matches still multiply them.

SaleLeft-join resultAmount contributionNorth 10North 10 A; North 10 B20North 5North 5 A; North 5 B10South 20South 20 NULL20Total5 output rows50, compared with original 35
32 / 60

Join strategies

A shuffle join moves rows by key; a broadcast join copies a suitable small lookup to executors. Both must preserve the checked join result.

Broadcast may avoid shuffling the large sales side, but lookup bytes, executor memory and join support matter. Confirm the chosen plan.

The join result must stay the same. Broadcast requires a lookup small enough for each executor.

Shuffle join · matching keys move togetherBroadcast join · copy a small lookupSalesN / S keysLookupN / S keysJoin matching groupsBoth inputs may shuffleN → A · S → BSales P0Local lookup copySales P1Local lookup copy
33 / 60

Performance diagnosis

Use task evidence to identify idle capacity, uneven work, repeated computation and spill.

34 / 60

Spark UI evidence

The Spark UI is the application’s runtime view of jobs, stages and tasks. It shows whether time is lost to idle capacity, skew or spill.

Use task duration, input size, shuffle bytes and spill together to identify the likely bottleneck before changing settings.

Observed patternInspect nextPossible explanationOne large, slow taskKey distribution + input sizeSkew or uneven workMany tasks spillTask size + executor memoryExecution memory pressureFew tasks; idle slotsPartitions + available capacityInsufficient parallel workDriver fails during collectResult size + driver memoryGathered result is too large
35 / 60

Task slots

A task slot means capacity to run one task at a time in this example. Four equal tasks and two available slots require two waves.

Two slots run two tasks together, then the next two. Adding partitions creates work units; adding capacity helps only if enough work can run.

Wave 1Wave 2Time →Slot 1Task · partition 0Task · partition 2Slot 2Task · partition 1Task · partition 3Illustration: equal task durations.
36 / 60

Partition sizing

Partitions determine work units. Available slots, data size and task costs determine useful parallelism.

Too few partitions leave slots idle; too many tiny partitions add scheduling and file overhead.

Too few: 1 taskSlot 1Slot 2Slot 3Slot 4Idle slots; large taskLimited parallelismEnough useful work: 4 tasksSlot 1Slot 2Slot 3Slot 4Useful task sizeManageable overheadToo many: many tiny tasksSlot 1Slot 2Slot 3Slot 4Gaps = launch overheadSmall tasks add up
37 / 60

repartition and coalesce

repartition redistributes rows and can increase or decrease partition count. It may improve balance, but it costs a shuffle.

coalesce can reduce partitions without a full rebalance. Fewer tasks can reduce overhead, but a single output partition can serialize a large write.

The extra South 7 row illustrates routing only; the original sales answer remains North 15 and South 20.

repartition(2, "region") · shufflecoalesce(2) · merge partitionsP0: N10, S20P1: N5P2: S7P0: N10, S20P1: N5P2: S7N: 10, 5S: 20, 7N10, S20, N5S7
38 / 60

Skew

Skew means one task receives much more data or work than its peers. A frequent key can dominate one destination partition.

The stage waits for its slowest task. More workers or hash partitions may leave that hot key together; inspect key sizes and task times.

Stage completion waits for the 10-second task. More workers alone may not split its key.

Illustrative task durations in one stageP02 sP12 sP2 · hot key10 sP32 s
39 / 60

Caching

Caching asks Spark to retain computed partitions for later actions. It is useful when the cost of repeated work exceeds cache population and storage costs.

Populate only when an action runs, reuse when valuable, and release with unpersist(). Cached data is not durable output.

Register cache → materialise through an action → reuse → unpersist when finished.

Read + cleanNorth 10 · South 20North 5populateCached intermediatePartition 0: N10, S20Partition 1: N5Available for reuseQuery A · regional sumN15 · S20Query B · sale countN2 · S1
40 / 60

Cache reuse in code

Here clean is reused by a row count and a regional aggregation.

Benchmark cold work and reused work separately; report whether cache population is included.

For a one-off query this extra materialisation may be wasteful.

clean = sales.filter("amount > 0").cache()
clean.count()  # populate for this example

clean.groupBy("region").sum("amount").show()
clean.groupBy("region").count().show()

clean.unpersist()

What each step means

cache(): register persistence

count(): request all rows for this example

Two queries reuse available cached data

unpersist(): release cached blocks

41 / 60

Spill

Spill occurs when execution work such as sorting or aggregation cannot remain entirely in executor memory and uses local disk. Cached partitions can compete for resources.

Use spill metrics and task sizes to test whether smaller tasks or releasing cache helps. Driver memory pressure from collect() is a separate problem.

Executor memory · finite capacityCached dataReused partitionsoccupy resourcesExecution memorySort / aggregate stateAdditional data exceeds capacityspillLocal diskSpilled task dataMore reads and writesLonger execution possible
42 / 60

Fair benchmarks

A fair performance comparison holds the query meaning, input and output checks fixed.

State the change, verify equal results, and explain the measured effect. Do not infer speed from shorter code.

Keep / recordWhy it mattersSame input, query meaning and output checksOtherwise you may be timing different work.An action that completes the intended computationTiming lazy plan construction misses execution.Same cluster resources; record relevant settingsCapacity differences change elapsed time.Cold versus warm cache; repeated runsReuse and run-to-run variation affect results.Elapsed time, shuffle, task spread and spillEvidence connects the bottleneck to the change.
43 / 60

Benchmark evidence

Illustrative repeated writes use the same input, result, cluster and cold-cache condition.

Compare repeated completed runs and connect the change to a measured reduction in work.

EvidenceBaselineAfter reducing unused dataCompleted write times30, 32, 31 seconds22, 24, 23 secondsMedian31 seconds23 secondsShuffle bytes120 MB60 MBRead-back checksSchema, count, totals matchSame checks match
44 / 60

Storage and output

Skip unnecessary input, then save and verify the regional totals.

45 / 60

Parquet

CSV stores row-oriented text; Parquet stores typed values in column chunks with metadata.

The query needs region and amount but not date, so Parquet can skip the date chunks. Actual I/O and time savings depend on the file layout and workload.

CSV · row-oriented textParquet · typed column chunksregion,amount,dateNorth,10,2026-10-01South,20,2026-10-01North,5,2026-10-02regionNorthSouthNorthamount10205dateOct 01Oct 01Oct 02
46 / 60

Pruning and filtering

Directory pruning skips directories; supported statistics can skip row groups; row filtering tests the surviving records.

A pushed predicate does not guarantee that all nonmatching bytes are skipped. The format, metadata and reader determine the savings.

Query: date = '2026-10-01' AND amount > 15Directory pruningdate=Oct 02Skip this directory.date=Oct 01Inspect its files.Row-group statisticsRG1 amount: min 5, max 10Skip: cannot exceed 15.RG2 amount: min 10, max 20Read: may contain matches.Row filteringRG2 values: 10, 20Test amount > 15.Keep: 20Reject: 10
47 / 60

What can be skipped?

Query: date = Oct 01 AND amount > 15. Dates are directory partitions.

Skip only data that cannot match; test rows in any surviving row group.

Stored dataEvidenceDecisionOct 02 directoryWrong partition dateSkip directoryOct 01, group A: 5, 10Maximum 10 ≤ 15Skip row groupOct 01, group B: 10, 20Maximum 20 > 15Read and test both rowsFinal matching row20 > 15Keep amount 20
48 / 60

Saved datasets

A write creates a dataset at a path, often with several part-files. Read that path back to verify the answer actually stored.

Read the whole dataset directory, then check schema, two regional rows and total 35. Save to independent storage before cluster shutdown.

A relative path uses the environment’s filesystem. Choose independent durable storage before cluster shutdown.

out = "results/region_totals"  # new path
totals.write.mode("errorifexists").parquet(out)

check = spark.read.parquet(out)
check.printSchema()
check.show()
check.count()

Verify the dataset

Schema: region and numeric total

2 rows: North 15; South 20

Grand total: 35

Several part-files may form one dataset.

49 / 60

Durable output

Durable output is stored independently of a cluster. Cluster-local files can disappear when the cluster is terminated.

Write to independent durable storage, read that copy back, then end the cluster.

Example independent destination: an approved S3 location outside cluster-local storage.

WriteIndependent storageRequired resultsRead + verifySame saved pathSchema, count, totalsFinishPreserve lab evidenceTerminate cluster
50 / 60

Spark in Labs 07 and 08

Lab 07 counts words using DataFrame split, explode and groupBy. Lab 08 reads typed CSV, compares SQL with DataFrame code and saves Parquet.

The same principles recur in both labs: define data, request work, check the result, preserve the saved copy and terminate the cluster.

The notebook submits the query; Spark executors do distributed work.

Lab 07DataFrame word countsplit → explode → groupByLab 08Typed CSV + SQLValidate → save ParquetBefore shutdownSave results + evidenceRead back → terminate
51 / 60

From query rules to a reliable Spark result

Connect query rules, execution, performance and verified output.

52 / 60

Sales to saved totals

For regional sales, query rules determine the answer; partition tasks and any shuffle produce it; a read-back check confirms it.

The data, query, execution and saved checks must agree on North 15 and South 20.

Storage provides data; the cluster manager provides resources; executors do the partition work.

ExpressDataFrame / SQLSchema + validationPlanLazy operationsDriver + optimizerExecutePartition tasksShuffle if neededVerifyCorrect resultCost + saved output
53 / 60

Execution roles

The driver coordinates; executors run tasks over partitions; slots limit how many tasks run at once.

Four partitions and two slots mean four tasks in two waves under the stated equal-duration assumptions.

TermMeaningDo not confuse it with…Driver / cluster managerPlan and schedule / allocate resourcesExecutors that compute partition tasksApplication / jobApplication processes / action-triggered workOne line of Python codeStage / taskParallel work phase / work on one partitionOne task per rowPartition / slotData work unit / concurrent task capacityA fixed machine or an HDFS block
54 / 60

Query correctness

Types, accepted rows, grouping keys and aggregation must match the sales question.

Trace actual rows and values. A successful run does not prove that the chosen filter, join or aggregation answers the question.

ConceptRule to retainRDD / DataFrame / SQLRecords / schema-based table / query language for the same engineTransformation / actionBuild a new plan / request a result or outputshow / count / collect / writeSample / row count / all driver rows / stored datasetgroupBy(...).count() / df.count()Lazy grouped aggregation / action returning row countJoin type + key multiplicityUnmatched rows can disappear; repeated matches multiply rows
55 / 60

Cross-partition totals

Original input: P0 has North 10 and South 20; P1 has North 5.

A shuffle can separate stages and add network/disk cost. Explain shows the plan; runtime evidence shows what it cost.

Same-key contributions meet; multiple keys may share a shuffle partition.

Local workFilter + selectPartial aggregationShuffle by regionNorth: 10, 5South: 20Merge groupsNorth = 15South = 20
56 / 60

Performance choices

Idle slots, uneven tasks, repeated work and spill point to different possible changes.

Reduce unnecessary data, diagnose the bottleneck, change one thing, and compare correct completed results.

Observed evidenceCandidate responseTrade-off / checkIdle slots; few large tasksMore useful partitionsExtra shuffle or scheduling costMany tiny tasks or filesSuitable coalesce / output layoutToo few tasks restrict parallelismOne very slow, large taskInvestigate skew and hot keysExtra workers alone may not helpRepeated expensive intermediateCache, then unpersistPopulation cost and memory useLarge join + small lookupConsider broadcastLookup must fit; verify chosen plan
57 / 60

Reuse and durability

Pruning skips input; cache reuses computed data; lineage can rebuild it; durable storage preserves output.

A cached DataFrame is not a backup. Before shutdown, verify the independent saved copy and required evidence.

MechanismWhat it providesImportant limitParquet + column pruningRead required column chunksBenefit depends on query and sourceDirectory / row-group pruningSkip data that cannot matchStill test surviving rowsCache / lineageReuse / recompute partitionsNeeds resources / available inputsDurable write + read-backPreserved, checked outputCluster-local data may disappearSchema + count + totalsEvidence of saved result meaningA sample alone is incomplete
58 / 60

Predict the saved regional totals

P0: North 10, South 20. P1: North 5, South −4. Keep amount > 0, then sum by region.

Name the rule and explain its consequence. Show enough arithmetic to justify the answer.

Assume out is a new path; the filter deliberately excludes the return. Two input partitions do not fix output-file count.

q = (sales.filter("amount > 0")
     .groupBy("region").sum("amount"))
q.write.mode("errorifexists").parquet(out)

Explain four things

1. What result is written?

2. Which line requests execution?

3. Why might a shuffle be needed?

4. What must be checked before shutdown?

59 / 60

Saved totals

The positive-sales rule keeps 10, 20 and 5 and excludes the South return of −4.

The write triggers execution; North contributions meet; a read-back check confirms two rows and total 35.

QuestionReasoned answerWritten resultNorth = 15; South = 20; 2 rows; grand total = 35.Execution triggerwrite requests output; filter and aggregation build the plan.MovementNorth contributions cross input partitions and must meet.Before shutdownRead saved data; check schema, count and totals; preserve an independent copy.