Big Data, Hadoop
and MapReduce
Scale, storage and distributed processing
j@forth.ai
Topic 1: what you should be able to do
Use these outcomes to check your understanding of Topic 1.
| Skill | Explain or demonstrate |
|---|---|
| Diagnose | Use the five Vs; distinguish data quality from capacity. |
| Compare | Scale up vs scale out; explain a benefit and a cost. |
| Explain | HDFS storage, YARN resources and MapReduce processing. |
| Trace | Read/write paths, task allocation and keyed aggregation. |
| Justify | Storage choice, combiner correctness and EMR data survival. |
Topic 1 study map
Study a section, answer without looking, then check the worked explanation.
| Topic | Revision task | Slides |
|---|---|---|
| 1 · Big Data + scaling | Diagnose pressures; compare capacity choices. | 8–26 |
| 2 · Hadoop + HDFS | Separate roles; trace storage and recovery. | 27–41 |
| 3 · YARN | Match each actor to its scope and actions. | 42–48 |
| 4 · MapReduce | Trace pairs, groups, tasks and output files. | 49–65 |
| 5 · EMR + conclusion | Explain node roles; practise and check answers. | 66–80 |
Five lectures: from distributed data to useful results
From sales data to a demand forecast
Practise the concepts in five labs
Save results and evidence. Terminate the cluster when the lab is complete.
Real-world example: turning web logs into user analytics
Store the records, share the work, and check the answer against the business question.
Big Data
What makes data useful, and when does its size become a problem?
Big Data: size matters relative to the task
Describe the data and the task before choosing tools.
| Question | Meaning | Example |
|---|---|---|
| What makes it big? | Data strains the available storage or processing. | No universal file-size threshold. |
| What makes it useful? | It can answer a defined question. | Orders measure sales; clicks measure interest. |
| What makes it trustworthy? | Identity, missing values and timing are checked. | A Tuesday release cannot inform Monday’s prediction. |
The five Vs describe different pressures
Volume: the amount of data
Volume is the quantity of data to store and process. It becomes a challenge when the dataset exceeds the capacity of the available system.
Partitioning divides the work; it does not reduce the total data. Here, all six records remain across three partitions.
Velocity: the rate of incoming data
Velocity describes how quickly data arrives and must be processed. A system must keep up with arrivals and meet the required response time.
Arrivals exceed processing by one event per interval, so the backlog grows. Buffering delays the problem; sustained throughput must catch up.
Variety: different forms of data
Variety means differences in data formats, structures and meaning. Tables, nested records and images require different ways to read and interpret them.
Parsing makes fields usable, but does not make their meanings equivalent. Image dimensions describe its size, not what the image shows.
Veracity: the trustworthiness of data
Veracity concerns data accuracy, completeness and reliability. Duplicates, missing values and incorrect records can make a technically correct calculation misleading.
Exclude the confirmed replay and keep the missing amount unknown. The accepted amount is 10; it is not a complete total for all orders.
Value: the usefulness of data
Value is the benefit gained by using data to answer a meaningful question or improve a decision. More data or faster processing does not automatically create value.
Demand of 7 minus stock of 4 suggests a gap of 3. The uncertainty range makes this evidence for a replenishment decision, not a guaranteed order quantity.
Scaling
How can we add capacity when one machine is no longer enough?
Compare scaling up with scaling out
More machines introduce coordination, networks and partial failures.
Scale up: add resources to one machine
A larger server helps only when the added resource is the bottleneck.
Scale out: divide work across machines
The benefit of parallel work must exceed scheduling and communication costs.
Trade-off: a shared result needs coordination
Workers process data in parallel, but they must exchange partial results to produce one correct answer.
Revision: identify the problem before the remedy
For a scenario answer: identify the pressure → explain the cause → justify the remedy.
| Distinction | Rule | Reason |
|---|---|---|
| Volume / velocity | Amount of data / rate of arrival. | A growing queue signals insufficient throughput. |
| Variety / veracity / value | Formats / trust / useful outcome. | Different pressures need different remedies. |
| Scale up | Add CPU, memory or storage to one host. | Helps only if that resource limits the work. |
| Scale out | Add hosts and divide work. | Communication and coordination reduce the gain. |
| Capacity / correctness | Measure speed; validate meaning separately. | More workers can still produce a wrong answer. |
Case study
How can we produce a correct hourly report before the next batch arrives?
Case study: an hourly report is late and wrong
Count distinct posts per account within 60 minutes. Diagnose the incorrect count and the missed deadline.
Remove confirmed duplicates before the join
Remove the replay of P1 and the repeated account row, then join two posts to one account record.
The queue grows when a batch takes too long
Batches arrive every 60 minutes, but each takes 90 minutes, so their waiting times grow from 0 to 30 to 60 minutes.
Accept the design only after these checks pass
The report must show two posts for U1, resist replay, and finish within the 60-minute deadline.
Hadoop architecture
How does Hadoop divide storage, resource management and processing?
Hadoop makes a cluster work together
Apache Hadoop is open-source software for distributed storage and processing across a cluster of computers.
Follow one job from stored data to a saved result
HDFS stores the input and output. YARN allocates resources. MapReduce tasks count the records.
One job connects storage, resources and processing
YARN allocates resources. MapReduce tasks read data from HDFS, process it, and save results to HDFS.
Hadoop separates storage, resources and processing
These components work together. Each component solves a different problem.
HDFS
Store blocks. Track locations. Recover replicas.
HDFS separates the file directory from its bytes
The NameNode tracks files and block locations; DataNodes store the block replicas.
HDFS splits a file into blocks and stores replicas
This example stores two copies of each block on DataNodes, while the NameNode records their locations.
Clients obtain metadata, then read block bytes
Clients get block locations from the NameNode, then read the data directly from DataNodes.
A write uses a replication pipeline
After the NameNode selects DataNodes, the client streams block data through the pipeline to create three replicas.
A failed replica changes the read path
The client reads a surviving replica while the NameNode arranges a new copy to restore replication.
HDFS supports large sequential workloads
HDFS suits large sequential reads. Combine small records into larger files to reduce file metadata overhead.
Case study: choose storage for three workloads
5 min · Your team must store these three workloads. Choose the best HDFS fit, explain the limits of the other two, and propose suitable storage designs.
Solution: use HDFS for the log archives
Use HDFS for batch scans, object storage for image retrieval, and a database for frequent profile updates.
HDFS revision: roles, paths and suitability
Explain both the mechanism and its consequence; a component name alone is incomplete.
| Question | Answer | Why it matters |
|---|---|---|
| Who stores what? | NameNode: metadata. DataNodes: block bytes. | Metadata lookup and data transfer are separate. |
| How does a read work? | Client asks NameNode, then reads a DataNode. | File bytes do not pass through the NameNode. |
| How does a write work? | NameNode selects nodes; client streams a pipeline. | Replicas store copies of the same block. |
| What if a replica fails? | Read a survivor; restore the missing replica. | Replication supports availability after node loss. |
| Which workload fits? | Large sequential scans; fewer larger files. | Tiny files add metadata and file-open overhead. |
YARN
Allocate resources. Coordinate applications.
YARN shares cluster resources among applications
The ResourceManager grants resources; each ApplicationMaster coordinates its job through worker NodeManagers.
YARN revision: actor, scope and responsibility
YARN allocates resources; the application’s tasks perform the calculation.
| Actor | Scope | Responsibility |
|---|---|---|
| ResourceManager | Whole cluster | Grants resources according to capacity and policy. |
| ApplicationMaster | One application | Requests resources; coordinates and tracks work. |
| NodeManager | One worker machine | Launches and monitors local containers. |
| Container | One resource allocation | CPU and memory for a process; not necessarily Docker. |
First, YARN starts the application coordinator
The ApplicationMaster needs its own container before requesting task resources.
Then, the coordinator launches and tracks work
Tasks finish. YARN releases their resources. The application unregisters.
Self-check: who allocates, launches and calculates?
Answer before moving to the worked explanation.
| Situation | Your task |
|---|---|
| A word-count application needs two task containers. | Name the actor that requests them and the actor that grants them. |
| The granted containers are on two worker machines. | Who launches the work? Who tracks application progress? |
| A reducer produces the final word counts. | Does the ResourceManager calculate those counts? Explain. |
Worked answer: separate allocation from computation
An allocation enables computation; it is not the computation itself.
| Action | Responsible actor | Reason |
|---|---|---|
| Request / grant | ApplicationMaster / ResourceManager | The application requests; the cluster allocator grants. |
| Launch / monitor locally | NodeManagers | Each manages containers on its own machine. |
| Track the application | ApplicationMaster | It coordinates the application’s tasks and progress. |
| Calculate counts | MapReduce task processes | YARN provides resources; tasks execute the logic. |
MapReduce
Map records. Group keys. Reduce values.
A key chooses the group; a value supplies the data
For word counts, emit the word as the key and 1 for each occurrence.
MapReduce groups records with matching keys
Local records become intermediate pairs, then grouped results.
An input split is logical work, not a stored block
With text input, each RecordReader delivers complete lines even when a line crosses a split or block boundary.
Mappers emit what the next stage needs
The mapper splits each line into words and emits one (word, 1) pair for every occurrence.
A combiner sums locally before records travel
An optional combiner reduces transferred records; the reducer must produce the same answer without it.
Combiner correctness: the result must stay the same
A combiner reduces transferred data only when local aggregation preserves the final answer.
| Check | Requirement | Example |
|---|---|---|
| Optional execution | Correct with no combine, one combine or repeated combines. | Counts stay correct when partial counts are added. |
| Order and grouping | Merge partial results without changing the answer. | For sums: (1 + 1) + 1 = 1 + (1 + 1). |
| Compatible state | Keep a representation that can be merged again. | For an average, carry (sum, count). |
Averages of averages can be wrong
Unequal group sizes break a naive average-of-averages.
Keep the sum and count until the final division
Merge the sums and counts. Calculate the average once at the end.
A partitioner routes many keys to each reducer
Equal keys use the same reducer. Different keys may also use that reducer.
Reducer tasks iterate over grouped keys
One reducer task processes both keys and writes both records to one part-file.
Trace six records through two reducer tasks
Count keys and tasks separately.
Two reducer tasks handle three distinct keys
Six emitted records form three key groups; two reducer tasks write three totals into two part-files.
MapReduce revision: keep these quantities separate
Trace records first, then count key groups, reducer tasks and output files separately.
| Concept | Meaning | Consequence |
|---|---|---|
| Block / input split | Stored bytes / logical input work. | A RecordReader handles complete records across boundaries. |
| Map / combine | Emit pairs / optionally aggregate locally. | A combiner must not change the final result. |
| Partition / group | Choose reducer / gather equal keys. | Equal keys go together; one reducer can handle many keys. |
| Task / reduce call | One running reducer / one grouped key. | Three keys need not mean three reducer tasks. |
| Output record / part-file | One emitted result / one task’s output file. | In our standard text-output examples, two reducers write two part-files. |
Execution trade-offs
What does MapReduce gain and lose by writing intermediate results to disk?
Separate shuffle storage from output between jobs
Map output uses local storage for shuffle; a later job reads the previous job’s saved output.
Disk-based execution: benefits and costs
Distinguish temporary shuffle storage from saved job output when explaining the trade-off.
| Storage | Benefit | Cost or limitation |
|---|---|---|
| Local map output | Buffers and spills support data larger than memory. | Disk I/O and network shuffle add work. |
| Saved output between jobs | A later job can reuse the stored dataset. | Writing and rereading adds I/O and latency. |
| Failure distinction | Lost map output can be recomputed by rerunning its task. | Local shuffle data is not an HDFS-replicated final result. |
Hadoop in the cloud
How do Hadoop responsibilities map to a managed cloud cluster?
EMR revision: three node roles
Choose a node type by the capacity it adds and the responsibilities it holds.
| Node type | Role | Consequence |
|---|---|---|
| Primary | Coordinates cluster services. | Coordination differs from worker computation. |
| Core | Runs tasks and HDFS DataNodes. | Removal affects both compute and stored replicas. |
| Task | Adds task-processing capacity without HDFS DataNodes. | More compute does not add HDFS storage. |
Cluster lifetime and data lifetime are different
Save and verify needed results outside the EMR cluster before terminating its compute and HDFS storage.
Check saved evidence. Terminate the cluster.
Closing a notebook does not terminate its cluster.
Explain storage, scheduling and keyed aggregation
5 min · Input: “car bus car”; one reducer. Explain storage, resource allocation, emitted pairs, groups and final file contents. What must you verify before and after shutdown?
Model answer: trace the file to a checked result
Storage supplies bytes, YARN supplies resources, and MapReduce groups and totals the emitted pairs.
Practice: apply MapReduce to sales totals
Write your answer before viewing the solution. Values are illustrative.
| Given | Question |
|---|---|
| Sales: (North,10), (South,7), (North,5). | What key and value should each mapper emit? |
| Partition rule: North → R0; South → R1. | Show grouped input and totals for each reducer. |
| Optional local aggregation. | Can a sum combiner be used? Explain why. |
| Standard text output; both reducers succeed. | How many key groups, reducer tasks and part-files? |
Worked answer: sales totals by region
The key changes with the business question; the map–group–reduce reasoning stays the same.
| Step | Result | Reason |
|---|---|---|
| Map | (North,10), (South,7), (North,5) | Region defines the group; amount is the value. |
| Group and reduce | R0: North [10,5] → 15; R1: South [7] → 7 | Add values within each region. |
| Combine | Sum partial amounts, if the combiner runs. | Regrouping sums preserves each region’s total. |
| Count | 2 key groups; 2 reducer tasks; 2 part-files | The supplied partition rule assigns one group to each task. |
Conclusion: Hadoop essentials
Review the whole lecture: the problem, the system, the processing and the result.
1 · Big Data and scaling
Choose a remedy that addresses the actual cause of the problem.
| Key point | What you should explain |
|---|---|
| Five Vs | Volume: amount. Velocity: arrival rate. Variety: formats. Veracity: trust. Value: useful outcome. |
| Big Data | The data and task strain available capacity; there is no universal size threshold. |
| Scale up / scale out | Add resources to one host / add hosts and divide the work. |
| Cost of distribution | Parallel work adds communication, coordination and partial-failure handling. |
| Correctness and capacity | Validate records to fix wrong results; measure bottlenecks to address slow results. |
2 · Hadoop roles and HDFS storage
A replica is another copy of a block, not another unique input record.
| Key point | What you should explain |
|---|---|
| Three responsibilities | HDFS stores files; YARN allocates resources; MapReduce processes records. |
| Metadata / bytes | NameNode tracks files, blocks and locations; DataNodes store block replicas. |
| Read / write | Read: get locations, then fetch bytes. Write: get placement, then stream a replica pipeline. |
| Replica failure | Read a surviving copy; restore replication. Copies improve availability after node loss. |
| Workload suitability | Large sequential scans fit HDFS. Tiny files add metadata overhead; frequent record updates need a suitable database. |
3 · YARN resource management
Keep cluster-wide allocation, application coordination and local task execution separate.
| Actor or step | What you should explain |
|---|---|
| ResourceManager | Allocates cluster resources; it does not execute the application’s calculation. |
| ApplicationMaster | Coordinates one application, requests containers and tracks task progress. |
| NodeManager | Launches and monitors containers on one worker machine. |
| Container | An allocation of resources such as CPU and memory; not necessarily Docker. |
| Application lifecycle | Submit → start AM → request task resources → launch and track work → release resources. |
4 · MapReduce: input to output
Example: car, bus, car → (car,1), (bus,1), (car,1) → car:[1,1], bus:[1] → car:2, bus:1.
| Stage or distinction | What you should explain |
|---|---|
| Block / split / record | Block: stored bytes. Split: logical mapper input. RecordReader delivers records. |
| Map | Emit key–value pairs. For word count: each occurrence emits (word, 1). |
| Partition and shuffle | Route equal keys to the same reducer; transfer, sort and group intermediate pairs. |
| Reduce | Process each grouped key and its values to emit final results. |
| Keys / tasks / files | One reducer can process several keys. In our standard text-output examples, each reducer writes one part-file. |
5 · Combiners and execution trade-offs
A performance optimisation is valid only if it preserves the required result.
| Key point | What you should explain |
|---|---|
| Optional combiner | Locally aggregate to reduce transfer; the answer must survive zero, one or repeated combines. |
| Safe partial results | Merging partial state must preserve the answer under regrouping and reordered inputs. |
| Average pitfall | [10,20] and [90]: mean of means = 52.5, but the true mean is 120 / 3 = 40. |
| Correct average state | Carry (sum, count): (30,2) + (90,1) = (120,3). Divide once at the end. |
| Disk distinction | Shuffle uses local map output; saved output can feed later jobs. Disk and network I/O add cost. |
6 · EMR and a complete Hadoop answer
You should be able to explain the system and work through a new example without copying a diagram.
| Key point | What you should explain |
|---|---|
| Primary / core / task | Primary coordinates. Core provides compute and HDFS storage. Task adds compute without HDFS DataNodes. |
| Cluster lifetime | Cluster HDFS is tied to the cluster; preserve required results outside it before termination. |
| Verified result | Read the saved output, check its meaning and verify the independent copy. |
| Finish the lab | Save required evidence and terminate the cluster. Closing a notebook does not stop the cluster. |
| Answer clearly | State the rule → trace the mechanism → show the result → justify the choice or trade-off. |
Spark architecture
and pipelines
From a query to distributed computation.
One pipeline, two views: code and execution
Predict when Spark runs the work and where it stores each result.
Our question: how much did each region sell?
Two North sales add to 15; the single South sale totals 20.
Spark execution
How does Spark organise several operations into one execution graph?
Spark uses a graph to plan multiple operations
Flexible APIs and data reuse can reduce work. Measure the execution time.
MapReduce materialises boundaries between jobs
Intermediate storage provides boundaries and also costs I/O.
Spark can plan multiple operations together
A shared engine can optimise a connected pipeline.
Spark still requires processing and data transfer
A different API does not remove hardware limits.
Spark architecture
Which work belongs to the driver, and which work belongs to executors?
The driver assigns work. Executors run it.
Each task processes one partition on an executor.
Each partition gets its own task
Within this stage, two partitions produce two tasks—even though they contain three rows.
Four partitions can run in two waves
With two available task slots, four equal-length tasks need two waves.
Assign responsibilities before drawing arrows
5 min · Who filters input partitions? Who schedules work? Where does collect() send the final rows?
Plan on the driver; process on the executors
The cluster manager supplies resources; collect() returns rows to the driver.
Data models
How do RDDs and DataFrames represent distributed data?
Two ways to represent distributed data
Structured information gives Spark SQL more options when it plans work.
An RDD represents records as objects
Object-level transformations make the logic explicit.
A DataFrame has named columns with data types
A schema gives the engine information for planning.
Use a schema when the question is relational
Choose an abstraction for the work. Measure its performance.
A schema defines how to interpret values
The schema assigns column names and types; validation checks the values.
Data ingestion
How do we turn source files into rows with predictable types?
Explicit schemas make a CSV read predictable
An explicit schema names the fields and tells Spark how to parse them.
A declared type does not guarantee a valid value
Inspect failed parses and missing values before filtering or aggregating.
Programming model
Which operations build a plan, and which operations execute it?
Transformations build a new plan
select, filter, limit and aggregation definitions describe work.
Actions request a result or an output
show, count, collect and writes have different destinations.
show and count return different evidence
A sample shows values. A count shows the number of rows.
collect returns every result row to the driver
Limit the result size before you use collect for display.
A distributed write keeps the result distributed
A path can describe a dataset containing several files.
Predict what limit(5) does before show()
Discuss your prediction; the next slide reveals the answer.
limit(5) changes the plan; show() requests rows
The variable b is a DataFrame, not a Python list containing five rows.
Execution dependencies
How does Spark turn a chain of transformations into jobs, stages and tasks?
Dependencies form a directed acyclic graph
Read this graph as data dependencies, not a timeline of machines.
A shuffle separates dependent stages
Spark can combine local operations in a pipeline. Grouping may redistribute rows.
Matching region keys must meet to form the total
This example routes partial sums by key, then merges each region’s values.
Application, jobs, stages and tasks describe different scales
Use the Spark UI to test the explanation against a real run.
Query planning
How can Spark change an execution plan while preserving the result?
The optimizer rewrites valid relational work
It searches for a better plan. It cannot correct business logic.
explain() reveals the plan before a large run
Find scan columns, filters and exchanges. Compare the plans. Measure performance.
Spark SQL
How can SQL and DataFrame code express the same computation?
SQL and DataFrame APIs can express the same query
A temporary view gives a DataFrame a name for queries in this session.
Word count
How does a familiar MapReduce example translate into Spark?
Word count still groups keys, with fewer API steps
The same word-count logic: tokenize the text, group equal words, then count.
Relational operations
How do joins and aggregations change the shape and number of rows?
A unique lookup adds attributes to each sale
A unique matching lookup adds attributes without multiplying a sale.
Before a join, define what each left row represents
A row’s meaning is part of the join specification.
Duplicate lookup keys can multiply input rows
A lookup needs an explicit uniqueness or multiplicity rule.
An inner join can duplicate and discard rows
An inner join keeps matches, multiplies repeated matches, and drops unmatched sales.
Predict the result before running the pipeline
5 min · Trace the amount > 0 filter. Join by customer ID. Sum by region. What if C1 occurs twice in the lookup?
Two positive sales become two regional totals
With unique lookup keys: North = 10 and South = 20.
Reliability
How can Spark recover a lost partition from the operations that produced it?
Spark can calculate lost partitions again from lineage
Lineage can rebuild lost computed data when its inputs remain available.
Pipeline output
How do we store distributed results without collecting all rows on the driver?
Write distributed results before you collect rows for display
Use a bounded aggregate for notebook visualization.
Read the saved dataset back and check its meaning
Verify schema, row count and totals at the output path—not only notebook success.
Spark lab
How does a Zeppelin notebook submit work to the Spark cluster?
Zeppelin sends work to the Spark cluster
A browser interface, a driver and executors have distinct roles.
Cluster lifecycle
What must happen after the notebook finishes?
A completed notebook does not terminate its cluster
Save evidence. Close connections. Check cluster termination.
Use a typed, traceable pipeline in Lab08
5 min · Explain when a census aggregation runs. Identify where tasks execute. State the checks before plotting and cluster termination.
Spark performance
and storage
Read the plan. Measure data movement.
A correct query can still use many resources
Find the resource cost before you change a setting.
Query planning
Where does the execution plan reveal the cost of a query?
The driver chooses a physical plan
The optimizer uses rules and available information. It does not guarantee the fastest execution.
Read the plan before guessing
Find scans, filters and exchanges in an explain output.
from pyspark.sql import functions as F
q = df.filter(F.col("age") >= 30)
q = q.groupBy("occupation").count()
q.explain(mode="formatted")Read less data
How can column selection and filtering reduce the work?
Select the columns the question needs
A source with many columns does not require all columns in the working dataset.
small = df.select(
"age", "occupation", "hours_per_week"
)
small.filter("age >= 30").show(5)A filter can move toward the source
Predicate pushdown and row filtering are related but different.
Predict which data the query can skip
A partition filter and a value filter may skip different work.
Data movement
Which operations keep data local, and which move it between workers?
Narrow work stays within an input partition
Filter and projection usually avoid redistributing records.
Grouping combines records with equal keys
A shuffle redistributes records so tasks can process each complete group.
Exchange separates stages
Tasks in one stage process partitions. The next stage reads shuffled output.
Parallelism
How do partition count and partition size affect the work?
Partitions divide work, not ownership
Too few or too many tasks can reduce performance.
Too few partitions can leave capacity idle
Available workers need independent runnable tasks.
Partitions let tasks process limited amounts of data concurrently
Balance task size against scheduling overhead.
Too many tiny partitions multiply overhead
More partitions can increase overhead after parallelism is sufficient.
Repartition redistributes data through an exchange
Use it when the benefit of a new layout exceeds the exchange cost.
balanced = df.repartition(8, "occupation")
summary = balanced.groupBy("occupation").count()
summary.explain("formatted")Coalesce can reduce tasks without a full shuffle
A drastic reduction can also reduce parallelism.
Many partitions can become many output fragments
Inspect output sizes before changing partition counts.
Coalesce combines existing task partitions
A narrow merge can reduce tasks without evenly redistributing rows.
One output task can become the bottleneck
Writing one output file may reduce throughput.
One frequent key can delay the entire stage
The stage finishes when its slowest required task finishes.
Join strategies
How does the join strategy change the data sent between workers?
The join strategy changes data movement
Large–large joins and small lookup joins have different costs.
A shuffle join redistributes matching keys
Large inputs can require significant exchange work.
A broadcast join replicates a small lookup
Replication uses network capacity and memory so workers can match records locally.
Choose a join strategy using evidence
Measure correctness and cost on equivalent work.
Broadcast a measured small lookup
Check the keys. Inspect the physical plan.
from pyspark.sql.functions import broadcast
joined = df.join(
broadcast(occupation_lookup), "occupation", "left"
)
joined.explain("formatted")Caching
When does reusing a stored result avoid enough work to justify caching?
Cache data that repeated operations reuse
Caching is optional. Cached data uses memory that execution also needs.
An action stores the data after cache registration
Release cached data after the repeated work.
clean = df.filter("age >= 18").cache()
clean.count() # materialize for this example
clean.groupBy("occupation").count().show()
clean.groupBy("income").count().show()
clean.unpersist()Memory pressure
What can disk spill tell us about the workload?
Disk spill is a symptom to investigate
Execution memory, cached data and task size interact.
Storage
How do file format and layout affect the amount of data a query reads?
Parquet stores columns together
Column selection and compression can reduce analytical I/O.
A CSV reader interprets row-oriented text
Projection does not make CSV a columnar file.
Parquet groups typed values by column
Column projection can reduce read work for supported queries.
Statistics can exclude a row group that cannot match
Statistics can exclude data. They do not replace correct row filtering.
A Spark write creates a dataset directory
Read the saved dataset. Check its schema and content.
path = "/user/hadoop/results/occupation_hours"
summary.write.mode("overwrite").parquet(path)
check = spark.read.parquet(path)
check.printSchema()
check.show(5)Partition directories target repeated filters
Choose useful keys without creating millions of tiny files.
path = "/tmp/teaching_events"
events.write.mode("overwrite") \
.partitionBy("dt").parquet(path)
recent = spark.read.parquet(path)
recent.filter("dt = '2026-10-01'").explain()Measurement
How do we compare performance using equivalent work and controlled conditions?
Time equivalent work under controlled conditions
Plan construction time is not query execution time.
A useful result explains both answer and cost
Numbers need the exact query and environment.
Pipeline lab preparation
Which data and format assumptions must we check before running the lab?
Read the lab data description critically
Lab08 downloads Adult census data. The original description differs.
Improve one measured bottleneck at a time
Correctness comes before a performance claim.
Spark ML and
valid evaluation
Build the pipeline. Check the evaluation.
A pipeline becomes a predictive model
Define the question before selecting an algorithm.
Different questions need different targets
Classification, regression and clustering solve different problems.
Classification predicts a category
Probability and the chosen class are different outputs.
Regression predicts a numeric quantity
Evaluate errors in units meaningful to the decision.
Clustering groups examples without target labels
A discovered group is not automatically a business label.
Data contract
What does one row represent, and what outcome will the model predict?
One row must match the prediction unit
Features must exist at the time of the decision.
Evaluation design
How do we keep information from the evaluation data out of training?
Split before learning preparation parameters
Do not use test data to fit pipeline stages.
Random splits suit exchangeable rows
A seed helps repeat an experiment, but row identity also matters.
train, valid, test = raw.randomSplit(
[0.6, 0.2, 0.2], seed=42
)
# Keep the same input and saved split identities.Use time boundaries for future predictions
Future records must not train a model evaluated in the past.
Leakage often looks like a helpful feature
A perfect score can indicate an invalid experiment.
Direct leakage exposes an outcome to the model
A feature must exist at the prediction time.
Preparation leakage lets held-out rows influence fit
Learned preprocessing belongs inside the training boundary.
Entity leakage repeats a person across the split
The split should match who or when the model must generalise to.
Spark ML
How does Spark carry features and outcomes through a learning workflow?
DataFrames carry both features and outcomes
spark.ml is the DataFrame-based pipeline API.
Estimators fit. Transformers transform.
Fitting an Estimator returns a fitted Model.
Features and labels
How do we convert source columns into the inputs and outcomes a model needs?
Declare the positive class explicitly
Set churn yes = 1 to avoid labels that depend on category frequency.
from pyspark.sql import functions as F
known = raw.filter(F.col("Churn").isin("Yes", "No"))
labeled = known.withColumn(
"label", (F.col("Churn") == "Yes").cast("double")
)Category numbers are identifiers
StringIndexer learns a category mapping from training data.
from pyspark.ml.feature import StringIndexer
indexer = StringIndexer(
inputCol="Contract", outputCol="contract_idx",
handleInvalid="keep"
)
index_model = indexer.fit(train)
encoded = index_model.transform(valid)Encode categories without an implied order
One-hot vectors separate category membership.
from pyspark.ml.feature import OneHotEncoder
encoder = OneHotEncoder(
inputCols=["contract_idx"],
outputCols=["contract_vec"]
)Cleaning changes which records train the model
Count exclusions and inspect types before assembly.
typed = raw.withColumn(
"TotalCharges", F.col("TotalCharges").cast("double")
)
usable = typed.filter(F.col("TotalCharges").isNotNull())
# Audit how many records this policy removes.The assembler fixes feature order
A vector must mean the same thing at training and prediction.
from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(
inputCols=["tenure", "MonthlyCharges", "contract_vec"],
outputCol="features"
)Scaling learns statistics from training
Centering sparse vectors can increase memory use.
from pyspark.ml.feature import StandardScaler
scaler = StandardScaler(
inputCol="features", outputCol="scaledFeatures",
withStd=True, withMean=False
)Convert text to a numeric representation
Tokenize the text. Create a numeric vector.
ML pipelines
How do we keep preparation and learning steps in the correct order?
A pipeline packages ordered stages
Fit the entire learned workflow on training rows.
train, valid, test = labeled_clean.randomSplit(
[0.6, 0.2, 0.2], seed=42
)
lr = LogisticRegression(labelCol="label")
pipeline = Pipeline(
stages=[indexer, encoder, assembler, lr])
model = pipeline.fit(train)
predictions = model.transform(valid)Training
What does logistic regression learn from the prepared training rows?
Logistic regression predicts a class probability
Learn coefficients rather than memorizing labels.
Prediction
What is the difference between a predicted class and its probability?
Prediction and probability answer different questions
A threshold changes labels without retraining the score model.
predictions.select(
"label", "rawPrediction", "probability", "prediction"
).show(5, truncate=False)Evaluation metrics
What does each metric tell us about the model's predictions?
ROC AUC measures ranking, not accuracy
BinaryClassificationEvaluator uses scores and known labels.
from pyspark.ml.evaluation import BinaryClassificationEvaluator
evaluator = BinaryClassificationEvaluator(
labelCol="label", rawPredictionCol="rawPrediction",
metricName="areaUnderROC"
)
roc_auc = evaluator.evaluate(predictions)
print("Validation ROC AUC:", roc_auc)Choose metrics for the decision costs
Rare positives can make accuracy misleading.
Precision asks whether alerts are correct
Precision measures correctness among predicted positives.
Recall measures the proportion of detected positives
Recall measures coverage among actual positives.
A PR curve traces precision and recall together
PR AUC summarises a curve. It does not measure accuracy at one threshold.
Model selection
How do we select model settings without using the final test set?
Validation chooses hyperparameters
Do not use test data to select settings.
from pyspark.ml.tuning import ParamGridBuilder
grid = ParamGridBuilder() \
.addGrid(lr.regParam, [0.0, 0.1]) \
.addGrid(lr.elasticNetParam, [0.0, 1.0]) \
.build()
# Four candidate settings; evaluate on validation only.Compare against a simple baseline
A more complex model must justify its additional cost.
A baseline provides a reference for model comparison
A trivial rule can reveal misleading accuracy.
Change the model within the same evaluation boundary
Additional complexity must justify its extra cost.
Choose with error costs and evidence
Selection requires a reason, not only a higher headline score.
Machine learning lab
What evidence makes the reported model result reproducible and valid?
Report the metric the lab actually computes
Check the metric in the unchanged lab code.
Valid evaluation is part of the pipeline
The result is useful only if the evaluation boundary is valid.
Hive and
distributed SQL
A table is a view over distributed data.
Hive gives shared files a SQL interface
A table definition connects names and types to stored data.
Hive architecture
How does the catalog connect SQL names to files and execution?
The catalog describes books and their locations
Keep metadata, execution and storage distinct.
The execution engine reads the records
The driver plans the query. Workers execute the distributed query.
EMR Hive normally uses Tez
Check the release and session settings. Do not assume that Hive uses Spark.
-- Read the actual Hive session setting.
SET hive.execution.engine;
-- Record EMR release and Hive version too.A metastore entry defines how to read stored data
Schema, format and location must agree with the files.
Column order and types interpret each field
A correct location cannot repair a wrong declared schema.
The delimiter must match the bytes
Format metadata tells the reader how to decode records.
A table location points to a dataset directory
Keep unrelated or incompatible files out of a table directory.
Schema on read
How does a table schema interpret data that is already stored?
Schema-on-read interprets existing data
Declared types do not correct malformed records.
Table lifecycle
What happens to the stored data when we remove a table?
Managed tables include a data lifecycle
Dropping a managed table normally removes its data and metadata.
External tables separate registration from ownership
Default DROP removes metadata. Separate controls must protect files from deletion.
Check deletion behavior before changing data
Table type alone is not the complete answer.
Stage the lab data
Where must the source files be before Hive can read them?
Stage files before declaring the table
Local download and HDFS storage are separate locations.
hadoop fs -mkdir -p /user/hive/data/movielens
hadoop fs -put ratings.tsv /user/hive/data/movielens/
hadoop fs -ls /user/hive/data/movielens/Table definitions
How do we describe the file format, columns and location in SQL?
Declare the TSV schema and location
One directory can contain several compatible part files.
CREATE EXTERNAL TABLE ratings (
user_id INT, movie_id INT,
rating INT, rating_ts BIGINT
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE
LOCATION '/user/hive/data/movielens/';Validate a table before aggregating
Inspect metadata, sampled values and row counts.
DESCRIBE FORMATTED ratings;
SELECT * FROM ratings LIMIT 5;
SELECT COUNT(*) FROM ratings;
SELECT MIN(rating), MAX(rating) FROM ratings;HiveQL
How do SQL queries turn the stored records into useful summaries?
Aggregation answers the source business question
Top rated by count is different from highest average rating.
SELECT movie_id, COUNT(*) AS rating_count
FROM ratings
GROUP BY movie_id
ORDER BY rating_count DESC, movie_id
LIMIT 10;WHERE filters rows. HAVING filters groups.
Apply thresholds after calculating aggregate values.
SELECT movie_id, COUNT(*) AS n, AVG(rating) AS avg_r
FROM ratings
WHERE rating BETWEEN 1 AND 5
GROUP BY movie_id
HAVING COUNT(*) >= 100 AND AVG(rating) > 4.0
ORDER BY avg_r DESC, movie_id
LIMIT 5;Choose the right filter location
Explain the intermediate relation, not only the syntax.
Joins connect identifiers to related data
Key uniqueness determines whether rows multiply.
SELECT m.title, COUNT(*) AS n
FROM ratings r
JOIN movies m ON r.movie_id = m.movie_id
GROUP BY m.title
ORDER BY n DESC
LIMIT 10;A left join can retain unmatched events
A WHERE filter on the right side may remove them again.
A left join starts by preserving left-side events
Every left row remains, including rows without a right match.
An unmatched lookup produces NULL right fields
NULL can describe a missing match rather than a missing left record.
A right-side WHERE filter can discard unmatched rows
SQL filtering can undo the preservation you expected from a left join.
Complex data
How can Hive expose fields inside nested JSON records?
STRUCT gives nested JSON named fields
The SerDe must match the file format and installed release.
CREATE EXTERNAL TABLE posts (
id BIGINT, text STRING,
author STRUCT<name:STRING, followers:INT>
)
ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION '/data/teaching/posts/';Dot notation selects a nested field
The result can remain a simple SQL relation.
SELECT text, author.name
FROM posts
WHERE author.followers > 1000000;Storage layout
How can directory partitions reduce the files a query must consider?
Partitioning makes directory exclusion possible
Partition keys describe the directory layout. They do not repeat in the row schema.
CREATE EXTERNAL TABLE sales (
order_id STRING, amount DOUBLE
)
PARTITIONED BY (dt STRING)
STORED AS PARQUET
LOCATION '/data/teaching/sales/';Folders need corresponding catalog partitions
Creating a partitioned table does not register every existing directory.
ALTER TABLE sales ADD IF NOT EXISTS
PARTITION (dt='2026-10-01')
LOCATION '/data/teaching/sales/dt=2026-10-01/';
SHOW PARTITIONS sales;Query planning
How do partition filters let the engine skip unrelated directories?
Pruning skips unrelated partition locations
Check the plan and measured bytes before you claim faster execution.
EXPLAIN
SELECT SUM(amount) FROM sales
WHERE dt = '2026-10-01';Match partition keys to the query pattern
A catalog and storage layout must work together.
Integration
How can several analytical tools use the same data and metadata?
Shared metadata enables several analytical tools
Sharing a catalog does not make every engine execute identically.
Hive lab preparation
Can we trace each lab query from SQL through metadata to the source files?
Follow the query from SQL to files
Keep screenshots and your reflection. Terminate the cluster.